-
Notifications
You must be signed in to change notification settings - Fork 1
Parallel Debuggers tools and Sessions process model considerations
- A debugger/tool can query for proctable information using the PMIx group name associated with the session - i.e., to get the proctable for all processes in the session, you would query for proctable passing the group name in place of the namespace and PMIX_RANK_WILDCARD for the rank. If you want the proctable info for a single process in the group, then you would use the group name for the namespace and the group rank of the proc.
This sounds good, but it might be that what the application programmer thinks is the PMIx group name may actually not be the name being used by an MPI implementation internally. Could add info to the debug info table that could assist the debugger in determining a set of processes to attach to. A debugger can currently query for group names associated with a PMIx session. To keep a simple relationship between an MPI application's session and the PMIx Group name, we would have to add some restrictions in terms of stringtag argument to MPI_Comm_create_from_group uniqueness.
- If you are spawning processes to form a group, then you can use the existing "stop at exec", "stop in PMIx init", and "stop in app" (usually used to stop in MPI_Init, though it can be used to stop anywhere in the app) to pause the new processes until debugger attach/release. I suspect you might also want to pause the parent process doing the spawn so the debugger can attach to it as well before releasing the new processes. We don't have anything to help here, so I propose to add a new "stop in spawn" attribute. When passed to PMIx_Spawn, it will hold the parent process just prior to returning from the blocking form of PMIx_Spawn, or just prior to executing the callback function for the non-blocking form of PMIx_Spawn.
Attaching at PMIx_Init would probably be simplest but could possibly change semantics of MPI_Session_init from being a kind of local operation to a synchronizing, depending on the behavior of the debugger. This option would be transparent to the application. The stop_at_exec may be better in that the application is paused before it hits main. DDT supports both of these options. stop_in_app is flexible but could have implications similar to stop_in_PMIx_Init in terms of adding synchronizing behavior to MPI_Session_init.
This leaves the question of how to handle construction of MPI sessions across existing processes. For this purpose, I propose to:
a. Add a "stop in group construct" attribute that would pause just before returning from the blocking form of group construct, or just before executing the callback function for the non-blocking form. In the case of async group formation (using PMIx_Group_invite/join), the pause would occur just before delivering the "group complete" event to each participant. Debugger notification occurs when the PMIx library indicates that all participants have reached the pause point.
Some application developers they want to have the parallel debugger to stop when a set of processes construct a PMIx group. Currently the thought is the user of the debugger will be able to supply these PMIx group names and can supply them to the tool. An alternative would be have the debugger query for PMIx groups and somehow figure out which ones to attach to.
b. Utilize the "stop in app" support by passing that attribute to the PMIx group construct operation. This would generate an event notifying a debugger that the app has paused during MPI "sessions init" or some other designated location after completing group construction. Remember, "stop in app" takes a string "tag" so the MPI library can not only look for the attribute, but also check the "tag" to know where to stop. The attribute/tag is included in the debugger notification as well as returned to each group member in the "group complete" info array. Debugger notification takes place once all processes indicate that they have reached the assigned location (mechanism identical to "stop in app" when spawning processes).
this approach may be usable with MPI_Comm_create_from_group and would not involve changes to PMIx. Somewhat like a specialized version of the debugger support in MPI_Init now. For the general case of non-blocking group construct or the invite/join model more work is need at the PMIx level.
Note all these considerations are dependent on debuggers making use of the PMIx tools interface.
Implementation of this support is pretty simple and can be made available in PMIx v5, due out this spring. Please feel free to provide comments/suggestions - none of this is in concrete!