MultiProc Plugin is double-submitting jobs

[mwaskom]

This problem is summarized in this thread: https://groups.google.com/forum/#!topic/nipy-user/CPjld7rIsMk

Basically, running a workflow with MultiProc and asking for a specific n_proc runs (2 * n_proc) processes, with each job duplicated. I do not see similar behavior with the Linear or IPython plugins, and the behavior is occurring (for me) on an older CPU that does not do any hyperthreading fanciness. The double computation is definitely happening, as asking for a MultiProc execution with as many procs as I have cores ground my computer to a complete halt.

I'll try to put together a test workflow to produce this behavior.

[chrisgorgo]

Can you replicate this bug on one of our example workflows?

[mwaskom]

Well, this makes no sense.

I am fairly certain that I have narrowed down the offending commit to this one. Every commit including and after it shows the bug, and every commit I've tried before it is fine.

If you haven't clicked on the link to the commit, do so now.

So, that's why I'm confused.

@satra, do you have any memory of this commit? Can you think of what might be going on?

Sorry I have been unable to concoct a dummy workflow to show the problem, although @toddt has replicated the issue with his setup on Mindhive.

[mwaskom]

Also has anyone successfully used the IPython plugin recently? It is totally crashing for me.

[satra]

see if this happens when you remove these lines from multiproc.py:

 +        try:
 +            if node.inputs.terminal_output == 'stream':
 +                node.inputs.terminal_output = 'file'
 +        except:
 +            pass

[mwaskom]

Huh. That does seem to fix it.

[chrisgorgo]

I can confirm it as well.

[mwaskom]

Current status: befuddled.

[chrisgorgo]

Surely you meant perplexed.

[satra]

the only piece that code does is to pass file handles to the subprocess call.

https://github.com/nipy/nipype/blob/master/nipype/interfaces/base.py#L1154

hence my real question whether these are just shared processes (check with pstree) or actually distinct processes.

[mwaskom]

I don't have pstree on my mac but they are both R+ processes in ps with
unique pids. The other evidence that they are distinct is that I can grind
my computer to a halt when asking for 75% of the available resources.

On Sat, Nov 2, 2013 at 1:54 PM, Satrajit Ghosh [email protected]:

the only piece that code does is to pass file handles to the subprocess
call.

https://github.com/nipy/nipype/blob/master/nipype/interfaces/base.py#L1154

hence my real question whether these are just shared processes (check with
pstree) or actually distinct processes.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/705#issuecomment-27632314
.

[satra]

so very likely the process is being forked, which results in memory duplication, which in turn is bringing your computer to a halt. replacing with alternate output mode.

[mwaskom]

yep, it's almost certainly swapping; my hard drive was making constant
noises at the time i put the box out of its misery (it was completely non
responsive so I couldn't really otherwise debug it)

On Sat, Nov 2, 2013 at 2:13 PM, Satrajit Ghosh [email protected]:

so very likely the process is being forked, which results in memory
duplication, which in turn is bringing your computer to a halt. replacing
with alternate output mode.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/705#issuecomment-27632649
.