sgegraph only submits the next processing node

[bloyl]

Summary

the SGEGraph plugin isn't submitting the entire workflow only the next required step.

Actual behavior

When calling workflow.run(plugin='SGEGraph', plugin_args = {'dont_resubmit_completed_jobs': True}) only the next processing node is submitted. subsequent calls are needed to iteratively processes subsequent nodes.

workflow.run(plugin='SGE) works as expected. submitting one job waiting until completion then submitting the next.

Expected behavior

I would expect that workflow.run(plugin='SGEGraph', plugin_args = {'dont_resubmit_completed_jobs': True}) would submit all processing jobs at once with later jobs being held until the earlier required jobs are finished.

How to replicate the behavior

I'm new to nipype so I'm not sure how to get sample/example data to provide a minimal example but I think that any of the workflows in the example folder would demonstrate this issue.

Platform details:

200207-13:43:08,275 nipype.utils INFO:
	 No new version available.
{'commit_hash': '792988f19',
 'commit_source': 'installation',
 'networkx_version': '2.4',
 'nibabel_version': '3.0.1',
 'nipype_version': '1.4.1',
 'numpy_version': '1.18.1',
 'pkg_path': '/mnt/isilon/meg_lab/Linux_software/anaconda3_lab/envs/nipype/lib/python3.8/site-packages/nipype',
 'scipy_version': '1.3.2',
 'sys_executable': '/mnt/isilon/meg_lab/Linux_software/anaconda3_lab/envs/nipype/bin/python',
 'sys_platform': 'linux',
 'sys_version': '3.8.1 (default, Jan  8 2020, 22:29:32) \n[GCC 7.3.0]',
 'traits_version': '5.2.0'}

Execution environment

Choose one

• My python environment outside container

[bloyl]

The problem was cause by a configuration on cluster. for security reasons LD_LIBRARY_PATH environment variables are stripped from the qsub environment. qsub prints a warning when this happens:

removed environment variable LD_LIBRARY_PATH from submit environment - it is considered a security issue
removed environment variable LD_LIBRARY_PATH_modshare from submit environment - it is considered a security issue
Your job 13917966 ("j0_dcm_dir_a0") has been submitted

and this breaks the awk syntax

nipype/nipype/pipeline/plugins/sgegraph.py

Lines 151 to 158 in b62d806

	full_line = "{jobNm}=$(qsub {outFileOption} {errFileOption} {extraQSubArgs} {dependantIndex} -N {jobNm} {batchscript} | awk '{{print $3}}')\n".format(
	jobNm=jobname,
	outFileOption=stdoutFile,
	errFileOption=stderrFile,
	extraQSubArgs=qsub_args,
	dependantIndex=deps,
	batchscript=batchscriptfile,
	)

so job ids aren't parsed correctly and the submit script crashes without a log message.

I'll hopefully push a PR in a few minutes that modifies the awk command.

[effigies]

Assuming this previously worked, then it may be that this is a change in SGE and the appropriate response is to condition on the version.

Do you have version information? We can at least start to keep track of which versions need what parsing approaches.

[bloyl]

I think that this is a fairly old gridengine issue: here is a discussion of the problem from 2012: http://gridengine.org/pipermail/users/2012-April/003271.html

So I imagine that it isn't a version issue so much as an interaction between how my HPC manages software, we use modules (which relies on manipulating LD_LIBRARY_PATH) and nipype passing the whole env via qsub.

I'll try to get some system documentation qsub sanitizing the environment variables an pass it along.

That being said take a look at the proposed fix in #3168 its pretty innocuous and should be fully backward compatible. Here is a similar function the had to deal with the same issue. https://github.com/RabadanLab/Pandora/blob/master/helpers/helpers.py#L178

[effigies]

Oh, I think I misunderstood. I thought you were stripping the initial "Your job", not matching, so I interpreted this as a change in output that was being accommodated.