Niftyreg regaverage

[byvernault]

Summary

Hi guys,

We realised that there is an issue with the RegAverage interface. After running the interface as a node in a workflow, the reg_average_cmd file and the output file does not exists. They are removed despite being used in the following nodes.

I also ran the interface as a node by itself and the file get created and stay there.

If we set the config tag remove_unnecessary_outputs = false, the outputs are present.

Do you know if something changed in the workflow core system that could explain this?

Actual behavior

170519-18:07:26,495 workflow INFO:
	 Workflow atlas_creation settings: ['check', 'execution', 'logging']
170519-18:07:26,528 workflow INFO:
	 Running in parallel.
170519-18:07:26,532 workflow INFO:
	 Executing: lin_reg ID: 0
170519-18:07:26,537 workflow INFO:
	 Adding 2 jobs for mapnode lin_reg
170519-18:07:26,542 workflow INFO:
	 Executing: _lin_reg0 ID: 5
170519-18:07:26,544 workflow INFO:
	 Executing: _lin_reg1 ID: 6
170519-18:07:26,550 workflow INFO:
	 Executing node _lin_reg0 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg0
170519-18:07:26,552 workflow INFO:
	 Executing node _lin_reg1 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg1
170519-18:07:26,571 workflow INFO:
170519-18:07:26,571 workflow INFO:
	 Running: reg_aladin -aff 20140701_08434709NXRZs006a1001_aff.txt -flo /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -omp 1 -ref /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -res 20140701_08434709NXRZs006a1001_res.nii.gz -rigOnly -voff
	 Running: reg_aladin -aff 20140701_08434709NXRZs006a1001_aff.txt -flo /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -omp 1 -ref /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -res 20140701_08434709NXRZs006a1001_res.nii.gz -rigOnly -voff
170519-18:09:42,504 workflow INFO:
	 [Job finished] jobname: _lin_reg1 jobid: 6
170519-18:09:42,544 workflow INFO:
	 [Job finished] jobname: _lin_reg0 jobid: 5
170519-18:09:42,547 workflow INFO:
	 Executing: lin_reg ID: 0
170519-18:09:42,553 workflow INFO:
	 Executing node lin_reg in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg
170519-18:09:42,569 workflow INFO:
	 Executing node _lin_reg0 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg0
170519-18:09:42,571 workflow INFO:
	 Collecting precomputed outputs
170519-18:09:42,582 workflow INFO:
	 Executing node _lin_reg1 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg1
170519-18:09:42,585 workflow INFO:
	 Collecting precomputed outputs
170519-18:09:42,621 workflow INFO:
	 [Job finished] jobname: lin_reg jobid: 0
170519-18:09:42,623 workflow INFO:
	 Executing: ave_ims ID: 1
170519-18:09:42,638 workflow INFO:
	 Executing node ave_ims in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/ave_ims
170519-18:09:42,651 workflow INFO:
	 Running: reg_average --cmd_file /Users/byvernault/test_grp/atlas_creation/lin_reg0/ave_ims/reg_average_cmd
170519-18:09:51,575 workflow INFO:
	 [Job finished] jobname: ave_ims jobid: 1
170519-18:09:51,578 workflow INFO:
	 Executing: lin_reg ID: 2
170519-18:09:51,588 workflow INFO:
	 Adding 2 jobs for mapnode lin_reg
170519-18:09:51,592 workflow INFO:
	 Executing: _lin_reg0 ID: 7
170519-18:09:51,594 workflow INFO:
	 Executing: _lin_reg1 ID: 8
170519-18:09:51,596 workflow INFO:
	 Executing node _lin_reg0 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg0
170519-18:09:51,598 workflow INFO:
	 Executing node _lin_reg1 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg1
170519-18:09:51,668 workflow ERROR:
	 [u'Node _lin_reg0 failed to run on host cmic-ben-laptop.']
170519-18:09:51,669 workflow INFO:
	 Saving crash info to /Users/byvernault/crash-20170519-180951-byvernault-_lin_reg0-7e88afb5-0196-490a-9718-7dab270b0ec4.pklz
170519-18:09:51,669 workflow INFO:
	 Traceback (most recent call last):
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/plugins/multiproc.py", line 52, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/engine/nodes.py", line 372, in run
    self._run_interface()
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/engine/nodes.py", line 482, in _run_interface
    self._result = self._run_command(execute)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/engine/nodes.py", line 603, in _run_command
    cmd = self._interface.cmdline
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/interfaces/base.py", line 1673, in cmdline
    self._check_mandatory_inputs()
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/interfaces/base.py", line 969, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: RegAladin requires a value for input 'ref_file'. For a list of required inputs, see RegAladin.help()

170519-18:09:51,683 workflow ERROR:
	 [u'Node _lin_reg1 failed to run on host cmic-ben-laptop.']
170519-18:09:51,684 workflow INFO:
	 Saving crash info to /Users/byvernault/crash-20170519-180951-byvernault-_lin_reg1-f5b11ccf-b0a3-4f0c-bfbc-c86dca97ff0f.pklz
170519-18:09:51,684 workflow INFO:
	 Traceback (most recent call last):
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/plugins/multiproc.py", line 52, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/engine/nodes.py", line 372, in run
    self._run_interface()
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/engine/nodes.py", line 482, in _run_interface
    self._result = self._run_command(execute)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/engine/nodes.py", line 603, in _run_command
    cmd = self._interface.cmdline
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/interfaces/base.py", line 1673, in cmdline
    self._check_mandatory_inputs()
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/interfaces/base.py", line 969, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: RegAladin requires a value for input 'ref_file'. For a list of required inputs, see RegAladin.help()

170519-18:09:51,699 workflow INFO:
	 ***********************************
170519-18:09:51,699 workflow ERROR:
	 could not run node: atlas_creation.lin_reg1.lin_reg
170519-18:09:51,699 workflow INFO:
	 crashfile: /Users/byvernault/crash-20170519-180951-byvernault-_lin_reg0-7e88afb5-0196-490a-9718-7dab270b0ec4.pklz
170519-18:09:51,699 workflow ERROR:
	 could not run node: atlas_creation.lin_reg1.lin_reg
170519-18:09:51,700 workflow INFO:
	 crashfile: /Users/byvernault/crash-20170519-180951-byvernault-_lin_reg1-f5b11ccf-b0a3-4f0c-bfbc-c86dca97ff0f.pklz
170519-18:09:51,700 workflow INFO:
	 ***********************************
Traceback (most recent call last):
  File "/Users/byvernault/home-local/code/anaconda/bin/perform_groupwise_registration.py", line 4, in <module>
    __import__('pkg_resources').run_script('niftypipe==2.0.0', u'perform_groupwise_registration.py')
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/pkg_resources/__init__.py", line 738, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1499, in run_script
    exec(code, namespace, namespace)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/niftypipe-2.0.0-py2.7.egg/EGG-INFO/scripts/perform_groupwise_registration.py", line 214, in <module>
    run_workflow(workflow=wf, qsubargs=qsubargs, parser=args)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/niftypipe-2.0.0-py2.7.egg/niftypipe/interfaces/base.py", line 221, in run_workflow
    workflow.run(plugin=plugin, plugin_args=pargs)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/engine/workflows.py", line 590, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/plugins/base.py", line 279, in run
    report_nodes_not_run(notrun)
  File "/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0-py2.7.egg/nipype/pipeline/plugins/base.py", line 101, in report_nodes_not_run
    raise RuntimeError(('Workflow did not execute cleanly. '
RuntimeError: Workflow did not execute cleanly. Check log for details

Expected behavior

170519-17:52:04,587 workflow INFO:
	 Workflow atlas_creation settings: ['check', 'execution', 'logging']
170519-17:52:04,660 workflow INFO:
	 Running in parallel.
170519-17:52:04,664 workflow INFO:
	 Executing: lin_reg ID: 0
170519-17:52:04,669 workflow INFO:
	 Adding 2 jobs for mapnode lin_reg
170519-17:52:04,673 workflow INFO:
	 Executing: _lin_reg0 ID: 5
170519-17:52:04,676 workflow INFO:
	 Executing: _lin_reg1 ID: 6
170519-17:52:04,682 workflow INFO:
	 Executing node _lin_reg0 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg0
170519-17:52:04,686 workflow INFO:
	 Executing node _lin_reg1 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg1
170519-17:52:04,705 workflow INFO:
	 Running: reg_aladin -aff /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg0/20140701_08434709NXRZs006a1001_aff.txt -flo /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -ref /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -res /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg0/20140701_08434709NXRZs006a1001_res.nii.gz -rigOnly -voff
170519-17:52:04,705 workflow INFO:
	 Running: reg_aladin -aff /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg1/20140701_08434709NXRZs006a1001_aff.txt -flo /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -ref /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -res /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg1/20140701_08434709NXRZs006a1001_res.nii.gz -rigOnly -voff
170519-17:54:26,563 workflow INFO:
	 [Job finished] jobname: _lin_reg0 jobid: 5
170519-17:54:27,273 workflow INFO:
	 [Job finished] jobname: _lin_reg1 jobid: 6
170519-17:54:27,275 workflow INFO:
	 Executing: lin_reg ID: 0
170519-17:54:27,283 workflow INFO:
	 Executing node lin_reg in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg
170519-17:54:27,299 workflow INFO:
	 Executing node _lin_reg0 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg0
170519-17:54:27,300 workflow INFO:
	 Collecting precomputed outputs
170519-17:54:27,310 workflow INFO:
	 Executing node _lin_reg1 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/lin_reg/mapflow/_lin_reg1
170519-17:54:27,311 workflow INFO:
	 Collecting precomputed outputs
170519-17:54:27,347 workflow INFO:
	 [Job finished] jobname: lin_reg jobid: 0
170519-17:54:27,349 workflow INFO:
	 Executing: ave_ims ID: 1
170519-17:54:27,363 workflow INFO:
	 Executing node ave_ims in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg0/ave_ims
170519-17:54:27,375 workflow INFO:
	 Running: reg_average --cmd_file /Users/byvernault/test_grp/atlas_creation/lin_reg0/ave_ims/reg_average_cmd
170519-17:54:36,405 workflow INFO:
	 [Job finished] jobname: ave_ims jobid: 1
170519-17:54:36,407 workflow INFO:
	 Executing: lin_reg ID: 2
170519-17:54:36,417 workflow INFO:
	 Adding 2 jobs for mapnode lin_reg
170519-17:54:36,422 workflow INFO:
	 Executing: _lin_reg0 ID: 7
170519-17:54:36,424 workflow INFO:
	 Executing: _lin_reg1 ID: 8
170519-17:54:36,427 workflow INFO:
	 Executing node _lin_reg0 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg0
170519-17:54:36,429 workflow INFO:
	 Executing node _lin_reg1 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg1
170519-17:54:36,444 workflow INFO:
	 Running: reg_aladin -aff /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg0/20140701_08434709NXRZs006a1001_aff.txt -flo /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -ref /Users/byvernault/test_grp/atlas_creation/lin_reg0/ave_ims/avg_out.nii.gz -res /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg0/20140701_08434709NXRZs006a1001_res.nii.gz -rigOnly -voff
170519-17:54:36,445 workflow INFO:
	 Running: reg_aladin -aff /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg1/20140701_08434709NXRZs006a1001_aff.txt -flo /Users/byvernault/Downloads/20140701_08434709NXRZs006a1001.nii.gz -ref /Users/byvernault/test_grp/atlas_creation/lin_reg0/ave_ims/avg_out.nii.gz -res /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg1/20140701_08434709NXRZs006a1001_res.nii.gz -rigOnly -voff
170519-17:56:56,83 workflow INFO:
	 [Job finished] jobname: _lin_reg0 jobid: 7
170519-17:56:56,84 workflow INFO:
	 [Job finished] jobname: _lin_reg1 jobid: 8
170519-17:56:56,86 workflow INFO:
	 Executing: lin_reg ID: 2
170519-17:56:56,92 workflow INFO:
	 Executing node lin_reg in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg
170519-17:56:56,106 workflow INFO:
	 Executing node _lin_reg0 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg0
170519-17:56:56,107 workflow INFO:
	 Collecting precomputed outputs
170519-17:56:56,115 workflow INFO:
	 Executing node _lin_reg1 in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg1/lin_reg/mapflow/_lin_reg1
170519-17:56:56,116 workflow INFO:
	 Collecting precomputed outputs
170519-17:56:56,159 workflow INFO:
	 [Job finished] jobname: lin_reg jobid: 2
170519-17:56:56,162 workflow INFO:
	 Executing: ave_ims ID: 3
170519-17:56:56,173 workflow INFO:
	 Executing node ave_ims in dir: /Users/byvernault/test_grp/atlas_creation/lin_reg1/ave_ims
170519-17:56:56,184 workflow INFO:
	 Running: reg_average --cmd_file /Users/byvernault/test_grp/atlas_creation/lin_reg1/ave_ims/reg_average_cmd
170519-17:57:04,899 workflow INFO:
	 [Job finished] jobname: ave_ims jobid: 3
170519-17:57:04,902 workflow INFO:
	 Executing: datasink ID: 4
170519-17:57:04,919 workflow INFO:
	 Executing node datasink in dir: /Users/byvernault/test_grp/atlas_creation/datasink
170519-17:57:05,148 workflow INFO:
	 [Job finished] jobname: datasink jobid: 4

How to replicate the behavior

Run the groupwise workflow in niftyreg like:

from nipype.workflows.smri.niftyreg import create_groupwise_average

wf = create_groupwise_average(name="atlas_creation",
                              itr_rigid=2,
                              itr_affine=0,
                              itr_non_lin=0)
wf.base_dir = result_dir
wf.inputs.inputspec.in_files = in_files

with in_files a list of two files (it can be the same file).

Script/Workflow details

{'nibabel_version': '2.1.0', 'sys_executable': '/Users/byvernault/home-local/code/anaconda/bin/python', 'networkx_version': '1.11', 'numpy_version': '1.12.1', 'sys_platform': 'darwin', 'sys_version': '2.7.12 |Anaconda 4.2.0 (x86_64)| (default, Jul 2 2016, 17:43:17) \n[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)]', 'commit_source': u'installation', 'commit_hash': u'bfdb164f2', 'pkg_path': '/Users/byvernault/home-local/code/anaconda/lib/python2.7/site-packages/nipype-0.13.0_gbfdb164f2.dev-py2.7.egg/nipype', 'nipype_version': u'0.13.0-dev', 'traits_version': '4.6.0', 'scipy_version': '0.19.0'}
0.13.0-dev

Execution environment

My python environment outside container

[byvernault]

It looks like for niftyreg RegAverage, the out_file is not set:

Execution Outputs

• out_file :

I will dig more to see why it's the case.
Cheers,

[mgxd]

@byvernault looks like the interface is missing _list_outputs - also, if you are using the reg_average_cmd file, you may want to add it to the OutputSpec

[byvernault]

OH, I removed it from base but I didn't put it back to this interface.

Good catch.

The reg_average_cmd is used internally. It's why I didn't put in the InputSpec.

We had some issues on the length of a reg_average command line where the OS was complaining. We decided to put the command line in a file and do the parsing inside the executable reg_average. For nipype, the user doesn't need to use it. He just specify the inputs like he will with reg_average and the interface will generate the cmd_file.

Thx again @mgxd

[satra]

closed via #2037