Problems with PBS plugin

[achetverikov]

Summary

PBS plugin doesn't seem to work for me on our Torque cluster. It waits forever for job completion. In addition, there are small errors in pbs.py and in base.py but fixing them does not help.

Actual behavior

The current nipype installed with conda raises an error 'e' is not defined at line 58 of pbs.py. The variable e (from this commit: 847879a ) seem to be not defined indeed. Changing it to 'stderr' on lines 58 and 61 resolves this error. Then lines 280-281 of base.py:
logger.info('Pending[%d] Submitting[%d] jobs Slots[%d]',
num_jobs, len(jobids[:slots]), slots or 'inf')
throw a logging error because 'inf' in slots is not valid for %d. I changed %d to %s to fix it.

But then nothing happens after the job is completed. For example, the log says:

171215-17:15:00,578 workflow INFO:
Workflow preprocessing settings: ['check', 'execution', 'logging', 'monitoring']
171215-17:15:00,642 workflow INFO:
Running in parallel.
171215-17:15:00,649 workflow INFO:
Pending[0] Submitting[1] jobs Slots[inf]
171215-17:15:00,654 workflow INFO:
Submitting: grabber ID: 0
171215-17:15:01,9 workflow INFO:
Finished submitting: grabber ID: 0

After that, nothing happens. Torque log shows that job was completed without errors and there's a pklz file in the node folder. The same job runs without problems without cluster.

Expected behavior

The results of submitted job should be processed.

How to replicate the behavior

Run any job with PBS using default settings.

Platform details:

{'pkg_path': '/home/visual/andche/.conda/envs/demo/lib/python3.6/site-packages/nipype', 'commit_source': 'archive substitution', 'commit_hash': '%h', 'nipype_version': '0.14.0', 'sys_version': '3.6.3 | packaged by conda-forge | (default, Dec 9 2017, 16:18:26) \n[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]', 'sys_executable': '/home/visual/andche/.conda/envs/demo/bin/python', 'sys_platform': 'linux', 'numpy_version': '1.13.3', 'scipy_version': '1.0.0', 'networkx_version': '2.0', 'nibabel_version': '2.2.1', 'traits_version': '4.6.0'}
0.14.0

[achetverikov]

Alright, so looks like the problem is that stderr output from qstat is empty on our cluster if the job is completed successfully. And because the job output is kept for sometime after the job is completed, qstat by itself also does not throw any errors. So here's the fix that I used:

def _is_pending(self, taskid):
        result = CommandLine('qstat -f {}'.format(taskid),
                             environ=dict(os.environ),
                             terminal_output='allatonce',
                             resource_monitor=False,
                             ignore_exception=True).run()

        stdout = result.runtime.stdout
        stderr = result.runtime.stderr
        errmsg = 'Unknown Job Id' 
        success = 'Job has finished'
        if (success in stderr) or ('job_state = C' in stdout):  
            return False
        else:
            return errmsg not in stderr`

qstat is run with an '-f' parameter to get easier to parse the output, then looking for 'job_state = C' in it. This seems to be working.

[mgxd]

@achetverikov nice investigating, would you like to submit a PR for this? It's always nice to make these plugins more flexible across different HPC configurations

[achetverikov]

@mgxd done