Incorrect extension identified when checking the ``extensions`` metadata of ``File`` traits

[servoz]

I observe the following message when defining the input parameter in_files (for example for the nipype.interfaces.spm.preprocess.smooth class / process) :

<class 'traits.trait_errors.TraitError'>: Each element of the 'in_files' trait of a Pipeline instance must be a pathlike object or string representing an existing file, but a value of '/Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii' <class 'str'> was specified.

The result is that the in_files remains at <undefined>.

It seems that the problem concerns the nipype.interfaces.spm.base.ImageFileSPM class and the nipype.interfaces.base.traits_extension.ImageFile class.

The problem was not observed until June 25 (I do not observe the problem with the nipype repository corresponding to commit e722d61) and has been observed since July 16 (observed at least since the nipype repository corresponding to commit b76860d).

[oesteban]

Does that file exist / what is the output of:

python -c "import os; print(os.path.exists('/Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii'))"

It seems to work for me:

$ touch file.nii
$ ipython
>>> from nipype.interfaces.spm.preprocess import SmoothInputSpec
>>> spec = SmoothInputSpec()
>>> spec.in_files = 'file.nii'
>>> spec
data_type = <undefined>
fwhm = <undefined>
implicit_masking = <undefined>
in_files = ['file.nii']
matlab_cmd = <undefined>
mfile = True
out_prefix = s
paths = <undefined>
use_mcr = <undefined>
use_v8struct = True
>>> from nipype import __version__
>>> __version__
'1.2.1-dev+g865adc915'

[servoz]

Yes the file exists:

>>> import os
>>> print(os.path.exists('/Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii'))
True

But:

>>> from nipype.interfaces.spm.preprocess import SmoothInputSpec
>>> spec = SmoothInputSpec()
>>> spec.in_files='/Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/nipype-1.2.1.dev0-py3.7.egg/nipype/interfaces/base/traits_extension.py", line 394, in validate
    value = super(MultiObject, self).validate(objekt, name, newvalue)
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_types.py", line 2573, in validate
    return TraitListObject(self, object, name, value)
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_handlers.py", line 2474, in __init__
    raise excp
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_handlers.py", line 2466, in __init__
    value = [validate(object, name, val) for val in value]
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_handlers.py", line 2466, in <listcomp>
    value = [validate(object, name, val) for val in value]
  File "/usr/local/lib/python3.7/site-packages/nipype-1.2.1.dev0-py3.7.egg/nipype/interfaces/base/traits_extension.py", line 315, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_handlers.py", line 236, in error
    object, name, self.full_info(object, name, value), value
traits.trait_errors.TraitError: Each element of the 'in_files' trait of a SmoothInputSpec instance must be a pathlike object or string representing an existing file, but a value of '/Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii' <class 'str'> was specified.

EDIT 1:
And on my side:

$ touch file.nii
$ python3
Python 3.7.3 (default, Jun 12 2019, 16:12:16) 
[Clang 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from nipype.interfaces.spm.preprocess import SmoothInputSpec
>>> spec = SmoothInputSpec()
>>> spec.in_files = 'file.nii'
>>> spec

data_type = <undefined>
fwhm = <undefined>
implicit_masking = <undefined>
in_files = ['file.nii']
matlab_cmd = <undefined>
mfile = True
out_prefix = s
paths = <undefined>
use_mcr = <undefined>
use_v8struct = True

I am sure that the alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii file was working fine before, because I always use this file when I test our codes. I can read and see what was changed in the nipype codes around middle of July but may be you will see faster than me what can induce the exception with your new codes ?

[oesteban]

Let's check three more options:

>>> from nipype.utils.filemanip import Path
>>> Path('/Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii').exists()

then:

>>> from nipype.interfaces.spm.preprocess import SmoothInputSpec
>>> from nipype.utils.filemanip import Path
>>> spec = SmoothInputSpec()
>>> spec.in_files = Path('/Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii')
>>> spec

and finally:

$ cd /Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/
$ ipython
>>> from nipype.interfaces.spm.preprocess import SmoothInputSpec
>>> spec = SmoothInputSpec()
>>> spec.in_files = 'alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii'
>>> spec

[oesteban]

Okay, I believe I found it:

$ touch 000.file.nii

In [1]: >>> from nipype.interfaces.spm.preprocess import SmoothInputSpec 
   ...: >>> spec = SmoothInputSpec() 
   ...: >>> spec.in_files = '000.file.nii' 
   ...: >>> spec                                                                                                                                                                                                                                                                                                                                                                                                   
/home/oesteban/.anaconda3/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
190805-10:24:08,775 nipype.interface DEBUG:
	 args_matlab
---------------------------------------------------------------------------
TraitError                                Traceback (most recent call last)
<ipython-input-1-dc262b2c1707> in <module>
      1 from nipype.interfaces.spm.preprocess import SmoothInputSpec
      2 spec = SmoothInputSpec()
----> 3 spec.in_files = '000.file.nii'
      4 spec

~/workspace/nipype/nipype/interfaces/base/traits_extension.py in validate(self, objekt, name, value)
    374                 not isinstance(value[0], list)):
    375             newvalue = [value]
--> 376         value = super(MultiObject, self).validate(objekt, name, newvalue)
    377 
    378         if value:

~/.anaconda3/lib/python3.7/site-packages/traits/trait_types.py in validate(self, object, name, value)
   2334                 return value
   2335 
-> 2336             return TraitListObject( self, object, name, value )
   2337 
   2338         self.error( object, name, value )

~/.anaconda3/lib/python3.7/site-packages/traits/trait_handlers.py in __init__(self, trait, object, name, value)
   2311             except TraitError as excp:
   2312                 excp.set_prefix( 'Each element of the' )
-> 2313                 raise excp
   2314 
   2315         self.len_error( len( value ) )

~/.anaconda3/lib/python3.7/site-packages/traits/trait_handlers.py in __init__(self, trait, object, name, value)
   2303                 validate = trait.item_trait.handler.validate
   2304                 if validate is not None:
-> 2305                     value = [ validate( object, name, val ) for val in value ]
   2306 
   2307                 list.__setitem__(self, slice(0, 0), value )

~/.anaconda3/lib/python3.7/site-packages/traits/trait_handlers.py in <listcomp>(.0)
   2303                 validate = trait.item_trait.handler.validate
   2304                 if validate is not None:
-> 2305                     value = [ validate( object, name, val ) for val in value ]
   2306 
   2307                 list.__setitem__(self, slice(0, 0), value )

~/workspace/nipype/nipype/interfaces/base/traits_extension.py in validate(self, objekt, name, value, return_pathlike)
    295             ext = ''.join(value.suffixes)
    296             if ext not in self._exts:
--> 297                 self.error(objekt, name, str(value))
    298 
    299         if not return_pathlike:

~/.anaconda3/lib/python3.7/site-packages/traits/trait_handlers.py in error(self, object, name, value)
    170         """
    171         raise TraitError( object, name, self.full_info( object, name, value ),
--> 172                           value )
    173 
    174     def full_info ( self, object, name, value ):

TraitError: Each element of the 'in_files' trait of a SmoothInputSpec instance must be a pathlike object or string representing an existing file, but a value of '000.file.nii' <class 'str'> was specified.

Regression introduced in #2962.

[servoz]

I'm afraid the nipype.utils.filemanip module won't include the Path class on the version I have in my station.

$ python3
Python 3.7.3 (default, Jun 12 2019, 16:12:16) 
[Clang 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from nipype.utils.filemanip import Path
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Path' from 'nipype.utils.filemanip' (/usr/local/lib/python3.7/site-packages/nipype-1.2.1.dev0-py3.7.egg/nipype/utils/filemanip.py)

For the last point:

$ cd /Users/invites/Documents/data/projects_populse_mia/test0/data/raw_data/
$ python3
Python 3.7.3 (default, Jun 12 2019, 16:12:16) 
[Clang 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getcwd()
'/Volumes/Users/invites/data/projects_populse_mia/test0/data/raw_data'
>>> os.listdir()
['alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-04-01-BOLD_CVR_7_53sl_ModeratePNS_SENSE-FEEPI-00-12-12.000.json', 'alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-04-01-BOLD_CVR_7_53sl_ModeratePNS_SENSE-FEEPI-00-12-12.000.nii', 'alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.json', 'alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii', 'logExport_2019-06-20_14-22-28', 'logExport_2019-06-20_14-22-28.json']
>>> from nipype.interfaces.spm.preprocess import SmoothInputSpec
>>> spec = SmoothInputSpec()
>>> spec.in_files = 'alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/nipype-1.2.1.dev0-py3.7.egg/nipype/interfaces/base/traits_extension.py", line 394, in validate
    value = super(MultiObject, self).validate(objekt, name, newvalue)
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_types.py", line 2573, in validate
    return TraitListObject(self, object, name, value)
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_handlers.py", line 2474, in __init__
    raise excp
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_handlers.py", line 2466, in __init__
    value = [validate(object, name, val) for val in value]
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_handlers.py", line 2466, in <listcomp>
    value = [validate(object, name, val) for val in value]
  File "/usr/local/lib/python3.7/site-packages/nipype-1.2.1.dev0-py3.7.egg/nipype/interfaces/base/traits_extension.py", line 315, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/lib/python3.7/site-packages/traits-5.1.1-py3.7-macosx-10.11-x86_64.egg/traits/trait_handlers.py", line 236, in error
    object, name, self.full_info(object, name, value), value
traits.trait_errors.TraitError: Each element of the 'in_files' trait of a SmoothInputSpec instance must be a pathlike object or string representing an existing file, but a value of 'alej170316_test24042018-IRM_Fonct._+_perfusion-2016-03-17_08-34-44-05-01-T1_3D_SENSE-T1TFE-00-04-25.000.nii' <class 'str'> was specified.

[oesteban]

I might have not imported it from the right module. Anyways, the problem is that your file ends with several sequences of characters separated by dots, which are interpreted as part of the extension.

EDIT: working on a fix right now

[servoz]

Oh I see !
Thanks for your very fast support !

[oesteban]

Can you check whether this fix is working for you?

$ python -m pip install -U git+https://github.com/oesteban/nipype.git@be28002e24e63c63df531d22f90524c39373dd29

And then any of the failing tests above.