Problem loading large CIFTI2 files in lastest updates

[demianw]

Hi!

it seems that one of the post-"Merge pull request #249" updates to the codebase is forcing the load function of cifti2 to bring into memory the whole loaded matrix. For large files (I'm working on files of ~10Gb) this forces a "cannot allocate memory" exception.

I don't know how is it possible to add such large files to the test-base such that we can have a control for this. In any case it will be great if the error could be fixed.

[matthew-brett]

Could you say more about why you think this is post-merge of #249 ? For example, do you get the same problem when you run from commit 6f58e02 ?

[demianw]

I know it's post-merge cause when I run it from the corresponding merge commit everything works. I will try and run from that commit later. Is there a way to establish tests for these cases?

[matthew-brett]

How are you detecting the memory usage at the moment? Maybe we could use something similar for a test?

[demianw]

Because I get the exception OSError: [Errno 12] Cannot allocate memory upon trying the nifti.cifti2.load(f).

I think that the key problem for the test is to have a large enough file on the test battery.

[demianw]

OK, it does work from 6f58e02 too. I tried tracing back the error but it seems to be deep into the memorymap and lazy loading data mechanisms

[matthew-brett]

Just to confirm - you mean there is no memory error from 6f58e02 ?

I'm guessing you are on Python 2.7?

Would you mind checking whether you get the memory error on Python 3, for this commit?

@effigies - I wonder whether we should just revert the BytesIO refactor, assuming that the cStringIO implementation from six is gentler with memory than io.BytesIO?

[demianw]

There's no memory error from commit 6f58e02

I'm working on python 3.5 not on 2.7.

[effigies]

Sure. Let's revert and just fix the import from externals. Feel free to do it, or I can when I get to a computer. Should we generally switch back to six.BytesIO? The main impulse I was indulging was a desire for consistency.

[matthew-brett]

@demianw - that's really odd, that you get this change in behavior on Python 3.5 - Chris' recent refactoring should only have an effect for Python 2.

What do you get when you checkout the master branch again, and you apply this patch:

diff --git a/nibabel/cifti2/parse_cifti2.py b/nibabel/cifti2/parse_cifti2.py
index 63fc0dd..774544f 100644
--- a/nibabel/cifti2/parse_cifti2.py
+++ b/nibabel/cifti2/parse_cifti2.py
@@ -11,7 +11,7 @@ from __future__ import division, print_function, absolute_import
 from distutils.version import LooseVersion
 
 import numpy as np
-from io import BytesIO
+from six import BytesIO
 
 from .cifti2 import (Cifti2MetaData, Cifti2Header, Cifti2Label,
                      Cifti2LabelTable, Cifti2VertexIndices,

?

[demianw]

@matthew-brett Mmmmmm. OK, I cannot reproduce the error now. The computer where I'm running everything is under much less memory stress (external conditions). Yet I dealt with this issue thorough the day of yesterday and today's morning (on a script that had worked flawlessly for weeks).

Thanks for such quick replies and I'll work on developing a test where we can reproduce this on other computers before crying wolf again.

[effigies]

@demianw If you could test pre and post refactor on each 3.5 and 2.7, that would be great. Thanks for the report.

[demianw]

Well, the best thing will be to try and add a memory stress test to the test battery and let travis do the work ;)

[matthew-brett]

@demianw - are you sure you weren't using Python 2 when you got the error?

[demianw]

@matthew-brett @effigies I'm absolutely sure I'm on python 3. Also the error happens again but only when the machine is under memory stress. Here you have the traceback which should help

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-117-599208c50a5e> in <module>()
----> 1 client.futures_of(full_cortex_admm_c2)[0].result()

/home/dwasserm/software/anaconda2/envs/root3/lib/python3.5/site-packages/distributed/client.py in result(self)
    111         result = sync(self.client.loop, self._result, raiseit=False)
    112         if self.status == 'error':
--> 113             six.reraise(*result)
    114         elif self.status == 'cancelled':
    115             raise result

/home/dwasserm/software/anaconda2/envs/root3/lib/python3.5/site-packages/six.py in reraise(tp, value, tb)
    683             value = tp()
    684         if value.__traceback__ is not tb:
--> 685             raise value.with_traceback(tb)
    686         raise value
    687 

<ipython-input-11-758a42c38a1f> in get_nibabel_slices()
      2 
      3 def get_nibabel_slices(filename, slice_):
----> 4     nib_dataobj = nib.load(filename).dataobj
      5     return nib_dataobj[slice_]
      6 

/home/dwasserm/software/nibabel/nibabel/loadsave.py in load()
     43         is_valid, sniff = image_klass.path_maybe_image(filename, sniff)
     44         if is_valid:
---> 45             img = image_klass.from_filename(filename, **kwargs)
     46             return img
     47 

/home/dwasserm/software/nibabel/nibabel/filebasedimages.py in from_filename()
    258     def from_filename(klass, filename):
    259         file_map = klass.filespec_to_file_map(filename)
--> 260         return klass.from_file_map(file_map)
    261 
    262     @classmethod

/home/dwasserm/software/nibabel/nibabel/cifti2/cifti2.py in from_file_map()
   1330          """
   1331         from .parse_cifti2 import _Cifti2AsNiftiImage, Cifti2Extension
-> 1332         nifti_img = _Cifti2AsNiftiImage.from_file_map(file_map)
   1333 
   1334         # Get cifti2 header

/home/dwasserm/software/nibabel/nibabel/keywordonly.py in wrapper()
     15                     '{0} takes at most {1} positional argument{2}'.format(
     16                         func.__name__, n, 's' if n > 1 else ''))
---> 17             return func(*args, **kwargs)
     18         return wrapper
     19     return decorator

/home/dwasserm/software/nibabel/nibabel/analyze.py in from_file_map()
    967         data = klass.ImageArrayProxy(imgf, hdr_copy, mmap=mmap)
    968         # Initialize without affine to allow header to pass through unmodified
--> 969         img = klass(data, None, header, file_map=file_map)
    970         # set affine from header though
    971         img._affine = header.get_best_affine()

/home/dwasserm/software/nibabel/nibabel/cifti2/parse_cifti2.py in __init__()
    148             raise ValueError('Nifti2 header does not contain a CIFTI2 '
    149                              'extension')
--> 150         self.cifti_img.data = self.get_data()
    151 
    152 

/home/dwasserm/software/nibabel/nibabel/dataobj_images.py in get_data()
    187         if self._data_cache is not None:
    188             return self._data_cache
--> 189         data = np.asanyarray(self._dataobj)
    190         if caching == 'fill':
    191             self._data_cache = data

/home/dwasserm/software/anaconda2/envs/root3/lib/python3.5/site-packages/numpy/core/numeric.py in asanyarray()
    581 
    582     """
--> 583     return array(a, dtype, copy=False, order=order, subok=True)
    584 
    585 

/home/dwasserm/software/nibabel/nibabel/arrayproxy.py in __array__()
    149     def __array__(self):
    150         # Read array and scale
--> 151         raw_data = self.get_unscaled()
    152         return apply_read_scaling(raw_data, self._slope, self._inter)
    153 

/home/dwasserm/software/nibabel/nibabel/arrayproxy.py in get_unscaled()
    144                                        offset=self._offset,
    145                                        order=self.order,
--> 146                                        mmap=self._mmap)
    147         return raw_data
    148 

/home/dwasserm/software/nibabel/nibabel/volumeutils.py in array_from_file()
    507                              shape=shape,
    508                              order=order,
--> 509                              offset=offset)
    510             # The error raised by memmap, for different file types, has
    511             # changed in different incarnations of the numpy routine

/home/dwasserm/software/anaconda2/envs/root3/lib/python3.5/site-packages/numpy/core/memmap.py in __new__()
    262         bytes -= start
    263         offset -= start
--> 264         mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
    265 
    266         self = ndarray.__new__(subtype, shape, dtype=descr, buffer=mm,

OSError: [Errno 12] Cannot allocate memory

[matthew-brett]

Can you also confirm you don't see this error with 6f58e02 ?