read_pickle error for multi-index: 'FrozenList' does not support mutable operations.

[wcbeard]

I'm in a bit of a pickle here. If I try to save and read back a multi-indexed dataframe, I get this error (and in some situations, can't reproduce when, I get a TypeError: Required argument 'shape' (pos 1) not found error).

The gist with the full traceback is here.

In [3]: import numpy as np

In [4]: import pandas as pd

In [5]: np.random.seed(10)

In [6]: a = np.random.randint(0, 20, (6, 5))

In [7]: df = pd.DataFrame(a).set_index([2,3,4])

In [8]: df
Out[8]:
           0   1
2  3  4
15 0  17   9   4
8  9  0   16  17
4  19 16  10   8
11 11 1    4  15
14 17 19   8   4
13 19 13  13   5

In [9]: df.to_pickle('~/Desktop/dummy.df')

In [10]: df2 = pd.read_pickle('~/Desktop/dummy.df')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
...
TypeError: 'FrozenList' does not support mutable operations.

When I reset_index it saves fine. I'm on Mac OSX, numpy 1.7, pandas 0.12.0-361-g53eec08.

In the meantime, if there's no quick fix, anyone know another way to save multi-indexes to file? csv doesn't look like it can preserve them. I could do something like appending _index to the columns and un-rename them after reading but would prefer something less hacky.

[jreback]

try with current master; this fixed the pickle issue: 725b195

[jreback]

you can save mi for both index/colmns via csv since 0.12, see here: http://pandas.pydata.org/pandas-docs/dev/io.html#reading-columns-with-a-multiindex

here for just mi on the index (ths has been for a while): http://pandas.pydata.org/pandas-docs/dev/io.html#reading-an-index-with-a-multiindex

[wcbeard]

pd.version.version tells me I'm on 0.12.0-361-g53eec08. Is that not the most recent commit? Is it possible it's really a different version installed? (I git pull'd today and did pip uninstall pandas + pip install . in the local repo).

[jtratner]

That said, I would not be surprised if the new MultiIndex setup needed to define some of the pickle magic methods, since we changed up its internal representation. Are there tests for roundtripping MI through pickle?

[jreback]

@jtratner should be from 0.10 on....I saved a pickled version from each (and 0.13)....

@d10genes do you see that commit I referenced (just git log and searchfor pickle); i think it was 3-4 days ago

@d10genes that error is a fall thru; I think there is another error

can you debug thru pdb and step thru the read_piickle ?

basically it tries the original pickle, then the fallback, then a version with an encoding, then fallback with encoding

are you using 2.7? (or a 3x) python?

[jreback]

@d10genes can you post a sample code?

[wcbeard]

@jreback This shows up in git log

commit 725b1951249a795fe01896dff4ce46bd9206021f
Merge: 2267fe4 0436809
Author: jreback <jeff@reback.net>
Date:   Fri Sep 6 16:06:27 2013 -0700

    Merge pull request #4755 from jreback/pickle_compat

    BUG: TimeSeries compat from < 0.13

[wcbeard]

As far as sample code, how do you want it different from the OP?

[jreback]

@d10genes sorry...you put it up already....hold on

[jreback]

I have a test for a mi and a frame with several kinds of index...but of course not a mi....let me fix.

thanks for the report! (some of this pickle code was pretty tricky and was trying to cover all the bases!)

[wcbeard]

Ok, thanks. And out of curiosity, would the underlying code for pickling treat Series and DataFrames differently? Going off of my above code, it looks like series MI pickling sets off that other error:

In [13]: s = df[0]

In [14]: s
Out[14]:
2   3   4
15  0   17     9
8   9   0     16
4   19  16    10
11  11  1      4
14  17  19     8
13  19  13    13
Name: 0, dtype: int64
In [16]: s.to_pickle('~/Desktop/s.df')
In [17]: s2 = pd.read_pickle('~/Desktop/s.df')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
...
TypeError: Required argument 'shape' (pos 1) not found

[jreback]

no...the problem is assigning the index to the pandas object, same code for all (but the deserialization of the MI is not 'defined' correctly)

[jtratner]

@jreback if you want me to look at it, I can... I have a hunch. (but you
usually figure these things out quickly anyways :))

[jreback]

i have the test case set and will look in a few

[jreback]

@jtratner

ok..this branch, last commti shows the test case: https://github.com/jreback/pandas/tree/pickle_fix

a bunch of things are commented out...but this will fail on a disabled method in FrozenList

pickle is trying to extend the list....

a possible solution is to have a context manager with certain classes (or can just do it with a try:except:finally (e.g. FrozenList), which reenables all/certain methods and then redisables them.....