BUG: v14.0 Error when unpickling DF with non-unique column multiindex

[code-of-kpp]

>>> d = pandas.Series({('1ab','2'): 3, ('1ab',3):4}, )
>>> d = pandas.concat([d,d])
>>> d = pandas.concat([d,d], axis=1) 
>>> pickle.loads(pickle.dumps(d))
       0  1
1ab 3  4  4
    2  3  3
    3  4  4
    2  3  3
>>> pickle.loads(pickle.dumps(d.T))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/usr/lib64/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib64/python2.7/pickle.py", line 1217, in load_build
    setstate(state)
  File "venv/lib/python2.7/site-packages/pandas/core/internals.py", line 2063, in __setstate__
    placement=self.axes[0].get_indexer(items))
  File "venv/lib/python2.7/site-packages/pandas/core/index.py", line 3200, in get_indexer
    raise Exception('Reindexing only valid with uniquely valued Index '
Exception: Reindexing only valid with uniquely valued Index objects

[jreback]

cc @immerrr

[immerrr]

The problem is that the pickle only contains block items which are not enough to tell which item must go where if they're non-unique. Luckily, there's no need to share items/ref_items anymore and thus blocks can be pickled/unpickled as usual objects. I hope I'll get some time later tonight to prepare a pull request with new pickle format.

[jreback]

@immerrr sounds good

[metakermit]

Can we hope for a new release with this fix anytime soon? I'm hitting this bug in 0.14.1.

[jreback]

this is fixed in 0.14.1

what seems to be the problem?

[metakermit]

Yes, I see that it's still affecting me with the current master version. So, I'm trying to pd.read_pickle this pickle of a time series with duplicate indices and am getting the above Exception.

Here is the pickle in a gist or if it's difficult to get a binary file from a gist on Dropbox.

[jreback]

how old is the pickle?

[metakermit]

I created it in Pandas 0.13.1 and I can read it normally with that version. I could serialise it to some different format, but the values are some custom objects, so pickling is really the fastest way. When I debug through the code, the offending line seems to be:

self.axes[0].get_indexer(items)

which seems odd, as the index is stored both in items and in self.axes[0]. The object seems OK, aside from the duplicate values.

ipdb> items
<class 'pandas.tseries.index.DatetimeIndex'>
[2010-01-02 03:00:00, ..., 2010-03-30 17:00:00]
Length: 124, Freq: None, Timezone: None

[jreback]

this issue is not fixable in that older pickles that are prior to this fix are not unpickle able

it will going forward allow pickles with dup indicies to work though

you can try picking that commit and putting it in an older install then repickling

or export from an older version of pandas in another format

their is simply not enough information in the older pickles to recreate unambiguously

[jreback]

much better is to put your ts in a frame
then reset_index and pickle
that will work

[metakermit]

Ah, OK. I will try doing that. Thanks!