BUG: DataFrame.to_records, DataFrame constructor broken for categoricals

[fkaufer]

to_records() and the df constructor are broken for data containing categoricals when created with dtype='category' or astype('category').

Broken (TypeError: data type not understood):

pd.DataFrame(list('abc'),dtype='category')

df=pd.DataFrame(list('abc'), columns=['A'])
df['A'] = df['A'].astype('category')
df.to_records()

Works:

pd.Series(list('abc'), dtype='category')

pd.DataFrame(list('abc'), dtype=pd.core.common.CategoricalDtype)

df=pd.DataFrame(list('abc'), columns=['A'])
df['A'] = df['A'].astype(pd.core.common.CategoricalDtype)
df.to_records()

pd.Series(list('abc'), dtype='category').to_frame().to_records()

to_frame seems to remove the category dtype though.

Pandas version: 0.15.0-20-g2737f5a

[jreback]

hmm, the second part to_frame() converting back to object is a bug (it should remain a categorical)

I am not sure I buy the first part. If you want to see if you can 'fix' and demonstrate that would help.

[fkaufer]

Actually, there are several issues. I only had the impression that they are somehow related in the sense that it makes a difference whether I use dtype=pd.core.common.CategoricalDtypeor dtype='category. But in fact for to_frame() this doesn't matter and removes the categorical either way.

My main problem is this

df=pd.DataFrame(list('abc'), columns=['A'])
df['A'] = df['A'].astype('category')
df.to_records()

Result:

[...]
//anaconda/envs/pd15/lib/python2.7/site-packages/pandas-0.15.0_20_g2737f5a-py2.7-macosx-10.5-x86_64.egg/pandas/core/common.pyc in __bytes__(self)
    163         Yields a bytestring in both py2/py3.
    164         """
--> 165         from pandas.core.config import get_option
    166 
    167         encoding = get_option("display.encoding")

TypeError: data type not understood

I have no problems when converting as follows:

df['A'] = df['A'].astype(pd.core.categorical.Categorical)
df.to_records()

The same is true for pd.DataFrame(list('abc'), dtype='category') which I discovered only coincidently when I was about to prepare a minimal example to report this supposedly to_records issue; I usually don't use dtype='category' in the df constructor.

Another example which shows it is not only related to to_records:

df['A'] = df['A'].astype(pd.core.common.CategoricalDtype)
pd.lib.fast_zip([df.A.values, df.B.values])

... works as expected.

df['A'] = df['A'].astype('category')
pd.lib.fast_zip([df.A.values, df.B.values])

... returns:

...
//anaconda/envs/pd15/lib/python2.7/site-packages/pandas-0.15.0_20_g2737f5a-py2.7-macosx-10.5-x86_64.egg/pandas/lib.so in pandas.lib.fast_zip (pandas/lib.c:9860)()

SystemError: numpy/core/src/multiarray/iterators.c:370: bad argument to internal function

[jreback]

you realize that using to_ records() defeats the entire purpose of categorical as numpy cannot support this

[fkaufer]

Sure, but I don't see point. Most to_... methods decode categoricals, but obviously there are reasons to use them.

Ironically I came across this issue when creating factor/key variables from multiple columns:

pd.factorize(df.to_records())

But I ended up using pd.lib.fast_zip anyway, which is less convenient but faster by an order of magnitude:

pd.factorize(pd.lib.fast_zip([df[c].values for c in df.columns]))

[jreback]

by your reply I am confused

to_records gives you a structured array
the only thing you could do with a categorical is make it into a regular object array
the back end has to support the categorical - in fact NO back ends currently support it and only HDF5 will actually be able to reproduce it faithfully

[fkaufer]

Sorry for the confusion.

Just to make sure: the main issue here is the inconsistency between using pandas.core.common.CategoricalDtype and its alias 'category' for various functions (e.g. to_records) and the fact the using 'category' sometimes triggers exceptions. IMO these exceptions are bugs. But reading your comments above and on my request (#8633) for graceful degradation behavior for categorical conversion in general, makes be believe you consider the exceptions are intentional.

But if this is the case and you think to_records shouldn't work for categoricals and raising TypeError: data type not understood is correct, then to_records should also fail for pandas.core.common.CategoricalDtype, right?

But instead

df=pd.DataFrame(list('abc'), columns=['A'])
df['A'] = df['A'].astype('category')
df.to_records()
Out[...]
...
TypeError: data type not understood

Whereas

df['A'] = df['A'].astype(pd.core.categorical.Categorical)
df.to_records()
Out[...]
rec.array([(0, 'a'), (1, 'b'), (2, 'c')], 
      dtype=[('index', '<i8'), ('A', 'O')])

[fkaufer]

Oh wait, it seems the reason for the inconsistency is even more weird. The second call of .astype(pd.core.categorical.Categorical) converts back to object.

df=pd.DataFrame(list('abc'), columns=['A'])
df['A'] = df['A'].astype(pd.core.categorical.Categorical)
df.to_records()
...
TypeError: data type not understood
df['A'] = df['A'].astype(pd.core.categorical.Categorical)
df.to_records()
rec.array([(0, 'a'), (1, 'b'), (2, 'c')], 
      dtype=[('index', '<i8'), ('A', 'O')])
df['A'] = df['A'].astype(pd.core.categorical.Categorical)
df.to_records()
...
TypeError: data type not understood

.astype('category') is - as expected - idempotent on categoricals.

[jreback]

@fkaufer

in the other thread, I was refering specifically to serialization/deserilization to other formats (e.g. to_*), that are actual formats, e.g. hdf,csv,stata,msgpack (NotImplemented)

while to_records is a bug, an is a conversion issue (this is a subtle point here). I agree this should simply coerce (and that's what this issue is about).

Further, you shoulld never need touch the actual CategoricalDtype object directly, and simply use 'category' (they should be interchangeable and if not that's a bug).

The reason we released categorical is its simply impossible to catch ALL possible cases. This is a massive addition and I think we got most, but that's why its helpful for you to find bugs! (which turn into tests and will get fixed for 0.15.1)

[jreback]

I realize u r using pd.core.categorical.Category as a dtype
that is undefined! (the dtype is CategoricalDtype)

Categorical is the actual object

technically it's actually ok and doesn't actually raise (nor does Numpy when u do this) - but prob should simply raise as its not correct at all

[jreback]

@fkaufer so #8652
should fix all 3 issues that you brought up (in future, FYI, its better to put check boxes in the top so can keep track).

pls test and lmk.