ERR: Categoricals should not allow non-strings when an object dtype is passed

[hnykda]

HDF Store (with table) doesn't support categories with mixed type inside a category

Even though it is possible to store category type using to_hdf with table, you can't do that when you have mixed types inside the category. It would be nice to mention this at least on docs.

Code Sample, a copy-pastable example if possible

import pandas as pd
df = pd.DataFrame({'a':[1, 'string']})
df['a'] = df['a'].astype('category')
df.to_hdf('play.hdf', 'main', format='table')

the error:

TypeError: Cannot serialize the column [values] because
its data contents are [mixed-integer] object dtype

Expected Output

No error, saved file.

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.6.4-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.0
nose: 1.3.7
pip: 8.1.1
setuptools: 20.3
Cython: None
numpy: 1.10.4
scipy: 0.17.0
statsmodels: None
xarray: None
IPython: 4.1.2
sphinx: 1.4.1
patsy: None
dateutil: 2.5.2
pytz: 2016.3
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.5.1
matplotlib: 1.5.1
openpyxl: None
xlrd: 1.0.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None

[jreback]

this is in general a completely bad idea. categories are by-definition single dtypes (and not object), except for strings.

These should raise actually on creation.

[jreback]

if you'd like to push a PR for error reporting in catetgorical creation that would be great.

[hnykda]

Yeah, I agree with you that it should be raised on creation.

I had a thousands of columns loaded from CSV and was converting them automatically to categories based on something like len(column.unique())>3: column.astype('category'). And when there is e.g. np.nan and 'string' in the column, it leads to what I have reported.

Will think about doing that PR.

Thanks Jeff!

[hnykda]

Sorry, I can't do it in a near future (say, month). But I am marking it down and maybe come back to it. If anyone wants to help here, it's very welcome.

[wcwagner]

@jreback Can you clarify what you mean by saying "Categoricals should not allow non-strings when an object dtype is passed"

My PR above takes this literally, which breaks many tests, particularly here . That list comprehension passes in arr to Categorical.from_array, which is often of type object, but all the individual values are homogenous

In my PR, should I just check if all the values are of the same type?

Thanks

[hnykda]

Hello @wcwagner . Thank you for taking a look on this.

I believe that it was more like the second option - individual values should be homogenous (in the terms of dtypes). So don't allow something like this: 1, '2', 3 (mixing int with string), while '1', '2', '3' (all str/objects) or 1, 2, 3 (all ints) are valid.

If I should implement it extremely naively, I would do something like:

categories_types = [type(x) for x in categories]
if len(categories_types) > 1: 
    raise ValueError('Categories must be all of the same type. They are %s', categories_types)