Skip to content

Concat on non-identical categories in categorical indexes raises TypeError #17629

Closed
@P-Tillmann

Description

@P-Tillmann

Code Sample, a copy-pastable example if possible

import pandas as pd
a = pd.DataFrame({'one':['foo','bar','bar'],'two':[1,2,3],'three':[1,3,4]})
b = pd.DataFrame({'one':['baz','bar','plob'],'two':[2,2,1],'three':[5,3,1]})

a['one'] = a['one'].astype('category')
b['one'] = b['one'].astype('category')
c = a.set_index('one')
d = b.set_index('one')
print(pd.concat([a, b]))
    one  three  two
0   foo      1    1
1   bar      3    2
2   bar      4    3
0   baz      5    2
1   bar      3    2
2  plob      1    1
print(pd.concat([c, d]))
raise TypeError("categories must match existing categories "
TypeError: categories must match existing categories when appending

Problem description

While concatenating categorical data with non-identical categories is now support for data columns by upcasting the data to an appropriate dtype it is still throwing an error when attempting to do the same thing with categorical indexes.
I think it would be convenient for users to implement the same behaviour for data columns and indexes.

Expected Output

print(pd.concat([c, d]))
      three  two
one             
foo       1    1
bar       3    2
bar       4    3
baz       5    2
bar       3    2
plob      1    1
print(pd.concat([c, d]).index.dtype)
dtype('O')

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.2.final.0
python-bits: 64
OS: Darwin
OS-release: 15.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_GB.UTF-8
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8

pandas: 0.20.3
pytest: 3.2.1
pip: None
setuptools: None
Cython: 0.26
numpy: 1.13.1
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 0.9.8
lxml: 3.8.0
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

Metadata

Metadata

Assignees

No one assigned

    Labels

    CategoricalCategorical Data TypeNeeds TestsUnit test(s) needed to prevent regressionsReshapingConcat, Merge/Join, Stack/Unstack, Explodegood first issue

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions