Description
Code Sample
import pandas as pd
cat1 = pd.Categorical(
["a", "b",],
categories=["a", "b"],
ordered=False)
cat2 = pd.Categorical(
["a", "b"],
categories=["b", "a"],
ordered=False)
df1 = pd.DataFrame({"A": ["a", "b"], "B": cat1})
df2 = pd.DataFrame({"A": ["a", "b"], "B": cat2})
expected = pd.DataFrame({"A": ["a", "b", "a", "b"],
"B": ["a", "b", "a", "b"]})
res = pd.concat([df1, df2], ignore_index=True)
print(res)
assert np.all(res == expected), "expected:\n{}\n res:\n{}".format(expected, res)
A B
0 a a
1 b b
2 a b
3 b aProblem description
In some previous versions, merging two categorical columns where the category and label did not agree would give the expected result of converting to string and using the values rather than the categories. In the new version, the merged columns keep the categories instead of the values. This is a major problem because it causes unexpected results in workflows which require reading from multiple parts, converting some of their columns to categories and then merging them.
Expected Output
A B
0 a a
1 b b
2 a a
3 b b
Output of pd.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-1049-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.22.0
pytest: 3.2.5
pip: 9.0.2
setuptools: 38.5.1
Cython: 0.25.1
numpy: 1.13.3
scipy: 0.18.1
pyarrow: None
xarray: None
IPython: 5.1.0
sphinx: None
patsy: None
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: 2.6.1
feather: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.1.2
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.5.0