Description
Code Sample, a copy-pastable example if possible
from pandas.api.types import union_categoricals
import pandas as pd
import numpy as np
dt1 = pd.Categorical([], categories=[pd.Timestamp("2020-01-01")])
dt2 = pd.Categorical([], categories=[pd.Timestamp("2020-01-02")])
dt3 = pd.Categorical([])
pd.CategoricalDtype([]).categories.dtype == np.dtype("O") # should probably be `None` or undefined
union_categoricals([dt1, dt2])
Out[113]: [], Categories (2, datetime64[ns]): [2020-01-01, 2020-01-02]
union_categoricals([dt1, dt3])
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-19-2496f54dfb94> in <module>
----> 1 union_categoricals([dt1, dt3])
~/miniconda3/envs/dask/lib/python3.6/site-packages/pandas/core/dtypes/concat.py in union_categoricals(to_union, sort_categories, ignore_order)
306 for other in to_union[1:]
307 ):
--> 308 raise TypeError("dtype of categories must be the same")
309
310 ordered = False
TypeError: dtype of categories must be the same
Problem description
At the moment, it is not possible to define the data type of categories
of a CategoricalDtype. This makes things difficult when trying to merge categorical data types e.g. before performing a concatenation of dataframes, especially when one dataframe has no defined categories.
Related: dask/dask#5756
Expected Output
# Add flag `dtype` to `pd.CategoricalDtype`
dt3 = pd.Categorical([], dtype=pd.CategoricalDtype(dtype=np.dtype("<M8[ns]"))
union_categoricals([dt1, dt3])
Out[113]: [], Categories (1, datetime64[ns]): [2020-01-01]
Output of pd.show_versions()
[paste the output of pd.show_versions()
here below this line]
INSTALLED VERSIONS
commit : None
python : 3.6.7.final.0
python-bits : 64
OS : Darwin
OS-release : 18.5.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.2.0.post20200209
Cython : None
pytest : 5.3.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.12.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None