Description
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of pandas (checked on 1.1.5, the latest Anaconda version)
- (optional) I have confirmed this bug exists on the master branch of pandas.
pd.CategoricalIndex(['A', 'B', 'C', 'A', 'B']).get_indexer(pd.CategoricalIndex(['B', 'C', 'D', 'E']))
# returns array([ 1, 4, 2, -1, -1], dtype=int64)
pd.Index(['A', 'B', 'C', 'A', 'B']).get_indexer(pd.Index(['B', 'C', 'D', 'E']))
# InvalidIndexError: Reindexing only valid with uniquely valued Index objectsProblem description
CategoricalIndex.get_indexer should fail if the index contains duplicates, just like the regular index does. Instead, it returns a nonsense array that does not even have the same length as the target.
Note this problem also affects the case where dtypes are equal or the target is not categorical i.e.:
# 1
dt = pd.CategoricalDtype(categories=['A', 'B', 'C', 'D', 'E'])
pd.CategoricalIndex(['A', 'B', 'C', 'A', 'B'], dtype=dt).get_indexer(pd.CategoricalIndex(['B', 'C', 'D', 'E'], dtype=dt))
# 2
pd.CategoricalIndex(['A', 'B', 'C', 'A', 'B']).get_indexer(pd.Index(['B', 'C', 'D', 'E']))I fixed a related bug in Sep 2019 as #28257 but that fix was merged in v1.0.0 so this must be a seperate issue or a regression. Could also be related to #25459.
Output of pd.show_versions()
INSTALLED VERSIONS
commit : b5958ee
python : 3.8.5.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : English_United Kingdom.1252
pandas : 1.1.5
numpy : 1.19.2
pytz : 2020.4
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.0.0.post20201207
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None