Closed
Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
from datashader.datatypes import RaggedArray
key = RaggedArray([[0, 1], [1, 2, 3, 4]], dtype='float64')
df = pd.DataFrame({"key": key, "val": [1, 2]})
pd.merge(df, df, on='key')Issue Description
We have an extension array, and after upgrading to Pandas 2.2. Some of our tests are beginning to fail (test_merge_on_extension_array). This is related to changes made in #56523. More specifically, these lines:
pandas/pandas/core/reshape/merge.py
Lines 1751 to 1755 in 1aeaf6b
This will raise the error below, with the example above
Traceback
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[1], line 7
4 key = RaggedArray([[0, 1], [1, 2, 3, 4]], dtype='float64')
5 df = pd.DataFrame({"key": key, "val": [1, 2]})
----> 7 pd.merge(df, df, on='key')
File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:184, in merge(left, right, how, on, left_on, right_on, left_index, right_index, sort, suffixes, copy, indicator, validate)
169 else:
170 op = _MergeOperation(
171 left_df,
172 right_df,
(...)
182 validate=validate,
183 )
--> 184 return op.get_result(copy=copy)
File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:886, in _MergeOperation.get_result(self, copy)
883 if self.indicator:
884 self.left, self.right = self._indicator_pre_merge(self.left, self.right)
--> 886 join_index, left_indexer, right_indexer = self._get_join_info()
888 result = self._reindex_and_concat(
889 join_index, left_indexer, right_indexer, copy=copy
890 )
891 result = result.__finalize__(self, method=self._merge_type)
File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:1151, in _MergeOperation._get_join_info(self)
1147 join_index, right_indexer, left_indexer = _left_join_on_index(
1148 right_ax, left_ax, self.right_join_keys, sort=self.sort
1149 )
1150 else:
-> 1151 (left_indexer, right_indexer) = self._get_join_indexers()
1153 if self.right_index:
1154 if len(self.left) > 0:
File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:1125, in _MergeOperation._get_join_indexers(self)
1123 # make mypy happy
1124 assert self.how != "asof"
-> 1125 return get_join_indexers(
1126 self.left_join_keys, self.right_join_keys, sort=self.sort, how=self.how
1127 )
File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/reshape/merge.py:1753, in get_join_indexers(left_keys, right_keys, sort, how)
1749 left = Index(lkey)
1750 right = Index(rkey)
1752 if (
-> 1753 left.is_monotonic_increasing
1754 and right.is_monotonic_increasing
1755 and (left.is_unique or right.is_unique)
1756 ):
1757 _, lidx, ridx = left.join(right, how=how, return_indexers=True, sort=sort)
1758 else:
File ~/miniconda3/envs/holoviz/lib/python3.12/site-packages/pandas/core/indexes/base.py:2251, in Index.is_monotonic_increasing(self)
2229 @property
2230 def is_monotonic_increasing(self) -> bool:
2231 """
2232 Return a boolean if the values are equal or increasing.
2233
(...)
2249 False
2250 """
-> 2251 return self._engine.is_monotonic_increasing
File index.pyx:262, in pandas._libs.index.IndexEngine.is_monotonic_increasing.__get__()
File index.pyx:283, in pandas._libs.index.IndexEngine._do_monotonic_check()
File index.pyx:297, in pandas._libs.index.IndexEngine._call_monotonic()
File algos.pyx:853, in pandas._libs.algos.is_monotonic()
ValueError: operands could not be broadcast together with shapes (4,) (2,) I can get things to run if I guard the check in a try/except like this:
try:
check = (
left.is_monotonic_increasing
and right.is_monotonic_increasing
and (left.is_unique or right.is_unique)
)
except Exception:
check = False
if check:
...Expected Behavior
It will work like Pandas 2.1 and merge the two dataframes.
Installed Versions
INSTALLED VERSIONS
------------------
commit : f538741432edf55c6b9fb5d0d496d2dd1d7c2457
python : 3.12.1.final.0
python-bits : 64
OS : Linux
OS-release : 6.6.10-76060610-generic
Version : #202401051437~1704728131~22.04~24d69e2 SMP PREEMPT_DYNAMIC Mon J
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.2.0
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 69.0.3
pip : 24.0
Cython : None
pytest : 7.4.4
hypothesis : None
sphinx : 5.3.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.1.0
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.21.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : None
dataframe-api-compat : None
fastparquet : 2023.10.1
fsspec : 2024.2.0
gcsfs : None
matplotlib : 3.8.2
numba : 0.59.0
numexpr : 2.9.0
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 15.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : 2024.2.0
scipy : 1.12.0
sqlalchemy : 2.0.25
tables : None
tabulate : 0.9.0
xarray : 2024.1.1
xlrd : None
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None