BUG: "Python int too large" in maybe_convert_objects with numpy 1.26

[bemoody]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

pip install numpy==1.26.4 pandas==2.2.3

import numpy, pandas
numpy._set_promotion_state("weak_and_warn")
x = pandas.DataFrame({"x": [1]})
print(x)

Issue Description

If using numpy 1.26, and numpy is set to "weak" or "weak_and_warn" promotion mode (meant to be compatible with the behavior of numpy 2.x), this causes internal pandas functions to fail.

For example, the above command to print a trivial DataFrame results in:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/frame.py", line 1214, in __repr__
    return self.to_string(**repr_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/util/_decorators.py", line 333, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/frame.py", line 1394, in to_string
    return fmt.DataFrameRenderer(formatter).to_string(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/io/formats/format.py", line 962, in to_string
    string = string_formatter.to_string()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/io/formats/string.py", line 29, in to_string
    text = self._get_string_representation()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/io/formats/string.py", line 53, in _get_string_representation
    return self._fit_strcols_to_terminal_width(strcols)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/io/formats/string.py", line 163, in _fit_strcols_to_terminal_width
    col_lens = Series([Series(ele).str.len().max() for ele in strcols])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/series.py", line 584, in __init__
    data = sanitize_array(data, index, dtype, copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/construction.py", line 654, in sanitize_array
    subarr = maybe_convert_platform(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/dtypes/cast.py", line 138, in maybe_convert_platform
    arr = lib.maybe_convert_objects(arr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib.pyx", line 2602, in pandas._libs.lib.maybe_convert_objects
OverflowError: Python int too large to convert to C long

This doesn't happen with numpy 1.26 in its default "legacy" mode. It doesn't happen with numpy 2.x in either "legacy" or "weak" mode.

More information about numpy 1.x versus 2.x and promotion modes is documented here: https://numpy.org/devdocs/numpy_2_0_migration_guide.html#changes-to-numpy-data-type-promotion

Expected Behavior

print(pandas.DataFrame({"x": [1]})) should not crash. It should work properly regardless of the global numpy promotion setting.

Installed Versions

INSTALLED VERSIONS
------------------
commit                : 0691c5cf90477d3503834d983f69350f250a6ff7
python                : 3.11.2
python-bits           : 64
OS                    : Linux
OS-release            : 6.1.0-7-amd64
Version               : #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-2 (2023-04-08)
machine               : x86_64
processor             : 
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

pandas                : 2.2.3
numpy                 : 1.26.4
pytz                  : 2024.2
dateutil              : 2.9.0.post0
pip                   : 23.0.1
Cython                : None
sphinx                : None
IPython               : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : None
blosc                 : None
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : None
fsspec                : None
html5lib              : None
hypothesis            : None
gcsfs                 : None
jinja2                : None
lxml.etree            : None
matplotlib            : None
numba                 : None
numexpr               : None
odfpy                 : None
openpyxl              : None
pandas_gbq            : None
psycopg2              : None
pymysql               : None
pyarrow               : None
pyreadstat            : None
pytest                : None
python-calamine       : None
pyxlsb                : None
s3fs                  : None
scipy                 : None
sqlalchemy            : None
tables                : None
tabulate              : None
xarray                : None
xlrd                  : None
xlsxwriter            : None
zstandard             : None
tzdata                : 2024.2
qtpy                  : None
pyqt5                 : None

[yuanx749]

I can confirm, but it seems working on main branch.

[bemoody]

Thanks. But do you mean the bug does not occur with the main branch of numpy, or do you mean the bug does not occur with the main branch of pandas and version 1.26.4 of numpy?

As far as I've seen, this bug doesn't occur with the 2.x releases of numpy, only with the 1.x releases.

I tried doing this:

git clone https://github.com/pandas-dev/pandas
virtualenv v1
./v1/bin/pip install ./pandas
./v1/bin/pip install 'numpy<2'

And I also tried doing this:

virtualenv v2
./v2/bin/pip install --pre --extra-index https://pypi.anaconda.org/scientific-python-nightly-wheels/simple pandas
./v2/bin/pip install 'numpy<2'

Both installations exhibit the bug above.

If you don't see the bug, what platform/interpreter and what versions of pandas and numpy are you using?

[bemoody]

It seems like this is an inconsistency in numpy. Looks like "weak promotion" in 2.x doesn't apply to comparisons, but "weak promotion" in 1.x does apply to comparisons?

• numpy 2.x

>>> import numpy
>>> numpy._set_promotion_state('weak')
>>> numpy.int8(1) < 1000
np.True_

• numpy 1.x

>>> import numpy
>>> numpy._set_promotion_state('weak')
>>> numpy.int8(1) < 1000
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: Python integer 1000 out of bounds for int8

In pandas, this causes an exception at either line 1457:

            or (oINT64_MIN <= val < 0)

or line 2631:

                        val > oUINT64_MAX or val < oINT64_MIN):

for example:

>>> pandas._libs.lib.maybe_convert_objects(numpy.array([numpy.int64(1)], dtype=object))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib.pyx", line 2631, in pandas._libs.lib.maybe_convert_objects
OverflowError: Python int too large to convert to C long
>>> pandas._libs.lib.maybe_convert_objects(numpy.array([numpy.uint32(1)], dtype=object))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib.pyx", line 2628, in pandas._libs.lib.maybe_convert_objects
  File "lib.pyx", line 1457, in pandas._libs.lib.Seen.saw_int
OverflowError: Python integer -9223372036854775808 out of bounds for uint32

[yuanx749]

I do not produce the bug on the main branch of pandas. My environment:

INSTALLED VERSIONS

commit : 49ca01b
python : 3.10.14
python-bits : 64
OS : Linux
OS-release : 5.10.16.3-microsoft-standard-WSL2
Version : #1 SMP Fri Apr 2 22:23:49 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 3.0.0.dev0+1580.g68d9dcab5b.dirty
numpy : 1.26.4
dateutil : 2.9.0
pip : 24.2
Cython : 3.0.11
sphinx : 8.0.2
IPython : 8.27.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : 1.4.0
fastparquet : 2024.5.0
fsspec : 2024.9.0
html5lib : 1.1
hypothesis : 6.112.1
gcsfs : 2024.9.0post1
jinja2 : 3.1.4
lxml.etree : 5.3.0
matplotlib : 3.9.2
numba : 0.60.0
numexpr : 2.10.0
odfpy : None
openpyxl : 3.1.5
psycopg2 : 2.9.9
pymysql : 1.4.6
pyarrow : 17.0.0
pyreadstat : 1.2.7
pytest : 8.3.3
python-calamine : None
pytz : 2024.2
pyxlsb : 1.0.10
s3fs : 2024.9.0
scipy : 1.14.1
sqlalchemy : 2.0.34
tables : 3.10.1
tabulate : 0.9.0
xarray : 2024.9.0
xlrd : 2.0.1
xlsxwriter : 3.2.0
zstandard : 0.23.0
tzdata : 2024.1
qtpy : None
pyqt5 : None