`test_maybe_promote_int_with_int` error on 32bit platform

[toddrme2178]

The test test_maybe_promote_int_with_int in tests/dtypes/cast/test_promote.py is failing for a subset of combinations where int8, uint8, int16, or uint16 values are promotoed to int32 or uint32 targets on i586 the platform (but not on x86_64). The error message is pretty consistent.

This failure is new in pandas 1.0.1, it was not present on 1.0.0.

The specific combination of values that fail are:

int16-32768-int32
int8-32768-int32
uint16--1-int32
uint16--129-int32
uint16-2147483648-uint32
uint16-65536-uint32
uint8-2147483648-uint32
uint8-65536-uint32

These do not fail:

int16--32769-int32
int8--32769-int32
uint16--32769-int32
uint8--32769-int32

A specific example of the error message is:

______________ test_maybe_promote_int_with_int[uint16--129-int32] ______________
linux -- Python 3.7.3 /usr/bin/python3

dtype = dtype('uint16'), fill_value = -129, expected_dtype = dtype('int32')

    @pytest.mark.parametrize(
        "dtype, fill_value, expected_dtype",
        [
            # size 8
            ("int8", 1, "int8"),
            ("int8", np.iinfo("int8").max + 1, "int16"),
            ("int8", np.iinfo("int16").max + 1, "int32"),
            ("int8", np.iinfo("int32").max + 1, "int64"),
            ("int8", np.iinfo("int64").max + 1, "object"),
            ("int8", -1, "int8"),
            ("int8", np.iinfo("int8").min - 1, "int16"),
            ("int8", np.iinfo("int16").min - 1, "int32"),
            ("int8", np.iinfo("int32").min - 1, "int64"),
            ("int8", np.iinfo("int64").min - 1, "object"),
            # keep signed-ness as long as possible
            ("uint8", 1, "uint8"),
            ("uint8", np.iinfo("int8").max + 1, "uint8"),
            ("uint8", np.iinfo("uint8").max + 1, "uint16"),
            ("uint8", np.iinfo("int16").max + 1, "uint16"),
            ("uint8", np.iinfo("uint16").max + 1, "uint32"),
            ("uint8", np.iinfo("int32").max + 1, "uint32"),
            ("uint8", np.iinfo("uint32").max + 1, "uint64"),
            ("uint8", np.iinfo("int64").max + 1, "uint64"),
            ("uint8", np.iinfo("uint64").max + 1, "object"),
            # max of uint8 cannot be contained in int8
            ("uint8", -1, "int16"),
            ("uint8", np.iinfo("int8").min - 1, "int16"),
            ("uint8", np.iinfo("int16").min - 1, "int32"),
            ("uint8", np.iinfo("int32").min - 1, "int64"),
            ("uint8", np.iinfo("int64").min - 1, "object"),
            # size 16
            ("int16", 1, "int16"),
            ("int16", np.iinfo("int8").max + 1, "int16"),
            ("int16", np.iinfo("int16").max + 1, "int32"),
            ("int16", np.iinfo("int32").max + 1, "int64"),
            ("int16", np.iinfo("int64").max + 1, "object"),
            ("int16", -1, "int16"),
            ("int16", np.iinfo("int8").min - 1, "int16"),
            ("int16", np.iinfo("int16").min - 1, "int32"),
            ("int16", np.iinfo("int32").min - 1, "int64"),
            ("int16", np.iinfo("int64").min - 1, "object"),
            ("uint16", 1, "uint16"),
            ("uint16", np.iinfo("int8").max + 1, "uint16"),
            ("uint16", np.iinfo("uint8").max + 1, "uint16"),
            ("uint16", np.iinfo("int16").max + 1, "uint16"),
            ("uint16", np.iinfo("uint16").max + 1, "uint32"),
            ("uint16", np.iinfo("int32").max + 1, "uint32"),
            ("uint16", np.iinfo("uint32").max + 1, "uint64"),
            ("uint16", np.iinfo("int64").max + 1, "uint64"),
            ("uint16", np.iinfo("uint64").max + 1, "object"),
            ("uint16", -1, "int32"),
            ("uint16", np.iinfo("int8").min - 1, "int32"),
            ("uint16", np.iinfo("int16").min - 1, "int32"),
            ("uint16", np.iinfo("int32").min - 1, "int64"),
            ("uint16", np.iinfo("int64").min - 1, "object"),
            # size 32
            ("int32", 1, "int32"),
            ("int32", np.iinfo("int8").max + 1, "int32"),
            ("int32", np.iinfo("int16").max + 1, "int32"),
            ("int32", np.iinfo("int32").max + 1, "int64"),
            ("int32", np.iinfo("int64").max + 1, "object"),
            ("int32", -1, "int32"),
            ("int32", np.iinfo("int8").min - 1, "int32"),
            ("int32", np.iinfo("int16").min - 1, "int32"),
            ("int32", np.iinfo("int32").min - 1, "int64"),
            ("int32", np.iinfo("int64").min - 1, "object"),
            ("uint32", 1, "uint32"),
            ("uint32", np.iinfo("int8").max + 1, "uint32"),
            ("uint32", np.iinfo("uint8").max + 1, "uint32"),
            ("uint32", np.iinfo("int16").max + 1, "uint32"),
            ("uint32", np.iinfo("uint16").max + 1, "uint32"),
            ("uint32", np.iinfo("int32").max + 1, "uint32"),
            ("uint32", np.iinfo("uint32").max + 1, "uint64"),
            ("uint32", np.iinfo("int64").max + 1, "uint64"),
            ("uint32", np.iinfo("uint64").max + 1, "object"),
            ("uint32", -1, "int64"),
            ("uint32", np.iinfo("int8").min - 1, "int64"),
            ("uint32", np.iinfo("int16").min - 1, "int64"),
            ("uint32", np.iinfo("int32").min - 1, "int64"),
            ("uint32", np.iinfo("int64").min - 1, "object"),
            # size 64
            ("int64", 1, "int64"),
            ("int64", np.iinfo("int8").max + 1, "int64"),
            ("int64", np.iinfo("int16").max + 1, "int64"),
            ("int64", np.iinfo("int32").max + 1, "int64"),
            ("int64", np.iinfo("int64").max + 1, "object"),
            ("int64", -1, "int64"),
            ("int64", np.iinfo("int8").min - 1, "int64"),
            ("int64", np.iinfo("int16").min - 1, "int64"),
            ("int64", np.iinfo("int32").min - 1, "int64"),
            ("int64", np.iinfo("int64").min - 1, "object"),
            ("uint64", 1, "uint64"),
            ("uint64", np.iinfo("int8").max + 1, "uint64"),
            ("uint64", np.iinfo("uint8").max + 1, "uint64"),
            ("uint64", np.iinfo("int16").max + 1, "uint64"),
            ("uint64", np.iinfo("uint16").max + 1, "uint64"),
            ("uint64", np.iinfo("int32").max + 1, "uint64"),
            ("uint64", np.iinfo("uint32").max + 1, "uint64"),
            ("uint64", np.iinfo("int64").max + 1, "uint64"),
            ("uint64", np.iinfo("uint64").max + 1, "object"),
            ("uint64", -1, "object"),
            ("uint64", np.iinfo("int8").min - 1, "object"),
            ("uint64", np.iinfo("int16").min - 1, "object"),
            ("uint64", np.iinfo("int32").min - 1, "object"),
            ("uint64", np.iinfo("int64").min - 1, "object"),
        ],
    )
    def test_maybe_promote_int_with_int(dtype, fill_value, expected_dtype):
        dtype = np.dtype(dtype)
        expected_dtype = np.dtype(expected_dtype)
    
        # output is not a generic int, but corresponds to expected_dtype
        exp_val_for_scalar = np.array([fill_value], dtype=expected_dtype)[0]
    
>       _check_promote(dtype, fill_value, expected_dtype, exp_val_for_scalar)

../../BUILDROOT/python-pandas-1.0.1-58.1.i386/usr/lib/python3.7/site-packages/pandas/tests/dtypes/cast/test_promote.py:237: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../BUILDROOT/python-pandas-1.0.1-58.1.i386/usr/lib/python3.7/site-packages/pandas/tests/dtypes/cast/test_promote.py:94: in _check_promote
    _assert_match(result_fill_value, expected_fill_value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

result_fill_value = -129, expected_fill_value = -129

    def _assert_match(result_fill_value, expected_fill_value):
        # GH#23982/25425 require the same type in addition to equality/NA-ness
        res_type = type(result_fill_value)
        ex_type = type(expected_fill_value)
        if res_type.__name__ == "uint64":
            # No idea why, but these (sometimes) do not compare as equal
            assert ex_type.__name__ == "uint64"
        elif res_type.__name__ == "ulonglong":
            # On some builds we get this instead of np.uint64
            # Note: cant check res_type.dtype.itemsize directly on numpy 1.18
            assert res_type(0).itemsize == 8
            assert ex_type == res_type or ex_type == np.uint64
        else:
            # On some builds, type comparison fails, e.g. np.int32 != np.int32
>           assert res_type == ex_type or res_type.__name__ == ex_type.__name__
E           AssertionError: assert (<class 'numpy.intc'> == <class 'numpy.int32'>
E             -<class 'numpy.intc'>
E             +<class 'numpy.int32'> or 'intc' == 'int32'
E             - intc
E             + int32)

The error message is consistent, except that intc and int32 are replaced with uintc and uint32 in some cases. I can provide all error messages if desired.

The tests are being run with numpy 1.18.1.

[TomAugspurger]

Thanks for the report. Are you able to bisect to the commit that caused the regression?

[TomAugspurger]

I'm updating our wheel-building infra for the new uploading stuff, and noticed these failures on 32-bit windows and linux jobs.

https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=29023&view=logs&j=dcd50a69-7bd8-59f7-72b1-a119e6ffb32f&t=d88b7d2e-2609-55f5-9005-b62435c7a771 and https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=29023&view=logs&j=517fe804-fa30-5dc2-1413-330699242c05&t=2e128ad5-2f7f-5333-3f34-c85b8fbc7250 for example.

[TomAugspurger]

Seems like these were added in #28777. @jbrockmendel do you have access to a 32-bit OS?

I would think this would be failing on our 32-bit job, but apparently not (or they aren't being run?).

[jbrockmendel]

do you have access to a 32-bit OS?

I'll take another try at building on a RasPi

I would think this would be failing on our 32-bit job, but apparently not (or they aren't being run?).

IIRC I un-xfailed this on the theory that the CI would tell us if that was a mistake. I'm surprised this hasn't shown up until now.

[TomAugspurger]

Perhaps @toddrme2178 nailed the reason in the original post

values are promotoed to int32 or uint32 targets on i586 the platform (but not on x86_64).

I'm not sure how to check that.

[TomAugspurger]

@toddrme2178 are you able to do any more debugging on why these tests are failing? Are you interested in figuring out why our CI is failing to catch these?

Generally speaking, we need help supporting 32-bit systems.

[toddrme2178]

@TomAugspurger I'll take a look when I get a chance.

[toddrme2178]

I have figured out at the very least that the issue only happens in numpy 1.18 and later. Currently only numpy 1.14 is being tested on 32bit. I have expanded the 32bit tests to cover numpy 1.14, 1.15, 1.16, 1.17, 1.18, and master. I don't think these should all be tested normally, but to catch these sorts of issues numpy dev might be tested on 32bit.

That doesn't really explain why the issue is happening, but at least it shows why it wasn't being caught.