BUG: pd.cut creates NaNs instead of expected bins for Timestamps. Behaviour is DIFFERENT for list and Series, with IntervalIndex and DatetimeIndex

[VladimirFokow]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
ts = pd.Series([pd.Timestamp('2000-01-01'),
                pd.Timestamp('2000-01-02'),
                pd.Timestamp('2000-01-02'),
                pd.Timestamp('2000-01-03')])
bins = pd.interval_range(ts[0], ts[3])

# 1) `IntervalIndex` with `pd.Series`:
pd.cut(ts, bins)

Output:

0    NaN
1    NaN
2    NaN
3    NaN
dtype: category
Categories (2, interval[datetime64[ns], right]): [(2000-01-01, 2000-01-02] < (2000-01-02, 2000-01-03]]

# 2) `IntervalIndex` with `np.ndarray`:
pd.cut(ts.to_numpy(), bins)

Output:

[NaN, NaN, NaN, NaN]
Categories (2, interval[datetime64[ns], right]): [(2000-01-01, 2000-01-02] < (2000-01-02, 2000-01-03]]

# 3) `DatetimeIndex` with list:
dt_indices = pd.date_range(ts[0], ts[3])
pd.cut(ts.to_list(), dt_indices)  # <-- produces TypeError

TypeError:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/Users/Vladimir/nb.ipynb Cell 30 in <cell line: 1>()
----> 1 pd.cut(ts.to_list(), dt_indices)

File ~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pandas/core/reshape/tile.py:292, in cut(x, bins, right, labels, retbins, precision, include_lowest, duplicates, ordered)
    289     if (np.diff(bins.astype("float64")) < 0).any():
    290         raise ValueError("bins must increase monotonically.")
--> 292 fac, bins = _bins_to_cuts(
    293     x,
    294     bins,
    295     right=right,
    296     labels=labels,
    297     precision=precision,
    298     include_lowest=include_lowest,
    299     dtype=dtype,
    300     duplicates=duplicates,
    301     ordered=ordered,
    302 )
    304 return _postprocess_for_cut(fac, bins, retbins, dtype, original)

File ~/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pandas/core/reshape/tile.py:427, in _bins_to_cuts(x, bins, right, labels, precision, include_lowest, dtype, duplicates, ordered)
    424         bins = unique_bins
    426 side: Literal["left", "right"] = "left" if right else "right"
--> 427 ids = ensure_platform_int(bins.searchsorted(x, side=side))
    429 if include_lowest:
    430     ids[np.asarray(x) == bins[0]] = 1

TypeError: '<' not supported between instances of 'int' and 'Timestamp'

Issue Description

• If bins is an IntervalIndex: pd.cut works as expected when x is a list of Timestamps, but produces NaNs when it is a pd.Series or np.ndarray.
• If bins is a DatetimeIndex : pd.cut works as expected when x is a pd.Series or np.ndarray of Timestamps, but raises a TypeError when it is a list.

pd.cut documentation clearly states that x can be any 1-dimensional array-like.

Examples that work as expected:

pd.cut(ts.to_list(), bins)
pd.cut(ts, dt_indices)
pd.cut(ts.to_numpy(), dt_indices)

This issue is a question on Stack Overflow.

Expected Behavior

[NaN, (2000-01-01, 2000-01-02], (2000-01-01, 2000-01-02], (2000-01-02, 2000-01-03]]

Installed Versions

INSTALLED VERSIONS

commit : e8093ba
python : 3.10.0.final.0
python-bits : 64
OS : Darwin
OS-release : 20.6.0
Version : Darwin Kernel Version 20.6.0: Tue Jun 21 20:50:28 PDT 2022; root:xnu-7195.141.32~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

pandas : 1.4.3
numpy : 1.23.2
pytz : 2022.1
dateutil : 2.8.2
setuptools : 61.2.0
pip : 22.2.2
Cython : 0.29.30
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.8.0
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.1
IPython : 8.2.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.1.1
matplotlib : 3.5.1
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.8.10
xarray : None
xlrd : None
xlwt : None
zstandard : None

[andrewchen1216]

take

[shiyu-coder]

I have also encountered this problem, and the current solution is:
first convert the elements of type datetime64[ns] in bins to INT through the pd.Timestamp.value method;
after obtained the result of pd.cut, and then pass the timestamp of INT type through pd.to_datetime(x/1e9, unit='s') to convert back.

[Saya47]

same issue

[Saya47]

import pandas as pd
np.array([  2,   3, 346,  10,  11,  12,  13,  14,  16,  17,  18,  22,  29,
        32, 287,  35,   6,  43,  45,  46,  47,  48,  58,  60,  63, 722,
        80,  84,  41,  85,  89,  90, 178,  93,  92,  97,  98, 101, 609,
       114, 284, 125, 126, 124, 130, 132, 133, 138, 145, 146, 149, 151,
       161, 162, 163, 164, 480, 166, 651, 168, 170, 174, 181, 183, 188,
       189, 193, 195, 198, 200, 202, 203, 206, 210, 212, 215, 216, 217,
       218, 221, 224, 267, 234, 238, 239, 737, 242, 250, 253, 257, 256,
       258, 233, 271, 273, 274, 279, 281, 282, 285, 291, 299, 301, 304,
       315, 321, 333, 335, 336, 370, 337, 339, 341, 342, 344, 352, 353,
       354, 359, 192, 374, 495, 367, 363, 364, 368, 385, 391, 396, 201,
       407, 136,  40, 112, 413, 219, 415, 417, 420, 428, 433, 437, 405,
       439, 440, 441, 448, 450, 451, 455, 456, 476, 477, 478, 481, 482,
       490, 371, 492, 654, 581, 515, 525, 107, 530, 533, 549, 550, 552,
       554, 555, 556, 560, 562, 565, 570, 580, 583, 653, 585, 595, 597,
       167, 612, 719, 252, 625, 626, 630, 636, 642, 643, 648, 649, 650,
       655, 746, 656, 657, 660, 673, 674, 675, 676, 684, 680, 682, 683,
       688, 338, 467, 696, 460, 697, 698, 699, 703, 704, 708, 711, 717,
       718, 724, 251, 728, 729, 732, 619, 735, 736, 740, 741, 365, 752,
       334])
pd.cut(x, bins=[i for i in range(min(x), max(x), 15)])

[aniruddhakal]

df.cut method is producing the NaN's of float types. They check True with np.is_nan(), but neither of df.fillna() or df.replace(float('nan'), replacement_value) can't replace it!