Skip to content

BUG: Inconsistent dtype selection for row-wise(axis=1) operations in pandas-2.0 for datetime & timedelta dtypes #52524

Closed
@galipremsagar

Description

@galipremsagar

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

In [2]: import pandas as pd

In [3]: from io import StringIO

In [4]: csv_str = ',t1,t2\n0,2020-08-01 09:00:00,1940-08-31 06:00:00\n1,1920-05-01 10:30:00,2020-08-02 10:00:00\n'

In [5]: df = pd.read_csv(StringIO(csv_str))

In [6]: df = df[['t1', 't2']]

In [7]: df['t1'] = df['t1'].astype('datetime64[us]')

In [8]: df['t2'] = df['t2'].astype('datetime64[us]')

In [9]: df
Out[9]: 
                   t1                  t2
0 2020-08-01 09:00:00 1940-08-31 06:00:00
1 1920-05-01 10:30:00 2020-08-02 10:00:00

In [10]: df.max(axis=1)
Out[10]: 
0   2020-08-01 09:00:00
1   2020-08-02 10:00:00
dtype: datetime64[us]

In [11]: df['t2'] = df['t2'].astype('datetime64[ms]')

In [12]: df.dtypes
Out[12]: 
t1    datetime64[us]
t2    datetime64[ms]
dtype: object

In [13]: df.max(axis=1)
Out[13]: 
0   2020-08-01 09:00:00
1   2020-08-02 10:00:00
dtype: datetime64[ns]

Issue Description

In the above example, a row-wise operation on different time resolutions seems to be defaulting to ns time resolution, whereas here we can clearly choose to return datetime64[us](the lowest possible time resolution depending on the operation performed-min).

Expected Behavior

In [13]: df.max(axis=1)
Out[13]: 
0   2020-08-01 09:00:00
1   2020-08-02 10:00:00
dtype: datetime64[us]

Installed Versions

INSTALLED VERSIONS

commit : c2a7f1a
python : 3.10.10.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-76-generic
Version : #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.0rc1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.6.1
pip : 23.0.1
Cython : 0.29.34
pytest : 7.2.2
hypothesis : 6.70.2
sphinx : 5.3.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.12.0
pandas_datareader: None
bs4 : 4.12.0
bottleneck : None
brotli :
fastparquet : None
fsspec : 2023.3.0
gcsfs : None
matplotlib : None
numba : 0.56.4
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 10.0.1
pyreadstat : None
pyxlsb : None
s3fs : 2023.3.0
scipy : 1.10.1
snappy :
sqlalchemy : 1.4.46
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : None
qtpy : None
pyqt5 : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugNon-Nanodatetime64/timedelta64 with non-nanosecond resolutionReduction Operationssum, mean, min, max, etc.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions