BUG: Calling to_datetime With errrors="coerce" Still Raises After 50 Rows

[jeremytwfortune]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd
from datetime import datetime

series = pd.Series(
    [datetime.fromisoformat("1446-04-12 00:00:00+00:00")] +
    ([datetime.fromisoformat("1991-10-20 00:00:00+00:00")] * 50)
)
pd.to_datetime(series, errors="coerce", utc=True)

Issue Description

Calling to_datetime on 51 or more rows of datetime values causes these nested errors:

ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

and

pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1446-04-12 00:00:00

This is despite being called with errors="coerce" and utc=True.

These errors do not appear if:

• There are <= 50 rows in the series
• The objects are typed differently before to_datetime is called: pd.to_datetime(series.astype("str"), errors="coerce", utc=True)

Note that even a simpler call like series.value_counts() produces the same error.

Expected Behavior

51 rows with the 1991 date converted to datetime and the 1446 date converted to NaT.

Installed Versions

1.4.0rc0:

INSTALLED VERSIONS

commit : d023ba7
python : 3.9.7.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22000
machine : AMD64
processor : Intel64 Family 6 Model 126 Stepping 5, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 1.4.0rc0
numpy : 1.22.0
pytz : 2021.3
dateutil : 2.8.2
pip : 21.2.3
setuptools : 57.4.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None
zstandard : None

1.3.5:

INSTALLED VERSIONS

commit : 66e3805
python : 3.9.7.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22000
machine : AMD64
processor : Intel64 Family 6 Model 126 Stepping 5, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 1.3.5
numpy : 1.22.0
pytz : 2021.3
dateutil : 2.8.2
pip : 21.2.3
setuptools : 57.4.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[jeremytwfortune]

This seems related to #31491 and #31520 (though those are both closed). I can confirm this behavior does not appear on 0.25 as #31491 suggests.

[pjireland]

I'm also running into this error with Pandas 1.4.0. Some example code is below.

s = pd.Series(dtype=object, index=range(1, 52))
s.at[30] = {}
pd.to_datetime(s, errors="coerce") # raises `TypeError: unhashable type: 'dict'`
pd.to_datetime(s.loc[2:52], errors="coerce") # doesn't raise an error

INSTALLED VERSIONS

commit : bb1f651
python : 3.9.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19042
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252

pandas : 1.4.0
numpy : 1.21.5
pytz : 2021.3
dateutil : 2.8.2
pip : 22.0.2
setuptools : 59.8.0
Cython : 0.29.27
pytest : 6.2.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.2
lxml.etree : 4.7.1
html5lib : None
pymysql : 1.0.2
psycopg2 : None
jinja2 : 3.0.3
IPython : 8.0.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
fastparquet : 0.8.0
fsspec : 2022.01.0
gcsfs : None
matplotlib : 3.5.1
numba : 0.55.1
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 3.0.0
pyreadstat : None
pyxlsb : None
s3fs : 0.4.2
scipy : 1.7.3
sqlalchemy : 1.4.31
tables : 3.7.0
tabulate : 0.8.9
xarray : 0.21.1
xlrd : 2.0.1
xlwt : None
zstandard : None

[jeremytwfortune]

Confirming that this is still an issue on 1.4.1.

[jreback]

@jeremytwfortune issues are solved with pull requests from the community

[srotondo]

take