Skip to content

BUG: to_datetime fails with np.datetime64 and non-ISO format #50038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Dec 4, 2022
Merged

BUG: to_datetime fails with np.datetime64 and non-ISO format #50038

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v2.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -658,7 +658,7 @@ Datetimelike
- Bug in subtracting a ``datetime`` scalar from :class:`DatetimeIndex` failing to retain the original ``freq`` attribute (:issue:`48818`)
- Bug in ``pandas.tseries.holiday.Holiday`` where a half-open date interval causes inconsistent return types from :meth:`USFederalHolidayCalendar.holidays` (:issue:`49075`)
- Bug in rendering :class:`DatetimeIndex` and :class:`Series` and :class:`DataFrame` with timezone-aware dtypes with ``dateutil`` or ``zoneinfo`` timezones near daylight-savings transitions (:issue:`49684`)
- Bug in :func:`to_datetime` was raising ``ValueError`` when parsing :class:`Timestamp` or ``datetime`` objects with non-ISO8601 ``format`` (:issue:`49298`)
- Bug in :func:`to_datetime` was raising ``ValueError`` when parsing :class:`Timestamp`, ``datetime``, or ``np.datetime64`` objects with non-ISO8601 ``format`` (:issue:`49298`, :issue:`50036`)
-

Timedelta
Expand Down
12 changes: 11 additions & 1 deletion pandas/_libs/tslibs/strptime.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,17 @@ from _thread import allocate_lock as _thread_allocate_lock
import numpy as np
import pytz

cimport numpy as cnp
from numpy cimport (
int64_t,
ndarray,
)

from pandas._libs.missing cimport checknull_with_nat_and_na
from pandas._libs.tslibs.conversion cimport convert_timezone
from pandas._libs.tslibs.conversion cimport (
convert_timezone,
get_datetime64_nanos,
)
from pandas._libs.tslibs.nattype cimport (
NPY_NAT,
c_nat_strings as nat_strings,
Expand All @@ -33,6 +37,9 @@ from pandas._libs.tslibs.np_datetime cimport (
pydatetime_to_dt64,
)
from pandas._libs.tslibs.timestamps cimport _Timestamp
from pandas._libs.util cimport is_datetime64_object

cnp.import_array()


cdef dict _parse_code_table = {"y": 0,
Expand Down Expand Up @@ -166,6 +173,9 @@ def array_strptime(
check_dts_bounds(&dts)
result_timezone[i] = val.tzinfo
continue
elif is_datetime64_object(val):
iresult[i] = get_datetime64_nanos(val, NPY_FR_ns)
continue
Comment on lines +176 to +178
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll planning to do more code sharing between the ISO and non-ISO paths, but for now this is a minimal fix

else:
val = str(val)

Expand Down
13 changes: 13 additions & 0 deletions pandas/tests/tools/test_to_datetime.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -758,6 +758,19 @@ def test_to_datetime_today_now_unicode_bytes(self, arg):
def test_to_datetime_dt64s(self, cache, dt):
assert to_datetime(dt, cache=cache) == Timestamp(dt)

@pytest.mark.parametrize(
"arg, format",
[
("2001-01-01", "%Y-%m-%d"),
("01-01-2001", "%d-%m-%Y"),
],
)
def test_to_datetime_dt64s_and_str(self, arg, format):
# https://github.com/pandas-dev/pandas/issues/50036
result = to_datetime([arg, np.datetime64("2020-01-01")], format=format)
expected = DatetimeIndex(["2001-01-01", "2020-01-01"])
tm.assert_index_equal(result, expected)

@pytest.mark.parametrize(
"dt", [np.datetime64("1000-01-01"), np.datetime64("5000-01-02")]
)
Expand Down