Skip to content

BUG: Fix near-minimum timestamp handling #57314

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Feb 22, 2024
19 changes: 16 additions & 3 deletions pandas/_libs/src/vendored/numpy/datetime/np_datetime.c
Original file line number Diff line number Diff line change
Expand Up @@ -483,14 +483,27 @@ npy_datetime npy_datetimestruct_to_datetime(NPY_DATETIMEUNIT base,
if (base == NPY_FR_ns) {
int64_t nanoseconds;

// for near-minimum timestamps, scaling microseconds to nanoseconds
// overflows but adding nanoseconds puts the timestamp back in a valid range
// Minimum valid timestamp in nanoseconds (1677-09-21 00:12:43.145224193).
const int64_t min_nanoseconds = NPY_MIN_INT64 + 1;

// For near-minimum timestamps (1677-09-21 00:12:43.145224193 through
// 1677-09-21 00:12:43.145224999), scaling microseconds to nanoseconds
// overflows (1677-09-21 00:12:43.145224 -> 1677-09-21 00:12:43.145224000),
// but adding nanoseconds can put the timestamp back in a valid range for
// nanosecond parts >= 193.

// (min_nanoseconds / 1000 - 1) * 1000 would overflow, so do not scale.
// This happens if microseconds corresponds to 1677-09-21 00:12:43.145224.
if (microseconds == min_nanoseconds / 1000 - 1) {
// calculate final nanoseconds from minimum without scaling microseconds
// Instead, use minimum nanosecond timestamp as base and offset it with
// nanosecond delta between dts and the minimum (_NS_MIN_DTS.ps = 193000).
// If dts->ps >= _NS_MIN_DTS.ps, timestamp is at/above the minimum.
// If dts->ps < _NS_MIN_DTS.ps, timestamp is below minimum and overflows.
PD_CHECK_OVERFLOW(checked_int64_add(
min_nanoseconds, (dts->ps - _NS_MIN_DTS.ps) / 1000, &nanoseconds));
} else {
// microseconds does not correspond to near-minimum timestamp, use default
// scaling and addition approach, handling any other overflows.
PD_CHECK_OVERFLOW(
scaleMicrosecondsToNanoseconds(microseconds, &nanoseconds));
PD_CHECK_OVERFLOW(
Expand Down