Skip to content

BUG: Fix mixed datetime dtype inference #33749

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Jun 1, 2020
Merged

BUG: Fix mixed datetime dtype inference #33749

merged 21 commits into from
Jun 1, 2020

Conversation

dsaxton
Copy link
Member

@dsaxton dsaxton commented Apr 23, 2020

@jreback jreback added Compat pandas objects compatability with Numpy or Python functions Datetime Datetime data dtype labels Apr 23, 2020
@jbrockmendel
Copy link
Member

is the docbuild failure real or a fluke?

@dsaxton
Copy link
Member Author

dsaxton commented Apr 23, 2020

is the docbuild failure real or a fluke?

It looks like something in user_guide/computation.rst which doesn't seem related, kicking off again to check

Nevermind, was a typo in whatsnew that caused it

other = DatetimeIndex(other)
try:
other = DatetimeIndex(other)
except OutOfBoundsDatetime:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is hitting this? why are we not bubbling this up?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cases where we have a datetime.date that's out of bounds for DatetimeIndex (e.g. https://github.com/pandas-dev/pandas/pull/33749/files#diff-0c5fadf83f91d24a931a051d82801917R569) are raising here instead of returning -1 which I believe is a bug on master (that's what @jschendel and I were talking about above). I don't think we'd want this to actually raise since it doesn't seem to be an essential step.

)
def test_infer_dtype_date_order_invariant(self, values):
# https://github.com/pandas-dev/pandas/issues/33741
result = lib.infer_dtype(values, skipna=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add NaT and parameterize on skipna as well

@jreback
Copy link
Contributor

jreback commented Apr 25, 2020

needs a rebase as well

Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i guess this is ok. cc @jbrockmendel

@@ -4662,7 +4662,11 @@ def _maybe_promote(self, other: "Index"):
"""

if self.inferred_type == "date" and isinstance(other, ABCDatetimeIndex):
return type(other)(self), other
try:
maybe_casted = type(other)(self)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

write this as

try:
    return type(other)(self)
except OutOfBoundsDatetime:
    pass
return Index(self), other

as it needs to copy

@jreback jreback added this to the 1.1 milestone Apr 26, 2020
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comment, ping on green.

return "datetime"
if is_date_array(values, skipna=skipna):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be elif

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@dsaxton
Copy link
Member Author

dsaxton commented Jun 1, 2020

@jreback ping

@jreback jreback merged commit a1eae0e into pandas-dev:master Jun 1, 2020
@dsaxton dsaxton deleted the datetime-type-inference branch June 1, 2020 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Compat pandas objects compatability with Numpy or Python functions Datetime Datetime data dtype
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: DatetimeIndex.get_indexer wrong result on mixed-type argument
4 participants