Skip to content

to_datetime broken for Tz-aware datetimes and NaTs in same array #5961

Closed
@cancan101

Description

@cancan101

This works:

In [400]: 
pd.to_datetime([pd.Timestamp("2013-1-1", tz=pytz.timezone('US/Eastern'))])

Out[400]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2013-01-01 00:00:00-05:00]
Length: 1, Freq: None, Timezone: US/Eastern

but this (adding an NaT) does not. At the very least, the error message is misleading:

In [401]:
pd.to_datetime([pd.Timestamp("2013-1-1", tz=pytz.timezone('US/Eastern')), pd.NaT])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-401-3b5b8a67d2e1> in <module>()
----> 1 pd.to_datetime([pd.Timestamp("2013-1-1", tz=pytz.timezone('US/Eastern')), pd.NaT])

/usr/local/lib/python2.7/dist-packages/pandas-0.13.0rc1_78_g142ca62-py2.7-linux-x86_64.egg/pandas/tseries/tools.pyc in to_datetime(arg, errors, dayfirst, utc, box, format, coerce, unit)
    137         return Series(values, index=arg.index, name=arg.name)
    138     elif com.is_list_like(arg):
--> 139         return _convert_listlike(arg, box=box)
    140 
    141     return _convert_listlike(np.array([ arg ]), box=box)[0]

/usr/local/lib/python2.7/dist-packages/pandas-0.13.0rc1_78_g142ca62-py2.7-linux-x86_64.egg/pandas/tseries/tools.pyc in _convert_listlike(arg, box)
    127                 return DatetimeIndex._simple_new(values, None, tz=tz)
    128             except (ValueError, TypeError):
--> 129                 raise e
    130 
    131     if arg is None:

ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

I tracked down this issue to a bug in datetime_to_datetime64

When iterating over the elements in datetime_to_datetime64 the check for nullness is util._checknull(val) which is False for NaT.

The correct null check is to use checknull from lib

PR on the way.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions