"IndexError: tuple index out of range" error with numpy array contain datetimes

[richard-bibb]

Code Sample, a copy-pastable example if possible

>>> from datetime import datetime as dt
>>> import numpy as np
>>> from pandas import DataFrame
>>> d=np.array([None, None, None, None, dt.now(), None])
>>> b = DataFrame(d)

Problem description

The above code works in an old version of pandas (0.7.3) but fails in the current version (0.19.2)

Expected Output

No error and dataframe containing the numpy date array

Output of pd.show_versions()

>>> pandas.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 2.7.12.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 20.10.1
Cython: None
numpy: 1.12.1
scipy: None
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: None
pandas_datareader: None

[chrisaycock]

Interestingly, this works:

pd.DataFrame(np.array([None, None, dt.now(), None]))

But this does not:

pd.DataFrame(np.array([None, None, None, dt.now(), None]))

Numpy doesn't seem to handle the input arrays any differently, but create_block_manager_from_blocks() definitely shows different blocks. The first is

[array([['NaT', 'NaT', '2017-04-04T13:54:03.236544000', 'NaT']], dtype='datetime64[ns]')]

while the second is

[array([None, None, None, datetime.datetime(2017, 4, 4, 13, 54, 10, 804563),
       None], dtype=object)]

I'm going to dig some more into this.

[jreback]

@chrisaycock here's why.

https://github.com/pandas-dev/pandas/blob/master/pandas/types/cast.py#L810

When we have a 1-d array that is object we want to see whether its datetimelike or not. So I did this old thing where I would sample the first 3 elements. This is bogus now as can simply do this (also need to tests for timedelta as well).

In [1]: from pandas._libs.lib import is_datetime_array

In [2]: is_datetime_array(np.array([None, None, datetime.datetime.now()]))
Out[2]: True

In [3]: is_datetime_array(np.array([None, None, None, datetime.datetime.now()]))
Out[3]: True

It might be better to simply modify is_possible_datetimelike_array as an alternative (it is called below as well).

These routines are robust to null values, check for the requested (datetime or timedelta-like), and don't read the entire routine if is just strings (which is the point here, we are trying to see w/o doing too much work if we can coerce this). This is called for every object array on construction, so this needs to be cheap.

This may seem like overly paranoid / extreme. But remember we can actually have mixed dtypes that get coerced, e.g.:

In [4]: Series(['NaT', None, pd.Timestamp('20130101'), datetime.datetime.now(), '20150101'])
Out[4]: 
0                          NaT
1                          NaT
2   2013-01-01 00:00:00.000000
3   2017-04-04 14:07:52.743198
4   2015-01-01 00:00:00.000000
dtype: datetime64[ns]

but we need it be robust to non-matching types.

In [6]: Series(['NaT', None, pd.Timestamp('20130101'), datetime.datetime.now(), '20150101', np.nan, pd.Timedelta('1 day')])
Out[6]: 
0                           NaT
1                          None
2           2013-01-01 00:00:00
3    2017-04-04 14:08:34.334113
4                      20150101
5                           NaN
6               1 days 00:00:00
dtype: object

[chrisaycock]

@jreback Agreed that the culprit is the "quick inference" on the first three elements.

Is your proposal to modify maybe_infer_to_datetimelike() to use is_possible_datetimelike_array() instead of the logic that begins with the "sample"? Are there other gotchas given all of the conditionals below it?

The alternative is to infer_dtype() the entire array instead of the first three elements. Would the performance impact be just too much then?

[jreback]

i think maybe best way is to trash all of the logic for the sample and below

then modify: is_possible_datetimelike_array to return 3 states (string is prob fine)

• datetime
• timedelta
• mixed

the u know if u should attempt to_datetime or to_timedelta or are done

it short circuits so should be performant
though maybe inside that routine you can take the first 3 valid points (non null) and actually try to convert them - if that fails then you are done

the issue is to try to not iterate (and convert) all strings

[jreback]

@chrisaycock this was a can of worms, see #15892