DatetimeIndex very slow repr performance

[wesm]

Hi folks, it's been a while. I would like to draw your attention to:

dr = pd.date_range('20000101', freq='D', periods=100000)
%time dr._is_dates_only

This takes about 500ms on my machine. Scales linearly with the size of the index (I interrupted a repr on a 5mm-length index and discovered this). Probably a good idea to fix this for 0.15.0 (this is on master)

[jorisvandenbossche]

@jreback This seems a very recent performance regression, as I hadn't updated my master on Windows since Tue Oct 7, and it timed only 80 ns. When I updated to latest master, it was indeed around 500 ms.
What I want to say, you fixed this specific method, but maybe there is an underlying reason for the performance regression that could also come up in other functions?

[behzadnouri]

note also the output here is wrong, because there is an overflow error:

>>> idx = pd.date_range('20000101', freq='D', periods=90000)
>>> idx._is_dates_only
True
>>> idx = pd.date_range('20000101', freq='D', periods=100000)
>>> idx._is_dates_only
False
>>> idx[-1]
Timestamp('1689-03-26 00:25:26.290448384', offset='D')

[jreback]

@jorisvandenbossche the method was 'poorly' written. Could be other issues, prob need to add an issue to systematically test the index repr's in vbench. (I didn't find any). We do test the whole frame, but sometimes not testing with all of the index types.

[jreback]

@behzadnouri

thats wraparound and results from passing negative values to date type, e.g.

In [7]: pd.DatetimeIndex(np.array([1,0,-1]).astype('M8[ns]')).tolist()
Out[7]: 
[Timestamp('1970-01-01 00:00:00.000000001'),
 Timestamp('1970-01-01 00:00:00'),
 Timestamp('1969-12-31 23:59:59.999999999')]

and is technically allowed (different issue if we want to raise on doing something like this - maybe).

[jreback]

@jorisvandenbossche this is their in 0.14.1 as well.

[jorisvandenbossche]

hmm, ok, should have done something wrong then (no time at the moment to further look at it)