AttributeError: 'numpy.ndarray' object has no attribute '_get_repr' with np.abs on a DatetimeIndex

[robintw]

I am trying to find the row of a DataFrame that is closest to a certain datetime. I have my data in a DataFrame which has a DatetimeIndex, and I can't use functions like asof because I need to get the closest row (either before or after the time), so I am trying to implement the standard way of doing this which is getting the absolute difference between each index time and the time I want to find, then finding the row with the minimum difference.

However, there seems to be a strange bug with using np.abs with the pandas index, and it seems to be in the printing of the results stage (relating to __get_repr). Pandas prints things differently depending on how many rows the DataFrame has, and I think this is related here - as running the code on a 200 row DataFrame fails, but on a 100 row DataFrame it works! I've checked that its not some strange value in the DataFrame which is causing the problem, as a 100 row segment from anywhere in my original input file works, whereas a 200 row segment from anywhere fails.

Two input files (aeronet_large.txt with 200 rows and aeronet.txt with 100 rows) and the code I've been using are available at this gist: https://gist.github.com/robintw/631b53fa0cd9dbdabb36, and I asked a StackOverflow question about this (http://stackoverflow.com/questions/15115547/find-closest-row-of-dataframe-to-given-time-in-pandas) where I was recommended to raise an issue.

[jreback]

try updating to master
this was fixed in #2899
and let me know

the printing is just a red herring its an exception on the subtraction

[robintw]

I downloaded the latest version from http://pandas.pydata.org/pandas-build/dev/ yesterday - looking at the times of that pull request, it looks like it should have been included. Is there a way to check from within pandas which version I am running?

[jreback]

print pandas.version
you are on windows?

which file?

[jreback]

all of the windows version looks pretty recent, so should be good

if you are windows, then this should work,
linux use git clone

[robintw]

The output of pandas.__version__ is: 0.11.0.dev-8ad9516

Yes, I am using Windows with Python 2.7.

[jreback]

yep...that looks right

[robintw]

Does it seem that this wasn't fixed in #2899 then? Is there any other information you'd like me to provide to help fix this issue?

[jreback]

I would need a test case that reproduces the error
just paste it in a comment

[robintw]

Hmm...I'm not sure entirely where the problem lies - but it definitely seems to be related to the length of the DataFrame. I've tried creating a generic sequence of dates and running np.abs on them, and it doesn't give an error:

n = 100
ind = pd.date_range(start='2009-07-01', periods=n, freq='H')
df = DataFrame(data=np.random.randn(n), index=ind)
df['time'] = df.index
image_time = dateutil.parser.parse('2009-07-02 13:04')
np.abs(df.time -image_time)

Also, the results look fairly sensible - differences are a maximum of 2 days. However, if I run the same code with n = 101 instead, I still don't get an error, but I get insane looking results - see the example of the last few lines of the results below:

Results where n = 100
2009-07-05 01:00:00 2 days, 11:56:00
2009-07-05 02:00:00 2 days, 12:56:00
2009-07-05 03:00:00 2 days, 13:56:00

Results where n = 101
2009-07-05 02:00:00 2536 days, 23:59:51.242751
2009-07-05 03:00:00 2580 days, 00:12:35.359744
2009-07-05 04:00:00 2622 days, 00:33:40.130816

Something strange definitely seems to be going on there!

Going back to the example that I provided when opening this issue (shown in the gist at https://gist.github.com/robintw/631b53fa0cd9dbdabb36), the results seem to be sensible (a max difference of 3 days) when run with the aeronet.txt file, but when run with aeronet_long.txt it gives the error below, which all seems to stem from the printing code:

AttributeError                            Traceback (most recent call last)
<ipython-input-35-97fdad570e66> in <module>()
     22 # Trying to do the absolute difference gives an error for aeronet_long.txt but not for aeronet.txt
     23 result = np.abs(aeronet.time - image_time)
---> 24 print result
     25 

C:\Python27\lib\site-packages\pandas\core\series.pyc in __str__(self)
   1021         if py3compat.PY3:
   1022             return self.__unicode__()
-> 1023         return self.__bytes__()
   1024 
   1025     def __bytes__(self):

C:\Python27\lib\site-packages\pandas\core\series.pyc in __bytes__(self)
   1031         """
   1032         encoding = com.get_option("display.encoding")
-> 1033         return self.__unicode__().encode(encoding, 'replace')
   1034 
   1035     def __unicode__(self):

C:\Python27\lib\site-packages\pandas\core\series.pyc in __unicode__(self)
   1044                     else get_option("display.max_rows"))
   1045         if len(self.index) > (max_rows or 1000):
-> 1046             result = self._tidy_repr(min(30, max_rows - 4))
   1047         elif len(self.index) > 0:
   1048             result = self._get_repr(print_header=True,

C:\Python27\lib\site-packages\pandas\core\series.pyc in _tidy_repr(self, max_vals)
   1069         """
   1070         num = max_vals // 2
-> 1071         head = self[:num]._get_repr(print_header=True, length=False,
   1072                                     name=False)
   1073         tail = self[-(max_vals - num):]._get_repr(print_header=False,

AttributeError: 'numpy.ndarray' object has no attribute '_get_repr'

That makes me suspect that the results given by the aeronet_test.py code running with aeronet_long.txt are also very strange results, and that is why an error is raised when we try to print them - because they're so insane that they can't be printed.

Does that help?

You're substracting a Datetime object from a TimeSeries and that causes an overflow
in _possibly_cast_to_timedelta where it calls tslib.array_to_timedelta64(value.astype(object), coerce=False)
there's a too broad Exception clause in Series._get_values which masks the error
and returns a numpy array (shouldn't this just die, what's the case here), which isn't
a TImeSeries Object and so the _get_repr call fails fails.
this only happens when you exceed pd.options.max_rows, because then the repr gocde path which
attempts this conversion is invoked.

Short Answer: use aeronet.time-pd.Timestamp(image_time) instead.
but you should have gotten a warning that you were doing something wrong.

Other then that, think you've got your date format wrong, most of the data is on consecutive days
but you've got some "01-07" days there, which probably should be july first, rather then january seventh.

you need to specigy a date format or store your date strings as iso8601 which is preferable, exacly
to prevent these kinds of ambiguity errors.

[jreback]

this is actually a very subtle error
I have push in 2955 to mostly fix this (not in master yet though)

this 'works', but the final answer you are looking for 'a'
is the wrong dtype (its timedelta64[us])...in microseconds..
I know it 'looks' right, but if you use the values they will be 1000 times off...(and THAT
is causing the overflow that @y-p pointed to above)

I think numpy 1.6.2 does the wrong thing on an np.abs(of a timedelta64)....
not easy right now to work around this...
but you can do some trickery depending on exactly what you are after

what should your final answer look like?

n [19]: df = pd.DataFrame(data=np.random.randn(5), index=pd.date_range('20090705',periods=5))

In [20]: df
Out[20]: 
                   0
2009-07-05 -0.659246
2009-07-06 -0.249913
2009-07-07 -0.025722
2009-07-08 -0.115015
2009-07-09 -0.123338

In [21]: df['time'] = df.index

In [22]: df
Out[22]: 
                   0                time
2009-07-05 -0.659246 2009-07-05 00:00:00
2009-07-06 -0.249913 2009-07-06 00:00:00
2009-07-07 -0.025722 2009-07-07 00:00:00
2009-07-08 -0.115015 2009-07-08 00:00:00
2009-07-09 -0.123338 2009-07-09 00:00:00

In [23]: df.dtypes
Out[23]: 
0              float64
time    datetime64[ns]
Dtype: object

In [24]: image_time
Out[24]: <Timestamp: 2009-07-02 13:04:00>

In [26]: df['diff'] = df['time']-image_time

In [27]: df
Out[27]: 
                   0                time             diff
2009-07-05 -0.659246 2009-07-05 00:00:00 2 days, 10:56:00
2009-07-06 -0.249913 2009-07-06 00:00:00 3 days, 10:56:00
2009-07-07 -0.025722 2009-07-07 00:00:00 4 days, 10:56:00
2009-07-08 -0.115015 2009-07-08 00:00:00 5 days, 10:56:00
2009-07-09 -0.123338 2009-07-09 00:00:00 6 days, 10:56:00

In [28]: df.dtypes
Out[28]: 
0               float64
time     datetime64[ns]
diff    timedelta64[ns]
Dtype: object

In [29]: a = df['diff'].abs()

In [30]: a
Out[30]: 
2009-07-05   2 days, 10:56:00
2009-07-06   3 days, 10:56:00
2009-07-07   4 days, 10:56:00
2009-07-08   5 days, 10:56:00
2009-07-09   6 days, 10:56:00
Freq: D, Name: diff, Dtype: timedelta64[us]

together with #2888, this is pointing at low test coverage for large integer cases. is there a way
to get coverage for cython code? I bet coverage would have exposed this.

[robintw]

Thanks for all the investigation of this.

@y-p: Using aeronet.time - pd.Timestamp(image_time) gives the same error for me.

I hadn't noticed that after doing np.abs the result was in us not ns - is that basically a NumPy bug? If so, should I report that separately to NumPy?

Basically, the results given by @jreback in the final line of his example seem right (apart from the fact that the datatype has been switched to us from ns). Once I've got the np.abs stuff working I was then going to find the minimum value and extract that row of the DataFrame, as that will be the row that has the closest time to the input time. So basically, if the result gives the right numbers (which the final result in the example above does), doesn't give an error, and works with something like np.min to get the minimum value, then that's all sorted for me.

Interestingly, there also seems to be a problem with the final bit of my requirement: getting the minimum value. I suspect it is related to the other problems that you've identified:

• Running a.argmin() gives an OverflowError: long too big to convert
• Running a.min() gives a TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule 'safe'

On a separate note: once we've sorted these problems, would it be worse adding a closest_time method to the DatetimeIndex object, in a similar manner to the asof method? Finding the closest row to a certain time is a fairly common operation with time-series data, so it may be worth including 'out-of-the-box'.

[jreback]

@robintw the np.abs is a numpy bug, its prob fixed in 1.7, I have a work around for you, min/max should work, will check argmin/max as well.....

I'll update master later today and let you know