numpy FutureWarning in df.equals

[jorisvandenbossche]

From the example in the v0.13.1 whatsnew docs (http://pandas.pydata.org/pandas-docs/version/0.15.0/whatsnew.html#id12):

df = DataFrame({'col':['foo', 0, np.nan]}).sort()
df2 = DataFrame({'col':[np.nan, 0, 'foo']}, index=[2,1,0])
df.equals(df)

gives:

In [3]: df.equals(df)
C:\Anaconda\lib\site-packages\numpy\core\numeric.py:2367: FutureWarning: numpy e
qual will not check object identity in the future. The comparison did not return
 the same result as suggested by the identity (`is`)) and will change.
  return bool(asarray(a1 == a2).all())
Out[3]: True

The example in basics.rst section does not have this warning (http://pandas.pydata.org/pandas-docs/version/0.15.0/basics.html#comparing-if-objects-are-equivalent):

df = DataFrame({'one' : Series(randn(3), index=['a', 'b', 'c']),
                'two' : Series(randn(4), index=['a', 'b', 'c', 'd']),
                'three' : Series(randn(3), index=['b', 'c', 'd'])})
(df+df).equals(df*2)

[jorisvandenbossche]

cc @unutbu

[jreback]

this has to do with the object dtype equality testing (only happens in 1.9), before that it is ok.

these warning are ALL over (because numpy put them in). Their is an issue about this ....

[jreback]

see here: #7065
and the linked numpy issue (which I ultimately closed as they didn't/couldn't fix it)

Its really a dumb warning

>>> import numpy as np
>>> np.__version__
'1.9.0'
>>> left = np.array([['foo', 0, np.nan]], dtype=object)
>>> right = np.array([['foo', 0, np.nan]], dtype=object)
>>> left == right
__main__:1: FutureWarning: numpy equal will not check object identity in the future. The comparison did not return the same result as suggested by the identity (`is`)) and will change.
array([[ True,  True,  True]], dtype=bool)

The only way around AFAICT is to essentially do is iterate and compare but will be slow and this routine should not be slow. We might have to do a cython routine for this (rather than a vectorized method)

[jorisvandenbossche]

@jreback ok, hadn't seen the other issue. Then this can be closed? (#7065 is still open).

I am not familiar with the internals of equals, but I understand that it are the nans that cause the problem. It isn't a possibility to mask the nans before doing the left == right?

[jreback]

you need to element-wise compare first (otherwise no point in nan detection), which is the rhs of that expression.

I think numpy is just doing something dumb.

yes, let's close this one. If its a problem in the docs, maybe just make the example non-object dtype.

[jorisvandenbossche]

It's not really a problem (the warning only shows when building, not in the output docs), and can indeed just change the example.

[jorisvandenbossche]

Ah, by the way, I think there was something wrong in the example: it did df.equals(df) while this should more logically be df.equals(df2), which does not give the warning

[unutbu]

If I recall correctly the warning is instigated by the use of (left == right) in core/common.py/array_equivalent. But the change in NumPy behavior will not affect array_equivalent because we use (left == right) in conjunction with

((left == right) | (pd.isnull(left) & pd.isnull(right))).all()

So the null case is already handled. Shall be quash this error message with warnings.filterwarnings as in:

import warnings
warnings.filterwarnings('ignore', 'numpy equal will not check object identity in the future')
import numpy as np
import pandas as pd
df = pd.DataFrame({'col':['foo', 0, np.nan]}).sort()
df2 = pd.DataFrame({'col':[np.nan, 0, 'foo']}, index=[2,1,0])
print(df.equals(df))

(which prints only True.)

[jreback]

yes I think we can just do that!

submit for the linked issue though