assert_almost_equal / equals should allow access to np.allclose

[rockg]

I am testing the equivalence of two large DataFrames (9084x367). The two are the same up to 1x10-13 but when np.array_equal fails there is a much slower code path (comparing these two frames takes upwards of 20 seconds). If I'm not mistaken, if the arrays aren't equivalent it does a more complicated version of np.allclose. I think a good intermediate step would be to check for array equivalence and then as a the second step call np.allclose--or maybe just do this on the outset. If that fails, which it will if there are any NaNs or if the tolerance is not met, then it will use the current logic. Or we could use np.isclose to consider NaNs as equivalent.

https://github.com/pydata/pandas/blob/master/pandas/src/testing.pyx#L85

[jreback]

this is not possible
the entire point is to do a simple element comparison for testing purposes to show where the elements are not equal considering any dtype

this is for testing only and should used at your own risk in production code

the Numpy routines do not consider nan positions correctly in any event

you can use np.allclosr if you wish on the numeric subset and it will work

[rockg]

I understand the other datatypes, but I'm thinking we can change array_equal with all(isclose(a, b)) and that would make a faster comparison of numpy types and would not have an issue with NaNs (this is essentially what is done anyways, numerics are compared at 5x10-6). This should create no difference to existing tests. I feel like the pandas testing api should provide a reliable way to test the equivalence of two dataframes. I am using it for my own unit tests and came across this being quite slow.

[jreback]

ok that could be reasonable
further I think passing thru keywords to .equals() would allow something similar
and is an exposed equivalents method (uses array_equivalent) under the hood

[rockg]

I agree that more control over .equals would be nice.

[jreback]

so signature could be:

def equals(self, other, strict=False, close=False, **kwargs)

strict would map to strict_nan in com.array_equivalent (currently its an internal parameter)
and close=False would trigger a call to np.allclose rather than np.array_equal, and pass
thru **kwarsgs (e.g. if one wanted to modify rtol/atol from the defaults)