Pandas test fails with Scipy 0.19

[zym1010]

Code Sample, a copy-pastable example if possible

import pandas as pd
pd.test()

Problem description

there are some errors w.r.t. interpolation related functions. Seems that this is related to the change of behavior for some Scipy functions in 0.19. But which is correct?

======================================================================
FAIL: test_interp_various (pandas.tests.frame.test_missing.TestDataFrameInterpolate)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/tests/frame/test_missing.py", line 513, in test_interp_various
    assert_frame_equal(result, expected)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1313, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1181, in assert_series_equal
    obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 59, in pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)
  File "pandas/src/testing.pyx", line 173, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 0] are different

DataFrame.iloc[:, 0] values are different (28.57143 %)
[left]:  [1.0, 2.0, 2.81547781415, 4.0, 5.0, 5.52964175305, 7.0]
[right]: [1.0, 2.0, 2.81621174, 4.0, 5.0, 5.64146581, 7.0]

======================================================================
FAIL: test_interp_scipy_basic (pandas.tests.series.test_missing.TestSeriesInterpolateData)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/tests/series/test_missing.py", line 711, in test_interp_scipy_basic
    assert_series_equal(result, expected)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1181, in assert_series_equal
    obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 59, in pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)
  File "pandas/src/testing.pyx", line 173, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Series are different

Series values are different (33.33333 %)
[left]:  [1.0, 3.0, 6.82352941176, 12.0, 18.0588235294, 25.0]
[right]: [1.0, 3.0, 6.769231, 12.0, 18.230769, 25.0]

======================================================================
FAIL: test_cmov_window_regular (pandas.tests.test_window.TestMoments)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/tests/test_window.py", line 878, in test_cmov_window_regular
    tm.assert_series_equal(xp, rs)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1181, in assert_series_equal
    obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 59, in pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)
  File "pandas/src/testing.pyx", line 173, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Series are different

Series values are different (60.0 %)
[left]:  [nan, nan, 9.28667, 10.34667, 12.00556, 13.33889, 13.38, 12.33667, nan, nan]
[right]: [nan, nan, 9.60058823529, 10.8523529412, 12.86, 13.52, 12.7629411765, 12.2070588235, nan, nan]

======================================================================
FAIL: test_cmov_window_regular_linear_range (pandas.tests.test_window.TestMoments)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/tests/test_window.py", line 895, in test_cmov_window_regular_linear_range
    tm.assert_series_equal(xp, rs)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1181, in assert_series_equal
    obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 59, in pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)
  File "pandas/src/testing.pyx", line 173, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Series are different

Series values are different (60.0 %)
[left]:  [nan, nan, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, nan, nan]
[right]: [nan, nan, 2.35294117647, 3.35294117647, 4.35294117647, 5.35294117647, 6.35294117647, 7.35294117647, nan, nan]

======================================================================
FAIL: test_cmov_window_regular_missing_data (pandas.tests.test_window.TestMoments)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/tests/test_window.py", line 928, in test_cmov_window_regular_missing_data
    tm.assert_series_equal(xp, rs)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1181, in assert_series_equal
    obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 59, in pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)
  File "pandas/src/testing.pyx", line 173, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Series are different

Series values are different (80.0 %)
[left]:  [nan, nan, 9.33167, 9.76125, 9.28667, 10.34667, 12.00556, 13.82125, 14.49429, 13.765]
[right]: [nan, nan, 9.61230769231, 9.24125, 9.60058823529, 10.8523529412, 12.86, 14.3857142857, 14.1308333333, 13.3433333333]

======================================================================
FAIL: test_cmov_window_special (pandas.tests.test_window.TestMoments)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/tests/test_window.py", line 955, in test_cmov_window_special
    tm.assert_series_equal(xp, rs)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1181, in assert_series_equal
    obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 59, in pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)
  File "pandas/src/testing.pyx", line 173, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Series are different

Series values are different (60.0 %)
[left]:  [nan, nan, 9.86851, 11.02969, 11.65161, 12.75129, 12.90702, 12.83757, nan, nan]
[right]: [nan, nan, 9.95592026795, 11.1636020642, 11.8220886145, 12.6904154014, 12.797644924, 12.8491264878, nan, nan]

======================================================================
FAIL: test_cmov_window_special_linear_range (pandas.tests.test_window.TestMoments)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/tests/test_window.py", line 973, in test_cmov_window_special_linear_range
    tm.assert_series_equal(xp, rs)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1181, in assert_series_equal
    obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 59, in pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)
  File "pandas/src/testing.pyx", line 173, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)
  File "/Users/yimengzh/miniconda2/envs/default35/lib/python3.5/site-packages/pandas/util/testing.py", line 1018, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Series are different

Series values are different (60.0 %)
[left]:  [nan, nan, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, nan, nan]
[right]: [nan, nan, 2.07210900201, 3.07210900201, 4.07210900201, 5.07210900201, 6.07210900201, 7.07210900201, nan, nan]

----------------------------------------------------------------------
Ran 10252 tests in 287.945s

FAILED (SKIP=627, failures=7)

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.5.2.final.0 python-bits: 64 OS: Darwin OS-release: 15.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 8.1.2
setuptools: 26.1.1.post20160901
Cython: None
numpy: 1.11.3
scipy: 0.19.0
statsmodels: None
xarray: None
IPython: 5.1.0
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[jreback]

well, since scipy 0.19.0 was just released and something I imagine changed I am not surprised. Would you have a look and see what changed and/or provide a PR to fix? (keeping in mind that we are back-compat thru scipy 0.13)

[jreback]

further scipy is not on the default conda channels as of yet, so this doesn't hit our tests yet.

[zym1010]

@jreback I will have a look at it later this week.

[zym1010]

@jreback after some digging, I think those cmov_window errors should be due to scipy/scipy#6483, and those interp errors should be due to change of implementation for quadratic and cubic interpolation.

To fix the first one, it should do to pass an additional argument False to

pandas/pandas/core/window.py

Line 547 in 648ae4f

return sig.get_window(win_type, window).astype(float)

. For the second, no idea.

[jreback]

@zym1010 for 1st one: so looks like they added an argument. Were we just ignoring this before? (so its now required). did they give any warning? is it back-compat to prior versions? (if so, let's just add the arg, no big deal).

For the 2nd just add in a conditional in the tests:

if scipy.__version__ >= LooseVersion('0.19.0'):
    # this results
else:
    # old results

We do this in a couple of other places as well for scipy results.

can you do a PR? (note that we are not actually testing with 0.19.0 yet as we don't have a fully conda-forge build for a couple of reasons), but I think I can change one of ours to do that.

[zym1010]

@jreback for 1st one: the argument is always there, and it's nullified if the window length is odd in previous versions. Now it's not the case. So in some sense I would say this is a bug on pandas' side. But yeah, they gave no warning on this nullifying behavior, and I don't think this change is in their release notes.

for second one. Alright. I will take your approach. BTW, the change is due to their spline generator (previously they used splmake, now not). See scipy/scipy#6710.

can you do a PR?

No problem. Should be done by midnight today when I'm more or less done with other work.

[jreback]

@zym1010 awesome! I will put in place a PR to have testing on conda-forge (so we use 0.19.0 for scipy soon). It will fail master, but hopefully your PR should fix!.

[jreback]

ok, merged #15668

change the skipping on scipy 0.19.0 when you push.