RLS: 2.1.3

[lithomas1]

Release date: 11/9

I am thinking that this should be the last release in the 2.1 series, since I want to release 2.2 (which should hopefully be compatible with numpy 2.0) before numpy does their 2.0 release.

cc @pandas-dev/pandas-core

[jorisvandenbossche]

Because of #55894, I am planning to do an earlier 2.1.3 tomorrow, if all goes well

[lithomas1]

Because of #55894, I am planning to do an earlier 2.1.3 tomorrow, if all goes well

@pandas-dev/pandas-core Please don't merge to 2.1.x anymore, if anything didn't get in in time we'll do a 2.1.4 release.

[jorisvandenbossche]

Done! -> https://github.com/pandas-dev/pandas/releases/tag/v2.1.3 and https://pypi.org/project/pandas/2.1.3

[lithomas1]

@jorisvandenbossche
Do you want me to trigger the conda-forge builds?

Their bot usually does it automatically but it takes a while for it to do it.

[jorisvandenbossche]

I was planning to wait on the bot, but feel free to trigger it manually.

[lithomas1]

OK, I did it manually.

(For future reference, you can open an issue on pandas-feedstock with the title @conda-forge-admin, please update version to trigger it).

[jorisvandenbossche]

Ah, that's easy ;) Something else to update in the release guide!

[lithomas1]

Looks like the segfault in rolling returned :(

  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/rolling.py", line 1824 in corr_func
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/common.py", line 20 in flex_binary_moment
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/rolling.py", line 558 in _apply_pairwise
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/rolling.py", line 1845 in corr
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/core/window/rolling.py", line 2852 in corr
  File "/Users/runner/miniforge3/conda-bld/pandas_1699651050096/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.10/site-packages/pandas/tests/window/test_pairwise.py", line 115 in test_rolling_corr_with_zero_variance

Here's the test that segfaults

pandas/pandas/tests/window/test_pairwise.py

Lines 110 to 115 in 09ed69e

	def test_rolling_corr_with_zero_variance(window):
	# GH 18430
	s = Series(np.zeros(20))
	other = Series(np.arange(20))
	assert s.rolling(window=window).corr(other=other).isna().all()

[WillAyd]

Here is the ASAN output - I assume this is an off by one in roll_mean_pandas

==92164==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e0000048a0 at pc 0x7faabc2f8f39 bp 0x7ffc691500b0 sp 0x7ffc691500a8
READ of size 8 at 0x60e0000048a0 thread T0
    #0 0x7faabc2f8f38 in __pyx_pf_6pandas_5_libs_6window_12aggregations_2roll_mean pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so.p/pandas/_libs/window/aggregations.pyx.cpp:5258
    #1 0x7faabc2f6c89 in __pyx_pw_6pandas_5_libs_6window_12aggregations_3roll_mean pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so.p/pandas/_libs/window/aggregations.pyx.cpp:5000
    #2 0x55926ff7bc55 in cfunction_call /usr/local/src/conda/python-3.11.6/Objects/methodobject.c:542
    #3 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #4 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #5 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #6 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #7 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #8 0x55926ff94d9f in _PyVectorcall_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:257
    #9 0x55926ff94d9f in _PyObject_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:328
    #10 0x55926ff94d9f in PyObject_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:355
    #11 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #12 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #13 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #14 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #15 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #16 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #17 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #18 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #19 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #20 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #21 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #22 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #23 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #24 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #25 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #26 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #27 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #28 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #29 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #30 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #31 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #32 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #33 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #34 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #35 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #36 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #37 0x55926ff94d2d in _PyObject_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:343
    #38 0x55926ff94d2d in PyObject_Call /usr/local/src/conda/python-3.11.6/Objects/call.c:355
    #39 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #40 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #41 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #42 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #43 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #44 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #45 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #46 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #47 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #48 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #49 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #50 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #51 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #52 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #53 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #54 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #55 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #56 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #57 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #58 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #59 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #60 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #61 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #62 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #63 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #64 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #65 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #66 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #67 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #68 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #69 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #70 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #71 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #72 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #73 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #74 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #75 0x55926ff6c55a in do_call_core /usr/local/src/conda/python-3.11.6/Python/ceval.c:7343
    #76 0x55926ff6c55a in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:5367
    #77 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #78 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #79 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #80 0x55926ff5f40f in _PyObject_FastCallDictTstate /usr/local/src/conda/python-3.11.6/Objects/call.c:152
    #81 0x55926ff92aa8 in _PyObject_Call_Prepend /usr/local/src/conda/python-3.11.6/Objects/call.c:482
    #82 0x559270063568 in slot_tp_call /usr/local/src/conda/python-3.11.6/Objects/typeobject.c:7623
    #83 0x55926ff5bbd2 in _PyObject_MakeTpCall /usr/local/src/conda/python-3.11.6/Objects/call.c:214
    #84 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #85 0x55927001ecbc in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #86 0x55927001ecbc in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #87 0x55927001e34e in PyEval_EvalCode /usr/local/src/conda/python-3.11.6/Python/ceval.c:1140
    #88 0x559270035c9d in builtin_exec_impl /usr/local/src/conda/python-3.11.6/Python/bltinmodule.c:1077
    #89 0x559270035c9d in builtin_exec /usr/local/src/conda/python-3.11.6/Python/clinic/bltinmodule.c.h:465
    #90 0x55926ff750ee in cfunction_vectorcall_FASTCALL_KEYWORDS /usr/local/src/conda/python-3.11.6/Objects/methodobject.c:443
    #91 0x55926ff74fdb in _PyObject_VectorcallTstate /usr/local/src/conda/python-3.11.6/Include/internal/pycore_call.h:92
    #92 0x55926ff74fdb in PyObject_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:299
    #93 0x55926ff68462 in _PyEval_EvalFrameDefault /usr/local/src/conda/python-3.11.6/Python/ceval.c:4760
    #94 0x55926ff8add0 in _PyEval_EvalFrame /usr/local/src/conda/python-3.11.6/Include/internal/pycore_ceval.h:73
    #95 0x55926ff8add0 in _PyEval_Vector /usr/local/src/conda/python-3.11.6/Python/ceval.c:6425
    #96 0x55926ff8add0 in _PyFunction_Vectorcall /usr/local/src/conda/python-3.11.6/Objects/call.c:393
    #97 0x559270048097 in pymain_run_module /usr/local/src/conda/python-3.11.6/Modules/main.c:300
    #98 0x559270047a08 in pymain_run_python /usr/local/src/conda/python-3.11.6/Modules/main.c:595
    #99 0x559270047a08 in Py_RunMain /usr/local/src/conda/python-3.11.6/Modules/main.c:680
    #100 0x55927000d006 in Py_BytesMain /usr/local/src/conda/python-3.11.6/Modules/main.c:734
    #101 0x7fab336280cf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #102 0x7fab33628188 in __libc_start_main_impl ../csu/libc-start.c:360
    #103 0x55927000ceac  (/home/willayd/mambaforge/envs/pandas-dev/bin/python3.11+0x28feac)

0x60e0000048a0 is located 0 bytes to the right of 160-byte region [0x60e000004800,0x60e0000048a0)
allocated by thread T0 here:
    #0 0x7fab33a955f8 in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7fab2f9105ce in PyDataMem_UserNEW (/home/willayd/mambaforge/envs/pandas-dev/lib/python3.11/site-packages/numpy/core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so+0x1105ce)

SUMMARY: AddressSanitizer: heap-buffer-overflow pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so.p/pandas/_libs/window/aggregations.pyx.cpp:5258 in __pyx_pf_6pandas_5_libs_6window_12aggregations_2roll_mean
Shadow bytes around the buggy address:
  0x0c1c7fff88c0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1c7fff88d0: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
  0x0c1c7fff88e0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fff88f0: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
  0x0c1c7fff8900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c1c7fff8910: 00 00 00 00[fa]fa fa fa fa fa fa fa 00 00 00 00
  0x0c1c7fff8920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fff8930: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1c7fff8940: 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
  0x0c1c7fff8950: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1c7fff8960: 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==92164==ABORTING

[WillAyd]

The offending line in Cython is this:

            /* "pandas/_libs/window/aggregations.pyx":286
 *                 compensation_add = compensation_remove = sum_x = 0
 *                 nobs = neg_ct = 0
 *                 prev_value = values[s]             # <<<<<<<<<<<<<<
 *                 num_consecutive_same_value = 0
 *                 for j in range(s, e):
 * ```