Error on Warnings From C Extensions

[WillAyd]

May differ on other platforms, but currently I see about 19 warnings from C extensions still left. I think we should aim to clean these up and set the Werror flag in CI as can help be proactive in identifying bugs from extensions

Here is the list of remaining warnings from a parallel build, so order may not be relevant

warning: pandas/_libs/groupby.pyx:1088:26: Unreachable code
warning: pandas/_libs/window/aggregations.pyx:59:47: Buffer unpacking not optimized away.
warning: pandas/_libs/window/aggregations.pyx:59:47: Buffer unpacking not optimized away.
warning: pandas/_libs/window/aggregations.pyx:60:47: Buffer unpacking not optimized away.
warning: pandas/_libs/window/aggregations.pyx:60:47: Buffer unpacking not optimized away.
pandas/_libs/hashing.c:25387:20: warning: unused function '__pyx_memview_get_object' [-Wunused-function]
  static PyObject *__pyx_memview_get_object(const char *itemp) {
                   ^
pandas/_libs/hashing.c:25392:12: warning: unused function '__pyx_memview_set_object' [-Wunused-function]
static int __pyx_memview_set_object(const char *itemp, PyObject *obj) {
           ^
pandas/_libs/hashtable.c:63498:20: warning: unused function '__pyx_memview_get_object' [-Wunused-function]
  static PyObject *__pyx_memview_get_object(const char *itemp) {
                   ^
pandas/_libs/hashtable.c:63503:12: warning: unused function '__pyx_memview_set_object' [-Wunused-function]
static int __pyx_memview_set_object(const char *itemp, PyObject *obj) {
           ^
pandas/_libs/algos.c:81415:3: warning: code will never be executed [-Wunreachable-code]
  __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_arr.rcbuffer->pybuffer);
  ^~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:83050:3: warning: code will never be executed [-Wunreachable-code]
  __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_arr.rcbuffer->pybuffer);
  ^~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:83956:3: warning: code will never be executed [-Wunreachable-code]
  __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_arr.rcbuffer->pybuffer);
  ^~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:84862:3: warning: code will never be executed [-Wunreachable-code]
  __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_arr.rcbuffer->pybuffer);
  ^~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:85039:3: warning: code will never be executed [-Wunreachable-code]
  __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_arr.rcbuffer->pybuffer);
  ^~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:85945:3: warning: code will never be executed [-Wunreachable-code]
  __Pyx_SafeReleaseBuffer(&__pyx_pybuffernd_arr.rcbuffer->pybuffer);
  ^~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:89901:19: warning: assigning to '__pyx_t_5numpy_uint8_t *' (aka 'unsigned char *') from 'const __pyx_t_5numpy_uint8_t *' (aka 'const unsigned char *') discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        __pyx_v_v = (&(*((__pyx_t_5numpy_uint8_t const  *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_values.data + __pyx_t_13 * __pyx_v_values.strides[0]) ) + __pyx_t_14 * __pyx_v_values.strides[1]) ))));
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:92699:19: warning: assigning to '__pyx_t_5numpy_int8_t *' (aka 'signed char *') from 'const __pyx_t_5numpy_int8_t *' (aka 'const signed char *') discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        __pyx_v_v = (&(*((__pyx_t_5numpy_int8_t const  *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_values.data + __pyx_t_13 * __pyx_v_values.strides[0]) ) + __pyx_t_14 * __pyx_v_values.strides[1]) ))));
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:98041:19: warning: assigning to '__pyx_t_5numpy_int16_t *' (aka 'short *') from 'const __pyx_t_5numpy_int16_t *' (aka 'const short *') discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        __pyx_v_v = (&(*((__pyx_t_5numpy_int16_t const  *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_values.data + __pyx_t_13 * __pyx_v_values.strides[0]) ) + __pyx_t_14 * __pyx_v_values.strides[1]) ))));
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:103383:19: warning: assigning to '__pyx_t_5numpy_int32_t *' (aka 'int *') from 'const __pyx_t_5numpy_int32_t *' (aka 'const int *') discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        __pyx_v_v = (&(*((__pyx_t_5numpy_int32_t const  *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_values.data + __pyx_t_13 * __pyx_v_values.strides[0]) ) + __pyx_t_14 * __pyx_v_values.strides[1]) ))));
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:107434:19: warning: assigning to '__pyx_t_5numpy_int64_t *' (aka 'long *') from 'const __pyx_t_5numpy_int64_t *' (aka 'const long *') discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        __pyx_v_v = (&(*((__pyx_t_5numpy_int64_t const  *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_values.data + __pyx_t_13 * __pyx_v_values.strides[0]) ) + __pyx_t_14 * __pyx_v_values.strides[1]) ))));
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:110194:19: warning: assigning to '__pyx_t_5numpy_float32_t *' (aka 'float *') from 'const __pyx_t_5numpy_float32_t *' (aka 'const float *') discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        __pyx_v_v = (&(*((__pyx_t_5numpy_float32_t const  *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_values.data + __pyx_t_13 * __pyx_v_values.strides[0]) ) + __pyx_t_14 * __pyx_v_values.strides[1]) ))));
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/algos.c:112955:19: warning: assigning to '__pyx_t_5numpy_float64_t *' (aka 'double *') from 'const __pyx_t_5numpy_float64_t *' (aka 'const double *') discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
        __pyx_v_v = (&(*((__pyx_t_5numpy_float64_t const  *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_values.data + __pyx_t_13 * __pyx_v_values.strides[0]) ) + __pyx_t_14 * __pyx_v_values.strides[1]) ))));
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 warnings generated.
2 warnings generated.
13 warnings generated.
pandas/_libs/tslibs/conversion.c:33343:20: warning: unused function '__pyx_memview_get_object' [-Wunused-function]
  static PyObject *__pyx_memview_get_object(const char *itemp) {
                   ^
pandas/_libs/tslibs/conversion.c:33348:12: warning: unused function '__pyx_memview_set_object' [-Wunused-function]
static int __pyx_memview_set_object(const char *itemp, PyObject *obj) {
           ^
2 warnings generated.

[jreback]

+1 in this in at least linux to start

[jbrockmendel]

the algos.c ones are a result of needing to filter out invalid pairs of dtypes from fused types.

The aggregations.pyx one should be straightforward.

[WillAyd]

PRs welcome to address any part of these @jbrockmendel

…

Sent from my iPhone

On Jan 1, 2020, at 9:22 PM, jbrockmendel ***@***.***> wrote:  the algos.c ones are a result of needing to filter out invalid pairs of dtypes from fused types. The aggregations.pyx one should be straightforward. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[jbrockmendel]

@WillAyd I can't build master on OSX with py38 bc of tp_print warnings and "Werror". Can you?

[WillAyd]

Haven’t set up a 38 env yet - can you post the warnings?

…

Sent from my iPhone

On Mar 27, 2020, at 6:13 PM, jbrockmendel ***@***.***> wrote:  @WillAyd I can't build master on OSX with py38 bc of tp_print warnings and "Werror". Can you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[jbrockmendel]

can you post the warnings?

They end with:

pandas/_libs/algos.c:144455:5: error: 'tp_print' is deprecated [-Werror,-Wdeprecated-declarations]
    0,
    ^
~/.pyenv/versions/3.8.2/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
    Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
    ^
~/.pyenv/versions/3.8.2/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                     ^
6 errors generated.
error: command 'clang' failed with exit status 1

[WillAyd]

Can you look at the C file and see the Cython code that generates the line throwing the warning?

[WillAyd]

@jbrockmendel can you confirm downgrading Cython to 0.29.13 fixes the issue? From what I can tell this started back with a fix in Cython 0.29.10 cython/cython#2976 which worked around an issue with the Py38 beta https://bugs.python.org/issue37250

Interestingly enough I noticed a change in Cython 0.29.14 that reintroduced code paths touching tp_print in cython/cython#3201

When trying locally things work in Cython 0.29.13 for me but not in 0.29.14 . If you can confirm as well can open an issue on the Cython side of things

[jbrockmendel]

Yes, downgrading to cython 0.29.13 allowed me to compile on 3.8

[WillAyd]

@jbrockmendel can you also try Cython 3.0? Worked for me as well; maybe easiest to just skip the specific combo of Cython 0.29.14 with Python 3.8 on macOS...