CLN: start using numpy-1.7 API

[immerrr]

Now that numpy-1.6 support is dropped we could proceed to use np-1.7 API.

This involves fixing a macro or two deep down in pandas internals, but otherwise should be pretty straightforward. As a bonus we can get rid of these annoying build warnings:

gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipandas/src/klib -Ipandas/src -I/home/immerrr/.conda/envs/pandas/lib/python2.7/site-packages/numpy/core/include -I/home/immerrr/.conda/envs/pandas/include/python2.7 -c pandas/index.c -o build/temp.linux-x86_64-2.7/pandas/index.o
In file included from /home/immerrr/.conda/envs/pandas/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0,
                 from /home/immerrr/.conda/envs/pandas/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /home/immerrr/.conda/envs/pandas/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from pandas/index.c:244:
/home/immerrr/.conda/envs/pandas/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

xref https://github.com/pydata/numexpr/pull/228/files

[jreback]

@immerrr would be gr8 (esp for 0.15)!

[immerrr]

I was going to suggest it for the hackathon, are you expecting anyone who's into low-level stuff there?

[jreback]

hmm, interesting, I will add this up their too. As their are going to be people who could hack on NumPy, but we'd rather co-op them to pandas :)

[devanshmehta]

I am trying to work on this bug. But half difficulty finding a starting point. It will be nice if you can point me in right direction. I have the forked the panda repo and added upstream.

[jreback]

are you at the bloomberg hackathon?

[devanshmehta]

Yes

[jreback]

@devanshmehta can you come over to my table?

[immerrr]

Ok, I gave this one a shot and it's nowhere near easy. After figuring out simple stuff, you run into two huge obstacles. One is that cython doesn't yet support numpy-1.7 api. Pandas has its own numpy.pxd, but it too has the necessary macros commented out with a description saying:

    # dtype PyArray_DESCR(ndarray) wrong refcount semantics

That one can be figured out with some interaction with Cython guys (and probably backported there), but then there are rolling functions.

Some of those functions in algos module do pure magic: they create one a single ndarray to refer to the window, pass it to python-level function and then alter its arr.data field to skip to the next element without incurring overhead of creating/destroying ndarrays. In np-1.7 api referring to data must be done via PyArray_DATA(arr) macro and it's probably not guaranteed to be assignable, so that optimization potential goes out of the window. I'm not sure if there's a way around this that doesn't lose performance.