Skip to content

ENH: added numexpr support for where operations #3154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 25, 2013

Conversation

jreback
Copy link
Contributor

@jreback jreback commented Mar 24, 2013

Added support for numexpr to do where operations, so this will have an
effect on most boolean indexing operations themselves
(inital commit would speedup ops like df>0)

this PR is in effect speeding up np.where

df[(df>0)&(df2>0)] (100k x 100 columns)

no_ne is no numexpr at all
st is single threaded

This operation is restricted to int64/float64 dtypes
(others would be upcasted, which we could deal with, but
not now)

note: the above operation could (and should) be much faster
if done as a single operation, but for now this is basically
4 calls to numexpr (3 boolean, then the where), but that's
for another day

----------------------------------------------------------------
Test name                      | target[ms] | base[ms] |   ratio
----------------------------------------------------------------
frame_multi_and                       128.5169   385.2890     0.3336
frame_multi_and_st                    215.7741   417.0649     0.5174
frame_multi_and_no_ne                 397.3501   401.0870     0.9907

     implemented internals/where in numexpr
@jreback
Copy link
Contributor Author

jreback commented Mar 25, 2013

any issues?

@ghost
Copy link

ghost commented Mar 25, 2013

I didn't test, but obviously the more the better.

jreback added a commit that referenced this pull request Mar 25, 2013
ENH: added numexpr support for where operations
@jreback jreback merged commit 08672e3 into pandas-dev:master Mar 25, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant