PERF: resample with max/min taking slow path

[jreback]

In [21]: df = DataFrame(np.random.randn(100000,2),index=date_range('20130101',periods=100000,freq='50L'))

In [19]: %timeit df.resample('1s',how=np.max)[0]
1 loops, best of 3: 902 ms per loop

mean is ok

In [22]: %timeit df.resample('1s',how=np.mean)[0]
100 loops, best of 3: 2.42 ms per loop

The way to do this actually is ohlc

In [20]: %timeit df.groupby(pd.TimeGrouper('1s'))[0].ohlc()
1000 loops, best of 3: 1.16 ms per loop

[dsm054]

I was wondering for a second why I hadn't seen this, but I tend to write how='max' instead, which seems to be fast-path:

>>> %timeit df.resample('1s',how=np.max)[0]
1 loops, best of 3: 1.18 s per loop
>>> %timeit df.resample('1s',how='max')[0]
100 loops, best of 3: 3.42 ms per loop

[jreback]

I think this is trivial actually.....

just add here: https://github.com/pydata/pandas/blob/master/pandas/core/groupby.py#L2990 (its not mpicking it up because of the mapping)

can you add some vbenches as well? (for both 'max' and np.max)...just to be sure...

[chrisaycock]

I came here to say that how='max' takes the fast path in my code, just like @dsm054 asserts.

[jreback]

@chrisaycock yep...its a 'bug' in that 'np.max' is not translated properly......pretty straightforward to fix...

any PR takers?

[dsm054]

I can have a go tonight-- haven't done anything vbenchy, but I can pattern-match relatively well, so as long as it's just "do the same thing as everything else" I should be okay. But if anyone else is dying to do it, he's welcome to it. :^)

[jreback]

@dsm054 gr8....put in vb_suite/groupby.py....add a new sub-section for various test of these

[dsm054]

@jreback: you sure you want the vbench stuff in groupby? Wouldn't it be more natural in timeseries, with the other resample stuff?

[jreback]

sorry
that's what I meant :)

[dsm054]

Looks good so far:

-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
dataframe_resample_min_numpy                 |   4.0100 | 1271.2740 |   0.0032 |
dataframe_resample_max_numpy                 |   4.4327 | 1280.4783 |   0.0035 |

[rockg]

Does count also take the slow path (I believe it does)?

[jreback]

@rockg can you put up an example? this was a very depedent path, e.g.

passing max and max (the function) worked, but np.max did not as it didn't map properly

[rockg]

This is how it looks on my machine. It is very convoluted how this ends up getting called. I'll need to see if I can figure out how to speed this up.

%timeit df.resample('1s',how=np.max)[0]
1 loops, best of 3: 1.99 s per loop
%timeit df.resample('1s', how='max')[0]
100 loops, best of 3: 2.95 ms per loop
%timeit df.resample('1s', how='count')[0]
1 loops, best of 3: 4.3 s per loop

[jreback]

@rockg count is not getting translated to call the correct cython method (related but not identical to the np.max problem)

[dsm054]

Does the count issue mean I shouldn't make a PR? (The code and tests are done, but I accidentally merged it into my github master and I'm not sure how to undo that yet.)

[jreback]

@dsm054 you can just rebase out the merged master

git fetch upstream/master
git rebase -i upstream/master

then just remove that commit

the count can be a separate issue or if you or @rockg figure it out...can be included here