Skip to content

ENH: Add table-wise numba rolling to other agg funcions #38995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jan 7, 2021

Conversation

mroeschke
Copy link
Member

@mroeschke mroeschke commented Jan 6, 2021

  • tests added / passed
  • Ensure all linting tests pass, see here for how to run them
  • whatsnew entry

Timings for a wide table

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.rand(10, 10**5))

roll_single = df.rolling(2, method="single")
roll_table = df.rolling(2, method="table")

%timeit roll_single.mean()

4.92 s ± 463 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

roll_single.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

%timeit roll_single.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

5.72 s ± 430 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

roll_table.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

%timeit roll_table.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

10.3 ms ± 1.23 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)	

@mroeschke mroeschke added Enhancement Window rolling, ewma, expanding labels Jan 6, 2021
@mroeschke mroeschke added this to the 1.3 milestone Jan 6, 2021
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you update the benchmark in the top of the PR (e.g. show single/table for cython/numba)

@jreback
Copy link
Contributor

jreback commented Jan 7, 2021

lgtm. test are failing :-< ping on green

@mroeschke
Copy link
Member Author

All green @jreback. Needed to mark some of the numba tests as slow.

@jreback jreback merged commit aa25da2 into pandas-dev:master Jan 7, 2021
@mroeschke mroeschke deleted the enh/table_rolling_other_aggs branch January 7, 2021 22:18
luckyvs1 pushed a commit to luckyvs1/pandas that referenced this pull request Jan 20, 2021
@8549
Copy link

8549 commented May 26, 2021

Sorry if this is not the appropriate place for asking this, but will this be usable with .aggregate? Like df.rolling('1y', on='Timestamp').aggregate([np.mean, np.std, np.sum], engine='numba', engine_kwargs={'nopython': True, 'parallel': True}). Thanks in advance.

@mroeschke
Copy link
Member Author

@8549 unfortunately no. PR's to support aggregate would be appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement Window rolling, ewma, expanding
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants