Skip to content

PERF: enhance HDFStore Table writing performance #4457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 5, 2013

Conversation

jreback
Copy link
Contributor

@jreback jreback commented Aug 3, 2013

Took out all of the cython code, turns out you can assign directly to numpy recarrays per
column if you put in the correct shape yourself

Invoked with :
--ncalls: 5
--repeats: 5
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
write_store_table_mixed                      |  45.6208 | 121.5352 |   0.3754 |
write_store_table_panel                      |  59.4140 |  99.5456 |   0.5969 |
write_store_table                            |  40.2264 |  65.8978 |   0.6104 |
write_store_table_dc                         | 143.3910 | 174.9424 |   0.8196 |
read_store_table_wide                        |  16.8930 |  17.5310 |   0.9636 |
query_store_table_wide                       |   9.6692 |   9.7564 |   0.9911 |
write_store_table_wide                       | 117.8906 | 118.8232 |   0.9922 |
read_store_table_mixed                       |   4.7604 |   4.7478 |   1.0026 |
query_store_table                            |   4.2646 |   4.2320 |   1.0077 |
read_store_table                             |   2.3678 |   2.3178 |   1.0216 |
read_store_table_panel                       |  20.9388 |  20.3526 |   1.0288 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [d11041d] : PERF: enhance HDFStore Table writing performance
Base   [aca1a42] : Merge pull request #4437 from cpcloud/fix-astype-calls

CLN: removed is_searchable (no longer needed)
jreback added a commit that referenced this pull request Aug 5, 2013
PERF: enhance HDFStore Table writing performance
@jreback jreback merged commit 8915ce6 into pandas-dev:master Aug 5, 2013
@dragoljub
Copy link

Jeff this looks fantastic for writing tables. Does this enhancement also apply when appending to an existing table?

I cant wait to give this a shot once 0.13 is out. 👍

@jreback
Copy link
Contributor Author

jreback commented Nov 27, 2013

http://pandas.pydata.org/pandas-build/dev/

there are are binaries out (iirc you use windows)

this works for appending / writing (it's the same operation )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants