read_hdf is much slower than it should be

[gdementen]

>>> from larray_eurostat import *
>>> arr = eurostat_get('migr_pop3ctb')
>>> arr.to_hdf('test_arr.h5', 'arr')
>>> import pandas as pd
>>> %timeit pd.read_hdf('test_arr.h5', 'arr')
114 ms ± 8.29 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>> %timeit read_hdf('test_arr.h5', 'arr')
1.24 s ± 35.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The problem is mostly in inout.array.cartesian_product_df which is very expensive while it should do nothing for .h5 arrays saved from an LArray (it was already "densified").

>>> from larray.inout.array import cartesian_product_df
>>> df = pd.read_hdf('test_arr.h5', 'arr')
>>> %timeit cartesian_product_df(df)
968 ms ± 20.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

>>> from larray.inout.array import df_labels
>>> %timeit labels = df_labels(df, False)
288 ms ± 3.85 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
>>> labels = df_labels(df, False)
>>> from itertools import product
>>> %timeit new_index = pd.MultiIndex.from_tuples(list(product(*labels)))
440 ms ± 3.83 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
>>> new_index = pd.MultiIndex.from_tuples(list(product(*labels)))
>>> columns = list(df.columns)
>>> %timeit np.array_equal(df.index.values, new_index.values)
23.3 ms ± 638 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

I think I had already fixed this (or at least partially) in the very old pandasbased3 branch.

[gdementen]

There are several ways to approach this problem. One way is to make cartesian_product_df faster (which is what I did in the pandasbased3 branch IF I recall correctly).

But there are two more options. I have some experimental (but working) code (in one of my branches 😉) which add a marker in the HDF file when we write it via larray so that we can bypass cartesian_product_df when loading it back as we know it is "dense" already. This is already much faster. But I just thought we could go even faster by bypassing Pandas altogether (using pytables directly) to read and write HDF files. It would be a bit more work (it would still be very reasonable though) because we could also bypass index_to_labels in that case which is costly too.

[alixdamman]

I agree you should bypass pandas and use pytables directly. My question is how to implement something that is not backward incompatible with our current pandas version?

IMO, you should first keep pandas and add a tag in the HDF file to bypass or not cartesian_product_df so that users can have a version of read_hdf a bit faster quickly. Then, we could move to implement a read/to_hdf based on pytables.