ENH: Iterate over HDF store hierarchically

[nileracecrew]

I am relatively new to pandas, but have been using raw HDF5 files for a long itme, so apologies if this feature or something equivalent already exists.

As stated in the documentation, it is possible to create hierarchies of data stores:

store.put('/first_group/df1', df1)
store.put('/first_group/df2', df2)
store.put('/second_group/df3', df3)

However, it seems that store.keys() can only provide a flattened list of all available groups. It would be nice to be able to walk the HDF5 file hierarchically. A couple examples (this is borrowing a bit from the h5py API):

# do stuff on each subgroup/substore in /first_group
for s in store['first_group'].values() :
    # do stuff on df1 and df2

# visit each group at the root level
for s in store.values() :
    if 'df1' in s :
        # do stuff on any store named 'df1'

The main use case I would have for something like this is that I have many dataframes with different schema but would like to group some of them together and then iterate through those groups.

Best way I can think to emulate this is to parse the strings in store.keys(), but that is pretty ugly.

[jreback]

wouldn't be opposed to adding a .walk() method, similar to how PyTables.Table.walk and os.walk works. You just iterate and yield each node, up to the user to do something with them.

[nileracecrew]

That would provide at least a basic hook for walking through the HDFStore.

Thinking more about my particular workflow, it might make sense instead to maintain a separate DataFrame with the paths to each DataFrame store, and some organizing metadata to query from.

[stephenpascoe]

I'd like to work on this during the EuroSciPy 2015 sprints #10877

[nileracecrew]

In case it's useful, I ended up using this construction to get all the
nodes at a particular location instead of walking the whole tree.

# simplified example from working code but not specifically tested so there may be typos

# added this method to pd.HDFStore
def iter_nodes(self, loc):
   return (n._v_name for n in self._handle.iter_nodes(loc))

# ...

with pd.HDFStore('mystore.h5') as store:
    for g in store.iter_nodes('/'):
        # do something to each node (group, dataset, etc) at the root level

[nileracecrew]

Related #6833

[stephenpascoe]

Created PR #10932