HDF5 (PyTables) fsync support

[benalexau]

I'm using Pandas HDF5 files on a shared network file system with a distributed lock manager (based on Kazoo) to ensure a single writer at a time.

When there is significant lock contention, a rapid series of flush, close, lock release and lock acquisition in a different process will cause corruption to the HDF5 file. This appears to be due to the operating system cache.

This is resolved by:

store = pd.HDFStore(filename, complevel=9, complib='blosc')
f_fd = store._handle.fileno()
store.flush()
os.fsync(f_fd)
store.close()

However this requires using _handle, which may be removed in the future.

Would you please either expose _handle for supported end user use, or the fileno, or perhaps modify flush() to accept a fsync=True parameter to support this use case. Thanks.

[jtratner]

so you're saying that simply adding an fsync=True kwarg that causes close to be:

f_fd = store._handle.fileno()
store.flush()
os.fsync(f_fd)
store.close()

Is enough to handle this? Is there any reason to not always do fsync first? (i.e., perf penalties or undesirable side effects)

[jreback]

easy to make an option (or u can simply use _handle; it's just private to avoid name collisions; we can expose this as well)

but shouldn't be the default
http://stackoverflow.com/questions/2333872/atomic-writing-to-file-with-python

[jreback]

I think allowing a kw arg would be fine

want to do a PR?

[benalexau]

I think it best to avoid fsync unless the user requests it, as there are scenarios where they might want to flush, close, and rapidly re-open without a fsync (eg a user relying on thread synchronisation in a single process to achieve a single writer at a time, as in that case the shared OS buffer would be reliable without fsync but it would still be necessary to flush, close and re-open as quickly as possible).

I'll make a PR later today.

[jreback]

gr8 thxs

[jreback]

closed by #5369