Interface mismatch between fastparquet and pyarrow for reading and writing Parquet

[malthe]

Problem description

The pandas.read_parquet and pandas.to_parquet methods defer operation to either pyarrow (first priority) or fastparquet. The issue is that these libraries have a slightly different interface:

For reading:

• pyarrow accepts IOBase and not bytes
• fastparquet accepts bytes and not IOBase

Pandas should support one or both and do the conversion automatically.

For writing:

• pyarrow accepts IOBase
• fastparquet doesn't really support writing to an ephemeral buffer because the stream is closed when using the open_with argument (see FR: Accept a file-like object in addition to a path in fastparquet.write dask/fastparquet#408)

[TomAugspurger]

Is this something to be done in pandas? Or should the backends standardize?

[mroeschke]

Thanks for the report, but agreed that I think this is a difference in the backends so closing