StringDtype not preserved when writing + reading parquet or feather

[gerritholl]

Code Sample, a copy-pastable example if possible

import tempfile
import pandas
df1 = pandas.DataFrame(
        ["a", "b", "c"], dtype=pandas.StringDtype(), columns=["test"])
with tempfile.NamedTemporaryFile() as fn:
    df1.to_parquet(fn)
    df2 = pandas.read_parquet(fn)
print(df1.equals(df2))
print(df1.dtypes)
print(df2.dtypes)

Problem description

When writing a DataFrame to parquet or feather, then reading the same file, columns with dtype StringDtype() are not preserved but read as object dtype.

The code above gives:

False
test    string
dtype: object
test    object
dtype: object

I have tested with with pandas 1.0.0 and the latest pandas git master. When using parquet or feather, the dtype is lost. When using pickle, it is preserved. HDF5 is not supported on writing (see #31199).

I'm not sure how this should be solved. At least, there should be a warning when attempting to write something that won't be preserved upon reading. Perhaps this should be an error (exception) unless an overwrite-flag is passed.

Expected Output

Ideally, I would expect to get the same as for pickle:

True
test    string
dtype: object
test    string
dtype: object

Output of pd.show_versions()

False
test string
dtype: object
test object
dtype: object

INSTALLED VERSIONS

commit : None
python : 3.8.1.final.0
python-bits : 64
OS : Linux
OS-release : 4.12.14-lp150.12.82-default
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 1.1.0.dev0+241.g761bceb77
numpy : 1.17.5
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.1.0.post20200119
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.0
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None
None

[TomAugspurger]

Thanks for the report. See #29752, and #31242 which is working on this.

[gerritholl]

Oops, I didn't find that one, sorry (it's hard with so many issues).

[jorisvandenbossche]

This should actually already work (the linked PR #31242 is for when you didn't write the data yourself), but you need pyarrow 0.16 for that, which will only be released in a few days.