Description
Example code
>>> df = pd.DataFrame(
... {'foo':[
... np.nan,
... np.inf,
... 1.1,
... 2.2
... ],
... 'bar':[
... 1,
... -np.inf,
... 2.1,
... 3.2
... ]
... })
>>> df
foo bar
0 NaN 1.0
1 inf -inf
2 1.1 2.1
3 2.2 3.2
>>> df.replace({np.nan: None, -np.inf:None, np.inf:None})
foo bar
0 NaN 1.0
1 NaN NaN
2 1.1 2.1
3 2.2 3.2
>>>
Problem description
When trying to replace all occurrences of NaN
s and Inf
s to None
s the resulting dataframe simply contains 'NaNs' for every occurrence, retaining the dtype of float64
rather than the expected object
. A work around is to simply issue the replace again with only np.nan
to None
.
This behavior was discovered when trying to convert the contents of a dataframe containing Inf
s, and NaN
s to None
s in preparation for JSON compliant output via a dictionary for compatibility with third party tools rather than directly through panda's to_json() method (which admittedly would make this process unnecessary as it produces the compliant output already)
I strongly suspect this problem is related to inconsistencies outlined here: #29024
Expected Output
I would expect a column type of object
with None
s in place as I see when replacing only NaNs.
Output of pd.show_versions()
[paste the output of pd.show_versions()
here below this line]
INSTALLED VERSIONS
commit : None
python : 3.8.1.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.13-arch1-1
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.0.0rc0+233.gec0996c675
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3
setuptools : 44.0.0
Cython : 0.29.14
pytest : None
hypothesis : None
sphinx : 2.2.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.4.2
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.11.1
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.4.2
matplotlib : 3.1.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : 1.3.12
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None