DataFrame.to_dict(orient='records') numeric inconsistency

[JohanKahrstrom]

Code Sample, a copy-pastable example if possible

Similar to #11247, but related to numerics. If there are any object columns in the DataFrame, then numerical columns are turned into native types, whereas if all columns are numpy numerics, they stay that way:

import pandas as pd

df = pd.DataFrame({
	'a': [1, 2, 3],
	'b': [1.0, 2.0, 3.0],
	'c': ['X', 'Y', 'Z']
})

dict_with_obj = df.to_dict('records')
print(f"Output types when an object column is present:")
print((type(dict_with_obj[0]['a']), type(dict_with_obj[0]['b'])))

dict_no_obj = df.drop(columns=['c']).to_dict('records')
print(f"Output types when no object column is present:")
print((type(dict_no_obj[0]['a']), type(dict_no_obj[0]['b'])))

Output:

Output types when an object column is present:
(<class 'int'>, <class 'float'>)
Output types when no object column is present:
(<class 'numpy.float64'>, <class 'numpy.float64'>)

Problem description

I need to combine the output dictionary with other dictionaries, and then convert it to JSON (using ujson.dumps). Since ujson cannot handle NumPy types, it crashes. Since I need to post process the output of to_dict before converting to JSON unfortunately df.to_json is not an option.

Expected Output

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.7.0.final.0
python-bits: 64
OS: Darwin
OS-release: 17.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8

pandas: 0.23.3
pytest: None
pip: 18.0
setuptools: 40.0.0
Cython: 0.28.5
numpy: 1.15.0
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[chris-b1]

ref to #13468, probably some other open issues - we should always be converting to native types.

As a workaround, iterating/constructing row-by-row seems to work.

dict_no_obj2 = [x.to_dict() 
                for _, x in df.drop(columns=['c']).iterrows()]

print((type(dict_no_obj2[0]['a']), type(dict_no_obj2[0]['b'])))

# (<class 'float'>, <class 'float'>)

[chris-b1]

ref #21256, likely same underlying issue

[mroeschke]

Looks fixed on master. Could use a test.

In [23]: df = pd.DataFrame({
    ...: ^I'a': [1, 2, 3],
    ...: ^I'b': [1.0, 2.0, 3.0],
    ...: ^I'c': ['X', 'Y', 'Z']
    ...: })
    ...:
    ...: dict_with_obj = df.to_dict('records')

In [24]: (type(dict_with_obj[0]['a']), type(dict_with_obj[0]['b']))
Out[24]: (int, float)

In [25]: dict_no_obj = df.drop(columns=['c']).to_dict('records')
    ...:

In [26]: (type(dict_no_obj[0]['a']), type(dict_no_obj[0]['b']))
Out[26]: (int, float)

In [27]: pd.__version__
Out[27]: '0.26.0.dev0+734.g0de99558b'