to_json index with Null Value Broken in 1.0

[WillAyd]

Discovered by @ArtificialQualia in #31748

Previous behavior on 0.25.3:

>>> pd.Series([1], index=[np.nan]).to_json()
'{"null":1}'
>>> pd.Series([1], index=[pd.NaT]).to_json()
'{"null":1}'
>>> pd.Series([1], index=[None]).to_json()
'{"null":1}'

And now on master:

>>> pd.Series([1], index=[np.nan]).to_json()
'{"nan":1}'
>>> pd.Series([1], index=[pd.NaT]).to_json()
'{"null":1}'
>>> pd.Series([1], index=[None]).to_json()
'{"None":1}'

Note that pd.NA also just has its str representation written out:

>>> pd.Series([1], index=[pd.NA]).to_json()
'{"<NA>":1}'

pd.NaT still works because of this logic:

pandas/pandas/_libs/src/ujson/python/objToJSON.c

Line 1455 in 6d3cc14

if (nanosecVal == get_nat()) {

So probably need to generalize for other "null" values

[jbrockmendel]

on master the pd.NA case is now giving '{"null":1}'

@WillAyd what is the desired behavior here? do you have a gameplan for this?

[WillAyd]

I'm not actively working on this but if someone wanted to I think the best solution is to take the NA check while encoding object keys:

pandas/pandas/_libs/src/ujson/python/objToJSON.c

Line 1455 in 6d3cc14

if (nanosecVal == get_nat()) {

And move it up in the function so it gets hit generally. The get_nat function should probably be a generic null check, importable from the missing cython file

[simonjayhawkins]

Previous behavior on 0.25.3:

>>> pd.Series([1], index=[np.nan]).to_json()
'{"null":1}'
>>> pd.Series([1], index=[pd.NaT]).to_json()
'{"null":1}'
>>> pd.Series([1], index=[None]).to_json()
'{"null":1}'

d75ee70 is the first bad commit
commit d75ee70
Author: William Ayd william.ayd@gmail.com
Date: Sat Aug 24 00:38:17 2019 +0200

Remove Encoding of values in char** For Labels (#27618)

[simonjayhawkins]

on master the pd.NA case is now giving '{"null":1}'

this was fixed by #32214

35537dd is the first new commit
commit 35537dd
Author: Daniel Saxton 2658661+dsaxton@users.noreply.github.com
Date: Wed Feb 26 06:39:55 2020 -0600

BUG: Cast pd.NA to pd.NaT in to_datetime (#32214)

[simonjayhawkins]

pd.NaT still works because of this logic:

pandas/pandas/_libs/src/ujson/python/objToJSON.c

Line 1455 in 6d3cc14

if (nanosecVal == get_nat()) {

this was also initially broken by #27618

>>> import pandas as pd
>>>
>>> pd.__version__
'0.26.0.dev0+228.gd75ee703e'
>>>
>>> pd.Series([1], index=[pd.NaT]).to_json()
'{"-9223372036854":1}'
>>>

however, this was fixed in #30977

9c33464 is the first new commit
commit 9c33464
Author: William Ayd will_ayd@innobi.io
Date: Sat Jan 18 07:34:12 2020 -0800

JSON Date Handling 1.0 Regressions (#30977)