pandas.DataFrame.to_json gives OverflowError when attempting to serialize datetime.time objects

[jamescasbon]

datetime.time objects cause to_json to fail, presumably because they are attempting to be converted in the same way as datetime.datetime objects.

For example:

>>> pd.DataFrame(data=dict(a = [datetime.time()])).to_json()
...
OverflowError: Maximum recursion level reached

In contrast:

 >>> pd.DataFrame(data=dict(a = [datetime.datetime.now()])).to_json()
'{"a":{"0":1379498870147}}'

Tested on 0.12 and HEAD (b039c1d)

[jreback]

cc @Komnomnomnom

since datetime.time are just object dtype not sure exactly what you could do (as you don't want to specifiy test all kinds of cases), maybe just fall thru. in this case datetime.time does have an isoformat method though

[jamescasbon]

What is meant to happen for objects? I will convert mine to datetime since that makes sense. However, I think there could be a better exception than OverflowErrror.

Taking str(x) might be a sensible default for objects?

[jreback]

@jamescasbon AFAIK its a bit more complicated, because some objects actually can be seriealized, e.g.

In [8]: pd.DataFrame(dict(A = [ ['foo','bar' ]]))
Out[8]: 
            A
0  [foo, bar]

In [9]: pd.DataFrame(dict(A = [ ['foo','bar' ]])).to_json()
Out[9]: '{"A":{"0":["foo","bar"]}}'

In [10]: pd.DataFrame(dict(A = [ ['foo','bar' ]])).dtypes
Out[10]: 
A    object
dtype: object

[Komnomnomnom]

Actually times aren't being treated the same as datetimes at all, they are hitting the default fallback behaviour (inherited from ujson) which if an object is unsupported is as follows:

1. if the object has a toDict method use it
2. otherwise try to serialize it to a dict using dir(...)

In my experience, unless the object is extremely simple, number 2 is next to useless. This case is a good example, time objects have a timedelta attribute and you end up with endless recursion of fallback serialisation, hence the OverflowError complaining about max recursion depth.

I'm open to improving the situation and adding a better fallback mechanism, most likely a default parameter to provide a callable to use if an object can't otherwise be serialised (ala the built-in JSON lib). If provided this would then be used instead of the fallback to using dir(...). I'm wary of replacing the existing fallback with str() as that would break existing code that relies on the current behaviour, and I'd prefer it to be explicit anyway as str can do some unexpected things in some cases.

Note on the deserialisation end I probably wouldn't change anything, so your times would be deserialised to strings and you'd have to deal with them manually.

Any opinions on how best to handle this?

[jreback]

I think it might be worth it to handle datetime.date and datetime.time natively, with serialization to a string type. and deserialization the same (no round-trip).In addition might be nice to do a comprehensive test of all python types to see what if anything 'breaks'.

Your default scheme sounds fine, but let's see what breaks first.

[jreback]

@Komnomnomnom any action on this in 0.13? push? up2u

[Komnomnomnom]

I'll definitely get datetime.time handling in for 0.13.

If 0.13 is now in 20 days (I thought it was this week) then I'll hope to get better default handling in too. Unless you want to freeze the code at some point before then?

[jreback]

just trying to get final code in......if you could do this week would be gr8