0.17 regression: convert_objects coercion with multiple dtypes

[AmbroseKeith]

I upgraded from 0.16.2 to 0.17/master and it broke working code.

0.16.2:

In [2]: import pandas as pd
   ...: from StringIO import StringIO
   ...: x="""foo,bar
   ...: 2015-09-14,True
   ...: 2015-09-15,
   ...: """
   ...: df=pd.read_csv(StringIO(x),sep=',').convert_objects('coerce')
   ...: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2 entries, 0 to 1
Data columns (total 2 columns):
foo    2 non-null datetime64[ns]
bar    1 non-null object
dtypes: datetime64[ns](1), object(1)
memory usage: 48.0+ bytes

0.17/master:

import pandas as pd
from StringIO import StringIO
x="""foo,bar
2015-09-14,True
2015-09-15,
"""
df=pd.read_csv(StringIO(x),sep=',').convert_objects('coerce')
df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2 entries, 0 to 1
Data columns (total 2 columns):
foo    2 non-null datetime64[ns]
bar    0 non-null datetime64[ns]
dtypes: datetime64[ns](2)
memory usage: 48.0 bytes
/home/ambk/work/pandas/pandas/core/generic.py:2584: FutureWarning: The use of 'coerce' as an input is deprecated. Instead set coerce=True.
  FutureWarning)

booleans get cast to datetimes now?! Usually deprecation means "avoid in new code" and not that your working code will break, otherwise it wouldn't be a deprecation but a breaking change. So that's not good, but ok, let's follow the helpful hint:

In [3]: import pandas as pd
   ...: from StringIO import StringIO
   ...: x="""foo,bar
   ...: 2015-09-14,True
   ...: 2015-09-15,
   ...: """
   ...: df=pd.read_csv(StringIO(x),sep=',').convert_objects(coerce=True)
   ...: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2 entries, 0 to 1
Data columns (total 2 columns):
foo    2 non-null object
bar    1 non-null float64
dtypes: float64(1), object(1)
memory usage: 48.0+ bytes

wut?, booleans become floats and almost-datetimes aren't converted at all anymore?

[TomAugspurger]

@AmbroseKeith thanks for the report. I changed to title to be more helpful than "utterly broken"

@jreback I think there's two bugs here. First the multiple dtypes, where the bools are getting mixed in with the Datetime block and getting coerced to NaN. At least that's my guess.

The last example, .convert_objects(coerce=True) shouldn't do anything right? Since datetime, numeric and timedelta are all False? btw @AmbroseKeith I don't think that does what you want (once we fix the bug). You want .convert_objects(datetime=True, coerce=True).

[jorisvandenbossche]

cc @bashtage

The original change occurred in PR #10265

[jreback]

This looks correct to me. You asked it to take an object column and force convert. Not sure what else you would return here. Bools are nothing special.

In [17]: df.convert_objects(coerce=True)
Out[17]: 
          foo  bar
0  2015-09-14    1
1  2015-09-15  NaN

df.convert_objects('coerce') could be caught as 'coerce' to datetime is invalid (though IIRC this was accept to provide back-compat).

In general expecting force converting to work always is simply user-error. It is impossible to guess things reliably (and this is why this was changed). If you have dates, use pd.to_datetime. .convert_objects has really a simple purpose, and that is to force convert mixed values in a known type.

Applying this on a DataFrame is quite dangerous and requires much guessing. Not sure what we could do about this.

[jreback]

Though I agree with @TomAugspurger

df.convert_objects(coerce=True) should raise a ValueError (and not just return the original frame). as this is a specification error.

[jorisvandenbossche]

In any case, the whatsnew entry will have to be much more explicit about these changes, as this has a big impact on the result if you use convert_objects on whole frames.

I think we should think about if it is possible to implement the new behaviour without breaking the existing usage.

[bashtage]

Trying to handle the legacy usage was a bad fit given the substantial change int he behavior of convert objects - it isn't just a keyword renaming, but a removal of the magic of the old version.

The examples in the original post are all correct as intended, although I agree that coerce=True while all of datetime==timedelta==numeric=False should raise since it doesn't make any sense.