Strange error message when summing datetime64 and datetime.time column

[jorisvandenbossche]

You get TypeError: Argument 'values' has incorrect type (expected numpy.ndarray, got Series):

In [29]: df = pd.DataFrame({'date':pd.date_range('2012-01-01', periods=3), 'time':[datetime.time(i, i, i) for i in range(3)]})

In [30]: df
Out[30]:
        date      time
0 2012-01-01  00:00:00
1 2012-01-02  01:01:01
2 2012-01-03  02:02:02

In [31]: df['date'] + df['time']
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-31-5101228e303e> in <module>()
----> 1 df['date'] + df['time']

C:\Anaconda\lib\site-packages\pandas\core\ops.pyc in wrapper(left, right, name)
    491             return NotImplemented
    492
--> 493         time_converted = _TimeOp.maybe_convert_for_time_op(left, right,
name)
    494
    495         if time_converted is None:

C:\Anaconda\lib\site-packages\pandas\core\ops.pyc in maybe_convert_for_time_op(c
ls, left, right, name)
    455         if name.startswith('__r'):
    456             name = "__" + name[3:]
--> 457         return cls(left, right, name)
    458
    459

C:\Anaconda\lib\site-packages\pandas\core\ops.pyc in __init__(self, left, right,
 name)
    272         self.right = right
    273         lvalues = self._convert_to_array(left, name=name)
--> 274         rvalues = self._convert_to_array(right, name=name, other=lvalues
)
    275
    276         self.is_timedelta_lhs = com.is_timedelta64_dtype(left)

C:\Anaconda\lib\site-packages\pandas\core\ops.pyc in _convert_to_array(self, val
ues, name, other)
    354             elif not (isinstance(values, (np.ndarray, pd.Series)) and
    355                       com.is_datetime64_dtype(values)):
--> 356                 values = tslib.array_to_datetime(values)
    357         elif inferred_type in ('timedelta', 'timedelta64'):
    358             # have a timedelta, convert to to ns here

TypeError: Argument 'values' has incorrect type (expected numpy.ndarray, got Series)

[jreback]

xref to #8698

[jorisvandenbossche]

@jreback it is indeed somewhat related in the sense that I should have converted it to a timedelta, but first didn't and got this error.
But for #8698, this will always be done explicitly by the user? And not automatically in something like df['date'] + df['time']. So, summing those should still generate a (more useful) error

[jreback]

well I think I tried to make this happen automatically and ran into a road-block, but don't remember exactly what the problem (aside from the pretty useless tz attribute in datetime.time). To be honest I think this is a completely useless type and can completely and transparently converted to Timedelta. I think it was python's attempt to make an offset like object.

[jorisvandenbossche]

well, this discussion is more related to #8343 then, I think.
But although it seems pretty useless, it exists and people do use it (you eg also can get it when reading in data from certain sources such as SQL and excel), so I don't think it is a good idea to convert it automatically in a Timedelta, since it has some different semantics

[jorisvandenbossche]

@jreback What I wanted to report here: not that this operation should work, but that the error message should be more clear, something in the sense of Operation '+' not supported for datetime64 and object/time

[jreback]

@jorisvandenbossche ok, I suppose it could raise the default message
TypeError: incompatible type [object] for a datetime/timedelta operation

[jorisvandenbossche]

yes, that is indeed what you get with other object typed columns

[jreback]

I think since datetime.time is a datetime sub-class it probably gets a bit further in the chain and actually tries to do the operation, but datetime.time doesn't know how to interoperate with a np.datetime64 so it fails. So in this case it could actually work w/o converting to a Timedelta. But would need some handling.

But if someone actually did this of course datetime.time are object dtype (for the foreseable future)