Cannot subtract tz-aware datetime.datetime from tz-aware datetime64 series.

[RobertasA]

Code Sample, a copy-pastable example if possible

import pytz
import datetime
import pandas as pd

foo = pd.Series(datetime.datetime(2016, 8, 23, 12, tzinfo=pytz.utc))
foo - datetime.datetime(2016, 8, 22, 12, tzinfo=pytz.utc)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-52-0639980e7d31> in <module>()
----> 1 foo - datetime.datetime(2016, 8, 1, 12, tzinfo=pytz.utc)

/Users/charon/.virtualenvs/iwoca-django/lib/python2.7/site-packages/pandas/core/ops.pyc in wrapper(left, right, name, na_op)
    607
    608         time_converted = _TimeOp.maybe_convert_for_time_op(left, right, name,
--> 609                                                            na_op)
    610
    611         if time_converted is None:

/Users/charon/.virtualenvs/iwoca-django/lib/python2.7/site-packages/pandas/core/ops.pyc in maybe_convert_for_time_op(cls, left, right, name, na_op)
    567             return None
    568
--> 569         return cls(left, right, name, na_op)
    570
    571

/Users/charon/.virtualenvs/iwoca-django/lib/python2.7/site-packages/pandas/core/ops.pyc in __init__(self, left, right, name, na_op)
    281
    282         lvalues = self._convert_to_array(left, name=name)
--> 283         rvalues = self._convert_to_array(right, name=name, other=lvalues)
    284
    285         self.name = name

/Users/charon/.virtualenvs/iwoca-django/lib/python2.7/site-packages/pandas/core/ops.pyc in _convert_to_array(self, values, name, other)
    419             elif not (isinstance(values, (np.ndarray, ABCSeries)) and
    420                       is_datetime64_dtype(values)):
--> 421                 values = tslib.array_to_datetime(values)
    422         elif inferred_type in ('timedelta', 'timedelta64'):
    423             # have a timedelta, convert to to ns here

/Users/charon/.virtualenvs/iwoca-django/lib/python2.7/site-packages/pandas/tslib.so in pandas.tslib.array_to_datetime (pandas/tslib.c:41972)()

/Users/charon/.virtualenvs/iwoca-django/lib/python2.7/site-packages/pandas/tslib.so in pandas.tslib.array_to_datetime (pandas/tslib.c:38943)()

ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

Expected Output

0   1 days
dtype: timedelta64[ns]

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Darwin
OS-release: 15.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8

pandas: 0.18.1
nose: 1.3.4
pip: 8.1.2
setuptools: 3.6
Cython: 0.23.2
numpy: 1.11.1
scipy: 0.14.0
statsmodels: 0.6.1
xarray: None
IPython: 2.3.1
sphinx: None
patsy: 0.4.1
dateutil: 2.3
pytz: 2014.10
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: 3.4.1
bs4: None
html5lib: 0.999
httplib2: 0.9
apiclient: 1.1
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.7.3
boto: 2.26.0
pandas_datareader: None

Note, doing

foo.astype(datetime.datetime) - datetime.datetime(2016, 8, 22, tzinfo=pytz.utc)

or

foo.astype(datetime.datetime) - pd.Timestamp(datetime.datetime(2016, 8, 22, tzinfo=pytz.utc)

works.

[jreback]

so in master only [18] fails. not getting coerced to Timestamp first. pull-requests welcome.

In [12]: foo = pd.Series(datetime.datetime(2016, 8, 23, 12, tzinfo=pytz.utc))
In [13]: foo
Out[13]:
0   2016-08-23 12:00:00+00:00
dtype: datetime64[ns, UTC]

In [14]: dt = datetime.datetime(2016, 8, 22, 12, tzinfo=pytz.utc)

In [15]: dt
Out[15]: datetime.datetime(2016, 8, 22, 12, 0, tzinfo=<UTC>)

In [16]: foo
Out[16]:
0   2016-08-23 12:00:00+00:00
dtype: datetime64[ns, UTC]

In [17]: foo-pd.Timestamp(dt)
Out[17]:
0   1 days
dtype: timedelta64[ns]

In [18]: foo-dt
ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

note that .astype(datetime.datetime) is quite non-idiomatic as you are coercing to an object array.

[shoyer]

.astype(datetime.datetime) should be a TypeError?