The error when as_datetime acts on multiple timezones is not useful

[dougthor42]

Code Sample, a copy-pastable example if possible

s = pd.Series([
    'nan',
    pd.Timestamp("1990-01-01"),
    "2015-03-14T16:15:14.123-08:00",
    "2019-03-04T21:56:32.620-07:00",         # Note different TimeZone
    None,
])
print(s)
#0                              nan
#1              1990-01-01 00:00:00
#2    2015-03-14T16:15:14.123-08:00
#3    2019-03-04T21:56:32.620-07:00
#4                             None
#dtype: object
pd.to_datetime(s)
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 592, in to_datetime
#    values = convert_listlike(arg._values, True, format)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 302, in _convert_listlike_datetimes
#    allow_object=True)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 1857, in objects_to_datetime64ns
#    require_iso8601=require_iso8601
#  File "pandas/_libs/tslib.pyx", line 460, in pandas._libs.tslib.array_to_datetime
#  File "pandas/_libs/tslib.pyx", line 712, in pandas._libs.tslib.array_to_datetime
#  File "pandas/_libs/tslib.pyx", line 813, in pandas._libs.tslib.array_to_datetime_object
#RuntimeError: No active exception to reraise

Problem description

Throwing RuntimeError does not give the user any feedback on how to fix things. The correct way to fix this is to add utc=True:

pd.to_datetime(s, utc=True)
#0                                NaT
#1          1990-01-01 00:00:00+00:00
#2   2015-03-15 00:15:14.123000+00:00
#3   2019-03-05 04:56:32.620000+00:00
#4                                NaT
#dtype: datetime64[ns, UTC]

Expected Output

The error message should be more descriptive, perhaps something like:

ValueError: array must be all same time zone

The above is the same error provided by pandas._libs.tslibs.conversion.datetime_to_datetime64 and can be seen using this example:

import numpy as np
import pandas as pd
df = pd.DataFrame([['1', "2019-04-03T16:13:27.123-08:00"], ['2', "2019-01-01T13:45:46.424-07:00"]])dd
print(df.types)
#0    object
#1    object
#dtype: object
df[1].astype(np.datetime64)
#Traceback (most recent call last):
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 1861, in objects_to_datetime64ns
#    values, tz_parsed = conversion.datetime_to_datetime64(data)
#  File "pandas/_libs/tslibs/conversion.pyx", line 185, in pandas._libs.tslibs.conversion.datetime_to_datetime64
#ValueError: Array must be all same time zone
#
#During handling of the above exception, another exception occurred:
#
#Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/generic.py", line 5691, in astype
#    **kwargs)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 531, in astype
#    return self.apply('astype', dtype=dtype, **kwargs)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 395, in apply
#    applied = getattr(b, f)(**kwargs)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 534, in astype
#    **kwargs)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 633, in _astype
#    values = astype_nansafe(values.ravel(), dtype, copy=True)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 690, in astype_nansafe
#    return astype_nansafe(to_datetime(arr).values, dtype, copy=copy)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/dtypes/cast.py", line 690, in astype_nansafe
#    return astype_nansafe(to_datetime(arr).values, dtype, copy=copy)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 609, in to_datetime
#    result = convert_listlike(arg, box, format)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 302, in _convert_listlike_datetimes
#    allow_object=True)
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 1866, in objects_to_datetime64ns
#    raise e
#  File "/c/gitlab/tph-admin/gitlab-activity-monitor/.venv-gam/lib/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 1857, in objects_to_datetime64ns
#    require_iso8601=require_iso8601
#  File "pandas/_libs/tslib.pyx", line 460, in pandas._libs.tslib.array_to_datetime
#  File "pandas/_libs/tslib.pyx", line 537, in pandas._libs.tslib.array_to_datetime
#ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

Output of pd.show_versions()

>>> import pandas as pd
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-17763-Microsoft
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: None
pip: 19.0.3
setuptools: 40.9.0
Cython: None
numpy: 1.16.2
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[mroeschke]

So the presence of pd.Timestamp("1990-01-01") causes this error to raise.

In [3]: s = pd.Series([
   ...:     'nan',
   ...:     "2015-03-14T16:15:14.123-08:00",
   ...:     "2019-03-04T21:56:32.620-07:00",         # Note different TimeZone
   ...:     None,
   ...: ])

In [4]: pd.to_datetime(s)
Out[4]:
0                                 NaT
1    2015-03-14 16:15:14.123000-08:00
2    2019-03-04 21:56:32.620000-07:00
3                                None
dtype: object

This parsing goes down the array_to_datetime_object path where it assumes all incoming objects are string-like or null-like. In this case, datetime-like objects should be let through since that the expected return type, so I'd call this a bug.

[mroeschke]

Actually I agree with your assessment @dougthor42, this should raise the ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True error.

[dougthor42]

So the presence of pd.Timestamp("1990-01-01") causes this error to raise.

Good catch! Though I could have sworn my testing didn't include that... ah well!

Glad to hear that it's an actionable item.