Skip to content

ENH: Support writing timestamps with timezones with to_sql #22654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 36 commits into from
Nov 8, 2018
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
776240b
ENH: Write timezone columns to SQL
Sep 9, 2018
befd200
add tests and change type to Timestamp
Sep 10, 2018
e9f122f
Lint error and comment our skipif
Sep 10, 2018
969d2da
Handle DatetimeTZ block
Sep 10, 2018
cc79b90
Ensure the datetimetz data is 2D first
Sep 11, 2018
24dbaa5
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Sep 11, 2018
6e86d58
Reading timezones returns timezones in UTC
Sep 11, 2018
c7c4a7a
Add whatsnew and some touchups
Sep 12, 2018
6aa4878
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Sep 14, 2018
513bbc8
Test other dbs
Sep 14, 2018
58772e1
timestamps are actually returned as naive local for myself, sqlite
Sep 14, 2018
1a29148
localize -> tz_localize
Sep 14, 2018
96e9188
sqlite doesnt support date types
Sep 15, 2018
ded5584
type
Sep 15, 2018
d575089
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Sep 15, 2018
a7d1b3e
retest
Sep 15, 2018
305759c
read_table vs read_query sqlite difference
Sep 16, 2018
7a79531
Add note in the to_sql docs
Sep 19, 2018
24823f8
Modify whatsnew
Sep 19, 2018
7db4eaa
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Sep 19, 2018
76e46dc
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Sep 21, 2018
978a0d3
Address review
Sep 21, 2018
8025248
Fix sqlalchemy ref
Sep 21, 2018
0e89370
clarify documentation and whatsnew
Sep 26, 2018
bab5cfb
Add an api breaking entry change as well
Sep 27, 2018
de62788
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Oct 10, 2018
e940279
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Oct 24, 2018
8c754b5
Add new section in whatsnew
Oct 25, 2018
e85842f
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Oct 26, 2018
5af83f7
Fix whatsnew to reflect prior bug
Oct 26, 2018
6b3a3f1
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Nov 6, 2018
c4304ec
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Nov 7, 2018
1054fdb
handle case when column is datetimeindex
Nov 7, 2018
f21c755
Add new whatsnew entry
Nov 7, 2018
f872ff7
Merge remote-tracking branch 'upstream/master' into writing_timezone_sql
Nov 7, 2018
ef3b20f
don't check name
Nov 7, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 3 additions & 29 deletions doc/source/whatsnew/v0.24.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -462,35 +462,6 @@ that the dates have been converted to UTC

pd.to_datetime(["2015-11-18 15:30:00+05:30", "2015-11-18 16:30:00+06:30"], utc=True)

.. _whatsnew_0240.api_breaking.timezone_databases:

Writing Timezone Aware Data to Databases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

:meth:`to_sql` can now write timezone aware datetime data (``datetime64[ns, tz]`` dtype)
as ``TIMESTAMP WITH TIME ZONE`` type to databases that support that type. (:issue:`9086`)

However, databases that do not support timezones will now store timezone aware data
as naive timestamps in local time instead of naive timestamps in UTC.

Therefore, round-tripping timezone aware data from pandas to a database and back
to pandas has changed behavior. Given timezone aware data that has been
written to a database from pandas, the following table summarizes
the timestamps that will be returned.

+----------+--------------------------------+-----------------------------------+
| | Database with timezone support | Database without timezone support |
+==========+================================+===================================+
| Before | tz-naive in UTC time | tz-naive in UTC time |
+----------+--------------------------------+-----------------------------------+
| After | tz-aware in UTC time | tz-naive in local time |
+----------+--------------------------------+-----------------------------------+

Converting timezones read from ``TIMESTAMP WITH TIME ZONE`` types to UTC is
consistent with prior behavior.

See the :ref:`io.sql_datetime_data` for more information.

.. _whatsnew_0240.api_breaking.calendarday:

CalendarDay Offset
Expand Down Expand Up @@ -1275,6 +1246,9 @@ MultiIndex
I/O
^^^

- Bug in :meth:`to_sql` when writing timezone aware data (``datetime64[ns, tz]`` dtype) would raise a ``TypeError`` (:issue:`9086`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would move this to the "enhancements" though, I would say that timezones were simply never supported, so it is a nice enhancement that we will now actually support it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, now looking at the full diff (and not only what changed recently), and see you actually already have that. It's a bit duplicated now, but I am fine with keeping it in both places.

- Bug in :meth:`to_sql` where a naive DatetimeIndex would be written as ``TIMESTAMP WITH TIMEZONE`` type in supported databases, e.g. PostgreSQL (:issue:`23510`)

.. _whatsnew_0240.bug_fixes.nan_with_str_dtype:

Proper handling of `np.NaN` in a string data-typed column with the Python engine
Expand Down
9 changes: 7 additions & 2 deletions pandas/io/sql.py
Original file line number Diff line number Diff line change
Expand Up @@ -852,8 +852,13 @@ def _sqlalchemy_type(self, col):
if col_type == 'datetime64' or col_type == 'datetime':
# GH 9086: TIMESTAMP is the suggested type if the column contains
# timezone information
if col.dt.tz is not None:
return TIMESTAMP(timezone=True)
try:
if col.dt.tz is not None:
return TIMESTAMP(timezone=True)
except AttributeError:
# The column is actually a DatetimeIndex
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just for my understanding: where was this attribute error catched before?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoop, sorry, again misunderstanding from not looking at the full diff :-)

if col.tz is not None:
return TIMESTAMP(timezone=True)
return DateTime
if col_type == 'timedelta64':
warnings.warn("the 'timedelta' type is not supported, and will be "
Expand Down
11 changes: 11 additions & 0 deletions pandas/tests/io/test_sql.py
Original file line number Diff line number Diff line change
Expand Up @@ -1391,6 +1391,17 @@ def test_datetime_with_timezone_roundtrip(self):
result['A'] = to_datetime(result['A'])
tm.assert_frame_equal(result, expected)

def test_naive_datetimeindex_roundtrip(self):
# GH 23510
# Ensure that a naive DatetimeIndex isn't converted to UTC
dates = date_range('2018-01-01', periods=5, freq='6H')
expected = DataFrame({'nums': range(5)}, index=dates)
expected.to_sql('foo_table', self.conn, index_label='info_date')
result = sql.read_sql_table('foo_table', self.conn,
index_col='info_date')
# result index with gain a name from a set_index operation; expected
tm.assert_frame_equal(result, expected, check_names=False)

def test_date_parsing(self):
# No Parsing
df = sql.read_sql_table("types_test_data", self.conn)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we rather test that it is not parsed instead of removing it? (but I agree this currently looks like this is not doing much)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see. My cursory glance thought it was duplicate of the line below; you're right, I will try to add an assert to this result as well

Expand Down