Error while deleting/dropping a series/dataframe entry with timezone aware timestamps.

[elboerto]

Code Sample, a copy-pastable example if possible

import pandas as pd
start = pd.Timestamp('20170101', tz='Europe/Berlin')
end = pd.Timestamp('20180101', tz='Europe/Berlin')
index = pd.date_range(start, end, freq='15min')
data = [1 for x in range(len(index))]
series = pd.Series(index=index, data=data)
dataframe = pd.DataFrame(series)
timestamp = pd.Timestamp('201701011515', tz='Europe/Berlin')

# Working
series[timestamp]
dataframe.loc[timestamp]

# Errors
del series[timestamp]
dataframe.drop(timestamp)
series.drop(timestamp)

Problem description

It not possible to delete/drop a series/dataframe entry with timezone aware timestamps, while accessing them is possible.

Expected Output

Deletion of the respective entry.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.4.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 142 Stepping 9, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None pandas: 0.23.1 pytest: 3.6.2 pip: 10.0.1 setuptools: 39.2.0 Cython: None numpy: 1.14.5 scipy: 1.1.0 pyarrow: None xarray: None IPython: 6.4.0 sphinx: None patsy: None dateutil: 2.7.3 pytz: 2018.5 blosc: None bottleneck: None tables: None numexpr: None feather: None matplotlib: 2.2.2 openpyxl: None xlrd: None xlwt: None xlsxwriter: None lxml: None bs4: None html5lib: None sqlalchemy: None pymysql: None psycopg2: None jinja2: None s3fs: None fastparquet: None pandas_gbq: None pandas_datareader: None

[gfyoung]

@elboerto : Can you try your code against master ? I can't seem to replicate this there.

[gnikonorov]

I was able to reproduce this issue with master's code. I have just installed pandas for development with anaconda as outlined here and received the following output when I ran the code:

(pandas-dev) ✔ ~/projects/pandas-gnikonorov/test [issue_21761|…1]
18:58 $ cat test_21761.py
import pandas as pd
start = pd.Timestamp('20170101', tz='Europe/Berlin')
end = pd.Timestamp('20180101', tz='Europe/Berlin')
index = pd.date_range(start, end, freq='15min')
data = [1 for x in range(len(index))]
series = pd.Series(index=index, data=data)
dataframe = pd.DataFrame(series)
timestamp = pd.Timestamp('201701011515', tz='Europe/Berlin')

# Working
series[timestamp]
dataframe.loc[timestamp]

# Errors
del series[timestamp]
dataframe.drop(timestamp)
series.drop(timestamp)
(pandas-dev) ✔ ~/projects/pandas-gnikonorov/test [issue_21761|…1]
18:58 $ python test_21761.py
Traceback (most recent call last):
  File "test_21761.py", line 15, in <module>
    del series[timestamp]
  File "/home/gnikonorov/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/pandas/core/generic.py", line 2740, in __delitem__
    self._data.delete(key)
  File "/home/gnikonorov/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/pandas/core/internals.py", line 4785, in delete
    self.axes[0] = self.axes[0].delete(loc)
  File "/home/gnikonorov/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 2225, in delete
    return DatetimeIndex(new_dates, name=self.name, freq=freq, tz=self.tz)
  File "/home/gnikonorov/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 445, in __new__
    ambiguous=ambiguous)
  File "pandas/_libs/tslibs/conversion.pyx", line 977, in pandas._libs.tslibs.conversion.tz_localize_to_utc
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from '2017-10-29 02:00:00', try using the 'ambiguous' argument
(pandas-dev) ✘-1 ~/projects/pandas-gnikonorov/test [issue_21761|…1]
18:58 $

I would like to attempt to fix this issue, if possible.

[gfyoung]

@gnikonorov : As I said earlier, I am (un)fortunately unable to replicate those errors. What version of pandas are you using i.e. what is your output when you run:

import pandas as pd
pd.__version__

[gnikonorov]

@gfyoung, I am running v0.24.0.dev0+243.g30eb48c.

(pandas-dev) ✘-1 ~/projects/pandas-gnikonorov [issue_21761|…2]
20:23 $ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> print(pandas.__version__)
0.24.0.dev0+243.g30eb48c
>>>

[gfyoung]

@gnikonorov : Hmmm...that's quite weird.

cc @jreback @jorisvandenbossche @TomAugspurger : Are any of you able to replicate this by any chance?

[elboerto]

Version

>>> import pandas
>>> print(pandas.__version__)
0.24.0.dev0+237.g3273309d2

Output:

(pandas-dev) C:\Users\Robert\Documents\Github\pandas-elboerto>python test/test_21761.py
Traceback (most recent call last):
  File "test/test_21761.py", line 15, in <module>
    del series[timestamp]
  File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\generic.py", line 2740, in __delitem__
    self._data.delete(key)
  File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\internals.py", line 4785, in delete
    self.axes[0] = self.axes[0].delete(loc)
  File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\indexes\datetimes.py", line 2225, in delete
    return DatetimeIndex(new_dates, name=self.name, freq=freq, tz=self.tz)
  File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\indexes\datetimes.py", line 445, in __new__
    ambiguous=ambiguous)
  File "pandas/_libs/tslibs/conversion.pyx", line 977, in pandas._libs.tslibs.conversion.tz_localize_to_utc
    elif is_dst:
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from '2017-10-29 02:00:00', try using the 'ambiguous' argument
``

[TomAugspurger]

I'm also able to reproduce.

…

On Mon, Jul 9, 2018 at 3:15 AM, elboerto ***@***.***> wrote: Version >>> import pandas>>> print(pandas.__version__)0.24.0.dev0+237.g3273309d2 Output: (pandas-dev) C:\Users\Robert\Documents\Github\pandas-elboerto>python test/test_21761.py Traceback (most recent call last): File "test/test_21761.py", line 15, in <module> del series[timestamp] File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\generic.py", line 2740, in __delitem__ self._data.delete(key) File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\internals.py", line 4785, in delete self.axes[0] = self.axes[0].delete(loc) File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\indexes\datetimes.py", line 2225, in delete return DatetimeIndex(new_dates, name=self.name, freq=freq, tz=self.tz) File "C:\Users\Robert\AppData\Local\conda\conda\envs\pandas-dev\lib\site-packages\pandas\core\indexes\datetimes.py", line 445, in __new__ ambiguous=ambiguous) File "pandas/_libs/tslibs/conversion.pyx", line 977, in pandas._libs.tslibs.conversion.tz_localize_to_utc elif is_dst: pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from '2017-10-29 02:00:00', try using the 'ambiguous' argument `` — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21761 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIuZGsGc7RYVmuV2FvcZzB5yz0goMks5uExEWgaJpZM4VFMTk> .