ENH: support zoneinfo tzinfo objects

[jbrockmendel]

ATM we support pytz, dateutil, and datetime.timezone tzinfos. Now that it is part of the stdlib py39, we should also support zoneinfo tzinfos.

[AlexKirko]

Congratulations to @pganssle for passing the PEP for this. Implementing the support in Timestamp and covering it in tests is going to be a PITA, I think, but it definitely needs to be done.

[pganssle]

Just pointing out - you shouldn't have to "support" it, and you wouldn't have to do a single thing if y'all had listened to me for the past 5 years when I said you shouldn't be digging into the internals of your time zone providers.

Whatever work is necessary for you to use the public tzinfo interface is worth it and long overdue.

[AlexKirko]

Yeah, I might be wrong, but I also don't think the current approach of querying the internals is going to work well here. I don't think there is any decent way forward except ditch that and move to only using public interfaces.

[jbrockmendel]

Whatever work is necessary for you to use the public tzinfo interface is worth it and long overdue.

There's a lot of upside to this. The question is if we can do it without a major perf loss.

[pganssle]

There's a lot of upside to this. The question is if we can do it without a major perf loss.

I guess it doesn't matter? It's harder to maintain what you have now than it would be to re-implement time zone handling from scratch in a performant way (also something I've suggested in the past as a workable solution).

Won't be long before people start complaining about the deprecation warnings from pytz, and I plan to revamp dateutil's internals to be a lot more like zoneinfo's, so you're about to start hitting issues with that route as well. I will not be issuing deprecation warnings for internal refactoring like I did last time. Pinning dateutil would be problematic since the current version doesn't work with "slim" binaries, which are becoming more commonly deployed.

[mroeschke]

It would be nice if we could coincide the long overdue timezone refactor with defaulting to the standard library timezones objects instead of pytz: #34916

[pganssle]

@mroeschke I would keep those separate. There are backwards-compatibility concerns with removing pytz as the default that aren't present with just adding support for using the public tzinfo interface.

Best to do the refactor first, then drop pytz as the default.

[AlexKirko]

@jorisvandenbossche said in #34916 that refactoring Timestamp to handle all tzinfo implementations through a public interface would be a breaking change (and, necessarily, a 2.0 thing). Haven't dug deep into the Timestamp implementation in a while, but it's not obvious to me. Could someone comment?

[jbrockmendel]

said in #34916 that refactoring Timestamp to handle all tzinfo implementations through a public interface would be a breaking change (and, necessarily, a 2.0 thing). Haven't dug deep into the Timestamp implementation in a while, but it's not obvious to me. Could someone comment?

The breaking change in #34916 is about what type of tzinfo objects we use when presented with a string like "US/Eastern" or "UTC". ATM we default to pytz; i opended #34916 because i think we should default to stdlib/zoneinfo instead. This issue is about supporting zoneinfo objects, e.g. ATM:

>>> tz = zoneinfo.ZoneInfo("US/Pacific")
>>> ts = pd.Timestamp.now(tz)
>>> ts.tz_localize(None)
Traceback (most recent call last):
  File "pandas/_libs/tslibs/timezones.pyx", line 246, in pandas._libs.tslibs.timezones.get_dst_info
    num = int(get_utcoffset(tz, None).total_seconds()) * 1_000_000_000
AttributeError: 'NoneType' object has no attribute 'total_seconds'
Exception ignored in: 'pandas._libs.tslibs.tzconversion.tz_convert_from_utc_single'
Traceback (most recent call last):
  File "pandas/_libs/tslibs/timezones.pyx", line 246, in pandas._libs.tslibs.timezones.get_dst_info
    num = int(get_utcoffset(tz, None).total_seconds()) * 1_000_000_000
AttributeError: 'NoneType' object has no attribute 'total_seconds'
Timestamp('1970-01-01 00:00:00')

Most or all of the relevant code is going to be in tslibs.timezones and tslibs.tzconversion.

We should be able to support zoneinfo using only the public API, then can measure perf and evaluate options.

[AlexKirko]

@jbrockmendel
Thanks for the clarification! I'm a bit swamped with a job search at the moment, but I should be able to play around with this a bit in about two or three weeks. If someone else decides to tackle this before then, I'll be happy to help with reviewing.

UPDATE: Timestamp is too tightly coupled with timezone tslibs code to refactor the latter without the former (at least, I think so after a few attempts to do that). I'll try to tackle both at the same time and post a PR if my solution works, but it will probably take a couple weeks. Will post here anyway.

[AlexKirko]

Apologies for the long absence: I had some health issues. Nothing too serious, but productivity was shot to hell for months. Now that I'm feeling better and should fully recover, I'm self-assigning this. Hope to have something for review in two weeks.