tz_localize should support is_dst input array

[rockg]

When storing datetimes with timezone information in mysql I split out the is_dst flag into a separate column. Then when reconstructing the Timestamps I am either forced to iterate through each row and call pytz.timezone.localize on every Timestamp which is very slow or do some magic with localizing what I can and then manually dealing with the fall transition time (note that infer_dst won't work because there could be many rows that have transitions in them). I would much rather create the DatetimeIndex from the column of dates and then call tz_localize with the is_dst column. This would then appropriately set the offset.

di = DatetimeIndex(frame['DateColumn'])
di = di.tz_localize(TimeZone, is_dst_flat=frame['IsDstColumn'])

Thoughts?

[jreback]

we already support infer_dst? how does that interact?

[rockg]

infer_dst only works when there are only two dates for the fall dst transition time. If there are more than two, then it cannot infer simply based on order. In that case, something else needs to be done to explicitly tell what times are dst times and what times are not.

[jreback]

ok, makes sense. would making infer_dst='force' make sense? or you simply want another keyword? (is_dst=None). I suppose its an error if is_dst is not None and infer_dst=True ?

[rockg]

I would like an is_dst keyword and this also mirrors pytz's localize which takes is_dst as a parameter. I agree that it is an error if is_dst is not None and infer_dst=True (they are mutually exclusive). I'll take a stab add adding it to localize. Should be straightforward if I recall.

[jreback]

ok, sounds good

[ischwabacher]

I suspect that is_dst is too entrenched for me to have a chance of painting this bikeshed my color, but I am against is_dst as a name because not all time zone discontinuities are DST boundaries. I think in this case, it is probably conceptually clearer to split off DST information as a timedelta64 offset rather than an is_dst flag, which among other things makes converting to UTC trivial.

That said, since the problem of converting local time and time zone to UTC (essentially inverting the time zone, if we view it as a map from UTC to local time) is simply one of selecting the correct UTC time from a list of zero, one or two times, and since in the case of an ambiguous time at a DST boundary the DST time must always come first, I propose that if we use is_dst as a parameter, it should mean "select the first (as opposed to the last) UTC time that maps to this local time, regardless of DST", and is_dst=True should be the default.

I also propose that we not put any effort into supporting the case where there are three or more UTC times corresponding to a given local time, because this case requires pathological government action and I don't know of any instance where it has happened.

[rockg]

Your second paragraph is not exactly correct and why this is needed. If you only had two ambiguous times, then the infer_dst argument is all that is needed and works like you say. However, when there are more times it becomes a problem. For example, when reading database records where there are multiple items that have ambiguous times associated with them, rather than discretizing among those items and inferring the right dst/non-dst time from for each item it will be much faster and easier to already use a pre-computed column when the data was input in my particular case. Here's a simple example of data I'm looking at.

Item,Time,IS_DST
1,2014-11-02 00:00,1
1,2014-11-02 01:00,1
1,2014-11-02 01:00,0
2,2014-11-02 00:00,1
2,2014-11-02 01:00,1
2,2014-11-02 01:00,0

[ischwabacher]

What I'm proposing is that (if you are in America/Chicago) you will have this:

Item,Time,Offset
1,2014-11-02 00:00,-05:00:00
1,2014-11-02 01:00,-05:00:00
1,2014-11-02 01:00,-06:00:00
2,2014-11-02 00:00,-05:00:00
2,2014-11-02 01:00,-05:00:00
2,2014-11-02 01:00,-06:00:00

Which is just as good.

[rockg]

Yes, I know that's your first point...I was more addressing your subsequent points.

I agree that it's just as good and contains the same information. My data is structured with bools and hence I want to make that work (it was only a couple lines of code to make this work). I don't know offhand how to make the offsets in the underlying tz_localize code. I don't see any harm in making both work if possible.

[mtrbean]

To @ischwabacher's point, I'm not against adding is_dst= to tz_localize(). But I also agree that a better solution to @rockg's problem is to store timestamps in UTC and timezone separately in two columns in a database table.

I always store timestamp in UTC in database to avoid ambiguity, unless the database has explicit support for timestamp with time zone like postgresql.