asfreq / resample do not set freq with regular timeseries

[jorisvandenbossche]

If your timeseries has a regularly spaced index, so if a frequency can be inferred, using asfreq or resample with this frequency does not set the frequency (it remains None):

In [1]: df = pd.DataFrame({'date':["2012-01-01", "2012-01-02", "2012-01-03"], 'col':[1,2,3]})
In [2]: df2 = df.set_index(pd.to_datetime(df.date))
In [3]: df2.index
Out[3]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2012-01-01 00:00:00, ..., 2012-01-03 00:00:00]
Length: 3, Freq: None, Timezone: None

In [4]: df2.index.inferred_freq
Out[4]: 'D'

In [5]: df2.asfreq('D').index
Out[5]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2012-01-01 00:00:00, ..., 2012-01-03 00:00:00]
Length: 3, Freq: None, Timezone: None

In [6]: df2.resample('D').index
Out[6]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2012-01-01 00:00:00, ..., 2012-01-03 00:00:00]
Length: 3, Freq: None, Timezone: None

Note: this is the same with a Series, however in 0.12 it works for Series (as then it is still a TimeSeries)

[jorisvandenbossche]

With the example of above, the problem is that the frequency is not considered when looking at equality of indices:

In [40]: idx_nofreq = df2.index
In [41]: idx_freq = pd.DatetimeIndex(df2.index, freq='D')
In [42]: idx_nofreq.equals(idx_freq)
Out[42]: True

With identical it checks:

In [43]: idx_nofreq.identical(idx_freq)
Out[43]: False

And if you track it down in reindex, equality is first checked with identical (https://github.com/pydata/pandas/blob/master/pandas/core/generic.py#L1331), so this is OK, but later in Index.reindex equals is used: https://github.com/pydata/pandas/blob/master/pandas/core/index.py#L1218. For this reason it will not look at the frequency and return the original index.

[jreback]

inferring the freq is not a free operation so it's done only if it matters

identical is just equals but checks is the freq has been set (hut doesn't infer it if it has not)

[jorisvandenbossche]

OK, but if I want to explicitly set the frequency of a timeseries with asfreq (so no inferring), I would suppose it does this, which is not the case in this example.

Or is there another way to do it?

[jreback]

yep I think it should explicity set the index, I would say this is a bug. Though the references you point at are not the 'reason' its not setting.

[jorisvandenbossche]

Maybe the title of the issue was not that clear, because it had not really do to something with inferring the freq (it was only the case in which we found the bug: trying to see if there was a not-None inferred_freq and then setting it to that frequency with asfreq). So changed it.

I also think it is a bug, asfreq should set the frequency, even when the index stays the same.
And I followed down the path of asfreq with pdb, and what I referenced above (the equality check in Index.reindex: https://github.com/pydata/pandas/blob/master/pandas/core/index.py#L1218) is what I think causes this behaviour. Of course, it is not that simple to change it to identical instead of equals, because eg the name does not have to be identical.

[jreback]

I get it
however the equals check is on purpose
you want to be able to reindex something that is equals but not necessarily identical (eg has none as a freq or a different name)

asfreq and maybe resample may just need to set the freq when the index is created

[jorisvandenbossche]

Yes, but the problem is that asfreq does not really creates an index. I creates one, and tries to reindex with it, but reindex will take a copy of the original and not the newly created because of this equality check that disregards the freq attribute.

[jreback]

yep this is tricky
as I said fixing this will break lots of things
so might just make sense to copy the returned index and set the freq their

[geryatejina]

jreback, would you be so kind to explain exactly how would you do that: so might just make sense to copy the returned index and set the freq their. I've been trying without success.
Thanks for any help.

[jreback]

setup a test as the example above. The result should have the same freq as the asfreq, iow, D in this case. I think tseries.resample.asfreq is called. The problem is that in the very bottom there: https://github.com/pydata/pandas/blob/master/pandas/tseries/resample.py#L445

the reindex can just pass thru the existing index (if its equal). Where the frequency is not already set. So would need to copy the index if this is the case and set the freq.

[geryatejina]

Thanks for the explanation, jreback! I guess I am still not clear how to "copy the index if this is the case and set the freq", but this is surely due to my still limited understanding Pandas. Thanks again.