ENH: Richer options for `interpolate` and `resample`

[TomAugspurger]

related #1892, #1479

Is there any interest in giving interpolate and resample (to higher frequency) some additional methods?

For example:

from scipy import interpolate
df = pd.DataFrame({'A': np.arange(10), 'B': np.exp(np.arange(10) + np.random.randn())})
xnew = np.arange(10) + .5

In [46]: df.interpolate(xnew, method='spline')

Could return something like

In [47]: pd.DataFrame(interpolate.spline(df.A, df.B, xnew, order=4), index=xnew)
Out[47]: 
               0
0.5     1.044413
1.5     0.798392
2.5     3.341909
3.5     8.000314
4.5    22.822819
5.5    60.957659
6.5   166.844351
7.5   451.760621
8.5  1235.969910
9.5     0.000000  # falls outside the original range so interpolate.spline sets it to 0.

I have never used the DataFrame's interpolate, but a quick glance says that something like the above wouldn't be backwards compatible with the current calling convention. Maybe a different name? This may be confusing two issues: interpolating over missing values and interpolating / predicting non-existent values. Or are they similar enought that they can be treated the same. I would think so.

These are just some quick thoughts before I forget. I haven't spent much time thinking a design through yet. I'd be happy to work on this in a month or so.

Also does this fall in the realm of statsmodels?

[cpcloud]

@jreback thought we deprecated DataFrame.interpolate...? should we bring it back? splines sort of blur the line between pandas and statsmodels (i think leaning more towards statsmodels) but i like the idea.

[danielballan]

Yes, this is a basic task that really should [edit:] not call for statsmodels, in my opinion.

[danielballan]

Ugly workaround I offered a few days ago: http://stackoverflow.com/a/18276030/1221924

[jreback]

since we do use statsmodels/scipy in other parts of the code why don't u peruse sm 5.0 for some available functions here?

[jreback]

@jseabold do u have direct support in sm 5.0 for interpolation? or do u defer to scipy?

[jreback]

http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html

should be straightforward to directly call these

[jreback]

via a kind argument (to pandas interpolate) with some kinds passing to scipy functions which are then wrapped on the return

[TomAugspurger]

@jreback agreed about the ease of wrapping scipy.interpolate. My example in the first post is just calling

interpolate.spline(df.index, df['A'], xnew)

to get the interpolated values and then wrapping them up in a Series.

I've assumed that the DatFrame's index is the original x-values, which is probably fine for a default but we'd want an argument to say "use this column".

I could probably start on this in a few weeks (I have to finish a paper, then I promised the statsmodels guys that I'd setup a vbench for them).

[danielballan]

Can we incorporate this into resample and reindex? Anywhere that ffill and bfill are accepted, linear and cubic should also be accepted.

And if we do that, can we give the same options that Series.interpolate provides?

[TomAugspurger]

@danielballan +1 on resuing parts or all of this for resample and reindex (and possibly fillna?). I think that it would be relatively easy to handle.

Not sure how this would fit in with Jeff's refactor of Series.

[jreback]

so there exists right now a Series.interpolate and a generic.interpolate; Series.interpolate should be basically scrapped and it will then use the generic one (needs only a very slight mod to do this).

interpolate calls the pandas.core.internals.interpolate (which is the same routine actually called by method ffill/bfill), so this can be handled at a lower level (e.g. other kinds of fillers)

its a bit to wrap your head around, but is pretty straightforward

[jreback]

the key is that both Series and DataFrame BOTH now have ._data (which is the BlockManager) and then call the methods on the blocks, so they end up calling the same methods

[jreback]

lmk if you want to take a stab (or I could set it up for you with the structure and you can add in the other methods)

[TomAugspurger]

If there's no urgency, I'm fine with going through the code to refactor generic.interpolate. I may have to bug you with a few questions though!

If Series.interpolate is refactored though, backwards compatibility may be a problem? Or on second thought maybe not... Right now Series.interploate is just for filling in NaNs. That could still be the default behavior but we could still accept an array of new values at which you want to interpolate, but default that to None. I think that should work.

[jreback]

@TomAugspurger that sounds fine; there should be no back compat issue (well...have to make sure, but in theory we have tests for that.....)

if the Series.interpolate is doing some that we want to keep (as the default?) which is prob linear interpolation that then can be moved to a lower level part of the code (e.g. core.common.interpolate_2d), where all of the interpolations will eventually happen

probl need some more tests to validate this