resample() should allow a "null" fill_method

[jd]

I'm using resample() to aggregate data in a timeframe.

>>> s = pandas.Series((3, 4), (pandas.Timestamp("2014-1-1"), pandas.Timestamp("2015-1-1")))
>>> x = s.resample("1s")
>>> len(x)
31536001

When doing such a call, resample fills with NaN all the (31536001 - 2) inexistent values, which ends up creating thousands of points and making Python using 500M+ RAM. The thing is that I don't care about the NaN point, so I would like to not fill them in the Series and having so much memory used. AFAICS resample does not offer such as fill_method.

[jreback]

this is trivially done with

s.resample('1s').dropna()

[jd]

@jreback Sure, but that still makes Pandas uses 500M+ RAM.

Things are actually worse for something like that:

>>> s = pandas.Series(range(1000), pandas.date_range('2014-1-1', periods=1000))
>>> s.resample('1s', how='median')

The result is pretty obvious and should take less than one second to compute. But since Pandas resample with thousands of points filled with NaN, it takes 10+ minutes to compute. And uses 1.5G+ RAM.

[jreback]

you need to show what you are going to do with this

why are you resampling such a large period? what's the point

[jd]

I don't understand why the reason is important here.
I treat statistics over a year with fine grained resolution up to the second.

[jreback]

a usecase/example of what you are doing would be helpful to understand the goal. e.g. are you merely snaping things to a close freq? or are you doing an actual resample but with sparse data?

the point of resample IS to give you a full-rank set for every point in time.

[TomAugspurger]

@jd does snap cover your needs?

[jd]

@jreback Yeah, I'm doing a resampling (aggregation) using sparse data, but I'm not only snapping – there might be several values in a time bucket.

@TomAugspurger It does not, though I was looking for such a function anyway so thanks for the pointer :)

[jreback]

@jd

so api for this could be 1 of the following:

• Series.resample(...., fill_method='sparse') (which could return a sparse Series)
• Series.resample(...., fill_method='drop') (which could return a Series, but with no NaN)
• SparseSeries(....).resample(....) (which would by definition return a sparse Series)

none of these are implemented, but potentially useful.

want to take a stab at this?

[jd]

@jreback Hum I've trouble figuring out the different between sparse and drop, but I'm sure I would need one of those.

I don't think I'm know enough of Pandas internal to implement that right now, but I'd be happy to test it and report if you write some code. :)

Thanks a lot!