Optionally use sys.getsizeof in DataFrame.memory_usage

[mrocklin]

I would like to know how many bytes my dataframe takes up in memory. The standard way to do this is the memory_usage method

df.memory_usage(index=True)

For object dtype columns this measures 8 bytes per element, the size of the reference not the size of the full object. In some cases this significantly underestimates the size of the dataframe.

It might be nice to optionally map sys.getsizeof on object dtype columns to get a better estimate of the size. If this ends up being expensive then it might be good to have this as an optional keyword argument.

df.memory_usage(index=True, measure_object=True)

[jreback]

In [13]: import sys

In [14]: df = DataFrame({'A' : ['foo','bar']*1000000})

In [15]: df['B'] = df['A'].astype('category')

In [16]: df.memory_usage()
Out[16]: 
A    16000000
B     2000016
dtype: int64

In [17]: %timeit df.memory_usage()
10000 loops, best of 3: 66.1 µs per loop

In [18]: %timeit sum([ sys.getsizeof(v) for v in df['A'].values ])
1 loops, best of 3: 467 ms per loop

[mrocklin]

Yup, probably shouldn't be default. I'd be quite happy to opt-in.

On the other side of the comparision, 500ms is very small compared to serialization and communication time if we mistakenly decide that it's a good idea to communicate this dataframe to another machine.

[jreback]

going to give you a cythonized comparison in a sec

[jreback]

In [8]: import sys

In [3]: pd.lib.memory_usage_of_array(np.array(['a'],dtype=object))
Out[3]: 46

In [4]: pd.lib.memory_usage_of_array(np.array(['ab'],dtype=object))
Out[4]: 47

In [5]: pd.lib.memory_usage_of_array(np.array(['abc'],dtype=object))
Out[5]: 48

In [6]: pd.lib.memory_usage_of_array(np.array(['abcd'],dtype=object))
Out[6]: 49

In [7]: df = DataFrame({'A' : ['foo','bar']*1000000})

In [9]: sum([ sys.getsizeof(v) for v in df['A'].values ]) + df['A'].values.nbytes
Out[9]: 96000000

In [10]: pd.lib.memory_usage_of_array(df['A'].values)
Out[10]: 96000000

In [11]: %timeit pd.lib.memory_usage_of_array(df['A'].values)
10 loops, best of 3: 108 ms per loop

In [12]: %timeit sum([ sys.getsizeof(v) for v in df['A'].values ]) + df['A'].values.nbytes
1 loops, best of 3: 481 ms per loop

[jreback]

I am giving back the original nbytes + the actual overhead (e.g. to store it costs you the pointer in the ndarray + the actual storage)

[mrocklin]

Seems like a good idea. The speedup there is nice.

[jreback]

yep, ok, easy enough (then would turn off the '+') if you opt-in

[jreback]

In [7]: df = DataFrame({'A' : ['foo','bar']*1000000})

In [8]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2000000 entries, 0 to 1999999
Data columns (total 1 columns):
A    object
dtypes: object(1)
memory usage: 30.5+ MB

In [9]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2000000 entries, 0 to 1999999
Data columns (total 1 columns):
A    object
dtypes: object(1)
memory usage: 106.8 MB

[max-sixty]

What do you think about overriding __sizeof__ with one of these?

[jreback]

ahh, so sys.getsizeof(DataFrame) gives a nice answer, sure. that would be ok.