ENH: showing memory used

[jreback]

maybe a function nbytes() to return memory usage

maybe as a tuple : the Series in [6] and then a series of the axis data?

In [5]: df = DataFrame({ 'float' : np.random.randn(1000), 'int' : np.random.randint(0,5,size=1000), 'date' : Timestamp('20130101'), 'string' : 'foo' })

In [6]: Series(dict([ (b.dtype,b.values.nbytes) for b in df._data.blocks ]))
Out[6]: 
datetime64[ns]    8000
int64             8000
float64           8000
object            8000
dtype: int64

In [7]: df.index.nbytes
Out[7]: 8000

In [8]: df.columns.nbytes
Out[8]: 32

[cpcloud]

my 2c. i think meminfo should return your series and a property nbytes should return df.meminfo().sum()

[asobrien]

Using the following DataFrame as an example:

In [11]:  df = pd.DataFrame({ 'float' : np.random.randn(10000000), 'int' : np.random.randint(0,5,size=10000000), 'date' : Timestamp('20130101'), 'string' : 'foo', 'smallint' : np.random.randint(0,5,size=10000000).astype('int16') })

I have implemented meminfo() as as DataFrame method, resulting in a Series. Note that units can be specified:

In [13]: # Default units = B
In [14]: df.meminfo()
Out[14]: 
date        80000000
float       80000000
int         80000000
smallint    20000000
string      80000000
dtype: float64

In [15]: df.meminfo(units="MB")
Out[15]: 
date        76.293945
float       76.293945
int         76.293945
smallint    19.073486
string      76.293945
dtype: float64

Total memory usage of the DataFrame has been implemented as part of df.info() where the units representation can be specified:

In [16]: df.info()             
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Data columns (total 5 columns):
date        datetime64[ns]
float       float64
int         int64
smallint    int16
string      object
dtypes: datetime64[ns](1), float64(1), int16(1), int64(1), object(1)
memory_usage: 340000000.00 B

In [18]: df.info(mem_unit='MB')
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Data columns (total 5 columns):
date        datetime64[ns]
float       float64
int         int64
smallint    int16
string      object
dtypes: datetime64[ns](1), float64(1), int16(1), int64(1), object(1)
memory_usage: 324.25 MB

Any more thoughts?

[jreback]

I think that would be useful!

would need a couple of display options I think:

• default unit (MB?)
• whether to display this by default (in say df.info()), default False for now

[asobrien]

Let's use MB as the default unit and add a memory_usage argument to the DataFrame info() method to specify unit type (where default is None).

Total memory usage of the DataFrame (by default in MB) can always be determined by using:

df.meminfo().sum()

[cpcloud]

Seems like it would be easy to infer the unit by taking the log10 of the number of bytes, dividing by 10 to that number and using the bisect module to lookup the corresponding name of the unit in an array.