Slow printing of large data frames

[cpcloud]

I have DataFrames with about 14 million rows and 16 columns and I have to wait at least 3-4 seconds for it to repr in an IPython session. Is there anything that can be done about this?

[wesm]

Set DataFrame._verbose_info to be False. This should be made configurable, or maybe even just disable the null-counts over 1 million rows

[cpcloud]

@wesm For the configurable, could this be done by registering an option such as:

# necessary imports

def check_verbose_info(x):
    try:
        bool(x)
    except:
        raise ValueError('invalid value for frame.verbose_info')

pd.config.register_option('frame.verbose_info', True, check_verbose_info)

If so, where would this need to go?

@cpcloud, have a look at pandas/core/config_init.py

[cpcloud]

With the latest release this is slow even after setting _verbose_info.

[wesm]

that's a bug then (probably it is checking whether the dataframe is "too wide" before reaching that code). if no one fixes the bug before i have a chance i will get to it in the next couple of weeks

The delay comes from the dtype count, which verbose_info doesn't disable.
Despite a lot of changes, this has been behaving like this at least since 0.9.1.

I intended to put of a fix for review but ended up pushing to master by mistake
(sorry about that). So I welcome review.

[jreback]

I changed the way the way get_dtype_counts works in 0.11. simple fix for this

[jreback]

@cpcloud do u have a test example?
what version/commit are u running?

[jreback]

@y-p saw your fix thanks.....this obviously shouldn't have been using as_blocks

df.blocks is quite slow for wide frames, and it can be a nasty surprise to get that hit
since, being a @Property, it just looks like you're getting a reference to some existing object.

I'll call this closed.

[cpcloud]

I'm always running the latest version :). @jreback You could do something like

df = DataFrame(rand(1e7, 16))
%timeit repr(df)

[cpcloud]

Interestingly, DataFrames with columns with all bool dtypes are twice as slow to repr as are those with columns with all float dtypes. I timed this using %timeit repr(df). This is not for an arbitrarily shaped DataFrame. It was only twice as slow on a DataFrame with about 14 M rows.