df.dtypes.values is not O(1) and repr(df) is therefore slow for large frames

For the FEC dataset, it takes about 1.5 sec to get a repr. and prun
puts it all in infer_dtype.

[jreback]

do you have a link to the dataset...can't seem to find mine

ftp://ftp.fec.gov/FEC/Presidential_Map/2012/P00000001/P00000001-ALL.zip

[jreback]

I believe that this is the problem.

It is trying to see if their are floats in an object array. I would simply not do this at all
or short-circuit it.

Breakpoint 2 at /mnt/home/jreback/pandas/pandas/core/format.py:1663
(Pdb) c
> /mnt/home/jreback/pandas/pandas/core/format.py(1663)_format_strings()
-> is_float = lib.map_infer(vals, com.is_float) & notnull(vals)
(Pdb) l
1658                    # object dtype
1659                    return '%s' % formatter(x)
1660 
1661            vals = self.values
1662 
1663B->         is_float = lib.map_infer(vals, com.is_float) & notnull(vals)
1664            leading_space = is_float.any()
1665 
1666            fmt_values = []
1667            for i, v in enumerate(vals):
1668                if not is_float[i] and leading_space:

It's probably there to support the float_format arg of to_string. I'll have to think about it.

[jreback]

for an object dtype you could warn if it 'looks' like float, but otherwise skip it
its checking strings that aren't numbers at all is the problem

isn't looks_like_float() exactly what map_infer does? how could it be faster if
I have to check each value for "appearence"?

It doesn't need to do this for values not displayed in the output. that's it.

[jreback]

right!

That's not where the bottleneck is.
What's this?

In [10]: %timeit df.dtypes.values
1 loops, best of 3: 178 ms per loop

aren't dtypes just a lookup?

[jreback]

this issue address this, but it needs reworking to make it more internal as I havev indicated: #5740

Related (djeavu): 3cb6961, #2807 (comment)

[jreback]

I have got a PR...give me a few

There's an off chance this might be the cause of a lot of the slowdowns we saw in 0.13
after the NDFrame refactor. is the change in behaviour related? if yes, hurrah.

[jreback]

anything with df.apply in it is generally bad (as dsm fixed for str.extract), when needed internally

#5660

frame_get_dtype_counts | 0.1843 | 0.1113 | 1.6552 |

less then what I expected, but should have sent bells ringing.

Unrelated in fact.