FIX? df.__repr__ takes for ever with big data

[dengemann]

Hi all,

after having updated my local branch with the current dev master ( pd.version == '0.11.0rc1' -- compiling under Mac OS 10.8 went smoothly) some of my scripts in which I handle large data frames (2.6 million rows, 10 columns) stopped working.
It turned out that accessing the df's repr method but also looking up docstrings seems to make ipython unresponsive and creates the impression as if it takes forever.

This little snippet reproduces the error with fake data ('tested' with EPD Python and IPython 7.2 (32-bit)):

import numpy as np
import pandas as pd

data  = np.random.random((3000000, 10))
df = pd.DataFrame(data)

print df  # does the same as `df.__repr__()` or `df?`

Can anyone confirm this?
It must have been some of the more recent changes as I did not have trouble with this earlier.
Any pointers and comments are highly welcome.

Cheers,
Denis

see #3326 (comment)
should be fixed by the time 0.11.0 final is out.

In the meantime, git reset --hard 7db1af4^ will get you practically everythin new
in 0.11.0.

Thanks for reporting it, wil leave open till it's sorted.

Can you provide a testcase for the docstring issue you mention? that's new.

[dengemann]

Thanks for your quick reply. Unfortunately I cannot give you a pure Python example as DataFrame instances don't employ But doing df? or df?? in IPython does the 'trick'. Maybe that's the point, IPython will certainly employ some inspection magic to find out about the doc string. Would it make sense to move the doc string from under DataFrame.init to DataFrame, so it's possible to referred to it as DataFrame.doc ?

The IPython docstring problem seems to be part of the same issue, so
there's probably no need for that, they'll both get fixed together.

[dengemann]

Yes, makes sense. Just out of curiosity, wouldn't it be more consistent with other scientific / data analysis packages in Python to have non-empty .doc attributes? Or is this already being addressed somewhere? (sorry, I'm quite new to tracking the dev version of pandas and it's not so easy to get an overview on the open issues)

I guess we all use ?? rather then doc , and IPython context help finds
it as well, so I never noticed that before.

Sure. PR welcome.

[dengemann]

I guess we all use ?? rather then doc , and IPython context help finds it as well, so I never noticed that before.

I agree, neither did I. When using the debugger however, IPython features aren't available and:

print foo.doc

is a helpful idiom.

Sure. PR welcome.

ok

[dengemann]

@y-p I will look for this 'issue' in the entire library. where ever I see pep8 violations I also would address those. (?)

Have a read on CONTRIBUTING.md for various notes, including plus and minuses for
"PEP8 storms".

[dengemann]

I just found that many violations are systematic and rather appear as
'dialects', thanks for the pointers, I'll take look.

On Sat, Apr 13, 2013 at 3:01 PM, y-p notifications@github.com wrote:

Have a read on CONTRIBUTING.md for various notes, including plus and
minuses for
"PEP8 storms".

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3337#issuecomment-16332706
.

[hayd]

@y-p I always feel like a PEP8 storm... :)

I guess it's not bad to hit the 'reset' button once per release, but more then that
becomes disruptive. I use emacs and dangling whitespace from lesser editors
is a constant annoyance too, so "formatting storms" are not all bad.

[dengemann]

After taking a tour I've learned ended up wondering about a few things.
It looks like local style to not separate args in a function and not to drop white spaces around kwargs.
also the typical 2 lines between functions rule is not complied with consistently.
Most importantly on many many instances lines are far beyond 80 characters.
I think this calls for a separate PR.
It would be good for new contributors to have a more detailed style guideline including a minimum agreed-upon set of pep8 rules. This might also keep down pep8 storms. If e.g. the 2 lines between functions rule is not among the core style rules then may can simply ignore it (despite eyes bleeding from the editor's pep8 mode).

Wdyt?

So CONTRIBUTING.md didn't convince you. oh well. :)

[dengemann]

So CONTRIBUTING.md didn't convince you. oh well. :)

Admittedly not, was more of a deterrent to me ;-)