New DataFrame display information?

[jseabold]

We are using DataFrames to hold information in a lot places now. E.g., ANOVA tables

            df  sum_sq     mean_sq          F    PR(>F)
C(Fitness)   2     672  336.000000  16.961538  0.000041
Residual    21     416   19.809524        NaN       NaN

[2 rows x 5 columns] 

Where'd that bottom [] business come from? Is this new? Isn't this something that's better included in the info method?

[jreback]

started in 0.13, you can turn off if you want by: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#output-formatting-enhancements (I think set show_dimensions to False)

[jseabold]

I figured you could, but that's a global option that we can't control after we pass back to a user. We're trying more and more to take advantage of DataFrames as containers to display information, to_latex, to_csv, etc. and this is just noise here. It is descriptive information, which is why I think it should be in info just not in the __repr__. If this is going to stay, then I'm going to have to go in and change all of our use of DataFrames as container/display objects, because it's pretty ugly for tables of information IMO.

[jseabold]

Also, .shape is just a tab-complete away...

[jreback]

this is only for interactive display, e.g. __repr__ so not sure what the issue is

its actually quite helpful when you have the default to display the actual data, but the frame itself is truncated (e.g. you are not using info, which you always have the option to do of course)

[jseabold]

Yes, I'm not saying it affects anything else. I'm saying that we like those features, but we also like a clean __repr__. In the case when we're using it as a table-like container, e.g., in ANOVA tables above, the clean __repr__ trumps everything for me. Trying quickly to scan a bunch of ANOVA tables, I'm interrupted with this noise about the rows and columns and it makes interactive use more difficult to the point where I'd rather fall back on our old, horrible to use summary tables. IIRC, we also have a PR to move our summary tables partially into DataFrames, which will also be affected.

It used to be that by default when the frame was truncated, that you got the info() view, which I preferred, but can live without. Why not show this new information only if the DataFrame is, in fact, truncated?

But, again, as you pointed out, you always have info, so you can just call it. And shapes is also just one tab-completion away. It just seems that there are so many other ways to get this information than forcing it into the default __repr__ of everything.

I hate to nit-pick here, but this is a pretty jarring visual change from what I'm used to, and there's not much gained.

[jreback]

I think their was a bit of discussion on the linked issues. http://pandas.pydata.org/pandas-docs/dev/whatsnew.html#dataframe-repr-changes.

I personally like it, but I can see your point. Hard to have everyone like everything all the time!

[cancan101]

@jseabold Why not just return a subclass of DataFrame with the repr methods changed?

Maybe the show_dimensions = get_option("display.show_dimensions") call should be moved to an instance methods making changing the behavior in a subclass easier?

[jseabold]

IMO the case for including that extra information has not been made well enough for me to start subclassing DataFrames to avoid seeing a piece of information I don't want or need. Are there any other data structures out there that do this? Matlab? data.frames? Numpy arrays? I think scipy.sparse matrices do, but are we really saying that a DataFrame that's truncated on output is like a sparse matrix? Originally, DFs showed the info view. But sometimes you want to peak at the data. So it grew a head and tail like data.frames. There's also shape. Is anyone really printing out the full __repr__ of a DataFrame just to find out about the rows and columns? This just seems like an addition for the sake of having an addition to me. No real benefit, potential negative side effects (more noise).

[cancan101]

@jseabold My opinion is that when no truncation of the dataframe occurs, printing the dimensions is in fact noise. When truncation does occur, I do believe that it makes sense to print the dimensions of the full frame.

[jseabold]

Yeah, I could live with that. I won't back up to argue whether there should even be this ... truncation vs just info, head, and tail :)

[cancan101]

@jreback What do you think about that?

[jreback]

I think these discussions are personal preferences and have long been discussed
you can simply set options to get the prior behavior if u want

no one is ever going to be happy

so will wait for more consensus to potentially change this again

[jseabold]

Ok, then here's another gripe. Discussions on github are not public enough IMO. There is a lot of a github noise from pandas. I try my best to keep up with development, but I don't see this until I install it. It seems to me, you're saying "I participated in the discussion, and it went on for a long time, so that's that."

This was discussed by 3 people on #4886 and #5550. @y-p made some strenuous initial objections (which are in line with my initial reaction) and then withdrew them. Particularly this bit

I strongly urge conducting a small usability study (have a few users adopt it for a week 
and report) before making potentially disruptive change like this to UX.

There are several other sensible comments to this effect.

It would've been great to see a ping on the mailing list about this. Something. Anything. Maybe I missed it and it's my fault. But, hey, let's think of this release as the usability study.

Coming back to the point, the issue of the footer was discussed only in passing on #5550. And here's what it says (my emphasis).

**When truncating**, having a footer with total row count would eliminate the need 
to use df.info in many cases and so reduce the impact of the change on existing users. 
(For example, after filtering a frame you're often interested in the size of the result).

[jreback]

@jseabold all for more discussion

esp on UX

pls post an issue to the mailing list if u would then