BUG: regression in max_info_columns behaviour?

[jorisvandenbossche]

Update:

• max_info_columns should not behave the same as max_info_rows. If max_info_columns is exceeded, it should flip to short summary (as verbose=False)
• we could add a keyword argument like show_counts (or another name) to specify if you want to show the non-null counts (to be able to override the max_info_rows option for a specific info call)

---

When you have more columns than specified in max_info_columns, df.info() will now still show all columns, but just without the information about the number of null values:

In [1]: pd.__version__
Out[1]: '0.13.1-656-gf30278e'

In [4]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 5 columns):
0    5 non-null float64
1    5 non-null float64
2    5 non-null float64
3    5 non-null float64
4    5 non-null float64
dtypes: float64(5)

In [5]: df.info(verbose=False)
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Columns: 5 entries, 0 to 4
dtypes: float64(5)

In [6]: pd.options.display.max_info_columns = 4

In [7]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 5 columns):
0    float64
1    float64
2    float64
3    float64
4    float64
dtypes: float64(5)

while previously in 0.13 this gave the same behaviour as for info(verbose=False):

In [15]: pd.__version__
Out[15]: '0.13.0'

In [16]: pd.options.display.max_info_columns = 4

In [17]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Columns: 5 entries, 0 to 4
dtypes: float64(5)

which seems much more logical to me.

I suppose this is related to #5682, which added the behaviour to also show the dtype per column.

Update: it was deliberately added in #5974

[jorisvandenbossche]

So it was added in #5974: rationale was options.display.max_info_rows was deprecated because of the new dataframe display, but this was then used in that PR to do the right behaviour in df.info whether to show the counts or not.
However, there is nothing about max_info_columns in that PR, and I personally also don't think that is what max_info_columns should be for.

Because now, if you have eg a dataframe with 1000 columns, df.info() will have 1000 lines in your terminal output.

[jorisvandenbossche]

This is the intended behaviour of that PR (as far as I understand):

In [17]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 5 columns):
0    5 non-null float64
1    5 non-null float64
2    5 non-null float64
3    5 non-null float64
4    5 non-null float64
dtypes: float64(5)

In [18]: pd.options.display.max_info_rows = 4

In [19]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5 entries, 0 to 4
Data columns (total 5 columns):
0    float64
1    float64
2    float64
3    float64
4    float64
dtypes: float64(5)

[jreback]

ok so why don't we hijack max_info_columns to determine info_verbose rather than a separate parm?

[jorisvandenbossche]

That was how it was in 0.13 and before, and that seems more logical to me personally.

[jreback]

cc @bjonen

so let's make that change rather than adding the info_verbose option in #6890

[jorisvandenbossche]

although it is still a little bit different. The columns argument sets the default for a threshold when using long and when short info output, while a info_verbose would set the default for info(verbose=..) for all dataframes.

EDIT: I suppose a user can then just set the max_info_columns=0 to have always the short summary info?

[jorisvandenbossche]

@cpcloud OK with this? (you were also involved a bit in the info view rework in 0.13.1, not exactly this but the showing of the dtypes)

[jreback]

max_info_columns=None/0 should effectively trigger what info_verbose=False was going to do
(and you can always pass it in too)