PERF: groupby with many empty groups memory blowup

[jbrockmendel]

Suppose we have a Categorical with many unused categories:

cat = pd.Categorical(range(24), categories=range(10**5))

df = pd.DataFrame({"A": cat, "B": range(24), "C": range(24), "D": 1})

gb = df.groupby(["A", "B", "C"])

>>> gb.size()  # memory balloons to 9+ GB before i kill it

There are only 24 rows in this DataFrame, so we shouldn't be creating millions of groups.

Without the Categorical, but just a large cross-product that implies many empty groups, this works fine:

df = pd.DataFrame({n: range(12) for n in range(8)})

gb = df.groupby(list(range(7)))
gb.size() # <-- works fine

[jreback]

this is the point of the observed keyword

[WillAyd]

So yea to second Jeff’s comment above is this a problem with observed=True?

[jbrockmendel]

passing observed=True solves the problem. id like to add a warning or something for users who find themselves about to hit a MemoryError

[WillAyd]

Did we ever discussion changed the default observed keyword? @TomAugspurger

…

Sent from my iPhone

On Jan 2, 2020, at 4:21 PM, jbrockmendel ***@***.***> wrote:  passing observed=True solves the problem. id like to add a warning or something for users who find themselves about to hit a MemoryError — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[TomAugspurger]

I don't recall a discussion about that.

I vaguely recall that this will be somewhat solved by having a DictEncodedArray that has a similar data model to Categorical, without the unobserved / fixed categories semantics.

[WillAyd]

Hmm I think that is orthogonal. IIUC the memory blowup is because by default we are generating Cartesian products. Would rather just deprecate that and switch the default value for observed

[TomAugspurger]

I think it’s a good default for the original design semantics of Categorical. It’s a bad default for the memory-saving aspect of categorical.

[jbrockmendel]

Sounds like we can close this as no-action?

…

On Thu, Jan 2, 2020 at 3:40 PM Tom Augspurger ***@***.***> wrote: I think it’s a good default for the original design semantics of Categorical. It’s a bad default for the memory-saving aspect of categorical. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#30552?email_source=notifications&email_token=AB5UM6AK4AKANOARO7MCO53Q3Z3NJA5CNFSM4KBD67Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH72B5I#issuecomment-570401013>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5UM6BQJEOLOICRGMW4KZTQ3Z3NJANCNFSM4KBD67QQ> .

[jreback]

i suppose we would show a PerformanceWarning if we detect this would happen which is pretty cheap to do

might be worthwhile

[paulrougieux]

I cannot reproduce the issue above with cat = pd.Categorical(range(24), categories=range(10**5)) on pandas 0.25.3.

When upgrading from pandas 0.24.2 to 0.25.3, I had a memory issue with a groupby().agg() on a data frame with 100 000 rows and 11 columns. I used 8 grouping variables with a mix of categorical and character variables and the grouping operation was using over 8Gb of memory.

Setting the argument observed=True:

df.groupby(index, observed=True)

fixed the memory issue.

Related Stack Overflow question: Pandas v 0.25 groupby with many columns gives memory error.

Maybe observed=True should be the default? At least beyond a certain ratio of observed / all possible combinations. When the observed combinations of categorical values is way lower than all possible combination of these categorical values, it is clear that it doesn't make sense to use observed=False. Is there a discussion on why the default was set to observed=False?