BUG: Dataframe.groupby aggregations with categorical columns lead to incorrect results.

[MarcoGorelli]

• closes Dataframe.groupby aggregations with categorical columns lead to incorrect results. #32494
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[jreback]

this should be in .group_info itself

[MarcoGorelli]

@WillAyd @jreback thanks for your reviews.

The solution I've pushed can either be:

if any(is_categorical(ping.grouper) for ping in self.grouper.groupings):
    result = result.reindex(self.grouper.result_index)

(as this is only necessary if there's a categorical grouping), or, if the if statement adds unnecessary complexity, just

result = result.reindex(self.grouper.result_index)

(which works all the time but in some cases isn't necessary), as I've currently done.

[MarcoGorelli]

This is currently wrong: it works for

>>> df = pd.DataFrame({"A": pd.Categorical(['a', 'b', 'a'], categories=['a', 'b', 'c']), "B": [1, 2, 3], "C": ['a', 'b', 'a']})
>>> df.groupby(['A', 'C']).transform('sum')

but not for

>>> df = pd.DataFrame({"A": pd.Categorical(['a', 'b', 'a'], categories=['a', 'b', 'c']), "B": [1, 2, 3], "C": ['a', 'b', 'a']})
>>> df.groupby(['A', 'C'])['B'].transform('sum')

will update when this is addressed

UPDATE

I think it's fine now

[jreback]

does this depend on if observed=True is passed?

[MarcoGorelli]

It's unnecessary if observed is True - have taken care of that now, thanks

[jreback]

can we *always just do
result = result.reindex(self.groupber.result_index, copy=False) ?

what breaks if we do that

[MarcoGorelli]

@jreback that's fine, we can do that always, I just thought it would be better not to if it wasn't necessary. Green now

[jreback]

ok looks good. can you run the groupby benchmarks to make sure this doesn't regress anything. ping on green (and post results)

[MarcoGorelli]

@jreback here are the results of

$ asv continuous -f 1.1 upstream/master HEAD -b ^groupby

:

       before           after         ratio
     [dede2c7a]       [d6043ecb]
     <32494^2>        <32494>   
+         185±7μs          216±8μs     1.17  groupby.GroupByMethods.time_dtype_as_field('object', 'count', 'direct')
-        826±60μs          746±6μs     0.90  groupby.GroupByMethods.time_dtype_as_group('float', 'cumprod', 'transformation')
-        208±20μs          186±1μs     0.89  groupby.GroupByMethods.time_dtype_as_group('float', 'cumcount', 'direct')
-      1.02±0.1ms         862±10μs     0.84  groupby.GroupByMethods.time_dtype_as_group('int', 'rank', 'transformation')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.

[MarcoGorelli]

@jreback

I just merged master and this time got

BENCHMARKS NOT SIGNIFICANTLY CHANGED.

running the same command

[jreback]

lgtm. comment on using a fixture, merge master and ping on green.

[jreback]

thanks @MarcoGorelli