Many groupby tests depend on a bug in DataFrameGroupBy.apply

[christopherzimmerman]

Discovered while fixing an issue in #28541

A large number of tests relating to DataFrameGroupBy.apply depend on a bug in apply where the grouped column is not removed from the returned DataFrame.

Here are some examples:

• test_apply_chunk_view
• test_apply_multiindex_fail

There are around ~20 cases of this.

---

In general, they expect the result of something like this:

df = pd.DataFrame({"key": [1, 1, 1, 2, 2, 2, 3, 3, 3], "value": range(9)})
df.groupby('key').apply(lambda x: x.sum())

To include the "key" column in the result, which should not be the case.

     key  value
key
1      3      3
2      6     12
3      9     21

The underlying issue with apply is a straightforward fix, all that needs to be done is change this return to use the _group_selection_context, but the PR will have to include updates to many tests.

[WillAyd]

I think worth double checking the comprehensive functionality of as_index, i.e. making sure it works consistently for the reduction functions to put the groupings in the index when True and put in as columns when False

Not sure off the top of my head how that is covered in tests but would be great if you can take a look while reviewing all of this

[jbrockmendel]

Not sure how much this overlaps, but I'm noticing in _aggregate_frame the for name, data in self: is yielding data DataFrames with more columns than I expect. I think this is explained by _GroupBy.__iter__ using self.obj instead of self._obj_with_exclusions. If we change this, a handful of tests are broken, but most of thsoe are for merge ops

[rhshadrach]

Closing as a duplicate of #7155.