REGR: Behavior change with empty apply in pandas 1.3.0rc1

[aberres]

Problem description

The following (toy) snippet worked with 1.2:

df = pd.DataFrame(columns=["a", "b"])
df["a"] = df.apply(lambda x: x["a"], axis=1)

With 1.3 it fails with ValueError: Columns must be same length as key

Technically this is correct - the apply on an empty frame returns the empty frame so things do not really match.

Expected Output

It still works? Just reporting it here if this is an unintended change. Maybe I missed it, but I did not see this mentioned in the changelog.

The fix is to only call apply when the frame is not empty I guess? I stumbled upon this one when running our test suite.

[simonjayhawkins]

Thanks @aberres for the report

Just reporting it here if this is an unintended change. Maybe I missed it, but I did not see this mentioned in the changelog.

first bad commit: [caf81fa] BUG: DataFrame.setitem not raising ValueError when rhs is df and has wrong number of columns (#39341)

cc @phofl

[phofl]

@simonjayhawkins I've relabeled, since this is an indexing issue, not related to apply.

The RHS is

Empty DataFrame
Columns: [a, b]
Index: []

This is basically the case we have changed the behavior for.

df = pd.DataFrame(columns=["a", "b"])
rhs = pd.DataFrame(columns=["a", "b"])
df["a"] = rhs

I think this behaves now as expected.

@aberres The note you are looking for is the one for #38604 in the whatsnew

Edit: The idea behind this is, that the number of columns has to match. Previously we assigned always the first column and silently dropped the rest.

[aberres]

@aberres The note you are looking for is the one for #38604 in the whatsnew

Thanks, makes sense.

Not sure if this happens a lot out in the wild - but maybe this case should be allowed for empty columns? I addd if not df.empty in my code which worked fine - as expected.

[phofl]

Don't think behavior based on data is desirable. I am not sure what you want to achieve here if this works for empty dfs. When you know, that your DataFrame is empty why bothering with the setitem call? If it is not empty, this will raise even if we would allow empty frames

@jbrockmendel thoughts?

[phofl]

Hm maybe I was wrong above and this is an apply issue.

df = pd.DataFrame([[1, 2]], columns=["a", "b"])
df.apply(lambda x: x["a"], axis=1)
0    1
dtype: int64


df = pd.DataFrame(columns=["a", "b"])
df.apply(lambda x: x["a"], axis=1)

Empty DataFrame
Columns: [a, b]
Index: []

Looks inconsistent, but I am not familiar enough with apply to asses this.

Nevertheless I am against catching the incosistency here in setitem. You can get into all sorts of trouble, if you expect a Series but receive a DataFrame.