BUG: Cannot assign values to DataFrame using mask after previous multi-column `__setitem__` value assignment

[zkrolikowski-vl]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

df = pd.DataFrame(data={'a': [1], 'b': [1]})
df[['a', 'b']] = [[2, 2]]
select_df = pd.DataFrame({'a': [True], 'b': [False]})
df[select_df] = [3, 3]

Issue Description

It's impossible to assign values to DataFrame using a DataFrame mask if the source DataFrame's values were assigned in a multi-column fashion by using __setitem__.

This seems to be a regression in pandas 1.4.0rc0 as the code is running fine under 1.3.5 or possibly an udocumented change - unless I missed this being mentioned in the changelog.

Comparatively the following code works:

df = pd.DataFrame(data={'a': [1], 'b': [1]})
select_df = pd.DataFrame({'a': [True], 'b': [False]})
df[select_df] = [3, 3]

Moreover the following assignment variants don't cause any problems:

df.loc[:, ['a', 'b']] = (1,2)
df.loc[0, ['a', 'b']] = (1,2)

I apologize if this issue is known under different keywords.

Traceback

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/base38/lib/python3.8/site-packages/pandas/core/frame.py", line 3642, in __setitem__
    self._setitem_frame(key, value)
  File "/opt/anaconda3/envs/base38/lib/python3.8/site-packages/pandas/core/frame.py", line 3765, in _setitem_frame
    self._where(-key, value, inplace=True)
  File "/opt/anaconda3/envs/base38/lib/python3.8/site-packages/pandas/core/generic.py", line 9155, in _where
    new_data = self._mgr.putmask(mask=cond, new=other, align=align)
  File "/opt/anaconda3/envs/base38/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 347, in putmask
    return self.apply(
  File "/opt/anaconda3/envs/base38/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "/opt/anaconda3/envs/base38/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 976, in putmask
    putmask_without_repeat(values.T, mask, new)
  File "/opt/anaconda3/envs/base38/lib/python3.8/site-packages/pandas/core/array_algos/putmask.py", line 147, in putmask_without_repeat
    raise ValueError("cannot assign mismatch length to masked array")
ValueError: cannot assign mismatch length to masked array

Expected Behavior

DataFrame should have it's values assigned based on the mask. An REPL example follows:

>>> df = pd.DataFrame(data={'a': [1], 'b': [1]})
>>> df[['a', 'b']] = [[2, 2]]
>>> select_df = pd.DataFrame({'a': [True], 'b': [False]})
>>> df[select_df] = [3, 3]
>>> df
   a  b
0  3  2

Installed Versions

INSTALLED VERSIONS ------------------ commit : bb1f651 python : 3.8.5.final.0 python-bits : 64 OS : Darwin OS-release : 20.2.0 Version : Darwin Kernel Version 20.2.0: Wed Dec 2 20:39:59 PST 2020; root:xnu-7195.60.75~1/RELEASE_X86_64 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : None LOCALE : en_US.UTF-8

pandas : 1.4.0
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 47.3.1
Cython : None
pytest : 6.0.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.22.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.3.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 3.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.5.3
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : 1.2.0
xlwt : None
zstandard : None

[jbrockmendel]

[3, 3] is treated as 1D while df[select_df] is 2D. If you use [[3, 3]] you should get the expected behavior.

Could probably use a more informative exception message.

[zkrolikowski-vl]

Thanks for your interest in this problem.

So performing df[['a', 'b']] = [[2, 2]] makes df 2D?

What's about the following example where we assign a flat list?

df = pd.DataFrame(data={'a': [1], 'b': [1]})
df[['a', 'b']] = [2, 2]
select_df = pd.DataFrame({'a': [True], 'b': [False]})
df[select_df] = [3, 3]

It suffers from the same problem.

I just wanted to understand what's the correct way to use this feature. Apparent behavior has changed since 1.3.5.

[jbrockmendel]

So performing df[['a', 'b']] = [[2, 2]] makes df 2D?

No. A DataFrame is always 2D. Setting values in it doesn't change that.