BUG: RecursionError when aligning column MultiIndex

[epizut]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

When trying to do arithmetic on two dataframes with a common MultiIndex level, I am having a max recursion error.
Please note that if using string column names (ie: labels) instead of ints, I have no recursion error but still a buggy NaN result.

df1 = pd.DataFrame([[1, 2, 3]], columns=['Level1', 'Level2', 'Value'])
df1 = df1.set_index(['Level1', 'Level2']).T
df1

df2 = pd.DataFrame([[1, 3]], columns=['Level1', 'Value'])
df2 = df2.set_index(['Level1']).T
df2

df1 - df2

RecursionError on atlest pandas 1.0.2 and 1.0.3

Expected Output, working fine on pandas 1.0.1

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.8.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 1.0.3
numpy : 1.18.3
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 41.2.0
Cython : None
pytest : 5.4.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.13.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.2.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.4.1
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[TomAugspurger]

Thanks for the report. We don't have a RecursionError anymore, instaed a ValueError.

Here's a slightly simpler reproducer.

In [95]: a = pd.DataFrame([[1]], index=['a'], columns=pd.MultiIndex.from_product([[0], [1]], names=['a', 'b']))

In [96]: b = pd.DataFrame([[1]], index=['a'], columns=pd.Index([0], name='a'))

In [97]: a - b
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/sandbox/pandas/pandas/core/indexing.py in _has_valid_tuple(self, key)
    712             try:
--> 713                 self._validate_key(k, i)
    714             except ValueError as err:

~/sandbox/pandas/pandas/core/indexing.py in _validate_key(self, key, axis)
   1379         else:
-> 1380             raise ValueError(f"Can only index by location with a [{self._valid_types}]")
   1381

ValueError: Can only index by location with a [integer, integer slice (START point is INCLUDED, END point is EXCLUDED), listlike of integers, boolean array]

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-97-09bd029d0285> in <module>
----> 1 a - b

~/sandbox/pandas/pandas/core/ops/__init__.py in f(self, other, axis, level, fill_value)
    634             self, other, op, axis, default_axis, fill_value, level
    635         ):
--> 636             return _frame_arith_method_with_reindex(self, other, op)
    637
    638         if isinstance(other, ABCSeries) and fill_value is not None:

~/sandbox/pandas/pandas/core/ops/__init__.py in _frame_arith_method_with_reindex(left, right, op)
    568     )
    569
--> 570     new_left = left.iloc[:, lcols]
    571     new_right = right.iloc[:, rcols]
    572     result = op(new_left, new_right)

~/sandbox/pandas/pandas/core/indexing.py in __getitem__(self, key)
    882                     # AttributeError for IntervalTree get_value
    883                     pass
--> 884             return self._getitem_tuple(key)
    885         else:
    886             # we by definition only have the 0th axis

~/sandbox/pandas/pandas/core/indexing.py in _getitem_tuple(self, tup)
   1452     def _getitem_tuple(self, tup: Tuple):
   1453
-> 1454         self._has_valid_tuple(tup)
   1455         try:
   1456             return self._getitem_lowerdim(tup)

~/sandbox/pandas/pandas/core/indexing.py in _has_valid_tuple(self, key)
    713                 self._validate_key(k, i)
    714             except ValueError as err:
--> 715                 raise ValueError(
    716                     "Location based indexing can only have "
    717                     f"[{self._valid_types}] types"

ValueError: Location based indexing can only have [integer, integer slice (START point is INCLUDED, END point is EXCLUDED), listlike of integers, boolean array] types

[jbrockmendel]

this is working on master, needs test