BUG: df[frozenset] fails if (other) column names are not unique

[RagBlufThim]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd

k = frozenset(["KEY"])

df1 = pd.DataFrame([[1,2,3,4], [5,6,7,8]], columns=[k, "B", "C", "D"])
df2 = pd.DataFrame([[1,2,3,4], [5,6,7,8]], columns=[k, "B", "C", "C"])

print(df1[k])  # works
print(df2[k])  # KeyError: "None of [Index(['KEY'], dtype='object')] are in the [columns]"

Problem description

Indexing a dataframe with a frozenset raises a KeyError if column names are not unique (the name of the indexed column is unique):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\test\venv124\lib\site-packages\pandas\core\frame.py", line 3030, in __getitem__
    indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
  File "C:\test\venv124\lib\site-packages\pandas\core\indexing.py", line 1266, in _get_listlike_indexer
    self._validate_read_indexer(keyarr, indexer, axis, raise_missing=raise_missing)
  File "C:\test\venv124\lib\site-packages\pandas\core\indexing.py", line 1308, in _validate_read_indexer
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['KEY'], dtype='object')] are in the [columns]"

Indexing another dataframe that only differs by having unique column names works, so using a frozenset as column name seems to be ok.

The traceback seems to indicate that a "listlike_indexer" is involved - maybe with non-unique column names a different indexer is used that has problems with non-atomic column names, turning the column name frozenset(["KEY"]) into "KEY". Just a guess based on the error message, I don't know anything about how pandas indexers work.

Expected Output

Indexing works with a column name of arbitrary type regardless of whether the other column names are unique or not.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.9.4.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
...

pandas : 1.2.4
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.8.1
...

[ma3da]

How should pandas actually behave here? Should arbitrary labels be accepted?
When given a container X, how does df.loc decide if it should return the series df[X] or a dataframe, say, df[[x_1, ..., x_n]]?

[RagBlufThim]

How should pandas actually behave here? Should arbitrary labels be accepted?

The user guide mentions "there’s nothing preventing you from using tuples as atomic labels on an axis". If a tuple is officially allowed as label, a frozenset should be as well. I seem to be not the only one who thinks so, see #41238.
More generally I wouldn't mind if any (hashable) object were allowed as label.

When given a container X, how does df.loc decide if it should return the series df[X] or a dataframe, say, df[[x_1, ..., x_n]]?

If the column labels of df are unique, df[X] currently returns the series with label X if X is a hashable container (e.g. tuple or frozenset), but a dataframe with the column labels given in X if X is not hashable, e.g. a list or set (see first section of example session below).

I expect(ed) df.loc[:, X] to behave like df[X]. However, it doesn't, it returns a dataframe even if X is a tuple or frozenset (see second section of example session below).

If the column labels of df are not unique, df[X] behaves differently again: It returns a dataframe for a frozenset label and raises a TypeError for a tuple label (see third section of example session below)..

This issue is about the discrepancy in behaviour dataframe with unique column labels vs. dataframe with non-unique column labels.
I think #41238 is (at least partly) about the discrepancy df[X] vs. df.loc[:, X].

My personal preference would be that a hashable container is treated as label, a non-hashable container as containing labels, and this regardless of whether the labels of df are unique or not, and consistently across df[X] vs. df.loc[:, X].

Example session:

>>> import pandas as pd
>>>
>>> dfu = pd.DataFrame([[1,2,3,4]], index=[1, 2], columns=["A", "B", ("A", "B"), frozenset(["A", "B"])])
>>> dfu
   A  B  (A, B)  (A, B)
1  1  2       3       4
2  1  2       3       4
>>> dfu[["A", "B"]]
   A  B
1  1  2
2  1  2
>>> dfu[("A", "B")]
1    3
2    3
Name: (A, B), dtype: int64
>>> dfu[frozenset(["A", "B"])]
1    4
2    4
Name: (A, B), dtype: int64
>>> dfu[set(["A", "B"])]
   A  B
1  1  2
2  1  2
>>>
>>> # Now .loc
>>> dfu.loc[:, ["A", "B"]]
   A  B
1  1  2
2  1  2
>>> dfu.loc[:, ("A", "B")]
   A  B
1  1  2
2  1  2
>>> dfu.loc[:, frozenset(["A", "B"])]
   A  B
1  1  2
2  1  2
>>> dfu.loc[:, set(["A", "B"])]
   A  B
1  1  2
2  1  2
>>>
>>> # Now a df with non-unique column labels
>>> dfn = pd.DataFrame([[1,2,3,4,5,6]], index=[1, 2], columns=["A", "B", ("A", "B"), frozenset(["A", "B"]), "C", "C"])
>>> dfn
   A  B  (A, B)  (A, B)  C  C
1  1  2       3       4  5  6
2  1  2       3       4  5  6
>>> dfn[["A", "B"]]
   A  B
1  1  2
2  1  2
>>> dfn[("A", "B")]
C:\test\venv124\lib\site-packages\pandas\core\indexes\base.py:3080: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  return self._engine.get_loc(casted_key)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\test\venv124\lib\site-packages\pandas\core\frame.py", line 3024, in __getitem__
    indexer = self.columns.get_loc(key)
  File "C:\test\venv124\lib\site-packages\pandas\core\indexes\base.py", line 3080, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas\_libs\index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 96, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 126, in pandas._libs.index.IndexEngine._get_loc_duplicates
  File "pandas\_libs\index.pyx", line 132, in pandas._libs.index.IndexEngine._maybe_get_bool_indexer
TypeError: Cannot convert bool to numpy.ndarray
>>> dfn[frozenset(["A", "B"])]
   A  B
1  1  2
2  1  2
>>> dfn[set(["A", "B"])]
   A  B
1  1  2
2  1  2
>>>