BUG: DataFrame.drop_duplicates method fails when a column with a list dtype is present

[zdafoe]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame(
    [ 
        {"number": 1, "item_ids": [1, 2, 3]},
        {"number": 1, "item_ids": [1, 2, 3]},
    ]
)
df.drop_duplicates()

Issue Description

DataFrame.drop_duplicates and DataFrame.duplicated fail if there is a column of the list dtype in the subset parameter and there is more than one column in subset

Expected Behavior

should output

   number   item_ids
0      1   [1, 2, 3]

Installed Versions

INSTALLED VERSIONS

commit : 04b45b1
python : 3.11.7.final.0
python-bits : 64
OS : Linux
OS-release : 6.5.0-14-generic
Version : #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023
machine : x86_64
processor :
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.3.0.dev0+54.g04b45b10b1
numpy : 1.26.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 69.0.3
pip : 23.3.1
Cython : None
pytest : 7.4.3
...
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report, but these methods rely on hashing and mutable objects are not hashable. As such, I don't think we can support such columns.

However, this limitation is not in the documentation. I think it would be good to add this documentation.

[EdDonlon]

take

[manlattan]

take