BUG: pandas.core.groupby.SeriesGroupBy.idxmax (and idxmin) - unexpected behavior with groups containing single NaN value

[semyonbok]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "var": ["A", "B", "C", "C", "D", "D"],
    "val": [11, np.nan, 12, 13, 14, np.nan]
})

df.groupby("var")["val"].idxmax()

Issue Description

The provided code snippet yields the following output:

var
A    0
B    0
C    3
D    4
Name: val, dtype: int64

Group 'B' with single np.nan value yields 0, which corresponds to the index of the first entry in the frame, relating to group "A".

Expected Behavior

The output with pandas 2.1.0

var
A    0.0
B    NaN
C    3.0
D    4.0
Name: val, dtype: float64

Installed Versions

INSTALLED VERSIONS

commit : f538741
python : 3.9.16.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United Kingdom.1252

pandas : 2.2.0
numpy : 1.26.3
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 69.0.3
pip : 23.3.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.9
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.18.1
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.2
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.12.0
sqlalchemy : None
tables : 3.9.2
tabulate : None
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None

[remiBoudreau]

I'm unable to produce regardless of version on Ubuntu 22.04.3 LTS and Docker container

[semyonbok]

@remiBoudreau, thank you for giving it a try. I can add that I use Miniconda package manager; and I installed the latest pandas through conda-forge channel. I've just tried it on another Windows machine, and observed the same behaviour. Furthermore, groups with single np.nan are not just zeroed-out, but assigned with the first object from the index in my case.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "var": ["A", "B", "C", "C", "D", "D", "E"],
    "val": [11, np.nan, 12, 13, 14, np.nan, np.nan]
}, index=list("abcdefg"))

df.groupby("var")["val"].idxmax()

outputs:

var
A    a
B    a
C    d
D    e
E    a
Name: val, dtype: object

Finally, I see that _idxmax_idxmin logic is quite new. Please let me know if there anything useful I can provide from my end.

[rhshadrach]

Thanks for the report. This was fixed in #57046 and has the same root cause. The fix will be part of 2.2.1.

[semyonbok]

@rhshadrach, great news, thank you very much! I'll do my best to try the main branch prior to raising an issue next time :)