BUG: `KeyError: '[nan] not in index'` when using `nan` to index

[ssche]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas. (tested on 2.0.0rc1, assuming that's close enough to main)

Reproducible Example

>>> import numpy as np
>>> import pandas as pd
>>>
>>> df = pd.DataFrame(index=pd.Index([np.nan, 100.0, 200.0, 300, 100.0]))
>>> ix = pd.Index([100, np.nan])
>>> df.loc[ix]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "venv/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1301, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1239, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1432, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "venv/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6133, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: '[nan] not in index'

Issue Description

I'm using a row index which can contain nan values, but I'm unable to use it to index the rows in the dataframe.

However when I try to convert the index to a mask, it seems to be working:

>>> m = df.index.isin(ix)
>>> m
array([ True,  True, False, False,  True])
>>> df[m]
Empty DataFrame
Columns: []
Index: [nan, 100.0]

Expected Behavior

>>> df.loc[ix]

returns

Empty DataFrame
Columns: []
Index: [nan, 100.0]

just like df[m] does.

Installed Versions

INSTALLED VERSIONS

commit : 2e218d1
python : 3.8.16.final.0
python-bits : 64
OS : Linux
OS-release : 6.1.18-200.fc37.x86_64
Version : #1 SMP PREEMPT_DYNAMIC Sat Mar 11 16:09:14 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8

pandas : 1.5.3
numpy : 1.23.2
pytz : 2020.4
dateutil : 2.8.1
setuptools : 59.6.0
pip : 22.2.2
Cython : 0.29.32
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 0.9.6
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6
jinja2 : 2.11.2
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : 1.3.5
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.8.1
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 1.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
snappy : None
sqlalchemy : 1.3.23
tables : 3.7.0
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
zstandard : None
tzdata : None

[ssche]

Interestingly, indexing with nan works, if it's the only element.

See this example for which I added an extra column 'a' compared to the initial post (to make it easier to follow):

>>> df = pd.DataFrame({'a': [1, 2, 3, 4, 5]}, index=pd.Index([np.nan, 100.0, 200.0, 300, 100.0]))
>>>
>>> df
       a
NaN    1
100.0  2
200.0  3
300.0  4
100.0  5

Indexing with np.nan works as expected...

>>> df.loc[np.nan, 'a']
1

... just like like indexing with a number...

>>> df.loc[200.0, 'a']
3

... or multiple index elements...

>>> df.loc[[100, 200], 'a']
100.0    2
100.0    5
200.0    3
Name: a, dtype: int64

... but as soon as np.nan is part of to the index, a KeyError is raised:

>>> df.loc[[100, 200, np.nan], 'a']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1067, in __getitem__
    return self._getitem_tuple(key)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1247, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 991, in _getitem_lowerdim
    return getattr(section, self.name)[new_key]
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1301, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1239, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "env/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1432, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "env/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "env/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6133, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: '[nan] not in index'

As soon as np.nan is used in a list (or similar), the KeyError occurs (Note: df.loc[np.nan, 'a'] works just fine).

>>> df.loc[[np.nan], 'a']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1067, in __getitem__
    return self._getitem_tuple(key)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1247, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 991, in _getitem_lowerdim
    return getattr(section, self.name)[new_key]
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1301, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1239, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexing.py", line 1432, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "env/blendopt-py38/lib64/python3.8/site-packages/pandas/core/indexes/base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Float64Index([nan], dtype='float64')] are in the [index]"

[ssche]

This is also an issue in pandas 2.0.0rc1

>>> df.loc[[np.nan], 'a']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1097, in __getitem__
    return self._getitem_tuple(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1280, in _getitem_tuple
    return self._getitem_lowerdim(tup)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1024, in _getitem_lowerdim
    return getattr(section, self.name)[new_key]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1103, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1332, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1272, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexing.py", line 1462, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexes/base.py", line 5873, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "envs/pandas2/lib64/python3.11/site-packages/pandas/core/indexes/base.py", line 5932, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index([nan], dtype='object')] are in the [index]"

INSTALLED VERSIONS
------------------
commit           : c2a7f1ae753737e589617ebaaff673070036d653
python           : 3.11.2.final.0
python-bits      : 64
OS               : Linux
OS-release       : 6.1.18-200.fc37.x86_64
Version          : #1 SMP PREEMPT_DYNAMIC Sat Mar 11 16:09:14 UTC 2023
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_AU.UTF-8
LOCALE           : en_AU.UTF-8

pandas           : 2.0.0rc1
numpy            : 1.24.2
pytz             : 2023.2
dateutil         : 2.8.2
setuptools       : 62.6.0
pip              : 22.2.2
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
brotli           : None
fastparquet      : None
fsspec           : None
gcsfs            : None
matplotlib       : None
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : None
snappy           : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
zstandard        : None
tzdata           : None
qtpy             : None
pyqt5            : None

[ltauro]

take

[ssche]

It seems that "all" that's missing to resolve this is to remove the check

    self._raise_if_missing(keyarr, indexer, axis_name)

in pandas/core/indexes/base.py, line 6070, in _get_indexer_strict

>>> import pandas as pd
>>> import numpy as np
>>> df = pd.DataFrame({'a': [1, 2, 3, 4, 5]}, index=pd.Index([np.nan, 100.0, 200.0, 300, 100.0]))
>>> df.loc[[np.nan], 'a']
100.0   NaN
Name: a, dtype: float64
>>> 
>>> 
>>> df.loc[[np.nan, np.nan], 'a']
100.0   NaN
100.0   NaN
Name: a, dtype: float64

Can this be done in pandas 1.5.x?

[ssche]

In the interim, would someone be able to propose a workaround that does the same thing (Index and MultiIndex)?

My current workaround is to monkeypatch Index, but that may hide required exceptions (although in base.py I couldn't find another place where _raise_if_missing is called):

from pandas.core.indexes.base import Index

def _ignore_if_missing(self, *args, **kwargs):
    pass

setattr(Index, '_raise_if_missing', _ignore_if_missing)

[ltauro]

@ssche I was able to reproduce this error. I am going to work on fixing this and submit a pull request as soon as possible.

[vanxLord90]

i had the same issue, i downloaded pandas in May from pip, what exactly is the solution to this issue?