different behaviour of df.isin() in 1.0.5/1.1.0, when df contains None

[johny-b]

>>> import pandas as pd
>>> x = pd.DataFrame([['foo', 'bar'], [1, None]])
>>> y = x[1].copy()

# PANDAS 1.0.5
>>> x.isin(y)
       0     1
0  False  True
1  False  True

# PANDAS 1.1.0
>>> x.isin(y)
       0      1
0  False   True
1  False  False

Is this change intended?

[simonjayhawkins]

Thanks @johny-b for the report.

Is this change intended?

I doubt it. running git bisect now to investigate.

[simonjayhawkins]

#31296 cc @jbrockmendel

19ae087 is the first bad commit
commit 19ae087
Author: jbrockmendel jbrockmendel@gmail.com
Date: Sun Mar 8 09:07:03 2020 -0700

PERF: do DataFrame.op(series, axis=0) blockwise (#31296)

[johny-b]

Take a look also on this, on 1.0.5:

>>> x = pd.DataFrame([['bar', None]])
>>> x.isin(x[1])
       0     1
0  False  True
>>> x = pd.DataFrame([[1, None]])
>>> x.isin(x[1])
       0      1
0  False  False

This looks like a bug to me, and this change fixes it (at least improves consistency, on 1.1.0 this is always False)

[simonjayhawkins]

possibly related to #19356 and #34125

tbh, I find the usage with Series and DataFrame non-intuitive anyway as they are not treated as unordered collections and if the Series is aligned on labels, I don't understand why isin doesn't have an axis keyword for broadcasting.

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isin.html

[rhshadrach]

The result is determined here:

pandas/pandas/_libs/ops.pyx

Lines 163 to 169 in 70c056b

	x = left[i]
	y = right[i]
	if checknull(x) or checknull(y):
	result[i] = False
	else:
	result[i] = PyObject_RichCompareBool(x, y, flag)

where checknull is True for both x and y. This is consistent with e.g. np.nan != np.nan but inconsistent with None == None.

[simonjayhawkins]

moved off 1.1.2 milestone (scheduled for this week) as no PRs to fix in the pipeline