ENH: Add option for DataFrame.compare and Series.compare to overlook differences involving missing values

[qmarcou]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

I'd like to use the compare() method to extract only contradictory data between datasets, not all changes in the dataset.
For instance:

import pandas as np
a = pd.Series([1, pd.NA, pd.NA])
b = pd.Series([2, "zz", pd.NA])
a.compare(b)
Out[3]: 
   self other
0     1     2
1  <NA>    zz

While the first row is a clear cut difference /contradiction (used to be 1 and is now 2), the second row is a bit more ambiguous and could be interpreted either as:

• used to be a pd.NA object ans is now a str with a set value (current behavior, completly sound as a default behavior imo)
• used to be something we didn't know but might as well have been "zz", and is now "zz", hence no difference reported (desired optional feature)

This behavior has not been discussed in #30852 and its source issue #30429

Feature Description

Add a new keyword argument such as skipna for aggregation functions to compare such that

a.compare(b, skipna=False)
Out: 
   self other
0     1     2
1  <NA>    zz
a.compare(b, skipna=True)
Out: 
   self other
0     1     2

skipna=False would be the default value to conform with the current default behavior (which is stricter/safer and thus more sound as default behavior)

Alternative Solutions

I can get the desired output by first erasing NA-linked differences using combine_first as a workaround:

def skipna_compare(a: pd.Series | pd.DataFrame,
                   b: pd.Series | pd.DataFrame
                   ) -> (pd.Series | pd.DataFrame):
    filled_a = a.combine_first(b)
    filled_b = b.combine_first(a)
    return filled_a.compare(filled_b)
skipna_compare(a,b)
Out: 
  self other
0    1     2

But I don't think this is really efficient and satisfactory.

Additional Context

Thanks for the amazing work!

[rhshadrach]

Thanks for the feature request. I am personally quite negative on it. It seems generally unusual to have NA values viewed as equal to non-NA values to be worth the cost of inclusion in pandas (code, review, tests, docs, bugfixes). As there is a 3-line alternative users can do, it seems best for me to live in user-code.

[qmarcou]

Thanks for your reply!
I can't argue with the code cost, although I could propose a PR for it :)
Codewise it seems there would only be few lines to change, beyond adding the actual parameters to the function and docs. The main change would be to this fillna line:

pandas/pandas/core/generic.py

Lines 9229 to 9236 in 642d244

	# error: Unsupported left operand type for & ("Self")
	mask = ~((self == other) | (self.isna() & other.isna())) # type: ignore[operator]
	mask.fillna(True, inplace=True)
	if not keep_equal:
	self = self.where(mask)
	other = other.where(mask)

However I'll add a slight precision. Sorry if my initial post was unclear on this, I didn't mean to consider NA values as equal to non-NA values (that would mean NA==obj is True). I meant to consider the result of this comparison as undetermined NA==obj is NA as per pandas NA propagation logic, and offer the possibility to neglect these undetermined results in the compare results, as is available for other pandas methods.

Regarding the 3-liner solution it took me some time to get there. I initially tried to post-process the df returned by compare, and while it is straightforward in the series case, the dataframe format requires a bit more fiddling.

[rhshadrach]

Sorry if my initial post was unclear on this, I didn't mean to consider NA values as equal to non-NA values (that would mean NA==obj is True). I meant to consider the result of this comparison as undetermined NA==obj is NA as per pandas NA propagation logic, and offer the possibility to neglect these undetermined results in the compare results, as is available for other pandas methods.

Isn't this effectively the same as considering NA to be truthy rather than falsey?

cc @mroeschke for any thoughts.

[mroeschke]

offer the possibility to neglect these undetermined results in the compare results, as is available for other pandas methods.

I guess this would be equivalent to calling .dropna after the compare call to ignore these results?

I would agree with @rhshadrach; I think since there are various methods to post-process the compare results, an option doesn't need to live in compare itself. Thanks for the suggestion but closing