DISCUSS: boolean dtype with missing value support

[jorisvandenbossche]

Part of the discussion on missing value handling in #28095, detailed proposal at https://hackmd.io/@jorisvandenbossche/Sk0wMeAmB.

if we go for a new NA value, we also need to decide the behaviour of this value in comparison operations. And consequently, we also need to decide on the behaviour of boolean values with missing data in logical operations and indexing operations.
So let's use this issue for that part of the discussion.

Some aspects of this:

• Behaviour in comparison operations: currently np.nan compares unequal (value == np.nan -> False, values > np.nan -> False, but we can also propagate missing values (value == NA -> NA, ...)
• Behaviour in logical operations: currently we always return False for | or & with missing data. But we could also use a "three-valued logic" like Julia and SQL (this has, eg, NA | True = True or NA & True = NA).
• Behaviour in indexing: currently you cannot do boolean indexing with a boolean series with missing values (which is object dtype right now). Do we want to change this? For example, interpret it as False (not select it)
  (TODO: should check how other languages do this)

Julia has a nice documentation page explain how they support missing values, the above ideas largely match with that.

Besides those behavioural API discussions, we also need to decide on how to approach this technically (boolean ExtensionArray with boolean numpy array + mask for missing values?) Shall we discuss that here as well, or keep that separate?

cc @pandas-dev/pandas-core

[jorisvandenbossche]

(also if we don't go for a new NA value, a boolean ExtensionArray with missing data support can be interesting, but in such a case it's probably more difficult to change the behaviour compared to what we currently have with np.nan)

[WillAyd]

I think I prefer what R and Julia do here but would be curious to hear counter arguments in support of existing behavior.

Just to clarify, you think any operation where one of the operands is NA should return NA right? But something like isna would return True in the presence of an NA? (assuming that from how Julia has the === operator)

[jbrockmendel]

Behaviour in logical operations: currently we always return False for | or & with missing data.

This is not accurate. These ops are basically nothing but corner cases, a handful of which do three-value logic. That's before considering DataFrame, which only sometimes behaves like Series.

I'll elaborate later this afternoon.

[jorisvandenbossche]

would be curious to hear counter arguments in support of existing behavior.

@WillAyd I think the existing behaviour is mainly a consequence of using np.nan (for which the existing behaviour makes sense). And an argument to keep the existing behaviour would be that we have done it like that for a long time..

Just to clarify, you think any operation where one of the operands is NA should return NA right?

For comparisons yes, for logical operations it depends. I pasted below the more elaborate explanation with code examples that I wrote in the proposal on hackmd.

But something like isna would return True in the presence of an NA?

Yes, that wouldn't change compared to the current behaviour I think.

---

Behaviour in comparison operations

In numerical operations, NA propagates (see also above). But for boolean operations the situation is less clear. Currently, we use the behaviour of np.nan for missing values in pandas. This means:

>>> np.nan == 1
False
>>> np.nan < 1
False
>>> np.nan != 1
True

However, a missing value could also propagate:

>>> pd.NA == 1
NA
>>> pd.NA < 1
NA
>>> PD.NA != 1
NA

This is for example what Julia and R do.

Boolean data type with missing values and logical operations

If we propagate NA in comparison operations (see above), the consequence is that you end up with boolean masks with missing values. This means that we need to support a boolean dtype with NA support, and define the behaviour in logical operations and indexing.

• What to return in logical operations? (eg True & NA)
• How to handle NA's in indexing operations (raise error, or consider as False ?)

Currently, the logical operations are not very consistently defined. On Series/DataFrame, it returns mostly False, and for scalars it is not defined:

>>> pd.Series([True, False, np.nan]) & True
0     True
1    False
2    False
dtype: bool

>>> pd.Series([True, False, np.nan]) | True
0     True
1     True
2    False
dtype: bool

>>> np.nan & True
TypeError: unsupported operand type(s) for &: 'float' and 'bool'

For those logical operations, Julia, R and SQL choose for the "three-valued logic" (only propagate missing values when it is logically required). See https://docs.julialang.org/en/v1/manual/missing/index.html for a good explanation. This would give:

>>> pd.Series([True, False, pd.NA]) & True
0     True
1    False
2       NA
dtype: bool

>>> pd.NA & True
NA

>>> pd.NA & False
False

>>> pd.NA | True
True

>>> pd.NA | False
NA

[jorisvandenbossche]

For the question around the indexing behaviour with boolean values in the presence of NAs, I think there are 3 options:

• Raise an exception
• Don't include in the output (interpret NA as False in the filtering operation)
• Propagate NAs (NA in mask gives NA in output)

I looked at some other languages / libraries that deal with this.

Postgres (SQL) filters only where True (thus interprets NULL as False in the filtering operation):

CREATE TABLE test_types (
    col1    integer,
    col2    integer
);

INSERT INTO test_types VALUES (1, 1);
INSERT INTO test_types VALUES (2, NULL);
INSERT INTO test_types VALUES (3, 3);

test_db=# SELECT col1, col2 > 2 AS mask FROM test_types2;
 col1 | mask 
------+------
    1 | f
    2 | 
    3 | t
(3 rows)

test_db=# SELECT * FROM test_types2 WHERE col2 > 2;
 col1 | col2 
------+------
    3 |    3
(1 row)

In R, it depends on function. dplyr's filter drops NAs: Unlike base subsetting with [, rows where the condition evaluates to NA are dropped. (from https://dplyr.tidyverse.org/reference/filter.html). Example:

> df <- tibble(col1 = c(1L, 2L, 3L), col2 = c(1L, NA, 3L))
> df
# A tibble: 3 x 2
   col1  col2
  <int> <int>
1     1     1
2     2    NA
3     3     3
> df %>% mutate(mask = col2 > 2)
# A tibble: 3 x 3
   col1  col2 mask 
  <int> <int> <lgl>
1     1     1 FALSE
2     2    NA NA   
3     3     3 TRUE 
> df %>% filter(col2 > 2)
# A tibble: 1 x 2
   col1  col2
  <int> <int>
1     3     3

But so in base R, it propagates NAs (missing value in the index always yields a missing value in the output, from https://adv-r.hadley.nz/subsetting.html):

> x <- c(1, 2, 3)
> mask <- c(FALSE, NA, TRUE)
> x[mask]
[1] NA  3

Julia currently raises an error (but not sure if this is on purpose or just not yet implemented. EDIT: based on https://julialang.org/blog/2018/06/missing this seems to be on purpose):

julia> arr = [1 2 3]
1×3 Array{Int64,2}:
 1  2  3

julia> mask = [false missing true]
1×3 Array{Union{Missing, Bool},2}:
 false  missing  true

julia> arr[mask]
ERROR: ArgumentError: unable to check bounds for indices of type Missing
Stacktrace:
 [1] checkindex(::Type{Bool}, ::Base.OneTo{Int64}, ::Missing) at ./abstractarray.jl:504
 [2] checkindex at ./abstractarray.jl:519 [inlined]
 [3] checkbounds at ./abstractarray.jl:434 [inlined]
 [4] checkbounds at ./abstractarray.jl:449 [inlined]
 [5] _getindex at ./multidimensional.jl:596 [inlined]
 [6] getindex(::Array{Int64,2}, ::Array{Union{Missing, Bool},2}) at ./abstractarray.jl:905
 [7] top-level scope at none:0

Apache Arrow C++ (pyarrow) has currently the same behaviour as base R (propagating):

In [2]: import pyarrow as pa 

In [4]: arr = pa.array([1, 2, 3])

In [5]: mask = pa.array([False, None, True])

In [6]: mask
Out[6]: 
<pyarrow.lib.BooleanArray object at 0x7fa0f1b9f768>
[
  false,
  null,
  true
]

In [7]: arr.filter(mask) 
Out[7]: 
<pyarrow.lib.Int64Array object at 0x7fa0f1b9fd68>
[
  null,
  3
]

[jorisvandenbossche]

(comment from the hackmd copied here)

[@Dr-Irv] I think you will have to be careful when people mix numpy and pandas operations. Does (pd.Series([pd.NA], dtype=int) + np.array([0])).values return an array of np.nan or a pandas typed array with pd.NA inside?

In our ops code, pandas objects always have "priority" on numpy arrays. So if you do series + array, the result is a series (and hence in your example will contain pd.NA).

But, it's certainly true that when actually performing similar operations on the equivalent numpy arrays, you can get different results, certainly if np.nan and pd.NA behave differently. So that is a clear drawback of choosing for such different behaviour.

Hypothetical example:

>>> pd.Series([1, 2, pd.NA]) == 2
0    False
1     True
2       NA
dtype: bool

>>> np.asarray(pd.Series([1, 2, pd.NA])) == 2
array([False,  True, False])

But, this also relates with the question: how do we convert to numpy? (which wasn't really discussed yet) By default, if there are NAs, we could also convert to object dtype (like now for IntegerArray) preserving the pd.NA, and then you wouldn't get this different behaviour. And then it could be an option for the user to ask for a conversion to np.nan (to get a non-object float array), but it would be an explicit request of the user to get something different than pd.NA.

[jbrockmendel]

I think I promised to offer examples of weird edge cases and then this got lost in my inbox while travelling. Is that still something that would be useful?

[xhochy]

I recently made a prototype BooleanArray that deals with missing in the current pandas logic: https://uwekorn.com/2019/09/02/boolean-array-with-missings.html It shouldn't be hard to adapt that to output results in Julia/Kleene logic and also implement other operations like | on top of that.

[xhochy]

(extracted from #28095 (comment))

(NA == NA) = ?

In this case, I would expect also NA as NA is typically interpreted as unknown (actually in the discussion here just a synonym to missing but it makes the intuition a bit better), thus the outcome of the comparison of two unknown values is also unknown.

Naming

In general, I like the Julia documentation but I would prefer when we would stick to using more widely known terms of the three value logic like Kleene logic. With them, we have a good theoretical foundation we can refer to on what a computation should return and this might make communication with e.g. the database community a lot easier.

[seberg]

I actually ran into this in numpy a few days back. The np.ma.masked constant behaves like 0. when it comes to type promotion in ufuncs (which triggers value based promotion, which does what was discussed here float32_arr + np.ma.masked gives float32).

However, the way this currently works np.ma.masked is an array with a .dtype and I want to deprecate value based logic for those. Which in turn means I will need to add hooks so that np.ma.masked works like a python float here for now (I can do that, by exposing a things that I probably need to expose anyway, I can even do that from python using public API).

I guess, the main question that I am wondering right now:

1. Do you/we really want float32_arr + NA to work (at least for some functions, maybe not all)?
   I can see just as well to say that you have to use float32_arr + NA[Float32].

2. Do we want pd.Series([1, NA, 2, 3]) to work? Or force pd.Series([1, NA], dtype=Int64) or pd.series([1, NA[Int64]]).

Note that this does not touch that series[3] = NA works fine of course. series[:] = list_including_NAs might need some care to get always right. But I think that should not be an issue, since we can probably make this call pd.Series(list_including_NAs, dtype=series.dtype) (There is one larger issue with the last one, in that numpy array coercion currently does not know about safe/same_kind/unsafe casting...).

[Dr-Irv]

The discussion above leads me to the following idea, which I think I may have posted elsewhere.

I think it is important to distinguish between NA meaning "data missing" versus NaN meaning "not a number". In the current pandas world, there is no differentiation. Consider this simple example:

In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: s=pd.Series([0,4,np.nan], dtype=float)

In [4]: s2=pd.Series([0,2,1], dtype=float)

In [5]: s
Out[5]:
0    0.0
1    4.0
2    NaN
dtype: float64

In [6]: s2
Out[6]:
0    0.0
1    2.0
2    1.0
dtype: float64

In [7]: s3=s/s2

In [8]: s3
Out[9]:
0    NaN
1    2.0
2    NaN
dtype: float64

In this example s has missing data, and the result of the division in s3 has NaN in two places. One (s3[0]) is due to an arithmetic error, and the other (s3[2]) is due to missing data. I have run into cases where I wanted to know the difference, to help hunt down a bug, which could be a bug in my arithmetic, or could be due to missing data.

IMHO, these should be represented differently. The arithmetic/boolean operations that occur with a floating point NaN are well defined in standard floating point specification documentation. But if pandas had a separate NA that meant "missing data", then we are free to define how arithmetic and boolean operations work with NA values, and those rules do not have to follow the rules for NaN. We should also determine how NaN and NA interact.