ROADMAP: Consistent missing value handling with new NA scalar

[jorisvandenbossche]

I cleaned up my initial write up on the consistent missing values proposal (#27825 (comment)), and incorporated the items brought up in the last video chat. So I think it is ready for some more detailed discussion.

The last version of the full proposal can be found here: https://hackmd.io/@jorisvandenbossche/Sk0wMeAmB

TL;DR:

• I propose to introduce a new scalar (singleton) pd.NA that can be used as the missing value indicator (when accessing a single value, not necessarily how it is stored under the hood).
• This can be used instead of np.nan or pd.NaT in new data types (eg nullable integers, potential string dtype)
• Long term, we can see if there is a migration possible to use this consistently for all data types.

cc @pandas-dev/pandas-core

[jorisvandenbossche]

Technical note: following our roadmap, I posted an issue with the proposal, but we need to see a bit how to have those discussions. As I posted the full proposal externally and not in a PR, it's more difficult to have inline comments (although I can also post it on eg google docs which is a bit friendlier for that). But alternatively, we could also do PRs instead of issues?)

[shoyer]

This is an interesting proposal!

I'm mostly concerned about type stability / predictability, e.g.,

• What is pd.Series([pd.NA]).dtype? (with np.nan this would be float64)
• What is pd.Timestamp('2000-01-01') - pd.NA? (could be either Timestamp or Timedelta)

We might need an NA dtype in order to have well-defined semantics for all of these operations.

It would also be worth consider whether we would like to deviate from NaN semantics in some cases. For example, it might make sense to define NA == other as NA rather than False.

Julia has a nice documentation page explain how they support missing values. It might be a good model to emulate for pandas. Two notable difference about Julia:

• They have robust union types and use them for missing values (e.g., Union{Missing, Int64}), which solves the isinstance problem.
• They have only have multi-methods/functions, not methods like Python, so they don't need to worry about defining type specific methods/properties on missing values, e.g., NA.year

[jorisvandenbossche]

Thanks for the feedback!

I'm mostly concerned about type stability / predictability, e.g.,

Yes, that's an item I briefly mentioned in the proposal with similar examples. We can (and will need to) decide on certain rules how to go about this, but they can still be unpredictable for users of course..

I also think we might need a NA dtype. If we do, that would probably mean that we return this NA type instead of guessing between eg timedelta or timestamp? However, but then what with operations that are less ambiguous? Eg float + 'someting' in principle either gives an error (if something is not a number) or gives a float. So if this gives a result (not errors), it is always float. But then should float + NA then give float? Similarly for timestamp + 'something' (while int + 'something' can have multiple return types depending on 'something')

I will need to think a bit more on usage implications of this.

It would also be worth consider whether we would like to deviate from NaN semantics in some cases. For example, it might make sense to define NA == other as NA rather than False.

That's also something I have been thinking about. I now included some additional text about this at the bottom of the proposal.

Ideally, I think we would do something like that and follow what Julia (and also mostly R and SQL) are doing (meaning: propagating NA on comparisons, having three-valued logic for boolean operations). However, this also has some additional complications, certainly if we want to introduce this gradually.
For example, assume we start using pd.NA in the new integer (and coming string) dtype, but keep the old np.nan for object / float. That would mean that NA behaviour in comparisons etc would work differently between both, initially leading to inconsistencies.

[WillAyd]

Read through but not sure about this. AFAIK for other languages (particularly Julia) there was an issue distinguishing between None and np.nan; there are some rough edges but I'd say that distinction is mostly clear in pandas.

For extension types we'd (I assume) mostly be masking the location of missing values anyway, so what particular advantage do you see to returning pd.NA from those instead of just np.nan?

[jorisvandenbossche]

(sorry for the slow reply, busy EuroScipy)

AFAIK for other languages (particularly Julia) there was an issue distinguishing between None and np.nan;

Can you elaborate a bit more on this sentence?

For extension types we'd (I assume) mostly be masking the location of missing values anyway, so what particular advantage do you see to returning pd.NA from those instead of just np.nan?

For sure, we could also return np.nan as the scalar value (you could basically replace pd.NA with np.nan in the proposal), and this is something our users are already familiar with.
Although somewhat subjectively, some reasons why I don't really like that solution:

• In general, np.nan is a float value (eg what you get from 0/0), and not a missing value indicator. This is maybe more a theoretical reason (as we have been using np.nan in practice for a long time as missing value indicator), but to me it "feels wrong" to start using this more broadly for new data types as the missing value indicator (because up to now, it was mostly used in actual float dtype, and in object dtype). It feels strange to return "not a number" for a missing value of strings.
  And a less theoretical reason is both R, Julia and SQL distinguish both and have a separate NA/NULL concept from float NaN.
• Personally (and given the above), I think for consistency reasons we should rather choose between either having a "not a value" sentinel per dtype (NaT, NaS(tring), NaI(nteger), which can behave more as times, strings or integers, respectively) or having a single pd.NA scalar (and from those two I personally prefer the last). This disucssion came also up in Tom experimenting with a NotAString value.
• Do we want, at some point, also support missing values in boolean columns? In that case, we need to decide the behaviour in certain operations, related to the questions that @shoyer brought up. If we want to follow there the behaviour that boolean arrays with missing values have in other languages, this might deviating from np.nan behaviour I think.

Originally I was planning to write a proposal combining the new sentinel value and consistently using mask-based approach for all dtypes (which more easily enables being consistent with the scalar value as well). Choosing a specific implementation (eg mask-based) will have much more impact on our code base, a possible choice for pd.NA over np.nan as the scalar value is much more a user-facing API question, where I find that pd.NA can give a more consistent, easier to teach pandas API.

[jbrockmendel]

In general, np.nan is a float value (eg what you get from 0/0), and not a missing value indicator.

So if I have a Series[float64] could it contain both np.nan and pd.NA, and these signify different things?

Misc thoughts:

• Assuming pd.NA is implemented in cython, checks for it would be more performant than checks for np.nan scalar (since that is not a singleton).
• Does consistency with arrow play a factor in your thought process? Note pa.NA == pa.NA
• Would it be worthwhile to implement a more limited pd.NA that was mostly just for object dtype? (maybe categorical. I think "non-arithmetic-supporting" would be the precise distinction)
  • maybe stop treating None as na, so that all recognized na scalars satisfy x != x.
  • stop casting np.nan to NaT when inserting into dt64 or td64 Series/DTA/TDA, but allow pd.NA to be inserted.
  • DTA[i] continues to return NaT instead of pd.NA, so arithmetic doesn't get broken

[jorisvandenbossche]

So if I have a Series[float64] could it contain both np.nan and pd.NA, and these signify different things?

In theory, yes (that is what R, Julia, SQL, .. do). In practice, though, we will certainly need some kind of option (and as default in the beginning) to still treat np.nan as a missing value.

Does consistency with arrow play a factor in your thought process?

Partly, yes, but not as the main reason (but you could see Arrow as "yet another example" next to other R/Julia/SQL that distinguish both concepts, and a python one). Note that Arrow nowadays uses "null" instead of "na" (but that's just the name, not a difference in concept).

I am using R regularly as example, but I am certainly not an expert in R. Eg they distinguish both, but it seems that NaN behaves more like NA there compared to numpy's np.nan (so it's maybe not the best reference to compare to).

Would it be worthwhile to implement a more limited pd.NA that was mostly just for object dtype? (maybe categorical. I think "non-arithmetic-supporting" would be the precise distinction)

I suppose with the idea to avoid the dubious situations in what the result type should be? (eg NA + 1 -> float or int or ?)

I don't really like that on the long term (for consistency reasons), but on the short term this might be what we will do in practice anyway. But, eg in that idea, what sentinel would we use for nullable integers? Still np.nan as today I suppose. But, that gives you already the same dubious situation (what is the result type of IntegerArray([1, 2]) + np.nan, int or float?)

maybe stop treating None as na, so that all recognized na scalars satisfy x != x.

That's certainly something to consider as well, I think. Although (given eg #28124), I suppose there are people that like to use None since that can play better with other python packages.

[jbrockmendel]

I suppose with the idea to avoid the dubious situations in what the result type should be? (eg NA + 1 -> float or int or ?)

That example works, but more problematic examples are Series[datetime64] - pd.NA and Series[timedelta64] / pd.NA. I know we discussed this on the last call, not sure if there is a thread on this.

[jorisvandenbossche]

Why is that more problematic? Because float and int, although different types, still can refer the same (semantic) value, while timestamp and timedelta are different concepts. That's indeed a bigger difference.
The proposal suggests the idea of a "NA dtype", which could potentially help remediate this case (I won't say "solve" ;)). Or at least, worth to think through a few use cases to see if it can help.

I think this thread is the place to discuss this.