Dispatch pd.is_na for scalar extension value

[TomAugspurger]

Right now, I don't believe there's a way for an ExtensionDtype to declare a custom scalar NA value and have pd.isna(scalar) do the right thing.

_nas = object()


class NaSType(str):
    """
    NA for String type.
    """

    # TODO: enforce singleton

    def __new__(cls, value):
        if value is not _nas:
            raise ValueError("Cannot create NaS from '{}'".format(value))
        return super().__new__(cls, value)

    def __eq__(self, other):
        # TODO: array comparisons, etc.
        return False

    def __str__(self):
        return "NaS"

    def __repr__(self):
        return str(self)


NaS = NaSType(_nas)


@register_extension_dtype
class StringDtype(ExtensionDtype):

    @property
    def na_value(self):
        return NaS

    @property
    def type(self) -> Type:
        return str

    @property
    def name(self) -> str:
        return "string"

    @classmethod
    def construct_from_string(cls, string: str):
        if string in {"string", "str"}:
            return cls()
        return super().construct_from_string(string)

    @classmethod
    def construct_array_type(cls) -> "Type[StringArray]":
        return StringArray

In [18]: NaS
Out[18]: NaS

In [19]: pd.isna(NaS)
Out[19]: False

That should be True. In https://github.com/pandas-dev/pandas/blob/master/pandas/core/dtypes/missing.py#L131-L132 we go straight to lib missing.checknull(obj) for scalar values.

[jbrockmendel]

One workaround would be to add a isna method to NaSType that always returns True.

[jorisvandenbossche]

Or: don't use such a custom NaS, but a default NA provided by pandas ?

[TomAugspurger]

We need to have a discussion about whether we want more or fewer NAs. If we want one, then yes that would be an option.

…

On Thu, Aug 8, 2019 at 3:31 PM Joris Van den Bossche < ***@***.***> wrote: Or: don't use such a custom NaS, but a default NA provided by pandas ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#27825?email_source=notifications&email_token=AAKAOITGUPOYE77XZ3OVTHLQDR7ATA5CNFSM4IKNWL2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD342DEQ#issuecomment-519676306>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIT5NUZDQUAA6FNOT4DQDR7ATANCNFSM4IKNWL2A> .

[jorisvandenbossche]

I have been pondering over a roadmap discussion post about that the last week, so I suppose I should try to write that :)

[TomAugspurger]

It'd be good to do that before the meeting next week :)

[jorisvandenbossche]

I couldn't finish it yet, but a very rough sketch can already be found here: https://hackmd.io/gPDsgTsKRlyHfoCRR1i_ng (in case you already want to think on it)