Should we change PEP 484 to disable implicit Optional when default = None?

[gvanrossum]

CC: @ddfisher @JukkaL @markshannon

PEP 484 currently says:

An optional type is also automatically assumed when the default value is
None, for example::

  def handle_employee(e: Employee = None): ...

This is equivalent to::

  def handle_employee(e: Optional[Employee] = None) -> None: ...

This was intended as saving some typing in a common case, but I've received strong feedback from some quarters that this is not consistent and a bad idea. There are other places where None is allowed but none of them automatically add Optional (to the contrary).

So far it hasn't mattered much for mypy users because mypy doesn't have Optional support, but that's soon going to change (the --strict-optional flag is becoming more reliable) so if we're going to change this, now would be a good time. Thoughts? If we don't change this soon it will probably be too late.

[ddfisher]

I'm for this change. If I recall correctly, this is the only place where the PEP suggests typecheckers understand types to be something other than what's written as the type, so I think this change would be a nice simplification.

I've also started to notice this cause a bit of user confusion. See python/typeshed#500, for example, for all the arguments to builtin functions that were accidentally marked Optional due to this.

[ilevkivskyi]

I have seen some confusions between optional arguments and optional types, and the convention in question amplifies such confusions. Therefore I am in favour of removing it.

[gvanrossum]

OK, let's bring this up on python-dev once Python 3.6 beta 1 is released
(i.e. after next weekend).

On Sat, Sep 3, 2016 at 7:32 AM, Ivan Levkivskyi notifications@github.com
wrote:

I have seen some confusions between optional arguments and optional types,
and the convention in question amplifies such confusions. Therefore I am in
favour of removing it.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#275 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMpUgUvnteOgZfJvI0g_D50N3TOthks5qmYTugaJpZM4J0K87
.

--Guido van Rossum (python.org/~guido)

[dmoisset]

IMO most of the confusion comes from the name being wrong. An argument is «optional» when the parameter has a default in python. An «optional» type is a type that you can choose to add or not (as in gradual typing). If we had Nullable[T] instead of Optional many of these confusions wold go away (and it's the name used mostly everywhere else, other than Maybe T in Haskell).

Other than that, there is an inconsistency that may require a change (I just wanted to clarify that the confusions shouldn't be part of the argument)

[prechelt]

@dmoisset: I agree that Nullable or Nonable (1 char shorter) or Maybe (3 chars shorter) would be better solutions than Optional.

On the other hand, even the short Maybe is cumbersome. Nonable parameters are a frequent case and I believe the =None is sufficiently explicit to warrant introducing an additional PEP rule to cope with it, even if that rule has the character of an exception.

[markshannon]

I like that the Optional[] can be omitted if a None default is provided. It is less typing, reads naturally (to me) and is unambiguous.
If users want to use Optional they can since Optional[Optional[T]] == Optional[T].
As syntactic sugar goes, it seems quite sweet :)

OOI, which quarters?

[JukkaL]

I haven't had enough experience with using mypy strict optional checking to decide whether the extra verbosity would become awkward. Maybe we should wait until we have a substantial codebase that can be type checked using strict optional checking.

[markshannon]

Why do you use the word "strict"? That suggests that there is something "loose" about the syntactic sugar. There isn't; it is precisely defined.

[JukkaL]

Strict optional checking means that mypy treats None as a regular type, and mypy enforces that you add guards like if x: foo(x + 1) for values with an Optional[] type. Without strict optional checking, None is implicitly compatible with every type, kind of like null in Java, and an Optional[X] type is equivalent to just X. It has nothing to do with syntax.

[markshannon]

So how is it relevant to default values for parameters making the type Optional?

[JukkaL]

The reason for adopting the syntactic sugar was the worry that otherwise type annotations would be too verbose and thus maybe also harder to read. I think that the main arguments for not having the syntactic sugar are consistency and conceptual simplicity.

If we have a codebase that uses Optional[] consistently, which can be enforced by the strict optional mode of mypy, we could evaluate, at least for that codebase, whether the arguments hold water. We can get a realistic of estimate of the actual verbosity of annotating code with or without the syntactic sugar. We can perhaps also get a better understanding of whether the inconsistency is an actual problem for people who are working on the code.

It may all boil down to personal preference, but having some empirical data could be useful for the discussion. Also, it doesn't seem to me that this is an urgent thing to resolve, as this doesn't require major typing implementation changes (though get_type_hints may be affected) and it's going to be easy for a static checker to support both modes via a configuration option. I don't see a need to rush the decision, and we can first wait for tools that can perform useful "strict optional" checking to be built.

[markshannon]

Whether it is easier to read is, as you say, personal preference, which is rarely influenced by empirical evidence 😄

Happy to wait and see what you come up with.

[ilevkivskyi]

@JukkaL I ma not sure what kinds of inconsistency you mean, but this one

x = None # type: int # here the type stays int, unlike for function params

will go away after PEP 526. I think there it is better to assume implicit Optional or not for both of these

def f(x: int = None): ...
x: int = None

but not separately for each of these.

[markshannon]

x = None # type: int

Ah, that thing. I think that should never have been added. It is just self-contradictory.

[JukkaL]

@ilevkivskyi There's still the question about annotating instance variables with mutable types in the class body in pre-3.6 code. This is what mypy users can write today:

class A:
    x = None  # type: List[int]  # Can't use x = [] (list is mutable) and don't want Optional

    def __init__(self) -> None:
        self.x = []

This is a common use case and I think we need to have a reasonable way to expressing this without PEP 526 syntax.