PEP 612 tracking issue

[hauntsaninja]

Hello!

I got a little scared off from working on this by the many other enthusiastic people volunteering their time in #8373. However, there doesn't seem to have been much progress; since I have some extra time now, I figured sharing what I've got could kickstart things. Apologies if someone — or whatever a GSoC is — is already working on it :-) Branch is visible at https://github.com/hauntsaninja/mypy/tree/pep612, happy to make it a draft PR if that would help anything.

Overall seems like the plan would be:

• Focus on generic functions first, implement generic class support later, as pyre plans to.
• Make the fairly straight forward changes in semanal.py and nodes.py so you have the AST. Make relevant visitors visit it (look at where we visit_type_var_expr).
• Change typeanal.py, tvar_scope.py, etc to get binding going to the types you create in types.py. Seems like this would be the place to enforce valid use location too.
• Implement the semantics of ParameterSpecification in checkexpr.py, infer.py, solve.py, etc.
• Add a runtime implementation to mypy_extensions

Since ParameterSpecification is just a juiced up type variable, it makes sense to try and re-use existing logic for TypeVar as much as possible (especially if later PEPs make use of the bounds and variance arguments that PEP 612 reserves). In my branch, I've chosen to add a base class TypeVarLikeX in a couple different class hierarchies, e.g. TypeVarLikeExpr, TypeVarLikeDef. Since I'm not super familiar with mypy's code, this also helps me stumble upon things that I might need to change (it feels great trusting mypy to help improve mypy!). But this is a fairly large refactor (the one-line change in the details snippet alone will require changes in a dozen files), and could maybe affect like plugins or something, so I thought I'd post this before exploring further.

--- a/mypy/types.py
+++ b/mypy/types.py
@@ -983,7 +983,7 @@ class CallableType(FunctionLike):
                  fallback: Instance,
                  name: Optional[str] = None,
                  definition: Optional[SymbolNode] = None,
-                 variables: Optional[List[TypeVarDef]] = None,
+                 variables: Optional[Sequence[TypeVarLikeDef]] = None,

I'm also a little worried about issues like #1317 and #5738. I don't yet know the code well enough to anticipate what issues we might run into, but I'd welcome any thoughts on interactions between existing TypeVar code and any PEP 612 implementation.

[JukkaL]

Your plan looks good to me!

I don't expect that #1317 or #5738 will be blockers for this, though they may affect some use cases (which likely don't work currently anyway). In particular, decorating generic functions might not work. I'm planning to work on one or both of those issues in the near future, since they are really quite bad and have been open for a long time.

Please don't hesitate to ask further questions here (or email be directly).

[gvanrossum]

I encourage you to start the work on this. Most likely I will just accept the PEP, given that we've seen no further feedback on typing-sig.

[hauntsaninja]

Thanks for the kick! I threw some hours at this today, in particular I managed to get through a good amount of the somewhat tedious refactoring caused by generalising TypeVar. I've made a good amount of progress for typeanal of ParamSpec. Branch is still at https://github.com/hauntsaninja/mypy/tree/pep612

[gvanrossum]

Maybe you can submit it as a PR and mark it Draft? I personally find it hard to follow a change like this as a branch in someone else's repo.

[hauntsaninja]

Sure, I opened a draft PR here: #9250 All feedback welcome!
Some notes:

• I tend to rewrite history a lot on my personal dev branches, but will avoid doing that if people start reviewing things or ask me not to
• I expect to be able to get things done on Saturdays, but aside from that am a little short on time. If need be, I'm happy to hand over whatever I've got — just let me know!
• As a summary of where thing stand:
  • semantic analysis for ParamSpec for generic functions is in a good place, eg testParamSpecLocations in the draft PR passes
  • my next step is figuring out the types and type var solving necessary to eg, allow mypy to understand how changes_return_type_to_str would work
  • semantic analysis for ParamSpec components, ParamSpec for generic classes and Concatenate mostly still remains

[gvanrossum]

So how far have you gotten with this? I see two merged PRs but am afraid to look.

One comment: we're currently looking into support for PEP 612 in typing.py for 3.10, and after that in typing_extensions.py for previous Python versions, so it would be nice if these were also supported:

from typing_extensions import ParamSpec, Concatenate
from typing import ParamSpec, Concatenate

We could actually use some feedback on the PR we're working on: the contributor is currently looking into fairly strict checking of the arguments, but I'm wondering if that's useful -- we can't go as far as what you did in mypy_extensions.pyi (ParamSpec = 0, Concatenate = 0 :-) but we could perhaps just accept anything once we see a ParamSpec or Concatenate as part of a type, rather than validating that the usage matches with the PEP (that could be considered mypy's responsibility). Thoughts?

[hauntsaninja]

There hasn't been much progress since my last comment; I sort of deprioritised this in favour of working on other mypy issues that seemed more pressing. But what exists should already support both typing_extensions and typing:

mypy/mypy/semanal.py

Line 3109 in 35e5031

call = self.get_typevarlike_declaration(

I very much agree that ParamSpec shouldn't validate much at runtime. I think Callable already validates too much, which will necessitate hacks for the backport of ParamSpec to typing_extensions like https://github.com/facebook/pyre-check/blob/0ba810e07635955cbcdef7cd2ab3722ab01dde5b/pyre_extensions/__init__.py#L75 I'll take a look at that PR soon.