Generic inference regression from #5699

[msullivan]

Since #5699, mypy now rejects the following program (minimized from a few failures at dropbox):

from typing import TypeVar, Callable, Tuple

T = TypeVar('T')

def f(x: Callable[..., T]) -> T:
    return x()

x: Tuple[str, ...] = f(tuple)

with the error message
Incompatible types in assignment (expression has type "Tuple[_T_co, ...]", variable has type "Tuple[str, ...]")

@ilevkivskyi thoughts?

To be honest I'm not totally sure how we used to get this right, since my understanding was that we don't do the sort of instantiation of all type variables and unification that I would expect to be necessary?

[ilevkivskyi]

OK, so here is the diagnosis: I remember that I checked that our repos are clean with that PR, but later @JukkaL proposed to only use the outer context invariant instances, which I didn't check. The point is that tuple is covariant (and Tuple[str, ...] translates to an instance type), so we previously (successfully) used it for inference, but not anymore.

It looks however, there is another bug this uncovered, because currently it should have inferred <nothing>, or failed with a reasonable error, like "Cannot infer type variable", instead of leaking type variable.

I believe this may be a manifestation of #1317 which is another high priority issue I always wanted to fix.

[ilevkivskyi]

Yes, I am pretty sure this is a duplicate of #1317, here is a similar repro:

[case testCallableArgInference]
from typing import TypeVar, Callable

T = TypeVar('T')

def f(x: Callable[..., T]) -> T:
    return x()

def tpl(x: T) -> T:  # Really any generic thing (class or just function)
    return x

reveal_type(f(tpl))  # Revealed type is 'T`-1' (???)
[out]

I can try looking into this more today and tomorrow, but I can't guarantee it is easy to fix.

[ilevkivskyi]

Continuing thinking about this and #1317 it seems there are actually two aspects here:

• Fix leaking type variables (which I find quite annoying)
• Support inferring generic types (which is more relevant for original use case in Generic decorators don't work with generic functions #1317), because currently inferred types can't have free type variables.

Fixing the first one could be easy, while the second point may be a bit tricky. Currently mypy gets almost there:

from typing import TypeVar, Callable, List

T = TypeVar('T')
S = TypeVar('S')
U = TypeVar('U')

def dec(f: Callable[[S], T]) -> Callable[[S], List[T]]: ...

def f(x: U) -> U: ...

g = dec(f)

reveal_type(g)  # Revealed type is 'def (U`-1) -> builtins.list[U`-1]'

we just need to make it def [U] ....

@JukkaL what do you think?

[JukkaL]

I agree that leaking type variables probably isn't hard to fix, once we figure out what the correct behavior would be.

I can't say how hard it would be to infer generic types. This would need some research.

[gvanrossum]

I'm not sure this needs to be ready for the release even if it technically is a regression -- IIUC the code that used to pass was questionable at best, right?

[ilevkivskyi]

IIUC the code that used to pass was questionable at best, right?

It is not incorrect although. I will try a bit more to see if there is a simple fix, I will have nothing by the end of the day, I would propose to just not worry about this for now.

[euresti]

This also affects a common usage of attrs:

import attr
from typing import Sequence

x: Sequence[int] = attr.Factory(list)  # Incompatible types in assignment (expression has type "List[_T]", variable has type "Sequence[int]")

Unsuprisingly, attr.Factory is defined as follows:

def Factory(factory: Callable[[], _T]) -> _T: ...

However

x: List[int] = attr.Factory(list) 

is ok.