mypy crashes with overloads of functools.singledispatch

[hauntsaninja]

I was trying out the advice I gave in #8354, and encountered a mypy crash:

~/delete λ cat test22.py                                                                                         2 
from typing import *
import functools


@overload
def foo(val: str) -> str: ...
@overload
def foo(val: int) -> int: ...


@functools.singledispatch
def foo(x):
    return x


@foo.register
def _(x: int) -> int:
    return x + 1
~/delete λ mypy test22.py --show-traceback
test22.py:5: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.770+dev.0b9089e9c9be2ad0b345b1d3743f808365d4aefc
Traceback (most recent call last):
  File "/usr/local/bin/mypy", line 11, in <module>
    load_entry_point('mypy', 'console_scripts', 'mypy')()
  File "/Users/shantanu/dev/mypy/mypy/__main__.py", line 8, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/Users/shantanu/dev/mypy/mypy/main.py", line 89, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 166, in build
    sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr, extra_plugins
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 234, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2612, in dispatch
    process_graph(graph, manager)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2919, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3018, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2111, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 293, in check_first_pass
    self.accept(d)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 400, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 515, in accept
    return visitor.visit_overloaded_func_def(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 433, in visit_overloaded_func_def
    self._visit_overloaded_func_def(defn)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 460, in _visit_overloaded_func_def
    self.check_overlapping_overloads(defn)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 481, in check_overlapping_overloads
    assert isinstance(inner_type, CallableType)
AssertionError: 
test22.py:5: : note: use --pdb to drop into pdb

[jeanmonet]

Same here:

Trying to combine typing.overload with functools.singledispatch, which gives a mypy internal error. The internal error disappears if I remove functools.singledispatch functionnality.

Using Python 3.7 with mypy 0.750

Python code:

from datetime import datetime, date
import typing
from typing import Union
from functools import singledispatch

from typing_extensions import Literal

DateInType = Union[date, str]

@typing.overload    #    <---- THIS IS ***line 50*** of td.py raising mypy internal error
def date2str(t: DateInType, out_format: Literal['str']) -> str: ...
@typing.overload
def date2str(t: DateInType, out_format: Literal['date']) -> date: ...

@singledispatch
def date2str(t, out_format):
    ''' Converts:
        - datetime.date to string (YYYY-MM-DD)
        - string (YYYY-MM-DD, YYYY/MM/DD, DD-MM-YYYY, ...) to datetime.date
        
        Second argument = 'out_format' can be 'str' or 'date'.
        If specified, returns output in string (YYYY-MM-DD) ['str'] or datetime.date format ['date'].
    '''

@date2str.register(date)
def _date2str_date(t, out_format='str'):
    ''' Input format = datetime.date.
        Output format, if string = YYYY-MM-DD'''
    if out_format == 'str':
        return t.strftime('%Y-%m-%d')
    elif isinstance(t, datetime):
        return t.date()
    else:
        return t

@date2str.register(str)
def _date2str_str(t, out_format='date'):
    ''' Input format = string (YYYY-MM-DD, DD-MM-YYYY, ...)'''
    dformats = ('%Y-%m-%d', '%Y/%m/%d', '%d-%m-%Y', '%d/%m/%Y', '%Y%m%d')
    for i, dformat in enumerate(dformats, start=1):
        try:
            d = datetime.strptime(t, dformat).date()
            break
        except ValueError:
            if i == len(dformats):
                raise
    if out_format == 'date':
        return d
    else:
        return date2str(d, out_format=out_format)


dt = datetime.now()
res = date2str(dt, out_format='date')
print(res, type(res))
assert type(res) == date

res2 = date2str(dt.date(), out_format='str')
print(res2, type(res2))
assert type(res2) == str

res3 = date2str('20061005', out_format='date')
print(res3, type(res3))
assert type(res3) == date

mypy traceback:

======== RESULTS ========
Traceback (most recent call last):
  File "/home/username/anaconda3/envs/py3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/__main__.py", line 12, in <module>
    main(None, sys.stdout, sys.stderr)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/main.py", line 89, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 167, in build
    sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr, extra_plugins
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 235, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 2620, in dispatch
    process_graph(graph, manager)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 2929, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 3028, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 2119, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 292, in check_first_pass
    self.accept(d)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 399, in accept
    stmt.accept(self)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/nodes.py", line 515, in accept
    return visitor.visit_overloaded_func_def(self)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 432, in visit_overloaded_func_def
    self._visit_overloaded_func_def(defn)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 459, in _visit_overloaded_func_def
    self.check_overlapping_overloads(defn)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 480, in check_overlapping_overloads
    assert isinstance(inner_type, CallableType)
AssertionError: 
typing_test/td.py:50: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.750

Am I doing something wrong?

[mattany]

Hello, I am an open source noob and would like to try and tackle this issue. Is that possible?

[hauntsaninja]

Yes, of course! Welcome. If you get stuck, feel free to ask questions here.

[mattany]

I couldn't replicate the bug on my machine. Using Python 3.8.2 and using my forked version of mypy (v0.770 I believe), copying @jeanmonet's code yielded the following output:

2020-04-08 <class 'datetime.date'>
2020-04-08 <class 'str'>
2006-10-05 <class 'datetime.date'>

Process finished with exit code 0

I also tried replicating the error using your (@hauntsaninja ) code but I admit I didn't really understand what was going on there (what does the lambda mean?)

cheers

[hauntsaninja]

My repro continues to repro for me (against mypy master). Note it contains the exact command I ran.

The output you copied looks like it would be the result of running the program (with the Python interpreter), which is expected to work — this is a bug in mypy, not in Python itself. Remember mypy is a static checker, it doesn't actually run your code. If you still can't get it to repro, try sharing the exact commands you're running here.

I assume by lambda you mean the def function named _? _ is a valid identifier, like any other, in Python. It doesn't have any special meaning, although it's commonly used as a name when you don't need to refer to the named thing (e.g. to make a list of 1s, [1 for _ in range(10)]).
What is special is the functools.singledispatch decorator applied to it, where use of functions named _ is a common idiom. I'd recommend reading https://docs.python.org/3/library/functools.html#functools.singledispatch and playing around with the foo function my repro defines.

If you need a reference for what overload does (and to understand the code better, since the mypy crash occurs while running some overload checking related code) , I recommend https://mypy.readthedocs.io/en/stable/more_types.html#function-overloading. In general, I've found the mypy docs really helpful.

[graingert]

see #2904

[radeklat]

I encountered this issue as well when I added @overload for PyCharm not recognising the different types. While it fixed the PyCharm warnings, it broke mypy :( Here is a minimal example:

from functools import singledispatch
from typing import Callable, overload

@overload
def overloaded_function(func: Callable):
    ...

@overload
def overloaded_function(format_string: str):
    ...

@singledispatch
def overloaded_function(func: Callable):
    pass

@overloaded_function.register
def _overloaded_function_for_strings(format_string: str):
    pass

[abbasidaniyal]

Is this issue still available? I would like to take this up.