[GSOC 2020] Project ideas

[JukkaL]

Update: See #10099 for GSoC 2021 project ideas.

Here are some ideas for larger mypy-related projects for contributors who want to tackle something fairly big (but also with a big potential impact).

Deep editor integrations

Currently it's possible to run mypy daemon from an editor and display the list of errors within the editor, but we could go much further. Possible ideas include going to the definition of an arbitrary reference (such as a method, variable, type, etc.), and displaying the inferred type of an expression. IDEs such as PyCharm can do some of this already, but mypy could support these features more reliably in some cases, since it maintains a very detailed representation of the program internally. Also, this could be very helpful with editors that have no or limited built-in support for these features.

Better decorator support

Mypy can't properly support decorators that change the return type of the decorated function, such as contextmanager (contextmanager is special-cased using a plugin, but this approach doesn't generalize to arbitrary functions). Add support for PEP 612 draft to make this better.

Generalize mypyc IR to allow non-C backends

Currently the IR of mypyc, the compiler we use to compile mypy, is tightly bound to C. This makes it impractical to experiment with alternative backends, such as an LLVM back end or a completely custom back end that directly generates assembly.

Related issue: mypyc/mypyc#709

Faster callables and nested functions in mypyc

Currently calling nested functions and variables with a callable type is pretty slow in compiled code. These limitations reduce the usefulness of mypyc significantly, especially when compiling code that wasn't originally written with mypyc in mind.

Related issues: mypyc/mypyc#713, mypyc/mypyc#712 (both of these would be implemented in a GSoC project)

NumPy support

This is a big topic, but this can be approached a feature at a time. One of the main missing things is "shape types" -- there needs to be a way to express the number of dimensions in an array, at the very least.

[hauntsaninja]

I'd be interested in adding support for PEP 612. I'll take a look at the pyre implementation this weekend, re-read the PEP, and scope out a plan.

[JukkaL]

@hauntsaninja Great! If you have any questions, I'm happy to help.

[JukkaL]

Another idea:

Detecting potentially undefined or misspelled locals

Some uses of undefined variables are not caught by mypy:

def f() -> None:
    if foo():
        x = 0
    print(x)  # No error

It would be useful to catch these (#2400). A related issue is reporting locals that are never read, as this is often an error (#76).

[TH3CHARLie]

GSOC 2020 organizations are announced and mypy is one of them! see https://summerofcode.withgoogle.com/organizations/6527008550420480/

I am interested in Generalize mypyc IR to allow non-C backends and I will manage to have a plan by the end of February.

[vpeiter]

Hi!
I'd be interested in working in detecting potentially undefined or misspelled locals.
I will work out a plan and get in contact soon...

[dubesar]

Hi!!
I am Sarvesh, I am interested in Faster callables and nested functions in mypyc and Numpy support.
I am working on Faster Callables and will get back soon with some solutuon.

[edwinjon]

Hello,
My name is Jonathan, I am actually focus on data science and I am really interested to work in Numpy Support. I once questioned regarding the unavailable of "shape types" . By working this project, will help my Data Science works too.

I am going to read the documentations and try to find out some insights which features can be improve for numpy. Thanks

Cheers,
Jonathan

[TH3CHARLie]

Hi, @edwinjon and @dubesar, glad to see someone interested in working on the mypy-Numpy topic!

As far as I know, there's a similar work that may help you develop your ideas and plans, check https://github.com/numpy/numpy-stubs.

[dubesar]

@JukkaL By numpy support I understand that the codes written in standard format needs to be changed to numpy format as for example changing arrays to numpy arrays.
Also by shapes you mean the shapes of the arrays?

Please elaborate that part!!

Also the Faster Callables part I checked the running time of the code and have addressed it in the issue. Please guide me further what has to be done in that, so that I can move in further with the work.

[egenedy97]

Hi @JukkaL my name is Eslam Genedy , I am at third year in University ASU Computer Engineering Department , it is my pleasure to apply for mypy . I have a good knowledge in oop in python and numpy . finally I hope to be contributed in this project thanks in advance

[JukkaL]

To support NumPy, we'd need some form of "shape type" support. We want to specify the number of dimensions and the item type of an array, at least. Even better, it would often be useful to specify the exact size of an array, but this will be much harder to implement.

This example is from @ilevkivskyi's presentation from the Typing Summit at PyCon 2019:

from typing import Shape, IntVar, TypeVar

N = IntVar('N')
T = TypeVar('T')

def diff(a: ndarray[T, Shape[N]]) -> ndarray[T, Shape[N - 1]]:
    ...
def sum2d(a: ndarray[T, Shape[:, :]) -> ndarray[T, Shape[:]]:
    ...

This means that diff takes a one-dimensional array of size N and item type T, and it returns a one-dimensional array of size N - 1 and item type T. sum2d, on the other hand, takes an arbitrary-sized two-dimensional array with item type T, and it returns a one-dimensional, arbitrary-sized array with item type T.

A reasonable goal for GSoC would be to support specifying the item type and the number of dimensions (e.g. ndarray[float, Shape[:, :]] for a two-dimensional array). This would involve at least these steps (I'm leaving a lot of details out):

1. Add support for the new shape type syntax.
2. Implement basic type operations involving shape types, such as displaying shapes in error messages and subtyping.
3. Implement simple stubs for NumPy that use the shape types.
4. If needed, add a mypy plugin to handle some common NumPy operations that can't be supported via existing type system features.

This is quite a challenging project and requires a deep understanding of mypy type checking internals, so it's really only well-suited for somebody who has substantial experience with working on mypy (or another type checker), or perhaps has completed a course on type theory (beyond a compilers course).

[joybh98]

@JukkaL , I'd like to integrate PEP 612 and take the better decorator support

[JukkaL]

@joybhallaa That would be a really useful project! I'd suggest starting by closing a few smaller issues first to get familiar with working on mypy. (The same advice applies to anybody who's interested in diving into a major project. It's best to gradually learn the codebase. Otherwise these's a big risk that there's too much to learn at once, and you'll get discouraged.)

Since PEP 612 involves type inference and type variables, tackling one (or some) of these issues could be a good option: https://github.com/python/mypy/issues?q=is%3Aopen+is%3Aissue+label%3Atopic-type-variables

If you find a promising issue but you are not sure where to start, you may want to ask for hints in the issue.

[joybh98]

Since PEP 612 involves type inference and type variables, tackling one (or some) of these issues could be a good option: https://github.com/python/mypy/issues?q=is%3Aopen+is%3Aissue+label%3Atopic-type-variables

@JukkaL I've taken up an issue and after I'm finished with that I'll take up issues that you have mentioned. 👍