Skip to content

Contributing Guide for Type Hints #27050

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Aug 25, 2019
Merged
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions doc/source/development/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -696,6 +696,108 @@ You'll also need to

See :ref:`contributing.warnings` for more.

Type Hints
----------

*pandas* strongly encourages the use of :pep:`484` style type hints. New development should contain type hints and pull requests to annotate existing code are accepted as well!

Syntax Requirements
~~~~~~~~~~~~~~~~~~~

Because *pandas* still supports Python 3.5, :pep:`526` does not apply and variables **must** be annotated with type comments. Specifically, this is a valid annotation within pandas:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put this comment lower, e.g. show the common / easy case first


.. code-block:: python

from typing import List

primes = [] # type: List[int]

Whereas this is **NOT** allowed:

.. code-block:: python

primes: List[int] = [] # not supported in Python 3.5!

Note that function signatures can always be annotated per :pep:`3107`:

.. code-block:: python

def sum_of_primes(primes: List[int] = []) -> int:
...

Style Guidelines
~~~~~~~~~~~~~~~~

Types imports should follow the ``from typing import ...`` convention. So rather than

.. code-block:: python

import typing

primes = [] # type: typing.List[int]

You should write

.. code-block:: python

from typing import List

primes = [] # type: List[int]

``Optional`` should be used where applicable, so instead of

.. code-block:: python

from typing import List, Union

maybe_primes = [] # type: List[Union[int, None]]

You should write

.. code-block:: python

from typing import List, Optional

maybe_primes = [] # type: List[Optional[int]]

When dealing with parameters with a default argument of ``None``, you should not use ``Optional`` as this will be inferred by the static type checker. So instead of:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have a code check for this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to revert this section. Mypy supports this by default but it appears to be only for backwards compatibility and is considered a "mistake" in PEP 484

python/typeshed#1420

So separately can look into enabling --no-implicit-optional to see what breaks


.. code-block:: python

def maybe_upcase_wrong(value: Optional[str] = None) -> Optional[str]:
...

You should write

.. code-block:: python

def maybe_upcase_right(value: str = None) -> Optional[str]:
...

Pandas-specific Types
~~~~~~~~~~~~~~~~~~~~~

Commonly used types specific to *pandas* will appear in pandas._typing and you should use these where applicable. This module is private for now but ultimately this should be exposed to third party libraries who want to implement type checking against pandas.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you put a reference to pandas._typing : https://github.com/pandas-dev/pandas/blob/master/pandas/_typing.py


For example, quite a few functions in *pandas* accept a ``dtype`` argument. This can be expressed as a string like ``"object"``, a numpy.dtype like ``np.int64`` or even a pandas ``ExtensionDtype`` like ``pd.CategoricalDtype``. Rather than burden the user with having to constantly annotate all of those options, this can simply be imported and reused from the pandas._typing module
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use a ref to numpy.dtype


.. code-block:: python

from pandas._typing import Dtype

def as_type(dtype: Dtype) -> ...:
...

This module will ultimately house types for repeatedly used concepts like "path-like", "array-like", "numeric", etc... and can also hold aliases for commonly appearing parameters like `axis`. Development of this module is active so be sure to refer to the source for the most up to date list of available types.

Validating Type Hints
~~~~~~~~~~~~~~~~~~~~~

*pandas* uses `mypy <http://mypy-lang.org>`_ to statically analyze the code base and type hints. After making any change you can ensure your type hints are correct by running

.. code-block:: shell

mypy pandas
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also type check single files or submodules? (for quicker development turnover, if you are trying out type checking whole pandas takes a while)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could but not generically useful as mypy doggedly follows all imports so wouldn't necessarily save much time:

https://mypy.readthedocs.io/en/latest/running_mypy.html#following-imports

If type checking speed is a concern the suggested approach is to use a daemon:

https://mypy.readthedocs.io/en/latest/mypy_daemon.html#mypy-daemon-mypy-server

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorisvandenbossche yes you can do something like this: mypy pandas/core/something.py, or mypy pandas/core/generic
It saves a bit of time but not much, but adding how to do that in the contributing guide might not be a bad idea. Personally when I am working with typing I run mypy for just the single file I am working on. For whole project I run it only before committing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t plan on adding this - it’s not value added to do

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well that's true, someone can easily figure out the command for a single file/folder by making a wise guess or going to mypy docs.


.. _contributing.ci:

Expand Down