Merge remote-tracking branch 'upstream/main' into pandas-arrays

Author: twoertwein

.github/workflows/optional.yml

@@ -0,0 +1,40 @@
+name: 'Optional'
+
+on:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+jobs:
+  nightly:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: Install project dependencies
+      uses: ./.github/setup
+      with:
+        os: ubuntu-latest
+        python-version: '3.11'
+
+    - name: Run pytest (against pandas nightly)
+      run: poetry run poe pytest --nightly
+
+  mypy_nightly:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: Install project dependencies
+      uses: ./.github/setup
+      with:
+        os: ubuntu-latest
+        python-version: '3.11'
+
+    - name: Run mypy tests with mypy nightly
+      run: poetry run poe mypy --mypy_nightly

.github/workflows/test.yml

@@ -1,7 +1,11 @@
 name: 'Test'
 
 on:
-  [push, pull_request, workflow_dispatch]
+  push:
+    branches:
+      - main
+  pull_request:
+  workflow_dispatch:
 
 env:
   MPLBACKEND: 'Agg'
@@ -14,7 +18,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest, macos-latest]
-        python-version: ['3.8', '3.9', '3.10.6', '3.11']
+        python-version: ['3.8', '3.9', '3.10', '3.11']
 
     steps:
     - uses: actions/checkout@v3
@@ -25,11 +29,6 @@ jobs:
         os: ${{ matrix.os }}
         python-version: ${{ matrix.python-version }}
 
-    - name: Show poetry python location (Windows)
-      shell: pwsh
-      run: poetry run where python
-      if: matrix.os == 'windows-latest'
-
     - name: Run mypy on 'tests' (using the local stubs) and on the local stubs
       run: poetry run poe mypy
 
@@ -42,22 +41,6 @@ jobs:
     - name: Install pandas-stubs and run tests on the installed stubs
       run: poetry run poe test_dist
 
-  nightly:
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-
-    steps:
-    - uses: actions/checkout@v3
-
-    - name: Install project dependencies
-      uses: ./.github/setup
-      with:
-        os: ubuntu-latest
-        python-version: '3.10'
-
-    - name: Run pytest (against pandas nightly)
-      run: poetry run poe pytest --nightly
-
   precommit:
     runs-on: ubuntu-latest
     timeout-minutes: 10

.pre-commit-config.yaml

@@ -3,34 +3,27 @@ ci:
     autofix_prs: false
 repos:
 -   repo: https://github.com/python/black
-    rev: 22.8.0
+    rev: 23.7.0
     hooks:
     -   id: black
 -   repo: https://github.com/PyCQA/isort
-    rev: 5.10.1
+    rev: 5.12.0
     hooks:
     -   id: isort
--   repo: https://github.com/asottile/pyupgrade
-    rev: v3.2.0
+-   repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.0.283
     hooks:
-    -   id: pyupgrade
-        types_or: [python, pyi]
-        types: [text]  # overwrite types: [python]
-        args: [--py38-plus]
--   repo: https://github.com/PyCQA/flake8
-    rev: 5.0.4
-    hooks:
-    -   id: flake8
-        name: flake8 (py)
-        types: [python]
-        args: [--ignore=E501 F841]
-    -   id: flake8
-        name: flake8 (pyi)
-        additional_dependencies:
-        - flake8-pyi==22.10.0
-        types: [pyi]
+    -   id: ruff
         args: [
-          --ignore=E301 E302 E305 E402 E501 E701 E704 F401 F811 W503 Y019 Y027 Y034 Y037 Y041 Y042,
-          # TypeVars in private files are already private
-          --per-file-ignores=_*.pyi:Y001
+          --exit-non-zero-on-fix,
+          --target-version, py38,
+          --extend-select, "PYI,UP,RUF100",
+          --ignore, "E501,E731,F841,PYI042",
+          --per-file-ignores, "_*.pyi:PYI001",
+          --fix
         ]
+-   repo: https://github.com/codespell-project/codespell
+    rev: v2.2.5
+    hooks:
+    - id: codespell
+      additional_dependencies: [ tomli ]

README.md

@@ -17,7 +17,7 @@ convention of providing stubs in a separate package, as specified in [PEP 561](h
 pandas.  In general, these stubs are narrower than what is possibly allowed by pandas,
 but follow a convention of suggesting best recommended practices for using pandas.
 
-The stubs are likely incomplete in terms of covering the published API of pandas.
+The stubs are likely incomplete in terms of covering the published API of pandas.  NOTE: The current 2.0.x releases of pandas-stubs do not support all of the new features of pandas 2.0.  See this [tracker](https://github.com/pandas-dev/pandas-stubs/issues/624) to understand the current compatibility with version 2.0.
 
 The stubs are tested with [mypy](http://mypy-lang.org/) and [pyright](https://github.com/microsoft/pyright#readme) and are currently shipped with the Visual Studio Code extension
 [pylance](https://github.com/microsoft/pylance-release#readme).
@@ -137,20 +137,19 @@ Documentation is a work-in-progress.
 
 ## Background
 
-These stubs are the result of a strategic effort lead by the core pandas team to integrate [Microsoft type stub repository](https://github.com/microsoft/python-type-stubs)
-together with the [VirtusLabs pandas_stubs repository](https://github.com/VirtusLab/pandas-stubs).
+These stubs are the result of a strategic effort led by the core pandas team to integrate [Microsoft type stub repository](https://github.com/microsoft/python-type-stubs) with the [VirtusLabs pandas_stubs repository](https://github.com/VirtusLab/pandas-stubs).
 
-These stubs were initially forked from the Microsoft project <https://github.com/microsoft/python-type-stubs> as of [this commit](https://github.com/microsoft/python-type-stubs/tree/6b800063bde687cd1846122431e2a729a9de625a).
+These stubs were initially forked from the Microsoft project at <https://github.com/microsoft/python-type-stubs> as of [this commit](https://github.com/microsoft/python-type-stubs/tree/6b800063bde687cd1846122431e2a729a9de625a).
 
-We are indebted to Microsoft and that project for the initial set of public type stubs.  We are also grateful for the original pandas-stubs project at <https://github.com/VirtusLab/pandas-stubs> that created the framework for testing the stubs.
+We are indebted to Microsoft and that project for providing the initial set of public type stubs.  We are also grateful for the original pandas-stubs project at <https://github.com/VirtusLab/pandas-stubs>, which created the framework for testing the stubs.
 
 ## Differences between type declarations in pandas and pandas-stubs
 
 The <https://github.com/pandas-dev/pandas/> project has type declarations for some parts of pandas, both for the internal and public API's.  Those type declarations are used to make sure that the pandas code is _internally_ consistent.
 
-The <https://github.com/pandas-dev/pandas-stubs/> project provides type declarations for the pandas _public_ API.  The philsophy of these stubs can be found at <https://github.com/pandas-dev/pandas-stubs/blob/main/docs/philosophy.md/> While it would be ideal if the `pyi` files in this project would be part of the `pandas` distribution, this would require consistency between the internal type declarations and the public declarations, and the scope of a project to create that consistency is quite large.  That is a long term goal.  Finally, another goal is to do more frequent releases of the pandas-stubs than is done for pandas, in order to make the stubs more useful.
+The <https://github.com/pandas-dev/pandas-stubs/> project provides type declarations for the pandas _public_ API.  The philosophy of these stubs can be found at <https://github.com/pandas-dev/pandas-stubs/blob/main/docs/philosophy.md/>. While it would be ideal if the `pyi` files in this project would be part of the `pandas` distribution, this would require consistency between the internal type declarations and the public declarations, and the scope of a project to create that consistency is quite large.  That is a long term goal.  Finally, another goal is to do more frequent releases of the pandas-stubs than is done for pandas, in order to make the stubs more useful.
 
-If issues are found with the public stubs, pull requests to correct those issues are welcome.  In addition, pull requests on the pandas repository to fix the same issue are welcome there as well.  However, since the goals of typing in the two projects are different (internal consistency vs. public usage), it may be a challenge to create consistent type declarations across both projects.  See <https://pandas.pydata.org/docs/development/contributing_codebase.html#type-hints/> for a discussion of typing standards used within the pandas code.
+If issues are found with the public stubs, pull requests to correct those issues are welcome.  In addition, pull requests on the pandas repository to fix the same issue are welcome there as well.  However, since the goals of typing in the two projects are different (internal consistency vs. public usage), it may be a challenge to create consistent type declarations across both projects.  See <https://pandas.pydata.org/docs/development/contributing_codebase.html#type-hints> for a discussion of typing standards used within the pandas code.
 
 ## Getting help
 

docs/philosophy.md

@@ -1,12 +1,12 @@
 # pandas-stubs Type Checking Philosophy
 
 The goal of the pandas-stubs project is to provide type stubs for the public API
-that represent the recommended ways of using pandas. This is opposed to the 
+that represent the recommended ways of using pandas.  This is opposed to the
 philosophy within the pandas source, as described [here](https://pandas.pydata.org/docs/development/contributing_codebase.html?highlight=typing#type-hints), which
 is to assist with the development of the pandas source code to ensure type safety within
 that source.
 
-Due to the methodology used by Microsoft to develop the original stubs, there are internal 
+Due to the methodology used by Microsoft to develop the original stubs, there are internal
 classes, methods and functions that are annotated within the pandas-stubs project
 that are incorrect with respect to the pandas source, but that have no effect on type
 checking user code that calls the public API.
@@ -27,12 +27,12 @@ s = pd.Series([1, 2, 3])
 lt = s < 3
 ```
 
-In the pandas source, `lt` is a `Series` with a `dtype` of `bool`. In the pandas-stubs,
-the type of `lt` is `Series[bool]`. This allows further type checking to occur in other
+In the pandas source, `lt` is a `Series` with a `dtype` of `bool`.  In the pandas-stubs,
+the type of `lt` is `Series[bool]`.  This allows further type checking to occur in other
 pandas methods.  Note that in the above example, `s` is typed as `Series[Any]` because
 its type cannot be statically inferred.
 
-This also allows type checking for operations on series that contain date/time data. Consider
+This also allows type checking for operations on series that contain date/time data.  Consider
 the following example that creates two series of datetimes with corresponding arithmetic.
 
 ```python
@@ -74,22 +74,22 @@ interval of `Timestamp`s.
 A set of (most likely incomplete) tests for testing the type stubs is in the pandas-stubs
 repository in the `tests` directory.  The tests are used with `mypy` and `pyright` to
 validate correct typing, and also with `pytest` to validate that the provided code
-actually executes. The recent decision for Python 3.11 to include `assert_type()`,
+actually executes.  The recent decision for Python 3.11 to include `assert_type()`,
 which is supported by `typing_extensions` version 4.2 and beyond makes it easier
-to test to validate the return types of functions and methods. Future work
+to test to validate the return types of functions and methods.  Future work
 is intended to expand the use of `assert_type()` in the test code.
 
 ## Narrow vs. Wide Arguments
 
-A consideration in creating stubs is too make the set of type annotations for
+A consideration in creating stubs is to make the set of type annotations for
 function arguments "just right", i.e.,
 not too narrow and not too wide.  A type annotation to an argument to a function or
 method is too narrow if it disallows valid arguments.  A type annotation to
 an argument to a function or method is too wide if
 it allows invalid arguments.  Testing for type annotations that are too narrow is rather
-straightforward. It is easy to create an example for which the type checker indicates
+straightforward.  It is easy to create an example for which the type checker indicates
 the argument is incorrect, and add it to the set of tests in the pandas-stubs
-repository after fixing the appropriate stub. However, testing for when type annotations 
+repository after fixing the appropriate stub.  However, testing for when type annotations
 are too wide is a bit more complicated.
 In this case, the test will fail when using `pytest`, but it is also desirable to
 have type checkers report errors for code that is expected to fail type checking.
@@ -100,14 +100,36 @@ Here is an example that illustrates this concept, from `tests/test_interval.py`:
     i1 = pd.Interval(
         pd.Timestamp("2000-01-01"), pd.Timestamp("2000-01-03"), closed="both"
     )
-    if TYPE_CHECKING:
-        i1 + pd.Timestamp("2000-03-03")  # type: ignore
+    if TYPE_CHECKING_INVALID_USAGE:
+        i1 + pd.Timestamp("2000-03-03")  # type: ignore[operator] # pyright: ignore[reportGeneralTypeIssues]
 
 ```
 
 In this particular example, the stubs consider that `i1` will have the type
 `pd.Interval[pd.Timestamp]`.  It is incorrect code to add a `Timestamp` to a
-time-based interval.  Without the `if TYPE_CHECKING` construct, the code would fail.
-However, it is also desirable to have the type checker pick up this failure, and by
-placing the `# type: ignore` on the line, an indication is made to the type checker
-that we expect this line to not pass the type checker.
+time-based interval.  Without the `if TYPE_CHECKING_INVALID_USAGE` construct, the
+code would fail at runtime.  Further, type checkers should report an error for this
+incorrect code.  By placing the `# type: ignore[operator] # pyright: ignore[reportGeneralTypeIssues]`
+on the line, type checkers are told to ignore the type error.  To ensure that the
+pandas-stubs annotations are not too wide (allow adding a `Timestamp` to a
+time-based interval), mypy and pyright are configured to report unused ignore
+statements.
+
+## Using ignore comments
+
+Type checkers report errors
+
+- when writing negative tests to reject invalid behavior (inside a
+  `TYPE_CHECKING_INVALID_USAGE` block),
+- when writing `overload`s that return incompatible return values, or
+- when type checkers have bugs themselves.
+
+Since mypy and pyright behave slightly differently, we use separate ignore comments
+for them.
+
+- If mypy reports an error, please use `# type: ignore[<error code>]`
+- If pyright reports an error, please use `# pyright: ignore[<error code>]`
+
+If mypy and pyright report errors, for example, inside a `TYPE_CHECKING_INVALID_USAGE`
+block, please ensure that the comment for mypy comes first:
+`# type: ignore[<error code>] # pyright: ignore[<error code>]`.

docs/release_procedure.md

@@ -7,7 +7,7 @@
 
 ```shell
 rm dist/*
-poetry pubish --build  # you will get prompted for your pypi username/password
+poetry publish --build  # you will get prompted for your pypi username/password
 git commit -a -m "Version a.b.c.yymmdd"
 git push upstream main
 git tag va.b.c.yymmdd

docs/tests.md

@@ -11,7 +11,11 @@ Here are the most important options. Fore more details, please use `poe --help`.
   - Run only pytest: `poe pytest`
   - Run only pre-commit: `poe style`
 - Run tests against the installed stubs (this will install and uninstall the stubs): `poe test_dist`
-- Optional: run pytest against pandas nightly (this might fail): `poe pytest --nightly`
-- Optional: Run stubtest to compare the installed pandas-stubs against pandas (this will fail): `poe stubtest`. If you have created an allowlist to ignore certain errors: `poe stubtest path_to_the_allow_list`
 
 These tests originally came from https://github.com/VirtusLab/pandas-stubs.
+
+The following tests are **optional**. Some of them are run by the CI but it is okay if they fail.
+
+- Run pytest against pandas nightly: `poe pytest --nightly`
+- Use mypy nightly to validate the annotations: `poe mypy --mypy_nightly`
+- Run stubtest to compare the installed pandas-stubs against pandas (this will fail): `poe stubtest`. If you have created an allowlist to ignore certain errors: `poe stubtest path_to_the_allow_list`