Merge remote-tracking branch 'upstream/master' into multiindex_at_fix

Author: rhshadrach

.github/CONTRIBUTING.md

@@ -2,7 +2,7 @@
 
 Whether you are a novice or experienced software developer, all contributions and suggestions are welcome!
 
-Our main contributing guide can be found [in this repo](https://github.com/pandas-dev/pandas/blob/master/doc/source/development/contributing.rst) or [on the website](https://pandas-docs.github.io/pandas-docs-travis/development/contributing.html). If you do not want to read it in its entirety, we will summarize the main ways in which you can contribute and point to relevant sections of that document for further information.
+Our main contributing guide can be found [in this repo](https://github.com/pandas-dev/pandas/blob/master/doc/source/development/contributing.rst) or [on the website](https://pandas.pydata.org/docs/dev/development/contributing.html). If you do not want to read it in its entirety, we will summarize the main ways in which you can contribute and point to relevant sections of that document for further information.
 
 ## Getting Started
 

.github/ISSUE_TEMPLATE/documentation_improvement.md

@@ -11,7 +11,7 @@ labels: "Docs, Needs Triage"
 
 [this should provide the location of the documentation, e.g. "pandas.read_csv" or the URL of the documentation, e.g. "https://dev.pandas.io/docs/reference/api/pandas.read_csv.html"]
 
-**Note**: You can check the latest versions of the docs on `master` [here](https://dev.pandas.io/docs).
+**Note**: You can check the latest versions of the docs on `master` [here](https://pandas.pydata.org/docs/dev/).
 
 #### Documentation problem
 

.github/workflows/ci.yml

@@ -125,32 +125,18 @@ jobs:
     - name: Check ipython directive errors
       run: "! grep -B1 \"^<<<-------------------------------------------------------------------------$\" sphinx.log"
 
-    - name: Install Rclone
-      run: sudo apt install rclone -y
-      if: github.event_name == 'push'
-
-    - name: Set up Rclone
+    - name: Install ssh key
       run: |
-        CONF=$HOME/.config/rclone/rclone.conf
-        mkdir -p `dirname $CONF`
-        echo "[ovh_host]" > $CONF
-        echo "type = swift" >> $CONF
-        echo "env_auth = false" >> $CONF
-        echo "auth_version = 3" >> $CONF
-        echo "auth = https://auth.cloud.ovh.net/v3/" >> $CONF
-        echo "endpoint_type = public" >> $CONF
-        echo "tenant_domain = default" >> $CONF
-        echo "tenant = 2977553886518025" >> $CONF
-        echo "domain = default" >> $CONF
-        echo "user = w4KGs3pmDxpd" >> $CONF
-        echo "key = ${{ secrets.ovh_object_store_key }}" >> $CONF
-        echo "region = BHS" >> $CONF
+        mkdir -m 700 -p ~/.ssh
+        echo "${{ secrets.server_ssh_key }}" > ~/.ssh/id_rsa
+        chmod 600 ~/.ssh/id_rsa
+        echo "${{ secrets.server_ip }} ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1Kkopomm7FHG5enATf7SgnpICZ4W2bw+Ho+afqin+w7sMcrsa0je7sbztFAV8YchDkiBKnWTG4cRT+KZgZCaY=" > ~/.ssh/known_hosts
       if: github.event_name == 'push'
 
-    - name: Sync web with OVH
-      run: rclone sync --exclude pandas-docs/** web/build ovh_host:prod
+    - name: Upload web
+      run: rsync -az --delete --exclude='pandas-docs' --exclude='docs' --exclude='Pandas_Cheat_Sheet*' web/build/ docs@${{ secrets.server_ip }}:/usr/share/nginx/pandas
       if: github.event_name == 'push'
 
-    - name: Sync dev docs with OVH
-      run: rclone sync doc/build/html ovh_host:prod/pandas-docs/dev
+    - name: Upload dev docs
+      run: rsync -az --delete doc/build/html/ docs@${{ secrets.server_ip }}:/usr/share/nginx/pandas/pandas-docs/dev
       if: github.event_name == 'push'

README.md

@@ -7,6 +7,7 @@
 # pandas: powerful Python data analysis toolkit
 [![PyPI Latest Release](https://img.shields.io/pypi/v/pandas.svg)](https://pypi.org/project/pandas/)
 [![Conda Latest Release](https://anaconda.org/conda-forge/pandas/badges/version.svg)](https://anaconda.org/anaconda/pandas/)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3509134.svg)](https://doi.org/10.5281/zenodo.3509134)
 [![Package Status](https://img.shields.io/pypi/status/pandas.svg)](https://pypi.org/project/pandas/)
 [![License](https://img.shields.io/pypi/l/pandas.svg)](https://github.com/pandas-dev/pandas/blob/master/LICENSE)
 [![Travis Build Status](https://travis-ci.org/pandas-dev/pandas.svg?branch=master)](https://travis-ci.org/pandas-dev/pandas)
@@ -158,7 +159,7 @@ Most development discussion is taking place on github in this repo. Further, the
 
 All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.
 
-A detailed overview on how to contribute can be found in the **[contributing guide](https://dev.pandas.io/docs/contributing.html)**. There is also an [overview](.github/CONTRIBUTING.md) on GitHub.
+A detailed overview on how to contribute can be found in the **[contributing guide](https://pandas.pydata.org/docs/dev/development/contributing.html)**. There is also an [overview](.github/CONTRIBUTING.md) on GitHub.
 
 If you are simply looking to start working with the pandas codebase, navigate to the [GitHub "issues" tab](https://github.com/pandas-dev/pandas/issues) and start looking through interesting issues. There are a number of issues listed under [Docs](https://github.com/pandas-dev/pandas/issues?labels=Docs&sort=updated&state=open) and [good first issue](https://github.com/pandas-dev/pandas/issues?labels=good+first+issue&sort=updated&state=open) where you could start out.
 

asv_bench/asv.conf.json

@@ -39,7 +39,7 @@
     // followed by the pip installed packages).
     "matrix": {
         "numpy": [],
-        "Cython": [],
+        "Cython": ["0.29.16"],
         "matplotlib": [],
         "sqlalchemy": [],
         "scipy": [],

asv_bench/benchmarks/array.py

@@ -9,6 +9,11 @@ def setup(self):
         self.values_float = np.array([1.0, 0.0, 1.0, 0.0])
         self.values_integer = np.array([1, 0, 1, 0])
         self.values_integer_like = [1, 0, 1, 0]
+        self.data = np.array([True, False, True, False])
+        self.mask = np.array([False, False, True, False])
+
+    def time_constructor(self):
+        pd.arrays.BooleanArray(self.data, self.mask)
 
     def time_from_bool_array(self):
         pd.array(self.values_bool, dtype="boolean")
@@ -21,3 +26,16 @@ def time_from_integer_like(self):
 
     def time_from_float_array(self):
         pd.array(self.values_float, dtype="boolean")
+
+
+class IntegerArray:
+    def setup(self):
+        self.values_integer = np.array([1, 0, 1, 0])
+        self.data = np.array([1, 2, 3, 4], dtype="int64")
+        self.mask = np.array([False, False, True, False])
+
+    def time_constructor(self):
+        pd.arrays.IntegerArray(self.data, self.mask)
+
+    def time_from_integer_array(self):
+        pd.array(self.values_integer, dtype="Int64")

asv_bench/benchmarks/multiindex_object.py

@@ -74,10 +74,38 @@ def setup(self):
             ],
             dtype=object,
         )
+        self.other_mi_many_mismatches = MultiIndex.from_tuples(
+            [
+                (-7, 41),
+                (-2, 3),
+                (-0.7, 5),
+                (0, 0),
+                (0, 1.5),
+                (0, 340),
+                (0, 1001),
+                (1, -4),
+                (1, 20),
+                (1, 1040),
+                (432, -5),
+                (432, 17),
+                (439, 165.5),
+                (998, -4),
+                (998, 24065),
+                (999, 865.2),
+                (999, 1000),
+                (1045, -843),
+            ]
+        )
 
     def time_get_indexer(self):
         self.mi_int.get_indexer(self.obj_index)
 
+    def time_get_indexer_and_backfill(self):
+        self.mi_int.get_indexer(self.other_mi_many_mismatches, method="backfill")
+
+    def time_get_indexer_and_pad(self):
+        self.mi_int.get_indexer(self.other_mi_many_mismatches, method="pad")
+
     def time_is_monotonic(self):
         self.mi_int.is_monotonic
 

asv_bench/benchmarks/rolling.py

@@ -165,4 +165,26 @@ def peakmem_fixed(self):
             self.roll.max()
 
 
+class ForwardWindowMethods:
+    params = (
+        ["DataFrame", "Series"],
+        [10, 1000],
+        ["int", "float"],
+        ["median", "mean", "max", "min", "kurt", "sum"],
+    )
+    param_names = ["constructor", "window_size", "dtype", "method"]
+
+    def setup(self, constructor, window_size, dtype, method):
+        N = 10 ** 5
+        arr = np.random.random(N).astype(dtype)
+        indexer = pd.api.indexers.FixedForwardWindowIndexer(window_size=window_size)
+        self.roll = getattr(pd, constructor)(arr).rolling(window=indexer)
+
+    def time_rolling(self, constructor, window_size, dtype, method):
+        getattr(self.roll, method)()
+
+    def peakmem_rolling(self, constructor, window_size, dtype, method):
+        getattr(self.roll, method)()
+
+
 from .pandas_vb_common import setup  # noqa: F401 isort:skip

ci/code_checks.sh

@@ -272,6 +272,10 @@ if [[ -z "$CHECK" || "$CHECK" == "doctests" ]]; then
     pytest -q --doctest-modules pandas/core/accessor.py
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
+    MSG='Doctests aggregation.py' ; echo $MSG
+    pytest -q --doctest-modules pandas/core/aggregation.py
+    RET=$(($RET + $?)) ; echo $MSG "DONE"
+
     MSG='Doctests base.py' ; echo $MSG
     pytest -q --doctest-modules pandas/core/base.py
     RET=$(($RET + $?)) ; echo $MSG "DONE"
@@ -288,10 +292,6 @@ if [[ -z "$CHECK" || "$CHECK" == "doctests" ]]; then
     pytest -q --doctest-modules pandas/core/generic.py
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
-    MSG='Doctests groupby.py' ; echo $MSG
-    pytest -q --doctest-modules pandas/core/groupby/groupby.py -k"-cumcount -describe -pipe"
-    RET=$(($RET + $?)) ; echo $MSG "DONE"
-
     MSG='Doctests series.py' ; echo $MSG
     pytest -q --doctest-modules pandas/core/series.py
     RET=$(($RET + $?)) ; echo $MSG "DONE"
@@ -314,6 +314,10 @@ if [[ -z "$CHECK" || "$CHECK" == "doctests" ]]; then
     pytest -q --doctest-modules pandas/core/dtypes/
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
+    MSG='Doctests groupby' ; echo $MSG
+    pytest -q --doctest-modules pandas/core/groupby/
+    RET=$(($RET + $?)) ; echo $MSG "DONE"
+
     MSG='Doctests indexes' ; echo $MSG
     pytest -q --doctest-modules pandas/core/indexes/
     RET=$(($RET + $?)) ; echo $MSG "DONE"
@@ -348,7 +352,7 @@ if [[ -z "$CHECK" || "$CHECK" == "docstrings" ]]; then
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Validate correct capitalization among titles in documentation' ; echo $MSG
-    $BASE_DIR/scripts/validate_rst_title_capitalization.py $BASE_DIR/doc/source/development/contributing.rst $BASE_DIR/doc/source/reference
+    $BASE_DIR/scripts/validate_rst_title_capitalization.py $BASE_DIR/doc/source/development $BASE_DIR/doc/source/reference
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
 fi

ci/deps/azure-36-32bit.yaml

@@ -22,5 +22,5 @@ dependencies:
   # see comment above
   - pip
   - pip:
-    - cython>=0.29.13
+    - cython>=0.29.16
     - pytest>=5.0.1

ci/deps/azure-36-locale.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.6.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - pytest-asyncio

ci/deps/azure-36-locale_slow.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.6.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0

ci/deps/azure-36-minimum_versions.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.6.1
 
   # tools
-  - cython=0.29.13
+  - cython=0.29.16
   - pytest=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0
@@ -22,7 +22,7 @@ dependencies:
   - numpy=1.13.3
   - openpyxl=2.5.7
   - pytables=3.4.2
-  - python-dateutil=2.6.1
+  - python-dateutil=2.7.3
   - pytz=2017.2
   - scipy=0.19.0
   - xlrd=1.1.0

ci/deps/azure-37-locale.yaml

@@ -5,7 +5,7 @@ dependencies:
   - python=3.7.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - pytest-asyncio

ci/deps/azure-37-numpydev.yaml

@@ -5,7 +5,6 @@ dependencies:
   - python=3.7.*
 
   # tools
-  - cython>=0.29.13
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0
@@ -15,6 +14,7 @@ dependencies:
   - pytz
   - pip
   - pip:
+    - cython>=0.29.16
     - "git+git://github.com/dateutil/dateutil.git"
     - "-f https://7933911d6844c6c53a7d-47bd50c35cd79bd838daf386af554a83.ssl.cf2.rackcdn.com"
     - "--pre"

ci/deps/azure-macos-36.yaml

@@ -5,7 +5,6 @@ dependencies:
   - python=3.6.*
 
   # tools
-  - cython>=0.29.13
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0
@@ -24,13 +23,14 @@ dependencies:
   - openpyxl
   - pyarrow>=0.13.0
   - pytables
-  - python-dateutil==2.6.1
+  - python-dateutil==2.7.3
   - pytz
   - xarray
   - xlrd
   - xlsxwriter
   - xlwt
   - pip
   - pip:
+    - cython>=0.29.16
     - pyreadstat
     - pyxlsb

ci/deps/azure-windows-36.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.6.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0

ci/deps/azure-windows-37.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.7.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0

ci/deps/travis-36-cov.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.6.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0
@@ -15,7 +15,7 @@ dependencies:
   # pandas dependencies
   - beautifulsoup4
   - botocore>=1.11
-  - cython>=0.29.13
+  - cython>=0.29.16
   - dask
   - fastparquet>=0.3.2
   - gcsfs

ci/deps/travis-36-locale.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.6.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0

ci/deps/travis-36-slow.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.6.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0

ci/deps/travis-37.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.7.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0

ci/deps/travis-38.yaml

@@ -6,7 +6,7 @@ dependencies:
   - python=3.8.*
 
   # tools
-  - cython>=0.29.13
+  - cython>=0.29.16
   - pytest>=5.0.1
   - pytest-xdist>=1.21
   - hypothesis>=3.58.0

doc/source/conf.py

@@ -416,6 +416,7 @@
         "python": ("https://docs.python.org/3/", None),
         "scipy": ("https://docs.scipy.org/doc/scipy/reference/", None),
         "statsmodels": ("https://www.statsmodels.org/devel/", None),
+        "pyarrow": ("https://arrow.apache.org/docs/", None),
     }
 
 # extlinks alias