Merge branch 'master' of https://github.com/pandas-dev/pandas into multiindex_union

� Conflicts:
�	doc/source/whatsnew/v1.3.0.rst
�	pandas/tests/indexes/test_setops.py

Author: phofl

asv_bench/benchmarks/groupby.py

@@ -480,7 +480,19 @@ class GroupByCythonAgg:
     param_names = ["dtype", "method"]
     params = [
         ["float64"],
-        ["sum", "prod", "min", "max", "mean", "median", "var", "first", "last"],
+        [
+            "sum",
+            "prod",
+            "min",
+            "max",
+            "mean",
+            "median",
+            "var",
+            "first",
+            "last",
+            "any",
+            "all",
+        ],
     ]
 
     def setup(self, dtype, method):

ci/azure/windows.yml

@@ -8,7 +8,7 @@ jobs:
     vmImage: ${{ parameters.vmImage }}
   strategy:
     matrix:
-      py37_np16:
+      py37_np17:
         ENV_FILE: ci/deps/azure-windows-37.yaml
         CONDA_PY: "37"
         PATTERN: "not slow and not network"

ci/deps/actions-37-minimum_versions.yaml

@@ -18,7 +18,7 @@ dependencies:
   - jinja2=2.10
   - numba=0.46.0
   - numexpr=2.6.8
-  - numpy=1.16.5
+  - numpy=1.17.3
   - openpyxl=3.0.0
   - pytables=3.5.1
   - python-dateutil=2.7.3

ci/deps/azure-macos-37.yaml

@@ -19,7 +19,7 @@ dependencies:
   - matplotlib=2.2.3
   - nomkl
   - numexpr
-  - numpy=1.16.5
+  - numpy=1.17.3
   - openpyxl
   - pyarrow=0.15.1
   - pytables

ci/deps/azure-windows-37.yaml

@@ -24,7 +24,7 @@ dependencies:
   - moto>=1.3.14
   - flask
   - numexpr
-  - numpy=1.16.*
+  - numpy=1.17.*
   - openpyxl
   - pyarrow=0.15
   - pytables

doc/source/getting_started/install.rst

@@ -222,7 +222,7 @@ Dependencies
 Package                                                          Minimum supported version
 ================================================================ ==========================
 `setuptools <https://setuptools.readthedocs.io/en/latest/>`__    38.6.0
-`NumPy <https://numpy.org>`__                                    1.16.5
+`NumPy <https://numpy.org>`__                                    1.17.3
 `python-dateutil <https://dateutil.readthedocs.io/en/stable/>`__ 2.7.3
 `pytz <https://pypi.org/project/pytz/>`__                        2017.3
 ================================================================ ==========================

doc/source/whatsnew/v1.2.5.rst

@@ -14,7 +14,7 @@ including other versions of pandas.
 
 Fixed regressions
 ~~~~~~~~~~~~~~~~~
-
+- Regression in :func:`concat` between two :class:`DataFrames` where one has an :class:`Index` that is all-None and the other is :class:`DatetimeIndex` incorrectly raising (:issue:`40841`)
 -
 -
 

doc/source/whatsnew/v1.3.0.rst

@@ -217,6 +217,9 @@ Other enhancements
 - :class:`RangeIndex` can now be constructed by passing a ``range`` object directly e.g. ``pd.RangeIndex(range(3))`` (:issue:`12067`)
 - :meth:`round` being enabled for the nullable integer and floating dtypes (:issue:`38844`)
 - :meth:`pandas.read_csv` and :meth:`pandas.read_json` expose the argument ``encoding_errors`` to control how encoding errors are handled (:issue:`39450`)
+- :meth:`.GroupBy.any` and :meth:`.GroupBy.all` use Kleene logic with nullable data types (:issue:`37506`)
+- :meth:`.GroupBy.any` and :meth:`.GroupBy.all` return a ``BooleanDtype`` for columns with nullable data types (:issue:`33449`)
+-
 
 .. ---------------------------------------------------------------------------
 
@@ -465,7 +468,7 @@ If installed, we now require:
 +-----------------+-----------------+----------+---------+
 | Package         | Minimum Version | Required | Changed |
 +=================+=================+==========+=========+
-| numpy           | 1.16.5          |    X     |         |
+| numpy           | 1.17.3          |    X     |    X    |
 +-----------------+-----------------+----------+---------+
 | pytz            | 2017.3          |    X     |         |
 +-----------------+-----------------+----------+---------+
@@ -678,7 +681,7 @@ Interval
 Indexing
 ^^^^^^^^
 
-- Bug in :meth:`Index.union` and :meth:`MultiIndex.union` dropping duplicate ``Index`` values when ``Index`` was not monotonic or ``sort`` was set to ``False`` (:issue:`36289`, :issue:`31326`, :issue:`38745`)
+- Bug in :meth:`Index.union` and :meth:`MultiIndex.union` dropping duplicate ``Index`` values when ``Index`` was not monotonic or ``sort`` was set to ``False`` (:issue:`36289`, :issue:`31326`, :issue:`40862`)
 - Bug in :meth:`CategoricalIndex.get_indexer` failing to raise ``InvalidIndexError`` when non-unique (:issue:`38372`)
 - Bug in inserting many new columns into a :class:`DataFrame` causing incorrect subsequent indexing behavior (:issue:`38380`)
 - Bug in :meth:`DataFrame.__setitem__` raising ``ValueError`` when setting multiple values to duplicate columns (:issue:`15695`)
@@ -711,7 +714,7 @@ Missing
 
 - Bug in :class:`Grouper` now correctly propagates ``dropna`` argument and :meth:`DataFrameGroupBy.transform` now correctly handles missing values for ``dropna=True`` (:issue:`35612`)
 - Bug in :func:`isna`, and :meth:`Series.isna`, :meth:`Index.isna`, :meth:`DataFrame.isna` (and the corresponding ``notna`` functions) not recognizing ``Decimal("NaN")`` objects (:issue:`39409`)
--
+- Bug in :meth:`DataFrame.fillna` not accepting dictionary for ``downcast`` keyword (:issue:`40809`)
 
 MultiIndex
 ^^^^^^^^^^
@@ -787,6 +790,8 @@ Groupby/resample/rolling
 - Bug in :meth:`Series.asfreq` and :meth:`DataFrame.asfreq` dropping rows when the index is not sorted (:issue:`39805`)
 - Bug in aggregation functions for :class:`DataFrame` not respecting ``numeric_only`` argument when ``level`` keyword was given (:issue:`40660`)
 - Bug in :class:`core.window.RollingGroupby` where ``as_index=False`` argument in ``groupby`` was ignored (:issue:`39433`)
+- Bug in :meth:`.GroupBy.any` and :meth:`.GroupBy.all` raising ``ValueError`` when using with nullable type columns holding ``NA`` even with ``skipna=True`` (:issue:`40585`)
+- Bug in :meth:`GroupBy.cummin` and :meth:`GroupBy.cummax` incorrectly rounding integer values near the ``int64`` implementations bounds (:issue:`40767`)
 
 Reshaping
 ^^^^^^^^^

environment.yml

@@ -3,7 +3,7 @@ channels:
   - conda-forge
 dependencies:
   # required
-  - numpy>=1.16.5
+  - numpy>=1.17.3
   - python=3
   - python-dateutil>=2.7.3
   - pytz

pandas/__init__.py

@@ -20,7 +20,6 @@
 
 # numpy compat
 from pandas.compat import (
-    np_version_under1p17 as _np_version_under1p17,
     np_version_under1p18 as _np_version_under1p18,
     is_numpy_dev as _is_numpy_dev,
 )

pandas/_libs/arrays.pyx

@@ -0,0 +1,167 @@
+"""
+Cython implementations for internal ExtensionArrays.
+"""
+cimport cython
+
+import numpy as np
+
+cimport numpy as cnp
+from numpy cimport ndarray
+
+cnp.import_array()
+
+
+@cython.freelist(16)
+cdef class NDArrayBacked:
+    """
+    Implementing these methods in cython improves performance quite a bit.
+
+    import pandas as pd
+
+    from pandas._libs.arrays import NDArrayBacked as cls
+
+    dti = pd.date_range("2016-01-01", periods=3)
+    dta = dti._data
+    arr = dta._ndarray
+
+    obj = cls._simple_new(arr, arr.dtype)
+
+    # for foo in [arr, dta, obj]: ...
+
+    %timeit foo.copy()
+    299 ns ± 30 ns per loop     # <-- arr underlying ndarray (for reference)
+    530 ns ± 9.24 ns per loop   # <-- dta with cython NDArrayBacked
+    1.66 µs ± 46.3 ns per loop  # <-- dta without cython NDArrayBacked
+    328 ns ± 5.29 ns per loop   # <-- obj with NDArrayBacked.__cinit__
+    371 ns ± 6.97 ns per loop   # <-- obj with NDArrayBacked._simple_new
+
+    %timeit foo.T
+    125 ns ± 6.27 ns per loop   # <-- arr underlying ndarray (for reference)
+    226 ns ± 7.66 ns per loop   # <-- dta with cython NDArrayBacked
+    911 ns ± 16.6 ns per loop   # <-- dta without cython NDArrayBacked
+    215 ns ± 4.54 ns per loop   # <-- obj with NDArrayBacked._simple_new
+
+    """
+    # TODO: implement take in terms of cnp.PyArray_TakeFrom
+    # TODO: implement concat_same_type in terms of cnp.PyArray_Concatenate
+
+    cdef:
+        readonly ndarray _ndarray
+        readonly object _dtype
+
+    def __init__(self, ndarray values, object dtype):
+        self._ndarray = values
+        self._dtype = dtype
+
+    @classmethod
+    def _simple_new(cls, ndarray values, object dtype):
+        cdef:
+            NDArrayBacked obj
+        obj = NDArrayBacked.__new__(cls)
+        obj._ndarray = values
+        obj._dtype = dtype
+        return obj
+
+    cpdef NDArrayBacked _from_backing_data(self, ndarray values):
+        """
+        Construct a new ExtensionArray `new_array` with `arr` as its _ndarray.
+
+        This should round-trip:
+            self == self._from_backing_data(self._ndarray)
+        """
+        # TODO: re-reuse simple_new if/when it can be cpdef
+        cdef:
+            NDArrayBacked obj
+        obj = NDArrayBacked.__new__(type(self))
+        obj._ndarray = values
+        obj._dtype = self._dtype
+        return obj
+
+    cpdef __setstate__(self, state):
+        if isinstance(state, dict):
+            if "_data" in state:
+                data = state.pop("_data")
+            elif "_ndarray" in state:
+                data = state.pop("_ndarray")
+            else:
+                raise ValueError
+            self._ndarray = data
+            self._dtype = state.pop("_dtype")
+
+            for key, val in state.items():
+                setattr(self, key, val)
+        elif isinstance(state, tuple):
+            if len(state) != 3:
+                if len(state) == 1 and isinstance(state[0], dict):
+                    self.__setstate__(state[0])
+                    return
+                raise NotImplementedError(state)
+
+            data, dtype = state[:2]
+            if isinstance(dtype, np.ndarray):
+                dtype, data = data, dtype
+            self._ndarray = data
+            self._dtype = dtype
+
+            if isinstance(state[2], dict):
+                for key, val in state[2].items():
+                    setattr(self, key, val)
+            else:
+                raise NotImplementedError(state)
+        else:
+            raise NotImplementedError(state)
+
+    def __len__(self) -> int:
+        return len(self._ndarray)
+
+    @property
+    def shape(self):
+        # object cast bc _ndarray.shape is npy_intp*
+        return (<object>(self._ndarray)).shape
+
+    @property
+    def ndim(self) -> int:
+        return self._ndarray.ndim
+
+    @property
+    def size(self) -> int:
+        return self._ndarray.size
+
+    @property
+    def nbytes(self) -> int:
+        return self._ndarray.nbytes
+
+    def copy(self):
+        # NPY_ANYORDER -> same order as self._ndarray
+        res_values = cnp.PyArray_NewCopy(self._ndarray, cnp.NPY_ANYORDER)
+        return self._from_backing_data(res_values)
+
+    def delete(self, loc, axis=0):
+        res_values = np.delete(self._ndarray, loc, axis=axis)
+        return self._from_backing_data(res_values)
+
+    def swapaxes(self, axis1, axis2):
+        res_values = cnp.PyArray_SwapAxes(self._ndarray, axis1, axis2)
+        return self._from_backing_data(res_values)
+
+    # TODO: pass NPY_MAXDIMS equiv to axis=None?
+    def repeat(self, repeats, axis: int = 0):
+        if axis is None:
+            axis = 0
+        res_values = cnp.PyArray_Repeat(self._ndarray, repeats, <int>axis)
+        return self._from_backing_data(res_values)
+
+    def reshape(self, *args, **kwargs):
+        res_values = self._ndarray.reshape(*args, **kwargs)
+        return self._from_backing_data(res_values)
+
+    def ravel(self, order="C"):
+        # cnp.PyArray_OrderConverter(PyObject* obj, NPY_ORDER* order)
+        # res_values = cnp.PyArray_Ravel(self._ndarray, order)
+        res_values = self._ndarray.ravel(order)
+        return self._from_backing_data(res_values)
+
+    @property
+    def T(self):
+        res_values = self._ndarray.T
+        return self._from_backing_data(res_values)