Merge branch 'master' of https://github.com/pandas-dev/pandas into ERR/imporve_error_message_cut/qcut

Author: ryankarlos

doc/source/development/contributing_docstring.rst

@@ -399,7 +399,7 @@ DataFrame:
 * DataFrame
 * pandas.Index
 * pandas.Categorical
-* pandas.SparseArray
+* pandas.arrays.SparseArray
 
 If the exact type is not relevant, but must be compatible with a numpy
 array, array-like can be specified. If Any type that can be iterated is

doc/source/getting_started/basics.rst

@@ -1951,7 +1951,7 @@ documentation sections for more on each type.
 | period            | :class:`PeriodDtype`      | :class:`Period`    | :class:`arrays.PeriodArray`   | ``'period[<freq>]'``,                   | :ref:`timeseries.periods`     |
 | (time spans)      |                           |                    |                               | ``'Period[<freq>]'``                    |                               |
 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
-| sparse            | :class:`SparseDtype`      | (none)             | :class:`SparseArray`          | ``'Sparse'``, ``'Sparse[int]'``,        | :ref:`sparse`                 |
+| sparse            | :class:`SparseDtype`      | (none)             | :class:`arrays.SparseArray`   | ``'Sparse'``, ``'Sparse[int]'``,        | :ref:`sparse`                 |
 |                   |                           |                    |                               | ``'Sparse[float]'``                     |                               |
 +-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
 | intervals         | :class:`IntervalDtype`    | :class:`Interval`  | :class:`arrays.IntervalArray` | ``'interval'``, ``'Interval'``,         | :ref:`advanced.intervalindex` |

doc/source/getting_started/dsintro.rst

@@ -676,11 +676,11 @@ similar to an ndarray:
    # only show the first 5 rows
    df[:5].T
 
+.. _dsintro.numpy_interop:
+
 DataFrame interoperability with NumPy functions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. _dsintro.numpy_interop:
-
 Elementwise NumPy ufuncs (log, exp, sqrt, ...) and various other NumPy functions
 can be used with no issues on Series and DataFrame, assuming the data within
 are numeric:
@@ -741,7 +741,7 @@ implementation takes precedence and a Series is returned.
    np.maximum(ser, idx)
 
 NumPy ufuncs are safe to apply to :class:`Series` backed by non-ndarray arrays,
-for example :class:`SparseArray` (see :ref:`sparse.calculation`). If possible,
+for example :class:`arrays.SparseArray` (see :ref:`sparse.calculation`). If possible,
 the ufunc is applied without converting the underlying data to an ndarray.
 
 Console display

doc/source/reference/arrays.rst

@@ -444,13 +444,13 @@ Sparse data
 -----------
 
 Data where a single value is repeated many times (e.g. ``0`` or ``NaN``) may
-be stored efficiently as a :class:`SparseArray`.
+be stored efficiently as a :class:`arrays.SparseArray`.
 
 .. autosummary::
    :toctree: api/
    :template: autosummary/class_without_autosummary.rst
 
-   SparseArray
+   arrays.SparseArray
 
 .. autosummary::
    :toctree: api/

doc/source/user_guide/missing_data.rst

@@ -920,3 +920,29 @@ filling missing values beforehand.
 
 A similar situation occurs when using Series or DataFrame objects in ``if``
 statements, see :ref:`gotchas.truth`.
+
+NumPy ufuncs
+------------
+
+:attr:`pandas.NA` implements NumPy's ``__array_ufunc__`` protocol. Most ufuncs
+work with ``NA``, and generally return ``NA``:
+
+.. ipython:: python
+
+   np.log(pd.NA)
+   np.add(pd.NA, 1)
+
+.. warning::
+
+   Currently, ufuncs involving an ndarray and ``NA`` will return an
+   object-dtype filled with NA values.
+
+   .. ipython:: python
+
+      a = np.array([1, 2, 3])
+      np.greater(a, pd.NA)
+
+   The return type here may change to return a different array type
+   in the future.
+
+See :ref:`dsintro.numpy_interop` for more on ufuncs.

doc/source/user_guide/sparse.rst

@@ -15,7 +15,7 @@ can be chosen, including 0) is omitted. The compressed values are not actually s
 
    arr = np.random.randn(10)
    arr[2:-2] = np.nan
-   ts = pd.Series(pd.SparseArray(arr))
+   ts = pd.Series(pd.arrays.SparseArray(arr))
    ts
 
 Notice the dtype, ``Sparse[float64, nan]``. The ``nan`` means that elements in the
@@ -51,7 +51,7 @@ identical to their dense counterparts.
 SparseArray
 -----------
 
-:class:`SparseArray` is a :class:`~pandas.api.extensions.ExtensionArray`
+:class:`arrays.SparseArray` is a :class:`~pandas.api.extensions.ExtensionArray`
 for storing an array of sparse values (see :ref:`basics.dtypes` for more
 on extension arrays). It is a 1-dimensional ndarray-like object storing
 only values distinct from the ``fill_value``:
@@ -61,7 +61,7 @@ only values distinct from the ``fill_value``:
    arr = np.random.randn(10)
    arr[2:5] = np.nan
    arr[7:8] = np.nan
-   sparr = pd.SparseArray(arr)
+   sparr = pd.arrays.SparseArray(arr)
    sparr
 
 A sparse array can be converted to a regular (dense) ndarray with :meth:`numpy.asarray`
@@ -144,7 +144,7 @@ to ``SparseArray`` and get a ``SparseArray`` as a result.
 
 .. ipython:: python
 
-   arr = pd.SparseArray([1., np.nan, np.nan, -2., np.nan])
+   arr = pd.arrays.SparseArray([1., np.nan, np.nan, -2., np.nan])
    np.abs(arr)
 
 
@@ -153,7 +153,7 @@ the correct dense result.
 
 .. ipython:: python
 
-   arr = pd.SparseArray([1., -1, -1, -2., -1], fill_value=-1)
+   arr = pd.arrays.SparseArray([1., -1, -1, -2., -1], fill_value=-1)
    np.abs(arr)
    np.abs(arr).to_dense()
 
@@ -194,7 +194,7 @@ From an array-like, use the regular :class:`Series` or
 .. ipython:: python
 
    # New way
-   pd.DataFrame({"A": pd.SparseArray([0, 1])})
+   pd.DataFrame({"A": pd.arrays.SparseArray([0, 1])})
 
 From a SciPy sparse matrix, use :meth:`DataFrame.sparse.from_spmatrix`,
 
@@ -256,10 +256,10 @@ Instead, you'll need to ensure that the values being assigned are sparse
 
 .. ipython:: python
 
-   df = pd.DataFrame({"A": pd.SparseArray([0, 1])})
+   df = pd.DataFrame({"A": pd.arrays.SparseArray([0, 1])})
    df['B'] = [0, 0]  # remains dense
    df['B'].dtype
-   df['B'] = pd.SparseArray([0, 0])
+   df['B'] = pd.arrays.SparseArray([0, 0])
    df['B'].dtype
 
 The ``SparseDataFrame.default_kind`` and ``SparseDataFrame.default_fill_value`` attributes

doc/source/whatsnew/v0.19.0.rst

@@ -1225,6 +1225,7 @@ Previously, sparse data were ``float64`` dtype by default, even if all inputs we
 As of v0.19.0, sparse data keeps the input dtype, and uses more appropriate ``fill_value`` defaults (``0`` for ``int64`` dtype, ``False`` for ``bool`` dtype).
 
 .. ipython:: python
+   :okwarning:
 
    pd.SparseArray([1, 2, 0, 0], dtype=np.int64)
    pd.SparseArray([True, False, False, False])

doc/source/whatsnew/v0.25.0.rst

@@ -354,6 +354,7 @@ When passed DataFrames whose values are sparse, :func:`concat` will now return a
 :class:`Series` or :class:`DataFrame` with sparse values, rather than a :class:`SparseDataFrame` (:issue:`25702`).
 
 .. ipython:: python
+   :okwarning:
 
    df = pd.DataFrame({"A": pd.SparseArray([0, 1])})
 
@@ -910,6 +911,7 @@ by a ``Series`` or ``DataFrame`` with sparse values.
 **New way**
 
 .. ipython:: python
+   :okwarning:
 
    df = pd.DataFrame({"A": pd.SparseArray([0, 0, 1, 2])})
    df.dtypes

doc/source/whatsnew/v1.0.0.rst

@@ -568,7 +568,7 @@ Deprecations
 - :func:`eval` keyword argument "truediv" is deprecated and will be removed in a future version (:issue:`29812`)
 - :meth:`DateOffset.isAnchored` and :meth:`DatetOffset.onOffset` are deprecated and will be removed in a future version, use :meth:`DateOffset.is_anchored` and :meth:`DateOffset.is_on_offset` instead (:issue:`30340`)
 - ``pandas.tseries.frequencies.get_offset`` is deprecated and will be removed in a future version, use ``pandas.tseries.frequencies.to_offset`` instead (:issue:`4205`)
-- :meth:`Categorical.take_nd` is deprecated, use :meth:`Categorical.take` instead (:issue:`27745`)
+- :meth:`Categorical.take_nd` and :meth:`CategoricalIndex.take_nd` are deprecated, use :meth:`Categorical.take` and :meth:`CategoricalIndex.take` instead (:issue:`27745`)
 - The parameter ``numeric_only`` of :meth:`Categorical.min` and :meth:`Categorical.max` is deprecated and replaced with ``skipna`` (:issue:`25303`)
 - The parameter ``label`` in :func:`lreshape` has been deprecated and will be removed in a future version (:issue:`29742`)
 - ``pandas.core.index`` has been deprecated and will be removed in a future version, the public classes are available in the top-level namespace (:issue:`19711`)
@@ -578,6 +578,7 @@ Deprecations
 - :meth:`DataFrame.to_stata`, :meth:`DataFrame.to_feather`, and :meth:`DataFrame.to_parquet` argument "fname" is deprecated, use "path" instead (:issue:`23574`)
 - The deprecated internal attributes ``_start``, ``_stop`` and ``_step`` of :class:`RangeIndex` now raise a ``FutureWarning`` instead of a ``DeprecationWarning`` (:issue:`26581`)
 - The ``pandas.util.testing`` module has been deprecated. Use the public API in ``pandas.testing`` documented at :ref:`api.general.testing` (:issue:`16232`).
+- ``pandas.SparseArray`` has been deprecated.  Use ``pandas.arrays.SparseArray`` (:class:`arrays.SparseArray`) instead. (:issue:`30642`)
 
 **Selecting Columns from a Grouped DataFrame**
 
@@ -833,6 +834,7 @@ Datetimelike
 - Bug in :meth:`Series.cummin` and :meth:`Series.cummax` with timezone-aware dtype incorrectly dropping its timezone (:issue:`15553`)
 - Bug in :class:`DatetimeArray`, :class:`TimedeltaArray`, and :class:`PeriodArray` where inplace addition and subtraction did not actually operate inplace (:issue:`24115`)
 - Bug in :func:`pandas.to_datetime` when called with ``Series`` storing ``IntegerArray`` raising ``TypeError`` instead of returning ``Series`` (:issue:`30050`)
+- Bug in :func:`date_range` with custom business hours as ``freq`` and given number of ``periods`` (:issue:`30593`)
 
 Timedelta
 ^^^^^^^^^
@@ -883,6 +885,7 @@ Interval
 - Bug in :meth:`IntervalIndex.get_indexer` where a :class:`Categorical` or :class:`CategoricalIndex` ``target`` would incorrectly raise a ``TypeError`` (:issue:`30063`)
 - Bug in ``pandas.core.dtypes.cast.infer_dtype_from_scalar`` where passing ``pandas_dtype=True`` did not infer :class:`IntervalDtype` (:issue:`30337`)
 - Bug in :class:`IntervalDtype` where the ``kind`` attribute was incorrectly set as ``None`` instead of ``"O"`` (:issue:`30568`)
+- Bug in :class:`IntervalIndex`, :class:`~arrays.IntervalArray`, and :class:`Series` with interval data where equality comparisons were incorrect (:issue:`24112`)
 
 Indexing
 ^^^^^^^^
@@ -907,7 +910,7 @@ Missing
 MultiIndex
 ^^^^^^^^^^
 
-- Constructior for :class:`MultiIndex` verifies that the given ``sortorder`` is compatible with the actual ``lexsort_depth``  if ``verify_integrity`` parameter is ``True`` (the default) (:issue:`28735`)
+- Constructor for :class:`MultiIndex` verifies that the given ``sortorder`` is compatible with the actual ``lexsort_depth``  if ``verify_integrity`` parameter is ``True`` (the default) (:issue:`28735`)
 - Series and MultiIndex `.drop` with `MultiIndex` raise exception if labels not in given in level (:issue:`8594`)
 -
 

pandas/__init__.py

@@ -115,7 +115,7 @@
     DataFrame,
 )
 
-from pandas.core.arrays.sparse import SparseArray, SparseDtype
+from pandas.core.arrays.sparse import SparseDtype
 
 from pandas.tseries.api import infer_freq
 from pandas.tseries import offsets
@@ -246,6 +246,19 @@ class Panel:
 
             return type(name, (), {})
 
+        elif name == "SparseArray":
+
+            warnings.warn(
+                "The pandas.SparseArray class is deprecated "
+                "and will be removed from pandas in a future version. "
+                "Use pandas.arrays.SparseArray instead.",
+                FutureWarning,
+                stacklevel=2,
+            )
+            from pandas.core.arrays.sparse import SparseArray as _SparseArray
+
+            return _SparseArray
+
         raise AttributeError(f"module 'pandas' has no attribute '{name}'")
 
 
@@ -308,6 +321,9 @@ def __getattr__(self, item):
 
     datetime = __Datetime().datetime
 
+    class SparseArray:
+        pass
+
 
 # module level doc-string
 __doc__ = """

pandas/_libs/algos_take_helper.pxi.in

@@ -116,7 +116,7 @@ def take_2d_axis0_{{name}}_{{dest}}(ndarray[{{c_type_in}}, ndim=2] values,
 
     IF {{True if c_type_in == c_type_out != "object" else False}}:
         cdef:
-            {{c_type_out}} *v
+            const {{c_type_out}} *v
             {{c_type_out}} *o
 
         # GH#3130

pandas/_libs/hashtable.pyx

@@ -1,7 +1,7 @@
 cimport cython
 
 from cpython.ref cimport PyObject, Py_INCREF
-from cpython.mem cimport PyMem_Malloc, PyMem_Realloc, PyMem_Free
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
 
 from libc.stdlib cimport malloc, free
 

pandas/_libs/missing.pyx

@@ -14,6 +14,7 @@ from pandas._libs.tslibs.np_datetime cimport (
     get_timedelta64_value, get_datetime64_value)
 from pandas._libs.tslibs.nattype cimport (
     checknull_with_nat, c_NaT as NaT, is_null_datetimelike)
+from pandas._libs.ops_dispatch import maybe_dispatch_ufunc_to_dunder_op
 
 from pandas.compat import is_platform_32bit
 
@@ -290,16 +291,29 @@ cdef inline bint is_null_period(v):
 # Implementation of NA singleton
 
 
-def _create_binary_propagating_op(name, divmod=False):
+def _create_binary_propagating_op(name, is_divmod=False):
 
     def method(self, other):
         if (other is C_NA or isinstance(other, str)
-                or isinstance(other, (numbers.Number, np.bool_))):
-            if divmod:
+                or isinstance(other, (numbers.Number, np.bool_))
+                or isinstance(other, np.ndarray) and not other.shape):
+            # Need the other.shape clause to handle NumPy scalars,
+            # since we do a setitem on `out` below, which
+            # won't work for NumPy scalars.
+            if is_divmod:
                 return NA, NA
             else:
                 return NA
 
+        elif isinstance(other, np.ndarray):
+            out = np.empty(other.shape, dtype=object)
+            out[:] = NA
+
+            if is_divmod:
+                return out, out.copy()
+            else:
+                return out
+
         return NotImplemented
 
     method.__name__ = name
@@ -369,8 +383,8 @@ class NAType(C_NAType):
     __rfloordiv__ = _create_binary_propagating_op("__rfloordiv__")
     __mod__ = _create_binary_propagating_op("__mod__")
     __rmod__ = _create_binary_propagating_op("__rmod__")
-    __divmod__ = _create_binary_propagating_op("__divmod__", divmod=True)
-    __rdivmod__ = _create_binary_propagating_op("__rdivmod__", divmod=True)
+    __divmod__ = _create_binary_propagating_op("__divmod__", is_divmod=True)
+    __rdivmod__ = _create_binary_propagating_op("__rdivmod__", is_divmod=True)
     # __lshift__ and __rshift__ are not implemented
 
     __eq__ = _create_binary_propagating_op("__eq__")
@@ -397,6 +411,8 @@ class NAType(C_NAType):
                 return type(other)(1)
             else:
                 return NA
+        elif isinstance(other, np.ndarray):
+            return np.where(other == 0, other.dtype.type(1), NA)
 
         return NotImplemented
 
@@ -408,6 +424,8 @@ class NAType(C_NAType):
                 return other
             else:
                 return NA
+        elif isinstance(other, np.ndarray):
+            return np.where((other == 1) | (other == -1), other, NA)
 
         return NotImplemented
 
@@ -440,6 +458,31 @@ class NAType(C_NAType):
 
     __rxor__ = __xor__
 
+    __array_priority__ = 1000
+    _HANDLED_TYPES = (np.ndarray, numbers.Number, str, np.bool_)
+
+    def __array_ufunc__(self, ufunc, method, *inputs, **kwargs):
+        types = self._HANDLED_TYPES + (NAType,)
+        for x in inputs:
+            if not isinstance(x, types):
+                return NotImplemented
+
+        if method != "__call__":
+            raise ValueError(f"ufunc method '{method}' not supported for NA")
+        result = maybe_dispatch_ufunc_to_dunder_op(
+            self, ufunc, method, *inputs, **kwargs
+        )
+        if result is NotImplemented:
+            # For a NumPy ufunc that's not a binop, like np.logaddexp
+            index = [i for i, x in enumerate(inputs) if x is NA][0]
+            result = np.broadcast_arrays(*inputs)[index]
+            if result.ndim == 0:
+                result = result.item()
+            if ufunc.nout > 1:
+                result = (NA,) * ufunc.nout
+
+        return result
+
 
 C_NA = NAType()   # C-visible
 NA = C_NA         # Python-visible