Merge branch 'main' into sas/decompress3

Author: jonashaag

doc/source/reference/testing.rst

@@ -26,6 +26,7 @@ Exceptions and warnings
 
    errors.AbstractMethodError
    errors.AccessorRegistrationWarning
+   errors.CSSWarning
    errors.DataError
    errors.DtypeWarning
    errors.DuplicateLabelError

doc/source/whatsnew/v1.4.4.rst

@@ -24,7 +24,7 @@ Fixed regressions
 Bug fixes
 ~~~~~~~~~
 - The :class:`errors.FutureWarning` raised when passing arguments (other than ``filepath_or_buffer``) as positional in :func:`read_csv` is now raised at the correct stacklevel (:issue:`47385`)
--
+- Bug in :meth:`DataFrame.to_sql` when ``method`` was a ``callable`` that did not return an ``int`` and would raise a ``TypeError`` (:issue:`46891`)
 
 .. ---------------------------------------------------------------------------
 

doc/source/whatsnew/v1.5.0.rst

@@ -275,6 +275,7 @@ Other enhancements
 - :class:`.DataError`, :class:`.SpecificationError`, :class:`.SettingWithCopyError`, :class:`.SettingWithCopyWarning`, :class:`.NumExprClobberingError`, :class:`.UndefinedVariableError`, and :class:`.IndexingError` are now exposed in ``pandas.errors`` (:issue:`27656`)
 - Added ``check_like`` argument to :func:`testing.assert_series_equal` (:issue:`47247`)
 - Allow reading compressed SAS files with :func:`read_sas` (e.g., ``.sas7bdat.gz`` files)
+- :class:`Series` reducers (e.g. ``min``, ``max``, ``sum``, ``mean``) will now successfully operate when the dtype is numeric and ``numeric_only=True`` is provided; previously this would raise a ``NotImplementedError`` (:issue:`47500`)
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_150.notable_bug_fixes:
@@ -766,7 +767,8 @@ Other Deprecations
 - Deprecated the argument ``na_sentinel`` in :func:`factorize`, :meth:`Index.factorize`, and :meth:`.ExtensionArray.factorize`; pass ``use_na_sentinel=True`` instead to use the sentinel ``-1`` for NaN values and ``use_na_sentinel=False`` instead of ``na_sentinel=None`` to encode NaN values (:issue:`46910`)
 - Deprecated :meth:`DataFrameGroupBy.transform` not aligning the result when the UDF returned DataFrame (:issue:`45648`)
 - Clarified warning from :func:`to_datetime` when delimited dates can't be parsed in accordance to specified ``dayfirst`` argument (:issue:`46210`)
-
+- Deprecated :class:`Series` and :class:`Resampler` reducers (e.g. ``min``, ``max``, ``sum``, ``mean``) raising a ``NotImplementedError`` when the dtype is non-numric and ``numeric_only=True`` is provided; this will raise a ``TypeError`` in a future version (:issue:`47500`)
+- Deprecated :meth:`Series.rank` returning an empty result when the dtype is non-numeric and ``numeric_only=True`` is provided; this will raise a ``TypeError`` in a future version (:issue:`47500`)
 
 .. ---------------------------------------------------------------------------
 .. _whatsnew_150.performance:
@@ -852,6 +854,7 @@ Conversion
 - Bug in metaclass of generic abstract dtypes causing :meth:`DataFrame.apply` and :meth:`Series.apply` to raise for the built-in function ``type`` (:issue:`46684`)
 - Bug in :meth:`DataFrame.to_records` returning inconsistent numpy types if the index was a :class:`MultiIndex` (:issue:`47263`)
 - Bug in :meth:`DataFrame.to_dict` for ``orient="list"`` or ``orient="index"`` was not returning native types (:issue:`46751`)
+- Bug in :meth:`DataFrame.apply` that returns a :class:`DataFrame` instead of a :class:`Series` when applied to an empty :class:`DataFrame` and ``axis=1`` (:issue:`39111`)
 
 Strings
 ^^^^^^^
@@ -882,6 +885,7 @@ Indexing
 - Bug in :meth:`Series.__setitem__` when setting ``boolean`` dtype values containing ``NA`` incorrectly raising instead of casting to ``boolean`` dtype (:issue:`45462`)
 - Bug in :meth:`Series.__setitem__` where setting :attr:`NA` into a numeric-dtype :class:`Series` would incorrectly upcast to object-dtype rather than treating the value as ``np.nan`` (:issue:`44199`)
 - Bug in :meth:`DataFrame.loc` when setting values to a column and right hand side is a dictionary (:issue:`47216`)
+- Bug in :meth:`DataFrame.loc` when setting a :class:`DataFrame` not aligning index in some cases (:issue:`47578`)
 - Bug in :meth:`Series.__setitem__` with ``datetime64[ns]`` dtype, an all-``False`` boolean mask, and an incompatible value incorrectly casting to ``object`` instead of retaining ``datetime64[ns]`` dtype (:issue:`45967`)
 - Bug in :meth:`Index.__getitem__`  raising ``ValueError`` when indexer is from boolean dtype with ``NA`` (:issue:`45806`)
 - Bug in :meth:`Series.__setitem__` losing precision when enlarging :class:`Series` with scalar (:issue:`32346`)
@@ -934,12 +938,14 @@ I/O
 - Bug in :func:`read_parquet` when ``engine="fastparquet"`` where the file was not closed on error (:issue:`46555`)
 - :meth:`to_html` now excludes the ``border`` attribute from ``<table>`` elements when ``border`` keyword is set to ``False``.
 - Bug in :func:`read_sas` with certain types of compressed SAS7BDAT files (:issue:`35545`)
+- Bug in :func:`read_excel` not forward filling :class:`MultiIndex` when no names were given (:issue:`47487`)
 - Bug in :func:`read_sas` returned ``None`` rather than an empty DataFrame for SAS7BDAT files with zero rows (:issue:`18198`)
 - Bug in :class:`StataWriter` where value labels were always written with default encoding (:issue:`46750`)
 - Bug in :class:`StataWriterUTF8` where some valid characters were removed from variable names (:issue:`47276`)
 - Bug in :meth:`DataFrame.to_excel` when writing an empty dataframe with :class:`MultiIndex` (:issue:`19543`)
 - Bug in :func:`read_sas` with RLE-compressed SAS7BDAT files that contain 0x40 control bytes (:issue:`31243`)
 - Bug in :func:`read_sas` that scrambled column names (:issue:`31243`)
+- Bug in :func:`read_sas` with RLE-compressed SAS7BDAT files that contain 0x00 control bytes (:issue:`47099`)
 -
 
 Period
@@ -995,6 +1001,7 @@ Reshaping
 - Bug in :func:`get_dummies` that selected object and categorical dtypes but not string (:issue:`44965`)
 - Bug in :meth:`DataFrame.align` when aligning a :class:`MultiIndex` to a :class:`Series` with another :class:`MultiIndex` (:issue:`46001`)
 - Bug in concatenation with ``IntegerDtype``, or ``FloatingDtype`` arrays where the resulting dtype did not mirror the behavior of the non-nullable dtypes (:issue:`46379`)
+- Bug in :func:`concat` losing dtype of columns when ``join="outer"`` and ``sort=True`` (:issue:`47329`)
 - Bug in :func:`concat` not sorting the column names when ``None`` is included (:issue:`47331`)
 - Bug in :func:`concat` with identical key leads to error when indexing :class:`MultiIndex` (:issue:`46519`)
 - Bug in :meth:`DataFrame.join` with a list when using suffixes to join DataFrames with duplicate column names (:issue:`46396`)

pandas/__init__.py

@@ -1,4 +1,5 @@
 # flake8: noqa
+from __future__ import annotations
 
 __docformat__ = "restructuredtext"
 
@@ -185,7 +186,7 @@
 __deprecated_num_index_names = ["Float64Index", "Int64Index", "UInt64Index"]
 
 
-def __dir__():
+def __dir__() -> list[str]:
     # GH43028
     # Int64Index etc. are deprecated, but we still want them to be available in the dir.
     # Remove in Pandas 2.0, when we remove Int64Index etc. from the code base.

pandas/_config/config.py

@@ -60,6 +60,7 @@
     Callable,
     Generic,
     Iterable,
+    Iterator,
     NamedTuple,
     cast,
 )
@@ -435,13 +436,13 @@ def __init__(self, *args) -> None:
 
         self.ops = list(zip(args[::2], args[1::2]))
 
-    def __enter__(self):
+    def __enter__(self) -> None:
         self.undo = [(pat, _get_option(pat, silent=True)) for pat, val in self.ops]
 
         for pat, val in self.ops:
             _set_option(pat, val, silent=True)
 
-    def __exit__(self, *args):
+    def __exit__(self, *args) -> None:
         if self.undo:
             for pat, val in self.undo:
                 _set_option(pat, val, silent=True)
@@ -733,7 +734,7 @@ def pp(name: str, ks: Iterable[str]) -> list[str]:
 
 
 @contextmanager
-def config_prefix(prefix):
+def config_prefix(prefix) -> Iterator[None]:
     """
     contextmanager for multiple invocations of API with a common prefix
 

pandas/_libs/algos.pyx

@@ -324,6 +324,7 @@ def kth_smallest(numeric_t[::1] arr, Py_ssize_t k) -> numeric_t:
 
 @cython.boundscheck(False)
 @cython.wraparound(False)
+@cython.cdivision(True)
 def nancorr(const float64_t[:, :] mat, bint cov=False, minp=None):
     cdef:
         Py_ssize_t i, j, xi, yi, N, K
@@ -356,8 +357,8 @@ def nancorr(const float64_t[:, :] mat, bint cov=False, minp=None):
                         nobs += 1
                         dx = vx - meanx
                         dy = vy - meany
-                        meanx += 1 / nobs * dx
-                        meany += 1 / nobs * dy
+                        meanx += 1. / nobs * dx
+                        meany += 1. / nobs * dy
                         ssqdmx += (vx - meanx) * dx
                         ssqdmy += (vy - meany) * dy
                         covxy += (vx - meanx) * dy

pandas/_testing/__init__.py

@@ -238,15 +238,15 @@
 _testing_mode_warnings = (DeprecationWarning, ResourceWarning)
 
 
-def set_testing_mode():
+def set_testing_mode() -> None:
     # set the testing mode filters
     testing_mode = os.environ.get("PANDAS_TESTING_MODE", "None")
     if "deprecate" in testing_mode:
         for category in _testing_mode_warnings:
             warnings.simplefilter("always", category)
 
 
-def reset_testing_mode():
+def reset_testing_mode() -> None:
     # reset the testing mode filters
     testing_mode = os.environ.get("PANDAS_TESTING_MODE", "None")
     if "deprecate" in testing_mode:
@@ -257,7 +257,7 @@ def reset_testing_mode():
 set_testing_mode()
 
 
-def reset_display_options():
+def reset_display_options() -> None:
     """
     Reset the display options for printing and representing objects.
     """
@@ -333,38 +333,38 @@ def to_array(obj):
 # Others
 
 
-def getCols(k):
+def getCols(k) -> str:
     return string.ascii_uppercase[:k]
 
 
 # make index
-def makeStringIndex(k=10, name=None):
+def makeStringIndex(k=10, name=None) -> Index:
     return Index(rands_array(nchars=10, size=k), name=name)
 
 
-def makeCategoricalIndex(k=10, n=3, name=None, **kwargs):
+def makeCategoricalIndex(k=10, n=3, name=None, **kwargs) -> CategoricalIndex:
     """make a length k index or n categories"""
     x = rands_array(nchars=4, size=n, replace=False)
     return CategoricalIndex(
         Categorical.from_codes(np.arange(k) % n, categories=x), name=name, **kwargs
     )
 
 
-def makeIntervalIndex(k=10, name=None, **kwargs):
+def makeIntervalIndex(k=10, name=None, **kwargs) -> IntervalIndex:
     """make a length k IntervalIndex"""
     x = np.linspace(0, 100, num=(k + 1))
     return IntervalIndex.from_breaks(x, name=name, **kwargs)
 
 
-def makeBoolIndex(k=10, name=None):
+def makeBoolIndex(k=10, name=None) -> Index:
     if k == 1:
         return Index([True], name=name)
     elif k == 2:
         return Index([False, True], name=name)
     return Index([False, True] + [False] * (k - 2), name=name)
 
 
-def makeNumericIndex(k=10, name=None, *, dtype):
+def makeNumericIndex(k=10, name=None, *, dtype) -> NumericIndex:
     dtype = pandas_dtype(dtype)
     assert isinstance(dtype, np.dtype)
 
@@ -382,21 +382,21 @@ def makeNumericIndex(k=10, name=None, *, dtype):
     return NumericIndex(values, dtype=dtype, name=name)
 
 
-def makeIntIndex(k=10, name=None):
+def makeIntIndex(k=10, name=None) -> Int64Index:
     base_idx = makeNumericIndex(k, name=name, dtype="int64")
     return Int64Index(base_idx)
 
 
-def makeUIntIndex(k=10, name=None):
+def makeUIntIndex(k=10, name=None) -> UInt64Index:
     base_idx = makeNumericIndex(k, name=name, dtype="uint64")
     return UInt64Index(base_idx)
 
 
-def makeRangeIndex(k=10, name=None, **kwargs):
+def makeRangeIndex(k=10, name=None, **kwargs) -> RangeIndex:
     return RangeIndex(0, k, 1, name=name, **kwargs)
 
 
-def makeFloatIndex(k=10, name=None):
+def makeFloatIndex(k=10, name=None) -> Float64Index:
     base_idx = makeNumericIndex(k, name=name, dtype="float64")
     return Float64Index(base_idx)
 
@@ -456,57 +456,57 @@ def all_timeseries_index_generator(k: int = 10) -> Iterable[Index]:
 
 
 # make series
-def make_rand_series(name=None, dtype=np.float64):
+def make_rand_series(name=None, dtype=np.float64) -> Series:
     index = makeStringIndex(_N)
     data = np.random.randn(_N)
     data = data.astype(dtype, copy=False)
     return Series(data, index=index, name=name)
 
 
-def makeFloatSeries(name=None):
+def makeFloatSeries(name=None) -> Series:
     return make_rand_series(name=name)
 
 
-def makeStringSeries(name=None):
+def makeStringSeries(name=None) -> Series:
     return make_rand_series(name=name)
 
 
-def makeObjectSeries(name=None):
+def makeObjectSeries(name=None) -> Series:
     data = makeStringIndex(_N)
     data = Index(data, dtype=object)
     index = makeStringIndex(_N)
     return Series(data, index=index, name=name)
 
 
-def getSeriesData():
+def getSeriesData() -> dict[str, Series]:
     index = makeStringIndex(_N)
     return {c: Series(np.random.randn(_N), index=index) for c in getCols(_K)}
 
 
-def makeTimeSeries(nper=None, freq="B", name=None):
+def makeTimeSeries(nper=None, freq="B", name=None) -> Series:
     if nper is None:
         nper = _N
     return Series(
         np.random.randn(nper), index=makeDateIndex(nper, freq=freq), name=name
     )
 
 
-def makePeriodSeries(nper=None, name=None):
+def makePeriodSeries(nper=None, name=None) -> Series:
     if nper is None:
         nper = _N
     return Series(np.random.randn(nper), index=makePeriodIndex(nper), name=name)
 
 
-def getTimeSeriesData(nper=None, freq="B"):
+def getTimeSeriesData(nper=None, freq="B") -> dict[str, Series]:
     return {c: makeTimeSeries(nper, freq) for c in getCols(_K)}
 
 
-def getPeriodData(nper=None):
+def getPeriodData(nper=None) -> dict[str, Series]:
     return {c: makePeriodSeries(nper) for c in getCols(_K)}
 
 
 # make frame
-def makeTimeDataFrame(nper=None, freq="B"):
+def makeTimeDataFrame(nper=None, freq="B") -> DataFrame:
     data = getTimeSeriesData(nper, freq)
     return DataFrame(data)
 
@@ -533,14 +533,19 @@ def makeMixedDataFrame():
     return DataFrame(getMixedTypeDict()[1])
 
 
-def makePeriodFrame(nper=None):
+def makePeriodFrame(nper=None) -> DataFrame:
     data = getPeriodData(nper)
     return DataFrame(data)
 
 
 def makeCustomIndex(
-    nentries, nlevels, prefix="#", names=False, ndupe_l=None, idx_type=None
-):
+    nentries,
+    nlevels,
+    prefix="#",
+    names: bool | str | list[str] | None = False,
+    ndupe_l=None,
+    idx_type=None,
+) -> Index:
     """
     Create an index/multindex with given dimensions, levels, names, etc'
 
@@ -637,7 +642,8 @@ def keyfunc(x):
     # convert tuples to index
     if nentries == 1:
         # we have a single level of tuples, i.e. a regular Index
-        index = Index(tuples[0], name=names[0])
+        name = None if names is None else names[0]
+        index = Index(tuples[0], name=name)
     elif nlevels == 1:
         name = None if names is None else names[0]
         index = Index((x[0] for x in tuples), name=name)
@@ -659,7 +665,7 @@ def makeCustomDataframe(
     dtype=None,
     c_idx_type=None,
     r_idx_type=None,
-):
+) -> DataFrame:
     """
     Create a DataFrame using supplied parameters.
 
@@ -780,7 +786,7 @@ def _gen_unique_rand(rng, _extra_size):
     return i.tolist(), j.tolist()
 
 
-def makeMissingDataframe(density=0.9, random_state=None):
+def makeMissingDataframe(density=0.9, random_state=None) -> DataFrame:
     df = makeDataFrame()
     i, j = _create_missing_idx(*df.shape, density=density, random_state=random_state)
     df.values[i, j] = np.nan
@@ -854,7 +860,7 @@ def skipna_wrapper(x):
     return skipna_wrapper
 
 
-def convert_rows_list_to_csv_str(rows_list: list[str]):
+def convert_rows_list_to_csv_str(rows_list: list[str]) -> str:
     """
     Convert list of CSV rows to single CSV-formatted string for current OS.
 

pandas/_testing/_io.py

@@ -250,7 +250,7 @@ def wrapper(*args, **kwargs):
     return wrapper
 
 
-def can_connect(url, error_classes=None):
+def can_connect(url, error_classes=None) -> bool:
     """
     Try to connect to the given url. True if succeeds, False if OSError
     raised
@@ -424,7 +424,7 @@ def write_to_compressed(compression, path, data, dest="test"):
 # Plotting
 
 
-def close(fignum=None):
+def close(fignum=None) -> None:
     from matplotlib.pyplot import (
         close as _close,
         get_fignums,