Updation of branch from pandas-dev/master

Updation of fork

Author: souris-dev

README.md

@@ -32,7 +32,7 @@ its way towards this goal.
 Here are just a few of the things that pandas does well:
 
   - Easy handling of [**missing data**][missing-data] (represented as
-    `NaN`) in floating point as well as non-floating point data
+    `NaN`, `NA`, or `NaT`) in floating point as well as non-floating point data
   - Size mutability: columns can be [**inserted and
     deleted**][insertion-deletion] from DataFrame and higher dimensional
     objects

asv_bench/asv.conf.json

@@ -26,7 +26,7 @@
     // The Pythons you'd like to test against.  If not provided, defaults
     // to the current version of Python used to run `asv`.
     // "pythons": ["2.7", "3.4"],
-    "pythons": ["3.6"],
+    "pythons": ["3.8"],
 
     // The matrix of dependencies to test.  Each key is the name of a
     // package (in PyPI) and the values are version numbers.  An empty

ci/deps/azure-37-locale.yaml

@@ -21,6 +21,7 @@ dependencies:
   - lxml
   - matplotlib>=3.3.0
   - moto
+  - flask
   - nomkl
   - numexpr
   - numpy=1.16.*

ci/deps/azure-37-slow.yaml

@@ -27,9 +27,11 @@ dependencies:
   - python-dateutil
   - pytz
   - s3fs>=0.4.0
+  - moto>=1.3.14
   - scipy
   - sqlalchemy
   - xlrd
   - xlsxwriter
   - xlwt
   - moto
+  - flask

ci/deps/azure-38-locale.yaml

@@ -6,14 +6,15 @@ dependencies:
 
   # tools
   - cython>=0.29.16
-  - pytest>=5.0.1,<6.0.0  # https://github.com/pandas-dev/pandas/issues/35620
+  - pytest>=5.0.1
   - pytest-xdist>=1.21
-  - pytest-asyncio
+  - pytest-asyncio>=0.12.0
   - hypothesis>=3.58.0
   - pytest-azurepipelines
 
   # pandas dependencies
   - beautifulsoup4
+  - flask
   - html5lib
   - ipython
   - jinja2
@@ -32,6 +33,7 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
+  - moto
   - pyarrow>=0.15
   - pip
   - pip:

ci/deps/azure-windows-37.yaml

@@ -8,28 +8,29 @@ dependencies:
   # tools
   - cython>=0.29.16
   - pytest>=5.0.1
-  - pytest-xdist>=1.21,<2.0.0 # GH 35737
+  - pytest-xdist>=1.21
   - hypothesis>=3.58.0
   - pytest-azurepipelines
 
   # pandas dependencies
   - beautifulsoup4
   - bottleneck
-  - fsspec>=0.7.4
+  - fsspec>=0.8.0
   - gcsfs>=0.6.0
   - html5lib
   - jinja2
   - lxml
   - matplotlib=2.2.*
-  - moto
+  - moto>=1.3.14
+  - flask
   - numexpr
   - numpy=1.16.*
   - openpyxl
   - pyarrow=0.15
   - pytables
   - python-dateutil
   - pytz
-  - s3fs>=0.4.0
+  - s3fs>=0.4.2
   - scipy
   - sqlalchemy
   - xlrd

ci/deps/azure-windows-38.yaml

@@ -8,15 +8,18 @@ dependencies:
   # tools
   - cython>=0.29.16
   - pytest>=5.0.1
-  - pytest-xdist>=1.21,<2.0.0 # GH 35737
+  - pytest-xdist>=1.21
   - hypothesis>=3.58.0
   - pytest-azurepipelines
 
   # pandas dependencies
   - blosc
   - bottleneck
   - fastparquet>=0.3.2
+  - flask
+  - fsspec>=0.8.0
   - matplotlib=3.1.3
+  - moto>=1.3.14
   - numba
   - numexpr
   - numpy=1.18.*
@@ -26,6 +29,7 @@ dependencies:
   - pytables
   - python-dateutil
   - pytz
+  - s3fs>=0.4.0
   - scipy
   - xlrd
   - xlsxwriter

ci/deps/travis-37-arm64.yaml

@@ -17,5 +17,6 @@ dependencies:
   - python-dateutil
   - pytz
   - pip
+  - flask
   - pip:
     - moto

ci/deps/travis-37-cov.yaml

@@ -23,7 +23,8 @@ dependencies:
   - geopandas
   - html5lib
   - matplotlib
-  - moto
+  - moto>=1.3.14
+  - flask
   - nomkl
   - numexpr
   - numpy=1.16.*

ci/deps/travis-37-locale.yaml

@@ -21,13 +21,14 @@ dependencies:
   - jinja2
   - lxml=4.3.0
   - matplotlib=3.0.*
-  - moto
   - nomkl
   - numexpr
   - numpy
   - openpyxl
   - pandas-gbq=0.12.0
+  - pyarrow>=0.17
   - psycopg2=2.7
+  - pyarrow>=0.15.0 # GH #35813
   - pymysql=0.7.11
   - pytables
   - python-dateutil

ci/deps/travis-37.yaml

@@ -20,8 +20,8 @@ dependencies:
   - pyarrow
   - pytz
   - s3fs>=0.4.0
+  - moto>=1.3.14
+  - flask
   - tabulate
   - pyreadstat
   - pip
-  - pip:
-    - moto

doc/source/development/roadmap.rst

@@ -53,6 +53,32 @@ need to implement certain operations expected by pandas users (for example
 the algorithm used in, ``Series.str.upper``). That work may be done outside of
 pandas.
 
+Consistent missing value handling
+---------------------------------
+
+Currently, pandas handles missing data differently for different data types. We
+use different types to indicate that a value is missing (``np.nan`` for
+floating-point data, ``np.nan`` or ``None`` for object-dtype data -- typically
+strings or booleans -- with missing values, and ``pd.NaT`` for datetimelike
+data). Integer data cannot store missing data or are cast to float. In addition,
+pandas 1.0 introduced a new missing value sentinel, ``pd.NA``, which is being
+used for the experimental nullable integer, boolean, and string data types.
+
+These different missing values have different behaviors in user-facing
+operations. Specifically, we introduced different semantics for the nullable
+data types for certain operations (e.g. propagating in comparison operations
+instead of comparing as False).
+
+Long term, we want to introduce consistent missing data handling for all data
+types. This includes consistent behavior in all operations (indexing, arithmetic
+operations, comparisons, etc.). We want to eventually make the new semantics the
+default.
+
+This has been discussed at
+`github #28095 <https://github.com/pandas-dev/pandas/issues/28095>`__ (and
+linked issues), and described in more detail in this
+`design doc <https://hackmd.io/@jorisvandenbossche/Sk0wMeAmB>`__.
+
 Apache Arrow interoperability
 -----------------------------
 

doc/source/getting_started/tutorials.rst

@@ -94,4 +94,4 @@ Various tutorials
 * `Intro to pandas data structures, by Greg Reda <http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/>`_
 * `Pandas and Python: Top 10, by Manish Amde <https://manishamde.github.io/blog/2013/03/07/pandas-and-python-top-10/>`_
 * `Pandas DataFrames Tutorial, by Karlijn Willems <https://www.datacamp.com/community/tutorials/pandas-tutorial-dataframe-python>`_
-* `A concise tutorial with real life examples <https://tutswiki.com/pandas-cookbook/chapter1>`_
+* `A concise tutorial with real life examples <https://tutswiki.com/pandas-cookbook/chapter1/>`_

doc/source/reference/frame.rst

@@ -343,6 +343,7 @@ Sparse-dtype specific methods and attributes are provided under the
 
 .. autosummary::
    :toctree: api/
+   :template: autosummary/accessor_method.rst
 
    DataFrame.sparse.from_spmatrix
    DataFrame.sparse.to_coo

doc/source/reference/series.rst

@@ -522,6 +522,7 @@ Sparse-dtype specific methods and attributes are provided under the
 
 .. autosummary::
    :toctree: api/
+   :template: autosummary/accessor_method.rst
 
    Series.sparse.from_coo
    Series.sparse.to_coo

doc/source/user_guide/timeseries.rst

@@ -2319,13 +2319,18 @@ you can use the ``tz_convert`` method.
     Instead, the datetime needs to be localized using the ``localize`` method
     on the ``pytz`` time zone object.
 
+.. warning::
+
+    Be aware that for times in the future, correct conversion between time zones
+    (and UTC) cannot be guaranteed by any time zone library because a timezone's
+    offset from UTC may be changed by the respective government.
+
 .. warning::
 
     If you are using dates beyond 2038-01-18, due to current deficiencies
     in the underlying libraries caused by the year 2038 problem, daylight saving time (DST) adjustments
     to timezone aware dates will not be applied. If and when the underlying libraries are fixed,
-    the DST transitions will be applied. It should be noted though, that time zone data for far future time zones
-    are likely to be inaccurate, as they are simple extrapolations of the current set of (regularly revised) rules.
+    the DST transitions will be applied.
 
     For example, for two dates that are in British Summer Time (and so would normally be GMT+1), both the following asserts evaluate as true:
 

doc/source/user_guide/visualization.rst

@@ -668,6 +668,7 @@ A ``ValueError`` will be raised if there are any negative values in your data.
    plt.figure()
 
 .. ipython:: python
+   :okwarning:
 
    series = pd.Series(3 * np.random.rand(4),
                       index=['a', 'b', 'c', 'd'], name='series')
@@ -742,6 +743,7 @@ If you pass values whose sum total is less than 1.0, matplotlib draws a semicirc
    plt.figure()
 
 .. ipython:: python
+   :okwarning:
 
    series = pd.Series([0.1] * 4, index=['a', 'b', 'c', 'd'], name='series2')
 

doc/source/whatsnew/index.rst

@@ -24,6 +24,7 @@ Version 1.1
 .. toctree::
    :maxdepth: 2
 
+   v1.1.2
    v1.1.1
    v1.1.0
 

doc/source/whatsnew/v0.25.0.rst

@@ -540,7 +540,7 @@ with :attr:`numpy.nan` in the case of an empty :class:`DataFrame` (:issue:`26397
 
 .. ipython:: python
 
-    df.describe()
+   df.describe()
 
 ``__str__`` methods now call ``__repr__`` rather than vice versa
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/source/whatsnew/v1.1.1.rst

@@ -1,7 +1,7 @@
 .. _whatsnew_111:
 
-What's new in 1.1.1 (?)
------------------------
+What's new in 1.1.1 (August 20, 2020)
+-------------------------------------
 
 These are the changes in pandas 1.1.1. See :ref:`release` for a full changelog
 including other versions of pandas.
@@ -15,20 +15,23 @@ including other versions of pandas.
 Fixed regressions
 ~~~~~~~~~~~~~~~~~
 
+- Fixed regression in :meth:`CategoricalIndex.format` where, when stringified scalars had different lengths, the shorter string would be right-filled with spaces, so it had the same length as the longest string (:issue:`35439`)
+- Fixed regression in :meth:`Series.truncate` when trying to truncate a single-element series (:issue:`35544`)
 - Fixed regression where :meth:`DataFrame.to_numpy` would raise a ``RuntimeError`` for mixed dtypes when converting to ``str`` (:issue:`35455`)
 - Fixed regression where :func:`read_csv` would raise a ``ValueError`` when ``pandas.options.mode.use_inf_as_na`` was set to ``True`` (:issue:`35493`)
 - Fixed regression where :func:`pandas.testing.assert_series_equal` would raise an error when non-numeric dtypes were passed with ``check_exact=True`` (:issue:`35446`)
-- Fixed regression in :class:`pandas.core.groupby.RollingGroupby` where column selection was ignored (:issue:`35486`)
-- Fixed regression where :meth:`DataFrame.interpolate` would raise a ``TypeError`` when the :class:`DataFrame` was empty (:issue:`35598`).
+- Fixed regression in ``.groupby(..).rolling(..)`` where column selection was ignored (:issue:`35486`)
+- Fixed regression where :meth:`DataFrame.interpolate` would raise a ``TypeError`` when the :class:`DataFrame` was empty (:issue:`35598`)
 - Fixed regression in :meth:`DataFrame.shift` with ``axis=1`` and heterogeneous dtypes (:issue:`35488`)
 - Fixed regression in :meth:`DataFrame.diff` with read-only data (:issue:`35559`)
 - Fixed regression in ``.groupby(..).rolling(..)`` where a segfault would occur with ``center=True`` and an odd number of values (:issue:`35552`)
 - Fixed regression in :meth:`DataFrame.apply` where functions that altered the input in-place only operated on a single row (:issue:`35462`)
 - Fixed regression in :meth:`DataFrame.reset_index` would raise a ``ValueError`` on empty :class:`DataFrame` with a :class:`MultiIndex` with a ``datetime64`` dtype level (:issue:`35606`, :issue:`35657`)
-- Fixed regression where :meth:`DataFrame.merge_asof` would raise a ``UnboundLocalError`` when ``left_index`` , ``right_index`` and ``tolerance`` were set (:issue:`35558`)
+- Fixed regression where :func:`pandas.merge_asof` would raise a ``UnboundLocalError`` when ``left_index``, ``right_index`` and ``tolerance`` were set (:issue:`35558`)
 - Fixed regression in ``.groupby(..).rolling(..)`` where a custom ``BaseIndexer`` would be ignored (:issue:`35557`)
 - Fixed regression in :meth:`DataFrame.replace` and :meth:`Series.replace` where compiled regular expressions would be ignored during replacement (:issue:`35680`)
-- Fixed regression in :meth:`~pandas.core.groupby.DataFrameGroupBy.agg` where a list of functions would produce the wrong results if at least one of the functions did not aggregate. (:issue:`35490`)
+- Fixed regression in :meth:`~pandas.core.groupby.DataFrameGroupBy.aggregate` where a list of functions would produce the wrong results if at least one of the functions did not aggregate (:issue:`35490`)
+- Fixed memory usage issue when instantiating large :class:`pandas.arrays.StringArray` (:issue:`35499`)
 
 .. ---------------------------------------------------------------------------
 
@@ -37,50 +40,11 @@ Fixed regressions
 Bug fixes
 ~~~~~~~~~
 
-- Bug in ``Styler`` whereby `cell_ids` argument had no effect due to other recent changes (:issue:`35588`) (:issue:`35663`).
-- Bug in :func:`pandas.testing.assert_series_equal` and :func:`pandas.testing.assert_frame_equal` where extension dtypes were not ignored when ``check_dtypes`` was set to ``False`` (:issue:`35715`).
-
-Categorical
-^^^^^^^^^^^
-
-- Bug in :meth:`CategoricalIndex.format` where, when stringified scalars had different lengths, the shorter string would be right-filled with spaces, so it had the same length as the longest string (:issue:`35439`)
-
-
-**Datetimelike**
-
--
--
-
-**Timedelta**
-
-- Bug in :meth:`to_timedelta` fails when arg is a :class:`Series` with `Int64` dtype containing null values (:issue:`35574`)
-
-
-**Numeric**
-
--
--
-
-**Groupby/resample/rolling**
-
-- Bug in :class:`pandas.core.groupby.RollingGroupby` where passing ``closed`` with column selection would raise a ``ValueError`` (:issue:`35549`)
-
-**Plotting**
-
--
-
-**Indexing**
-
-- Bug in :meth:`Series.truncate` when trying to truncate a single-element series (:issue:`35544`)
-
-**DataFrame**
-- Bug in :class:`DataFrame` constructor failing to raise ``ValueError`` in some cases when data and index have mismatched lengths (:issue:`33437`)
--
-
-**Strings**
-
-- fix memory usage issue when instantiating large  :class:`pandas.arrays.StringArray` (:issue:`35499`)
-
+- Bug in :class:`~pandas.io.formats.style.Styler` whereby ``cell_ids`` argument had no effect due to other recent changes (:issue:`35588`) (:issue:`35663`)
+- Bug in :func:`pandas.testing.assert_series_equal` and :func:`pandas.testing.assert_frame_equal` where extension dtypes were not ignored when ``check_dtypes`` was set to ``False`` (:issue:`35715`)
+- Bug in :meth:`to_timedelta` fails when ``arg`` is a :class:`Series` with ``Int64`` dtype containing null values (:issue:`35574`)
+- Bug in ``.groupby(..).rolling(..)`` where passing ``closed`` with column selection would raise a ``ValueError`` (:issue:`35549`)
+- Bug in :class:`DataFrame` constructor failing to raise ``ValueError`` in some cases when ``data`` and ``index`` have mismatched lengths (:issue:`33437`)
 
 .. ---------------------------------------------------------------------------
 
@@ -89,4 +53,4 @@ Categorical
 Contributors
 ~~~~~~~~~~~~
 
-.. contributors:: v1.1.0..v1.1.1|HEAD
+.. contributors:: v1.1.0..v1.1.1