Merge pull request #8 from pandas-dev/master

Updation of fork

Author: souris-dev

README.md

@@ -32,7 +32,7 @@ its way towards this goal.
 Here are just a few of the things that pandas does well:
 
   - Easy handling of [**missing data**][missing-data] (represented as
-    `NaN`) in floating point as well as non-floating point data
+    `NaN`, `NA`, or `NaT`) in floating point as well as non-floating point data
   - Size mutability: columns can be [**inserted and
     deleted**][insertion-deletion] from DataFrame and higher dimensional
     objects

asv_bench/asv.conf.json

@@ -26,7 +26,7 @@
     // The Pythons you'd like to test against.  If not provided, defaults
     // to the current version of Python used to run `asv`.
     // "pythons": ["2.7", "3.4"],
-    "pythons": ["3.6"],
+    "pythons": ["3.8"],
 
     // The matrix of dependencies to test.  Each key is the name of a
     // package (in PyPI) and the values are version numbers.  An empty

ci/deps/azure-37-locale.yaml

@@ -21,6 +21,7 @@ dependencies:
   - lxml
   - matplotlib>=3.3.0
   - moto
+  - flask
   - nomkl
   - numexpr
   - numpy=1.16.*

ci/deps/azure-37-slow.yaml

@@ -27,9 +27,11 @@ dependencies:
   - python-dateutil
   - pytz
   - s3fs>=0.4.0
+  - moto>=1.3.14
   - scipy
   - sqlalchemy
   - xlrd
   - xlsxwriter
   - xlwt
   - moto
+  - flask

ci/deps/azure-38-locale.yaml

@@ -6,14 +6,15 @@ dependencies:
 
   # tools
   - cython>=0.29.16
-  - pytest>=5.0.1,<6.0.0  # https://github.com/pandas-dev/pandas/issues/35620
+  - pytest>=5.0.1
   - pytest-xdist>=1.21
-  - pytest-asyncio
+  - pytest-asyncio>=0.12.0
   - hypothesis>=3.58.0
   - pytest-azurepipelines
 
   # pandas dependencies
   - beautifulsoup4
+  - flask
   - html5lib
   - ipython
   - jinja2
@@ -32,6 +33,7 @@ dependencies:
   - xlrd
   - xlsxwriter
   - xlwt
+  - moto
   - pyarrow>=0.15
   - pip
   - pip:

ci/deps/azure-windows-37.yaml

@@ -8,28 +8,29 @@ dependencies:
   # tools
   - cython>=0.29.16
   - pytest>=5.0.1
-  - pytest-xdist>=1.21,<2.0.0 # GH 35737
+  - pytest-xdist>=1.21
   - hypothesis>=3.58.0
   - pytest-azurepipelines
 
   # pandas dependencies
   - beautifulsoup4
   - bottleneck
-  - fsspec>=0.7.4
+  - fsspec>=0.8.0
   - gcsfs>=0.6.0
   - html5lib
   - jinja2
   - lxml
   - matplotlib=2.2.*
-  - moto
+  - moto>=1.3.14
+  - flask
   - numexpr
   - numpy=1.16.*
   - openpyxl
   - pyarrow=0.15
   - pytables
   - python-dateutil
   - pytz
-  - s3fs>=0.4.0
+  - s3fs>=0.4.2
   - scipy
   - sqlalchemy
   - xlrd

ci/deps/azure-windows-38.yaml

@@ -8,15 +8,18 @@ dependencies:
   # tools
   - cython>=0.29.16
   - pytest>=5.0.1
-  - pytest-xdist>=1.21,<2.0.0 # GH 35737
+  - pytest-xdist>=1.21
   - hypothesis>=3.58.0
   - pytest-azurepipelines
 
   # pandas dependencies
   - blosc
   - bottleneck
   - fastparquet>=0.3.2
+  - flask
+  - fsspec>=0.8.0
   - matplotlib=3.1.3
+  - moto>=1.3.14
   - numba
   - numexpr
   - numpy=1.18.*
@@ -26,6 +29,7 @@ dependencies:
   - pytables
   - python-dateutil
   - pytz
+  - s3fs>=0.4.0
   - scipy
   - xlrd
   - xlsxwriter

ci/deps/travis-37-arm64.yaml

@@ -17,5 +17,6 @@ dependencies:
   - python-dateutil
   - pytz
   - pip
+  - flask
   - pip:
     - moto

ci/deps/travis-37-cov.yaml

@@ -23,7 +23,8 @@ dependencies:
   - geopandas
   - html5lib
   - matplotlib
-  - moto
+  - moto>=1.3.14
+  - flask
   - nomkl
   - numexpr
   - numpy=1.16.*

ci/deps/travis-37-locale.yaml

@@ -21,13 +21,14 @@ dependencies:
   - jinja2
   - lxml=4.3.0
   - matplotlib=3.0.*
-  - moto
   - nomkl
   - numexpr
   - numpy
   - openpyxl
   - pandas-gbq=0.12.0
+  - pyarrow>=0.17
   - psycopg2=2.7
+  - pyarrow>=0.15.0 # GH #35813
   - pymysql=0.7.11
   - pytables
   - python-dateutil

ci/deps/travis-37.yaml

@@ -20,8 +20,8 @@ dependencies:
   - pyarrow
   - pytz
   - s3fs>=0.4.0
+  - moto>=1.3.14
+  - flask
   - tabulate
   - pyreadstat
   - pip
-  - pip:
-    - moto

doc/source/development/contributing.rst

@@ -204,6 +204,7 @@ You will need `Build Tools for Visual Studio 2017
 	You DO NOT need to install Visual Studio 2019.
 	You only need "Build Tools for Visual Studio 2019" found by
 	scrolling down to "All downloads" -> "Tools for Visual Studio 2019".
+	In the installer, select the "C++ build tools" workload.
 
 **Mac OS**
 

doc/source/development/roadmap.rst

@@ -53,6 +53,32 @@ need to implement certain operations expected by pandas users (for example
 the algorithm used in, ``Series.str.upper``). That work may be done outside of
 pandas.
 
+Consistent missing value handling
+---------------------------------
+
+Currently, pandas handles missing data differently for different data types. We
+use different types to indicate that a value is missing (``np.nan`` for
+floating-point data, ``np.nan`` or ``None`` for object-dtype data -- typically
+strings or booleans -- with missing values, and ``pd.NaT`` for datetimelike
+data). Integer data cannot store missing data or are cast to float. In addition,
+pandas 1.0 introduced a new missing value sentinel, ``pd.NA``, which is being
+used for the experimental nullable integer, boolean, and string data types.
+
+These different missing values have different behaviors in user-facing
+operations. Specifically, we introduced different semantics for the nullable
+data types for certain operations (e.g. propagating in comparison operations
+instead of comparing as False).
+
+Long term, we want to introduce consistent missing data handling for all data
+types. This includes consistent behavior in all operations (indexing, arithmetic
+operations, comparisons, etc.). We want to eventually make the new semantics the
+default.
+
+This has been discussed at
+`github #28095 <https://github.com/pandas-dev/pandas/issues/28095>`__ (and
+linked issues), and described in more detail in this
+`design doc <https://hackmd.io/@jorisvandenbossche/Sk0wMeAmB>`__.
+
 Apache Arrow interoperability
 -----------------------------
 

doc/source/getting_started/overview.rst

@@ -9,9 +9,9 @@ Package overview
 **pandas** is a `Python <https://www.python.org>`__ package providing fast,
 flexible, and expressive data structures designed to make working with
 "relational" or "labeled" data both easy and intuitive. It aims to be the
-fundamental high-level building block for doing practical, **real world** data
+fundamental high-level building block for doing practical, **real-world** data
 analysis in Python. Additionally, it has the broader goal of becoming **the
-most powerful and flexible open source data analysis / manipulation tool
+most powerful and flexible open source data analysis/manipulation tool
 available in any language**. It is already well on its way toward this goal.
 
 pandas is well suited for many different kinds of data:
@@ -21,7 +21,7 @@ pandas is well suited for many different kinds of data:
   - Ordered and unordered (not necessarily fixed-frequency) time series data.
   - Arbitrary matrix data (homogeneously typed or heterogeneous) with row and
     column labels
-  - Any other form of observational / statistical data sets. The data actually
+  - Any other form of observational / statistical data sets. The data
     need not be labeled at all to be placed into a pandas data structure
 
 The two primary data structures of pandas, :class:`Series` (1-dimensional)
@@ -57,7 +57,7 @@ Here are just a few of the things that pandas does well:
     Excel files, databases, and saving / loading data from the ultrafast **HDF5
     format**
   - **Time series**-specific functionality: date range generation and frequency
-    conversion, moving window statistics, date shifting and lagging.
+    conversion, moving window statistics, date shifting, and lagging.
 
 Many of these principles are here to address the shortcomings frequently
 experienced using other languages / scientific research environments. For data
@@ -101,12 +101,12 @@ fashion.
 
 Also, we would like sensible default behaviors for the common API functions
 which take into account the typical orientation of time series and
-cross-sectional data sets. When using ndarrays to store 2- and 3-dimensional
+cross-sectional data sets. When using the N-dimensional array (ndarrays) to store 2- and 3-dimensional
 data, a burden is placed on the user to consider the orientation of the data
 set when writing functions; axes are considered more or less equivalent (except
 when C- or Fortran-contiguousness matters for performance). In pandas, the axes
 are intended to lend more semantic meaning to the data; i.e., for a particular
-data set there is likely to be a "right" way to orient the data. The goal,
+data set, there is likely to be a "right" way to orient the data. The goal,
 then, is to reduce the amount of mental effort required to code up data
 transformations in downstream functions.
 
@@ -148,8 +148,8 @@ pandas possible. Thanks to `all of our contributors <https://github.com/pandas-d
 If you're interested in contributing, please visit the :ref:`contributing guide <contributing>`.
 
 pandas is a `NumFOCUS <https://www.numfocus.org/open-source-projects/>`__ sponsored project.
-This will help ensure the success of development of pandas as a world-class open-source
-project, and makes it possible to `donate <https://pandas.pydata.org/donate.html>`__ to the project.
+This will help ensure the success of the development of pandas as a world-class open-source
+project and makes it possible to `donate <https://pandas.pydata.org/donate.html>`__ to the project.
 
 Project governance
 ------------------

doc/source/getting_started/tutorials.rst

@@ -94,4 +94,4 @@ Various tutorials
 * `Intro to pandas data structures, by Greg Reda <http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/>`_
 * `Pandas and Python: Top 10, by Manish Amde <https://manishamde.github.io/blog/2013/03/07/pandas-and-python-top-10/>`_
 * `Pandas DataFrames Tutorial, by Karlijn Willems <https://www.datacamp.com/community/tutorials/pandas-tutorial-dataframe-python>`_
-* `A concise tutorial with real life examples <https://tutswiki.com/pandas-cookbook/chapter1>`_
+* `A concise tutorial with real life examples <https://tutswiki.com/pandas-cookbook/chapter1/>`_

doc/source/reference/frame.rst

@@ -343,6 +343,7 @@ Sparse-dtype specific methods and attributes are provided under the
 
 .. autosummary::
    :toctree: api/
+   :template: autosummary/accessor_method.rst
 
    DataFrame.sparse.from_spmatrix
    DataFrame.sparse.to_coo

doc/source/reference/series.rst

@@ -522,6 +522,7 @@ Sparse-dtype specific methods and attributes are provided under the
 
 .. autosummary::
    :toctree: api/
+   :template: autosummary/accessor_method.rst
 
    Series.sparse.from_coo
    Series.sparse.to_coo

doc/source/user_guide/sparse.rst

@@ -87,14 +87,15 @@ The :attr:`SparseArray.dtype` property stores two pieces of information
    sparr.dtype
 
 
-A :class:`SparseDtype` may be constructed by passing each of these
+A :class:`SparseDtype` may be constructed by passing only a dtype
 
 .. ipython:: python
 
    pd.SparseDtype(np.dtype('datetime64[ns]'))
 
-The default fill value for a given NumPy dtype is the "missing" value for that dtype,
-though it may be overridden.
+in which case a default fill value will be used (for NumPy dtypes this is often the
+"missing" value for that dtype). To override this default an explicit fill value may be
+passed instead
 
 .. ipython:: python
 

doc/source/user_guide/timeseries.rst

@@ -2319,13 +2319,18 @@ you can use the ``tz_convert`` method.
     Instead, the datetime needs to be localized using the ``localize`` method
     on the ``pytz`` time zone object.
 
+.. warning::
+
+    Be aware that for times in the future, correct conversion between time zones
+    (and UTC) cannot be guaranteed by any time zone library because a timezone's
+    offset from UTC may be changed by the respective government.
+
 .. warning::
 
     If you are using dates beyond 2038-01-18, due to current deficiencies
     in the underlying libraries caused by the year 2038 problem, daylight saving time (DST) adjustments
     to timezone aware dates will not be applied. If and when the underlying libraries are fixed,
-    the DST transitions will be applied. It should be noted though, that time zone data for far future time zones
-    are likely to be inaccurate, as they are simple extrapolations of the current set of (regularly revised) rules.
+    the DST transitions will be applied.
 
     For example, for two dates that are in British Summer Time (and so would normally be GMT+1), both the following asserts evaluate as true:
 

doc/source/user_guide/visualization.rst

@@ -668,6 +668,7 @@ A ``ValueError`` will be raised if there are any negative values in your data.
    plt.figure()
 
 .. ipython:: python
+   :okwarning:
 
    series = pd.Series(3 * np.random.rand(4),
                       index=['a', 'b', 'c', 'd'], name='series')
@@ -742,6 +743,7 @@ If you pass values whose sum total is less than 1.0, matplotlib draws a semicirc
    plt.figure()
 
 .. ipython:: python
+   :okwarning:
 
    series = pd.Series([0.1] * 4, index=['a', 'b', 'c', 'd'], name='series2')
 

doc/source/whatsnew/index.rst

@@ -24,6 +24,7 @@ Version 1.1
 .. toctree::
    :maxdepth: 2
 
+   v1.1.2
    v1.1.1
    v1.1.0
 

doc/source/whatsnew/v0.25.0.rst

@@ -540,7 +540,7 @@ with :attr:`numpy.nan` in the case of an empty :class:`DataFrame` (:issue:`26397
 
 .. ipython:: python
 
-    df.describe()
+   df.describe()
 
 ``__str__`` methods now call ``__repr__`` rather than vice versa
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^