Merge branch 'master' into ndarray_tolerance

Author: buntwo

MANIFEST.in

@@ -3,6 +3,7 @@ include LICENSE
 include RELEASE.md
 include README.rst
 include setup.py
+include pyproject.toml
 
 graft doc
 prune doc/build

doc/source/overview.rst

@@ -6,7 +6,11 @@
 Package overview
 ****************
 
-:mod:`pandas` consists of the following things
+:mod:`pandas` is an open source, BSD-licensed library providing high-performance, 
+easy-to-use data structures and data analysis tools for the `Python <https://www.python.org/>`__
+programming language.
+
+:mod:`pandas` consists of the following elements
 
  * A set of labeled array data structures, the primary of which are
    Series and DataFrame
@@ -21,27 +25,23 @@ Package overview
  * Memory-efficient "sparse" versions of the standard data structures for storing
    data that is mostly missing or mostly constant (some fixed value)
  * Moving window statistics (rolling mean, rolling standard deviation, etc.)
- * Static and moving window linear and `panel regression
-   <http://en.wikipedia.org/wiki/Panel_data>`__
 
-Data structures at a glance
----------------------------
+Data Structures
+---------------
 
 .. csv-table::
     :header: "Dimensions", "Name", "Description"
     :widths: 15, 20, 50
 
-    1, Series, "1D labeled homogeneously-typed array"
-    2, DataFrame, "General 2D labeled, size-mutable tabular structure with
-    potentially heterogeneously-typed columns"
-    3, Panel, "General 3D labeled, also size-mutable array"
+    1, "Series", "1D labeled homogeneously-typed array"
+    2, "DataFrame", "General 2D labeled, size-mutable tabular structure with potentially heterogeneously-typed column"
 
-Why more than 1 data structure?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Why more than one data structure?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The best way to think about the pandas data structures is as flexible
 containers for lower dimensional data. For example, DataFrame is a container
-for Series, and Panel is a container for DataFrame objects. We would like to be
+for Series, and Series is a container for scalars. We would like to be
 able to insert and remove objects from these containers in a dictionary-like
 fashion.
 
@@ -85,36 +85,41 @@ The first stop for pandas issues and ideas is the `Github Issue Tracker
 pandas community experts can answer through `Stack Overflow
 <http://stackoverflow.com/questions/tagged/pandas>`__.
 
-Longer discussions occur on the `developer mailing list
-<http://groups.google.com/group/pystatsmodels>`__, and commercial support
-inquiries for Lambda Foundry should be sent to: support@lambdafoundry.com
+Community
+---------
 
-Credits
--------
+pandas is actively supported today by a community of like-minded individuals around 
+the world who contribute their valuable time and energy to help make open source 
+pandas possible. Thanks to `all of our contributors <https://github.com/pandas-dev/pandas/graphs/contributors>`__.
+
+If you're interested in contributing, please
+visit `Contributing to pandas webpage <http://pandas.pydata.org/pandas-docs/stable/contributing.html>`__.
 
-pandas development began at `AQR Capital Management <http://www.aqr.com>`__ in
-April 2008. It was open-sourced at the end of 2009. AQR continued to provide
-resources for development through the end of 2011, and continues to contribute
-bug reports today.
+pandas is a `NUMFocus <https://www.numfocus.org/open-source-projects/>`__ sponsored project.
+This will help ensure the success of development of pandas as a world-class open-source
+project, and makes it possible to `donate <https://pandas.pydata.org/donate.html>`__ to the project.
 
-Since January 2012, `Lambda Foundry <http://www.lambdafoundry.com>`__, has
-been providing development resources, as well as commercial support,
-training, and consulting for pandas.
+Project Governance
+------------------
 
-pandas is only made possible by a group of people around the world like you
-who have contributed new code, bug reports, fixes, comments and ideas. A
-complete list can be found `on Github <http://www.github.com/pandas-dev/pandas/contributors>`__.
+The governance process that pandas project has used informally since its inception in 2008 is formalized in `Project Governance documents <https://github.com/pandas-dev/pandas-governance>`__ .
+The documents clarify how decisions are made and how the various elements of our community interact, including the relationship between open source collaborative development and work that may be funded by for-profit or non-profit entities.
+
+Wes McKinney is the Benevolent Dictator for Life (BDFL).
 
 Development Team
-----------------
+-----------------
+
+The list of the  Core Team members and more detailed information can be found on the `people’s page <https://github.com/pandas-dev/pandas-governance/blob/master/people.md>`__ of the governance repo.
+ 
 
-pandas is a part of the PyData project. The PyData Development Team is a
-collection of developers focused on the improvement of Python's data
-libraries. The core team that coordinates development can be found on `Github
-<http://github.com/pydata>`__. If you're interested in contributing, please
-visit the `project website <http://pandas.pydata.org>`__.
+Institutional Partners
+----------------------
+
+The information about current institutional partners can be found on `pandas website page <https://pandas.pydata.org/about.html>`__
 
 License
 -------
 
 .. literalinclude:: ../../LICENSE
+

doc/source/whatsnew/v0.21.0.txt

@@ -112,6 +112,7 @@ Other Enhancements
 ^^^^^^^^^^^^^^^^^^
 
 - The ``validate`` argument for :func:`merge` function now checks whether a merge is one-to-one, one-to-many, many-to-one, or many-to-many. If a merge is found to not be an example of specified merge type, an exception of type ``MergeError`` will be raised. For more, see :ref:`here <merging.validation>` (:issue:`16270`)
+- Added support for `PEP 518 <https://www.python.org/dev/peps/pep-0518/>`_ to the build system (:issue:`16745`)
 - :func:`Series.to_dict` and :func:`DataFrame.to_dict` now support an ``into`` keyword which allows you to specify the ``collections.Mapping`` subclass that you would like returned.  The default is ``dict``, which is backwards compatible. (:issue:`16122`)
 - :func:`RangeIndex.append` now returns a ``RangeIndex`` object when possible (:issue:`16212`)
 - :func:`Series.rename_axis` and :func:`DataFrame.rename_axis` with ``inplace=True`` now return ``None`` while renaming the axis inplace. (:issue:`15704`)
@@ -272,6 +273,30 @@ named ``.isna()`` and ``.notna()``, these are included for classes ``Categorical
 
 The configuration option ``pd.options.mode.use_inf_as_null`` is deprecated, and ``pd.options.mode.use_inf_as_na`` is added as a replacement.
 
+.. _whatsnew_210.api.multiindex_single:
+
+MultiIndex Constructor with a Single Level
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``MultiIndex`` constructors no longer squeeze a MultiIndex with all
+length-one levels down to a regular ``Index``. This affects all the
+``MultiIndex`` constructors. (:issue:`17178`)
+
+Previous behavior:
+
+.. code-block:: ipython
+
+   In [2]: pd.MultiIndex.from_tuples([('a',), ('b',)])
+   Out[2]: Index(['a', 'b'], dtype='object')
+
+Length 1 levels are no longer special-cased. They behave exactly as if you had
+length 2+ levels, so a :class:`MultiIndex` is always returned from all of the
+``MultiIndex`` constructors:
+
+.. ipython:: python
+
+   pd.MultiIndex.from_tuples([('a',), ('b',)])
+
 .. _whatsnew_0210.api:
 
 Other API Changes
@@ -357,6 +382,7 @@ Indexing
 - Allow unicode empty strings as placeholders in multilevel columns in Python 2 (:issue:`17099`)
 - Bug in ``.iloc`` when used with inplace addition or assignment and an int indexer on a ``MultiIndex`` causing the wrong indexes to be read from and written to (:issue:`17148`)
 - Bug in ``.isin()`` in which checking membership in empty ``Series`` objects raised an error (:issue:`16991`)
+- Bug in ``CategoricalIndex`` reindexing in which specified indices containing duplicates were not being respected (:issue:`17323`)
 
 I/O
 ^^^
@@ -404,6 +430,7 @@ Reshaping
 - Bug in :func:`crosstab` where passing two ``Series`` with the same name raised a ``KeyError`` (:issue:`13279`)
 - :func:`Series.argmin`, :func:`Series.argmax`, and their counterparts on ``DataFrame`` and groupby objects work correctly with floating point data that contains infinite values (:issue:`13595`).
 - Bug in :func:`unique` where checking a tuple of strings raised a ``TypeError`` (:issue:`17108`)
+- Bug in :func:`concat` where order of result index was unpredictable if it contained non-comparable elements (:issue:`17344`)
 
 Numeric
 ^^^^^^^
@@ -422,3 +449,4 @@ Categorical
 Other
 ^^^^^
 - Bug in :func:`eval` where the ``inplace`` parameter was being incorrectly handled (:issue:`16732`)
+- Several ``NaT`` method docstrings (e.g. :func:`NaT.ctime`) were incorrect (:issue:`17327`)

pandas/_libs/src/ujson/python/objToJSON.c

@@ -329,7 +329,7 @@ static Py_ssize_t get_attr_length(PyObject *obj, char *attr) {
     return ret;
 }
 
-npy_int64 get_long_attr(PyObject *o, const char *attr) {
+static npy_int64 get_long_attr(PyObject *o, const char *attr) {
   npy_int64 long_val;
   PyObject *value = PyObject_GetAttrString(o, attr);
   long_val = (PyLong_Check(value) ?
@@ -338,15 +338,12 @@ npy_int64 get_long_attr(PyObject *o, const char *attr) {
   return long_val;
 }
 
-npy_float64 total_seconds(PyObject *td) {
-  // Python 2.6 compat
-  // TODO(anyone): remove this legacy workaround with a more
-  // direct td.total_seconds()
-  npy_int64 microseconds = get_long_attr(td, "microseconds");
-  npy_int64 seconds = get_long_attr(td, "seconds");
-  npy_int64 days = get_long_attr(td, "days");
-  npy_int64 days_in_seconds = days * 24LL * 3600LL;
-  return (microseconds + (seconds + days_in_seconds) * 1000000.0) / 1000000.0;
+static npy_float64 total_seconds(PyObject *td) {
+  npy_float64 double_val;
+  PyObject *value = PyObject_CallMethod(td, "total_seconds", NULL);
+  double_val = PyFloat_AS_DOUBLE(value);
+  Py_DECREF(value);
+  return double_val;
 }
 
 static PyObject *get_item(PyObject *obj, Py_ssize_t i) {

pandas/_libs/tslib.pyx

@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 # cython: profile=False
 
 import warnings
@@ -3922,7 +3923,7 @@ for _method_name in _nat_methods:
         def f(*args, **kwargs):
             return NaT
         f.__name__ = func_name
-        f.__doc__ = _get_docstring(_method_name)
+        f.__doc__ = _get_docstring(func_name)
         return f
 
     setattr(NaTType, _method_name, _make_nat_func(_method_name))
@@ -3934,7 +3935,7 @@ for _method_name in _nan_methods:
         def f(*args, **kwargs):
             return np.nan
         f.__name__ = func_name
-        f.__doc__ = _get_docstring(_method_name)
+        f.__doc__ = _get_docstring(func_name)
         return f
 
     setattr(NaTType, _method_name, _make_nan_func(_method_name))
@@ -3952,7 +3953,7 @@ for _maybe_method_name in dir(NaTType):
             def f(*args, **kwargs):
                 raise ValueError("NaTType does not support " + func_name)
             f.__name__ = func_name
-            f.__doc__ = _get_docstring(_method_name)
+            f.__doc__ = _get_docstring(func_name)
             return f
 
         setattr(NaTType, _maybe_method_name,

pandas/core/accessor.py

@@ -0,0 +1,35 @@
+# -*- coding: utf-8 -*-
+"""
+
+accessor.py contains base classes for implementing accessor properties
+that can be mixed into or pinned onto other pandas classes.
+
+"""
+
+
+class DirNamesMixin(object):
+    _accessors = frozenset([])
+
+    def _dir_deletions(self):
+        """ delete unwanted __dir__ for this object """
+        return self._accessors
+
+    def _dir_additions(self):
+        """ add addtional __dir__ for this object """
+        rv = set()
+        for accessor in self._accessors:
+            try:
+                getattr(self, accessor)
+                rv.add(accessor)
+            except AttributeError:
+                pass
+        return rv
+
+    def __dir__(self):
+        """
+        Provide method name lookup and completion
+        Only provide 'public' methods
+        """
+        rv = set(dir(type(self)))
+        rv = (rv - self._dir_deletions()) | self._dir_additions()
+        return sorted(rv)

pandas/core/base.py

@@ -19,6 +19,7 @@
 from pandas.util._decorators import (Appender, cache_readonly,
                                      deprecate_kwarg, Substitution)
 from pandas.core.common import AbstractMethodError
+from pandas.core.accessor import DirNamesMixin
 
 _shared_docs = dict()
 _indexops_doc_kwargs = dict(klass='IndexOpsMixin', inplace='',
@@ -73,7 +74,7 @@ def __repr__(self):
         return str(self)
 
 
-class PandasObject(StringMixin):
+class PandasObject(StringMixin, DirNamesMixin):
 
     """baseclass for various pandas objects"""
 
@@ -92,23 +93,6 @@ def __unicode__(self):
         # Should be overwritten by base classes
         return object.__repr__(self)
 
-    def _dir_additions(self):
-        """ add addtional __dir__ for this object """
-        return set()
-
-    def _dir_deletions(self):
-        """ delete unwanted __dir__ for this object """
-        return set()
-
-    def __dir__(self):
-        """
-        Provide method name lookup and completion
-        Only provide 'public' methods
-        """
-        rv = set(dir(type(self)))
-        rv = (rv - self._dir_deletions()) | self._dir_additions()
-        return sorted(rv)
-
     def _reset_cache(self, key=None):
         """
         Reset cached properties. If ``key`` is passed, only clears that key.
@@ -141,7 +125,7 @@ class NoNewAttributesMixin(object):
 
     Prevents additional attributes via xxx.attribute = "something" after a
     call to `self.__freeze()`. Mainly used to prevent the user from using
-    wrong attrirbutes on a accessor (`Series.cat/.str/.dt`).
+    wrong attributes on a accessor (`Series.cat/.str/.dt`).
 
     If you really want to add a new attribute at a later time, you need to use
     `object.__setattr__(self, key, value)`.

pandas/core/common.py

@@ -629,3 +629,17 @@ def _random_state(state=None):
     else:
         raise ValueError("random_state must be an integer, a numpy "
                          "RandomState, or None")
+
+
+def _get_distinct_objs(objs):
+    """
+    Return a list with distinct elements of "objs" (different ids).
+    Preserves order.
+    """
+    ids = set()
+    res = []
+    for obj in objs:
+        if not id(obj) in ids:
+            ids.add(id(obj))
+            res.append(obj)
+    return res

pandas/core/frame.py

@@ -67,7 +67,8 @@
                                 _dict_compat,
                                 standardize_mapping)
 from pandas.core.generic import NDFrame, _shared_docs
-from pandas.core.index import Index, MultiIndex, _ensure_index
+from pandas.core.index import (Index, MultiIndex, _ensure_index,
+                               _ensure_index_from_sequences)
 from pandas.core.indexing import (maybe_droplevels, convert_to_index_sliceable,
                                   check_bool_indexer)
 from pandas.core.internals import (BlockManager,
@@ -1155,9 +1156,9 @@ def from_records(cls, data, index=None, exclude=None, columns=None,
             else:
                 try:
                     to_remove = [arr_columns.get_loc(field) for field in index]
-
-                    result_index = MultiIndex.from_arrays(
-                        [arrays[i] for i in to_remove], names=index)
+                    index_data = [arrays[i] for i in to_remove]
+                    result_index = _ensure_index_from_sequences(index_data,
+                                                                names=index)
 
                     exclude.update(index)
                 except Exception:
@@ -3000,7 +3001,7 @@ def set_index(self, keys, drop=True, append=False, inplace=False,
                     to_remove.append(col)
             arrays.append(level)
 
-        index = MultiIndex.from_arrays(arrays, names=names)
+        index = _ensure_index_from_sequences(arrays, names)
 
         if verify_integrity and not index.is_unique:
             duplicates = index.get_duplicates()

pandas/core/generic.py

@@ -192,8 +192,9 @@ def __unicode__(self):
 
     def _dir_additions(self):
         """ add the string-like attributes from the info_axis """
-        return set([c for c in self._info_axis
-                    if isinstance(c, string_types) and isidentifier(c)])
+        additions = set([c for c in self._info_axis
+                         if isinstance(c, string_types) and isidentifier(c)])
+        return super(NDFrame, self)._dir_additions().union(additions)
 
     @property
     def _constructor_sliced(self):