Merge remote-tracking branch 'upstream/master' into timestamp

Author: saurav-chakravorty

Makefile

@@ -23,4 +23,3 @@ doc:
 	cd doc; \
 	python make.py clean; \
 	python make.py html
-	python make.py spellcheck

ci/code_checks.sh

@@ -206,7 +206,7 @@ if [[ -z "$CHECK" || "$CHECK" == "doctests" ]]; then
 
     MSG='Doctests frame.py' ; echo $MSG
     pytest -q --doctest-modules pandas/core/frame.py \
-        -k"-axes -combine -itertuples -join -pivot_table -query -reindex -reindex_axis -round"
+        -k" -itertuples -join -reindex -reindex_axis -round"
     RET=$(($RET + $?)) ; echo $MSG "DONE"
 
     MSG='Doctests series.py' ; echo $MSG

doc/source/whatsnew/v0.24.1.rst

@@ -13,65 +13,56 @@ Whats New in 0.24.1 (February XX, 2019)
 {{ header }}
 
 These are the changes in pandas 0.24.1. See :ref:`release` for a full changelog
-including other versions of pandas.
+including other versions of pandas. See :ref:`whatsnew_0240` for the 0.24.0 changelog.
 
-.. _whatsnew_0241.regressions:
-
-Fixed Regressions
-^^^^^^^^^^^^^^^^^
+.. _whatsnew_0241.api:
 
-- Bug in :meth:`DataFrame.itertuples` with ``records`` orient raising an ``AttributeError`` when the ``DataFrame`` contained more than 255 columns (:issue:`24939`)
-- Bug in :meth:`DataFrame.itertuples` orient converting integer column names to strings prepended with an underscore (:issue:`24940`)
-- Fixed regression in :func:`read_sql` when passing certain queries with MySQL/pymysql (:issue:`24988`).
-- Fixed regression in :class:`Index.intersection` incorrectly sorting the values by default (:issue:`24959`).
-- Fixed regression in :func:`merge` when merging an empty ``DataFrame`` with multiple timezone-aware columns on one of the timezone-aware columns (:issue:`25014`).
+API Changes
+~~~~~~~~~~~
 
-.. _whatsnew_0241.enhancements:
+Changing the ``sort`` parameter for :class:`Index` set operations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Enhancements
-^^^^^^^^^^^^
+The default ``sort`` value for :meth:`Index.union` has changed from ``True`` to ``None`` (:issue:`24959`).
+The default *behavior*, however, remains the same: the result is sorted, unless
 
+1. ``self`` and ``other`` are identical
+2. ``self`` or ``other`` is empty
+3. ``self`` or ``other`` contain values that can not be compared (a ``RuntimeWarning`` is raised).
 
-.. _whatsnew_0241.bug_fixes:
-
-Bug Fixes
-~~~~~~~~~
-- Bug in :meth: `Timestamp` supporting %z (:issue:`21257`). 
-**Conversion**
-
--
--
--
+This change will allow ``sort=True`` to mean "always sort" in a future release.
 
-**Indexing**
+The same change applies to :meth:`Index.difference` and :meth:`Index.symmetric_difference`, which
+would not sort the result when the values could not be compared.
 
--
--
--
+The `sort` option for :meth:`Index.intersection` has changed in three ways.
 
-**I/O**
+1. The default has changed from ``True`` to ``False``, to restore the
+   pandas 0.23.4 and earlier behavior of not sorting by default.
+2. The behavior of ``sort=True`` can now be obtained with ``sort=None``.
+   This will sort the result only if the values in ``self`` and ``other``
+   are not identical.
+3. The value ``sort=True`` is no longer allowed. A future version of pandas
+   will properly support ``sort=True`` meaning "always sort".
 
--
--
--
-
-**Categorical**
+.. _whatsnew_0241.regressions:
 
--
--
--
+Fixed Regressions
+~~~~~~~~~~~~~~~~~
 
-**Timezones**
+- Fixed regression in :meth:`DataFrame.to_dict` with ``records`` orient raising an
+  ``AttributeError`` when the ``DataFrame`` contained more than 255 columns, or
+  wrongly converting column names that were not valid python identifiers (:issue:`24939`, :issue:`24940`).
+- Fixed regression in :func:`read_sql` when passing certain queries with MySQL/pymysql (:issue:`24988`).
+- Fixed regression in :class:`Index.intersection` incorrectly sorting the values by default (:issue:`24959`).
+- Fixed regression in :func:`merge` when merging an empty ``DataFrame`` with multiple timezone-aware columns on one of the timezone-aware columns (:issue:`25014`).
+- Fixed regression in :meth:`Series.rename_axis` and :meth:`DataFrame.rename_axis` where passing ``None`` failed to remove the axis name (:issue:`25034`)
+- Fixed regression in :func:`to_timedelta` with `box=False` incorrectly returning a ``datetime64`` object instead of a ``timedelta64`` object (:issue:`24961`)
 
--
--
--
+.. _whatsnew_0241.bug_fixes:
 
-**Timedelta**
-- Bug in :func:`to_timedelta` with `box=False` incorrectly returning a ``datetime64`` object instead of a ``timedelta64`` object (:issue:`24961`)
--
--
--
+Bug Fixes
+~~~~~~~~~
 
 **Reshaping**
 
@@ -81,11 +72,9 @@ Bug Fixes
 
 - Fixed the warning for implicitly registered matplotlib converters not showing. See :ref:`whatsnew_0211.converters` for more (:issue:`24963`).
 
-
 **Other**
 
 - Fixed AttributeError when printing a DataFrame's HTML repr after accessing the IPython config object (:issue:`25036`)
--
 
 .. _whatsnew_0.241.contributors:
 

doc/source/whatsnew/v0.25.0.rst

@@ -19,17 +19,9 @@ including other versions of pandas.
 Other Enhancements
 ^^^^^^^^^^^^^^^^^^
 
+- :meth:`Timestamp.replace` now supports the ``fold`` argument to disambiguate DST transition times (:issue:`25017`)
 -
 -
--
-
-.. _whatsnew_0250.performance:
-
-Performance Improvements
-~~~~~~~~~~~~~~~~~~~~~~~~
- - Significant speedup in `SparseArray` initialization that benefits most operations, fixing performance regression introduced in v0.20.0 (:issue:`24985`)
-
-
 
 .. _whatsnew_0250.api_breaking:
 
@@ -69,8 +61,8 @@ Removal of prior version deprecations/changes
 Performance Improvements
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
--
--
+- Significant speedup in `SparseArray` initialization that benefits most operations, fixing performance regression introduced in v0.20.0 (:issue:`24985`)
+- `DataFrame.to_stata()` is now faster when outputting data with any string or non-native endian columns (:issue:`25045`)
 -
 
 
@@ -165,6 +157,7 @@ MultiIndex
 I/O
 ^^^
 
+- Fixed bug in missing text when using :meth:`to_clipboard` if copying utf-16 characters in Python 3 on Windows (:issue:`25040`)
 -
 -
 -

pandas/_libs/lib.pyx

@@ -233,10 +233,11 @@ def fast_unique_multiple(list arrays, sort: bool=True):
             if val not in table:
                 table[val] = stub
                 uniques.append(val)
-    if sort:
+    if sort is None:
         try:
             uniques.sort()
         except Exception:
+            # TODO: RuntimeWarning?
             pass
 
     return uniques

pandas/_libs/tslibs/nattype.pyx

@@ -669,7 +669,6 @@ class NaTType(_NaT):
         nanosecond : int, optional
         tzinfo : tz-convertible, optional
         fold : int, optional, default is 0
-            added in 3.6, NotImplemented
 
         Returns
         -------

pandas/_libs/tslibs/offsets.pyx

@@ -18,6 +18,7 @@ from numpy cimport int64_t
 cnp.import_array()
 
 
+from pandas._libs.tslibs cimport util
 from pandas._libs.tslibs.util cimport is_string_object, is_integer_object
 
 from pandas._libs.tslibs.ccalendar import MONTHS, DAYS
@@ -408,6 +409,10 @@ class _BaseOffset(object):
         return self.apply(other)
 
     def __mul__(self, other):
+        if hasattr(other, "_typ"):
+            return NotImplemented
+        if util.is_array(other):
+            return np.array([self * x for x in other])
         return type(self)(n=other * self.n, normalize=self.normalize,
                           **self.kwds)
 
@@ -458,6 +463,9 @@ class _BaseOffset(object):
         TypeError if `int(n)` raises
         ValueError if n != int(n)
         """
+        if util.is_timedelta64_object(n):
+            raise TypeError('`n` argument must be an integer, '
+                            'got {ntype}'.format(ntype=type(n)))
         try:
             nint = int(n)
         except (ValueError, TypeError):
@@ -533,12 +541,20 @@ class _Tick(object):
     can do isinstance checks on _Tick and avoid importing tseries.offsets
     """
 
+    # ensure that reversed-ops with numpy scalars return NotImplemented
+    __array_priority__ = 1000
+
     def __truediv__(self, other):
         result = self.delta.__truediv__(other)
         return _wrap_timedelta_result(result)
 
+    def __rtruediv__(self, other):
+        result = self.delta.__rtruediv__(other)
+        return _wrap_timedelta_result(result)
+
     if PY2:
         __div__ = __truediv__
+        __rdiv__ = __rtruediv__
 
 
 # ----------------------------------------------------------------------

pandas/_libs/tslibs/timestamps.pyx

@@ -1,4 +1,5 @@
 # -*- coding: utf-8 -*-
+import sys
 import warnings
 
 from cpython cimport (PyObject_RichCompareBool, PyObject_RichCompare,
@@ -44,10 +45,11 @@ from pandas._libs.tslibs.timezones import UTC
 # Constants
 _zero_time = datetime_time(0, 0)
 _no_input = object()
-
+PY36 = sys.version_info >= (3, 6)
 
 # ----------------------------------------------------------------------
 
+
 def maybe_integer_op_deprecated(obj):
     # GH#22535 add/sub of integers and int-arrays is deprecated
     if obj.freq is not None:
@@ -1203,7 +1205,6 @@ class Timestamp(_Timestamp):
         nanosecond : int, optional
         tzinfo : tz-convertible, optional
         fold : int, optional, default is 0
-            added in 3.6, NotImplemented
 
         Returns
         -------
@@ -1260,12 +1261,16 @@ class Timestamp(_Timestamp):
             # see GH#18319
             ts_input = _tzinfo.localize(datetime(dts.year, dts.month, dts.day,
                                                  dts.hour, dts.min, dts.sec,
-                                                 dts.us))
+                                                 dts.us),
+                                        is_dst=not bool(fold))
             _tzinfo = ts_input.tzinfo
         else:
-            ts_input = datetime(dts.year, dts.month, dts.day,
-                                dts.hour, dts.min, dts.sec, dts.us,
-                                tzinfo=_tzinfo)
+            kwargs = {'year': dts.year, 'month': dts.month, 'day': dts.day,
+                      'hour': dts.hour, 'minute': dts.min, 'second': dts.sec,
+                      'microsecond': dts.us, 'tzinfo': _tzinfo}
+            if PY36:
+                kwargs['fold'] = fold
+            ts_input = datetime(**kwargs)
 
         ts = convert_datetime_to_tsobject(ts_input, _tzinfo)
         value = ts.value + (dts.ps // 1000)

pandas/compat/pickle_compat.py

@@ -201,7 +201,7 @@ def load_newobj_ex(self):
     pass
 
 
-def load(fh, encoding=None, compat=False, is_verbose=False):
+def load(fh, encoding=None, is_verbose=False):
     """load a pickle, with a provided encoding
 
     if compat is True:
@@ -212,7 +212,6 @@ def load(fh, encoding=None, compat=False, is_verbose=False):
     ----------
     fh : a filelike object
     encoding : an optional encoding
-    compat : provide Series compatibility mode, boolean, default False
     is_verbose : show exception output
     """
 

pandas/core/accessor.py

@@ -16,11 +16,15 @@ class DirNamesMixin(object):
         ['asobject', 'base', 'data', 'flags', 'itemsize', 'strides'])
 
     def _dir_deletions(self):
-        """ delete unwanted __dir__ for this object """
+        """
+        Delete unwanted __dir__ for this object.
+        """
         return self._accessors | self._deprecations
 
     def _dir_additions(self):
-        """ add additional __dir__ for this object """
+        """
+        Add additional __dir__ for this object.
+        """
         rv = set()
         for accessor in self._accessors:
             try:
@@ -33,7 +37,7 @@ def _dir_additions(self):
     def __dir__(self):
         """
         Provide method name lookup and completion
-        Only provide 'public' methods
+        Only provide 'public' methods.
         """
         rv = set(dir(type(self)))
         rv = (rv - self._dir_deletions()) | self._dir_additions()
@@ -42,7 +46,7 @@ def __dir__(self):
 
 class PandasDelegate(object):
     """
-    an abstract base class for delegating methods/properties
+    An abstract base class for delegating methods/properties.
     """
 
     def _delegate_property_get(self, name, *args, **kwargs):
@@ -65,10 +69,10 @@ def _add_delegate_accessors(cls, delegate, accessors, typ,
         ----------
         cls : the class to add the methods/properties to
         delegate : the class to get methods/properties & doc-strings
-        acccessors : string list of accessors to add
+        accessors : string list of accessors to add
         typ : 'property' or 'method'
         overwrite : boolean, default False
-           overwrite the method/property in the target class if it exists
+           overwrite the method/property in the target class if it exists.
         """
 
         def _create_delegator_property(name):
@@ -117,7 +121,7 @@ def delegate_names(delegate, accessors, typ, overwrite=False):
     ----------
     delegate : object
         the class to get methods/properties & doc-strings
-    acccessors : Sequence[str]
+    accessors : Sequence[str]
         List of accessor to add
     typ : {'property', 'method'}
     overwrite : boolean, default False

pandas/core/arrays/categorical.py

@@ -2321,8 +2321,7 @@ def _values_for_factorize(self):
     @classmethod
     def _from_factorized(cls, uniques, original):
         return original._constructor(original.categories.take(uniques),
-                                     categories=original.categories,
-                                     ordered=original.ordered)
+                                     dtype=original.dtype)
 
     def equals(self, other):
         """
@@ -2674,9 +2673,7 @@ def _factorize_from_iterable(values):
     if is_categorical(values):
         if isinstance(values, (ABCCategoricalIndex, ABCSeries)):
             values = values._values
-        categories = CategoricalIndex(values.categories,
-                                      categories=values.categories,
-                                      ordered=values.ordered)
+        categories = CategoricalIndex(values.categories, dtype=values.dtype)
         codes = values.codes
     else:
         # The value of ordered is irrelevant since we don't use cat as such,

pandas/core/base.py

@@ -1234,7 +1234,7 @@ def value_counts(self, normalize=False, sort=True, ascending=False,
             If True then the object returned will contain the relative
             frequencies of the unique values.
         sort : boolean, default True
-            Sort by values.
+            Sort by frequencies.
         ascending : boolean, default False
             Sort in ascending order.
         bins : integer, optional