Merge branch 'master' into issue_24980

Author: charlesdong1991

.github/CONTRIBUTING.md

@@ -8,16 +8,16 @@ Our main contributing guide can be found [in this repo](https://github.com/panda
 
 If you are looking to contribute to the *pandas* codebase, the best place to start is the [GitHub "issues" tab](https://github.com/pandas-dev/pandas/issues). This is also a great place for filing bug reports and making suggestions for ways in which we can improve the code and documentation.
 
-If you have additional questions, feel free to ask them on the [mailing list](https://groups.google.com/forum/?fromgroups#!forum/pydata) or on [Gitter](https://gitter.im/pydata/pandas). Further information can also be found in the "[Where to start?](https://github.com/pandas-dev/pandas/blob/master/doc/source/contributing.rst#where-to-start)" section.
+If you have additional questions, feel free to ask them on the [mailing list](https://groups.google.com/forum/?fromgroups#!forum/pydata) or on [Gitter](https://gitter.im/pydata/pandas). Further information can also be found in the "[Where to start?](https://github.com/pandas-dev/pandas/blob/master/doc/source/development/contributing.rst#where-to-start)" section.
 
 ## Filing Issues
 
-If you notice a bug in the code or documentation, or have suggestions for how we can improve either, feel free to create an issue on the [GitHub "issues" tab](https://github.com/pandas-dev/pandas/issues) using [GitHub's "issue" form](https://github.com/pandas-dev/pandas/issues/new). The form contains some questions that will help us best address your issue. For more information regarding how to file issues against *pandas*, please refer to the "[Bug reports and enhancement requests](https://github.com/pandas-dev/pandas/blob/master/doc/source/contributing.rst#bug-reports-and-enhancement-requests)" section.
+If you notice a bug in the code or documentation, or have suggestions for how we can improve either, feel free to create an issue on the [GitHub "issues" tab](https://github.com/pandas-dev/pandas/issues) using [GitHub's "issue" form](https://github.com/pandas-dev/pandas/issues/new). The form contains some questions that will help us best address your issue. For more information regarding how to file issues against *pandas*, please refer to the "[Bug reports and enhancement requests](https://github.com/pandas-dev/pandas/blob/master/doc/source/development/contributing.rst#bug-reports-and-enhancement-requests)" section.
 
 ## Contributing to the Codebase
 
-The code is hosted on [GitHub](https://www.github.com/pandas-dev/pandas), so you will need to use [Git](http://git-scm.com/) to clone the project and make changes to the codebase. Once you have obtained a copy of the code, you should create a development environment that is separate from your existing Python environment so that you can make and test changes without compromising your own work environment. For more information, please refer to the "[Working with the code](https://github.com/pandas-dev/pandas/blob/master/doc/source/contributing.rst#working-with-the-code)" section.
+The code is hosted on [GitHub](https://www.github.com/pandas-dev/pandas), so you will need to use [Git](http://git-scm.com/) to clone the project and make changes to the codebase. Once you have obtained a copy of the code, you should create a development environment that is separate from your existing Python environment so that you can make and test changes without compromising your own work environment. For more information, please refer to the "[Working with the code](https://github.com/pandas-dev/pandas/blob/master/doc/source/development/contributing.rst#working-with-the-code)" section.
 
-Before submitting your changes for review, make sure to check that your changes do not break any tests. You can find more information about our test suites in the "[Test-driven development/code writing](https://github.com/pandas-dev/pandas/blob/master/doc/source/contributing.rst#test-driven-development-code-writing)" section. We also have guidelines regarding coding style that will be enforced during testing, which can be found in the "[Code standards](https://github.com/pandas-dev/pandas/blob/master/doc/source/contributing.rst#code-standards)" section.
+Before submitting your changes for review, make sure to check that your changes do not break any tests. You can find more information about our test suites in the "[Test-driven development/code writing](https://github.com/pandas-dev/pandas/blob/master/doc/source/contributing.rst#test-driven-development-code-writing)" section. We also have guidelines regarding coding style that will be enforced during testing, which can be found in the "[Code standards](https://github.com/pandas-dev/pandas/blob/master/doc/source/development/contributing.rst#code-standards)" section.
 
-Once your changes are ready to be submitted, make sure to push your changes to GitHub before creating a pull request. Details about how to do that can be found in the "[Contributing your changes to pandas](https://github.com/pandas-dev/pandas/blob/master/doc/source/contributing.rst#contributing-your-changes-to-pandas)" section. We will review your changes, and you will most likely be asked to make additional changes before it is finally ready to merge. However, once it's ready, we will merge it, and you will have successfully contributed to the codebase!
+Once your changes are ready to be submitted, make sure to push your changes to GitHub before creating a pull request. Details about how to do that can be found in the "[Contributing your changes to pandas](https://github.com/pandas-dev/pandas/blob/master/doc/source/development/contributing.rst#contributing-your-changes-to-pandas)" section. We will review your changes, and you will most likely be asked to make additional changes before it is finally ready to merge. However, once it's ready, we will merge it, and you will have successfully contributed to the codebase!

asv_bench/benchmarks/algorithms.py

@@ -5,7 +5,6 @@
 import pandas as pd
 from pandas.util import testing as tm
 
-
 for imp in ['pandas.util', 'pandas.tools.hashing']:
     try:
         hashing = import_module(imp)
@@ -142,4 +141,4 @@ def time_quantile(self, quantile, interpolation, dtype):
         self.idx.quantile(quantile, interpolation=interpolation)
 
 
-from .pandas_vb_common import setup  # noqa: F401
+from .pandas_vb_common import setup  # noqa: F401 isort:skip

asv_bench/benchmarks/strings.py

@@ -102,10 +102,10 @@ def setup(self, repeats):
         N = 10**5
         self.s = Series(tm.makeStringIndex(N))
         repeat = {'int': 1, 'array': np.random.randint(1, 3, N)}
-        self.repeat = repeat[repeats]
+        self.values = repeat[repeats]
 
     def time_repeat(self, repeats):
-        self.s.str.repeat(self.repeat)
+        self.s.str.repeat(self.values)
 
 
 class Cat(object):

doc/source/whatsnew/v0.24.1.rst

@@ -23,6 +23,7 @@ Fixed Regressions
 - Bug in :meth:`DataFrame.itertuples` with ``records`` orient raising an ``AttributeError`` when the ``DataFrame`` contained more than 255 columns (:issue:`24939`)
 - Bug in :meth:`DataFrame.itertuples` orient converting integer column names to strings prepended with an underscore (:issue:`24940`)
 - Bug in :meth:`to_string` with ``index`` set to ``False``, leading space is added. (:issue:`24980`)
+- Fixed regression in :class:`Index.intersection` incorrectly sorting the values by default (:issue:`24959`).
 
 .. _whatsnew_0241.enhancements:
 
@@ -73,8 +74,7 @@ Bug Fixes
 
 **Reshaping**
 
--
--
+- Bug in :meth:`DataFrame.groupby` with :class:`Grouper` when there is a time change (DST) and grouping frequency is ``'1d'`` (:issue:`24972`)
 
 **Visualization**
 

pandas/core/indexes/base.py

@@ -2333,7 +2333,7 @@ def union(self, other, sort=True):
     def _wrap_setop_result(self, other, result):
         return self._constructor(result, name=get_op_result_name(self, other))
 
-    def intersection(self, other, sort=True):
+    def intersection(self, other, sort=False):
         """
         Form the intersection of two Index objects.
 
@@ -2342,11 +2342,15 @@ def intersection(self, other, sort=True):
         Parameters
         ----------
         other : Index or array-like
-        sort : bool, default True
+        sort : bool, default False
             Sort the resulting index if possible
 
             .. versionadded:: 0.24.0
 
+            .. versionchanged:: 0.24.1
+
+               Changed the default from ``True`` to ``False``.
+
         Returns
         -------
         intersection : Index

pandas/core/indexes/datetimes.py

@@ -594,14 +594,22 @@ def _wrap_setop_result(self, other, result):
         name = get_op_result_name(self, other)
         return self._shallow_copy(result, name=name, freq=None, tz=self.tz)
 
-    def intersection(self, other, sort=True):
+    def intersection(self, other, sort=False):
         """
         Specialized intersection for DatetimeIndex objects. May be much faster
         than Index.intersection
 
         Parameters
         ----------
         other : DatetimeIndex or array-like
+        sort : bool, default True
+            Sort the resulting index if possible.
+
+            .. versionadded:: 0.24.0
+
+            .. versionchanged:: 0.24.1
+
+               Changed the default from ``True`` to ``False``.
 
         Returns
         -------

pandas/core/indexes/interval.py

@@ -1093,8 +1093,8 @@ def equals(self, other):
     def overlaps(self, other):
         return self._data.overlaps(other)
 
-    def _setop(op_name):
-        def func(self, other, sort=True):
+    def _setop(op_name, sort=True):
+        def func(self, other, sort=sort):
             other = self._as_like_interval_index(other)
 
             # GH 19016: ensure set op will not return a prohibited dtype
@@ -1128,7 +1128,7 @@ def is_all_dates(self):
         return False
 
     union = _setop('union')
-    intersection = _setop('intersection')
+    intersection = _setop('intersection', sort=False)
     difference = _setop('difference')
     symmetric_difference = _setop('symmetric_difference')
 

pandas/core/indexes/multi.py

@@ -2910,7 +2910,7 @@ def union(self, other, sort=True):
         return MultiIndex.from_arrays(lzip(*uniq_tuples), sortorder=0,
                                       names=result_names)
 
-    def intersection(self, other, sort=True):
+    def intersection(self, other, sort=False):
         """
         Form the intersection of two MultiIndex objects.
 
@@ -2922,6 +2922,10 @@ def intersection(self, other, sort=True):
 
             .. versionadded:: 0.24.0
 
+            .. versionchanged:: 0.24.1
+
+               Changed the default from ``True`` to ``False``.
+
         Returns
         -------
         Index

pandas/core/indexes/range.py

@@ -343,7 +343,7 @@ def equals(self, other):
 
         return super(RangeIndex, self).equals(other)
 
-    def intersection(self, other, sort=True):
+    def intersection(self, other, sort=False):
         """
         Form the intersection of two Index objects.
 
@@ -355,6 +355,10 @@ def intersection(self, other, sort=True):
 
             .. versionadded:: 0.24.0
 
+            .. versionchanged:: 0.24.1
+
+               Changed the default from ``True`` to ``False``.
+
         Returns
         -------
         intersection : Index

pandas/core/resample.py

@@ -30,8 +30,7 @@
 from pandas.core.indexes.timedeltas import TimedeltaIndex, timedelta_range
 
 from pandas.tseries.frequencies import to_offset
-from pandas.tseries.offsets import (
-    DateOffset, Day, Nano, Tick, delta_to_nanoseconds)
+from pandas.tseries.offsets import DateOffset, Day, Nano, Tick
 
 _shared_docs_kwargs = dict()
 
@@ -1613,20 +1612,20 @@ def _get_timestamp_range_edges(first, last, offset, closed='left', base=0):
     A tuple of length 2, containing the adjusted pd.Timestamp objects.
     """
     if isinstance(offset, Tick):
-        is_day = isinstance(offset, Day)
-        day_nanos = delta_to_nanoseconds(timedelta(1))
-
-        # #1165 and #24127
-        if (is_day and not offset.nanos % day_nanos) or not is_day:
-            first, last = _adjust_dates_anchored(first, last, offset,
-                                                 closed=closed, base=base)
-            if is_day and first.tz is not None:
-                # _adjust_dates_anchored assumes 'D' means 24H, but first/last
-                # might contain a DST transition (23H, 24H, or 25H).
-                # Ensure first/last snap to midnight.
-                first = first.normalize()
-                last = last.normalize()
-            return first, last
+        if isinstance(offset, Day):
+            # _adjust_dates_anchored assumes 'D' means 24H, but first/last
+            # might contain a DST transition (23H, 24H, or 25H).
+            # So "pretend" the dates are naive when adjusting the endpoints
+            tz = first.tz
+            first = first.tz_localize(None)
+            last = last.tz_localize(None)
+
+        first, last = _adjust_dates_anchored(first, last, offset,
+                                             closed=closed, base=base)
+        if isinstance(offset, Day):
+            first = first.tz_localize(tz)
+            last = last.tz_localize(tz)
+        return first, last
 
     else:
         first = first.normalize()

pandas/tests/indexes/test_base.py

@@ -765,6 +765,11 @@ def test_intersect_str_dates(self, sort):
 
         assert len(result) == 0
 
+    def test_intersect_nosort(self):
+        result = pd.Index(['c', 'b', 'a']).intersection(['b', 'a'])
+        expected = pd.Index(['b', 'a'])
+        tm.assert_index_equal(result, expected)
+
     @pytest.mark.parametrize("sort", [True, False])
     def test_chained_union(self, sort):
         # Chained unions handles names correctly
@@ -1595,20 +1600,27 @@ def test_drop_tuple(self, values, to_drop):
         for drop_me in to_drop[1], [to_drop[1]]:
             pytest.raises(KeyError, removed.drop, drop_me)
 
-    @pytest.mark.parametrize("method,expected", [
+    @pytest.mark.parametrize("method,expected,sort", [
+        ('intersection', np.array([(1, 'A'), (2, 'A'), (1, 'B'), (2, 'B')],
+                                  dtype=[('num', int), ('let', 'a1')]),
+         False),
+
         ('intersection', np.array([(1, 'A'), (1, 'B'), (2, 'A'), (2, 'B')],
-                                  dtype=[('num', int), ('let', 'a1')])),
+                                  dtype=[('num', int), ('let', 'a1')]),
+         True),
+
         ('union', np.array([(1, 'A'), (1, 'B'), (1, 'C'), (2, 'A'), (2, 'B'),
-                            (2, 'C')], dtype=[('num', int), ('let', 'a1')]))
+                            (2, 'C')], dtype=[('num', int), ('let', 'a1')]),
+         True)
     ])
-    def test_tuple_union_bug(self, method, expected):
+    def test_tuple_union_bug(self, method, expected, sort):
         index1 = Index(np.array([(1, 'A'), (2, 'A'), (1, 'B'), (2, 'B')],
                                 dtype=[('num', int), ('let', 'a1')]))
         index2 = Index(np.array([(1, 'A'), (2, 'A'), (1, 'B'),
                                  (2, 'B'), (1, 'C'), (2, 'C')],
                                 dtype=[('num', int), ('let', 'a1')]))
 
-        result = getattr(index1, method)(index2)
+        result = getattr(index1, method)(index2, sort=sort)
         assert result.ndim == 1
 
         expected = Index(expected)

pandas/tests/resample/test_datetime_index.py

@@ -1278,6 +1278,21 @@ def test_resample_across_dst():
     assert_frame_equal(result, expected)
 
 
+def test_groupby_with_dst_time_change():
+    # GH 24972
+    index = pd.DatetimeIndex([1478064900001000000, 1480037118776792000],
+                             tz='UTC').tz_convert('America/Chicago')
+
+    df = pd.DataFrame([1, 2], index=index)
+    result = df.groupby(pd.Grouper(freq='1d')).last()
+    expected_index_values = pd.date_range('2016-11-02', '2016-11-24',
+                                          freq='d', tz='America/Chicago')
+
+    index = pd.DatetimeIndex(expected_index_values)
+    expected = pd.DataFrame([1.0] + ([np.nan] * 21) + [2.0], index=index)
+    assert_frame_equal(result, expected)
+
+
 def test_resample_dst_anchor():
     # 5172
     dti = DatetimeIndex([datetime(2012, 11, 4, 23)], tz='US/Eastern')

setup.cfg

@@ -114,7 +114,6 @@ force_sort_within_sections=True
 skip=
     pandas/core/api.py,
     pandas/core/frame.py,
-    asv_bench/benchmarks/algorithms.py,
     asv_bench/benchmarks/attrs_caching.py,
     asv_bench/benchmarks/binary_ops.py,
     asv_bench/benchmarks/categoricals.py,