Skip to content

BUG: DataFrame.diff(axis=1) with mixed (or EA) dtypes #32995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Apr 10, 2020
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.1.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,7 @@ Numeric
- Bug in :meth:`to_numeric` with string argument ``"uint64"`` and ``errors="coerce"`` silently fails (:issue:`32394`)
- Bug in :meth:`to_numeric` with ``downcast="unsigned"`` fails for empty data (:issue:`32493`)
- Bug in :meth:`DataFrame.mean` with ``numeric_only=False`` and either ``datetime64`` dtype or ``PeriodDtype`` column incorrectly raising ``TypeError`` (:issue:`32426`)
- Bug in :meth:`DataFrame.diff` with ``axis=1`` returning incorrect results with mixed dtypes; this will now raise ``NotImplementedError`` (:issue:`32995`)
-

Conversion
Expand Down
7 changes: 7 additions & 0 deletions pandas/core/internals/blocks.py
Original file line number Diff line number Diff line change
Expand Up @@ -1840,7 +1840,14 @@ def interpolate(
)

def diff(self, n: int, axis: int = 1) -> List["Block"]:
if axis == 0 and n != 0:
# n==0 case will be a no-op so let is fall through
# Since we only have one column, the result will be all-NA.
# Create this result by shifting along axis=0 past the length of
# our values.
return super().diff(len(self.values), axis=0)
if axis == 1:
# TODO(EA2D): unnecessary with 2D EAs
# we are by definition 1D.
axis = 0
return super().diff(n, axis)
Expand Down
7 changes: 7 additions & 0 deletions pandas/core/internals/managers.py
Original file line number Diff line number Diff line change
Expand Up @@ -579,6 +579,13 @@ def putmask(
)

def diff(self, n: int, axis: int) -> "BlockManager":
self._consolidate_inplace()
if axis == 0 and self.nblocks > 1:
# This check needs to occur after the consolidation above
raise NotImplementedError(
"Cannot perform DataFrame.diff with axis=1 with mixed dtypes. "
"Try `frame.T.diff(periods, axis=0).T` instead."
)
return self.apply("diff", n=n, axis=axis)

def interpolate(self, **kwargs) -> "BlockManager":
Expand Down
21 changes: 21 additions & 0 deletions pandas/tests/frame/methods/test_diff.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,24 @@ def test_diff_axis(self):
tm.assert_frame_equal(
df.diff(axis=0), DataFrame([[np.nan, np.nan], [2.0, 2.0]])
)

def test_diff_period(self):
# GH#32995 Don't pass an incorrect axis
# TODO(EA2D): this bug wouldn't have happened with 2D EA
pi = pd.date_range("2016-01-01", periods=3).to_period("D")
df = pd.DataFrame({"A": pi})

result = df.diff(1, axis=1)

# TODO: should we make Block.diff do type inference? or maybe algos.diff?
expected = (df - pd.NaT).astype(object)
tm.assert_frame_equal(result, expected)

def test_diff_axis1_mixed_dtypes(self):
# GH#32995 We cannot diff across Blocks with axis=1

df = pd.DataFrame({"A": range(3), "B": np.arange(3, dtype=np.float64)})

msg = "Cannot perform DataFrame.diff with axis=1 with mixed dtypes"
with pytest.raises(NotImplementedError, match=msg):
df.diff(axis=1)