Skip to content

BUG: DataFrame repr with ArrowDtype with extension #54130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v2.1.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -598,6 +598,7 @@ ExtensionArray
- Bug in :meth:`Series.quantile` for pyarrow temporal types raising ArrowInvalid (:issue:`52678`)
- Bug in :meth:`Series.rank` returning wrong order for small values with ``Float64`` dtype (:issue:`52471`)
- Bug in :meth:`~arrays.ArrowExtensionArray.__iter__` and :meth:`~arrays.ArrowExtensionArray.__getitem__` returning python datetime and timedelta objects for non-nano dtypes (:issue:`53326`)
- Bug where the :class:`DataFrame` repr would not work when a column would have an :class:`ArrowDtype` with an ``pyarrow.ExtensionDtype`` (:issue:`54063`)
- Bug where the ``__from_arrow__`` method of masked ExtensionDtypes(e.g. :class:`Float64Dtype`, :class:`BooleanDtype`) would not accept pyarrow arrays of type ``pyarrow.null()`` (:issue:`52223`)

Styler
Expand Down
5 changes: 3 additions & 2 deletions pandas/core/dtypes/dtypes.py
Original file line number Diff line number Diff line change
Expand Up @@ -2102,8 +2102,9 @@ def type(self):
elif pa.types.is_null(pa_type):
# TODO: None? pd.NA? pa.null?
return type(pa_type)
else:
raise NotImplementedError(pa_type)
elif isinstance(pa_type, pa.ExtensionType):
return type(self)(pa_type.storage_type).type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this going to match the storage_type?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the python type of the storage_type

raise NotImplementedError(pa_type)

@property
def name(self) -> str: # type: ignore[override]
Expand Down
15 changes: 15 additions & 0 deletions pandas/tests/extension/test_arrow.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
pa = pytest.importorskip("pyarrow", minversion="7.0.0")

from pandas.core.arrays.arrow.array import ArrowExtensionArray
from pandas.core.arrays.arrow.extension_types import ArrowPeriodType


@pytest.fixture(params=tm.ALL_PYARROW_DTYPES, ids=str)
Expand Down Expand Up @@ -3143,3 +3144,17 @@ def test_to_numpy_temporal(pa_type):
expected = np.array(expected, dtype=object)
assert result[0].unit == expected[0].unit
tm.assert_numpy_array_equal(result, expected)


def test_arrowextensiondtype_dataframe_repr():
# GH 54062
df = pd.DataFrame(
pd.period_range("2012", periods=3),
columns=["col"],
dtype=ArrowDtype(ArrowPeriodType("D")),
)
result = repr(df)
# TODO: repr value may not be expected; address how
# pyarrow.ExtensionType values are displayed
expected = " col\n0 15340\n1 15341\n2 15342"
Copy link
Member

@jbrockmendel jbrockmendel Jul 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where does "15340" come from? i'd expect something like "2012-01-01"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is because ArrowPeriodType stores period data as pa.int64. Since pyarrow.ExtensionType is only used for serialization it appears it doesn't go through period reprs

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im unclear on if this should be considered "wrong"? or just orthogonal to what this PR is doing? seems like this would be an unhelpful repr to give a user

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a orthogonal incorrect behavior. This PR addresses a bug where pyarrow.ExtensionTypes didn't even show a repr

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sense

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a comment that this test shouldn't be interpreted as saying this repr is "correct"?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Added a comment

assert result == expected