ENH: Expand index types in Series.struct.field method

[TomAugspurger]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

As mentioned in #54977 (comment), pc.struct_field takes more types than we allow in Series.struct.field: https://arrow.apache.org/docs/python/generated/pyarrow.compute.struct_field.html

Feature Description

Allow more types in Series.struct.field, including lists to supported nested selection.

        >>> import pyarrow as pa; import pandas as pd
        >>> version_type = pa.struct([
        ...     ("major", pa.int64()),
        ...     ("minor", pa.int64()),
        ... ])
        >>> s = pd.Series(
        ...     [
        ...         {"version": {"major": 1, "minor": 5}, "project": "pandas"},
        ...         {"version": {"major": 2, "minor": 1}, "project": "pandas"},
        ...         {"version": {"major": 1, "minor": 26}, "project": "numpy"},
        ...     ],
        ...     dtype=pd.ArrowDtype(pa.struct(
        ...         [("version", version_type), ("project", pa.string())]
        ...     ))
        ... )
        >>> s.struct.field(["version", "minor"])
        0     5
        1     1
        2    26
        Name: minor, dtype: int64[pyarrow]

Alternative Solutions

Repeated s.struct.field(0).struct.field(1) gives the same answer, but is slower since each intermediate object needs to be created.

Otherwise, not possible without using pyarrow directly.

Additional Context

I've started on a branch implementing this.

The one bit that needs discussion is how to handle the name when doing nested selection. For now, I think we should just return the name of the last field.

You might consider something like ".".join([names]) where names is the list of intermediate fields, so my earlier example would have a name like version.minor to indicate that you sliced multiple levels. However, it'd be hard to use that reliably since any intermediate name might have a . in it, so you can be sure that result.name.split(".") gives the right fields from the original array.

[TomAugspurger]

cc @tswast

[TomAugspurger]

pc.struct_field also allows for pyarrow.compute.Expression objects. I'm not too sure what's allowed here, but for now I'd suggest just setting the output name to str(expression). That does the expected thing for pc.field("field_name"), setting the name to "field_name".

[tswast]

Thanks!

You might consider something like ".".join([names]) where names is the list of intermediate fields, so my earlier example would have a name like version.minor to indicate that you sliced multiple levels.

This is the method I was pursuing but it became a little more complex than I wanted to get into at the time. You make a good point that you can't actually recover the original names anyway since intermediate steps might actually contain . in the name itself.