Refactor format_array in pandas\io\formats\format.py

[simonjayhawkins]

see #26833 (comment)

to make format_array more generic we need to push more functionality to the objects to be formatted themselves (and PandasArray) rather than encoding the conversions and special casing in format_array.

[simonjayhawkins]

as per #26833 (comment)

IIUC correctly, before EAs, format_array formatted an numpy array. and GenericArrayFormatter expects an numpy array, used mainly for numpy arrays with dtype=object

but we now have a ExtensionArrayFormatter that includes logic to convert to a numpy array

        values = self.values
        if isinstance(values, (ABCIndexClass, ABCSeries)):
            values = values._values

        formatter = values._formatter(boxed=True)

        if is_categorical_dtype(values.dtype):
            # Categorical is special for now, so that we can preserve tzinfo
            array = values.get_values()
        else:
            array = np.asarray(values)

        fmt_values = format_array(array,

whereas before, _formatting_values was used and conversion to a numpy array looked like

    def _format_col(self, i):
        frame = self.tr_frame
        formatter = self._get_formatter(i)
        values_to_format = frame.iloc[:, i]._formatting_values()
        return format_array(values_to_format, formatter,
                            float_format=self.float_format, na_rep=self.na_rep,
                            space=self.col_space, decimal=self.decimal)

even though _formatting_values is depecated:

    def _formatting_values(self) -> np.ndarray:
        # At the moment, this has to be an array since we use result.dtype
        """
        An array of values to be printed in, e.g. the Series repr

        .. deprecated:: 0.24.0

           Use :meth:`ExtensionArray._formatter` instead.
        """

it is still used. but does not necessarily return an numpy array

>>> import pandas as pd
>>> df = pd.DataFrame({"c": [2,3,float('nan')]})
>>> df = df.astype("Int64")
>>> df.c._formatting_values()
<IntegerArray>
[2, 3, NaN]
Length: 3, dtype: Int64

so when creating a repr_html of a DataFrame which uses _format_col, we pass a <IntegerArray> to format_array, which then uses ExtensionArrayFormatter to get the values as an numpy array and then call
format_array again which then formats the numpy array using GenericArrayFormatter.

It seems a bit convoluted.

i'm not sure why a numpy array is not returned from _formatting_values.

I think the issues are related since the discussions revolve around numpy arrays from extension arrays and formatting probably works best if the formatting is done on a numpy array with am object dtype. performance is probably not an issue for formatting.

[jorisvandenbossche]

whereas before, _formatting_values was used and conversion to a numpy array looked like
even though _formatting_values is depecated:
it is still used. but does not necessarily return an numpy array

I think you are confusing two different _formatting_values: the one defined on Series, and the one defined on an EA.
It is the one on EA that is deprecated, in favor of the EA._formatter.

And it is the Series._formatting_values that can potentially return an EA, as that is what ExtensionBlock.formatting_values decides to return (this is where the EA._formatting_values deprecation is handled)

so when creating a repr_html of a DataFrame which uses _format_col, we pass a to format_array, which then uses ExtensionArrayFormatter to get the values as an numpy array and then call
format_array again which then formats the numpy array using GenericArrayFormatter.

It's a bit of indirection, but that is how the EAs are currently displayed: the values itself are converted to a numpy array (the __array__ that the EA defines) and the display is determined by the EA._formatter. And then it is using our format_array machinery to format those values with the custom formatter.

[jorisvandenbossche]

I think the general discussion items are:

• Who should be responsible of the actual formatting?

  • Currently, the EA has partial responsibility: it gives you an array to format and a custom formatter, but is not responsible for the full formatting pipeline, it is still format_array that combines both elements (eg it does not return an array of strings to be composed in a repr). We could also decide to give it full control over the pipeline.
    For the rest, the formatting is centralized in format.py.
  • Do we want our internal EAs to follow the EA interface for formatting?
    Currently, that is not yet possible I think. Eg the datetime formatting functionality takes the resolution of the full array into account. So it is not possible to format this scalar by scalar (such as the current EA interface does)

• How do we handle all the formatting options, and how do we ensure the different objects (eg EAs) are following those options?

  • What are all the options we are talking about here?
  • Do we want to pass some of those to EA._formatter so that EA backed Series/DataFrame follows our general options?

[simonjayhawkins]

I think you are confusing two different _formatting_values: the one defined on Series, and the one defined on an EA.

i guess. is #24858 related?

[simonjayhawkins]

It's a bit of indirection, but that is how the EAs are currently displayed: the values itself are converted to a numpy array (the __array__ that the EA defines) and the display is determined by the EA._formatter. And then it is using our format_array machinery to format those values with the custom formatter.

but also pass through ExtensionArrayFormatter. Is this generic enough for EA developers?

[jorisvandenbossche]

Is this generic enough for EA developers?

What do you mean exactly?

I think you are confusing two different _formatting_values: the one defined on Series, and the one defined on an EA.

i guess. is #24858 related?

Well, that was an issue prompted by the deprecated _formatting_values, but it is not really related to this discussion I think. The issue there is that the new way (to replace the deprecated one) requires np.array(EA) to work (which is probably generally not a problem).

[simonjayhawkins]

• Do we want our internal EAs to follow the EA interface for formatting?

that would presumably be less performant. on the other hand, shouldn't EA developers have a mechanism to be able to achieve the same performance?

[jorisvandenbossche]

Yes, and additionally, EAs might also want to use the full array context to format their values.

A possible conclusion from this discussion might be that it was maybe the wrong direction we took when deprecating _formatting_values (returning a to be printed array) for _formatter (a formatter for scalar objects).
We might want to go back to an array-based formatting, so the end-goal can be that all formatting lives on the Array itself (of course the helper functions for that does not need to live there, only the responsible caller).

@TomAugspurger do you remember what the motivation was to move from the formatting_values to the scalar formatter?

[simonjayhawkins]

Is this generic enough for EA developers?

What do you mean exactly?

ExtensionArrayFormatter has special casing for internal EAs. That's why this issue was opened. The title of this issue maybe misleading now since the PR where the contents of ExtensionArrayFormatter was moved to format_array has been reverted.

A possible conclusion from this discussion might be that it was maybe the wrong direction we took when deprecating _formatting_values (returning a to be printed array) for _formatter (a formatter for scalar objects).

That maybe a separate issue. but could solve the special casing?

[jorisvandenbossche]

That maybe a separate issue

Not sure, as I think any refactor of format_array should at least involve thinking about how to handle EAs / how this integrates with our own formatting. From your initial post: " we need to push more functionality to the objects to be formatted themselves "
What is the issue otherwise about ? ;)

[simonjayhawkins]

What is the issue otherwise about ? ;)

the issue is raised directly from #26833 (comment), where the PR was a precursor to allow a custom formatter to override the defaults #26000

The EA formatters are 'incomplete' as custom formatters as they don't account for other formatting options. I have now updated #26000, to allow EA formatters and custom formatters specified in to_string, to_latex and to_html to share the same format_array machinery.