Using Array API functions on DataFrame objects

[jakirkham]

In some cases users like to use Array API functions (for example where) on DataFrame objects (in particular Series). Is this something that we would like to support in the API? If not, how would we recommend users approach these kinds of problems.

For an example of this please see issue ( dask/distributed#5224 )

[rgommers]

In some cases users like to use Array API functions (for example where) on DataFrame objects (in particular Series). Is this something that we would like to support in the API? If not, how would we recommend users approach these kinds of problems.

We had (/have) a pretty strong consensus that there should not be a separate Series-like object, but only a DataFrame object with a single column. IIRC the key issue is that Series and DataFrame have so much API duplication, for little benefit.

That's not a complete answer though. Your question can be rephrased as something like:

• "Can array API functions be applied to dataframes?", or
• "Is there a connection between the array and dataframe APIs?"

It in principle makes sense to me to have array API functions work on dataframes with a homogeneous dtype (which includes single-column dataframes). I'm not sure there's a good way to pick and choose what functions in the array API make sense. I can imagine a dataframe library providing the whole array API somehow, or to reuse an existing array library that is a dependency.

[jakirkham]

When discussing this in the call a few weeks back (and please feel free to correct me), we explored a few options. In the end gravitated towards having some way to share/convert data between DataFrames and Arrays with the idea one could then use Array operations on the converted Array. If the underlying library doesn't have an actually Array, they could just return some object that implements the Array API. Here are some related discussions on this topic ( #25 ) ( #39 ) ( #48 )

cc @jorisvandenbossche (in case I missed anything here 🙂)

[jbrockmendel]

is something analogous to __array_ufunc__ or __array_function__ an option? IIUC the numpy devs haven't always been happy with these designs

[rgommers]

is something analogous to __array_ufunc__ or __array_function__ an option?

The array API standard has __array_namespace__. If Column is array API standard compliant and has an __array_namespace__ method, things will work.

IIUC the numpy devs haven't always been happy with these designs

Indeed. __array_namespace__ is based on the design ideas that were trying to address the limitations of __array_ufunc/function__.

[rgommers]

We had (/have) a pretty strong consensus that there should not be a separate Series-like object, but only a DataFrame object with a single column.

We collectively changed our minds on this one. I think that came over time with (a) the realization that we're really building a library author-focused API that is quite different from the public APIs in current dataframe libraries, (b) that there are actually things that a one-column dataframe does not do well enough (e.g. we now are introducing unique values function for Column which is pretty clean; unique for DataFrame would be different and much more complex), and (c) that a column object which is array-like but with solid missing data support and string/datetime/categorical dtypes is a useful primitive.

So, I think we still want the thing here that @jakirkham originally asked for - but on Column rather than on DataFrame.