ExtensionDtype API for preferred type when converting to NumPy array

[TomAugspurger]

This is coming out of SparseArray. Right now if you have a homogeneous DataFrame of EAs, then doing something like np.asarray(df) will always convert to object.

pandas/pandas/core/internals/managers.py

Lines 787 to 794 in 32a74f1

	def _interleave(self):
	"""
	Return ndarray from blocks with specified item order
	Items must be contained in the blocks
	"""
	dtype = _interleaved_dtype(self.blocks)
	result = np.empty(self.shape, dtype=dtype)

.

Should we give ExtensionDtype some input on the dtype used here? This would allow for some consistency between np.asarray(series) and np.asarray(homogeneous_dataframe).

[jorisvandenbossche]

Not sure if that was from the same discussion, but I recall a discussion where we said that we basically wanted to know the "numpy dtype if I would be converted to a numpy array" of an ExtensionArray, without actually converting it.
That seems something useful in general I think, and could then be used to determine the best interleaved dtype in the use case above?

[jorisvandenbossche]

It was indeed from the sparse discussion, my comment there: #22325 (comment)

Basically, we want to know np.array(EA).dtype without doing the conversion to array.

We could add a numpy_dtype attribute to the Dtype object?

This might also overlap with #22224

[TomAugspurger]

There's some overlap with #22224, but I don't know if we can re-use the same attribute for both unfortunately. IntegerArray.numpy_type can't hold values with NaNs.

[jorisvandenbossche]

Yes, I don't think we can re-use it, but the numpy_dtype attribute of IntegerArray can still be changed of course.

The other issue is more about if you have a composite array, what is the dtype of the underlying array (which of course can also be multiple arrays .., so then this might also be multiple dtypes), while here it is the dtype when converted to a numpy array.

But I would use a numpy_dtype name rather for the use case of this issue.

[jorisvandenbossche]

Although the question is, what if this dtype depends on the data?
For example, for IntegerArray, we now decided to return an object array. But, you could also opt for actually returning the integer array if there are no NAs, and float otherwise.

[TomAugspurger]

But, you could also opt for actually returning the integer array

I assume you meant integer dtype.

That's a good point. It's not clear what's best here. I think IntegerArray may not be the best one to consider here, since the "best type" so clearly depends on the values. There's no way to put that on the ExtensionDtype.

Let's consider, say, Categorical and Sparse. In this case, there's always a clearly best dtype for the resulting ndarray, CategoricalDtype.categories.dtype and SparseDtype.subdtype (or whatever we end up calling the type of sp_values).

[jorisvandenbossche]

But, you could also opt for actually returning the integer array

I assume you meant integer dtype.

Sorry, I was speaking about the return value for np.array(EA), that was not fully clear. So I did mean "integer (numpy) array".

The question then is, what should the value for this attribute be for IntegerArray? None to indicate it is not known ?

[jreback]

I think we have this now? e.g. can just specify the dtype= in np.asarray(), or something else needed here?

[jorisvandenbossche]

I don't think this is solved. It is not about passing a dtype in np.asarray(..), the original question is: predicting for an array what the dtype will be of np.array(dtype) (so the default when not passing a dtype= kwarg).

One use case is to define the interleaved dtype (see top post)

(but it's not necessarily a blocker for 0.24.0)

[TomAugspurger]

Agreed: not solved, but a blocker.

…

On Fri, Jan 4, 2019 at 10:22 AM Joris Van den Bossche < ***@***.***> wrote: I don't think this is solved. It is not about passing a dtype in np.asarray(..), the original question is: predicting for an array what the dtype will be of np.array(dtype) (so the default when not passing a dtype= kwarg). One use case is to define the interleaved dtype (see top post) (but it's not necessarily a blocker for 0.24.0) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#22791 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHItXsV-jbeGWEwnblFUqboJFQtLmfks5u_3_ggaJpZM4WydvG> .