ENH: Indexes with any numpy int/uint/float dtype

[topper-123]

I've made a (draft) PR in #41153 that implements an index class for all the normal numpy numeric dtypes (int64/32/16/8, uint64/32/16/8 an dfloat65/32). There was some discussion in that PR how the public API for this should be, so I'm opening this issue for discussing that.

My plan is to get #41153 merged when its ready, possibly without any public-facing API changes. The public-facing changes would then come afterwards, and after there is an conclusion on how the API should be.

Summary of the options

As I see it the options for the API is:

Nr.	Option	Is backwards compat.	complexity
1.	Have a single, public NumericIndex class for all numeric index types	Yes	Easy
2.	Have dtype-specific subclasses (Int8Index, Int16Index, Int32Index, etc), and those can still be backed by a single internal NumIndex	Yes	Easy
3.	Use the existing, general-purpose Index class for these numeric numpy dtypes	No	Harder
4.	Use the existing, numeric index classes for these numeric numpy dtypes, so e.g. Int64Index can take int32 etc.	No	Easy

Options 1. and 3. would mean that the existing Int64Index, UInt64Index and Float64Index would be deprecated and removed in pandas 2.0, because their functionality would bw covered by the other index classes.

Option 2. would increase the number of numeric index classes.

Options 4. would extend the functionality of the existing numeric index classes.

So there are quite a few possibilities. Hopefully we can come to a common conclusion.

@pandas-dev/pandas-core

[topper-123]

My opinion:

The current index types are very ingrained in pandas, and I think we should prioritize backwards compatibility in pandas 1.x, even at the possible cost of having a worse API short-term. The "optimal" API could then be implemented for pandas 2.0.

In that light I would prefer different short-term and long-term choices:

Short-term: Implement a seperate NumericIndex, i.e. keeping Int64Index etc. working unchanged, but deprecate them in due time for pandas 2.0.
Long-term: For pandas 2.0, integrate the numeric indexes into the base Index and removing Int64Index, NumericIndex etc.

I think new index classes should only be implemented for differing API and there will not be API differences between working on the various indexes with different numpy numeric dtypes. So having a index class for each dtype subtype is IMO excessive, and in e.g. in Float32Index(..., dtype="float32") the dtype parameter would be superflous and doesn't add value to the API because the dtype is encoded in the index class.

[toobaz]

Short-term: Implement a seperate NumericIndex, i.e. keeping Int64Index etc. working unchanged, but deprecate them in due time for pandas 2.0.

I agree, with the existing numeric indexes being actually implemented as NumericIndex

Long-term: For pandas 2.0, integrate the numeric indexes into the base Index and removing Int64Index, NumericIndex etc.

I still fail to see why e.g. NumericIndex should be integrated inside Index but not RangeIndex, IntervalIndex... maybe there is a clear (and easily documentable) API distinction... but I'm failing to see it, and hence I don't like this idea. Sure, one could argue that IntervalIndex is different because its elements are (in some sense) "not scalar", but already the distinction with RangeIndex is more in terms of implementation... and what about strings?!

In general, while I understand that aggregating in Index more functionalities allows to clean the API, i think it's not a consistent approach in particular now that types and implementations are becoming more modular. What about alternative (even third-party) indexes that store numbers in different ways (like EA for values)? What becomes the rule of "what Index does?" if it's not just about object, nor about "indexing with numbers" (RangeIndex does too and possibly more in the future)?

[jbrockmendel]

Use the existing, numeric index classes for these numeric numpy dtypes, so e.g. Int64Index can take int32 etc.

My kneejerk reaction is very negative on the idea of Int64Index holding anything other than int64.

Use the existing, general-purpose Index class for these numeric numpy dtypes

In 2.0, if we could get combine all Index subclasses back into Index, that'd be pretty neat. Failing that, having only a handful of dtypes return the base class seems klunky.

Have dtype-specific subclasses (Int8Index, Int16Index, Int32Index, etc), and those can still be backed by a single internal NumIndex

The proliferation of classes is unfortunate, but this is the most straightforward option.

Have a single, public NumericIndex class for all numeric index types

I'd be fine with this.

[jorisvandenbossche]

I still fail to see why e.g. NumericIndex should be integrated inside Index but not RangeIndex, IntervalIndex... maybe there is a clear (and easily documentable) API distinction... but I'm failing to see it

@toobaz there is indeed a very clear API distinction between Index and IntervalIndex: IntervalIndex has additional public methods and properties compared the base Index class, and it wouldn't make sense to add those interval-specific methods and properties to the base class.
The different numeric Index classes (Int64Index, UInt64Index, Float64Index) have no such API distinction with the base Index class. So basically swapping NumericIndex with Index doesn't have any user-visible impact, except for the repr being "Index([..., dtype='int64')" instead of "NumericIndex([...], dtype='int64')".

What about alternative (even third-party) indexes that store numbers in different ways (like EA for values)?

IMO that's actually a good reason to use the base Index class for other dtypes than just object dtype. That way, a third-party EA can actually be used as Index using the Index class (currently this is not possible, and we convert the EA to a numpy array when storing it in the index)

[toobaz]

@toobaz there is indeed a very clear API distinction between Index and IntervalIndex: IntervalIndex has additional public methods and properties compared the base Index class, and it wouldn't make sense to add those interval-specific methods and properties to the base class.

You have a point, and one that in principle is also easy to document ("it's a separate class iff it provides additional methods").

I'm a bit afraid (in terms of implementation, but also of users expectations) of deciding that whether a given type/storage mechanism is supported by Index has nothing to do with the code actually implementing it below the surface. But it's true that as long as we state that any type/storage mechanism will be supported only via EA, it's just a matter of supporting EA.

a third-party EA can actually be used as Index using the Index class (currently this is not possible, and we convert the EA to a numpy array when storing it in the index)

I'm a bit confused by what pd.core.indexes.extension.ExtensionIndex is for. Isn't it able to keep the EA as such?

Notice that if Index will need to support EA, some Index subclasses might also have to (think about an EA providing an alternative implementation of datetimes, or intervals). But maybe this is not too difficult by doing inheritance right.

[jorisvandenbossche]

I'm a bit confused by what pd.core.indexes.extension.ExtensionIndex is for. Isn't it able to keep the EA as such?

It's (currently) only meant as a base class for a few of our own existing Index subclasses that are backed by an ExtensionArray (i.e. CategoricalIndex, IntervalIndex, Datetime/Timedelta/PeriodIndex), but you currently can't use it for any EA (it could be extended for this use case, or we could simply use Index for that, which relates to point 1) in #39133)

[toobaz]

you currently can't use it for any EA

I see. And now that I think about it,

some Index subclasses might also have to (think about an EA providing an alternative implementation of datetimes, or intervals). But maybe this is not too difficult by doing inheritance right.

Inheritance must be done right, because we might legitimally want to have different subclasses (i.e., with specific methods) to support EA. So whether or not we merge as much as possible in Index, we would like Index and any of its subclasses to support EA.

Still not a fan of the merge, but I finished my arguments ;-) @jorisvandenbossche your general API distinction makes sense, I still think it complicates the codebase but it might be worth it. Bonus points if Index learns to manage non-predefined numpy dtypes (e.g. fixed length strings - clearly not as an alternative to #35169 ).

[jbrockmendel]

there is indeed a very clear API distinction between Index and IntervalIndex: IntervalIndex has additional public methods and properties compared the base Index class, and it wouldn't make sense to add those interval-specific methods and properties to the base class.

If we wanted to get down to a single Index class, one option for these methods/properties would be to hide them behind an accessor.