DOC: `CategoricalDtype` equality semantics aren't completely described

[VladimirFokow]

Pandas version checks

• I have checked that the issue still exists on the latest versions of the docs on main here

Location of the documentation

https://pandas.pydata.org/docs/user_guide/categorical.html#equality-semantics

https://github.com/pandas-dev/pandas/blob/main/pandas/core/dtypes/dtypes.py#L407

Documentation problem

Problematic statement 1:

Two instances of CategoricalDtype compare equal whenever they have the same categories and order.

Problematic statement 2:

5. A CDT with ordered={False, None} is only equal to another CDT with
   ordered={False, None} and identical categories.

Counter-example:

>>> a = pd.Categorical(np.full(2, np.nan, dtype=object))
>>> b = pd.Categorical(np.full(2, np.nan))

>>> a, b
([NaN, NaN]
 Categories (0, object): [],
 [NaN, NaN]
 Categories (0, float64): [])

>>> a.dtype, b.dtype
(CategoricalDtype(categories=[], ordered=False, categories_dtype=object),
 CategoricalDtype(categories=[], ordered=False, categories_dtype=float64))

>>> a.dtype == b.dtype
False

As we can see, they both have ordered=False, and their categories are same.
Following the documentation, they should be equal.

Suggested fix for documentation

to have accurate and exhaustive descriptions

[rhshadrach]

Thanks for the report.

As we can see, they both have ordered=False, and their categories are same.

Though both empty, two arrays of different dtypes are not the same.

I suppose "including their dtypes" would be okay to add.

[luke396]

take

[VladimirFokow]

thanks @rhshadrach !
I don't think this is a good first issue, because someone with an accurate and full conceptual understanding of Categorical and CategoricalDtype should help edit these descriptions.
I expand on this in PR #57273
(but on the other hand, maybe it is good - for people to figure this out and then write their findings)

[dvl-mehnaz]

#57259

[VladimirFokow]

@Mehnaz02
dtype in b in my example is float64
(the whole point is a and b having different dtypes in their CategoricalDtypes):

# when
a.categories.dtype
# is different from:
b.categories.dtype

(btw, in the latest pandas version a.dtype also prints the categories_dtype at the end)

[VladimirFokow]

I can try starting to fix it, but definitely much later..
and would need a technical check for accuracy.

[rhshadrach]

I don't think this is a good first issue, because someone with an accurate and full conceptual understanding of Categorical and CategoricalDtype should help edit these descriptions.

I disagree with this assessment. I believe one just needs to modify the documentation to state that the categories must be the equal as indexes, and that in particular includes their dtype. Am I missing something?

In addition, all PRs are reviewed, and so help can be given as part of that review.