Make categories and ordered part of CategoricalDtype

[TomAugspurger]

This is to discuss pushing the Categorical.categories and
Categorical.ordered information into the extension type CategoricalDtype.

pd.CategoricalDtype(categories, ordered=False)

Note that there is no values argument. This is a type constructor, that
isn't attached to any specific Categorical instance.

Why?

Several times now (read_csv(..., dtype=...), .astype(...), Series([], dtype=...))
we have places where we accept dtype='category' which takes the values
in the method (the series, or column from the CSV, etc.)
and hands it off to the value constructor, with no control over the
categories and ordered arguments.

Categorical(values, categories=None, ordered=False)

The proposal here would add the categories and ordered
attributes / arguments to CategoricalDtype and provide a common API
for specifying non-default parameters for the Categorical constructor
in methods like read_csv, astype, etc.

t = pd.CategoricalDtype(['low', 'med', 'high'], ordered=True)
pd.read_csv('foo.csv', dtype={'A': int, 'B': t)
pd.Series(['high', 'low', 'high'], dtype=t)

s = pd.Series(['high', 'low', 'high'])
s.astype(t)

We would continue to accept dtype='category'.

This becomes even more import when doing operations on larger than memory datasets with something like dask or even (read_csv(..., chunksize=N)). Right now you don't have an easy way to specify the categories or ordered for columns (assuming you know them ahead of time).

Issues

1. CategoricalDtype currently isn't part of the public API. Which methods
   on it do we make public?
2. Equality semantics: For backwards compat, I think all instances
   of CategoricalDtype should compare equal with all others. You can use
   identity to check if two types are the same

t1 = pd.CategoricalDtype(['a', 'b'], ordered=True)
t2 = pd.CategoricalDtype(['a', 'b'], ordered=False)

t1 == t2  # True
t1 is t2  # False
t1 is t1  # True

3. Should the categories argument be required? Currently dtype='category'
   says 1.) infer the categories based on the values, and 2.) it's unordered.
   Would CategoricalDtype(None, ordered=False) be allowed?
4. Strictness? If I say

pd.Series(['a', 'b', 'c'], dtype=pd.CategoricalDtype(['a', 'b']))

What happens? I would probably expect a TypeError or ValueError as c
isn't "supposed" to be there. Or do we replace 'c' with NA? Should
strict be another parameter to CategoricalDtype (I don't think so).

I'm willing to work on this over the next couple weeks.

xref #14676 (astype)
xref #14503 (read_csv)

[TomAugspurger]

cc @JanSchulz (let me know if you want me to stop pinging you on this categorical issues 😄 ). Also @mrocklin I think this may interest you as the fastparquet stuff could use CategoricalDtype since you know the categories ahead of time.

[jankatins]

Will this also mean that Categorical is changed to save the categories/order information in the the dtype object? As pandas2 seems to go the "we ave our own dtypes"-way, it seems that would be a sensible way (if that would also happening with the rest of the dtypes)

[Thanks for pinging me!]

[TomAugspurger]

Will this also mean that Categorical is changed to save the categories/order information in the the dtype object?

Yeah, pd.Categorical(['a', 'b'], categories=['a', 'b', 'c'], ordered=True).dtype would be CategoricalDtype(['a', 'b', 'c'], ordered=True).

Dunno if we want to expand the pd.Categorical constructor to take a dtype=CategoricalDtype option. Perhaps.

[jreback]

yes, this was the original plan when I first did this, but too many moving parts.

If this can be implemented in a back-compat manner (as this is pretty much an implementation detail). I think would be great.

[jorisvandenbossche]

1. CategoricalDtype currently isn't part of the public API. Which methods on it do we make public?

Specific for categorical, I suppose we would have a categories and ordered attribute? But for the existing attributes/methods, maybe to start with we could say in the docs that none of these are public? (not sure if there are that you would want to use)

2. Equality semantics: For backwards compat, I think all instances of CategoricalDtype should compare equal with all others. You can use identity to check if two types are the same

Sounds OK

3. Should the categories argument be required? Would CategoricalDtype(None, ordered=False) be allowed?

I would not allow this to start with. Is the idea that you can this way let the categories be inferred from the values, but set the ordered attribute through the dtype? Or what would be the application?

4. Strictness? I would probably expect a TypeError or ValueError as c isn't "supposed" to be there. Or do we replace 'c' with NA?

It seems that we currently replace with NA:

In [53]: pd.Categorical(['a', 'b', 'c'], categories=['a', 'b'])
Out[53]: 
[a, b, NaN]
Categories (2, object): [a, b]

In [54]: pd.Series(['a', 'b', 'c']).astype('category', categories=['a', 'b'])
Out[54]: 
0      a
1      b
2    NaN
dtype: category
Categories (2, object): [a, b]

although I also think I would like a TypeError more as the default behaviour.

[jorisvandenbossche]

@TomAugspurger How is this issue different from #14676, can that one be closed?

[TomAugspurger]

That one will be a trivial change to .astype once this is fixed. I think leave it open as a reminder to implement it once the CategoricalDtype changes are done.

…

On Nov 23, 2016, at 05:25, Joris Van den Bossche ***@***.***> wrote: @TomAugspurger How is this issue different from #14676, can that one be closed? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[TomAugspurger]

Small update for future reference. I'm having trouble with the current approach where CategoricalDtype[categories, ordered] is a singleton, i.e. one instance of CategoricalDtype for each category and ordered. The problem is _cache being a (weakref) dictionary. We need

CategoricalDtype([1, 2])

to be different from

CategoricalDtype([1., 2.])

but if we use a dictionary to do singleton stuff, the keys compare equal so we only ever get one. @jreback do you have any links to info on what-all requirements the numpy duck-type has to satisfy?