Ordered vs. Unordered Categoricals

[jseabold]

I was looking at supporting Categorical objects in statsmodels [1] recently and again now in seaborn [2]. I think it would be great to support both ordered and unordered Categorical variables (via a flag or something). IIRC, there was some discussion about this. Is it possible? From what I could tell all Categorical types are assumed to have some order.

[1] statsmodels/statsmodels#2133
[2] mwaskom/seaborn#361

[jreback]

these are directly supported via Categorical.ordered

see a tutorial here:

http://nbviewer.ipython.org/github/jreback/pydata2014-pandas/blob/master/notebooks/Categorical.ipynb

[jseabold]

Ah, nice. Thanks!

[jseabold]

What's the preferred idiom for checking for these new first-class citizen Categoricals? Checking for the cat attribute raises an error (i.e., hasattr(series, 'cat'), if it's not a Categorical. AFAICT, the old Categorical and the new Categorical type aren't compatible. So I guess that leaves a version check?

[jreback]

see also more full discussinon in #8814 as this needs a section in the docs

In [10]: df = DataFrame({'A' : list('abc'),'B':Series(list('def'),dtype='category')})

In [11]: df
Out[11]: 
   A  B
0  a  d
1  b  e
2  c  f

In [12]: df.dtypes
Out[12]: 
A      object
B    category
dtype: object

In [13]: df.select_dtypes(include=['category'])
Out[13]: 
   B
0  d
1  e
2  f

In [18]: pandas.core.common.is_categorical_dtype(df['B'])
Out[18]: True

In [19]: pandas.core.common.is_categorical_dtype(df['A'])
Out[19]: False

In [20]: df['A'].dtype.name
Out[20]: 'object'

In [21]: df['B'].dtype.name
Out[21]: 'category'

[mwaskom]

Not sure if this is the right place for this, but the behavior of unordered categorical surprised me a bit:

import string
import numpy as np
import pandas as pd

cat = pd.Categorical(list(string.uppercase), ordered=False)
num = np.random.randn(26)
df = pd.DataFrame(dict(cat=cat, num=num))

df.sort("num").cat.unique()

This returns

array(['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
       'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'], dtype=object)

I was expecting to get the order of letters that now appears in the DataFrame (which would happen if cat just had a regular object dtype), rather than the original alphabetical order.

This is relevant because I want to leverage Categorical objects in seaborn to get plot elements ordered in good ways. I was hoping that Categorical.unique() would give me the category order, if it exists, and the item order in the DataFrame if not.

Is this the correct behavior? (Probably). Is there a way to get what I want out of the object?

[jreback]

In [24]: np.array(df.sort('num')['cat'].values)
Out[24]:
array(['K', 'J', 'X', 'P', 'Q', 'U', 'O', 'L', 'H', 'N', 'T', 'R', 'A',
       'S', 'E', 'W', 'M', 'G', 'B', 'V', 'Z', 'Y', 'I', 'C', 'F', 'D'], dtype=object)

[mwaskom]

That gives me the array of values, but I want the array of unique values, and if I use np.unique() it will sort

[mwaskom]

Also I'd have to handle the "is this column a categorical dtype series, and if so, is it already ordered?" logic on my end, where it would be preferable to get a consistent answer from the series itself.

[jreback]

unique by definition is not ordered (and arbitrary)

[mwaskom]

When I use an ordered category object, it is:

cat = pd.Categorical(["b", "c", "c", "a", "b", "c"], categories=["c", "b", "a"])
cat.unique()

gives

array(['c', 'b', 'a'], dtype=object)

[jreback]

In [34]: c = df['cat'].values

In [35]: c.reorder_categories(c.take(df.sort('num').index.unique())).categories
Out[35]: Index([u'C', u'U', u'H', u'A', u'O', u'E', u'X', u'F', u'D', u'S', u'L', u'Y', u'M', u'V', u'G', u'B', u'N', u'R', u'W', u'Q', u'I', u'J', u'K', u'T', u'P', u'Z'], dtype='object')

[mwaskom]

From the docstring:

ordered : boolean, optional
    Whether or not this categorical is treated as a ordered categorical. If not given,
    the resulting categorical will be ordered if values can be sorted.

This caused me to think that setting ordered to False would not lexicographically sort the levels, but would instead fall back to the normal unique() behavior of giving me the values in the order they appear in the dataframe