less than, greater than cuts on categorical don't follow order

[olgabot]

When subsetting an ordered pd.Categorical object using less than/greater than on the ordered values, the less/than greater than follow lexicographical order, not categorical order.

If you create a dataframe and assign categories, you can subset:

In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: df = pd.DataFrame({'x':np.arange(10), 'y':list('AAAABBBCCC')})

In [4]: df.y = df.y.astype('category')

In [5]: df.ix[(df.y >= "A") & (df.y <= "B")]
Out[5]: 
   x  y
0  0  A
1  1  A
2  2  A
3  3  A
4  4  B
5  5  B
6  6  B

But if you try to subset on an ordered category, it does the lexicographical order instead:

In [6]: df = pd.DataFrame({'x':np.arange(10), 'y':list('AAAABBBCCC')})

In [7]: df.y = pd.Categorical(df.y, categories=['A', 'C', 'B'], ordered=True)

In [8]: df.ix[(df.y >= "A") & (df.y <= "C")]
Out[8]: 
   x  y
0  0  A
1  1  A
2  2  A
3  3  A
4  4  B
5  5  B
6  6  B
7  7  C
8  8  C
9  9  C

[jreback]

ordered=True just means that the categories themselves are in order (and its meaningful).

A subsetting operation does selection which is ordered by the index of the frame.
You need to do an operation that matters to the actual order, e.g. sort/min/max

In [18]: In [8]: df.ix[(df.y >= "A") & (df.y <= "C")].sort('y')
Out[18]: 
   x  y
0  0  A
1  1  A
2  2  A
3  3  A
7  7  C
8  8  C
9  9  C
4  4  B
5  5  B
6  6  B

Note if you had a CategoricalIndex (merging very shorty), then this would work as you expect

[jorisvandenbossche]

@jreback I don't fully understand your explanation of the behaviour.

I think the point of having an ordered categorical is that this order is used in comparisons ?

[jorisvandenbossche]

Relabeled the issue until we agree on that :-)

cc @JanSchulz @shoyer

So the point is this:

In [34]: cat = Series([1, 2, 3]).astype("category", categories=[3, 2, 1], ordered=True)

In [35]: cat
Out[35]:
0    1
1    2
2    3
dtype: category
Categories (3, object): [3 < 2 < 1]

In [37]: cat[[0]]
Out[37]:
0    1
dtype: category
Categories (3, object): [3 < 2 < 1]

In [38]: cat[[0]] > 2
Out[38]:
0    False
dtype: bool

Should the categorical value 1 be larger as 2 (it is larger based on the categorical order, it is not larger based on normal order)

[jankatins]

I agree with @jorisvandenbossche: in the last example, the comparison cat[[0]] > 2 should be True and df.y <= "C" should be False for all B in df.y.

Not sure what happened here, but the docs also show this "wrong" behavior: http://pandas.pydata.org/pandas-docs/version/0.15.2/categorical.html#comparisons
-> Both comparisons should have the same output

On the other hand, with the ordered = False default change they should raise an error :-(

[kay1793]

The implementation of categories in pandas doesn't distinguish a (single) catgeory from its underlying value:

cat = Series([1, 2, 3]).astype("category", categories=[3, 2, 1], ordered=True)
type(cat[0])
Out[7]: numpy.int64
# *not* "pandas.Category(value=1,parent=Foo)"

I expect users sometimes want to subset based on value and other times by order (just as loc/iloc selects based on label or location) but the way a category and its value are lumped into one makes it difficult to support both. If you did have singleton categories as objects, you would need to have them reference the parent set of categories they belong to in order to be meaningful.

[jankatins]

@kay1793: ok, good catch, this should account for cat[0] > 2, but not for df.y > "C"

[kay1793]

That's true.

If the cat series comparison adhered to order, users could still get a value based comparison
by "lowering" the category series into a series of values. That's bound to confuse newcomers so hopefully the error message will be informative.

[jorisvandenbossche]

@kay1793 Indeed, but that is the limitation of numpy we have to live with at the moment (and that is also the reason I used cat[[0]] in my example ..)

[jorisvandenbossche]

@JanSchulz regarding the question on "raising for ordered=False" -> at the moment we said that sorting such a category should work. So maybe it is a bit unconsistent that a direct comparison of greater/smaller than would raise?

[kay1793]

That's true.

That is, unless you think what olga is asking for should work as:

cats[cats > pd.category(1,parent=<Cats>)]

or similar, rather then forcing anyone who wants cats[cats < 2] to downcast the entire array first in order to compare by value. it's a question of whether you choose the semantics based on the type of the category array or the type of other (when comparing with a category array: 2 < cats and cats >2).