API: preferred way to check if column/Series has Categorical dtype

[jorisvandenbossche]

From http://stackoverflow.com/questions/26924904/check-if-dataframe-column-is-categorical/26925340#26925340

What is the preferred way to check for categorical dtype?

I now answered:

In [42]: isinstance(df.cat_column.dtype, pd.core.common.CategoricalDtype)
Out[42]: True

In [43]: pd.core.common.is_categorical_dtype(df.cat_column)
Out[43]: True

But:

• this seems somewhat buried in pandas. Should there be a more top-level function to do this?
• we should add the preferred way to the categorical docs.

[onesandzeroes]

That was me asking the question. I originally started writing up the question because I was working on a PR for pandas, and while writing I discovered is_categorical_dtype(). Since I was working on internal pandas code that works for my current usage.

Having something that's not so deeply buried would be good though. I tried df.col.dtype == 'category' because I thought that was a pretty standard way of doing a quick type check. Unless there are strong reasons not to use this method, it should probably work the same for categoricals as it does for other types (e.g. df.col.dtype == 'float64')

[shoyer]

df.col.dtype == 'category' does appear to work for me on pandas 0.15.1. As @onesandzeroes says, I think should be the preferred way to check for categorical types.

It does looks like Categorical.dtype.__ne__ needs to be defined, though -- currently it's not, so Python defaults to something arbitrary.

[jorisvandenbossche]

@shoyer as you can see in my answer on SO, it does indeed work, but the problem is it raises for other dtypes instead of giving False, which is not very handy (and that is a numpy thing).
With the example of SO:

In [86]: df
Out[86]:
  cat_column   x   y
0          c   0   0
1          d  10   4
2          f  20   8
3          a  30  12
4          b  40  16
5          e  50  20

In [87]: df.cat_column.dtype == 'category'
Out[87]: True

In [88]: df.x.dtype
Out[88]: dtype('float64')

In [89]: df.x.dtype == 'category'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-89-aff611b16544> in <module>()
----> 1 df.x.dtype == 'category'

TypeError: data type "category" not understood

So exactly because that is not working (as you would expect: returning False), I think we should provide a common way to do this (or at least document this in the categorical docs what is the best way to do this)

[shoyer]

Ah, I see. A reasonable solution might be to wrap the dtype in str, e.g., str(df.x.dtype) == 'category'.

[jorisvandenbossche]

cc @JanSchulz

[jreback]

In [10]: df = DataFrame({'A' : np.random.randn(5), 'B' : Series(list('aabbc')).astype('category')})

In [11]: df
Out[11]: 
          A  B
0 -0.064981  a
1  0.852717  a
2  0.693611  b
3  0.411486  b
4 -1.425537  c

In [12]: df.dtypes
Out[12]: 
A     float64
B    category
dtype: object

In [13]: df.dtypes == 'category'
Out[13]: 
A    False
B     True
dtype: bool

In [14]: df.select_dtypes(include=['category'])
Out[14]: 
   B
0  a
1  a
2  b
3  b
4  c

n [16]: pd.core.common.is_categorical_dtype(df.A.dtype)
Out[16]: False

So the preferred method of 'cheking' dtypes is simply to use select_dtypes or [13] works as well. Check a np.dtype('float64')=='category' blows up - I think maybe should create a bug report to have this fixed upstream. Not much we can do for this. Of course as @jtratner pointed out
``np.dtype('float64').dtype.name=='category' (will work correctly with numpy dtypes).

So I don't think it necessary to have the user actually use anything internal.

If pressed, com.is_categorical_dtype(...) would be ok

would not suggestion any mention/use of com.CategoricalDType (as an instance check) though - this is TOO internal.

To be honest this rarely should if ever come up. If the OP is trying to check individual dtypes for category then this is the wrong approach (and mostly certainly .select_dtypes() is the correct method.

So if someone wants to add a small doc section, ok.

[jankatins]

Shouldn't this work?

if s.cat: 
   print("It's a categorical!")

[jreback]

@JanSchulz

that will raise for non-cat types (as does .dt)

prevents user error. I think this is correct (these should raise)

[jorisvandenbossche]

@JanSchulz no, because it gives a TypeError instead of False if it is not a categorical:

In [143]: df
Out[143]:
          A  B
0  0.299586  a
1  0.335853  a
2 -0.135405  b
3  1.247738  b
4 -0.232270  c

In [144]: bool(df.B.cat)
Out[144]: True

In [145]: bool(df.A.cat)
....
TypeError: Can only use .cat accessor with a 'category' dtype

[jreback]

I think is an issue that should be raised to numpy

In [9]: np.dtype('i8') == 'foo'
TypeError: data type "foo" not understood

closing on pandas side as this is sane on the pandas side