Performance: .unique / .nunique of categorical series slow on large data set

[joshlk]

I've noticed that Series.unique and Series.nunique when used with a categorical series can be slow on large dataset. Presumably its not utilising the shortcuts:

Series.unique = Series.cat.categories
Series.nunique = len(Series.cat.categories)

Heres an example in iPhython:

s = pd.Series(np.random.choice(['a','b','c'], 100000000)).astype('category')
%time s.nunique()
896 ms
%time len(s.cat.categories)
55.1 µs

Its significantly slower indicating its not using the above shortcut.

pd.show_versions()

pandas: 0.17.1
Cython: 0.23.4
numpy: 1.10.1
IPython: 4.0.1

[jreback]

pls pd.show_versions()
and an example

[joshlk]

Sorry @jreback more details have been added above

[jreback]

so, .unique on a categorical has a couple of guarantees, namely that it is in the order of appearance, and it only includes values that are actually present, e.g.

In [3]: s = Series(list('babc')).astype('category',categories=list('abcd'))

In [4]: s.unique()
Out[4]: 
[b, a, c]
Categories (3, object): [b, a, c]

In [6]: s.cat.categories
Out[6]: Index([u'a', u'b', u'c', u'd'], dtype='object')

so if you want .categories, then just use that.

[jreback]

I suppose it is possible to add an option (to all .unique), not just this one, to do some sort of quick uniqueness like this, but not sure of the utility beyond categoricals.

[joshlk]

I didn't realise that, I thought they were equivalent. Thanks

[jorisvandenbossche]

It would be good to clarify this in the docs I think:

• add a note to the docstring of categories that this includes all categories and not only those present in the data
• add the above example (Performance: .unique / .nunique of categorical series slow on large data set #12278 (comment)) to the tutorial docs categorical.rst

[jreback]

ok, let's repurpose this then. @joshlk want to do a PR?