ENH: Add sparse option in pivot_table / pivot / unstack

[yupbank]

A small, complete example of the issue

# Your code here
sparse_df = df.pivot(index='a', columns='b', values='c', aggfunc=lambda x: len(x), sparse=True)

Expected Output

Output of pd.show_versions()

# Paste the output here

[jreback]

this is possible; would have to be integrated into unstacking proper. We do do something like this in get_dummies, but that is not fully generalizable. So this would be an effort. A community pull-request would be needed here.

[yupbank]

i might try my best for a pull-request :)

[yupbank]

@jreback i noticed that there isn't many options for sparse_matrix. (which in my case want something df.to_csr_matrix) is that ok, i add some sparse_matrix to current limited scipy.coo_matrix only collection ?
or do you suggest i talk to someone who knows the pandas.sparse better?
and also DataFrame.groupby people, cause i notice the actually agg_func/value assign of pivot happened in groupby?

[jreback]

adding sparse means pandas sparse structures: http://pandas.pydata.org/pandas-docs/stable/sparse.html

[jnothman]

I have implemented, I think, most of the logic for sparse unstacking of series (or homogeneous dtypes) at master...jnothman:sparse-unstack

So far I am:

• currently overwriting the existing _Unstacker.get_result() implementation for ease of testing.
• not handling the categorical case yet (cf. Behaviour of Categorical inputs to sparse data structures #19278).
• not currently correctly outputting SparseDataFrame where it would be output
• not adding a sparse parameter to allow the user to switch on sparse output

I'd be very happy for someone else to complete the work!

[jnothman]

Maybe then it can be downgraded to Effort Medium :P