non-NDFFrame object error using pandas.SparseSeries.from_coo() function

[francescoferroni]

There appears to be an issue with the .from_coo() sparse function. If the frame is viewed, it gives a "non-NDFFrame error". It could potentially be due to overlapping entries on the same index (this is handled in sparse.coo_matrix by adding the entries by default).

More details here:
http://stackoverflow.com/questions/31970070/non-ndfframe-object-error-using-pandas-sparseseries-from-coo-function

Example:

import pandas as pd 
import scipy.sparse as ss 
import numpy as np 
row = (np.random.random(100)*100).astype(int) 
col = (np.random.random(100)*100).astype(int) 
val = np.random.random(100)*100 
sparse = ss.coo_matrix((val,(row,col)),shape=(100,100)) 
pd.SparseSeries.from_coo(sparse)
TypeError: cannot concatenate a non-NDFrame object

[jreback]

looks like a bug. needs someone to dig in on these sparse issues.

[hpaulj]

I've deduced that it's a size issue. When the underlying Series starts to display with ellipsis, I get this error. So it has to do with the number of terms (nonzero elements in scipy sparse matrix). I did my testing in Py3 and got a different error (from the .__unicode__ branch rather than the .__bytes__ one).

It may affect any SparseSeries, regardless of whether it is constructed with from_coo, since it occurs after the to_sparse step.

s = Series(A.data, MultiIndex.from_arrays((A.row, A.col)))
s = s.sort_index()
s = s.to_sparse()  # TODO: specify kind?

[jreback]

The fundamental issue is that slicing is broken in sparse.

In [11]: s = Series([1]+[np.nan]*5).to_sparse()

In [12]: s
Out[12]: 
0     1
1   NaN
2   NaN
3   NaN
4   NaN
5   NaN
dtype: float64
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([1], dtype=int32)

In [13]: s.iloc[0:3]
Out[13]: 
[1.0, nan, nan]
Fill: nan
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([1], dtype=int32)

In [14]: type(s.iloc[0:3])
Out[14]: pandas.sparse.array.SparseArray

In [15]: type(s)
Out[15]: pandas.sparse.series.SparseSeries

[14] should be a SparseSeries (not SparseArray which is the underlying object the SparseSeries holds). its not getting wrapped when sliced somewhere.

[jreback]

this would be fixed by #10627

[sinhrks]

As @jreback said, this is a displaying issue of SparseSeries created with from_coo. Dupe of #10560.

s = pd.SparseSeries.from_coo(sparse)
type(s)
# pandas.sparse.series.SparseSeries