Handle sparse matrix

[glemaitre]

Some of the proposed methods should handle sparse matrix. We should review which methods can be improved to handle these data.

Are there any plans to add support soon, even for the very basic stuff like "Random Selection"?

[glemaitre]

We would like too. Any PR is welcome.

[dbarrundiag]

Say I have a pipeline that looks like this:

pipeline = Pipeline([
               ('features', FeatureUnion([ . . . ])),
               ('under_sampler', RandomUnderSampler()),
               ('predictor', SVC())
     ])

This will fail as the FeatureUnion creates a sparse matrix correct? Is there any hacky way I can make this work?

[glemaitre]

I don't have a real solution there apart of forcing the output of FeatureUnion to be dense if this is not an issue for you. We are almost on the corner to start moving the method to support sparse matrix but we want to finish the user guide first.

[ocelot256]

Hi,

I wanted to ask if there has been any progress on this?

I am working with a large sparse dataset that is too big to actually load in a dense format, and I'm running into issues with this using version 0.2.1 of the library.

What would be involved in this, and would it require a lot of changes?

Thanks!

[glemaitre]

@ocelot256 It will depend of the algorithm and if the scikit-learn algorithm used underneath is supporting the sparsity. I might find a bit of time to look at it during the summer

[ocelot256]

@glemaitre I think the RandomUnderSampler is a good starting point.

I was looking at the code and it seems the base class calls sklearn's check_X_y for every subclass. So this needs to be fixed if sparse matrix support is going to be added gradually.

I can imagine this is a real issue for many people who need to use this in practical scenarios. I'm going to try and look into it. Feel free to let me know any ideas you had about implementing this.

[glemaitre]

I would think that check_X_y could take sparse='csr' and we could enforce check_X_y in each concrete class to be less permissive if needed. Anyway the idea will be to accept sparse everywhere.