Random Samplers fail for Nested Numpy Arrays

[dgrahn]

Description

I'm getting the following error when attempting to produce use the Random Over Sampler.

  File ".../lib/python3.7/site-packages/imblearn/base.py", line 79, in fit_resample
    X, y, binarize_y = self._check_X_y(X, y)
  File ".../lib/python3.7/site-packages/imblearn/over_sampling/_random_over_sampler.py", line 94, in _check_X_y
    X, y = check_X_y(X, y, accept_sparse=['csr', 'csc'], dtype=None)
  File ".../lib/python3.7/site-packages/sklearn/utils/validation.py", line 719, in check_X_y
    estimator=estimator)
  File ".../lib/python3.7/site-packages/sklearn/utils/validation.py", line 542, in check_array
    allow_nan=force_all_finite == 'allow-nan')
  File ".../lib/python3.7/site-packages/sklearn/utils/validation.py", line 59, in _assert_all_finite
    if _object_dtype_isnan(X).any():
AttributeError: 'bool' object has no attribute 'any'

This happened right after I updated sklearn.

Steps/Code to Reproduce

Both over and under sample fail.

from imblearn import over_sampling, under_sampling
import numpy  as np
import pandas as pd

df = pd.DataFrame({
    'val': [
        np.zeros((1, 2)),
        np.zeros((3, 4)),
        np.zeros((5, 6)),
    ]
})

x = df.values
y = np.array([ 0, 0, 1])

sampler = over_sampling.RandomOverSampler(random_state=42)
# sampler = under_sampling.RandomUnderSampler(random_state=42)
sampler.fit_resample(x, y)

Versions

Linux-3.10.0-957.21.3.el7.x86_64-x86_64-with-centos-7.6.1810-Core
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0]
NumPy 1.16.5
SciPy 1.3.1
Scikit-Learn 0.21.2

[dgrahn]

I will update with test steps, hopefully tomorrow.

[dgrahn]

Installing sklearn==0.20.4 resolves the problem, but the functionality may be as intended. I'm going to open an issue with them, but I think calling check_X_y with force_all_finite=False would solve the issue.

Is it necessary for this flag to be True?

[glemaitre]

We should avoid forcing for finite value since we are not using them here for any computation.

[dgrahn]

@glemaitre check_X_y is used in the following places with a number of different options. It looks like the BaseSampler._check_X_y method could be updated to handle the different options and thereby reduce the amount of duplicated code. Here's roughly what it could look like.

BaseSampler               = check_X_y(X, y, binarize=True)
FunctionSampler         = check_X_y(X, y, sparse=self.accept_sparse)
SMOTEENN                 = check_X_y(X, y, binarize=False)
SMOTENC                   = check_X_y(X, y, binarize=True, dtype=None)
RandomOverSampler = check_X_y(X, y, binarize=True, dtype=None, force_all_finite=False)
SMOTETomek             = check_X_y(X, y, binarize=False)
SamplerMixin             = check_X_y(X, y)

def _check_X_y(X, y, binarize=False, dtype='numeric', accept_sparse=True, force_all_finite=True):
    if binarize:
        y, binarize_y = check_target_type(y, indicate_one_vs_all=True)
    else:
        y = check_target_type(y)

    sparse = ['csr', 'csc'] if sparse else False
    X, y = check_X_y(X, y, accept_sparse=sparse, dtype=dtype, force_all_finite=force_all_finite)

    if binarize:
        return X, y, binarize_y
    else:
        return X, y

I'm not entirely happy with how complex that is...

[glemaitre]

I think that the idea was to have a conservative _check_X_y for the majority of the case and overwrite it for a specific use case. I am unsure that passing the parameter will solve the issue because you will need to call _check_X_y with the right parameter which is delegated to BaseSampler up to know.