-
-
Notifications
You must be signed in to change notification settings - Fork 18.5k
BUG: Series.corr/cov raising with masked dtype #51422
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 6 commits
192d20d
f9fb872
caab0d7
14e32ba
5fd5030
3b47e75
7e78103
1a94a2c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2647,9 +2647,12 @@ def corr( | |
if len(this) == 0: | ||
return np.nan | ||
|
||
this_values = np.asarray(this.values) | ||
other_values = np.asarray(other.values) | ||
|
||
if method in ["pearson", "spearman", "kendall"] or callable(method): | ||
return nanops.nancorr( | ||
this.values, other.values, method=method, min_periods=min_periods | ||
this_values, other_values, method=method, min_periods=min_periods | ||
) | ||
|
||
raise ValueError( | ||
|
@@ -2702,8 +2705,10 @@ def cov( | |
this, other = self.align(other, join="inner", copy=False) | ||
if len(this) == 0: | ||
return np.nan | ||
this_values = np.asarray(this.values) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we use ._values here? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes I think so. For my knowledge, if we need an ndarray for There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. its probably harmless in this case but in general .values can be either a) lossy (dt64tz) or b) convert to object (period) while ._values is safe from those failure modes There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. got it, thanks! I'll update just in case There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. values also creates references and is read_only (in the future) for CoW. Better to use ._values |
||
other_values = np.asarray(other.values) | ||
return nanops.nancov( | ||
this.values, other.values, min_periods=min_periods, ddof=ddof | ||
this_values, other_values, min_periods=min_periods, ddof=ddof | ||
) | ||
|
||
@doc( | ||
|
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -42,13 +42,14 @@ def test_cov(self, datetime_series): | |||
assert isna(ts1.cov(ts2, min_periods=12)) | ||||
|
||||
@pytest.mark.parametrize("test_ddof", [None, 0, 1, 2, 3]) | ||||
def test_cov_ddof(self, test_ddof): | ||||
@pytest.mark.parametrize("dtype", ["float64", "Float64"]) | ||||
def test_cov_ddof(self, test_ddof, dtype): | ||||
# GH#34611 | ||||
np_array1 = np.random.rand(10) | ||||
np_array2 = np.random.rand(10) | ||||
|
||||
s1 = Series(np_array1) | ||||
s2 = Series(np_array2) | ||||
s1 = Series(np_array1, dtype=dtype) | ||||
s2 = Series(np_array2, dtype=dtype) | ||||
|
||||
result = s1.cov(s2, ddof=test_ddof) | ||||
expected = np.cov(np_array1, np_array2, ddof=test_ddof)[0][1] | ||||
|
@@ -57,9 +58,12 @@ def test_cov_ddof(self, test_ddof): | |||
|
||||
class TestSeriesCorr: | ||||
@td.skip_if_no_scipy | ||||
def test_corr(self, datetime_series): | ||||
@pytest.mark.parametrize("dtype", ["float64", "Float64"]) | ||||
def test_corr(self, datetime_series, dtype): | ||||
from scipy import stats | ||||
|
||||
datetime_series = datetime_series.astype(dtype) | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what happens here if there are NAs? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Both left and right are filtered for Line 1600 in 95a087d
so this works:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. cool. what about if there is an nan? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you mean an nan within a masked array? e.g.
It works either way since the |
||||
|
||||
# full overlap | ||||
tm.assert_almost_equal(datetime_series.corr(datetime_series), 1) | ||||
|
||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you move to 2.1 as well?