API: Index/Series/DataFrame op 1-d list-like coercion

[sinhrks]

xref #1134. There is another inconsistencies related to op. Index/Series/DataFrame op can accept 1-d list-like as input and coercing to Index/Series. However, supported 1-d list-likes differ depending on the class and kind of op.

Code Sample, a copy-pastable example if possible

# Series + equal length list, OK
pd.Series([1, 2, 3]) + [2, 2, 2]
#0    3
#1    4
#2    5
# dtype: int64

# Series + equal length Index, OK
pd.Series([1, 2, 3]) + pd.Index([2, 2, 2])
#0    3
#1    4
#2    5
# dtype: int64

# DataFrame + equal length list, OK
pd.DataFrame([[1, 2, 3]]) + [2, 2, 2]
#    0  1  2
#0  3  4  5

# DataFrame + equal length Index, NG
pd.DataFrame([[1, 2, 3]]) + pd.Index([2, 2, 2])
# ValueError: cannot evaluate a numeric op with unequal lengths

I've organize the result once as the below table. I think all these must be supported and consistent.

Class	op	list	tuple	ndarray(1dim)	Index
Index	Arithmetic	x	x	o	o
Index	Comparison	o	o	o	o
Index	Boolean	o	o	o	o
Series	Arithmetic	o	o	o	o
Series	Comparison	o	o	o	o
Series	Boolean	o	o	o	x
Frame	Arithmetic	o	o	o	x
Frame	Comparison	o	o	o	o
Frame	Boolean	o	o	o	x

NOTE: Index result may depends on its type.

output of pd.show_versions()

0.18.1

[jorisvandenbossche]

@sinhrks Nice overview! It are the 'x' that fail?
But Series boolean op with Index is indicated with 'x', but seems to work:

In [42]: pd.Series([1, 2, 3]) == pd.Index([2, 2, 2])
Out[42]:
0    False
1     True
2    False
dtype: bool

I agree these should probably all work consistently

[sinhrks]

@jorisvandenbossche I meant boolean op with logical op. Though using bool Index is not very effective, it should be consistent (or should output understandable error at least).

pd.Series([True, True, False]) & pd.Index([True, False, True])
# TypeError: cannot compare a dtyped [bool] array with a scalar of type [Index]

[jorisvandenbossche]

@sinhrks Whoops, yes, == is of course 'comparison' :-)

[sinhrks]

Found 2 related issues which will be broken if we introduce the above rules:

#4576 Pandas 0.12 unexpected results when comparing dataframe to list or tuple

on current master: list comparison performed per columns

df = pd.DataFrame([[1, 2], [3, 4], [5, 6]])
df
#    0  1
# 0  1  2
# 1  3  4
# 2  5  6

df > [2, 2]
# ValueError: operands could not be broadcast together with shapes (2,3) (2,) 

df > [2, 2, 2]
#        0      1
# 0  False  False
# 1   True   True
# 2   True   True

after the change: list comparison performed performed per row

df > [2, 2]
       0      1
0  False  False
1   True   True
2   True   True

df > [2, 2, 2]
# ValueError

# ref for comparison: UNCHANGED by the fix
df > pd.Series([2, 2])
       0      1
0  False  False
1   True   True
2   True   True

#11339 Boolean comparison vs tuple fails in 0.17.0

on current master: comparison may be broadcasted regarding tuple as "single" element.

s = pd.Series([(1,1),(1,2)])
s
# 0    (1, 1)
# 1    (1, 2)
# dtype: object

s == (1, 1)
# 0     True
# 1    False
# dtype: bool

after the change: tuple will no longer be regarded as a "single" element. should be the same as == pd.Series([1, 1])

s == (1, 1)
# 0    False
# 1    False
# dtype: bool

# ref for comparison: UNCHANGED by the fix
s == pd.Series([1, 1])
# 0    False
# 1    False
# dtype: bool