Now boolean operators work with NDFrames?

[cpcloud]

Awkward side effect of the new and improved Series probably, I typed something wrong and was shocked to not get the usual numpy ValueError barf:

In [57]: a, b = Series(rand(10) > 0.5), Series(rand(10) > 0.5)

In [58]: a
Out[58]:
0     True
1     True
2    False
3    False
4     True
5    False
6    False
7     True
8     True
9     True
dtype: bool

In [59]: b
Out[59]:
0     True
1     True
2     True
3     True
4    False
5     True
6    False
7     True
8     True
9     True
dtype: bool

In [60]: a and b # what?! no way!!
Out[60]:
0     True
1     True
2     True
3     True
4    False
5     True
6    False
7     True
8     True
9     True
dtype: bool

In [61]: a & b # whew
Out[61]:
0     True
1     True
2    False
3    False
4    False
5    False
6    False
7     True
8     True
9     True
dtype: bool

[jreback]

In [11]: dfa, dfb = DataFrame(randn(10,2)),DataFrame(randn(10,2))

In [13]: dfa
Out[13]: 
          0         1
0  1.032736 -0.981451
1  0.154600  1.458225
2  0.688022  0.793732
3 -1.171007 -0.643027
4 -0.559731 -0.019199
5  1.079594 -0.549702
6 -1.803250  1.319622
7  0.602654  0.550604
8 -0.908063 -1.304829
9  0.103192  1.542427

In [14]: dfb
Out[14]: 
          0         1
0 -0.445965 -1.055636
1  0.929273  0.160086
2  0.302718 -0.653348
3 -1.227953 -1.627101
4  1.832395 -0.036561
5  0.806457  1.730636
6  1.805333 -0.299774
7  0.283562 -0.263815
8  0.749068  0.589955
9  0.523333 -2.085325

In [15]: dfa and dfb
Out[15]: 
          0         1
0 -0.445965 -1.055636
1  0.929273  0.160086
2  0.302718 -0.653348
3 -1.227953 -1.627101
4  1.832395 -0.036561
5  0.806457  1.730636
6  1.805333 -0.299774
7  0.283562 -0.263815
8  0.749068  0.589955
9  0.523333 -2.085325

[hayd]

This seems like a good thing. Now truthy-ness of Series/DataFrames is consistent, empty Series are Falsey.

In [6]: s1 = pd.Series([1])

In [7]: s_ = pd.Series()

In [8]: s_ or s
Out[8]: 
0    1
dtype: int64

Not sure I can use this anywhere, but I'll try.

[miketkelly]

Consistency aside, this seems likely to lead to lots of subtle bugs when people use df[a or b] when they mean df[a | b].

[jtratner]

Along the lines of @mtkni's comment : is it a good thing that Series behaves more like a Python container vs a numpy container?

[hayd]

Ew:

In [1]: bool(np.array([]))
Out[1]: False

In [2]: bool(np.array([1]))
Out[2]: True

In [3]: bool(np.array([1, 2]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-5ba97928842c> in <module>()
----> 1 bool(np.array([1, 2]))

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

That said, you're probably right that and/or should raise. :(

[kjordahl]

You also get fun things like this:

>>> a = Series(rand(10))
>>> bool(a == 0)
True
>>> bool(a == 1)
True

I think it should raise as well. It's doing numpy-like comparisons and broadcasting, not throwing an exception can allow subtle bugs to go undetected.

Possibly related: #1069 (old but apparently reverted), #4576, #4537

[jreback]

this like this I think are easily caught

I had to add a couple of coercion method which can handle these cases

eg bool , int etc

[kjordahl]

What's the use case for allowing bool(s) to return True for any series with nonzero length? It has never been the case in previous releases, since Series inherited from numpy.ndarray. This is new behavior, and IMO it does not add anything useful, only a source of bugs.

[jreback]

oh I agree

however we have a couple of tests that would fail otherwise so need to fix those

iIIRC there was a couple of places this is relied upon (which is just wrong)

@cpcloud pls mark this and we can take a look

[kjordahl]

OK, thanks.

It looks like raising ValueError for truth tests on DataFrames (which had been introduced in PR #1073) was reverted by PR #3696. I would argue that should be restored as well.

[jreback]

yeh these should all be consistent
I'll take alook tomorrow

[jreback]

In [17]: s = Series(randn(4))

In [18]: df = DataFrame(randn(10,2))

In [19]: s_empty = Series()

In [20]: df_empty = DataFrame()

This is from 0.12

In [21]: bool(s)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

In [22]: bool(s_empty)
Out[22]: False

In [23]: bool(df)
Out[23]: True

In [24]: bool(df_empty)
Out[24]: False

master

In [5]: bool(s)
Out[5]: True

In [6]: bool(s_empty)
Out[6]: False

In [7]: bool(df)
Out[7]: True

In [8]: bool(df_empty)
Out[8]: False

These are the same in 0.12 and master, e.g. a 1-element Series is special cased
to evaluate to return the boolean of a single value

In [9]: bool(Series([True]))
Out[9]: True

In [10]: bool(Series([False]))
Out[10]: False

I am going to argue that 0.12 is inconsistent and master is correct

However, one could have a point of view that ALL of these should raise

any thoughts?

[jreback]

@wesm ?

[kjordahl]

pandas 0.11.0:

>>> bool(s)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

>>> bool(s_empty)
False

>>> bool(df)
ValueError: Cannot call bool() on DataFrame.

>>> bool(df_empty)
ValueError: Cannot call bool() on DataFrame.

To me, it looks like this is the right behavior (with the possible exception of bool(df_empty)).