Adjust Bool evaluation for empty DataFrames to match PEP8

[tstriker]

Right now, as per the gotchas section, pandas raises a value error on an empty frame
http://pandas.pydata.org/pandas-docs/dev/gotchas.html#gotchas-truth

While documentation describes ambiguity, in the python world there is none.
PEP8 (http://legacy.python.org/dev/peps/pep-0008/) recommends: "For sequences, (strings, lists, tuples), use the fact that empty sequences are false."

The check returns False for None and empty values for all iterables. Further, there are any (https://docs.python.org/2/library/functions.html#any) and all(https://docs.python.org/2/library/functions.html#all) functions for the proposed ambiguity checks.

As currently it's just raising a ValueError, fixing the problem to correctly evaluate representing DataFrame.empty would be backwards compatible.

The check becomes important when working in a mixed environment with pandas and other structures - a simple if foo can become as convoluted as
empty = foo.empty() if (foo is not None and isinstance(foo, pd.DataFrame)) else not foo

[jreback]

You can simply do this (this would need an adjustement to handle scalars)

handling a mixed None and even lists/scalars is a bit non-trivial in python alone
e.g. len(None) is not defined

In [47]: def f(x):
   ....:     return x is not None and bool(len(x))
   ....: 

In [48]: f(None)
Out[48]: False

In [49]: f([])
Out[49]: False

In [50]: f(DataFrame())
Out[50]: False

In [51]: f([True])
Out[51]: True

In [52]: f([False])
Out[52]: True

In [53]: f(DataFrame(True,index=range(2),columns=range(2)))
Out[53]: True

I am not sure why you are confused about the ambiguity. Lists/tuples/dicts dont' care about the contents so by definition their isn't a problem. However pandas objects (and numpy objects) are simply different; the DO care about the contents and act on them.

[tstriker]

I'm not confused about ambiguity - i'm saying that there is not ambiguity if you approach the problem from the perspective of python - the default check in python is for None/empty, while the other two (.any and .all) are to be provided explicitly.

To your point - handling a mixed None and even lists/scalars in python is trivial in python - it's if foo as [None, [], (,), {}, False] all evaluate to False.

I'm arguing that providing a sane default that matches the behavior of the environment would make it more standard in the pythonic sense.

Finally - it's not objects that care about data - it's the programmers - pandas and numpy sure have special properties, but from my perspective they all can be boiled down to python primitives and unless there is strong reason, on the basic level should also behave as such.

[jreback]

you just made my point, these are NOT python primitives, in fact they are much more feature rich. Trying to treat a frame (or a numpy array) as a list is meaningless. Yes it sometimes can act like a list, but that doesn't mean it IS one. Duck typing has its limits.

so are you also advocating that numpy change this?

[tstriker]

Eesh, i guess i've gotten myself into trouble.

In short - I'm using pandas for a web app, and it just trips me personally up every time when i do an "if foo" in the templates and then pandas is waving hands and claiming that it doesn't know what i'm talking about. The next steps then are plugging in .empty and then making sure that the object is always defined as DataFrame, as it would fail when the code branch that sets it from None would not execute, etc. Your suggested function would solve the problem, but shortly we would find ourselves following the magic of a cargo cult calling the f function for all the things.

The benefit of allowing simple evaluation would be that of simplicity of code.
As for downsides - i don't see any - so maybe that's a valid point for investigation - is there really ambiguity, considering how the environment treats empty objects? Or is it inertia?

[jreback]

@tbaugis you have a valid point, but I think its a combination of inertia (back compat), and refusing to guess

Here is the change issue with lots of sub-issues and commentary
#4657

if you want to review and come back with comments. Always like to hear a view.

[immerrr]

@tbaugis considering numpy, there is some danger factor given how it can broadcast a single False value to a full container which will lose its "falseness". It is not an ambiguity per se, but you're in for a lot of surprises if you one day decide to replace a single constant with per-element one and don't have this particular scenario well tested (see #6966 for example).

Pandas containers are not so malleable, so there's less to be afraid of, but consistency with numpy is a plus.

As for the magical f function, I'm sticking to using empty lists as non-existent DataFrames and then len(self.data) > 0 check works in both cases.

[armaganthis3]

The OP is about Empty dataframes. As mentioned all empty python sequences give false. So are empty numpy arrays:

In [320]: np.empty(shape=[0])
Out[320]: array([], dtype=float64)

In [321]: bool(np.empty(shape=[0]))
Out[321]: False

In [322]: bool(np.empty(shape=[0,0]))
Out[322]: False

Numpy will not broadcast on an dimensions which have zero elements so what @immerrr
notes doesn't seem ralevent (correct me if I'm wrong). Furthermore this was broken by a fix for #4633 about binary boolean operators between NDframes, but none of the examples in that issues have to do with empty frames or numpy arrays. They are all about binary boolean operators between containers having at least some data.

#4657 by @jreback that changed this itself broke long existing behaviuor, and did it despite objections from users (@jseabold of statsmodels was one ) and actually overruled the original call wes mckiney made on this specific issue. Especially when considering the amount of breaking changes that have been going into master since 0.12.1 and for the most threadbare reasons, the "concerns" about backward-compatibility seem kinda hypocritical.

What's an actual good reason why bool(pd.DataFrame()) shouldn't be false like everything else in python?

[jreback]

@armaganthis3

The changes were all about consistency; @jseabold objection had to do with IMHO a numpy inconcistency. Pandas actually hasn't changed AT ALL, with the exception of the single-element case.
so numpy treats the 0,1 element cases differently from all others.

In [18]: bool(np.array([]))
Out[18]: False

In [19]: bool(np.array([True]))
Out[19]: True

In [20]: bool(np.array([False]))
Out[20]: False

In [21]: bool(np.array([False,False]))
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

In [22]: bool(np.array([True,True]))
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Pandas is entirely consistent

In [23]: bool(pd.DataFrame())
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

In [24]: bool(pd.DataFrame([[True]]))
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

In [25]: bool(pd.DataFrame([[False]]))
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

[armaganthis3]

... and we're back to discussing non-empty dataframes again? The OP put it in the title, discussed it in his description and I hyper-emphasized the point again in my comment. The question is about empty dataframes.

Consistency. By which you mean doing the same thing whether it makes sense or not? Even if by doing so it's inconsistent with every other python data type out there? That's just bizzare.

what's the point of this "consistency"? whom does it benefit?

the OP, discussions in previous issue, the ml discussion and the PEP are all explicit about
the obvious fact that bool(empty)=false is convenient. It's certainly pythonic by acclamation. how is all that trumped by some bogus definition of "consistency"? what precisely is ambiguous about the truth value of an empty dataframe as the exception message suggests?

[jreback]

so you think that bool(DataFrame()) == False but if its non empty you get a raise is better? how is that?

if I have an if statement I don't want it working when the test is False, but raising when it is True. *that makes no sense.

That would be inconsistent

[tstriker]

If it's nonempty, you would get a True, irregardless of the contents of the frame. The bool check would be evaluating if the DataFrame (or other sequences for that matter, unless they are generators) is empty or not.

@armaganthis3 raised a good point, that the issue is not with consistency, but rather matching expectations (pep8).

I ran through the archive of the pointed out issue (thanks for the link!) and the discussion (https://groups.google.com/forum/#!topic/pydata/XzSHSLlTSZ8) seems to have flushed the baby (empty checks) with the bathwater (evaluating dataframe contents to determine the bool value), not considering the option to leave empty checks as the sane default.

[armaganthis3]

Yes, That's exactly what I think. So does the OP. So did the numpy developers.
So did several users on the ml/comments in the other issues.

I have no idea where you came up with this definition of consistency, and I sure don't know
why you're clinging to it with so very little to back it up with except some personal aesthetic
you personally hold.

Who says a method can't return a value when it's appropriate and raise otherwise? that's
really... common, come to think of it.

a method that always raises whenever it's called regardless of circumstances?
yup, that sure is consistent. also useless.

[tstriker]

@armaganthis3 btw, couldn't not notice that you have created a github burner account - while you are certainly helping with argumentation, i'd beg you to stay constructive if possible :)

[jreback]

alright,

@armaganthis3

so let' have a PR from you that fixes the issue (pretty trivial), but changes the documentation with an explanation as well

look forward to it

[jreback]

So to repeat @jseabold points (which I think are very rational), and were addressed, correctly me if I am wrong

Behavior and reasoning: 

1. Empty series raises. Maybe you screwed up your index? What is the 
'correct' output of this? 

if pd.isnull(pd.DataFrame([])): 
  print 'this dataframe has no missing values?' 

This seems ambiguous. You can't answer the question because there's no 
information to evaluate the statement. 

2. 1 element is fine. You know what you're doing, carry on. Also 
.all() == .any() in this case, so it's not ambiguous. 

3. Length > 1 raises . This is ambiguous. Ask for all, any, or empty. 
Maybe you screwed up your index? 

Skipper