BUG: fix HDFStore iterator to handle a where properly (GH8014)

[jreback]

closes #8014

[jreback]

cc @bboerner
give this a try

[bboerner]

@jreback, the fix looks good, thanks!

[jreback]

@bboerner ok this is fixed for a partial selection case.

note when you put up a fix, it should be on top of the current PR and only show the changes. e.g. just create whatever the changes are; these are local to you and don't modify anything else (unless they are pulled in).

[bboerner]

@jreback, thanks I verified the fix. And for the explanation about submitting fixes.

Several thoughts:

1. I think the current = current + self.chunksize should revert back to above the check for if value is None else if None the loop continues w/o updating current.
2. current = current + self.chunksize could be current = stop - same effect and IMHO clearer.
3. As currently (and previously in the PR) implemented this can return one (or possibly many more) empty DataFrames through the generator. concat([df for df in it]) handles that case but it doesn't seem natural to get back a large number of empty DataFrames from an iterator. That could be handled by modifying the check to be if value is None or value.empty: Thoughts? I can put together a test case if you want.

Regards.

[jreback]

2. easy

3. and 3) do you have something that breaks this?

None should never actually be returned (it will raise if something bad happens or return an empty frame), never None (I may have had that for some reason but don't remember).

I don't think this can return a list of empty frames, but just a list of a single empty frame

e.g. your selection is completely invalid, then you will get a single empty frame. (but should test this).

[jreback]

updated

[jreback]

actually I think its correct to yield empties if the query doesn't select anything (you have to remember the columns are returned correct). Otherwise this is odd I think. You should get back something for each iteration and empty is valid.

[bboerner]

Thanks.

1. I encountered the empty DataFrames a while back selecting ~10K rows from a file containing 1.74e7 rows and got back 1 non-empty DataFrame and 173 emptys. That was unexpected and not intuitive.
2. More troubling is that it breaks with Python idiom e.g.

for e in []:
    print True

never prints True so I suggest that for a where clause which selects no rows that:

for df in store.select('df',where=where,iterator=True):
    print True

should also not print True.

Commit bboerner/pandas@7fedf5b contains the test cases illustrating.

If you'd like me to open this as a seperate issue let me know.

Regards.

[jreback]

ok, updated. (changed 1 item in your test which seemed wrong, see my comment).

[bboerner]

Thanks, look good. Is there a performance preference for not len(value): vs. value.empty?

Regards.

[jreback]

doesn't make any difference
this is all dwarfed by the actual operation of selection

[bboerner]

Thanks.