Read from HDF with empty `where` throws an error

[BeforeFlight]

Code Sample

df = pd.DataFrame(np.random.rand(4,4))

where = ''
with pd.HDFStore('test.h5') as store:
    store.put('df', df, 't')
    store.select('df', where = where)

Problem description

Wanted to be able construct "by hands" and save where condition for later, so declare it as variable. But some times constructed where becomes empty and code throws an error.

Traceback (most recent call last):

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-101-48181c3b59fb>", line 6, in <module>
    store.select('df', where = where)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 740, in select
    return it.get_result()

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 1518, in get_result
    results = self.func(self.start, self.stop, where)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 733, in func
    columns=columns)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 4254, in read
    if not self.read_axes(where=where, **kwargs):

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 3443, in read_axes
    self.selection = Selection(self, where=where, **kwargs)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 4815, in __init__
    self.terms = self.generate(where)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 4828, in generate
    return Expr(where, queryables=q, encoding=self.table.encoding)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/core/computation/pytables.py", line 548, in __init__
    self.terms = self.parse()

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/core/computation/expr.py", line 766, in parse
    return self._visitor.visit(self.expr)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/core/computation/expr.py", line 331, in visit
    return visitor(node, **kwargs)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/core/computation/expr.py", line 335, in visit_Module
    raise SyntaxError('only a single expression is allowed')

  File "<string>", line unknown
SyntaxError: only a single expression is allowed

Expected Output

When empty string is passed to where - just select whole DataFrame. It may be easily achieved by changing last statement to store.select('df', where = where if where else None). But it would be better to add this checking inside pandas, so user may not worry about it all the times using selection from HDF with where.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.7.3.final.0 python-bits: 64 OS: Linux OS-release: 5.0.0-16-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 4.5.0
pip: 19.1.1
setuptools: 41.0.1
Cython: 0.29.7
numpy: 1.16.3
scipy: 1.2.1
pyarrow: None
xarray: 0.12.1
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.8.0
pytz: 2019.1
blosc: None
bottleneck: None
tables: 3.5.1
numexpr: 2.6.9
feather: None
matplotlib: 3.0.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10.1
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

[WillAyd]

Where is documented as accepting a list so if you use an empty list instead of the string you should be able to manage this the way you want

[BeforeFlight]

In API reference it is stated it accepts list, yes. But in user_guide all examples are with using where as string. And also user_guide states: "If a list/tuple of expressions is passed they will be combined via &". The latter may become problem if one would create empty where = [], and starts to populate it with conditions - all of them will be forced to be combined via '&' (not '|' as may be wished). So in this case it would be ended to amending single condition inside where = [condition] list.

But anyway even here problem is the same. If where will ends up as empty list after all processing:

df = pd.DataFrame(np.random.rand(4,4))

where = []
with pd.HDFStore('test.h5') as store:
    store.put('df', df, 't')
    store.select('df', where = where)

Same error will be raised:

Traceback (most recent call last):

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-90-507edb4b117e>", line 6, in <module>
    store.select('df', where = where)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 740, in select
    return it.get_result()

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 1518, in get_result
    results = self.func(self.start, self.stop, where)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 733, in func
    columns=columns)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 4254, in read
    if not self.read_axes(where=where, **kwargs):

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 3443, in read_axes
    self.selection = Selection(self, where=where, **kwargs)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 4815, in __init__
    self.terms = self.generate(where)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/io/pytables.py", line 4828, in generate
    return Expr(where, queryables=q, encoding=self.table.encoding)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/core/computation/pytables.py", line 548, in __init__
    self.terms = self.parse()

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/core/computation/expr.py", line 766, in parse
    return self._visitor.visit(self.expr)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/core/computation/expr.py", line 331, in visit
    return visitor(node, **kwargs)

  File "/home/beforeflight/Coding/Python/_venvs_/main/lib/python3.7/site-packages/pandas/core/computation/expr.py", line 335, in visit_Module
    raise SyntaxError('only a single expression is allowed')

  File "<string>", line unknown
SyntaxError: only a single expression is allowed

[WillAyd]

Thanks for the additional references. If you'd like to take a look and clean up implementation / documentation PRs would certainly be welcome!

[TomAugspurger]

To make sure I understand, the proposed fix is for where=[] to be treaded the same as where=None, i.e. no filtering?

[BeforeFlight]

@TomAugspurger, if you are asking me, yes I think it should be that way. Empty where=[] -> no filtering -> whole df will be return.

[TomAugspurger]

Sounds right. Can you submit a PR with tests?

…

On Jun 3, 2019, at 17:17, BeforeFlight ***@***.***> wrote: If you are asking me, yes I think it should be that way. Empty where=[] -> no filtering -> whole df will be return. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[BeforeFlight]

@TomAugspurger as I explained in neighbour theme with groupby issue - just don't know how to do it correctly.

[WillAyd]

@BeforeFlight we have a contributing guide which could be helpful:

https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#testing-with-continuous-integration

If you would like to try but run into specific issues we are of course here to help. You can also use Gitter for development questions