Dataframe.query uses numexpr even when 'compute.use_numexpr' is False

[teto]

Code Sample, a copy-pastable example if possible

# Your code here
import pandas as pd
import io

content = io.StringIO('''
time,val
212.23, 32
''')

pd.set_option('compute.use_numexpr', False)
print("use numexpr? %d" % pd.get_option('compute.use_numexpr'))

df = pd.read_csv(
    content,
    sep=',',
    usecols=['time', 'val'],
    dtype= {'val': 'Int64'},
    # parse_dates=date_cols,
)
q = "val == 32"
# passing `engine="python"` fixes it
df_dest = df.query(q, )

throws with pandas 1.0

Traceback (most recent call last):
  File "test.py", line 21, in <module>
    df_dest = df.query(q, )
  File "/nix/store/0yyjswv58jk0fv1miyklmppvpd0p45sy-python3.7-pandas-1.0.1/lib/python3.7/sit
e-packages/pandas/core/frame.py", line 3231, in query
    res = self.eval(expr, **kwargs)
  File "/nix/store/0yyjswv58jk0fv1miyklmppvpd0p45sy-python3.7-pandas-1.0.1/lib/python3.7/sit
e-packages/pandas/core/frame.py", line 3346, in eval
    return _eval(expr, inplace=inplace, **kwargs)
  File "/nix/store/0yyjswv58jk0fv1miyklmppvpd0p45sy-python3.7-pandas-1.0.1/lib/python3.7/sit
e-packages/pandas/core/computation/eval.py", line 337, in eval
    ret = eng_inst.evaluate()
  File "/nix/store/0yyjswv58jk0fv1miyklmppvpd0p45sy-python3.7-pandas-1.0.1/lib/python3.7/sit
e-packages/pandas/core/computation/engines.py", line 73, in evaluate
    res = self._evaluate()
  File "/nix/store/0yyjswv58jk0fv1miyklmppvpd0p45sy-python3.7-pandas-1.0.1/lib/python3.7/sit
e-packages/pandas/core/computation/engines.py", line 114, in _evaluate
    return ne.evaluate(s, local_dict=scope)
  File "/nix/store/k6wr7w16q0aq9mfkf0w15wxlhx76sxqc-python3.7-numexpr-2.7.1/lib/python3.7/si
te-packages/numexpr/necompiler.py", line 822, in evaluate
    zip(names, arguments)]
  File "/nix/store/k6wr7w16q0aq9mfkf0w15wxlhx76sxqc-python3.7-numexpr-2.7.1/lib/python3.7/si
te-packages/numexpr/necompiler.py", line 821, in <listcomp>
    signature = [(name, getType(arg)) for (name, arg) in
  File "/nix/store/k6wr7w16q0aq9mfkf0w15wxlhx76sxqc-python3.7-numexpr-2.7.1/lib/python3.7/si
te-packages/numexpr/necompiler.py", line 703, in getType
    raise ValueError("unknown type %s" % a.dtype.name)
ValueError: unknown type object

Problem description

Dataframe.query fails when using UInt* dtypes see #25369 (comment).
My first idea to solve this was to disable numexpr but this isn't enough, one has to pass engine="python" to forcefully disable numexpr when calling query

[glemaitre]

I stumbled into this issue as well. I am adding a minimal example which can be used as a non-regression test:

import pandas as pd
pd.set_option("compute.use_numexpr", False)
df = pd.DataFrame(
    {"A": [True, False, True, False, None, None],
     "B": [1, 2, 3, 4, 5, 6]}
)
df.query('A.isnull()')

[glemaitre]

After checking #33006 I am not sure what is the right fix and what was the initial semantic behind compute.use_numexpr. The doc is quite general regarding the impact of the option. However, maybe it was intended for evaluate only. This said it would be really handy to have an option to force the engine globally for the query method.

@jreback Since you commented on the PR, maybe you can have some insights.

[quangngd]

I stumbled into this issue as well. I am adding a minimal example which can be used as a non-regression test:

import pandas as pd
pd.set_option("compute.use_numexpr", False)
df = pd.DataFrame(
    {"A": [True, False, True, False, None, None],
     "B": [1, 2, 3, 4, 5, 6]}
)
df.query('A.isnull()')

@glemaitre
This test could be fixed by either fixing this issue or #25369.
If the dtype bug is fixed (numexpr works), the test could silently pass but this issue still remains, aka numexpr is still being used despite pd.set_option("compute.use_numexpr", False) has been set.

[rhshadrach]

The following demonstrates issue and raises on master:

from pandas.core.computation.eval import _check_engine
with pd.option_context('compute.use_numexpr', False):
    result = _check_engine(None)
    assert result == 'python'

Can resolve this as in #33006 with the above code as a test.

[yashhd]

Hello, I'm Yash and I've been asked to contribute to an open source project for my Software Development course. I'd be interested to work on this issue and it'd be great if I could be assigned to this. Thank you.

[jreback]

you can assign yourself with /take

[yashhd]

@jreback I tried doing that but I'm still unable to assign myself.

[jreback]

see https://pandas.pydata.org/docs/development/contributing.html#where-to-start

[yashhd]

@jreback Thank you so much. I'll have a look.

[yashhd]

take