ERR: better error reporting for missing numexpr

[twiecki]

From the example at http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.query.html:

from numpy.random import randn
from pandas import DataFrame
df = DataFrame(randn(10, 2), columns=list('ab'))
df.query('a > b')

gives me:

---------------------------------------------------------------------------
NameResolutionError                       Traceback (most recent call last)
<ipython-input-19-47040e53b0e7> in <module>()
      2 from pandas import DataFrame
      3 df = DataFrame(randn(10, 2), columns=list('ab'))
----> 4 df.query('a > b')

/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in query(self, expr, **kwargs)
   1778                              "query expression")
   1779 
-> 1780         res = self.eval(expr, **kwargs)
   1781 
   1782         try:

/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in eval(self, expr, **kwargs)
   1829         kwargs['local_dict'] = _ensure_scope(resolvers=resolvers, **kwargs)
   1830         kwargs['target'] = self
-> 1831         return _eval(expr, **kwargs)
   1832 
   1833     def _slice(self, slobj, axis=0, raise_on_error=False, typ=None):

/usr/local/lib/python2.7/dist-packages/pandas/computation/eval.pyc in eval(expr, parser, engine, truediv, local_dict, global_dict, resolvers, level, target)
    206     eng = _engines[engine]
    207     eng_inst = eng(parsed_expr)
--> 208     ret = eng_inst.evaluate()
    209 
    210     # assign if needed

/usr/local/lib/python2.7/dist-packages/pandas/computation/engines.pyc in evaluate(self)
     48 
     49         # make sure no names in resolvers and locals/globals clash
---> 50         self.pre_evaluate()
     51         res = self._evaluate()
     52         return _reconstruct_object(self.result_type, res, self.aligned_axes,

/usr/local/lib/python2.7/dist-packages/pandas/computation/engines.pyc in pre_evaluate(self)
     31 
     32     def pre_evaluate(self):
---> 33         self.expr.check_name_clashes()
     34 
     35     def evaluate(self):

/usr/local/lib/python2.7/dist-packages/pandas/computation/expr.pyc in check_name_clashes(self)
    797         lcl_keys = frozenset(env.locals.keys()) & names
    798         gbl_keys = frozenset(env.globals.keys()) & names
--> 799         _check_disjoint_resolver_names(res_keys, lcl_keys, gbl_keys)
    800 
    801     def add_resolvers_to_locals(self):

/usr/local/lib/python2.7/dist-packages/pandas/computation/expr.pyc in _check_disjoint_resolver_names(resolver_keys, local_keys, global_keys)
     39     if res_locals:
     40         msg = "resolvers and locals overlap on names {0}".format(res_locals)
---> 41         raise NameResolutionError(msg)
     42 
     43     res_globals = list(com.intersection(resolver_keys, global_keys))

NameResolutionError: resolvers and locals overlap on names ['a']

This is with 0.13.

[jreback]

In [2]: df = DataFrame(randn(10, 2), columns=list('ab'))

In [3]: df.query('a > b')
Out[3]: 
          a         b
0 -1.300378 -2.176283
1  0.281861 -1.441128
5 -0.425845 -0.597904
6  2.110389  0.941799
7 -0.693335 -1.215670

[5 rows x 2 columns]

You must have a or b defined somewhere in scope (that is the meaning of the error)

[twiecki]

What do you mean by "in scope"? My understanding is that it will assume 'a' and 'b' are columns of the df. Do I have to have 'a' and 'b' defined as python variables somwhere?

[twiecki]

This is quite weird, I now purged pandas and reinstalled and now I get:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-3d30a42056f6> in <module>()
      1 df = pd.DataFrame(numpy.random.randn(10, 2), columns=list('ab'))
----> 2 df.query('a > b')

/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in query(self, expr, **kwargs)
   1778                              "query expression")
   1779 
-> 1780         res = self.eval(expr, **kwargs)
   1781 
   1782         try:

/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in eval(self, expr, **kwargs)
   1829         kwargs['local_dict'] = _ensure_scope(resolvers=resolvers, **kwargs)
   1830         kwargs['target'] = self
-> 1831         return _eval(expr, **kwargs)
   1832 
   1833     def _slice(self, slobj, axis=0, raise_on_error=False, typ=None):

/usr/local/lib/python2.7/dist-packages/pandas/computation/eval.pyc in eval(expr, parser, engine, truediv, local_dict, global_dict, resolvers, level, target)
    206     eng = _engines[engine]
    207     eng_inst = eng(parsed_expr)
--> 208     ret = eng_inst.evaluate()
    209 
    210     # assign if needed

/usr/local/lib/python2.7/dist-packages/pandas/computation/engines.pyc in evaluate(self)
     49         # make sure no names in resolvers and locals/globals clash
     50         self.pre_evaluate()
---> 51         res = self._evaluate()
     52         return _reconstruct_object(self.result_type, res, self.aligned_axes,
     53                                    self.expr.terms.return_type)

/usr/local/lib/python2.7/dist-packages/pandas/computation/engines.pyc in _evaluate(self)
     95 
     96         try:
---> 97             return ne.evaluate(s, local_dict=self.expr.env.locals,
     98                                global_dict=self.expr.env.globals,
     99                                truediv=self.expr.truediv)

AttributeError: 'module' object has no attribute 'evaluate'

/usr/local/lib/python2.7/dist-packages/pandas/computation/ops.py:62: DeprecationWarning: object.__new__() takes no parameters
  return supr_new(klass, name, env, side=side, encoding=encoding)
/usr/local/lib/python2.7/dist-packages/pandas/computation/ops.py:62: DeprecationWarning: object.__new__() takes no parameters
  return supr_new(klass, name, env, side=side, encoding=encoding)

[jreback]

what I mean is if a or b is defined as a local variable before this call it will complain

In [5]: a = 5

In [6]: In [4]: df.query('a > b')
NameResolutionError: resolvers and locals overlap on names ['a']

you can pass local_dict=dict() to avoid it looking at the scope at all

[twiecki]

Oh, I see. yes, that makes sense.

Although now I am getting the second error (AttributeError: 'module' object has no attribute 'evaluate') in a clean interpreter.

[cpcloud]

I wonder if this error message could be a bit more transparent. When I wrote this it made perfect sense, but now that 0.13 is out and query/eval will get some more face time I'm thinking that exposing the internal details of the scope mechanism is a bit rude. @jreback Thoughts?

[cpcloud]

Maybe something along the lines of,

Local variables and column names <list of clashes> overlap

[jreback]

maybe also have easy way to turn this off, e.g. local_dict=False or use_locals=False?

[jreback]

maybe should be off by default?

[cpcloud]

Resolvers is more general (since you don't have to use a DataFrame as the extra context for eval/query) but I think it's too much detail.

@twiecki As a side note, if you want to use the local variable a in your expression just throw an @ in front of it

a = 5
df.query('@a > b')

[cpcloud]

Hm like allow_locals? If False ignore locals always? Similar to R I believe

[twiecki]

Any thoughts on my current problem #5969 (comment)?

[jreback]

@twiecki make clean first. then build again..(just to be sure)

[twiecki]

I installed with pip install so should be clean no matter what, no?

[cpcloud]

@twiecki Can you show the output of

import pandas
pandas.util.print_versions()

I think that's where it is... @jreback pls correct me if I'm wrong