Maximum recursion depth exceeded in comparison error

[github392]

I have this definition

@state_trigger("sensor.grid_consumed_energy_hour")
def accumulate_purchased_cost_daily(value,old_value):
    log.info(f"got arguments {value} {sensor.electricity_price}"  )
    temp = float(input_number.accumulate_purchased_cost) \
        + (float(sensor.electricity_price)*(abs(float(value)-float(old_value))))
    state.set('input_number.accumulate_purchased_cost',temp)

This works for a while but after 1-2h I get "maximum recursion depth exceeded in comparison". I can't find the place where I have introduced a recursion call.

I have also tried

 input_number.accumulate_purchased_cost = float(input_number.accumulate_purchased_cost) \
      + (float(sensor.electricity_price)*(abs(float(value)-float(old_value))))

but the result is the same. Any insight what I have done wrong is appreciated.

Pyscript version: 0.32

debug log:

2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.state] state.update({'sensor.grid_consumed_energy_hour': '0.90', 'sensor.grid_consumed_energy_hour.old': '0.89'}, {'trigger_type': 'state', 'var_name': 'sensor.grid_consumed_energy_hour', 'value': '0.90', 'old_value': '0.89'})
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.trigger] trigger file.energy.accumulate_purchased_cost_daily got state trigger, running action (kwargs = {'trigger_type': 'state', 'var_name': 'sensor.grid_consumed_energy_hour', 'value': '0.90', 'old_value': '0.89'})
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.trigger] trigger file.energy.accumulate_purchased_cost_daily waiting for state change or event
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling info("got arguments 0.90 1.059", {})
2020-11-09 18:40:00 INFO (MainThread) [custom_components.pyscript.file.energy.accumulate_purchased_cost_daily] got arguments 0.90 1.059
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling float("5.332900000000042", {})
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling float("1.059", {})
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling float("0.90", {})
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling float("0.89", {})
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling abs(0.010000000000000009, {})
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling set("input_number.accumulate_purchased_cost", 5.343490000000042, {})
2020-11-09 18:40:00 DEBUG (MainThread) [custom_components.pyscript.state] setting input_number.accumulate_purchased_cost = 5.343490000000042, attr = {'initial': None, 'editable': False, 'min': 0.0, 'max': 10000000.0, 'step': 1.0, 'mode': 'slider', 'unit_of_measurement': 'SEK', 'icon': 'mdi:cash'}
2020-11-09 18:40:00 ERROR (MainThread) [custom_components.pyscript.file.energy.accumulate_purchased_cost_daily] Exception in <file.energy.accumulate_purchased_cost_daily> line 14:
state.set('input_number.accumulate_purchased_cost',temp)
^
RecursionError: maximum recursion depth exceeded in comparison

[craigbarratt]

That's not good. I can't see a problem with the code you posted. However, there are ways recursion can happen, for example if you set a state variable (eg, input_number.accumulate_purchased_cost) that in turn causes a trigger that ends up causing that variable to be set again. But I would expect to see a lot of log messages before the recursion exception. It might be helpful to look at all your code; could you post to gist?

Also, in 0.32 there is a bug that the "trigger on any change" form of @state_trigger (like you are using for sensor.grid_consumed_energy_hour) will also trigger on any attribute change too. That's fixed in master. You could try the master version.

Or it could be a bug in pyscript. If so, trying to recreate the bug with as simple a test case as possible would be the goal. If you are willing to edit the code for pyscript, the first thing to do would be to get more information about where the internal error occurs. In custom_components/pyscript/eval.py add import traceback near the top of the file, and at the end of the format_exc function (here for 0.32), replace:

return mesg

with

return mesg + "\n" + traceback.format_exc(10)

[github392]

I did the change in the core py file and run test with both 0.32 and the new 1.0 and I get the same result but the trace shows where. I have put the definitions for the part that are included in this scenario. If we need more deep dive I can see if I can clone my HA instance and create a small test case.

input_number:
  accumulate_purchased_cost:
    min: 0
    max: 10000000
    icon: mdi:cash
    unit_of_measurement: "SEK" 

platform: integration
    source: sensor.real_time_consumption
    name: energy_grid_consumed
    unit_prefix: k
    unit_time: h
    round: 2
    method: left

utility_meter:
  # calculate hourly energy consumed from grid
  grid_consumed_energy_hour:
    source: sensor.energy_grid_consumed
    cycle: hourly

2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.state] state.update({'sensor.grid_consumed_energy_hour': '0.55', 'sensor.grid_consumed_energy_hour.old': '0.54'}, {'trigger_type': 'state', 'var_name': 'sensor.grid_consumed_energy_hour', 'value': '0.55', 'old_value': '0.54', 'context': Context(user_id=None, parent_id=None, id='77d06ca904b78bf1fb81c1aefe5f99d9')})
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.trigger] trigger file.energy.accumulate_purchased_cost_daily got state trigger, running action (kwargs = {'trigger_type': 'state', 'var_name': 'sensor.grid_consumed_energy_hour', 'value': '0.55', 'old_value': '0.54', 'context': Context(user_id=None, parent_id=None, id='77d06ca904b78bf1fb81c1aefe5f99d9')})
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.trigger] trigger file.energy.accumulate_purchased_cost_daily waiting for state change or event
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling info("got arguments 0.55 0.967", {})
2020-11-10 13:13:04 INFO (MainThread) [custom_components.pyscript.file.energy.accumulate_purchased_cost_daily] got arguments 0.55 0.967
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling float("31.681950000000104", {})
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling float("0.967", {})
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling float("0.55", {})
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling float("0.54", {})
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling abs(0.010000000000000009, {})
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.eval] file.energy.accumulate_purchased_cost_daily: calling set("input_number.accumulate_purchased_cost", 31.691620000000103, {})
2020-11-10 13:13:04 DEBUG (MainThread) [custom_components.pyscript.state] setting input_number.accumulate_purchased_cost = 31.691620000000103, attr = {'initial': None, 'editable': False, 'min': 0.0, 'max': 10000000.0, 'step': 1.0, 'mode': 'slider', 'unit_of_measurement': 'SEK', 'icon': 'mdi:cash'}
2020-11-10 13:13:04 ERROR (MainThread) [custom_components.pyscript.file.energy.accumulate_purchased_cost_daily] Exception in <file.energy.accumulate_purchased_cost_daily> line 14:
state.set('input_number.accumulate_purchased_cost',temp)
^
RecursionError: maximum recursion depth exceeded in comparison
Traceback (most recent call last):
File "/config/custom_components/pyscript/eval.py", line 802, in aeval
val = await getattr(self, name, self.ast_not_implemented)(arg)
File "/config/custom_components/pyscript/eval.py", line 1691, in ast_call
return await self.call_func(func, func_name, *args, **kwargs)
File "/config/custom_components/pyscript/eval.py", line 1711, in call_func
return await func(*args, **kwargs)
File "/config/custom_components/pyscript/state.py", line 190, in set
cls.hass.states.async_set(var_name, value, new_attributes, context=context)
File "/usr/src/homeassistant/homeassistant/core.py", line 1174, in async_set
same_attr = old_state.attributes == MappingProxyType(attributes)
RecursionError: maximum recursion depth exceeded in comparison

[dlashua]

This won't likely fix the issue, but the debugging data it provides might help resolve this.

Instead of using state.set on the input_number, can you use the service call designed for updating an input_number?

In PyScript 1.0.0, do this:

# state.set('input_number.accumulate_purchased_cost',temp)
input_number.accumulate_purchased_cost.set_value(value=temp)

[craigbarratt]

Thanks for the trace information. The recursion error appears to be inside MappingProxyType in HASS core.py when it is called with the attributes from state.set(). I'm not familiar with MappingProxyType but I just read a bit about it.

This issue reports the same error from python_scripts. While it wasn't fixed or resolved, the workaround was to copy the attributes and use that copy when calling state.set. However, from the log message the attributes look benign (I was thinking there could be some strange circular reference in the attributes):

setting input_number.accumulate_purchased_cost = 31.691620000000103, attr = {'initial': None, 'editable': False, 'min': 0.0, 'max': 10000000.0, 'step': 1.0, 'mode': 'slider', 'unit_of_measurement': 'SEK', 'icon': 'mdi:cash'}

As @dlashua points out, the better way to set an input_number value is via a service call, rather than setting the entity_id directly. In 0.32 it would just be this service call:

input_number.set_value(entity_id="input_number.accumulate_purchased_cost", value=temp)

This works in 1.0.0 too, and as @dlashua notes, in 1.0.0 you can also use the new short-form service-as-method call instead. The input_number service call will also do error checking on the value against min and max.

Setting aside the service call workaround, I don't yet see what attributes can cause MappingProxyType to get a recursion error. Based on the issue referenced above, it is possible this fix to state.py will solve the problem, although I don't yet know why. For 1.0.0, it's line 174; replace this:

new_attributes = state_value.attributes

with this:

new_attributes = state_value.attributes.copy()

For 0.32, it's at line 108.

[github392]

Just a short feedback. input_number.accumulate_purchased_cost.set_value(value=temp) works ok. It has now running for more then 16h without any issue. I will do the change above and rerun the old way and see what will happen.

[github392]

I have now done the change in state.py and rerun the test and below statements now works ok.

input_number.accumulate_purchased_cost = temp
state.set('input_number.accumulate_purchased_cost',temp)

So with this change it look like everything works as expected. Which statement is the preferred one?

[craigbarratt]

Either should be fine, but for input_number it's better to use the service call instead.

Thanks for checking that the attribute copy avoids the issue. I don't know what the root cause is. It's some sort of interaction between HASS and Python related to MappingProxyType acting on the attributes, and from the issue above, people have experienced it with python_scripts too. I looked at the Python source code (I'm not familiar with Python internals), and tried some tests, but wasn't able to replicate the problem or even see how it could happen.

I'll commit a fix to add the attribute copy.