Feature Request: task.ensure_future or task.create_task

[dlashua]

I would like a way to say "run this other pyscript method without blocking execution of the current code path". Even better if this new method returns some kind of Future-like object that I can, later, get the result of, or cancel, or whatever. This method should take a pyscript method as an argument:

So I can, for instance do :

for value in range(4):
  task.create_task(report_a)

def report_a():
  task.unique('report_a')
  task.sleep(10)
  log.info('reporting a')

When running this, the log.info should only happen once and it should only sleep for 10-ish seconds.

Even better if I can pass arguments too: task.create_task(report_a, param1, param2, other="thing")

[craigbarratt]

The new functions seem to work. task.create() takes arguments as you suggest (although kwargs is not tested yet). task.wait() is just asyncio.wait() so it takes a list or set of tasks as a first argument, with optional timeout:

t = task.create(report_a, 10)
# do something else for a while, then...
done, pending = task.wait({t})
t.result()

Assuming t is done, t.result() is the function return value.

I haven't written the docs or added tests yet.

[dlashua]

Excellent! I should have a chance to test this later today.

[dlashua]

Limited testing shows this works perfectly.

However, there isn't an easy way to add a done callback to a task.

I was able to add this as a native python module to gain this functionality:

import asyncio

def task_done_callback(task, pyscript_func, *args, **kwargs):
    def task_done_callback_inner(f):
        loop = asyncio.get_running_loop()
        loop.create_task(pyscript_func(f, *args, **kwargs))

    task.add_done_callback(task_done_callback_inner)

It's a slight departure from the real add_done_callback, as it allows arbitrary args and kwargs as long as the task is the first argument. But, I think this is an improvement.

Used like this:

import sys
import importlib
if "/config/pyscript_modules" not in sys.path:
    sys.path.append("/config/pyscript_modules")

import done_callback
importlib.reload(done_callback)
task_add_done_callback = done_callback.task_done_callback

@time_trigger('startup')
def startup():
    log.info('starting')
    t1 = task.create(do_sleep, 1)
    t5 = task.create(do_sleep, 5)
    t10 = task.create(do_sleep, 10)
    task_add_done_callback(t1, report_done, 't1')
    task_add_done_callback(t5, report_done, 't5')
    task_add_done_callback(t10, report_done, 't10')
    task.sleep(2)
    log.info(f"t1 done? {t1.done()}")
    log.info(f"t5 done? {t5.done()}")
    log.info(f"t10 done? {t10.done()}")
    log.info('wait for t5')
    task.wait([t5])
    task.cancel(t10)
    log.info('done')


def report_done(f, name):
    if f.cancelled():
        log.info(f'{name} task was cancelled')
        return

    if f.exception() is not None:
        log.info(f'{name} task had exception {f.exception()}')
        return

    if f.done():
        log.info(f'{name} task completed with result {f.result()}')
        return

    log.info(f'{name} task is in unknown state')

def do_sleep(secs):
    log.info(f"Starting Sleep {secs}")
    task.sleep(secs)
    log.info(f"Finished Sleep {secs}")
    return secs

Perhaps we can add this as well? It'd be ideal if t10.add_done_callback() worked as is, but, it doesn't because it requires a sync method and all pyscript methods are async. I think syntax like this should be good enough:

t = task.create(whatever)
task.add_done_callback(t, somemethod, somearg1, somearg2, kwarg1=1, kwarg2=2)

def somemethod(f, a, b, kwarg1=None, kwarg2=None):
  whatever

[dlashua]

Just saw this Exception:

2020-12-11 06:08:00 ERROR (MainThread) [custom_components.pyscript.function] task_reaper: got exception Traceback (most recent call last):
  File "/config/custom_components/pyscript/function.py", line 97, in task_reaper
    await cmd[1]
RuntimeError: await wasn't used with future

It happened when I called pyscript.reload after running some test code using task.create() on methods that use task.unique() therefore resulting in cancelled tasks. I'm going to try to reproduce it now.

[dlashua]

I figured out where it's happening, I still don't know why.

task_reaper in function.py awaits each task after it cancels them. task_reaper itself is also in the task list (at least when pyscript.reload() is called in a pyscript). I believe it's awaiting a cancelled task that, for whatever reason, still hasn't cancelled. So another call to pysript.reload() fails because it's trying to cancel this already cancelling (but not yet cancelled) task.

I think.

I was able to get this far by logging what is being cancelled (in function.py around line 97). However, it's always run_coro which isn't very informative. So, I altered run_coro to set the task name to coro.__name__ which is where I was able to see task_reaper and that the task was cancelling instead of cancelled.

It's definitely the combination of my task_add_done_callback (above), task.unique(), and calling pyscript.reload() from in pyscript. Because if I take any one of those things out, the issue stops happening.

[dlashua]

This Exception was caused because I had a task running that was relying on task.unique in a shutdown function to kill it.

I changed the way that operated (instead of while True: I used while somevar is not None: and then had the shutdown function set somevar to None). This resolved the issue.

I'm not sure if my code is bad, or if there's something pyscript should be cleaning up that it isn't.

You can see the code here. Look for the word "Reaper" to see what I changed to make it work.

https://gist.github.com/dlashua/f7d88f9a5afdcf7af17ce24266925a0b

[dlashua]

I've been able to reproduce this. But it has nothing to do with task.create. Opening a new issue and closing this one.