Best practices for long running tools?

[ostegm]

I'm building a few tools which require waiting on long running operations (10's of minutes or so), and I'm looking for suggestions on the best way to implement those in this framework.

One idea I have is to have the tool return a pointer that says, "Come back and check in in the future, and I can give you some intermediate results possibly."

Is there anything like this already built into the SDK or anything coming? ⁠

[rm-openai]

I am thinking about this actively, but I want to make sure to do it right - so it may take a couple of weeks. For now, your main options are:

1. Just block on the long running operation - easy, but bad for obvious reasons.
2. Return "operation in progress, will let you know when done" as the tool result. Asynchronously kickoff the real operation. When done, insert a user message into the input with the result. This is a hack but works ok.
3. Use something like temporal for durable execution

If you (or anyone) has suggestions on implementation, open to it!

[ostegm]

Thanks for the reply - its not clear what the right approach is. Tne idea I'm exploring right now:

1. Define the original tool (start_X) which starts an async job that can stream output or post events to a queue and returns an id.
2. Define a second tool (check_X) which takes the id from the output of the first tool.
3. Optionally define a third: (kill_X)which can stop it?

The LLM could be instructed to use the ID to check in on the progress and get the intermediate outputs while its progressing and report back to the user. It would allow the LLM to work on other things in parallel and keep the user appraised of the progress?

[rm-openai]

That could work - it totally depends on the UX you want. For example - Deep Research in ChatGPT kicks off a process, and sends occasional updates to the frontend to display to the user. But the user can't send messages while things are in progress.

On the other hand, if you wanted to allow the user to be able to keep chatting while the task ran, your approach sounds great to me.

[ostegm]

The part that I'm not sure how to implement with my approach is the case where you trigger this long-running task, the LLM responds to the user, and then the user walks away, which means the LLM never gets a chance to come check on the task progress. So you may never get a status update until the user comes back and triggers another interaction with the agent. ⁠

[rm-openai]

There's a few pieces there:

1. You're probably storing the conversation in a database. You should probably insert a new item into the DB when the task is complete. That way, when the user refreshes the page, things appear correctly.
2. You might also want to notify the user via push notifications, so they see it.

the LLM responds to the user, and then the user walks away, which means the LLM never gets a chance to come check on the task progress

I guess I'd say - the LLM should only check on the task progress for the user. As long as the user is still active, you can periodically check for updates and update the message history. If they walk away, then only update the history when the task is done.

(Sorry if this is abstract - I'm speaking in general terms since I don't know the specifics of your application)

[BenjaminChoou]

There's a few pieces there:

1. You're probably storing the conversation in a database. You should probably insert a new item into the DB when the task is complete. That way, when the user refreshes the page, things appear correctly.
2. You might also want to notify the user via push notifications, so they see it.

the LLM responds to the user, and then the user walks away, which means the LLM never gets a chance to come check on the task progress

I guess I'd say - the LLM should only check on the task progress for the user. As long as the user is still active, you can periodically check for updates and update the message history. If they walk away, then only update the history when the task is done.

(Sorry if this is abstract - I'm speaking in general terms since I don't know the specifics of your application)

@rm-openai Thanks for providing this idea. Recently, I am trying to use this SDK to build a voice agent. I encountered into similar case too, where:

• I have long running tools
• Voice agent should actively talk to users during long running tools.

According to examples given in SDK docs, the voice agent will not interact with the user until it gets the tool result.
I believe, this is a common case for people who are building voice agent. Alternatives could be an extra agent generating the filler message in between. However, this is not very straightforward in my opinion.

Could you kindly give any suggestions on this case?