automatic graph retries

[jtoman]

Retry policies

Retries happen at 2 levels:

• Transient, provider level errors. When these occur a retry policy will wait (with exponential backoff) and then try resuming the graph from exactly the last point at execution.
• "Wedged", "bad state" errors. These indicate the graph state is likely corrupted. When this occurs; the "fresh start" policy rebuilds a fresh input from the most recent state, mints a new thread id, and then kicks off again.

These policies are independent; you can have one but not the other or both at the same time. When both retry policies are installed, the transient provider exception catching is nested within the "bad state" error handler. In practice, this shouldn't matter as they are intended to catch different errors.

Further, "transient error" retry can be (and usually is) applied to every graph execution in a run of the application. The "retry bad state" handler is, necessarily, graph specific, as it involves reconstructing an input state I from the last run S.

Retry Policy and Subagents

If there is a global, run wide retry policy installed, a subagent spawned from within a tool node will first exhaust its retry policy. Once its retries are exhausted, it will bubble that exception up to the parent graph via its ToolNode. The provider level exception (propagated from the subgraph) will then hit the parent graph's retry policy. At this point, the parent graph will be retried from its most recent checkpoint; which is the tool node that spawns the subgraph. Importantly, (by convention) subagents are spawned with fresh thread ids each time, so latest attempt at the subagent will itself start with a fresh state. NB this can have a multiplicative effect on backoffs if it turns out we are spending 2 hours retrying runs that are lost causes we can revisit.

Policy installation

As stated above, the transient retry policy is "run global", and is intended to be installed during app startup. The infrastructure is resilient to it not existing, without the context var set, all runs of the graph run without a retry policy.

Individual graph runs can, optionally, install their own transient retry handler or their own "bad state", fresh retry handler, or both. It is strongly expected that only a handful of "critical" agents will opt into the fresh retry handling, and no one will ever override the run global retry policy.

No one currently uses the state restart functionality but it is tested.