Building durable agents with Temporal
tl;dr
Building reliable AI agents means handling failures, running tasks in parallel, waiting for human approval, and keeping long-running work alive. This article shares what I’ve learned about using Temporal for these tasks, with practical Go examples. It also covers worker discovery with Kyverno and scaling workers with KEDA to reduce idle resource use, along with the trade-offs to consider.
Let's start from the beginning
Distributed systems are not new. Early discussions go back to the mid-1960s. For example, Paul Baran’s paper On Distributed Communications explores distributed network communication, redundancy, and failover. Dijkstra's work on mutual exclusion also helped shape how we think about concurrency and coordinating processes.
Over time, these ideas developed into theory and then into practical systems. One example is the Network File System, released by Sun Microsystems in 1984. Systems like NFS brought challenges such as consistency, write durability, performance, and failure recovery into everyday use.
The need to solve these problems only grew as cloud services became the standard way to run modern systems. Today, it is relatively easy to set up resources across regions, but distributed transactions, locks, process synchronisation, and many other challenges remain.
Agentic era
For better or worse, we are entering a new chapter where personalised AI assistants, agentic flows, and automations play a growing part in our daily lives, work, and entertainment. I look at these changes mainly from an engineering perspective. The pace is extreme. We have quickly moved from “Ask on Stack Overflow” to “Ask ChatGPT” to “Run agentic loops”. Now we are building personalised AI assistants, harnesses, and remote coding agent platforms.
Building these systems well is not easy. Some agents run for hours, some actions need manual approval, and agent failures or network issues need to be handled gracefully. Preserving history and context also matters for reliability, token usage, and cost. Familiar distributed systems patterns, such as child workflows, parallel execution, and long-running activities, fit naturally into agentic systems. They are building blocks for durable workflows and what are often called durable agents.
This is how Restate, a durable runtime for agents, describes a durable workflow:
A durable workflow is a programming paradigm and architecture that lets long-running code execute reliably across crashes, server restarts, network failures, or human-in-the-loop pauses without losing its state or starting over from scratch
Durable workflows
There are many tools for building durable workflows, including workflows for agentic systems. The list keeps growing, and their capabilities keep changing. A few I have looked into are Apache Airflow, Temporal, Vercel Workflows, and Cloudflare Workflows. Each has its own features and trade-offs. Some work best within a specific ecosystem, such as Vercel or Cloudflare, while others offer self-hosting options and fit into existing infrastructure.
Most of my experience with durable workflows comes from using Temporal at Miro for a range of use cases. Along the way, I have learned a lot about its patterns and possible optimisations. Temporal is also used by other companies building agentic systems. For example, Lee Robinson highlights its use for Grok Bot, and Temporal offers an integration with OpenAI’s Agents SDK.
The Temporal documentation covers a wide range of patterns. I’ll focus on those commonly used in agentic systems and those I’ve worked with firsthand. For each pattern, I’ll link to a real implementation and explain its use case.
Child Workflows
Child Workflows split complex business logic into smaller, reusable units. Each child runs as an independent Workflow with its own ID, event history, and lifecycle. This gives you isolation, independent tracking, queries, timeouts, retries, and the option for a child to outlive its parent. Child Workflows can run synchronously, asynchronously, or only when certain conditions are met. This makes them a useful pattern for a parent agent working with sub-agents.
Agentic platforms use child workflows for several tasks. A common example is a parent agent coordinating sub-agents, each with its own context window. A session workflow can also start a SandboxLifecycle child workflow to provision a VM, keep it running, and tear it down. A PullRequest child workflow can wait for CI results and review comments even after the chat session ends.
Practical example: A user uploads a PDF and asks, “What are the main points in this document?” The conversation workflow handles the request and starts a DocumentAnalysisWorkflow child workflow. The child prepares and analyzes the document, then returns a result the main agent can use in its reply.
The child has its own workflow history, task queue, timeout, and processing budget, so progress and failures can be tracked separately from the conversation workflow.
for _, a := range s.Batch {
if a.Outcome == "authorized" {
if a.Scope.ToolName == "file_upload" {
// Pass the approved scope and processing budget to the child.
in := chat.DocumentAnalysisInput{
Scope: a.Scope,
Model: model,
Budget: chat.DefaultDocumentBudget(),
}
// Give the child its own ID, task queue, and execution limit.
child := workflow.WithChildOptions(
turnCtx,
workflow.ChildWorkflowOptions{
WorkflowID: in.ChildID(),
TaskQueue: WebDocumentsQueue,
WorkflowExecutionTimeout: 15 * time.Minute,
WaitForCancellation: true,
ParentClosePolicy: enumspb.PARENT_CLOSE_POLICY_REQUEST_CANCEL,
WorkflowIDReusePolicy: enumspb.WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE,
},
)
futures = append(futures, workflow.ExecuteChildWorkflow(
child,
DocumentAnalysisName,
in,
))
} else {
// Other approved tools run as activities.
futures = append(futures, workflow.ExecuteActivity(
workflow.WithActivityOptions(turnCtx, opts),
WebToolActivity,
chat.WebToolExecution{Scope: a.Scope},
))
}
}
}Sources: document_analysis.go · web_approval.go
Parallel Execution
Parallel execution lets a workflow start several independent Activities or Child Workflows before waiting for their results. This can reduce waiting time when one task does not need the output of another. Temporal records their progress, so the workflow can recover after a worker restart without losing track of the scheduled work.
Durable agents can use this pattern to fetch several web pages, analyze different parts of a document, or ask specialist agents to work on separate questions. Remote coding agents can run independent checks, such as linting, type checking, and tests, in parallel. Work that modifies the same files or shared state needs more care and may need to run sequentially.
Practical example: A user uploads a report and asks for a summary. After approval, the demo’s DocumentAnalysisWorkflow prepares the document for analysis. When the document contains multiple chunks, it starts a batch of AnalyzeDocumentChunk Activities without waiting for each one individually.
The batch size is limited by in.Budget.Parallelism. Once all Activities in the batch finish, the workflow moves to the next batch. This keeps the amount of scheduled work bounded instead of submitting every chunk at once. Actual execution speed still depends on worker capacity and the local model server.
for base := 0; base < result.Chunks; base += in.Budget.Parallelism {
if ctx.Err() != nil {
return result, ctx.Err()
}
// Schedule this batch before waiting for any results.
var futures []workflow.Future
for i := base; i < result.Chunks && i < base+in.Budget.Parallelism; i++ {
futures = append(futures, workflow.ExecuteActivity(
work,
"AnalyzeDocumentChunk",
chat.DocumentChunkInput{
Analysis: in,
Index: i,
},
))
}
// Wait for every activity, requesting cancellation on the first error.
var first error
for _, f := range futures {
if err := f.Get(join, nil); err != nil && first == nil {
first = err
cancel()
}
}
if first != nil {
return result, first
}
}ExecuteActivity returns a future, so the first inner loop schedules the whole batch without blocking on each Activity. The second loop waits for completion. If it encounters an error, it requests cancellation of the remaining work and still waits for all futures before returning.
Source: document_analysis.go
Approval
The Approval pattern lets a workflow wait for a person’s decision before continuing. Temporal Signals or Updates can deliver that decision without keeping an Activity running or repeatedly polling for an answer. The workflow can recover after a worker restart and continue waiting, subject to its deadline.
Durable agents can use approvals before fetching external content, sending messages, or changing data. Remote coding agents can ask before running a destructive command, pushing changes, or deploying an application. An approval should be tied to the specific action, so an old decision cannot accidentally authorize different work.
Practical example: A user asks to summarize a web page and provides its URL. When the model requests the web_fetch tool, the conversation workflow checks whether permission has already been granted for the conversation’s toolset. If not, it records an approval request and waits for the user’s decision before fetching the page.
if s.PendingApproval == nil {
a.Scope.PolicyRevision = s.Policy.Revision
// Bind the request to this turn, tool call, and policy revision.
s.PendingApproval = &chat.WebApproval{
ID: chat.LogicalID(
s.ActiveTurn,
"approval/"+a.Scope.ToolCallID+"/"+
strconv.FormatInt(s.Policy.Revision, 10),
),
Scope: a.Scope,
Deadline: workflow.Now(ctx).Add(24 * time.Hour),
}
}
p := s.PendingApproval
if p.Scope != a.Scope {
return temporal.NewNonRetryableApplicationError(
"unbound pending approval",
"InvalidRequest",
nil,
)
}
// Persist the request so the application can show it to the user.
s.Status = "awaiting_approval"
if err := workflow.ExecuteActivity(
webStorageContext(ctx),
WebApprovalActivity,
chat.WebApprovalRecord{Approval: *p},
).Get(ctx, nil); err != nil {
return err
}
// Wait for a decision, a policy change, or a workflow pause.
_, err := workflow.AwaitWithTimeout(
turnCtx,
p.Deadline.Sub(workflow.Now(ctx)),
func() bool {
return p.Decision != "" ||
p.Scope.PolicyRevision != s.Policy.Revision ||
pause()
},
)Source: web_approval.go
Long-Running Activity
Some agent operations take longer than a normal web request. A model may spend several minutes generating a response, or a tool may need time to process a large amount of data. Running this work as a Temporal Activity gives it explicit timeouts, retry rules, and a way to report that it is still alive.
Durable agents can use long-running Activities for model generation, document processing, or external jobs. Remote coding agents can use them to run a test suite, build a project, or wait for a sandbox operation to finish. Regular heartbeats help Temporal detect an interrupted worker and deliver cancellation requests.
Practical example: A user asks the demo a question that takes the local Ollama model a while to answer. Model generation runs inside a Temporal Activity, independently of the browser request. While it works, the activity sends heartbeats and the application can stream partial output to the user.
If the user presses Stop, the workflow requests cancellation. If the worker shuts down instead, the implementation treats that as a retryable interruption rather than a user cancellation. Before invoking the model on a retry, it checks whether a complete result has already been saved.
The conversation workflow gives each model attempt a 10-minute execution timeout and requires a heartbeat within 15 seconds. The total activity window, including queueing and retries, is limited to 25 minutes.
mctx := workflow.WithActivityOptions(
turnCtx,
workflow.ActivityOptions{
TaskQueue: WebLLMQueue,
StartToCloseTimeout: 10 * time.Minute,
ScheduleToCloseTimeout: 25 * time.Minute,
HeartbeatTimeout: 15 * time.Second,
WaitForCancellation: true,
RetryPolicy: &temporal.RetryPolicy{
MaximumAttempts: 2,
InitialInterval: time.Second,
NonRetryableErrorTypes: []string{
"InvalidRequest",
"ModelNotFound",
},
},
},
)
// Identify the model step being executed.
modelRef := ref
modelRef.Step = state.Step
var output chat.ModelStepResult
err := workflow.ExecuteActivity(
mctx,
WebModelActivity,
modelRef,
).Get(join, &output)WaitForCancellation makes the workflow wait for the activity’s cancellation to finish rather than immediately moving on when cancellation is requested. The surrounding implementation also tracks invocation cleanup before allowing replacement work.
Inside model execution, the activity starts a heartbeat helper. This runs independently of token arrival, so the model can remain alive even while it is thinking and has not produced new output.
var tokens atomic.Int64
// Report liveness and the current token count while generation runs.
stop := startStreamHeartbeats(ctx, func() {
if activity.IsActivity(ctx) {
activity.RecordHeartbeat(ctx, tokens.Load())
}
})
defer stop()
// Reuse a completed result instead of invoking the model again.
if _, err := a.Repository.GetChatModelResult(ctx, ref); err == nil {
return nil
} else if !errors.Is(err, storage.ErrNotFound) {
return err
}
// ...
func startStreamHeartbeats(ctx context.Context, beat func()) func() {
done := make(chan struct{})
exited := make(chan struct{})
go func() {
defer close(exited)
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
beat()
for {
select {
case <-done:
return
case <-ctx.Done():
return
case <-ticker.C:
beat()
}
}
}()
// Stop the helper and wait for its goroutine to exit.
var once sync.Once
return func() {
once.Do(func() { close(done) })
<-exited
}
}Sources: web_session.go · chat_model.go
The following sections are reserved for optimisations with KEDA and Kyverno. These techniques can work alongside the Temporal patterns above, but they are also useful for workloads that do not use Temporal.
Worker autodiscovery with Kyverno
Worker autodiscovery can be a useful addition to the Child Workflows and Parallel Execution patterns. As a system grows, keeping a central list of workers and their dependencies becomes harder. Adding a worker can mean updating both its deployment and the orchestrator’s configuration.
Kyverno is a Kubernetes policy engine that can validate resources and generate other resources from their metadata. This makes it useful for building a lightweight service registry. Services declare their roles through annotations, and policies generate registrations that an application can watch. This pattern works without Temporal too, but the application still needs to understand those registrations.
This setup uses two policies. A ValidatingPolicy checks worker annotations, while a GeneratingPolicy creates a ConfigMap containing the worker’s task queue, workflow type, and dependencies. The orchestrator watches these registrations, validates the dependency graph, and starts Temporal Child Workflows in dependency order.
For an agentic platform, this could register specialist workers for code review, testing, or security checks without maintaining a separate central list. It discovers capabilities declared in annotations, not capabilities inferred from an agent’s code.
The example below shows how a worker Service declares its dependencies:
apiVersion: v1
kind: Namespace
metadata:
name: temporal-demo-worker-d
labels:
temporal-demo.io/worker-namespace: "true"
---
apiVersion: v1
kind: Service
metadata:
name: worker-d
namespace: temporal-demo-worker-d
annotations:
temporal-worker: "true"
temporal-worker-run-after: "worker-b,worker-c"
spec:
selector:
app: worker-d
ports:
- name: health
port: 8081
targetPort: healthThe following shortened policy turns matching Services into registrations:
apiVersion: policies.kyverno.io/v1
kind: GeneratingPolicy
metadata:
name: temporal-generate-registrations
spec:
evaluation:
generateExisting:
enabled: true
synchronize:
enabled: true
orphanDownstreamOnPolicyDelete:
enabled: false
matchConstraints:
namespaceSelector:
matchLabels:
temporal-demo.io/worker-namespace: "true"
resourceRules:
- apiGroups: [""]
apiVersions: [v1]
operations: [CREATE, UPDATE]
resources: [services]
matchConditions:
- name: worker-service
expression: >-
object.metadata.?annotations[?'temporal-worker']
.orValue('') == 'true'
variables:
- name: registration
expression: >-
{
'apiVersion': dyn('v1'),
'kind': dyn('ConfigMap'),
'metadata': dyn({
'name': 'temporal-worker-' + object.metadata.name,
'namespace': object.metadata.namespace,
'labels': {'temporal-demo.io/registration': 'true'}
}),
'data': dyn({
'workerName': object.metadata.name,
'taskQueue': object.metadata.name + '.v1.temporal',
'workflowType': 'ExampleChildWorkflow',
'runAfter': object.metadata.?annotations[
?'temporal-worker-run-after'
].orValue('')
})
}
generate:
- expression: >-
generator.apply(
object.metadata.namespace,
[variables.registration]
)Here, CEL expressions replace the earlier {{ request.object... }} templates. Optional lookups with orValue('') handle missing annotations. The dyn(...) casts allow the generated resource to contain values of different types.
generateExisting covers Services that already exist. synchronize keeps registrations aligned with their sources and repairs modified or deleted ConfigMaps. Removing the worker annotation, setting it to "false", or deleting the Service removes its registration. With orphaning disabled, deleting the policy also removes its generated registrations.
This shortened example leaves out the schema-version and source-identity fields required by the demo’s registration contract. Use the full policy when running it.
A separate ValidatingPolicy rejects malformed annotations. Its dependency check uses CEL like this:
# Excerpt: "annotations", "dependencies", and trimmed "tokens"
# are variables defined earlier in the full ValidatingPolicy.
validationActions: [Deny]
validations:
- expression: >-
variables.annotations[?'temporal-worker'].orValue('') != 'true' ||
variables.dependencies.trim() == '' ||
variables.tokens.all(token,
token.matches('^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$') &&
token != object.metadata.name)
message: Dependencies must contain valid names and no self-reference.Trade-offs and edge cases:
Kyverno can help with worker and service autodiscovery, but there are limits to what its policies can check.
- Local validation is not graph validation. The policy checks dependency names, empty entries, and direct self-references, but it cannot validate the full dependency graph. The orchestrator still needs to reject missing dependencies, duplicate worker identities, and indirect cycles such as
A → B → C → A. CEL does not turn Kyverno into a recursive dependency-graph engine. - Discovery is eventually consistent. Policy reconciliation and Kubernetes watches take time.
- Registration is not readiness. A registered worker may have no running Pods because KEDA scaled it to zero, as described in the next section. Registration describes how to route work, not whether a process is currently available.
Sources: GeneratingPolicy · ValidatingPolicy · Discovery behavior · Graph validation
Worker scaling with KEDA
Temporal workers are not always busy. Keeping a Pod running for every worker can waste resources, especially when tasks arrive in bursts.
KEDA scales Kubernetes workloads using external signals, such as a queue backlog. Starting with version 2.17, KEDA supports scaling applications based on Temporal task queues.
With Temporal, KEDA watches workflow and activity task queues. When work arrives, it scales up the corresponding worker Deployment. Temporal keeps tasks queued while the Pod starts. Once the queue stays idle and the cooldown expires, KEDA can scale the worker back to zero.
For agentic systems, this is useful for occasional jobs such as repository indexing, code review, or sandbox setup. Workers only need capacity when there is work to process, at the cost of some startup latency.
The ScaledObject below connects worker D’s Deployment to its Temporal task queue:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: worker-d
namespace: temporal-demo-worker-d
spec:
# Existing Deployment in the same Kubernetes namespace.
scaleTargetRef:
name: worker-d
minReplicaCount: 0
maxReplicaCount: 1 # Demo value, production can use different
pollingInterval: 5 # Check for work every five seconds.
cooldownPeriod: 120 # Wait before returning to zero.
triggers:
- type: temporal
metadata:
endpoint: temporal.temporal.svc.cluster.local:7233
namespace: default # Temporal namespace, not Kubernetes.
taskQueue: worker-d.v1.temporal
queueTypes: "workflow,activity"
activationTargetQueueSize: "0"
targetQueueSize: "1"
includeRunningWorkflowCount: "false"
enableTLS: "false" # Use false for local / demo onlyThe worker polls worker-d.v1.temporal for both workflow and activity tasks. A backlog above zero activates the Deployment. targetQueueSize sets the backlog target used for scaling.
In the demo’s dependency graph, worker D runs after B and C finish. When the parent starts D’s Child Workflow, Temporal queues its first task. KEDA detects that backlog and starts a worker Pod to process it.
Trade-offs and edge cases:
- Cold starts add latency. The first task waits for polling, scheduling, image loading, and worker startup. Keep a minimum replica count for latency-sensitive workers, and account for startup delays in task and workflow timeouts.
- Shutdown and retries still matter. Workers need graceful shutdown, Activity heartbeats, and retry-safe side effects. Temporal preserves workflow history, but an interrupted Activity can run again.
- Zero Pods does not mean zero infrastructure cost. Temporal, PostgreSQL, controllers, and cluster nodes remain running. Production connections also need appropriate TLS and authentication.
Sources: ScaledObject template · Worker Deployment · Scaling behavior and limitations.