What the Infra?: The Real Architecture Behind Production AI Agents
Agent demos stop at the prompt; production agents live or die on the runtime, isolation, and telemetry planes underneath it. This is a concrete architecture walkthrough — capability-scoped tool execution, streaming real-time analysis, and the security boundaries that keep a breaking security incident from becoming your incident.
AI Generated with Qwen 3.0 via CometAPIEvery agent demo you have seen on a conference stage hides the same uncomfortable truth: the model is the easy part. The hard part is the infrastructure that schedules the loop, isolates the side effects, streams the state, and tells you — in seconds, not in a postmortem — that something is wrong. This article dissects a reference architecture for AI agent infrastructure: the three planes, the runtime contract, the telemetry control loop, and the security boundaries that determine whether a breaking security disclosure becomes your incident or someone else's.
What the Infra? Why Agents Break Below the Model
A chat completion is a pure function. An agent is a distributed system with a natural-language input surface, non-deterministic control flow, and side effects on the real world. Those three properties invert almost every assumption your existing platform makes:
- Non-determinism breaks caching and replay unless you capture prompt, tool outputs, and sampling parameters as a reproducible trace.
- Side effects break at-least-once semantics unless every tool call carries an idempotency key.
- Multi-step loops break latency SLOs because p50 is meaningless when a single run fans out into 40 model calls and 12 tool invocations.
When a breaking security disclosure hits — a prompt-injection technique, a sandbox escape, a token-leak pattern — the blast radius is decided long before the disclosure. It is decided by how you scoped credentials, what your tools were allowed to touch, and whether your real-time analysis pipeline could detect an anomalous tool call graph while it was still happening. If your answer to 'what is the infra?' is 'a queue and a Lambda', you are the blast radius.
The Three-Plane Agent Architecture
Production agent systems decompose into three planes with different failure modes and different SLOs.
Control plane: orchestration, policy, registry
The control plane decides what may happen: the run graph scheduler, the tool registry with typed schemas, the policy engine (OPA/Cedar), and the model gateway that centralizes routing, quotas, and provider failover. It must be stateless and horizontally scalable, because it is on the hot path of every step.
Execution plane: sandboxes, adapters, egress
The execution plane does the thing: ephemeral sandboxes (Firecracker microVMs, gVisor user-space kernels, or hardened containers), tool adapters that translate model output into typed calls, and an egress proxy that allowlists destinations. This is where isolation tiers matter and where your cost model lives.
Observation plane: traces, evals, control signals
The observation plane closes the loop: OpenTelemetry spans per model call and tool call, token accounting, and a streaming analyzer that can trip circuit breakers. Note the direction of data flow — telemetry is not a dashboard for humans, it is an input to the control plane.
Building a Runtime That Can't Be Talked Into Self-Destruction
The single highest-leverage design decision: the model never holds authority. It emits a proposal; the runtime validates it against a capability set minted at run creation. The model can request net:api.stripe.com; it cannot grant it.
Code: capability-scoped tool invocation with deadline propagation
import { z } from 'zod';
type Capability = `tool:${string}` | `net:${string}` | `fs:ro` | `fs:rw`;
interface ToolContext {
readonly runId: string;
readonly deadline: number; // epoch ms, inherited from the run budget
readonly caps: ReadonlySet<Capability>;
readonly signal: AbortSignal;
}
interface ToolDefinition<I, O> {
readonly name: string;
readonly requires: readonly Capability[];
readonly idempotent: boolean;
readonly inputSchema: z.ZodType<I>;
readonly execute: (input: I, ctx: ToolContext) => Promise<O>;
}
export class PolicyError extends Error {
constructor(readonly tool: string, readonly missing: Capability) {
super(`capability ${missing} not granted for ${tool}`);
}
}
export class BudgetError extends Error {
constructor(readonly runId: string, readonly remainingMs: number) {
super(`run ${runId} has ${remainingMs}ms left; aborting`);
}
}
export async function invokeTool<I, O>(
def: ToolDefinition<I, O>,
rawInput: unknown,
parent: Omit<ToolContext, 'signal' | 'deadline'>,
): Promise<O> {
for (const cap of def.requires) {
if (!parent.caps.has(cap)) throw new PolicyError(def.name, cap);
}
const remainingMs = parent.deadline - Date.now();
if (remainingMs <= 50) throw new BudgetError(parent.runId, remainingMs);
const parsed = def.inputSchema.safeParse(rawInput);
if (!parsed.success) {
throw new z.ZodError(parsed.error.issues); // never feed raw model JSON to a tool
}
const signal = AbortSignal.any([
AbortSignal.timeout(Math.min(remainingMs, 30_000)),
AbortSignal.timeout(remainingMs),
]);
return def.execute(parsed.data, { ...parent, signal });
}Two details that matter in production. First, AbortSignal.any composes the per-tool timeout with the run-level deadline, so a runaway tool cannot starve the remaining steps. Second, schema validation happens before the side effect — the model output is untrusted input, exactly like an HTTP body.
Retries, idempotency, and the at-least-once trap
Every non-idempotent tool (payments.charge, email.send, deploy.rollback) must take a client-generated idempotency key derived from (runId, stepId, toolName). Retry with exponential backoff and full jitter, cap at 3 attempts, and treat 4xx as terminal except 429 and 409. The failure mode you are preventing is the agent retrying a charge because the response timed out, not the request.
Real-time Analysis: Traces as a Control Signal
Dashboards answer questions you already knew to ask. A control loop answers the question you did not: is this run behaving unlike every run before it? The pattern is a per-run rolling window over tool latency, token burn, error rate, and, critically, the tool call graph — repeated identical calls (http.fetch x 200 to the same host) are the signature of both injection loops and runaway retries.
Code: a streaming analyzer with cardinality control
from __future__ import annotations
import time
from collections import deque
from dataclasses import dataclass
from typing import Deque, Iterable
from opentelemetry import metrics
meter = metrics.get_meter("agent.runtime")
breaker = meter.create_counter("agent.run.breaker_tripped")
@dataclass(frozen=True, slots=True)
class AgentEvent:
run_id: str
tool: str
latency_ms: float
tokens: int
ok: bool
ts: float
class RunAnalyzer:
"""Per-run rolling window. Emits a verdict, not a metric."""
def __init__(self, window: int = 64, max_repeat: int = 8) -> None:
self._events: Deque[AgentEvent] = deque(maxlen=window)
self._window = window
self._max_repeat = max_repeat
def observe(self, event: AgentEvent) -> str | None:
self._events.append(event)
recent = list(self._events)
if len(recent) < 8:
return None
errors = sum(1 for e in recent if not e.ok) / len(recent)
if errors > 0.5:
return "trip:error_rate"
# Loop detection: same tool repeated within the window.
counts: dict[str, int] = {}
for e in recent:
counts[e.tool] = counts.get(e.tool, 0) + 1
if any(c >= self._max_repeat for c in counts.values()):
return "trip:tool_loop"
tokens = sum(e.tokens for e in recent)
if tokens > 120_000:
return "trip:token_burn"
return None
def enforce(self, run_id: str, verdict: str | None) -> None:
if verdict is None:
return
breaker.add(1, {"run": run_id[:8], "reason": verdict})
raise RuntimeError(f"run {run_id} halted: {verdict}")Cardinality is the operational trap. Never use raw run_id or user prompt text as a metric attribute — hash and truncate it (run_id[:8]), and keep high-cardinality data in traces, not metrics or logs. A single agent fleet can generate more unique label values in a minute than the rest of your platform does in a month.
Breaking Security in the Agent Era: Four Attack Classes
1. Prompt injection is a supply-chain defect
Any text your agent ingests — a fetched page, a PDF, a GitHub issue — is untrusted input with the same authority as a tool response. Treat retrieved content as data, never as instructions, and never concatenate it into a system-level context slot. OWASP's LLM Top 10 ranks this first for a reason: it has no complete fix, only containment.
2. The confused deputy: tool-call authorization
The model is a low-privilege principal that must not be able to borrow the runtime's identity. Authorization belongs in policy code, evaluated per call, with the caller's capability set as input — not in the prompt.
package agent.authz
default allow := false
allow if {
input.tool == "http.fetch"
some host in input.args.hosts
host in data.egress.allowlist
sprintf("net:%s", [host]) in input.context.caps
}
allow if {
input.tool == "repo.write"
input.context.caps[_] == "fs:rw"
startswith(input.args.path, "/workspace/")
}3. Sandbox escape and egress
MicroVM-per-run is the default recommendation for untrusted code execution; gVisor is a pragmatic middle tier; plain containers are acceptable only for fully trusted, first-party code. Pair every sandbox with a deny-by-default egress proxy — most exfiltration paths need outbound network, and most agents need exactly two or three hosts.
4. Credential exfiltration through context
Never place long-lived secrets in a prompt or a tool result. Mint short-lived, run-scoped tokens via OAuth 2.0 Token Exchange (RFC 8693), bind them to the workload with DPoP (RFC 9449) or mTLS, and set TTLs measured in minutes. If a token appears in a trace attribute, you have a logging incident.
Trade-offs: Isolation Tiers, Latency Budgets, and Cost
| Isolation tier | Cold start | Blast radius | Right for |
|---|---|---|---|
| Process + seccomp | ~5 ms | Host namespace shared | Pure-compute tools, no network |
| Container | ~200 ms | Kernel shared | First-party, audited code |
| gVisor | ~400 ms | User-space kernel | Third-party tool adapters |
| Firecracker microVM | ~125 ms boot | Hardware-virtualized | Untrusted generated code |
The second trade-off is budget allocation. A useful default: 60% of the run deadline to model calls, 30% to tool calls, 10% reserved for finalization. Without an explicit split, retries silently consume the finalization budget and you ship truncated answers under load.
Production Readiness Checklist
- Every tool has a typed input schema and fails closed on validation errors.
- Capabilities are minted at run creation, never escalated mid-run.
- All non-idempotent tools require an idempotency key.
- Per-run deadline is propagated to every model and tool call.
- Deny-by-default egress allowlist on every sandbox.
- Traces capture prompt hash, model version, sampling params, and tool I/O.
- A streaming analyzer can halt a run in under 5 seconds.
- Metrics use hashed, bounded-cardinality attributes only.
- Secrets are run-scoped, short-TTL, and never logged.
- A kill switch exists that revokes capabilities fleet-wide without a deploy.
Pitfalls That Bite in Week Three
- Logging full prompts for debugging. It becomes a PII and secret reservoir within days. Log hashes and redacted spans.
- Retry storms on 429. Without jitter and a circuit breaker, one provider hiccup becomes a self-inflicted DDoS.
- Treating tool output as trusted. A calendar invite body is an injection vector as surely as a web page.
- Unbounded agent loops. Always cap steps, tokens, and wall clock per run — three independent limits, not one.
- Chasing the Twitter tech trend of the week. When a new framework tops the Twitter tech trend cycle, benchmark it against your trace data before it touches the control plane. Architecture decisions age better than hype cycles.
Sources & Technical References
- OWASP Top 10 for LLM Applications — https://genai.owasp.org/llm-top-10/
- NIST AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework
- OpenTelemetry Semantic Conventions — https://opentelemetry.io/docs/specs/semconv/
- RFC 8693: OAuth 2.0 Token Exchange — https://www.rfc-editor.org/rfc/rfc8693
- RFC 9449: OAuth 2.0 Demonstrating Proof of Possession (DPoP) — https://www.rfc-editor.org/rfc/rfc9449
- Open Policy Agent Documentation — https://www.openpolicyagent.org/docs/latest/
- Agache et al., 'Firecracker: Lightweight Virtualization for Serverless Applications', NSDI 2020 — https://www.usenix.org/conference/nsdi20/presentation/agache
- gVisor Documentation — https://gvisor.dev/docs/
- Anthropic, 'Building Effective Agents' — https://www.anthropic.com/research/building-effective-agents
- MDN Web Docs,
AbortSignalandAbortSignal.timeout()— https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal - Google SRE Book, 'Handling Overload' — https://sre.google/sre-book/handling-overload/
- AWS Builders' Library, 'Timeouts, Retries, and Backoff with Jitter' — https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
Discussion (0)
Continue Reading
More production-grade engineering deep dives and architectural guides

The Gemini Decisions API: Architecting a Deterministic Decision Service on a Probabilistic Model
A Hacker News thread argued about whether an LLM belongs on the critical path of a business decision. This is the production reference architecture that settles the argument: schema-bound structured output, bounded tool loops, deterministic policy gates, and a latency budget you can defend in a design review.

Brotli vs. Zstandard vs. Gzip at 1 PB/Month: Measured Ratios, CPU Seconds per GB, and the Real CDN Bill
At petabyte scale, compression is no longer a config flag — it is a P&L line item. Here are the measured ratios, real CPU-seconds-per-GB numbers, and the break-even cache-hit math that decides whether Brotli q11 pays for itself or quietly bankrupts your origin.

Micro-Frontends at 5,000 Routes: Module Federation vs single-spa vs Web Components, Measured
We rebuilt the same 5,000-route enterprise shell three times — Module Federation 2.0, single-spa, and native Web Components — and measured p95 route transitions, marginal memory per mounted remote, duplicated vendor bytes, and the fully loaded cost of 1M deployments. The results invert most vendor claims: the fastest architecture on cold start is the one that duplicates the most code.