What the Infra?: The Real Architecture Behind Production AI Agents

Every agent demo you have seen on a conference stage hides the same uncomfortable truth: the model is the easy part. The hard part is the infrastructure that schedules the loop, isolates the side effects, streams the state, and tells you — in seconds, not in a postmortem — that something is wrong. This article dissects a reference architecture for AI agent infrastructure: the three planes, the runtime contract, the telemetry control loop, and the security boundaries that determine whether a breaking security disclosure becomes your incident or someone else's.

What the Infra? Why Agents Break Below the Model

A chat completion is a pure function. An agent is a distributed system with a natural-language input surface, non-deterministic control flow, and side effects on the real world. Those three properties invert almost every assumption your existing platform makes:

• Non-determinism breaks caching and replay unless you capture prompt, tool outputs, and sampling parameters as a reproducible trace.
• Side effects break at-least-once semantics unless every tool call carries an idempotency key.
• Multi-step loops break latency SLOs because p50 is meaningless when a single run fans out into 40 model calls and 12 tool invocations.

When a breaking security disclosure hits — a prompt-injection technique, a sandbox escape, a token-leak pattern — the blast radius is decided long before the disclosure. It is decided by how you scoped credentials, what your tools were allowed to touch, and whether your real-time analysis pipeline could detect an anomalous tool call graph while it was still happening. If your answer to 'what is the infra?' is 'a queue and a Lambda', you are the blast radius.

The Three-Plane Agent Architecture

Production agent systems decompose into three planes with different failure modes and different SLOs.

Control plane: orchestration, policy, registry

The control plane decides what may happen: the run graph scheduler, the tool registry with typed schemas, the policy engine (OPA/Cedar), and the model gateway that centralizes routing, quotas, and provider failover. It must be stateless and horizontally scalable, because it is on the hot path of every step.

Execution plane: sandboxes, adapters, egress

The execution plane does the thing: ephemeral sandboxes (Firecracker microVMs, gVisor user-space kernels, or hardened containers), tool adapters that translate model output into typed calls, and an egress proxy that allowlists destinations. This is where isolation tiers matter and where your cost model lives.

Observation plane: traces, evals, control signals

The observation plane closes the loop: OpenTelemetry spans per model call and tool call, token accounting, and a streaming analyzer that can trip circuit breakers. Note the direction of data flow — telemetry is not a dashboard for humans, it is an input to the control plane.

Building a Runtime That Can't Be Talked Into Self-Destruction

The single highest-leverage design decision: the model never holds authority. It emits a proposal; the runtime validates it against a capability set minted at run creation. The model can request net:api.stripe.com; it cannot grant it.

Code: capability-scoped tool invocation with deadline propagation

import { z } from 'zod';

type Capability = `tool:${string}` | `net:${string}` | `fs:ro` | `fs:rw`;

interface ToolContext {
  readonly runId: string;
  readonly deadline: number;        // epoch ms, inherited from the run budget
  readonly caps: ReadonlySet<Capability>;
  readonly signal: AbortSignal;
}

interface ToolDefinition<I, O> {
  readonly name: string;
  readonly requires: readonly Capability[];
  readonly idempotent: boolean;
  readonly inputSchema: z.ZodType<I>;
  readonly execute: (input: I, ctx: ToolContext) => Promise<O>;
}

export class PolicyError extends Error {
  constructor(readonly tool: string, readonly missing: Capability) {
    super(`capability ${missing} not granted for ${tool}`);
  }
}
export class BudgetError extends Error {
  constructor(readonly runId: string, readonly remainingMs: number) {
    super(`run ${runId} has ${remainingMs}ms left; aborting`);
  }
}

export async function invokeTool<I, O>(
  def: ToolDefinition<I, O>,
  rawInput: unknown,
  parent: Omit<ToolContext, 'signal' | 'deadline'>,
): Promise<O> {
  for (const cap of def.requires) {
    if (!parent.caps.has(cap)) throw new PolicyError(def.name, cap);
  }

  const remainingMs = parent.deadline - Date.now();
  if (remainingMs <= 50) throw new BudgetError(parent.runId, remainingMs);

  const parsed = def.inputSchema.safeParse(rawInput);
  if (!parsed.success) {
    throw new z.ZodError(parsed.error.issues); // never feed raw model JSON to a tool
  }

  const signal = AbortSignal.any([
    AbortSignal.timeout(Math.min(remainingMs, 30_000)),
    AbortSignal.timeout(remainingMs),
  ]);

  return def.execute(parsed.data, { ...parent, signal });
}

Two details that matter in production. First, AbortSignal.any composes the per-tool timeout with the run-level deadline, so a runaway tool cannot starve the remaining steps. Second, schema validation happens before the side effect — the model output is untrusted input, exactly like an HTTP body.

Retries, idempotency, and the at-least-once trap

Every non-idempotent tool (payments.charge, email.send, deploy.rollback) must take a client-generated idempotency key derived from (runId, stepId, toolName). Retry with exponential backoff and full jitter, cap at 3 attempts, and treat 4xx as terminal except 429 and 409. The failure mode you are preventing is the agent retrying a charge because the response timed out, not the request.

Real-time Analysis: Traces as a Control Signal

Dashboards answer questions you already knew to ask. A control loop answers the question you did not: is this run behaving unlike every run before it? The pattern is a per-run rolling window over tool latency, token burn, error rate, and, critically, the tool call graph — repeated identical calls (http.fetch x 200 to the same host) are the signature of both injection loops and runaway retries.

Code: a streaming analyzer with cardinality control

from __future__ import annotations

import time
from collections import deque
from dataclasses import dataclass
from typing import Deque, Iterable

from opentelemetry import metrics

meter = metrics.get_meter("agent.runtime")
breaker = meter.create_counter("agent.run.breaker_tripped")

@dataclass(frozen=True, slots=True)
class AgentEvent:
    run_id: str
    tool: str
    latency_ms: float
    tokens: int
    ok: bool
    ts: float

class RunAnalyzer:
    """Per-run rolling window. Emits a verdict, not a metric."""

    def __init__(self, window: int = 64, max_repeat: int = 8) -> None:
        self._events: Deque[AgentEvent] = deque(maxlen=window)
        self._window = window
        self._max_repeat = max_repeat

    def observe(self, event: AgentEvent) -> str | None:
        self._events.append(event)

        recent = list(self._events)
        if len(recent) < 8:
            return None

        errors = sum(1 for e in recent if not e.ok) / len(recent)
        if errors > 0.5:
            return "trip:error_rate"

        # Loop detection: same tool repeated within the window.
        counts: dict[str, int] = {}
        for e in recent:
            counts[e.tool] = counts.get(e.tool, 0) + 1
        if any(c >= self._max_repeat for c in counts.values()):
            return "trip:tool_loop"

        tokens = sum(e.tokens for e in recent)
        if tokens > 120_000:
            return "trip:token_burn"
        return None

    def enforce(self, run_id: str, verdict: str | None) -> None:
        if verdict is None:
            return
        breaker.add(1, {"run": run_id[:8], "reason": verdict})
        raise RuntimeError(f"run {run_id} halted: {verdict}")

Cardinality is the operational trap. Never use raw run_id or user prompt text as a metric attribute — hash and truncate it (run_id[:8]), and keep high-cardinality data in traces, not metrics or logs. A single agent fleet can generate more unique label values in a minute than the rest of your platform does in a month.

Breaking Security in the Agent Era: Four Attack Classes

1. Prompt injection is a supply-chain defect

Any text your agent ingests — a fetched page, a PDF, a GitHub issue — is untrusted input with the same authority as a tool response. Treat retrieved content as data, never as instructions, and never concatenate it into a system-level context slot. OWASP's LLM Top 10 ranks this first for a reason: it has no complete fix, only containment.

2. The confused deputy: tool-call authorization

The model is a low-privilege principal that must not be able to borrow the runtime's identity. Authorization belongs in policy code, evaluated per call, with the caller's capability set as input — not in the prompt.

package agent.authz

default allow := false

allow if {
  input.tool == "http.fetch"
  some host in input.args.hosts
  host in data.egress.allowlist
  sprintf("net:%s", [host]) in input.context.caps
}

allow if {
  input.tool == "repo.write"
  input.context.caps[_] == "fs:rw"
  startswith(input.args.path, "/workspace/")
}

3. Sandbox escape and egress

MicroVM-per-run is the default recommendation for untrusted code execution; gVisor is a pragmatic middle tier; plain containers are acceptable only for fully trusted, first-party code. Pair every sandbox with a deny-by-default egress proxy — most exfiltration paths need outbound network, and most agents need exactly two or three hosts.

4. Credential exfiltration through context

Never place long-lived secrets in a prompt or a tool result. Mint short-lived, run-scoped tokens via OAuth 2.0 Token Exchange (RFC 8693), bind them to the workload with DPoP (RFC 9449) or mTLS, and set TTLs measured in minutes. If a token appears in a trace attribute, you have a logging incident.

Trade-offs: Isolation Tiers, Latency Budgets, and Cost

The second trade-off is budget allocation. A useful default: 60% of the run deadline to model calls, 30% to tool calls, 10% reserved for finalization. Without an explicit split, retries silently consume the finalization budget and you ship truncated answers under load.

Production Readiness Checklist

• Every tool has a typed input schema and fails closed on validation errors.
• Capabilities are minted at run creation, never escalated mid-run.
• All non-idempotent tools require an idempotency key.
• Per-run deadline is propagated to every model and tool call.
• Deny-by-default egress allowlist on every sandbox.
• Traces capture prompt hash, model version, sampling params, and tool I/O.
• A streaming analyzer can halt a run in under 5 seconds.
• Metrics use hashed, bounded-cardinality attributes only.
• Secrets are run-scoped, short-TTL, and never logged.
• A kill switch exists that revokes capabilities fleet-wide without a deploy.

Pitfalls That Bite in Week Three

• Logging full prompts for debugging. It becomes a PII and secret reservoir within days. Log hashes and redacted spans.
• Retry storms on 429. Without jitter and a circuit breaker, one provider hiccup becomes a self-inflicted DDoS.
• Treating tool output as trusted. A calendar invite body is an injection vector as surely as a web page.
• Unbounded agent loops. Always cap steps, tokens, and wall clock per run — three independent limits, not one.
• Chasing the Twitter tech trend of the week. When a new framework tops the Twitter tech trend cycle, benchmark it against your trace data before it touches the control plane. Architecture decisions age better than hype cycles.

Sources & Technical References

• OWASP Top 10 for LLM Applications — https://genai.owasp.org/llm-top-10/
• NIST AI Risk Management Framework (AI RMF 1.0) — https://www.nist.gov/itl/ai-risk-management-framework
• OpenTelemetry Semantic Conventions — https://opentelemetry.io/docs/specs/semconv/
• RFC 8693: OAuth 2.0 Token Exchange — https://www.rfc-editor.org/rfc/rfc8693
• RFC 9449: OAuth 2.0 Demonstrating Proof of Possession (DPoP) — https://www.rfc-editor.org/rfc/rfc9449
• Open Policy Agent Documentation — https://www.openpolicyagent.org/docs/latest/
• Agache et al., 'Firecracker: Lightweight Virtualization for Serverless Applications', NSDI 2020 — https://www.usenix.org/conference/nsdi20/presentation/agache
• gVisor Documentation — https://gvisor.dev/docs/
• Anthropic, 'Building Effective Agents' — https://www.anthropic.com/research/building-effective-agents
• MDN Web Docs, AbortSignal and AbortSignal.timeout() — https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal
• Google SRE Book, 'Handling Overload' — https://sre.google/sre-book/handling-overload/
• AWS Builders' Library, 'Timeouts, Retries, and Backoff with Jitter' — https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/