Service Mesh at 100k Pods: Istio Ambient vs Linkerd vs Cilium Mesh vs Kuma — p99 Latency, Memory per Node, and the True Cost per Million Requests

The 100k-Pod Mesh Is a Different Engineering Problem

Almost every "Linkerd vs Istio" post you have read was measured on a three-node kind cluster with a single namespace and a handful of services. Those results are not wrong — they are just measuring a different system than the one you operate. At 100,000 pods you cross several hard thresholds simultaneously:

• Control-plane fan-out — 100k proxies (or ztunnels) each holding a watch on thousands of ServiceEntry/EndpointSlice objects.
• Identity churn — SPIFFE SVIDs with 24h TTLs at 100k identities is ~1.2 CSRs/second average, with thundering-herd bursts after a control-plane restart.
• Aggregate memory tax — a 30 MB proxy is irrelevant on one pod and 3 TB of RSS across a fleet.
• Datapath state — conntrack tables, iptables sync times, and eBPF map pressure at 10k+ services.

The architectural question that dominates all four is simple: do you pay per pod, or per node? Everything else — p99, memory, cost — follows from that answer.

Four Architectures, One Datapath Question

Istio Ambient: Split the Proxy, Split the Cost

Ambient removes the sidecar entirely and splits the mesh into two layers. A ztunnel (Rust, per node) terminates L4 mTLS over HBONE and enforces L4 authorization policy; it never parses HTTP. Waypoint proxies (Envoy, per namespace or per service account) are deployed only where you need L7 policy, retries, or traffic shifting.

The cost model shifts from O(pods) to O(nodes) + O(waypoints). The trade-off is two TLS legs for L7 traffic: client → ztunnel (mTLS) → waypoint (re-originated mTLS) → ztunnel → server. That is two handshakes to amortize via connection pooling, and two hops of latency on the first request of a connection.

Linkerd: The Micro-Proxy Discipline

Linkerd keeps a per-pod sidecar, but it is a purpose-built Rust proxy (linkerd2-proxy) rather than a general-purpose Envoy. That buys a materially smaller footprint and a simpler config surface: no VirtualService DSL, just HTTPRoute-style policy and service profiles. The destination and identity controllers are lean and designed to be horizontally sharded by namespace watch scope.

The cost model is unambiguously O(pods). That is Linkerd's strength (predictable, minimal per-pod blast radius) and its weakness (it does not amortize).

Cilium Service Mesh: The Kernel Is the Data Plane

Cilium does L3/L4 service routing and identity-based policy in eBPF, attached at the socket layer (connect() hooks) or TC ingress. No sidecar, no ztunnel, no extra hop for L4 — the packet path is the host network stack with a BPF program spliced in. L7 policy and Envoy-based features run in a per-node embedded Envoy, not per pod. Mutual authentication uses Cilium's SPIFFE-based identity with cilium-agent acting as the SVID provider.

The trade-off is operational: you are now running the mesh and the CNI in the same DaemonSet, so a bad cilium-agent rollout is a node-level networking outage, not a degraded mesh.

Kuma: Envoy Everywhere, With Escape Hatches

Kuma is the CNCF-graduated control plane that runs Envoy sidecars by default, with a multi-zone (global + zone CP) topology built for heterogeneous and multi-cluster estates. It also ships an eBPF datapath mode (built on Merbridge-style connect() redirection) that bypasses the sidecar for L4 to cut a hop. If you need a single mesh spanning VMs, Kubernetes, and multiple clouds with per-zone failure domains, Kuma's topology is the differentiator — at the cost of Envoy's per-pod footprint.

Side-by-Side

Benchmark Methodology That Survives Review

Before any number, the harness. Three rules separate a measurement from a marketing slide.

1. Open-loop load, always. Closed-loop generators (fixed concurrency, no think time) suffer coordinated omission: when the system stalls, the generator stalls with it, and your p99 silently excludes the stall. Drive a fixed request rate and measure the achieved rate versus target.

2. Pin the confounders. Same node pool, same kernel (5.15 vs 6.1 changes conntrack and eBPF behavior), same MTU, tracing off, access logs off, explicit CPU requests on every proxy with no CPU limits (throttling produces artificial p99 cliffs).

3. Compare like hops. A ztunnel-only L4 comparison against a Linkerd sidecar doing full L7 is not a comparison. Match the feature set.

#!/usr/bin/env bash
set -euo pipefail

TARGET="http://echo.default.svc.cluster.local:8080"
RATE="${RATE:-2000}"       # fixed requests/sec — open loop
DURATION="${DURATION:-300}"
CONN="${CONN:-64}"         # keep-alive connections to spread the rate over

# 60s warmup populates connection pools and TLS sessions, then measure.
kubectl exec -i deploy/fortio-client -- fortio load \
  -qps "${RATE}" -t 60s -c "${CONN}" -keepalive \
  -payload-size 1024 "${TARGET}" >/dev/null

kubectl exec -i deploy/fortio-client -- fortio load \
  -qps "${RATE}" -t "${DURATION}s" -c "${CONN}" -keepalive \
  -payload-size 1024 -json /tmp/ambient-l4.json "${TARGET}"

kubectl cp "$(kubectl get pod -l app=fortio-client -o name | head -1)":/tmp/ambient-l4.json \
  "./results/ambient-l4.json"

For fleet-scale control-plane tests, generate churn with kube-burner or a synthetic endpoint scaler rather than hand-creating pods — you need the watch event rate, not the steady state.

p99 Latency: Where the Microseconds Actually Go

Tail latency in a mesh is not dominated by the proxy's per-request work. It is dominated by four things: CPU throttling of the proxy, TLS session resumption behavior, connection pool warmth, and queueing at the hop you added.

Representative Added p99 (intra-cluster, 1 KiB, keep-alive)

These are order-of-magnitude ranges consistent with published vendor benchmarks and reproduced lab runs on 8-vCPU nodes. Treat them as a prior to validate, not a specification — payload size, concurrency, and kernel version move these by 2x routinely.

The single largest p99 risk in every sidecar architecture is CPU limits. A proxy throttled at 100m will show p99 spikes into the tens of milliseconds while p50 looks perfect. Set CPU requests to at least 2x steady-state usage and never set a limit below 4x.

Measuring It Yourself

# Ambient: L7 latency is reported by the waypoint, not the ztunnel.
histogram_quantile(
  0.99,
  sum by (le, destination_workload) (
    rate(istio_request_duration_milliseconds_bucket{reporter="destination"}[5m])
  )
)

# Linkerd: proxy-reported end-to-end latency per route.
histogram_quantile(
  0.99,
  sum by (le, route, dst_service) (
    rate(response_latency_ms_bucket[5m])
  )
)

# Cilium: L7 (Envoy) request duration and BPF datapath drops.
histogram_quantile(0.99, sum by (le, destination_workload) (
  rate(cilium_envoy_http_request_duration_seconds_bucket[5m])))
sum(rate(cilium_drop_count_total{reason="Policy denied"}[5m])) by (node)

Memory per Node: The Aggregate Tax

This is where architecture becomes invoice.

The Math That Matters

Fleet:        100,000 pods  /  ~40 pods per node  =  2,500 nodes

Sidecar @ 30 MiB/pod  ->  100,000 x 30 MiB  =  3.0 TB aggregate RSS
Ambient  @ 60 MiB/node ->    2,500 x 60 MiB  =  0.15 TB aggregate RSS
Cilium   @ 400 MiB/node->    2,500 x 400 MiB =  1.0 TB aggregate RSS (agent)

That 3 TB is not free capacity you can ignore — on a 32 GiB node it is 1.2 GiB of the 32, which at 40 pods per node means you fit roughly 2 fewer workload pods per node, or about 5% stranded capacity fleet-wide.

Representative Footprints (RSS, steady state)

Instrumenting It

# Per-node mesh memory in GiB — the number your capacity planner needs.
sum by (node) (
  container_memory_working_set_bytes{
    namespace=~"linkerd|istio-system|kube-system",
    container=~"linkerd-proxy|istio-proxy|ztunnel|cilium-agent"
  }
) / 1e9

Control Plane Behavior at 100k Pods

xDS Fan-Out and Push Throttling

At 100k proxies, a single VirtualService edit can trigger 100k xDS pushes. Istio's debounce and throttle knobs are not optional at this scale:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    pilot:
      env:
        PILOT_PUSH_THROTTLE: "100"        # concurrent pushes per istiod
        PILOT_DEBOUNCE_AFTER: "100ms"
        PILOT_DEBOUNCE_MAX: "10s"
        PILOT_ENABLE_EDS_DEBOUNCE: "true"  # batch endpoint churn
        PILOT_ENABLE_ANALYSIS: "false"     # off in prod, costs CPU
  meshConfig:
    accessLogFile: ""                      # logs are a silent p99 tax
    enableTracing: false
    defaultConfig:
      holdApplicationUntilProxyStarts: true

Run revision-based canary upgrades (istio.io/rev) so a control-plane upgrade never restarts 100k data planes at once.

CiliumEndpointSlice and the CRD Explosion

At 100k endpoints, the CiliumEndpoint CRD set becomes a genuine etcd problem: watch events, list cost, and API server memory all scale linearly. CiliumEndpointSlice (CES) batches many endpoints into one object and is the required configuration at this scale on recent releases (enable via the ciliumEndpointSlice.enabled Helm value / operator flag on 1.17+).

Identity and Certificate Churn

100k SVIDs at 24h TTL is ~1.2 CSR/s average, but a control-plane restart produces a synchronized burst. Stagger issuance TTLs per node (jitter), and for Istio verify istiod CA throughput before scaling past ~50k identities per replica — split by revision or shard namespaces.

The True Cost per Million Requests

A defensible model has exactly three inputs per architecture: node hourly cost, baseline throughput, and overhead as a fraction of that throughput.

from dataclasses import dataclass
from decimal import Decimal, getcontext
from typing import Iterable

getcontext().prec = 28


@dataclass(frozen=True)
class NodeProfile:
    name: str
    hourly_usd: Decimal
    memory_gib: int
    baseline_rps: int          # no-mesh throughput for this workload on this node
    pods_per_node: int
    cpu_overhead_pct: float    # datapath CPU tax, as a fraction (0.12 == 12%)
    mem_per_pod_mib: float     # proxy RSS attributable per pod (0 for per-node models)
    mem_per_node_mib: float    # ztunnel / cilium-agent RSS

    def __post_init__(self) -> None:
        if self.hourly_usd <= 0:
            raise ValueError(f"{self.name}: hourly_usd must be positive")
        if self.baseline_rps <= 0:
            raise ValueError(f"{self.name}: baseline_rps must be positive")
        if not 0.0 <= self.cpu_overhead_pct < 1.0:
            raise ValueError(f"{self.name}: cpu_overhead_pct must be in [0, 1)")


def cost_per_million(node: NodeProfile) -> Decimal:
    effective_rps = Decimal(node.baseline_rps) * (Decimal(1) - Decimal(str(node.cpu_overhead_pct)))
    if effective_rps <= 0:
        raise ValueError(f"{node.name}: effective throughput collapsed to zero")
    return (node.hourly_usd / (effective_rps * 3600)) * Decimal(1_000_000)


def memory_tax_fraction(node: NodeProfile) -> float:
    used_mib = node.pods_per_node * node.mem_per_pod_mib + node.mem_per_node_mib
    used_gib = used_mib / 1024.0
    if used_gib >= node.memory_gib:
        raise ValueError(f"{node.name}: mesh overhead exceeds node memory")
    return used_gib / node.memory_gib


def report(profiles: Iterable[NodeProfile]) -> None:
    baseline = None
    for p in profiles:
        cpm = cost_per_million(p)
        baseline = baseline or cpm
        delta = ((cpm / baseline) - 1) * 100
        print(
            f"{p.name:<22} ${cpm:.5f}/M  ({delta:+5.1f}% vs baseline)  "
            f"mem tax {memory_tax_fraction(p) * 100:4.1f}% of node RAM"
        )

Worked Example: 2,500 Nodes of m6i.2xlarge

Baseline node: $0.384/hr, 32 GiB, 40 pods, 6,000 RPS of no-mesh capacity.

Fleet baseline spend: 2,500 × 730h × $0.384 ≈ $700k/month. A 14-point delta between sidecar and eBPF models is ~$100k/month — and that is before counting the ~5% of node RAM the sidecars strand, which pushes sidecar models another 4–6 points higher.

The honest counterpoint: if your cluster is 30% badly bin-packed, mesh choice is the second-order term. Fix requests/limits and pod density first, then buy the mesh.

Decision Matrix

Production Checklist and Pitfalls

• Never benchmark closed-loop. Fixed concurrency hides stalls; report achieved RPS next to target RPS.
• Disable access logs and tracing before measuring. They routinely account for more p99 than the proxy itself.
• No CPU limits on data planes. Throttling is the number one cause of sidecar p99 cliffs.
• Warm connections and TLS sessions. First-request latency is a handshake measurement, not a mesh measurement.
• Watch iptables sync time on non-eBPF CNIs once you exceed ~5k services; it shows up as kube-proxy CPU spikes.
• Set `holdApplicationUntilProxyStarts` or you will chase startup race failures at scale.
• Load-test the control plane separately with endpoint churn, not just steady-state request load.
• Budget for identity bursts after control-plane restarts; jitter SVID TTLs.
• Verify MTU end-to-end. WireGuard/IPsec overlays reduce usable MTU; silent fragmentation looks like random p99 outliers.
• Canary the mesh with revisions/zones so no single rollout touches more than 10% of data planes.

Sources & Technical References

• Istio — Ambient Mesh documentation: https://istio.io/latest/docs/ambient/
• Istio — Performance and Scalability: https://istio.io/latest/docs/ops/deployment/performance-and-scalability/
• Istio — Introducing Ambient Mesh (Howard & Jackson): https://istio.io/latest/blog/2022/introducing-ambient-mesh/
• Istio — Benchmark tooling (istio/tools/perf/benchmark): https://github.com/istio/tools/tree/master/perf/benchmark
• Linkerd — Architecture and proxy documentation: https://linkerd.io/2/architecture/
• Linkerd — Independent benchmarks of Linkerd and Istio: https://linkerd.io/2021/10/13/linkerd-vs-istio-benchmarks/
• Cilium — Service Mesh documentation: https://docs.cilium.io/en/stable/network/servicemesh/
• Cilium — eBPF datapath and socket-level load balancing: https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/
• Cilium — CiliumEndpointSlice scaling guidance: https://docs.cilium.io/en/stable/network/kubernetes/ciliumendpointslice/
• Kuma — Documentation (policies, multi-zone, eBPF mode): https://kuma.io/docs/
• SPIFFE — Secure Production Identity Framework specification: https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE-ID.md
• RFC 8446 — The Transport Layer Security (TLS) Protocol Version 1.3: https://www.rfc-editor.org/rfc/rfc8446
• RFC 7540 — Hypertext Transfer Protocol Version 2 (HTTP/2): https://www.rfc-editor.org/rfc/rfc7540
• Kubernetes — Services, EndpointSlices, and kube-proxy: https://kubernetes.io/docs/concepts/services-networking/
• Google SRE Book — Service Level Objectives: https://sre.google/sre-book/service-level-objectives/
• Gil Tene — How NOT to Measure Latency (coordinated omission): https://www.youtube.com/watch?v=lJ8ydIuPFeU