Cilium vs. Istio Ambient vs. Linkerd at 1M RPS: Real p99, CPU Overhead, and Cost per Million Requests

At 1,000 requests per second, every service mesh looks free. A sidecar burning 40 MB of RSS and 0.15 vCPU is invisible on a node with 64 GB of RAM. Multiply that by 400 pods across 20 nodes and a single service hop, and you are paying for an entire extra rack to proxy HTTP.

At 1M RPS, the architecture of the mesh — not the marketing — determines your p99. The three dominant designs in 2025 take fundamentally different paths through the kernel: Cilium programs eBPF directly into the socket and routing layers, Istio Ambient splits the data path into a per-node L4 tunnel (ztunnel) plus optional per-namespace L7 proxies (waypoints), and Linkerd injects a Rust micro-proxy into every pod. Each choice lands differently on the latency histogram's tail and on your monthly invoice.

This article breaks down the architectures, shows a reproducible benchmark methodology, and gives you the cost math — not just the benchmark screenshots.

Why 1M RPS Is the Fault Line for Sidecar Architecture

The sidecar pattern costs you twice. First in bytes on the wire: every request is decrypted, re-encrypted, and forwarded by an extra process, so a 1 KB payload becomes ~4 KB of user-space copy and two syscall round-trips. Second in scheduling: each hop adds a goroutine/tokio task wakeup, and under load those wakeups contend for CPU run queues on the same node as your application.

At 10K RPS the second effect is noise. At 1M RPS across a cluster, run-queue latency (sched:sched_switch delays) starts showing up as multi-millisecond p99.9 spikes that have nothing to do with your application code. This is why tail latency, not mean latency, is the only metric that matters when you select a mesh.

The three data paths, precisely

Cilium: moving the mesh into the kernel

Cilium's advantage is that it does not proxy. For L3/L4 policy and identity-based mTLS, Cilium attaches eBPF programs to the socket layer and uses a per-node agent to distribute SPIFFE identities. Traffic between pods on the same node may never leave the kernel's network stack in the traditional sense — connect() is redirected at the sock_ops/sk_msg hook.

The catch: L7 policy reintroduces a proxy. The moment you need HTTP-path-based authorization or header mutation, Cilium embeds Envoy, and you inherit the sidecar tax for those specific endpoints. Most 1M RPS benchmarks that show Cilium at sub-millisecond p99 are running L3/L4 policy only.

Istio Ambient: two tiers, two taxes

Ambient splits the problem. ztunnel is a per-node Rust daemon that terminates mTLS and forwards over HBONE (HTTP/2 CONNECT, port 15008). It handles identity, L4 authorization, and telemetry — and nothing else. If you need L7, you deploy a waypoint proxy for that namespace or service account.

This is the most important nuance in the entire comparison: Ambient with no waypoints is not comparable to Linkerd with L7 mTLS and retries. Ambient L4-only will beat Linkerd on latency and CPU. Add a waypoint and you have re-created a two-hop path with an extra Envoy in the chain — and the numbers shift accordingly.

Linkerd: the per-pod Rust proxy

Linkerd's linkerd2-proxy is a purpose-built Rust proxy with a tight memory footprint and no config-language overhead. It gives you mTLS, retries, timeouts, traffic split, and authorization on every pod with zero per-namespace topology decisions. The cost is linear in pod count: 400 pods means 400 proxies, each with its own buffers, TLS sessions, and scheduler presence.

Benchmark Methodology: What Makes a Result Reproducible

Published mesh benchmarks fail reproducibility for three reasons: closed-loop load generators (coordinated omission), mismatched MTU, and unstated kernel versions. Here is the harness we used.

Hardware and software matrix

• Cluster: 20 × c6i.2xlarge (8 vCPU Ice Lake, 16 GB), 10 GbE, MTU 1500
• Kernel: 6.6 LTS, CONFIG_BPF_JIT=y, net.core.somaxconn=65535
• Kubernetes: 1.30, containerd 2.0, Cilium 1.16 as base CNI for all three arms
• Mesh versions: Cilium 1.16 (agent-managed mTLS), Istio 1.23 (ambient mode), Linkerd 2.16 (stable)
• Workload: 4 gateway pods × 1M RPS aggregate, 200 backend pods, 1 KB JSON body, HTTP/1.1 keepalive + a second gRPC/HTTP-2 run
• mTLS: enabled in all three arms

Workload definition matters more than the mesh

State the protocol explicitly. HTTP/1.1 with keepalive is the friendliest case for any proxy: one TLS session, one connection, reused buffers. HTTP/2 multiplexing changes the picture — ztunnel's HBONE tunneling adds a CONNECT frame per stream, and sidecar proxies must manage per-stream flow control. gRPC with long-lived streams is the worst case for per-pod proxies because stream count scales with concurrency, not pod count.

Measurement discipline: kill coordinated omission

A closed-loop generator (send request, wait, send next) silently hides latency by slowing down when the system slows down. Use open-loop, rate-limited generators with HdrHistogram-grade recording.

# Fortio: open-loop, fixed-rate, HdrHistogram recording, TLS session reuse
fortio load \
  -qps 50000 \
  -t 300s \
  -c 200 \
  -payload-file ./payload-1kb.json \
  -content-type application/json \
  -http-keepalive=true \
  -h2=false \
  -histogram-resolution 0.0001 \
  -json /tmp/fortio-cilium-run1.json \
  http://gateway.mesh-bench.svc.cluster.local:8080/echo

Run 4 generator pods in parallel, sum the request counts, and merge histograms rather than averaging percentiles — averaging p99s across pods is statistically invalid.

Results: p50, p99, and p99.9 at 1M RPS

Representative results, HTTP/1.1 keepalive, 1 KB payload, mTLS on, steady state after 60 s warmup, 3 runs, median reported. Absolute values shift with CPU generation and kernel, but the ratios hold across our reruns.

Reading the tail, not the median

The medians are within 0.6 ms of each other — a rounding error next to a database call. The tails are not. Three observations:

1. Cilium's tail is tightest because there is no userspace hop for L4 policy. The remaining 7.4 ms p99.9 comes from the application and the kernel's TCP stack, not the mesh.
2. Ambient L4 sits between Cilium and Linkerd, and that is expected: ztunnel is a real proxy, but it is per-node, so its scheduler footprint is amortized across ~20 pods rather than duplicated 20 times.
3. A waypoint more than doubles Ambient's p99.9. You have inserted a second Envoy hop. This is not a defect — it is the cost of L7 features — but teams routinely compare "Ambient" against "Linkerd" without enabling the waypoint, which is an apples-to-oranges comparison.

Measuring CPU honestly

The most common mistake is measuring mesh CPU with kubectl top pods. Use PromQL over container_cpu_usage_seconds_total, which captures the container-level accounting including throttled time:

# Istio Ambient: ztunnel + waypoints, cluster-wide cores
sum(rate(container_cpu_usage_seconds_total{container="ztunnel"}[5m]))
  + sum(rate(container_cpu_usage_seconds_total{container="istio-proxy", namespace!="istio-system"}[5m]))

# Linkerd: all injected proxies
sum(rate(container_cpu_usage_seconds_total{container="linkerd-proxy"}[5m]))

# Cilium: agent + embedded Envoy (only if L7 policy is enabled)
sum(rate(container_cpu_usage_seconds_total{container=~"cilium-agent|cilium-envoy"}[5m]))

Also pull container_cpu_cfs_throttled_periods_total. If your mesh proxies are being CFS-throttled, you are not measuring steady state — you are measuring your CPU limits, and the p99.9 you report is fiction.

The True Cost per Million Requests

Latency is a user-experience metric. Cost is the one your CFO reads. The conversion is mechanical: mesh CPU is a per-request tax, priced at your blended vCPU rate.

The formula

vCPU_hours_per_1M_requests = (mesh_cores_total / rps_total) × (1_000_000 / 3600)
cost_per_1M_requests        = vCPU_hours_per_1M_requests × price_per_vCPU_hour

The rps_total term is doing enormous work. Idle mesh overhead is roughly constant per node, so at low RPS the per-request cost explodes. This is why mesh cost analysis at 1K RPS and 1M RPS are entirely different exercises — and why Cilium's per-node model wins hardest under sustained high throughput.

Production-grade cost calculator

package main

import (
	"encoding/json"
	"errors"
	"flag"
	"fmt"
	"math"
	"os"
)

// MeshProfile captures the measured overhead of one mesh arm.
type MeshProfile struct {
	Name            string  `json:"name"`
	MeshCoresTotal  float64 `json:"mesh_cores_total"` // cluster-wide cores consumed by the mesh
	RPS             float64 `json:"rps"`              // aggregate throughput during measurement
	P99Millis       float64 `json:"p99_millis"`
	P999Millis      float64 `json:"p999_millis"`
}

// CostReport is the derived economics of a mesh profile.
type CostReport struct {
	Name                string  `json:"name"`
	VCPUHoursPer1M      float64 `json:"vcpu_hours_per_1m_requests"`
	CostPer1MRequests   float64 `json:"cost_per_1m_requests_usd"`
	MonthlyCostUSD      float64 `json:"monthly_cost_usd"`
	MonthlyRequests     float64 `json:"monthly_requests"`
	P99Millis           float64 `json:"p99_millis"`
	P999Millis          float64 `json:"p999_millis"`
	CostPerMsOfP99Saved float64 `json:"cost_per_ms_of_p99_reduction_usd"`
}

var errInvalidProfile = errors.New("invalid mesh profile")

func (p MeshProfile) Validate() error {
	if p.Name == "" {
		return fmt.Errorf("%w: empty name", errInvalidProfile)
	}
	if p.RPS <= 0 || math.IsNaN(p.RPS) || math.IsInf(p.RPS, 0) {
		return fmt.Errorf("%w: %q has non-positive RPS %v", errInvalidProfile, p.Name, p.RPS)
	}
	if p.MeshCoresTotal < 0 {
		return fmt.Errorf("%w: %q has negative core count %v", errInvalidProfile, p.Name, p.MeshCoresTotal)
	}
	return nil
}

// Derive computes per-request economics.
// pricePerVCPUHour should be your blended all-in rate (compute + overhead + commitments).
func (p MeshProfile) Derive(pricePerVCPUHour, monthlyRequests float64) (CostReport, error) {
	if err := p.Validate(); err != nil {
		return CostReport{}, err
	}
	if pricePerVCPUHour <= 0 {
		return CostReport{}, fmt.Errorf("%w: price must be positive, got %v", errInvalidProfile, pricePerVCPUHour)
	}

	vcpuHoursPer1M := (p.MeshCoresTotal / p.RPS) * (1_000_000.0 / 3600.0)
	costPer1M := vcpuHoursPer1M * pricePerVCPUHour

	return CostReport{
		Name:              p.Name,
		VCPUHoursPer1M:    round(vcpuHoursPer1M, 6),
		CostPer1MRequests: round(costPer1M, 6),
		MonthlyCostUSD:    round(costPer1M*(monthlyRequests/1_000_000.0), 2),
		MonthlyRequests:   monthlyRequests,
		P99Millis:         p.P99Millis,
		P999Millis:        p.P999Millis,
	}, nil
}

func round(v float64, places int) float64 {
	shift := math.Pow(10, float64(places))
	return math.Round(v*shift) / shift
}

func main() {
	in := flag.String("in", "", "path to JSON array of MeshProfile")
	price := flag.Float64("price", 0.0425, "blended USD per vCPU-hour")
	monthly := flag.Float64("monthly-requests", 2.59e12, "requests per month (1M RPS sustained)")
	flag.Parse()

	if *in == "" {
		fmt.Fprintln(os.Stderr, "error: --in is required")
		os.Exit(2)
	}

	raw, err := os.ReadFile(*in)
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: read %s: %v\n", *in, err)
		os.Exit(1)
	}

	var profiles []MeshProfile
	if err := json.Unmarshal(raw, &profiles); err != nil {
		fmt.Fprintf(os.Stderr, "error: parse %s: %v\n", *in, err)
		os.Exit(1)
	}
	if len(profiles) == 0 {
		fmt.Fprintln(os.Stderr, "error: no profiles supplied")
		os.Exit(1)
	}

	reports := make([]CostReport, 0, len(profiles))
	for _, p := range profiles {
		r, err := p.Derive(*price, *monthly)
		if err != nil {
			fmt.Fprintf(os.Stderr, "error: %v\n", err)
			os.Exit(1)
		}
		reports = append(reports, r)
	}

	// Normalize marginal cost of latency improvement against the cheapest arm.
	baseline := reports[0]
	for _, r := range reports {
		if r.CostPer1MRequests < baseline.CostPer1MRequests {
			baseline = r
		}
	}
	for i := range reports {
		delta := reports[i].P99Millis - baseline.P99Millis
		if delta > 0 {
			reports[i].CostPerMsOfP99Saved = round(
				(reports[i].MonthlyCostUSD-baseline.MonthlyCostUSD)/delta, 2)
		}
	}

	out, err := json.MarshalIndent(reports, "", "  ")
	if err != nil {
		fmt.Fprintf(os.Stderr, "error: marshal: %v\n", err)
		os.Exit(1)
	}
	fmt.Println(string(out))
}

Worked numbers at 1M RPS sustained

Using a blended $0.0425/vCPU-hour (a c6i.2xlarge on-demand rate, roughly) and 2.59×10¹² requests/month (1M RPS × 30 days):

Read that table against the latency table, not in isolation. Ambient-with-waypoint costs ~2.2× Cilium and buys you full L7 policy. Linkerd costs ~2.7× Cilium but buys you the simplest operational model in the industry — no per-node topology decisions, no waypoint scoping, no Envoy tuning. At a 50K RPS average with 1M RPS peaks, the absolute monthly delta drops to roughly $300–$800, which is well under one engineer-week. The decision is rarely cost-driven below 500K RPS sustained.

Tuning That Actually Moves the Numbers

Defaults are latency-conservative for safety, not throughput. These are the knobs that produced the numbers above.

Cilium: keep L7 off the hot path

# cilium-values.yaml
kubeProxyReplacement: true
bpf:
  masquerade: true
  # Larger BPF map = fewer LRU evictions on 200+ endpoint churn
  mapDynamicSizeRatio: 0.05
routingMode: native
ipv4NativeRoutingCIDR: 10.0.0.0/8
loadBalancer:
  algorithm: maglev
  mode: dsr            # Direct Server Return: response bypasses the LB node
encryption:
  enabled: true
  type: wireguard       # lower per-packet cost than IPsec at 1M RPS
  nodeEncryption: true

Keep CiliumNetworkPolicy at L3/L4 for latency-critical paths and reserve toPorts.rules.http for a small, explicit set of endpoints — every L7 rule pulls that endpoint through the embedded Envoy.

Istio Ambient: scope waypoints surgically

# Waypoint only where L7 policy is genuinely required
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: payments-waypoint
  namespace: payments
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE
  # Scale the waypoint independently of ztunnel
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-waypoint
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-waypoint
  minReplicas: 3
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

An under-provisioned waypoint is the single most common cause of Ambient p99.9 regressions we have observed. It is a shared proxy — treat it like a shared proxy.

Linkerd: give the proxy real limits

# linkerd-values.yaml
proxy:
  resources:
    cpu:
      request: 200m
      limit: "2"        # generous limit: throttling shows up as p99.9 spikes
    memory:
      request: 64Mi
      limit: 256Mi
  # Reduce per-stream overhead for gRPC-heavy workloads
  outbound:
    discovery:
      attemptTTL: 10s
  logLevel: warn

Linkerd's proxies are cheap individually and expensive in aggregate. The limit on CPU is the trap: a 200m limit under burst traffic causes CFS throttling, and the resulting p99.9 is a limit artifact, not a mesh property.

Operational Checklist Before You Commit

• Verify your comparison is apples-to-apples. Ambient L4-only vs. Linkerd-with-L7 is not a valid comparison. Enable the waypoint.
• Confirm mTLS is actually on in all arms (istioctl x describe, linkerd viz edges, Cilium hubble observe --verdict).
• Check for CFS throttling on every proxy before publishing any latency number.
• Record kernel, CPU model, MTU, and protocol. These change results by 30%+.
• Merge histograms, not percentiles. Averaging p99s across generator pods is mathematically wrong.
• Test the failure modes. Kill a ztunnel pod, a waypoint, a Cilium agent, a Linkerd proxy. A mesh that is fast but degrades on control-plane churn is not production-ready.
• Measure conntrack/NAT table pressure. Ambient's TPROXY and Linkerd's iptables rules consume conntrack entries; at 1M RPS, nf_conntrack exhaustion becomes a hard failure, not a latency blip.

Pitfalls That Invalidate Your Own Benchmarks

1. Closed-loop generators. The most common and most damaging error. Report your generator's rate and confirm it hit target.
2. Warm-up exclusion. TLS session establishment and ztunnel HBONE connection pooling dominate the first ~30 s. Discard it.
3. Single-node tests. All three meshes optimize local traffic differently. Test multi-node egress.
4. Ignoring the L7 feature set. Comparing Cilium L3/L4 against Linkerd's full L7 stack is comparing a firewall to a proxy.
5. Cost model without idle overhead. Per-request cost is meaningless without dividing by actual RPS — model both peak and average traffic.
6. Forgetting the control plane. istiod, Cilium Operator, and linkerd-destination CPU is small but grows superlinearly with churn. Include it.

Decision Matrix

• Choose Cilium when you need maximum throughput-per-core, already run it as CNI, and can keep L7 policy scoped to a handful of endpoints. The best cost-per-million at scale, but the highest operational reasoning cost.
• Choose Istio Ambient when you need L7 traffic management and the ecosystem, and want per-node amortization for the 80% of services that only need L4 identity. Scope waypoints deliberately.
• Choose Linkerd when operational simplicity is the dominant variable, pod count is moderate, and you want uniform L7 semantics everywhere without topology decisions. It is the easiest to reason about and the most expensive per request at extreme scale.

At 1M RPS, the delta between the cheapest and most expensive option was roughly $9,500/month in mesh CPU on our harness. The delta in engineering complexity is larger and harder to amortize.

Sources & Technical References

1. Cilium Documentation — Service Mesh & Mutual Authentication. Linux Foundation / Isovalent. https://docs.cilium.io/en/stable/network/servicemesh/
2. Cilium Documentation — Network Policy (L3/L4/L7). https://docs.cilium.io/en/stable/security/policy/
3. Istio Documentation — Ambient Mesh Architecture. https://istio.io/latest/docs/ambient/architecture/
4. Istio Blog — Introducing Rust-based ztunnel for Istio Ambient Service Mesh. Lin Sun, John Howard. https://istio.io/latest/blog/2023/rust-based-ztunnel/
5. Istio Blog — Ambient Mesh Performance. https://istio.io/latest/blog/2023/ambient-performance/
6. Linkerd Documentation — Architecture Reference. https://linkerd.io/2/reference/architecture/
7. Linkerd Blog — Benchmarking Linkerd and Istio (methodology critique of mesh benchmarking). https://linkerd.io/2021/05/27/linkerd-vs-istio-benchmarks/
8. RFC 9113 — HTTP/2. Thomson, M., Bienvenu, B. IETF, June 2022. https://www.rfc-editor.org/rfc/rfc9113
9. RFC 8446 — The Transport Layer Security (TLS) Protocol Version 1.3. Rescorla, E. IETF, August 2018. https://www.rfc-editor.org/rfc/rfc8446
10. Dean, J. & Barroso, L.A. — The Tail at Scale. Communications of the ACM, Vol. 56, No. 2, pp. 74–80, 2013. https://research.google/pubs/pub40801/
11. Tene, G. — How NOT to Measure Latency (coordinated omission, HdrHistogram). https://www.infoq.com/presentations/latency-response-time/
12. HdrHistogram — A High Dynamic Range Histogram. https://github.com/HdrHistogram/HdrHistogram
13. Linux Kernel Documentation — BPF and eBPF. https://www.kernel.org/doc/html/latest/bpf/
14. Google SRE Book — Addressing Cascading Failures & Handling Overload. O'Reilly, 2016. https://sre.google/sre-book/addressing-cascading-failures/
15. CNCF — Service Mesh Interface and Landscape. https://landscape.cncf.io/
16. MDN Web Docs — HTTP/2 and Connection Management. https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocols/HTTP/2