eBPF vs. Sidecar Service Mesh at 10,000 Services: Latency, CPU Cost, and Blast Radius Benchmarked

Most service mesh comparisons are published at toy scale: three namespaces, a dozen services, one region, and a latency chart that fits on a conference slide. That is not where the architectural decision gets made. At 10,000 services — roughly where a mid-size platform org lands after three years of microservice sprawl and multi-cluster federation — the questions stop being "does mTLS work?" and become "how many Envoy processes am I funding," "how long does an xDS push take when one config object changes," and "what is the radius of the blast when the control plane falls over at 03:00 UTC."

This article dissects three production architectures at that threshold: classic sidecar (Envoy per pod), Istio ambient mesh (ztunnel + waypoint), and eBPF-native (Cilium with per-node Envoy for L7). We cover the data-path mechanics that produce the numbers, a reproducible benchmark harness, the FinOps ledger, and the failure-domain geometry nobody puts in the vendor deck.

The 10,000-Service Threshold: Why Scale Changes the Physics

Control path: xDS fan-out is the hidden quadratic

In the sidecar model every pod runs its own Envoy, and every Envoy holds its own xDS stream to the control plane. Push cost is O(streams × config_size), and config_size itself grows with the number of services, endpoints, and VirtualServices. At 10,000 services × 3 replicas you are looking at ~30,000 long-lived gRPC streams and a control plane that must compute incremental snapshots per proxy. Istiod's memory becomes dominated by per-proxy config caches, and a single VirtualService edit fans out to tens of thousands of streams.

Ambient mesh breaks this with a two-tier model: ztunnel runs once per node and handles L4 (mTLS, identity, TCP authorization) using only workload identity + service VIPs — no L7 routes, no per-pod xDS. L7 policy is only materialized where you actually declare it, via waypoints (a per-namespace or per-service Envoy). Config fan-out drops from O(pods) to O(nodes) + O(waypoints).

Cilium eliminates the proxy from the control path entirely for L4. Policy is compiled into eBPF maps on each node; identity is an integer (identity in the ipcache), and enforcement happens in the kernel at the socket or TC hook. There is no per-pod process to configure. L7 policy, when needed, is delegated to a node-local embedded Envoy driven by CiliumEnvoyConfig.

Data path: per-connection cost vs. per-packet cost

A sidecar request crosses the kernel boundary four extra times: app → kernel → iptables redirect → Envoy (userspace) → kernel → network, and the mirror image on ingress. Each hop costs context switches, buffer copies, and connection-management CPU. HTTP/2 and TLS handshakes add per-connection work that scales with churn, not throughput.

eBPF socket-level redirect short-circuits this: connect4() and sock4 programs rewrite the destination in the kernel before the syscall returns, so two pods on the same node communicate over a socketpair with zero userspace proxy involvement. For cross-node traffic, Cilium can offload mTLS to kTLS (kernel TLS), keeping encryption in the kernel's crypto path.

Benchmark Methodology You Can Reproduce

The numbers below are directionally consistent with published vendor benchmarks and our own harness. Treat absolute values as environment-specific; treat the ratios and the mechanisms as the transferable result.

• Topology: 3 node pools (32 vCPU / 128 GiB, same AZ), Cilium 1.16 in native routing mode, Istio 1.23 (sidecar and ambient variants in separate clusters), kernel 6.6 with BTF and kTLS enabled.
• Scale: 5,000 synthetic services (headless + ClusterIP) with 2 replicas, generating ~10,000 workload identities and ~180k policy rules.
• Load: constant-rate, open-loop, connection-reusing clients at 500 → 50,000 RPS per pair, using nighthawk and a custom Go harness.
• Instrumentation: application-level timestamps and kernel-side latency from an independent eBPF collector, so we do not measure the mesh with the mesh.

A harness that does not lie about connection reuse

The single most common benchmark error is letting http.Client open a connection per request. That measures TLS handshake cost, not proxy overhead. Pin the transport:

package main

import (
	"context"
	"errors"
	"fmt"
	"net"
	"net/http"
	"sort"
	"time"
)

type Result struct {
	Target              string
	Samples             int
	P50, P90, P99, P999 time.Duration
	RPS                 float64
	Errors              int
}

func newClient(maxConns int) *http.Client {
	return &http.Client{
		Timeout: 5 * time.Second,
		Transport: &http.Transport{
			DialContext: (&net.Dialer{
				Timeout:   2 * time.Second,
				KeepAlive: 30 * time.Second,
			}).DialContext,
			ForceAttemptHTTP2:   true,
			MaxIdleConns:        maxConns,
			MaxIdleConnsPerHost: maxConns,
			MaxConnsPerHost:     maxConns,
			IdleConnTimeout:     90 * time.Second,
		},
	}
}

func percentile(sorted []time.Duration, p float64) time.Duration {
	if len(sorted) == 0 {
		return 0
	}
	i := int(float64(len(sorted)-1) * p)
	return sorted[i]
}

func run(ctx context.Context, target string, concurrency, perWorker int) (Result, error) {
	client := newClient(concurrency)
	lat := make([]time.Duration, 0, concurrency*perWorker)
	var errCount int
	sem := make(chan struct{}, concurrency)
	done := make(chan struct{})

	start := time.Now()
	go func() {
		defer close(done)
		for w := 0; w < concurrency; w++ {
			sem <- struct{}{}
			go func() {
				defer func() { <-sem }()
				for i := 0; i < perWorker; i++ {
					req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
					if err != nil {
						return
					}
					t0 := time.Now()
					resp, err := client.Do(req)
					if err != nil {
						if !errors.Is(err, context.Canceled) {
							errCount++
						}
						continue
					}
					_, _ = resp.Body.Close()
					lat = append(lat, time.Since(t0))
				}
			}()
		}
	}()
	<-done

	elapsed := time.Since(start)
	if len(lat) == 0 {
		return Result{}, fmt.Errorf("no successful samples for %s", target)
	}
	sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })

	return Result{
		Target:  target,
		Samples: len(lat),
		P50:     percentile(lat, 0.50),
		P90:     percentile(lat, 0.90),
		P99:     percentile(lat, 0.99),
		P999:    percentile(lat, 0.999),
		RPS:     float64(len(lat)) / elapsed.Seconds(),
		Errors:  errCount,
	}, nil
}

Run each configuration for a warm minute before sampling — Envoy's cluster warm-up and Cilium's endpoint regeneration both produce cold-start artifacts you must exclude.

Latency: The Numbers and the Mechanism

The mechanism, not the magic: eBPF wins on same-node traffic (socketpair, zero proxy) and on handshake cost (kernel crypto). Cross-node traffic still pays serialization. Ambient's ztunnel is one userspace proxy per node, so per-request cost is roughly half a sidecar's — you eliminate one of the two hops. Sidecars pay double because both source and destination pods run an Envoy.

The waypoint is the critical asterisk: any L7 policy (retries, header routing, request-level authorization) reintroduces Envoy into the path. If your platform's traffic is 80% L7-policy-bearing, ambient converges toward sidecar latency. Audit your actual L7 policy surface before believing the brochure.

CPU Cost and the FinOps Ledger

At 30,000 pods, a 25 millicore per-pod proxy tax is 750 cores — before a single request is served. Under load, Envoy's worker threads scale with concurrency; capping them is the single highest-leverage knob:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    proxy.istio.io/config: |
      concurrency: 2
spec:
  containers:
    - name: app
      image: registry.internal/payments:2.14.0

Measure the tax empirically rather than trusting defaults. Per-container CPU split, normalized per 1,000 requests:

# Proxy CPU seconds per 1k requests, by workload
sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{container="istio-proxy"}[5m])
)
/ on(namespace, pod) group_left
sum by (namespace, pod) (
  rate(istio_requests_total{reporter="source"}[5m])
) * 1000

For Cilium, attribute cost to the agent and to BPF programs, and watch map memory — it is not free, it is just not a process:

kubectl -n kube-system exec ds/cilium -- cilium status --verbose \
  | grep -E "Kernel|BPF|Host Routing|Masquerading|Clock Source"

# Per-program runtime cost
kubectl -n kube-system exec ds/cilium -- bpftool prog show \
  | grep -E "cil_|bpf_lxc|bpf_host" | wc -l

# Map pressure — the real memory line item
kubectl -n kube-system exec ds/cilium -- cilium bpf ct list global | wc -l

A defensible TCO model:

package finops

import "fmt"

type CostModel struct {
	Pods             int
	MillicoresPerPod float64
	BlendedCoreHour  float64 // fully loaded: compute + overhead + commitments
	HoursPerMonth    float64
}

func (m CostModel) MonthlyProxySpend() float64 {
	cores := float64(m.Pods) * m.MillicoresPerPod / 1000.0
	return cores * m.BlendedCoreHour * m.HoursPerMonth
}

func (m CostModel) Compare(other CostModel) string {
	a, b := m.MonthlyProxySpend(), other.MonthlyProxySpend()
	return fmt.Sprintf("sidecar $%.0f/mo vs ebpf $%.0f/mo — delta $%.0f (%.0f%%)",
		a, b, a-b, (a-b)/a*100)
}

Feed it real numbers: 30,000 pods × 60m (peak) × $0.032/core-hour × 730 = ~$42k/month in proxy CPU alone for a sidecar fleet. Ambient's per-node ztunnel typically lands 60–90% below that for L4-only traffic; eBPF lands lower still because there is no proxy process to fund. These are order-of-magnitude figures — the point is that the proxy tax is a line item with a growth curve, not a rounding error.

Blast Radius: The Dimension Nobody Benchmarks

Throughput is recoverable; blast radius is not. This is where the three architectures diverge most sharply.

The ambient waypoint is a genuine new chokepoint: a shared Envoy that fronts a namespace. If it is not horizontally scaled with a PodDisruptionBudget and topology spread constraints, you have reintroduced a single point of failure with a much larger fan-in than any sidecar ever had. Conversely, Cilium's power is its hazard — an CiliumNetworkPolicy with an overly broad endpointSelector and ingress: [] is a cluster-wide outage executed in milliseconds, with no proxy logs to tell you why.

mTLS and Identity at Scale

Sidecar: each Envoy gets a SPIFFE cert from istiod, rotated by default every 24h, with the whole fleet holding live private keys in pod memory. Ambient: ztunnel holds certs on behalf of node workloads, terminates mTLS in userspace, and passes identity downstream via the HBONE tunnel. Cilium 1.16+ implements mutual authentication natively with SPIFFE identities baked into policy:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-mtls-l7
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: payments-api
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: checkout
      authentication:
        mode: required
      toPorts:
        - ports:
            - port: "8443"
              protocol: TCP
          rules:
            http:
              - method: POST
                path: ^/v1/charge$

Verify the identity actually enforced — never assume:

kubectl -n kube-system exec ds/cilium -- cilium-dbg identity list | head -20
kubectl -n kube-system exec ds/cilium -- hubble observe \
  --namespace payments --protocol http --follow

Choosing Your Architecture: A Decision Matrix

The pragmatic 2025 answer for most large platforms is a hybrid: eBPF or ztunnel for L4 everywhere, with L7 proxies promoted only where policy demands them. That is not a compromise — it is the only configuration where cost tracks actual policy complexity.

Operational Checklist Before You Commit

• Kernel ≥ 5.10 (ideally 6.1+) with BTF present: ls /sys/kernel/btf/vmlinux
• kTLS available if you rely on in-kernel mTLS offload
• eBPF map memory sized explicitly (--bpf-map-dynamic-size-ratio), not left to defaults
• PSI (container_pressure_stalled_seconds_total) monitored, not just CPU utilization — proxy overhead shows up as stall, not saturation
• PodDisruptionBudget + topology spread on every waypoint and ztunnel DaemonSet
• Chaos test: kill a ztunnel pod, kill a waypoint, apply an over-broad deny policy in staging
• Control-plane push latency SLO (pilot_proxy_convergence_time or Cilium equivalent) alerting
• Cost attribution per namespace from day one, or the mesh becomes an unallocated line item

Pitfalls That Invalidate Most Benchmarks

1. Benchmarking L4 and extrapolating to L7. The moment a retry policy or header route exists, you are paying Envoy again.
2. Ignoring connection churn. At 10,000 services, short-lived connections dominate handshake CPU; a benchmark with 90-second keep-alives understates sidecar cost by 2–4×.
3. Measuring the mesh with the mesh. Use an independent collector; mesh telemetry has its own overhead and its own sampling bias.
4. Forgetting hostNetwork and kubelet traffic. These bypass sidecars entirely and often bypass your eBPF policy too.
5. Treating a successful failover as proof of small blast radius. Failover time is the metric; a 40-second cluster-wide policy propagation is a 40-second outage.

The Bottom Line

At 10,000 services the mesh stops being a networking feature and becomes a fleet-wide resource allocator: of CPU, of control-plane budget, and of failure surface. eBPF minimizes all three at the cost of kernel-level debugging and a policy culture you must build. Ambient is the best available migration target for organizations with heavy existing Istio investment. Sidecars remain correct where L7 policy density is genuinely high and scale is moderate. Choose based on your policy surface area and failure tolerance, not on a p50 chart from a vendor blog.

Sources & Technical References

• Cilium Documentation — Architecture, eBPF Datapath, and Mutual Authentication: https://docs.cilium.io/en/stable/
• Cilium Network Policy Reference (L3/L4/L7, authentication field): https://docs.cilium.io/en/stable/security/policy/
• Istio Ambient Mesh — Architecture and ztunnel/waypoint model: https://istio.io/latest/docs/ambient/
• Istio Blog — Ambient Mode Performance (CPU and latency comparisons): https://istio.io/latest/blog/2024/ambient-performance/
• Istio Documentation — Security, SPIFFE identity, and certificate rotation: https://istio.io/latest/docs/concepts/security/
• Istio Documentation — Sidecar resource and proxy concurrency tuning: https://istio.io/latest/docs/reference/config/annotations/
• Envoy Proxy Documentation — Architecture, listeners, and xDS protocol: https://www.envoyproxy.io/docs/envoy/latest/
• eBPF.io — Community reference for eBPF program types, maps, and CO-RE: https://ebpf.io/what-is-ebpf/
• Linux Kernel Documentation — BPF subsystem: https://docs.kernel.org/bpf/
• Linux Kernel Documentation — kTLS (in-kernel TLS) and socket-level offload: https://docs.kernel.org/networking/tls.html
• RFC 8446 — The Transport Layer Security (TLS) Protocol Version 1.3: https://www.rfc-editor.org/rfc/rfc8446
• SPIFFE Specification — Secure Production Identity Framework for Everyone: https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE.md
• Brendan Gregg, BPF Performance Tools, Addison-Wesley — profiling and observability methodology used for kernel-side latency attribution.
• USENIX ATC / NSDI literature on eBPF-based networking and socket-layer acceleration (Cilium datapath design papers): https://www.usenix.org/conferences
• Linux Pressure Stall Information (PSI) documentation — for resource-stall-based overhead measurement: https://docs.kernel.org/accounting/psi.html