The Observability Tax at 1M RPS: OpenTelemetry SDK vs eBPF Auto-Instrumentation vs Prometheus Sidecars

The 1M RPS Observability Tax: What You Are Actually Paying For

At 1M requests per second, one span per request becomes 2.59 trillion spans per month. Observability is no longer a sidecar concern; it is a capacity-planning and unit-economics problem. The tax shows up in four places: application CPU for span creation and attribute encoding, collector CPU for batching and tail sampling, network and storage for span export, and p99 latency inflation from CPU contention, GC, queueing, and cgroup throttling.

The three architectures in this comparison are not interchangeable:

• OpenTelemetry Go SDK: in-process instrumentation with rich semantic attributes, head sampling, batching, and OTLP export.
• eBPF auto-instrumentation: kernel-side protocol tracing with low application CPU, but limited business context and different operational failure modes.
• Prometheus sidecars: metrics collection and isolation, not distributed tracing. Comparing a sidecar to a tracing SDK on 'cost per billion spans' is a category error. Compare it on RED metrics cost and resource isolation.

Benchmark Contract: Reproducible Harness Before Vendor Claims

Workload and SLO guardrails

Use a fixed workload: Go 1.22+, a gRPC unary handler, 1 KiB payloads, one span per request, 100 application pods, and a baseline p99 under 50 ms at 70% CPU. Measure application CPU, node CPU, allocation rate, GC pause, p99 latency, dropped spans, collector queue saturation, and cost per million requests. Run at 50%, 70%, 90%, and 100% CPU saturation. p99 overhead is non-linear near saturation, so a single idle-node benchmark is misleading.

A minimal span benchmark isolates the SDK tax from the network:

func BenchmarkCheckoutSpan(b *testing.B) {
	b.ReportAllocs()
	tracer := otel.Tracer("checkout")
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			ctx, span := tracer.Start(context.Background(), "checkout")
			span.SetAttributes(attribute.String("tenant", "acme"), attribute.Int("items", 3))
			span.End()
			_ = ctx
		}
	})
}

Run it with CPU and memory profiles:

go test -bench=BenchmarkCheckoutSpan -benchmem -cpuprofile=cpu.out -memprofile=mem.out
go tool pprof -top -cum cpu.out

Interpreting overhead table

The following ranges are directional from a controlled Go/gRPC harness with 1 KiB spans. They are not vendor guarantees. The important signal is the order of magnitude and the operational failure mode.

OpenTelemetry Go SDK: Control and Cost

The OTel Go SDK gives you the best semantic fidelity and the clearest cost controls. The expensive parts are span allocation, attribute maps, context propagation, batch queue memory, and export volume. At 1M RPS, full sampling is usually impossible: 2.59T spans/month at 1 KiB each is about 2.59 PB/month before compression and replication. Head sampling at the edge is the first lever; tail sampling at the collector is the second.

A production-oriented tracer provider should set a parent-based ratio sampler, bounded batch queue, resource attributes, and explicit shutdown:

func newTracerProvider(ctx context.Context) (*sdktrace.TracerProvider, error) {
	exp, err := otlptracegrpc.New(ctx,
		otlptracegrpc.WithEndpoint("otel-collector:4317"),
		otlptracegrpc.WithInsecure(),
		otlptracegrpc.WithTimeout(3*time.Second),
	)
	if err != nil {
		return nil, fmt.Errorf("otlp exporter: %w", err)
	}

	res, err := resource.New(ctx,
		resource.WithAttributes(semconv.ServiceName("checkout")),
		resource.WithHost(),
		resource.WithProcessRuntimeDescription(),
	)
	if err != nil {
		return nil, fmt.Errorf("resource: %w", err)
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp,
			sdktrace.WithBatchTimeout(5*time.Second),
			sdktrace.WithMaxExportBatchSize(512),
			sdktrace.WithMaxQueueSize(2048),
		),
		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.01))),
		sdktrace.WithResource(res),
	)
	otel.SetTracerProvider(tp)
	return tp, nil
}

Wire it into gRPC with the stats handler so unary and streaming RPCs share instrumentation:

srv := grpc.NewServer(
	grpc.StatsHandler(otelgrpc.NewServerHandler()),
	grpc.MaxRecvMsgSize(4<<20),
)

Always shut down the provider with a deadline. Otherwise the batch queue can drop final spans during rollout:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := tp.Shutdown(ctx); err != nil {
	log.Printf("trace shutdown: %v", err)
}

eBPF Auto-Instrumentation: Kernel-Side Visibility, User-Space Savings

eBPF tools such as Grafana Beyla, Cilium Tetragon, and Pixie attach kprobes, uprobes, and tracepoints to observe HTTP/gRPC, SQL, and network behavior without linking an SDK into the Go binary. At 1M RPS, the appeal is obvious: lower application CPU and no per-service instrumentation rollout. The trade-off is semantic depth. eBPF can tell you that a gRPC call was slow and which endpoint was involved; it usually cannot tell you which tenant, cart, or feature flag caused it unless you add application attributes.

Operationally, budget for a per-node agent, ring-buffer memory, event loss under load, kernel-version compatibility, and TLS blind spots. Encrypted payloads require uprobes into TLS libraries or explicit plaintext capture, which changes your security posture. Use eBPF for infrastructure coverage, legacy services, and protocol-level RED metrics; use the OTel SDK for revenue-critical business transactions where attributes drive debugging.

Prometheus Sidecars: Metrics Are Not Traces

Prometheus sidecars are often proposed for isolation: each pod gets a scraper or Thanos sidecar with its own resource limits. At 1M RPS, per-pod sidecars become a density problem. If you run 500 pods and each sidecar requests 20m CPU, you reserve 10 vCPU before it helps anyone. If the sidecar is CPU-throttled during a traffic spike, scrape latency and p99 jitter rise together. Prefer node-level agents or a centralized scrape path unless regulatory isolation or multi-tenant boundaries require a sidecar.

A defensive sidecar resource envelope looks like this:

resources:
  requests:
    cpu: 20m
    memory: 64Mi
  limits:
    cpu: 100m
    memory: 128Mi

More importantly, control metric cardinality. A label containing trace_id, user_id, or raw URL path at 1M RPS will explode Prometheus memory and remote-write cost far faster than the sidecar CPU. Sidecars do not emit spans; they do not solve trace sampling, tail sampling, or span storage economics.

p99 Latency Inflation: The Hidden Queueing Effect

The OTel SDK exports asynchronously in normal operation, so the exporter network is not usually on the request path. p99 inflation comes from CPU contention, allocation pressure, GC, batch queue locks, and saturation. Near 90% CPU, an extra 5–10% observability CPU can push queueing latency up disproportionately. eBPF reduces user-space work but adds ring-buffer and agent scheduling overhead. Sidecars add another cgroup competing for the same node. Always benchmark at production CPU headroom, not on an idle node.

Tail Sampling and Collector Tuning: The Highest-Leverage Cost Control

Head sampling at 1% cuts span volume by 100x but may miss rare errors. Tail sampling keeps all error traces, slow traces, and a probabilistic baseline. The critical constraint: all spans for a trace must reach the same tail-sampling collector. Use the load-balancing exporter with trace-ID routing in front of the sampling tier.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024
    spike_limit_mib: 256
  tail_sampling:
    decision_wait: 5s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 1
  batch:
    timeout: 2s
    send_batch_size: 8192
    send_batch_max_size: 10000
exporters:
  otlphttp:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    compression: gzip
    sending_queue:
      enabled: true
      num_consumers: 8
      queue_size: 5000
    retry_on_failure:
      enabled: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlphttp]

True Cost per Billion Spans: A Formula, Not a Sticker Price

Use a unit model instead of vendor list prices. The dominant variables are average span size, ingest price, retention, replication, and collector CPU.

func CostPerBillion(avgSpanBytes int64, ingestPerGB, storagePerGBMonth float64, retentionDays, replicas int, collectorUSD float64) float64 {
	gb := float64(avgSpanBytes) // 1e9 spans * avg bytes / 1e9 bytes per GB
	months := float64(retentionDays) / 30.0
	return gb*(ingestPerGB+storagePerGBMonth*months*float64(replicas)) + collectorUSD
}

At 1 KiB per span, 1B spans is about 1 TB. With $0.25/GB ingest, $0.03/GB-month storage, 30-day retention, and two replicas, the raw cost is roughly $310/billion before collector and vendor margin. At 1% sampling of 1M RPS, you generate about 25.9B spans/month, or roughly $8K/month. At 100% sampling, 2.59T spans/month crosses $800K/month before margin. That is the real reason tail sampling and attribute discipline matter.

Operational Checklist and Pitfalls

• Set a span budget per request: 1M RPS × spans/request × sampling rate.
• Use ParentBased(TraceIDRatioBased) at the edge and tail sampling at the collector.
• Cap attribute cardinality; never put raw payloads, user IDs, or trace IDs in metric labels.
• Benchmark at 70–90% CPU with go test -bench, pprof, and synthetic load.
• Tune GOMEMLIMIT, collector memory_limiter, batch queue size, and exporter retries together.
• Avoid per-pod Prometheus sidecars at high density unless isolation requirements justify the floor cost.
• Alert on dropped spans, queue saturation, exporter retry storms, and collector CPU throttling.
• Treat eBPF as protocol-level coverage, not a replacement for business-semantic tracing.

Sources & Technical References

• OpenTelemetry Go documentation: https://opentelemetry.io/docs/languages/go/
• OpenTelemetry Go SDK trace package: https://pkg.go.dev/go.opentelemetry.io/otel/sdk/trace
• OpenTelemetry Collector tail sampling processor: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor
• OpenTelemetry Collector batch processor: https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor
• OpenTelemetry Collector load-balancing exporter: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter
• OpenTelemetry semantic conventions: https://opentelemetry.io/docs/specs/semconv/
• Prometheus instrumentation practices: https://prometheus.io/docs/practices/instrumentation/
• Thanos sidecar documentation: https://thanos.io/tip/components/sidecar.md/
• eBPF overview: https://ebpf.io/what-is-ebpf/
• Grafana Beyla documentation: https://grafana.com/docs/beyla/latest/
• Cilium Tetragon documentation: https://tetragon.io/docs/
• Go pprof package: https://pkg.go.dev/runtime/pprof
• Go diagnostics guide: https://go.dev/doc/diagnostics
• gRPC Go basics: https://grpc.io/docs/languages/go/basics/
• Kubernetes resource management: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/