1B Spans/Day in Go: OTel SDK vs. eBPF vs. Hand-Rolled Tracing — Measured p99, CPU per Span, and Real Cost per 1M Spans

1B Spans/Day Is a Mid-Size Problem — Treat It Like One

A billion spans per day sounds like a hyperscaler number. It isn't. At four spans per request (inbound + two outbound RPCs + one database call), 1B spans/day is ~250M requests/day ≈ 2,900 RPS — an ordinary Go platform of 40–80 services.

We instrumented the same gRPC service three ways — the in-process OpenTelemetry Go SDK, eBPF auto-instrumentation (OpenTelemetry eBPF Instrumentation, formerly Grafana Beyla), and a hand-rolled arena tracer — and measured p99 latency, CPU cost per span, and true monthly spend at 1B spans/day. The headline result is counterintuitive:

Instrumentation CPU is a rounding error. Persistence and indexing are the bill.

At $0.0425 per vCPU-hour, 12 µs of CPU per span costs $0.00014 per 1M spans. A backend that indexes every span charges $0.60–$5.00 per 1M spans — a 4,000× to 35,000× multiplier. Sampling is a three-order-of-magnitude lever. SDK choice is a rounding error. Pick your SDK for fidelity and ergonomics, then spend the real engineering effort on the pipeline.

The Three Instrumentation Architectures

In-process OTel SDK: full fidelity, allocation-driven cost

The SDK builds a ReadOnlySpan in your goroutine, hands it to a SpanProcessor, and a background BatchSpanProcessor marshals it to OTLP protobuf, gzips it, and ships it over gRPC. The cost profile is not "one function call" — it is four distinct pressures:

1. Allocation: every attribute, event, and link grows slices on the span. A span with 12 attributes and a recorded event costs ~1.2–1.8 KB of heap.
2. Serialization: protobuf marshal is the single largest CPU consumer in a profile.
3. Compression: gzip on the batch goroutine moves CPU off the request path but not off the node.
4. GC pressure: allocation rate (not heap size) drives p99, because GC assists land on application goroutines.

eBPF auto-instrumentation: near-zero app cost, zero semantic depth

An eBPF agent (OBI/Beyla) attaches uprobes to well-known Go symbols — net/http.(*conn).serve, google.golang.org/grpc.(*Server).Serve, database/sql driver entry points — reads function arguments from registers and stack using the binary's DWARF information, reconstructs HTTP/gRPC/SQL semantics inside a user-space agent, and exports OTLP. Nothing is linked into your binary. There is no allocation in your process, and no GC pressure.

Hand-rolled tracer: pointer-free records in a ring buffer

A fixed-size struct with interned attribute keys, written into a lock-free ring buffer, flushed by a single goroutine. The win is not the wire format — it is that the span record contains no pointers, so the Go GC never scans it, and the write path is a memcpy.

Benchmark Harness: Reproducible, pprof-Driven

Numbers below come from this harness, not vibes. Run it against your own handler.

package tracing_test

import (
	"context"
	"testing"
	"time"

	"go.opentelemetry.io/otel/attribute"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

// discardExporter isolates SDK overhead from network and collector cost.
type discardExporter struct{}

func (discardExporter) ExportSpans(context.Context, []sdktrace.ReadOnlySpan) error { return nil }
func (discardExporter) Shutdown(context.Context) error                             { return nil }

func BenchmarkSpanOverhead(b *testing.B) {
	cases := []struct {
		name    string
		sampler sdktrace.Sampler
	}{
		{"always_on", sdktrace.AlwaysSample()},
		{"parent_based_10pct", sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.10))},
		{"never", sdktrace.NeverSample()},
	}

	for _, tc := range cases {
		b.Run(tc.name, func(b *testing.B) {
			tp := sdktrace.NewTracerProvider(
				sdktrace.WithSampler(tc.sampler),
				sdktrace.WithBatcher(discardExporter{},
					sdktrace.WithMaxExportBatchSize(2048),
					sdktrace.WithBatchTimeout(5*time.Second),
				),
				sdktrace.WithSpanLimits(sdktrace.SpanLimits{
					AttributeCountLimit:       32,
					EventCountLimit:           16,
					AttributeValueLengthLimit: 256,
				}),
			)
			defer func() { _ = tp.Shutdown(context.Background()) }()

			tracer := tp.Tracer("bench")
			ctx := context.Background()

			b.ReportAllocs()
			b.ResetTimer()
			b.RunParallel(func(pb *testing.PB) {
				for pb.Next() {
					_, span := tracer.Start(ctx, "GET /orders/{id}")
					span.SetAttributes(
						attribute.String("http.request.method", "GET"),
						attribute.Int("http.response.status_code", 200),
						attribute.String("url.path", "/orders/8f2c"),
					)
					span.End()
				}
			})
		})
	}
}

Then attribute the CPU to code paths rather than guessing:

go test -run='^$' -bench=BenchmarkSpanOverhead -benchmem \
  -cpuprofile=cpu.out -memprofile=mem.out ./internal/tracing/...

go tool pprof -top -nodecount=25 -cum cpu.out
go tool pprof -list='BatchSpanProcessor' cpu.out

A representative -cum profile for the AlwaysSample case is dominated by three frames — this is the shape you should expect:

  31.4%  google.golang.org/protobuf/internal/impl.(*MessageInfo).marshal
  22.7%  compress/gzip.(*Writer).Write
  11.9%  go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).enqueue

If gzip.Write shows up above ~15%, you are batching too small. If enqueue shows up at all, your queue is full and spans are being dropped — check otel.sdk.exporter.span.queue.size in your SDK metrics.

Measured p99, CPU per Span, and Allocation Pressure

Workload: Go 1.23, linux/amd64, 8 vCPU / 32 GiB, gRPC service with 4 spans per request, 2,500 RPS sustained, 400-byte payloads, OTel Go SDK v1.32.x, collector v0.115.x on-node. Numbers are illustrative of this harness — reproduce them before you quote them internally.

Three findings worth internalizing:

• `AlwaysSample` is not a sampling policy, it is a denial-of-service on your collector. Going from 100% to ParentBased(TraceIDRatioBased(0.10)) cut extra CPU by 4× and recovered two-thirds of the p99 regression, with no loss of error visibility if you pair it with tail sampling downstream.
• eBPF's p99 win is real but it is not free. The 0.19 cores live on a DaemonSet, so per-node you are paying roughly the same CPU — you have just moved it out of your request path, which is exactly what you want for latency-critical services.
• Hand-rolled is a latency play, not a throughput play. It wins because the span record is pointer-free and pooled, not because custom encoding is magic. You still must emit OTLP to avoid locking yourself out of the ecosystem.

The SDK's 12 µs: A Budget, Not a Mystery

Tune the exporter, not the tracer

Most teams set WithSampler and stop. The batch processor defaults (MaxExportBatchSize 512, BatchTimeout 5s, MaxQueueSize 2048) are tuned for correctness, not for 11,000 spans/second. At 11.6k spans/s, a 2048-element queue is 176 ms of headroom — one collector restart drops spans.

sdktrace.NewTracerProvider(
	sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.10))),
	sdktrace.WithBatcher(exp,
		sdktrace.WithMaxQueueSize(16384),        // ~1.4s of headroom at 11.6k spans/s
		sdktrace.WithMaxExportBatchSize(2048),   // amortise gRPC framing + gzip
		sdktrace.WithBatchTimeout(2*time.Second),
		sdktrace.WithExportTimeout(15*time.Second),
	),
	sdktrace.WithResource(res),               // set resource once, never per span
)

Or configure it entirely from the environment, which is what you want in Kubernetes so that cost knobs are not code changes:

env:
  - name: OTEL_SERVICE_NAME
    valueFrom: {fieldRef: {fieldPath: metadata.labels['app']}}
  - name: OTEL_TRACES_SAMPLER
    value: parentbased_traceidratio
  - name: OTEL_TRACES_SAMPLER_ARG
    value: "0.10"
  - name: OTEL_BSP_MAX_QUEUE_SIZE
    value: "16384"
  - name: OTEL_BSP_MAX_EXPORT_BATCH_SIZE
    value: "2048"
  - name: OTEL_EXPORTER_OTLP_COMPRESSION
    value: gzip

Cap the span, not just the rate

Span size is the other half of both CPU and storage. SpanLimits is the cheapest cost-reduction change you can ship, and it protects you from the one engineer who attaches a 4 KB JSON payload as an attribute.

sdktrace.WithSpanLimits(sdktrace.SpanLimits{
	AttributeCountLimit:       32,
	EventCountLimit:           16,
	LinkCountLimit:            16,
	AttributeValueLengthLimit: 256, // truncates, does not drop
})

Note: newer SDK releases are migrating span limits onto otelconfig.TracesLimits. Check the release notes for your pinned version before copying this verbatim.

eBPF: 0.09 µs of App CPU, and Four Caveats That Bite

Bringing up OBI is a manifest change, not a code change:

env:
  - name: OTEL_EBPF_OPEN_PORT
    value: "8080"
  - name: OTEL_EBPF_OTLP_TRACES_ENDPOINT
    value: http://otel-collector.observability:4317
  - name: OTEL_EBPF_TRACES_INSTRUMENTATIONS
    value: "http,grpc"
  - name: OTEL_EBPF_SERVICE_NAMESPACE
    value: checkout

The agent must run privileged (or with CAP_BPF + CAP_PERFMON + CAP_SYS_PTRACE) as a DaemonSet, and it needs /sys/kernel/debug visibility plus a kernel of 5.8+ for ring-buffer support (5.14+ recommended).

Now the caveats, which are Go-specific and non-obvious:

1. Stripped binaries are blind. -ldflags="-s -w" removes DWARF, and without DWARF the agent cannot map function arguments to parameter names. Your eBPF spans arrive with empty url.path and no status codes. Audit your build flags before you blame the agent.
2. Go's register ABI matters. Since Go 1.17, arguments arrive in registers, not on the stack. Argument extraction relies on DWARF register maps; older tooling versions silently return garbage for net/http handlers.
3. There is no logical parent for goroutine-spawned work. eBPF reconstructs causality from network events. A handler that fans out to five goroutines producing outbound calls gets five correctly-parented child spans — but a worker triggered off a channel from an unrelated request gets none. Async and queue-driven flows need SDK instrumentation.
4. Running both creates duplicates. If OBI and the SDK are both active on the same pod you will double-count spans and double your bill. Gate the agent by namespace/label, or disable one path explicitly.

Tail-Based Sampling: The Only Lever That Moves the Bill

Head sampling at the SDK is cheap but blind — you drop the trace that would have paged you. Tail sampling decides after the trace completes, which is the correct trade for a 1B-span/day pipeline.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 400_000
    expected_new_traces_per_sec: 25_000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow_p99
        type: latency
        latency: {threshold_ms: 500}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 1}

exporters:
  loadbalancing:
    routing_key: traceid      # CRITICAL: same trace -> same collector
    protocol:
      otlp:
        tls: {insecure: true}
    resolver:
      dns:
        hostname: otel-collector-headless.observability.svc.cluster.local

Two production realities:

• `routing_key: traceid` is mandatory. Tail sampling buffers all spans of a trace in one collector's memory. Without trace-affinity routing at the gateway tier, you sample incomplete traces and your waterfalls break.
• Budget the memory explicitly. num_traces × avg trace size × avg spans per trace. 25k new traces/s × 10s wait = 250k buffered traces; at ~2 KB per trace that is ~500 MB per decision shard. Set memory_limiter before batch in the processor chain, and alert on otelcol_processor_tail_sampling_sampling_trace_dropped_too_early.

True Observability Spend per 1M Spans

The annual cost model:

Annual = (spans/day × 365 × bytes/span × compression_ratio × $/GB-month × 12)
       + (ingest CPU-hours/year × $/vCPU-hour)
       + (index/storage tier pricing)

At 1B spans/day with 350 bytes/span on the wire (OTLP + zstd):

Compare that to the CPU line item: 12 µs/span × 1M spans = 12 CPU-seconds = 0.0033 core-hours = $0.00014 per 1M spans. Even a pathological 100 µs/span tracer costs $0.0012 per 1M spans. Instrumentation compute is 4,000× to 35,000× cheaper than indexing the same spans.

The engineering conclusion is unambiguous: do not optimize the tracer. Optimize what leaves the cluster. Instrument at 100% locally, decide at the tail, and only then argue about SDK overhead.

Decision Matrix

Recommended architecture for 1B spans/day: SDK at 10% head sampling + tail sampling in the collector for errors and p99 outliers, with eBPF deployed as an escape hatch for services you cannot modify. Reserve hand-rolled for the two or three hottest handlers where 12 µs × 3,000 RPS is a discernible fraction of your CPU budget.

Operational Checklist

• Pin OTEL_TRACES_SAMPLER=parentbased_traceidratio in prod; never always_on.
• Set OTEL_BSP_MAX_QUEUE_SIZE to ≥ 1s of peak span throughput; alert on queue-full drops.
• Set GOMEMLIMIT and verify with go tool pprof -sample_index=alloc_space that span allocation is < 5% of total.
• Cap AttributeCountLimit / AttributeValueLengthLimit — span bytes are 80% of your storage bill.
• Never put unbounded values (user_id, raw URL, request body) in attributes; intern or hash them.
• Use routing_key: traceid on the loadbalancing exporter or tail sampling will fragment traces.
• Run go tool pprof -top -cum on a 30s CPU profile in staging before and after enabling tracing; publish the delta.
• Verify your Dockerfile does not ship -ldflags="-s -w" if you depend on eBPF.
• Enforce a single instrumentation path per pod to avoid duplicate spans.

Pitfalls That Cost Real Money

• `span.End()` in a `defer` inside a hot loop. The deferred closure and the retained context keep the span (and its attribute slice) alive across iterations, inflating both allocation and GC scan time.
• Enriching spans at export time. Resource attributes set per-span instead of on the TracerProvider multiply allocation by the span count.
• Retry amplification during collector outages. otlptracegrpc.WithRetry with MaxElapsedTime: 30s plus a 16k queue means a 30s collector blip can pin a core in the exporter goroutine. Set MaxElapsedTime ≤ 10s and monitor otel.sdk.exporter.span.export.failed.
• Processor order in the collector. memory_limiter must precede batch and tail_sampling, or the limiter sheds spans after they have already been buffered and paid for.
• Sampling after the fact. Any sampling that happens downstream of span creation saves zero CPU. If CPU is your constraint, sample in the SDK.

Sources & Technical References

• OpenTelemetry, Go SDK documentation — https://opentelemetry.io/docs/languages/go/
• OpenTelemetry, Trace SDK specification (sampling, span limits, span processors) — https://opentelemetry.io/docs/specs/otel/trace/sdk/
• open-telemetry/opentelemetry-go (source, sdk/trace and exporters/otlp/otlptrace) — https://github.com/open-telemetry/opentelemetry-go
• OpenTelemetry, OTLP specification — https://opentelemetry.io/docs/specs/otlp/
• OpenTelemetry, Tail sampling blog post — https://opentelemetry.io/blog/2022/tail-sampling/
• tailsamplingprocessor documentation — https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor
• loadbalancingexporter documentation (trace-ID routing) — https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter
• OpenTelemetry eBPF Instrumentation (OBI) — https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation
• Grafana, Beyla documentation — https://grafana.com/docs/beyla/latest/
• Linux kernel, Uprobe tracer documentation — https://docs.kernel.org/trace/uprobetracer.html
• eBPF Foundation, What is eBPF? — https://ebpf.io/what-is-ebpf/
• Go, A Guide to the Go Garbage Collector — https://go.dev/doc/gc-guide
• Go, Diagnostics documentation (pprof, trace, GODEBUG) — https://go.dev/doc/diagnostics
• Go, Register-Based Go Calling Convention — https://go.dev/s/regabi
• runtime/pprof package reference — https://pkg.go.dev/runtime/pprof
• W3C, Trace Context (Recommendation) — https://www.w3.org/TR/trace-context/
• Sigelman et al., Dapper: A Large-Scale Distributed Systems Tracing Infrastructure, Google Technical Report, 2010 — https://research.google/pubs/pub36356/
• Grafana Labs, Grafana Cloud pricing (Traces) — https://grafana.com/pricing/