RUM SDKs at 1 Billion Sessions: Datadog vs Sentry vs New Relic vs web-vitals

Why Your RUM SDK Becomes the Regression It Measures

Every real user monitoring SDK is a performance-critical script that executes on the same main thread as your application, during the single most contended window of the page lifecycle. At 1 billion sessions per month, an extra 12 ms of blocking per session is 12,000,000 seconds of aggregate blocked time — and, critically, that time lands in the input delay and presentation delay components of Interaction to Next Paint (INP), not in some neutral measurement lane.

The question that actually matters at scale is not which dashboard renders prettier funnels. It is five numbers per SDK: bytes parsed, observers attached, tasks queued on the main thread, bytes beaconed, and dollars billed. This article gives you a reproducible harness to measure the first four and the arithmetic to derive the fifth.

The Four-Layer Overhead Model

Treat any RUM SDK as a stack of four costs. They compound multiplicatively on low-end Android hardware, which is where your p75 Core Web Vitals actually live.

Layer 1 — Boot, parse, and compile

The agent bundle must be fetched, parsed, and compiled before it can observe anything. A 2 KB library and a 50 KB library do not differ by 25x in perceived cost — they differ by whether the parse lands inside or outside the critical rendering window.

Layer 2 — Observers and listeners

PerformanceObserver subscriptions (longtask, event, largest-contentful-paint, layout-shift, resource) are cheap individually. The cost is in swallowed event types: passive listeners on click, pointerdown, and keydown used for interaction attribution add per-interaction processing time, and any listener that is not explicitly passive: true on a scroll or touch target risks adding scroll-blocking time.

Layer 3 — Serialization and queueing

Session state, view state, and buffered events are serialized into JSON on the main thread. This is where large agents quietly pay: JSON.stringify over a growing breadcrumb or resource buffer is O(n) main-thread work that recurs on every flush.

Layer 4 — Transport

Beacons consume bandwidth and, more subtly, compete for the browser's limited keepalive quota. Chromium enforces a 64 KiB total in-flight limit for keepalive fetches, and navigator.sendBeacon() is spec-advised to cap payloads around 64 KiB, returning false when the queue is full. A vendor that fires three beacons on page hide will drop data on slow connections — silently.

Building a Reproducible SDK Benchmark Harness

Stop trusting vendor micro-sites. Measure on your own HTML, on a throttled mid-tier device profile, with each SDK enabled in isolation. This harness captures long tasks, total blocking time above the 50 ms threshold, and real beacon bytes.

// rum-harness.ts — run once per SDK variant under identical network/CPU throttling
type Sample = {
  sdk: string;
  longTaskCount: number;
  totalBlockingMs: number;  // sum of (duration - 50) for tasks > 50ms
  beaconCalls: number;
  beaconBytes: number;
};

const supported = (PerformanceObserver as unknown as {
  supportedEntryTypes?: readonly string[];
}).supportedEntryTypes ?? [];

async function measure(sdk: string, run: () => Promise<void>): Promise<Sample> {
  const durations: number[] = [];
  let observer: PerformanceObserver | undefined;

  if (supported.includes('longtask')) {
    observer = new PerformanceObserver((list) => {
      for (const entry of list.getEntries()) durations.push(entry.duration);
    });
    observer.observe({ type: 'longtask', buffered: false });
  }

  const encoder = new TextEncoder();
  let beaconCalls = 0;
  let beaconBytes = 0;
  const originalSendBeacon = navigator.sendBeacon?.bind(navigator);

  if (originalSendBeacon) {
    navigator.sendBeacon = (url: string | URL, data?: BodyInit | null): boolean => {
      beaconCalls += 1;
      if (typeof data === 'string') beaconBytes += encoder.encode(data).byteLength;
      else if (data instanceof Blob) beaconBytes += data.size;
      else if (data instanceof ArrayBuffer) beaconBytes += data.byteLength;
      return originalSendBeacon(url, data);
    };
  }

  await run(); // load page, script a fixed interaction script, wait for idle + flush

  observer?.disconnect();
  if (originalSendBeacon) navigator.sendBeacon = originalSendBeacon;

  const totalBlockingMs = durations
    .filter((d) => d > 50)
    .reduce((sum, d) => sum + (d - 50), 0);

  return { sdk, longTaskCount: durations.length, totalBlockingMs, beaconCalls, beaconBytes };
}

Drive it with a fixed script of interactions (a scroll, a click on a non-trivial handler, a route transition) so INP attribution is comparable across vendors. Run each variant at least 30 times; long-task counts are noisy and medians lie less than means.

INP Regression: Where SDK Overhead Actually Hurts

INP measures the full interaction latency from input to next paint, decomposed into input delay, processing time, and presentation delay. Your SDK can degrade all three:

• Input delay — a long task from your SDK's serialization is already running when the user taps. The input queues behind it.
• Processing time — the SDK's own passive listeners run inside your event handler's frame budget.
• Presentation delay — the SDK's observer callbacks run after the handler but before the next frame, delaying paint.

The 50 ms long-task threshold is the right lens because any task over 50 ms is a candidate to delay input, and the 75th percentile INP threshold of 200 ms means a handful of bad frames per session can tip you from good to needs improvement across an entire population.

Attributing SDK-caused tasks with LoAF

Chrome 123+ ships the Long Animation Frames API, which attributes long frames to specific scripts by URL. This is the fastest way to prove whether a long task belongs to your application or to the agent.

if (PerformanceObserver.supportedEntryTypes?.includes('long-animation-frame')) {
  new PerformanceObserver((list) => {
    for (const frame of list.getEntries() as PerformanceLongAnimationFrameTiming[]) {
      if (frame.duration < 50) continue;

      for (const script of frame.scripts) {
        const isVendorAgent = /(datadoghq|sentry-cdn|nr-data|newrelic)/.test(script.sourceURL ?? '');
        if (!isVendorAgent) continue;

        // Report to a *separate*, cheap pipeline — not your RUM vendor.
        void fetch('/internal/rum-attribution', {
          method: 'POST',
          keepalive: true,
          headers: { 'content-type': 'application/json' },
          body: JSON.stringify({
            invoker: script.invoker,
            source: script.sourceURL,
            duration: Math.round(script.duration),
            styleAndLayout: Math.round(script.forcedStyleAndLayoutDuration),
            frameDuration: Math.round(frame.duration),
          }),
        });
      }
    }
  }).observe({ type: 'long-animation-frame', buffered: true });
}

Fixing agent-induced delay with scheduler.yield

If you own the integration layer (or you are evaluating web-vitals as the baseline), never run serialization or multi-event flushes in one task. Break them and yield back so input can be processed.

declare global {
  interface Scheduler {
    yield?: () => Promise<void>;
  }
}

async function flushQueue<T>(queue: T[], send: (batch: T[]) => void, batchSize = 25): Promise<void> {
  for (let i = 0; i < queue.length; i += batchSize) {
    send(queue.slice(i, i + batchSize));
    // Yield to the event loop so pending input is dispatched before the next batch.
    if (typeof scheduler?.yield === 'function') await scheduler.yield();
    else await new Promise<void>((r) => setTimeout(r, 0));
  }
}

The web-vitals Baseline: Raw Measurement With Zero Vendor Tax

web-vitals v4 is under 2 KB gzipped and ships INP attribution (interaction target, LoAF entries) in the core bundle — a genuinely strong signal for the byte cost. It has no transport, no session identity, no backend. You own the pipeline.

// vitals-sender.ts — production-grade, bfcache-safe, sampled
import { onCLS, onINP, onLCP, onTTFB, type Metric } from 'web-vitals';

const ENDPOINT = 'https://rum.example.com/v1/vitals';
const SAMPLE_RATE = 0.1; // 10% of sessions is statistically sufficient at 1B scale
const sampled = Math.random() < SAMPLE_RATE;

function emit(metric: Metric): void {
  if (!sampled) return;

  const payload = JSON.stringify({
    name: metric.name,
    value: Math.round(metric.value),
    rating: metric.rating,
    id: metric.id,
    navigationType: metric.navigationType,
    target: metric.attribution?.interactionTarget ?? null,
  });

  const blob = new Blob([payload], { type: 'application/json' });
  const queued = navigator.sendBeacon?.(ENDPOINT, blob) ?? false;

  if (!queued) {
    // keepalive is capped at 64KiB total in flight — keep this body tiny.
    void fetch(ENDPOINT, { method: 'POST', body: payload, keepalive: true, headers: { 'content-type': 'application/json' } }).catch(() => {
      /* never let telemetry throw into the host app */
    });
  }
}

onCLS(emit);
onINP(emit);
onLCP(emit);
onTTFB(emit);

The trade-off is everything above the metric: no error correlation, no session replay, no distributed traces into your backend, no per-route regression detection. You are buying that capability from a vendor, or building it (ClickHouse plus a 2–4 FTE platform team).

Vendor-by-Vendor Engineering Assessment

Exact gzipped sizes shift every release — measure them in your bundle with webpack-bundle-analyzer or rollup-plugin-visualizer, not from marketing pages.

Datadog RUM

The browser agent is the heaviest of the four in default configuration, in the tens-of-KB gzipped range for @datadog/browser-rum alone, and materially larger once session replay is enabled. It collects views, resources, long tasks, errors, and user actions by default. Two configuration decisions dominate your INP impact: trackUserInteractions (adds passive listeners and per-action spans) and sessionReplaySampleRate (replay records DOM mutations continuously). Use trackViewsManually in SPAs to avoid spurious view boundaries, and set a proxy URL if you care about ad-blocker loss rates. It is the most operationally complete of the three, and correspondingly the one most likely to appear in your own LoAF attribution data.

Sentry Performance

Sentry's browserTracingIntegration is span-based: one root pageload or navigation span, with an idle timeout that keeps the transaction open until activity settles. The overhead model is therefore per-transaction duration, not per-session — a page that idles for four seconds with background activity accumulates span work. tracesSampleRate is your primary cost and CPU lever; set it well below 1.0 at scale. The base SDK is the lightest of the three commercial agents in typical web configurations, and its error-first heritage means breadcrumbs are the dominant memory consumer.

New Relic Browser

New Relic's loader architecture (nr-loader-spa, nr-loader-rum, nr-loader-lite) is the most tunable of the three: the lite loader strips SPA instrumentation, which is often the single largest CPU reduction available without leaving the platform. Its harvest cycle batches events and flushes on visibilitychange, which is the correct pattern. Watch for the server-side agent injection path — it adds a synchronous-ish script reference that competes with your critical path unless served async with a preconnect.

web-vitals

Not a RUM product. It is the control group. If a commercial agent does not beat it on INP p75 by a margin large enough to justify its byte and CPU cost, you are paying for dashboards, not signal.

The True Cost Per Million Sessions

Use a formula, not a blog table — list prices change quarterly and volume discounts at 1B sessions are typically 40–60% off list.

monthly_cost = sessions × P_session
             + events   × P_event        (spans, browser events, replays)
             + ingress_GB × P_GB
             + retention_TB × P_TB_month

Order-of-magnitude anchors (verify against current vendor pricing pages):

The counterintuitive finding: raw bandwidth is a rounding error. At 30 KB per session, 1B sessions is ~30 TB/month — roughly $1,500 of egress. The entire cost is license, not bytes. That is why sampling and event-volume reduction beat payload optimization by two orders of magnitude on the invoice, and why tracesSampleRate and sessionReplaySampleRate are the two most valuable numbers in your config.

Break-even for building in-house sits roughly at 20–50M sessions/month at list price, rising to 100M+ once enterprise discounts apply and you price your platform team honestly at $250k–$400k loaded per engineer per year.

Operational Checklist Before You Sign

• Benchmark each agent against web-vitals only, on your own page, on a 4x-throttled mid-tier device — not on a laptop.
• Publish INP p75 with the agent on and off; require the delta to be stated in the contract review.
• Confirm the agent uses passive: true on touchstart/touchmove/wheel handlers.
• Verify beacon payloads stay under the keepalive cap; confirm sendBeacon returning false is handled with a fallback.
• Confirm flushing is bound to visibilitychange → hidden, not unload (bfcache-safe).
• Set tracesSampleRate / sessionReplaySampleRate before launch, not after the first invoice.
• Run LoAF attribution in production for two weeks; attribute every >100 ms frame to a script URL.
• Model cost at 1x, 5x, and 20x current traffic using the formula above.
• Negotiate a data-retention clause; retention is the silent second invoice.

Pitfalls to Avoid

Benchmarking on an idle page. An agent that costs 30 ms on an idle page can cost 200 ms on a page with 60 resources and three analytics tags, because observer callbacks contend for the same frames.

Ignoring the sampling paradox. Session replay and full tracing rarely fail on average hardware — they fail in the p75 tail on low-end devices, which is exactly the population Core Web Vitals scores. Sample by device class, not uniformly.

Double-instrumenting. Running a commercial agent alongside a hand-rolled web-vitals sender is common and usually wasteful; it doubles observers and produces two conflicting INP numbers. Pick one, or make the second a debug-only build flag.

Treating INP as a frontend-only metric. INP regression is frequently caused by server response time pushing the input handler's data dependency late. Correlate interaction spans with backend traces before you blame the SDK.

Assuming the loader is free. Server-side agent injection paths are convenient and hostile to LCP if they are not async with a preconnect to the collector origin.

Sources & Technical References

• W3C Event Timing API (INP specification and durationThreshold semantics) — https://w3c.github.io/event-timing/
• W3C Long Tasks API 1.0 (50 ms threshold, longtask entry type) — https://w3c.github.io/longtasks/
• W3C Long Animation Frames API (script-level attribution) — https://w3c.github.io/long-animation-frames/
• web.dev: Interaction to Next Paint (INP) — https://web.dev/articles/inp
• web.dev: Optimize long tasks — https://web.dev/articles/optimize-long-tasks
• Chrome for Developers: scheduler.yield() — https://developer.chrome.com/blog/use-scheduler-yield
• MDN: Navigator.sendBeacon() (payload limits, false return semantics) — https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon
• WHATWG Fetch Standard: keepalive flag and in-flight quota — https://fetch.spec.whatwg.org/#request-keepalive-flag
• MDN: Page Visibility API (visibilitychange / hidden) — https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API
• web-vitals library (source, v4 attribution, bundle size) — https://github.com/GoogleChrome/web-vitals
• Datadog RUM Browser Monitoring documentation — https://docs.datadoghq.com/real_user_monitoring/browser/
• Datadog Real User Monitoring pricing — https://www.datadoghq.com/pricing/?product=real-user-monitoring
• Sentry for JavaScript: BrowserTracing integration and sampling — https://docs.sentry.io/platforms/javascript/tracing/
• New Relic Browser agent: loaders, harvest cycle, SPA instrumentation — https://docs.newrelic.com/docs/browser/browser-monitoring/getting-started/introduction-browser-monitoring/
• W3C Performance Timeline (buffered entries, supportedEntryTypes) — https://w3c.github.io/performance-timeline/