Landing Page A/B Testing at 1B Visitors: Bucketing Latency, Flicker Rate, SDK Overhead, and the True Cost per Million Experiments

The Funnel Math Nobody Runs Before Choosing a Platform

A landing page doing 1B visitors/year with 4 concurrent experiments, 3 flags per page, and 2.5 evaluations per session produces roughly 10 billion bucketing decisions per year — about 315M/day, ~3,650/second on average, and 5–8× that at campaign peak.

That number is the one that matters. Not the seat count, not the visual editor, not the Slack integration. Vendors sell you "A/B testing," but what they actually ship is a deterministic hash function plus a config distribution system. Every property that hurts at 1B visitors — bucketing latency, flicker, SDK overhead, unit cost — falls out of those two primitives.

This article quantifies all four with a reproducible harness, an architecture comparison, and a cost model you can paste into a spreadsheet before signing a nine-figure-traffic contract.

Defining the Four Cost Centers Precisely

Ambiguity is how vendors win procurement. Pin these down first.

1. Bucketing latency

Time from "I have a stable unit ID" to "I have a variation key." Sub-millisecond when evaluation is local (in-process, against a cached config). Tens to hundreds of milliseconds when it is a network round trip to a /decide endpoint — and that round trip is on your LCP critical path.

2. Flicker rate

Percentage of sessions where the browser paints the control variant before the experiment variant applies. On a landing page, flicker is not cosmetic — it is a measurable conversion tax, a CLS contributor, and a silent source of mis-attributed events (users who converted during the control paint get bucketed inconsistently).

3. SDK overhead

Four separate costs: gzipped bundle bytes, main-thread parse/compile time, resident memory per evaluation context, and background polling/streaming traffic (which at 3,650 rps is not negligible egress).

4. Cost per million experiments

Dollars per million evaluated impressions served. Not per seat. Not per MTU. This is where platforms diverge by 10–100×, and where self-hosting flips the economics entirely.

How Deterministic Bucketing Actually Works

Every serious platform does the same thing: hash experimentSeed + unitId into a uniform 32-bit integer, modulo it into a bucket space (usually 10,000 buckets), then walk weighted ranges. Here is a production-grade implementation using MurmurHash3 — the same primitive GrowthBook and several others use.

// lib/hash.ts — MurmurHash3 x86 32-bit
export function murmurhash3(key: string, seed = 0): number {
  let remainder = key.length & 3; // key.length % 4
  let bytes = key.length - remainder;
  let h1 = seed;
  const c1 = 0xcc9e2d51;
  const c2 = 0x1b873593;
  let i = 0;
  let k1 = 0;

  while (i < bytes) {
    k1 =
      (key.charCodeAt(i) & 0xff) |
      ((key.charCodeAt(i + 1) & 0xff) << 8) |
      ((key.charCodeAt(i + 2) & 0xff) << 16) |
      ((key.charCodeAt(i + 3) & 0xff) << 24);
    i += 4;

    k1 = Math.imul(k1, c1);
    k1 = (k1 << 15) | (k1 >>> 17);
    k1 = Math.imul(k1, c2);

    h1 ^= k1;
    h1 = (h1 << 13) | (h1 >>> 19);
    h1 = (Math.imul(h1, 5) + 0xe6546b64) | 0;
  }

  k1 = 0;
  switch (remainder) {
    case 3:
      k1 ^= (key.charCodeAt(i + 2) & 0xff) << 16;
    // falls through
    case 2:
      k1 ^= (key.charCodeAt(i + 1) & 0xff) << 8;
    // falls through
    case 1:
      k1 ^= key.charCodeAt(i) & 0xff;
      k1 = Math.imul(k1, c1);
      k1 = (k1 << 15) | (k1 >>> 17);
      k1 = Math.imul(k1, c2);
      h1 ^= k1;
  }

  h1 ^= key.length;
  h1 ^= h1 >>> 16;
  h1 = Math.imul(h1, 0x85ebca6b);
  h1 ^= h1 >>> 13;
  h1 = Math.imul(h1, 0xc2b2ae35);
  h1 ^= h1 >>> 16;

  return h1 >>> 0;
}

// lib/bucketing.ts
import { murmurhash3 } from "./hash";

export type Variation = { key: string; weight: number }; // weights must sum to 1

export interface Assignment {
  experimentKey: string;
  variationKey: string;
  bucket: number; // integer 0..9999
  inExperiment: boolean;
}

export class BucketingError extends Error {
  constructor(message: string, readonly experimentKey: string) {
    super(message);
    this.name = "BucketingError";
  }
}

const BUCKETS = 10_000;

export function assign(
  experimentKey: string,
  unitId: string,
  variations: Variation[],
  coverage = 1,
): Assignment {
  if (!unitId) throw new BucketingError("unitId is empty", experimentKey);

  const totalWeight = variations.reduce((s, v) => s + v.weight, 0);
  if (Math.abs(totalWeight - 1) > 1e-6) {
    throw new BucketingError(`weights sum to ${totalWeight}, expected 1`, experimentKey);
  }

  const bucket = murmurhash3(`${experimentKey}:${unitId}`) % BUCKETS;
  const n = bucket / BUCKETS;

  if (n >= coverage) {
    return { experimentKey, variationKey: variations[0].key, bucket, inExperiment: false };
  }

  const scaled = n / coverage;
  let cumulative = 0;
  for (const v of variations) {
    cumulative += v.weight;
    if (scaled < cumulative) {
      return { experimentKey, variationKey: v.key, bucket, inExperiment: true };
    }
  }

  const last = variations[variations.length - 1];
  return { experimentKey, variationKey: last.key, bucket, inExperiment: true };
}

The non-negotiable property here is cross-runtime determinism. If your edge worker and your origin disagree by a single bucket, you get split-brain assignment, duplicated exposures, and a permanent SRM you will spend a quarter debugging.

Benchmarking Bucketing Latency Without Fooling Yourself

Most "benchmarks" people publish are wrong because they include JIT warm-up, hash the same ID repeatedly (perfect cache behavior), or measure wall-clock across an event loop tick. Here is a harness that avoids all three.

// bench/bucketing.bench.ts
import { performance } from "node:perf_hooks";
import { assign, type Variation } from "../lib/bucketing";

const VARIATIONS: Variation[] = [
  { key: "control", weight: 0.34 },
  { key: "hero_v2", weight: 0.33 },
  { key: "hero_v3", weight: 0.33 },
];

const EXPERIMENTS = ["lp_hero", "lp_cta", "lp_pricing"];

export interface LatencyStats {
  n: number;
  p50: number;
  p90: number;
  p99: number;
  max: number;
}

export function summarize(samples: number[]): LatencyStats {
  const s = [...samples].sort((a, b) => a - b);
  const at = (q: number) => s[Math.min(s.length - 1, Math.floor(q * s.length))];
  return { n: s.length, p50: at(0.5), p90: at(0.9), p99: at(0.99), max: s[s.length - 1] };
}

export function run(iterations = 1_000_000): LatencyStats {
  // Warm up the JIT before recording.
  for (let i = 0; i < 50_000; i++) assign("lp_hero", `warm_${i}`, VARIATIONS);

  const samples = new Array<number>(iterations);
  for (let i = 0; i < iterations; i++) {
    const unitId = `u_${i % 50_000}`; // models a 50k-visitor recurring population
    const t0 = performance.now();
    for (const key of EXPERIMENTS) assign(key, unitId, VARIATIONS);
    samples[i] = performance.now() - t0;
  }
  return summarize(samples);
}

console.table(run());

Representative results on a c7g.large, in-process, three evaluations per sample (reproduce it yourself — do not take these as gospel):

The conclusion is unambiguous: remote evaluation is 3–4 orders of magnitude slower than local evaluation. If your landing page awaits a /decide call before rendering, you have already lost more conversion than the experiment will ever recover.

Architecture Shootout: Where Each Platform Actually Wins

GrowthBook — self-hosted economics

Open-source core, warehouse-native analytics, and a hashing spec documented well enough that you can implement bucketing yourself and stay bit-compatible. At 10B evaluations/year, self-hosting turns marginal cost into approximately zero. You pay in SRE time and SLA ownership.

Statsig — the strongest default for high-volume SaaS

Server SDKs pull a ruleset on init and evaluate locally; updates arrive via background polling or streaming. The evaluation model is fast and the experimentation statistics (CUPED, sequential testing) are genuinely strong. The pricing unit — events/impressions — is where 1B visitors becomes a budget conversation.

LaunchDarkly — the reliability reference implementation

Streaming (SSE) config propagation with strong consistency guarantees, excellent audit/approval workflows, and mature local evaluation in server SDKs. You are paying for operational trust more than for statistics. At landing-page scale, its cost is defensible only if flags — not experiments — are the primary workload.

Optimizely — optimize for the visual editor, not for 1B rps

Feature Experimentation with datafile-based local evaluation is solid. Web Experimentation historically relies on a client-side snippet with an anti-flicker pattern, which is the wrong primitive at this scale: you are paying main-thread cost and a hidden-render window on every landing page view.

Killing Flicker on Landing Pages: Edge Rewrite Is the Only Real Answer

There are exactly three patterns, ordered by correctness:

1. Edge bucketing + URL rewrite. HTML arrives already correct. Flicker rate is structurally 0%.
2. SSR with an inlined flag payload. Correct, but couples flag freshness to your deploy cadence.
3. Anti-flicker snippet (hide, wait, reveal). Last resort. It hides body until flags resolve or a timeout fires, costing LCP and risking a blank page.

Pattern 1, in a Next.js Edge middleware:

// middleware.ts — Next.js App Router, Edge runtime
import { NextRequest, NextResponse } from "next/server";
import { assign, type Variation } from "./lib/bucketing";

const HERO_VARIANTS: Variation[] = [
  { key: "control", weight: 0.5 },
  { key: "hero_v2", weight: 0.5 },
];

const STICKY_COOKIE = "lp_exp_hero";

export const config = {
  matcher: ["/lp/:path*"],
  runtime: "edge",
};

export function middleware(req: NextRequest): NextResponse {
  // RSC and prefetch requests must reuse the assigned variant, never re-bucket.
  if (req.headers.get("rsc") === "1" || req.headers.get("next-router-prefetch")) {
    return NextResponse.next();
  }

  const uid = req.cookies.get("uid")?.value ?? crypto.randomUUID();
  const sticky = req.cookies.get(STICKY_COOKIE)?.value;
  const { variationKey } = assign("lp_hero", uid, HERO_VARIANTS);
  const variant = sticky ?? variationKey;

  const url = req.nextUrl.clone();
  url.pathname = `/lp/${variant}${req.nextUrl.pathname.replace("/lp", "")}`;

  const res = NextResponse.rewrite(url);
  res.cookies.set(STICKY_COOKIE, variant, {
    httpOnly: false,
    sameSite: "lax",
    maxAge: 60 * 60 * 24 * 30,
  });
  res.headers.set("x-exp-lp-hero", variant);
  res.headers.set("cache-control", "private, no-store, must-revalidate");
  return res;
}

If you must use an anti-flicker snippet, at least make it fail open — never leave the page hidden forever:

<style>
  html.exp-anti-flicker body { visibility: hidden !important; opacity: 0 !important; }
</style>
<script>
  (function () {
    var root = document.documentElement;
    var TIMEOUT_MS = 1200;
    root.classList.add("exp-anti-flicker");
    function reveal() {
      root.classList.remove("exp-anti-flicker");
      clearTimeout(timer);
    }
    var timer = setTimeout(reveal, TIMEOUT_MS); // fail-open guard
    window.__expReady = reveal;                // call from your decide() callback
  })();
</script>

SDK Overhead: Approximate Browser Footprint

Order-of-magnitude figures, gzipped, current major versions — verify against Bundlephobia before contracting, since these change every quarter:

At 1B visitors, a 30 kB delta is ~30 TB of additional egress per full-population load. Model it.

The True Cost per Million Experiments

Impressions, not seats, drive the bill. Here is a model you can run:

// ops/cost-model.ts
export interface PlatformCostModel {
  name: string;
  monthlyPlatformFee: number;   // seats + contractual minimums
  includedImpressions: number;  // per month
  pricePerMillionOverage: number;
  infraMonthly: number;         // self-host: compute + Postgres + Redis + CDN
  sreCostMonthly: number;       // loaded engineer hours
}

export interface Workload {
  monthlyImpressions: number;
}

export function monthlyCost(m: PlatformCostModel, w: Workload): number {
  const overageMillions =
    Math.max(0, w.monthlyImpressions - m.includedImpressions) / 1_000_000;
  return (
    m.monthlyPlatformFee +
    overageMillions * m.pricePerMillionOverage +
    m.infraMonthly +
    m.sreCostMonthly
  );
}

export function costPerMillion(m: PlatformCostModel, w: Workload): number {
  if (w.monthlyImpressions <= 0) throw new RangeError("impressions must be > 0");
  return (monthlyCost(m, w) / w.monthlyImpressions) * 1_000_000;
}

For 1B visitors/year with 3 experiments each — 250M impressions/month — the shape of the answer:

• Managed, event-priced SaaS: typically low-to-mid single-digit dollars per million at committed volume; list pricing is far higher. At 250M impressions that is a five-figure monthly line item, before analytics warehouse costs.
• Self-hosted GrowthBook: ~$600–900/month of infrastructure (3× c7g.xlarge, Postgres, Redis, CDN) with near-zero marginal cost per impression. Add 0.25–0.5 FTE of SRE time.

The crossover point is roughly where your impressions exceed ~50–100M/month. Below that, buy the SaaS; above it, self-hosting pays for its own SRE time within a quarter.

Pre-Launch Operational Checklist

• Bucketing is local, not a per-request network call, on every render-critical path.
• The same hash function and salt run on edge, origin, and client — verified by a golden-vector test.
• Sticky assignment is cookie- and server-side persisted; expect 5–20% cookie loss from ITP/ETP.
• cache-control: private, no-store on bucketed HTML; variant is in the cache key, not the cookie alone.
• Exposure events are sent via navigator.sendBeacon() on visibilitychange, not blocking fetch.
• SRM check runs daily and alerts at p < 0.001.
• Guardrail metrics (LCP, INP, CLS, JS error rate) are hard-stop conditions, not footnotes.

Pitfalls That Quietly Invalidate Results at This Scale

Sample Ratio Mismatch from cache poisoning. If a CDN caches the bucketed HTML without a variant-keyed cache, your 50/50 split silently becomes 92/8 — and every downstream statistic is garbage. Fabijan et al. (KDD '19) catalog this as one of the most common causes of invalid experiments.

Bot and prefetch traffic. At 1B visitors, crawler and link-preview traffic can be 3–8% of "users." Excluding it after bucketing is fine; excluding it before bucketing creates a biased hash population.

Novelty and primacy effects. A landing page headline experiment needs at least two full business cycles before you trust the lift. Most platforms' sequential tests will happily stop you early on noise.

Instrumenting the wrong denominator. Cost per impression and lift per converted session are different denominators. Mixed denominators are the most common reason a "winning" landing page variant produces no revenue lift.

Sources & Technical References

• GrowthBook Documentation — SDK reference and experiment configuration.
• GrowthBook: Build Your Own SDK (hashing spec) — the canonical bucketing/hash specification.
• Statsig Documentation — server SDK local evaluation and ruleset refresh semantics.
• Statsig Node Server SDK — initialization, background sync, and evaluation model.
• Optimizely Developer Documentation — Feature Experimentation datafile and local evaluation.
• Optimizely Web Experimentation: Anti-Flicker Snippet — official behavior and timeout semantics.
• LaunchDarkly Documentation — streaming architecture and SDK evaluation modes.
• LaunchDarkly: Client-side vs Server-side SDKs — evaluation and connectivity trade-offs.
• MurmurHash3 reference implementation (Austin Appleby / smhasher) — the hashing primitive used by most experimentation SDKs.
• MDN: Performance API — performance.now() and the Performance Timeline.
• MDN: Navigator.sendBeacon() — non-blocking exposure telemetry.
• web.dev: Core Web Vitals — LCP, INP, and CLS thresholds used as guardrails.
• Next.js Middleware Documentation — edge rewrite semantics and matcher configuration.
• Cloudflare Workers Runtime APIs — Web Crypto and edge execution constraints.
• W3C Performance Timeline Level 2 — the spec underpinning latency instrumentation.
• Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
• Fabijan, A., et al. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments. KDD '19.