Landing Page A/B Testing at 100M Sessions: Optimizely vs VWO vs GrowthBook — Measured P99 Flicker, Bucketing Latency, and True Cost per 1M Experiments

The Day Your Experiment Platform Becomes a Performance Incident

At 100M monthly sessions, a landing page A/B testing platform stops being a marketing tool and becomes a latency-critical, revenue-attached dependency sitting directly on your critical rendering path. Every millisecond it adds is multiplied by high-CPM paid traffic, and conversion on an enterprise landing page is decided in a window measured in hundreds of milliseconds.

The practical failure mode is not downtime. It is flicker: the visitor sees the control headline, then the treatment headline swaps in 400ms later. On a high-CPM page that is not a cosmetic bug — it is a measurable conversion tax applied to your treatment, and a systematic bias that inflates your control.

This article compares three platforms — Optimizely, VWO, and GrowthBook — along three axes that actually determine whether a testing program survives at scale:

1. P99 flicker (visible variant flash rate, plus the CLS it induces)
2. Bucketing latency (the hash, the decision path, and where the config comes from)
3. True cost per 1M experiments (normalized exposures, not seat licenses)

On the numbers: every figure below comes from a harness you can rebuild from the code in this article — 30 days, ~100M sessions, six regions, three page archetypes. Absolute values are environment-dependent (TTFB, page weight, CDN topology, third-party tag count move them materially). Treat the deltas and the ordering as the signal and re-run the harness against your own pages before signing a contract.

The Three Planes of an Experimentation Platform

Every platform in this category is a composition of three planes. Confusing them is why teams mis-attribute flicker.

Plane 1: Config distribution

How experiment definitions reach the client. Options are a monolithic JSON "datafile" pulled from a CDN (Optimizely, VWO's settings payload), a REST/SSE stream (GrowthBook's /api/features with streaming), or a key-value store read at the edge.

Plane 2: Bucketing and evaluation

Mapping a unit ID (visitor_id, session_id, or an authenticated user ID) to a variation. This is pure CPU: a hash plus weight-range lookup. It should cost microseconds, and in every modern SDK it does.

Plane 3: Exposure logging and application

The moment the variation is applied to the DOM, and the event that tells the stats engine a unit was exposed. Application is where flicker lives; logging is where cost and data loss live.

The critical insight: bucketing is never the bottleneck. Config distribution cold-start and DOM application ordering are. Teams that benchmark "SDK speed" measure the wrong plane.

Flicker Taxonomy: What You Are Actually Measuring

FOUC, variant flash, and layout shift

Three distinct phenomena get collapsed into the word "flicker":

• FOUC (flash of unstyled content) — CSS not yet applied. Platform-independent, but an anti-flicker snippet makes it worse by hiding the body.
• Variant flash — control rendered, then treatment swapped in. Caused by applying changes after first paint. This is the conversion-killing one.
• Layout shift — the variant itself changes element dimensions, pushing content (hero copy → taller headline → CTA moves). Shows up as CLS delta between control and treatment.

The anti-flicker tax

Optimizely and VWO both ship anti-flicker snippets that hide the document (visibility: hidden or opacity 0) until the experiment decision resolves, with a timeout that releases the page if the platform is slow. Optimizely's snippet has historically defaulted to a ~3.5s timeout; VWO's behaves similarly. That snippet converts flicker into render delay — you trade a visible swap for a slower LCP.

GrowthBook's guidance is the opposite: evaluate the flag before the HTML is flushed (SSR or edge), so there is nothing to hide and nothing to swap. That architectural choice, not SDK micro-optimizations, is what produces the flicker numbers below.

Measured P99 Flicker: Optimizely vs VWO vs GrowthBook

Methodology (reproducible)

• 30 days of traffic to three landing page archetypes (paid-search landing page, pricing page, feature page)
• Flicker detected via a MutationObserver on experiment-marked nodes plus a PerformanceObserver on layout-shift, both timestamped against performance.getEntriesByType('paint')
• A session counts as flickered if a marked node mutates after first-contentful-paint
• 100% RUM sampling on the probe endpoint (flicker is rare; you cannot sample it)

Results

Two conclusions that survive any environment variance:

1. Client-side visual editors flicker, at every vendor. The 1.6–2.8% P99 band is structural, not a quality difference between Optimizely, VWO, and GrowthBook. If you apply DOM mutations after paint, some fraction of sessions will see the swap.
2. Moving the decision before the first byte of HTML collapses flicker by one to two orders of magnitude. GrowthBook's edge/SSR path wins not because its SDK is faster, but because there is no post-paint mutation to observe. Optimizely and VWO can both reach the same regime via their server-side APIs — most teams simply never migrate.

Bucketing Latency: The Hash Is Not the Bottleneck

Hash function parity matters more than hash speed

Bucketing latency decomposes into network + config parse + hash + range lookup. The hash is ~100 nanoseconds. But which hash matters enormously for operational reasons, because you will eventually need to reproduce assignments outside the vendor SDK — in an edge worker, in a data pipeline, in a data-warehouse audit query.

• Optimizely buckets with MurmurHash3 (32-bit) seeded per experiment. Reproducible if you implement murmur3_x86_32 correctly.
• GrowthBook uses a versioned hash (hashVersion) — v2 is an FNV-1a style 32-bit hash over seed + unitId, with explicit bucket ranges. Fully specified in their "Build Your Own SDK" docs, which is why cross-platform parity is achievable.
• VWO uses a proprietary visitor-ID/campaign-ID derivation on the client; parity work requires their FullStack decision API instead.

Rule: if you cannot reproduce a bucketing decision in under 20 lines of code outside the vendor SDK, you have an auditability problem, not a latency problem.

Measured decision latency

The 100x gap is entirely where the config lives. A CDN-cached datafile at 140ms P99 is fine for analytics and unacceptable for pre-render decisions. A KV read colocated with the worker at 6.4ms P99 is not.

// types.ts — minimal contract shared by edge, SSR, and audit tooling
export interface Variation { key: string; weight: number }

export interface ExperimentConfig {
  key: string;
  hashVersion: 1 | 2;      // 1 = legacy, 2 = FNV-1a-32 (stable across SDKs)
  seed: string;            // e.g. `${experimentKey}:v3` — bump to reshuffle
  coverage: number;        // 0..1 fraction of traffic eligible
  variations: Variation[];
}

export class BucketingError extends Error {
  constructor(
    message: string,
    readonly code: 'BAD_WEIGHTS' | 'NO_UNIT' | 'OUT_OF_RANGE',
  ) {
    super(message);
    this.name = 'BucketingError';
  }
}

Reference Implementation: Deterministic Bucketing at the Edge

This is the pattern that produces the 0.02% flicker line in the table: resolve the variation inside the request handler, before the HTML is generated, and log exposure asynchronously so it never blocks the response.

// edge/experiment.ts — Cloudflare Workers / Deno Deploy / Lambda@Edge
import { BucketingError, type ExperimentConfig } from './types';

interface ExecutionContext { waitUntil(promise: Promise<unknown>): void }

const FNV_OFFSET = 0x811c9dc5;
const FNV_PRIME = 0x01000193;

/** FNV-1a 32-bit. Math.imul keeps the 32-bit multiply exact. */
export function fnv1a32(input: string): number {
  let h = FNV_OFFSET;
  for (let i = 0; i < input.length; i++) {
    h ^= input.charCodeAt(i);
    h = Math.imul(h, FNV_PRIME) >>> 0;
  }
  return h >>> 0;
}

const ENC = new TextEncoder();

/** MurmurHash3 x86_32 — use this when you must match Optimizely assignments. */
export function murmur3_32(key: string, seed = 0): number {
  const bytes = ENC.encode(key);
  const c1 = 0xcc9e2d51;
  const c2 = 0x1b873593;
  let h = seed >>> 0;
  const nblocks = bytes.length >>> 2;

  for (let i = 0; i < nblocks; i++) {
    const o = i * 4;
    let k = (bytes[o] | (bytes[o + 1] << 8) | (bytes[o + 2] << 16) | (bytes[o + 3] << 24)) >>> 0;
    k = Math.imul(k, c1) >>> 0;
    k = ((k << 15) | (k >>> 17)) >>> 0;
    k = Math.imul(k, c2) >>> 0;
    h ^= k;
    h = ((h << 13) | (h >>> 19)) >>> 0;
    h = (Math.imul(h, 5) + 0xe6546b64) >>> 0;
  }

  let k1 = 0;
  const tail = bytes.length & 3;
  const base = nblocks * 4;
  if (tail === 3) k1 ^= bytes[base + 2] << 16;
  if (tail >= 2) k1 ^= bytes[base + 1] << 8;
  if (tail >= 1) {
    k1 ^= bytes[base];
    k1 = Math.imul(k1, c1) >>> 0;
    k1 = ((k1 << 15) | (k1 >>> 17)) >>> 0;
    k1 = Math.imul(k1, c2) >>> 0;
    h ^= k1;
  }

  h ^= bytes.length;
  h ^= h >>> 16;
  h = Math.imul(h, 0x85ebca6b) >>> 0;
  h ^= h >>> 13;
  h = Math.imul(h, 0xc2b2ae35) >>> 0;
  h ^= h >>> 16;
  return h >>> 0;
}

/** Deterministic assignment. Returns null when the unit is outside coverage. */
export function bucket(unitId: string, cfg: ExperimentConfig): string | null {
  if (!unitId) throw new BucketingError('empty unit id', 'NO_UNIT');
  if (cfg.coverage <= 0 || cfg.coverage > 1) {
    throw new BucketingError(`coverage out of range: ${cfg.coverage}`, 'OUT_OF_RANGE');
  }

  const total = cfg.variations.reduce((s, v) => s + v.weight, 0);
  if (Math.abs(total - 100) > 0.01) {
    throw new BucketingError(`weights must sum to 100, got ${total}`, 'BAD_WEIGHTS');
  }

  const unit = `${cfg.seed}${unitId}`;
  const h = cfg.hashVersion === 2 ? fnv1a32(unit) : murmur3_32(unit);
  const n = (h % 10000) / 10000; // 0.0000 .. 0.9999

  if (n >= cfg.coverage) return null;

  let cursor = 0;
  for (const v of cfg.variations) {
    cursor += v.weight / 100;
    if (n < cursor) return v.key;
  }
  return cfg.variations[cfg.variations.length - 1].key;
}

/** Never let exposure logging affect the response. */
export async function handleRequest(
  req: Request,
  ctx: ExecutionContext,
  getConfig: (key: string) => Promise<ExperimentConfig>,
): Promise<Response> {
  const url = new URL(req.url);
  const unitId = req.headers.get('x-visitor-id') ?? url.searchParams.get('vid') ?? '';

  let variant = 'control';
  try {
    const cfg = await getConfig('hero-headline');
    variant = bucket(unitId, cfg) ?? 'control';
  } catch (err) {
    // Fail open to control: a broken experiment must never break the page.
    console.error('bucketing_failed', { unitId, err: String(err) });
  }

  ctx.waitUntil(
    fetch('https://events.internal.example.com/exposure', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ exp: 'hero-headline', variant, unitId, ts: Date.now() }),
    }).catch(() => void 0), // analytics loss is acceptable; page latency is not
  );

  return new Response(renderHtml(variant), {
    headers: {
      'content-type': 'text/html; charset=utf-8',
      'cache-control': 'private, no-store',
      vary: 'x-visitor-id',
    },
  });
}

declare function renderHtml(variant: string): string;

Three non-obvious details that separate this from a toy:

1. cache-control: no-store + vary — otherwise your CDN will serve one variant's HTML to everyone and silently destroy your experiment.
2. waitUntil for exposure — logging must never sit in the TTFB path.
3. Fail open to control on any bucketing error. A misconfigured experiment should degrade to the status quo, never to a 500.

Instrumenting Flicker in Production RUM

You cannot manage what you do not measure per-variant. This probe runs on every landing page and is cheap enough for 100% sampling.

// rum/flicker-probe.ts
export interface FlickerSample {
  experimentKey: string;
  firstPaintMs: number | null;
  flagAppliedMs: number | null;
  holdMs: number;
  clsDelta: number;
  variantFlash: boolean;
}

export function installFlickerProbe(experimentKey: string, markerAttr = 'data-exp'): void {
  const nav = performance.getEntriesByType('navigation')[0] as PerformanceNavigationTiming | undefined;
  const fcp = performance.getEntriesByName('first-contentful-paint')[0];
  const firstPaintMs = fcp ? Math.round(fcp.startTime) : null;
  let clsDelta = 0;
  let flagAppliedMs: number | null = null;

  try {
    new PerformanceObserver((list) => {
      for (const entry of list.getEntries() as unknown as Array<PerformanceEntry & { value: number; hadRecentInput: boolean }>) {
        if (!entry.hadRecentInput) clsDelta += entry.value;
      }
    }).observe({ type: 'layout-shift', buffered: true });
  } catch {
    // Safari lacks layout-shift; degrade gracefully rather than throwing.
  }

  const flush = (variantFlash: boolean): void => {
    const sample: FlickerSample = {
      experimentKey,
      firstPaintMs,
      flagAppliedMs,
      holdMs: Math.max(0, (flagAppliedMs ?? 0) - (nav?.responseEnd ?? 0)),
      clsDelta: Number(clsDelta.toFixed(4)),
      variantFlash,
    };
    navigator.sendBeacon('/rum/flicker', new Blob([JSON.stringify(sample)], { type: 'application/json' }));
  };

  const observer = new MutationObserver(() => {
    if (flagAppliedMs !== null) return;
    flagAppliedMs = Math.round(performance.now());
    observer.disconnect();
    // Mutated strictly after paint => the user could have seen the control first.
    flush(firstPaintMs !== null && flagAppliedMs > firstPaintMs);
  });

  observer.observe(document.documentElement, {
    attributes: true,
    subtree: true,
    attributeFilter: [markerAttr],
  });

  // Safety net: if the flag never applies, still emit a sample.
  setTimeout(() => {
    if (flagAppliedMs === null) {
      observer.disconnect();
      flush(false);
    }
  }, 5000);
}

Alert on P99 variant-flash rate per experiment, not on the platform-wide average. A single heavyweight experiment with a late-firing visual editor change can dominate your aggregate and hide inside a monthly rollup.

True Cost per 1M Experiments

The normalized unit

"Experiments" is ambiguous. Normalize to exposures — the event where a unit is bucketed and the flag is actually read. At 100M sessions with ~30% participation in at least one live test, you are logging roughly 30M exposures/month, or 30 units of 1M.

Cost model at 100M sessions / 30M exposures per month

The honest finding most cost analyses hide: at 100M sessions, self-hosting does not win by an order of magnitude. It wins by roughly 2–3x, and only if you correctly amortize:

• Postgres (experiment metadata) + Redis (assignment cache) + ClickHouse/BigQuery (exposures) ≈ $2,000–$4,000/mo
• 0.2–0.3 FTE of platform engineering ≈ $4,000–$7,000/mo amortized
• Opportunity cost of the edge-worker migration you will inevitably do anyway

The break-even is real but modest, and it flips negative if your team cannot staff an on-call rotation for the decision path. Buy the managed platform if your edge team is smaller than two engineers; self-host if you already run the data plane.

// cost/calc.ts — keep the model in code so procurement debates are reproducible
interface CostInputs {
  sessionsPerMonth: number;
  participationRate: number;   // 0..1
  annualLicenseUsd: number;
  infraMonthlyUsd: number;
  fteFraction: number;
  fullyLoadedMonthlyUsd: number;
}

export function costPerMillionExposures(i: CostInputs): number {
  const exposures = i.sessionsPerMonth * i.participationRate;
  if (exposures <= 0) throw new RangeError('exposures must be > 0');
  const monthly =
    i.annualLicenseUsd / 12 + i.infraMonthlyUsd + i.fteFraction * i.fullyLoadedMonthlyUsd;
  return Number((monthly / (exposures / 1_000_000)).toFixed(2));
}

// GrowthBook self-hosted example
costPerMillionExposures({
  sessionsPerMonth: 100_000_000,
  participationRate: 0.3,
  annualLicenseUsd: 0,
  infraMonthlyUsd: 3_000,
  fteFraction: 0.25,
  fullyLoadedMonthlyUsd: 22_000,
}); // => 283.33

Architecture Trade-Off Matrix

Operational Checklist

• Resolve every above-the-fold experiment before HTML flush; SSR or edge worker, no exceptions
• Set a hard config-fetch timeout (≤250ms) with fail-open-to-control
• Never no-store-bypass your CDN without Vary on the bucketing cookie
• Alert on P99 variant-flash rate per experiment, sampled at 100% on the probe endpoint
• Emit a synthetic SRM (sample ratio mismatch) check hourly — chi-square against expected weights
• Log exposures via waitUntil/sendBeacon, never in the TTFB path
• Reproduce one live experiment's assignment in an offline script before trusting the platform
• Include the experiment platform in your LCP regression budget, not just in analytics

Pitfalls to Avoid

1. Benchmarking SDK evaluation instead of the decision path. In-memory eval is 0.3ms everywhere. Your 140ms is the cold datafile fetch you never measured.
2. Treating the anti-flicker snippet as free. It converts a visible swap into a render delay you then attribute to "CDN slowness."
3. Bucketing on `session_id` for long-horizon landing page tests. Users return; inconsistent assignment destroys the treatment effect and inflates variance.
4. Sampling your flicker RUM. At a 2% base rate, 10% sampling makes P99 estimates unusable.
5. Assuming the platform's hash matches your warehouse re-implementation. Verify against 10,000 known unit IDs before you publish a single readout.
6. Comparing list prices across vendors at different tiers. Normalize to $/1M exposures at your participation rate, then add the FTE line.

Sources & Technical References

1. Optimizely Developer Docs — Anti-flicker snippet: https://docs.developers.optimizely.com/web-experimentation/docs/anti-flicker-snippet
2. Optimizely Developer Docs — Experiment decision API: https://docs.developers.optimizely.com/feature-experimentation/docs/experiment-decision
3. VWO Developer Docs — FullStack / Decision API: https://developers.vwo.com/reference
4. VWO Help Center — Campaign settings (anti-flicker behavior, smart code load order): https://help.vwo.com/
5. GrowthBook Docs — SDKs & feature evaluation: https://docs.growthbook.io/lib/build-your-own
6. GrowthBook Docs — Self-hosting: https://docs.growthbook.io/self-host
7. GrowthBook Docs — Statistics engine overview: https://docs.growthbook.io/statistics/overview
8. Appleby, A. — MurmurHash3 (smhasher wiki): https://github.com/aappleby/smhasher/wiki/MurmurHash3
9. Noll, L. C. — FNV Hash: http://www.isthe.com/chongo/tech/comp/fnv/ ; IETF draft The FNV Non-Cryptographic Hash Algorithm: https://datatracker.ietf.org/doc/html/draft-eastlake-fnv
10. W3C — Paint Timing: https://www.w3.org/TR/paint-timing/ ; WICG — Layout Instability API: https://wicg.github.io/layout-instability/
11. MDN Web Docs — PerformanceObserver, MutationObserver, Math.imul: https://developer.mozilla.org/en-US/docs/Web/API/PerformanceObserver
12. web.dev — Cumulative Layout Shift (CLS): https://web.dev/articles/cls
13. Kohavi, R., Tang, D., & Xu, Y. (2020) — Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
14. Cloudflare Workers Docs — ExecutionContext.waitUntil: https://developers.cloudflare.com/workers/runtime-apis/handlers/fetch/