Landing Page Experimentation at 50M Sessions: Client-Side A/B Testing vs. Edge Decisioning vs. Server-Side Flags — Flicker, INP, and Cost per Million Decisions Benchmarked

The Forcing Function: Why 50M Sessions Breaks Naive Experimentation

At 5 million sessions a month, you can afford architectural sloppiness. The anti-flicker snippet adds 400ms of blank white paint, nobody notices, and your landing page A/B testing program ships variants weekly. At 50 million sessions, that same snippet is a revenue line item. A 100ms regression compounding across 50M sessions is roughly 1.6 million session-requests worth of degraded first impression per month — before you count the mobile long-tail where a mid-tier Android device turns 400ms into 1.2 seconds.

The architectural question is deceptively simple: at what point in the request lifecycle does the variant get chosen? The answer determines three things simultaneously — whether the page flickers (CLS), whether interactions feel fast (INP), and whether your experimentation platform bill scales with compute or with humans.

This article benchmarks three production topologies against each other and gives you the arithmetic to pick between them.

Three Architectures, One Rendering Pipeline

Client-Side Experimentation: The DOM Mutator

The experiment SDK loads asynchronously, the original DOM paints, then the SDK rewrites it. Optimizely Web, VWO, and most GA-based tooling live here. It is the easiest to deploy and the hardest to make fast, because the variant is applied after first paint by definition.

Edge Decisioning: The Middleware Bucketer

A Cloudflare Worker, Fastly Compute service, Akamai EdgeWorker, or Vercel Edge Middleware intercepts the HTML request, evaluates the bucket, and injects the variant before the origin renders. The browser receives one coherent document. There is no rewrite and no flicker, because there is no second paint.

Server-Side Flags: The Origin Evaluator

The origin (Node, Go, Rails, PHP) evaluates the flag during render using a local SDK cache. GrowthBook, LaunchDarkly, Statsig, and Unleash all support this. It is the most precise option and the most coupled: your flag evaluation is now inside your TTFB budget and your deployment topology.

The Decision-to-Paint Timeline

The distinction that matters:

Everything downstream — flicker, INP, cost — follows from that table.

Flicker: The CLS Tax You Pay for Late Decisions

Cumulative Layout Shift measures unexpected movement of visible elements. A client-side test that swaps a hero image, changes a headline length, or inserts a social-proof banner is a textbook CLS generator. Google's guidance is a CLS budget of 0.1 at p75; a single unmitigated hero swap routinely produces 0.12–0.35 because the new element has different intrinsic dimensions than the one it replaced.

Why the Anti-Flicker Snippet Is a Loan Shark

Vendors ship an anti-flicker snippet that hides <html> until the SDK resolves or a timeout fires. This converts a visual metric into a latency metric: CLS goes to zero, LCP absorbs the entire cost. You have not fixed the problem — you have refinanced it.

<script>
  // The default pattern most client-side vendors ship.
  // Trades paint latency for variant-application determinism.
  (function () {
    var HIDE_MS = 1200; // vendor default; usually not tuned
    document.documentElement.style.visibility = 'hidden';
    window.__unhide = function () {
      document.documentElement.style.visibility = '';
    };
    // Fail-open: never leave a page invisible if the SDK stalls.
    setTimeout(window.__unhide, HIDE_MS);
  })();
</script>

Measure the delta. On a throttled 4G profile with a Moto G4-class CPU, the SDK bundle plus network round-trip to the vendor's decision API typically lands between 350ms and 900ms at p75. That is LCP budget consumed before a single byte of your own HTML is parsed.

The Deterministic Alternative: Bucket Before Paint

Edge middleware makes the decision in the request path, so the HTML that arrives is already the correct variant. No hiding, no timeout, no repaint. Here is a Next.js middleware implementation with signed-cookie identity, deterministic hashing, and a fail-open control path:

// middleware.ts — deterministic bucketing at the edge, pre-paint.
import { NextRequest, NextResponse } from 'next/server';

interface Experiment {
  key: string;
  /** Share of traffic assigned to the treatment, 0..1 */
  traffic: number;
  /** Salt must never change mid-experiment or you re-randomize cohorts. */
  salt: string;
}

const HERO_V3: Experiment = { key: 'hero_v3', traffic: 0.5, salt: 'hero_v3:2024-11' };

/** Maps an identity to a uniform value in [0,1) using 52 bits of SHA-256. */
async function bucket(identity: string, salt: string): Promise<number> {
  const bytes = new TextEncoder().encode(`${salt}:${identity}`);
  const digest = await crypto.subtle.digest('SHA-256', bytes);
  const view = new DataView(digest);
  // 52-bit mantissa keeps the result an exact safe integer.
  const hi = view.getUint32(0);
  const lo = view.getUint32(4) & 0xfffff;
  return (hi * 2 ** 20 + lo) / 2 ** 52;
}

export async function middleware(req: NextRequest) {
  const requestHeaders = new Headers(req.headers);
  let variant = 'a';
  let uid = req.cookies.get('exp_uid')?.value;
  const isNewUser = !uid;

  try {
    if (!uid) uid = crypto.randomUUID();
    const roll = await bucket(uid, HERO_V3.salt);
    variant = roll < HERO_V3.traffic ? 'b' : 'a';
  } catch (err) {
    // Fail-open to control. A broken experiment must never break the page.
    console.error(JSON.stringify({ evt: 'bucket_failed', err: String(err) }));
    variant = 'a';
  }

  // Expose to RSC / SSR so the origin renders the correct tree.
  requestHeaders.set('x-exp-hero', variant);
  const res = NextResponse.next({ request: { headers: requestHeaders } });
  res.headers.set('x-exp-hero', variant);

  if (isNewUser && uid) {
    res.cookies.set('exp_uid', uid, {
      httpOnly: true,
      sameSite: 'lax',
      secure: true,
      maxAge: 60 * 60 * 24 * 365,
    });
  }
  return res;
}

export const config = { matcher: ['/', '/pricing', '/lp/:path*'] };

The critical detail is matcher. Edge middleware runs on every matching request; scoping it to landing pages only keeps the cost and latency surface minimal.

INP: The Interaction Cost of Client-Side Variant Application

Interaction to Next Paint replaced FID in March 2024 and is far less forgiving. INP measures the worst-case (approximately 98th percentile) interaction latency across the page lifetime, and it punishes long tasks on the main thread.

Client-side experimentation is structurally hostile to INP for three reasons:

1. Bundle execution. The vendor SDK parses and executes on the main thread, typically 60–180KB compressed.
2. MutationObserver churn. SDKs that re-apply variants on dynamic content keep a live observer running for the page lifetime.
3. Custom-code interception. Many client-side tools wrap setTimeout, fetch, and DOM APIs to track element visibility, which adds per-call overhead to every interaction handler.

Edge and server-side flagging have zero INP contribution because the variant tree is what the browser parses — no mutation, no observer, no wrapper.

Instrument it rather than assuming it:

// INP attribution beacon. Captures the interaction target and the variant
// that produced it so regressions can be blamed on a specific experiment.
interface InpPayload {
  name: string;
  duration: number;
  variant: string;
  target: string | null;
}

if ('PerformanceObserver' in window) {
  const observer = new PerformanceObserver((list) => {
    for (const entry of list.getEntries() as PerformanceEventTiming[]) {
      if (entry.duration < 40) continue; // below the 'good' threshold band
      const payload: InpPayload = {
        name: entry.name,
        duration: Math.round(entry.duration),
        variant: document.documentElement.dataset.expHero ?? 'a',
        target: (entry as PerformanceEventTiming & { target?: Element }).target?.tagName ?? null,
      };
      navigator.sendBeacon('/rum/inp', JSON.stringify(payload));
    }
  });
  observer.observe({ type: 'event', buffered: true, durationThreshold: 40 });
}

Pair this with the web-vitals library's onINP() for the aggregated field value and use the beacon above for attribution.

Benchmark Methodology: Measuring Without Lying to Yourself

Synthetic lab numbers mislead because they run on fast hardware with warm caches. The table below is a representative harness result for a content landing page (~180KB HTML+CSS, one hero image, three CTAs), served from a single origin region, measured with Lighthouse CI throttling plus 30 days of RUM at p75 on mobile. Treat the magnitudes as a reproduction target, not as gospel — re-run it against your own page.

Two conclusions follow. First, flicker and LCP are a see-saw for client-side tooling — you pick which one you sacrifice. Second, edge decisioning costs about one round-trip inside your own infrastructure, which is usually cheaper than the vendor's decision API call anyway.

Statistically, all three architectures produce equivalent experiment validity if bucketing is deterministic and the cookie boundary is respected. The differences are performance and cost, not causal inference.

Cost Per Million Decisions: The Arithmetic That Decides Your Architecture

Here is the counterintuitive result. At 50M sessions/month, edge compute is effectively free and platform licensing is the entire bill.

Assume 50M sessions/month and 1.4 decision-bearing requests per session (HTML document + one client-side flag fetch on SPAs):

monthly_decisions   = 50,000,000 × 1.4 = 70,000,000
edge_requests_billable = 70,000,000 − 10,000,000 (included tier) = 60,000,000
workers_cost        = 60 × $0.30            = $18.00
kv_reads            = 50 × $0.50            = $25.00  (1 read/decision, cacheTtl=60)
base_plan           =                      =  $5.00
-----------------------------------------------------
edge_total/month    =                      ≈ $48.00

Against a managed experimentation platform priced on monthly active users, 50M MAU at even $0.10/MAU is $5,000/month; at enterprise rates it is frequently an order of magnitude higher. That is a three-order-of-magnitude delta for the same statistical outcome.

Pricing changes quarterly across every vendor. Do not copy these numbers — rebuild the model with current published rates from Cloudflare Workers pricing, Cloudflare KV pricing, Vercel pricing, Fastly pricing, and your vendor's MAU contract.

Reference Architecture: Edge Decisioning With Origin Fallback

This is the topology I recommend for landing pages specifically: pure edge decisioning, KV-backed config with a short cacheTtl, and a hard fail-open to control. Note the Vary: Cookie header — forgetting it means your edge CDN can serve one variant to everyone.

// Cloudflare Worker — edge A/B decisioning with KV config + fail-open control.
export interface Env {
  EXP_CONFIG: KVNamespace;
  ORIGIN: Fetcher;
}

interface ExperimentConfig {
  key: string;
  traffic: number;
  salt: string;
  enabled: boolean;
}

const CONTROL: ExperimentConfig = {
  key: 'hero_v3',
  traffic: 0.5,
  salt: 'hero_v3:2024-11',
  enabled: false,
};

async function bucket(identity: string, salt: string): Promise<number> {
  const digest = await crypto.subtle.digest(
    'SHA-256',
    new TextEncoder().encode(`${salt}:${identity}`),
  );
  const view = new DataView(digest);
  return (view.getUint32(0) * 2 ** 20 + (view.getUint32(4) & 0xfffff)) / 2 ** 52;
}

function readUid(req: Request): string {
  const cookie = req.headers.get('Cookie') ?? '';
  const match = /(?:^|;\s*)exp_uid=([^;]+)/.exec(cookie);
  return match?.[1] ?? crypto.randomUUID();
}

export default {
  async fetch(req: Request, env: Env): Promise<Response> {
    const url = new URL(req.url);

    let cfg: ExperimentConfig = CONTROL;
    try {
      const cached = await env.EXP_CONFIG.get<ExperimentConfig>(CONTROL.key, {
        type: 'json',
        cacheTtl: 60,
      });
      if (cached) cfg = cached;
    } catch (err) {
      // Never block a page render on config retrieval.
      console.error(JSON.stringify({ evt: 'kv_read_failed', err: String(err) }));
    }

    let variant = 'a';
    if (cfg.enabled) {
      try {
        const uid = readUid(req);
        variant = (await bucket(uid, cfg.salt)) < cfg.traffic ? 'b' : 'a';
      } catch (err) {
        console.error(JSON.stringify({ evt: 'bucket_failed', err: String(err) }));
      }
    }

    const headers = new Headers(req.headers);
    headers.set('x-exp-hero', variant);
    const originRes = await env.ORIGIN.fetch(new Request(req, { headers }));

    const response = new Response(originRes.body, originRes);
    response.headers.set('x-exp-hero', variant);
    // CRITICAL: without Vary, a shared cache serves one variant to all users.
    response.headers.append('Vary', 'Cookie');
    return response;
  },
} satisfies ExportedHandler<Env>;

For teams that want a managed evaluation layer, GrowthBook's Node SDK can be evaluated inside SSR with a sub-200ms feature-load timeout and a control fallback:

import { GrowthBook } from '@growthbook/growthbook';

interface EvalResult {
  variation: string;
  source: 'growthbook' | 'fallback';
}

export async function evaluateHero(ctx: { userId: string; url: string }): Promise<EvalResult> {
  const gb = new GrowthBook({
    apiHost: process.env.GB_API_HOST,
    clientKey: process.env.GB_CLIENT_KEY,
    attributes: { id: ctx.userId, url: ctx.url },
    trackingCallback: (exp, result) => {
      // Never block render on an analytics HTTP call.
      queueMicrotask(() => trackExposure(exp.key, result.variationId, ctx.userId));
    },
  });

  try {
    await gb.loadFeatures({ timeout: 200 });
    return { variation: String(gb.getFeatureValue('hero-headline', 'control')), source: 'growthbook' };
  } catch (err) {
    console.error(JSON.stringify({ evt: 'flag_eval_failed', err: String(err) }));
    return { variation: 'control', source: 'fallback' };
  }
}

Statistical Integrity: Bucketing, SRM, and the Cookie Boundary

Performance gains mean nothing if the experiment is invalid. Three failure modes dominate at scale:

Sample Ratio Mismatch (SRM). If your 50/50 split lands at 51.2/48.8 over 70M decisions, the lift estimate is biased by selection effects. Always run a chi-square check before reading results:

interface SrmResult {
  chi2: number;
  df: number;
  mismatch: boolean;
}

export function chiSquareSrm(
  observed: Record<string, number>,
  expected: Record<string, number>,
): SrmResult {
  const total = Object.values(observed).reduce((a, b) => a + b, 0);
  const df = Object.keys(expected).length - 1;
  if (total < 1000) return { chi2: 0, df, mismatch: false }; // underpowered

  let chi2 = 0;
  for (const [key, ratio] of Object.entries(expected)) {
    const exp = total * ratio;
    if (exp <= 0) continue;
    chi2 += (observed[key] ?? 0 - exp) ** 2 / exp;
  }
  // p < 0.001 for df=1 is 10.83; use a table for higher df.
  return { chi2, df, mismatch: df === 1 && chi2 > 10.83 };
}

Salt drift. Changing the salt string mid-experiment re-randomizes the cohort. Users who saw variant B at first paint may see A on the next session, contaminating both arms. Version your salts and freeze them until the experiment concludes.

Cache poisoning. The single most common edge-experimentation bug: setting Cache-Control: public on a personalized response without Vary: Cookie. One user's variant becomes everyone's variant. Set private, no-store on experiment-bearing HTML or add Vary on the bucketing cookie — never both neither.

Operational Checklist Before You Ship

• Bucketing is deterministic on a stable identity (cookie UUID, not IP or UA).
• Vary: Cookie or Cache-Control: private is set on every experiment-bearing response.
• Fail-open path returns the control variant on KV timeout, digest failure, or SDK error.
• Edge matcher is scoped to landing-page routes only — not /*.
• Exposure events fire exactly once per session, from the server when possible.
• SRM check runs automatically at 10% and 100% of planned sample.
• x-exp-* header is mirrored into <html data-exp-*> for RUM attribution.
• Kill switch lives in KV/flag config, not in a deploy pipeline.
• CLS and INP are segmented by variant in your RUM dashboard.
• Cost model is re-validated against current vendor pricing each quarter.

Pitfalls That Cost Real Revenue

1. Treating the anti-flicker snippet as a fix. It converts CLS into LCP. If your p75 LCP is already 2.4s, hiding the page for 600ms pushes you into the "poor" band and you will lose more to ranking and bounce than you gain from the test.
2. Running edge middleware globally. A worker on /* multiplies your request count by every asset and API call. Scope the matcher.
3. Synchronous analytics from the edge. A fetch to your warehouse inside the worker adds 30–200ms to TTFB. Use ctx.waitUntil() or a queue.
4. Ignoring the origin's own caching. If your CDN caches HTML for 300s, a variant assigned mid-window will not render. Invalidate or reduce TTL during experiments.
5. Comparing vendor-reported lift across platforms. Different exposure definitions (impression vs. assignment) produce different denominators. Standardize on assignment-based exposure and re-baseline.

The Bottom Line

For landing pages specifically — high traffic, low route count, conversion-critical — edge decisioning wins on every axis that matters. It eliminates flicker by construction, contributes nothing to INP, adds roughly one round-trip inside your own infrastructure, and at 50M sessions costs tens of dollars per month rather than tens of thousands. Reserve server-side flags for authenticated product surfaces where you already own the render, and reserve client-side tooling for the long tail of low-traffic pages where engineering time genuinely outweighs the CWV cost.

Sources & Technical References

• Google web.dev — Cumulative Layout Shift (CLS): https://web.dev/articles/cls
• Google web.dev — Interaction to Next Paint (INP): https://web.dev/articles/inp
• Google web.dev — Optimize INP: https://web.dev/articles/optimize-inp
• MDN Web Docs — LayoutShift interface: https://developer.mozilla.org/en-US/docs/Web/API/LayoutShift
• MDN Web Docs — PerformanceEventTiming: https://developer.mozilla.org/en-US/docs/Web/API/PerformanceEventTiming
• MDN Web Docs — SubtleCrypto.digest(): https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest
• W3C — Web Cryptography API Recommendation: https://www.w3.org/TR/WebCryptoAPI/
• Cloudflare — Workers Pricing: https://developers.cloudflare.com/workers/platform/pricing/
• Cloudflare — KV Pricing: https://developers.cloudflare.com/kv/platform/pricing/
• Cloudflare — Workers Runtime & Request Lifecycle: https://developers.cloudflare.com/workers/
• Vercel — Edge Middleware: https://vercel.com/docs/functions/edge-middleware
• Vercel — Pricing: https://vercel.com/pricing
• Next.js — Middleware: https://nextjs.org/docs/app/building-your-application/routing/middleware
• Fastly — Compute Pricing: https://www.fastly.com/pricing
• GrowthBook — Documentation: https://docs.growthbook.io/
• GrowthBook — JavaScript/Node SDK: https://docs.growthbook.io/lib/js
• LaunchDarkly — Documentation: https://docs.launchdarkly.com/
• Optimizely — Web Experimentation Developer Docs: https://docs.developers.optimizely.com/web-experimentation/docs
• IETF RFC 6265 — HTTP State Management Mechanism (cookies): https://www.rfc-editor.org/rfc/rfc6265
• IETF RFC 9110 — HTTP Semantics, §12.5.5 Vary: https://www.rfc-editor.org/rfc/rfc9110
• IETF RFC 9111 — HTTP Caching: https://www.rfc-editor.org/rfc/rfc9111
• IETF RFC 4122 — A Universally Unique IDentifier (UUID) URN Namespace: https://www.rfc-editor.org/rfc/rfc4122
• Kohavi, Tang & Xu — Trustworthy Online Controlled Experiments (Cambridge University Press, 2020): https://www.cambridge.org/core/books/trustworthy-online-controlled-experiments/D97B26382EB0EB2DC2019A7A7B5183E8
• Kohavi & Longbotham — Online Controlled Experiments and A/B Testing (Encyclopedia of Machine Learning, Springer): https://link.springer.com/referenceworkentry/10.1007/978-1-4899-7687-1_891
• Fabijan et al. — Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (KDD '19): https://arxiv.org/abs/1905.02057