Pingora vs Envoy vs Nginx vs HAProxy at 1M RPS: Rust's Proxy Revolution — p99 Latency, Memory, and True Cost per Million Requests

At 1M requests per second, a reverse proxy stops being a commodity checkbox. It becomes a latency tax, a memory multiplier, and a line item in your cloud bill. The comparison between Pingora, Envoy, NGINX, and HAProxy is no longer just C versus C++ versus Rust; it is about tail latency under connection churn, allocator behavior, TLS session reuse, and how much headroom you must buy to keep p99 inside an SLA.

This guide gives you a reproducible way to evaluate all four proxies at 1M RPS, with Rust harness code you can run, a cost model for true dollars per million requests, and the operational details that separate a lab chart from production capacity planning.

The 1M RPS Benchmark Problem: Measure Tails, Not Averages

A 1M RPS target is usually an aggregate fleet number, not a single node. Per-node capacity of 50k–200k RPS is more realistic once TLS, HTTP/2, and payload parsing are in play. The benchmark must therefore report p99 and p99.9 at a fixed concurrency and connection-reuse profile, not a single average.

Closed-loop load generators are the most common mistake. If your generator sends the next request only after the previous response, it hides queueing delay. Use an open-loop generator such as wrk2 or a constant-rate Rust harness, and record latency with HdrHistogram. Warm up TLS sessions, upstream pools, and allocator arenas for 5–10 minutes before sampling.

Architecture Showdown: Four Proxies, Four Failure Modes

Pingora: Rust Ownership as a Latency Strategy

Pingora is Cloudflare's Rust proxy framework built on Tokio. Its advantage is not merely “Rust is fast”; it is predictable memory ownership, no stop-the-world GC, and a threaded async runtime that can use all cores without a process-per-core reload model. At 1M RPS, Pingora's lower allocation churn often shows up as tighter p99.9 and lower RSS under connection churn. The trade-off is ecosystem maturity: fewer off-the-shelf filters and a smaller hiring pool than Envoy or NGINX.

Envoy: The Control Plane Is the Product

Envoy's C++ event loop and filter chain are fast, but its real value is dynamic configuration through xDS, rich observability, and L7 traffic management. At 1M RPS, Envoy's memory footprint is usually the highest of the four because of its filter chains, stats, and per-worker structures. That is not a defect; it is the cost of a programmable data plane. If you need per-route retries, outlier detection, and global rate limiting, Envoy earns its memory.

NGINX: Compact, Mature, and Still Relevant

NGINX's C event loop and worker-process model remain extremely efficient for static content, TLS termination, and reverse proxying. Its RSS is often lower than Envoy's, and its configuration model is familiar. The OSS edition's active health checks and dynamic upstream reconfiguration are more limited than HAProxy's or Envoy's; many teams pair NGINX with Consul Template or NGINX Plus. At 1M RPS, watch worker connections, keepalive upstream pools, and TLS session cache size.

HAProxy: Predictable Load Balancing at Line Rate

HAProxy is a focused L4/L7 load balancer with an event-driven C core, excellent ACLs, and first-class health checking. It often produces the most predictable p99 for pure load-balancing workloads because it does less per request by default. Its HTTP/3 and filter ecosystem are less rich than Envoy's, and complex L7 transformations become ACL-heavy. If your problem is “balance 1M RPS across upstreams with fast failover,” HAProxy is hard to beat.

Representative Results at 1M RPS: What the Curves Usually Show

The table below is an illustrative planning range from reproducible lab patterns and public benchmark discussions, not a vendor guarantee. TLS, HTTP/2, payload size, CPU model, NIC, and connection reuse dominate the result.

*Aggregate fleet target; per-node RPS varies. Always reproduce with your own TLS and payload profile.

A Production-Grade Rust Harness for p99 and Cost per Million

The harness below uses a semaphore to cap in-flight requests, discards warmup samples, and records only successful responses. Replace reqwest with hyper if you need to remove client overhead, but keep the open-loop or bounded-concurrency model explicit.

use std::{sync::Arc, time::Duration};
use anyhow::{Context, Result};
use hdrhistogram::Histogram;
use reqwest::Client;
use tokio::{sync::Semaphore, task::JoinSet, time::Instant};

#[derive(Debug, Clone)]
struct BenchConfig {
    url: String,
    duration: Duration,
    warmup: Duration,
    concurrency: usize,
}

#[derive(Debug)]
struct Percentiles { p50: u64, p99: u64, p999: u64, count: u64 }

async fn run_benchmark(cfg: BenchConfig) -> Result<Percentiles> {
    let client = Client::builder()
        .pool_max_idle_per_host(cfg.concurrency)
        .tcp_nodelay(true)
        .timeout(Duration::from_secs(5))
        .build()
        .context("build reqwest client")?;

    let sem = Arc::new(Semaphore::new(cfg.concurrency));
    let deadline = Instant::now() + cfg.warmup + cfg.duration;
    let mut tasks = JoinSet::new();

    for _ in 0..cfg.concurrency {
        let client = client.clone();
        let sem = sem.clone();
        let url = cfg.url.clone();
        let warmup_end = Instant::now() + cfg.warmup;

        tasks.spawn(async move {
            let mut local = Histogram::<u64>::new(3)?;
            loop {
                let permit = sem.clone().acquire_owned().await?;
                let start = Instant::now();
                let res = client.get(&url).send().await;
                let elapsed_us = start.elapsed().as_micros() as u64;
                drop(permit);

                if Instant::now() > deadline { break; }
                if Instant::now() > warmup_end {
                    if let Ok(response) = res {
                        if response.status().is_success() {
                            local.record(elapsed_us)?;
                        }
                    }
                }
            }
            Ok::<_, anyhow::Error>(local)
        });
    }

    let mut hist = Histogram::<u64>::new(3)?;
    while let Some(joined) = tasks.join_next().await {
        let local = joined??;
        hist.add(&local)?;
    }

    Ok(Percentiles {
        p50: hist.value_at_quantile(0.50),
        p99: hist.value_at_quantile(0.99),
        p999: hist.value_at_quantile(0.999),
        count: hist.len(),
    })
}

Cost per Million Requests: The Formula That Matters

At 1M RPS, raw compute cost per million is tiny; the differentiator is how much headroom you must buy to keep p99 inside the SLA. Use effective RPS, not peak RPS:

cost_per_million = fleet_hourly_cost * 1_000_000 / (effective_rps * 3600)
                 + egress_cost_per_million
                 + ops_cost_per_million

If one node handles 200k RPS and costs $1.00/hour, compute cost is $0.00139 per million requests. At $10/hour it is $0.0139 per million. Egress, load-balancer data processing, and on-call time often exceed that. Pingora and HAProxy tend to win when memory headroom and p99.9 stability let you run fewer nodes; Envoy wins when dynamic control-plane features reduce operational toil.

Tuning Checklist for 1M RPS

• Use open-loop or bounded-concurrency load generation; never compare averages only.
• Pin proxy workers and load generators to separate NUMA nodes; account for softirq CPU.
• Enable SO_REUSEPORT, size NIC queues, and verify RSS/RPS/XPS settings.
• Benchmark TLS 1.3 resumption, HTTP/1.1 keepalive, and HTTP/2 multiplexing separately.
• Keep upstream keepalive pools larger than worker count and monitor pool exhaustion.
• Track RSS, page faults, context switches, and p99.9—not just CPU utilization.
• Reserve 30–40% capacity headroom for failover, connection storms, and config reloads.
• Canary proxy configuration changes and compare p99 across identical traffic slices.

Pitfalls That Invalidate Proxy Benchmarks

Coordinated omission is the silent killer: if the generator waits for a response before scheduling the next request, it under-reports queueing. Comparing plaintext against TLS, or HTTP/1.1 against HTTP/2, produces misleading wins. Running the load generator on the same host as the proxy steals CPU and distorts p99. Container CPU limits and throttling can turn a 4 ms p99 into a 40 ms p99 without any proxy regression. Finally, do not benchmark one node and call it 1M RPS; measure the fleet at the SLA boundary.

Sources & Technical References

• Pingora open-source repository and documentation: https://github.com/cloudflare/pingora
• Cloudflare: “Pingora, the proxy that connects Cloudflare to the Internet”: https://blog.cloudflare.com/pingora-open-source/
• Envoy Proxy official documentation: https://www.envoyproxy.io/docs/envoy/latest/
• NGINX official documentation: https://nginx.org/en/docs/
• HAProxy official documentation: https://docs.haproxy.org/
• HdrHistogram: https://github.com/HdrHistogram/HdrHistogram
• wrk2 and coordinated omission: https://github.com/giltene/wrk2
• Tokio documentation: https://docs.rs/tokio/latest/tokio/
• Rust Async Book: https://rust-lang.github.io/async-book/
• Linux network performance parameters: https://github.com/leandromoreira/linux-network-performance-parameters