Rust Async Runtimes at 10M Concurrent Tasks: Tokio vs Monoio vs Glommio vs Smol

At ten million concurrent tasks, the async runtime stops being plumbing and becomes the single largest variable in your latency budget. The gap between your p50 and your p99 is rarely in your handler — it lives in the work-stealing queue, the epoll wait lock, or the cross-core wakeup path.

This article compares four runtimes with genuinely different architectures: Tokio (work-stealing, epoll/kqueue), Monoio (thread-per-core, io_uring), Glommio (thread-per-core, io_uring, DataDog), and Smol (minimalist, portable). We'll cover the architecture that drives their tail behavior, build a reproducible benchmark harness, measure memory per task, and derive a defensible cost-per-million-connections figure for capacity planning.

The 10M Concurrency Problem: Why Runtime Choice Now Matters

Ten million idle TCP connections is a real workload — push notification fan-out, multiplayer session registries, WebSocket gateways, and long-poll queues all land there. At that scale three constraints dominate, in order:

1. Kernel socket memory — typically 1–4 GiB per million established TCP sockets depending on tcp_rmem/tcp_wmem and sk_buff overhead.
2. Task/future heap cost — 50–150 bytes per idle task adds 0.5–1.5 GiB across ten million.
3. Scheduler tail behavior — one contended lock or cross-core wakeup adds 10–100 µs at p99.

Note the ordering. Most engineers obsess over (3) or (2), but (1) is usually the bill. The runtime still matters enormously: it decides how much userspace you burn and whether your p99 stays flat as cores scale.

Comparing the Four Runtimes Architecturally

Tokio — Work-Stealing, Epoll-Based, Battle-Tested

Tokio's default multi-threaded scheduler moves work between runtimes via a per-worker run queue and a global injection queue. This gives excellent CPU utilization for unbalanced workloads — one hot connection doesn't pin one core. The cost is CAS contention on the steal path and futex/eventfd wakeups when a task migrates or a worker parks.

At ten million tasks with uniform load, that overhead is amortized and Tokio wins on operational simplicity. At ten million with a skewed load (thundering herd, hot shards), stealing becomes visible in p99.

Monoio — Thread-Per-Core with io_uring

Monoio (ByteDance) pins one executor per core and uses io_uring for I/O. Because tasks are !Send, there's no wakeup migration — timers, TcpStream, and buffers are all local to the driver thread. This eliminates the cross-core wake path entirely. The trade-off: you must explicitly shard state (e.g., sharded maps or per-core registries) and accept Linux-only io_uring +

Glommio — io_uring, DataDog's Thread-Per-Core Executor

Glommio offers a richer I/O surface (direct I/O files, io_uring native TPC listener, per-executor latencies). Its task metadata is heavier because it carries executor/queue identity for locality guarantees, so per-idle-task memory is the highest of the four.

Smol — Minimalist, Composable, Portable

Smol is the smallest executor here: a thin async-executor over the polling crate, plus futures-lite. It's ideal when you want async without an opinionated runtime — but it ships no io_uring path, so per-syscall cost under heavy I/O is higher than the io_uring runtimes.

Why p99 Tail Latency Diverges From Median

Three effects separate the runtimes at the tail:

1. Cross-core wakeup — Tokio's wake() may notify a worker on another core; the latency floor is the IPI + run-queue scan, typically 2–15 µs, spiking under CAS contention.
2. `epoll_wait` contention — Tokio uses eventfd to wake parked workers; a spinning accept loop can produce thundering-herd syscalls. Glommio and Monoio each have a single io_uring per core.
3. Syscall amplification — per-I/O, io_uring can batch submissions and completions; epoll does one read/write syscall per operation (plus epoll_wait). For a 10k-RPS-per-core handler this is measurable; for 10M mostly-idle connections, it isn't.

That last point is the honest answer most benchmarks omit: if your workload is 10M idle long-lived connections, the runtime's I/O backend barely moves p99 — scheduler overhead does. io_uring dominates once each connection is actually reading/writing data.

Building a Reproducible 10M-Task Benchmark Harness

Measure latency with an HDR histogram, log at high resolution, and keep load-generator and SUT on separate machines. Pin cores with taskset and disable C-states.

use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};

use hdrhistogram::Histogram;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;

#[derive(Debug, Clone, Copy)]
struct LatencyReport {
    p50_ns: u64,
    p99_ns: u64,
    p999_ns: u64,
    max_ns: u64,
}

fn summarize(h: &Histogram<u64>) -> LatencyReport {
    LatencyReport {
        p50_ns: h.value_at_quantile(0.50),
        p99_ns: h.value_at_quantile(0.99),
        p999_ns: h.value_at_quantile(0.999),
        max_ns: h.max(),
    }
}

async fn probe(
    target: &'static str,
    h: Arc<Mutex<Histogram<u64>>>,
) -> std::io::Result<()> {
    let mut stream = TcpStream::connect(target).await?;
    let mut buf = [0u8; 4];
    loop {
        let start = Instant::now();
        stream.write_all(b"PING").await?;
        stream.read_exact(&mut buf).await?;
        let elapsed = start.elapsed().as_nanos() as u64;
        h.lock()
            .map_err(|e| std::io::Error::other(e.to_string()))?
            .record(elapsed)
            .map_err(|e| std::io::Error::other(e.to_string()))?;
        tokio::time::sleep(Duration::from_millis(100)).await;
    }
}

For the SUT, spawn immediately and detach — measure RSS before/after the spawn loop to isolate per-task overhead.

Memory Per Task: Measurement and Breakdown

/// Resident set size in KiB, read from /proc/self/statm on Linux.
fn rss_kib() -> std::io::Result<u64> {
    let statm = std::fs::read_to_string("/proc/self/statm")?;
    let field = statm
        .split_whitespace()
        .nth(1)
        .ok_or_else(|| std::io::Error::new(std::io::ErrorKind::InvalidData, "statm too short"))?;
    let pages: u64 = field
        .parse()
        .map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, format!("parse: {e}")))?;
    Ok(pages * 4) // 4 KiB pages on x86-64
}

Measured ranges on x86-64, glibc + jemalloc, 10M idle tasks with a zero-capture future (order-of-magnitude illustration; your future size dominates):

The real lesson: your handler's future size dwarfs the runtime's header. A future capturing two 64 KiB buffers (common when you eagerly allocate a read/write buffer per connection) costs 128 KiB × 10M = 1.2 TiB. Allocate lazily, size buffers to tcp_rmem, and share read-only buffers via Arc.

True Cost Per Million Connections

Userspace task memory is the cheap part. Here's a defensible model for a 1M-connection gateway with 4 KiB receive buffers:

Userspace task + future          ~100 MB
Kernel sk_buff + socket struct    ~1–4 GB (tcp_rmem min × 2)
FD table + page tables            ~200–400 MB
Total per million                 ~1.3–4.5 GB

That kernel number is the bill — and it moves with net.ipv4.tcp_rmem, net.ipv4.tcp_wmem, and net.core.somaxconn. Tune those first, before choosing a runtime. Then subtract the runtime overhead by going thread-per-core: Monoio and Glommio shave ~0.5 GiB per 10M tasks, which is real money at hyperscale but noise at 1M.

Selection Guide and Production Hardening Checklist

Choose Tokio unless you have a measurable io_uring or thread-locality reason not to. It has the mature ecosystem (tokio-util, tower, hyper), runtime-agnostic task cancellation, and the smoothest migration to distributed systems.

If you're going thread-per-core, use Monoio for new networking services (it's cleaner to embed and has sane sharding ergonomics) and Glommio where you need its richer I/O surface. Reach for Smol when you're a library author who must not force a runtime on users.

Hardening checklist for 10M:

• ulimit -n and fs.file-max ≥ target × 1.2.
• sysctl -w net.core.somaxconn=65535 and net.ipv4.tcp_max_syn_backlog.
• vm.overcommit_memory=1 for 1M+ thread-per-core tasks with jemalloc.
• mlock io_uring rings or set IORING_SETUP_SQPOLL after io_uring_disabled check.
• Pin workers with taskset/cgroups; disable turbo and C2+ states (measure with cyclictest).
• Instrument with tokio-metrics (Tokio) or per-executor latency histograms, not just application timers.

Pitfalls to Avoid

• Benchmarking with `#[tokio::main]` on a 32-core box while the load generator runs on the same machine. Everything is fake.
• Comparing io_uring runtimes on `kernel 5.x` without `IORING_SETUP_SQPOLL`. You measure the slow path.
• Assuming thread-per-core is `Send`. You'll rewrite state management; budget for it.
• Reading p99 from a single run. Run 5× and report the median of the histograms, not the best.
• Forgetting `tokio::task::yield_now` between mass spawns. You'll measure your spawn harness, not the runtime.

Sources & Technical References

• Tokio documentation: https://docs.rs/tokio/latest/tokio/
• Tokio task scheduling internals: https://tokio.rs/blog/2019-10-scheduler
• Monoio GitHub (ByteDance): https://github.com/bytedance/monoio
• Monoio API docs: https://docs.rs/monoio/
• Glommio GitHub (DataDog): https://github.com/DataDog/glommio
• Glommio API docs: https://docs.rs/glommio/
• Smol GitHub: https://github.com/smol-rs/smol
• Smol API docs: https://docs.rs/smol/
• Jens Axboe, Efficient IO with io_uring: https://kernel.dk/io_uring.pdf
• Linux io_uring(7) man page: https://man7.org/linux/man-pages/man7/io_uring.7.html
• Linux epoll(7) man page: https://man7.org/linux/man-pages/man7/epoll.7.html
• HdrHistogram project: https://github.com/HdrHistogram/HdrHistogram
• Rust Async Book: https://rust-lang.github.io/async-book/
• RFC 2394, async/await: https://rust-lang.github.io/rfcs/2394-async_await.html