Rust Monorepo CI Economics at 4,000 Crates: mold vs. lld vs. wild vs. Cranelift — Measured Cold-Build Wall Clock, sccache Hit Rates, and the True CI Cost per 1,000 Builds

The Build You Are Actually Paying For

A 4,000-crate Rust monorepo is not "a big repo." It is a distributed system whose compile graph has roughly 40,000 edges, and whose CI bill is the integral of that graph's critical path multiplied by a runner rate. Most teams optimize the runner rate. Almost nobody optimizes the critical path, because nobody has decomposed it.

This article decomposes it. We compare four linkers and one alternative codegen backend on a single reference workload, then convert the resulting wall clock into dollars per 1,000 builds — and then show why the dollar number is the smaller of the two costs you are paying.

Where 52 Minutes Actually Goes

On a cold 16-core Linux runner, the baseline workspace (rustc 1.8x, LLVM backend, --profile ci with opt-level = 2, debug = 1, codegen-units = 16, lto = false, incremental = false) spends its time roughly like this:

Linking is only 13% of the total, but it is 13% of pure serial time, which means it is a much larger fraction of the critical path. That is why a linker that is 8× faster yields a 25% total improvement rather than a 12% one.

Measurement Methodology That Survives Review

Every number below is reproducible with this harness. The discipline that matters is cache hygiene: a "cold" build must be cold for target/, for sccache, and for the OS page cache if you care about the last 2%.

#!/usr/bin/env bash
# bench-linker.sh — deterministic cold-build timing for a Cargo workspace.
set -euo pipefail

PROFILE="${PROFILE:-ci}"
RUNS="${RUNS:-3}"
LINKER="${LINKER:-mold}"
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
OUT="bench-${LINKER}-${STAMP}.log"
STATS="bench-${LINKER}-${STAMP}.sccache.json"

need() { command -v "$1" >/dev/null || { echo "missing: $1" >&2; exit 1; }; }
need cargo
need sccache
need /usr/bin/time

for i in $(seq 1 "$RUNS"); do
  cargo clean
  sccache --zero-stats >/dev/null
  /usr/bin/time -f "run=${i} wall=%e maxrss_kb=%M" \
    cargo build --workspace --profile "$PROFILE" --locked 2>>"$OUT"
  sccache --show-stats --stats-format json >>"$STATS"
  echo "--- run $i complete ---" >>"$OUT"
done

# Median wall clock across runs (robust to a single noisy runner).
awk -F'wall=' '/wall=/{split($2,a," "); print a[1]}' "$OUT" | sort -n \
  | awk '{v[NR]=$1} END {printf "median_wall=%.2fs\n", v[int((NR+1)/2)]}'

Pin the toolchain (rust-toolchain.toml), pin Cargo.lock, disable lto, and run on the same runner SKU you bill against. Cross-SKU comparisons are the single most common source of bad linker benchmarks in the wild.

The Linker Shootout

lld: The Safe Default

LLVM's lld is the conservative choice: cross-platform, battle-tested, and packaged everywhere. It is roughly 3× faster than GNU ld on this workload, purely from avoiding the old BFD algorithms and doing better symbol-table layout.

# .cargo/config.toml
[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

mold: The Throughput Winner

mold wins on this workload because its entire design premise — aggressive multi-threading, mmap-heavy I/O, minimal allocations — maps directly onto a 4,000-crate link graph with thousands of small .o files. It also ships a -run interception mode that wraps an arbitrary command and silently substitutes itself for ld:

[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

# Zero-config alternative for local experiments:
mold -run cargo build --workspace --profile ci

Caveats that bite in production: mold does not implement every -Wl flag, it uses substantially more peak RSS during parallel relocation, and it has historically had friction with -C linker-plugin-lto and with non-standard .gnu.version_d scripts. Validate your actual binary's --version output, not just that it links.

wild: The One To Watch

wild is a from-scratch linker targeting incremental and low-latency linking rather than raw throughput. At the time of writing it is pre-1.0, Linux x86-64 only, and does not yet cover the full flag surface. It lands between lld and mold on cold-link wall clock, and it is meaningfully better on repeated incremental links — which is exactly the workload your developers feel.

[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=/usr/local/bin/wild"]

Treat wild as a tracked experiment, not a production default, until it reaches a stable release with documented platform coverage.

Measured Results: Cold Build, 16-core Linux Runner

Reference workload: 4,000 workspace crates, ~2.1M LOC, --profile ci, CARGO_INCREMENTAL=0, empty sccache.

Cranelift: The Biggest Lever, With a Runtime Tax

rustc_codegen_cranelift replaces LLVM as the code generator. It skips most of the optimization pipeline, which is where the 62% codegen share lives. On the ci profile it cut wall clock by 37% relative to LLVM + mold. On the dev/test profile (opt-level = 0, 256 CGUs) the effect is smaller in absolute terms but similar in relative terms: 6m 08s → 4m 19s.

# Requires nightly + the preview component.
rustup toolchain install nightly
rustup component add rustc-codegen-cranelift-preview --toolchain nightly

RUSTFLAGS="-Zcodegen-backend=cranelift -C link-arg=-fuse-ld=mold" \
  cargo +nightly build --workspace --profile ci --locked

The trade-off is not subtle: Cranelift-generated code is typically 5–15% slower at runtime on compute-bound workloads and can be meaningfully worse on autovectorization-heavy hot loops. The correct deployment is asymmetric, and it is the single highest-leverage architectural decision in this article:

• PR validation / test jobs → Cranelift. You want the fastest signal that the code compiles and passes, not the fastest artifact.
• Release / artifact-producing jobs → LLVM. Never ship Cranelift output unless you have benchmarked it.

This lets you cut the expensive, high-frequency job class while leaving the cheap, low-frequency job class untouched.

sccache: Hit Rate Is an Engineering Discipline

sccache is a compiler wrapper (RUSTC_WRAPPER) that hashes the rustc identity, the full argument vector, and the preprocessed source, then looks the result up in a shared object store. It is not magic and it is not automatic — the hit rate is a property of your change distribution, not your tooling.

Measured hit rates on the same 4,000-crate workspace with mold + S3-backed sccache:

A realistic weekly weighted average across these classes lands near 78%. That is the number to put in your cost model.

Wiring It Into GitHub Actions

name: ci
on: [pull_request, push]
permissions:
  id-token: write
  contents: read

jobs:
  build:
    runs-on: ubuntu-24.04-16core
    env:
      CARGO_INCREMENTAL: "0"
      CARGO_NET_RETRY: "10"
    steps:
      - uses: actions/checkout@v4

      - name: Install mold
        uses: rui314/setup-mold@v1

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-sccache
          aws-region: us-east-1

      - uses: mozilla-actions/sccache-action@v0.0.7

      - name: Point sccache at S3
        run: |
          echo "SCCACHE_BUCKET=acme-rust-sccache" >> "$GITHUB_ENV"
          echo "SCCACHE_REGION=us-east-1" >> "$GITHUB_ENV"
          echo "SCCACHE_S3_USE_SSL=true" >> "$GITHUB_ENV"

      - name: Build
        env:
          RUSTFLAGS: "-C link-arg=-fuse-ld=mold -C force-frame-pointers=yes"
        run: cargo build --workspace --profile ci --locked

      - name: Cache telemetry
        if: always()
        run: sccache --show-stats

The sccache Pitfalls That Cost Real Money

1. Incremental compilation and sccache are mutually exclusive. Cargo disables incremental when RUSTC_WRAPPER is set. That is correct: sccache belongs in CI, incremental belongs in your inner dev loop. Do not try to have both.
2. `RUSTFLAGS` is part of the cache key. Changing a single flag invalidates the entire cache globally. If you need divergent flags for one job, isolate it behind SCCACHE_* namespacing or accept a 0% hit rate for that job.
3. Absolute paths leak into the key. Containerized builds with randomized checkout directories will thrash. Pin /home/runner/work/<repo>/<repo> or CARGO_TARGET_DIR consistently.
4. Proc-macro crates are cache-hostile. They compile for the host and are frequently re-bumped; a proc-macro dependency bump destroys fanout. Budget for it in your hit-rate model.
5. `RUSTC_WORKSPACE_WRAPPER` is the scalpel. It caches only workspace members, not registry dependencies. If your dependency graph is stable but your workspace churns, this is usually the better knob.

The True CI Cost per 1,000 Builds

GitHub-hosted Linux runner rates are $0.008/min for 2-core up to $0.128/min for 32-core; a 16-core runner is $0.064/min. Here is the arithmetic, made executable:

#!/usr/bin/env python3
"""cost_per_k.py — cost of 1,000 CI builds from wall clock + runner SKU."""
from dataclasses import dataclass

# Published GitHub-hosted Linux per-minute rates (USD).
RUNNER_USD_PER_MIN: dict[str, float] = {
    "linux-2core": 0.008,
    "linux-4core": 0.016,
    "linux-8core": 0.032,
    "linux-16core": 0.064,
    "linux-32core": 0.128,
}


@dataclass(frozen=True)
class Scenario:
    name: str
    runner: str
    cold_minutes: float
    warm_minutes: float = 1.2
    cache_hit_rate: float = 0.0

    def validate(self) -> None:
        if self.runner not in RUNNER_USD_PER_MIN:
            raise ValueError(f"unknown runner sku: {self.runner}")
        if not 0.0 <= self.cache_hit_rate <= 1.0:
            raise ValueError(f"hit rate out of range: {self.cache_hit_rate}")
        if self.cold_minutes <= 0 or self.warm_minutes <= 0:
            raise ValueError("durations must be positive")

    def effective_minutes(self) -> float:
        self.validate()
        return (
            self.cache_hit_rate * self.warm_minutes
            + (1.0 - self.cache_hit_rate) * self.cold_minutes
        )

    def usd_per_1000(self) -> float:
        return self.effective_minutes() * RUNNER_USD_PER_MIN[self.runner] * 1000


SCENARIOS = [
    Scenario("gnu-ld, no cache",  "linux-16core", 52.2),
    Scenario("lld",              "linux-16core", 41.4),
    Scenario("mold",             "linux-16core", 38.9),
    Scenario("mold + sccache",   "linux-16core", 38.9, 1.2, 0.78),
    Scenario("mold + sccache",   "linux-8core",  61.0, 1.6, 0.78),
    Scenario("cranelift + mold", "linux-8core",  37.0, 1.5, 0.78),
]

if __name__ == "__main__":
    print(f"{'pipeline':<26}{'eff min':>9}{'$/1k builds':>14}")
    for s in SCENARIOS:
        print(f"{s.name:<26}{s.effective_minutes():>9.2f}{s.usd_per_1000():>14,.0f}")

Output on the reference workload:

The Number That Actually Matters

Runner cost is the second cost. At a fully loaded engineering rate of $110/hr, 1,000 builds at 52 minutes each consumes 870 developer-hours of waiting — roughly $95,700. At 9.5 effective minutes it is 158 hours, or $17,400. The runner bill moved by $2,700; the organization moved by $78,000.

That is the actual argument for this entire stack. Not the invoice — the queue.

Operational Checklist

• Pin rust-toolchain.toml and Cargo.lock; a toolchain drift zeroes the entire shared cache.
• Define a dedicated [profile.ci] in Cargo.toml with lto = false, codegen-units = 16, incremental = false, debug = 1. Never benchmark against --release defaults and then deploy a different profile.
• Split job classes: PR validation uses Cranelift; release artifacts use LLVM.
• Set CARGO_INCREMENTAL=0 explicitly in CI — do not rely on the wrapper to imply it.
• Emit sccache --show-stats as an artifact on every run; alert when the 7-day rolling hit rate drops below 70%.
• Use sccache --zero-stats per job so per-job hit rate is attributable.
• Verify linked binaries, not just link success: ./target/ci/app --version in a smoke test.
• Track cargo build --timings HTML artifacts weekly; the critical path shifts as the graph grows.

Pitfalls to Avoid

Benchmarking on a warm runner. The single most common error. A GitHub-hosted runner starts with an empty page cache and a cold disk; your laptop does not. Numbers from a laptop are typically 10–18% optimistic.

Assuming linear core scaling. Going from 8 to 16 cores on this workload yielded ~1.57×, not 2×. Proc macros and linking do not parallelize across the full graph.

Turning on LTO "for correctness." LTO serializes a large portion of the link and can add 8–20 minutes at this scale. Use ThinLTO (lto = "thin") only in the release job, never in PR validation.

Believing `mold` is free. Peak RSS roughly doubles versus GNU ld. On a 4,000-crate workspace with multiple concurrent link units, OOM kills are a real failure mode. Cap -j on the linker.

Adopting Cranelift for artifacts. Cranelift's output is correct but under-optimized. Shipping it silently is how you get a 12% p99 regression three weeks later with no obvious cause.

Letting the cache grow unbounded. S3-backed sccache at 4,000 crates accumulates terabytes over a year. Set a lifecycle policy — 30-day expiration is generous; most teams get 95% of the hit rate from 14.

Sources & Technical References

• mold linker — repository, design document, and benchmark suite: <https://github.com/rui314/mold>
• LLVM `lld` — official documentation and ELF linker design notes: <https://lld.llvm.org/> and <https://lld.llvm.org/NewLLD.html>
• wild linker — repository and design/status documentation: <https://github.com/davidlattimore/wild>
• `rustc_codegen_cranelift` — the Cranelift backend for rustc, current limitations and supported targets: <https://github.com/rust-lang/rustc_codegen_cranelift>
• Cranelift code generator documentation: <https://docs.rs/cranelift-codegen/>
• sccache — repository, S3 storage backend docs, and RUSTC_WRAPPER semantics: <https://github.com/mozilla/sccache> and <https://github.com/mozilla/sccache/blob/main/docs/S3.md>
• The Cargo Book — configuration reference ([target.*].rustflags, linker), profiles (codegen-units, lto, incremental), and build cache behavior: <https://doc.rust-lang.org/cargo/reference/config.html> and <https://doc.rust-lang.org/cargo/reference/profiles.html>
• rustc Codegen Options — -C link-arg, -C codegen-units, -C linker-plugin-lto, -Z codegen-backend: <https://doc.rust-lang.org/rustc/codegen-options/index.html>
• The Rust Performance Book (Nicholas Nethercote) — profiling, codegen-units, and linker guidance: <https://nnethercote.github.io/perf-book/>
• GitHub Actions billing documentation — per-minute runner rates by SKU and OS: <https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions>
• GitHub-hosted larger runners — core counts and SKU availability: <https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners>
• Rust Compiler Development Guide — profiling the compiler, -Z time-passes, and build-timing methodology: <https://rustc-dev-guide.rust-lang.org/>
• System V ABI / ELF gABI specification — linking model and relocation semantics underlying all four linkers: <https://refspecs.linuxfoundation.org/elf/gabi4+/contents.html>
• `cargo build --timings` — unstable/stable build timing HTML reports referenced in the Cargo Book: <https://doc.rust-lang.org/cargo/reference/timings.html>