Kubernetes Manifest Templating at 10k Services: Helm vs Kustomize vs Jsonnet vs CUE — Render Latency, Memory, and True Cost per 100k Deploys

At 10,000 services, a 150 ms render is not a rendering problem. It is a queueing, retry, and developer-wait problem. The wrong templating choice rarely breaks a small cluster; at platform scale it quietly taxes every pipeline, every GitOps refresh, and every on-call incident.

This article benchmarks four mainstream Kubernetes manifest templating approaches—Helm, Kustomize, Jsonnet, and CUE—on a reference 10k-service estate. It focuses on p99 render latency, peak RSS, and the true cost per 100k deploys, including the hidden costs that average-latency benchmarks miss.

The Real Cost Model for 100k Deploys

Render latency is only one term in the cost equation. A useful model is:

cost_100k = (100000 / throughput_rps) * worker_hourly_cost / 3600 + retry_rate * pipeline_cost + developer_wait_cost

At 10k services, p99 matters more than p50. If 1% of 100k deploys hit a 300 ms tail, that is 1,000 slow renders. In a GitOps controller, those tails become reconciliation backlog. In CI, they become flaky timeouts and retries.

The true cost per 100k deploys includes CI runner minutes, controller memory, cache misses, failed renders, and the human cost of waiting for feedback. Render compute is often less than 5% of the total. Optimizing the other 95% is where platform teams win.

Benchmark Methodology: 10k Services, 24 Resources, and p99

Our reference workload models 10,000 services, each with 24 Kubernetes resources: Deployment, Service, ConfigMap, Secret reference, HPA, PDB, NetworkPolicy, ServiceAccount, RoleBinding, and environment-specific patches across four environments. The harness runs on a 16 vCPU, 32 GiB cloud instance with local NVMe and warm caches after the first render.

We measure p50, p95, p99, peak RSS per worker, allocations, and throughput. We do not measure cold CLI startup unless explicitly noted. For production, measure the same path your GitOps controller or CI runner actually uses—ideally in-process SDKs, not shelled-out CLIs.

A minimal Go benchmark harness should report allocations and fail fast:

func BenchmarkRenderer(b *testing.B, r Renderer) {
    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        if err := r.Render("testdata/app", io.Discard); err != nil {
            b.Fatal(err)
        }
    }
}

Helm: Chart Semantics and In-Process Rendering

Helm remains the packaging standard for Kubernetes. It gives you values, subcharts, dependencies, release semantics, and a vast ecosystem. Its cost is Go template execution and, in many pipelines, the overhead of shelling out to helm template.

In-process rendering with the Helm SDK removes process spawn, kubeconfig discovery, and CLI flag parsing. A minimal renderer looks like this:

package render

import (
    "fmt"
    "io"

    "helm.sh/helm/v3/pkg/chart/loader"
    "helm.sh/helm/v3/pkg/chartutil"
    "helm.sh/helm/v3/pkg/engine"
)

type HelmRenderer struct {
    ReleaseName string
    Namespace   string
    Values      map[string]interface{}
}

func (r HelmRenderer) Render(chartPath string, out io.Writer) error {
    ch, err := loader.Load(chartPath)
    if err != nil {
        return fmt.Errorf("load helm chart: %w", err)
    }
    vals, err := chartutil.ToRenderValues(ch, r.Values, chartutil.ReleaseOptions{
        Name:      r.ReleaseName,
        Namespace: r.Namespace,
        IsInstall: true,
    }, nil)
    if err != nil {
        return fmt.Errorf("coalesce values: %w", err)
    }
    rendered, err := engine.Render(ch, vals)
    if err != nil {
        return fmt.Errorf("render helm templates: %w", err)
    }
    for name, manifest := range rendered {
        if _, err := fmt.Fprintf(out, "---\n# Source: %s\n%s\n", name, manifest); err != nil {
            return fmt.Errorf("write manifest %s: %w", name, err)
        }
    }
    return nil
}

Helm p99 is usually acceptable for small charts, but subchart traversal, values coalescing, and template function calls add up. Avoid shelling out to helm template in a hot loop. That measures process startup and filesystem discovery as much as rendering.

Kustomize: Overlays and Deep-Merge Costs

Kustomize avoids a templating language. It composes YAML with bases, overlays, patches, and generators. That makes it Git-friendly and easy to review, but overlay sprawl and deep strategic merges still consume CPU.

A production Go renderer can call the Kustomize API directly:

package render

import (
    "fmt"
    "io"

    "sigs.k8s.io/kustomize/api/krusty"
    "sigs.k8s.io/kustomize/kyaml/filesys"
)

type KustomizeRenderer struct {
    Options *krusty.Options
}

func (r KustomizeRenderer) Render(dir string, out io.Writer) error {
    if r.Options == nil {
        r.Options = krusty.MakeDefaultOptions()
    }
    k := krusty.MakeKustomizer(r.Options)
    resMap, err := k.Run(filesys.MakeFsOnDisk(), dir)
    if err != nil {
        return fmt.Errorf("run kustomize: %w", err)
    }
    yaml, err := resMap.AsYaml()
    if err != nil {
        return fmt.Errorf("serialize kustomize resources: %w", err)
    }
    if _, err := out.Write(yaml); err != nil {
        return fmt.Errorf("write kustomize output: %w", err)
    }
    return nil
}

Kustomize is usually the fastest option for straightforward overlays. Its risk is structural: hundreds of overlays with remote bases and complex patches can turn a fast engine into a slow repository problem. Cache immutable bases and avoid remote imports on the critical path.

Jsonnet: Hermetic Power at a Latency Premium

Jsonnet is a data templating language with functions, imports, mixins, and lazy evaluation. It is excellent for generating complex configuration families, but that power has a cost. Deep imports, large mixin graphs, and lazy evaluation can increase p99 and memory.

A minimal go-jsonnet renderer:

package render

import (
    "fmt"
    "io"

    "github.com/google/go-jsonnet"
)

type JsonnetRenderer struct {
    ImportPaths []string
}

func (r JsonnetRenderer) Render(path string, out io.Writer) error {
    vm := jsonnet.MakeVM()
    if len(r.ImportPaths) > 0 {
        vm.Importer(&jsonnet.FileImporter{JPaths: r.ImportPaths})
    }
    jsonStr, err := vm.EvaluateFile(path)
    if err != nil {
        return fmt.Errorf("evaluate jsonnet %s: %w", path, err)
    }
    if _, err := io.WriteString(out, jsonStr); err != nil {
        return fmt.Errorf("write jsonnet output: %w", err)
    }
    return nil
}

Jsonnet is not a drop-in Helm replacement unless you already invest in a Jsonnet platform. At 10k services, the win comes from eliminating duplication across service families, not from raw render speed.

CUE: Validation-First Rendering

CUE is a constraint language and data validator. It unifies schemas, values, and policies, making it attractive for platform guardrails. It can render or export configuration, but its primary strength is validation before rendering.

A basic CUE export in Go:

package render

import (
    "fmt"
    "io"
    "os"

    "cuelang.org/go/cue/cuecontext"
)

type CUERenderer struct{}

func (r CUERenderer) Render(path string, out io.Writer) error {
    src, err := os.ReadFile(path)
    if err != nil {
        return fmt.Errorf("read cue file %s: %w", path, err)
    }
    ctx := cuecontext.New()
    val := ctx.CompileString(string(src))
    if err := val.Err(); err != nil {
        return fmt.Errorf("compile cue %s: %w", path, err)
    }
    b, err := val.MarshalJSON()
    if err != nil {
        return fmt.Errorf("marshal cue to json: %w", err)
    }
    if _, err := out.Write(b); err != nil {
        return fmt.Errorf("write cue output: %w", err)
    }
    return nil
}

For production CUE with imports and packages, use cuelang.org/go/cue/load with proper build context and import resolution. CUE often has lower render latency than Helm or Jsonnet, but it is not a full substitute for Helm release semantics or Kustomize overlay workflows.

Benchmark Results: p99, RSS, and Cost per 100k Deploys

Representative results from our reference workload. Your numbers will vary with chart complexity, disk cache, concurrency, and import graphs.

Render compute is calculated from normalized vCPU-hour cost and 70% utilization. True cost includes CI runner minutes, retries, cache misses, and developer wait. The gap between render compute and true cost is the lesson: templating engine choice matters, but pipeline architecture and failure rates matter more.

What the Numbers Mean for Platform Teams

Kustomize is the pragmatic default for pure Kubernetes overlays and environment promotion. Helm is the default when you need packaging, values, subcharts, and third-party ecosystem compatibility. Jsonnet is a deliberate investment for complex configuration families and should be treated as a platform product. CUE is strongest as a validation and policy layer, not merely a renderer.

At 10k services, the winning architecture is often hybrid: Helm for distribution, Kustomize for environment overlays, CUE for schema and policy validation, and Jsonnet only where hermetic generation pays for itself. Do not measure one engine in isolation and declare a universal winner.

GitOps Controller Implications: Argo CD and Flux CD at 10k Apps

In Argo CD, the repo-server renders manifests and caches artifacts. At 10k applications, repo-server memory, cache churn, and refresh concurrency become the bottleneck. Shard repos and applications across repo-server replicas, keep caches warm, and avoid rendering thousands of independent apps from one enormous repo on every webhook.

Flux CD's kustomize-controller is efficient for overlay-driven workflows, but controller concurrency and source artifact size still matter. Render in CI for fast feedback, and let GitOps controllers reconcile immutable, already-rendered artifacts when possible. The controller should apply validated output, not become your primary templating bottleneck.

Production Checklist for Kubernetes Manifest Templating

• Pin Helm, Kustomize, Jsonnet, and CUE versions in CI and controller images.
• Measure p99 and peak RSS with the same concurrency model as production.
• Separate cold-cache and warm-cache benchmarks; report both.
• Avoid remote bases, remote imports, and network fetches on the render hot path.
• Cache rendered manifests by content hash and engine version.
• Render once per commit, apply many times; do not re-render during every apply.
• Use --dry-run=server for admission validation, not as a templating benchmark.
• Add a render-time SLO to platform dashboards and alert on p99 regression.
• Shard GitOps controllers by team, repo, or environment before scaling vertically.

Pitfalls That Invalidate Templating Benchmarks

Comparing helm template CLI startup against an in-process Kustomize API is not apples-to-apples. Measuring average latency hides queueing tails. Ignoring cache state produces optimistic results. Benchmarking a three-resource toy chart tells you nothing about a 24-resource service with subcharts and patches. Finally, treating render compute as the whole cost misses retries, developer wait, and controller memory pressure.

Sources & Technical References

• Helm documentation: https://helm.sh/docs/
• Helm Go SDK: https://pkg.go.dev/helm.sh/helm/v3
• Kustomize documentation: https://kubectl.docs.kubernetes.io/guides/introduction/kustomize/
• Kustomize Go API: https://pkg.go.dev/sigs.k8s.io/kustomize/api/krusty
• Jsonnet language reference: https://jsonnet.org/
• go-jsonnet package: https://pkg.go.dev/github.com/google/go-jsonnet
• CUE documentation: https://cuelang.org/docs/
• CUE Go API: https://pkg.go.dev/cuelang.org/go/cue
• Argo CD high availability and repo-server guidance: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/
• Flux kustomize-controller: https://fluxcd.io/flux/components/kustomize/
• OpenGitOps principles: https://opengitops.dev/
• Kubernetes documentation: https://kubernetes.io/docs/