GPU Scheduling at 10k Nodes: Kubernetes vs Ray vs Slurm vs Volcano

A 10,000-node cluster of 8-GPU H100 servers holds 80,000 accelerators. At a list price of two dollars and fifty cents per GPU-hour, that is $200,000 per hour of allocation — $4.8M per day, before power. Every percentage point of utilization you lose to scheduler pathologies costs about $48,000 per day.

That framing is the entire reason this comparison matters. At one node, every scheduler works. At one hundred nodes, they all mostly work. At ten thousand nodes, the differences stop being stylistic and become architectural: whether the control plane is a single writer or a distributed quorum, whether gang admission is a first-class primitive or a plugin bolted onto a node-fit loop, and whether the scheduler understands that 256 GPUs must be placed as an indecomposable unit across a rail-optimized InfiniBand fabric.

This article dissects all four systems at that scale, with the measurement methodology and cost math needed to make a defensible platform decision.

The Three Latency Budgets Nobody Separates

Teams routinely conflate scheduling latency with time-to-first-token-of-training. They are different budgets with different owners and different fixes.

1. Admission latency — submission to binding. Scheduler-bound. Dominated by filter/score complexity, gang deadlock, and queue fairness churn.
2. Placement-to-ready latency — binding to container running. Kubelet-bound. Dominated by image pull size (a 40 GB CUDA training image is a twelve-minute pull on a cold node), NVIDIA driver injection, and device plugin registration.
3. Rendezvous latency — process start to first NCCL all-reduce. Fabric-bound. Dominated by rank discovery, MASTER_ADDR propagation, and NCCL topology detection.

At 10k nodes, budget (1) is typically sub-second to low tens of seconds in healthy systems, but it has a fat tail. Budget (2) is where 80% of perceived slowness lives. Budget (3) is where distributed training jobs die silently with NCCL timeout on rail-misaligned placements.

Measure all three. Report them as p50, p99, and p99.9. A scheduler with a 300 ms mean and a 90-second p99 is a broken scheduler.

The Control Plane Ceiling: What Actually Breaks at 10k Nodes

The first bottleneck is never the scheduling algorithm. It is the state store and the fan-out.

• etcd keeps the entire cluster object graph in one Raft group. The default 2 GiB backend quota is generous for 1,000 nodes and cramped for 10,000 nodes running high-churn batch jobs. Set --quota-backend-bytes=8589934592, enable --auto-compaction-retention, and defragment on a schedule. Watch event fan-out and large object churn (every pod status update) are the real cost drivers.
• kube-apiserver applies API Priority and Fairness (APF) so that a controller storm from a Ray autoscaler cannot starve the scheduler or the node lifecycle controller. Give kube-scheduler its own PriorityLevelConfiguration with reserved concurrency.
• kube-scheduler historically shipped with --kube-api-qps=50 and --kube-api-burst=100. Those values are fine for hundreds of nodes and catastrophic for thousands of nodes with high pod churn. Documented scaling SLOs target p99 end-to-end pod startup latency in the low seconds at 5,000 nodes — which is only achievable with tuned QPS, disabled default plugins, and a percentageOfNodesToScore that respects the cluster size.

Here is a production kube-scheduler profile for a GPU-heavy 10k-node cluster:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
parallelism: 32
# Default formula collapses to the 5% floor at 10k nodes (500 nodes scored).
# Raise deliberately only if your bin-packing is weak; each increase is linear cost.
percentageOfNodesToScore: 10
profiles:
  - schedulerName: gpu-scheduler
    plugins:
      score:
        enabled:
          - name: NodeResourcesBalancedAllocation
            weight: 5
          - name: InterPodAffinity
            weight: 3
          - name: NodeAffinity
            weight: 3
        disabled:
          - name: NodeResourcesLeastAllocated
      # No QueueSort override: default PrioritySort is deterministic and cheap.
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: LeastAllocated
            resources:
              - name: nvidia.com/gpu
                weight: 10
              - name: cpu
                weight: 2
              - name: memory
                weight: 1

And the API server flags that actually matter:

# kube-apiserver flags (excerpt)
- --goaway-chance=0.001
- --max-requests-inflight=600
- --max-mutating-requests-inflight=300
- --enable-priority-and-fairness=true
- --watch-cache=true

The critical missing primitive in vanilla Kubernetes is gang scheduling. kube-scheduler binds pods one at a time. A 32-node training job with replicas: 256 will happily schedule 200 pods, then fail to place the rest due to resource fragmentation, leaving a partial job holding 1,600 GPUs at zero throughput. On 10k nodes with mixed workload sizes, this is a daily occurrence. You either add the coscheduling scheduler plugin, use Kueue for quota-aware job admission, or move to a scheduler that treats the job as the unit of work.

Note also that NVIDIA's Dynamic Resource Allocation (DRA) is now the strategic path for structured GPU claims — MIG slices, ComputeDomains, and topology-aware allocation expressed as first-class API objects rather than the flat nvidia.com/gpu: 8 integer of the device plugin era.

Volcano: Gang Admission and Queue Fairness on Kubernetes

Volcano is the CNCF-hosted answer to Kubernetes' gang-blindness. It replaces the binding loop with a session-based scheduler: enqueue → allocate → preempt → reclaim → backfill, with plugin-driven policy at each stage.

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: gpu-prod
spec:
  reclaimable: false
  weight: 10
  capability:
    nvidia.com/gpu: 4096
    cpu: 200000
---
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: llama-pretrain-70b
  namespace: research
spec:
  # Atomic: no pod binds unless all 32 are feasible.
  minAvailable: 32
  queue: gpu-prod
  schedulerName: volcano
  maxRetry: 3
  policies:
    - event: PodEvicted
      action: RestartJob
    - event: JobReady
      action: CompleteJob
  plugins:
    ssh: []
    svc: []
  tasks:
    - name: worker
      replicas: 32
      template:
        spec:
          containers:
            - name: trainer
              image: registry.internal/llm-train:2.4.1
              resources:
                limits:
                  nvidia.com/gpu: 8

The proportion plugin implements weighted fair-share across queues, with capability acting as a hard ceiling so a research team cannot starve inference serving. The drf plugin adds dominant-resource fairness for heterogeneous jobs. binpack reduces fragmentation by concentrating allocations, which matters enormously on rail-optimized fabrics where 8-GPU blocks should land in the same leaf.

Operationally, Volcano's weak points are the same as Kubernetes': it inherits etcd's ceiling, and its preemption loop can generate significant API server churn during large reclaim events. It also lacks Slurm's decades of topology-aware constraint expressiveness — --switches=1@00:20:00 has no clean equivalent.

Ray: The Distributed Runtime That Happens to Schedule

Ray is not a cluster scheduler in the Slurm sense; it is a distributed Python/C++ runtime with a control plane. The relevant primitives are placement groups, which provide atomic multi-node gang allocation.

from __future__ import annotations

import logging
import sys
from dataclasses import dataclass

import ray
from ray.exceptions import GetTimeoutError, RaySystemError
from ray.util.placement_group import placement_group, remove_placement_group
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy

logger = logging.getLogger('train.launcher')


@dataclass(frozen=True)
class GangSpec:
    nodes: int
    gpus_per_node: int
    cpus_per_node: int
    strategy: str = 'STRICT_SPREAD'


def reserve_gang(spec: GangSpec, timeout_s: float = 300.0):
    bundles = [
        {'GPU': spec.gpus_per_node, 'CPU': spec.cpus_per_node}
        for _ in range(spec.nodes)
    ]
    pg = placement_group(bundles, strategy=spec.strategy)
    try:
        ray.get(pg.ready(), timeout=timeout_s)
    except GetTimeoutError:
        remove_placement_group(pg)
        raise RuntimeError(
            f'gang of {spec.nodes}x{spec.gpus_per_node} GPU not schedulable '
            f'within {timeout_s}s; queue pressure or fragmentation'
        )
    except RaySystemError as exc:
        remove_placement_group(pg)
        raise RuntimeError('GCS error while reserving placement group') from exc
    logger.info('gang reserved: %d bundles', len(bundles))
    return pg


def main() -> int:
    ray.init(address='auto', namespace='training')
    spec = GangSpec(nodes=32, gpus_per_node=8, cpus_per_node=96)
    pg = reserve_gang(spec)
    strategy = PlacementGroupSchedulingStrategy(
        placement_group=pg, placement_group_bundle_index=0
    )
    trainer = ray.remote(num_gpus=0)(_noop)
    ray.get(trainer.options(num_cpus=0, scheduling_strategy=strategy).remote())
    return 0


def _noop() -> None:
    return None


if __name__ == '__main__':
    sys.exit(main())

Architecturally, Ray's control plane (GCS) is a logically centralized singleton backed by a key-value store, with per-node raylets handling the actual task placement. This scales well into the low thousands of nodes; beyond that, GCS load, placement group churn, and the autoscaler's polling loop become the dominant tail-latency sources. Ray's strength is not raw admission throughput — it is that the scheduler is the runtime, so data locality, object store affinity, and actor placement are all decided together.

Run Ray on Kubernetes via KubeRay (RayCluster, RayJob CRDs) and the autoscaler, and you inherit both stacks' failure modes. Budget for that.

Slurm: Still the Fastest Gang Scheduler in the Room

Slurm's slurmctld is a single-writer, single-threaded C controller, and it is still the benchmark to beat for throughput and topology constraint expressiveness. It has run 158,976-node systems.

# slurm.conf (excerpt)
ClusterName=gpu-prod
SlurmctldHost=ctld01
gresTypes=gpu
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
TopologyPlugin=topology/tree
SchedulerType=sched/backfill
SchedulerParameters=bf_max_job_start=200,bf_window=1440,bf_resolution=120,bf_continue
PriorityType=priority/multifactor
PriorityWeightFairshare=10000
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity,task/cray
Epilog=/etc/slurm/epilog.d/cleanup-gpu

#!/usr/bin/env bash
#SBATCH --job-name=llama70b
#SBATCH --nodes=32
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --cpus-per-task=16
#SBATCH --exclusive
#SBATCH --constraint=h100-80gb
#SBATCH --switches=1@00:20:00   # all nodes under one leaf switch: rail safety
#SBATCH --time=12:00:00
#SBATCH --output=%x-%j.out
set -euo pipefail
srun --kill-on-bad-exit=1 --cpu-bind=cores \
  bash -lc 'exec python -m train.main --config configs/70b.yaml'

--switches is the killer feature no Kubernetes-native scheduler replicates cleanly: it asks the interconnect topology for a set of nodes within N hops, which directly determines whether NCCL finds an all-to-all rail or falls back to spine oversubscription. Slurm's weaknesses are equally concrete — no cloud-native API surface, slurmrestd as an afterthought relative to the CRD ecosystem, and painful interactive multi-tenancy for notebook-style workloads.

Head-to-Head: The Only Comparison Table That Matters

Measuring It Honestly: The Benchmark Harness

Most published comparisons are unfalsifiable. Here is a harness that produces numbers you can defend.

from __future__ import annotations

import json
import os
import sys
from dataclasses import dataclass
from typing import Final
from urllib.error import HTTPError, URLError
from urllib.parse import urlencode
from urllib.request import Request, urlopen

PROM: Final[str] = os.environ.get('PROM_URL', 'http://prometheus.monitoring:9090')
TIMEOUT_S: Final[float] = 15.0


def prom_query(expr: str) -> list[dict]:
    url = f'{PROM}/api/v1/query'
    body = urlencode({'query': expr}).encode('utf-8')
    req = Request(url, data=body,
                  headers={'Content-Type': 'application/x-www-form-urlencoded'})
    try:
        with urlopen(req, timeout=TIMEOUT_S) as resp:
            payload = json.loads(resp.read().decode('utf-8'))
    except (HTTPError, URLError, TimeoutError) as exc:
        raise RuntimeError(f'prometheus unreachable for query: {expr!r}') from exc
    if payload.get('status') != 'success':
        raise RuntimeError(
            f'prometheus error {payload.get("status")}: {payload.get("error")}'
        )
    return payload['data']['result']


def p99_admission_latency_seconds(window: str = '1h') -> float:
    expr = (
        'histogram_quantile(0.99, sum by (le) (rate('
        'scheduler_pod_scheduling_sli_duration_seconds_bucket'
        f'[{window}])))'
    )
    result = prom_query(expr)
    if not result:
        raise RuntimeError('no scheduling SLI series; is the scheduler scraped?')
    return float(result[0]['value'][1])


def gpu_active_utilization(window: str = '1h') -> float:
    # Fraction of allocated GPUs doing real work over the window.
    expr = (
        'avg(avg_over_time(DCGM_FI_PROF_SM_ACTIVE'
        f'[{window}]) > 0.20)'
    )
    result = prom_query(expr)
    if not result:
        raise RuntimeError('dcgm-exporter not scraped or GPUs not labeled')
    return float(result[0]['value'][1])

Run it continuously against all clusters with identical workload traces. A one-shot benchmark is a marketing artifact; a rolling p99 dashboard is an operational instrument.

Utilization and the True Cost per Million GPU-Hours

Two utilization numbers matter, and multiplying them is the only honest cost model:

• Allocation utilization = GPUs bound to pods / total GPUs. Measures scheduler waste: fragmentation, queue starvation, over-provisioned reservations.
• Active utilization = allocated GPUs with SM activity above threshold / allocated GPUs. Measures workload waste: data-loader starvation, checkpoint stalls, bad batch sizing, NCCL bubbles.

from __future__ import annotations

from dataclasses import dataclass
from decimal import Decimal, ROUND_HALF_UP

@dataclass(frozen=True)
class ClusterEconomics:
    nodes: int
    gpus_per_node: int
    list_price_per_gpu_hour: Decimal
    live_price_per_gpu_hour: Decimal
    alloc_util: Decimal
    active_util: Decimal

    @property
    def total_gpus(self) -> int:
        return self.nodes * self.gpus_per_node

    def cost_per_million_gpu_hours(self, tax: Decimal = Decimal('0.00')) -> Decimal:
        if not (Decimal('0') < self.alloc_util <= Decimal('1')):
            raise ValueError('alloc_util must be in (0, 1]')
        if not (Decimal('0') < self.active_util <= Decimal('1')):
            raise ValueError('active_util must be in (0, 1]')
        if not (Decimal('0') <= tax < Decimal('1')):
            raise ValueError('scheduling tax must be in [0, 1)')

        effective_per_hour = self.live_price_per_gpu_hour / (self.alloc_util * self.active_util)
        adjusted = effective_per_hour / (Decimal('1') - tax)
        return (adjusted * Decimal(1_000_000)).quantize(
            Decimal('1'), rounding=ROUND_HALF_UP
        )


if __name__ == '__main__':
    cluster = ClusterEconomics(
        nodes=10_000,
        gpus_per_node=8,
        list_price_per_gpu_hour=Decimal('2.50'),
        live_price_per_gpu_hour=Decimal('2.50'),
        alloc_util=Decimal('0.92'),
        active_util=Decimal('0.61'),
    )
    print(cluster.cost_per_million_gpu_hours(tax=Decimal('0.04')))
    # ~$4,455,000 per million GPU-hours versus a $2.5M nominal

The lesson: active utilization dominates everything else. A 4% scheduling tax is noise next to a 61% active-utilization number. Teams burn months tuning schedulers when the real loss is a data pipeline that cannot feed 8,192 GPUs. Fix the pipeline first, then the scheduler.

Production Checklist Before You Commit

• Pin scheduler QPS/burst and APF priority levels; load-test at 3x expected churn.
• Size etcd for 8 GiB, automate defragmentation, monitor etcd_server_quota_backend_bytes.
• Require gang admission for any job spanning more than one node — no exceptions.
• Enforce queue capability ceilings; unbounded queues always end in preemption storms.
• Export scheduler_pod_scheduling_sli_duration_seconds and DCGM SM activity to the same Prometheus as alerts.
• Pin --switches / topology constraints for any job larger than one leaf switch.
• Pre-pull training images via a DaemonSet; a cold pull is the single largest non-scheduler latency contributor.
• Track cost per million GPU-hours weekly, not quarterly — the feedback loop must be faster than the budget cycle.

Pitfalls That Cost Real Money

• Treating placement groups as free. Every un-released placement group on Ray is reserved capacity that no other workload can use. Always remove_placement_group in a finally block.
• Assuming `nvidia.com/gpu: 8` means one NUMA-consistent node. On multi-NUMA hosts, GPU-to-CPU affinity errors silently cost 5-15% throughput. Pin with the NVIDIA GPU Operator's topology manager.
• Benchmarking with uniform job sizes. Real clusters have bimodal distributions; fragmentation only appears with mixed sizes.
• Comparing scheduler latency across clusters with different image pull warm-up. Normalize or you are measuring registries.
• Ignoring MIG granularity in the cost model. A 1g.10gb MIG slice at 1/7th of the GPU is not 1/7th of the price unless you price it that way.

Sources & Technical References

• Kubernetes SIG Scheduling — kube-scheduler configuration API v1: https://kubernetes.io/docs/reference/config-api/kube-scheduler-config.v1/
• Kubernetes — Scheduling Framework (PreFilter/Filter/Score/Reserve/Permit/Bind): https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/
• Kubernetes SIG Scalability — published SLOs and scalability thresholds: https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md
• Kubernetes — Dynamic Resource Allocation (DRA): https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/
• Kubernetes — API Priority and Fairness: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/
• etcd — Maintenance, defragmentation, and backend quota: https://etcd.io/docs/v3.5/op-guide/maintenance/
• Volcano — Official documentation: https://volcano.sh/en/docs/
• Volcano — Scheduler plugins (gang, drf, proportion, binpack): https://volcano.sh/en/docs/plugins/
• Volcano — Queue and capability semantics: https://volcano.sh/en/docs/queue/
• Kueue — Job queueing and quota management for Kubernetes: https://kueue.sigs.k8s.io/
• Kubernetes scheduler-plugins — coscheduling (gang) plugin: https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling
• Ray — Placement Groups and scheduling strategies: https://docs.ray.io/en/latest/ray-core/scheduling/placement-group.html
• Ray — Architecture whitepaper (GCS, raylets, ownership): https://docs.ray.io/en/latest/ray-contribute/whitepaper.html
• KubeRay — Ray on Kubernetes operator: https://docs.ray.io/en/latest/cluster/kubernetes/index.html
• Slurm — slurm.conf reference (SelectType, TopologyPlugin, SchedulerParameters): https://slurm.schedmd.com/slurm.conf.html
• Slurm — Generic Resource (GRES) scheduling for GPUs: https://slurm.schedmd.com/gres.html
• Slurm — Network topology guide (topology/tree, topology/block): https://slurm.schedmd.com/topology.html
• Slurm — Backfill scheduling configuration: https://slurm.schedmd.com/sched_config.html
• NVIDIA — DCGM Exporter metrics reference: https://github.com/NVIDIA/dcgm-exporter
• NVIDIA — GPU Operator documentation (topology manager, device plugin): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html
• NVIDIA — Multi-Instance GPU (MIG) user guide: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
• Project HAMi — Heterogeneous AI computing virtualization middleware (vGPU on Volcano): https://github.com/Project-HAMi/HAMi
• Prometheus — histogram_quantile and native histogram semantics: https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile
• Chowdhery et al., PaLM: Scaling Language Modeling with Pathways (definition of Model FLOPs Utilization): https://arxiv.org/abs/2204.02311
• Burns et al., Borg, Omega, and Kubernetes, ACM Queue, 2016: https://queue.acm.org/detail.cfm?id=2898444