Kubernetes GPU FinOps at 1,000 A100s: Time-Slicing vs. MIG vs. MPS, Benchmarked

The $30M Question Nobody Budgeted For

A thousand NVIDIA A100 80GB SXM modules at a blended on-demand rate of roughly $3.40/GPU-hour is a $29.8M/year line item before you pay for a single CPU node, NIC, or storage byte. And in the overwhelming majority of clusters we have audited, mean streaming multiprocessor (SM) utilization sits between 25% and 35%. You are paying for a thousand GPUs and effectively running three hundred.

The reason is structural, not cultural. Kubernetes' default GPU contract is whole-device allocation: nvidia.com/gpu: 1 grants exclusive ownership of an entire 80GB A100 for the pod's lifetime. A Jupyter notebook that idles at 2% SM occupancy while a data scientist reads documentation occupies the same accelerator as a 40-hour pre-training job. Static partitioning by namespace or team makes it worse — it converts temporal fragmentation into permanent fragmentation.

This article is the writeup of a 12-week, three-arm benchmark we ran across 1,000 A100s to answer a deceptively simple question: which GPU sharing mechanism gives the best cost-per-useful-work at enterprise scale, and at what isolation cost? We measured time-slicing, MIG, and MPS against whole-device baselines on scheduling latency, fault and memory isolation, throughput, and finally dollars. The tiered architecture that came out of it reduced annualized GPU spend by 60.1% without a single SLA breach.

The Three Sharing Primitives (And What They Actually Partition)

Before benchmarking, you must internalize that these three mechanisms partition fundamentally different physical resources. Conflating them is the #1 source of bad GPU FinOps decisions.

Time-Slicing: Cooperative Context Switching, Zero Enforcement

Time-slicing is implemented entirely inside the CUDA driver's scheduler. Multiple processes submit work to the same GPU; the driver preempts at kernel-launch boundaries and round-robins contexts. Critically:

• No memory isolation. Every process shares the full 80GB HBM address space. One OOM kills the whole device.
• No fault isolation. An illegal memory access or an ECC error takes down every co-tenant.
• No QoS guarantee. A bursty neighbor can starve you indefinitely; there is no fairness contract.
• No scheduler visibility. This is the killer for latency-sensitive workloads — see the next section.

MIG: Hardware Partitioning via the GA100 Die

Multi-Instance GPU carves the A100's 108 SMs and 8 HBM slices into up to seven isolated instances using hardware-level partitioning of the memory controllers and L2 cache slices. On an A100 80GB, valid profiles include 1g.10gb (×7), 2g.20gb (×3), 3g.40gb (×2), 4g.40gb (×1 + 3g.40gb), and 7g.80gb (×1).

Each MIG instance gets its own dedicated SMs, its own memory bandwidth slice, and — this is the crucial part — its own error domain. A crash in 1g.10gb instance 3 does not touch instance 4. MIG instances are advertised by the NVIDIA device plugin as first-class Kubernetes extended resources (nvidia.com/mig-1g.10gb), which means the kube-scheduler has ground truth about capacity.

MPS: The Client/Server Concurrency Model

The Multi-Process Service runs a control daemon that multiplexes CUDA contexts from many client processes onto one hardware context, eliminating the context-switch and register-file save/restore overhead that time-slicing incurs. MPS gives true concurrent kernel execution — a 10% SM-utilization kernel from client A and a different kernel from client B can co-reside on the same SM.

MPS is the right tool for many small, cooperating, homogeneous workloads (ensemble inference, hyperparameter sweeps of tiny models). It is not a tenancy boundary: all clients share one address space unless you pin memory. You can, however, enforce a compute quota per client:

# Enforce per-client limits on the MPS server
export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=12      # ~13 SMs of 108
export CUDA_MPS_PINNED_DEVICE_MEM_LIMIT="0=10737418240"  # 10 GiB
nvidia-cuda-mps-control -d

Benchmark Methodology

We ran three 8-week arms on identical hardware (DGX A100 nodes, 8× A100-80GB SXM4, NVLink 3, ConnectX-6 200GbE, Ubuntu 22.04, containerd 1.7, Kubernetes 1.30, GPU Operator 24.9, driver 550.90.07).

Workload mix (fixed across arms):

Instruments: DCGM Exporter 3.3 (DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_XID_ERRORS), kube-state-metrics, scheduler_framework_extension_point_duration_seconds, kube_pod_status_scheduled_time, and OpenCost 1.109 for allocation. Utilization is measured as SM-active occupancy (fraction of cycles at least one warp resident), not the misleading DCGM_FI_DEV_GPU_UTIL sampling counter.

Result 1: The Scheduler Is Lying To You About Time-Slicing

This is the single most important operational finding. With time-slicing at replicaCount: 4, the device plugin advertises 4× nvidia.com/gpu per physical A100. The kube-scheduler sees free capacity and binds the pod in milliseconds. The pod goes Running. Your platform dashboard turns green.

And then the pod sits in the CUDA driver's queue for eleven minutes.

Time-slicing moves the queue from the scheduler into the driver, where Kubernetes, the Horizontal Pod Autoscaler, and the Cluster Autoscaler cannot observe it. Your autoscaler concludes the cluster is full of running pods and refuses to add nodes, while every one of those pods is blocked on a GPU mutex. We have seen this pattern produce 40-minute p99 latency on "autoscaled" inference platforms.

Mitigation: if you use time-slicing for interactive/batch work, you must instrument application-level queue depth and expose it to a custom autoscaler metric. Do not trust Running as a proxy for "making progress."

MIG does not have this problem — pending pods are genuinely Pending, visible to kube_pod_status_phase, and Cluster Autoscaler will react correctly.

Result 2: Isolation Is Not A Slider, It's A Contract

We ran a deliberate fault-injection arm: 500 pods executing a CUDA kernel with an out-of-bounds write, and 500 pods allocating 12GB on a device where co-tenants held 70GB.

For anything multi-tenant — and at 1,000 GPUs, everything is multi-tenant — MIG is the only mechanism that satisfies a real availability contract. Time-slicing and MPS are performance-optimization tools for trusted co-tenants, not security or availability boundaries.

The Tiered Architecture That Cut 60%

The mistake most teams make is picking one mechanism cluster-wide. The correct answer is a tier hierarchy mapped to workload classes, enforced by admission control and node pools.

Tier 0 — MIG 7g.80gb: Distributed Pre-Training

Multi-node gang-scheduled jobs need full NVLink bandwidth and full HBM. Dedicated node pool, nvidia.com/mig.config=all-disabled label, taints to prevent co-scheduling.

Tier 1 — MIG 3g.40gb / 2g.20gb: Fine-Tuning & Latency-Sensitive Inference

Hard isolation, guaranteed SMs, scheduler-visible pending state. This is where the bulk of the savings live because fine-tuning jobs at 90% SM util on a 3g.40gb slice cost 43% less than the same job on a whole A100.

Tier 2 — Time-Slicing ×4: Interactive Notebooks

Throwaway dev environments where 400ms of queueing is invisible and isolation is irrelevant. 71% utilization vs. 31% baseline.

Tier 3 — MPS ×8: CI / Unit Tests

Seconds-long jobs that spend most of their life in context setup. MPS eliminates per-process context overhead, which is precisely the bottleneck here.

Wiring It Up: Device Plugin Configuration

Time-slicing via GPU Operator:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  a100-dev: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4
---
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  devicePlugin:
    config:
      name: time-slicing-config
      default: a100-dev

MIG profiles via `mig-parted`:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
  namespace: gpu-operator
data:
  config.yaml: |-
    version: v1
    mig-configs:
      all-1g.10gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
      inference-mix:
        - devices: all
          mig-enabled: true
          mig-devices:
            "3g.40gb": 1
            "2g.20gb": 1
            "1g.10gb": 2
      all-disabled:
        - devices: all
          mig-enabled: false

# Apply profile to a node pool — GPU Operator reconciles the rest
kubectl label node gpu-infer-07 nvidia.com/mig.config=inference-mix --overwrite
kubectl label node gpu-train-01 nvidia.com/mig.config=all-disabled --overwrite

Use migStrategy: single for homogeneous pools (advertises only nvidia.com/mig-3g.40gb) and migStrategy: mixed when a pool serves heterogeneous profiles.

Admission Control: Route Pods To The Right Tier

A mutating webhook that translates a semantic tier label into the correct extended resource and node affinity. This eliminates the entire class of "data scientist requested nvidia.com/gpu: 1 and got a whole A100" incidents.

package gpuclass

import (
	"context"
	"encoding/json"
	"fmt"
	"log"
	"net/http"

	admissionv1 "k8s.io/api/admission/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/resource"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

type tier struct {
	Resource   string
	NodePool   string
	DefaultQty int64
}

// tierMap is the single source of truth for GPU allocation policy.
var tierMap = map[string]tier{
	"dev":       {"nvidia.com/gpu", "gpu-dev", 1},
	"inference": {"nvidia.com/mig-3g.40gb", "gpu-infer", 1},
	"finetune":  {"nvidia.com/mig-2g.20gb", "gpu-infer", 1},
	"train":     {"nvidia.com/gpu", "gpu-train", 1},
	"ci":        {"nvidia.com/gpu", "gpu-ci", 1},
}

func handleMutate(w http.ResponseWriter, r *http.Request) {
	var review admissionv1.AdmissionReview
	if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
		http.Error(w, fmt.Sprintf("decode admission review: %v", err), http.StatusBadRequest)
		return
	}
	if review.Request == nil {
		http.Error(w, "nil admission request", http.StatusBadRequest)
		return
	}

	var pod corev1.Pod
	if err := json.Unmarshal(review.Request.Object.Raw, &pod); err != nil {
		http.Error(w, fmt.Sprintf("decode pod: %v", err), http.StatusBadRequest)
		return
	}

	patch, err := buildPatch(&pod)
	if err != nil {
		log.Printf("gpu-tier: skipping pod %s/%s: %v", pod.Namespace, pod.Name, err)
		respond(w, review.Request.UID, nil) // fail-open: never block scheduling on webhook bugs
		return
	}
	respond(w, review.Request.UID, patch)
}

func buildPatch(pod *corev1.Pod) ([]byte, error) {
	tierName, ok := pod.Labels["platform.example.com/gpu-tier"]
	if !ok {
		return nil, fmt.Errorf("no gpu-tier label")
	}
	t, ok := tierMap[tierName]
	if !ok {
		return nil, fmt.Errorf("unknown gpu-tier %q", tierName)
	}

	patches := []map[string]any{}
	for _, c := range pod.Spec.Containers {
		if _, wants := c.Resources.Limits[corev1.ResourceName("nvidia.com/gpu")]; !wants {
			continue
		}
		patches = append(patches,
			map[string]any{"op": "remove", "path": fmt.Sprintf("/spec/containers/%d/resources/limits/nvidia.com~1gpu", indexOf(pod, c.Name))},
			map[string]any{"op": "add", "path": fmt.Sprintf("/spec/containers/%d/resources/limits/%s", indexOf(pod, c.Name), escape(t.Resource)), "value": resource.MustParse("1").String()},
		)
	}

	patches = append(patches, map[string]any{
		"op": "add", "path": "/spec/nodeSelector/platform.example.com~1gpu-pool", "value": t.NodePool,
	})

	return json.Marshal(patches)
}

func respond(w http.ResponseWriter, uid types.UID, patch []byte) {
	resp := admissionv1.AdmissionReview{
		TypeMeta: metav1.TypeMeta{APIVersion: "admission.k8s.io/v1", Kind: "AdmissionReview"},
		Response: &admissionv1.AdmissionResponse{UID: uid, Allowed: true},
	}
	if patch != nil {
		resp.Response.PatchType = ptr(admissionv1.PatchTypeJSONPatch)
		resp.Response.Patch = patch
	}
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(resp); err != nil {
		log.Printf("gpu-tier: encode response: %v", err)
	}
}

Note the fail-open design: a webhook outage must never block pod creation. Return Allowed: true on any internal error and alert on the log line.

FinOps Instrumentation: Measuring Dollars, Not Just Cycles

Utilization is a proxy. The metric your CFO cares about is cost per successful unit of work. Build it with DCGM + OpenCost and a recording rule that attributes GPU-seconds to namespace and tier.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gpu-finops
  namespace: monitoring
spec:
  groups:
    - name: gpu.finops
      interval: 60s
      rules:
        - record: gpu:sm_active:avg_1h
          expr: |
            avg by (node, gpu, Hostname) (
              DCGM_FI_PROF_SM_ACTIVE{gpu=~".+"}
            )

        - record: gpu:cost_per_hour:by_namespace
          expr: |
            sum by (namespace, gpu_class) (
              kube_pod_container_resource_requests{resource=~"nvidia\\.com/.*"}
              * on (pod, namespace) group_left()
              label_replace(kube_pod_labels, "gpu_class", "$1", "label_platform_example_com_gpu_tier", "(.*)")
            ) * on (gpu_class) group_left() gpu_class_hourly_rate

        - alert: GPUStrandedCapacity
          expr: |
            (gpu:sm_active:avg_1h < 0.15)
            and on (node) (kube_node_status_allocated_gpu > 0)
          for: 6h
          labels: {severity: warning, team: platform}
          annotations:
            summary: "Node {{ $labels.node }} allocated but idle — reclaim or downscale"

Feed gpu:cost_per_hour:by_namespace into chargeback. Within one quarter, teams that can see their own dollar burn reduce it 15–25% voluntarily — that is often the cheapest 5% of your savings.

Pitfalls That Will Cost You Real Money

1. Trusting `DCGM_FI_DEV_GPU_UTIL`. It is a sampling counter that reports "a kernel was resident," not "SMs were busy." A memcpy-bound job reports 100% while using 3% of compute. Always use DCGM_FI_PROF_SM_ACTIVE for FinOps.
2. Time-slicing without application queue metrics. Covered above; it silently converts capacity scarcity into unmeasurable latency.
3. MIG fragmentation in the Cluster Autoscaler. A node with three 1g.10gb slices free cannot satisfy a 3g.40gb request. Configure --expander=least-waste and align node pools to profile so CA actually helps.
4. Mixing MIG and MPS on one device. Unsupported. MPS requires exclusive device access.
5. `migStrategy: mixed` plus time-slicing. These are mutually exclusive — the device plugin rejects the combination.
6. Forgetting MIG is static. Reconfiguration requires draining the node. Set profiles at node-pool provisioning time, not on demand.
7. Under-provisioning CI with MPS pinned memory. CUDA_MPS_PINNED_DEVICE_MEM_LIMIT failures surface as opaque CUDA_ERROR_OUT_OF_MEMORY; add it to your docs.
8. No preemption policy. Use PriorityClasses so pre-training jobs preempt notebook pods on Tier 0 nodes rather than queueing behind them.

The 60% Math

Against a $6.87M equivalent whole-device baseline for the same throughput, that is a 60.1% reduction — roughly $4.1M/year — with zero SLO violations across the 8-week production validation window.

Operational Checklist Before You Ship

• Baseline DCGM_FI_PROF_SM_ACTIVE for 14 days before changing anything. You cannot prove savings without a baseline.
• Classify every GPU workload into a tier; block unclassified pods with a validating webhook (Deny on missing platform.example.com/gpu-tier).
• Provision MIG node pools at creation time with an explicit mig.config label per pool.
• Enable XID error alerting (DCGM_FI_DEV_XID_ERRORS > 0) routed to node-level auto-remediation.
• Configure PriorityClass ladder: pre-training > inference > fine-tune > dev > CI.
• Emit application-level GPU queue depth as a custom metric and wire it to HPA/KEDA.
• Ship namespace-level dollar chargeback to engineering leadership before you enforce quotas.
• Validate Cluster Autoscaler behavior with fragmented MIG capacity in a staging pool.
• Document the MIG reconfiguration runbook (cordon → drain → relabel → verify nvidia-smi mig -lgip).
• Set a 90-day review cadence — profile mix drifts, and a tier map that was right in Q1 is often wrong by Q3.

Looking Forward: DRA

Kubernetes 1.32 graduated Dynamic Resource Allocation to GA, and NVIDIA's DRA driver exposes ComputeDomain and GPU attributes (memory, SM count, NVLink topology) as a structured ResourceClaim. DRA will eventually let the scheduler express "I need ≥20GB HBM and same-NVLink-domain as this other claim" — something no current sharing mechanism can do. Design your admission layer so the resource name is an implementation detail you can swap; do not hardcode nvidia.com/gpu into 4,000 Helm charts.

Key Takeaways

• Time-slicing maximizes raw utilization but hides queueing from Kubernetes — use only for latency-insensitive, trust-boundary-free workloads.
• MIG is the only mechanism that provides real memory, fault, and QoS isolation, and the only one the kube-scheduler can reason about correctly. It is the default choice for multi-tenant production.
• MPS wins for high-churn, tiny-job workloads where context-switch overhead dominates.
• The savings are not in picking a winner — they are in tiering: matching each workload class to the cheapest mechanism that satisfies its actual isolation and latency contract.

Sources & Technical References

1. NVIDIA, Multi-Instance GPU (MIG) User Guide, CUDA Toolkit Documentation — https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
2. NVIDIA, Multi-Process Service (MPS) Documentation — https://docs.nvidia.com/deploy/mps/index.html
3. NVIDIA, GPU Operator Documentation: GPU Sharing (Time-Slicing, MIG, MPS) — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
4. NVIDIA, Kubernetes Device Plugin for NVIDIA GPUs (GitHub: NVIDIA/k8s-device-plugin) — https://github.com/NVIDIA/k8s-device-plugin
5. NVIDIA, mig-parted: MIG Partition Editor for GPUs — https://github.com/NVIDIA/mig-parted
6. NVIDIA, DCGM Exporter Metrics Reference — https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/default-counters.csv
7. NVIDIA, A100 Tensor Core GPU Architecture Whitepaper (GA100 die, SM and HBM slice topology) — https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf
8. Kubernetes SIG Node, Dynamic Resource Allocation (KEP-4381), kubernetes/enhancements — https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4381-dra-structured-parameters
9. Kubernetes Documentation, Scheduling Framework & Extension Points — https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/
10. Kubernetes Documentation, Node Resources Fit & Node Declared Features — https://kubernetes.io/docs/concepts/scheduling-eviction/resource-bin-packing/
11. OpenCost, Specification: GPU Cost Allocation — https://opencost.io/docs/specification
12. CNCF, FinOps for Kubernetes: OpenCost Whitepaper — https://www.cncf.io/blog/2023/06/14/opencost-graduates/
13. FinOps Foundation, FinOps Framework: Allocate — Unit Economics — https://www.finops.org/framework/capabilities/unit-economics/
14. Li, H. et al., Multi-Tenant GPU Clusters for Deep Learning Workloads (USENIX ATC), on interference and fragmentation in shared GPU scheduling.
15. Weng, Q. et al., MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters, USENIX NSDI 2022 — https://www.usenix.org/conference/nsdi22/presentation/weng