Most AI engineering teams assume that Kubernetes with auto-scaling is the default path to high GPU utilization. That assumption costs some organizations 25–40% of their compute budget. Meanwhile, hyperscalers like Google, Amazon, and Microsoft have quietly built custom, manually tuned schedulers that squeeze 85–95% utilization out of their GPU fleets. The gap isn't about magic — it's about fundamental design choices in scheduling algorithms, bin-packing, and preemption models.
This comparison dissects how each approach handles the three most painful GPU scheduling problems: memory fragmentation, gang scheduling for multi-GPU training, and tail-latency spikes during inference. By the end, you'll know which model fits your infrastructure scale — and whether you can bridge the gap without building your own scheduler from scratch.
Hyperscalers treat GPU scheduling as a first-class systems design problem, not an extension of CPU scheduling. Their schedulers — Borg at Google, Tupperware at Meta, and internal variants at Microsoft — embed explicit knowledge of GPU topology into every scheduling decision.
Custom hyperscaler schedulers accept a YAML-like spec that declares the exact adjacency requirements: which GPU instances must share an NVLink bridge, which CPU socket minimizes cross-NUMA memory access, and which network adapter provides the highest bandwidth to the parameter server. When a training job requests 32 H100 GPUs, the scheduler computes a feasible placement offline using a graph-matching algorithm that respects these constraints. The decision takes 200–500 milliseconds but runs only once per job launch.
Instead of simple priority queues, these schedulers use preemption strategies borrowed from queuing theory. When a high-priority inference job arrives, the scheduler does not randomly kill pods. It selects the lowest-value training task whose resource consumption best fits the incoming job's requirements — a technique called reservoir sampling over resource footprints. Microsoft's internal scheduler reduced inference tail latency by 62% using this method while keeping training throughput loss under 7%.
Kubernetes was designed for stateless microservices, not GPU-intensive training where a single pod consumes half a node's memory bandwidth. The standard approach — a combination of the Cluster Autoscaler (CA) and the Kubernetes Scheduler — works adequately for inference but starts breaking under training workloads.
The default Kubernetes scheduler uses a scoring algorithm that favors nodes with the most available resources. On a heterogeneous cluster with H100s, A100s, and A10s, this quickly leads to fragmentation: small inference pods accumulate on the best GPUs, blocking training jobs that need contiguous H100 pods. The gpu-operator from NVIDIA helps by labeling nodes with GPU model and memory, but it does not solve the topology-aware bin-packing problem.
CA scales node pools up or down based on pending pods. For batch training, this works — a pending pod triggers node addition, and the job starts 60–90 seconds later. But for inference with a 10 ms SLO, that delay is unacceptable. Organizations running production LLM inference on K8s often pre-warm node pools using a custom controller that predicts demand from API metrics. The open-source Karpenter project improves on CA by allowing in-flight bin-packing across instance types, but it still lacks GPU topology awareness.
Multi-GPU training requires all pods of a distributed job to start simultaneously — gang scheduling. The Volcano project adds this to K8s, but its implementation uses a simple queue-and-reserve pattern that can deadlock under high contention. Koordinator (from Alibaba) provides finer-grained colocation, but both tools rely on heuristic timeouts rather than constraint solving. Real-world tests show Volcano gang scheduling loses 8–12% utilization compared to a custom scheduler during high-contention periods because it reserves resources too conservatively.
The numbers come from a 2024 study published by the EuroSys conference, which compared scheduler performance across three clusters running LLM training and recommendation inference.
| Metric | Hyperscaler Custom Scheduler | K8s (CA + Volcano) |
|---|---|---|
| Average GPU utilization (training) | 89% | 67% |
| Peak utilization (inference) | 94% | 82% |
| P99 inference tail latency (ms) | 22 | 41 |
| Job scheduling delay (training) | ~350 ms | ~4 s |
| Fragmentation overhead | 4% | 18% |
The fragmentation overhead difference is the most instructive. K8s leaves small GPU memory gaps — say 4 GB on an 80 GB H100 — that cannot fit any pending job. The custom schedulers actively defragment by migrating small inference loads to a shared inference-only pool, a feature entirely absent from vanilla K8s.
Building a custom GPU scheduler is not cheap. The hyperscaler teams behind these systems average 8–12 engineers dedicated to scheduler development and maintenance. For most organizations, that investment is only justified above 500 GPUs in a single cluster.
Several organizations have built hybrid schedulers that use K8s for lifecycle management while offloading placement decisions to a custom orchestrator. The pattern involves three components:
A dedicated service, written in Go or Rust, maintains a live graph of GPU topology and job requirements. When a user submits a job to K8s, a mutating webhook intercepts the pod spec and adds node affinity rules that encode the placement decision. This keeps the K8s scheduler as a simple executor while the placement service handles the constraint solving. ByteDance uses this pattern and reports 82% utilization — closer to custom schedulers — without forking the K8s codebase.
For inference workloads, deploy a separate controller that reads historical API request patterns and provisions GPU nodes 2 minutes before expected traffic spikes. The popular open-source Keda project supports this via Prometheus metrics, but its scaling interval is too coarse for sub-second spikes. A custom prediction model using a lightweight LSTM on a 30-second window reduces cold-start latency by 70% in production tests.
Start by measuring your current GPU fragmentation. Run nvidia-smi pmon or use the NVIDIA DCGM exporter in Prometheus to log per-GPU memory allocation over 48 hours. If your average allocation per GPU drops below 60% of available memory, standard K8s scheduling is leaving money on the table. In that case, implement the hybrid pattern described above: a placement service that enforces topology constraints through K8s node affinity, plus a predictive pre-warmer for inference. That two-step adjustment typically recovers 10–15% utilization within a week, without requiring a full scheduler rewrite.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse