In 2025, AI workloads are no longer single-GPU experiments. They are sprawling, distributed systems pulling data from object stores, shuttling tensors between accelerators, and coordinating thousands of parallel workers. Yet most Kubernetes schedulers still treat the cluster as a flat pool of resources. That disconnect is silently costing teams 20–40% of their training throughput and inflating inference latency. The root cause: topology ignorance. When a scheduler ignores where data physically resides, which NUMA node a GPU sits on, or how many hops a network packet must travel, every job pays for it. This guide explains why topology-aware scheduling is now a mandatory design pattern for production AI—and how to implement it without rewriting your entire stack.
Non-Uniform Memory Access (NUMA) is the quiet multiplier in AI performance. On a dual-socket server, each CPU has its own memory controller and its own PCIe lanes directly attached to GPUs and NICs. When a process runs on Socket 0 but allocates memory on Socket 1, every memory access traverses the interconnect (Intel UPI or AMD Infinity Fabric), adding 150–300 nanoseconds of latency. For a single-threaded benchmark, that is noise. For a distributed training step iterating over millions of parameters, it becomes a hidden bottleneck.
Consider an example: a PyTorch job using NCCL (NVIDIA Collective Communications Library) on an 8-GPU node. NCCL uses shared memory for intra-node communication. If the scheduler assigns containers to CPUs on Socket 0 while half the GPUs sit on Socket 1, the shared memory buffers are allocated on the wrong socket. The result: inter-GPU communication bypasses the fast path, dropping PCIe bandwidth from 64 GB/s (per socket) to 32 GB/s (across sockets). In a 100-Gbps cluster, that mismatch can reduce training throughput by 30%.
The fix is not exotic. Modern Linux kernels expose NUMA topology via lscpu and numactl, and the OpenMPI community has long used `–map-by socket` to pin processes. For Kubernetes, a topology-aware scheduler can read the node's NUMA layout and ensure that a pod's CPU, memory, and GPU are co-located on the same socket. The Topology Manager in kubelet is a step in that direction, but its default policy—`best-effort`—rarely guarantees strict alignment. In 2025, production clusters need the `Single-NUMA-Node` or `Multi-NUMA-Node` policy combined with a custom scheduler that understands accelerator affinity. The performance gap is too large to ignore.
Networking is the new bottleneck in distributed AI. Every gradient synchronization in data-parallel training sends all-reduce traffic across the network. On a standard Mellanox ConnectX-6 NIC delivering 200 Gbps, the physical bandwidth is ample—but only if the NIC is directly connected to the GPU's PCIe switch. Most servers route NIC traffic through the CPU's PCIe root complex, which adds latency and consumes CPU cycles for packet processing.
Topology-aware scheduling solves this by matching GPU placement with NIC location. For example, NVIDIA's DGX A100 server has eight GPUs and eight NVIDIA ConnectX-7 NICs, one per GPU. The optimal configuration places each GPU in the same NUMA domain as its paired NIC, allowing GPUDirect RDMA (Remote Direct Memory Access) to copy tensors directly from GPU memory to the NIC without CPU involvement. But a naive scheduler might place the GPU on Socket 0 and the NIC on Socket 1, defeating the purpose. With GPUDirect RDMA, the latency of a 512 MB all-reduce drops from 1.1 ms (CPU-staged) to 0.4 ms (direct), as shown in Mellanox's official benchmarks. Across hundreds of iterations, that 0.7 ms saving shaves minutes off training runs.
A practical checklist for GPU/NIC affinity:
AI training is data-centric, yet most schedulers treat data as an external dependency. When a training pod is scheduled on a node that does not host its dataset, the data loader pulls from a remote object store over the network. A 100 GB dataset (common for image classification at scale) transferred over a 10 Gbps link takes 80 seconds. A good scheduler with data locality can pre-place that dataset on a node's NVMe drive during the container image pull phase, cutting load time to under 5 seconds.
In 2025, tools like Kubernetes' `podtopologyspread` and the open-source Koordinator project include data-aware scheduling policies. They track which nodes have cached copies of specific datasets, and the scheduler uses this metadata to place pods on those nodes. For example, a training job for a recommendation engine might use a feature store with a cached version on node A and node B. The scheduler should prefer node A because it also has a free GPU of the required model type, whereas node B's GPU is occupied. This is not trivial, but it is implementable: label nodes with dataset identifiers and let the scheduler match node affinity rules.
Another approach is to use a data-loading sidecar that prefetches data during the pulling phase. That sidecar can run on any node, but if the scheduler guarantees it runs on the same node as the training pod, the data is already local when the pod starts. The orchestration becomes a two-stage scheduling problem: first reserve the node, then launch the data loader and the training pod with a localhost channel.
Kubernetes does not natively understand NUMA, PCIe topology, or data locality. To implement topology-aware scheduling, you need to extend the scheduler. There are three common paths, each with trade-offs.
The most flexible route is to write a custom scheduler plugin using the Kubernetes scheduling framework. You implement a `Score` extension point that reads node topology annotations (e.g., `topology.kubernetes.io/zone` and `node.kubernetes.io/nv-topology`) and scores nodes based on how well they match the pod's requirements. For instance, if a pod requests an NVIDIA A100 with GPU index 0, the plugin checks if the node has an annotation like `gpu-index-0-pcie-socket: 0`. The scoring function adds 100 points if the pod's CPU request is on the same socket.
This gives you full control but requires Go programming and continuous maintenance. A simpler alternative is to use the Topology Spread Constraints feature to force pods onto nodes that share a specific label, but that is not dynamic enough for metric-driven scheduling.
For teams without platform engineering bandwidth, static node labels can mimic topology awareness. Label each node with `cpu-socket` and `gpu-pci-switch`. Then, for GPU jobs, set node affinity to match the GPU's socket. This works well for stable cluster configurations but fails when nodes are upgraded or replaced—the labels drift out of sync.
Another emerging practice is to use runtime telemetry to inform future placements. Prometheus scrapes metrics like `node_numa_byte_total` and `process_numa_miss_local` from the node exporter and the kernel. If a scheduler plugin scrapes these metrics, it can prefer nodes where the NUMA miss rate is low, indicating good locality in recent jobs. This is indirect but effective, and it adapts to environmental drift. The drawback is that it is reactive, not proactive: the first batch of jobs might run poorly before the scheduler learns.
Topology-aware scheduling is not just about performance—it is about money. In the public cloud, you pay per vCPU-hour, per GB of memory, and per GB of network transfer. When jobs run faster, you use fewer core-hours, and when they use local data, you consume less network bandwidth. For example, a training job that normally runs 2 hours on 4 nodes with a remote dataset might run 1.4 hours with locality—saving 0.6 hours of compute on 4 nodes, a 30% cost cut. Over a month with hundreds of jobs, that adds up to tens of thousands of dollars.
More importantly, topology-aware scheduling reduces waste on spot instances. When you request spot instances, you often have no control over underlying hardware. A topology-aware scheduler can adapt by selecting a different data replica or by preprocessing model weights to fit the available topology. This flexibility prevents job failures or slowdowns that lead to idle time and retry costs.
However, there is an overhead: the additional CPU and networking resources used to coordinate topology decisions. In practice, that overhead is negligible—a scheduler plugin running on the control plane adds a few hundred milliseconds per scheduling decision, which is under 1% of job execution time. The net benefit is overwhelmingly positive.
To solidify these ideas, consider a realistic scenario from an NLP team training a 7B-parameter transformer on 8 nodes with 8 A100 GPUs each. The cluster is orchestrated by Kubernetes with a custom scheduler plugin enabled. The plugin is configured to enforce GPU/NIC affinity: each GPU must be paired with its local NIC. It also instructs the data loader to cache the tokenized dataset on the node's NVMe drive.
During the job submission, the scheduler filters out nodes with insufficient GPUs or memory. It then scores remaining nodes based on whether the GPU index matches the NIC index. For each candidate node, it checks if the dataset is already locally cached; if not, it adds a pointer to the sidecar that will prefetch the data. Once the job starts, the NCCL communication uses GPUDirect RDMA, and the data loader reads from local disk; the result is a 25% reduction in training wall-clock time compared to baseline (no topology awareness). The team also observes a 12% reduction in network egress costs, as less data is pulled from the object store.
This experiment was run in API version 2.7 of the scheduler, and the plugin code is available as an open-source reference on GitHub. The key takeaway: the change requires no modification to the training code—only scheduling policy.
The ecosystem is moving beyond Kubernetes. Slurm, the HPC scheduler, has long supported topology-aware allocation via `--topology=node`, but it lacks the elasticity of Kubernetes. New frameworks like Singularity and KubeFlow are integrating topology hints into their job specs. On the hardware side, compute express link (CXL) is introducing cache-coherent memory pools that can be shared across sockets; future schedulers will need to understand CXL-attached memory regions and manage them.
For edge AI, topology awareness takes another form: the scheduler must account for energy consumption and thermal headroom, not just latency. A mobile device with a CPU and a neural accelerator will have a specific package layout, and scheduling a vision model to run on the NPU with local memory will extend battery life. The same principles of locality apply, but the metrics change.
In short, topology-aware scheduling is an essential tool for any serious AI infrastructure in 2025. Start by profiling your own workloads—find the bottlenecks that come from placement. Then adopt the strategies discussed here. Your next training run—and your cloud bill—will thank you.
Your first step is to run `nvidia-smi topo -m` on each node type in your cluster and document the NUMA and PCIe topology. Use that map to design scheduling constraints for your highest-priority jobs. Once you see the performance improvements, extend the practice to your entire cluster.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse