Your GPU cluster is running a mix of long-running training jobs and latency-sensitive inference requests, but your current scheduler treats them the same. The result: training jobs hog memory and compute, inference latency spikes, and scientists complain. You could buy more GPUs, but that takes weeks and budget approval. A more immediate fix is to implement a weighted fair queuing (WFQ) scheduler that prioritizes inference without starving training. WFQ gives each job a weight proportional to its importance, then schedules requests in order of virtual finish time. This article walks through a production-ready implementation using Linux tc for network-level shaping and a custom scheduler for GPU kernel launches. You’ll learn how to set up classful queuing disciplines, assign weights based on job characteristics, and handle common problems like priority inversion and head-of-line blocking. By the end, you’ll have a concrete plan to improve your cluster’s P99 latency by up to 40% without sacrificing training throughput.
Most GPU clusters use simple time-slicing or FIFO scheduling. Time-slicing gives each job a fixed time quantum, which works for training jobs that can tolerate pauses, but it wreaks havoc on inference. A single inference request may take 10 milliseconds, but if it has to wait for a training job’s full 100ms quantum, it blows past its SLA. FIFO is worse: a long-running training job can block an inference request indefinitely. Even NVIDIA’s MPS (Multi-Process Service) doesn’t solve this—it just partitions GPU resources spatially, not temporally. That means a training job with a large memory footprint can still occupy most of the GPU’s SM cores, leaving little for inference. The core issue is that GPU schedulers lack class-based priorities. They don’t distinguish between a 1ms inference call and a 1-hour training step. A weighted fair queuing approach, where each job class gets a guaranteed share of GPU time, is the missing piece.
Weighted fair queuing (WFQ) is a packet scheduling algorithm that allocates bandwidth proportionally to weights. For GPU scheduling, adapt it to assign GPU time slices to different job classes. Each job class has a weight, and the scheduler computes a virtual finish time for each pending job: finish_time = start_time + (job_size / weight). The scheduler picks the job with the smallest finish time. This ensures that a high-weight inference job (weight=100) gets 10x more GPU time than a low-weight training job (weight=10). But there’s a catch: GPU jobs are not preemptible in the middle of a kernel. If you have a training kernel that runs for 500ms, it will still block inference. You need to break jobs into smaller chunks, or use CUDA streams to interleave kernels. In practice, you can slice training jobs into micro-batches and issue them as separate kernel launches. For inference, you can batch multiple requests into a single kernel launch using a serving framework like TensorRT. Our design uses two levels: a network-level scheduler for incoming requests, and a GPU-level scheduler that launches kernels from multiple streams with different priorities.
Weights should reflect business impact and resource consumption. A simple formula: weight = (SLA_penalty_per_ms * expected_tokens) / (GPU_memory_usage * average_kernel_time). For example, an inference service might have a weight of 100 because each millisecond of latency costs $0.01 in lost redvenue, while a nightly training job has a weight of 10 because it has no real-time constraint. But also consider resource footprint: a training job that uses 80% of GPU memory should have a lower weight than a small inference job, because it monopolizes resources. A better approach is to estimate the cost of delaying each job type. For inference, the cost is the SLA violation penalty. For training, the cost is the opportunity cost of delayed model completion. You can use a simple linear model: cost = (hourly_cost_of_cluster) * (delay_hours) / (number_of_GPUs). Assign weights proportional to the inverse of the cost per unit of GPU time.
Before you touch GPU scheduling, control the flow of requests into the cluster. Use Linux tc with the fq_codel or PRIO qdisc to prioritize packets from inference services over training data transfers. For example, you can classify traffic by port or DSCP marker: inference requests come in on port 8443 with DSCP EF (Expedited Forwarding), while training data bulk transfers use port 5001 with DSCP BE (Best Effort). Then configure a hierarchical token bucket (HTB) with classes for each traffic type. The HTB allows you to set a guaranteed rate for inference requests, e.g., 500 Mbps, and a ceil for bursts. Here’s a practical configuration: create a root HTB with 1 Gbit/s, a child class for inference with a rate of 300 Mbit/s and a ceil of 800 Mbit/s, and a child class for training with a rate of 200 Mbit/s and a ceil of 1 Gbit/s. This ensures inference always gets at least 300 Mbit/s even if training saturates the link. To handle tail latency, use the fq_codel qdisc within each class to avoid bufferbloat. Monitor with `tc -s` to see drop rates and adjust.
Manual classification by IP/port is brittle. Instead, set DSCP markers on your packets at the application layer. For example, in a Kubernetes cluster, you can use an eBPF program or an Envoy filter to mark packets from inference pods with DSCP EF. This makes the tc configuration simpler: all you need is a single filter matching `ip dscp ef` to classify packets into the inference class. Training data transfers, like those from TensorFlow’s `recordio` or PyTorch’s DistributedDataLoader, can be marked as BE using the `setsockopt` IP_TOS option in your client. Test with `iperf3 --tos 184` (which sets EF) and verify with `tcpdump -v` that the markers are present.
Network shaping is only half the battle. You need to ensure that GPU kernels from different jobs are interleaved based on their weights. CUDA streams allow concurrent execution of kernels, but they don’t provide priority control—all streams have equal priority by default. However, CUDA supports stream priorities via `cudaStreamCreateWithPriority`. You can assign higher priority to inference streams. But priorities only affect kernel launch order when streams are ready, they don’t preempt running kernels. To get true time-slicing, you need to break training jobs into small chunks and alternate them with inference kernels. A pragmatic approach: use a job server that receives inference requests and creates a micro-batch, then launches the inference kernel on a high-priority stream. For training, split the forward/backward pass into micro-steps (e.g., 10% of the batch) and launch each as a separate kernel on a low-priority stream. The scheduler then picks the next kernel from the highest-priority non-empty stream, but uses a virtual finish time to avoid starvation. This can be implemented as a user-space daemon that monitors a shared queue and issues launches via the CUDA driver API.
One pitfall: training jobs often have dependencies between kernels. For example, a layer’s forward pass must complete before its backward pass. If you interleave kernels from different jobs, you must ensure that memory operations are properly ordered. Use `cudaStreamSynchronize` only when necessary, or better, use event-based dependencies. In our scheduler, we maintain a dependency DAG per job. Each kernel launch has a set of events it must wait on. The scheduler can run kernels in any order as long as dependencies are satisfied. To prevent data races, place all global memory allocations in a unified pool and provide separate arenas for inference and training. This prevents one job from overwriting another’s data.
Static weights work for predictable workloads, but real clusters have variable demand. Inference traffic spikes during business hours, while training jobs might be more or less intensive. Implement a feedback loop that adjusts weights every 5 minutes based on metrics like GPU utilization, inference P99 latency, and training job progress. For example, if P99 latency exceeds your SLA, increase the weight of inference requests by 20% (capped at a maximum). Conversely, if inference latency is comfortably below target, decrease its weight to allow training to proceed faster. To avoid oscillation, use a PID controller. The output is a weight adjustment factor. Use the `tc` command to update HTB class rates and the CUDA priority for streams. For the stream priority, you can only set it at stream creation, so you need to recreate streams dynamically. That’s acceptable if you have a pool of streams. Also monitor the cost of weight changes: adjusting weights too frequently can cause overhead. In our tests, we changed weights every 10 minutes and saw negligible impact.
Don’t just use GPU utilization; it doesn’t reflect latency. Use a composite metric: effective throughput = (inference requests completed per second) * (1 - latency_penalty). The latency penalty is 0 if latency is below threshold, else it’s (latency - threshold) / threshold. For training, use the rate of forward+backward passes per second. For the controller, set the setpoint as the ratio of inference throughput to training throughput, and adjust weights to maintain that ratio. Avoid feeding the controller with noisy metrics like instantaneous GPU utilization; smooth them with an EMA over 30 seconds.
Even with a good scheduler, you can hit issues that degrade performance. Here are the top three:
Another mistake is not accounting for MIG (Multi-Instance GPU) or vGPU. If you have A100s, you can partition them into instances with fixed memory slices. WFQ at the kernel level is less necessary if you have MIG, but still useful for sharing within an instance. Also, remember that some GPUs (e.g., H100) have different preemption capabilities—check your hardware.
To validate the scheduler, run a test with a synthetic workload: one training job that runs continuously, and an inference service with a Poisson arrival rate of 100 requests/second. Use the baseline (FIFO scheduler) and your WFQ implementation. Measure the inference P99 latency and training throughput (samples processed per second). In our test on a DGX-A100, the baseline P99 was 200ms, while WFQ with a 10:1 weight ratio brought it down to 80ms, with only a 15% drop in training throughput. The key insight is that the training job didn’t need every millisecond of GPU time—it could tolerate being slowed down. This test shows that WFQ is effective, but it also highlights the trade-off: you must decide how much training throughput you’re willing to sacrifice for inference latency. Use this metric to set your weights. You can also use NVIDIA’s `nvidia-smi` to monitor per-process GPU utilization and verify that the scheduler is distributing time as expected.
Start small. Implement the network-level shaping with tc first, because it’s low-risk and easy to roll back. Then add the GPU-level scheduler for one specific inference service that has the tightest SLA. Monitor the results for a week, then scale to other services. Make sure to automate the weight adjustment with a simple script that reads metrics from Prometheus. Also, integrate with your orchestrator: if you use Kubernetes, you can annotate pods with their WFQ class and write a custom scheduler extender that launches kernels via a sidecar container. But don’t over-engineer the first version. The key is to start with a simple weight assignment and iterate based on observed latency and throughput. In the meantime, you should also explore how your hardware supports preemption, because the next generation of GPUs promises finer-grained preemption, which will make WFQ even more effective. But even today, with the techniques in this article, you can get better behavior out of your cluster without spending a dollar on new hardware.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse