Every AI hardware vendor has a benchmark slide that shows jaw-dropping teraflops or tokens-per-second. The footnote, buried in fine print, reveals the unspoken truth: those numbers were achieved at maximum power draw, with thermal headroom few data centers can afford. A chip that hits 2000 TFLOPS at 700 watts is not the same as one that hits 1500 TFLOPS at 350 watts — and in a real deployment, the power cap often dictates the real throughput. This article breaks down why power-aware benchmarking is the only honest way to compare hardware for AI workloads in 2025, and offers practical methods for measuring it yourself.
Peak performance numbers are measured in a burst — typically for a few seconds or minutes, long enough to complete a single batch of matrix multiplications before the silicon temperature becomes a limiting factor. Once the chip heats up to its thermal junction limit (typically 85–105°C), the clock speed gets throttled. The result is that a chip sustaining 80% of its peak throughput for 24 hours is often more valuable than a chip sustaining 50% of its peak after the first hour.
The concept you need to measure is sustained performance under thermal steady state. If you are evaluating an NVIDIA H100 vs. a custom ASIC, run your actual inference workload for at least 3 hours and record throughput at 15-minute intervals. You will see a curve, not a line. Where that curve flattens is the number you should use for capacity planning. Some cloud providers, like CoreWeave, now publish thermal profiles for their rented GPUs, but most on-prem evaluation requires you to do this yourself.
Power capping (also called power limiting) is the practice of setting a maximum wattage that a chip can draw, typically via a driver-level setting. NVIDIA GPUs support nvidia-smi -pl to set a power limit between the minimum and maximum for the board. AMD offers similar controls via rocm-smi. Most ML engineers never touch these settings, which is a mistake.
A 2024 analysis by the MLCommons Science Working Group observed that capping an A100 from 400W to 300W only reduced throughput on BERT-style language model inference by roughly 7%. The energy efficiency (inferences per watt) improved by over 28%. For a 100-GPU cluster running 24/7, that means a 25 kW reduction in power draw per rack, which translates to roughly 18 tonnes of CO2 avoided per month in a standard US grid mix (based on the EPA’s 0.85 lbs/kWh average). The trade-off is that you need more GPUs to hit a given target throughput, but the total cost of ownership — including electricity, cooling, and cabinet space — often nets out in favour of lower power caps.
Practical tip: train your models with the power cap set to 80% of the chip’s specification. Use the final 20% headroom only for short, latency-sensitive bursts like interactive inference requests.
MLPerf is the industry-standard benchmark suite for AI training and inference. Yet its power measurement methodology has known blind spots. In the MLPerf Inference v4.1 results (published mid-2024), several vendors reported energy consumption separately from performance, but the “time to benchmark completion” is not always public. The key anomaly: the benchmark allows for cooling to be off or the ambient temperature to be lower than typical data center conditions. A vendor submitting results at 18°C ambient will show better power efficiency than one at 25°C, but the real deployment will likely run at 25°C or higher.
Another issue is the power unplugged phenomenon. Some submissions include batttery or capacitor-based “power shelters” that smooth out transient spikes, allowing boards to exceed their thermal design power (TDP) for short bursts without thermal shutdown. This is legal within benchmark rules, but it is not replicable in a standard rack with a 15A/208V circuit.
When you read a claim like “X accelerator achieves 5,000 inferences per second,” ask three questions: (1) at what sustained power draw? (2) at what ambient temperature? (3) how long did the run take? If a vendor cannot answer those three numbers with precision, treat the benchmark claim as a marketing figure, not an engineering specification.
You do not need a $200,000 thermal chamber to get meaningful data. A power-metered PDU (power distribution unit) at the rack level, plus a temperature sensor at the intake vent, will cover 80% of what you need. For a single node, you can also use a watt meter with a high sampling rate (e.g., the Watts Up? .NET meter at 1 Hz sampling) or use the chip’s internal energy counters via NVIDIA’s nvmlDeviceGetTotalEnergyConsumption API.
Your test should follow a simple protocol: (1) warm up the hardware for 20 minutes with a dummy workload to reach thermal equilibrium; (2) run your production inference or training script for 1 hour while logging throughput, power, and temperature every 5 seconds; (3) plot power against throughput over time; (4) compute the median and 95th percentile for power draw, and the harmonic mean of throughput (to give lower-weight to bursty highs).
One subtlety worth capturing is memory power separate from compute power. Some accelerators, especially those with high-bandwidth memory (HBM) like the AMD Instinct MI300X, draw a significant fraction of power just to keep the memory refreshed, even when idle. An idle H100 typically draws 50–70W just for HBM refresh. If your workload is memory-bound (large embeddings, long context windows), this idle power becomes a major cost driver. Your harness should measure idle power and active power separately to model total cost accurately.
TDP is the amount of cooling the chip needs to operate at base clock without tripping overheating protection. It is not the maximum power draw, and it is not the power draw under all workloads. In practice, TDP is a specification for the cooling system designer, not for the capacity planner. For instance, an Intel Xeon 8592+ has a TDP of 350W, but a fully-loaded AVX-512 workload can transiently draw 450W for a few milliseconds. If you size your rack circuit based on TDP instead of peak power, your circuit breakers will trip.
The more relevant metric is steady-state thermal power (SSP), which is the sustained power draw you see after 30 minutes of continuous, heavy load. For most commercial accelerators, SSP is 85–95% of TDP. But for a small subset of designs, primarily those with aggressive boost algorithms (e.g., Intel’s Thermal Velocity Boost on some Xeon parts), SSP can be as high as 110% of TDP at low ambient temperatures.
Design your data center power budget using SSP plus a 20% safety margin, not TDP. This buffer covers ambient temperature swings, aging of thermal paste, and the occasional spike caused by mixed workloads hitting multiple compute units simultaneously.
In late 2024, a financial services client ran a benchmark of two candidate inference servers for their low-latency fraud detection model. Server A used a single NVIDIA L40S (350W TDP), server B used two Intel Gaudi 2 accelerators (each 500W TDP). The vendor benchmark sheets suggested Server A had 30% more raw throughput on a ResNet-50 variant.
We set both servers to a 300W node-level power cap (the client’s rack budget). Server A sustained 1,200 inferences per second; Server B sustained 1,950. Server B was genuinely slower on paper, but its architecture was better at maintaining performance under the power cap. The lesson: raw chip performance does not linearly translate to performance under a capped power envelope.
In production, Server B handled 1.35x the request volume with the same power consumption. This forced a complete re-evaluation of the procurement criteria — moving weight from “peak ops” to “ops per watt under a fixed cap,” which is the only metric that pays your electricity bill.
Vendors love to advertise efficiency as TFLOPS per watt, but this ratio is workload-dependent and usually quoted for a single precision format (often FP16 with sparse tensor cores enabled). A chip may deliver 2x the FP16 TFLOPS per watt of a competitor, but only 1.1x the FP8 or INT8 efficiency. Since most production AI pipelines use mixed precision — FP32 for some layers, BF16 for others, INT8 for quantization — the single-number claim is only partially relevant.
To avoid this trap, build a composite efficiency metric for your specific pipeline: sum the power-drawn per layer type, then divide by the total number of inferences per hour. Multiply by your electricity cost to get real operational cost per inference. Only this bottom-up number should drive hardware purchases. A lower efficiency number at the chip level can still beat a higher one once you account for actual precision mixes.
A practical way to approximate this without running full pipeline benchmarks: profile your model with PyTorch’s torch.profiler to get per-operator time, then use vendor power models (e.g., NVIDIA’s Power Efficiency Estimator) to sum up expected energy per operator. It will not be perfect, but it will be better than trusting a single marketing number.
The next time you evaluate AI hardware, start by asking for three documents: the power-cap sweep curve (performance vs. wattage), the thermal step response (how quickly performance degrades after a cold start), and the energy consumption by precision type. If a vendor cannot provide those, ask for a trial unit and run your test harness. In 2025, the difference between a good accelerator system and a great one is not measured in teraflops — it is measured in what keeps running after the breaks are slammed on and the heat rises.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse