When your application starts hitting an LLM API at production scale, the first surprise is often the bill. A single chatbot session might consume hundreds of tokens per turn, and without guardrails, a misbehaving client or a sudden traffic spike can burn through your monthly budget in hours. Rate limiting isn't just about staying within vendor caps — it's about predictable costs, fair resource allocation, and protecting downstream infrastructure from cascading failures. The token bucket algorithm is a well-established approach for smoothing traffic, but applying it to LLM inference requires careful consideration of token-level cost, variable latency, and burst behavior that differs significantly from traditional REST API throttling.
Fixed-window counters reset at arbitrary boundaries (every minute, every hour), which creates a problem: near the end of a window, a client can accumulate unused capacity and then send a massive burst right after reset. Sliding-window logs solve the boundary issue but store every request timestamp, which becomes memory-intensive under high throughput. Token buckets offer a simpler, memory-efficient compromise.
A token bucket holds a configurable number of tokens (the burst capacity). Tokens are added at a steady rate (the refill rate). Each request consumes one or more tokens. If the bucket is empty, the request is either queued or rejected. For LLM inference, you can map one token to one input token in the prompt or to one output token generated, or you can assign a weighted cost based on model size (e.g., GPT-4 consumes 3 tokens per inference token vs. GPT-3.5's 1). This granularity prevents a single request from draining the entire budget.
Unlike fixed-window counters, token buckets naturally handle short bursts: if the bucket is full (e.g., 100 tokens), a client can send 100 tokens worth of prompts instantly, then must wait for refills. This matches the bursty nature of LLM chatbots where a user might paste a long document followed by silence.
The two critical parameters are refill rate (tokens per second) and burst size (maximum token capacity). For external API keys (OpenAI, Anthropic), use the published rate limits: OpenAI's tier 1 allows 5,000 RPM on GPT-4, but each request varies wildly in token consumption. Set your burst size to the maximum tokens allowed per minute (e.g., 300,000 tokens/min for tier 1 GPT-4) and refill rate to burst size / 60. For internal services, start with a refill rate that matches your average expected usage plus 20% headroom, and burst size equal to 2x the refill rate to handle natural spikes.
One nuance: LLM APIs often have separate limits for prompts and completions. OpenAI's documentation specifies tokens-per-minute (TPM) and requests-per-minute (RPM). A token bucket that tracks only RPM ignores the real cost driver. Implement a dual-token bucket: one for request count and one for total token consumption. The request bucket ensures fair queueing; the token bucket prevents cost runaway.
Single-process token buckets work for low-traffic apps, but production AI inference runs on multiple containers behind a load balancer. You need a shared state. Redis is the standard choice because it supports atomic operations like INCR and EXPIRE with sub-millisecond latency. The classic approach uses a sorted set or a simple counter with time-bucketed tokens.
A cleaner pattern for LLM workloads uses Redis's CELL (Generic Cell Rate Algorithm) via the redis-cell module, which implements a token bucket natively. If you can't load the module, implement a Lua script that atomically checks and decrements a token counter. Example Lua skeleton:
local tokens = redis.call("GET", KEYS[1])
if not tokens then
redis.call("SET", KEYS[1], ARGV[1])
tokens = ARGV[1]
end
if tonumber(tokens) >= tonumber(ARGV[2]) then
redis.call("DECRBY", KEYS[1], ARGV[2])
return 1
else
return 0
end
Wrap this with a background refill job that increments tokens every second using EXPIRE to reset the counter on inactivity. For fault tolerance, use Redis Cluster or Sentinel, but note that failover during an INCR can double-count. Accept this tiny risk or implement idempotency keys per request.
Strictly rejecting requests returns HTTP 429 to the client, which may trigger retries that worsen congestion. Instead, implement a virtual waiting room: when the bucket is empty, calculate the estimated wait time based on refill rate and return a Retry-After header. For async workloads (e.g., batch summarization), push the request to a queue with a delay. For real-time chatbots, serve a fallback model (e.g., switch from GPT-4 to GPT-3.5) or a canned response: "I'm currently at capacity. Please retry in a few seconds."
Not all LLM inference costs are equal. GPT-4 pricing is roughly 20–30x higher per token than GPT-3.5-turbo. A token bucket that treats all tokens equally will let a burst of GPT-4 requests drain your budget while blocking cheaper GPT-3.5 requests. Solve this with weighted tokens: define a cost multiplier per model. For instance:
When a request arrives, compute weighted_cost = (input_tokens + output_tokens) * multiplier. Deduct that amount from the token bucket. If the bucket doesn't have enough, reject the request or downgrade model. This ensures your budget allocation reflects dollar cost, not just request volume.
Aggregate token bucket metrics (rejected requests, average token cost per hour) into a monitoring system like Prometheus or Datadog. Set alerts when rejection rate exceeds 5% — that signals either under-provisioned capacity or a client with excessive retry logic. Also track token burn rate: if your bucket is consistently empty, you are running at maximum budget tolerance; consider raising the refill rate or negotiating higher API limits.
LLM inference introduces two burst patterns that differ from regular APIs. First, a single prompt with 128,000 input tokens (e.g., analyzing a whole codebase) could consume your entire hourly budget in one request. Second, streaming responses mean the output tokens arrive over seconds or minutes, making it impossible to know the final cost upfront.
For long-context prompts, perform a pre-check: estimate the prompt cost before sending the request. If it exceeds the bucket's current balance, reject early. For streaming, reserve a token allowance upfront based on the maximum allowed output tokens (e.g., 4,096). As tokens stream back, refund the unused portion. Implementation steps:
prompt_tokens + max_output_tokens from bucket.max_output_tokens - actual_used back to bucket.This prevents a single streaming session from hogging capacity while allowing refunds for short responses. The refund logic must be atomic — use a Redis Lua script that reads the current token count, adds the refund, and sets the new value.
If your platform serves multiple users or integrations, a global token bucket lets one aggressive tenant starve others. Implement hierarchical token buckets: a parent bucket for the total budget (e.g., 1M tokens/min for the whole account), and child buckets per API key with proportional capacities. When a request arrives, first check the child bucket. If the child bucket has tokens, deduct from both child and parent. If the parent bucket is empty, reject even if the child bucket still has tokens (preventing total budget overshoot).
Choose child bucket sizes dynamically: allocate a baseline (e.g., 10% of parent capacity) plus a bonus based on recent usage history. This prevents a new tenant from grabbing excessive share while allowing active tenants to grow. Redis sorted sets with ZINCRBY let you track trailing 1-hour token consumption per key and adjust child bucket refill rates every 5 minutes.
Before deploying, simulate traffic that matches LLM inference patterns: bursty arrivals with Pareto-distributed inter-request times, variable token counts (prompts range from 500 to 50,000 tokens), and streaming vs. non-streaming requests. Tools like locust or k6 can generate such workloads. Verify these behaviors:
A common failure mode: the background refill job runs every second but clock drift across Redis nodes causes over-refilling. Mitigate by storing the last refill timestamp alongside the token count and computing refill based on actual elapsed time since last check.
Token buckets control volume but not concurrency. LLM APIs have limits on simultaneous connections (e.g., OpenAI's 3 concurrent requests for some tiers). A token bucket alone could allow 20 requests to pass if each consumes few tokens, leading to connection-level throttling. Pair the token bucket with a semaphore (e.g., Redis SETNX with TTL) that limits concurrent in-flight requests to your API limit. Deduct tokens only when the request starts, not when it enters the queue, so your capacity reflects actual in-flight cost.
Also set a latency budget: if a request has been queued for longer than, say, 30 seconds, serve a fallback or reject. LLM inference degrades quickly with latency; users abandon chatbots that take more than a few seconds to respond. Track queue wait time as a metric and adjust burst capacity downward if p95 queue time exceeds your SLA.
Now, you have a production-ready token bucket rate limiter tailored to LLM inference. Start by instrumenting a single model endpoint with weighted tokens and the streaming refund mechanism. Deploy it behind a Redis-backed middleware in your API gateway. Monitor rejection rates and token burn daily. Once stable, expand to hierarchical buckets for multi-tenant support. Your credit card will thank you.
Browse the latest reads across all four sections — published daily.
← Back to BestLifePulse