How limits combine
One call to a model route can be refused by five different limits. They are checked in a fixed order, each answers in its own shape, and only one of them sends the X-RateLimit-* headers. This page puts them side by side so a 429 can be read and the right thing retried.
Who can use it: anyone holding an API key. The key's own limit applies to every route; the other four apply to the model routes: chat completions, completions, embeddings, rerank and image generation.
The five limits
| Set by | Scope | What you see when it refuses | |
|---|---|---|---|
| Key rate limit | The key's Requests / min and Requests / day, or the platform default of 60 and 1,000 | One key, on every route | 429 {"code": "RATE_LIMITED"} from the public API gateway, with X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset and X-RateLimit-Window (minute or day) |
| Gateway request limit | The AI Gateway's Rate Limits dialog: requests per minute per user, per tenant and per model (defaults 30, 200 and 100) | All traffic in the tenant | 429 in the OpenAI shape, error.code rate_limit_exceeded, a message naming rpm and the scope; the gateway's Retry-After is not relayed |
| Gateway token limit | The same dialog: tokens per minute per user, tenant and model (defaults 60,000, 500,000 and 200,000) | The same | 429 rate_limit_exceeded naming tpm. Tokens are counted after each answer, so the call that crosses the cap completes and the next one is refused |
| Budget | A budget alert in the Control Room: a spend threshold per day, week or month on the tenant, a user, a team, an agent or a model | The rule's scope | Warn and soft block: the call proceeds, and the gateway's X-Budget-Warning header is not relayed. Hard block: 429 error.code budget_exceeded |
| Token quota | The licence's token cap for the period, in hard or soft mode | The tenant | Hard: 429 token_quota_exceeded. Soft: the call proceeds; X-Token-Quota-Warning is not relayed |
For a chat completion the checks run in this order: the key's limit at the edge; the gateway's request limit for the user and the tenant; the input guardrails; the budget; the token quota; model resolution; then, per candidate model, the per-model request limit and a reservation of the call's worst-case cost against the budget. When two would fail, the earliest wins, so a call over both the rate limit and the budget is refused as rate_limit_exceeded. Embeddings and rerank check the budget first, then the rate limit, then the quota; image generation checks the rate limit, then the budget, and never the quota.
Read a 429
| Body | Header | Which limit, and what to do |
|---|---|---|
{"code": "RATE_LIMITED", "detail": "Rate limit exceeded"} | X-RateLimit-Window | The key's own. Wait for X-RateLimit-Reset, a Unix timestamp in seconds. |
error.code rate_limit_exceeded; the message names rpm or tpm and user, tenant or model | none: the gateway's Retry-After is dropped at the edge | The AI Gateway. Wait for the next clock minute. |
error.code budget_exceeded; the message names the rule and the spend | none | A budget's hard block. Do not retry until the period resets. |
error.code token_quota_exceeded | none | The tenant's token quota. |
error.code all_providers_failed, error_class rate_limit or quota | none: the gateway's Retry-After is dropped at the edge | The model provider's own limit, after every candidate was tried. Wait a few seconds. |
Quick start
A wrapper that reads the headers on success and tells the refusals apart:
- cURL
- Python
- TypeScript
curl -i -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"tier": "fast", "messages": [{"role": "user", "content": "ping"}]}' \
| grep -i -E "^HTTP|^x-ratelimit|^\{"
import time, requests
BASE = "https://<your tenant host>/public/v1"
HEADERS = {"Authorization": "Bearer kapi_..."}
def complete(body):
r = requests.post(f"{BASE}/chat/completions", headers=HEADERS, json=body)
if r.status_code != 429:
r.raise_for_status()
print("key minute allowance left:", r.headers.get("X-RateLimit-Remaining"))
return r.json()
err = r.json()
if err.get("code") == "RATE_LIMITED": # the key's own limit
wait = int(r.headers["X-RateLimit-Reset"]) - int(time.time())
elif err.get("error", {}).get("code") == "rate_limit_exceeded": # the gateway's limit
wait = 60 - int(time.time()) % 60 # a fixed minute window; the gateway's Retry-After is not relayed
else: # budget or quota: do not loop
raise RuntimeError(err["error"]["message"])
time.sleep(max(wait, 1))
return complete(body)
print(complete({"tier": "fast", "messages": [{"role": "user", "content": "ping"}]})["choices"][0]["message"]["content"])
const BASE = "https://<your tenant host>/public/v1";
const headers = { Authorization: "Bearer kapi_...", "Content-Type": "application/json" };
const sleep = (s: number) => new Promise((res) => setTimeout(res, s * 1000));
async function complete(body: unknown): Promise<any> {
const r = await fetch(`${BASE}/chat/completions`, { method: "POST", headers, body: JSON.stringify(body) });
if (r.status !== 429) {
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
console.log("key minute allowance left:", r.headers.get("X-RateLimit-Remaining"));
return r.json();
}
const err = await r.json();
let wait: number;
if (err.code === "RATE_LIMITED") wait = Number(r.headers.get("X-RateLimit-Reset")) - Math.floor(Date.now() / 1000);
else if (err.error?.code === "rate_limit_exceeded") wait = 60 - (Math.floor(Date.now() / 1000) % 60); // fixed minute window; Retry-After is not relayed
else throw new Error(err.error?.message ?? "budget or quota exhausted");
await sleep(Math.max(wait, 1));
return complete(body);
}
console.log((await complete({ tier: "fast", messages: [{ role: "user", content: "ping" }] })).choices[0].message.content);
Warnings before a refusal
X-RateLimit-Remainingon every successful response counts down the key's minute allowance.- The gateway sets
X-Budget-Warningat 80 percent of a rule's threshold and on a soft block, but the edge does not relay it, so the first sign of a budget is its hard block. X-Token-Quota-Warning, set when the quota is in soft mode and used up, is not relayed either.- Nothing exposes the gateway's per-user, per-tenant or per-model counters before they refuse.
Limits and behaviour
- Through the public API the gateway's per-user limit falls on the platform's service identity, so on the current build every key in the tenant shares one allowance.
- The key's windows are fixed on the clock: the minute allowance resets at the next clock minute, the day allowance at midnight UTC. The gateway's windows are fixed sixty seconds too; with
Retry-Afterunrelayed, wait for the next minute. - A call refused by a later check gives its request slot back. A call refused by the key's limit never reaches the gateway and counts against nothing else.
- A per-model refusal does not fall through to another model in the tier. A budget rule scoped to a model cannot match a call that named a tier, and a rule scoped to an agent matches only calls that carry the
X-Agent-IDheader. - Budgets count spend the gateway can price; a model without a price adds nothing, and a daily or weekly rule sees a completion only after its counter is rebuilt from the cost ledger.
- The warning percentage entered on a budget alert is not applied on the current build; the 80 percent threshold is the same for every rule.
- If the platform's counter store is unavailable, the AI Gateway admits calls rather than refusing them and the key's limit is counted per gateway replica, so briefly more calls than the limit can get through.
Troubleshooting
- 429
RATE_LIMITEDright after a burst: the key's minute allowance. Spread the calls, or ask the key's creator to raise Requests / min on the API keys tab. - 429
rate_limit_exceededonuseralthough this key sent little: the shared per-user allowance described above. An administrator raises Per User in the AI Gateway's Rate Limits dialog. - 429
budget_exceeded: the rule named in the message has reached its threshold; an administrator sees it on the Budgets page and can raise or delete the alert. - 429
token_quota_exceeded: the tenant's licence quota for the period is spent; nothing on the API side clears it. - More calls got through than the key's limit: the counter store was unavailable and each gateway replica counted on its own.
Where the limits are set