Skip to main content

How limits combine

One call to a model route can be refused by five different limits. They are checked in a fixed order, each answers in its own shape, and only one of them sends the X-RateLimit-* headers. This page puts them side by side so a 429 can be read and the right thing retried.

Who can use it: anyone holding an API key. The key's own limit applies to every route; the other four apply to the model routes: chat completions, completions, embeddings, rerank and image generation.

The five limits​

Set byScopeWhat you see when it refuses
Key rate limitThe key's Requests / min and Requests / day, or the platform default of 60 and 1,000One key, on every route429 {"code": "RATE_LIMITED"} from the public API gateway, with X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset and X-RateLimit-Window (minute or day)
Gateway request limitThe AI Gateway's Rate Limits dialog: requests per minute per user, per tenant and per model (defaults 30, 200 and 100)All traffic in the tenant429 in the OpenAI shape, error.code rate_limit_exceeded, a message naming rpm and the scope; the gateway's Retry-After is not relayed
Gateway token limitThe same dialog: tokens per minute per user, tenant and model (defaults 60,000, 500,000 and 200,000)The same429 rate_limit_exceeded naming tpm. Tokens are counted after each answer, so the call that crosses the cap completes and the next one is refused
BudgetA budget alert in the Control Room: a spend threshold per day, week or month on the tenant, a user, a team, an agent or a modelThe rule's scopeWarn and soft block: the call proceeds, and the gateway's X-Budget-Warning header is not relayed. Hard block: 429 error.code budget_exceeded
Token quotaThe licence's token cap for the period, in hard or soft modeThe tenantHard: 429 token_quota_exceeded. Soft: the call proceeds; X-Token-Quota-Warning is not relayed

For a chat completion the checks run in this order: the key's limit at the edge; the gateway's request limit for the user and the tenant; the input guardrails; the budget; the token quota; model resolution; then, per candidate model, the per-model request limit and a reservation of the call's worst-case cost against the budget. When two would fail, the earliest wins, so a call over both the rate limit and the budget is refused as rate_limit_exceeded. Embeddings and rerank check the budget first, then the rate limit, then the quota; image generation checks the rate limit, then the budget, and never the quota.

Read a 429​

BodyHeaderWhich limit, and what to do
{"code": "RATE_LIMITED", "detail": "Rate limit exceeded"}X-RateLimit-WindowThe key's own. Wait for X-RateLimit-Reset, a Unix timestamp in seconds.
error.code rate_limit_exceeded; the message names rpm or tpm and user, tenant or modelnone: the gateway's Retry-After is dropped at the edgeThe AI Gateway. Wait for the next clock minute.
error.code budget_exceeded; the message names the rule and the spendnoneA budget's hard block. Do not retry until the period resets.
error.code token_quota_exceedednoneThe tenant's token quota.
error.code all_providers_failed, error_class rate_limit or quotanone: the gateway's Retry-After is dropped at the edgeThe model provider's own limit, after every candidate was tried. Wait a few seconds.

Quick start​

A wrapper that reads the headers on success and tells the refusals apart:

curl -i -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"tier": "fast", "messages": [{"role": "user", "content": "ping"}]}' \
| grep -i -E "^HTTP|^x-ratelimit|^\{"

Warnings before a refusal​

  • X-RateLimit-Remaining on every successful response counts down the key's minute allowance.
  • The gateway sets X-Budget-Warning at 80 percent of a rule's threshold and on a soft block, but the edge does not relay it, so the first sign of a budget is its hard block.
  • X-Token-Quota-Warning, set when the quota is in soft mode and used up, is not relayed either.
  • Nothing exposes the gateway's per-user, per-tenant or per-model counters before they refuse.

Limits and behaviour​

  • Through the public API the gateway's per-user limit falls on the platform's service identity, so on the current build every key in the tenant shares one allowance.
  • The key's windows are fixed on the clock: the minute allowance resets at the next clock minute, the day allowance at midnight UTC. The gateway's windows are fixed sixty seconds too; with Retry-After unrelayed, wait for the next minute.
  • A call refused by a later check gives its request slot back. A call refused by the key's limit never reaches the gateway and counts against nothing else.
  • A per-model refusal does not fall through to another model in the tier. A budget rule scoped to a model cannot match a call that named a tier, and a rule scoped to an agent matches only calls that carry the X-Agent-ID header.
  • Budgets count spend the gateway can price; a model without a price adds nothing, and a daily or weekly rule sees a completion only after its counter is rebuilt from the cost ledger.
  • The warning percentage entered on a budget alert is not applied on the current build; the 80 percent threshold is the same for every rule.
  • If the platform's counter store is unavailable, the AI Gateway admits calls rather than refusing them and the key's limit is counted per gateway replica, so briefly more calls than the limit can get through.

Troubleshooting​

  • 429 RATE_LIMITED right after a burst: the key's minute allowance. Spread the calls, or ask the key's creator to raise Requests / min on the API keys tab.
  • 429 rate_limit_exceeded on user although this key sent little: the shared per-user allowance described above. An administrator raises Per User in the AI Gateway's Rate Limits dialog.
  • 429 budget_exceeded: the rule named in the message has reached its threshold; an administrator sees it on the Budgets page and can raise or delete the alert.
  • 429 token_quota_exceeded: the tenant's licence quota for the period is spent; nothing on the API side clears it.
  • More calls got through than the key's limit: the counter store was unavailable and each gateway replica counted on its own.

Where the limits are set

Describes the platform as of .

Was this page helpful?