Tiers and models
The model routes take a model id or a tier. A tier is a label an administrator puts on models in the gateway (fast, balanced, reasoning, guard, code and vision for chat; embedding, reranking and image for the other routes), and the gateway serves the highest-priority healthy model carrying it, moving to the next when one fails. A model id pins one model and turns that movement off. Choose by whether you want the platform to keep the call working, or one model's exact behaviour.
Who can use it: anyone holding an API key with the Chat scope for chat completions or the Completions scope for text completions, and Read models to list what the tenant serves.
Tier or model
tier | model | |
|---|---|---|
| What you send | A label such as fast, balanced (the default when both are empty) or reasoning; auto lets the gateway classify | A model id exactly as GET /public/v1/models returns it |
| What answers | The highest-priority enabled model tagged with that tier whose provider has a verified credential | That model and no other |
| When the first choice fails | The next model in the tier, then the neighbouring tiers, then any enabled chat model | No second model is tried; the failure comes back as an error |
| When nothing matches | 422 with no_models_for_tier and a reason | 422 with model_not_available |
| Reach for it when | You want the platform to choose, and to keep working through a provider incident | You depend on one model's behaviour, or on a feature only it supports |
Sending both pins the model; tier is then only echoed. A tier name in model works too: fast, balanced, reasoning and auto there are read as tiers, and default means no explicit model.
Quick start
List the chat models, then let the tier choose and read back what served the call:
- cURL
- Python
- TypeScript
curl "https://<your tenant host>/public/v1/models?kind=chat" \
-H "Authorization: Bearer kapi_..."
curl -i -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"tier": "reasoning",
"messages": [{"role": "user", "content": "Plan the migration of a 2 TB Postgres database with under a minute of downtime."}]}'
import requests
BASE = "https://<your tenant host>/public/v1"
HEADERS = {"Authorization": "Bearer kapi_..."}
models = requests.get(f"{BASE}/models", headers=HEADERS, params={"kind": "chat"}).json()["data"]
print({m["id"]: m["tier"] for m in models})
r = requests.post(f"{BASE}/chat/completions", headers=HEADERS, json={
"tier": "reasoning",
"messages": [{"role": "user", "content": "Plan the migration of a 2 TB Postgres database with under a minute of downtime."}],
})
body = r.json()
print(body["served_model"], "on", body["served_provider"], "tier", body["tier"])
if body["is_fallback"]:
print("fallback from", body["fallback_from"], "because", body["fallback_reason"])
const BASE = "https://<your tenant host>/public/v1";
const headers = { Authorization: "Bearer kapi_...", "Content-Type": "application/json" };
const models = (await (await fetch(`${BASE}/models?kind=chat`, { headers })).json()).data;
console.log(Object.fromEntries(models.map((m: { id: string; tier: string }) => [m.id, m.tier])));
const r = await fetch(`${BASE}/chat/completions`, {
method: "POST", headers,
body: JSON.stringify({
tier: "reasoning",
messages: [{ role: "user", content: "Plan the migration of a 2 TB Postgres database with under a minute of downtime." }],
}),
});
const body = await r.json();
console.log(body.served_model, "on", body.served_provider, "tier", body.tier);
if (body.is_fallback) console.log("fallback from", body.fallback_from, "because", body.fallback_reason);
What balanced means
A tier is catalogue data, not a fixed list of models. An administrator tags each model in the AI Gateway with one tier and a priority, and balanced resolves to the models tagged exactly balanced, in priority order, whose provider is enabled and holds a verified credential. Retired models, deprecated models and fine-tuning templates are skipped by tier routing and reachable only by pin.
When a tier resolves to nothing, the gateway walks to its neighbours first: balanced tries fast then reasoning; reasoning tries balanced then fast; fast tries balanced then reasoning; code, vision and guard try balanced then fast; after that, any enabled chat model. Every substitution is disclosed in the response, so a tenant whose default tier has not been tagged yet is served by a neighbouring tier and told so on every call. Only when nothing at all resolves does the call fail, with 422 and a reason of no_catalog_rows, no_provider_enabled, no_credential, unverified_credential or no_models_for_tier.
auto, and balanced on a request without tools, adds one step: the gateway classifies the last user message and routes it to fast, balanced or reasoning before resolving. The tier in the response is the one that served the call.
Read what answered
| Field | Meaning |
|---|---|
model, served_model, served_provider | The model that answered and its provider. model is overwritten with the served model, so it is safe to log. |
tier, provider | The tier that served the call and the provider type. Not stamped on streamed chunks. |
is_fallback | True when the model that served is not the one the tier put first, including when the tier itself was substituted. Always present on a non-streamed answer; on a stream it rides the first chunk. |
fallback_from | The primary model's id, or the requested tier when the tier was substituted. Present only on a fallback. |
fallback_reason | unhealthy, circuit_open, credential_unresolvable, quota_exhausted, attempt_budget_exhausted or upstream_error for a runtime fallback; no_models_for_requested_tier, caller_tier_cascade or runtime_cross_tier_fallback for a tier substitution. |
The AI Gateway also stamps these facts into the headers X-Served-Model, X-Served-Provider, X-Fallback-From and X-Fallback-Reason, but the public edge relays only the status, the body and the content type, so on the current build none of them reach a public API caller. Read the fields from the body, and on a stream from the first chunk.
Limits and behaviour
- A model id must match the catalogue entry exactly. A provider prefix such as
openai/works only when the catalogue stores the id that way; the models route shows the spelling to use. - The gateway makes at most two upstream attempts per call by default, and once the first token of a streamed answer has been sent it does not switch models; a failure after that point ends the stream with an error event.
- A model tagged to an environment other than production is not a candidate for public API calls, and a pin on one answers 422 as if the model did not exist.
- Two refusals are decided before any model is called and arrive as 403
FORBIDDEN: platform policy forbidding the model for the organisation, and a team not entitled to it. Thedetailtext tells them apart. - The models route lists chat models by default;
kind=alladds every other modality, each row carrying itsmodality, andtier=<label>filters by exact tag. - That route, not this page, is the live catalogue: it is per tenant, and
has_keyreports a credential, not a model the provider serves.
Troubleshooting
- 422 with
model_not_available: the id is not served for this tenant. List the models and copy the id; a disabled model, a provider without a verified credential, or a model in another lane all answer this way. - 422 with
no_models_for_tier: readreason.no_credentialandunverified_credentialare for an administrator;no_catalog_rowsmeans nothing is tagged with that tier or its neighbours. - A different model answered than the tier usually serves: read
is_fallbackandfallback_reason.unhealthyandcircuit_openclear on their own;credential_unresolvableneeds an administrator. - 403
FORBIDDENwhosedetailsays the model is not permitted for your organisation: platform policy, which your organisation's administrator cannot change. Your team(s) are not entitled: ask your organisation's administrator. - The response
tieris not the one you sent: the tier was substituted or, forauto, chosen from the question;fallback_reasonsays which.
The same choice elsewhere