Skip to main content

Tiers and models

The model routes take a model id or a tier. A tier is a label an administrator puts on models in the gateway (fast, balanced, reasoning, guard, code and vision for chat; embedding, reranking and image for the other routes), and the gateway serves the highest-priority healthy model carrying it, moving to the next when one fails. A model id pins one model and turns that movement off. Choose by whether you want the platform to keep the call working, or one model's exact behaviour.

Who can use it: anyone holding an API key with the Chat scope for chat completions or the Completions scope for text completions, and Read models to list what the tenant serves.

Tier or model​

tiermodel
What you sendA label such as fast, balanced (the default when both are empty) or reasoning; auto lets the gateway classifyA model id exactly as GET /public/v1/models returns it
What answersThe highest-priority enabled model tagged with that tier whose provider has a verified credentialThat model and no other
When the first choice failsThe next model in the tier, then the neighbouring tiers, then any enabled chat modelNo second model is tried; the failure comes back as an error
When nothing matches422 with no_models_for_tier and a reason422 with model_not_available
Reach for it whenYou want the platform to choose, and to keep working through a provider incidentYou depend on one model's behaviour, or on a feature only it supports

Sending both pins the model; tier is then only echoed. A tier name in model works too: fast, balanced, reasoning and auto there are read as tiers, and default means no explicit model.

Quick start​

List the chat models, then let the tier choose and read back what served the call:

curl "https://<your tenant host>/public/v1/models?kind=chat" \
-H "Authorization: Bearer kapi_..."

curl -i -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"tier": "reasoning",
"messages": [{"role": "user", "content": "Plan the migration of a 2 TB Postgres database with under a minute of downtime."}]}'

What balanced means​

A tier is catalogue data, not a fixed list of models. An administrator tags each model in the AI Gateway with one tier and a priority, and balanced resolves to the models tagged exactly balanced, in priority order, whose provider is enabled and holds a verified credential. Retired models, deprecated models and fine-tuning templates are skipped by tier routing and reachable only by pin.

When a tier resolves to nothing, the gateway walks to its neighbours first: balanced tries fast then reasoning; reasoning tries balanced then fast; fast tries balanced then reasoning; code, vision and guard try balanced then fast; after that, any enabled chat model. Every substitution is disclosed in the response, so a tenant whose default tier has not been tagged yet is served by a neighbouring tier and told so on every call. Only when nothing at all resolves does the call fail, with 422 and a reason of no_catalog_rows, no_provider_enabled, no_credential, unverified_credential or no_models_for_tier.

auto, and balanced on a request without tools, adds one step: the gateway classifies the last user message and routes it to fast, balanced or reasoning before resolving. The tier in the response is the one that served the call.

Read what answered​

FieldMeaning
model, served_model, served_providerThe model that answered and its provider. model is overwritten with the served model, so it is safe to log.
tier, providerThe tier that served the call and the provider type. Not stamped on streamed chunks.
is_fallbackTrue when the model that served is not the one the tier put first, including when the tier itself was substituted. Always present on a non-streamed answer; on a stream it rides the first chunk.
fallback_fromThe primary model's id, or the requested tier when the tier was substituted. Present only on a fallback.
fallback_reasonunhealthy, circuit_open, credential_unresolvable, quota_exhausted, attempt_budget_exhausted or upstream_error for a runtime fallback; no_models_for_requested_tier, caller_tier_cascade or runtime_cross_tier_fallback for a tier substitution.

The AI Gateway also stamps these facts into the headers X-Served-Model, X-Served-Provider, X-Fallback-From and X-Fallback-Reason, but the public edge relays only the status, the body and the content type, so on the current build none of them reach a public API caller. Read the fields from the body, and on a stream from the first chunk.

Limits and behaviour​

  • A model id must match the catalogue entry exactly. A provider prefix such as openai/ works only when the catalogue stores the id that way; the models route shows the spelling to use.
  • The gateway makes at most two upstream attempts per call by default, and once the first token of a streamed answer has been sent it does not switch models; a failure after that point ends the stream with an error event.
  • A model tagged to an environment other than production is not a candidate for public API calls, and a pin on one answers 422 as if the model did not exist.
  • Two refusals are decided before any model is called and arrive as 403 FORBIDDEN: platform policy forbidding the model for the organisation, and a team not entitled to it. The detail text tells them apart.
  • The models route lists chat models by default; kind=all adds every other modality, each row carrying its modality, and tier=<label> filters by exact tag.
  • That route, not this page, is the live catalogue: it is per tenant, and has_key reports a credential, not a model the provider serves.

Troubleshooting​

  • 422 with model_not_available: the id is not served for this tenant. List the models and copy the id; a disabled model, a provider without a verified credential, or a model in another lane all answer this way.
  • 422 with no_models_for_tier: read reason. no_credential and unverified_credential are for an administrator; no_catalog_rows means nothing is tagged with that tier or its neighbours.
  • A different model answered than the tier usually serves: read is_fallback and fallback_reason. unhealthy and circuit_open clear on their own; credential_unresolvable needs an administrator.
  • 403 FORBIDDEN whose detail says the model is not permitted for your organisation: platform policy, which your organisation's administrator cannot change. Your team(s) are not entitled: ask your organisation's administrator.
  • The response tier is not the one you sent: the tier was substituted or, for auto, chosen from the question; fallback_reason says which.

The same choice elsewhere

Describes the platform as of .

Was this page helpful?