Use an OpenAI client library
Public API
The model routes of the public API accept the request shapes OpenAI client libraries already send. Change two settings, the base URL and the key, and the official libraries work against your tenant.
Who can use it: anyone holding an API key with the Chat scope for chat completions and the Read models scope to list models; the other model routes name their scope on their reference page. The tenant must be licensed for the public API.
Prerequisites
- An API key. The Chat only preset covers chat completions; add Read models if the application lists models itself. See Authentication.
- The
openaipackage for Python or Node.js, or any client that lets you set a base URL.
Configure the library
-
Find a model id. The models route returns the models the AI Gateway serves for your tenant in the OpenAI list shape, chat models by default.
curl "https://<your tenant host>/public/v1/models" \-H "Authorization: Bearer kapi_..." -
Point the library at your tenant. The base URL is
https://<your tenant host>/public/v1and the key is the bearer token.from openai import OpenAIclient = OpenAI(base_url="https://<your tenant host>/public/v1", api_key="kapi_...")completion = client.chat.completions.create(model="<your-model-id>",messages=[{"role": "user", "content": "Summarise our onboarding guide in three bullet points."}],)print(completion.choices[0].message.content)import OpenAI from "openai";const client = new OpenAI({ baseURL: "https://<your tenant host>/public/v1", apiKey: "kapi_..." });const completion = await client.chat.completions.create({model: "<your-model-id>",messages: [{ role: "user", content: "Summarise our onboarding guide in three bullet points." }],});console.log(completion.choices[0].message.content); -
Let the tier choose, if you prefer. Called directly, the route accepts a request without
modeland routes bytier(balancedunless you say otherwise). The client libraries require a model argument, so pass a model id from step 1 when you go through them.
Stream tokens
Set stream to true and the route answers with a text/event-stream: each event is a data: line carrying a chat completion chunk as JSON, and the stream ends with data: [DONE]. The libraries iterate the chunks for you.
The chunks arrive as the model produces them, so the loop below prints the reply word by word. The exception is a guardrail profile with output rails (output checks or secret redaction): the gateway then holds the whole answer so those rails can read it, and the chunks arrive in one burst. Stream a reply has the event catalogue for both the model routes and the agent routes.
with client.chat.completions.create(
model="<your-model-id>",
messages=[{"role": "user", "content": "Write a short welcome note for new starters."}],
stream=True,
) as stream:
for chunk in stream:
delta = chunk.choices[0].delta.content or ""
print(delta, end="", flush=True)
Generation routes have a longer time budget than the rest of the API, so a long answer is not cut off at the sixty-second mark that applies elsewhere.
Attribute usage to an agent
Send an X-Agent-ID header carrying an agent id and the gateway records the usage against that agent in analytics.
client = OpenAI(
base_url="https://<your tenant host>/public/v1",
api_key="kapi_...",
default_headers={"X-Agent-ID": "<agent-id>"},
)
Tools, JSON output and provider fields
tools, tool_choice and response_format take the OpenAI shapes, with three differences from a direct provider call. On a first turn with tools bound, an absent or auto tool choice is upgraded to required, so the model must call a tool; none and a named tool are respected, and the requirement is relaxed on later turns. response_format is forwarded as sent, and dropped without an error for a model that does not support it. extra_body is not a passthrough: only ten sampling parameters for self-hosted engines are kept (min_tokens, ignore_eos, top_k, min_p, repetition_penalty, length_penalty, stop_token_ids, include_stop_str_in_output, skip_special_tokens, spaces_between_special_tokens) and every other key is dropped.
user and metadata are accepted and discarded: neither reaches the model, the usage record or the logs, so attribute usage with the X-Agent-ID header above. The other model routes, completions, embeddings, rerank and image generation, are listed on the Chat and completions page with the scope each one needs; Call tools from a completion and Structured outputs go deeper.
Verify
The response is an OpenAI-compatible chat completion object with choices[0].message.content, and the key's row on the Developer Hub's API keys tab shows the usage.
Troubleshooting
- 401 with
AUTH_ERROR: check that the library sends the key as the bearer token and that the base URL ends in/public/v1. - 401 with
AUTH_ERRORandInvalid or expired API key, but the key works elsewhere: the key lacks the scope of the route you called, and the gateway reports a missing scope as an invalid key. Chat completions need Chat; the models list needs Read models. - 429 with
RATE_LIMITED: the key is over its own per-minute or per-day limit; readX-RateLimit-Resetand retry after it. See API overview. - 429 with
error.codeset torate_limit_exceeded: a per-user, per-tenant or per-model limit inside the AI Gateway; the gateway'sRetry-Afteris not relayed, so wait for the next clock minute. See How limits combine. - 502 or 504: the gateway could not get an answer from the provider in time. Retry with backoff; if it persists, an administrator can check the provider in the Control Room's AI Gateway.
Related
- Chat and completions
- Tools and models
- Chat from your application
- Errors
- Integrations, one verified page per client library and front end