OpenAI Python SDK
Public API
The openai package is the client most Python code already uses for chat completions. Two constructor arguments, a base URL and a key, point it at your tenant's public API. From then on models.list() returns the models your tenant serves, chat.completions.create() runs through your tenant's AI Gateway with its tiers, guardrails, budgets and logs, and every call is recorded against your key.
Who can use it: anyone holding an API key with the Chat scope; add Read models if the application lists models itself. The tenant must be licensed for the public API.
Prerequisites
- Python 3.9 or later and
pip install openai. Verified withopenai2.54.0. - Your tenant's public API address,
https://<your tenant host>/public/v1. Studio's Developer Hub shows it at the top of the page. - A key from the Developer Hub's API keys tab. The Chat only preset is enough to complete; add Read models to list models.
Connect
from openai import OpenAI
client = OpenAI(
base_url="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
)
models = client.models.list()
print([m.id for m in models.data][:5])
completion = client.chat.completions.create(
model="<a model id from the list>",
messages=[{"role": "user", "content": "Summarise our onboarding guide in three bullet points."}],
max_tokens=300,
)
print(completion.model, completion.usage.completion_tokens)
print(completion.choices[0].message.content)
The library needs a model argument. The models route lists the chat models the AI Gateway serves for your tenant, and Tiers and models explains how a model id relates to a tier. Keep the key out of source: the library reads OPENAI_API_KEY and OPENAI_BASE_URL from the environment when the arguments are omitted, so a shell that exports both runs unchanged code against your tenant.
To record the usage against one of your agents in analytics, add default_headers={"X-Agent-ID": "<agent id>"} to the constructor.
Stream a reply
with client.chat.completions.create(
model="<a model id from the list>",
messages=[{"role": "user", "content": "Write a short welcome note for new starters."}],
max_tokens=300,
stream=True,
stream_options={"include_usage": True},
) as stream:
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
if chunk.usage:
print()
print(chunk.usage.completion_tokens, "completion tokens")
The chunks arrive as the model produces them, unless your guardrail profile has output rails: the gateway then holds the whole answer so those rails can read it, and the chunks arrive in one burst. With include_usage set, the last chunk carries usage and an empty choices list, which is why the loop checks chunk.choices first.
Verify
models.list()returns your tenant's models rather than OpenAI's catalogue; only the models the AI Gateway serves appear.completion.modelnames the model that answered andcompletion.usagecarries the token counts. The streamed reply's last chunk carries the same usage wheninclude_usageis set.- The key's row on the Developer Hub's API keys tab shows a new last-used time, and each completion appears in the Control Room's AI Gateway under Logs as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.
What does not carry over
- Only the model routes. The public API serves chat completions, completions, embeddings, rerank, image generation and the models list in the OpenAI shapes. The
responses,beta,files,batches,audioandmoderationsclients on the same object have no route behind them. userandmetadata. Both are accepted and discarded; neither reaches the model, the usage record or the logs. Attribute usage with theX-Agent-IDheader instead.extra_body. Not a passthrough: ten sampling parameters for self-hosted engines are kept and every other key is dropped. Use an OpenAI client library lists them.- Streaming pace. Chunks arrive as written, or in one burst after the output rails have read the answer.
- Tools on a first turn. With
toolsbound andtool_choiceabsent orauto, the first turn is upgraded torequired, so the model must call a tool;noneand a named tool are respected, and later turns are not upgraded. See Call tools from a completion.
Troubleshooting
AuthenticationErroron every call. Check that the key is passed asapi_keyand that the base URL ends in/public/v1. The same error with a key that works elsewhere means the key lacks the route's scope: the gateway reports a missing scope as an invalid key. Chat completions need Chat; the models list needs Read models.RateLimitError. Read the error body.RATE_LIMITEDis the key's own per-minute or per-day limit: wait forX-RateLimit-Reset.rate_limit_exceededis a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway'sRetry-Afteris not relayed, so wait for the next clock minute. See How limits combine.- A call through
client.responses,client.filesor another non-model client fails. Those clients have no route on the public API; usechat.completions,completions,embeddingsorimages. APITimeoutErroron a long answer. The library's default timeout is ten minutes, and the generation routes have a longer budget than the sixty seconds that applies elsewhere. If you loweredtimeouton the constructor, raise it for long answers.APIConnectionError. The host is wrong or a proxy blocks it.curl https://<your tenant host>/public/v1/modelswith the key in anAuthorizationheader should return JSON.