Skip to main content

OpenAI Python SDK

Public API

The openai package is the client most Python code already uses for chat completions. Two constructor arguments, a base URL and a key, point it at your tenant's public API. From then on models.list() returns the models your tenant serves, chat.completions.create() runs through your tenant's AI Gateway with its tiers, guardrails, budgets and logs, and every call is recorded against your key.

Who can use it: anyone holding an API key with the Chat scope; add Read models if the application lists models itself. The tenant must be licensed for the public API.

Prerequisites​

  • Python 3.9 or later and pip install openai. Verified with openai 2.54.0.
  • Your tenant's public API address, https://<your tenant host>/public/v1. Studio's Developer Hub shows it at the top of the page.
  • A key from the Developer Hub's API keys tab. The Chat only preset is enough to complete; add Read models to list models.

Connect​

from openai import OpenAI

client = OpenAI(
base_url="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
)

models = client.models.list()
print([m.id for m in models.data][:5])

completion = client.chat.completions.create(
model="<a model id from the list>",
messages=[{"role": "user", "content": "Summarise our onboarding guide in three bullet points."}],
max_tokens=300,
)
print(completion.model, completion.usage.completion_tokens)
print(completion.choices[0].message.content)

The library needs a model argument. The models route lists the chat models the AI Gateway serves for your tenant, and Tiers and models explains how a model id relates to a tier. Keep the key out of source: the library reads OPENAI_API_KEY and OPENAI_BASE_URL from the environment when the arguments are omitted, so a shell that exports both runs unchanged code against your tenant.

To record the usage against one of your agents in analytics, add default_headers={"X-Agent-ID": "<agent id>"} to the constructor.

Stream a reply​

with client.chat.completions.create(
model="<a model id from the list>",
messages=[{"role": "user", "content": "Write a short welcome note for new starters."}],
max_tokens=300,
stream=True,
stream_options={"include_usage": True},
) as stream:
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
if chunk.usage:
print()
print(chunk.usage.completion_tokens, "completion tokens")

The chunks arrive as the model produces them, unless your guardrail profile has output rails: the gateway then holds the whole answer so those rails can read it, and the chunks arrive in one burst. With include_usage set, the last chunk carries usage and an empty choices list, which is why the loop checks chunk.choices first.

Verify​

  1. models.list() returns your tenant's models rather than OpenAI's catalogue; only the models the AI Gateway serves appear.
  2. completion.model names the model that answered and completion.usage carries the token counts. The streamed reply's last chunk carries the same usage when include_usage is set.
  3. The key's row on the Developer Hub's API keys tab shows a new last-used time, and each completion appears in the Control Room's AI Gateway under Logs as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.

What does not carry over​

  • Only the model routes. The public API serves chat completions, completions, embeddings, rerank, image generation and the models list in the OpenAI shapes. The responses, beta, files, batches, audio and moderations clients on the same object have no route behind them.
  • user and metadata. Both are accepted and discarded; neither reaches the model, the usage record or the logs. Attribute usage with the X-Agent-ID header instead.
  • extra_body. Not a passthrough: ten sampling parameters for self-hosted engines are kept and every other key is dropped. Use an OpenAI client library lists them.
  • Streaming pace. Chunks arrive as written, or in one burst after the output rails have read the answer.
  • Tools on a first turn. With tools bound and tool_choice absent or auto, the first turn is upgraded to required, so the model must call a tool; none and a named tool are respected, and later turns are not upgraded. See Call tools from a completion.

Troubleshooting​

  • AuthenticationError on every call. Check that the key is passed as api_key and that the base URL ends in /public/v1. The same error with a key that works elsewhere means the key lacks the route's scope: the gateway reports a missing scope as an invalid key. Chat completions need Chat; the models list needs Read models.
  • RateLimitError. Read the error body. RATE_LIMITED is the key's own per-minute or per-day limit: wait for X-RateLimit-Reset. rate_limit_exceeded is a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway's Retry-After is not relayed, so wait for the next clock minute. See How limits combine.
  • A call through client.responses, client.files or another non-model client fails. Those clients have no route on the public API; use chat.completions, completions, embeddings or images.
  • APITimeoutError on a long answer. The library's default timeout is ten minutes, and the generation routes have a longer budget than the sixty seconds that applies elsewhere. If you lowered timeout on the constructor, raise it for long answers.
  • APIConnectionError. The host is wrong or a proxy blocks it. curl https://<your tenant host>/public/v1/models with the key in an Authorization header should return JSON.

Describes the platform as of .

Was this page helpful?