LiteLLM
Public API
LiteLLM's completion function gives one calling convention across providers. Prefix the model id with openai/, pass your tenant's base URL as api_base and the key as api_key, and LiteLLM speaks the chat completions protocol to your tenant's AI Gateway. This page covers LiteLLM as a library inside your own code; its proxy server is a separate deployment and is not covered here.
Who can use it: anyone holding an API key with the Chat scope. The tenant must be licensed for the public API.
Prerequisites
- Python 3.9 or later and
pip install litellm. Verified withlitellm1.100.1. - Your tenant's public API address,
https://<your tenant host>/public/v1, from the top of Studio's Developer Hub. - A key from the Developer Hub's API keys tab, and a model id your tenant serves. The models route lists them; Use an OpenAI client library shows the call.
Connect
import litellm
response = litellm.completion(
model="openai/<a model id your tenant serves>",
api_base="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
messages=[{"role": "user", "content": "Summarise our onboarding guide in three bullet points."}],
max_tokens=300,
)
print(response.model, response.usage.completion_tokens)
print(response.choices[0].message.content)
The openai/ prefix selects LiteLLM's OpenAI-compatible provider; everything after the slash is sent to your tenant as the model id. Keep the prefix even for ids LiteLLM would recognise on its own, and pass api_base on every call or set litellm.api_base once, because without it the provider uses OpenAI's address. acompletion is the asynchronous twin with the same arguments. To record the usage against one of your agents in analytics, add extra_headers={"X-Agent-ID": "<agent id>"}.
Stream a reply
for chunk in litellm.completion(
model="openai/<a model id your tenant serves>",
api_base="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
messages=[{"role": "user", "content": "Write a short welcome note for new starters."}],
max_tokens=300,
stream=True,
):
print(chunk.choices[0].delta.content or "", end="", flush=True)
The chunks arrive as the model produces them, unless your guardrail profile has output rails: the gateway then holds the whole answer so those rails can read it, and the chunks arrive in one burst.
Verify
response.modelnames the model that answered andresponse.usagecarries the token counts, as the gateway returned them.- The key's row on the Developer Hub's API keys tab shows a new last-used time.
- Each call appears in the Control Room's AI Gateway under Logs as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.
What does not carry over
- Cost. LiteLLM computes its own cost estimate from its price map, keyed by the model id, which is not what your tenant charges. The cost that counts is on the gateway log line and in the Control Room's analytics.
- Routing, fallbacks and budgets. Your tenant's AI Gateway already applies tiers, fallbacks, budgets and rate limits to every call. A LiteLLM router in front of it adds a second layer of the same decisions, so keep the library's role to the call itself.
- Tools on a first turn. With
toolspassed and no tool choice set, the first turn is upgraded torequired, so the model must call a tool; later turns are not upgraded. See Call tools from a completion. userandmetadata. Both are accepted and discarded; attribute usage with theX-Agent-IDheader instead.- Streaming pace. Chunks arrive as written, or in one burst after the output rails have read the answer.
Troubleshooting
AuthenticationErroron every call. Check the key and thatapi_baseends in/public/v1. The same error with a key that works elsewhere means the key lacks the Chat scope: the gateway reports a missing scope as an invalid key.RateLimitError.RATE_LIMITEDin the body is the key's own per-minute or per-day limit: wait forX-RateLimit-Reset.rate_limit_exceededis a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway'sRetry-Afteris not relayed, so wait for the next clock minute. See How limits combine.BadRequestErrorsaying the provider was not given. Theopenai/prefix is missing and LiteLLM could not guess a provider from the model id. Add the prefix.- A recognised model id was answered by OpenAI rather than your tenant.
api_basewas omitted. Pass it on every call or setlitellm.api_base. - An
InternalServerErrorwraps another error. LiteLLM wraps the underlying client's exception; read the message inside it, which names the real cause.