Skip to main content

LiteLLM

Public API

LiteLLM's completion function gives one calling convention across providers. Prefix the model id with openai/, pass your tenant's base URL as api_base and the key as api_key, and LiteLLM speaks the chat completions protocol to your tenant's AI Gateway. This page covers LiteLLM as a library inside your own code; its proxy server is a separate deployment and is not covered here.

Who can use it: anyone holding an API key with the Chat scope. The tenant must be licensed for the public API.

Prerequisites​

  • Python 3.9 or later and pip install litellm. Verified with litellm 1.100.1.
  • Your tenant's public API address, https://<your tenant host>/public/v1, from the top of Studio's Developer Hub.
  • A key from the Developer Hub's API keys tab, and a model id your tenant serves. The models route lists them; Use an OpenAI client library shows the call.

Connect​

import litellm

response = litellm.completion(
model="openai/<a model id your tenant serves>",
api_base="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
messages=[{"role": "user", "content": "Summarise our onboarding guide in three bullet points."}],
max_tokens=300,
)
print(response.model, response.usage.completion_tokens)
print(response.choices[0].message.content)

The openai/ prefix selects LiteLLM's OpenAI-compatible provider; everything after the slash is sent to your tenant as the model id. Keep the prefix even for ids LiteLLM would recognise on its own, and pass api_base on every call or set litellm.api_base once, because without it the provider uses OpenAI's address. acompletion is the asynchronous twin with the same arguments. To record the usage against one of your agents in analytics, add extra_headers={"X-Agent-ID": "<agent id>"}.

Stream a reply​

for chunk in litellm.completion(
model="openai/<a model id your tenant serves>",
api_base="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
messages=[{"role": "user", "content": "Write a short welcome note for new starters."}],
max_tokens=300,
stream=True,
):
print(chunk.choices[0].delta.content or "", end="", flush=True)

The chunks arrive as the model produces them, unless your guardrail profile has output rails: the gateway then holds the whole answer so those rails can read it, and the chunks arrive in one burst.

Verify​

  1. response.model names the model that answered and response.usage carries the token counts, as the gateway returned them.
  2. The key's row on the Developer Hub's API keys tab shows a new last-used time.
  3. Each call appears in the Control Room's AI Gateway under Logs as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.

What does not carry over​

  • Cost. LiteLLM computes its own cost estimate from its price map, keyed by the model id, which is not what your tenant charges. The cost that counts is on the gateway log line and in the Control Room's analytics.
  • Routing, fallbacks and budgets. Your tenant's AI Gateway already applies tiers, fallbacks, budgets and rate limits to every call. A LiteLLM router in front of it adds a second layer of the same decisions, so keep the library's role to the call itself.
  • Tools on a first turn. With tools passed and no tool choice set, the first turn is upgraded to required, so the model must call a tool; later turns are not upgraded. See Call tools from a completion.
  • user and metadata. Both are accepted and discarded; attribute usage with the X-Agent-ID header instead.
  • Streaming pace. Chunks arrive as written, or in one burst after the output rails have read the answer.

Troubleshooting​

  • AuthenticationError on every call. Check the key and that api_base ends in /public/v1. The same error with a key that works elsewhere means the key lacks the Chat scope: the gateway reports a missing scope as an invalid key.
  • RateLimitError. RATE_LIMITED in the body is the key's own per-minute or per-day limit: wait for X-RateLimit-Reset. rate_limit_exceeded is a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway's Retry-After is not relayed, so wait for the next clock minute. See How limits combine.
  • BadRequestError saying the provider was not given. The openai/ prefix is missing and LiteLLM could not guess a provider from the model id. Add the prefix.
  • A recognised model id was answered by OpenAI rather than your tenant. api_base was omitted. Pass it on every call or set litellm.api_base.
  • An InternalServerError wraps another error. LiteLLM wraps the underlying client's exception; read the message inside it, which names the real cause.

Describes the platform as of .

Was this page helpful?