# LiteLLM

Call your tenant's models through LiteLLM's completion function with the openai provider prefix, a base URL and a key, and stream replies the same way.

LiteLLM's `completion` function gives one calling convention across providers. Prefix the model id with `openai/`, pass your tenant's base URL as `api_base` and the key as `api_key`, and LiteLLM speaks the chat completions protocol to your tenant's AI Gateway. This page covers LiteLLM as a library inside your own code; its proxy server is a separate deployment and is not covered here.

**Who can use it:** anyone holding an API key with the **Chat** scope. The tenant must be licensed for the public API.

## Prerequisites

- Python 3.9 or later and `pip install litellm`. Verified with `litellm` 1.100.1.
- Your tenant's public API address, `https://<your tenant host>/public/v1`, from the top of Studio's **Developer Hub**.
- A key from the Developer Hub's **API keys** tab, and a model id your tenant serves. The models route lists them; [Use an OpenAI client library](/developers/guides/use-an-openai-client-library) shows the call.

## Connect

```python

response = litellm.completion(
    model="openai/<a model id your tenant serves>",
    api_base="https://<your tenant host>/public/v1",
    api_key="kapi_YOUR_KEY",
    messages=[{"role": "user", "content": "Summarise our onboarding guide in three bullet points."}],
    max_tokens=300,
)
print(response.model, response.usage.completion_tokens)
print(response.choices[0].message.content)
```

The `openai/` prefix selects LiteLLM's OpenAI-compatible provider; everything after the slash is sent to your tenant as the model id. Keep the prefix even for ids LiteLLM would recognise on its own, and pass `api_base` on every call or set `litellm.api_base` once, because without it the provider uses OpenAI's address. `acompletion` is the asynchronous twin with the same arguments. To record the usage against one of your agents in analytics, add `extra_headers={"X-Agent-ID": "<agent id>"}`.

## Stream a reply

```python
for chunk in litellm.completion(
    model="openai/<a model id your tenant serves>",
    api_base="https://<your tenant host>/public/v1",
    api_key="kapi_YOUR_KEY",
    messages=[{"role": "user", "content": "Write a short welcome note for new starters."}],
    max_tokens=300,
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)
```

The chunks arrive as the model produces them, unless your guardrail profile has output rails: the gateway then holds the whole answer so those rails can read it, and the chunks arrive in one burst.

## Verify

1. `response.model` names the model that answered and `response.usage` carries the token counts, as the gateway returned them.
2. The key's row on the Developer Hub's **API keys** tab shows a new last-used time.
3. Each call appears in the Control Room's **AI Gateway** under **Logs** as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.

## What does not carry over

- **Cost.** LiteLLM computes its own cost estimate from its price map, keyed by the model id, which is not what your tenant charges. The cost that counts is on the gateway log line and in the Control Room's analytics.
- **Routing, fallbacks and budgets.** Your tenant's AI Gateway already applies tiers, fallbacks, budgets and rate limits to every call. A LiteLLM router in front of it adds a second layer of the same decisions, so keep the library's role to the call itself.
- **Tools on a first turn.** With `tools` passed and no tool choice set, the first turn is upgraded to `required`, so the model must call a tool; later turns are not upgraded. See [Call tools from a completion](/developers/guides/call-tools-from-a-completion).
- **`user` and `metadata`.** Both are accepted and discarded; attribute usage with the `X-Agent-ID` header instead.
- **Streaming pace.** Chunks arrive as written, or in one burst after the output rails have read the answer.

## Troubleshooting

- **`AuthenticationError` on every call.** Check the key and that `api_base` ends in `/public/v1`. The same error with a key that works elsewhere means the key lacks the **Chat** scope: the gateway reports a missing scope as an invalid key.
- **`RateLimitError`.** `RATE_LIMITED` in the body is the key's own per-minute or per-day limit: wait for `X-RateLimit-Reset`. `rate_limit_exceeded` is a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway's `Retry-After` is not relayed, so wait for the next clock minute. See [How limits combine](/developers/guides/how-limits-combine).
- **`BadRequestError` saying the provider was not given.** The `openai/` prefix is missing and LiteLLM could not guess a provider from the model id. Add the prefix.
- **A recognised model id was answered by OpenAI rather than your tenant.** `api_base` was omitted. Pass it on every call or set `litellm.api_base`.
- **An `InternalServerError` wraps another error.** LiteLLM wraps the underlying client's exception; read the message inside it, which names the real cause.

## Related

- [Use an OpenAI client library](/developers/guides/use-an-openai-client-library)
- [How limits combine](/developers/guides/how-limits-combine)
- [OpenAI Python SDK](/developers/integrations/openai-python)
- [Integrations](/developers/integrations/overview)

Source: https://docs.katonic.ai/developers/integrations/litellm
