# LlamaIndex

Use the OpenAILike class from LlamaIndex against your tenant's public API so query engines, chat engines and agents call your tenant's models with one base URL and one key.

LlamaIndex reaches OpenAI-compatible servers through the `OpenAILike` class in the `llama-index-llms-openai-like` package. Give it your tenant's base URL, a key and a model id, mark it as a chat model and set it as the default LLM; every query engine, chat engine and agent built afterwards answers through your tenant's AI Gateway.

**Who can use it:** anyone holding an API key with the **Chat** scope. The tenant must be licensed for the public API.

## Prerequisites

- Python 3.10 or later and `pip install llama-index-llms-openai-like`, which brings `llama-index-core` with it. Verified with `llama-index-llms-openai-like` 0.8.0 on `llama-index-core` 0.14.24.
- Your tenant's public API address, `https://<your tenant host>/public/v1`, from the top of Studio's **Developer Hub**.
- A key from the Developer Hub's **API keys** tab, and a model id your tenant serves. The models route lists them; [Use an OpenAI client library](/developers/guides/use-an-openai-client-library) shows the call.

## Connect

```python
from llama_index.core import Settings
from llama_index.core.llms import ChatMessage
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="<a model id your tenant serves>",
    api_base="https://<your tenant host>/public/v1",
    api_key="kapi_YOUR_KEY",
    is_chat_model=True,
    max_tokens=300,
)
Settings.llm = llm

reply = llm.complete("Summarise our onboarding guide in three bullet points.")
print(reply.text)

turn = llm.chat([ChatMessage(role="user", content="What does the onboarding guide cover?")])
print(turn.message.content)
```

`is_chat_model=True` sends requests to the chat completions route, where your tenant serves conversational models; left off, the class calls the legacy completions route instead. `Settings.llm` makes the model the default for everything built afterwards, so a `VectorStoreIndex` query engine or a function-calling agent needs no further wiring for the LLM itself. Set `context_window` to the model's limit, which the models route reports for each model, so LlamaIndex packs prompts to the real window rather than the class's small default.

To record the usage against one of your agents in analytics, add `default_headers={"X-Agent-ID": "<agent id>"}` to the constructor.

## Stream a reply

```python
for chunk in llm.stream_complete("Write a short welcome note for new starters."):
    print(chunk.delta or "", end="", flush=True)
```

Each chunk carries the text so far in `text` and the new part in `delta`. The chunks arrive in one burst, not as the model produces them: the gateway holds the whole answer so the output guardrails can read it, then replays the chunks. Stream for the event shape, not for a typing effect.

## Verify

1. `reply.raw` is the gateway's response as the underlying client parsed it: its `model` names the model that answered and its `usage` carries the token counts.
2. The key's row on the Developer Hub's **API keys** tab shows a new last-used time.
3. Each call appears in the Control Room's **AI Gateway** under **Logs** as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.

## What does not carry over

- **Embeddings.** LlamaIndex's default embedding model is OpenAI's own, reached at OpenAI's address, so an index built with defaults fails when the only key you hold is your tenant's. Set `Settings.embed_model` to an embedding model you run yourself, or to a model your tenant serves through an OpenAI-compatible embedding class pointed at the same base URL and key. [Embeddings and rerank](/developers/guides/embeddings-and-rerank) names the route and the models.
- **Tools on a first turn.** A function-calling agent binds tools, and the gateway upgrades a first turn whose tool choice is absent or `auto` to `required`, so the model must call a tool; later turns are not upgraded. See [Call tools from a completion](/developers/guides/call-tools-from-a-completion).
- **`user` and `metadata`.** Both are accepted and discarded; attribute usage with the `X-Agent-ID` header instead.
- **Streaming pace.** Chunks arrive in one burst after the guardrails have read the answer.

## Troubleshooting

- **`AuthenticationError` on every call.** Check the key and that `api_base` ends in `/public/v1`. The same error with a key that works elsewhere means the key lacks the **Chat** scope: the gateway reports a missing scope as an invalid key.
- **`RateLimitError`.** `RATE_LIMITED` in the body is the key's own per-minute or per-day limit: wait for `X-RateLimit-Reset`. `rate_limit_exceeded` is a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway's `Retry-After` is not relayed, so wait for the next clock minute. See [How limits combine](/developers/guides/how-limits-combine).
- **The request went to the completions route instead of chat completions.** `is_chat_model` defaults to false. Set it to True for conversational models.
- **An index build fails while embedding.** The default embedding model is OpenAI's. Set `Settings.embed_model` as described above before building the index.
- **Long prompts are truncated or the packer complains about the window.** `context_window` is still the class default. Set it to the model's limit from the models route.

## Related

- [Use an OpenAI client library](/developers/guides/use-an-openai-client-library)
- [Embeddings and rerank](/developers/guides/embeddings-and-rerank)
- [LangChain](/developers/integrations/langchain)
- [Integrations](/developers/integrations/overview)

Source: https://docs.katonic.ai/developers/integrations/llamaindex
