Skip to main content

LlamaIndex

Public API

LlamaIndex reaches OpenAI-compatible servers through the OpenAILike class in the llama-index-llms-openai-like package. Give it your tenant's base URL, a key and a model id, mark it as a chat model and set it as the default LLM; every query engine, chat engine and agent built afterwards answers through your tenant's AI Gateway.

Who can use it: anyone holding an API key with the Chat scope. The tenant must be licensed for the public API.

Prerequisites​

  • Python 3.10 or later and pip install llama-index-llms-openai-like, which brings llama-index-core with it. Verified with llama-index-llms-openai-like 0.8.0 on llama-index-core 0.14.24.
  • Your tenant's public API address, https://<your tenant host>/public/v1, from the top of Studio's Developer Hub.
  • A key from the Developer Hub's API keys tab, and a model id your tenant serves. The models route lists them; Use an OpenAI client library shows the call.

Connect​

from llama_index.core import Settings
from llama_index.core.llms import ChatMessage
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
model="<a model id your tenant serves>",
api_base="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
is_chat_model=True,
max_tokens=300,
)
Settings.llm = llm

reply = llm.complete("Summarise our onboarding guide in three bullet points.")
print(reply.text)

turn = llm.chat([ChatMessage(role="user", content="What does the onboarding guide cover?")])
print(turn.message.content)

is_chat_model=True sends requests to the chat completions route, where your tenant serves conversational models; left off, the class calls the legacy completions route instead. Settings.llm makes the model the default for everything built afterwards, so a VectorStoreIndex query engine or a function-calling agent needs no further wiring for the LLM itself. Set context_window to the model's limit, which the models route reports for each model, so LlamaIndex packs prompts to the real window rather than the class's small default.

To record the usage against one of your agents in analytics, add default_headers={"X-Agent-ID": "<agent id>"} to the constructor.

Stream a reply​

for chunk in llm.stream_complete("Write a short welcome note for new starters."):
print(chunk.delta or "", end="", flush=True)

Each chunk carries the text so far in text and the new part in delta. The chunks arrive in one burst, not as the model produces them: the gateway holds the whole answer so the output guardrails can read it, then replays the chunks. Stream for the event shape, not for a typing effect.

Verify​

  1. reply.raw is the gateway's response as the underlying client parsed it: its model names the model that answered and its usage carries the token counts.
  2. The key's row on the Developer Hub's API keys tab shows a new last-used time.
  3. Each call appears in the Control Room's AI Gateway under Logs as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.

What does not carry over​

  • Embeddings. LlamaIndex's default embedding model is OpenAI's own, reached at OpenAI's address, so an index built with defaults fails when the only key you hold is your tenant's. Set Settings.embed_model to an embedding model you run yourself, or to a model your tenant serves through an OpenAI-compatible embedding class pointed at the same base URL and key. Embeddings and rerank names the route and the models.
  • Tools on a first turn. A function-calling agent binds tools, and the gateway upgrades a first turn whose tool choice is absent or auto to required, so the model must call a tool; later turns are not upgraded. See Call tools from a completion.
  • user and metadata. Both are accepted and discarded; attribute usage with the X-Agent-ID header instead.
  • Streaming pace. Chunks arrive in one burst after the guardrails have read the answer.

Troubleshooting​

  • AuthenticationError on every call. Check the key and that api_base ends in /public/v1. The same error with a key that works elsewhere means the key lacks the Chat scope: the gateway reports a missing scope as an invalid key.
  • RateLimitError. RATE_LIMITED in the body is the key's own per-minute or per-day limit: wait for X-RateLimit-Reset. rate_limit_exceeded is a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway's Retry-After is not relayed, so wait for the next clock minute. See How limits combine.
  • The request went to the completions route instead of chat completions. is_chat_model defaults to false. Set it to True for conversational models.
  • An index build fails while embedding. The default embedding model is OpenAI's. Set Settings.embed_model as described above before building the index.
  • Long prompts are truncated or the packer complains about the window. context_window is still the class default. Set it to the model's limit from the models route.

Describes the platform as of .

Was this page helpful?