Skip to main content

LangChain

Public API

LangChain's ChatOpenAI class speaks the chat completions protocol, and your tenant's public API answers it. Construct the model with a base URL and a key, and every chain, agent or graph that takes a chat model runs through your tenant's AI Gateway, with its tiers, guardrails, budgets and logs, and without a provider key of its own.

Who can use it: anyone holding an API key with the Chat scope. The tenant must be licensed for the public API.

Prerequisites​

  • Python 3.10 or later and pip install langchain-openai. Verified with langchain-openai 1.6.2 on langchain-core 1.6.3.
  • Your tenant's public API address, https://<your tenant host>/public/v1, from the top of Studio's Developer Hub.
  • A key from the Developer Hub's API keys tab, and a model id your tenant serves. The models route lists them; Use an OpenAI client library shows the call.

Connect​

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
model="<a model id your tenant serves>",
base_url="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
max_tokens=300,
)

reply = llm.invoke("Summarise our onboarding guide in three bullet points.")
print(reply.content)
print(reply.response_metadata["model_name"], reply.usage_metadata["output_tokens"])

invoke accepts a string, a list of messages or a prompt value and returns an AIMessage. Always pass model: the class's default is one of OpenAI's model names, which your tenant may not serve. The object drops into any place LangChain expects a chat model, for example a prompt piped into it:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
("system", "You answer in one sentence."),
("human", "{question}"),
])
chain = prompt | llm
print(chain.invoke({"question": "What does the onboarding guide cover?"}).content)

To record the usage against one of your agents in analytics, add default_headers={"X-Agent-ID": "<agent id>"} to the constructor.

Stream a reply​

for chunk in llm.stream("Write a short welcome note for new starters."):
print(chunk.content, end="", flush=True)

The chunks arrive in one burst, not as the model produces them: the gateway holds the whole answer so the output guardrails can read it, then replays the chunks. Stream for the event shape, not for a typing effect. Some chunks carry no content, the first with only a role and the last with only metadata, so a consumer that concatenates chunk.content gets the whole answer without special cases.

Verify​

  1. reply.response_metadata names the model that answered and reply.usage_metadata carries the token counts, as the gateway returned them.
  2. The key's row on the Developer Hub's API keys tab shows a new last-used time.
  3. Each call appears in the Control Room's AI Gateway under Logs as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.

What does not carry over​

  • Tools on a first turn. After bind_tools, a first turn whose tool choice is absent or auto is upgraded to required, so the model must call a tool; none and a named tool are respected, and later turns are not upgraded. An agent loop built on the model behaves accordingly. See Call tools from a completion.
  • user and metadata. Both are accepted and discarded; attribute usage with the X-Agent-ID header instead.
  • Streaming pace. Chunks arrive in one burst after the guardrails have read the answer.
  • Only the model routes. Chat completions, completions, embeddings, rerank, image generation and the models list are served in the OpenAI shapes; nothing else on OpenAI's API is. Embeddings and rerank covers the embedding models your tenant serves.

Troubleshooting​

  • AuthenticationError on every call. Check the key and that the base URL ends in /public/v1. The same error with a key that works elsewhere means the key lacks the Chat scope: the gateway reports a missing scope as an invalid key.
  • RateLimitError. RATE_LIMITED in the body is the key's own per-minute or per-day limit: wait for X-RateLimit-Reset. rate_limit_exceeded is a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway's Retry-After is not relayed, so wait for the next clock minute. See How limits combine.
  • An error names a model you did not choose. model was omitted and the class used its own default. Pass a model id your tenant serves.
  • Errors surface late while debugging. The class retries transient failures twice before raising. Construct it with max_retries=0 while you diagnose a connection, then restore the default.
  • A chain works in one environment and fails with a connection error in another. The host is wrong or a proxy blocks it. curl https://<your tenant host>/public/v1/models with the key in an Authorization header should return JSON.

Describes the platform as of .

Was this page helpful?