LangChain
Public API
LangChain's ChatOpenAI class speaks the chat completions protocol, and your tenant's public API answers it. Construct the model with a base URL and a key, and every chain, agent or graph that takes a chat model runs through your tenant's AI Gateway, with its tiers, guardrails, budgets and logs, and without a provider key of its own.
Who can use it: anyone holding an API key with the Chat scope. The tenant must be licensed for the public API.
Prerequisites
- Python 3.10 or later and
pip install langchain-openai. Verified withlangchain-openai1.6.2 onlangchain-core1.6.3. - Your tenant's public API address,
https://<your tenant host>/public/v1, from the top of Studio's Developer Hub. - A key from the Developer Hub's API keys tab, and a model id your tenant serves. The models route lists them; Use an OpenAI client library shows the call.
Connect
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="<a model id your tenant serves>",
base_url="https://<your tenant host>/public/v1",
api_key="kapi_YOUR_KEY",
max_tokens=300,
)
reply = llm.invoke("Summarise our onboarding guide in three bullet points.")
print(reply.content)
print(reply.response_metadata["model_name"], reply.usage_metadata["output_tokens"])
invoke accepts a string, a list of messages or a prompt value and returns an AIMessage. Always pass model: the class's default is one of OpenAI's model names, which your tenant may not serve. The object drops into any place LangChain expects a chat model, for example a prompt piped into it:
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_messages([
("system", "You answer in one sentence."),
("human", "{question}"),
])
chain = prompt | llm
print(chain.invoke({"question": "What does the onboarding guide cover?"}).content)
To record the usage against one of your agents in analytics, add default_headers={"X-Agent-ID": "<agent id>"} to the constructor.
Stream a reply
for chunk in llm.stream("Write a short welcome note for new starters."):
print(chunk.content, end="", flush=True)
The chunks arrive in one burst, not as the model produces them: the gateway holds the whole answer so the output guardrails can read it, then replays the chunks. Stream for the event shape, not for a typing effect. Some chunks carry no content, the first with only a role and the last with only metadata, so a consumer that concatenates chunk.content gets the whole answer without special cases.
Verify
reply.response_metadatanames the model that answered andreply.usage_metadatacarries the token counts, as the gateway returned them.- The key's row on the Developer Hub's API keys tab shows a new last-used time.
- Each call appears in the Control Room's AI Gateway under Logs as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.
What does not carry over
- Tools on a first turn. After
bind_tools, a first turn whose tool choice is absent orautois upgraded torequired, so the model must call a tool;noneand a named tool are respected, and later turns are not upgraded. An agent loop built on the model behaves accordingly. See Call tools from a completion. userandmetadata. Both are accepted and discarded; attribute usage with theX-Agent-IDheader instead.- Streaming pace. Chunks arrive in one burst after the guardrails have read the answer.
- Only the model routes. Chat completions, completions, embeddings, rerank, image generation and the models list are served in the OpenAI shapes; nothing else on OpenAI's API is. Embeddings and rerank covers the embedding models your tenant serves.
Troubleshooting
AuthenticationErroron every call. Check the key and that the base URL ends in/public/v1. The same error with a key that works elsewhere means the key lacks the Chat scope: the gateway reports a missing scope as an invalid key.RateLimitError.RATE_LIMITEDin the body is the key's own per-minute or per-day limit: wait forX-RateLimit-Reset.rate_limit_exceededis a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway'sRetry-Afteris not relayed, so wait for the next clock minute. See How limits combine.- An error names a model you did not choose.
modelwas omitted and the class used its own default. Pass a model id your tenant serves. - Errors surface late while debugging. The class retries transient failures twice before raising. Construct it with
max_retries=0while you diagnose a connection, then restore the default. - A chain works in one environment and fails with a connection error in another. The host is wrong or a proxy blocks it.
curl https://<your tenant host>/public/v1/modelswith the key in anAuthorizationheader should return JSON.