Vercel AI SDK
Public API
The Vercel AI SDK talks to any OpenAI-compatible server through its @ai-sdk/openai-compatible provider. Create the provider with your tenant's base URL and the key in a header, take a chat model from it, and generateText and streamText run through your tenant's AI Gateway from a route handler, a worker or a script.
Who can use it: anyone holding an API key with the Chat scope. The tenant must be licensed for the public API.
Prerequisites
- Node.js 18 or later and
npm install ai @ai-sdk/openai-compatible. Verified withai7.0.99 and@ai-sdk/openai-compatible3.0.48. - Your tenant's public API address,
https://<your tenant host>/public/v1, from the top of Studio's Developer Hub. - A key from the Developer Hub's API keys tab, and a model id your tenant serves. The models route lists them; Use an OpenAI client library shows the call.
Connect
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { generateText } from "ai";
const katonic = createOpenAICompatible({
name: "katonic",
baseURL: "https://<your tenant host>/public/v1",
headers: { Authorization: `Bearer ${process.env.KATONIC_API_KEY}` },
});
const model = katonic.chatModel("<a model id your tenant serves>");
const result = await generateText({
model,
prompt: "Summarise our onboarding guide in three bullet points.",
maxOutputTokens: 300,
});
console.log(result.text);
console.log(result.usage.outputTokens, "completion tokens");
Use this provider rather than @ai-sdk/openai: that package's default model class calls OpenAI's Responses route, which the public API does not serve. name labels the provider in the SDK's metadata, and the model id is passed through to your tenant unchanged. The file must be a module ("type": "module" in package.json, or a .mjs extension) for the top-level await.
Stream a reply
import { streamText } from "ai";
const stream = streamText({
model,
prompt: "Write a short welcome note for new starters.",
maxOutputTokens: 300,
});
for await (const part of stream.textStream) {
process.stdout.write(part);
}
In a route handler, hand the result to the SDK's response helpers instead of iterating it yourself. The parts arrive in one burst, not as the model produces them: the gateway holds the whole answer so the output guardrails can read it, then replays the chunks. Stream for the event shape, not for a typing effect.
Verify
result.textis the model's answer andresult.usage.outputTokenscarries the completion tokens the gateway counted.- The key's row on the Developer Hub's API keys tab shows a new last-used time.
- Each call appears in the Control Room's AI Gateway under Logs as a line naming the provider, model, tier, tokens, cost and latency. That line does not name the key or the user.
What does not carry over
- The OpenAI provider's defaults.
@ai-sdk/openaiassumes OpenAI's Responses route; the public API serves chat completions, completions, embeddings, rerank, image generation and the models list in the OpenAI shapes, so stay on the OpenAI-compatible provider. - Tools on a first turn. With
toolspassed togenerateTextorstreamTextand no tool choice set, the first turn is upgraded torequired, so the model must call a tool; later turns are not upgraded. See Call tools from a completion. - Streaming pace. Parts arrive in one burst after the guardrails have read the answer.
- The browser. The provider runs anywhere
fetchdoes, but the key would then live in the page. Keep it in a route handler or server component and let your own backend hold the key.
Troubleshooting
APICallErrorwith status 401. Check the header and that the base URL ends in/public/v1. The same error with a key that works elsewhere means the key lacks the Chat scope: the gateway reports a missing scope as an invalid key.APICallErrorwith status 429.RATE_LIMITEDin the body is the key's own per-minute or per-day limit: wait forX-RateLimit-Reset.rate_limit_exceededis a per-user, per-tenant or per-model limit inside the AI Gateway: the gateway'sRetry-Afteris not relayed, so wait for the next clock minute. See How limits combine.Invalid JSON response. The address returned something other than JSON, usually a sign-in or error page from the wrong host or path. The base URL must end in/public/v1.- A 404 from
@ai-sdk/openai. Its default model class calls the Responses route. Switch to@ai-sdk/openai-compatibleas above. SyntaxError: Cannot use import statement outside a module. Add"type": "module"topackage.jsonor rename the file to.mjs.