Stream a reply
Two public routes stream, and they stream differently. Chat completions sends OpenAI chunk events as the model writes them, unless your organisation's guardrail profile must read the whole answer first. The agent chat route relays the agent's run as it happens, as typed events that cover text, tool calls, steps and receipts. Both use server-sent events with data: lines only and end with data: [DONE].
Who can use it: anyone holding an API key with the Chat scope; text completions stream the same way as chat completions under the Completions scope.
Which stream you get
POST /public/v1/chat/completions with stream | POST /public/v1/chat with stream | |
|---|---|---|
| Timing | As the model writes; generated, checked, then sent at once when the guardrail profile has output rails | Relayed as the agent produces it |
| Event shape | OpenAI chat.completion.chunk objects | Objects with a type field |
| Text | choices[0].delta.content | TEXT_MESSAGE_CONTENT events with delta |
| A failure after the start | One chunk with finish_reason set to error and an error object, then [DONE]; status stays 200 | A RUN_ERROR event with error, error_code and retryable; status stays 200 |
| Usage | A late chunk, usually the last, with a usage object; its choices holds one entry with an empty delta | RUN_FINISHED with a usage object |
| Heartbeats | None | None |
| Does not stream | POST /public/v1/<namespace>/<slug>/chat ignores stream |
Read a completion stream
Each event is a chunk as the provider shaped it, stamped with model, served_model and served_provider, plus is_fallback, fallback_from and fallback_reason on the first when the tier's primary did not serve. A thinking model (the Qwen3, QwQ and DeepSeek-R1 families, not their instruct or coder variants) streams its reasoning live on delta.reasoning_content from its first token, then the answer on delta.content; the </think> tag never appears in the answer. Tool calls arrive as delta.tool_calls with index and function filled in. The last data chunk before [DONE] has empty choices and the usage object; the gateway asks every provider that can send it and synthesises one for self-hosted engines. Anthropic and Bedrock are not asked, so it can be absent there.
- cURL
- Python
- TypeScript
curl -N -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"tier": "fast", "stream": true, "messages": [{"role": "user", "content": "Three tips for a first day at work."}]}'
import json, requests
BASE = "https://<your tenant host>/public/v1"
HEADERS = {"Authorization": "Bearer kapi_..."}
with requests.post(f"{BASE}/chat/completions", headers=HEADERS, stream=True, json={
"tier": "fast", "stream": True,
"messages": [{"role": "user", "content": "Three tips for a first day at work."}],
}) as r:
r.raise_for_status()
for line in r.iter_lines():
if not line.startswith(b"data: "):
continue
payload = line[6:]
if payload == b"[DONE]":
break
chunk = json.loads(payload)
if "error" in chunk: # failure after the start; status was 200
raise RuntimeError(chunk["error"]["message"])
if chunk.get("usage"):
print("\nusage:", chunk["usage"], "served by", chunk["served_model"])
for choice in chunk.get("choices", []):
print(choice["delta"].get("content") or "", end="", flush=True)
const BASE = "https://<your tenant host>/public/v1";
const headers = { Authorization: "Bearer kapi_...", "Content-Type": "application/json" };
const r = await fetch(`${BASE}/chat/completions`, {
method: "POST", headers,
body: JSON.stringify({ tier: "fast", stream: true, messages: [{ role: "user", content: "Three tips for a first day at work." }] }),
});
if (!r.ok || !r.body) throw new Error(`${r.status} ${await r.text()}`);
const reader = r.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";
for (;;) {
const { value, done } = await reader.read();
if (done) break;
buffer += value;
let nl: number;
while ((nl = buffer.indexOf("\n\n")) >= 0) {
const frame = buffer.slice(0, nl); buffer = buffer.slice(nl + 2);
if (!frame.startsWith("data: ")) continue;
const payload = frame.slice(6);
if (payload === "[DONE]") break;
const chunk = JSON.parse(payload);
if (chunk.error) throw new Error(chunk.error.message); // failure after the start; status was 200
if (chunk.usage) console.log("\nusage:", chunk.usage, "served by", chunk.served_model);
for (const c of chunk.choices ?? []) process.stdout.write(c.delta?.content ?? "");
}
}
A failure after the start is a chunk with finish_reason error and an error with message and type, plus code STREAM_ERROR when the gateway failed, then [DONE]. Before the first token, reasoning included, the gateway can still move to another model in the tier; after it, a failure ends the stream.
When the answer is held
Chunks leave as written only when the guardrail profile has no output rails: no output checks and no secret redaction. Otherwise, or when the profile cannot be read, the gateway reads the whole answer, redacts or blocks it, and sends the chunks together. On AI Cloud each organisation's profile decides.
An input rail that blocks the question sends one chunk with the refusal text, finish_reason content_filter and a guardrail object with direction input, then [DONE]. A zero usage chunk follows only if you asked for usage in stream_options. guardrail.rails never includes the matched value.
Read an agent stream
Every event carries type and timestamp; fields that are not set are absent. A normal turn runs RUN_STARTED, then text and tool events, then STEP_STARTED and STEP_FINISHED pairs, then RUN_FINISHED, then [DONE].
| Event | Fields | Meaning |
|---|---|---|
RUN_STARTED | run_id, thread_id | The turn was admitted; thread_id is the session's canonical id |
TEXT_MESSAGE_START, TEXT_MESSAGE_CONTENT, TEXT_MESSAGE_END | message_id, role; delta | The reply, in word-sized deltas |
TOOL_CALL_START, TOOL_CALL_ARGS, TOOL_CALL_RESULT, TOOL_CALL_END | tool_call_id, tool_call_name; args as a string; result as a string with is_error, truncated and duration_ms | A tool the agent used |
STEP_STARTED, STEP_FINISHED | step_name, step_index, step_type, tokens_input, tokens_output, step_latency_ms | One model step each, after the text |
CUSTOM | name, value | sources (the citations), knowledge_retrieval, guardrail_block, guardrail_redaction, guardrail_defense, receipt, ui.block_limit |
STATE_DELTA, ARTIFACT_CREATED, ARTIFACT_UPDATED, ACTIVITY_SNAPSHOT | state; name and value; activity | Agents that keep state or write artifacts |
RUN_FINISHED | usage with prompt_tokens, completion_tokens, total_tokens, model, cost, estimated | The turn succeeded |
RUN_ERROR | error, error_code, retryable, and run_id and thread_id when known | The turn failed. error_code is one of provider_failure, gateway, aborted, agent_not_found, model_not_vision_capable, agent_type_cannot_read_images or unknown |
- cURL
- Python
- TypeScript
curl -N -X POST "https://<your tenant host>/public/v1/chat" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"agent_id": "support-assistant", "session_id": "<session-id>", "stream": true,
"messages": [{"role": "user", "content": "And for annual plans?"}]}'
with requests.post(f"{BASE}/chat", headers=HEADERS, stream=True, json={
"agent_id": "support-assistant", "session_id": "<session-id>", "stream": True,
"messages": [{"role": "user", "content": "And for annual plans?"}],
}) as r:
r.raise_for_status()
for line in r.iter_lines(): # read to the end, not to [DONE]
if not line.startswith(b"data: ") or line == b"data: [DONE]":
continue
event = json.loads(line[6:])
kind = event["type"]
if kind == "TEXT_MESSAGE_CONTENT":
print(event["delta"], end="", flush=True)
elif kind == "CUSTOM" and event["name"] == "sources":
print("\nsources:", event["value"])
elif kind == "RUN_FINISHED":
print("\nusage:", event.get("usage"))
elif kind == "RUN_ERROR":
raise RuntimeError(f"{event['error_code']}: {event['error']} (retryable={event.get('retryable')})")
const r = await fetch(`${BASE}/chat`, {
method: "POST", headers,
body: JSON.stringify({ agent_id: "support-assistant", session_id: "<session-id>", stream: true,
messages: [{ role: "user", content: "And for annual plans?" }] }),
});
if (!r.ok || !r.body) throw new Error(`${r.status} ${await r.text()}`);
const reader = r.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";
for (;;) { // read to the end, not to [DONE]
const { value, done } = await reader.read();
if (done) break;
buffer += value;
let nl: number;
while ((nl = buffer.indexOf("\n\n")) >= 0) {
const frame = buffer.slice(0, nl); buffer = buffer.slice(nl + 2);
if (!frame.startsWith("data: ") || frame === "data: [DONE]") continue;
const event = JSON.parse(frame.slice(6));
if (event.type === "TEXT_MESSAGE_CONTENT") process.stdout.write(event.delta);
else if (event.type === "CUSTOM" && event.name === "sources") console.log("\nsources:", event.value);
else if (event.type === "RUN_FINISHED") console.log("\nusage:", event.usage);
else if (event.type === "RUN_ERROR") throw new Error(`${event.error_code}: ${event.error}`);
}
}
What a strict client needs
- Only
data:lines, separated by a blank line. There are noevent:orid:fields, and[DONE]is not JSON. - No heartbeats on either route, so set the read timeout by the route. A completion stream held by output rails is silent until the whole answer is ready; the gateway allows up to fifteen seconds to the first token for chat models, longer for reasoning models, and two minutes of idle after that. An agent stream can pause during a tool call, and the agent's own limit on a turn is about five minutes.
- The status is 200 once the stream has started. Treat a chunk with
finish_reasonset toerror, or aRUN_ERRORevent, as the failure. - On the agent route, read until the connection closes, not to
[DONE]: an answer that produced reasoning and no text sends aRUN_ERRORafter the terminal frame. A failure before the run is admitted is aRUN_ERRORwith noRUN_STARTED. - On the agent route an input refusal streams as ordinary text with zero usage, and an output block arrives as a
CUSTOMevent namedguardrail_blockwith the replacement text. - A service that stops sending for five minutes is cut without an error event. Treat a stream that ends without
RUN_FINISHEDor[DONE]as failed.
Troubleshooting
- Every completion chunk arrives at the same moment: the guardrail profile has output rails, so the gateway checks the whole answer before sending it. That is by design.
- A chunk carries
errorand the request returned 200: the provider failed after the stream started. Retry the call. RUN_ERRORwithagent_not_found: theagent_iddoes not exist; the buffered form of the same call answers 404.RUN_ERRORwithgatewayafter a long pause: the agent's turn ran out of time. Shorten the question or the tools involved.- No events at all from the registry-path route: it ignores
stream. UsePOST /public/v1/chatwithagent_idfor a stream.
The same reply elsewhere