Skip to main content

Stream a reply

Two public routes stream, and they stream differently. Chat completions sends OpenAI chunk events as the model writes them, unless your organisation's guardrail profile must read the whole answer first. The agent chat route relays the agent's run as it happens, as typed events that cover text, tool calls, steps and receipts. Both use server-sent events with data: lines only and end with data: [DONE].

Who can use it: anyone holding an API key with the Chat scope; text completions stream the same way as chat completions under the Completions scope.

Which stream you get​

POST /public/v1/chat/completions with streamPOST /public/v1/chat with stream
TimingAs the model writes; generated, checked, then sent at once when the guardrail profile has output railsRelayed as the agent produces it
Event shapeOpenAI chat.completion.chunk objectsObjects with a type field
Textchoices[0].delta.contentTEXT_MESSAGE_CONTENT events with delta
A failure after the startOne chunk with finish_reason set to error and an error object, then [DONE]; status stays 200A RUN_ERROR event with error, error_code and retryable; status stays 200
UsageA late chunk, usually the last, with a usage object; its choices holds one entry with an empty deltaRUN_FINISHED with a usage object
HeartbeatsNoneNone
Does not streamPOST /public/v1/<namespace>/<slug>/chat ignores stream

Read a completion stream​

Each event is a chunk as the provider shaped it, stamped with model, served_model and served_provider, plus is_fallback, fallback_from and fallback_reason on the first when the tier's primary did not serve. A thinking model (the Qwen3, QwQ and DeepSeek-R1 families, not their instruct or coder variants) streams its reasoning live on delta.reasoning_content from its first token, then the answer on delta.content; the </think> tag never appears in the answer. Tool calls arrive as delta.tool_calls with index and function filled in. The last data chunk before [DONE] has empty choices and the usage object; the gateway asks every provider that can send it and synthesises one for self-hosted engines. Anthropic and Bedrock are not asked, so it can be absent there.

curl -N -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"tier": "fast", "stream": true, "messages": [{"role": "user", "content": "Three tips for a first day at work."}]}'

A failure after the start is a chunk with finish_reason error and an error with message and type, plus code STREAM_ERROR when the gateway failed, then [DONE]. Before the first token, reasoning included, the gateway can still move to another model in the tier; after it, a failure ends the stream.

When the answer is held​

Chunks leave as written only when the guardrail profile has no output rails: no output checks and no secret redaction. Otherwise, or when the profile cannot be read, the gateway reads the whole answer, redacts or blocks it, and sends the chunks together. On AI Cloud each organisation's profile decides.

An input rail that blocks the question sends one chunk with the refusal text, finish_reason content_filter and a guardrail object with direction input, then [DONE]. A zero usage chunk follows only if you asked for usage in stream_options. guardrail.rails never includes the matched value.

Read an agent stream​

Every event carries type and timestamp; fields that are not set are absent. A normal turn runs RUN_STARTED, then text and tool events, then STEP_STARTED and STEP_FINISHED pairs, then RUN_FINISHED, then [DONE].

EventFieldsMeaning
RUN_STARTEDrun_id, thread_idThe turn was admitted; thread_id is the session's canonical id
TEXT_MESSAGE_START, TEXT_MESSAGE_CONTENT, TEXT_MESSAGE_ENDmessage_id, role; deltaThe reply, in word-sized deltas
TOOL_CALL_START, TOOL_CALL_ARGS, TOOL_CALL_RESULT, TOOL_CALL_ENDtool_call_id, tool_call_name; args as a string; result as a string with is_error, truncated and duration_msA tool the agent used
STEP_STARTED, STEP_FINISHEDstep_name, step_index, step_type, tokens_input, tokens_output, step_latency_msOne model step each, after the text
CUSTOMname, valuesources (the citations), knowledge_retrieval, guardrail_block, guardrail_redaction, guardrail_defense, receipt, ui.block_limit
STATE_DELTA, ARTIFACT_CREATED, ARTIFACT_UPDATED, ACTIVITY_SNAPSHOTstate; name and value; activityAgents that keep state or write artifacts
RUN_FINISHEDusage with prompt_tokens, completion_tokens, total_tokens, model, cost, estimatedThe turn succeeded
RUN_ERRORerror, error_code, retryable, and run_id and thread_id when knownThe turn failed. error_code is one of provider_failure, gateway, aborted, agent_not_found, model_not_vision_capable, agent_type_cannot_read_images or unknown
curl -N -X POST "https://<your tenant host>/public/v1/chat" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"agent_id": "support-assistant", "session_id": "<session-id>", "stream": true,
"messages": [{"role": "user", "content": "And for annual plans?"}]}'

What a strict client needs​

  • Only data: lines, separated by a blank line. There are no event: or id: fields, and [DONE] is not JSON.
  • No heartbeats on either route, so set the read timeout by the route. A completion stream held by output rails is silent until the whole answer is ready; the gateway allows up to fifteen seconds to the first token for chat models, longer for reasoning models, and two minutes of idle after that. An agent stream can pause during a tool call, and the agent's own limit on a turn is about five minutes.
  • The status is 200 once the stream has started. Treat a chunk with finish_reason set to error, or a RUN_ERROR event, as the failure.
  • On the agent route, read until the connection closes, not to [DONE]: an answer that produced reasoning and no text sends a RUN_ERROR after the terminal frame. A failure before the run is admitted is a RUN_ERROR with no RUN_STARTED.
  • On the agent route an input refusal streams as ordinary text with zero usage, and an output block arrives as a CUSTOM event named guardrail_block with the replacement text.
  • A service that stops sending for five minutes is cut without an error event. Treat a stream that ends without RUN_FINISHED or [DONE] as failed.

Troubleshooting​

  • Every completion chunk arrives at the same moment: the guardrail profile has output rails, so the gateway checks the whole answer before sending it. That is by design.
  • A chunk carries error and the request returned 200: the provider failed after the stream started. Retry the call.
  • RUN_ERROR with agent_not_found: the agent_id does not exist; the buffered form of the same call answers 404.
  • RUN_ERROR with gateway after a long pause: the agent's turn ran out of time. Shorten the question or the tools involved.
  • No events at all from the registry-path route: it ignores stream. Use POST /public/v1/chat with agent_id for a stream.

The same reply elsewhere

Describes the platform as of .

Was this page helpful?