Skip to main content

Structured outputs

response_format takes the OpenAI shapes, {"type": "json_object"} and {"type": "json_schema", "json_schema": {...}}, and the gateway forwards it to the provider exactly as sent. It does not translate between providers, validate the answer against your schema, or repair broken JSON, and for a model that does not support the field it drops it without a word. So there are two jobs on your side: pick a model that supports it, and validate what comes back.

Who can use it: anyone holding an API key with the Chat scope, and Read models to check a model's capabilities first.

Which models honour it​

The models route reports a capabilities object per model. Its json flag comes from a static table of what each model supports, not from a live check, and is left out when the table does not know.

capabilities.jsonWhat to do
trueSend response_format, with a schema if you have one, and validate the result.
falseThe field will be dropped. Ask for JSON in the prompt and validate harder.
absentUnknown. Try it and validate; treat prose as the field having been dropped.

Pin the model for anything that depends on this. A tier can fall back to another model on the next call, and the fallback may not support the field; served_model says which one answered.

Quick start​

Ask for a fixed shape and check it before use:

curl -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"model": "<your-model-id>",
"messages": [{"role": "user", "content": "Extract the fields from: Invoice 4471 from Northwind, total 1,280.50 EUR, due 30 September 2026."}],
"response_format": {"type": "json_schema", "json_schema": {"name": "invoice", "schema": {
"type": "object", "properties": {"number": {"type": "string"}, "supplier": {"type": "string"},
"total": {"type": "number"}, "currency": {"type": "string"}, "due": {"type": "string"}},
"required": ["number", "supplier", "total", "currency", "due"], "additionalProperties": false}}}}'

Limits and behaviour​

  • The gateway makes no distinction between json_object and json_schema; whatever the provider does with each is what you get. A strict flag inside json_schema reaches the provider unchanged and is enforced there or not at all.
  • Check finish_reason before parsing. An output guardrail that blocks replaces the content with a banner and sets finish_reason to content_filter; a redaction swaps text inside the JSON, which can break it. Both are announced in the guardrail object.
  • A refusal counts as an answer, not as an empty response: a model that declines to fill the schema returns a refusal and does not trigger a fallback to another model.
  • Tools and response_format together follow the tool rules: on a first turn with tools bound the model is made to call a tool, so the JSON answer arrives on the turn after the tool result. See Call tools from a completion.
  • Streaming buffers the whole answer before replaying it, so a JSON answer never arrives half-formed mid-stream; it arrives whole, in one burst, then [DONE].

Troubleshooting​

  • The answer is prose, not JSON: the served model does not support the field and it was dropped. Read served_model, check its capabilities.json, and pin a model that has it.
  • The JSON does not match the schema: nothing in the gateway enforces it. Validate, and on failure send the answer back with the validation error as the next user message.
  • finish_reason is content_filter: a guardrail blocked the answer; the guardrail object names the rail. This is a policy outcome, not a formatting one.
  • The same request works on one call and not the next: a tier fallback served the second call. Pin the model.

The same call elsewhere

Describes the platform as of .

Was this page helpful?