Structured outputs
response_format takes the OpenAI shapes, {"type": "json_object"} and {"type": "json_schema", "json_schema": {...}}, and the gateway forwards it to the provider exactly as sent. It does not translate between providers, validate the answer against your schema, or repair broken JSON, and for a model that does not support the field it drops it without a word. So there are two jobs on your side: pick a model that supports it, and validate what comes back.
Who can use it: anyone holding an API key with the Chat scope, and Read models to check a model's capabilities first.
Which models honour it
The models route reports a capabilities object per model. Its json flag comes from a static table of what each model supports, not from a live check, and is left out when the table does not know.
capabilities.json | What to do |
|---|---|
true | Send response_format, with a schema if you have one, and validate the result. |
false | The field will be dropped. Ask for JSON in the prompt and validate harder. |
| absent | Unknown. Try it and validate; treat prose as the field having been dropped. |
Pin the model for anything that depends on this. A tier can fall back to another model on the next call, and the fallback may not support the field; served_model says which one answered.
Quick start
Ask for a fixed shape and check it before use:
- cURL
- Python
- TypeScript
curl -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"model": "<your-model-id>",
"messages": [{"role": "user", "content": "Extract the fields from: Invoice 4471 from Northwind, total 1,280.50 EUR, due 30 September 2026."}],
"response_format": {"type": "json_schema", "json_schema": {"name": "invoice", "schema": {
"type": "object", "properties": {"number": {"type": "string"}, "supplier": {"type": "string"},
"total": {"type": "number"}, "currency": {"type": "string"}, "due": {"type": "string"}},
"required": ["number", "supplier", "total", "currency", "due"], "additionalProperties": false}}}}'
import json, requests
BASE = "https://<your tenant host>/public/v1"
HEADERS = {"Authorization": "Bearer kapi_..."}
SCHEMA = {"type": "object", "additionalProperties": False,
"properties": {"number": {"type": "string"}, "supplier": {"type": "string"}, "total": {"type": "number"},
"currency": {"type": "string"}, "due": {"type": "string"}},
"required": ["number", "supplier", "total", "currency", "due"]}
r = requests.post(f"{BASE}/chat/completions", headers=HEADERS, json={
"model": "<your-model-id>",
"messages": [{"role": "user", "content": "Extract the fields from: Invoice 4471 from Northwind, total 1,280.50 EUR, due 30 September 2026."}],
"response_format": {"type": "json_schema", "json_schema": {"name": "invoice", "schema": SCHEMA}},
}).json()
choice = r["choices"][0]
if choice["finish_reason"] == "content_filter":
raise RuntimeError(f"blocked by guardrails: {r.get('guardrail')}")
invoice = json.loads(choice["message"]["content"]) # raises on prose or broken JSON
missing = [k for k in SCHEMA["required"] if k not in invoice]
if missing:
raise ValueError(f"model answered without {missing}; served by {r['served_model']}")
print(invoice)
const BASE = "https://<your tenant host>/public/v1";
const headers = { Authorization: "Bearer kapi_...", "Content-Type": "application/json" };
const schema = { type: "object", additionalProperties: false,
properties: { number: { type: "string" }, supplier: { type: "string" }, total: { type: "number" },
currency: { type: "string" }, due: { type: "string" } },
required: ["number", "supplier", "total", "currency", "due"] };
const r = await (await fetch(`${BASE}/chat/completions`, {
method: "POST", headers,
body: JSON.stringify({
model: "<your-model-id>",
messages: [{ role: "user", content: "Extract the fields from: Invoice 4471 from Northwind, total 1,280.50 EUR, due 30 September 2026." }],
response_format: { type: "json_schema", json_schema: { name: "invoice", schema } },
}),
})).json();
const choice = r.choices[0];
if (choice.finish_reason === "content_filter") throw new Error(`blocked by guardrails: ${JSON.stringify(r.guardrail)}`);
const invoice = JSON.parse(choice.message.content); // throws on prose or broken JSON
const missing = schema.required.filter((k) => !(k in invoice));
if (missing.length) throw new Error(`model answered without ${missing.join(", ")}; served by ${r.served_model}`);
console.log(invoice);
Limits and behaviour
- The gateway makes no distinction between
json_objectandjson_schema; whatever the provider does with each is what you get. Astrictflag insidejson_schemareaches the provider unchanged and is enforced there or not at all. - Check
finish_reasonbefore parsing. An output guardrail that blocks replaces the content with a banner and setsfinish_reasontocontent_filter; a redaction swaps text inside the JSON, which can break it. Both are announced in theguardrailobject. - A refusal counts as an answer, not as an empty response: a model that declines to fill the schema returns a
refusaland does not trigger a fallback to another model. - Tools and
response_formattogether follow the tool rules: on a first turn with tools bound the model is made to call a tool, so the JSON answer arrives on the turn after the tool result. See Call tools from a completion. - Streaming buffers the whole answer before replaying it, so a JSON answer never arrives half-formed mid-stream; it arrives whole, in one burst, then
[DONE].
Troubleshooting
- The answer is prose, not JSON: the served model does not support the field and it was dropped. Read
served_model, check itscapabilities.json, and pin a model that has it. - The JSON does not match the schema: nothing in the gateway enforces it. Validate, and on failure send the answer back with the validation error as the next user message.
finish_reasoniscontent_filter: a guardrail blocked the answer; theguardrailobject names the rail. This is a policy outcome, not a formatting one.- The same request works on one call and not the next: a tier fallback served the second call. Pin the model.
The same call elsewhere