Skip to main content

Call tools from a completion

The chat completions route accepts OpenAI-shaped tools: you describe functions, the model answers with tool calls instead of text, you run them and send the results back for the final answer. The gateway forwards the shapes but is not a pass-through. It decides when the model must call a tool, strips tools a provider rejects, repairs malformed calls from some open models, and says what it changed in response headers.

Who can use it: anyone holding an API key with the Chat scope. These are your own functions, run by your code; the platform's governed tool catalogue is a separate route, covered in Execute a governed tool.

What the gateway changes​

You sendWhat happens
tools, up to 128Forwarded. If the provider answers that the model cannot use tools, the call is retried once without tools, tool_choice and their relatives. The gateway marks such a response with X-Tools-Stripped, a header the public edge does not relay on the current build, so through the public API the only sign is an answer without tool_calls.
tool_choice absent or autoOn the first turn, with real tools bound, upgraded to required: the model must call a tool. Relaxed back to auto once the conversation holds a tool result or an assistant turn in the current exchange, and for greetings.
tool_choice: "required"Kept, with the same relaxations.
tool_choice: "none" or a named functionRespected as sent.
parallel_tool_callsDropped; the gateway does not know the field.
A tool message with tool_call_idForwarded. Several system messages are merged into one leading system message for every provider.

Self-hosted engines that use guided decoding never receive a forced choice; they get auto. A model that refuses a forced choice gets auto with X-Tool-Choice-Relaxed: model_rejects_forced_choice, and a provider that rejects the value gets auto on a retry, disclosed as X-Tool-Choice-Relaxed: provider_rejected.

Quick start​

Define one function, let the model call it, run it, and send the result back:

curl -X POST "https://<your tenant host>/public/v1/chat/completions" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"tier": "balanced",
"messages": [{"role": "user", "content": "Is order 4471 shipped?"}],
"tools": [{"type": "function", "function": {"name": "order_status",
"description": "Look up an order by id",
"parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}}]}'

The reply carries choices[0].message.tool_calls; run the function and send the whole exchange back with a tool message carrying the same tool_call_id.

Read the response​

A turn that calls tools has finish_reason set to tool_calls and choices[0].message.tool_calls, each with an id, type of function, and function.name and function.arguments, the arguments as a JSON string to parse. On a stream the tool-call deltas pass through with their index and function filled in; the whole answer is buffered and replayed anyway, as every stream on this route is. The characters of the arguments count toward completion_tokens.

For several open-weight model families the gateway repairs a call the model wrote as text into a structured call. When a streamed call cannot be repaired, the gateway drops it and appends a notice to the content instead of failing the turn, so parse tool_calls, not the text.

Limits and behaviour​

  • With tools bound, a first-turn question is answered with a tool call, not text, unless you send tool_choice as none or name a function. Send none for turns where a text answer is wanted.
  • A model answer with no content, no tool call and no refusal counts as a failure and moves the call to the next candidate in the tier.
  • The gateway never runs a function. Whatever tool_calls names, your code executes it and decides what to send back.
  • The model may call a tool that is not in your list only if the provider does; the gateway does not filter the names on the way back.
  • The output guardrails read the text of the final answer, not the tool arguments; the tool rails of a guardrail profile apply to the platform's governed tools, not to these.

Troubleshooting​

  • The model always calls a tool on the first turn, even for small talk: the upgrade to required. Send tool_choice as none for those turns.
  • tool_calls is missing although you sent tools: the served model cannot use tools and the call was retried without them; the X-Tools-Stripped header that says so does not reach public API callers on the current build. Pin a model whose capabilities.tools is true on the models route.
  • A forced tool_choice came back as text: the model or the provider refused the forced choice and was given auto. The gateway records this in an X-Tool-Choice-Relaxed header that is not relayed on the current build, so handle both shapes.
  • function.arguments is not valid JSON: the model wrote it badly and the repair did not apply. Validate before running anything and send the error back as the tool result so the model can retry.
  • Several system prompts behave as one: they are merged before the provider sees them; put per-turn instructions in a user message.

The same job elsewhere

Describes the platform as of .

Was this page helpful?