Skip to main content

Embeddings and rerank

Two model routes serve retrieval work outside the knowledge engine. Embeddings turn text into vectors for an index you keep yourself; rerank scores a list of passages against a query. Both route by tier (embedding and reranking) or by model id, and both carry the fallback fields described in Tiers and models. If the passages already live in a Katonic knowledge source, Search retrieval options reranks for you.

Who can use it: anyone holding an API key with the Chat or the Completions scope; either reaches both routes.

The two routes​

POST /public/v1/embeddingsPOST /public/v1/rerank
Inputinput: one string or a list of stringsquery, and documents as a list of strings
Model choicemodel, or tier (default embedding)model, or tier (default reranking)
Optionsencoding_format, dimensions, input_typetop_n
ResponseThe OpenAI embedding list, data[].embedding and usage, plus tier and providerresults[] with index and relevance_score, plus model, tier and provider
What is recordedTokens and spendNothing: the route records no tokens and no spend

Quick start​

Embed a list of passages for storage, then rerank a shortlist against a question:

curl -X POST "https://<your tenant host>/public/v1/embeddings" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"input": ["Carer leave is ten days a year.", "Annual leave accrues monthly."], "input_type": "passage"}'

curl -X POST "https://<your tenant host>/public/v1/rerank" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"query": "How much carer leave do I get?",
"documents": ["Carer leave is ten days a year.", "Annual leave accrues monthly."], "top_n": 1}'

The options​

FieldRouteWhat it does
input_typeembeddingsquery or passage, for models that embed questions and documents differently. Forwarded to the providers that distinguish them, each in its own vocabulary: NVIDIA (which assumes query when you send nothing), Cohere, Google and Voyage. Every other provider drops it.
dimensionsembeddingsForwarded when set; the provider decides whether to honour it.
encoding_formatembeddingsForwarded when set.
userembeddingsAccepted and discarded.
top_nrerankReturn only the best n results. Left out, every document comes back scored.

Embedding rows come back in the order of input. A rerank result carries the index of the document in your list and a relevance_score between 0 and 1; use index to get back to the text. Both responses carry the served_model headers, so a fallback is visible.

Limits and behaviour​

  • Pin model for anything you store. A tier fallback can answer with a different embedding model, and its vectors do not match the ones already in your index; served_model says which one answered.
  • There is no per-request cap on the number of inputs or documents beyond the 64 MiB body limit. Provider limits still apply and come back as provider errors.
  • Embedding tokens count toward the AI Gateway's tokens-per-minute limit and are priced into budgets. Rerank calls are checked against budgets before they run but add nothing to them, because the route returns no token count.
  • The gateway's per-model rate limit is keyed on the tier name when no model is named.
  • An empty data list from a provider is treated as a failure and the next candidate in the tier is tried.
  • The checks run in the order budget, rate limit, token quota, so a call over budget is refused as budget_exceeded even when it is also over the rate limit. See How limits combine.

Troubleshooting​

  • 422 with no_models_for_tier: no model is tagged embedding or reranking for the tenant. An administrator tags one on the AI Gateway's Models page.
  • 403 with FORBIDDEN: the key has neither the Chat nor the Completions scope.
  • dimensions had no effect: the provider that served the call does not support it; check served_model and pin one that does.
  • Two embeddings of the same text differ between calls: a fallback served the second call. Read X-Fallback-Reason and pin the model.
  • A rerank relevance_score sits near 0.5 for everything: the served model returned raw logits, which the gateway squashes into the 0 to 1 range; compare scores within one call, not across models.

The same task in the platform

Describes the platform as of .

Was this page helpful?