Embeddings and rerank
Two model routes serve retrieval work outside the knowledge engine. Embeddings turn text into vectors for an index you keep yourself; rerank scores a list of passages against a query. Both route by tier (embedding and reranking) or by model id, and both carry the fallback fields described in Tiers and models. If the passages already live in a Katonic knowledge source, Search retrieval options reranks for you.
Who can use it: anyone holding an API key with the Chat or the Completions scope; either reaches both routes.
The two routes
POST /public/v1/embeddings | POST /public/v1/rerank | |
|---|---|---|
| Input | input: one string or a list of strings | query, and documents as a list of strings |
| Model choice | model, or tier (default embedding) | model, or tier (default reranking) |
| Options | encoding_format, dimensions, input_type | top_n |
| Response | The OpenAI embedding list, data[].embedding and usage, plus tier and provider | results[] with index and relevance_score, plus model, tier and provider |
| What is recorded | Tokens and spend | Nothing: the route records no tokens and no spend |
Quick start
Embed a list of passages for storage, then rerank a shortlist against a question:
- cURL
- Python
- TypeScript
curl -X POST "https://<your tenant host>/public/v1/embeddings" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"input": ["Carer leave is ten days a year.", "Annual leave accrues monthly."], "input_type": "passage"}'
curl -X POST "https://<your tenant host>/public/v1/rerank" \
-H "Authorization: Bearer kapi_..." \
-H "Content-Type: application/json" \
-d '{"query": "How much carer leave do I get?",
"documents": ["Carer leave is ten days a year.", "Annual leave accrues monthly."], "top_n": 1}'
import requests
BASE = "https://<your tenant host>/public/v1"
HEADERS = {"Authorization": "Bearer kapi_..."}
passages = ["Carer leave is ten days a year.", "Annual leave accrues monthly."]
emb = requests.post(f"{BASE}/embeddings", headers=HEADERS,
json={"input": passages, "input_type": "passage"}).json()
vectors = [row["embedding"] for row in emb["data"]] # same order as passages
print(len(vectors), "vectors of", len(vectors[0]), "dims from", emb["provider"])
rr = requests.post(f"{BASE}/rerank", headers=HEADERS, json={
"query": "How much carer leave do I get?", "documents": passages, "top_n": 1}).json()
best = rr["results"][0]
print(passages[best["index"]], round(best["relevance_score"], 3), "scored by", rr["model"])
const BASE = "https://<your tenant host>/public/v1";
const headers = { Authorization: "Bearer kapi_...", "Content-Type": "application/json" };
const passages = ["Carer leave is ten days a year.", "Annual leave accrues monthly."];
const emb = await (await fetch(`${BASE}/embeddings`, {
method: "POST", headers, body: JSON.stringify({ input: passages, input_type: "passage" }),
})).json();
const vectors: number[][] = emb.data.map((row: { embedding: number[] }) => row.embedding);
console.log(vectors.length, "vectors of", vectors[0].length, "dims from", emb.provider);
const rr = await (await fetch(`${BASE}/rerank`, {
method: "POST", headers,
body: JSON.stringify({ query: "How much carer leave do I get?", documents: passages, top_n: 1 }),
})).json();
const best = rr.results[0];
console.log(passages[best.index], best.relevance_score.toFixed(3), "scored by", rr.model);
The options
| Field | Route | What it does |
|---|---|---|
input_type | embeddings | query or passage, for models that embed questions and documents differently. Forwarded to the providers that distinguish them, each in its own vocabulary: NVIDIA (which assumes query when you send nothing), Cohere, Google and Voyage. Every other provider drops it. |
dimensions | embeddings | Forwarded when set; the provider decides whether to honour it. |
encoding_format | embeddings | Forwarded when set. |
user | embeddings | Accepted and discarded. |
top_n | rerank | Return only the best n results. Left out, every document comes back scored. |
Embedding rows come back in the order of input. A rerank result carries the index of the document in your list and a relevance_score between 0 and 1; use index to get back to the text. Both responses carry the served_model headers, so a fallback is visible.
Limits and behaviour
- Pin
modelfor anything you store. A tier fallback can answer with a different embedding model, and its vectors do not match the ones already in your index;served_modelsays which one answered. - There is no per-request cap on the number of inputs or documents beyond the 64 MiB body limit. Provider limits still apply and come back as provider errors.
- Embedding tokens count toward the AI Gateway's tokens-per-minute limit and are priced into budgets. Rerank calls are checked against budgets before they run but add nothing to them, because the route returns no token count.
- The gateway's per-model rate limit is keyed on the tier name when no model is named.
- An empty
datalist from a provider is treated as a failure and the next candidate in the tier is tried. - The checks run in the order budget, rate limit, token quota, so a call over budget is refused as
budget_exceededeven when it is also over the rate limit. See How limits combine.
Troubleshooting
- 422 with
no_models_for_tier: no model is taggedembeddingorrerankingfor the tenant. An administrator tags one on the AI Gateway's Models page. - 403 with
FORBIDDEN: the key has neither the Chat nor the Completions scope. dimensionshad no effect: the provider that served the call does not support it; checkserved_modeland pin one that does.- Two embeddings of the same text differ between calls: a fallback served the second call. Read
X-Fallback-Reasonand pin the model. - A rerank
relevance_scoresits near 0.5 for everything: the served model returned raw logits, which the gateway squashes into the 0 to 1 range; compare scores within one call, not across models.
The same task in the platform