Skip to main content

Configure the AI Gateway and models

LLM gateway

Every model call an agent makes goes through the AI Gateway. In this stage you connect the providers your organisation may use, decide which models sit in each routing tier, cap how fast anyone can call them, and learn where to watch the traffic.

Who can use it: the Admin role or the Operator grant, in a tenant licensed for the LLM gateway. On AI Cloud, providers and models are managed by your distributor.

VideoConfigure the AI Gateway and Models3:06

Recorded on .

Prerequisites​

  • API keys for the providers you plan to use, or the URL of a self-hosted endpoint.
  • To import the provider catalogue, Ops Hub connected under Platform Settings; see Set up platform settings.

Add a provider​

  1. Open the gateway

    In the Control Room, open AI Gateway. A header chip says whether Ops Hub is connected, next to the buttons Rate Limits, Logs and Settings. The Providers tab opens first, with tiles counting providers, models and providers With API Keys and Without Keys, then request, cost, success-rate and latency tiles.

    The AI Gateway on its Providers tab: header chips for the Ops Hub connection and the provider and model counts, the Rate Limits, Logs and Settings buttons, the CONFIGURE tabs Providers, Models and Tiers and the OBSERVE tabs Usage and Costs, tiles for Providers, Total Models, With API Keys and Without Keys, then Total Requests, Total Cost, Avg Success Rate and Max Avg Latency, the Search providers box with Sync from Ops Hub and Add Provider, and provider rows each showing a KEY badge, a health badge, a model count, a success rate and a latency.The AI Gateway on its Providers tab: header chips for the Ops Hub connection and the provider and model counts, the Rate Limits, Logs and Settings buttons, the CONFIGURE tabs Providers, Models and Tiers and the OBSERVE tabs Usage and Costs, tiles for Providers, Total Models, With API Keys and Without Keys, then Total Requests, Total Cost, Avg Success Rate and Max Avg Latency, the Search providers box with Sync from Ops Hub and Add Provider, and provider rows each showing a KEY badge, a health badge, a model count, a success rate and a latency.
    The Providers tab of the AI Gateway. Each row shows whether the provider has a key and how healthy it is; Refresh Catalog imports the catalogue and Add Provider adds one by hand.
  2. Import or add providers

    Refresh Catalog imports the provider catalogue; when the list is empty and Ops Hub is reachable, the page does this once by itself. Add Provider opens a dialog: enter a Provider Name, pick a Provider Type and set the Base URL, which is prefilled for hosted services, built from fields such as a region or resource name for some, and required for a self-hosted endpoint. Add an API Key (optional here; Test checks it), use Discover Models to list the provider's models and tick the ones to import with a tier each, then select Add Provider.

  3. Configure a key

    Rows without a credential show NO KEY and a Configure Key button, which opens Manage API Key. Enter the API Key, choose its Environment, select Validate, then Save Key: one key per provider, shared by all its models. A keyed row shows KEY and a health badge, Healthy, Degraded or Unhealthy, judged against success-rate thresholds set under Settings. PLATFORM-MANAGED marks a provider created by the platform's own LLM deployments; it needs no key. Select a row to open the provider's page.

Enable models and assign tiers​

The Models tab lists every model under three origins, Provider, Deployed and Hosted, with Search models... and an All Tiers filter. The columns are Model, Tier, Priority, Context, Cost ($/1M), Status, Health and Latency, and Status reads Active, Disabled or No Key. The pencil at the end of a row opens Edit, where you set the model's Tier, switch it between Enabled and Disabled and record its costs. A provider's page adds models with Add Model or Discover.

A tier is an abstract routing target: when an agent asks for Reasoning, the gateway routes to the highest-priority healthy model in that tier. The Tiers tab lists Reasoning, Fast, Balanced, Guard, Embedding, Reranking, Stt, Tts, Vision and Code. In each, the top model is PRIMARY and the rest are fallbacks; use Move up (higher priority) and Move down (lower priority) or drag to reorder, and Tier settings (the gear) to pick the Primary Model. Models that are disabled, have no API key or whose key has not been validated are excluded from routing and counted as inactive.

The Tiers tab of the AI Gateway: a sentence explaining that tiers are routing targets, then a section per tier, Reasoning first, with its model count and inactive count, and the models in routing order, the first marked PRIMARY and the rest FALLBACK, each with its provider, origin, lanes, status and health.The Tiers tab of the AI Gateway: a sentence explaining that tiers are routing targets, then a section per tier, Reasoning first, with its model count and inactive count, and the models in routing order, the first marked PRIMARY and the rest FALLBACK, each with its provider, origin, lanes, status and health.
Each tier is a routing target. The top model in a tier is the primary and the others are fallbacks; Tier settings picks the primary, and the arrows or dragging change the order.

Set rate limits​

Select Rate Limits. Rate Limit Configuration has three scopes, Per User, Per Tenant and Per Model, each with Requests/Min (RPM) and Tokens/Min (TPM). The dialog describes them as sliding windows; the gateway counts them in fixed one-minute windows, so a burst that empties a minute's allowance is refused until the next minute begins. The shipped values are 30 and 60,000 per user, 200 and 500,000 per tenant, and 100 and 200,000 per model. Save Changes applies yours; Reset Defaults restores those values.

The Rate Limit Configuration dialog: a line saying limits are counted in fixed one-minute windows, then the Per User and Per Tenant sections with their Requests/Min (RPM) and Tokens/Min (TPM) fields, and the top of the Per Model section.The Rate Limit Configuration dialog: a line saying limits are counted in fixed one-minute windows, then the Per User and Per Tenant sections with their Requests/Min (RPM) and Tokens/Min (TPM) fields, and the top of the Per Model section.
Rate Limits sets requests and tokens per minute for each user, for the whole tenant and per model. Save Changes applies them; Reset Defaults restores the shipped values.

Watch it​

Usage shows requests, success rate, latency and provider health for a time range, plus traffic by origin. Costs shows total and per-request cost, a breakdown by model and by origin, and budget alerts. Logs lists recent gateway requests, each opening to its Request Info and Tokens & Cost. Budgets belong to the next stage: see Govern the platform day to day.

Verify​

On Tiers, the tier your agents use lists at least one model marked PRIMARY with the status Active, and its provider row reads KEY and Healthy. In the Work Room, a message to an agent gets an answer, and Usage counts the request.

Troubleshooting​

  • Ops Hub is not connected. Please connect Ops Hub from Platform Settings to sync providers.: connect Ops Hub on its Platform Settings tab, or add providers by hand.
  • No providers configured yet. or No models configured. Add a provider first.: select Refresh Catalog or Add Provider.
  • Invalid API key. or Saved key could not be decrypted. Re-enter the key.: fix the key in Manage API Key.
  • Provider is unreachable. Check the Base URL and network., Base URL is required for this provider. or Base URL is invalid. Check for unresolved placeholders or missing scheme.: correct the endpoint on the provider's page.
  • Provider's TLS certificate is not trusted.: for a trusted internal endpoint, tick the TLS verification option in the provider dialog to skip certificate checks.
  • Provider catalog is empty. Sync providers, then retry.: run Refresh Catalog; if the catalogue was never pushed to your workspace, ask your platform operator.
  • Usage or Costs shows a lock: your account lacks the permission or licence to view usage and cost data; the panel says which.

Go deeper​

The Control Room pages behind this journey, each with every field, state and message:

Describes the platform as of .

Was this page helpful?