Operate¶

The day-to-day screens: who can use the gateway, what models it offers, how it picks one, and how you cap usage. Everything here assumes the core concepts (scopes, roles, keys).

Organizations¶

Where: Operate ▸ Organizations · Who: members view; MANAGE_TEAMS / MANAGE_MEMBERS to manage; MANAGE_ORG for org settings · Editions: all

The Organizations page is where you run your tenant: its teams, its people, and its invitations. It has three things going on:

Orgs and teams. Create and rename teams, archive them, or (platform admin, when empty) hard-delete. A member is locked to their own org; the platform admin picks which org to work in.
Members. See everyone in the org, change a person's role (only to a role below your own), and remove people. Roles are the ones in Roles and capabilities.
Invitations. Invite someone by email. They get an accept-link that turns into a membership. This is how people join without you sharing credentials.
Team membership. Assign org members into a team and appoint a lead. The lead gets the delegated, team-scoped authority described in concepts.

Pick entities by name, never by ID

Anywhere you select an org, team, or project, the dashboard uses a searchable picker that resolves names to IDs for you and lets you create one inline. You never type a UUID.

Deleting an org or team is a true purge

A hard delete is allowed only when the scope is empty (no sub-entities, no active keys). It publishes a ScopeDeleted event that cascades a tenant purge: budgets, key caps and keys, guardrail rules, and the audit trail for that scope are erased. Prefer archive (reversible) unless you truly mean to erase. See org deletion.

Virtual keys¶

Where: Operate ▸ Virtual keys · Who: ISSUE_KEYS (user and up) · Editions: all

A virtual key is the credential your application uses to call the gateway. It is what you hand to a service instead of a real vendor key. You never expose your OpenAI/Anthropic master keys to applications; those stay sealed in the catalog.

Issuing a key¶

Click Issue key.
Choose the placement (the scope the key belongs to): the org, and optionally a team and a project. This sets the key's scope chain, which decides which budgets and limits apply to it.
Give it a label so you can recognise it later.
Issue. The secret is shown once. Copy it now; only its hash is stored, so it cannot be shown again. The list afterward shows a non-secret hint (a prefix), the scope, and the status.

# What your application then does (the data plane):
curl https://your-gateway/v1/chat/completions \
  -H "Authorization: Bearer sk-...your-virtual-key..." \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hello"}]}'

Placement rules and limits (the hidden parts)¶

Placement authority. Putting a key directly at the org level (no team) draws on the org's shared pool, so it needs budget authority (MANAGE_BUDGETS, i.e. admin/owner) or the platform admin. A regular member without that must place the key in a team they belong to. This stops a member from minting org-level spend.
Per-scope key-count caps. There is an always-on ceiling on how many keys a scope may hold (default 1 000 000 - high enough that it never bites a real organization, so it acts as a runaway backstop rather than a budget), plus admin-set caps via the key-limits API, which are how you actually prevent key sprawl. Hitting either returns a clear 409.
Visibility is scoped. A plain member sees only their teams' keys; an overseer with VIEW_SECURITY sees the whole org; the platform admin sees all. So the list you see is already filtered to what you are allowed to manage.
Revoking is immediate: the hash is marked revoked and the next request with that key is rejected. (SOAR can also quarantine a key, a reversible SUSPENDED state, see SOAR.)

Catalog (providers, deployments, and routing)¶

Where: Operate ▸ Catalog · Who: everyone views; MANAGE_PROVIDERS (admin/owner) for vendors + keys; MANAGE_DEPLOYMENTS (operator and up) for models + pricing · Editions: all

The catalog is your per-org map of "what models exist and where they come from." It has two layers, split by sensitivity on purpose.

Providers (the vendor connections)¶

A provider is a connection to a vendor: a base URL, a protocol, and the master credential for that vendor. Because it holds a secret, creating or editing one needs MANAGE_PROVIDERS.

The credential is AES-256-GCM encrypted at rest; the app refuses to start without its master key. Applications never see it.
The protocol selects the outbound adapter: openai (passthrough, covers any OpenAI-compatible server including Ollama, Groq, vLLM, LiteLLM), anthropic, gemini, vertex, vertex-anthropic.
Editing a provider supports key rotation: re-encrypt and swap the credential, effective on the next request (the catalog reads no cache).
Deleting a provider cascades to its deployments.

Connecting a local Ollama

Add a provider with protocol openai, base URL http://ollama:11434/v1, and any placeholder key (Ollama ignores it). Then add a deployment whose alias is what your apps will request and whose upstream model is the Ollama model name (e.g. gemma3:4b). No code change anywhere else.

Deployments (the model aliases)¶

A deployment maps a model alias (what your apps ask for) to a provider plus an upstream model (what the vendor runs), with optional pricing. Defining one needs MANAGE_DEPLOYMENTS.

Alias vs upstream. Your apps call the alias (fast-cheap); the deployment points it at, say, OpenAI's gpt-4o-mini. Re-point it later with zero app changes.
Pricing is per-token, in the org's base currency: input, output, and optionally cached-read and cache-write rates. If set, every call gets a computed Money cost that flows into budgets, the audit trail, and reports. No pricing means usage is still metered in tokens, just not in money.
drop_params lets you strip request params a particular upstream rejects.
Load balancing is implicit. Register two or more deployments under the same alias and the gateway automatically spreads requests across them and falls back between them. The catalog flags this as load-balanced ×N and offers a group-by-alias view.

Ceilings

Registration is bounded by always-on caps (default 1 000 providers per org, 5 000 deployments per provider); a breach is a 409.

How routing works (the hidden engine)¶

This is the part that is invisible in the UI but does the heavy lifting. When a request names an alias, here is exactly how a vendor gets chosen. Yes, it is both budget-aware and health-aware.

flowchart TD
    REQ["Request for alias 'fast-cheap'"] --> RES["catalog.resolve(alias, org)<br/>→ all deployments under the alias"]
    RES --> BUD{"Per-deployment budget filter<br/>(cost-aware)"}
    BUD -->|"drop any deployment whose<br/>own budget is exhausted"| HEALTH["Health partition<br/>(circuit breaker)"]
    HEALTH --> HE["Healthy candidates"]
    HEALTH --> CD["Cooling-down candidates"]
    HE --> LBH["Load-balance (random order)"]
    CD --> LBC["Load-balance (random order)"]
    LBH --> ORDER["Try healthy first…"]
    LBC --> ORDER2["…then cooling-down last<br/>(half-open probe)"]
    ORDER --> CALL["Call vendor"]
    ORDER2 --> CALL
    CALL -->|success| DONE["Return + record success"]
    CALL -->|"transient 5xx / dropped conn"| RETRY["Retry w/ backoff + jitter<br/>(streaming-safe)"]
    RETRY --> CALL
    CALL -->|"hard failure"| NEXT["Record failure → next candidate (fallback)"]
    NEXT --> CALL
    CALL -->|"every candidate over budget"| B429["429 Over budget"]

Step by step:

Resolve. The alias resolves to the org's list of candidate deployments. If none is registered, a configured fallback provider is used.
Budget-aware filtering (cost-aware failover). Before routing, the gateway drops any candidate whose own per-deployment budget is exhausted, in one batched read. So if you cap the expensive vendor and leave the cheap one open, traffic spills to the cheap one instead of being denied. The request is denied with a 429 only when every candidate is over budget. (Aggregate, all-deployment budgets are checked earlier, at the pre-call gate; see Budgets.)
Health-aware ordering (circuit breaker). A per-deployment circuit breaker tracks failures. After N consecutive failures a deployment enters a cooldown window; a success clears it. Candidates are partitioned into healthy and cooling-down.
Load balancing. Each group is shuffled by a stateless random load balancer, so traffic spreads evenly across equivalent backends.
Order and fall back. Healthy candidates are tried first; cooling-down ones are kept at the end of the list as half-open probes (still reachable, so a recovered backend rejoins automatically). The pipeline tries each in order until one succeeds, reporting every attempt's outcome back to the breaker.
Transient retries. Within a single attempt, transient failures (502/503/504/529 and dropped connections) are retried with exponential backoff and jitter, in a streaming-safe way, before the attempt is counted as a failure.

Tuning

The breaker is configured with aim.routing.health.failure-threshold and aim.routing.health.cooldown. It is in-memory per instance. Full internals: routing.

The net effect: register two cheap backends and one premium one under an alias, price them, put a budget on the premium one, and the gateway will load-balance the cheap pair, fail over on errors, cool down a flaky backend, and spill off the premium one when its budget runs out, all without your application knowing.

Playground¶

Where: Operate ▸ Playground · Who: open to every role (the key you paste authorises the call) · Editions: all

The Playground is a real chat client wired to the real gateway, so it exercises the full pipeline (auth, budgets, guardrails, routing) exactly as an application would. Use it to smoke-test a deployment, see guardrails act, or watch token usage.

Paste a virtual key. It is held in memory only; nothing is minted or proxied on your behalf.
Pick the org (platform admin chooses; a member is locked to theirs), which scopes the model menu, then pick a model alias.
Toggle the wire surface (OpenAI /v1/chat/completions or Anthropic /v1/messages) and optionally turn on the built-in demo tools.
Chat. You get streaming markdown, reasoning/"thinking" blocks with a token count, copy / regenerate / edit / branch, attachments (images + text), a stop button, and a live token counter.

It is governed like everything else

Because the Playground calls /v1/** with your key, the call is rate-limited, budgeted, guardrailed, and audited. If a guardrail redacts the reply or a budget denies the call, you will see it here too. That is the point: it is a faithful test, not a bypass.

Budgets¶

Where: Operate ▸ Budgets · Who: VIEW_BUDGETS to view; MANAGE_BUDGETS (admin/owner, or a team lead for a project in their team) to set · Editions: all

Budgets ▸ Manage. A scope's at-a-glance summary above live spend meters. The second card is a per-deployment cap (note the badge), the lever behind cost-aware failover.

Budgets cap spend, in tokens and/or money, over a reset period, at any scope. They are enforced pre-call on accumulated spend and accrued post-call from the event stream, so streaming is fine and the one request that crosses the line still completes.

A budget is identified by four things, so a single scope can hold several at once:

Part	Values	Meaning
Scope	ORG / TEAM / PROJECT / KEY	Where it attaches (and where it is enforced in the chain)
Model	all, or one alias	Cap everything, or just one model
Provider	all, or one	All deployments of the alias, or one specific deployment (for cost-aware failover)
Period	NONE / HOURLY / DAILY / WEEKLY / MONTHLY	The reset window (lazy reset)

Each budget sets a hard tokenLimit and/or costLimit (denies when exceeded) plus an optional soft cost limit (warns, but proceeds). So you can run "monthly $5k hard cap on everything, plus a daily $200 soft warning, plus an hourly burst cap on the expensive model" all on one scope.

The two tabs¶

Manage (per scope). Pick an ORG/TEAM/PROJECT (platform admin picks the org; members locked to theirs). You get an at-a-glance summary (counts of budgets, near-limit, over, and the next reset) beside a paginated list of budget cards, each with live spend meters for cost and tokens against their soft and hard thresholds. Add an all-models, per-model, or per-deployment cap (a deployment picker appears for a load-balanced alias), edit, or delete.
Explore (cross-scope FinOps). A filterable table over every budget in the org with client-side burn-rate, projected end-of-period, and time-to-exhaust, a spend trend + 7-day forecast chart, and model leaderboards (most expensive, most efficient by actual $/Mtok, cheapest by catalog rate).

Aggregate vs per-deployment, and why failover works

A provider-less budget (all deployments of an alias, or all models) is an aggregate cap, checked at the pre-routing gate: if it is exhausted the whole alias is denied. A budget restricted to one provider is a per-deployment cap, checked by the per-candidate filter during routing: exhausting it makes the gateway fail over to a sibling deployment rather than deny. That distinction is what makes cost-aware routing work: cap the premium backend per-deployment, leave an aggregate headroom, and traffic spills to the cheap backend.

Where the trends come from

Budget rows hold only the current window's spend. The trend, forecast, and leaderboards come from the analytics usage API (/admin/usage/spend and /spend-series), so they are full history, not just this window.

Rate limits¶

Where: Operate ▸ Rate limits · Who: MANAGE_GATEWAY (operator and up) to set/remove; members view · Editions: all (Valkey-backed counters on Distributed)

Rate limits ▸ Limits, with a SOAR throttle live (the amber banner and its Stop throttle action). Meters read against the effective cap, not just the configured one.

Rate limits cap requests per minute (RPM) and tokens per minute (TPM) per scope, via a per-minute sliding window. Exceeding a limit returns 429 with a Retry-After header. Like budgets, a limit attaches to a scope and is enforced across the whole scope chain.

RPM admits-and-counts on the hot path. TPM is a pre-call read plus a post-call token accrual.
Counters live behind a pluggable backend: in-JVM (single instance / dev, the default) or Valkey (the only one that scales an org-wide counter across instances, via atomic increments on TTL'd minute buckets). On the Distributed tier, use Valkey.
An unconfigured scope touches no counter store at all (the config is cached in memory), so rate limiting adds nothing to a call until you actually set a limit.

The two tabs¶

Limits. Pick a scope, see live RPM/TPM usage meters against the effective cap, and set / update / remove a limit. If a SOAR throttle is active on the scope, an amber "SOAR throttle active" banner appears (the limiter takes the tighter of the configured and the throttle cap), with a Stop throttle action to lift it early.
Activity. The scope's request density, with a view selector (Day / Week / Month) and a range selector (1m / 3m / 6m): Day is an hour-of-day bar, Week is a day-of-week × hour heatmap, Month is a calendar heatmap of calls per day. All UTC, scoped to the selected org/team/project.

Throttle is also a SOAR action

The same per-scope throttle mechanism backs SOAR's automated throttle containment. You can apply one manually here (operator and up) to seed or exercise the banner, and lift it early. It auto-reverts on its own. See SOAR.

Next: Govern → (guardrails, notifications, audit & usage, reports, sign-in activity), or back to the section map.