LLM API Virtual Keys

TL;DR

Virtual key versus provider key

Applications present a virtual key to the gateway. The gateway authenticates it, resolves identity and policy, then selects an approved upstream route using a provider credential that never leaves the trusted boundary.

A provider key usually grants access according to the provider account or project that issued it. A virtual key is an internal authorization object: it can represent one service, tenant, environment, team or temporary workload, even when several virtual keys ultimately use the same upstream account. That additional identity layer is what makes per-application budgets, model allowlists and audit ownership possible.

Do not encode sensitive policy in the key itself. Store a one-way identifier and versioned server-side policy so permissions can change and keys can be revoked without disclosure. The visible token should carry enough entropy to resist guessing, while ownership, scope, expiry and budget remain server-side records.

Virtual keys are most valuable when their lifecycle is shorter and more specific than the upstream credential. Issue separate keys for production, staging, CI jobs and local development; name an accountable owner; set an expiry where practical; and avoid copying a human user's key into a long-running service. If two workloads need different data regions or model families, they should not share one virtual key merely for convenience.

Credential design choices

Virtual key policy controls

Validate the control plane

Control-plane testing must cover both authorization correctness and operational propagation. Creating a key in one region, changing its policy in another and revoking it during active traffic should produce deterministic results within documented time bounds. Test against multiple replicas so a stale cache cannot continue accepting a key that another replica has already revoked.

• Attempt cross-tenant, wrong-model, expired and revoked-key requests.
• Verify limits under bursts and ensure decisions are atomic across replicas.
• Rotate upstream credentials without changing application virtual keys.
• Trace every accepted and denied request to identity, policy version and route.

Budget tests need concurrency, not just sequential calls. Reserve spend before dispatch when several requests can race against the same limit, reconcile the reservation with actual token usage afterward, and define whether a request is rejected, queued or routed to a cheaper model when the budget is exhausted. Verify that retries do not charge the virtual-key budget twice for one logical attempt.

Terminate virtual keys at the gateway

A user or service authenticates with a virtual key or identity token. The gateway resolves an internal principal, evaluates versioned policy and selects an allowed model route. Provider secrets are fetched from a protected vault and never returned to the application.

Store only a hash or keyed digest of the virtual key where possible, and show the plaintext value only at creation. Requests should be authenticated before large bodies are buffered or expensive work begins. After authentication, attach the resolved principal and policy version to the request context so every downstream routing, logging and billing decision uses the same identity.

Design for partial control-plane failure. Runtime gateways can cache signed policy snapshots for a short, bounded interval, but revocation and high-risk policy changes need a fast invalidation channel. Define fail-open versus fail-closed behavior by workload: a low-risk internal summarizer may tolerate a brief cached policy, while regulated data access should normally stop when policy freshness cannot be established.

Use separate scopes for models and external capabilities

Model virtual keys govern inference access. QVeris governs external APIs, tools, services and live data through Discover → Inspect → Call. Bind both to the same identity where useful, but issue separate scopes, budgets and audit records.

A support assistant, for example, may be allowed to use several text models but only read ticket metadata and public documentation. A finance agent may use the same models yet require market data, filings and portfolio tools with different entitlements. Model access alone cannot express those distinctions, so external capability policy should name the allowed tools, permitted data classes, argument limits and any required human approval.

Join the records at the workflow level rather than forcing them into one credential. A shared trace can show that a virtual key authorized model inference and that a separately scoped QVeris capability produced the live result. This gives operators one investigation path while preserving separate revocation, cost attribution and least-privilege controls.

FAQ

Official sources and further reading