One API Key for All LLMs: Keep Secrets Isolated

TL;DR

What one key should and should not mean

One app key simplifies client configuration and centralizes governance. It should not become a broad shared secret with permanent access to every provider, model and environment.

Treat “one key” as one stable application identity, not one credential copied across systems. The application authenticates once to the gateway; the gateway evaluates who is calling, from which environment, for which workload and under which budget. Only then does it select an eligible model route and obtain the matching upstream credential inside the trusted boundary.

The gateway maps the authenticated principal to a versioned policy. Provider adapters use separate credentials with the least scope available. Rotation or provider changes do not require distributing new secrets to every application.

This abstraction also limits blast radius. A leaked application key can be revoked for one tenant or environment without disabling every upstream account. A provider credential can be rotated after an incident without changing client configuration. The design fails, however, if all virtual keys resolve to the same unrestricted provider secret or if production and test traffic share one policy.

Unified credential patterns

Policy attached to the key

Red-team unified access

A happy-path model response proves only that routing works. Security testing must prove that the gateway rejects requests when identity, scope, lifecycle state or policy does not match. Run these tests against the same authentication path and policy store used by production, not a simplified mock.

• Attempt wrong-tenant, wrong-model, expired and revoked-key requests.
• Verify provider secrets never enter responses, logs, traces or client bundles.
• Rotate every upstream credential while applications keep their virtual key.
• Test policy rollback, budget exhaustion and a compromised-key kill switch.

Also test concurrency and replay behavior. A key with a monthly budget can still cause an incident if thousands of requests arrive before spend counters converge. Use atomic reservations or bounded local allowances, define idempotency behavior for retries, and verify that a revoked key stops new requests within the documented propagation window.

Resolve one key into many isolated credentials

The gateway authenticates the application virtual key, resolves an internal principal and evaluates workload policy. A capability router selects an approved adapter. The adapter retrieves only the matching provider credential from a vault, executes the call and returns normalized output plus native evidence.

Keep the request lifecycle observable. The audit record should identify the application principal, policy version, requested capability, candidate routes, selected provider, provider-native request ID, token usage, cost and final status. It should record which policy allowed the call without recording the provider secret itself. This makes a single-key experience easier for developers without creating a single point of audit ambiguity.

Separate the control plane from the request path. Key creation, ownership changes, policy updates, rotation and revocation belong to a versioned control plane with approvals. Runtime validation should use a cached, bounded copy of that policy so a temporary control-plane outage does not automatically expose unrestricted access or stop every healthy inference request.

Use separate scopes for inference and tools

The model gateway uses the virtual key for inference policy. QVeris governs external APIs, tools, services and live data through Discover → Inspect → Call. Bind both to the same principal when useful, but keep credentials, budgets and audit trails separate.

Inference permission does not imply action permission. An application allowed to use a reasoning model may still be prohibited from sending email, placing an order, querying regulated data or retrieving customer records. Give tool scopes explicit capability names, argument constraints, data classifications and approval requirements rather than inheriting the broad model scope.

Keep cost controls independent as well. Model spend is usually measured in tokens, while tools may charge by request, record, document, data entitlement or downstream transaction. Joining both totals under one dashboard is useful, but independent budgets prevent a burst of tool calls from consuming the inference allowance or hiding an expensive external action inside a cheap model request.

FAQ

Official sources and further reading