Model Router vs Gateway
A router chooses an eligible model; a gateway authenticates, executes, governs, and observes the request.
Published: March 22, 2026 · By the QVeris Team
TL;DR
It filters, scores and selects an eligible model or endpoint for the workload.
It authenticates, adapts, limits, executes, observes and governs the request lifecycle.
Routing can be a policy module inside the broader gateway control plane.
The selected candidate, reasons, attempts, transformations and result belong in one trace.
Decision plane versus execution plane
The model router receives a versioned workload contract, candidate catalog, tenant policy and current operational signals. It removes ineligible endpoints, scores the remainder and emits a route plan with the selected endpoint, adapter, fallback limit and rationale. Its core output is a reproducible decision—not a network response from a model.
The gateway owns the execution boundary: caller identity, credentials, provider adapters, deadlines, quotas, retries, cache, policy enforcement, telemetry and final response. It validates the route plan before use because the execution layer must not trust a stale or compromised decision to bypass hard constraints. A thin gateway may use a static router; a richer gateway may host workload-specific policies.
Use a standalone router when several execution systems need the same decision service or when routing evolves independently. Embed it when one gateway owns the entire lifecycle and low latency matters. In both cases, keep the plan contract explicit so teams can test selection separately from provider execution.
Router and gateway responsibilities
| Responsibility | Best fit | Verify before choosing |
|---|---|---|
| Candidate eligibility | Router filters by capability, policy, region and budget. | Gateway must supply a versioned catalog and enforce the resulting constraints. |
| Scoring and selection | Router compares quality, cost, latency, health or preference signals. | Gateway records the policy version and selected endpoint before execution. |
| Provider translation | Router may declare an adapter requirement. | Gateway maps request, stream, response, error and usage semantics. |
| Operational controls | Router can consider quota and health as inputs. | Gateway owns auth, keys, rate limits, cache, retries, circuit breakers and cancellation. |
| Evidence and governance | Router explains candidate exclusions and score. | Gateway joins route evidence with attempts, native IDs, usage, policy and final disposition. |
Define a clean component contract
Workload contract, eligible catalog snapshot, policy version and fresh operational signals.
Selected endpoint, ranked alternates, exclusions, reason codes and adapter requirements.
Authorize identity, obtain credentials, translate, enforce budgets and execute bounded attempts.
One parent trace connects decision, execution, provider response and downstream tool work.
Test the router and gateway together
Router unit tests prove deterministic selection from fixed inputs. End-to-end tests prove that the selected route can actually authenticate, preserve the workload contract and return within budget. You need both: a perfect decision is useless if the gateway cannot execute it, while successful execution should not excuse an unexplainable choice.
- Freeze workload, catalog, policy and health fixtures so route decisions are reproducible.
- Test hard exclusions before score weights, preferences and tie-breaking.
- Verify every selected endpoint has a valid adapter, scoped credential and policy-approved network path.
- Inject stale health, quota exhaustion, malformed plans, adapter errors and execution failures.
- Confirm fallback re-enters selection with the failed route excluded and the same parent budget.
- Verify the trace explains candidates, exclusions, choice, attempts, validation and final outcome.
Embed the router, separate the contract
The gateway constructs a routing context from caller identity, workload requirements, catalog and policy versions, remaining budget and fresh health state. The router returns a versioned route plan plus the candidate set, exclusions and score inputs. Even when implemented in the same process, pass this through a typed internal contract rather than sharing mutable global state.
The gateway validates plan age and hard constraints, resolves credentials and adapters, executes one bounded attempt and validates the output. It records both decision evidence and provider-native execution evidence under one trace. This separation lets routing policy evolve without fragmenting authentication, safety, accounting or telemetry.
Production rule: the gateway must independently enforce every hard constraint the router used to choose a candidate.
Model routing does not route external capabilities
A router chooses inference; a gateway governs inference execution. QVeris complements both by discovering and calling external APIs, tools, services and live data. Keep capability requirements in the workload contract so the model route supports the required tool shape, but inspect capability schema, permissions, health and freshness independently.
Connect model and tool calls to the same parent workflow trace while keeping separate credentials, action IDs, retries and fallback pools. A different model may reuse a verified tool result, but it must not automatically gain permission to repeat a consequential action or switch to a semantically different data source.
Continue with related implementation guides
Use these adjacent guides to compare the next architectural decision, implementation boundary, or operating control without starting the research again.
FAQ
No. It can be a library, plugin, policy module or independent service.
Yes. It can govern one fixed upstream while still providing auth, limits, adapters and evidence.
The execution layer, usually the gateway, because it knows attempts, deadlines and side effects.