Dify + Custom Model API
Configure, Verify, and Keep a Rollback
Add a custom model provider to Dify by defining the credential and data boundary, testing required capabilities, and keeping rollback available.
TL;DR
Do not share provider keys or logs across unrelated workspaces.
Test models, chat, streams, tools, schemas and errors.
Keep provider, model, request ID, usage and failure evidence.
Canary one app and retain the previous provider configuration.
Custom provider setup flow
Register an approved base URL, scoped key and model alias in the provider configuration. Verify the endpoint's actual routes and capabilities before exposing it to an app.
Start with the provider contract, not the display name. Confirm the exact path Dify will call, authentication header, model identifier, request and response fields, timeout behavior and error format. “OpenAI compatible” often covers basic chat completion but not necessarily streaming, tool calling, vision, JSON schema, embeddings or token accounting.
Use a stable internal alias for the application and keep the physical provider model behind it. This lets operators change endpoints or model versions without editing every workflow. Record the alias-to-endpoint mapping, provider configuration version and owner so an incident can be traced to the exact configuration that served a request.
Promote through a test workspace, canary selected applications and watch native errors, usage and output quality. Roll back configuration without rewriting prompts. Keep production and test credentials separate, and verify that a workflow exported from one workspace cannot silently inherit a higher-privilege provider record in another.
Configuration fields to govern
| Field | Best fit | Verify before choosing |
|---|---|---|
| Base URL | Approved HTTPS compatible endpoint. | Validate operator, host, region and data terms. |
| API key | Tenant-scoped least-privilege credential. | Store in secrets, rotate and audit access. |
| Model alias | Stable app-facing workload name. | Map to a versioned model and verified capabilities. |
| Retries and timeout | Bounded execution policy. | Give one layer ownership and preserve idempotency. |
Capability verification
Check model listing and each required inference path.
Test streams, tools, JSON schema, errors and usage.
Confirm keys, configs, logs and budgets remain tenant-scoped.
Version the provider config and test one-click rollback.
Promote one application at a time
A provider can pass a simple playground prompt and still break a production Dify application. Build the test set from actual workflow nodes: system prompts, conversation history, structured variables, knowledge context, file inputs, streaming, tool calls and downstream parsers. Compare both the raw provider response and the final workflow output.
- Run provider contract tests outside production.
- Compare target and previous providers on representative app workflows.
- Canary one tenant and monitor errors, routes and accepted outputs.
- Exercise rollback and credential revocation.
Define acceptance thresholds before the canary. Track transport errors, first-token and total latency, schema-valid output, tool-call argument validity, token usage and task-level quality. A lower error rate is not enough if the new route drops citations, changes JSON field types or produces much longer responses that break cost assumptions.
Rollback must restore the full previous provider record, not only the model name. Test base URL, credential reference, capability flags, context limits, timeout, retry policy and output adapter as one versioned unit. Revoke the rejected credential after rollback so stale workers cannot continue sending traffic to the failed route.
Treat provider configuration as code
Dify applications reference a stable alias. A tenant-scoped provider record holds the approved endpoint and secret. Capability tests gate promotion, telemetry preserves native evidence and configuration versions make rollback immediate.
Store non-secret configuration in version control or a governed configuration registry. Review changes to endpoint, model mapping, context limits and capability flags like code. Secrets remain in a vault and are referenced by identifier. The deployed record should include a checksum or version so the request trace can prove which configuration was active.
Separate discovery from promotion. A provider administrator may create and test a candidate record, but production activation should require an approval or automated quality gate. This prevents a convenient dashboard edit from bypassing tenant policy, data residency or application-level regression checks.
Production rule: never mark a capability supported until its production path has passed a contract test.
Add QVeris for external APIs and tools
The custom model API supplies inference. QVeris supplies external APIs, tools, services and live data through Discover → Inspect → Call. Keep their provider records, secrets and evidence separate.
This matters when a Dify workflow moves from generating text to completing a real task. A model route may be allowed to summarize a request, while the following capability call needs a separate permission to query a filing, fetch live market data, parse a document or update an external system. Inspecting the capability before execution exposes required arguments, schema, cost, latency and permission requirements.
Join the two layers with a workflow trace rather than one shared secret. Record the Dify application and model attempt, then attach the QVeris search, capability and execution identifiers to the same trace. Operators can reconstruct the complete workflow while still rotating, revoking and budgeting inference and external actions independently.
Start with the provider contract Dify actually loads
Dify's current plugin system separates provider metadata, credential fields, model-type code, invocation, token counting, and error mapping. A custom model is not only a URL field: package the contract, validate credentials with a lightweight call, and test both streaming and non-streaming paths.
provider: company_gateway
label:
en_US: Company LLM Gateway
zh_Hans: 企业 LLM 网关
supported_model_types:
- llm
configurate_methods:
- customizable-model
provider_credential_schema:
credential_form_schemas:
- variable: server_url
type: text-input
required: true
- variable: api_key
type: secret-input
required: true
extra:
python:
provider_source: provider/company_gateway.py
model_sources:
- models/llm/llm.py- Implement validate_credentials and map auth, rate-limit, timeout, and provider errors.
- Return normalized text, finish reason, usage, and native request identifiers.
- Remote-debug the plugin, package it, and promote one Dify application before workspace-wide rollout.
Verified implementation reference: Dify custom model integration.
FAQ
Use the provider contract Dify supports and verify the current integration documentation.
Prefer scoped credentials and explicit isolation unless your governance permits sharing.
Restore the previous provider config and revoke the target credential if needed.