LLM Provider Inventory
The LLM provider inventory is a service-level list of reusable LLM provider configurations declared under the ai/providers namespace. You define each provider once, giving it a unique name together with the vendor, model, and credentials it needs, and then reference that provider by name from the ai/llm component or the AI Router instead of repeating the full configuration on every backend. Use it whenever several endpoints connect to the same providers, so credentials, models, and endpoints live in a single place. A usable entry needs only two or three fields, and it is the recommended way to configure LLM connections in KrakenD.
credentials of each provider live in your configuration. Inject them through environment variables with Flexible Configuration instead of writing the keys in plain text, and restrict access to the configuration file.Declaring Providers
You declare the inventory in the root extra_config using the ai/providers namespace. The namespace holds a single providers list, where each entry describes one provider configuration. The minimum a usable entry needs is a name and a provider (the vendor: openai, gemini, anthropic, mistral, bedrock, or alibaba). Set credentials to the API key of the vendor. When you leave it empty, KrakenD sends no authentication header, which only suits servers that require no key, such as a local OpenAI-compatible server. When you omit endpoint or model, KrakenD uses the vendor’s defaults.
The following configuration declares seven providers across five vendors:
{
"$schema": "https://www.krakend.io/schema/krakend.json",
"version": 4,
"extra_config": {
"ai/providers": {
"providers": [
{
"name": "cheap-quick",
"provider": "openai",
"endpoint": "https://custom-self-hosted.example.com/v1/responses",
"model": "gpt-4o-mini",
"credentials": "xxx"
},
{
"name": "gpt-default",
"provider": "openai",
"credentials": "xxx"
},
{
"name": "gemini-flash",
"provider": "gemini",
"model": "gemini-2.5-flash",
"credentials": "xxx"
},
{
"name": "coding-basic",
"provider": "anthropic",
"model": "claude-haiku-4-5",
"credentials": "xxx"
},
{
"name": "coding-team",
"provider": "anthropic",
"model": "claude-sonnet-4-6",
"credentials": "xxx"
},
{
"name": "reasoning",
"provider": "mistral",
"credentials": "xxx"
},
{
"name": "summaries",
"provider": "bedrock",
"model": "eu.amazon.nova-lite-v1:0",
"credentials": "xxx",
"debug": true,
"variables": {
"max_tokens": 1500
}
}
]
}
}
}
The cheap-quick entry points to a self-hosted OpenAI-compatible endpoint, while the rest rely on each vendor’s default endpoint. The gpt-default and reasoning entries set only a vendor and credentials, so they also use the vendor’s default model. Replace every "xxx" with your real credentials, preferably through an environment variable.
Besides the vendor, model, endpoint, and credentials, a provider entry accepts these options:
variablescarries provider-specific parameters such astemperatureormax_tokens. The accepted keys depend on theproviderand match the variables documented on each vendor’s page, such as OpenAI or Anthropic.input_templateandoutput_templatepoint to Go template files that replace the default request payload and response transformation. See Prompt Templates.streamingandformatturn the provider into a streaming one, as explained in Streaming Providers.debuglogs the provider activity for troubleshooting. Do not enable it in production, as it may log sensitive data.
These are all the fields of the ai/providers namespace:
Fields of AI Providers
providers* array of objects- The array of AI providers available for referencing from the backends. Each object is a different provider declaration.Each item of providers accepts the following properties:
credentialsstring- The API key used to authenticate against the provider. Inject it with an environment variable instead of writing it in the configuration.
debugboolean- Enables the debug mode to log activity for troubleshooting. Do not set this value to
truein production as it may log sensitive data.Defaults tofalse endpointstring- Overrides the default API endpoint of the provider. Set it when you need to reach a proxy, a private deployment, or a region-specific host instead of the vendor’s public API. When you don’t set it, each provider uses its own default endpoint. Gemini and Bedrock build theirs from the
modelandstreamingvalues, so their default shown here corresponds to the defaultmodelwithout streaming. format- Wire format KrakenD exposes to the client for streaming responses, when not defined, the request and response format is not modified. Only anthropic and openai stream formats are supported client-side, regardless of the upstream connector. When defined, KrakenD translates the client’s request into the upstream connector’s format, and translates the streamed response back into this format before returning it.Possible values are:
"anthropic","openai" input_templatestring- A path to a custom Go template that sets the payload format sent to the provider. You don’t need to set this value unless you want to override the default template making use of all the
variableslisted in this configuration. modelstring- The name of the model you want to use. The value you provide is passed as is to the provider and KrakenD does not check if the model is currently accepted by the vendor. Check the available models on the vendor documentation. When you don’t set it, each provider uses its own default model.
name* string- A unique name for this provider declaration. This is the name you use to reference the provider from the backends, so you declare its settings once and reuse them everywhere.Example:
"fast-chat" output_templatestring- A path to a custom Go template that sets how the response from the provider is transformed before being sent to the client. The default template extracts the text from the first choice returned by the provider so in most cases you don’t need to set a custom output template.
provider*- The AI vendor this declaration connects to. The chosen vendor determines which
variablesare accepted.Possible values are:"anthropic","bedrock","gemini","mistral","openai","alibaba" streamingboolean- Enables streaming responses, so the tokens generated by the model are forwarded to the client as they are produced instead of waiting for the complete response.Defaults to
false variablesobject- The variables KrakenD uses to build the payload for the chosen
provider.
Referencing a Provider from a Backend
Once a provider exists in the inventory, the ai/llm component references it by name with a single provider property, instead of an inline vendor block:
{
"endpoint": "/chat",
"method": "POST",
"backend": [
{
"extra_config": {
"ai/llm": {
"provider": "cheap-quick"
}
}
}
]
}
The backend above resolves the cheap-quick provider from the inventory and sends the request to its endpoint, with its model and credentials, so it needs neither host nor url_pattern. Because you reference the provider by name, you can reuse it across many backends and change its definition in one place.
When ai/llm references a provider, it accepts no other option: every setting, including format, belongs to the inventory entry. To choose among several providers on each request, based on headers, policies, or a classifier, use the AI Router, which references the same inventory.
Streaming Providers
Set streaming to true in a provider entry to forward the tokens to the client as the model produces them. Add format to normalize the streamed output to the openai or anthropic wire format, regardless of the vendor behind the provider. LLM Streaming explains both options in detail.
The streaming flag of the provider must match the endpoint that uses it: a streaming provider requires an endpoint with a streaming output_encoding, and a non-streaming provider requires an endpoint without it. When they do not match, the endpoint answers every request with a 500 status code, and KrakenD logs an error like this one at startup:
KRAKEND ERROR: [BACKEND: POST /chat -> POST /][AI/LLM] Failed to load provider: gemini-stream provider gemini-stream streaming flag true does not match backend encoding
Declare two entries for the same model when you need it both with and without streaming.
The following configuration streams Gemini responses in the OpenAI format:
{
"$schema": "https://www.krakend.io/schema/krakend.json",
"version": 4,
"extra_config": {
"ai/providers": {
"providers": [
{
"name": "gemini-stream",
"provider": "gemini",
"model": "gemini-2.5-flash",
"credentials": "xxx",
"streaming": true,
"format": "openai"
}
]
}
},
"endpoints": [
{
"endpoint": "/v1/responses",
"method": "POST",
"output_encoding": "streaming",
"backend": [
{
"extra_config": {
"ai/llm": {
"provider": "gemini-stream"
}
}
}
]
}
]
}
Clients send OpenAI Responses requests to /v1/responses and receive an OpenAI-formatted stream, while Gemini serves the request. Because streaming is true, KrakenD calls Gemini’s streaming endpoint automatically. For example:
$curl -N -XPOST --json '{"input": "Tell me a joke", "stream": true}' http://localhost:8080/v1/responsesThe response arrives as Server-Sent Events with the OpenAI Responses event types, such as response.output_text.delta, and ends with a response.completed event that carries the token usage.
Default Endpoints and Models
When you omit model in a provider entry, KrakenD uses the default model of the vendor:
| Provider | Default model |
|---|---|
OpenAI (openai) | gpt-5.4-mini |
Google Gemini (gemini) | gemini-3.5-flash |
Anthropic (anthropic) | claude-haiku-4-5 |
Mistral (mistral) | mistral-small-latest |
AWS Bedrock (bedrock) | eu.amazon.nova-lite-v1:0 |
Alibaba Cloud (alibaba) | qwen3.8-flash |
When you omit endpoint, KrakenD sends the requests to the default endpoint of the provider:
openai https://api.openai.com/v1/responses
gemini https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent
anthropic https://api.anthropic.com/v1/messages
mistral https://api.mistral.ai/v1/chat/completions
bedrock https://bedrock-runtime.eu-west-1.amazonaws.com/model/{model}/converse
alibaba https://token-plan.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1/responses
The Gemini and Bedrock default endpoints embed the selected model in the URL. With streaming enabled, they also switch to the vendor’s streaming method: Gemini uses {model}:streamGenerateContent?alt=sse and Bedrock uses {model}/converse-stream. The other vendors stream through the same endpoint. The Bedrock default targets the eu-west-1 region; set endpoint to use a different region.
