Document updated on Oct 8, 2026

Unified LLM Interface

The Unified LLM Interface of KrakenD allows you to interact with one or more LLMs, removing the complexity of dealing with LLMs for the end user. It allows you to set the grounds to communicate with LLMs, authenticate, and treat requests and responses back and forth.

The unified LLM interface helps you to:

• Route to the right LLM
• Abstract the request interface
• Abstract the response interface
• Declare every provider once in a central inventory

Routing to the right LLM

KrakenD’s LLM Routing and AI Proxy feature enables the distribution of AI requests across one or multiple Large Language Model. It supports various routing strategies including direct proxying, conditional routing based on request headers or policies, JWT claim-based routing, and path-based routing.

Read more about LLM Routing, or use the AI Router to select a provider from the inventory on every request.

Abstracting the request and response interfaces

The namespace ai/llm makes communication with several AI providers simpler as each vendor will require a different payload and communication style. The ai/llm namespace is able to pre-generate Prompt Templates for a series of vendors:

• Google Gemini
• OpenAI / ChatGPT
• Mistral
• Anthropic
• Bedrock
• Alibaba Cloud (Qwen)
• Other vendors

When talking to the LLM through KrakenD, the user can concentrate in the query rather than in the whole payload needed to interact with the vendor API. The gateway takes care of formatting the API request and response, and allows you to treat different LLMs using the same format, so switching a provider is an easy task.

To stream the response back to the client instead of waiting for the full completion, see Streaming LLM Responses with the AI Gateway.

Prompt Templates

Prompt Templates define how the system talks to each individual LLM, respecting their format and payload characteristics, allowing the user to define only the prompt.

When you talk to an LLM you need to send a body in a format that is different for each vendor. We call Prompt Templates the skeleton used to build the payload you will sned to the LLM. Prompt Templates are predefined templates that contain the variables the user needs to pass. Instead of allowing the end-user to construct the full-body that the LLM will receive (and opening the door to injecting malicious system directives), we ask just for the prompt. This is not only writing a text that you send to the LLM, but actually have the opportunity to insert place holders, values that came in the request body, headers, parameters, etc. or even apply scripting over it.

The Prompt Templates offer more control on the types of requests sent to the AI and allow you to specify system and user input to ensure the final output.

See each vendor documentation.

LLM Provider Inventory

The LLM provider inventory is a service-level list of reusable LLM provider configurations declared under the ai/providers namespace. You define each provider once, giving it a unique name together with the vendor, model, and credentials it needs, and then reference that provider by name from other components instead of repeating the full configuration. Use it when several endpoints or backends connect to the same providers, so you keep credentials, models, and endpoints in a single source of truth. The inventory is intentionally minimal: a usable entry needs only a handful of options.

Defining Providers

You declare the inventory in the root extra_config using the ai/providers namespace. The namespace holds a single providers list, where each entry describes one provider configuration. The minimum a usable entry needs is a name and a provider (the vendor: openai, gemini, anthropic, mistral, bedrock, or alibaba). Most vendors also require credentials to authenticate, but providers that rely on external authentication, such as AWS for Bedrock, can omit them. When you omit endpoint or model, KrakenD falls back to the vendor’s defaults, so entries like gpt-default and reasoning below are valid with only two or three fields:

{
  "$schema": "https://www.krakend.io/schema/krakend.json",
  "version": 4,
  "extra_config": {
    "ai/providers": {
      "providers": [
        {
          "name": "cheap-quick",
          "provider": "openai",
          "endpoint": "https://custom-self-hosted.example.com/v1/responses",
          "model": "gpt-4o-mini",
          "credentials": "xxx"
        },
        {
          "name": "gpt-default",
          "provider": "openai",
          "credentials": "xxx"
        },
        {
          "name": "gemini-flash",
          "provider": "gemini",
          "model": "gemini-2.5-flash",
          "credentials": "xxx"
        },
        {
          "name": "coding-basic",
          "provider": "anthropic",
          "model": "claude-haiku-4-5",
          "credentials": "xxx"
        },
        {
          "name": "coding-team",
          "provider": "anthropic",
          "model": "claude-sonnet-4-6",
          "credentials": "xxx"
        },
        {
          "name": "reasoning",
          "provider": "mistral",
          "credentials": "xxx"
        },
        {
          "name": "summaries",
          "provider": "bedrock",
          "model": "eu.amazon.nova-lite-v1:0",
          "debug": true,
          "variables": {
            "max_tokens": 1500
          }
        }
      ]
    }
  }
}

The configuration above declares seven providers across five vendors. The cheap-quick entry points to a self-hosted OpenAI-compatible endpoint, while the rest rely on each vendor’s default endpoint. The summaries entry omits credentials because Bedrock authenticates through AWS. Replace every "xxx" with your real credentials, preferably through an environment variable.

A provider entry can also carry custom templates, streaming, debugging, and vendor-specific variables:

• input_template and output_template point to Go template files that customize the request payload and the response transformation, the same mechanism as the Prompt Templates.
• streaming forwards the tokens to the client as the model produces them, and format normalizes the streamed output to the anthropic or openai wire format, regardless of the vendor. See LLM streaming.
• variables carries provider-specific parameters such as temperature. The accepted keys depend on the provider and match the variables documented on each vendor’s page.
• debug logs the provider activity for troubleshooting. Do not enable it in production, as it may log sensitive data.

These are all the fields of the ai/providers namespace:

Referencing a Provider from the LLM Component

Once a provider exists in the inventory, components reference it by name. The ai/llm component replaces its inline vendor block with a single provider property:

{
  "endpoint": "/",
  "method": "POST",
  "backend": [
    {
      "extra_config": {
        "ai/llm": {
          "provider": "cheap-quick"
        }
      }
    }
  ]
}

The backend above resolves the cheap-quick provider from the inventory and uses its vendor, endpoint, model, and credentials, so it needs neither host nor url_pattern. Because the provider is referenced by name, you can reuse it across many backends and change its definition in one place. To choose among several providers on each request based on headers, policies, or a classifier, use the AI Router.

Default Endpoints and Models

When you omit endpoint or model in a provider entry, KrakenD applies these defaults:

The Gemini and Bedrock default endpoints embed the selected model in the URL, and also change when you enable streaming; the defaults shown correspond to the default model without streaming. The Bedrock default targets the eu-west-1 region; set endpoint to use a different region. See each provider’s page for the available model names and the variables accepted in the variables object.