Semantic Cache for LLM and Agent Responses

The semantic cache stores responses keyed by the meaning of the request rather than by an exact match. KrakenD turns the request content into an embedding vector and, when a new request is close enough in meaning to one it has already seen, returns the stored response instead of calling the backend again. Use it in front of LLMs and agents, where users phrase the same question in many different ways and every call costs tokens and time. The result is lower token spend and latency, and you can enable it on any endpoint or backend.

How the Semantic Cache Works

For each request, KrakenD takes the content you choose with req_content and converts it into a vector using an embedder (an ONNX model). It then queries Redis for the nearest stored vector. When the cosine distance between the incoming vector and a stored one is within max_distance, KrakenD treats it as a hit and returns the cached response without contacting the backend. When no stored vector is close enough, it is a miss: KrakenD calls the backend, returns the response, and stores it together with its vector for future requests.

A max_distance of 0 only matches identical vectors. A lower value requires requests to be more similar before they match, so it produces fewer but more precise hits. A higher value returns more hits at the risk of matching less related requests.

The semantic cache needs three things in place: an embedding model in ONNX format, a Redis connection to store the vectors and responses, and an embedder definition that links the two. You can optionally add OpenTelemetry to collect hit and miss metrics.

Downloading an Embedding Model

KrakenD does not include the embedding model in its distribution; you must provide it. Either add it manually to the models directory, or download it from Hugging Face with the built-in command. The model must be in ONNX format.

$krakend ai download-model -h

╓▄█                          ▄▄▌                               ╓██████▄µ
▐███  ▄███╨▐███▄██H╗██████▄  ║██▌ ,▄███╨ ▄██████▄  ▓██▌█████▄  ███▀╙╙▀▀███╕
▐███▄███▀  ▐█████▀"╙▀▀"╙▀███ ║███▄███┘  ███▀""▀███ ████▀╙▀███H ███     ╙███
▐██████▌   ▐███⌐  ,▄████████M║██████▄  ║██████████M███▌   ███H ███     ,███
▐███╨▀███µ ▐███   ███▌  ,███M║███╙▀███  ███▄```▄▄` ███▌   ███H ███,,,╓▄███▀
▐███  ╙███▄▐███   ╙█████████M║██▌  ╙███▄`▀███████╨ ███▌   ███H █████████▀
                     ``                     `'`

Version: 3.0.4-ee

Download model from Hugging Face. Model needs to have onnx format

Usage:
  krakend ai download-model [flags]

Examples:
krakend ai download-model --model sentence-transformers/all-MiniLM-L6-v2

Flags:
  -d, --cache-dir string   path to use for model download cache, defaults to a temp dir
  -h, --help               help for download-model
  -m, --model string       model to download (default "sentence-transformers/all-MiniLM-L6-v2")
  -o, --onnx-path string   model .onnx file (default "onnx/model.onnx")
  -p, --path string        path where to download models to (default "/etc/krakend/models")
  -t, --timeout string     download timeout (default "1m")
  -v, --verbose            verbose mode

The command accepts the following flags:

• --model (-m): The Hugging Face model to download. Defaults to sentence-transformers/all-MiniLM-L6-v2.
• --onnx-path (-o): The path of the .onnx file inside the model repository. Defaults to onnx/model.onnx.
• --path (-p): The directory where KrakenD writes the model. Defaults to /etc/krakend/models. The model ends up in a subfolder named <org>_<model>, such as /etc/krakend/models/sentence-transformers_all-MiniLM-L6-v2.
• --cache-dir (-d): The directory that caches the downloaded files. Defaults to a temporary directory that the command deletes when it finishes. See model download cache directory below.
• --timeout (-t): The maximum time for the whole download, all files together, as a duration with a unit such as 30s, 5m or 1h. Defaults to 1m.
• --verbose (-v): Prints additional details while downloading.

To download the default model into the default /etc/krakend/models directory:

$krakend ai download-model --model sentence-transformers/all-MiniLM-L6-v2

With Docker, mount a host directory on /etc/krakend/models so the model stays on your host after the container exits. The container runs as the krakend user (UID 1000), so the host directory must exist and be writable by UID 1000. Otherwise, Docker creates it as root and the download fails with a permission denied error. Create the directory first, and change its owner with sudo chown 1000 models if your host user is not UID 1000:

$mkdir -p models
docker run --rm -v "$PWD/models:/etc/krakend/models" krakend/krakend-ee:3.0 ai download-model

The container downloads the default model and writes it to the models directory on your host, inside the sentence-transformers_all-MiniLM-L6-v2 subfolder. When the download finishes, the command prints:

OK: model successfully downloaded to:   /etc/krakend/models/sentence-transformers_all-MiniLM-L6-v2

Model Download Cache Directory

The download-model command first downloads the model files into a cache directory, and then copies them to the --path directory. By default, the cache is a temporary directory under the operating system’s temp folder ($TMPDIR, or /tmp on Linux) that the command deletes when it finishes. As a result, every run downloads the full model again (about 87 MiB for the default model) and copies it to --path again, even when --path already contains it.

To keep the downloaded files between runs, set --cache-dir to a persistent directory. When the model is already in the cache, the command reuses it and does not use the network at all. With Docker, mount a host directory for the cache. As with the models directory, both directories must exist and be writable by UID 1000:

$mkdir -p models hf-cache
docker run --rm \
  -v "$PWD/hf-cache:/cache" \
  -v "$PWD/models:/etc/krakend/models" \
  krakend/krakend-ee:3.0 ai download-model --cache-dir /cache

The first run downloads the model into hf-cache and copies it to models. Later runs find the files in hf-cache and skip the download. Every run ends with the same message:

OK: model successfully downloaded to:   /etc/krakend/models/sentence-transformers_all-MiniLM-L6-v2

$mkdir -p models
docker run --rm -e XDG_CACHE_HOME=/tmp/cache -v "$PWD/models:/etc/krakend/models" krakend/krakend-ee:3.0.3 ai download-model

Service Configuration

Declare the shared resources in the root extra_config. You need a Redis connection pool, which the semantic cache uses as its vector storage, and the embedders under the ai/semantic-cache namespace. The optional OpenTelemetry block below exposes the cache hit and miss metrics through the Prometheus exporter:

{
  "$schema": "https://www.krakend.io/schema/krakend.json",
  "version": 4,
  "extra_config": {
    "redis": {
      "connection_pools": [
        {
          "name": "default",
          "address": "localhost:6379"
        }
      ]
    },
    "ai/semantic-cache": {
      "embedders": [
        {
          "name": "default",
          "model_path": "/etc/krakend/models/sentence-transformers_all-MiniLM-L6-v2",
          "vector_dimension": 384
        }
      ]
    },
    "telemetry/opentelemetry": {
      "service_name": "krakend_prometheus_service",
      "metric_reporting_period": 1,
      "trace_sample_rate": 1,
      "exporters": {
        "prometheus": [
          {
            "name": "local_prometheus",
            "port": 9090,
            "process_metrics": true,
            "go_metrics": true
          }
        ]
      }
    }
  }
}

The configuration above defines a default Redis connection and a default embedder pointing to a downloaded model. Endpoints and backends reference these resources by name. The vector_dimension must match the output of the model, for example 384 for all-MiniLM-L6-v2.

These are the embedder options at the service level:

If you don’t specify embedders, or it contains no embedder items, KrakenD defines a default embedder with default values instead. This makes the following configuration equivalent to declaring the default embedder of the example above:

{
  "$schema": "https://www.krakend.io/schema/krakend.json",
  "version": 4,
  "extra_config": {
    "ai/semantic-cache": {}
  }
}

Enabling the Cache on Endpoints and Backends

Add the ai/semantic-cache namespace to the extra_config of the endpoint or backend you want to cache. Every field is optional, so the minimum valid configuration is an empty object, which uses the default Redis connection, the default embedder, and the default distance and TTL:

{
  "extra_config": {
    "ai/semantic-cache": {}
  }
}

The following endpoint caches requests using the prompt in the body, scopes the cache per path parameter, and keeps entries for five minutes:

{
  "endpoint": "/1/{ids}",
  "input_headers": ["x-user"],
  "input_query_strings": ["*"],
  "method": "POST",
  "extra_config": {
    "ai/semantic-cache": {
      "redis_connection": "default",
      "embedder": "default",
      "max_distance": 0.15,
      "req_content": "req_body.prompt",
      "key_variants": [
        "req_params.Ids"
      ],
      "ttl": "300s"
    }
  },
  "backend": [
    {
      "url_pattern": "/__echo/1/{ids}"
    }
  ]
}

This configuration uses the default Redis connection and embedder, vectorizes only the prompt field of the request body, and treats vectors within a cosine distance of 0.15 as a match. It also keys the cache on the Ids path parameter through key_variants, and expires entries after 300s.

You can place the same configuration in a backend’s extra_config instead of the endpoint’s, to cache a single backend within the endpoint:

{
  "endpoint": "/3/{ids}",
  "input_headers": ["x-user"],
  "input_query_strings": ["*"],
  "method": "POST",
  "backend": [
    {
      "url_pattern": "/__echo/3/{ids}",
      "extra_config": {
        "ai/semantic-cache": {
          "redis_connection": "default",
          "embedder": "default",
          "max_distance": 0.15,
          "req_content": "req_params.Ids",
          "key_variants": [
            "req_headers.x-user"
          ],
          "ttl": "300s"
        }
      }
    }
  ]
}

These are the configuration options at the endpoint and backend levels:

Selecting the Content to Match

The req_content field decides what KrakenD vectorizes. Use it to focus the matching on the content that varies between requests: for example, if every request carries a large shared system prompt, select only the user prompt. By default it uses the full req_body, but you can point it at any of the request input sources:

• req_body: the request body. Use dot-notation to reach nested fields, for example req_body.content.prompt.
• req_params: a URL path parameter. The key is capitalized, so the endpoint /1/{ids} exposes its parameter as req_params.Ids.
• req_query_string: a query string value, for example req_query_string.user.
• req_headers: a request header, for example req_headers.x-user.

The key_variants field takes a list of selectors and uses them to partition the cache. Requests with different values for these fields are stored and matched separately, even when their vectorized content is identical, which lets you keep separate cache entries per user, tenant, or any other dimension. Unlike req_content, key_variants reads only from req_params, req_query_string, and req_headers; it does not support req_body. For example, caching per user with the x-user header:

{
  "extra_config": {
    "ai/semantic-cache": {
      "req_content": "req_body.prompt",
      "key_variants": [
        "req_headers.x-user"
      ],
      "ttl": "300s"
    }
  }
}

Sharing Cache Between Endpoints and Backends

Different endpoints never share cache between them. No configuration option enables it.

The same backend shared across different endpoints shares its cache when the HTTP method, host, and URL match, the vector dimension and key variants are the same, and the encoding is compatible.

Hit and Miss Metrics

When you configure OpenTelemetry, the semantic cache reports cache hit and miss metrics. Export them through the Prometheus exporter, as shown in the service configuration above, to monitor the cache effectiveness and tune max_distance and ttl.