Document updated on Mar 5, 2026

Kafka Driver for the Asynchronous Agent

The Kafka driver for async agents allows KrakenD to consume Kafka queues autonomously. Routines listening to Kafka topics react to new events and push data to your backends.

This driver differs from the Kafka backend consumer. Unlike endpoints, async agents do not require a client request to trigger an action. Instead, the agent subscribes to the topics and fires an action when an event is delivered.

Driver Configuration

Place the Kafka driver inside the extra_config of the async component to subscribe to one or more topics:

{
    "async/kafka": {
        "cluster": {
            "brokers": [
                "localhost:9092"
            ],
            "client_id": "cid_stocksconsumer"
        },
        "group": {
            "group_id": "my_group_id",
            "isolation_level": "read_commited"
        },
        "key_meta": "Message-Id"
    }
}

KrakenD consumes messages following the “latest” offset policy, meaning it processes only messages produced after startup.

The Kafka driver configuration accepts the same fields as the reader field in backend kafka consumer (backend/pubsub/kafka/subscriber), except for the topics field that is defined in the general async agent configuration:

The cluster field describes details about the connection to the Kafka cluster. It is the same configuration used for the Kafka PubSub publisher and subscriber:

The cluster field includes a client_tls field that allows you to authenticate with the Kafka service, as described in Service Client TLS Settings.

You can also use Kafka user and password authentication with the SASL config:

{
    "cluster": {
        "brokers": [
            "localhost:9092"
        ],
        "sasl": {
            "user": "johnsmith",
            "password": "myp4ssword"
        }
    }
}

The group field has details about the Kafka consumer group behaviour:

ACK Behaviour in Kafka

Kafka has no per-message ACK: acknowledging a Kafka message commits the client’s partition offset.

KrakenD ACKs every message read, even when the response of the pipeline results in a non-successful status code.