Document updated on Mar 5, 2026

Kafka PubSub with Extended Connectivity

Kafka PubSub enables KrakenD to publish messages to and consume messages from Kafka topics directly from backend configurations. Use this component when you need to secure your Kafka connections with mutual TLS (mTLS) or SASL authentication. It provides two backend namespaces: backend/pubsub/publisher/kafka for publishing and backend/pubsub/subscriber/kafka for subscribing.

Publisher Configuration

Add the backend/pubsub/publisher/kafka namespace under a backend’s extra_config to publish messages to a Kafka topic. KrakenD converts the backend’s input_headers into message metadata.

The following configuration enables publishing to the orderplacement topic with mTLS authentication:

{
  "backend/pubsub/publisher/kafka": {
    "success_status_code": 201,
    "writer": {
        "topic": "orderplacement",
        "key_meta": "X-My-Key",
        "cluster": {
            "brokers": [
                "localhost:49092"
            ],
            "client_id": "krakend_async_agent",
            "client_tls": {
                "allow_insecure_connections": false,
                "ca_certs": [
                    "./config/ca/cacert.pem"
                ],
                "client_certs": [
                    {
                        "certificate": "./config/certs/client/client.signed.pem",
                        "private_key": "./config/certs/client/client.key"
                    }
                ]
            }
        },
        "producer": {
            "idempotent": true
        }
    }
  }
}

The top-level fields for backend/pubsub/publisher/kafka are:

The writer field contains the configuration for writing to a Kafka topic:

The cluster field describes details about the connection to the Kafka cluster. It is the same configuration used for the Kafka PubSub subscriber and the Kafka async agent:

The cluster field includes a client_tls field that allows you to authenticate with the Kafka service, as described in Service Client TLS Settings.

You can also use user and password authentication with the SASL config:

The producer field has details about how to write messages:

OpenTelemetry

Metrics

When OpenTelemetry is enabled, KrakenD reports the following metrics for all messages:

• messaging.write.body.size: histogram of body sizes in bytes (does not include metadata).
• messaging.write.body.duration: histogram of duration taken to write a message.
• messaging.write.failure.count: histogram of the time taken to fail to write a message.

All metrics include the following attributes:

• kind: identifies the messaging system (in this case kafka).
• topic: the topic where the message is written to.

Traces

KrakenD adds information about the write process as a new span if a trace is active, with the following fields:

• kind: identifies the messaging system (in this case kafka).
• topic: the topic where the message is written to.
• messaging.write.body.size: size of the message body.
• messaging.write.body.duration: time taken to write the message in seconds.

Subscriber Configuration

Add the backend/pubsub/subscriber/kafka namespace under a backend’s extra_config to consume messages from a Kafka topic. It contains a reader field with the same configuration as the Kafka async agent driver.

The following configuration subscribes to the orderplacement topic with mTLS authentication:

{
    "backend/pubsub/subscriber/kafka": {
        "reader": {
            "topics": ["orderplacement"],
            "key_meta": "X-Idempotency-Key",
            "cluster": {
                "brokers": [
                    "localhost:49092"
                ],
                "client_id": "krakend_async_agent",
                "client_tls": {
                    "allow_insecure_connections": false,
                    "ca_certs": [
                        "./config/ca/cacert.pem"
                    ],
                    "client_certs": [
                        {
                            "certificate": "./config/certs/client/client.signed.pem",
                            "private_key": "./config/certs/client/client.key"
                        }
                    ]
                }
            },
            "group": {
                "id": "k_endpoint_read",
                "isolation_level": "read_commited"
            }
        }
    }
}

KrakenD consumes messages following the “latest” offset policy, meaning it processes only messages produced after KrakenD starts.

The reader field defines how to read from a Kafka cluster:

The cluster field describes details about the connection to the Kafka cluster. It is the same configuration used for the Kafka PubSub publisher and the Kafka async agent:

The cluster field includes a client_tls field that allows you to authenticate with the Kafka service, as described in Service Client TLS Settings.

You can also use user and password authentication with the SASL config:

The group field has details about the Kafka consumer group:

OpenTelemetry

Metrics

When OpenTelemetry is enabled, KrakenD reports the following metrics for all messages read:

• messaging.read.body.size: histogram of body sizes in bytes (does not include metadata).
• messaging.read.body.duration: histogram of the duration taken to read a message.
• messaging.read.ack.duration: histogram of the time taken to acknowledge the message.
• messaging.read.failure.count: histogram of time taken to fail reading a message.

All metrics include the following attributes:

• kind: identifies the messaging system (for example kafka).
• topic: the topic where the message is read from.
• origin: always set to subscriber, to differentiate reading metrics from those of an async agent (which can take longer as it waits for new events).

Traces

KrakenD adds information about the read process as a new span if a trace is active, with the following fields:

• kind: identifies the messaging system (for example kafka).
• topic: the topic where the message is read from.
• messaging.read.body.size: size of the read message body.
• messaging.read.body.duration: time taken to read a message in seconds.

ACK Behaviour in Kafka

Kafka has no per-message ACK: acknowledging a Kafka message commits the client’s partition offset.

KrakenD ACKs every message read, even when the response of the pipeline results in a non-successful status code.