Document updated on Oct 11, 2024

Publisher/Subscribe with Kafka, NATS and cloud systems

You can connect an endpoint to multiple publish/subscribe backends, helping you integrate with event driven architectures.

For instance, a frontend client can push events to a queue using a REST interface. Or a client could consume a REST endpoint that is plugged to the last events pushed in a backend. You can even validate messages and formats as all the KrakenD available middleware can be used. The list of supported backend technologies is:

• NATS.io
• RabbitMQ
• Apache Kafka

Cloud-specific:

• AWS SNS (Simple Notification Service)
• AWS SQS (Simple Queueing Service)
• Azure Service Bus Topics and Subscriptions
• GCP PubSub

Configuration

To add pub/sub functionality to your backends include the namespaces backend/pubsub/subscriber and backend/pubsub/publisher under the extra_config of your backend section.

The host key defines the desired driver, and the actual host is usually set in an environment variable outside of KrakenD. The url_pattern is a mandatory field according to its Schema, but it is not used and you can type any value in it:

For a subscriber:

{
	"host": ["schema://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
			"subscription_url": "url"
		}
	}
}

For a publisher:

{
	"host": ["schema://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/publisher": {
			"topic_url": "url"
		}
	}
}

See the specification of each individual technology.

Example (RabbitMQ):

Set the envvar RABBIT_SERVER_URL='guest:guest@localhost:5672' and add in the configuration:

{
	"host": ["amqp://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
		"subscription_url": "myexchange"
		}
	}
}

Supported PubSub Drivers

GCP PubSub

Google’s Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between independent applications.

The configuration you need to use is:

• host: gcppubsub://
• url for topics: "projects/myproject/topics/mytopic" or the shortened form "myproject/mytopic"
• url for subscriptions: "projects/myproject/subscriptions/mysub" or the shortened form "myproject/mysub"
• Environment variables: GOOGLE_APPLICATION_CREDENTIALS, see Google documentation.

Example:

{
	"host": ["gcppubsub://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
		"subscription_url": "projects/myproject/subscriptions/mysub"
		}
	}
}

NATS

NATS.io is a simple, secure and high performance open source messaging system for cloud native applications, IoT messaging, and microservices architectures.

Configuration:

• host: nats://
• Environment variable: NATS_SERVER_URL
• url: mysubject

No query parameters are supported.

Example:

{
	"host": ["nats://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
			"subscription_url": "mysubject"
		}
	}
}

AWS SNS

Amazon Simple Notification Service (SNS) is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Amazon SNS provides topics for high-throughput, push-based, many-to-many messaging

AWS SNS sets the url without any host or environment variables, e.g:

{
	"host": ["awssns:///arn:aws:sns:us-east-2:123456789012:mytopic"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
			"subscription_url": "?region=us-east-2"
		}
	}
}

AWS SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.

AWS SQS sets the url without any host or environment variables, e.g:

Url: awssqs://sqs-queue-url

{
	"host": ["awssqs://sqs.us-east-2.amazonaws.com/123456789012"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
			"subscription_url": "/myqueue?region=us-east-2"
		}
	}
}

Azure Service Bus Topic and Subscription

Microsoft Azure Service Bus supports a set of cloud-based, message-oriented middleware technologies including reliable message queuing and durable publish/subscribe messaging. These “brokered” messaging capabilities can be thought of as decoupled messaging features that support publish-subscribe, temporal decoupling, and load balancing scenarios using the Service Bus messaging workload.

Configuration:

• host: azuresb://
• Environment variable: SERVICEBUS_CONNECTION_STRING
• Topics: mytopic
• Subscriptions: mytopic?subscription=mysubscription

Note that for subscriptions, the subscription name must be provided in the ?subscription= query parameter.

Example:

{
	"host": ["azuresb://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
			"subscription_url": "mytopic"
		}
	}
}

RabbitMQ

RabbitMQ is one of the most popular open source message brokers.

Rabbit can alternatively be configured using the AMQP component.

Configuration:

• host: rabbit://
• Environment variable: RABBIT_SERVER_URL
• url for topics: myexchange
• url for subscriptions: myqueue

No query parameters are supported.

Example:

{
	"host": ["rabbit://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
			"subscription_url": "myexchange"
		}
	}
}

Kafka

Apache Kafka is a distributed streaming platform.

Kafka connection requires KrakenD >= 1.1.

• host: kafka://
• Environment var: KAFKA_BROKERS pointing to the server(s), e.g: KAFKA_BROKERS=192.168.1.100:9092

Kafka subscriptions:

{
	"host": ["kafka://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/subscriber": {
			"subscription_url": "group?topic=mytopic"
		}
	}
}

Kafka topics:

{
	"host": ["kafka://"],
	"url_pattern": "/ignored",
	"disable_host_sanitize": true,
	"extra_config": {
		"backend/pubsub/publisher": {
			"topic_url": "mytopic"
		}
	}
}