Telemetry and Monitoring through OpenTelemetry

Document updated on Jan 29, 2024

OpenTelemetry (for short OTEL) offers a comprehensive, unified, and vendor-neutral approach to collecting and managing telemetry data, providing enhanced observability and deeper insights into application performance and behavior. It’s particularly beneficial in complex, distributed, and cloud-native environments.

OpenTelemetry captures detailed, contextual information about the operation of your applications. This includes not only metrics but also tracing data that shows the full lifecycle of requests as they flow through your systems, providing insights into performance bottlenecks, latency issues, and error diagnostics.

It supports auto-instrumentation and can be integrated seamlessly into cloud-native deployments, making it easier to monitor these dynamic environments.

Collecting metrics and traces

The telemetry/opentelemetry component in KrakenD collects the activity generated for the enabled layers and pushes or exposes the data for pulling. There are two ways of publishing metrics:

• OpenTelemetry protocol (OTLP) - push
• Prometheus - pull

You can use both simultaneously if needed, and even multiple instances of each.

When you add OpenTelemetry in the configuration, you will have different metrics available.

Prometheus exporter (pull)

Choose the prometheus exporter when you want KrakenD to expose a new port offering a /metrics endpoint. So, an external Prometheus job can connect to a URL like http://krakend:9090/metrics and retrieve all the data.

See how to configure Prometheus

OTLP exporter (push)

Choose the otlp exporter when you want to push the metrics to a local or remote collector or directly to a SaaS or storage system that supports native OTLP (there is a large number of supported providers). The following diagram represents this idea:

The host where your collector lives can also point to an external load balancer between KrakenD and multiple collectors if needed:

Enterprise users can push directly to external storage passing auth credentials using the telemetry/opentelemetry-security component, so the collector is not needed anymore:

This strategy saves a lot of time during the setup of KrakenD.

OpenTelemetry Configuration

To enable OpenTelemetry, you will need a Prometheus or an OTEL Collector (or both) and add the telemetry/opentelemetry namespace at the top level of your configuration.

The configuration of the telemetry/opentelemetry namespace is very extensive, but the two key entries are:

• exporters, defining the different technologies you will use
• layers, the amount of data you want to report

The entire configuration is as follows:

Here’s an example with a Grafana Tempo and a Prometheus.

{
    "version": 3,
    "$schema": "https://www.krakend.io/schema/v2.7/krakend.json",
    "extra_config": {
        "telemetry/opentelemetry": {
            "service_name": "krakend_middle_service",
            "service_version": "commit-sha-ACBDE1234",
            "exporters": {
                "prometheus": [
                    {
                        "name": "my_prometheus",
                        "port": 9092,
                        "listen_ip": "::1",
                        "process_metrics": false,
                        "go_metrics": false
                    }
                ],
                "otlp": [
                    {
                        "name": "local_tempo",
                        "host": "localhost",
                        "port": 4317,
                        "use_http": false
                    }
                ]
            },
            "layers": {
                "global": {
                    "disable_metrics": false,
                    "disable_traces": false,
                    "disable_propagation": false
                },
                "proxy": {
                    "disable_metrics": false,
                    "disable_traces": false
                },
                "backend": {
                    "metrics": {
                        "disable_stage": false,
                        "round_trip": true,
                        "read_payload": true,
                        "detailed_connection": true,
                        "static_attributes": [
                            {
                                "key": "my_metric_attr",
                                "value": "my_middle_metric"
                            }
                        ]
                    },
                    "traces": {
                        "disable_stage": false,
                        "round_trip": true,
                        "read_payload": true,
                        "detailed_connection": true,
                        "static_attributes": [
                            {
                                "key": "my_metric_attr",
                                "value": "my_middle_metric"
                            }
                        ]
                    }
                }
            },
            "skip_paths": [
                "/foo/{bar}"
            ]
        }
    }
}

Examples of integrations

• Push metrics to InfluxDB
• Pull metrics from Prometheus
• Push metrics to Datadog
• Push metrics to Zipkin
• Push metrics to Jaeger
• Push metrics to Azure Monitor