Understanding OpenTelemetry layers and metrics

Document updated on Mar 1, 2024

You can add several exporters to your OpenTelemetry configuration (the more, the hungrier the gateway will be), and KrakenD will send data to all the declared exporters and layers by default.

While exporters define where you will have the metrics, the layers define which metrics you want to have. The layers contain the traces and metrics for a subset of the execution flow. These are the layers you can use:

Diagram showing global, proxy, and backend sequence

global: The global layer contains everything that KrakenD saw in and out. It includes the total timings for a request hitting the service until it is delivered to the client.
proxy: The proxy layer starts acting after processing the HTTP request and comprehends the internal work of dealing with the endpoint, including spawning multiple requests to the backends, aggregating their results, and manipulating the final response.
backend: The backend layer monitors the activity of a single backend request and response with any additional components in its level. It contains mainly the timings between KrakenD and your service. It is the richest layer of all.

It is vital to see that backend is a subset of proxy, and proxy is a subset of global. Do not generate metrics for the layers you will not use.

CPU and memory consumption

As you will see below, the amount of telemetry data you can send for each request is massive. The more layers and details you add, the more resources KrakenD will consume and the more storage you will need to save metrics and traces. Choose carefully a balance between observability power and resource consumption to save money!

In production, you should only enable a small sample rate of the traces (e.g., 5%). The data generated in telemetry is usually more extensive than the responses of the services you expose.

Once you configure the OpenTelemetry component, KrakenD will start reporting metrics when there is activity. The section below describes different metrics in each layer, briefly explaining their meaning.

The presence of the metrics and the traces described below depend on the OpenTelemetry configuration.

Data exposed in the global layer

The global layer reveals all timings in and out of KrakenD. If you have custom server plugins, their timings are also included here. Enable this layer when you want to see the complete picture of the gateway.

You will have metrics, traces, or both available depending on the exporter you use.

Metrics of the `global` layer

Present when you have an exporter able to collect metrics and the global attribute configured inside layers

http.server.duration float Histogram of the time in seconds that it takes to produce the response

→ http.response.status_code integer Status code of the produced response

→ http.route string The matched endpoint path

→ http.request.method string The method sent by the user (GET, POST, etc.)

http.server.response.size integer Histogram of the size of the body produced for the response in bytes

→ http.response.status_code integer Status code of the produced response

→ http.route string The matched endpoint path

→ http.request.method string The method sent by the user (GET, POST, etc.)

Traces of the `global` layer

Present when you have an exporter able to collect traces and the global attribute configured inside layers. In case an error happens, the span will record the error and set the span status to error (value = 1), in case of success the span status is set to ok (value = 2) as per the OTEL definition.

/foo/123 string A span is created with the name of the requested path and is depicted as /foo/123 as an example.

→ krakend.stage string Contains the value global. It’s meant to filter traces of this stage.

→ http.route string The associated endpoint definition for this route. E.g., /foo/{bar}. In case no matching is made, for example in 404 results, it would be empty.

→ http.response.body.size integer Histogram of the size in bytes of the body produced for the response.

→ http.response.status_code integer Status code of the produced response

→ http.request.method string The method sent by the user (GET, POST, etc.)

→ http.request.header.xxx array When the report_headers is set in the configuration, you will have one extra span for each incoming header (xxx represents here any header name in lower case).

→ http.connection.hijacked bool If an HTTP request is upgraded to WebSockets, gRPC, etc. and cannot be longer treated as HTTP.

Data exposed in the proxy layer

The proxy layer starts counting after processing the HTTP request and comprehends the internal work of dealing with the endpoint, including spawning multiple requests to the backends, aggregating their results, and manipulating the final response.

Metrics of the `proxy` layer

Present when you have an exporter able to collect metrics and the proxy attribute configured inside layers

krakend.proxy.duration float Histogram of the time in seconds that it takes to produce the response

→ error bool When the endpoint failed to fetch any data from the backend(s).

→ canceled bool When the client aborted the request

→ complete bool When all backends successfully returned their information. Matches the header X-KrakenD-Complete.

→ http.response.status_code integer Status code of the produced response after aggregation

→ http.route string The associated endpoint definition for this route. E.g., /foo/{bar}.

→ http.request.method string The method sent by the user (GET, POST, etc.)

Traces of the `proxy` layer

Present when you have an exporter able to collect traces and the proxy attribute configured inside layers. The spans revealed here include aggregation.

/foo/123 string A span is created with the name of the requested path and is depicted as /foo/123 as an example.

→ krakend.stage string Contains the value proxy. It’s meant to filter traces of this stage.

→ error string When there is an error, the description

→ canceled bool It appears with a true value when the request has been canceled (usually when concurrent_calls are used).

→ complete bool When all backends successfully returned their information. Matches the header X-KrakenD-Complete.

→ http.request.header.xxx array When the report_headers is set in the configuration, you will have one extra span for each incoming header (xxx represents here any header name in lower case).

→ http.request.method string The method declared in the endpoint

→ http.route string The associated endpoint definition for this route. E.g., /foo/{bar}. In case no matching is made, for example in 404 results, it would be empty.

→ http.response.status_code integer Status code of the produced response after aggregation

Data exposed in the backend layer

The backend layer monitors the activity of a single backend request and response with any additional components in its level. It contains mainly the timings between KrakenD and your service. It is the richest layer of all.

Metrics of the `backend` layer

Present when you have an exporter able to collect metrics and the backend attribute configured inside layers. To see all http.client metrics you need to enable the round_trip in the backend layer. To see the http.client.request. header you must enable the read_payload flag

krakend.backend.duration float Histogram of the time in seconds that it takes to produce the response. The metric includes the whole timing, including any used to perform manipulations at the backend level.

→ error bool When the backend request failed.

→ canceled bool When the gateway aborted the request to the backend (e.g., got an answer from another thread using concurrent_calls)

→ complete bool When the backend successfully returned the information.

→ http.response.status_code integer Status code of the produced response after aggregation

→ http.route string The associated url_pattern for this route.

→ krakend.endpoint.route string The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.stage string Contains the value backend. It’s meant to filter traces of this stage.

→ server.address string Targeted hosts. In case more than one is provided, these are joined with _.

http.client.request.started.count integer The number of requests started.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.failed.count integer The number of requests that failed.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.canceled.count integer The number of requests canceled.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.timedout.count integer The number of requests that timed-out.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.content.length integer Counter with the sum of bytes in the Content-Length header.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.duration integer Histogram with the seconds taken until receiving the first byte of the response.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.response.size integer Histogram with the size of response bodies in bytes as reported in the Content-Lenght header. When the header is absent, the metric does not exist.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.response.no-content-length integer When the Content-Length is not set, this metric is created. If you sum the number of http.client.response.size and the value of http.client.response.no-content-length they represent the total responses.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.get-conn.duration float Reports the duration of the connection with the backend in seconds, excluding any data processing time. Requires the flag detailed_connection

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.tls.duration float Reports the duration of the TLS negotiation in seconds.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.dns.duration float Reports the duration of the DNS retrieval in seconds.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.read.size integer Number of read bytes.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.endpoint.route string The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string The associated method of the parent endpoint.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string Targeted host.

http.client.request.read.size-hist integer Histogram with the read bytes.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.endpoint.route string The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string The associated method of the parent endpoint.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string Targeted host.

http.client.request.read.time integer Counter with seconds spent reading the body payload of the response.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.endpoint.route string The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string The associated method of the parent endpoint.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string Targeted host.

http.client.request.read.time-hist integer Histogram with the seconds spent reading the body.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.endpoint.route string The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string The associated method of the parent endpoint.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string Targeted host.

http.client.request.read.errors integer A counter with the number of errors that happened reading the response body.

→ http.request.method string The method used to connect to the backend

→ http.route string The resolved url_pattern for this route.

→ krakend.endpoint.route string The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string The associated method of the parent endpoint.

→ krakend.stage string Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string Targeted host.

Traces of the `backend` layer

Present when you have an exporter able to collect traces and the backend attribute configured inside layers. There is a span for the backend part, which includes possible backend manipulations, and another span for the backend-request, which include the interaction with the backend.

/my-pattern string A span is created with the name of the resolved url_pattern is depicted as /my-pattern as an example.