News KrakenD CE v2.8 released with improved Lua and OpenTelemetry

Community Documentation

You are viewing a previous version of KrakenD Community Edition (v2.6), go to the latest version

Document updated on Mar 1, 2024

Understanding OpenTelemetry layers and metrics

You can add several exporters to your OpenTelemetry configuration (the more, the hungrier the gateway will be), and KrakenD will send data to all the declared exporters and layers by default.

While exporters define where you will have the metrics, the layers define which metrics you want to have. The layers contain the traces and metrics for a subset of the execution flow. These are the layers you can use:

Diagram showing global, proxy, and backend sequence

  • global: The global layer contains everything that KrakenD saw in and out. It includes the total timings for a request hitting the service until it is delivered to the client.
  • proxy: The proxy layer starts acting after processing the HTTP request and comprehends the internal work of dealing with the endpoint, including spawning multiple requests to the backends, aggregating their results, and manipulating the final response.
  • backend: The backend layer monitors the activity of a single backend request and response with any additional components in its level. It contains mainly the timings between KrakenD and your service. It is the richest layer of all.

It is vital to see that backend is a subset of proxy, and proxy is a subset of global. Do not generate metrics for the layers you will not use.

CPU and memory consumption

As you will see below, the amount of telemetry data you can send for each request is massive. The more layers and details you add, the more resources KrakenD will consume and the more storage you will need to save metrics and traces. Choose carefully a balance between observability power and resource consumption to save money!

In production, you should only enable a small sample rate of the traces (e.g., 5%). The data generated in telemetry is usually more extensive than the responses of the services you expose.

Once you configure the OpenTelemetry component, KrakenD will start reporting metrics when there is activity. The section below describes different metrics in each layer, briefly explaining their meaning.

The presence of the metrics and the traces described below depend on the OpenTelemetry configuration.

Data exposed in the global layer

The global layer reveals all timings in and out of KrakenD. If you have custom server plugins, their timings are also included here. Enable this layer when you want to see the complete picture of the gateway.

You will have metrics, traces, or both available depending on the exporter you use.

Metrics of the global layer

Present when you have an exporter able to collect metrics and the global attribute configured inside layers

http.server.duration float

Histogram of the time in seconds that it takes to produce the response

→ http.response.status_code integer

Status code of the produced response

→ http.route string

The matched endpoint path

→ http.request.method string

The method sent by the user (GET, POST, etc.)

http.server.response.size integer

Histogram of the size of the body produced for the response in bytes

→ http.response.status_code integer

Status code of the produced response

→ http.route string

The matched endpoint path

→ http.request.method string

The method sent by the user (GET, POST, etc.)

Traces of the global layer

Present when you have an exporter able to collect traces and the global attribute configured inside layers. In case an error happens, the span will record the error and set the span status to error (value = 1), in case of success the span status is set to ok (value = 2) as per the OTEL definition.

/foo/123 string

A span is created with the name of the requested path and is depicted as /foo/123 as an example.

→ krakend.stage string

Contains the value global. It’s meant to filter traces of this stage.

→ http.route string

The associated endpoint definition for this route. E.g., /foo/{bar}. In case no matching is made, for example in 404 results, it would be empty.

→ http.response.body.size integer

Histogram of the size in bytes of the body produced for the response.

→ http.response.status_code integer

Status code of the produced response

→ http.request.method string

The method sent by the user (GET, POST, etc.)

→ http.request.header.xxx array

When the report_headers is set in the configuration, you will have one extra span for each incoming header (xxx represents here any header name in lower case).

→ http.connection.hijacked bool

If an HTTP request is upgraded to WebSockets, gRPC, etc. and cannot be longer treated as HTTP.

Data exposed in the proxy layer

The proxy layer starts counting after processing the HTTP request and comprehends the internal work of dealing with the endpoint, including spawning multiple requests to the backends, aggregating their results, and manipulating the final response.

Metrics of the proxy layer

Present when you have an exporter able to collect metrics and the proxy attribute configured inside layers

krakend.proxy.duration float

Histogram of the time in seconds that it takes to produce the response

→ error bool

When the endpoint failed to fetch any data from the backend(s).

→ canceled bool

When the client aborted the request

→ complete bool

When all backends successfully returned their information. Matches the header X-KrakenD-Complete.

→ http.response.status_code integer

Status code of the produced response after aggregation

→ http.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ http.request.method string

The method sent by the user (GET, POST, etc.)

Traces of the proxy layer

Present when you have an exporter able to collect traces and the proxy attribute configured inside layers. The spans revealed here include aggregation.

/foo/123 string

A span is created with the name of the requested path and is depicted as /foo/123 as an example.

→ krakend.stage string

Contains the value proxy. It’s meant to filter traces of this stage.

→ error string

When there is an error, the description

→ canceled bool

It appears with a true value when the request has been canceled (usually when concurrent_calls are used).

→ complete bool

When all backends successfully returned their information. Matches the header X-KrakenD-Complete.

→ http.request.header.xxx array

When the report_headers is set in the configuration, you will have one extra span for each incoming header (xxx represents here any header name in lower case).

→ http.request.method string

The method declared in the endpoint

→ http.route string

The associated endpoint definition for this route. E.g., /foo/{bar}. In case no matching is made, for example in 404 results, it would be empty.

→ http.response.status_code integer

Status code of the produced response after aggregation

Data exposed in the backend layer

The backend layer monitors the activity of a single backend request and response with any additional components in its level. It contains mainly the timings between KrakenD and your service. It is the richest layer of all.

Metrics of the backend layer

Present when you have an exporter able to collect metrics and the backend attribute configured inside layers. To see all http.client metrics you need to enable the round_trip in the backend layer. To see the http.client.request. header you must enable the read_payload flag

krakend.backend.duration float

Histogram of the time in seconds that it takes to produce the response. The metric includes the whole timing, including any used to perform manipulations at the backend level.

→ error bool

When the backend request failed.

→ canceled bool

When the gateway aborted the request to the backend (e.g., got an answer from another thread using concurrent_calls)

→ complete bool

When the backend successfully returned the information.

→ http.response.status_code integer

Status code of the produced response after aggregation

→ http.route string

The associated url_pattern for this route.

→ krakend.endpoint.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.stage string

Contains the value backend. It’s meant to filter traces of this stage.

→ server.address string

Targeted hosts. In case more than one is provided, these are joined with _.

http.client.request.started.count integer

The number of requests started.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.failed.count integer

The number of requests that failed.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.canceled.count integer

The number of requests canceled.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.timedout.count integer

The number of requests that timed-out.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.content.length integer

Counter with the sum of bytes in the Content-Length header.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.duration integer

Histogram with the seconds taken until receiving the first byte of the response.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.response.size integer

Histogram with the size of response bodies in bytes as reported in the Content-Lenght header. When the header is absent, the metric does not exist.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.response.no-content-length integer

When the Content-Length is not set, this metric is created. If you sum the number of http.client.response.size and the value of http.client.response.no-content-length they represent the total responses.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.get-conn.duration float

Reports the duration of the connection with the backend in seconds, excluding any data processing time. Requires the flag detailed_connection

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.tls.duration float

Reports the duration of the TLS negotiation in seconds.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.dns.duration float

Reports the duration of the DNS retrieval in seconds.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

http.client.request.read.size integer

Number of read bytes.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.endpoint.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string

The associated method of the parent endpoint.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string

Targeted host.

http.client.request.read.size-hist integer

Histogram with the read bytes.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.endpoint.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string

The associated method of the parent endpoint.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string

Targeted host.

http.client.request.read.time integer

Counter with seconds spent reading the body payload of the response.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.endpoint.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string

The associated method of the parent endpoint.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string

Targeted host.

http.client.request.read.time-hist integer

Histogram with the seconds spent reading the body.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.endpoint.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string

The associated method of the parent endpoint.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string

Targeted host.

http.client.request.read.errors integer

A counter with the number of errors that happened reading the response body.

→ http.request.method string

The method used to connect to the backend

→ http.route string

The resolved url_pattern for this route.

→ krakend.endpoint.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string

The associated method of the parent endpoint.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

→ server.address string

Targeted host.

Traces of the backend layer

Present when you have an exporter able to collect traces and the backend attribute configured inside layers. There is a span for the backend part, which includes possible backend manipulations, and another span for the backend-request, which include the interaction with the backend.

/my-pattern string

A span is created with the name of the resolved url_pattern is depicted as /my-pattern as an example.

→ krakend.stage string

Contains the value backend. It’s meant to filter traces of this stage.

→ error string

When there is an error, the description

→ canceled bool

It appears with a true value when the request has been canceled (usually when concurrent_calls are used).

→ complete bool

When the backend successfully returned the information.

→ http.request.header.xxx array

When the report_headers is set in the configuration, you will have one extra span for each incoming header (xxx represents here any header name in lower case).

→ http.request.method string

The method declared in the endpoint

→ http.route string

The associated url_pattern for this route.

→ krakend.endpoint.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ krakend.endpoint.method string

The associated method of the parent endpoint.

→ server.address string

Targeted hosts. In case more than one is provided, these are joined with _.

/my-pattern string

A span is created with the name of the resolved url_pattern is depicted as /my-pattern as an example. Notice that this span shares the name with the previous but is for the backend-request instead of backend.

→ krakend.stage string

Contains the value backend-request. It’s meant to filter traces of this stage.

→ dns-duration integer

The duration in seconds of the DNS resolution. Requires the flag detailed_connection

→ get-conn-duration integer

The duration in seconds to establish the connection. Requires the flag detailed_connection

→ tls-duration integer

The duration in seconds to perform the TLS handshake. Requires the flag detailed_connection

→ http.request.body.size integer

Size of the request body (payload) in bytes.

→ http.request.header.xxx array

When the report_headers is set in the configuration, you will have one extra span for each incoming header (xxx represents here any header name in lower case).

→ http.response.status_code integer

Status code of the produced response after aggregation

→ http.route string

The associated url_pattern for this route.

→ krakend.endpoint.route string

The associated endpoint definition for this route. E.g., /foo/{bar}.

→ http.request.method string

The method declared in the endpoint

→ krakend.endpoint.method string

The associated method of the parent endpoint.

→ http.route string

The associated url_pattern for this route.

→ server.address string

Targeted host.

→ url.full string

The complete address used to retrieve the content, including protocol, port and path (e.g., http://example.com:9876/path)

→ user_agent.original string

The user agent used when connecting

http.client.request.read.tracker string

When the read_payload option is set to true in the configuration, the time spent to read the body. Enable only in debugging scenarios.

→ http.client.request.read.size integer

Number of read bytes.

→ http.client.request.read.size integer

Seconds spent to read the request

Scarf

Unresolved issues?

The documentation is only a piece of the help you can get! Whether you are looking for Open Source or Enterprise support, see more support channels that can help you.

See all support channels