# Granular OpenTelemetry by endpoint

> Set different metrics and traces settings per endpoint or per backend individually, overriding existing OpenTelemetry settings that are defined at the service level

Source: https://www.krakend.io/docs/enterprise/telemetry/opentelemetry-by-endpoint/

The OpenTelemetry configuration is declared [at the service level](https://www.krakend.io/docs/enterprise/telemetry/opentelemetry/), but in the Enterprise edition you have more granularity and you can override metrics and traces per endpoint and per backend as follows.

## Endpoint Override of metrics and traces

The following example overrides some properties that could be declared at the service level, and changes the exporters defined there to use a different set.

While the `proxy` section can be defined in the Community edition, the `exporters_override` and `backend` sections are Enterprise only. Only a small set of settings are displayed in this example:

```json
{
  "endpoints": [
    {
      "endpoint": "/example",
      "backend": [
        {
          "host": [
            "example.com"
          ],
          "url_pattern": "/example"
        }
      ],
      "extra_config": {
        "telemetry/opentelemetry": {
          "global": {
            "traces_static_attributes": [
              {
                "key": "owner",
                "value": "team-charlie"
              }
            ],
            "metrics_static_attributes": [
              {
                "key": "owner",
                "value": "team-charlie"
              }
            ]
          },
          "proxy": {
            "disable_metrics": false,
            "disable_traces": false
          },
          "backend": {
            "metrics": {
              "read_payload": false,
              "static_attributes": [
                {
                  "key": "my_metric_attr",
                  "value": "my_metric_val"
                }
              ]
            },
            "exporters_override": {
              "metric_reporting_period": 10,
              "trace_sample_rate": 1,
              "metric_exporters": [
                "local_prometheus"
              ],
              "trace_exporters": [
                "debug_jaeger",
                "newrelic",
                "local_tempo"
              ]
            }
          }
        }
      }
    }
  ]
}
```

Exporter names must be typed exactly as in the service level. The full set of options are:

##### Options to override OpenTelemetry settings at the endpoint level

\* required fields

##### [`backend`](https://www.krakend.io/docs/enterprise/telemetry/opentelemetry-by-endpoint/#backend "Click to copy link to backend") object

**Enterprise only.** Reports the activity between KrakenD and each of your backend services. This is the more granular layer.

##### [`metrics`](#backend-metrics "Click to copy link to metrics") object

##### [`detailed_connection`](#metrics-detailed_connection "Click to copy link to detailed_connection") boolean

Whether you want to enable detailed metrics for the HTTP connection phase or not. Includes times to connect, DNS querying, and the TLS handshake.

Defaults to `false`

##### [`disable_stage`](#metrics-disable_stage "Click to copy link to disable_stage") boolean

Whether to turn off the metrics or not. Setting this to `true` means stop reporting any data.

Defaults to `false`

##### [`read_payload`](#metrics-read_payload "Click to copy link to read_payload") boolean

Whether you want to enable metrics for the response reading payload or not (HTTP connection not taken into account).

Defaults to `false`

##### [`round_trip`](#metrics-round_trip "Click to copy link to round_trip") boolean

Whether you want to enable metrics for the actual HTTP request for the backend or not (manipulation not taken into account). This is the time your backend needs to produce a result.

Defaults to `false`

##### [`static_attributes`](#metrics-static_attributes "Click to copy link to static_attributes") array of objects Show properties

A list of tags or labels you want to associate with these metrics.

Example: `[{"key":"my_metric_attr","value":"my_metric_val"}]`

Each item of static\_attributes accepts the following properties:

##### [`key`](#static_attributes-key "Click to copy link to key") * string

The key, tag, or label name you want to use.

##### [`value`](#static_attributes-value "Click to copy link to value") * string

The static value you want to assign to this key.

##### [`traces`](#backend-traces "Click to copy link to traces") object

##### [`detailed_connection`](#traces-detailed_connection "Click to copy link to detailed_connection") boolean

Whether you want to add detailed trace attributes for the HTTP connection phase or not. Includes times to connect, DNS querying, and the TLS handshake.

Defaults to `false`

##### [`disable_stage`](#traces-disable_stage "Click to copy link to disable_stage") boolean

Whether to turn off the traces or not. Setting this to `true` means stop reporting any data.

Defaults to `false`

##### [`read_payload`](#traces-read_payload "Click to copy link to read_payload") boolean

Whether you want to add trace attributes for the response reading payload or not (HTTP connection not taken into account).

Defaults to `false`

##### [`report_headers`](#traces-report_headers "Click to copy link to report_headers") boolean

Whether you want to report the final headers that reached the backend.

Defaults to `false`

##### [`round_trip`](#traces-round_trip "Click to copy link to round_trip") boolean

Whether you want to add trace attributes for the actual HTTP request for the backend or not (manipulation not taken into account). This is the time your backend needs to produce a result.

Defaults to `false`

##### [`skip_headers`](#traces-skip_headers "Click to copy link to skip_headers") array of strings

A list of headers you want to skip when reporting the headers that reached the backend. This is useful to avoid reporting sensitive data.

Example: `["X-Backend-Secret","X-All-Secret"]`

##### [`static_attributes`](#traces-static_attributes "Click to copy link to static_attributes") array of objects Show properties

A list of tags or labels you want to associate to these traces.

Example: `[{"key":"my_trace_attr","value":"my_trace_val"}]`

Each item of static\_attributes accepts the following properties:

##### [`key`](#static_attributes-key-2 "Click to copy link to key") * string

The key, tag, or label name you want to use.

##### [`value`](#static_attributes-value-2 "Click to copy link to value") * string

The static value you want to assign to this key.

##### [`exporters_override`](https://www.krakend.io/docs/enterprise/telemetry/opentelemetry-by-endpoint/#exporters_override "Click to copy link to exporters_override") object

**Enterprise only.** Override exporter configuration for this endpoint

##### [`metric_exporters`](#exporters_override-metric_exporters "Click to copy link to metric_exporters") array of strings

Overrides the metrics exporters used in this endpoint

Example: `["local_prometheus"]`

##### [`metric_reporting_period`](#exporters_override-metric_reporting_period "Click to copy link to metric_reporting_period") integer

Override how often you want to report and flush the metrics in seconds.

##### [`trace_exporters`](#exporters_override-trace_exporters "Click to copy link to trace_exporters") array of strings

Overrides the trace exporters used in this endpoint

Example: `["debug_jaeger","newrelic","local_tempo"]`

##### [`trace_sample_rate`](#exporters_override-trace_sample_rate "Click to copy link to trace_sample_rate") number

Overrides the sample rate for traces defines the percentage of reported traces. This option is key to reduce the amount of data generated (and resource usage), while you still can debug and troubleshoot issues. For instance, a number of `0.25` will report a 25% of the traces seen in the system. Do not rely on a value of `0` to stop sampling on this endpoint: at the service level, KrakenD treats `0` as unset and reports every trace.

Example: `0.25`

##### [`global`](https://www.krakend.io/docs/enterprise/telemetry/opentelemetry-by-endpoint/#global "Click to copy link to global")

Overrides the global configuration for this endpoint.

##### [`metrics_static_attributes`](#global-metrics_static_attributes "Click to copy link to metrics_static_attributes") array

Static attributes you want to pass for metrics. Overrides the `metrics_static_attributes` defined at the service level.

Each item of metrics\_static\_attributes accepts the following properties:

##### [`key`](#metrics_static_attributes-key "Click to copy link to key") string

The key of the static attribute you want to send

##### [`value`](#metrics_static_attributes-value "Click to copy link to value") string

The value of the static attribute you want to send

##### [`traces_static_attributes`](#global-traces_static_attributes "Click to copy link to traces_static_attributes") array

Static attributes you want to pass for traces. Overrides the `traces_static_attributes` defined at the service level.

Each item of traces\_static\_attributes accepts the following properties:

##### [`key`](#traces_static_attributes-key "Click to copy link to key") string

The key of the static attribute you want to send

##### [`value`](#traces_static_attributes-value "Click to copy link to value") string

The value of the static attribute you want to send

##### [`proxy`](https://www.krakend.io/docs/enterprise/telemetry/opentelemetry-by-endpoint/#proxy "Click to copy link to proxy") object

Reports the activity at the beginning of the proxy layer, including spawning the required requests to multiple backends, merging, endpoint transformation and any other internals of the proxy between the request processing and the backend communication

##### [`disable_metrics`](#proxy-disable_metrics "Click to copy link to disable_metrics") boolean

Whether you want to disable all metrics in this endpoint or not.

Defaults to `false`

##### [`disable_traces`](#proxy-disable_traces "Click to copy link to disable_traces") boolean

Whether you want to disable all traces in this endpoint or not.

Defaults to `false`

##### [`metrics_static_attributes`](#proxy-metrics_static_attributes "Click to copy link to metrics_static_attributes") array

Static attributes you want to pass for metrics.

Each item of metrics\_static\_attributes accepts the following properties:

##### [`key`](#metrics_static_attributes-key-2 "Click to copy link to key") string

The key of the static attribute you want to send

##### [`value`](#metrics_static_attributes-value-2 "Click to copy link to value") string

The value of the static attribute you want to send

##### [`report_headers`](#proxy-report_headers "Click to copy link to report_headers") boolean

Whether you want to report all headers that passed from the request to the proxy layer (`input_headers` policy in the endpoint plus KrakenD’s headers).

Defaults to `false`

##### [`skip_headers`](#proxy-skip_headers "Click to copy link to skip_headers") array of strings

A list of headers you want to skip when reporting headers passed to the proxy layer. This is useful to avoid reporting sensitive data.

Example: `["X-Sensitive-Data","X-Signature"]`

##### [`traces_static_attributes`](#proxy-traces_static_attributes "Click to copy link to traces_static_attributes") array

Static attributes you want to pass for traces.

Each item of traces\_static\_attributes accepts the following properties:

##### [`key`](#traces_static_attributes-key-2 "Click to copy link to key") string

The key of the static attribute you want to send

##### [`value`](#traces_static_attributes-value-2 "Click to copy link to value") string

The value of the static attribute you want to send

Schema: [https://www.krakend.io/schema/v3.0/telemetry/opentelemetry-endpoint.json](https://www.krakend.io/schema/v3.0/telemetry/opentelemetry-endpoint.json "JSON schema definition")

Do not use a trace sample rate of 0 to stop tracing

Do not rely on a `trace_sample_rate` of `0` to stop sampling on an endpoint. At the service level, KrakenD treats `0` as an unset value and reports every trace (see [OpenTelemetry](https://www.krakend.io/docs/enterprise/telemetry/opentelemetry/)). To keep only a small share of traces, use a small positive rate such as `0.01`.

## Backend Override of metrics and traces

For instance, you have a specific backend that is adding noise to your dashboards and you’d like to disable all layers:

```json
{
    "backend": [
        {
            "extra_config": {
                "telemetry/opentelemetry": {
                    "proxy": {
                        "disable_metrics": true,
                        "disable_traces": true
                    },
                    "backend": {
                        "metrics": {
                            "disable_stage": true
                        },
                        "traces": {
                            "disable_stage": true
                        }
                    }
                }
            },
            "url_pattern": "/noise",
            "host": [
                "example.com"
            ]
        }
    ]
}
```

These are the options:

##### Options to override OpenTelemetry settings at the backend level

\* required fields

##### [`backend`](https://www.krakend.io/docs/enterprise/telemetry/opentelemetry-by-endpoint/#backend "Click to copy link to backend") object

Reports the activity between KrakenD and each of your backend services. This is the more granular layer.

##### [`metrics`](#backend-metrics "Click to copy link to metrics") object

##### [`detailed_connection`](#metrics-detailed_connection "Click to copy link to detailed_connection") boolean

Whether you want to enable detailed metrics for the HTTP connection phase or not. Includes times to connect, DNS querying, and the TLS handshake.

Defaults to `false`

##### [`disable_stage`](#metrics-disable_stage "Click to copy link to disable_stage") boolean

Whether to turn off the metrics or not. Setting this to `true` means stop reporting any data.

Defaults to `false`

##### [`read_payload`](#metrics-read_payload "Click to copy link to read_payload") boolean

Whether you want to enable metrics for the response reading payload or not (HTTP connection not taken into account).

Defaults to `false`

##### [`round_trip`](#metrics-round_trip "Click to copy link to round_trip") boolean

Whether you want to enable metrics for the actual HTTP request for the backend or not (manipulation not taken into account). This is the time your backend needs to produce a result.

Defaults to `false`

##### [`static_attributes`](#metrics-static_attributes "Click to copy link to static_attributes") array of objects Show properties

a list of tags or labels you want to associate with these metrics.

see: [https://www.krakend.io/docs/telemetry/opentelemetry/](https://www.krakend.io/docs/telemetry/opentelemetry/)

Example: `[{"key":"my_metric_attr","value":"my_metric_val"}]`

Each item of static\_attributes accepts the following properties:

##### [`key`](#static_attributes-key "Click to copy link to key") * string

the key, tag, or label name you want to use.

see: [https://www.krakend.io/docs/telemetry/opentelemetry/](https://www.krakend.io/docs/telemetry/opentelemetry/)

##### [`value`](#static_attributes-value "Click to copy link to value") * string

the static value you want to assign to this key.

see: [https://www.krakend.io/docs/telemetry/opentelemetry/](https://www.krakend.io/docs/telemetry/opentelemetry/)

##### [`traces`](#backend-traces "Click to copy link to traces") object

##### [`detailed_connection`](#traces-detailed_connection "Click to copy link to detailed_connection") boolean

Whether you want to add detailed trace attributes for the HTTP connection phase or not. Includes times to connect, DNS querying, and the TLS handshake.

Defaults to `false`

##### [`disable_stage`](#traces-disable_stage "Click to copy link to disable_stage") boolean

Whether to turn off the traces or not. Setting this to `true` means stop reporting any data.

Defaults to `false`

##### [`read_payload`](#traces-read_payload "Click to copy link to read_payload") boolean

Whether you want to add trace attributes for the response reading payload or not (HTTP connection not taken into account).

Defaults to `false`

##### [`report_headers`](#traces-report_headers "Click to copy link to report_headers") boolean

Whether you want to report the final headers that reached the backend.

Defaults to `false`

##### [`round_trip`](#traces-round_trip "Click to copy link to round_trip") boolean

Whether you want to add trace attributes for the actual HTTP request for the backend or not (manipulation not taken into account). This is the time your backend needs to produce a result.

Defaults to `false`

##### [`skip_headers`](#traces-skip_headers "Click to copy link to skip_headers") array of strings

A list of headers you want to skip when reporting the headers that reached the backend. This is useful to avoid reporting sensitive data.

Example: `["X-Backend-Secret","X-All-Secret"]`

##### [`static_attributes`](#traces-static_attributes "Click to copy link to static_attributes") array of objects Show properties

a list of tags or labels you want to associate with these metrics.

see: [https://www.krakend.io/docs/telemetry/opentelemetry/](https://www.krakend.io/docs/telemetry/opentelemetry/)

Example: `[{"key":"my_metric_attr","value":"my_metric_val"}]`

Each item of static\_attributes accepts the following properties:

##### [`key`](#static_attributes-key-2 "Click to copy link to key") * string

the key, tag, or label name you want to use.

see: [https://www.krakend.io/docs/telemetry/opentelemetry/](https://www.krakend.io/docs/telemetry/opentelemetry/)

##### [`value`](#static_attributes-value-2 "Click to copy link to value") * string

the static value you want to assign to this key.

see: [https://www.krakend.io/docs/telemetry/opentelemetry/](https://www.krakend.io/docs/telemetry/opentelemetry/)

Schema: [https://www.krakend.io/schema/v3.0/telemetry/opentelemetry-backend.json](https://www.krakend.io/schema/v3.0/telemetry/opentelemetry-backend.json "JSON schema definition")
