Distributed Tracing

Configure distributed tracing with OpenTelemetry to visualize request flows across your infrastructure using Jaeger, Zipkin, and more.

Distributed Tracing

Distributed tracing provides end-to-end visibility into request flows across your infrastructure. Traefik uses OpenTelemetry, an open standard for distributed tracing compatible with Jaeger, Zipkin, and other tracing backends.

Overview

Tracing allows you to:

• Visualize request propagation across services
• Identify performance bottlenecks
• Analyze latency distribution
• Track errors through the request lifecycle
• Understand service dependencies

Traefik follows OpenTelemetry semantic conventions v1.37.0.

Quick Start

Enable basic tracing:

YAML

tracing:
  otlp: {}

By default, traces are sent via HTTPS to https://localhost:4318/v1/traces.

Common Configuration

Service Name

serviceName · string · default: "traefik"

Service name identifier in the tracing backend.

YAML

tracing:
  serviceName: traefik-prod
  otlp: {}

Sample Rate

sampleRate · float · default: "1.0"

Proportion of requests to trace (0.0 to 1.0).

Set to 0.1 to trace 10% of requests:

YAML

tracing:
  sampleRate: 0.1

The OpenTelemetry SDK also supports the OTEL_TRACES_SAMPLER environment variable for custom sampling strategies.

Add Internals

addInternals · boolean · default: "false"

Enable tracing for internal resources like ping@internal.

YAML

tracing:
  addInternals: true

Resource Attributes

resourceAttributes · object · default: "{}"

Additional resource attributes sent to the collector.

YAML

tracing:
  resourceAttributes:
    environment: production
    datacenter: us-east-1
    version: v2.10

Captured Headers

Capture request and response headers as span attributes:

capturedRequestHeaders · array · default: "[]"

List of request headers to add as span attributes.

capturedResponseHeaders · array · default: "[]"

List of response headers to add as span attributes.

YAML

tracing:
  capturedRequestHeaders:
    - User-Agent
    - X-Request-Id
    - Authorization
  capturedResponseHeaders:
    - X-Response-Time
    - Content-Type

Safe Query Parameters

safeQueryParams · array · default: "[]"

Query parameters to NOT redact (all others are redacted by default).

By default, all query parameters are redacted for security. Specify safe parameters:

YAML

tracing:
  safeQueryParams:
    - page
    - limit
    - sort

OpenTelemetry Configuration

HTTP Protocol

Send traces via HTTP to an OpenTelemetry Collector:

YAML

tracing:
  otlp:
    http:
      endpoint: https://otel-collector:4318/v1/traces
      headers:
        X-API-Key: secret-key
      tls:
        ca: /path/to/ca.crt
        cert: /path/to/client.cert
        key: /path/to/client.key
        insecureSkipVerify: false

HTTP Configuration Options

otlp.http.endpoint · string · default: "https://localhost:4318/v1/traces"

URL of the OpenTelemetry Collector.

To disable TLS, use http:// instead of https:// in the endpoint URL.

otlp.http.headers · object · default: "{}"

Additional headers sent with traces.

otlp.http.tls.ca · string

Path to certificate authority file.

otlp.http.tls.cert · string

Path to client certificate file.

otlp.http.tls.key · string

Path to client private key file.

otlp.http.tls.insecureSkipVerify · boolean · default: "false"

Skip TLS certificate verification.

gRPC Protocol

Send traces via gRPC to an OpenTelemetry Collector:

YAML

tracing:
  otlp:
    grpc:
      endpoint: otel-collector:4317
      insecure: false
      headers:
        Authorization: Bearer token123
      tls:
        ca: /path/to/ca.crt
        cert: /path/to/client.cert
        key: /path/to/client.key

gRPC Configuration Options

otlp.grpc.endpoint · string · default: "localhost:4317"

Address of the OpenTelemetry Collector (format: host:port).

otlp.grpc.insecure · boolean · default: "false"

Send traces without using a secured protocol.

otlp.grpc.headers · object · default: "{}"

Additional headers sent with traces.

Trace Propagation

Traefik supports multiple trace context propagation formats via the OTEL_PROPAGATORS environment variable:

• tracecontext (default) - W3C Trace Context
• baggage (default) - W3C Baggage
• b3 - B3 single header
• b3multi - B3 multi header
• jaeger - Jaeger propagation
• xray - AWS X-Ray
• ottrace - OT Trace

Configure Propagators

Set multiple propagators:

OTEL_PROPAGATORS=b3,jaeger

In Docker Compose:

services:
  traefik:
    image: traefik:v3.0
    environment:
      - OTEL_PROPAGATORS=b3,jaeger
    command:
      - --tracing.otlp=true

Backend Integration Examples

Jaeger

Connect to Jaeger via OpenTelemetry Collector:

YAML

tracing:
  serviceName: traefik
  sampleRate: 1.0
  otlp:
    grpc:
      endpoint: jaeger:4317
      insecure: true

Or use the Jaeger agent directly with the OpenTelemetry Collector as a sidecar.

Zipkin

Connect to Zipkin via OpenTelemetry Collector:

YAML

tracing:
  serviceName: traefik
  otlp:
    http:
      endpoint: http://otel-collector:4318/v1/traces

Configure the OpenTelemetry Collector to export to Zipkin:

# otel-collector-config.yaml
exporters:
  zipkin:
    endpoint: http://zipkin:9411/api/v2/spans

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [zipkin]

AWS X-Ray

Configure for AWS X-Ray:

YAML

tracing:
  serviceName: traefik
  otlp:
    grpc:
      endpoint: xray-collector:4317
      insecure: true

Set the X-Ray propagator:

OTEL_PROPAGATORS=xray

Complete Examples

Production Tracing Setup

YAML

tracing:
  serviceName: traefik-prod
  sampleRate: 0.1
  addInternals: false
  resourceAttributes:
    environment: production
    datacenter: us-east-1
    cluster: prod-1
  capturedRequestHeaders:
    - User-Agent
    - X-Request-Id
  capturedResponseHeaders:
    - X-Response-Time
  safeQueryParams:
    - page
    - limit
  otlp:
    grpc:
      endpoint: otel-collector:4317
      headers:
        Authorization: Bearer prod-token
      tls:
        ca: /etc/traefik/certs/ca.crt
        cert: /etc/traefik/certs/client.crt
        key: /etc/traefik/certs/client.key

Development Setup

YAML

tracing:
  serviceName: traefik-dev
  sampleRate: 1.0
  addInternals: true
  otlp:
    http:
      endpoint: http://localhost:4318/v1/traces

Docker Compose with Jaeger

version: '3'

services:
  traefik:
    image: traefik:v3.0
    command:
      - --api.insecure=true
      - --providers.docker=true
      - --entryPoints.web.address=:80
      - --tracing.serviceName=traefik
      - --tracing.sampleRate=1.0
      - --tracing.otlp.grpc.endpoint=jaeger:4317
      - --tracing.otlp.grpc.insecure=true
    environment:
      - OTEL_PROPAGATORS=jaeger
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  jaeger:
    image: jaegertracing/all-in-one:latest
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP

Access Jaeger UI at http://localhost:16686.

Span Attributes

Traefik creates spans with OpenTelemetry semantic conventions:

HTTP Span Attributes

• http.request.method - HTTP method (GET, POST, etc.)
• http.response.status_code - HTTP status code
• network.protocol.name - Protocol name (http/1.1, h2, etc.)
• server.address - Server hostname
• server.port - Server port
• url.scheme - URL scheme (http, https)
• url.path - Request path
• user_agent.original - User-Agent header

Traefik-Specific Attributes

• traefik.router.name - Router that handled the request
• traefik.service.name - Service that handled the request
• traefik.entrypoint.name - EntryPoint that received the request

Best Practices

Performance Optimization

• Use sampling (e.g., sampleRate: 0.1) in high-traffic environments
• Avoid capturing large headers or sensitive data
• Monitor collector resource usage

Security

• Redact sensitive query parameters (default behavior)
• Use TLS for collector connections in production
• Avoid capturing Authorization headers unless necessary
• Secure the collector endpoint with authentication

Troubleshooting

• Verify collector connectivity with --log.level=DEBUG
• Check trace sampling configuration
• Validate propagation headers in requests
• Monitor collector for errors