Distributed Tracing

Distributed tracing provides end-to-end visibility into request flows across your infrastructure. Traefik uses OpenTelemetry, an open standard for distributed tracing compatible with Jaeger, Zipkin, and other tracing backends.

Overview

Tracing allows you to:

Visualize request propagation across services
Identify performance bottlenecks
Analyze latency distribution
Track errors through the request lifecycle
Understand service dependencies

Traefik follows OpenTelemetry semantic conventions v1.37.0.

Quick Start

Enable basic tracing:

tracing:
  otlp: {}

[tracing]
  [tracing.otlp]

--tracing=true
--tracing.otlp=true

By default, traces are sent via HTTPS to https://localhost:4318/v1/traces.

Common Configuration

Service Name

string

default:"traefik"

Service name identifier in the tracing backend.

tracing:
  serviceName: traefik-prod
  otlp: {}

[tracing]
  serviceName = "traefik-prod"
  [tracing.otlp]

--tracing.serviceName=traefik-prod
--tracing.otlp=true

Sample Rate

float

default:"1.0"

Proportion of requests to trace (0.0 to 1.0).

Set to 0.1 to trace 10% of requests:

tracing:
  sampleRate: 0.1

[tracing]
  sampleRate = 0.1

--tracing.sampleRate=0.1

The OpenTelemetry SDK also supports the OTEL_TRACES_SAMPLER environment variable for custom sampling strategies.

Add Internals

boolean

default:"false"

Enable tracing for internal resources like ping@internal.

tracing:
  addInternals: true

[tracing]
  addInternals = true

--tracing.addinternals=true

Resource Attributes

object

default:"{}"

Additional resource attributes sent to the collector.

tracing:
  resourceAttributes:
    environment: production
    datacenter: us-east-1
    version: v2.10

[tracing.resourceAttributes]
  environment = "production"
  datacenter = "us-east-1"
  version = "v2.10"

--tracing.resourceAttributes.environment=production
--tracing.resourceAttributes.datacenter=us-east-1
--tracing.resourceAttributes.version=v2.10

Captured Headers

Capture request and response headers as span attributes:

array

default:"[]"

List of request headers to add as span attributes.

array

default:"[]"

List of response headers to add as span attributes.

tracing:
  capturedRequestHeaders:
    - User-Agent
    - X-Request-Id
    - Authorization
  capturedResponseHeaders:
    - X-Response-Time
    - Content-Type

[tracing]
  capturedRequestHeaders = ["User-Agent", "X-Request-Id", "Authorization"]
  capturedResponseHeaders = ["X-Response-Time", "Content-Type"]

--tracing.capturedRequestHeaders=User-Agent,X-Request-Id,Authorization
--tracing.capturedResponseHeaders=X-Response-Time,Content-Type

Safe Query Parameters

array

default:"[]"

Query parameters to NOT redact (all others are redacted by default).

By default, all query parameters are redacted for security. Specify safe parameters:

tracing:
  safeQueryParams:
    - page
    - limit
    - sort

[tracing]
  safeQueryParams = ["page", "limit", "sort"]

--tracing.safeQueryParams=page,limit,sort

OpenTelemetry Configuration

HTTP Protocol

Send traces via HTTP to an OpenTelemetry Collector:

tracing:
  otlp:
    http:
      endpoint: https://otel-collector:4318/v1/traces
      headers:
        X-API-Key: secret-key
      tls:
        ca: /path/to/ca.crt
        cert: /path/to/client.cert
        key: /path/to/client.key
        insecureSkipVerify: false

[tracing.otlp.http]
  endpoint = "https://otel-collector:4318/v1/traces"
  
  [tracing.otlp.http.headers]
    X-API-Key = "secret-key"
  
  [tracing.otlp.http.tls]
    ca = "/path/to/ca.crt"
    cert = "/path/to/client.cert"
    key = "/path/to/client.key"
    insecureSkipVerify = false

--tracing.otlp.http.endpoint=https://otel-collector:4318/v1/traces
--tracing.otlp.http.headers.X-API-Key=secret-key
--tracing.otlp.http.tls.ca=/path/to/ca.crt

HTTP Configuration Options

string

default:"https://localhost:4318/v1/traces"

URL of the OpenTelemetry Collector.

To disable TLS, use http:// instead of https:// in the endpoint URL.

object

default:"{}"

Additional headers sent with traces.

string

Path to certificate authority file.

string

Path to client certificate file.

string

Path to client private key file.

boolean

default:"false"

Skip TLS certificate verification.

gRPC Protocol

Send traces via gRPC to an OpenTelemetry Collector:

tracing:
  otlp:
    grpc:
      endpoint: otel-collector:4317
      insecure: false
      headers:
        Authorization: Bearer token123
      tls:
        ca: /path/to/ca.crt
        cert: /path/to/client.cert
        key: /path/to/client.key

[tracing.otlp.grpc]
  endpoint = "otel-collector:4317"
  insecure = false
  
  [tracing.otlp.grpc.headers]
    Authorization = "Bearer token123"
  
  [tracing.otlp.grpc.tls]
    ca = "/path/to/ca.crt"
    cert = "/path/to/client.cert"
    key = "/path/to/client.key"

--tracing.otlp.grpc.endpoint=otel-collector:4317
--tracing.otlp.grpc.insecure=false
--tracing.otlp.grpc.headers.Authorization="Bearer token123"
--tracing.otlp.grpc.tls.ca=/path/to/ca.crt

gRPC Configuration Options

string

default:"localhost:4317"

Address of the OpenTelemetry Collector (format: host:port).

boolean

default:"false"

Send traces without using a secured protocol.

object

default:"{}"

Additional headers sent with traces.

Trace Propagation

Traefik supports multiple trace context propagation formats via the OTEL_PROPAGATORS environment variable:

tracecontext (default) - W3C Trace Context
baggage (default) - W3C Baggage
b3 - B3 single header
b3multi - B3 multi header
jaeger - Jaeger propagation
xray - AWS X-Ray
ottrace - OT Trace

Configure Propagators

Set multiple propagators:

OTEL_PROPAGATORS=b3,jaeger

In Docker Compose:

services:
  traefik:
    image: traefik:v3.0
    environment:
      - OTEL_PROPAGATORS=b3,jaeger
    command:
      - --tracing.otlp=true

Backend Integration Examples

Jaeger

Connect to Jaeger via OpenTelemetry Collector:

tracing:
  serviceName: traefik
  sampleRate: 1.0
  otlp:
    grpc:
      endpoint: jaeger:4317
      insecure: true

[tracing]
  serviceName = "traefik"
  sampleRate = 1.0
  
  [tracing.otlp.grpc]
    endpoint = "jaeger:4317"
    insecure = true

Or use the Jaeger agent directly with the OpenTelemetry Collector as a sidecar.

Zipkin

Connect to Zipkin via OpenTelemetry Collector:

tracing:
  serviceName: traefik
  otlp:
    http:
      endpoint: http://otel-collector:4318/v1/traces

[tracing]
  serviceName = "traefik"
  
  [tracing.otlp.http]
    endpoint = "http://otel-collector:4318/v1/traces"

Configure the OpenTelemetry Collector to export to Zipkin:

# otel-collector-config.yaml
exporters:
  zipkin:
    endpoint: http://zipkin:9411/api/v2/spans

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [zipkin]

AWS X-Ray

Configure for AWS X-Ray:

tracing:
  serviceName: traefik
  otlp:
    grpc:
      endpoint: xray-collector:4317
      insecure: true

[tracing]
  serviceName = "traefik"
  
  [tracing.otlp.grpc]
    endpoint = "xray-collector:4317"
    insecure = true

Set the X-Ray propagator:

OTEL_PROPAGATORS=xray

Complete Examples

Production Tracing Setup

tracing:
  serviceName: traefik-prod
  sampleRate: 0.1
  addInternals: false
  resourceAttributes:
    environment: production
    datacenter: us-east-1
    cluster: prod-1
  capturedRequestHeaders:
    - User-Agent
    - X-Request-Id
  capturedResponseHeaders:
    - X-Response-Time
  safeQueryParams:
    - page
    - limit
  otlp:
    grpc:
      endpoint: otel-collector:4317
      headers:
        Authorization: Bearer prod-token
      tls:
        ca: /etc/traefik/certs/ca.crt
        cert: /etc/traefik/certs/client.crt
        key: /etc/traefik/certs/client.key

[tracing]
  serviceName = "traefik-prod"
  sampleRate = 0.1
  addInternals = false
  
  [tracing.resourceAttributes]
    environment = "production"
    datacenter = "us-east-1"
    cluster = "prod-1"
  
  capturedRequestHeaders = ["User-Agent", "X-Request-Id"]
  capturedResponseHeaders = ["X-Response-Time"]
  safeQueryParams = ["page", "limit"]
  
  [tracing.otlp.grpc]
    endpoint = "otel-collector:4317"
    
    [tracing.otlp.grpc.headers]
      Authorization = "Bearer prod-token"
    
    [tracing.otlp.grpc.tls]
      ca = "/etc/traefik/certs/ca.crt"
      cert = "/etc/traefik/certs/client.crt"
      key = "/etc/traefik/certs/client.key"

Development Setup

tracing:
  serviceName: traefik-dev
  sampleRate: 1.0
  addInternals: true
  otlp:
    http:
      endpoint: http://localhost:4318/v1/traces

[tracing]
  serviceName = "traefik-dev"
  sampleRate = 1.0
  addInternals = true
  
  [tracing.otlp.http]
    endpoint = "http://localhost:4318/v1/traces"

Docker Compose with Jaeger

version: '3'

services:
  traefik:
    image: traefik:v3.0
    command:
      - --api.insecure=true
      - --providers.docker=true
      - --entryPoints.web.address=:80
      - --tracing.serviceName=traefik
      - --tracing.sampleRate=1.0
      - --tracing.otlp.grpc.endpoint=jaeger:4317
      - --tracing.otlp.grpc.insecure=true
    environment:
      - OTEL_PROPAGATORS=jaeger
    ports:
      - "80:80"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  jaeger:
    image: jaegertracing/all-in-one:latest
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"  # Jaeger UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP

Access Jaeger UI at http://localhost:16686.

Span Attributes

Traefik creates spans with OpenTelemetry semantic conventions:

HTTP Span Attributes

http.request.method - HTTP method (GET, POST, etc.)
http.response.status_code - HTTP status code
network.protocol.name - Protocol name (http/1.1, h2, etc.)
server.address - Server hostname
server.port - Server port
url.scheme - URL scheme (http, https)
url.path - Request path
user_agent.original - User-Agent header

Traefik-Specific Attributes

traefik.router.name - Router that handled the request
traefik.service.name - Service that handled the request
traefik.entrypoint.name - EntryPoint that received the request

Best Practices

Performance Optimization

Use sampling (e.g., sampleRate: 0.1) in high-traffic environments
Avoid capturing large headers or sensitive data
Monitor collector resource usage

Security

Redact sensitive query parameters (default behavior)
Use TLS for collector connections in production
Avoid capturing Authorization headers unless necessary
Secure the collector endpoint with authentication

Troubleshooting

Verify collector connectivity with --log.level=DEBUG
Check trace sampling configuration
Validate propagation headers in requests
Monitor collector for errors

Getting Started

Core Configuration

Providers

Routing

Middlewares

HTTPS & TLS

Observability

Operations

Distributed Tracing

Distributed Tracing

Overview

Quick Start

Common Configuration

Service Name

Sample Rate

Add Internals

Resource Attributes

Captured Headers

Safe Query Parameters

OpenTelemetry Configuration

HTTP Protocol

HTTP Configuration Options

gRPC Protocol

gRPC Configuration Options

Trace Propagation

Configure Propagators

Backend Integration Examples

Jaeger

Zipkin

AWS X-Ray

Complete Examples

Production Tracing Setup

Development Setup

Docker Compose with Jaeger

Span Attributes

HTTP Span Attributes

Traefik-Specific Attributes

Best Practices

​Distributed Tracing

​Overview

​Quick Start

​Common Configuration

​Service Name

​Sample Rate

​Add Internals

​Resource Attributes

​Captured Headers

​Safe Query Parameters

​OpenTelemetry Configuration

​HTTP Protocol

​HTTP Configuration Options

​gRPC Protocol

​gRPC Configuration Options

​Trace Propagation

​Configure Propagators

​Backend Integration Examples

​Jaeger

​Zipkin

​AWS X-Ray

​Complete Examples

​Production Tracing Setup

​Development Setup

​Docker Compose with Jaeger

​Span Attributes

​HTTP Span Attributes

​Traefik-Specific Attributes

​Best Practices

Distributed Tracing

Overview

Quick Start

Common Configuration

Service Name

Sample Rate

Add Internals

Resource Attributes

Captured Headers

Safe Query Parameters

OpenTelemetry Configuration

HTTP Protocol

HTTP Configuration Options

gRPC Protocol

gRPC Configuration Options

Trace Propagation

Configure Propagators

Backend Integration Examples

Jaeger

Zipkin

AWS X-Ray

Complete Examples

Production Tracing Setup

Development Setup

Docker Compose with Jaeger

Span Attributes

HTTP Span Attributes

Traefik-Specific Attributes

Best Practices