lans.cloud Blog
tutorial11 min read

Building a Modern Reverse Proxy with Traefik - Part 5: Production Best Practices

#traefik#docker#devops#production#monitoring#security#troubleshooting

Welcome to the final part of our Traefik series! We've covered the fundamentals, setup, deployments, and advanced features. Now we'll make it bulletproof for production.

What We'll Cover

  • Troubleshooting Guide - Solve common and rare issues
  • Security Hardening - Lock down your infrastructure
  • Monitoring & Alerting - Know what's happening
  • Backup Strategies - Protect your configuration
  • Performance Optimization - Make it fast
  • High Availability - Eliminate single points of failure
  • Maintenance Procedures - Keep it running smoothly

Comprehensive Troubleshooting Guide

Issue 1: 502 Bad Gateway

Symptoms: Traefik returns 502 error, app seems unreachable

Common Causes & Solutions:

Cause 1: App not listening on correct interface

# Check what app is actually listening on
docker exec myapp-container ss -tlnp

# Look for the port - should show 0.0.0.0:PORT or :::PORT
# If it shows 127.0.0.1:PORT, that's the problem

Solution: For Node.js/Next.js apps:

environment:
  - HOSTNAME=0.0.0.0  # Add this!

Cause 2: Wrong port in labels

# Check the label
- "traefik.http.services.myapp.loadbalancer.server.port=3000"

# Verify app actually runs on this port
docker exec myapp-container netstat -tlnp | grep 3000

Solution: Update port to match what app actually uses

Cause 3: Container not on traefik-network

# Check which networks container is on
docker inspect myapp-container | grep -A 20 Networks

# Should show traefik-network

Solution:

networks:
  - traefik-network  # Make sure this is added

Cause 4: App not started yet

# Check if container is actually running
docker ps | grep myapp

# Check app logs
docker logs myapp-container

# Look for "Server started" or similar message

Solution: Add health check or wait longer

Cause 5: Firewall blocking internal communication

# Test from Traefik container
docker exec traefik wget -O- http://myapp-container:3000

# Should return app response

Solution: Check Docker firewall rules

Issue 2: SSL Certificate Errors

Symptom: "Your connection is not private" or "SEC_ERROR_UNKNOWN_ISSUER"

Debug Steps:

# Check Traefik logs for ACME errors
docker logs traefik | grep -i acme
docker logs traefik | grep -i letsencrypt

# Check certificate file
cat ~/traefik/letsencrypt/acme.json
# Should contain certificates

# Check DNS is pointing to your server
dig yourdomain.com

Cause 1: DNS not configured

Let's Encrypt needs to verify domain ownership via HTTP challenge. If DNS doesn't point to your server, verification fails.

Solution:

# Verify DNS points to your IP
dig yourdomain.com

# Should return your server IP
# If not, update DNS and wait for propagation

Cause 2: Port 80 blocked

Let's Encrypt HTTP challenge requires port 80.

Solution:

# Check firewall
ufw status | grep 80

# Should show: 80/tcp ALLOW

Cause 3: Rate limit hit

Let's Encrypt allows 5 certificates per week per domain.

Check if you're rate limited:

  • Visit https://crt.sh
  • Search for your domain
  • Count recent certificates

Solution: Wait a week, or use a different subdomain

Cause 4: acme.json permission error

# Check permissions
ls -la ~/traefik/letsencrypt/acme.json

# Should be: -rw------- (600)

Solution:

chmod 600 ~/traefik/letsencrypt/acme.json
docker compose restart traefik

Cause 5: Wrong email in config

Let's Encrypt requires a valid email.

Solution: Update traefik.yml:

certificatesResolvers:
  letsencrypt:
    acme:
      email: valid-email@example.com  # Change this

Issue 3: Service Not Discovered

Symptom: New container starts but doesn't appear in Traefik dashboard

Debug checklist:

# 1. Verify traefik.enable label
docker inspect myapp-container | grep traefik.enable
# Should show: "traefik.enable": "true"

# 2. Check if container is running
docker ps | grep myapp

# 3. Verify on correct network
docker inspect myapp-container | grep -A 10 Networks
# Should include traefik-network

# 4. Check Traefik logs
docker logs traefik --tail 50

# 5. Verify Docker socket mounted
docker inspect traefik | grep docker.sock
# Should show mount

Common fixes:

# Missing traefik.enable
labels:
  - "traefik.enable=true"  # Add this!

# Wrong network specified
labels:
  - "traefik.docker.network=traefik-network"  # If on multiple networks

# Network not external
networks:
  traefik-network:
    external: true  # Not "external: false"

Issue 4: Redirect Loop

Symptom: Browser shows "Too many redirects" error

Cause: Usually HTTP→HTTPS redirect misconfiguration

Debug:

# Check if both HTTP and HTTPS routers exist
docker inspect myapp-container | grep -i router

# Test with curl
curl -I http://yourdomain.com
curl -I https://yourdomain.com

Solution:

Only use websecure entrypoint:

labels:
  - "traefik.http.routers.myapp.entrypoints=websecure"  # Only this!
  # Don't add web entrypoint

Traefik's global HTTP→HTTPS redirect (in traefik.yml) handles HTTP.

Issue 5: Slow Performance

Symptom: Website loads slowly through Traefik

Debug:

# Check Traefik resource usage
docker stats traefik

# High CPU or memory indicates a problem

# Test direct connection (bypassing Traefik)
docker exec -it myapp-container curl http://localhost:3000

# Compare to Traefik route
curl https://yourdomain.com

Solutions:

Solution 1: Enable compression

labels:
  - "traefik.http.middlewares.compress.compress=true"
  - "traefik.http.routers.myapp.middlewares=compress"

Solution 2: Reduce middleware Too much middleware = slow requests. Only use what you need.

Solution 3: Increase resources

services:
  traefik:
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 1G

Solution 4: Check app itself

# Profile app response time
docker exec myapp-container curl -w "@-" -o /dev/null -s http://localhost:3000 <<< '
time_total:  %{time_total}
'

If app is slow, Traefik can't fix that.

Security Hardening Checklist

1. Secure the Docker Socket

Problem: Traefik needs access to /var/run/docker.sock, which is very powerful.

Risk: If Traefik is compromised, attacker could control all containers.

Solution: Use Docker Socket Proxy

services:
  # Socket proxy (only allows read operations)
  docker-socket-proxy:
    image: tecnativa/docker-socket-proxy
    container_name: docker-socket-proxy
    restart: unless-stopped
    networks:
      - traefik-socket
    environment:
      CONTAINERS: 1
      NETWORKS: 1
      SERVICES: 1
      TASKS: 1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  traefik:
    # ... other config ...
    environment:
      - DOCKER_HOST=tcp://docker-socket-proxy:2375
    networks:
      - traefik-network
      - traefik-socket
    # Remove docker.sock mount

networks:
  traefik-socket:
    driver: bridge

Now Traefik can't execute privileged Docker commands.

2. Limit Dashboard Access

Option 1: IP Whitelist

labels:
  # Only allow from your office/home IP
  - "traefik.http.middlewares.dashboard-whitelist.ipwhitelist.sourcerange=203.0.113.45/32"
  - "traefik.http.routers.dashboard.middlewares=dashboard-auth,dashboard-whitelist"

Option 2: VPN Only

Don't expose dashboard to internet at all. Access via VPN (WireGuard, Tailscale, etc.)

Option 3: Strong Authentication

# Use a very strong password
htpasswd -nb admin $(openssl rand -base64 32)

3. Enable Automatic Security Headers

Add to traefik.yml:

# Global security headers
http:
  middlewares:
    security-headers:
      headers:
        frameDeny: true
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 315360000
        customFrameOptionsValue: "SAMEORIGIN"

Apply to all services:

labels:
  - "traefik.http.routers.myapp.middlewares=security-headers"

4. Regular Updates

# Create update script
cat > ~/update-traefik.sh << 'EOF'
#!/bin/bash
cd ~/traefik
docker compose pull
docker compose up -d
docker image prune -f
echo "Traefik updated successfully"
EOF

chmod +x ~/update-traefik.sh

# Run weekly via cron
crontab -e
# Add: 0 3 * * 0 /home/yourname/update-traefik.sh

5. Fail2ban for Brute Force Protection

# Install fail2ban
apt install -y fail2ban

# Create Traefik jail
cat > /etc/fail2ban/jail.d/traefik.conf << 'EOF'
[traefik-auth]
enabled = true
port = http,https
filter = traefik-auth
logpath = /home/yourname/traefik/logs/access.log
maxretry = 5
bantime = 3600
EOF

# Create filter
cat > /etc/fail2ban/filter.d/traefik-auth.conf << 'EOF'
[Definition]
failregex = ^.* ".*" 401 .*$
ignoreregex =
EOF

# Restart fail2ban
systemctl restart fail2ban

6. Secrets Management

Don't put secrets in docker-compose.yml!

Use environment variables:

# Create .env file
cat > ~/traefik/.env << 'EOF'
TRAEFIK_DASHBOARD_PASSWORD=very-secure-password
DB_PASSWORD=another-secure-password
API_KEY=yet-another-secret
EOF

# Secure it
chmod 600 ~/traefik/.env

# Reference in docker-compose.yml
environment:
  - DB_PASSWORD=${DB_PASSWORD}

Or use Docker secrets:

secrets:
  db_password:
    file: ./secrets/db_password.txt

services:
  myapp:
    secrets:
      - db_password

Monitoring and Alerting

1. Enable Metrics

Add to traefik.yml:

metrics:
  prometheus:
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0

Traefik now exposes Prometheus metrics at :8080/metrics

2. Set Up Prometheus

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    networks:
      - traefik-network
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: unless-stopped
    volumes:
      - grafana-data:/var/lib/grafana
    networks:
      - traefik-network
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.rule=Host(`monitoring.example.com`)"
      - "traefik.http.routers.grafana.entrypoints=websecure"
      - "traefik.http.routers.grafana.tls.certresolver=letsencrypt"
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"

volumes:
  prometheus-data:
  grafana-data:

Create prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'traefik'
    static_configs:
      - targets: ['traefik:8080']

3. Important Metrics to Monitor

  • traefik_entrypoint_requests_total - Total requests
  • traefik_entrypoint_request_duration_seconds - Response time
  • traefik_service_requests_total{service="myapp"} - Per-service requests
  • traefik_service_requests_bytes_total - Bandwidth usage
  • traefik_tls_certs_not_after - Certificate expiration time

4. Set Up Alerts

Create alertmanager.yml:

route:
  receiver: 'email'

receivers:
  - name: 'email'
    email_configs:
      - to: 'your-email@example.com'
        from: 'alerts@example.com'
        smarthost: smtp.gmail.com:587
        auth_username: 'alerts@example.com'
        auth_password: 'app-password'

Create alert rules in prometheus.yml:

rule_files:
  - 'alerts.yml'

alerts.yml:

groups:
  - name: traefik
    rules:
      - alert: HighErrorRate
        expr: rate(traefik_service_requests_total{code=~"5.."}[5m]) > 0.1
        for: 5m
        annotations:
          summary: "High error rate on {{ $labels.service }}"

      - alert: CertExpiringSoon
        expr: (traefik_tls_certs_not_after - time()) / 86400 < 7
        annotations:
          summary: "Certificate expiring in less than 7 days"

      - alert: ServiceDown
        expr: up{job="traefik"} == 0
        for: 1m
        annotations:
          summary: "Traefik is down!"

5. Log Aggregation

Use Loki for centralized logging:

services:
  loki:
    image: grafana/loki
    container_name: loki
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki-data:/loki
    networks:
      - traefik-network

volumes:
  loki-data:

Configure Grafana to use Loki as data source.

Backup Strategies

What to Back Up

Critical:

  • /home/yourname/traefik/traefik.yml - Configuration
  • /home/yourname/traefik/docker-compose.yml - Container definition
  • /home/yourname/traefik/letsencrypt/acme.json - SSL certificates

Important:

  • /home/yourname/traefik/logs/ - Logs (for debugging)
  • /var/lib/docker/volumes/ - Application data

Automated Backup Script

cat > ~/backup-traefik.sh << 'EOF'
#!/bin/bash

BACKUP_DIR="/home/yourname/backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/traefik_backup_$DATE.tar.gz"

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup Traefik configuration
tar -czf $BACKUP_FILE \
  /home/yourname/traefik/traefik.yml \
  /home/yourname/traefik/docker-compose.yml \
  /home/yourname/traefik/letsencrypt/acme.json \
  /home/yourname/traefik/.env

# Optional: Upload to S3
# aws s3 cp $BACKUP_FILE s3://my-bucket/traefik-backups/

# Keep only last 30 days
find $BACKUP_DIR -name "traefik_backup_*.tar.gz" -mtime +30 -delete

echo "Backup completed: $BACKUP_FILE"
EOF

chmod +x ~/backup-traefik.sh

# Run daily via cron
crontab -e
# Add: 0 2 * * * /home/yourname/backup-traefik.sh

Restore Procedure

# Stop Traefik
cd ~/traefik
docker compose down

# Extract backup
tar -xzf ~/backups/traefik_backup_YYYYMMDD_HHMMSS.tar.gz -C /

# Restore permissions
chmod 600 ~/traefik/letsencrypt/acme.json

# Restart
docker compose up -d

Performance Optimization

1. Enable HTTP/2

Already enabled by default in Traefik 2.x with TLS.

Verify:

curl -I --http2 https://yourdomain.com | grep HTTP
# Should show: HTTP/2 200

2. Enable Caching (For Static Content)

Use Traefik's cache middleware:

labels:
  - "traefik.http.middlewares.cache.plugin.cache.maxage=3600"
  - "traefik.http.routers.myapp.middlewares=cache"

Or use a dedicated cache (Varnish, Redis):

services:
  varnish:
    image: varnish
    # ...configure...

3. Connection Pooling

Traefik reuses connections to backends. Configure limits:

# In traefik.yml
serversTransport:
  maxIdleConnsPerHost: 200

4. Load Balancing

Scale services horizontally:

services:
  api:
    deploy:
      replicas: 3  # Run 3 instances

Traefik automatically load balances.

Load balancing strategies:

labels:
  # Round-robin (default)
  - "traefik.http.services.api.loadbalancer.server.port=8000"

  # Weighted round-robin
  - "traefik.http.services.api.loadbalancer.servers[0].url=http://api1:8000"
  - "traefik.http.services.api.loadbalancer.servers[0].weight=3"
  - "traefik.http.services.api.loadbalancer.servers[1].url=http://api2:8000"
  - "traefik.http.services.api.loadbalancer.servers[1].weight=1"

  # Sticky sessions
  - "traefik.http.services.api.loadbalancer.sticky.cookie=true"

5. Resource Limits

Prevent resource exhaustion:

services:
  traefik:
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M

High Availability Setup

For critical production environments, eliminate single points of failure.

Architecture

         Load Balancer (Hetzner/Cloudflare)
                    │
        ┌───────────┼───────────┐
        ↓           ↓           ↓
    Traefik 1   Traefik 2   Traefik 3
        │           │           │
        └───────────┼───────────┘
                    │
              Your Apps (scaled)

Implementation

1. Use Docker Swarm or Kubernetes

Docker Swarm (simpler):

# Initialize swarm
docker swarm init

# Deploy stack
docker stack deploy -c docker-compose.yml traefik

2. Use external load balancer

  • Hetzner Load Balancer
  • AWS ELB
  • Cloudflare Load Balancing
  • HAProxy

3. Share SSL certificates

Use Consul or Redis for certificate storage:

# In traefik.yml
certificatesResolvers:
  letsencrypt:
    acme:
      storage: consul://consul:8500/traefik/acme

Health Checks

healthcheck:
  test: ["CMD", "traefik", "healthcheck", "--ping"]
  interval: 10s
  timeout: 3s
  retries: 3

Migration Strategies

From Nginx to Traefik

1. Run both simultaneously

Keep Nginx on port 80/443, run Traefik on different ports initially.

2. Migrate one service at a time

Update DNS for one subdomain to point to new setup, test, repeat.

3. Use Traefik's multiple providers

Traefik can watch both Docker AND read Nginx-style configs during transition.

Zero-Downtime Updates

# Update strategy:

# 1. Pull new image
docker pull traefik:latest

# 2. Start new container with different name
docker run -d --name traefik-new \
  # ... same config as traefik ...

# 3. Verify it works
curl http://localhost:8080/ping

# 4. Swap containers
docker rename traefik traefik-old
docker rename traefik-new traefik
docker stop traefik-old
docker rm traefik-old

Maintenance Checklist

Daily

  • [ ] Check Grafana dashboards for anomalies
  • [ ] Review error rate metrics
  • [ ] Check disk space usage

Weekly

  • [ ] Review Traefik logs for errors
  • [ ] Check certificate expiration dates
  • [ ] Update Traefik and apps
  • [ ] Review security alerts

Monthly

  • [ ] Test backup restoration
  • [ ] Review and rotate secrets
  • [ ] Check for Traefik CVEs
  • [ ] Audit access logs for suspicious activity
  • [ ] Review and optimize middleware

Quarterly

  • [ ] Load test infrastructure
  • [ ] Review and update disaster recovery plan
  • [ ] Security audit
  • [ ] Capacity planning

Production Deployment Checklist

Before going live:

  • [ ] DNS configured with TTL 300
  • [ ] Firewall rules configured (22, 80, 443 only)
  • [ ] SSL certificates obtained and valid
  • [ ] All services have health checks
  • [ ] Rate limiting configured
  • [ ] Security headers enabled
  • [ ] Authentication on sensitive endpoints
  • [ ] Monitoring and alerting set up
  • [ ] Backups automated and tested
  • [ ] Documentation updated
  • [ ] Disaster recovery plan written
  • [ ] Load testing completed
  • [ ] Logs configured and aggregated
  • [ ] Secrets stored securely (not in git)
  • [ ] Team trained on procedures

Conclusion

You've completed the entire Traefik series! You now know:

  • Part 1: What Traefik is and why to use it
  • Part 2: How to set up Traefik from scratch
  • Part 3: How to deploy applications with Traefik
  • Part 4: Advanced features and middleware
  • Part 5: Production best practices and troubleshooting

You have everything needed to run a production-grade reverse proxy infrastructure.

Further Resources

Official Documentation:

Community:

  • Traefik Community Forum
  • Reddit: r/traefik
  • Discord: Traefik Official

Related Tools:

Final Thoughts

Traefik has transformed how I deploy and manage applications. What used to take hours of Nginx configuration now takes minutes with Docker labels. SSL certificates that required manual renewal now happen automatically. Service discovery that didn't exist is now built-in.

I hope this series has given you both the knowledge and confidence to build your own infrastructure. Remember:

  • Start simple and add complexity as needed
  • Always test changes in staging first
  • Document your setup (future you will thank present you)
  • Monitor everything
  • Keep security in mind from day one

Happy deploying!

Questions? Review the troubleshooting section, check the official docs, or search the community forums. Chances are someone has faced the same issue!