Skip to content

Troubleshooting Guide

This guide covers common issues encountered in the Docker infrastructure and their solutions, with real examples from our setup.

Quick Checklist for Common Issues

Before diving into detailed troubleshooting, run through this quick checklist:

  1. Is the service running? 

    docker ps | grep <service-name>

  2. Is it listening on the expected port? 

    sudo ss -tlnup | grep <port>
# Or inside container:
docker exec <container> ss -tlnup

  3. Is Proxmox forwarding the port?

  4. Check on Proxmox host: Network > Firewall/NAT rules

  5. Common missing forwards: 53/UDP (DNS), 80/TCP (HTTP), 443/TCP (HTTPS)

  6. Can you access it from within the VM? 

    # For web services:
curl -I http://localhost:<port>
# For DNS:
dig @localhost google.com

  7. Can you access it externally? 

    # From your home/external network:
curl -I https://<service>.rbnk.uk
dig @<public-ip> google.com

  8. Can you access via Tailscale IP? 

    # From a device in your Tailscale network:
curl -I http://100.98.20.43:<port>
dig @100.98.20.43 google.com

Table of Contents

Service Won't Start

Symptom

Container exits immediately or never reaches "running" state.

Common Causes and Solutions

1. Port Already in Use

# Check if port is in use
sudo netstat -tlnp | grep :3000

# Or using ss
sudo ss -tlnp | grep :3000

# Find and stop conflicting container
docker ps -a | grep :3000
docker stop <container_id>

2. Missing Environment Variables

# Check if all required vars are set
docker compose -f myapp/docker-compose.yml config

# Common error in logs:
# "Error: DATABASE_URL environment variable not set"

# Solution: Ensure .env file exists and is readable
ls -la myapp/.env
# Should show: -rw-r----- 1 user user 423 Nov 11 10:30 .env

3. Image Pull Failures

# Manual pull to see detailed error
docker pull myimage:tag

# Common issues:
# - Rate limiting: "toomanyrequests: You have reached your pull rate limit"
# - Auth required: "pull access denied"

# Solution for private registries:
docker login registry.example.com

4. Volume Mount Issues

# Error: "Cannot start service: error while creating mount source path"

# Check if source path exists
ls -la /srv/dockerdata/myapp/data

# Create if missing
mkdir -p /srv/dockerdata/myapp/data

# Fix permissions
sudo chown -R $USER:docker /srv/dockerdata/myapp/data

Real Example: Supabase Auth Service

# Auth service wasn't starting
docker compose -f supabase/docker-compose.yml logs supabase-auth

# Error: "GOTRUE_SITE_URL is required"

# Solution: Added to .env
echo "SITE_URL=https://supabase.rbnk.uk" >> supabase/.env

# Restart service
docker compose -f supabase/docker-compose.yml up -d supabase-auth

Bad Gateway Errors

Symptom

502 Bad Gateway or 504 Gateway Timeout from Traefik.

Common Causes and Solutions

1. Service Not Running

# Check if backend service is running
docker ps | grep myapp

# If not running, check why
docker compose -f myapp/docker-compose.yml ps
docker compose -f myapp/docker-compose.yml logs --tail=50

2. Wrong Network Configuration

# Verify service is on traefik_proxy network
docker inspect myapp-web-1 | jq '.[0].NetworkSettings.Networks | keys'

# Should include "traefik_proxy"
# If missing, add to docker-compose.yml:
networks:
  - traefik_proxy

3. Incorrect Port in Traefik Labels

# Check what port the service is actually listening on
docker compose -f myapp/docker-compose.yml exec myapp-web netstat -tlnp

# Update Traefik label to match
# Wrong: - "traefik.http.services.myapp.loadbalancer.server.port=80"
# Right: - "traefik.http.services.myapp.loadbalancer.server.port=3000"

4. Service Still Starting

# Check if service has health check
docker inspect myapp-web-1 | jq '.[0].State.Health'

# Monitor startup
docker logs -f myapp-web-1

# Add startup delay in docker-compose.yml:
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
  start_period: 60s

Real Example: Supabase Studio Path Routing

# Issue: 502 on https://supabase.rbnk.uk/rest/v1/

# Debugging:
curl http://localhost:8080/api/http/routers | jq '.[] | select(.rule | contains("rest"))'

# Found: Wrong service name in label
# Fixed: Changed loadbalancer service name to match router name
- "traefik.http.routers.supabase-rest.service=supabase-rest"
- "traefik.http.services.supabase-rest.loadbalancer.server.port=3000"

SSL Certificate Issues

Symptom

Certificate errors, SSL handshake failures, or browser warnings.

Common Causes and Solutions

1. Certificate Not Issued

# Check Traefik logs for cert issues
docker logs traefik 2>&1 | grep -i "certificate\|acme"

# Common error: "Unable to obtain ACME certificate"
# Cause: Cloudflare API token incorrect

# Verify Cloudflare credentials
docker exec traefik env | grep CF_

# Test Cloudflare API access
curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "Authorization: Bearer $CF_DNS_API_TOKEN"

2. DNS Challenge Failing

# Check DNS propagation
dig TXT _acme-challenge.myapp.rbnk.uk

# Ensure Cloudflare DNS API token has correct permissions:
# - Zone:Read
# - DNS:Edit

3. Wrong Certificate Resolver

# Check if using correct resolver in labels
grep certresolver myapp/docker-compose.yml

# Should be:
- "traefik.http.routers.myapp.tls.certresolver=cloudflare"

# Not:
- "traefik.http.routers.myapp.tls.certresolver=letsencrypt"

Real Example: Wildcard Certificate Setup

# Needed *.rbnk.uk certificate
# Added to traefik.yml:
certificatesResolvers:
  cloudflare:
    acme:
      dnsChallenge:
        provider: cloudflare
        resolvers:
          - "1.1.1.1:53"
          - "8.8.8.8:53"

# Verified working:
docker exec traefik cat /letsencrypt/acme.json | jq '.cloudflare.Certificates[0].domain'

Network Connectivity Problems

Symptom

Services can't communicate with each other or external services.

Common Causes and Solutions

0. Check Proxmox Port Forwarding (First Step!)

# If service works locally but not externally, it's likely a Proxmox NAT issue
# From within VM:
sudo ss -tlnup | grep <port>  # Verify service is listening

# Test locally:
curl http://localhost:<port>  # Should work

# Test externally (from home):
curl http://<public-ip>:<port>  # Fails? Check Proxmox NAT rules

# Common missing port forwards:
# - 53/UDP for DNS services (AdGuard, Pi-hole)
# - 80/TCP for HTTP (usually forwarded to Traefik)
# - 443/TCP for HTTPS (usually forwarded to Traefik)
# - Custom service ports (3000, 8080, etc.)

1. Services on Different Networks

# List all networks
docker network ls

# Check which networks a container is connected to
docker inspect myapp-web-1 | jq '.[0].NetworkSettings.Networks | keys'

# Connect to additional network
docker network connect supabase_default myapp-web-1

2. Using Localhost Instead of Service Names

# Wrong in myapp:
DATABASE_URL=postgresql://user:pass@localhost:5432/db

# Correct:
DATABASE_URL=postgresql://user:pass@myapp-db:5432/db

# Or for Supabase:
DATABASE_URL=postgresql://user:pass@supa-db:5432/db

3. Network Isolation

# Check if network is internal only
docker network inspect internal | jq '.[0].Internal'

# If true, services can't reach internet
# Solution: Use different network or remove internal flag

Real Example: App Connecting to Supabase

# App couldn't connect to Supabase PostgreSQL

# Added to app's docker-compose.yml:
networks:
  traefik_proxy:
    external: true
  supabase_default:
    external: true

services:
  myapp:
    networks:
      - traefik_proxy
      - supabase_default
    environment:
      - DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD}@supa-db:5432/postgres

Tailscale Access Issues

Overview

Tailscale is a VPN service that creates a secure network between your devices. When running services accessible via Tailscale, you may encounter unique connectivity challenges due to the overlay network architecture.

Key Insight

Tailscale IPs (like 100.98.20.43) are only accessible from devices that are part of your Tailscale network. This is why setting your router's DNS to a Tailscale IP will fail - the router itself is not part of the Tailscale network and cannot reach the Tailscale IP.

Common Tailscale Connectivity Problems

1. Service Accessible via Public IP but Not Tailscale IP

Symptom: Service works on public IP/domain but not on Tailscale IP (100.x.x.x).

Common Causes: - Service only bound to specific interface (not tailscale0) - Firewall blocking Tailscale interface - Service explicitly configured for public interface only

Solutions: 

# Check if service is listening on all interfaces
sudo ss -tlnup | grep :<port>
# Should show 0.0.0.0:<port> not just specific IP

# Check Tailscale interface
ip addr show tailscale0
# Note the IP address (e.g., 100.98.20.43)

# Test binding - if service only listens on specific IP, update config:
# Wrong: bind_address = "10.0.0.10"
# Right: bind_address = "0.0.0.0"  # Listen on all interfaces

# For Docker services, ensure they bind to all interfaces:
ports:
  - "0.0.0.0:53:53/udp"  # Explicit binding to all interfaces
  - "53:53/tcp"          # This also binds to all by default

2. DNS Resolution Issues with MagicDNS

Symptom: Can't resolve Tailscale hostnames (e.g., myserver.tail-scale.ts.net).

Debugging: 

# Check if MagicDNS is enabled
tailscale status
# Look for "MagicDNS: enabled"

# Test MagicDNS resolution
nslookup myserver.tail-scale.ts.net 100.100.100.100

# Check local DNS configuration
cat /etc/resolv.conf
# Should include Tailscale DNS (100.100.100.100) if using MagicDNS

# Force DNS query through Tailscale
dig @100.100.100.100 myserver.tail-scale.ts.net

Common Issues: - Local DNS resolver (like AdGuard) intercepting queries - MagicDNS disabled in Tailscale admin - Conflicting search domains

3. DERP Relay vs Direct Connections

Symptom: Slow connection or high latency to Tailscale services.

Understanding Connection Types: - Direct: Peer-to-peer connection (fast, low latency) - DERP: Relayed through Tailscale servers (slower, when direct fails)

Debugging: 

# Check connection type to a peer
tailscale netcheck

# See detailed peer information
tailscale status -peers
# Look for "relay" in the connection type

# Test direct connectivity
tailscale ping <tailscale-ip>
# Shows latency and connection type

# Force a direct connection attempt
tailscale ping --until-direct <tailscale-ip>

Common Causes of DERP Usage: - Strict NAT/firewall blocking UDP - CGNAT preventing direct connections - Corporate firewall policies

4. Tailscale Conflicts with AdGuard DNS

Symptom: DNS queries fail when both Tailscale and AdGuard are running.

The Conflict: - AdGuard binds to port 53 on all interfaces - Tailscale DNS wants to handle certain queries - Both services compete for DNS resolution

Solutions: 

# Option 1: Configure AdGuard to not interfere with Tailscale
# In AdGuard settings, add upstream for Tailscale domains:
# [/tail-scale.ts.net/]100.100.100.100

# Option 2: Use split DNS in Tailscale
# Configure Tailscale to only handle specific domains

# Option 3: Bind AdGuard to specific interfaces only
# In AdGuard config:
bind_hosts:
  - 10.0.0.10    # Local network interface only
  - 127.0.0.1    # Localhost
  # Don't bind to 0.0.0.0 or Tailscale interface

# Verify resolution works
dig @localhost google.com         # Should use AdGuard
dig @100.98.20.43 google.com     # Should use AdGuard via Tailscale
dig myserver.tail-scale.ts.net   # Should resolve via Tailscale

Debugging Commands

Essential Tailscale Status Commands

# Overall Tailscale status
tailscale status
# Shows: your IP, peer connections, relay status

# Detailed network check
tailscale netcheck
# Shows: UDP connectivity, nearest DERP, NAT type

# Test connectivity to specific peer
tailscale ping <peer-ip-or-hostname>
# Shows: latency, direct/relay status, packet loss

# Debug mode for detailed logs
sudo tailscaled --debug

Checking Service Binding

# See what interfaces exist
ip link show

# Check if service listens on Tailscale interface
sudo ss -tlnup | grep :53
# Look for entries with 100.98.20.43:53 or 0.0.0.0:53

# Test from another Tailscale device
curl http://100.98.20.43:<port>/health

# Check iptables for Tailscale rules
sudo iptables -L -n | grep -i tailscale

DNS Debugging with Tailscale

# Check current DNS configuration
resolvectl status

# Test DNS resolution via different servers
# Via Tailscale MagicDNS:
dig @100.100.100.100 myserver.tail-scale.ts.net

# Via your service on Tailscale IP:
dig @100.98.20.43 google.com

# Via public IP:
dig @<public-ip> google.com

# Check for DNS loops or conflicts
sudo tcpdump -i any -n port 53
# Then make a DNS query and watch the traffic flow

Common Scenarios and Solutions

Scenario 1: Web Service on Tailscale

# docker-compose.yml adjustments for Tailscale access
services:
  web:
    ports:
      - "0.0.0.0:8080:80"  # Bind to all interfaces
    environment:
      # Ensure app accepts connections from Tailscale network
      - ALLOWED_HOSTS=localhost,100.98.20.43,myserver.tail-scale.ts.net

Scenario 2: DNS Server (AdGuard) on Tailscale

# Proper configuration for DNS access via Tailscale
services:
  adguard:
    ports:
      - "0.0.0.0:53:53/tcp"
      - "0.0.0.0:53:53/udp"
      - "0.0.0.0:3000:3000"  # Web UI
    environment:
      # Configure upstream DNS carefully
      - UPSTREAM_DNS=1.1.1.1,8.8.8.8

Scenario 3: Database Access via Tailscale

# PostgreSQL configuration for Tailscale access
# In postgresql.conf:
listen_addresses = '*'  # or '0.0.0.0,100.98.20.43'

# In pg_hba.conf, add Tailscale network:
host    all    all    100.64.0.0/10    md5  # Tailscale CGNAT range

# Test connection:
psql -h 100.98.20.43 -U postgres -d mydb

Best Practices for Tailscale Services

  1. Always bind to all interfaces unless you have specific security requirements
  2. Document Tailscale IPs in your service documentation
  3. Test from both public and Tailscale IPs during deployment
  4. Monitor DERP usage - excessive relay usage indicates network issues
  5. Use Tailscale ACLs to control access instead of service-level filtering
  6. Configure health checks that work from both networks: 
    # Health check that works from anywhere
curl -f http://localhost:<port>/health || \
curl -f http://100.98.20.43:<port>/health || \
curl -f https://service.rbnk.uk/health

Prevention and Monitoring

  1. Set up alerts for Tailscale connectivity: 

    # Simple monitoring script
#!/bin/bash
if ! tailscale ping -c 1 100.98.20.43 >/dev/null 2>&1; then
  echo "Tailscale connectivity issue detected"
  # Send alert
fi

  2. Regular connectivity tests: 

    # Add to crontab
*/5 * * * * /usr/local/bin/check-tailscale-services.sh

  3. Document your Tailscale topology:

  4. Which services are accessible via Tailscale
  5. Which devices are in the Tailscale network
  6. Any special routing or ACL rules

Remember: Tailscale creates a private network overlay. Services accessible via Tailscale IPs are only reachable from devices authenticated and connected to your Tailscale network. This is a feature, not a bug - it provides an additional layer of security for internal services.

Proxmox VM Specific Issues

Overview

Running Docker inside a Proxmox VM introduces additional networking layers that can cause unique issues. The VM uses NAT for internet access, and Proxmox must forward ports from the host to the VM.

Common Issues and Solutions

1. DNS Port Forwarding Issue

Symptom: DNS service (like AdGuard) works inside VM but not from external networks.

Cause: Proxmox not forwarding port 53/UDP (only 53/TCP).

Solution: 

# Inside VM - verify AdGuard is listening
sudo ss -tlnup | grep :53
# Should show:
# udp   UNCONN 0      0      0.0.0.0:53         0.0.0.0:*    users:(("AdGuardHome",pid=1234,fd=15))
# tcp   LISTEN 0      4096   0.0.0.0:53         0.0.0.0:*    users:(("AdGuardHome",pid=1234,fd=14))

# Test from inside VM (should work)
dig @localhost google.com

# Test from outside (will fail if UDP not forwarded)
dig @<public-ip> google.com

# Fix: Add UDP port forward in Proxmox
# Datacenter > Node > VM > Firewall > Add rule:
# - Direction: in
# - Action: ACCEPT  
# - Protocol: UDP
# - Dest. port: 53
# - Comment: DNS UDP forward

2. NAT Loopback Issues

Symptom: Can't test public IP/domain from within the VM.

Cause: NAT hairpinning not configured in Proxmox.

Workarounds: 

# Option 1: Use SSH tunnel for testing
ssh -L 8080:localhost:80 user@proxmox-vm
# Then browse to http://localhost:8080

# Option 2: Add hosts entry for internal testing
echo "10.0.0.10 myservice.rbnk.uk" >> /etc/hosts

# Option 3: Test from external network
# Use phone hotspot or external VPS

3. Service Accessible via SSH Tunnel but Not Publicly

Symptom: Service works through SSH tunnel but not via public IP.

Debugging Steps: 

# 1. Verify service is running and listening
docker ps
sudo netstat -tlnup | grep <port>

# 2. Test locally in VM
curl http://localhost:<port>  # Should work

# 3. Check iptables/firewall in VM
sudo iptables -L -n | grep <port>

# 4. Verify Proxmox firewall rules
# Check both VM firewall and datacenter firewall
# Ensure port is forwarded in Network > Firewall settings

# 5. Test from Proxmox host
ssh root@proxmox-host
curl http://<vm-internal-ip>:<port>  # Should work if VM firewall OK

# 6. Check Proxmox NAT rules
# In Proxmox: Network > Edit vmbr0 > Post-up script
# Should contain iptables rules for port forwarding

4. Common Missing Port Forwards and Symptoms

Service Port Protocol Symptom if Missing
DNS (AdGuard) 53 UDP DNS queries timeout, dig fails
DNS (AdGuard) 53 TCP Large DNS responses fail
HTTP 80 TCP Can't access websites
HTTPS 443 TCP SSL sites unreachable
SSH 22 TCP Can't SSH to VM
PostgreSQL 5432 TCP Database connection refused
Custom APIs 3000-9000 TCP API calls fail

Case Study: AdGuard DNS Debugging Journey

Problem: AdGuard DNS worked perfectly inside the VM but external DNS queries failed.

Investigation Process: 

# 1. Verified AdGuard was running
docker ps | grep adguard
# ✓ Container running

# 2. Checked if listening on port 53
sudo ss -tlnup | grep :53
# ✓ Listening on both TCP and UDP

# 3. Tested DNS locally
dig @localhost google.com
# ✓ Worked perfectly

# 4. Tested from home network
dig @<public-ip> google.com
# ✗ Timeout - no response

# 5. Tested TCP DNS (uncommon but worth trying)
dig @<public-ip> +tcp google.com
# ✓ Worked! This pointed to UDP issue

# 6. Checked Proxmox port forwards
# Found: Only TCP/53 was forwarded, not UDP/53

Solution Applied: 1. Added UDP port 53 forward in Proxmox firewall 2. Applied the new firewall rules 3. DNS queries immediately started working

Lessons Learned: - Always check both TCP and UDP when dealing with services that use both - Proxmox firewall rules are applied at the hypervisor level, before traffic reaches the VM - Use protocol-specific testing tools (dig +tcp for DNS TCP) - Document all port forwards to avoid future confusion

Debugging Commands Specific to Proxmox VMs

# Check VM's IP configuration
ip addr show
ip route show

# Verify DNS resolution inside VM
cat /etc/resolv.conf
nslookup google.com

# Test connectivity from VM to internet
ping -c 4 8.8.8.8
curl -I https://google.com

# Check if Proxmox host can reach VM service
# From Proxmox host:
curl http://<vm-ip>:<port>

# Monitor traffic (inside VM)
sudo tcpdump -i any port <port>

# Check for dropped packets
sudo iptables -L -v -n | grep DROP

Prevention Tips for Proxmox Environments

  1. Document all port forwards: Maintain a list of all forwarded ports
  2. Test both protocols: If a service uses both TCP and UDP, forward both
  3. Use health checks: Implement external monitoring to catch issues early
  4. Regular firewall audits: Review Proxmox firewall rules monthly
  5. Test after Proxmox updates: Firewall rules might change after updates

Permission Errors

Symptom

Permission denied errors when accessing files or creating directories.

Common Causes and Solutions

1. Incorrect File Ownership

# Check current ownership
ls -la /srv/dockerdata/myapp/

# Fix ownership
sudo chown -R $USER:docker /srv/dockerdata/myapp/

# For specific containers running as non-root
docker exec myapp-web id
# uid=1000(node) gid=1000(node)

# Match host permissions
sudo chown -R 1000:1000 /srv/dockerdata/myapp/data/

2. SELinux or AppArmor

# Check if SELinux is enforcing
getenforce

# Temporary fix for testing
sudo setenforce 0

# Permanent fix: Add :Z to volume mount
volumes:
  - ./data:/data:Z

3. Read-Only Filesystem

# Check if filesystem is read-only
mount | grep /srv

# Remount as read-write if needed
sudo mount -o remount,rw /srv

Real Example: Supabase Storage Permissions

# Storage uploads failing with permission denied

# Check storage container user
docker exec supabase-storage id
# uid=101(storage) gid=101(storage)

# Fix permissions
sudo chown -R 101:101 /srv/dockerdata/supabase/volumes/storage/

# Also needed correct S3 permissions in .env
STORAGE_S3_ACL=private

Disk Space Issues

Symptom

No space left on device, containers crashing, or write failures.

Common Causes and Solutions

1. Check Disk Usage

# Overall disk usage
df -h

# Docker specific usage
docker system df

# Find large directories
du -sh /srv/dockerdata/*/ | sort -hr | head -10

2. Clean Up Docker Resources

# Remove stopped containers
docker container prune -f

# Remove unused images
docker image prune -a -f

# Remove unused volumes (CAREFUL!)
docker volume prune -f

# Complete cleanup
docker system prune -a --volumes -f

3. Log Rotation Issues

# Find large log files
find /srv/dockerdata -name "*.log" -size +100M

# Check container logs
docker ps -q | xargs -I {} sh -c 'echo "=== {} ===" && docker logs {} 2>&1 | wc -l'

# Add log rotation to docker-compose.yml:
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"

Real Example: Supabase Database Logs

# PostgreSQL logs filled disk

# Added to supabase/docker-compose.yml:
services:
  supa-db:
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "5"

# Recreate container to apply
docker compose -f supabase/docker-compose.yml up -d supa-db

Container Restart Loops

Symptom

Container continuously restarts, never staying up.

Common Causes and Solutions

1. Check Restart Policy and Logs

# See restart count
docker ps -a | grep myapp

# Check last few restarts
docker logs --tail=50 myapp-web-1

# Temporarily disable restart to debug
docker update --restart=no myapp-web-1

2. Dependency Issues

# Check if dependencies are ready
docker compose -f myapp/docker-compose.yml ps

# Add depends_on with health check:
depends_on:
  myapp-db:
    condition: service_healthy

3. Memory Limits Too Low

# Check if OOMKilled
docker inspect myapp-web-1 | jq '.[0].State.OOMKilled'

# Increase memory limit:
deploy:
  resources:
    limits:
      memory: 2G  # Increase from 512M

Real Example: LiteLLM Configuration Loop

# LiteLLM kept restarting

# Found in logs:
# "Error: OPENAI_API_KEY environment variable not set"

# But later:
# "Error: Invalid API key format"

# Solution: Validate all API keys before starting
# Added healthcheck that validates configuration

Database Connection Issues

Symptom

Connection refused, timeout, or authentication failures.

Common Causes and Solutions

1. Connection String Format

# PostgreSQL format:
# postgresql://[user]:[password]@[host]:[port]/[database]

# Common mistakes:
# Wrong: postgresql://user:pass@localhost:5432/db
# Right: postgresql://user:pass@container-name:5432/db

# Test connection:
docker run --rm -it --network=myapp_default postgres:15 \
  psql postgresql://user:pass@myapp-db:5432/db -c "SELECT 1"

2. Database Not Ready

# Add init wait script:
until pg_isready -h myapp-db -p 5432; do
  echo "Waiting for database..."
  sleep 2
done

3. Authentication Issues

# Check PostgreSQL logs
docker logs myapp-db 2>&1 | grep -i "authentication\|connection"

# Common fixes:
# 1. Ensure POSTGRES_PASSWORD matches in all services
# 2. Check pg_hba.conf allows connections
# 3. Verify user exists in database

Real Example: Supabase Multi-Database Setup

# Services couldn't connect after bootstrap

# Issue: Bootstrap SQL created new passwords
# Solution: Sync passwords in SQL with .env

# In bootstrap-sql/01-auth.sql:
ALTER USER authenticator WITH PASSWORD :'postgres_password';

# Pass env var to SQL:
docker exec -it supa-db psql -U postgres \
  -v postgres_password="'$POSTGRES_PASSWORD'"

Debugging Techniques and Tools

1. Container Inspection

# Full container details
docker inspect myapp-web-1 | jq

# Specific fields
docker inspect myapp-web-1 | jq '.[0].State'
docker inspect myapp-web-1 | jq '.[0].NetworkSettings.Networks'
docker inspect myapp-web-1 | jq '.[0].Mounts'

2. Execute Commands in Containers

# Get shell access
docker exec -it myapp-web-1 /bin/bash
# Or for Alpine:
docker exec -it myapp-web-1 /bin/sh

# Run diagnostic commands
docker exec myapp-web-1 ps aux
docker exec myapp-web-1 netstat -tlnp
docker exec myapp-web-1 env | sort

3. Network Debugging

# Test connectivity between containers
docker exec myapp-web-1 ping myapp-db
docker exec myapp-web-1 nc -zv myapp-db 5432
docker exec myapp-web-1 wget -O- http://myapp-api:3000/health

# DNS resolution
docker exec myapp-web-1 nslookup myapp-db
docker exec myapp-web-1 cat /etc/resolv.conf

4. Resource Monitoring

# Real-time stats
docker stats

# Historical data for specific container
docker logs myapp-web-1 2>&1 | grep -i "memory\|cpu"

# System resources
htop
iotop

5. Traefik Debugging

# Access Traefik dashboard (local only)
ssh -L 8080:localhost:8080 server
# Browse to http://localhost:8080

# API debugging
curl http://localhost:8080/api/http/routers | jq
curl http://localhost:8080/api/http/services | jq
curl http://localhost:8080/api/http/middlewares | jq

# Check specific route
curl http://localhost:8080/api/http/routers/myapp@docker | jq

Log Analysis Strategies

1. Aggregate Logs

# Combine logs from multiple services
docker compose -f myapp/docker-compose.yml logs -f

# Save logs for analysis
docker compose -f myapp/docker-compose.yml logs > myapp-logs.txt

# Search for errors
docker logs myapp-web-1 2>&1 | grep -i error

# Show context around errors
docker logs myapp-web-1 2>&1 | grep -B5 -A5 -i error

# Time-based filtering
docker logs --since="2024-01-11T10:00:00" myapp-web-1

3. Log Patterns to Watch For

# Memory issues
docker logs myapp-web-1 2>&1 | grep -i "out of memory\|oom\|killed"

# Connection issues
docker logs myapp-web-1 2>&1 | grep -i "connection refused\|timeout\|unreachable"

# Permission issues
docker logs myapp-web-1 2>&1 | grep -i "permission denied\|access denied\|forbidden"

# Startup issues
docker logs myapp-web-1 2>&1 | grep -i "failed to start\|exit\|crash"

4. Structured Log Analysis

# If service outputs JSON logs
docker logs myapp-web-1 2>&1 | jq 'select(.level == "error")'

# Count errors by type
docker logs myapp-web-1 2>&1 | jq -r '.error' | sort | uniq -c | sort -nr

5. Multi-Service Correlation

# Create timeline of events across services
for service in web db redis; do
  echo "=== myapp-$service ==="
  docker logs --timestamps myapp-$service-1 2>&1 | tail -20
done | sort -k2,3

# Watch specific operation across services
docker compose -f myapp/docker-compose.yml logs -f | grep -i "transaction-id-123"

Quick Reference Commands

# Service health check
docker compose -f <stack>/docker-compose.yml ps

# View recent logs
docker compose -f <stack>/docker-compose.yml logs --tail=100

# Restart problematic service
docker compose -f <stack>/docker-compose.yml restart <service>

# Rebuild and restart
docker compose -f <stack>/docker-compose.yml up -d --build <service>

# Emergency stop
docker compose -f <stack>/docker-compose.yml down

# Full reset (CAREFUL!)
docker compose -f <stack>/docker-compose.yml down -v
docker compose -f <stack>/docker-compose.yml up -d

Prevention Tips

  1. Always test locally first - Never deploy untested configurations
  2. Use health checks - Detect issues before they cascade
  3. Set resource limits - Prevent single service from taking down system
  4. Monitor logs - Set up alerts for error patterns
  5. Document changes - Keep track of what changed and why
  6. Backup before major changes - Always have a rollback plan
  7. Use version tags - Avoid latest tag in production

Remember: Most issues have been seen before. Check logs, verify configuration, and test connectivity systematically.