Watchtower Container Auto-Update Documentation

Table of Contents

1. Overview
2. Architecture
3. Configuration
4. Excluded Services
5. Common Operations
6. Security Considerations
7. Best Practices
8. Troubleshooting
9. Monitoring & Notifications

Overview

Watchtower is an automated Docker container update service that monitors running containers for newer image versions. When updates are detected, Watchtower pulls the new image, gracefully stops the old container, and starts a new one with the same configuration.

Key Features

• Automated Updates: Checks for image updates every 24 hours
• Graceful Restarts: Stops containers cleanly before starting updated versions
• Email Notifications: Sends detailed reports via Gmail SMTP
• Selective Updates: Supports exclusion labels for critical services
• Cleanup: Automatically removes old Docker images after successful updates
• Zero-Downtime: Minimal service interruption during updates

Why Watchtower?

In a Docker-based infrastructure with 20+ services, manual updates become time-consuming and error-prone. Watchtower provides:

1. Security: Ensures containers run latest security patches
2. Efficiency: Eliminates manual update workflows
3. Consistency: Standardized update process across all services
4. Monitoring: Email reports keep you informed of all changes
5. Safety: Critical services can be excluded from automatic updates

Architecture

Core Components

Watchtower Container
├── Docker Socket Access (read-only)
├── Image Monitoring (every 24h)
├── Update Process
│   ├── Pull new image
│   ├── Stop old container
│   ├── Start new container
│   └── Cleanup old image
└── Notification System (SMTP)

Docker Socket Access

Watchtower requires access to the Docker socket to: - Monitor running containers - Pull new images from registries - Stop and start containers - Clean up old images

The socket is mounted read-only for security: /var/run/docker.sock:/var/run/docker.sock:ro

Update Process Flow

1. Discovery: Watchtower scans all running containers
2. Filter: Excludes containers with com.centurylinklabs.watchtower.enable=false label
3. Check: Compares local image digest with registry digest
4. Pull: Downloads newer image if available
5. Stop: Gracefully stops the current container (30s timeout)
6. Start: Creates new container with identical configuration
7. Cleanup: Removes old image to save disk space
8. Notify: Sends email report of all actions taken

Configuration

Environment Variables

Located in /srv/dockerdata/watchtower/docker-compose.yml:

environment:
  # Update frequency (86400 = 24 hours)
  - WATCHTOWER_POLL_INTERVAL=86400

  # Cleanup old images after update
  - WATCHTOWER_CLEANUP=true

  # Don't update stopped containers
  - WATCHTOWER_INCLUDE_STOPPED=false

  # Include containers that are restarting
  - WATCHTOWER_INCLUDE_RESTARTING=true

  # Email notifications
  - WATCHTOWER_NOTIFICATIONS=email
  - [email protected]
  - [email protected]
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.gmail.com
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=587
  - [email protected]
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=${WATCHTOWER_EMAIL_PASSWORD}  # Set in .env file
  - WATCHTOWER_NOTIFICATION_EMAIL_DELAY=2

  # Logging and behavior
  - WATCHTOWER_LOG_LEVEL=info
  - WATCHTOWER_ROLLING_RESTART=false
  - WATCHTOWER_TIMEOUT=30s
  - TZ=Europe/London

Configuration Options Explained

Update Frequency

• WATCHTOWER_POLL_INTERVAL=86400: Check every 24 hours
• Alternative values: 43200 (12h), 3600 (1h), 1800 (30min)

Cleanup Behavior

• WATCHTOWER_CLEANUP=true: Remove old images after successful update
• Saves disk space but removes rollback capability

Container Selection

• WATCHTOWER_INCLUDE_STOPPED=false: Ignore stopped containers
• WATCHTOWER_INCLUDE_RESTARTING=true: Update containers in restart loop

Notification Template

• WATCHTOWER_NOTIFICATION_TEMPLATE: Custom email format
• Current: Simple line-by-line message format

Safety Features

• WATCHTOWER_TIMEOUT=30s: Maximum time to wait for graceful shutdown
• WATCHTOWER_ROLLING_RESTART=false: Update all containers simultaneously

Gmail SMTP Configuration

Watchtower uses Gmail with an App Password for notifications:

1. SMTP Server: smtp.gmail.com:587 (TLS)
2. Authentication: App Password (not regular password)
3. From Address: [email protected] (display name)
4. To Address: [email protected]

Note: Gmail App Passwords provide secure access without exposing the main account password.

Excluded Services

Critical infrastructure services are excluded from automatic updates using the label:

labels:
  - "com.centurylinklabs.watchtower.enable=false"

Currently Excluded Services

Database Services

• supabase (supa-db): PostgreSQL database with critical data
• paperless-db: Document management database
• monitoring (prometheus, loki): Time-series databases with historical data
• Redis instances: Fast data stores requiring manual coordination

Infrastructure Services

• traefik: Reverse proxy - updates could break SSL certificates or routing
• prometheus: Metrics database - data loss risk during updates
• grafana: Dashboard service with custom configurations
• loki: Log aggregation service with indexed data
• alertmanager: Alert routing service requiring configuration stability

Security Services

• supabase-auth: Authentication service - updates could break user sessions
• adguard: DNS filtering - critical for network security
• ntfy: Notification service - required for monitoring alerts

Self-Excluded

• watchtower: Excludes itself to prevent update-during-update conflicts

Rationale for Exclusions

1. Data Integrity: Database services require backup verification before updates
2. Service Continuity: Infrastructure services need planned maintenance windows
3. Configuration Preservation: Services with complex configs need manual validation
4. Security Verification: Auth services require testing before production updates
5. Dependency Management: Services with tight coupling need coordinated updates

Common Operations

View Update Status

# Check Watchtower logs for recent activity
docker logs watchtower

# View last 100 log entries
docker logs --tail 100 watchtower

# Follow live updates (useful during scheduled check times)
docker logs -f watchtower

Manual Update Trigger

# Force immediate update check (without waiting for schedule)
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower --run-once --cleanup

# Update specific container only
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower --run-once --cleanup container_name

Emergency Operations

# Stop Watchtower (prevent automatic updates)
docker compose -f /srv/dockerdata/watchtower/docker-compose.yml stop

# Restart Watchtower service
docker compose -f /srv/dockerdata/watchtower/docker-compose.yml restart

# Temporarily disable for maintenance
docker compose -f /srv/dockerdata/watchtower/docker-compose.yml down

Configuration Changes

# Edit environment variables
nano /srv/dockerdata/watchtower/docker-compose.yml

# Apply changes (restart required)
docker compose -f /srv/dockerdata/watchtower/docker-compose.yml up -d

# Verify new configuration
docker logs watchtower | tail -20

Check Excluded Services

# Find all containers excluded from Watchtower
find /srv/dockerdata -name "docker-compose.yml" -exec grep -l "com.centurylinklabs.watchtower.enable=false" {} \;

# Verify exclusion labels on running containers
docker ps --format "table {{.Names}}\t{{.Labels}}" | grep watchtower.enable

Security Considerations

Docker Socket Access

Risk: Watchtower has privileged access to the Docker daemon Mitigation: - Read-only socket mount - Runs in isolated container - No host network access - Regular security updates via auto-update

Email Credentials

Risk: SMTP credentials stored in plain text Mitigation: - App Password instead of main password - Container-only environment variables - File permissions restricted to root

Update Risks

Risk: Automatic updates could introduce breaking changes Mitigation: - Excluded critical services from auto-updates - Email notifications for all changes - Docker image cleanup provides quick rollback - Non-rolling updates for immediate detection

Network Security

Risk: Outbound connections for image pulls and email Mitigation: - Only connects to trusted registries (Docker Hub, etc.) - SMTP over TLS encryption - No inbound network access

Best Practices

Service Exclusion Guidelines

Always Exclude: - Database services (data integrity risk) - Authentication services (session disruption) - Infrastructure services (system stability) - Services with persistent state

Consider Excluding: - Services with complex configurations - Services with tight version dependencies - Low-change, high-stability services - Services requiring coordinated updates

Safe for Auto-Update: - Stateless web applications - API services without complex state - Monitoring agents and exporters - Development/testing services

Version Pinning Strategy

For critical services, use specific image tags instead of latest:

# Good: Specific version
image: postgres:15.8

# Avoid: Latest tag (unpredictable updates)
image: postgres:latest

# Alternative: Major version pinning
image: postgres:15

Update Scheduling

Current Schedule: Daily at varying times based on container start order Recommendation: Consider scheduled updates during maintenance windows

# Alternative: Weekly updates on Sunday at 2 AM
WATCHTOWER_POLL_INTERVAL=604800  # 7 days
# Use external cron job to start/stop Watchtower at specific times

Notification Management

Current: All updates generate emails Optimization: Configure notification levels

# Only notify on warnings and errors
WATCHTOWER_NOTIFICATION_LEVEL=warn

# Custom notification template for better formatting
WATCHTOWER_NOTIFICATION_TEMPLATE=|
  {{range .}}
  Container: {{.Name}}
  Image: {{.ImageName}}
  Status: {{.State}}
  {{if .Error}}Error: {{.Error}}{{end}}
  {{println}}
  {{end}}

Backup Integration

Before relying on automatic updates:

1. Verify Backups: Ensure all critical data is backed up
2. Test Restores: Validate backup restoration procedures
3. Monitor Results: Check email notifications for update status
4. Document Dependencies: Know which services depend on each other

Troubleshooting

Common Issues and Solutions

1. Updates Not Happening

Symptoms: No email notifications, containers not updating despite available images Diagnosis:

# Check Watchtower is running
docker ps | grep watchtower

# Verify schedule and logs
docker logs watchtower | grep -i "poll\|schedule\|check"

# Check last activity timestamp
docker logs watchtower | tail -20

Solutions: - Verify WATCHTOWER_POLL_INTERVAL is reasonable (not too high) - Check Docker socket permissions - Restart Watchtower service - Verify internet connectivity for image pulls

2. Email Notifications Not Working

Symptoms: Updates happen but no emails received Diagnosis:

# Check email configuration in logs
docker logs watchtower | grep -i "email\|smtp\|notification"

# Verify SMTP settings
docker exec watchtower env | grep WATCHTOWER_NOTIFICATION

Solutions: - Verify Gmail App Password is correct - Check spam folder for notifications - Test SMTP connectivity from container - Verify WATCHTOWER_NOTIFICATIONS=email is set

3. Container Update Failures

Symptoms: Emails show update attempts but containers not updated Diagnosis:

# Check for specific error messages
docker logs watchtower | grep -i "error\|failed\|timeout"

# Verify image registry accessibility
docker pull <failing-image>

# Check container dependencies
docker inspect <container> | jq .HostConfig.Links

Solutions: - Increase WATCHTOWER_TIMEOUT for slow-starting services - Check image registry availability - Verify container has no conflicting labels - Check for port conflicts during restart

4. Disk Space Issues

Symptoms: Updates fail with "no space left on device" Diagnosis:

# Check disk usage
df -h /srv/dockerdata
docker system df

# Check if cleanup is enabled
docker logs watchtower | grep -i "cleanup\|prune\|remove"

Solutions: - Verify WATCHTOWER_CLEANUP=true is set - Manually clean old images: docker image prune -a - Monitor disk usage before scheduled updates - Consider reducing update frequency

5. Service Dependencies Broken

Symptoms: Updated service fails to start or connect to dependencies Diagnosis:

# Check container startup logs
docker logs <updated-container>

# Verify network connectivity
docker exec <container> ping <dependency>

# Check if dependent services need updates too
docker images | grep <image-name>

Solutions: - Use WATCHTOWER_ROLLING_RESTART=true for ordered updates - Exclude services with tight coupling - Implement health checks in dependent services - Coordinate manual updates for dependent service groups

Debug Mode

Enable verbose logging for troubleshooting:

# Temporarily change log level
WATCHTOWER_LOG_LEVEL=debug

# Or use trace for maximum verbosity
WATCHTOWER_LOG_LEVEL=trace

# Apply changes and monitor
docker compose -f /srv/dockerdata/watchtower/docker-compose.yml up -d
docker logs -f watchtower

Manual Container Updates

If automatic updates fail, update manually:

# Stop service
docker compose -f /srv/dockerdata/<service>/docker-compose.yml down

# Pull latest image
docker compose -f /srv/dockerdata/<service>/docker-compose.yml pull

# Start service with new image
docker compose -f /srv/dockerdata/<service>/docker-compose.yml up -d

# Verify service health
docker logs <service-container>

Monitoring & Notifications

Email Notification Format

Current notifications include: - Container Name: Which service was updated - Image Details: Old and new image information - Update Status: Success or failure - Timestamp: When the update occurred - Error Details: If update failed

Integration with Monitoring Stack

Watchtower integrates with the existing monitoring infrastructure:

Prometheus Metrics

Watchtower doesn't expose Prometheus metrics directly, but container restart events are visible in: - Container uptime metrics - Docker daemon metrics
- Application-specific health checks

Log Aggregation

Watchtower logs are captured by the Docker daemon and can be: - Forwarded to Loki via Docker logging driver - Parsed for alert conditions - Analyzed for update patterns

Notification Stack Integration

For enhanced notifications, consider routing through the notification stack:

# Send Watchtower status to ntfy
docker logs watchtower --since 1h | grep -i "updated\|failed" | \
  while read line; do
    notify "Watchtower Update" "$line" info deployment
  done

Update Reports

Weekly summary of auto-update activity:

# Generate weekly update report
docker logs watchtower --since 168h | grep -E "updated|failed|error" > /tmp/watchtower-weekly.log

# Send to notification system
notify "Watchtower Weekly Report" "$(cat /tmp/watchtower-weekly.log)" normal maintenance

Future Enhancements

Planned improvements to Watchtower monitoring:

1. Prometheus Exporter: Custom exporter for update metrics
2. Grafana Dashboard: Visual update tracking and failure rates
3. Advanced Notifications: Integration with Slack/Discord via notification stack
4. Update Scheduling: Cron-based updates during maintenance windows
5. Rollback Automation: Automatic rollback on health check failures

---

Summary

Watchtower provides essential automated container updates for this Docker infrastructure, keeping services secure and up-to-date while protecting critical components through selective exclusions. The current configuration balances automation with safety, ensuring that routine updates happen automatically while requiring manual intervention for infrastructure-critical services.

The 24-hour update cycle with email notifications provides a good balance between staying current and maintaining system stability. The exclusion of databases, authentication services, and core infrastructure ensures that critical services remain stable while less critical services benefit from automatic security updates.

For additional monitoring and management capabilities, Watchtower integrates well with the existing notification stack and monitoring infrastructure, providing multiple layers of update tracking and alerting.

Regular review of excluded services and update patterns helps maintain an optimal balance between automation and control, ensuring the infrastructure remains both secure and stable.