Infrastructure Maintenance Guide¶

Disk Status: System disk at 79% capacity - monitoring recommended

Disk Space Management¶

1. Regular Cleanup Commands¶

# Check current disk usage
df -h /srv

# Clean Docker system (removes unused containers, images, volumes)
docker system prune -a --volumes -f

# Check Docker disk usage details
docker system df

# Clean old logs
find /var/lib/docker/containers -name "*.log" -type f -size +100M -exec truncate -s 0 {} \;

# Remove old backups (keeping last 7 days instead of 14)
find /srv/dockerdata/_backup -type f -mtime +7 -delete

# Check what's using the most space
du -h /srv/dockerdata | sort -rh | head -20

2. Identify Large Files¶

# Find files larger than 1GB
find /srv -type f -size +1G -exec ls -lh {} \; | sort -k5 -rh

# Check container logs size
find /var/lib/docker/containers -name "*.log" -exec ls -lh {} \; | sort -k5 -rh

Daily Maintenance Tasks¶

Health Checks (5-10 minutes)¶

Service Status Check

# Check all running containers
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Size}}"

# Check Traefik routing
curl -s http://localhost:8080/api/http/routers | jq '.[] | {name: .name, rule: .rule, status: .status}'

# Check Supabase health
docker compose -f /srv/dockerdata/supabase/docker-compose.yml ps

Log Review

# Check for errors in last 24h
docker compose -f /srv/dockerdata/traefik/docker-compose.yml logs --since 24h 2>&1 | grep -i error
docker compose -f /srv/dockerdata/supabase/docker-compose.yml logs --since 24h 2>&1 | grep -i error

# Check authentication failures
docker logs supabase-auth --since 24h 2>&1 | grep -i "failed\|denied"

Disk Space Monitoring

# Create monitoring script
cat > /srv/dockerdata/_scripts/daily-health-check.sh << 'EOF'
#!/bin/bash

THRESHOLD=90
DISK_USAGE=$(df /srv | awk 'NR==2 {print $5}' | sed 's/%//')

if [ $DISK_USAGE -gt $THRESHOLD ]; then
    echo "WARNING: Disk usage at ${DISK_USAGE}%"
    docker system df
fi

# Check container health
docker ps --format "table {{.Names}}\t{{.Status}}" | grep -v "healthy\|Up"
EOF

chmod +x /srv/dockerdata/_scripts/daily-health-check.sh

Weekly Maintenance Tasks¶

1. Container Updates Check¶

# Check for available updates
cat > /srv/dockerdata/_scripts/check-updates.sh << 'EOF'
#!/bin/bash

echo "=== Checking for container updates ==="

# List all images in use
docker compose -f /srv/dockerdata/traefik/docker-compose.yml config --images
docker compose -f /srv/dockerdata/supabase/docker-compose.yml config --images

# Pull latest versions (without applying)
for compose in /srv/dockerdata/*/docker-compose.yml; do
    echo "Checking updates for: $compose"
    docker compose -f "$compose" pull --dry-run 2>/dev/null || true
done
EOF

chmod +x /srv/dockerdata/_scripts/check-updates.sh

2. Backup Verification¶

# Verify backup integrity
cat > /srv/dockerdata/_scripts/verify-backups.sh << 'EOF'
#!/bin/bash

BACKUP_DIR="/srv/dockerdata/_backup/supabase"
LATEST_BACKUP=$(ls -t $BACKUP_DIR/*.sql.gz 2>/dev/null | head -1)

if [ -z "$LATEST_BACKUP" ]; then
    echo "ERROR: No backups found!"
    exit 1
fi

# Test backup integrity
gunzip -t "$LATEST_BACKUP" && echo "Backup integrity: OK" || echo "Backup integrity: FAILED"

# Check backup age
BACKUP_AGE=$(( ($(date +%s) - $(stat -c %Y "$LATEST_BACKUP")) / 3600 ))
if [ $BACKUP_AGE -gt 48 ]; then
    echo "WARNING: Latest backup is $BACKUP_AGE hours old"
fi
EOF

chmod +x /srv/dockerdata/_scripts/verify-backups.sh

3. Security Patches Review¶

# Check system updates
apt update && apt list --upgradable

# Review Docker security updates
docker version
docker scout recommendations

4. Resource Usage Analysis¶

# Create resource analysis script
cat > /srv/dockerdata/_scripts/resource-analysis.sh << 'EOF'
#!/bin/bash

echo "=== Container Resource Usage ==="
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}"

echo -e "\n=== Top CPU Consumers ==="
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}" | sort -k2 -rn | head -5

echo -e "\n=== Top Memory Consumers ==="
docker stats --no-stream --format "table {{.Name}}\t{{.MemPerc}}" | sort -k2 -rn | head -5
EOF

chmod +x /srv/dockerdata/_scripts/resource-analysis.sh

Monthly Maintenance Tasks¶

1. Full System Audit¶

# Create audit script
cat > /srv/dockerdata/_scripts/monthly-audit.sh << 'EOF'
#!/bin/bash

echo "=== Monthly System Audit ==="
echo "Date: $(date)"

# Service inventory
echo -e "\n--- Running Services ---"
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

# Network configuration
echo -e "\n--- Docker Networks ---"
docker network ls

# Volume usage
echo -e "\n--- Docker Volumes ---"
docker volume ls
docker volume prune -f

# Security audit
echo -e "\n--- Exposed Ports ---"
docker ps --format "table {{.Names}}\t{{.Ports}}" | grep -E "0.0.0.0|:::"

# Certificate status
echo -e "\n--- SSL Certificate Status ---"
docker exec traefik cat /letsencrypt/acme.json | jq '.Certificates[].domain'
EOF

chmod +x /srv/dockerdata/_scripts/monthly-audit.sh

2. Performance Review¶

Analyze PostgreSQL query performance
Review application response times
Check Traefik access logs for slow requests
Analyze container restart frequency

3. Capacity Planning¶

# Growth tracking
cat > /srv/dockerdata/_scripts/capacity-planning.sh << 'EOF'
#!/bin/bash

# Database growth
echo "=== Database Size Growth ==="
docker exec supa-db psql -U postgres -c "SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database ORDER BY pg_database_size(pg_database.datname) DESC;"

# Disk usage trend (requires historical data)
echo -e "\n=== Disk Usage Trend ==="
df -h /srv

# Backup size trend
echo -e "\n=== Backup Size Trend ==="
ls -lh /srv/dockerdata/_backup/supabase/*.sql.gz | tail -10
EOF

chmod +x /srv/dockerdata/_scripts/capacity-planning.sh

4. Certificate Renewal Checks¶

# Check certificate expiration
docker exec traefik cat /letsencrypt/acme.json | jq -r '.Certificates[] | "\(.domain.main): \(.Store)"'

Container Updates¶

Watchtower Automated Updates¶

Watchtower provides automated container updates for non-critical services:

Schedule: Checks for updates every 24 hours
Scope: All containers except critical infrastructure services
Notifications: Email reports sent to admin after each update cycle
Cleanup: Automatically removes old images after successful updates

Excluded Services (Manual Updates Only)¶

Critical services excluded from auto-updates for stability: - Databases: supa-db, paperless-db, Redis instances - Infrastructure: traefik, prometheus, grafana, loki, alertmanager - Security: supabase-auth, adguard, ntfy - Self: watchtower (prevents update-during-update conflicts)

Watchtower Management¶

# Check Watchtower status and recent activity
docker logs watchtower --tail 50

# Force immediate update check (all containers)
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower --run-once --cleanup

# Update specific container only
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower --run-once --cleanup container_name

# Temporarily stop automated updates
docker compose -f /srv/dockerdata/watchtower/docker-compose.yml stop

# Resume automated updates
docker compose -f /srv/dockerdata/watchtower/docker-compose.yml start

Manual Update Procedures¶

For critical services excluded from Watchtower:

Pre-update Checklist

# 1. Create backup
/srv/dockerdata/_scripts/supabase-backup.sh

# 2. Check current versions
docker compose -f /srv/dockerdata/supabase/docker-compose.yml config --images

# 3. Review changelogs
# Visit container repositories for breaking changes

Update Process

# Single service update
cd /srv/dockerdata/supabase
docker compose pull supabase-auth
docker compose up -d supabase-auth

# Full stack update
docker compose pull
docker compose up -d

Post-update Verification

# Check service health
docker compose ps

# Verify functionality
curl -f https://supabase.rbnk.uk/auth/v1/health

Rollback Strategies¶

# Create rollback script
cat > /srv/dockerdata/_scripts/rollback-service.sh << 'EOF'
#!/bin/bash

SERVICE=$1
COMPOSE_FILE=$2
PREVIOUS_IMAGE=$3

if [ -z "$SERVICE" ] || [ -z "$COMPOSE_FILE" ] || [ -z "$PREVIOUS_IMAGE" ]; then
    echo "Usage: $0 <service> <compose-file> <previous-image>"
    exit 1
fi

# Stop current version
docker compose -f "$COMPOSE_FILE" stop "$SERVICE"

# Start with previous image
docker run -d --name "${SERVICE}_rollback" "$PREVIOUS_IMAGE"

echo "Rolled back $SERVICE to $PREVIOUS_IMAGE"
echo "Verify service, then update docker-compose.yml"
EOF

chmod +x /srv/dockerdata/_scripts/rollback-service.sh

Version Pinning Best Practices¶

# Always pin to specific versions in production
services:
  supa-db:
    image: supabase/postgres:15.8.1.129  # Specific version
    # NOT: image: supabase/postgres:latest

  traefik:
    image: traefik:v3.3.0  # Specific version
    # NOT: image: traefik:latest

Disk Management¶

Git Repository Cleanup (July 28, 2025)¶

Problem Solved: Repository size reduced from 400MB+ to 22MB by removing accidentally committed large files from Git history.

The Issue¶

OpenWebUI embedding model cache files (90MB+ each) were accidentally committed to the repository
AdGuard query logs were also taking up significant space
These files were removed from the working directory but remained in Git history
Every clone of the repository downloaded these unnecessary large files

The Solution: git-filter-repo¶

We used git-filter-repo, the officially recommended tool by Git for rewriting history (replacing the deprecated git filter-branch).

Installation:

# Install git-filter-repo (Python-based tool)
pip3 install git-filter-repo

Cleanup Process:

# 1. Create a fresh clone (git-filter-repo requires this)
git clone /srv/dockerdata /tmp/dockerdata-clean
cd /tmp/dockerdata-clean

# 2. Analyze repository to find large files
git filter-repo --analyze
# This creates a report in .git/filter-repo/analysis/

# 3. Remove large files from entire history
git filter-repo --path open-webui/data/cache/ --invert-paths --force
git filter-repo --path open-webui/data/models/ --invert-paths --force
git filter-repo --path adguard/data/querylog.json --invert-paths --force
git filter-repo --path adguard/data/querylog.json.1 --invert-paths --force

# 4. Verify cleanup
du -sh .git  # Should show significant reduction

# 5. Force push to remotes (history rewrite requires force push)
git remote add origin [email protected]:admin/infrastructure.git
git push --force origin main

Prevention: Updated .gitignore¶

Added proper entries to prevent future issues:

# OpenWebUI - Model cache and data
open-webui/data/cache/
open-webui/data/models/
open-webui/data/uploads/
open-webui/data/vector_db/
open-webui/data/*.db
open-webui/data/*.db-*

# AdGuard - Query logs and stats
adguard/data/querylog.json*
adguard/data/stats.db
adguard/data/sessions.db
adguard/data/filters/

Important Notes¶

This was a one-time cleanup operation - Only needed when large files are accidentally committed
History rewriting is destructive - All commit hashes changed after the cleanup
Coordination required - All team members needed to re-clone after the cleanup
Automatic GitHub sync - The cleaned history was automatically synced to GitHub mirror

Alternative Tools¶

While we used git-filter-repo, other tools are available:

BFG Repo-Cleaner: Faster for simple removals

java -jar bfg.jar --delete-files '*.db' --no-blob-protection repo.git

git filter-branch: Deprecated but still available

git filter-branch --index-filter 'git rm --cached --ignore-unmatch path/to/file' --all

Best Practices to Avoid Large Files¶

Review before committing:

# Check file sizes before adding
find . -type f -size +10M -exec ls -lh {} \;

# Use git status with size info
git status --porcelain | while read status file; do 
  echo "$status $file $(ls -lh "$file" 2>/dev/null | awk '{print $5}')"
done

Use Git LFS for large files:

git lfs track "*.model"
git lfs track "*.db"

Pre-commit hooks:

# Add to .git/hooks/pre-commit
files=$(git diff --cached --name-only)
for file in $files; do
  if [ -f "$file" ] && [ $(stat -c%s "$file") -gt 10485760 ]; then
    echo "Error: $file is larger than 10MB"
    exit 1
  fi
done

Cleanup Procedures¶

# Comprehensive cleanup script
cat > /srv/dockerdata/_scripts/disk-cleanup.sh << 'EOF'
#!/bin/bash

echo "=== Starting Disk Cleanup ==="

# 1. Docker cleanup
echo "Cleaning Docker resources..."
docker system prune -a --volumes -f
docker image prune -a -f

# 2. Log cleanup
echo "Cleaning logs..."
find /var/lib/docker/containers -name "*.log" -type f -size +100M -exec truncate -s 0 {} \;

# 3. Old backups
echo "Cleaning old backups..."
find /srv/dockerdata/_backup -type f -mtime +14 -delete

# 4. Temporary files
echo "Cleaning temp files..."
find /tmp -type f -atime +7 -delete 2>/dev/null

# 5. Package cache
echo "Cleaning package cache..."
apt-get clean
apt-get autoremove -y

# Show results
echo -e "\n=== Disk Usage After Cleanup ==="
df -h /srv
docker system df
EOF

chmod +x /srv/dockerdata/_scripts/disk-cleanup.sh

Log Rotation¶

# Configure Docker log rotation
cat > /etc/docker/daemon.json << 'EOF'
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "5"
  }
}
EOF

# Restart Docker to apply
systemctl restart docker

Growth Monitoring¶

# Create growth monitoring script
cat > /srv/dockerdata/_scripts/monitor-growth.sh << 'EOF'
#!/bin/bash

LOG_FILE="/var/log/disk-growth.log"

# Log current usage
echo "$(date '+%Y-%m-%d %H:%M:%S') $(df /srv | awk 'NR==2 {print $5}')" >> $LOG_FILE

# Alert if growth rate is high
CURRENT=$(df /srv | awk 'NR==2 {print $5}' | sed 's/%//')
YESTERDAY=$(tail -24 $LOG_FILE 2>/dev/null | head -1 | awk '{print $3}' | sed 's/%//')

if [ ! -z "$YESTERDAY" ]; then
    GROWTH=$((CURRENT - YESTERDAY))
    if [ $GROWTH -gt 5 ]; then
        echo "WARNING: Disk usage increased by ${GROWTH}% in last 24 hours"
    fi
fi
EOF

chmod +x /srv/dockerdata/_scripts/monitor-growth.sh

# Add to crontab
(crontab -l 2>/dev/null; echo "0 * * * * /srv/dockerdata/_scripts/monitor-growth.sh") | crontab -

Emergency Procedures¶

Service Recovery¶

# Service recovery script
cat > /srv/dockerdata/_scripts/service-recovery.sh << 'EOF'
#!/bin/bash

SERVICE=$1

case $SERVICE in
    "supabase")
        echo "Recovering Supabase..."
        cd /srv/dockerdata/supabase
        docker compose down
        docker compose up -d
        sleep 10
        docker compose ps
        ;;
    "traefik")
        echo "Recovering Traefik..."
        cd /srv/dockerdata/traefik
        docker compose down
        docker compose up -d
        ;;
    "all")
        echo "Recovering all services..."
        for compose in /srv/dockerdata/*/docker-compose.yml; do
            dir=$(dirname "$compose")
            echo "Recovering $dir..."
            docker compose -f "$compose" down
            docker compose -f "$compose" up -d
        done
        ;;
    *)
        echo "Usage: $0 {supabase|traefik|all}"
        exit 1
        ;;
esac
EOF

chmod +x /srv/dockerdata/_scripts/service-recovery.sh

System Recovery¶

Boot Issues

# Check Docker service
systemctl status docker
systemctl restart docker

# Check disk space
df -h

# Check system logs
journalctl -xe

Network Issues

# Reset Docker networks
docker network prune -f
docker network create traefik_proxy
docker network create supabase_default

Data Recovery¶

# PostgreSQL recovery from backup
cat > /srv/dockerdata/_scripts/restore-postgres.sh << 'EOF'
#!/bin/bash

BACKUP_FILE=$1

if [ -z "$BACKUP_FILE" ]; then
    echo "Usage: $0 <backup-file.sql.gz>"
    exit 1
fi

echo "Restoring PostgreSQL from $BACKUP_FILE..."

# Stop dependent services
docker compose -f /srv/dockerdata/supabase/docker-compose.yml stop supabase-rest supabase-auth

# Restore database
gunzip -c "$BACKUP_FILE" | docker exec -i supa-db psql -U postgres

# Restart services
docker compose -f /srv/dockerdata/supabase/docker-compose.yml up -d

echo "Restore complete"
EOF

chmod +x /srv/dockerdata/_scripts/restore-postgres.sh

Incident Response¶

Service Outage
Check container status: docker ps -a
Review logs: docker logs <container> --tail 100
Check resources: docker stats
Restart if needed: docker restart <container>
Security Incident
Isolate affected containers: docker stop <container>
Review access logs: docker logs traefik | grep <suspicious-ip>
Check for unauthorized changes: docker diff <container>
Rotate secrets if compromised

Automation¶

Maintenance Scripts¶

# Create master automation script
cat > /srv/dockerdata/_scripts/automated-maintenance.sh << 'EOF'
#!/bin/bash

LOG_FILE="/var/log/maintenance.log"

echo "=== Automated Maintenance Started: $(date) ===" >> $LOG_FILE

# Daily tasks
/srv/dockerdata/_scripts/daily-health-check.sh >> $LOG_FILE 2>&1

# Weekly tasks (on Sundays)
if [ $(date +%w) -eq 0 ]; then
    echo "Running weekly maintenance..." >> $LOG_FILE
    /srv/dockerdata/_scripts/check-updates.sh >> $LOG_FILE 2>&1
    /srv/dockerdata/_scripts/verify-backups.sh >> $LOG_FILE 2>&1
fi

# Monthly tasks (on the 1st)
if [ $(date +%d) -eq 01 ]; then
    echo "Running monthly maintenance..." >> $LOG_FILE
    /srv/dockerdata/_scripts/monthly-audit.sh >> $LOG_FILE 2>&1
fi

# Disk cleanup if usage > 85%
DISK_USAGE=$(df /srv | awk 'NR==2 {print $5}' | sed 's/%//')
if [ $DISK_USAGE -gt 85 ]; then
    echo "Disk usage at ${DISK_USAGE}%, running cleanup..." >> $LOG_FILE
    /srv/dockerdata/_scripts/disk-cleanup.sh >> $LOG_FILE 2>&1
fi

echo "=== Automated Maintenance Completed: $(date) ===" >> $LOG_FILE
EOF

chmod +x /srv/dockerdata/_scripts/automated-maintenance.sh

Scheduled Tasks¶

# Setup systemd timer for automation
cat > /etc/systemd/system/docker-maintenance.service << 'EOF'
[Unit]
Description=Docker Infrastructure Maintenance
After=docker.service

[Service]
Type=oneshot
ExecStart=/srv/dockerdata/_scripts/automated-maintenance.sh
User=root
StandardOutput=journal
StandardError=journal
EOF

cat > /etc/systemd/system/docker-maintenance.timer << 'EOF'
[Unit]
Description=Run Docker Maintenance Daily
Requires=docker-maintenance.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 04:00:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Enable timer
systemctl daemon-reload
systemctl enable docker-maintenance.timer
systemctl start docker-maintenance.timer

Health Check Automation¶

# Create health check endpoint
cat > /srv/dockerdata/_scripts/health-endpoint.sh << 'EOF'
#!/bin/bash

# Simple HTTP health check server
while true; do
    STATUS="OK"

    # Check critical services
    docker ps | grep -q "supa-db" || STATUS="FAIL"
    docker ps | grep -q "traefik" || STATUS="FAIL"

    # Check disk space
    DISK_USAGE=$(df /srv | awk 'NR==2 {print $5}' | sed 's/%//')
    [ $DISK_USAGE -gt 90 ] && STATUS="DISK_CRITICAL"

    echo -e "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\nStatus: $STATUS" | nc -l -p 8888 -q 1
done
EOF

chmod +x /srv/dockerdata/_scripts/health-endpoint.sh

Update Notifications¶

# Create update notification script
cat > /srv/dockerdata/_scripts/notify-updates.sh << 'EOF'
#!/bin/bash

# Check for updates and notify
UPDATES=$(/srv/dockerdata/_scripts/check-updates.sh | grep "newer" | wc -l)

if [ $UPDATES -gt 0 ]; then
    echo "Container updates available: $UPDATES"
    # Add notification method here (email, webhook, etc.)
fi
EOF

chmod +x /srv/dockerdata/_scripts/notify-updates.sh

Quick Reference¶

Critical Commands¶

# Emergency disk cleanup
docker system prune -a --volumes -f && find /srv/dockerdata/_backup -type f -mtime +3 -delete

# Service restart
docker compose -f /srv/dockerdata/<service>/docker-compose.yml restart

# Check all services
for d in /srv/dockerdata/*/; do echo "=== $d ==="; docker compose -f "$d/docker-compose.yml" ps; done

# Emergency backup
/srv/dockerdata/_scripts/supabase-backup.sh

# View recent errors
docker compose logs --since 1h 2>&1 | grep -i error

Monitoring Checklist¶

[ ] Disk usage < 85%
[ ] All containers running
[ ] No error logs in last hour
[ ] Backup completed today
[ ] SSL certificates valid
[ ] Response times normal
[ ] Memory usage stable
[ ] No security alerts

Support Contacts¶

System Administrator: [Your contact]
Emergency Escalation: [Emergency contact]
Documentation: /srv/dockerdata/docs/
Scripts Location: /srv/dockerdata/_scripts/

Last Updated: $(date) Auto-generated maintenance procedures - customize as needed

Important System Changes¶

Docker Data Root Migration (July 18, 2025)¶

Action: Moved Docker data root from /var/lib/docker to /srv/dockerdata/docker-root
Result: System disk usage reduced from 93% to 45%
Method: Symlink created from old location to new
Impact: All containers successfully migrated and running