Backup and Restore Operations Guide

Overview

This guide covers comprehensive backup and restoration procedures for all services in the infrastructure, including automated backup strategies, disaster recovery planning, and point-in-time recovery.

Current Backup Infrastructure

Existing Scripts

1. Supabase Backup (/srv/dockerdata/_scripts/supabase-backup.sh)
2. Performs logical dumps using pg_dumpall
3. Backs up configuration files and bootstrap SQL
4. 14-day retention policy

5. Target: /srv/backups/supabase/

6. Pocketbase Backup (/srv/dockerdata/_scripts/pocketbase-backup.sh)

7. Stops container for SQLite consistency
8. Archives all PocketBase data directories
9. 14-day retention policy

10. Target: /srv/backups/pocketbase/

11. Gitea Backup (/srv/dockerdata/_scripts/gitea-backup.sh)

12. Uses Gitea's built-in dump command
13. Backs up repositories, LFS data, database, and configuration
14. 14-day retention policy
15. Target: /srv/backups/gitea/
16. Automated via systemd timer (2 AM daily)

Service-Specific Backup Procedures

Supabase PostgreSQL

Manual Backup

# Run existing backup script
/srv/dockerdata/_scripts/supabase-backup.sh

# Alternative: Direct pg_dump for specific database
docker exec supa-db pg_dump -U postgres -d postgres > backup-$(date +%Y%m%d%H%M%S).sql

# Physical backup using pg_basebackup
docker exec supa-db pg_basebackup -U postgres -D /tmp/backup -Ft -z -P
docker cp supa-db:/tmp/backup ./physical-backup-$(date +%Y%m%d%H%M%S)

Restoration Process

# Stop all Supabase services except database
cd /srv/dockerdata/supabase
docker compose stop supabase-auth supabase-rest supabase-realtime supabase-storage supabase-meta supabase-studio

# Restore from logical backup
gunzip -c /srv/backups/supabase/db/pgdumpall-TIMESTAMP.sql.gz | \
  docker exec -i supa-db psql -U postgres

# Verify restoration
docker exec supa-db psql -U postgres -c "\l"
docker exec supa-db psql -U postgres -c "\dt+ public.*"

# Restart services
docker compose up -d

Traefik

Backup

# Manual backup
cd /srv/dockerdata
tar -czf /srv/backups/traefik-$(date +%Y%m%d%H%M%S).tgz \
  traefik/docker-compose.yml \
  traefik/traefik.yml \
  traefik/dynamic/ \
  traefik/.env

# Backup certificates (handle with care)
sudo tar -czf /srv/backups/traefik-certs-$(date +%Y%m%d%H%M%S).tgz \
  traefik/letsencrypt/acme.json

Restoration

# Stop Traefik
cd /srv/dockerdata/traefik
docker compose down

# Restore configuration
tar -xzf /srv/backups/traefik-TIMESTAMP.tgz -C /srv/dockerdata

# Restore certificates (if needed)
tar -xzf /srv/backups/traefik-certs-TIMESTAMP.tgz -C /srv/dockerdata
chmod 600 /srv/dockerdata/traefik/letsencrypt/acme.json

# Start Traefik
docker compose up -d

Gitea

Backup

# Automated backup (recommended)
/srv/dockerdata/_scripts/gitea-backup.sh

# Manual backup using Gitea's built-in command
docker exec -u git gitea gitea dump -c /data/gitea/conf/app.ini \
  -w /tmp -f /tmp/gitea-dump-$(date +%Y%m%d%H%M%S).zip

# Extract dump from container
docker cp gitea:/tmp/gitea-dump-*.zip /srv/backups/gitea/app/

# Verify backup integrity
unzip -t /srv/backups/gitea/app/gitea-dump-*.zip

Restoration

# Stop Gitea service
cd /srv/dockerdata/gitea
docker compose down

# Extract the backup
cd /tmp
unzip /srv/backups/gitea/app/gitea-dump-TIMESTAMP.zip

# Restore database (SQLite)
docker run --rm -v gitea_gitea_data:/data -v /tmp:/backup \
  alpine sh -c "rm -f /data/gitea/gitea.db && cp /backup/gitea-db.sql /data/gitea/"

# Restore repositories
docker run --rm -v gitea_gitea_data:/data -v /tmp:/backup \
  alpine sh -c "rm -rf /data/git && tar -xzf /backup/gitea-repo.zip -C /data/"

# Restore configuration
docker run --rm -v gitea_gitea_data:/data -v /tmp:/backup \
  alpine sh -c "cp /backup/app.ini /data/gitea/conf/"

# Restore LFS data if exists
if [[ -f /tmp/data-lfs.zip ]]; then
  docker run --rm -v gitea_gitea_data:/data -v /tmp:/backup \
    alpine sh -c "rm -rf /data/git/lfs && unzip /backup/data-lfs.zip -d /data/git/"
fi

# Fix permissions
docker run --rm -v gitea_gitea_data:/data \
  alpine chown -R 1000:1000 /data

# Start Gitea
docker compose up -d

# Verify restoration
docker exec gitea gitea admin user list

Paperless-AI

Backup

# Automated backup via infrastructure script (recommended)
/srv/dockerdata/rclone/backup-infrastructure.sh

# Manual backup of Docker volume
docker run --rm -v paperless-ai_data:/source -v /srv/backups:/backup \
  alpine tar -czf /backup/paperless-ai-data-$(date +%Y%m%d%H%M%S).tar.gz -C /source .

# Backup configuration files
cd /srv/dockerdata
tar -czf /srv/backups/paperless-ai-config-$(date +%Y%m%d%H%M%S).tgz \
  paperless-ai/docker-compose.yml \
  paperless-ai/.env

Restoration

# Stop Paperless-AI service
cd /srv/dockerdata/paperless-ai
docker compose down

# Restore configuration
tar -xzf /srv/backups/paperless-ai-config-TIMESTAMP.tgz -C /srv/dockerdata

# Restore data volume
docker run --rm -v paperless-ai_data:/dest -v /srv/backups:/backup \
  alpine tar -xzf /backup/paperless-ai-data-TIMESTAMP.tar.gz -C /dest

# Start service
docker compose up -d

# Verify restoration
docker logs paperless-ai

Watchtower

Backup

# Backup configuration only (no persistent data)
cd /srv/dockerdata
tar -czf /srv/backups/watchtower-$(date +%Y%m%d%H%M%S).tgz \
  watchtower/docker-compose.yml \
  watchtower/config.json \
  watchtower/.env

Restoration

# Restore configuration
tar -xzf /srv/backups/watchtower-TIMESTAMP.tgz -C /srv/dockerdata

# Start Watchtower
cd /srv/dockerdata/watchtower
docker compose up -d

# Verify operation
docker logs watchtower

Metube

Backup

# Note: Downloads directory can be large - backup selectively
# Backup configuration only
cd /srv/dockerdata
tar -czf /srv/backups/metube-config-$(date +%Y%m%d%H%M%S).tgz \
  metube/docker-compose.yml \
  metube/config/ \
  metube/.env

# Manual backup of important downloads (optional)
# WARNING: This can be very large
rsync -av --progress /srv/dockerdata/metube/downloads/ /srv/backups/metube-downloads/

Restoration

# Stop Metube service
cd /srv/dockerdata/metube
docker compose down

# Restore configuration
tar -xzf /srv/backups/metube-config-TIMESTAMP.tgz -C /srv/dockerdata

# Restore downloads (if backed up)
rsync -av --progress /srv/backups/metube-downloads/ /srv/dockerdata/metube/downloads/

# Start service
docker compose up -d

# Verify access
curl -I https://metube.rbnk.uk

Generic Docker Volume Backup

# Backup any Docker volume
VOLUME_NAME="volume_name"
docker run --rm -v $VOLUME_NAME:/data -v /srv/backups:/backup \
  alpine tar -czf /backup/${VOLUME_NAME}-$(date +%Y%m%d%H%M%S).tgz -C /data .

# Restore Docker volume
docker run --rm -v $VOLUME_NAME:/data -v /srv/backups:/backup \
  alpine tar -xzf /backup/${VOLUME_NAME}-TIMESTAMP.tgz -C /data

Automated Backup Setup

Existing Automated Backups

Gitea has its own systemd timer already configured: - gitea-backup.timer: Runs daily at 2 AM - gitea-backup.service: Executes /srv/dockerdata/_scripts/gitea-backup.sh - Installation: /srv/dockerdata/gitea/systemd/install-systemd-services.sh

Create Systemd Service and Timer

# Create backup service
sudo tee /etc/systemd/system/dockerdata-backup.service > /dev/null << 'EOF'
[Unit]
Description=Docker Data Backup Service
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/srv/dockerdata/_scripts/run-all-backups.sh
StandardOutput=journal
StandardError=journal
SyslogIdentifier=dockerdata-backup

[Install]
WantedBy=multi-user.target
EOF

# Create timer
sudo tee /etc/systemd/system/dockerdata-backup.timer > /dev/null << 'EOF'
[Unit]
Description=Run Docker Data Backup at 3 AM daily
Requires=dockerdata-backup.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

# Create master backup script
sudo tee /srv/dockerdata/_scripts/run-all-backups.sh > /dev/null << 'EOF'
#!/bin/bash
set -euo pipefail

LOGFILE="/var/log/dockerdata-backup.log"
exec 1> >(tee -a "$LOGFILE")
exec 2>&1

echo "=== Starting backup run at $(date) ==="

# Send notification that backup is starting
notify "Backup Started" "Daily backup job has started" 2 backup-info-daily

# Track failures
FAILED_SCRIPTS=""

# Run all backup scripts
for script in /srv/dockerdata/_scripts/*-backup.sh; do
    if [[ -x "$script" ]]; then
        echo "Running: $script"
        if ! "$script"; then
            echo "WARNING: $script failed with exit code $?"
            FAILED_SCRIPTS="${FAILED_SCRIPTS}$(basename $script), "
        fi
    fi
done

# Send completion notification
if [ -z "$FAILED_SCRIPTS" ]; then
    notify "Backup Success" "All backup jobs completed successfully" 3 backup-info-daily
else
    notify "Backup Failed" "Failed scripts: ${FAILED_SCRIPTS%%, }" 5 backup-error-daily
fi

echo "=== Backup run completed at $(date) ==="
EOF

sudo chmod +x /srv/dockerdata/_scripts/run-all-backups.sh

# Enable and start timer
sudo systemctl daemon-reload
sudo systemctl enable dockerdata-backup.timer
sudo systemctl start dockerdata-backup.timer

# Check timer status
sudo systemctl list-timers dockerdata-backup.timer

Backup Notifications

The backup system integrates with the notification stack to provide real-time status updates:

Notification Events

• Backup Start: Notifies when backup jobs begin
• Backup Success: Confirms successful completion
• Backup Failure: Alerts on any failures with specific script names
• Storage Warnings: Alerts when backup storage exceeds thresholds

Configure Backup Notifications

# Notifications are sent via the notify command
# Format: notify "title" "message" priority topic

# Topics used:
# - backup-info-daily: Regular status updates
# - backup-error-daily: Failure alerts
# - backup-warning-storage: Storage warnings

# To receive notifications, configure your preferred channels in:
# - Apprise: https://apprise.rbnk.uk
# - ntfy: Subscribe to topics via mobile app or web

Backup Verification

Automated Verification Script

# Create verification script
sudo tee /srv/dockerdata/_scripts/verify-backups.sh > /dev/null << 'EOF'
#!/bin/bash
set -euo pipefail

BACKUP_ROOT="/srv/backups"
VERIFY_DIR="/tmp/backup-verify-$$"
ERRORS=0

# Function to verify tar archives
verify_tar() {
    local file=$1
    echo "Verifying tar archive: $file"
    if tar -tzf "$file" > /dev/null 2>&1; then
        echo "✓ Archive is valid"
        return 0
    else
        echo "✗ Archive is corrupted!"
        return 1
    fi
}

# Function to verify SQL dumps
verify_sql() {
    local file=$1
    local tempfile="${file%.gz}"

    echo "Verifying SQL dump: $file"

    # Decompress and check syntax
    if gunzip -c "$file" > "$tempfile" 2>/dev/null; then
        # Basic SQL syntax check
        if head -n 100 "$tempfile" | grep -q "PostgreSQL database dump"; then
            echo "✓ SQL dump appears valid"
            rm -f "$tempfile"
            return 0
        else
            echo "✗ SQL dump format invalid!"
            rm -f "$tempfile"
            return 1
        fi
    else
        echo "✗ Failed to decompress SQL dump!"
        return 1
    fi
}

# Check Supabase backups
echo "=== Verifying Supabase Backups ==="
for backup in $(find $BACKUP_ROOT/supabase -name "*.sql.gz" -mtime -1); do
    verify_sql "$backup" || ((ERRORS++))
done

for backup in $(find $BACKUP_ROOT/supabase -name "*.tgz" -mtime -1); do
    verify_tar "$backup" || ((ERRORS++))
done

# Check other service backups
echo "=== Verifying Other Service Backups ==="
for backup in $(find $BACKUP_ROOT -name "*.tgz" -mtime -1 | grep -v supabase); do
    verify_tar "$backup" || ((ERRORS++))
done

# Cleanup
rm -rf "$VERIFY_DIR"

# Report
if [[ $ERRORS -eq 0 ]]; then
    echo "✓ All backups verified successfully"
    exit 0
else
    echo "✗ Found $ERRORS backup errors!"
    exit 1
fi
EOF

sudo chmod +x /srv/dockerdata/_scripts/verify-backups.sh

Disaster Recovery Plan

Full System Recovery Procedure

1. Prepare New System

   # Install Docker and dependencies
   curl -fsSL https://get.docker.com | sh
   sudo usermod -aG docker $USER
   
   # Create directory structure
   sudo mkdir -p /srv/dockerdata /srv/backups
   cd /srv/dockerdata
   

2. Restore Configuration

   # Copy latest config backup
   scp backup-server:/srv/backups/config/latest.tgz .
   tar -xzf latest.tgz
   
   # Set correct permissions
   find . -name "*.env" -exec chmod 640 {} \;
   find . -name "docker-compose.yml" -exec chmod 644 {} \;
   

3. Restore Services in Order

   # 1. Restore Traefik first (for networking)
   cd /srv/dockerdata/traefik
   docker compose up -d
   
   # 2. Restore Supabase database
   cd /srv/dockerdata/supabase
   docker compose up -d supa-db
   # Wait for database to be ready
   sleep 30
   
   # 3. Restore database content
   gunzip -c /srv/backups/supabase/db/latest.sql.gz | \
     docker exec -i supa-db psql -U postgres
   
   # 4. Start remaining Supabase services
   docker compose up -d
   
   # 5. Restore other services in priority order
   # Priority 1: Essential services
   for service in gitea monitoring; do
       if [[ -f "/srv/dockerdata/$service/docker-compose.yml" ]]; then
           cd "/srv/dockerdata/$service"
           docker compose up -d
           sleep 10  # Wait for service to start
       fi
   done
   
   # Priority 2: Application services
   for service in paperless-ai n8n open-webui litellm; do
       if [[ -f "/srv/dockerdata/$service/docker-compose.yml" ]]; then
           cd "/srv/dockerdata/$service"
           docker compose up -d
           sleep 5
       fi
   done
   
   # Priority 3: Utility services
   for service in watchtower metube glance rclone; do
       if [[ -f "/srv/dockerdata/$service/docker-compose.yml" ]]; then
           cd "/srv/dockerdata/$service"
           docker compose up -d
       fi
   done
   

Point-in-Time Recovery (PostgreSQL)

1. Enable WAL Archiving

   # Add to PostgreSQL configuration
   docker exec supa-db bash -c "cat >> /var/lib/postgresql/data/postgresql.conf << EOF
   wal_level = replica
   archive_mode = on
   archive_command = 'test ! -f /archive/%f && cp %p /archive/%f'
   EOF"
   
   # Create archive directory
   docker exec supa-db mkdir -p /archive
   docker exec supa-db chown postgres:postgres /archive
   
   # Restart PostgreSQL
   docker compose restart supa-db
   

2. Perform PITR

   # Stop PostgreSQL
   docker compose stop supa-db
   
   # Restore base backup
   docker run --rm -v supabase_db_data:/data -v /srv/backups:/backup \
     alpine tar -xzf /backup/pg-base-backup.tgz -C /data
   
   # Copy WAL files
   docker run --rm -v supabase_db_data:/data -v /srv/backups/wal:/wal \
     alpine cp /wal/* /data/pg_wal/
   
   # Create recovery.conf
   docker run --rm -v supabase_db_data:/data alpine sh -c "cat > /data/recovery.conf << EOF
   restore_command = 'cp /archive/%f %p'
   recovery_target_time = '2024-01-18 12:00:00'
   EOF"
   
   # Start PostgreSQL
   docker compose start supa-db
   

Offsite Backup Strategy

Rclone Containerized Service

Rclone is deployed as a containerized service providing automated cloud backups with web GUI management:

• Web Interface: https://rclone.rbnk.uk
• Configuration: /srv/dockerdata/rclone/config/rclone.conf
• Data Directory: /srv/dockerdata/rclone/data/
• Backup Source: Read-only mount of /srv/dockerdata

See Rclone Service Documentation for detailed configuration and operations.

Legacy Rclone Setup (Host-based)

For systems still using host-based rclone:

# Install rclone
curl https://rclone.org/install.sh | sudo bash

# Configure remote (interactive)
rclone config

# Create sync script
sudo tee /srv/dockerdata/_scripts/offsite-sync.sh > /dev/null << 'EOF'
#!/bin/bash
set -euo pipefail

RCLONE_REMOTE="backblaze:dockerdata-backups"
LOCAL_BACKUP="/srv/backups"

echo "Starting offsite sync at $(date)"

# Sync with bandwidth limit
rclone sync "$LOCAL_BACKUP" "$RCLONE_REMOTE" \
  --bwlimit 10M \
  --transfers 4 \
  --checkers 8 \
  --log-level INFO \
  --log-file /var/log/rclone-backup.log

# Verify remote
rclone size "$RCLONE_REMOTE"

echo "Offsite sync completed at $(date)"
EOF

sudo chmod +x /srv/dockerdata/_scripts/offsite-sync.sh

# Add to cron (run at 5 AM)
echo "0 5 * * * /srv/dockerdata/_scripts/offsite-sync.sh" | sudo crontab -

Backup Monitoring

# Create monitoring script
sudo tee /srv/dockerdata/_scripts/backup-monitor.sh > /dev/null << 'EOF'
#!/bin/bash

WEBHOOK_URL="https://discord.com/api/webhooks/YOUR_WEBHOOK_HERE"
BACKUP_DIR="/srv/backups"
MAX_AGE_HOURS=26

send_alert() {
    local message=$1
    curl -H "Content-Type: application/json" \
         -d "{\"content\": \"🚨 Backup Alert: $message\"}" \
         "$WEBHOOK_URL"
}

# Check for recent backups
for service in supabase traefik pocketbase gitea; do
    latest=$(find "$BACKUP_DIR/$service" -type f -name "*.tgz" -o -name "*.sql.gz" | \
             xargs ls -t | head -1)

    if [[ -z "$latest" ]]; then
        send_alert "No backups found for $service!"
        continue
    fi

    age_hours=$(( ($(date +%s) - $(stat -c %Y "$latest")) / 3600 ))

    if [[ $age_hours -gt $MAX_AGE_HOURS ]]; then
        send_alert "Backup for $service is $age_hours hours old!"
    fi
done

# Check backup disk space
usage=$(df -h "$BACKUP_DIR" | awk 'NR==2 {print $5}' | sed 's/%//')
if [[ $usage -gt 80 ]]; then
    send_alert "Backup disk usage is at ${usage}%!"
fi
EOF

sudo chmod +x /srv/dockerdata/_scripts/backup-monitor.sh

# Add to cron (check every 6 hours)
echo "0 */6 * * * /srv/dockerdata/_scripts/backup-monitor.sh" | sudo crontab -

Testing and Maintenance

Quarterly Restoration Test

1. Create test environment
2. Restore from backup
3. Verify functionality
4. Document any issues
5. Update procedures as needed

Backup Maintenance Checklist

• [ ] Verify all services have recent backups
• [ ] Check backup sizes for anomalies
• [ ] Test restoration on non-production system
• [ ] Review and update retention policies
• [ ] Verify offsite sync is working
• [ ] Check available disk space
• [ ] Review backup logs for errors
• [ ] Update documentation with any changes

Emergency Contacts and Resources

• Backup storage locations: /srv/backups/
• Offsite backups: Configured in rclone
• Recovery documentation: This file
• System architecture: /srv/dockerdata/docs/architecture/
• Service credentials: Stored in .env files (check permissions!)