Backup Alerts Runbook

Overview

This runbook provides step-by-step troubleshooting procedures for all backup-related alerts in the infrastructure. Each alert includes immediate actions, investigation steps, and resolution procedures.

Alert Categories

Alert Name	Severity	Type	SLA
CloudBackupMissing	Critical	Missing backup	48 hours
InfrastructureBackupMissing	Critical	Missing backup	48 hours
CloudBackupFailed	Critical	Backup failure	Immediate
InfrastructureBackupFailed	Critical	Backup failure	Immediate
BackupErrors	Warning	Error accumulation	3 errors/24h
R2StorageHigh	Warning	Storage capacity	80% usage
BackupDurationHigh	Warning	Performance	2x normal duration

---

CloudBackupMissing Alert

Alert Definition

alert: CloudBackupMissing
expr: time() - cloud_backup_last_run_timestamp > 172800  # 48 hours
for: 5m
labels:
  severity: critical
  service: backup
  category: cloud

Immediate Actions (< 5 minutes)

1. Check Current Status

   # Check last backup timestamp
   curl -s localhost:9100/metrics | grep cloud_backup_last_run_timestamp
   
   # Calculate hours since last backup
   last_backup=$(curl -s localhost:9100/metrics | grep cloud_backup_last_run_timestamp | awk '{print $2}')
   current=$(date +%s)
   hours_ago=$(( (current - last_backup) / 3600 ))
   echo "Last backup: $hours_ago hours ago"
   

2. Verify Service Status

   # Check if Rclone service is running
   docker ps | grep rclone
   
   # Check Rclone service logs for errors
   docker compose -f rclone/docker-compose.yml logs -f --tail=50
   

3. Check Cron Job Status

   # Verify cron daemon is running
   systemctl status cron
   
   # Check recent cron activity
   journalctl -u cron | grep backup | tail -10
   

Investigation Steps (5-15 minutes)

1. Examine Backup Logs

   # Check main backup log
   tail -50 /srv/dockerdata/rclone/backup.log
   
   # Look for recent errors
   grep -i error /srv/dockerdata/rclone/backup.log | tail -10
   
   # Check for authentication issues
   grep -i "auth\|token\|permission" /srv/dockerdata/rclone/backup.log | tail -5
   

2. Verify Cloud Service Connectivity

   # Test Google Drive connection
   docker exec rclone rclone lsd gdrive: --max-depth 1
   
   # Test OneDrive connection
   docker exec rclone rclone lsd onedrive: --max-depth 1
   
   # Test Hetzner Storage Box connection
   docker exec rclone rclone lsd hetzner: --max-depth 1
   

3. Check Wrapper Script Status

   # Verify wrapper script exists and is executable
   ls -la /srv/dockerdata/rclone/backup-clouds-with-metrics.sh
   
   # Check for syntax errors
   bash -n /srv/dockerdata/rclone/backup-clouds-with-metrics.sh
   

Resolution Procedures

If Cron Job Failed:

# Check crontab entry
crontab -l | grep backup

# Re-add missing cron entry if needed
echo "0 2 * * * /srv/dockerdata/rclone/backup-clouds-with-metrics.sh" | crontab -

# Restart cron service
systemctl restart cron

If Cloud Authentication Failed:

# Re-authenticate with cloud services
docker exec -it rclone rclone config

# Test authentication
docker exec rclone rclone about gdrive:
docker exec rclone rclone about onedrive:

If Hetzner Storage Issues:

# Check Hetzner connectivity
docker exec rclone rclone about hetzner:

# Verify storage box isn't full
docker exec rclone rclone size hetzner:/backups

# Check for network issues
docker exec rclone ping storage-box.de -c 3

Manual Recovery:

# Run backup manually to clear alert
/srv/dockerdata/rclone/backup-clouds-with-metrics.sh

# Verify metrics updated
cat /var/lib/node_exporter/textfile_collector/cloud_backup.prom

# Check alert status in Prometheus
curl -s 'localhost:9090/api/v1/query?query=cloud_backup_last_run_timestamp' | jq

---

InfrastructureBackupMissing Alert

Alert Definition

alert: InfrastructureBackupMissing
expr: time() - infrastructure_backup_last_run_timestamp > 172800  # 48 hours
for: 5m
labels:
  severity: critical
  service: backup
  category: infrastructure

Immediate Actions (< 5 minutes)

1. Check Backup Status

   # Check last infrastructure backup
   curl -s localhost:9100/metrics | grep infrastructure_backup_last_run_timestamp
   
   # Check R2 connection
   docker exec rclone rclone about cloudflare:
   

2. Verify Critical Services

   # Check PostgreSQL (critical for backup)
   docker ps | grep supa-db
   
   # Check Rclone service
   docker compose -f rclone/docker-compose.yml ps
   

Investigation Steps (5-15 minutes)

1. Examine Infrastructure Backup Logs

   # Check infrastructure backup log
   tail -50 /srv/dockerdata/rclone/backup-infrastructure.log
   
   # Look for R2 errors
   grep -i "r2\|cloudflare\|error" /srv/dockerdata/rclone/backup-infrastructure.log | tail -10
   

2. Check Database Connectivity

   # Test PostgreSQL connection
   docker exec supa-db psql -U postgres -c "SELECT version();"
   
   # Check database size (for backup planning)
   docker exec supa-db psql -U postgres -c "SELECT pg_size_pretty(pg_database_size('postgres'));"
   

3. Verify R2 Access

   # List recent backups in R2
   docker exec rclone rclone lsl cloudflare:infrastructure-backups | tail -5
   
   # Check R2 storage usage
   docker exec rclone rclone size cloudflare:infrastructure-backups
   

Resolution Procedures

If Database Backup Failed:

# Test manual database dump
docker exec supa-db pg_dumpall -U postgres > test-dump.sql

# Check dump file size and integrity
ls -lh test-dump.sql
head -20 test-dump.sql
rm test-dump.sql

If R2 Connection Failed:

# Check R2 credentials in environment
docker exec rclone printenv | grep -i cloudflare

# Test R2 connection
docker exec rclone rclone about cloudflare:

# Re-configure R2 if needed
docker exec -it rclone rclone config

If Storage Full:

# Clean old backups manually
docker exec rclone rclone delete cloudflare:infrastructure-backups \
  --min-age 30d --dry-run  # Remove --dry-run to execute

# Check space after cleanup
docker exec rclone rclone size cloudflare:infrastructure-backups

Manual Recovery:

# Run infrastructure backup manually
/srv/dockerdata/rclone/backup-infrastructure-with-metrics.sh

# Verify backup completed
docker exec rclone rclone lsl cloudflare:infrastructure-backups | tail -1

# Check updated metrics
cat /var/lib/node_exporter/textfile_collector/infrastructure_backup.prom

---

BackupErrors Alert

Alert Definition

alert: BackupErrors
expr: increase(cloud_backup_errors_total[24h]) > 3 or increase(infrastructure_backup_errors_total[24h]) > 3
for: 5m
labels:
  severity: warning
  service: backup

Investigation Steps

1. Identify Error Patterns

   # Check error metrics
   curl -s localhost:9100/metrics | grep backup_errors_total
   
   # Look for recent errors in logs
   grep -i error /srv/dockerdata/rclone/backup.log | tail -20
   grep -i error /srv/dockerdata/rclone/backup-infrastructure.log | tail -20
   

2. Categorize Errors

   # Network errors
   grep -i "network\|timeout\|connection" /srv/dockerdata/rclone/backup*.log | tail -10
   
   # Authentication errors
   grep -i "auth\|token\|permission\|forbidden" /srv/dockerdata/rclone/backup*.log | tail -10
   
   # Storage errors
   grep -i "space\|quota\|full" /srv/dockerdata/rclone/backup*.log | tail -10
   

Resolution Based on Error Type

Network/Timeout Errors:

# Test connectivity
docker exec rclone ping google.com -c 3
docker exec rclone ping storage-box.de -c 3

# Check network interface
ip route show

Authentication Errors:

# Re-authenticate services
docker exec -it rclone rclone config

# Check token expiration
docker exec rclone rclone config show | grep token

Storage Capacity Errors:

# Check local disk space
df -h /srv/dockerdata
df -h /srv/backups

# Check cloud storage quotas
docker exec rclone rclone about gdrive:
docker exec rclone rclone about hetzner:

---

R2StorageHigh Alert

Alert Definition

alert: R2StorageHigh
expr: infrastructure_backup_r2_usage_percentage > 80
for: 5m
labels:
  severity: warning
  service: backup
  category: storage

Immediate Actions

1. Check Current Usage

   # Get current R2 usage
   curl -s localhost:9100/metrics | grep r2_usage
   
   # Get detailed R2 information
   docker exec rclone rclone about cloudflare: --json | jq
   

2. Analyze Backup Growth

   # List backups by size
   docker exec rclone rclone lsl cloudflare:infrastructure-backups | \
     awk '{print $1/1048576 " MB", $2, $3, $4}' | sort -nr | head -10
   
   # Check backup retention
   docker exec rclone rclone lsl cloudflare:infrastructure-backups | \
     awk '{print $2, $3}' | sort | tail -20
   

Resolution Steps

1. Clean Old Backups

   # Remove backups older than 14 days (dry run first)
   docker exec rclone rclone delete cloudflare:infrastructure-backups \
     --min-age 14d --dry-run
   
   # Execute cleanup if dry run looks correct
   docker exec rclone rclone delete cloudflare:infrastructure-backups \
     --min-age 14d
   

2. Optimize Backup Size

   # Find largest files in recent backup
   backup_date=$(docker exec rclone rclone lsl cloudflare:infrastructure-backups | \
     tail -1 | awk '{print $4}' | cut -d'/' -f1)
   
   docker exec rclone rclone ls cloudflare:infrastructure-backups/$backup_date \
     --min-size 50M | sort -nr
   

3. Adjust Retention Policy

   # Edit backup script to reduce retention
   nano /srv/dockerdata/rclone/backup-infrastructure-with-metrics.sh
   
   # Change RETENTION_DAYS from 14 to 10 or 7 days
   # Then run cleanup
   

---

BackupDurationHigh Alert

Alert Definition

alert: BackupDurationHigh
expr: cloud_backup_duration_seconds > 14400 or infrastructure_backup_duration_seconds > 7200
for: 0m
labels:
  severity: warning
  service: backup
  category: performance

Investigation Steps

1. Check Current Duration

   # Get recent backup durations
   curl -s localhost:9100/metrics | grep backup_duration_seconds
   
   # Convert to hours for readability
   duration=$(curl -s localhost:9100/metrics | grep cloud_backup_duration_seconds | awk '{print $2}')
   echo "Last backup took: $(($duration / 3600)) hours $(($duration % 3600 / 60)) minutes"
   

2. Analyze Performance Factors

   # Check system load during backup time
   uptime
   iostat -x 1 5
   
   # Check network performance
   iperf3 -c speedtest.net -p 5201 -t 10
   
   # Check disk performance
   hdparm -tT /dev/sda
   

3. Identify Bottlenecks

   # Check for large files in backup
   find /srv/dockerdata -type f -size +100M -exec ls -lh {} \; | head -10
   
   # Check backup transfer rate
   grep -i "transferred\|rate" /srv/dockerdata/rclone/backup.log | tail -5
   

Optimization Actions

1. Optimize Rclone Settings

   # Edit backup script to add performance flags
   nano /srv/dockerdata/rclone/backup-clouds-with-metrics.sh
   
   # Add these flags to rclone sync:
   # --transfers 8 --checkers 16 --fast-list
   

2. Exclude Large Non-Critical Files

   # Add exclusions to backup script
   # --exclude "*.tmp" --exclude "*.log" --exclude "cache/**"
   

3. Schedule Optimization

   # Stagger backup times to reduce system load
   # Edit crontab to spread backups across different hours
   crontab -e
   

---

Common Backup Issues and Solutions

Issue: Backup Stuck/Hanging

Symptoms: - Backup process running for hours without completion - No progress in logs - High system load

Solution:

# Kill stuck backup process
pkill -f backup-clouds-with-metrics.sh
pkill -f backup-infrastructure-with-metrics.sh

# Clear any lock files
rm -f /tmp/backup-*.lock

# Check for zombie rclone processes
ps aux | grep rclone | grep -v grep

# Restart rclone service if needed
docker compose -f rclone/docker-compose.yml restart

Issue: Metrics Not Updating

Symptoms: - Backup runs successfully but metrics show old timestamps - Alert continues firing despite successful backup

Solution:

# Check textfile collector directory
ls -la /var/lib/node_exporter/textfile_collector/

# Check file permissions
chmod 644 /var/lib/node_exporter/textfile_collector/*.prom

# Verify metrics file content
cat /var/lib/node_exporter/textfile_collector/cloud_backup.prom

# Restart node exporter
docker compose -f monitoring/docker-compose.yml restart node-exporter

Issue: Authentication Expired

Symptoms: - "token expired" or "authentication failed" errors - Backup fails at cloud service connection

Solution:

# Re-run rclone config for affected service
docker exec -it rclone rclone config

# For Google Drive (browser-based auth):
# 1. Select existing config
# 2. Choose to edit
# 3. Accept all defaults
# 4. Complete browser authentication

# For OneDrive (similar process):
# Follow same steps but for OneDrive config

# Test authentication
docker exec rclone rclone lsd gdrive:
docker exec rclone rclone lsd onedrive:

Issue: Insufficient Storage Space

Symptoms: - "no space left on device" errors - Backup fails during file operations

Solution:

# Check disk usage
df -h

# Clean up old backups
find /srv/backups -name "*.tgz" -mtime +14 -delete
find /srv/backups -name "*.sql.gz" -mtime +14 -delete

# Clean Docker unused resources
docker system prune -f

# Clean log files
find /srv/dockerdata -name "*.log" -size +100M -exec truncate -s 10M {} \;

---

Escalation Procedures

Level 1: Automated Resolution

• Automatic retry logic in backup scripts (3 attempts)
• Temporary error tolerance (up to 3 errors per 24h)
• Self-healing for transient network issues

Level 2: Administrator Notification

• Trigger: Critical alerts (missing backups, failures)
• Channels: ntfy push notification, email alert
• Response Time: 4 hours during business hours, 24 hours otherwise

Level 3: Emergency Response

• Trigger: Data loss risk, multiple backup failures
• Actions:
• Immediate manual backup execution
• Alternative backup strategy activation
• Infrastructure health assessment

Emergency Contact Information

Primary Administrator: - ntfy topic: backup-critical-failure - Email: configured in AlertManager - Response SLA: 4 hours

Backup Recovery Team: - Available during critical data loss scenarios - Access to all backup credentials and procedures - Authority to implement emergency recovery procedures

---

Prevention and Maintenance

Daily Checks

# Automated health check script
/srv/dockerdata/_scripts/backup-health-check.sh

Weekly Reviews

• Review backup success rates in Grafana
• Check storage growth trends
• Verify alert rule effectiveness
• Test random restore procedures

Monthly Tasks

• Full disaster recovery test
• Backup strategy review
• Performance optimization assessment
• Documentation updates

Quarterly Assessments

• Backup retention policy review
• Storage cost optimization
• Recovery time objective validation
• Technology stack updates

---

Related Documentation

• Backup Metrics Implementation - Technical implementation details
• Backup Strategy Guide - Overall backup architecture
• Grafana Dashboard Configuration - Dashboard management
• Prometheus Alert Configuration - Alert rule management