Backup Strategy Guide¶

Overview¶

This document outlines the comprehensive backup strategy for the infrastructure, covering both local service backups and cloud storage synchronization. The strategy ensures data protection through automated backups, retention policies, and offsite storage.

Backup Architecture¶

Two-Tier Backup System¶

Local Service Backups: Database dumps, configuration files, and application data
Cloud Storage Backups: Synchronization of cloud services (Google Drive, OneDrive) to Hetzner Storage Box

Backup Classification and Priority¶

Services are classified into different backup priorities based on criticality and data characteristics:

Critical - Automated Daily Backups¶

Supabase PostgreSQL: Primary application database
Gitea: Source code repositories and Git data
Paperless-AI: AI application data and configurations
Infrastructure Configs: All service configurations, environment files, certificates

Important - Automated Daily Backups¶

Pocketbase Instances: Application-specific databases
Traefik Certificates: SSL certificates and ACME data
Monitoring Data: Grafana dashboards, Prometheus configurations

Low Priority - Configuration Only¶

Watchtower: Minimal configuration backup (no persistent data)
Service Configs: Docker compose files and basic configurations

Optional - Manual/On-Demand¶

Metube Downloads: User media files (/srv/dockerdata/metube/downloads/)
Classification: User media content (videos, audio files)
Size: Variable, can be large (GBs per download)
Backup recommendation: Manual backup of important downloads only
Restoration: Not critical for infrastructure operation

Backup Schedule¶

Service	Schedule	Time	Retention	Location
Cloud Storage (Rclone)	Daily	2:00 AM	Full sync	Hetzner Storage Box
Gitea	Daily	2:00 AM	14 days	`/srv/backups/gitea/`
Supabase	Daily	3:00 AM	14 days	`/srv/backups/supabase/`
Pocketbase	Daily	3:00 AM	14 days	`/srv/backups/pocketbase/`
Paperless-AI	Daily	3:00 AM	14 days	Included in Infrastructure
Metube Downloads	Manual	On-demand	User discretion	Optional backup
Infrastructure (Rclone)	Daily	4:00 AM	14 days	Cloudflare R2
R2 → Google Drive Archive	Weekly	Sunday 6:00 AM	60 days	Google Drive

Cloud Storage Backups¶

Overview¶

Automated daily backups of cloud storage services to ensure data availability and protection against cloud service outages or data loss. Enhanced with comprehensive metrics collection for monitoring and alerting.

Configuration¶

Script Location: /srv/dockerdata/rclone/backup-clouds-with-metrics.sh (wrapper with metrics)
Original Script: /srv/dockerdata/rclone/backup-clouds.sh (preserved)
Log File: /srv/dockerdata/rclone/backup.log
Schedule: Daily at 2:00 AM via cron
Destination: Hetzner Storage Box (hetzner:/backups/)
Monitoring: Prometheus metrics via textfile collector

Backed Up Services¶

Google Drive
Full sync to hetzner:/backups/gdrive
Estimated size: ~217GB+
Includes all files and folder structure
OneDrive
Full sync to hetzner:/backups/onedrive
Includes all files and folder structure

Sync Behavior¶

Method: rclone sync (one-way mirror)
Direction: Cloud → Hetzner Storage Box
Deletions: Files deleted from cloud are removed from backup
Transfers: 4 concurrent transfers
Checkers: 8 parallel file checkers

Manual Backup Commands¶

# Backup Google Drive manually
docker exec rclone rclone sync gdrive: hetzner:/backups/gdrive \
  --transfers 4 \
  --checkers 8 \
  --log-level INFO \
  --progress

# Backup OneDrive manually
docker exec rclone rclone sync onedrive: hetzner:/backups/onedrive \
  --transfers 4 \
  --checkers 8 \
  --log-level INFO \
  --progress

# Dry run to see what would be synced (no changes made)
docker exec rclone rclone sync gdrive: hetzner:/backups/gdrive \
  --dry-run \
  --log-level INFO

Storage Requirements¶

Google Drive: ~217GB (as of last check)
OneDrive: Variable based on usage
Growth Rate: Monitor monthly for capacity planning
Storage Box Capacity: 1TB available

Infrastructure Backups to R2¶

Overview¶

Automated daily backups of all Docker services and infrastructure components to Cloudflare R2 object storage, providing offsite disaster recovery capabilities with cost-effective storage. Enhanced with comprehensive monitoring and metrics collection.

Configuration¶

Script Location: /srv/dockerdata/rclone/backup-infrastructure-with-metrics.sh (wrapper with metrics)
Original Script: /srv/dockerdata/rclone/backup-infrastructure.sh (preserved)
Schedule: Daily at 4:00 AM via cron
Destination: Cloudflare R2 bucket (cloudflare:infrastructure-backups)
Retention: 14 days with automatic cleanup
Storage Usage: Currently using 6.7GB of 10TB R2 free tier
Monitoring: Prometheus metrics including R2 usage tracking

What's Included¶

PostgreSQL Database Dumps
Supabase PostgreSQL database (pg_dumpall)
Compressed with gzip for efficient storage
Includes all databases, roles, and permissions
Service Configurations
Docker compose files for all services
Environment files (.env) with sensitive data
Service-specific configuration directories
Custom scripts and automation tools
Traefik Certificates
Let's Encrypt SSL certificates
ACME account data
Certificate renewal metadata
Application Data
Gitea repositories and settings
Grafana dashboards and datasources
Prometheus configuration and rules
Paperless-AI application data and configurations
Authentik media files
Other service-specific data
Home Assistant (Added Nov 2025)
Configuration files via SMB mount
Excludes large database files
Excludes TTS cache and restore states
n8n Workflows (Added Nov 2025)
Exported workflows JSON
Exported credentials (encrypted)
Monitoring Data (Enhanced Nov 2025)
Grafana dashboards and provisioning
Prometheus rules and alert configurations

Backup Process¶

The backup script performs the following steps:

Creates timestamped backup directory
Dumps PostgreSQL database
Archives service configurations
Syncs to Cloudflare R2 with encryption
Cleans up local temporary files
Removes backups older than 14 days from R2
Updates Prometheus metrics for monitoring

Manual Backup Commands¶

# Run infrastructure backup manually (with metrics)
/srv/dockerdata/rclone/backup-infrastructure-with-metrics.sh

# Run original backup script (without metrics)
/srv/dockerdata/rclone/backup-infrastructure.sh

# Check backup metrics
curl -s localhost:9100/metrics | grep infrastructure_backup

# Check R2 backup status
docker exec rclone rclone ls cloudflare:infrastructure-backups

# View backup sizes
docker exec rclone rclone size cloudflare:infrastructure-backups --json | jq

# List recent backups with timestamps
docker exec rclone rclone lsl cloudflare:infrastructure-backups | sort -k2,3

# Download specific backup
docker exec rclone rclone copy \
  cloudflare:infrastructure-backups/backup-20250127-040000/ \
  /local/restore/path \
  --progress

# Check current R2 usage percentage
curl -s localhost:9100/metrics | grep r2_usage_percentage

R2 Storage Management¶

# Check total R2 usage
docker exec rclone rclone about cloudflare:

# Find large files in backups
docker exec rclone rclone ls cloudflare:infrastructure-backups \
  --min-size 100M

# Manually clean old backups (if needed)
docker exec rclone rclone delete cloudflare:infrastructure-backups \
  --min-age 14d \
  --dry-run  # Remove --dry-run to execute

Weekly Google Drive Archive (Added Nov 2025)¶

Overview¶

Weekly archival of R2 infrastructure backups to Google Drive for long-term offsite storage. This provides a second cloud copy with extended retention, implementing a true 3-2-1 backup strategy.

Configuration¶

Script: /srv/dockerdata/rclone/archive-to-gdrive.sh
Schedule: Weekly on Sunday at 6:00 AM
Source: cloudflare:infrastructure-backups
Destination: gdrive:server-backups/infrastructure
Retention: 60 days (vs 14 days on R2)

Benefits¶

3-2-1 Backup Strategy: 3 copies (local, R2, GDrive), 2 media types, 1 offsite
Extended Retention: 60 days vs 14 days on R2
Cost Effective: Uses existing Google Drive quota
Fast Recovery: R2 for quick restores, GDrive for disaster recovery

Manual Operations¶

# Run archive manually
/srv/dockerdata/rclone/archive-to-gdrive.sh

# Check archive size
docker exec rclone rclone size gdrive:server-backups/infrastructure

# List archived backups
docker exec rclone rclone ls gdrive:server-backups/infrastructure

# Restore from Google Drive archive
docker exec rclone rclone copy \
  gdrive:server-backups/infrastructure/postgres/20251122-040000/ \
  /local/restore/path \
  --progress

Local Service Backups¶

Supabase PostgreSQL¶

Automated Backup¶

Script: /srv/dockerdata/_scripts/supabase-backup.sh
Schedule: Daily at 3:00 AM (systemd timer)
Components:
Full database dump (pg_dumpall)
Configuration files (.env, docker-compose.yml)
Bootstrap SQL scripts
Retention: 14 days rolling

Manual Operations¶

# Run backup manually
/srv/dockerdata/_scripts/supabase-backup.sh

# Direct database dump
docker exec supa-db pg_dump -U postgres -d postgres > \
  backup-$(date +%Y%m%d%H%M%S).sql

Gitea¶

Automated Backup¶

Script: /srv/dockerdata/_scripts/gitea-backup.sh
Schedule: Daily at 2:00 AM (systemd timer)
Components:
Git repositories
LFS data
SQLite database
Configuration files
Method: Gitea's built-in gitea dump command

Pocketbase¶

Automated Backup¶

Script: /srv/dockerdata/_scripts/pocketbase-backup.sh
Schedule: Daily at 3:00 AM
Method: Container stop for SQLite consistency
Components:
SQLite database files
Uploaded files
Configuration

Monitoring and Verification¶

Enhanced Monitoring System¶

The backup monitoring system was completely redesigned to address the CloudBackupMissing alert issue and provide comprehensive backup observability.

Problem Solved¶

CloudBackupMissing Alert: Previously showed "20309d 1h 34m 52s ago" due to missing metrics
Solution: Implemented wrapper scripts that execute backups and update Prometheus metrics
Result: Real-time backup status tracking with accurate timestamps and comprehensive error reporting

Grafana Dashboard¶

Monitor all backup operations through the centralized Grafana dashboard: - URL: https://grafana.rbnk.uk - Dashboard: Infrastructure > Rclone Backup Monitoring - Metrics Update: Real-time via Prometheus textfile collector - Enhanced Panels: Backup status, error rates, duration trends, R2 usage

Prometheus Metrics Collection¶

Both cloud and infrastructure backup scripts now export comprehensive metrics:

Cloud Backup Metrics¶

# Check cloud backup metrics
curl -s localhost:9100/metrics | grep cloud_backup

# Available metrics:
# - cloud_backup_last_run_timestamp: Unix timestamp of last cloud backup
# - cloud_backup_success: 1 if successful, 0 if failed  
# - cloud_backup_duration_seconds: Time taken for cloud backup
# - cloud_backup_errors_total: Cumulative error count

Infrastructure Backup Metrics¶

# Check infrastructure backup metrics  
curl -s localhost:9100/metrics | grep infrastructure_backup

# Available metrics:
# - infrastructure_backup_last_run_timestamp: Unix timestamp of last backup
# - infrastructure_backup_success: 1 if successful, 0 if failed
# - infrastructure_backup_duration_seconds: Time taken for backup
# - infrastructure_backup_size_bytes: Total size of backup
# - infrastructure_backup_files_total: Number of files backed up
# - infrastructure_backup_r2_usage_bytes: Current R2 storage usage
# - infrastructure_backup_r2_usage_percentage: Percentage of R2 free tier used
# - infrastructure_backup_errors_total: Cumulative error count

Alert Rules¶

Comprehensive Prometheus alerts are configured for backup monitoring:

Critical Alerts: - CloudBackupMissing: Triggers if cloud backup hasn't succeeded in 48 hours - InfrastructureBackupMissing: Triggers if infrastructure backup hasn't succeeded in 48 hours
- CloudBackupFailed: Immediate alert on cloud backup failure - InfrastructureBackupFailed: Immediate alert on infrastructure backup failure

Warning Alerts: - BackupErrors: Alerts on error accumulation (>3 errors per 24h) - R2StorageHigh: Warns when R2 usage exceeds 80% of free tier - BackupDurationHigh: Performance alert for unusually long backup times

Notification Integration: - ntfy Topics: Real-time push notifications to mobile devices - Email Alerts: Critical backup failures sent to administrator - Grafana Annotations: Visual markers on dashboards for backup events

Backup Status Checks¶

Infrastructure Backup Status¶

# Check R2 backup status
docker exec rclone rclone ls cloudflare:infrastructure-backups

# View backup sizes and verify growth  
docker exec rclone rclone size cloudflare:infrastructure-backups --json | jq

# Check metrics for last backup status (NEW - enhanced metrics)
curl -s localhost:9100/metrics | grep infrastructure_backup

# View backup log
tail -50 /srv/dockerdata/rclone/backup-infrastructure.log

# Verify latest backup timestamp
docker exec rclone rclone lsl cloudflare:infrastructure-backups | \
  sort -k2,3 | tail -1

# Check backup success status (NEW)
curl -s localhost:9100/metrics | grep infrastructure_backup_success

# Monitor R2 usage (NEW)
curl -s localhost:9100/metrics | grep infrastructure_backup_r2_usage

Cloud Backup Monitoring¶

# Check latest backup log
tail -n 50 /srv/dockerdata/rclone/backup.log

# Check cloud backup metrics (NEW - enhanced monitoring)
curl -s localhost:9100/metrics | grep cloud_backup

# Check backup timestamps
ls -la /srv/dockerdata/rclone/backup.log

# Verify Hetzner storage usage
docker exec rclone rclone size hetzner:/backups

# List recent backups
docker exec rclone rclone lsl hetzner:/backups/gdrive --max-depth 1
docker exec rclone rclone lsl hetzner:/backups/onedrive --max-depth 1

# Check for backup errors in log
grep -i error /srv/dockerdata/rclone/backup.log | tail -20

# Check backup success status (NEW)
curl -s localhost:9100/metrics | grep cloud_backup_success

# Monitor backup duration trends (NEW)  
curl -s localhost:9100/metrics | grep cloud_backup_duration_seconds

Local Backup Monitoring¶

# Check backup directories
ls -la /srv/backups/*/

# Find backups older than 14 days (should be cleaned up)
find /srv/backups -name "*.tgz" -o -name "*.sql.gz" -mtime +14

# Check backup sizes for anomalies
du -sh /srv/backups/*

# Verify latest backup for each service
for service in supabase gitea pocketbase; do
  echo "Latest $service backup:"
  ls -t /srv/backups/$service/*/* | head -1
done

Integrity Verification¶

Cloud Backup Verification¶

# Compare file counts
echo "Google Drive file count:"
docker exec rclone rclone size gdrive: --json | jq .count

echo "Backup file count:"
docker exec rclone rclone size hetzner:/backups/gdrive --json | jq .count

# Check for sync differences
docker exec rclone rclone check gdrive: hetzner:/backups/gdrive \
  --one-way \
  --size-only

Local Backup Verification¶

# Test archive integrity
tar -tzf /srv/backups/gitea/app/gitea-dump-*.zip > /dev/null && \
  echo "Archive OK" || echo "Archive corrupted"

# Verify SQL dump
gunzip -t /srv/backups/supabase/db/*.sql.gz && \
  echo "SQL dump OK" || echo "SQL dump corrupted"

Restoration Procedures¶

Cloud Storage Restoration¶

# Restore specific files from Google Drive backup
docker exec rclone rclone copy \
  hetzner:/backups/gdrive/path/to/file \
  gdrive:/restore/path \
  --log-level INFO

# Full restoration (use with caution)
docker exec rclone rclone sync \
  hetzner:/backups/gdrive \
  gdrive-restore: \
  --dry-run  # Remove --dry-run to perform actual restore

Service Restoration¶

See the comprehensive Backup and Restore Operations Guide for detailed service restoration procedures.

Disaster Recovery¶

Priority Order¶

Traefik: Network routing and SSL certificates
Supabase PostgreSQL: Critical application data
Essential Services: Gitea (source code), Monitoring stack
Application Services: Paperless-AI, n8n workflows, LiteLLM, OpenWebUI
Utility Services: Watchtower, Metube, Glance dashboard, Rclone
Cloud Storage: User data from Google Drive/OneDrive

Detailed Recovery Priority¶

Tier 1 - Infrastructure (RTO: 30 minutes) - Traefik proxy and SSL certificates - Core networking and routing

Tier 2 - Critical Data (RTO: 1-2 hours) - Supabase PostgreSQL database - Gitea repositories and configuration

Tier 3 - Essential Services (RTO: 2-3 hours) - Monitoring stack (Prometheus, Grafana) - Pocketbase instances

Tier 4 - Application Services (RTO: 3-4 hours) - Paperless-AI application and data - n8n workflow automation - LiteLLM proxy - OpenWebUI interface

Tier 5 - Utility Services (RTO: 4-6 hours) - Watchtower auto-updates - Glance dashboard - Rclone cloud sync - Metube media downloads (optional)

Recovery Time Objectives (RTO)¶

Cloud Storage: 2-4 hours (depending on data size)
Database Services: 1-2 hours
Application Services: 30 minutes
Full System: 4-6 hours

Maintenance Tasks¶

Daily¶

Monitor backup logs for errors
Verify backup completion via timestamps

Weekly¶

Check storage utilization
Verify at least one backup integrity

Monthly¶

Test restoration procedure for one service
Review and update documentation
Check backup retention cleanup

Quarterly¶

Full disaster recovery drill
Review backup strategy effectiveness
Update RTO/RPO targets based on testing

Storage Management¶

Capacity Planning¶

# Check overall backup storage usage
df -h /srv/backups
df -h /srv/dockerdata/rclone

# Check Hetzner Storage Box usage
docker exec rclone rclone about hetzner:

# Check Cloudflare R2 usage
docker exec rclone rclone about cloudflare:

# Monitor R2 infrastructure backup growth
docker exec rclone rclone size cloudflare:infrastructure-backups --json | \
  jq '{total_size_gb: (.bytes / 1073741824), total_files: .count}'

# Project growth rate
# (Compare monthly snapshots of these commands)

R2 Free Tier Monitoring¶

Cloudflare R2 provides 10GB free storage. Monitor usage to stay within limits:

# Current R2 usage breakdown
docker exec rclone rclone size cloudflare: --json | \
  jq '{
    used_gb: (.bytes / 1073741824),
    free_tier_gb: 10,
    percentage_used: ((.bytes / 1073741824) / 10 * 100)
  }'

# List backup sizes by date
docker exec rclone rclone lsl cloudflare:infrastructure-backups | \
  awk '{print $2, $3, $1/1048576 " MB", $4}' | sort

Cleanup Procedures¶

# Manual cleanup of old cloud backups (if needed)
# Note: sync automatically manages this

# Clean old local backups beyond retention
find /srv/backups -name "*.tgz" -o -name "*.sql.gz" -mtime +14 -delete

# Clear old logs
find /srv/dockerdata/rclone -name "*.log" -mtime +30 -delete

# Optional: Clean large Metube downloads (manual review recommended)
# WARNING: Review files before deletion
find /srv/dockerdata/metube/downloads -name "*.mp4" -o -name "*.mkv" -mtime +90 -ls
# To delete (uncomment after review):
# find /srv/dockerdata/metube/downloads -name "*.mp4" -o -name "*.mkv" -mtime +90 -delete

Special Considerations¶

Metube Downloads Management¶

Metube downloads can consume significant disk space. Consider these strategies:

# Check current download directory size
du -sh /srv/dockerdata/metube/downloads/

# Find largest files
find /srv/dockerdata/metube/downloads -type f -exec ls -lhS {} + | head -20

# Archive old downloads to external storage
rsync -av --remove-source-files \
  /srv/dockerdata/metube/downloads/ \
  /external/storage/metube-archive/

# Set up automated cleanup of downloads older than 30 days (optional)
# Add to cron: 0 2 * * 0 find /srv/dockerdata/metube/downloads -name "*" -mtime +30 -delete

Security Considerations¶

Encryption in Transit: All Rclone transfers use HTTPS
Access Control: Rclone uses OAuth2 tokens stored securely in container
Hetzner Storage: Accessed via SFTP with key-based authentication
Local Backups: Restricted permissions (root:root 600)

Troubleshooting¶

Common Issues¶

Cloud Backup Failures¶

# Check Rclone configuration
docker exec rclone rclone config show

# Test cloud service connectivity
docker exec rclone rclone lsd gdrive:
docker exec rclone rclone lsd onedrive:

# Test Hetzner connectivity
docker exec rclone rclone lsd hetzner:

# Check for rate limiting
grep -i "rate" /srv/dockerdata/rclone/backup.log

Storage Space Issues¶

# Find large files in cloud backups
docker exec rclone rclone ls hetzner:/backups \
  --min-size 1G

# Clean up partial transfers
docker exec rclone rclone cleanup hetzner:

Backup Metrics Implementation - Technical details of the enhanced monitoring system
Backup Alerts Runbook - Step-by-step troubleshooting for backup alerts
Backup and Restore Operations - Detailed restoration procedures
Maintenance Guide - System maintenance procedures
Monitoring Guide - System monitoring and alerting

Recent Improvements (2025-08-09)¶

CloudBackupMissing Alert Resolution¶

The persistent CloudBackupMissing alert issue showing "20309d 1h 34m 52s ago" has been completely resolved through the implementation of comprehensive backup monitoring:

What Was Fixed: - Missing Prometheus metrics from backup operations - Inaccurate backup status tracking - Lack of real-time backup monitoring - No automated alerting for backup failures

Implementation: - Created wrapper scripts that execute backups and update Prometheus metrics - Implemented textfile collector integration for real-time metrics - Enhanced Grafana dashboards with backup status panels - Added comprehensive alert rules for backup failures and warnings - Integrated with notification system (ntfy, email) for immediate alerts

Benefits: - Real-time backup status visibility - Accurate timestamp tracking for all backup operations
- Comprehensive error monitoring and reporting - Automated alerting for backup issues - Enhanced troubleshooting capabilities through detailed metrics

For technical implementation details, see: Backup Metrics Implementation Guide

For alert troubleshooting procedures, see: Backup Alerts Runbook

Backup Strategy Guide¶

Overview¶

Backup Architecture¶

Two-Tier Backup System¶

Backup Classification and Priority¶

Critical - Automated Daily Backups¶

Important - Automated Daily Backups¶

Low Priority - Configuration Only¶

Optional - Manual/On-Demand¶

Backup Schedule¶

Cloud Storage Backups¶

Overview¶

Configuration¶

Backed Up Services¶

Sync Behavior¶

Manual Backup Commands¶

Storage Requirements¶

Infrastructure Backups to R2¶

Overview¶

Configuration¶

What's Included¶

Backup Process¶

Manual Backup Commands¶

R2 Storage Management¶

Weekly Google Drive Archive (Added Nov 2025)¶

Overview¶

Configuration¶

Benefits¶

Manual Operations¶

Local Service Backups¶

Supabase PostgreSQL¶

Automated Backup¶

Manual Operations¶

Gitea¶

Automated Backup¶

Pocketbase¶

Automated Backup¶

Monitoring and Verification¶

Enhanced Monitoring System¶

Problem Solved¶

Grafana Dashboard¶

Prometheus Metrics Collection¶

Cloud Backup Metrics¶

Infrastructure Backup Metrics¶

Alert Rules¶

Backup Status Checks¶

Infrastructure Backup Status¶

Cloud Backup Monitoring¶

Local Backup Monitoring¶

Integrity Verification¶

Cloud Backup Verification¶

Local Backup Verification¶

Restoration Procedures¶

Cloud Storage Restoration¶

Service Restoration¶

Disaster Recovery¶

Priority Order¶

Detailed Recovery Priority¶

Recovery Time Objectives (RTO)¶

Maintenance Tasks¶

Daily¶

Weekly¶

Monthly¶

Quarterly¶

Storage Management¶

Capacity Planning¶

R2 Free Tier Monitoring¶

Cleanup Procedures¶

Special Considerations¶

Metube Downloads Management¶

Security Considerations¶

Troubleshooting¶

Common Issues¶

Cloud Backup Failures¶

Storage Space Issues¶

Related Documentation¶

Recent Improvements (2025-08-09)¶

CloudBackupMissing Alert Resolution¶