Skip to content

Rclone Service

Overview

Rclone is a command line program and web service to manage files on cloud storage. It's used in this infrastructure as a critical backup solution, automatically syncing important data from the Hetzner server to cloud storage providers for disaster recovery.

Key Features

  • 70+ Cloud Storage Backends: Supports major providers like Google Drive, S3, Backblaze B2, Dropbox, OneDrive, and many more
  • Encrypted Transfers: All data transfers are encrypted in transit
  • Encrypted Storage: Optional client-side encryption before uploading to cloud storage
  • Bandwidth Control: Limit upload/download speeds to prevent network saturation
  • Scheduled Backups: Automated backup jobs via systemd timers
  • Web GUI: Browser-based interface for monitoring and management
  • Resume Support: Interrupted transfers can be resumed
  • Verification: MD5/SHA-1 hashes checked for data integrity

Architecture

Container Setup

rclone:
  image: rclone/rclone:latest
  container_name: rclone
  restart: unless-stopped
  volumes:
    - ./config:/config/rclone
    - ./data:/data
    - /srv/dockerdata:/backup:ro  # Read-only access to all Docker data
  environment:
    - RCLONE_CONFIG=/config/rclone/rclone.conf
  networks:
    - traefik_proxy
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.rclone.rule=Host(`rclone.rbnk.uk`)"
    - "traefik.http.services.rclone.loadbalancer.server.port=5572"
  command: rcd --rc-web-gui --rc-addr :5572 --rc-user admin --rc-pass ${RCLONE_PASSWORD}

Network Configuration

  • Network: Connected to traefik_proxy for web GUI access
  • Web GUI Port: 5572 (internal), exposed via Traefik
  • Access URL: https://rclone.rbnk.uk

Backup Strategy

What Gets Backed Up

  1. Critical Infrastructure Data:
  2. Supabase PostgreSQL dumps (including Authentik data)
  3. Gitea repositories and data
  4. Service configurations (.env files)

  5. Docker volume data

  6. Application Data:

  7. Pocketbase SQLite databases
  8. Paperless-AI Docker volume data
  9. n8n workflow and credential exports

  10. User uploads and media files

  11. System Configuration:

  12. Docker Compose files
  13. Traefik certificates
  14. Grafana dashboards and provisioning

  15. Prometheus rules and alert configurations

  16. External Systems (via SMB):

  17. Home Assistant configuration (excluding database)

Backup Schedule

The infrastructure runs three complementary backup schedules implementing a 3-2-1 strategy:

  1. Cloud-to-Hetzner Backups (2 AM daily):
  2. Syncs data from Google Drive and OneDrive to Hetzner storage
  3. Ensures cloud data is backed up locally

  4. Runs via /srv/dockerdata/rclone/backup-clouds.sh

  5. Infrastructure-to-R2 Backups (4 AM daily):

  6. Backs up all Docker services to Cloudflare R2
  7. Includes PostgreSQL, configs, certificates, Pocketbase data, Home Assistant, Grafana, Prometheus, and n8n workflows
  8. Runs via /srv/dockerdata/rclone/backup-infrastructure.sh

  9. 14-day retention policy with automatic cleanup

  10. Weekly Google Drive Archive (6 AM Sundays):

  11. Archives R2 backups to Google Drive for long-term offsite storage
  12. Provides second offsite copy for disaster recovery (3-2-1 backup strategy)
  13. Runs via /srv/dockerdata/rclone/archive-to-gdrive.sh
  14. 60-day retention on Google Drive

Automated via systemd timers:

# Cloud backup timer (2 AM)
# /etc/systemd/system/rclone-backup.timer
[Unit]
Description=Run Rclone cloud backup daily at 2 AM
Requires=rclone-backup.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

# Infrastructure backup timer (4 AM)
# /etc/systemd/system/rclone-infrastructure-backup.timer
[Unit]
Description=Run Rclone infrastructure backup daily at 4 AM
Requires=rclone-infrastructure-backup.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 04:00:00
Persistent=true

[Install]
WantedBy=timers.target

# Weekly Google Drive archive (6 AM Sundays)
# Cron job in root crontab:
# 0 6 * * 0 /srv/dockerdata/rclone/archive-to-gdrive.sh >> /srv/dockerdata/rclone/archive-gdrive.log 2>&1

Backup Script Example

#!/bin/bash
# /srv/dockerdata/rclone/scripts/backup.sh

# Backup PostgreSQL
docker exec supa-db pg_dumpall -U postgres > /tmp/postgres-backup.sql
docker exec rclone rclone copy /tmp/postgres-backup.sql remote:backups/postgres/$(date +%Y%m%d)/ --progress

# Backup Gitea
docker exec rclone rclone sync /backup/gitea/data remote:backups/gitea/ --progress

# Backup all Docker volumes (incremental)
docker exec rclone rclone sync /backup remote:backups/dockerdata/ \
  --exclude "*.log" \
  --exclude "*/cache/*" \
  --exclude "*/temp/*" \
  --progress

Infrastructure Backup

The infrastructure backup system provides comprehensive disaster recovery by backing up all critical Docker services to Cloudflare R2 storage.

Overview

The backup script at /srv/dockerdata/rclone/backup-infrastructure.sh performs daily backups of:

  1. PostgreSQL Databases:
  2. Full database dumps from Supabase
  3. All schemas, users, and permissions

  4. Compressed with gzip for efficiency

  5. Service Configurations:

  6. All .env files across services
  7. Docker Compose configurations

  8. Nginx/Traefik routing rules

  9. SSL Certificates:

  10. Let's Encrypt certificates from Traefik
  11. Certificate renewal information

  12. ACME account data

  13. Pocketbase Applications:

  14. SQLite databases for all Pocketbase stacks
  15. User authentication data
  16. Application-specific configurations

Backup Destination

All infrastructure backups are stored in Cloudflare R2: - Bucket: infrastructure-backups - Structure: YYYY-MM-DD/service-name/ - Retention: 14 days (automatic cleanup of older backups) - Encryption: Data encrypted in transit and at rest

Systemd Integration

The infrastructure backup runs automatically via systemd:

# Service unit
# /etc/systemd/system/rclone-infrastructure-backup.service
[Unit]
Description=Backup infrastructure to Cloudflare R2
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/srv/dockerdata/rclone/backup-infrastructure.sh
StandardOutput=journal
StandardError=journal
SyslogIdentifier=rclone-infrastructure-backup
User=root

# Timer unit (4 AM daily)
# /etc/systemd/system/rclone-infrastructure-backup.timer
[Unit]
Description=Run infrastructure backup daily at 4 AM
Requires=rclone-infrastructure-backup.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 04:00:00
Persistent=true

[Install]
WantedBy=timers.target

Manual Operations

# Run infrastructure backup manually
/srv/dockerdata/rclone/backup-infrastructure.sh

# Check backup status in R2
docker exec rclone rclone ls cloudflare:infrastructure-backups

# List today's backups
docker exec rclone rclone ls cloudflare:infrastructure-backups/$(date +%Y-%m-%d)/

# Verify backup integrity
docker exec rclone rclone check /srv/dockerdata cloudflare:infrastructure-backups/latest/ --one-way

# Download specific backup
docker exec rclone rclone copy cloudflare:infrastructure-backups/2024-03-15/postgres/ /restore/postgres/

Weekly Google Drive Archive

The weekly archive provides a second offsite copy of infrastructure backups, implementing the 3-2-1 backup strategy (3 copies, 2 media types, 1 offsite).

Overview

The archive script at /srv/dockerdata/rclone/archive-to-gdrive.sh syncs R2 backups to Google Drive every Sunday at 6 AM:

  • Source: cloudflare:infrastructure-backups
  • Destination: gdrive:server-backups/infrastructure
  • Schedule: Weekly on Sundays at 6 AM (cron)
  • Retention: 60 days on Google Drive

Benefits

  1. Geographic Redundancy: Google Drive provides a separate offsite location from R2
  2. Provider Diversity: Different cloud provider reduces single-vendor risk
  3. Long-term Retention: 60 days vs 14 days on R2
  4. Cost Optimization: Uses Google Drive's 100GB free tier efficiently

Manual Operations

# Run archive manually
/srv/dockerdata/rclone/archive-to-gdrive.sh

# Check Google Drive archive size
docker exec rclone rclone size gdrive:server-backups/infrastructure

# List archived backups
docker exec rclone rclone ls gdrive:server-backups/infrastructure

# View archive logs
tail -f /srv/dockerdata/rclone/archive-gdrive.log

# Restore from Google Drive (if R2 unavailable)
docker exec rclone rclone copy gdrive:server-backups/infrastructure/postgres/20241115/ /restore/postgres/

# Check archive metrics
cat /srv/dockerdata/monitoring/prometheus/textfile_collector/rclone_metrics.prom | grep gdrive

Monitoring

The archive script exports metrics to Prometheus: - rclone_gdrive_archive_last_success_timestamp: Last successful archive time - rclone_gdrive_archive_size_bytes: Total size of Google Drive archive

Cloud Storage Configuration

Setting Up a Remote

  1. Enter Rclone container: 

    docker exec -it rclone sh

  2. Configure remote storage: 

    rclone config
# Follow interactive setup for your provider

  3. Common Remote Types:

Backblaze B2 (recommended for cost-effectiveness): 

Type: b2
Account ID: <your-account-id>
Application Key: <your-app-key>

Google Drive: 

Type: drive
Client ID: <leave blank for default>
Client Secret: <leave blank for default>
Scope: drive

S3-Compatible (for Supabase Storage backup): 

Type: s3
Provider: Other
Access Key ID: <your-key>
Secret Access Key: <your-secret>
Endpoint: <your-endpoint>

Encryption Setup

For sensitive data, enable client-side encryption:

# In rclone config
Name: secure-remote
Type: crypt
Remote: remote:encrypted  # Points to folder on existing remote
Password: <strong-password>
Password2: <salt-password>  # Optional

Operations

Manual Backup

# Backup specific directory
docker exec rclone rclone copy /backup/important-data remote:backups/manual/ --progress

# Sync with deletion (mirror)
docker exec rclone rclone sync /backup/gitea remote:backups/gitea/ --progress

# Dry run to see what would be copied
docker exec rclone rclone copy /backup/data remote:backups/ --dry-run

# Run infrastructure backup manually
/srv/dockerdata/rclone/backup-infrastructure.sh

# Check backup status in R2
docker exec rclone rclone ls cloudflare:infrastructure-backups

# View backup metrics
curl -s localhost:9100/metrics | grep rclone_

Restore Operations

# List available backups
docker exec rclone rclone ls remote:backups/

# Restore specific backup
docker exec rclone rclone copy remote:backups/postgres/20240315/ /restore/postgres/

# Restore with specific date filter
docker exec rclone rclone copy remote:backups/dockerdata/ /restore/ \
  --max-age 7d  # Files modified within last 7 days

Monitoring

  1. Web GUI Access:
  2. URL: https://rclone.rbnk.uk
  3. Username: admin

  4. Password: Set in .env file

  5. Prometheus Metrics:

The infrastructure includes comprehensive backup monitoring via Prometheus:

  • Metrics Collection: /srv/dockerdata/rclone/collect-metrics.sh
  • Metrics Endpoint: localhost:9100/metrics (node_exporter)
  • Grafana Dashboard: Infrastructure > Rclone Backup Monitoring

Key Metrics Collected: 

# Backup timestamps
rclone_backup_last_run_timestamp    # When backup last started
rclone_backup_last_success_timestamp # When backup last succeeded

# Backup status
rclone_backup_last_run_status       # 1 = success, 0 = failure
rclone_backup_errors_total          # Total error count

# Storage metrics
rclone_backup_size_bytes            # Total backup size
rclone_backup_files_total           # Number of files backed up

  1. Check Transfer Status: 

    # View current transfers
docker exec rclone rclone rc core/stats

# Check remote storage usage
docker exec rclone rclone size remote:backups/

# Check R2 infrastructure backups
docker exec rclone rclone size cloudflare:infrastructure-backups

  2. View Logs: 

    # Rclone service logs
docker compose -f rclone/docker-compose.yml logs -f

# Infrastructure backup logs
journalctl -u rclone-infrastructure-backup -f

# Cloud backup logs  
journalctl -u rclone-backup -f

  3. Alert Rules:

The monitoring stack includes automated alerts for backup failures:

# Infrastructure backup failure (24 hours)
- alert: RcloneInfrastructureBackupFailed
  expr: time() - rclone_backup_last_success_timestamp{job="infrastructure"} > 86400
  annotations:
    summary: "Infrastructure backup to R2 hasn't succeeded in 24 hours"

# Cloud backup failure (48 hours)
- alert: RcloneCloudBackupFailed
  expr: time() - rclone_backup_last_success_timestamp{job="cloud"} > 172800
  annotations:
    summary: "Cloud backup hasn't succeeded in 48 hours"

# Backup errors detected
- alert: RcloneBackupErrors
  expr: increase(rclone_backup_errors_total[1h]) > 0
  annotations:
    summary: "Rclone backup errors detected in the last hour"

Performance Optimization

Bandwidth Limiting

Prevent backup jobs from saturating network:

# Limit to 10 MB/s
docker exec rclone rclone copy /backup remote:backups/ --bwlimit 10M

# Time-based limits (business hours vs off-hours)
docker exec rclone rclone copy /backup remote:backups/ \
  --bwlimit "08:00,2M 18:00,10M 23:00,1M"

Transfer Optimization

# Parallel transfers for small files
docker exec rclone rclone copy /backup remote:backups/ \
  --transfers 32 \
  --checkers 16

# Large file optimization
docker exec rclone rclone copy /backup remote:backups/ \
  --s3-chunk-size 128M \
  --s3-upload-concurrency 10

Security Considerations

  1. Access Control:
  2. Web GUI protected by username/password
  3. Accessible only through Traefik with HTTPS

  4. Consider adding Authelia for 2FA

  5. Data Protection:

  6. Use client-side encryption for sensitive data
  7. Rotate cloud storage API keys regularly

  8. Store credentials in .env file with proper permissions

  9. Network Security:

  10. All transfers use TLS/SSL
  11. Web GUI should not be exposed without authentication
  12. Consider IP whitelisting for admin access

Disaster Recovery

Recovery Priority

  1. Tier 1 - Critical (Restore first):
  2. PostgreSQL databases
  3. Authentication configurations

  4. SSL certificates

  5. Tier 2 - Important:

  6. Git repositories
  7. User uploads

  8. Application configurations

  9. Tier 3 - Replaceable:

  10. Logs
  11. Caches
  12. Temporary files

Full Recovery Process

# 1. Restore PostgreSQL
docker exec rclone rclone copy remote:backups/postgres/latest/ /restore/postgres/
docker exec -i supa-db psql -U postgres < /restore/postgres/dump.sql

# 2. Restore Gitea
docker compose -f gitea/docker-compose.yml down
docker exec rclone rclone sync remote:backups/gitea/ /srv/dockerdata/gitea/data/
docker compose -f gitea/docker-compose.yml up -d

# 3. Restore configurations
docker exec rclone rclone copy remote:backups/configs/ /srv/dockerdata/ --include "*.env"

Monitoring Integration

Prometheus Metrics

The infrastructure uses a custom metrics collection approach via shell scripts that export metrics to the node_exporter textfile collector:

# Metrics collection script
/srv/dockerdata/rclone/collect-metrics.sh

# Runs after each backup to update metrics
# Exports to: /var/lib/node_exporter/textfile_collector/rclone_backup.prom

Metrics Exported: - rclone_backup_last_run_timestamp: Unix timestamp of last backup attempt - rclone_backup_last_success_timestamp: Unix timestamp of last successful backup - rclone_backup_last_run_status: Status of last run (1=success, 0=failure) - rclone_backup_errors_total: Cumulative error count - rclone_backup_size_bytes: Total size of backed up data - rclone_backup_files_total: Number of files in backup

Grafana Dashboard

The Rclone Backup Monitoring dashboard provides: - Backup success/failure timeline - Time since last successful backup - Backup size trends - Error rate graphs - Storage usage by destination

Access via: Grafana > Infrastructure > Rclone Backup Monitoring

Alerting Rules

# In Prometheus alerts
- alert: RcloneInfrastructureBackupFailed
  expr: time() - rclone_backup_last_success_timestamp{job="infrastructure"} > 86400
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Infrastructure backup to R2 hasn't succeeded in 24 hours"
    description: "The daily infrastructure backup to Cloudflare R2 has failed. Check logs: journalctl -u rclone-infrastructure-backup"

- alert: RcloneCloudBackupFailed
  expr: time() - rclone_backup_last_success_timestamp{job="cloud"} > 172800
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Cloud sync backup hasn't succeeded in 48 hours"
    description: "The cloud-to-Hetzner backup sync has failed. Check logs: journalctl -u rclone-backup"

- alert: RcloneBackupErrors
  expr: increase(rclone_backup_errors_total[1h]) > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Rclone backup errors detected"
    description: "{{ $value }} errors occurred during backup operations in the last hour"

- alert: CloudStorageFull
  expr: rclone_backup_size_bytes / 1073741824 > 900  # 900 GB threshold
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Backup storage approaching limit"
    description: "Backup storage usage is {{ $value | humanize }}B. Consider cleanup or storage expansion."

Troubleshooting

Common Issues

  1. Authentication Failures: 

    # Re-authenticate with cloud provider
docker exec -it rclone rclone config reconnect remote:

  2. Slow Transfers:

  3. Check bandwidth limits
  4. Verify network connectivity

  5. Consider using different cloud region

  6. Failed Backups: 

    # Check detailed logs
docker exec rclone rclone copy /backup remote:backups/ -vv

# Test connectivity
docker exec rclone rclone lsd remote:

Debug Commands

# Test remote configuration
docker exec rclone rclone config show remote

# Check for errors in specific backup
docker exec rclone rclone check /backup remote:backups/ --one-way

# View transfer statistics
docker exec rclone rclone rc core/stats --json

Best Practices

  1. Regular Testing:
  2. Perform monthly restore tests
  3. Verify backup integrity

  4. Document recovery procedures

  5. Multiple Destinations:

  6. Use 3-2-1 backup rule
  7. Configure multiple cloud remotes

  8. Consider different geographic regions

  9. Automation:

  10. Use systemd timers for scheduling
  11. Implement health checks

  12. Set up alerting for failures

  13. Cost Management:

  14. Use lifecycle policies for old backups
  15. Compress before uploading
  16. Choose appropriate storage classes

Integration with Infrastructure

Rclone integrates seamlessly with the existing infrastructure:

  • Traefik: Provides HTTPS access to web GUI
  • Monitoring: Exports metrics to Prometheus
  • Docker Volumes: Read-only access to all container data
  • Systemd: Automated scheduling and service management
  • Alerting: Integrated with AlertManager for backup failures

This makes Rclone a critical component of the disaster recovery strategy, ensuring that all important data is safely backed up to cloud storage and can be quickly restored when needed.