Backup Metrics Implementation Guide¶

Overview¶

This document details the implementation of comprehensive backup monitoring metrics that resolved the persistent CloudBackupMissing alert issue and provides robust backup status tracking through Prometheus.

The Problem¶

The CloudBackupMissing alert was continuously showing "20309d 1h 34m 52s ago" due to missing metrics from backup operations. The original backup scripts ran independently without integrating with the monitoring system, leaving Prometheus without visibility into backup status.

Symptoms¶

CloudBackupMissing alert perpetually active with unrealistic timestamps
No visibility into backup success/failure rates
Missing backup duration and size metrics
No automated alerting for backup failures
Unable to track backup trends over time

The Solution¶

Created wrapper scripts that execute backups and simultaneously update Prometheus metrics using the textfile collector mechanism. This provides comprehensive backup observability while maintaining existing backup functionality.

Architecture Overview¶

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Cron Job      │───▶│  Wrapper Script  │───▶│ Original Backup │
│ (scheduled)     │    │                  │    │    Script       │
└─────────────────┘    └──────────┬───────┘    └─────────────────┘
                                  │
                                  ▼
                       ┌──────────────────┐
                       │ Prometheus       │
                       │ Metrics Update   │
                       │ (textfile        │
                       │  collector)      │
                       └──────────────────┘
                                  │
                                  ▼
                       ┌──────────────────┐    ┌─────────────────┐
                       │   Prometheus     │───▶│    Grafana      │
                       │   (scrapes)      │    │  (dashboards)   │
                       └──────────────────┘    └─────────────────┘
                                  │
                                  ▼
                       ┌──────────────────┐    ┌─────────────────┐
                       │  Alert Manager   │───▶│  Notifications  │
                       │   (rules)        │    │   (ntfy/email)  │
                       └──────────────────┘    └─────────────────┘

Implementation Details¶

Wrapper Scripts Created¶

1. Cloud Backup Wrapper (`/srv/dockerdata/rclone/backup-clouds-with-metrics.sh`)¶

Purpose: Wraps the original cloud backup script with metrics collection.

Key Features: - Captures start and end timestamps - Tracks backup success/failure status - Measures backup duration - Updates Prometheus metrics via textfile collector - Preserves all original backup functionality

Metrics Generated:

# HELP cloud_backup_last_run_timestamp Unix timestamp of last cloud backup run
# TYPE cloud_backup_last_run_timestamp gauge
cloud_backup_last_run_timestamp 1643723400

# HELP cloud_backup_success Success status of last cloud backup (1=success, 0=failure)
# TYPE cloud_backup_success gauge
cloud_backup_success 1

# HELP cloud_backup_duration_seconds Duration of cloud backup operation
# TYPE cloud_backup_duration_seconds gauge
cloud_backup_duration_seconds 3654

# HELP cloud_backup_errors_total Total errors encountered during cloud backup
# TYPE cloud_backup_errors_total counter
cloud_backup_errors_total 0

2. Infrastructure Backup Wrapper (`/srv/dockerdata/rclone/backup-infrastructure-with-metrics.sh`)¶

Purpose: Wraps the infrastructure backup script with comprehensive metrics.

Additional Metrics:

# R2 storage usage metrics
infrastructure_backup_r2_usage_bytes 7200000000
infrastructure_backup_r2_usage_percentage 72

# Backup size tracking
infrastructure_backup_size_bytes 856000000
infrastructure_backup_files_total 15432

Textfile Collector Integration¶

The metrics are written to files that Prometheus node exporter automatically scrapes:

File Locations: - Cloud backup metrics: /var/lib/node_exporter/textfile_collector/cloud_backup.prom - Infrastructure backup metrics: /var/lib/node_exporter/textfile_collector/infrastructure_backup.prom

Update Process: 1. Wrapper script creates temporary .prom.tmp file 2. Writes metrics in Prometheus format 3. Atomically moves to final .prom file (ensures consistency) 4. Node exporter scrapes on next collection cycle (15 seconds)

Cron Job Updates¶

Updated cron jobs to use wrapper scripts instead of direct backup scripts:

# Original cron entries (replaced)
# 0 2 * * * /srv/dockerdata/rclone/backup-clouds.sh

# New cron entries with metrics
0 2 * * * /srv/dockerdata/rclone/backup-clouds-with-metrics.sh
0 4 * * * /srv/dockerdata/rclone/backup-infrastructure-with-metrics.sh

Metrics Collection Details¶

Timestamp Handling¶

Challenge: Ensuring accurate timestamp collection across different execution contexts.

Solution: Capture timestamps at script start and use consistent Unix epoch format:

start_time=$(date +%s)
# ... backup operations ...
end_time=$(date +%s)
duration=$((end_time - start_time))

# Write metrics
echo "cloud_backup_last_run_timestamp $end_time" > metrics.prom

Error Detection¶

Challenge: Reliably detecting backup failures across different backup tools.

Solution: Multi-level error checking:

# 1. Check script exit code
if ! /srv/dockerdata/rclone/backup-clouds.sh; then
    success=0
    error_count=$((error_count + 1))
else
    success=1
fi

# 2. Check log file for errors
if grep -i error /srv/dockerdata/rclone/backup.log | tail -10 | grep -q "$(date +%Y-%m-%d)"; then
    error_count=$((error_count + 1))
fi

# 3. Verify expected output files exist
if [ ! -f "/expected/backup/file" ]; then
    success=0
    error_count=$((error_count + 1))
fi

R2 Usage Monitoring¶

Special handling for Cloudflare R2 usage tracking:

# Get R2 usage information
r2_info=$(docker exec rclone rclone about cloudflare: 2>/dev/null | grep -E "Total|Used")
r2_used_bytes=$(echo "$r2_info" | grep "Used" | awk '{print $2}' | sed 's/[^0-9]//g')
r2_total_bytes=10737418240  # 10GB free tier limit

# Calculate percentage
r2_percentage=$(( (r2_used_bytes * 100) / r2_total_bytes ))

# Write metrics
echo "infrastructure_backup_r2_usage_bytes $r2_used_bytes" >> metrics.prom
echo "infrastructure_backup_r2_usage_percentage $r2_percentage" >> metrics.prom

Integration with Notification System¶

Alert Rule Configuration¶

Created specific alert rules in Prometheus for backup monitoring:

# Cloud backup alerts
- alert: CloudBackupMissing
  expr: time() - cloud_backup_last_run_timestamp > 172800  # 48 hours
  for: 5m
  labels:
    severity: critical
    service: backup
    category: cloud
  annotations:
    summary: "Cloud backup missing for {{ $value }}s"
    description: "Cloud backup hasn't run successfully for {{ humanizeDuration $value }}"

- alert: CloudBackupFailed
  expr: cloud_backup_success == 0
  for: 0m
  labels:
    severity: critical
    service: backup
    category: cloud
  annotations:
    summary: "Cloud backup failed"
    description: "Last cloud backup attempt failed. Check logs for details."

# Infrastructure backup alerts
- alert: InfrastructureBackupMissing
  expr: time() - infrastructure_backup_last_run_timestamp > 172800  # 48 hours
  for: 5m
  labels:
    severity: critical
    service: backup
    category: infrastructure
  annotations:
    summary: "Infrastructure backup missing for {{ $value }}s"
    description: "Infrastructure backup hasn't run successfully for {{ humanizeDuration $value }}"

- alert: R2StorageHigh
  expr: infrastructure_backup_r2_usage_percentage > 80
  for: 5m
  labels:
    severity: warning
    service: backup
    category: storage
  annotations:
    summary: "R2 storage usage high: {{ $value }}%"
    description: "Cloudflare R2 storage usage is {{ $value }}% of free tier limit"

Notification Channels¶

Alerts route through multiple notification channels:

ntfy Topics: Immediate push notifications
backup-critical-failure for critical backup failures
backup-warning-storage for storage warnings
Email: Daily digest and critical alerts
Sends to administrator email configured in AlertManager
Grafana Annotations: Visual indicators on dashboards
Backup start/end events
Failure markers
Storage threshold crossings

Dashboard Updates¶

Enhanced Grafana Dashboard¶

The Rclone Backup Monitoring dashboard now includes:

Panels Added: 1. Backup Status Overview: Current status of all backup jobs 2. Last Successful Backup: Time since last successful backup 3. Backup Duration Trends: Historical backup performance 4. Error Rate: Backup failure percentage over time 5. R2 Storage Usage: Current and trending storage consumption 6. Alert Status: Active backup-related alerts

Panel Queries:

# Backup recency
(time() - cloud_backup_last_run_timestamp) / 3600

# Success rate over 7 days
rate(cloud_backup_success[7d]) * 100

# R2 usage trend
infrastructure_backup_r2_usage_percentage

# Backup duration moving average
avg_over_time(cloud_backup_duration_seconds[24h]) / 3600

Alert Panel Integration¶

Added alert status panel showing: - Currently firing backup alerts - Alert severity and duration - Recent alert history - Acknowledgment status

Verification and Testing¶

Manual Verification Commands¶

Check Metrics Collection:

# Verify metrics files exist and have recent timestamps
ls -la /var/lib/node_exporter/textfile_collector/*.prom

# Check metric content
cat /var/lib/node_exporter/textfile_collector/cloud_backup.prom
cat /var/lib/node_exporter/textfile_collector/infrastructure_backup.prom

# Verify Prometheus is scraping metrics
curl -s localhost:9100/metrics | grep -E "(cloud_backup|infrastructure_backup)"

Test Alert Rules:

# Check if alerts are loaded
curl -s localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name=="backup")'

# View active backup alerts
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.service=="backup")'

# Test alert firing conditions
curl -s 'localhost:9090/api/v1/query?query=time()-cloud_backup_last_run_timestamp' | jq

Validate Backup Execution:

# Run wrapper script manually
/srv/dockerdata/rclone/backup-clouds-with-metrics.sh

# Check updated metrics
cat /var/lib/node_exporter/textfile_collector/cloud_backup.prom

# Verify log updates
tail -20 /srv/dockerdata/rclone/backup.log

Automated Testing¶

Created test script for comprehensive validation:

#!/bin/bash
# /srv/dockerdata/_scripts/test-backup-metrics.sh

echo "Testing backup metrics implementation..."

# Test 1: Check wrapper scripts exist and are executable
echo "Checking wrapper scripts..."
if [ -x "/srv/dockerdata/rclone/backup-clouds-with-metrics.sh" ]; then
    echo "✓ Cloud backup wrapper found"
else
    echo "✗ Cloud backup wrapper missing or not executable"
fi

# Test 2: Verify textfile collector directory
echo "Checking textfile collector setup..."
if [ -d "/var/lib/node_exporter/textfile_collector" ]; then
    echo "✓ Textfile collector directory exists"
else
    echo "✗ Textfile collector directory missing"
fi

# Test 3: Check current metrics
echo "Checking current metrics..."
if [ -f "/var/lib/node_exporter/textfile_collector/cloud_backup.prom" ]; then
    age=$(stat -c %Y "/var/lib/node_exporter/textfile_collector/cloud_backup.prom")
    current=$(date +%s)
    hours_old=$(( (current - age) / 3600 ))
    echo "✓ Cloud backup metrics exist (${hours_old}h old)"
else
    echo "✗ Cloud backup metrics file missing"
fi

# Test 4: Validate metric format
echo "Validating metric format..."
if grep -q "^cloud_backup_last_run_timestamp [0-9]\+$" /var/lib/node_exporter/textfile_collector/cloud_backup.prom 2>/dev/null; then
    echo "✓ Metrics format valid"
else
    echo "✗ Invalid metrics format"
fi

# Test 5: Check Prometheus scraping
echo "Checking Prometheus scraping..."
if curl -s localhost:9100/metrics | grep -q "cloud_backup_last_run_timestamp"; then
    echo "✓ Prometheus is scraping backup metrics"
else
    echo "✗ Prometheus not scraping backup metrics"
fi

echo "Backup metrics test complete."

Performance Impact¶

Resource Usage¶

Storage: Minimal impact - Each metrics file: <1KB - Total additional storage: <10KB

CPU: Negligible overhead - Additional processing: ~0.1% of backup runtime - Prometheus scraping: ~0.01% CPU per scrape

Network: No additional network usage - Metrics collected locally via filesystem

Memory: Minimal impact - Additional metrics in Prometheus: ~50KB total

Backup Performance¶

Timing Analysis: - Original backup script runtime: Unchanged - Wrapper script overhead: <5 seconds - Total impact: <0.1% of backup duration

No Functional Changes: - All original backup functionality preserved - Same backup files created - Same error handling maintained - Same log file updates

Troubleshooting Guide¶

Common Issues¶

1. Metrics Not Updating

# Check wrapper script permissions
ls -la /srv/dockerdata/rclone/backup-*-with-metrics.sh

# Verify textfile collector permissions
ls -la /var/lib/node_exporter/textfile_collector/

# Check for script errors
journalctl -u cron | grep backup

2. Prometheus Not Scraping

# Verify node exporter configuration
docker logs monitoring_node-exporter_1 | grep textfile

# Check metrics endpoint
curl localhost:9100/metrics | head -20

3. Incorrect Metric Values

# Check timestamp format
date +%s  # Should match metric timestamp

# Verify calculation logic
tail -20 /srv/dockerdata/rclone/backup.log
cat /var/lib/node_exporter/textfile_collector/cloud_backup.prom

Recovery Procedures¶

If Metrics Stop Updating: 1. Check cron job status: systemctl status cron 2. Verify wrapper script exists and is executable 3. Run wrapper script manually to test 4. Check textfile collector directory permissions 5. Restart node exporter if needed

If Alerts Keep Firing: 1. Verify backup actually completed successfully 2. Check metric file timestamps and content 3. Confirm Prometheus is scraping updated metrics 4. Review alert rule thresholds

Future Enhancements¶

Planned Improvements¶

Advanced Error Classification
Categorize errors by type (network, storage, authentication)
Track error patterns over time
Implement automatic retry logic for transient failures
Performance Metrics
Bandwidth utilization during backups
File transfer rates
Backup efficiency metrics (deduplication ratios)
Predictive Alerting
Trend analysis for storage growth
Backup duration anomaly detection
Capacity planning alerts
Enhanced Dashboards
Backup calendar view showing success/failure patterns
Comparative analysis between backup types
Cost analysis dashboard for cloud storage usage

Integration Opportunities¶

n8n Workflow Integration
Automated backup health reports
Intelligent retry logic
Backup scheduling optimization
StackWiz MCP Integration
Natural language backup management
AI-powered backup troubleshooting
Automated backup strategy recommendations

Backup Strategy Guide - Overall backup architecture
Backup Alerts Runbook - Alert troubleshooting procedures
Grafana Dashboard Configuration - Dashboard setup and customization
Prometheus Configuration - Metrics collection and alert rules