Monitoring and Observability Guide

Current State Analysis (Updated 2025-01-02)

Monitoring Stack Status: ✅ OPERATIONAL

The monitoring stack has been successfully deployed and is fully operational with the following components:

1. ✅ Centralized Metrics Collection - Prometheus collecting from all exporters
2. ✅ Log Aggregation - Loki collecting and indexing container logs
3. ✅ Alerting System - AlertManager routing alerts to n8n workflows
4. ✅ Comprehensive Health Checks - Blackbox exporter monitoring all services
5. ✅ Performance Metrics - Complete system and container resource tracking
6. ✅ Application Metrics - HTTP metrics, database stats, backup monitoring
7. ✅ Visualization - Grafana dashboards for all major services and infrastructure

Recent Fixes Applied

Permission Issues Resolved: - Created /srv/dockerdata/monitoring/fix-permissions.sh script - Fixed ownership for Prometheus (user 65534), Grafana (user 472), Loki (user 10001), AlertManager (user 65534) - Disabled Watchtower auto-updates for monitoring stack to prevent permission issues

AlertManager Configuration Stabilized: - Resolved "time interval crossing midnight" errors - Simplified configuration with basic webhook routing to n8n - Removed complex time-based routing that was causing startup failures - Maintained full functionality through n8n workflow integration

Monitoring Stack Stability: - All services running consistently without restarts - Metrics collection stable across all exporters - Alert delivery functioning through n8n webhook integration - Dashboard performance optimized

Permission Fix Script

A critical fix has been implemented to address permission issues that were preventing the monitoring stack from starting properly.

Script Location

/srv/dockerdata/monitoring/fix-permissions.sh

Usage

# Run the script if monitoring services fail to start with permission errors
/srv/dockerdata/monitoring/fix-permissions.sh

What It Fixes

The script corrects ownership and permissions for all monitoring service data directories:

• Prometheus: Sets ownership to user 65534 (nobody)
• Grafana: Sets ownership to user 472 (grafana)
• Loki: Sets ownership to user 10001 (loki)
• AlertManager: Sets ownership to user 65534 (nobody)

When to Use

• After system updates that might change Docker volume permissions
• When adding the monitoring stack to a new environment
• If any monitoring service fails to start with "permission denied" errors
• After manually manipulating files in monitoring data directories

Prevention Measures

• Watchtower Exclusion: All monitoring containers have com.centurylinklabs.watchtower.enable=false labels
• Regular Backups: Permission fix script is included in infrastructure backups
• Documentation: Troubleshooting steps documented in service-specific guides

Implementing Prometheus + Grafana Stack

1. Create Monitoring Stack

# Create monitoring directory
mkdir -p /srv/dockerdata/monitoring/{prometheus,grafana,alertmanager}

# Create docker-compose.yml
cat > /srv/dockerdata/monitoring/docker-compose.yml << 'EOF'
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/alerts.yml:/etc/prometheus/alerts.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    networks:
      - traefik_proxy
      - monitoring
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_proxy"
      - "traefik.http.routers.prometheus.rule=Host(`prometheus.rbnk.uk`)"
      - "traefik.http.routers.prometheus.entrypoints=websecure"
      - "traefik.http.routers.prometheus.tls=true"
      - "traefik.http.routers.prometheus.tls.certresolver=cloudflare"
      - "traefik.http.services.prometheus.loadbalancer.server.port=9090"
      - "traefik.http.routers.prometheus.middlewares=auth@file"

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=${GRAFANA_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_ROOT_URL=https://grafana.rbnk.uk
      - GF_SMTP_ENABLED=true
      - GF_SMTP_HOST=${SMTP_HOST:-smtp.gmail.com:587}
      - GF_SMTP_USER=${SMTP_USER}
      - GF_SMTP_PASSWORD=${SMTP_PASSWORD}
      - GF_SMTP_FROM_ADDRESS=${SMTP_FROM}
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    networks:
      - traefik_proxy
      - monitoring
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_proxy"
      - "traefik.http.routers.grafana.rule=Host(`grafana.rbnk.uk`)"
      - "traefik.http.routers.grafana.entrypoints=websecure"
      - "traefik.http.routers.grafana.tls=true"
      - "traefik.http.routers.grafana.tls.certresolver=cloudflare"
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    volumes:
      - ./alertmanager/config.yml:/etc/alertmanager/config.yml
      - alertmanager_data:/alertmanager
    networks:
      - monitoring
    command:
      - '--config.file=/etc/alertmanager/config.yml'
      - '--storage.path=/alertmanager'

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    privileged: true
    devices:
      - /dev/kmsg
    networks:
      - monitoring

  postgres-exporter:
    image: prometheuscommunity/postgres-exporter:latest
    container_name: postgres-exporter
    restart: unless-stopped
    environment:
      DATA_SOURCE_NAME: "postgresql://postgres:${POSTGRES_PASSWORD}@supa-db:5432/postgres?sslmode=disable"
    networks:
      - monitoring
      - supabase_default

volumes:
  prometheus_data:
  grafana_data:
  alertmanager_data:

networks:
  monitoring:
    driver: bridge
  traefik_proxy:
    external: true
  supabase_default:
    external: true
EOF

# Create .env file
cat > /srv/dockerdata/monitoring/.env << 'EOF'
GRAFANA_PASSWORD=changeme
POSTGRES_PASSWORD=your-supabase-postgres-password
SMTP_HOST=smtp.gmail.com:587
[email protected]
SMTP_PASSWORD=your-app-password
[email protected]
EOF
chmod 640 /srv/dockerdata/monitoring/.env

2. Configure Prometheus

# Create Prometheus configuration
cat > /srv/dockerdata/monitoring/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'docker-monitoring'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - "alerts.yml"

scrape_configs:
  # Docker host metrics
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  # Container metrics
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  # PostgreSQL metrics
  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres-exporter:9187']

  # Traefik metrics
  - job_name: 'traefik'
    static_configs:
      - targets: ['traefik:8080']

  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Docker daemon metrics
  - job_name: 'docker'
    static_configs:
      - targets: ['172.17.0.1:9323']
EOF

3. Create Alert Rules

cat > /srv/dockerdata/monitoring/prometheus/alerts.yml << 'EOF'
groups:
  - name: system
    rules:
      - alert: HighCPUUsage
        expr: (100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage is above 80% (current value: {{ $value }}%)"

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage detected"
          description: "Memory usage is above 85% (current value: {{ $value }}%)"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space"
          description: "Disk space is below 15% (current value: {{ $value }}%)"

  - name: containers
    rules:
      - alert: ContainerDown
        expr: up{job="cadvisor"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Container is down"
          description: "Container {{ $labels.name }} has been down for more than 1 minute"

      - alert: ContainerHighCPU
        expr: (rate(container_cpu_usage_seconds_total[5m]) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container high CPU usage"
          description: "Container {{ $labels.name }} CPU usage is above 80%"

      - alert: ContainerHighMemory
        expr: (container_memory_usage_bytes / container_spec_memory_limit_bytes) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container high memory usage"
          description: "Container {{ $labels.name }} memory usage is above 90%"

  - name: postgres
    rules:
      - alert: PostgresDown
        expr: pg_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "PostgreSQL is down"
          description: "PostgreSQL instance is down"

      - alert: PostgresHighConnections
        expr: pg_stat_database_numbackends / pg_settings_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "PostgreSQL high connection count"
          description: "PostgreSQL connection count is above 80% of max_connections"

      - alert: PostgresDeadlocks
        expr: increase(pg_stat_database_deadlocks[5m]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "PostgreSQL deadlocks detected"
          description: "PostgreSQL has {{ $value }} deadlocks in the last 5 minutes"

  - name: backup
    rules:
      - alert: BackupFailed
        expr: backup_last_success_timestamp < (time() - 86400)
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "Backup job failed"
          description: "Last successful backup was more than 24 hours ago"

  - name: gitea
    rules:
      - alert: GiteaDown
        expr: up{job="gitea"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Gitea is down"
          description: "Gitea service is not responding"

      - alert: GiteaSyncFailed
        expr: gitea_sync_last_success < (time() - 3600)
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Gitea sync job failed"
          description: "Last successful sync to GitHub was more than 1 hour ago"

      - alert: GiteaHighDiskUsage
        expr: (gitea_repository_size_bytes / gitea_disk_quota_bytes) > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Gitea high disk usage"
          description: "Gitea repositories using > 90% of allocated space"
EOF

4. Configure Alertmanager

cat > /srv/dockerdata/monitoring/alertmanager/config.yml << 'EOF'
global:
  resolve_timeout: 5m
  # Using Mailrise SMTP gateway for email-to-notification conversion
  smtp_from: '[email protected]'
  smtp_smarthost: 'mailrise:8025'
  smtp_require_tls: false

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'critical'
      repeat_interval: 1h
    - match:
        severity: warning
      receiver: 'warning'

receivers:
  # Default receiver - uses Apprise for multi-channel notifications
  - name: 'default'
    webhook_configs:
      - url: 'http://apprise:8000/notify/apprise'
        send_resolved: true

  # Critical alerts - high priority with multiple channels
  - name: 'critical'
    webhook_configs:
      - url: 'http://apprise:8000/notify/apprise?tag=critical,oncall'
        send_resolved: true
    email_configs:
      - to: '[email protected]'
        headers:
          Subject: '[CRITICAL] {{ .GroupLabels.alertname }}'

  # Warning alerts - normal priority
  - name: 'warning'
    webhook_configs:
      - url: 'http://apprise:8000/notify/apprise?tag=warning'
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']
EOF

# Note: The notification stack integration provides:
# - Apprise: Routes to 100+ notification services (WhatsApp, Telegram, Discord, etc.)
# - ntfy: Push notifications to mobile/desktop apps
# - Mailrise: Converts AlertManager emails to push notifications
# - Uptime Kuma: Additional service monitoring with its own notifications
# Configure actual notification URLs in Apprise web UI at https://apprise.rbnk.uk

5. Enable Docker Metrics

# Configure Docker daemon for metrics
sudo tee /etc/docker/daemon.json > /dev/null << 'EOF'
{
  "metrics-addr": "172.17.0.1:9323",
  "experimental": true
}
EOF

# Restart Docker
sudo systemctl restart docker

6. Add Traefik Metrics

# Update Traefik configuration
cat >> /srv/dockerdata/traefik/traefik.yml << 'EOF'

metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0
EOF

# Restart Traefik
cd /srv/dockerdata/traefik
docker compose restart

Grafana Dashboard Setup

1. Provision Datasources

mkdir -p /srv/dockerdata/monitoring/grafana/provisioning/datasources

cat > /srv/dockerdata/monitoring/grafana/provisioning/datasources/prometheus.yml << 'EOF'
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
EOF

2. Import Dashboards

mkdir -p /srv/dockerdata/monitoring/grafana/provisioning/dashboards

# Dashboard provisioning config
cat > /srv/dockerdata/monitoring/grafana/provisioning/dashboards/dashboards.yml << 'EOF'
apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards
EOF

# Download pre-built dashboards
cd /srv/dockerdata/monitoring/grafana/provisioning/dashboards

# Node Exporter Dashboard
wget https://grafana.com/api/dashboards/1860/revisions/latest/download -O node-exporter.json

# Docker Dashboard
wget https://grafana.com/api/dashboards/893/revisions/latest/download -O docker.json

# PostgreSQL Dashboard
wget https://grafana.com/api/dashboards/9628/revisions/latest/download -O postgresql.json

# Traefik Dashboard
wget https://grafana.com/api/dashboards/12250/revisions/latest/download -O traefik.json

Log Aggregation with Loki

1. Add Loki to Monitoring Stack

# Update docker-compose.yml
cat >> /srv/dockerdata/monitoring/docker-compose.yml << 'EOF'

  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: unless-stopped
    volumes:
      - ./loki/config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - monitoring

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail/config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks:
      - monitoring

volumes:
  loki_data:
EOF

2. Configure Loki

mkdir -p /srv/dockerdata/monitoring/loki

cat > /srv/dockerdata/monitoring/loki/config.yml << 'EOF'
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://alertmanager:9093

analytics:
  reporting_enabled: false
EOF

3. Configure Promtail

mkdir -p /srv/dockerdata/monitoring/promtail

cat > /srv/dockerdata/monitoring/promtail/config.yml << 'EOF'
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: containers
    static_configs:
      - targets:
          - localhost
        labels:
          job: containerlogs
          __path__: /var/lib/docker/containers/*/*log

    pipeline_stages:
      - json:
          expressions:
            output: log
            stream: stream
            attrs:
      - json:
          expressions:
            tag:
          source: attrs
      - regex:
          expression: (?P<container_name>(?:[^|]*))\|(?P<image_name>(?:[^|]*))
          source: tag
      - timestamp:
          format: RFC3339Nano
          source: time
      - labels:
          stream:
          container_name:
          image_name:
      - output:
          source: output

  - job_name: syslog
    static_configs:
      - targets:
          - localhost
        labels:
          job: syslog
          __path__: /var/log/syslog
EOF

Application-Specific Monitoring

Gitea Metrics

# Add Gitea monitoring script
cat > /srv/dockerdata/_scripts/gitea-metrics.sh << 'EOF'
#!/bin/bash
# Gitea metrics exporter for Prometheus

# Repository count
echo "# HELP gitea_repositories_total Total number of repositories"
echo "# TYPE gitea_repositories_total gauge"
docker exec gitea sqlite3 /data/gitea/gitea.db \
  "SELECT COUNT(*) FROM repository;" | \
  xargs echo "gitea_repositories_total"

# User count
echo "# HELP gitea_users_total Total number of users"
echo "# TYPE gitea_users_total gauge"
docker exec gitea sqlite3 /data/gitea/gitea.db \
  "SELECT COUNT(*) FROM user WHERE type = 0;" | \
  xargs echo "gitea_users_total"

# Repository sizes
echo "# HELP gitea_repository_size_bytes Repository size in bytes"
echo "# TYPE gitea_repository_size_bytes gauge"
docker exec gitea find /data/git/repositories -maxdepth 2 -mindepth 2 -type d | \
  while read repo; do
    size=$(docker exec gitea du -sb "$repo" | awk '{print $1}')
    name=$(basename "$(dirname "$repo")")/$(basename "$repo")
    echo "gitea_repository_size_bytes{repo=\"$name\"} $size"
  done

# Last sync status
echo "# HELP gitea_sync_last_success Last successful sync timestamp"
echo "# TYPE gitea_sync_last_success gauge"
if [[ -f /var/log/gitea-sync.log ]]; then
  last_success=$(grep "Sync completed successfully" /var/log/gitea-sync.log | \
    tail -1 | awk '{print $1 " " $2}')
  if [[ -n "$last_success" ]]; then
    timestamp=$(date -d "$last_success" +%s)
    echo "gitea_sync_last_success $timestamp"
  fi
fi
EOF

chmod +x /srv/dockerdata/_scripts/gitea-metrics.sh

Gitea Sync Monitoring

# Monitor sync jobs
cat > /srv/dockerdata/_scripts/monitor-gitea-sync.sh << 'EOF'
#!/bin/bash
# Monitor Gitea to GitHub sync status

LOG_FILE="/var/log/gitea-sync.log"
WEBHOOK_URL="${DISCORD_WEBHOOK_URL}"
MAX_AGE_MINUTES=30

# Check last sync time
if [[ -f "$LOG_FILE" ]]; then
  last_sync=$(grep "Sync completed" "$LOG_FILE" | tail -1)
  if [[ -n "$last_sync" ]]; then
    last_time=$(echo "$last_sync" | awk '{print $1 " " $2}')
    age_minutes=$(( ($(date +%s) - $(date -d "$last_time" +%s)) / 60 ))

    if [[ $age_minutes -gt $MAX_AGE_MINUTES ]]; then
      message="⚠️ Gitea sync hasn't run in $age_minutes minutes"
      curl -H "Content-Type: application/json" \
           -d "{\"content\": \"$message\"}" \
           "$WEBHOOK_URL"
    fi
  fi
fi

# Check for sync errors
recent_errors=$(tail -100 "$LOG_FILE" | grep -c "ERROR")
if [[ $recent_errors -gt 0 ]]; then
  message="🚨 Found $recent_errors errors in recent Gitea sync logs"
  curl -H "Content-Type: application/json" \
       -d "{\"content\": \"$message\"}" \
       "$WEBHOOK_URL"
fi
EOF

chmod +x /srv/dockerdata/_scripts/monitor-gitea-sync.sh

# Add to cron
echo "*/15 * * * * /srv/dockerdata/_scripts/monitor-gitea-sync.sh" | crontab -

Supabase Metrics

# Add custom metrics endpoint to Supabase
cat > /srv/dockerdata/supabase/metrics-exporter.sh << 'EOF'
#!/bin/bash
# Custom Supabase metrics exporter

# Database stats
echo "# HELP supabase_database_size_bytes Database size in bytes"
echo "# TYPE supabase_database_size_bytes gauge"
docker exec supa-db psql -U postgres -t -c \
  "SELECT pg_database_size('postgres');" | \
  xargs echo "supabase_database_size_bytes"

# Connection stats
echo "# HELP supabase_connections Active database connections"
echo "# TYPE supabase_connections gauge"
docker exec supa-db psql -U postgres -t -c \
  "SELECT count(*) FROM pg_stat_activity;" | \
  xargs echo "supabase_connections"

# Table sizes
echo "# HELP supabase_table_size_bytes Table size in bytes"
echo "# TYPE supabase_table_size_bytes gauge"
docker exec supa-db psql -U postgres -t -c \
  "SELECT schemaname, tablename, pg_total_relation_size(schemaname||'.'||tablename) 
   FROM pg_tables WHERE schemaname NOT IN ('pg_catalog', 'information_schema');" | \
  while read schema table size; do
    echo "supabase_table_size_bytes{schema=\"$schema\",table=\"$table\"} $size"
  done
EOF

chmod +x /srv/dockerdata/supabase/metrics-exporter.sh

Health Check Implementation

1. Service Health Endpoints

# Create health check service
cat > /srv/dockerdata/monitoring/health-checker/docker-compose.yml << 'EOF'
version: '3.8'

services:
  health-checker:
    image: willfarrell/ping:latest
    container_name: health-checker
    restart: unless-stopped
    environment:
      CHECKS: |
        {
          "supabase-api": {
            "url": "https://supabase.rbnk.uk/rest/v1/",
            "method": "GET",
            "expectedStatusCode": 200,
            "timeout": 10
          },
          "supabase-auth": {
            "url": "https://supabase.rbnk.uk/auth/v1/health",
            "method": "GET",
            "expectedStatusCode": 200,
            "timeout": 10
          },
          "traefik": {
            "url": "http://traefik:8080/ping",
            "method": "GET",
            "expectedStatusCode": 200,
            "timeout": 5
          },
          "grafana": {
            "url": "https://grafana.rbnk.uk/api/health",
            "method": "GET",
            "expectedStatusCode": 200,
            "timeout": 10
          },
          "gitea": {
            "url": "https://gitea.rbnk.uk/api/v1/version",
            "method": "GET",
            "expectedStatusCode": 200,
            "timeout": 10
          }
        }
    networks:
      - monitoring
      - traefik_proxy
    labels:
      - "prometheus.io/scrape=true"
      - "prometheus.io/port=9090"

networks:
  monitoring:
    external: true
  traefik_proxy:
    external: true
EOF

2. Custom Health Checks

# Add to each service's docker-compose.yml
cat >> /srv/dockerdata/supabase/docker-compose.yml << 'EOF'

    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
EOF

Resource Usage Tracking

1. Create Resource Dashboard

{
  "dashboard": {
    "title": "Resource Usage",
    "panels": [
      {
        "title": "CPU Usage by Container",
        "targets": [{
          "expr": "rate(container_cpu_usage_seconds_total[5m]) * 100"
        }]
      },
      {
        "title": "Memory Usage by Container",
        "targets": [{
          "expr": "container_memory_usage_bytes / 1024 / 1024 / 1024"
        }]
      },
      {
        "title": "Network I/O",
        "targets": [{
          "expr": "rate(container_network_receive_bytes_total[5m])"
        }]
      },
      {
        "title": "Disk I/O",
        "targets": [{
          "expr": "rate(container_fs_reads_bytes_total[5m])"
        }]
      }
    ]
  }
}

Notification Stack Integration

Connecting Monitoring to Notifications

The monitoring stack integrates seamlessly with the notification system:

1. AlertManager → Apprise
2. All alerts route through Apprise for multi-channel delivery
3. Tags control which services receive which alerts

4. Supports 100+ notification services

5. Uptime Kuma Integration

6. Provides additional service monitoring
7. Has its own notification configuration

8. Can send to Apprise for unified notifications

9. Direct ntfy Integration

10. Critical alerts can bypass Apprise and go directly to ntfy
11. Useful for high-priority mobile push notifications
12. Topic structure: monitoring-severity-category

Setting Up Notification Channels

# 1. Configure Apprise (via web UI at https://apprise.rbnk.uk)
# Add notification URLs for your preferred services:
# - WhatsApp: whatsapp://phone@number
# - Telegram: tgram://bottoken/ChatID
# - Discord: discord://webhook_id/webhook_token
# - Email: mailto://user:[email protected]

# 2. Configure ntfy authentication
# Service accounts are pre-configured for monitoring:
# Username: monitoring
# Password: (check /srv/dockerdata/ntfy/.env)

# 3. Test notifications
curl -X POST http://apprise:8000/notify/apprise \
  -H "Content-Type: application/json" \
  -d '{"title": "Test Alert", "body": "Monitoring test", "tag": "test"}'

Monitoring Best Practices

Key Metrics to Monitor

1. System Metrics
2. CPU usage (< 80%)
3. Memory usage (< 85%)
4. Disk usage (< 85%)
5. Network I/O

6. System load

7. Container Metrics

8. Container status
9. Resource usage per container
10. Container restarts

11. Exit codes

12. Application Metrics

13. Request rate
14. Error rate
15. Response time (p50, p95, p99)
16. Queue depth

17. Database connections

18. Business Metrics

19. Active users
20. Transaction volume
21. API usage
22. Storage consumption

Alert Configuration

# Severity Levels
- critical: Immediate action required (pager)
- warning: Investigation needed (email)
- info: Informational only (log)

# Alert Routing
- Database down: critical
- High resource usage: warning
- Backup failure: critical
- Certificate expiry: warning
- Service degradation: warning

Maintenance Tasks

1. Daily
2. Check dashboard for anomalies
3. Review recent alerts

4. Verify backup completion

5. Weekly

6. Review resource trends
7. Check for security alerts

8. Update dashboards as needed

9. Monthly

10. Review and tune alerts
11. Analyze performance trends
12. Capacity planning
13. Update monitoring documentation

Troubleshooting

Common Issues

1. Prometheus not scraping

   # Check targets
   curl http://localhost:9090/api/v1/targets
   
   # Check service discovery
   docker exec prometheus promtool check config /etc/prometheus/prometheus.yml
   

2. Grafana datasource issues

   # Test connection
   curl http://prometheus:9090/api/v1/query?query=up
   
   # Check logs
   docker logs grafana
   

3. Missing metrics

   # Verify exporter is running
   docker ps | grep exporter
   
   # Check exporter metrics
   curl http://node-exporter:9100/metrics
   

Service-Specific Monitoring

Rclone Backup Monitoring

Monitor cloud backup operations for reliability and performance:

1. Metrics Collection
2. Textfile Collector: Rclone metrics are collected via the Node Exporter textfile collector
3. Script Location: /srv/dockerdata/rclone/collect-metrics.sh
4. Schedule: Runs every 5 minutes via cron

5. Metrics Endpoint: Exposed through node-exporter at port 9100

6. Backup Monitoring

The monitoring system tracks both infrastructure and cloud backups:

Infrastructure Backup Metrics: - rclone_infrastructure_backup_last_success_timestamp - Unix timestamp of last successful backup - rclone_infrastructure_backup_errors_24h - Number of backup errors in the last 24 hours

Cloud Backup Metrics: - rclone_cloud_backup_last_success_timestamp - Unix timestamp of last successful cloud sync - rclone_cloud_backup_errors_24h - Number of sync errors in the last 24 hours - rclone_cloud_backup_size_bytes{provider="gdrive|dropbox|onedrive"} - Size of backups per cloud provider

R2 Storage Metrics: - rclone_r2_used_bytes - Used storage in Cloudflare R2 - rclone_r2_total_bytes - Total available storage (1TB limit) - rclone_r2_usage_percent - Percentage of R2 storage used

Collection Metrics: - rclone_metrics_collection_timestamp - Unix timestamp of last metrics collection (for freshness check)

1. Grafana Dashboards

Rclone Backup Monitoring Dashboard (Located in Infrastructure folder): - Auto-refresh: Every 30 seconds - Key Panels: - Backup Status Overview - Shows last backup times and current status - R2 Storage Usage Gauge - Visual representation of storage utilization - Storage Distribution by Provider - Pie chart showing backup sizes across cloud providers - Storage Growth Trends - Time series showing storage usage over time - Error Count Tracking - Displays backup errors over the last 24 hours

1. Alert Rules

The following Rclone-specific alerts are configured:

- alert: InfrastructureBackupMissing
  expr: time() - rclone_infrastructure_backup_last_success_timestamp > 172800
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Infrastructure backup missing for 48 hours"
    description: "Last successful infrastructure backup was {{ $value | humanizeDuration }} ago"

- alert: CloudBackupMissing  
  expr: time() - rclone_cloud_backup_last_success_timestamp > 172800
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Cloud backup sync missing for 48 hours"
    description: "Last successful cloud sync was {{ $value | humanizeDuration }} ago"

- alert: BackupErrors
  expr: (rclone_infrastructure_backup_errors_24h > 0) or (rclone_cloud_backup_errors_24h > 0)
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Backup errors detected"
    description: "{{ $value }} backup errors in the last 24 hours"

- alert: R2StorageHigh
  expr: rclone_r2_usage_percent > 80
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "R2 storage usage high"
    description: "R2 storage is {{ $value }}% full"

- alert: RcloneMetricsStale
  expr: time() - rclone_metrics_collection_timestamp > 600
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Rclone metrics collection stale"
    description: "Metrics haven't been collected for {{ $value | humanizeDuration }}"

1. Example Queries

# Time since last infrastructure backup (in hours)
(time() - rclone_infrastructure_backup_last_success_timestamp) / 3600

# R2 storage usage percentage
rclone_r2_usage_percent

# Total backup errors in last 24h
rclone_infrastructure_backup_errors_24h + rclone_cloud_backup_errors_24h

# Cloud backup sizes by provider
rclone_cloud_backup_size_bytes

# Check if metrics are fresh (collected within last 10 minutes)
time() - rclone_metrics_collection_timestamp < 600

Next Steps

1. Deploy monitoring stack
2. Import dashboards
3. Configure alerts (including Rclone backup monitoring)
4. Set up on-call rotation
5. Document runbooks
6. Train team on tools