Skip to content

Monitoring Configuration for New Services

This document provides monitoring configurations for Metube, Watchtower, and Paperless-AI services integrated with the existing Prometheus, Grafana, and AlertManager infrastructure.

Table of Contents

Metube Monitoring

Key Metrics

  • Download queue size
  • Download success/failure rate
  • Storage usage
  • Active downloads count
  • Download speed

Prometheus Configuration

Add to /srv/dockerdata/monitoring/prometheus/prometheus.yml:

  - job_name: 'metube'
    static_configs:
      - targets: ['metube:8081']
    metrics_path: '/metrics'
    scrape_interval: 30s

Alert Rules

Add to /srv/dockerdata/monitoring/prometheus/alerts.yml:

  - name: metube
    interval: 30s
    rules:
      - alert: MetubeDown
        expr: up{job="metube"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Metube service is down"
          description: "Metube has been down for more than 2 minutes"

      - alert: MetubeHighQueueSize
        expr: metube_queue_size > 50
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Metube download queue is large"
          description: "Download queue has {{ $value }} items pending"

      - alert: MetubeHighFailureRate
        expr: |
          rate(metube_downloads_failed_total[5m]) /
          rate(metube_downloads_total[5m]) > 0.25
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High download failure rate"
          description: "Download failure rate is {{ $value | humanizePercentage }}"

      - alert: MetubeStorageFull
        expr: |
          (metube_storage_used_bytes / metube_storage_total_bytes) > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Metube storage almost full"
          description: "Storage usage is at {{ $value | humanizePercentage }}"

Grafana Dashboard

{
  "title": "Metube Video Downloader",
  "panels": [
    {
      "title": "Download Queue Size",
      "targets": [
        {
          "expr": "metube_queue_size",
          "legendFormat": "Queue Size"
        }
      ],
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }
    },
    {
      "title": "Download Success Rate",
      "targets": [
        {
          "expr": "rate(metube_downloads_success_total[5m]) / rate(metube_downloads_total[5m]) * 100",
          "legendFormat": "Success Rate %"
        }
      ],
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }
    },
    {
      "title": "Storage Usage",
      "targets": [
        {
          "expr": "(metube_storage_used_bytes / metube_storage_total_bytes) * 100",
          "legendFormat": "Storage Used %"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 70 },
              { "color": "red", "value": 85 }
            ]
          },
          "unit": "percent"
        }
      },
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }
    },
    {
      "title": "Active Downloads",
      "targets": [
        {
          "expr": "metube_active_downloads",
          "legendFormat": "Active Downloads"
        }
      ],
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }
    }
  ]
}

Watchtower Monitoring

Key Metrics

  • Containers monitored
  • Containers updated
  • Update failures
  • Last update check timestamp
  • Update frequency

Prometheus Configuration

Add to /srv/dockerdata/monitoring/prometheus/prometheus.yml:

  - job_name: 'watchtower'
    static_configs:
      - targets: ['watchtower:8080']
    metrics_path: '/v1/metrics'
    scrape_interval: 60s

Alert Rules

Add to /srv/dockerdata/monitoring/prometheus/alerts.yml:

  - name: watchtower
    interval: 30s
    rules:
      - alert: WatchtowerDown
        expr: up{job="watchtower"} == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Watchtower is down"
          description: "Container update service has been down for 5 minutes"

      - alert: WatchtowerUpdateFailed
        expr: watchtower_updates_failed_total > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Container update failed"
          description: "{{ $value }} container updates have failed"

      - alert: WatchtowerNotChecking
        expr: time() - watchtower_last_check_timestamp_seconds > 86400
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Watchtower hasn't checked for updates"
          description: "Last update check was {{ $value | humanizeDuration }} ago"

      - alert: WatchtowerHighUpdateCount
        expr: increase(watchtower_updates_total[1h]) > 10
        labels:
          severity: info
        annotations:
          summary: "Many containers updated"
          description: "{{ $value }} containers were updated in the last hour"

Grafana Dashboard

{
  "title": "Watchtower Container Updates",
  "panels": [
    {
      "title": "Containers Monitored",
      "targets": [
        {
          "expr": "watchtower_containers_monitored",
          "legendFormat": "Monitored"
        }
      ],
      "gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 }
    },
    {
      "title": "Updates (24h)",
      "targets": [
        {
          "expr": "increase(watchtower_updates_total[24h])",
          "legendFormat": "Updated"
        },
        {
          "expr": "increase(watchtower_updates_failed_total[24h])",
          "legendFormat": "Failed"
        }
      ],
      "gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 }
    },
    {
      "title": "Time Since Last Check",
      "targets": [
        {
          "expr": "(time() - watchtower_last_check_timestamp_seconds) / 3600",
          "legendFormat": "Hours"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "h",
          "thresholds": {
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 12 },
              { "color": "red", "value": 24 }
            ]
          }
        }
      },
      "gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 }
    },
    {
      "title": "Update History",
      "targets": [
        {
          "expr": "rate(watchtower_updates_total[1h])",
          "legendFormat": "Updates/hour"
        }
      ],
      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 }
    }
  ]
}

Paperless-AI Monitoring

Key Metrics

  • Document processing queue
  • OCR success rate
  • API response time
  • Storage usage
  • Active processing tasks
  • Document count by type

Prometheus Configuration

Add to /srv/dockerdata/monitoring/prometheus/prometheus.yml:

  - job_name: 'paperless'
    static_configs:
      - targets: ['paperless-webserver:8000']
    metrics_path: '/api/metrics/'
    scrape_interval: 30s
    bearer_token: 'YOUR_PAPERLESS_API_TOKEN'

Alert Rules

Add to /srv/dockerdata/monitoring/prometheus/alerts.yml:

  - name: paperless
    interval: 30s
    rules:
      - alert: PaperlessDown
        expr: up{job="paperless"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Paperless-AI is down"
          description: "Document management service is not responding"

      - alert: PaperlessHighQueue
        expr: paperless_documents_pending > 100
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Large document processing queue"
          description: "{{ $value }} documents waiting to be processed"

      - alert: PaperlessOCRFailures
        expr: |
          rate(paperless_ocr_failed_total[5m]) /
          rate(paperless_ocr_total[5m]) > 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High OCR failure rate"
          description: "OCR failure rate is {{ $value | humanizePercentage }}"

      - alert: PaperlessAPISlowResponse
        expr: |
          histogram_quantile(0.95,
            rate(paperless_api_request_duration_seconds_bucket[5m])
          ) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Paperless API slow response"
          description: "95th percentile API response time is {{ $value | humanizeDuration }}"

      - alert: PaperlessStorageFull
        expr: |
          (paperless_storage_used_bytes / paperless_storage_total_bytes) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Paperless storage getting full"
          description: "Storage usage is at {{ $value | humanizePercentage }}"

Grafana Dashboard

{
  "title": "Paperless-AI Document Management",
  "panels": [
    {
      "title": "Processing Queue",
      "targets": [
        {
          "expr": "paperless_documents_pending",
          "legendFormat": "Pending"
        },
        {
          "expr": "paperless_documents_processing",
          "legendFormat": "Processing"
        }
      ],
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }
    },
    {
      "title": "OCR Success Rate",
      "targets": [
        {
          "expr": "rate(paperless_ocr_success_total[5m]) / rate(paperless_ocr_total[5m]) * 100",
          "legendFormat": "Success Rate %"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "thresholds": {
            "steps": [
              { "color": "red", "value": null },
              { "color": "yellow", "value": 80 },
              { "color": "green", "value": 90 }
            ]
          }
        }
      },
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }
    },
    {
      "title": "API Response Time (p95)",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, rate(paperless_api_request_duration_seconds_bucket[5m]))",
          "legendFormat": "95th percentile"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "s"
        }
      },
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }
    },
    {
      "title": "Documents by Type",
      "targets": [
        {
          "expr": "paperless_documents_total",
          "legendFormat": "{{ type }}"
        }
      ],
      "options": {
        "pieType": "pie",
        "displayLabels": ["name", "percent"]
      },
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }
    },
    {
      "title": "Storage Usage",
      "targets": [
        {
          "expr": "(paperless_storage_used_bytes / paperless_storage_total_bytes) * 100",
          "legendFormat": "Used %"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "thresholds": {
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 70 },
              { "color": "red", "value": 85 }
            ]
          }
        }
      },
      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 16 }
    }
  ]
}

Integration Steps

1. Enable Metrics Endpoints

Metube: Ensure the container exposes metrics on port 8081: 

# metube/docker-compose.yml
services:
  metube:
    environment:
      - ENABLE_METRICS=true
      - METRICS_PORT=8081
    ports:
      - "8081:8081"  # Metrics port

Watchtower: Enable metrics endpoint: 

# watchtower/docker-compose.yml
services:
  watchtower:
    environment:
      - WATCHTOWER_HTTP_API_METRICS=true
      - WATCHTOWER_HTTP_API_TOKEN=${WATCHTOWER_API_TOKEN}
    ports:
      - "8080:8080"  # API/Metrics port

Paperless-AI: Configure API access: 

# paperless/docker-compose.yml
services:
  webserver:
    environment:
      - PAPERLESS_ENABLE_METRICS=true

2. Update Prometheus Configuration

  1. Add the job configurations to /srv/dockerdata/monitoring/prometheus/prometheus.yml
  2. Add the alert rules to /srv/dockerdata/monitoring/prometheus/alerts.yml
  3. Restart Prometheus: 
    docker compose -f monitoring/docker-compose.yml restart prometheus

3. Import Grafana Dashboards

  1. Access Grafana at https://grafana.rbnk.uk
  2. Navigate to Dashboards → Import
  3. Create a new dashboard for each service using the provided JSON configurations
  4. Save dashboards to the "Services" folder

4. Configure AlertManager Routing

Add service-specific routing in /srv/dockerdata/monitoring/alertmanager/alertmanager.yml:

route:
  routes:
    - match:
        alertname: "Metube.*"
      receiver: "media-alerts"
    - match:
        alertname: "Watchtower.*"
      receiver: "infrastructure-alerts"
    - match:
        alertname: "Paperless.*"
      receiver: "document-alerts"

5. Verify Integration

  1. Check Prometheus targets: 

    curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.job | contains("metube", "watchtower", "paperless"))'

  2. Test alerts: 

    docker exec prometheus promtool check rules /etc/prometheus/alerts.yml

  3. Verify metrics collection: 

    # Check Metube metrics
curl http://localhost:9090/api/v1/query?query=metube_queue_size

# Check Watchtower metrics
curl http://localhost:9090/api/v1/query?query=watchtower_containers_monitored

# Check Paperless metrics
curl http://localhost:9090/api/v1/query?query=paperless_documents_pending

6. Add to Glance Dashboard

Update /srv/dockerdata/glance/config/glance.yml to include the new services:

services:
  - name: Media Services
    columns: 3
    services:
      - title: Metube
        url: https://metube.rbnk.uk
        icon: si-youtube
        health-check:
          url: https://metube.rbnk.uk/health
      - title: Watchtower
        description: Container Updates
        icon: si-docker
        health-check:
          url: http://watchtower:8080/v1/metrics
      - title: Paperless-AI
        url: https://paperless.rbnk.uk
        icon: si-files
        health-check:
          url: https://paperless.rbnk.uk/api/health/

Monitoring Best Practices

  1. Metric Naming: Follow Prometheus naming conventions
  2. Use _total suffix for counters
  3. Use _bytes suffix for sizes

  4. Use _seconds suffix for durations

  5. Label Usage: Keep cardinality low

  6. Avoid high-cardinality labels (IDs, timestamps)

  7. Use meaningful labels (service, type, status)

  8. Alert Tuning: Start with conservative thresholds

  9. Monitor for false positives

  10. Adjust based on actual usage patterns

  11. Dashboard Organization: Group related metrics

  12. Use consistent color schemes
  13. Include relevant thresholds

  14. Add helpful descriptions

  15. Performance Impact: Monitor scraping duration

  16. Keep scrape intervals appropriate
  17. Use recording rules for complex queries

Grafana Dashboard Recommendations

Metube (YouTube Downloader)

Dashboard Structure

Create a dedicated dashboard with 4 main sections: 1. Service Health Overview (Row 1) 2. Download Performance Metrics (Row 2) 3. Storage Analytics (Row 3) 4. Format Distribution (Row 4)

Key Metrics to Monitor

  1. Download Performance 

    # Active downloads (based on CPU activity)
rate(container_cpu_usage_seconds_total{name="metube"}[1m]) > 0.1

# Download success rate (custom metric)
rate(metube_downloads_success_total[5m]) / rate(metube_downloads_total[5m])

# Average download duration
rate(metube_download_duration_seconds_sum[5m]) / rate(metube_download_duration_seconds_count[5m])

# Queue length (via custom exporter)
metube_queue_length

  2. Storage Metrics 

    # Storage used by downloads
node_filesystem_used_bytes{mountpoint="/srv/dockerdata"} - node_filesystem_used_bytes{mountpoint="/srv/dockerdata"} offset 24h

# Storage growth rate (GB/day)
rate(node_filesystem_used_bytes{mountpoint="/srv/dockerdata"}[24h]) * 86400 / 1e9

# Available storage percentage
100 - (node_filesystem_avail_bytes{mountpoint="/srv/dockerdata"} / node_filesystem_size_bytes{mountpoint="/srv/dockerdata"} * 100)

# File count estimation (via custom script)
metube_total_files

  3. Format Distribution 

    # Video quality distribution
metube_downloads_by_quality{quality=~"1080p|720p|480p|360p"}

# Audio vs Video ratio
sum(metube_downloads_by_type{type="audio"}) / sum(metube_downloads_by_type{type="video"})

# Format popularity
topk(5, metube_downloads_by_format)


# Row 1: Service Health Overview
panels:
  - title: "Service Status"
    type: stat
    targets:
      - expr: up{job="blackbox", instance="https://metube.rbnk.uk"}
    mappings:
      - value: 1
        text: "ONLINE"
        color: green
      - value: 0
        text: "OFFLINE"
        color: red
    gridPos: {h: 4, w: 4, x: 0, y: 0}

  - title: "Current Downloads"
    type: stat
    targets:
      - expr: metube_active_downloads
    unit: short
    gridPos: {h: 4, w: 4, x: 4, y: 0}

  - title: "24h Success Rate"
    type: gauge
    targets:
      - expr: |
          100 * (
            sum(increase(metube_downloads_success_total[24h])) /
            sum(increase(metube_downloads_total[24h]))
          )
    unit: percent
    thresholds:
      - value: 0
        color: red
      - value: 80
        color: yellow
      - value: 95
        color: green
    gridPos: {h: 4, w: 4, x: 8, y: 0}

  - title: "Storage Available"
    type: gauge
    targets:
      - expr: |
          node_filesystem_avail_bytes{mountpoint="/srv/dockerdata"} / 1e9
    unit: decgbytes
    thresholds:
      - value: 0
        color: red
      - value: 50
        color: yellow
      - value: 100
        color: green
    gridPos: {h: 4, w: 4, x: 12, y: 0}

  - title: "Response Time"
    type: stat
    targets:
      - expr: probe_duration_seconds{job="blackbox", instance="https://metube.rbnk.uk"}
    unit: s
    gridPos: {h: 4, w: 4, x: 16, y: 0}

# Row 2: Download Performance
  - title: "Download Activity Timeline"
    type: timeseries
    targets:
      - expr: rate(container_cpu_usage_seconds_total{name="metube"}[5m]) * 100
        legendFormat: "CPU Usage %"
      - expr: metube_active_downloads
        legendFormat: "Active Downloads"
    gridPos: {h: 8, w: 12, x: 0, y: 4}

  - title: "Download Queue"
    type: graph
    targets:
      - expr: metube_queue_length
        legendFormat: "Queue Length"
      - expr: rate(metube_downloads_started_total[5m]) * 60
        legendFormat: "Downloads/min"
    gridPos: {h: 8, w: 12, x: 12, y: 4}

Watchtower (Container Updates)

Dashboard Structure

Create a dedicated dashboard with 3 main sections: 1. Update Activity Overview (Row 1) 2. Container Health Metrics (Row 2) 3. Notification & Error Tracking (Row 3)

Key Metrics to Monitor

  1. Update Activity 

    # Containers checked (based on logs)
watchtower_containers_checked

# Successful updates counter
increase(watchtower_updates_success_total[24h])

# Failed updates counter
increase(watchtower_updates_failed_total[24h])

# Time since last check
time() - watchtower_last_check_timestamp

  2. Container Management 

    # Total containers monitored
count(container_last_seen{name!="watchtower"})

# Watchtower memory usage
container_memory_usage_bytes{name="watchtower"} / 1024 / 1024

# Watchtower restart count
container_restart_count{name="watchtower"}

# Container update lag
max by (name) (time() - container_start_time_seconds)

  3. Notification Metrics 

    # Notification attempts
watchtower_notifications_sent_total

# Notification failures
watchtower_notifications_failed_total

# Alert frequency
rate(watchtower_notifications_sent_total[1h])


# Row 1: Update Activity Overview
panels:
  - title: "Watchtower Status"
    type: stat
    targets:
      - expr: up{job="watchtower"} OR vector(1)
    mappings:
      - value: 1
        text: "RUNNING"
        color: green
      - value: 0
        text: "DOWN"
        color: red
    gridPos: {h: 4, w: 4, x: 0, y: 0}

  - title: "Last Update Check"
    type: stat
    targets:
      - expr: time() - watchtower_last_check_timestamp
    unit: s
    thresholds:
      - value: 0
        color: green
      - value: 3600
        color: yellow
      - value: 86400
        color: red
    gridPos: {h: 4, w: 4, x: 4, y: 0}

  - title: "24h Update Success"
    type: stat
    targets:
      - expr: increase(watchtower_updates_success_total[24h])
    unit: short
    gridPos: {h: 4, w: 4, x: 8, y: 0}

  - title: "24h Update Failures"
    type: stat
    targets:
      - expr: increase(watchtower_updates_failed_total[24h])
    unit: short
    thresholds:
      - value: 0
        color: green
      - value: 1
        color: yellow
      - value: 5
        color: red
    gridPos: {h: 4, w: 4, x: 12, y: 0}

  - title: "Containers Monitored"
    type: gauge
    targets:
      - expr: count(container_last_seen{name!="watchtower"})
    unit: short
    gridPos: {h: 4, w: 4, x: 16, y: 0}

# Row 2: Container Health Metrics
  - title: "Container Update Timeline"
    type: timeseries
    targets:
      - expr: |
          sum by (name) (
            changes(container_start_time_seconds[24h])
          )
        legendFormat: "{{name}}"
    gridPos: {h: 8, w: 12, x: 0, y: 4}

  - title: "Update Activity Log"
    type: table
    targets:
      - expr: |
          topk(20,
            max by (name, image) (
              container_start_time_seconds
            )
          )
    format: table
    transformations:
      - id: organize
        options:
          columns:
            - name: Time
              type: time
            - name: Container
              field: name
            - name: Image
              field: image
    gridPos: {h: 8, w: 12, x: 12, y: 4}

# Row 3: Notification & Error Tracking
  - title: "Notification Success Rate"
    type: gauge
    targets:
      - expr: |
          100 * (
            watchtower_notifications_sent_total - watchtower_notifications_failed_total
          ) / watchtower_notifications_sent_total
    unit: percent
    thresholds:
      - value: 0
        color: red
      - value: 90
        color: yellow
      - value: 99
        color: green
    gridPos: {h: 6, w: 6, x: 0, y: 12}

  - title: "Memory Usage"
    type: graph
    targets:
      - expr: container_memory_usage_bytes{name="watchtower"} / 1024 / 1024
        legendFormat: "Memory (MB)"
    gridPos: {h: 6, w: 9, x: 6, y: 12}

  - title: "Network Activity"
    type: graph
    targets:
      - expr: rate(container_network_transmit_bytes_total{name="watchtower"}[5m])
        legendFormat: "TX Bytes/s"
      - expr: rate(container_network_receive_bytes_total{name="watchtower"}[5m])
        legendFormat: "RX Bytes/s"
    gridPos: {h: 6, w: 9, x: 15, y: 12}

Paperless-AI (Document Management)

Dashboard Structure

Create a dedicated dashboard with 4 main sections: 1. Service Health Overview (Row 1) 2. Document Processing Metrics (Row 2) 3. RAG Performance Analytics (Row 3) 4. Storage & Resource Usage (Row 4)

Key Metrics to Monitor

  1. Document Processing 

    # Documents in processing queue
paperless_documents_queue_length

# OCR processing rate
rate(paperless_documents_processed_total[5m])

# Average OCR duration
rate(paperless_ocr_duration_seconds_sum[5m]) / rate(paperless_ocr_duration_seconds_count[5m])

# Failed document processing
increase(paperless_documents_failed_total[1h])

# Document classification accuracy
paperless_classification_accuracy_percent

  2. RAG Performance 

    # Query response time (p95)
histogram_quantile(0.95, rate(paperless_rag_query_duration_seconds_bucket[5m]))

# Semantic search latency
rate(paperless_rag_search_duration_seconds_sum[5m]) / rate(paperless_rag_search_duration_seconds_count[5m])

# Vector index size
paperless_rag_index_size_bytes / 1024 / 1024

# RAG service memory usage
container_memory_usage_bytes{name="paperless-ai-rag"} / 1024 / 1024 / 1024

# Embeddings generation rate
rate(paperless_rag_embeddings_created_total[5m])

  3. Storage & Database 

    # Document storage usage
paperless_storage_documents_bytes / 1024 / 1024 / 1024

# Database size
pg_database_size_bytes{datname="paperless"} / 1024 / 1024

# Active database connections
pg_stat_activity_count{datname="paperless"}

# Index optimization status
paperless_index_optimization_timestamp


# Row 1: Service Health Overview
panels:
  - title: "Main Service Status"
    type: stat
    targets:
      - expr: up{job="paperless-ai"}
    mappings:
      - value: 1
        text: "ONLINE"
        color: green
      - value: 0
        text: "OFFLINE"
        color: red
    gridPos: {h: 4, w: 4, x: 0, y: 0}

  - title: "RAG Service Status"
    type: stat
    targets:
      - expr: up{job="paperless-rag"}
    mappings:
      - value: 1
        text: "ONLINE"
        color: green
      - value: 0
        text: "OFFLINE"
        color: red
    gridPos: {h: 4, w: 4, x: 4, y: 0}

  - title: "Processing Queue"
    type: gauge
    targets:
      - expr: paperless_documents_queue_length
    unit: short
    thresholds:
      - value: 0
        color: green
      - value: 10
        color: yellow
      - value: 50
        color: red
    gridPos: {h: 4, w: 4, x: 8, y: 0}

  - title: "Documents Processed (24h)"
    type: stat
    targets:
      - expr: increase(paperless_documents_processed_total[24h])
    unit: short
    gridPos: {h: 4, w: 4, x: 12, y: 0}

  - title: "AI Model Memory"
    type: gauge
    targets:
      - expr: container_memory_usage_bytes{name="paperless-ai-rag"} / 1024 / 1024 / 1024
    unit: decgbytes
    thresholds:
      - value: 0
        color: green
      - value: 4
        color: yellow
      - value: 8
        color: red
    gridPos: {h: 4, w: 4, x: 16, y: 0}

# Row 2: Document Processing Metrics
  - title: "Document Processing Rate"
    type: timeseries
    targets:
      - expr: rate(paperless_documents_processed_total[5m]) * 60
        legendFormat: "Docs/min"
      - expr: rate(paperless_documents_failed_total[5m]) * 60
        legendFormat: "Failed/min"
    gridPos: {h: 8, w: 12, x: 0, y: 4}

  - title: "OCR Performance"
    type: graph
    targets:
      - expr: |
          histogram_quantile(0.95, 
            rate(paperless_ocr_duration_seconds_bucket[5m])
          )
        legendFormat: "p95 OCR Time"
      - expr: |
          histogram_quantile(0.50, 
            rate(paperless_ocr_duration_seconds_bucket[5m])
          )
        legendFormat: "Median OCR Time"
    yaxes:
      - format: s
        label: "Processing Time"
    gridPos: {h: 8, w: 12, x: 12, y: 4}

# Row 3: RAG Performance Analytics
  - title: "RAG Query Response Time"
    type: heatmap
    targets:
      - expr: |
          sum by (le) (
            rate(paperless_rag_query_duration_seconds_bucket[5m])
          )
        format: heatmap
    dataFormat: tsbuckets
    gridPos: {h: 8, w: 12, x: 0, y: 12}

  - title: "Semantic Search Performance"
    type: graph
    targets:
      - expr: |
          histogram_quantile(0.95,
            rate(paperless_rag_search_duration_seconds_bucket[5m])
          )
        legendFormat: "p95 Search Time"
      - expr: |
          rate(paperless_rag_search_requests_total[5m])
        legendFormat: "Searches/s"
        yaxis: 2
    yaxes:
      - format: s
        label: "Response Time"
      - format: short
        label: "Requests/s"
    gridPos: {h: 8, w: 12, x: 12, y: 12}

# Row 4: Storage & Resource Usage
  - title: "Storage Breakdown"
    type: piechart
    targets:
      - expr: paperless_storage_documents_bytes
        legendFormat: "Documents"
      - expr: paperless_storage_thumbnails_bytes
        legendFormat: "Thumbnails"
      - expr: paperless_storage_index_bytes
        legendFormat: "Search Index"
      - expr: pg_database_size_bytes{datname="paperless"}
        legendFormat: "Database"
    pieType: donut
    unit: decbytes
    gridPos: {h: 8, w: 8, x: 0, y: 20}

  - title: "CPU & Memory Usage"
    type: graph
    targets:
      - expr: |
          rate(container_cpu_usage_seconds_total{name="paperless-ai"}[5m]) * 100
        legendFormat: "Main Service CPU %"
      - expr: |
          rate(container_cpu_usage_seconds_total{name="paperless-ai-rag"}[5m]) * 100
        legendFormat: "RAG Service CPU %"
      - expr: |
          container_memory_usage_bytes{name="paperless-ai"} / 1024 / 1024 / 1024
        legendFormat: "Main Memory (GB)"
        yaxis: 2
      - expr: |
          container_memory_usage_bytes{name="paperless-ai-rag"} / 1024 / 1024 / 1024
        legendFormat: "RAG Memory (GB)"
        yaxis: 2
    yaxes:
      - format: percent
        label: "CPU Usage"
        max: 100
      - format: decgbytes
        label: "Memory Usage"
    gridPos: {h: 8, w: 16, x: 8, y: 20}

Prometheus Alert Rules

Alert Configuration File

Create /srv/dockerdata/monitoring/prometheus/alerts/new-services.yml:

groups:
  - name: metube_alerts
    interval: 30s
    rules:
      # Service Health
      - alert: MetubeDown
        expr: up{job="blackbox", instance="https://metube.rbnk.uk"} == 0
        for: 5m
        labels:
          severity: warning
          service: metube
          category: availability
        annotations:
          summary: "Metube service is down"
          description: "Metube has been unreachable for {{ $value }} minutes"
          runbook_url: "https://docs.rbnk.uk/monitoring/runbooks/metube-down"

      - alert: MetubeSlowResponse
        expr: probe_duration_seconds{job="blackbox", instance="https://metube.rbnk.uk"} > 5
        for: 10m
        labels:
          severity: warning
          service: metube
          category: performance
        annotations:
          summary: "Metube slow response time"
          description: "Response time is {{ $value }}s (threshold: 5s)"

      # Storage Alerts
      - alert: MetubeHighStorageUsage
        expr: |
          (1 - (node_filesystem_avail_bytes{mountpoint="/srv/dockerdata"} / 
           node_filesystem_size_bytes{mountpoint="/srv/dockerdata"})) > 0.9
        for: 15m
        labels:
          severity: critical
          service: metube
          category: storage
        annotations:
          summary: "Critical: Low storage for downloads"
          description: "Only {{ printf \"%.2f\" (mul $value 100) }}% storage remaining on /srv/dockerdata"
          action: "Clean up old downloads or increase storage capacity"

      - alert: MetubeStorageGrowthRate
        expr: |
          predict_linear(node_filesystem_avail_bytes{mountpoint="/srv/dockerdata"}[4h], 86400) < 0
        for: 30m
        labels:
          severity: warning
          service: metube
          category: storage
        annotations:
          summary: "Storage will be full within 24 hours"
          description: "At current rate, storage will be exhausted in < 24h"

      # Download Performance
      - alert: MetubeHighFailureRate
        expr: |
          (rate(metube_downloads_failed_total[1h]) / 
           rate(metube_downloads_total[1h])) > 0.25
        for: 30m
        labels:
          severity: warning
          service: metube
          category: quality
        annotations:
          summary: "High download failure rate"
          description: "{{ printf \"%.2f\" (mul $value 100) }}% of downloads failing"

  - name: watchtower_alerts
    interval: 30s
    rules:
      # Service Health
      - alert: WatchtowerNotRunning
        expr: |
          up{job="cadvisor"} == 1 AND 
          absent(container_last_seen{name="watchtower"}) == 1
        for: 1h
        labels:
          severity: warning
          service: watchtower
          category: availability
        annotations:
          summary: "Watchtower container not running"
          description: "Watchtower has been down for over 1 hour"
          action: "Check container logs and restart if necessary"

      - alert: WatchtowerCrashLooping
        expr: |
          rate(container_restart_count{name="watchtower"}[15m]) > 0
        for: 5m
        labels:
          severity: critical
          service: watchtower
          category: stability
        annotations:
          summary: "Watchtower is crash looping"
          description: "Container restarted {{ $value }} times in 15 minutes"

      # Update Activity
      - alert: WatchtowerNoRecentActivity
        expr: |
          time() - watchtower_last_check_timestamp > 86400
        for: 2h
        labels:
          severity: warning
          service: watchtower
          category: operations
        annotations:
          summary: "No update checks in 24 hours"
          description: "Last update check was {{ humanizeDuration $value }} ago"

      - alert: WatchtowerUpdateFailures
        expr: |
          increase(watchtower_updates_failed_total[24h]) > 5
        for: 30m
        labels:
          severity: warning
          service: watchtower
          category: operations
        annotations:
          summary: "Multiple container update failures"
          description: "{{ $value }} update failures in the last 24 hours"

      # Resource Usage
      - alert: WatchtowerHighMemoryUsage
        expr: |
          container_memory_usage_bytes{name="watchtower"} > 500 * 1024 * 1024
        for: 10m
        labels:
          severity: warning
          service: watchtower
          category: resources
        annotations:
          summary: "Watchtower high memory usage"
          description: "Using {{ humanize $value }}B memory (threshold: 500MB)"

      # Notification Issues
      - alert: WatchtowerNotificationFailures
        expr: |
          (watchtower_notifications_failed_total / watchtower_notifications_sent_total) > 0.1
        for: 30m
        labels:
          severity: warning
          service: watchtower
          category: notifications
        annotations:
          summary: "Notification delivery failures"
          description: "{{ printf \"%.2f\" (mul $value 100) }}% of notifications failing"

  - name: paperless_ai_alerts
    interval: 30s
    rules:
      # Service Health
      - alert: PaperlessAIDown
        expr: up{job="blackbox", instance="https://paperless-ai.rbnk.uk"} == 0
        for: 5m
        labels:
          severity: critical
          service: paperless-ai
          category: availability
        annotations:
          summary: "Paperless-AI main service is down"
          description: "Service has been unreachable for {{ $value }} minutes"
          runbook_url: "https://docs.rbnk.uk/monitoring/runbooks/paperless-down"

      - alert: PaperlessRAGServiceDown
        expr: |
          up{job="blackbox", instance="http://paperless-ai:8000/health"} == 0
        for: 5m
        labels:
          severity: critical
          service: paperless-ai
          category: availability
          component: rag
        annotations:
          summary: "RAG service is down"
          description: "AI/RAG backend not responding for {{ $value }} minutes"
          impact: "Document search and AI features unavailable"

      # Performance
      - alert: PaperlessHighCPU
        expr: |
          rate(container_cpu_usage_seconds_total{name="paperless-ai"}[5m]) > 0.8
        for: 15m
        labels:
          severity: warning
          service: paperless-ai
          category: performance
        annotations:
          summary: "High CPU usage"
          description: "CPU at {{ printf \"%.2f\" (mul $value 100) }}% for 15 minutes"

      - alert: PaperlessRAGHighMemory
        expr: |
          container_memory_usage_bytes{name="paperless-ai-rag"} > 8 * 1024 * 1024 * 1024
        for: 10m
        labels:
          severity: critical
          service: paperless-ai
          category: resources
          component: rag
        annotations:
          summary: "RAG service high memory usage"
          description: "Using {{ humanize $value }}B memory (threshold: 8GB)"
          action: "Consider restarting RAG service or upgrading memory"

      # Processing Performance
      - alert: PaperlessProcessingQueueBacklog
        expr: paperless_documents_queue_length > 50
        for: 30m
        labels:
          severity: warning
          service: paperless-ai
          category: performance
        annotations:
          summary: "Large document processing backlog"
          description: "{{ $value }} documents waiting in queue"

      - alert: PaperlessOCRSlow
        expr: |
          histogram_quantile(0.95, 
            rate(paperless_ocr_duration_seconds_bucket[5m])
          ) > 60
        for: 20m
        labels:
          severity: warning
          service: paperless-ai
          category: performance
        annotations:
          summary: "OCR processing is slow"
          description: "95th percentile OCR time is {{ humanizeDuration $value }}"

      - alert: PaperlessHighFailureRate
        expr: |
          (rate(paperless_documents_failed_total[1h]) / 
           rate(paperless_documents_processed_total[1h])) > 0.1
        for: 30m
        labels:
          severity: warning
          service: paperless-ai
          category: quality
        annotations:
          summary: "High document processing failure rate"
          description: "{{ printf \"%.2f\" (mul $value 100) }}% failure rate"

      # RAG Performance
      - alert: PaperlessRAGSlowQueries
        expr: |
          histogram_quantile(0.95, 
            rate(paperless_rag_query_duration_seconds_bucket[5m])
          ) > 5
        for: 15m
        labels:
          severity: warning
          service: paperless-ai
          category: performance
          component: rag
        annotations:
          summary: "RAG queries are slow"
          description: "95th percentile query time is {{ humanizeDuration $value }}"

      # Database
      - alert: PaperlessDatabaseConnections
        expr: pg_stat_activity_count{datname="paperless"} > 50
        for: 10m
        labels:
          severity: warning
          service: paperless-ai
          category: database
        annotations:
          summary: "High database connection count"
          description: "{{ $value }} active connections to paperless database"

      - alert: PaperlessDatabaseSize
        expr: pg_database_size_bytes{datname="paperless"} > 10 * 1024 * 1024 * 1024
        for: 30m
        labels:
          severity: info
          service: paperless-ai
          category: database
        annotations:
          summary: "Database size exceeds 10GB"
          description: "Database size is {{ humanize $value }}B"
          action: "Consider archiving old documents"

Alert Routing Configuration

Update /srv/dockerdata/monitoring/alertmanager/alertmanager.yml:

global:
  resolve_timeout: 5m
  smtp_require_tls: false

route:
  group_by: ['alertname', 'service', 'category']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'default'
  routes:
    # Critical alerts - immediate notification
    - match:
        severity: critical
      group_wait: 0s
      repeat_interval: 4h
      receiver: critical-alerts

    # Service-specific routing
    - match:
        service: metube
      receiver: media-alerts
      routes:
        - match:
            category: storage
          receiver: storage-alerts

    - match:
        service: watchtower
      receiver: infrastructure-alerts

    - match:
        service: paperless-ai
      receiver: document-alerts
      routes:
        - match:
            component: rag
          receiver: ai-alerts

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://apprise:8000/notify/alerts'
        send_resolved: true

  - name: 'critical-alerts'
    webhook_configs:
      - url: 'http://ntfy:80/critical-alerts'
        http_config:
          basic_auth:
            username: prometheus
            password: '${NTFY_PROMETHEUS_PASSWORD}'
        send_resolved: true
        title: '🚨 CRITICAL: {{ .GroupLabels.alertname }}'

  - name: 'media-alerts'
    webhook_configs:
      - url: 'http://ntfy:80/metube-alerts'
        http_config:
          basic_auth:
            username: prometheus
            password: '${NTFY_PROMETHEUS_PASSWORD}'
        send_resolved: true

  - name: 'storage-alerts'
    webhook_configs:
      - url: 'http://ntfy:80/storage-alerts'
        http_config:
          basic_auth:
            username: prometheus
            password: '${NTFY_PROMETHEUS_PASSWORD}'
        send_resolved: true
        title: '💾 Storage Alert: {{ .GroupLabels.alertname }}'

  - name: 'infrastructure-alerts'
    webhook_configs:
      - url: 'http://ntfy:80/infrastructure-critical'
        http_config:
          basic_auth:
            username: prometheus
            password: '${NTFY_PROMETHEUS_PASSWORD}'
        send_resolved: true

  - name: 'document-alerts'
    webhook_configs:
      - url: 'http://ntfy:80/paperless-alerts'
        http_config:
          basic_auth:
            username: prometheus
            password: '${NTFY_PROMETHEUS_PASSWORD}'
        send_resolved: true

  - name: 'ai-alerts'
    webhook_configs:
      - url: 'http://ntfy:80/ai-alerts'
        http_config:
          basic_auth:
            username: prometheus
            password: '${NTFY_PROMETHEUS_PASSWORD}'
        send_resolved: true
        title: '🤖 AI Service: {{ .GroupLabels.alertname }}'

# Inhibition rules to prevent alert storms
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['service']

  - source_match:
      alertname: 'PaperlessAIDown'
    target_match_re:
      alertname: 'Paperless.*'
    equal: ['service']

  - source_match:
      alertname: 'WatchtowerNotRunning'
    target_match:
      alertname: 'WatchtowerNoRecentActivity'
    equal: ['service']

Custom Metrics Collection

Metube Metrics Exporter

Create /srv/dockerdata/monitoring/textfile_collector/metube_metrics.sh:

#!/bin/bash
# Metube custom metrics collector

METUBE_DIR="/srv/dockerdata/metube/downloads"
PROM_FILE="/var/lib/prometheus/textfile_collector/metube.prom"

# Count downloads by status
COMPLETED=$(find "$METUBE_DIR" -name "*.mp4" -o -name "*.webm" -o -name "*.mkv" | wc -l)
QUEUED=$(find "$METUBE_DIR" -name "*.part" | wc -l)

# Calculate storage usage
STORAGE_USED=$(du -sb "$METUBE_DIR" | cut -f1)
STORAGE_FILES=$(find "$METUBE_DIR" -type f | wc -l)

# Get largest files
LARGEST_FILE=$(find "$METUBE_DIR" -type f -exec du -b {} + | sort -nr | head -1 | cut -f1)

# Write metrics
cat > "$PROM_FILE" << EOF
# HELP metube_downloads_completed_total Total completed downloads
# TYPE metube_downloads_completed_total counter
metube_downloads_completed_total $COMPLETED

# HELP metube_downloads_queued Current downloads in queue
# TYPE metube_downloads_queued gauge
metube_downloads_queued $QUEUED

# HELP metube_storage_used_bytes Storage used by downloads
# TYPE metube_storage_used_bytes gauge
metube_storage_used_bytes $STORAGE_USED

# HELP metube_total_files Total number of files
# TYPE metube_total_files gauge
metube_total_files $STORAGE_FILES

# HELP metube_largest_file_bytes Size of largest file
# TYPE metube_largest_file_bytes gauge
metube_largest_file_bytes ${LARGEST_FILE:-0}
EOF

# Add to cron for regular updates
# */5 * * * * /srv/dockerdata/monitoring/textfile_collector/metube_metrics.sh

Watchtower Metrics Exporter

Create /srv/dockerdata/monitoring/textfile_collector/watchtower_metrics.sh:

#!/bin/bash
# Watchtower activity metrics from logs

PROM_FILE="/var/lib/prometheus/textfile_collector/watchtower.prom"

# Get last activity timestamp
LAST_LOG=$(docker logs watchtower --since 24h 2>&1 | tail -1)
if [ ! -z "$LAST_LOG" ]; then
  LAST_TIMESTAMP=$(date +%s)
else
  LAST_TIMESTAMP=0
fi

# Count containers monitored
CONTAINERS=$(docker ps --format "{{.Names}}" | grep -v watchtower | wc -l)

# Parse recent updates from logs
UPDATES_24H=$(docker logs watchtower --since 24h 2>&1 | grep -c "Updated")
FAILURES_24H=$(docker logs watchtower --since 24h 2>&1 | grep -c "Failed")

# Write metrics
cat > "$PROM_FILE" << EOF
# HELP watchtower_last_check_timestamp Last time watchtower checked for updates
# TYPE watchtower_last_check_timestamp gauge
watchtower_last_check_timestamp $LAST_TIMESTAMP

# HELP watchtower_containers_monitored Number of containers being monitored
# TYPE watchtower_containers_monitored gauge
watchtower_containers_monitored $CONTAINERS

# HELP watchtower_updates_success_total Successful updates in last 24h
# TYPE watchtower_updates_success_total counter
watchtower_updates_success_total $UPDATES_24H

# HELP watchtower_updates_failed_total Failed updates in last 24h
# TYPE watchtower_updates_failed_total counter
watchtower_updates_failed_total $FAILURES_24H
EOF

Paperless-AI Metrics Exporter

Create /srv/dockerdata/monitoring/textfile_collector/paperless_metrics.py:

#!/usr/bin/env python3
import psycopg2
import os
import json
from datetime import datetime

# Database connection
conn = psycopg2.connect(
    host="paperless-db",
    database="paperless",
    user=os.environ.get("POSTGRES_USER", "paperless"),
    password=os.environ.get("POSTGRES_PASSWORD")
)

metrics = []

try:
    cur = conn.cursor()

    # Document statistics
    cur.execute("SELECT COUNT(*) FROM documents_document")
    total_docs = cur.fetchone()[0]
    metrics.append(f"paperless_documents_total {total_docs}")

    # Processing queue
    cur.execute("SELECT COUNT(*) FROM documents_document WHERE status='processing'")
    queue_length = cur.fetchone()[0]
    metrics.append(f"paperless_documents_queue_length {queue_length}")

    # Failed documents
    cur.execute("SELECT COUNT(*) FROM documents_document WHERE status='failed'")
    failed_docs = cur.fetchone()[0]
    metrics.append(f"paperless_documents_failed_total {failed_docs}")

    # Storage usage
    cur.execute("SELECT SUM(filesize) FROM documents_document")
    storage_bytes = cur.fetchone()[0] or 0
    metrics.append(f"paperless_storage_documents_bytes {storage_bytes}")

    # Tags and correspondents
    cur.execute("SELECT COUNT(*) FROM documents_tag")
    tags_count = cur.fetchone()[0]
    metrics.append(f"paperless_tags_total {tags_count}")

    # Write metrics
    with open("/var/lib/prometheus/textfile_collector/paperless.prom", "w") as f:
        for metric in metrics:
            metric_name = metric.split()[0]
            f.write(f"# TYPE {metric_name} gauge\n")
            f.write(f"{metric}\n")

finally:
    conn.close()

Dashboard JSON Examples

Complete Metube Dashboard JSON

{
  "dashboard": {
    "title": "Metube Media Downloader",
    "uid": "metube-monitoring",
    "tags": ["metube", "media", "downloads"],
    "timezone": "browser",
    "schemaVersion": 38,
    "version": 1,
    "refresh": "30s",
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "panels": [
      {
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "fieldConfig": {
          "defaults": {
            "mappings": [
              {
                "options": {
                  "0": {
                    "color": "red",
                    "index": 1,
                    "text": "OFFLINE"
                  },
                  "1": {
                    "color": "green",
                    "index": 0,
                    "text": "ONLINE"
                  }
                },
                "type": "value"
              }
            ],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                }
              ]
            }
          }
        },
        "gridPos": {
          "h": 4,
          "w": 4,
          "x": 0,
          "y": 0
        },
        "id": 1,
        "options": {
          "colorMode": "background",
          "graphMode": "none",
          "justifyMode": "center",
          "orientation": "auto",
          "reduceOptions": {
            "calcs": ["lastNotNull"],
            "fields": "",
            "values": false
          },
          "textMode": "value"
        },
        "pluginVersion": "9.5.0",
        "targets": [
          {
            "expr": "up{job=\"blackbox\", instance=\"https://metube.rbnk.uk\"}",
            "refId": "A"
          }
        ],
        "title": "Service Status",
        "type": "stat"
      },
      {
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "fieldConfig": {
          "defaults": {
            "color": {
              "mode": "palette-classic"
            },
            "custom": {
              "axisCenteredZero": false,
              "axisColorMode": "text",
              "axisLabel": "",
              "axisPlacement": "auto",
              "barAlignment": 0,
              "drawStyle": "line",
              "fillOpacity": 10,
              "gradientMode": "none",
              "hideFrom": {
                "tooltip": false,
                "viz": false,
                "legend": false
              },
              "lineInterpolation": "smooth",
              "lineWidth": 2,
              "pointSize": 5,
              "scaleDistribution": {
                "type": "linear"
              },
              "showPoints": "never",
              "spanNulls": true,
              "stacking": {
                "group": "A",
                "mode": "none"
              },
              "thresholdsStyle": {
                "mode": "off"
              }
            },
            "unit": "percent"
          }
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 4,
          "y": 0
        },
        "id": 2,
        "options": {
          "legend": {
            "calcs": ["mean", "lastNotNull"],
            "displayMode": "table",
            "placement": "bottom",
            "showLegend": true
          },
          "tooltip": {
            "mode": "multi",
            "sort": "desc"
          }
        },
        "targets": [
          {
            "expr": "rate(container_cpu_usage_seconds_total{name=\"metube\"}[5m]) * 100",
            "legendFormat": "CPU Usage %",
            "refId": "A"
          }
        ],
        "title": "Download Activity (CPU Usage)",
        "type": "timeseries"
      }
    ],
    "templating": {
      "list": [
        {
          "current": {
            "selected": false,
            "text": "Prometheus",
            "value": "prometheus"
          },
          "hide": 0,
          "includeAll": false,
          "label": "Data Source",
          "multi": false,
          "name": "datasource",
          "options": [],
          "query": "prometheus",
          "refresh": 1,
          "regex": "",
          "skipUrlSync": false,
          "type": "datasource"
        }
      ]
    }
  }
}

Watchtower Complete Dashboard

{
  "dashboard": {
    "title": "Watchtower Container Updates",
    "uid": "watchtower-monitoring",
    "tags": ["watchtower", "updates", "containers"],
    "timezone": "browser",
    "panels": [
      {
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "fieldConfig": {
          "defaults": {
            "custom": {
              "align": "auto",
              "displayMode": "auto",
              "inspect": false
            },
            "mappings": [],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                }
              ]
            }
          }
        },
        "gridPos": {
          "h": 8,
          "w": 24,
          "x": 0,
          "y": 0
        },
        "id": 1,
        "options": {
          "showHeader": true,
          "sortBy": [
            {
              "desc": true,
              "displayName": "Last Update"
            }
          ]
        },
        "pluginVersion": "9.5.0",
        "targets": [
          {
            "expr": "sort_desc(container_start_time_seconds{name!=\"watchtower\"})",
            "format": "table",
            "instant": true,
            "refId": "A"
          }
        ],
        "title": "Container Update History",
        "transformations": [
          {
            "id": "organize",
            "options": {
              "excludeByName": {
                "Time": true,
                "__name__": true,
                "job": true,
                "instance": true
              },
              "indexByName": {},
              "renameByName": {
                "Value": "Last Update",
                "container_label_com_docker_compose_project": "Project",
                "container_label_com_docker_compose_service": "Service",
                "image": "Image",
                "name": "Container"
              }
            }
          },
          {
            "id": "convertFieldType",
            "options": {
              "conversions": [
                {
                  "destinationType": "time",
                  "targetField": "Last Update"
                }
              ],
              "fields": {}
            }
          }
        ],
        "type": "table"
      }
    ]
  }
}

Paperless-AI RAG Performance Dashboard

{
  "dashboard": {
    "title": "Paperless-AI Document Intelligence",
    "uid": "paperless-ai-monitoring",
    "tags": ["paperless", "ai", "documents", "rag"],
    "panels": [
      {
        "datasource": {
          "type": "prometheus",
          "uid": "${datasource}"
        },
        "fieldConfig": {
          "defaults": {
            "custom": {
              "hideFrom": {
                "tooltip": false,
                "viz": false,
                "legend": false
              },
              "scaleDistribution": {
                "type": "linear"
              }
            }
          }
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 0,
          "y": 0
        },
        "id": 1,
        "options": {
          "calculate": false,
          "cellGap": 1,
          "color": {
            "exponent": 0.5,
            "fill": "dark-orange",
            "mode": "scheme",
            "reverse": false,
            "scale": "exponential",
            "scheme": "Oranges",
            "steps": 64
          },
          "exemplars": {
            "color": "rgba(255,0,255,0.7)"
          },
          "filterValues": {
            "le": 1e-9
          },
          "legend": {
            "show": true
          },
          "rowsFrame": {
            "layout": "auto"
          },
          "tooltip": {
            "show": true,
            "yHistogram": false
          },
          "yAxis": {
            "axisPlacement": "left",
            "reverse": false,
            "unit": "s"
          }
        },
        "pluginVersion": "9.5.0",
        "targets": [
          {
            "expr": "sum by (le) (rate(paperless_rag_query_duration_seconds_bucket[5m]))",
            "format": "heatmap",
            "refId": "A"
          }
        ],
        "title": "RAG Query Response Time Distribution",
        "type": "heatmap"
      }
    ]
  }
}

Implementation Guide

Step 1: Enable Prometheus Scraping

Add to /srv/dockerdata/monitoring/prometheus/prometheus.yml:

scrape_configs:
  - job_name: 'metube'
    static_configs:
      - targets: ['metube:8081']
    metrics_path: '/metrics'
    scrape_interval: 30s

  - job_name: 'paperless-ai'
    static_configs:
      - targets: ['paperless-ai:3000']
    metrics_path: '/metrics'
    scrape_interval: 30s

  - job_name: 'paperless-rag'
    static_configs:
      - targets: ['localhost:8000']
    metrics_path: '/metrics'
    scrape_interval: 30s

Step 2: Configure Blackbox Exporter

For services without native metrics endpoints:

modules:
  http_2xx_metube:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200, 301, 302]
      follow_redirects: true
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false

Step 3: Create Grafana Dashboards

  1. Import dashboard JSON through Grafana UI
  2. Set appropriate data source (Prometheus)
  3. Configure refresh intervals
  4. Add to dashboard playlists
  5. Set up dashboard links

Step 4: Test Alert Rules

# Reload Prometheus configuration
docker exec prometheus kill -HUP 1

# Check rule syntax
docker exec prometheus promtool check rules /etc/prometheus/alerts/new-services.yml

# Verify alerts in Prometheus UI
curl http://localhost:9090/api/v1/alerts | jq

# Test alert routing
docker exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml

Step 5: Integration with Notification Stack

Configure ntfy topics for each service:

# Create service-specific topics
curl -u admin:${NTFY_ADMIN_PASSWORD} \
  -X PUT https://ntfy.rbnk.uk/metube-alerts

curl -u admin:${NTFY_ADMIN_PASSWORD} \
  -X PUT https://ntfy.rbnk.uk/paperless-alerts

# Subscribe monitoring service account
curl -u monitoring:${NTFY_MONITORING_PASSWORD} \
  -X POST https://ntfy.rbnk.uk/metube-alerts/subscribe

Best Practices

  1. Dashboard Organization
  2. Group related panels together
  3. Use consistent color schemes
  4. Add helpful descriptions to panels

  5. Use variables for flexibility

  6. Alert Tuning

  7. Start with conservative thresholds
  8. Monitor for false positives
  9. Adjust based on actual usage patterns

  10. Document threshold reasoning

  11. Performance Optimization

  12. Limit query time ranges
  13. Use recording rules for complex queries
  14. Aggregate data appropriately

  15. Cache dashboard queries

  16. Maintenance

  17. Regular dashboard review
  18. Update queries as services evolve
  19. Archive unused dashboards
  20. Document custom panels

Troubleshooting

Common Issues

  1. No data in Grafana
  2. Verify Prometheus is scraping the target
  3. Check service is exposing metrics
  4. Ensure network connectivity

  5. Validate query syntax

  6. Alerts not firing

  7. Check alert rule evaluation in Prometheus
  8. Verify AlertManager is receiving alerts
  9. Test notification channel connectivity

  10. Review alert routing rules

  11. High cardinality warnings

  12. Review label usage in queries
  13. Implement recording rules
  14. Adjust scrape intervals
  15. Use metric relabeling