Health Check Endpoint Documentation

Overview

This document provides comprehensive health check information for infrastructure services, including available endpoints, Docker health checks, monitoring integration, and troubleshooting guidance.

Service Health Checks

Metube (YouTube Downloader)

Service URL: https://metube.rbnk.uk Container Port: 8081

Available Health Check Methods

1. HTTP Endpoint Check
2. Endpoint: https://metube.rbnk.uk/ (Main web interface)
3. Method: GET
4. Expected Response: HTTP 200 with HTML content

5. Response Content: Web interface should load successfully

6. Docker Health Check

   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:8081/"]
     interval: 30s
     timeout: 10s
     retries: 3
     start_period: 40s
   

7. Alternative Health Check (without curl)

   healthcheck:
     test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8081/"]
     interval: 30s
     timeout: 10s
     retries: 3
     start_period: 40s
   

Integration with Monitoring Systems

Prometheus Blackbox Exporter:

# Add to prometheus.yml
- job_name: 'metube-health'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - https://metube.rbnk.uk
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115

Uptime Kuma Configuration: - Monitor Type: HTTP(s) - URL: https://metube.rbnk.uk - Interval: 60 seconds - Expected Status Code: 200

Example curl Commands

# Basic availability check
curl -f -s -o /dev/null -w "%{http_code}" https://metube.rbnk.uk

# Detailed response check
curl -I https://metube.rbnk.uk

# Internal container check (from within Docker network)
curl -f http://metube:8081/

Common Failure Scenarios

• 502 Bad Gateway: Container is down or Traefik cannot reach it
• 503 Service Unavailable: Container is starting up or overloaded
• Connection Timeout: Network issues or container hanging
• SSL Certificate Errors: Traefik SSL configuration issues

---

Watchtower (Container Updater)

Service: Background service (no web interface) Container: watchtower

Available Health Check Methods

1. Docker Native Health Check

   healthcheck:
     test: ["CMD", "/watchtower", "--health-check"]
     interval: 600s  # 10 minutes - longer interval for background service
     timeout: 30s
     retries: 3
     start_period: 60s
   

2. Process-based Health Check

   healthcheck:
     test: ["CMD", "pgrep", "-f", "watchtower"]
     interval: 300s
     timeout: 10s
     retries: 3
   

3. Log-based Health Check Script

   #!/bin/bash
   # Check if watchtower has logged recently (within last 30 minutes)
   if docker logs watchtower --since 30m 2>&1 | grep -q "watchtower"; then
     exit 0
   else
     exit 1
   fi
   

Integration with Monitoring Systems

Prometheus Container Metrics (via cAdvisor):

# Monitor container status
container_last_seen{name="watchtower"}

# Monitor container restarts
increase(container_start_time_seconds{name="watchtower"}[1h])

Custom Exporter Script (/textfile_collector/watchtower_health.prom):

#!/bin/bash
# Generate metrics for Prometheus textfile collector
echo "# HELP watchtower_container_running Watchtower container status"
echo "# TYPE watchtower_container_running gauge"
if docker ps --filter "name=watchtower" --filter "status=running" | grep -q watchtower; then
  echo "watchtower_container_running 1"
else
  echo "watchtower_container_running 0"
fi

# Check last activity from logs
LAST_LOG_TIME=$(docker logs watchtower --since 24h 2>&1 | tail -1 | grep -o '[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}T[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}' | head -1)
if [ ! -z "$LAST_LOG_TIME" ]; then
  TIMESTAMP=$(date -d "$LAST_LOG_TIME" +%s)
  NOW=$(date +%s)
  AGE=$((NOW - TIMESTAMP))
  echo "# HELP watchtower_last_activity_seconds Time since last watchtower activity"
  echo "# TYPE watchtower_last_activity_seconds gauge"
  echo "watchtower_last_activity_seconds $AGE"
fi

Example Monitoring Commands

# Check container status
docker ps --filter "name=watchtower" --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"

# View recent logs
docker logs watchtower --since 1h

# Check if watchtower is responding to health check
docker exec watchtower /watchtower --health-check
echo $?  # Should return 0 if healthy

# Monitor container restart count
docker inspect watchtower --format='{{.RestartCount}}'

Common Failure Scenarios

• Container Exit Code 1: Configuration error or permission issues
• Container Constantly Restarting: Docker socket permission problems
• No Recent Log Activity: Watchtower may be stuck or sleeping
• Email Notification Failures: SMTP configuration issues

---

Paperless-AI (Document Analyzer)

Service URL: https://paperless-ai.rbnk.uk Container Port: 3000

Available Health Check Methods

1. HTTP Endpoint Check
2. Endpoint: https://paperless-ai.rbnk.uk/
3. Method: GET
4. Expected Response: HTTP 200 with application interface

5. Response Content: Web application should load successfully

6. API Health Check (if available)

7. Endpoint: https://paperless-ai.rbnk.uk/health (may not exist)

8. Alternative: https://paperless-ai.rbnk.uk/api/status

9. Docker Health Check

   healthcheck:
     test: ["CMD", "curl", "-f", "http://localhost:3000/"]
     interval: 30s
     timeout: 15s  # Longer timeout for AI processing
     retries: 3
     start_period: 60s  # AI initialization takes time
   

10. Advanced Health Check with Dependencies

    healthcheck:
      test: |
        curl -f http://localhost:3000/ && \
        curl -f http://localhost:8000/ || exit 1  # Check RAG service if enabled
      interval: 60s
      timeout: 20s
      retries: 3
      start_period: 120s
    

Integration with Monitoring Systems

Prometheus Blackbox Exporter:

# Add to prometheus.yml
- job_name: 'paperless-ai-health'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - https://paperless-ai.rbnk.uk
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115

Custom AI Service Monitor:

#!/bin/bash
# Monitor RAG service availability
echo "# HELP paperless_ai_rag_service_available RAG service availability"
echo "# TYPE paperless_ai_rag_service_available gauge"

if curl -f -s -m 10 http://paperless-ai:8000/ >/dev/null 2>&1; then
  echo "paperless_ai_rag_service_available 1"
else
  echo "paperless_ai_rag_service_available 0"
fi

# Monitor main service
echo "# HELP paperless_ai_main_service_available Main service availability"
echo "# TYPE paperless_ai_main_service_available gauge"

if curl -f -s -m 10 http://paperless-ai:3000/ >/dev/null 2>&1; then
  echo "paperless_ai_main_service_available 1"
else
  echo "paperless_ai_main_service_available 0"
fi

Example curl Commands

# Basic availability check
curl -f -s -o /dev/null -w "%{http_code}" https://paperless-ai.rbnk.uk

# Check with timeout (AI services can be slow)
curl -f -s -m 20 https://paperless-ai.rbnk.uk

# Internal container checks
curl -f http://paperless-ai:3000/
curl -f http://paperless-ai:8000/  # RAG service if enabled

# Verbose connection info
curl -v -I https://paperless-ai.rbnk.uk

Common Failure Scenarios

• Long Response Times: AI processing causing timeouts
• RAG Service Unavailable: Backend AI service not responding
• Memory Issues: High memory usage from AI models
• Model Loading Failures: AI models not properly initialized
• API Key Issues: Invalid or expired OpenAI/Ollama API keys

---

Global Monitoring Configuration

Prometheus Alert Rules

# Add to alerts-enhanced.yml
groups:
  - name: service_health_checks
    rules:
      - alert: ServiceDown
        expr: probe_success == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been down for more than 2 minutes"

      - alert: ServiceSlowResponse
        expr: probe_duration_seconds > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Service {{ $labels.instance }} is responding slowly"
          description: "{{ $labels.instance }} response time is {{ $value }}s"

      - alert: ContainerUnhealthy
        expr: container_health_status != 1
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} is unhealthy"
          description: "Container health check is failing"

Grafana Dashboard Queries

# Service availability percentage
avg_over_time(probe_success[24h]) * 100

# Response time 95th percentile
histogram_quantile(0.95, probe_duration_seconds_bucket)

# Container health status
container_health_status{name=~"metube|watchtower|paperless-ai"}

# Service uptime
up{job=~"blackbox|.*-health"}

Uptime Kuma Integration

Notification Channels: - ntfy: https://ntfy.rbnk.uk/monitoring-critical - Apprise: https://apprise.rbnk.uk/notify/apprise

Monitor Configuration Template:

{
  "name": "Service Health Check",
  "type": "http",
  "url": "https://service.rbnk.uk",
  "interval": 60,
  "retryInterval": 60,
  "resendInterval": 1440,
  "maxretries": 3,
  "timeout": 30,
  "ignoreTls": false,
  "upsideDown": false,
  "maxredirects": 10,
  "accepted_statuscodes": ["200-299"],
  "dns_resolve_type": "A",
  "dns_resolve_server": "1.1.1.1",
  "notificationIDList": []
}

Troubleshooting Guide

General Health Check Issues

1. Timeout Errors

   # Increase timeout values
   # Check network connectivity
   docker network inspect traefik_proxy
   
   # Test internal connectivity
   docker exec prometheus curl -v http://service:port/
   

2. SSL Certificate Issues

   # Check certificate status
   curl -vI https://service.rbnk.uk 2>&1 | grep -E "(SSL|certificate|TLS)"
   
   # Test without SSL verification
   curl -k https://service.rbnk.uk
   

3. Container Health Check Failures

   # Check container logs
   docker logs container-name --tail 50
   
   # Inspect health check command
   docker inspect container-name | jq '.[0].Config.Healthcheck'
   
   # Manual health check execution
   docker exec container-name /health-check-command
   

4. DNS Resolution Issues

   # Test DNS resolution
   nslookup service.rbnk.uk
   
   # Check from container perspective
   docker exec monitoring-container nslookup service.rbnk.uk
   

Service-Specific Troubleshooting

Metube: - Check download directory permissions - Verify yt-dlp functionality: docker exec metube yt-dlp --version

Watchtower: - Verify Docker socket access: docker exec watchtower ls -la /var/run/docker.sock - Check email configuration for notifications

Paperless-AI: - Monitor memory usage: docker stats paperless-ai - Check AI model initialization in logs - Verify API key configuration

Best Practices

1. Health Check Intervals
2. Critical services: 30-60 seconds
3. Background services: 5-10 minutes

4. AI services: 60-120 seconds (longer processing times)

5. Timeout Configuration

6. Web services: 10-15 seconds
7. AI services: 20-30 seconds

8. Background services: 30-60 seconds

9. Retry Logic

10. Minimum 3 retries before marking as failed
11. Exponential backoff for external monitoring

12. Different retry counts for different service types

13. Alerting Thresholds

14. Immediate alerts for critical services
15. Grouped alerts to prevent spam
16. Different severity levels based on service importance

---

For additional monitoring configuration examples, see: - Monitoring Operations Guide - Prometheus Configuration - Grafana Dashboards