Skip to content

Prometheus Documentation

Overview

Prometheus is the metrics collection and storage backend for the monitoring stack. It scrapes metrics from various exporters and services, stores them in a time-series database, and provides a query language (PromQL) for data analysis.

Configuration

  • URL: https://prometheus.rbnk.uk
  • Internal Port: 9090
  • User ID: 65534 (nobody)
  • Data Directory: /srv/dockerdata/docker-root/volumes/monitoring_prometheus_data/_data

Current State

The Prometheus installation is fully operational with the following key features:

Data Collection

  • Node Exporter: System-level metrics (CPU, memory, disk, network)
  • cAdvisor: Container-level metrics (Docker container resource usage)
  • PostgreSQL Exporter: Database metrics from Supabase
  • Blackbox Exporter: HTTP/HTTPS endpoint monitoring
  • Promtail: Log collection agent for Loki integration

Configuration Files

  • Main Config: /srv/dockerdata/monitoring/prometheus/prometheus.yml
  • Alert Rules: /srv/dockerdata/monitoring/prometheus/alerts.yml
  • Scrape Interval: 15 seconds (configurable)
  • Retention Period: 30 days (default)

Permission Fix Applied

Issue Resolved: Prometheus was failing to start due to permission denied errors accessing its data directory.

Solution Implemented: - Created permission fix script at /srv/dockerdata/monitoring/fix-permissions.sh - Prometheus runs as user 65534 (nobody) - Data directory ownership corrected: chown -R 65534:65534 /prometheus/data - Permissions set to 755 for proper read/write access

Prevention: - Watchtower auto-updates disabled for monitoring stack - Label added: com.centurylinklabs.watchtower.enable=false - Permission fix script available for future use if needed

Usage

Accessing Prometheus

  1. Navigate to https://prometheus.rbnk.uk
  2. Authentication handled by Traefik middleware
  3. Use the web UI for:
  4. Query building and testing
  5. Target health monitoring
  6. Alert rule status
  7. Configuration validation

Common PromQL Queries

# System CPU usage
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage  
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Container memory usage
container_memory_usage_bytes{name!~"POD|k8s_.*"}

# Disk space remaining
node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100

# HTTP request rate by service
rate(traefik_service_requests_total[5m])

# Database connections
pg_stat_database_numbackends

Target Health

Monitor scrape target status at: - URL: https://prometheus.rbnk.uk/targets - Healthy Targets: Should show "UP" status - Unhealthy Targets: Will show error messages and last scrape time

Alert Rules

Current alert rules cover:

System Alerts

  • HighCPUUsage: CPU > 80% for 5 minutes
  • HighMemoryUsage: Memory > 85% for 5 minutes
  • DiskSpaceLow: Disk < 15% remaining for 5 minutes

Container Alerts

  • ContainerDown: Container not responding for 1 minute
  • ContainerHighCPU: Container CPU > 80% for 5 minutes
  • ContainerHighMemory: Container memory > 90% limit for 5 minutes

Database Alerts

  • PostgresDown: Database unreachable for 1 minute
  • PostgresHighConnections: Connections > 80% of max for 5 minutes
  • PostgresDeadlocks: Deadlocks detected in 5-minute window

Backup Alerts

  • BackupFailed: No successful backup in 24 hours
  • InfrastructureBackupMissing: Infrastructure backup missing 48+ hours
  • CloudBackupMissing: Cloud sync missing 48+ hours

Troubleshooting

Permission Issues

If Prometheus fails to start with permission errors:

# Run the permission fix script
/srv/dockerdata/monitoring/fix-permissions.sh

# Verify ownership
ls -la /srv/dockerdata/docker-root/volumes/monitoring_prometheus_data/_data

# Check container logs
docker logs prometheus

Configuration Validation

# Check configuration syntax
docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

# Validate alert rules
docker exec prometheus promtool check rules /etc/prometheus/alerts.yml

# Test alert rule queries
docker exec prometheus promtool query instant 'up'

Performance Issues

# Check resource usage
docker stats prometheus

# Monitor scrape duration
# Query in Prometheus UI: prometheus_target_scrape_duration_seconds

# Check for slow targets
# Query: increase(prometheus_target_scrapes_exceeded_sample_limit_total[5m]) > 0

Storage Issues

# Check disk usage
du -sh /srv/dockerdata/docker-root/volumes/monitoring_prometheus_data/_data

# Monitor storage growth
# Query: prometheus_tsdb_data_replay_duration_seconds

# Check for storage errors in logs
docker logs prometheus | grep -i storage

Integration with n8n

Prometheus integrates with the n8n automation platform through AlertManager:

  1. Alert Rules: Defined in Prometheus, evaluated every 15 seconds
  2. AlertManager: Routes alerts to n8n webhooks based on severity
  3. n8n Workflows: Process alerts and trigger appropriate notifications
  4. Notification Stack: Final delivery via Apprise, ntfy, and other channels

Maintenance

Regular Tasks

  • Daily: Check target health and recent alerts
  • Weekly: Review storage usage and performance metrics
  • Monthly: Analyze long-term trends and optimize retention

Configuration Changes

  1. Edit configuration files in /srv/dockerdata/monitoring/prometheus/
  2. Validate changes: promtool check config prometheus.yml
  3. Restart Prometheus: docker compose -f monitoring/docker-compose.yml restart prometheus
  4. Verify targets are healthy after restart

Backup Considerations

  • Configuration: Backed up as part of infrastructure backup
  • Data: Time-series data can be regenerated, but consider backing up for historical analysis
  • Alert Rules: Critical to backup - contain all monitoring logic

Best Practices

  1. Query Optimization: Use appropriate time ranges and aggregations
  2. Alert Tuning: Set realistic thresholds to avoid alert fatigue
  3. Label Management: Consistent labeling across all metrics
  4. Resource Monitoring: Monitor Prometheus's own resource usage
  5. Security: Keep authentication enabled and restrict network access

Future Enhancements

  • Recording Rules: Pre-calculate complex queries for better performance
  • Federation: Consider multi-instance setup for scaling
  • Remote Storage: Evaluate long-term storage solutions
  • Custom Exporters: Add application-specific metrics collection