Monitoring Stack Documentation¶

Overview¶

The monitoring stack provides comprehensive observability for the entire Docker infrastructure, enabling proactive monitoring, alerting, and troubleshooting. It combines industry-standard tools to collect metrics, aggregate logs, visualize data, and manage alerts.

Architecture¶

graph TB
    subgraph "Data Collection"
        NE[Node Exporter<br/>System Metrics]
        CA[cAdvisor<br/>Container Metrics]
        PE[PostgreSQL Exporter<br/>Database Metrics]
        BE[Blackbox Exporter<br/>Endpoint Monitoring]
        PT[Promtail<br/>Log Collection]
    end

    subgraph "Storage & Processing"
        P[Prometheus<br/>Metrics Storage]
        L[Loki<br/>Log Storage]
    end

    subgraph "Visualization & Alerting"
        G[Grafana<br/>Dashboards]
        AM[AlertManager<br/>Alert Routing]
    end

    NE --> P
    CA --> P
    PE --> P
    BE --> P
    PT --> L

    P --> G
    L --> G
    P --> AM

    AM --> E[Email/Discord/Webhook]

Components¶

Core Services¶

Prometheus (https://prometheus.rbnk.uk)¶

Purpose: Time-series database for metrics storage and querying
Features:
Collects metrics from various exporters
Evaluates alert rules
Provides PromQL query language
30-day retention by default
Access: Protected by Traefik authentication

Grafana (https://grafana.rbnk.uk)¶

Purpose: Visualization platform for metrics and logs
Features:
Pre-configured dashboards
Custom dashboard creation
Multiple data sources (Prometheus, Loki)
Alert visualization
Default Credentials: admin / (configured in .env)

Loki¶

Purpose: Log aggregation system
Features:
Collects logs from all containers
Indexes metadata, not log content (cost-effective)
Integrates with Grafana for visualization
30-day retention

AlertManager¶

Purpose: Alert routing and management
Features:
Groups and deduplicates alerts
Routes alerts based on severity
Supports multiple notification channels
Silence and inhibition rules

Data Collectors¶

Node Exporter¶

Collects host-level metrics:
CPU usage and load
Memory statistics
Disk I/O and space
Network statistics
System information

cAdvisor¶

Collects container-level metrics:
Container CPU usage
Memory consumption
Network I/O per container
Disk usage per container

PostgreSQL Exporter¶

Monitors Supabase database:
Connection statistics
Database sizes
Query performance
Replication status
Lock information

Blackbox Exporter¶

Monitors service endpoints:
HTTP/HTTPS availability
Response times
SSL certificate expiry
DNS resolution

Promtail¶

Log collection agent:
Tails container logs
Adds metadata labels
Ships to Loki
Handles log rotation

Access and Usage¶

Accessing Services¶

Grafana Dashboard: https://grafana.rbnk.uk
Main entry point for monitoring
View dashboards, explore metrics, search logs
Prometheus UI: https://prometheus.rbnk.uk
Direct metric queries
Target health status
Alert state viewing

Common Tasks¶

Viewing System Health¶

Open Grafana: https://grafana.rbnk.uk
Navigate to Dashboards → Infrastructure Overview
Key panels to check:
System resource usage
Container status
Service availability
Recent alerts

Monitoring PostgreSQL Database¶

Open Grafana: https://grafana.rbnk.uk
Navigate to Dashboards → PostgreSQL Database
Monitor key database metrics:
Active/idle connections
Query performance (SELECT, INSERT, UPDATE, DELETE)
Cache hit rates (should be > 90%)
Database sizes and growth
Lock contention
Checkpoint and WAL statistics

Searching Logs¶

In Grafana, go to Explore
Select Loki as data source

Use LogQL queries:

# All logs from a specific container
{container_name="supa-db"}

# Error logs across all services
{job="containerlogs"} |= "ERROR"

# Logs from last hour with specific text
{container_name="traefik"} |= "404" | __timestamp__ > (now() - 1h)

Creating Custom Dashboards¶

Click + → Create → Dashboard

Add panels with PromQL queries:

# Container memory usage
container_memory_usage_bytes{name!~"POD|k8s_.*"}

# HTTP request rate
rate(traefik_service_requests_total[5m])

# Database connections
pg_stat_database_numbackends

Managing Alerts¶

View active alerts: Alerting → Alert rules
Silence alerts: Alerting → Silences
Configure contact points: Alerting → Contact points

Pre-configured Dashboards¶

Infrastructure Overview¶

System resource utilization
Container health status
Network traffic overview
Disk usage trends

Service Health¶

Service availability status
Response time percentiles
Error rates by service
SSL certificate status

Resource Optimization¶

Container resource usage ranking
Memory and CPU trends
Disk I/O patterns
Network bandwidth usage

Security Monitoring¶

Failed authentication attempts
Unusual network activity
Container restart patterns
Access logs analysis

PostgreSQL Database (ID: 9628)¶

Database connections and activity
Query performance and slow queries
Cache hit ratios
Database size and growth
Transaction rates (commits/rollbacks)
Lock statistics and deadlocks
Checkpoint performance
Background writer statistics
Temporary file usage

Alert Management¶

Alert Severity Levels¶

Critical: Immediate action required
Service down
Database unreachable
Disk space < 10%
Backup failure
Warning: Investigation needed
High resource usage
Slow queries
Certificate expiring soon
Container restarts
Info: Informational only
Scheduled maintenance
Non-critical updates
Usage statistics

Notification Channels¶

AlertManager integrates with the notification stack for flexible alert routing:

Primary Integration - Apprise: - AlertManager sends webhooks to http://apprise:8000/notify/apprise - Apprise distributes to configured channels (WhatsApp, Telegram, Email, Discord, etc.) - Tag-based routing allows selective notifications

Configuration Example:

# alertmanager.yml
receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://apprise:8000/notify/apprise'
        send_resolved: true

  - name: 'critical'
    webhook_configs:
      - url: 'http://apprise:8000/notify/apprise?tag=critical,oncall'
        send_resolved: true

Available Channels via Apprise: - WhatsApp: Via WhatsApp Business API - Telegram: Direct messages and group notifications - Email: SMTP-based email notifications - Discord: Webhooks to channels - ntfy: Push notifications to mobile/desktop - Slack: Workspace notifications - Custom: Any service supported by Apprise

Common Alert Scenarios¶

High Memory Usage¶

Alert: HighMemoryUsage
Severity: warning
Action: 
1. Check Grafana memory dashboard
2. Identify memory-consuming containers
3. Consider scaling or optimization

Service Down¶

Alert: ServiceDown
Severity: critical
Action:
1. Check container status: docker ps
2. Review logs: docker logs [container]
3. Restart if needed: docker compose restart [service]

Disk Space Low¶

Alert: DiskSpaceLow
Severity: critical
Action:
1. Check disk usage: df -h
2. Clean Docker: docker system prune -a
3. Review backup retention
4. Consider disk expansion

Monitoring Best Practices¶

Dashboard Usage¶

Start with Overview: Always check the Infrastructure Overview first
Drill Down: Use detailed dashboards for specific issues
Time Ranges: Adjust time ranges to identify patterns
Annotations: Mark deployments and changes

Query Optimization¶

Use Recording Rules: For frequently-used complex queries
Limit Time Ranges: Avoid queries over large time spans
Use Aggregations: Reduce data points with avg(), max(), etc.
Label Filtering: Be specific with label selectors

Alert Tuning¶

Avoid Alert Fatigue: Set appropriate thresholds
Use Time Windows: Require conditions to persist (e.g., "for: 5m")
Group Related Alerts: Use inhibition rules
Document Runbooks: Link alerts to troubleshooting guides

Maintenance Tasks¶

Daily¶

Check Grafana dashboard for anomalies
Review any triggered alerts
Verify backup job completion
Ensure all monitoring services are running

Weekly¶

Review resource usage trends
Check for security-related alerts
Update dashboards based on needs
Review and acknowledge resolved alerts
Verify permission fix script is accessible

Monthly¶

Analyze long-term trends
Capacity planning review
Alert rule optimization
Dashboard cleanup
Review Watchtower exclusions for monitoring stack

After System Updates¶

Run permission fix script if monitoring services fail to start
Verify AlertManager configuration is valid
Check that Watchtower exclusions are maintained
Test alert delivery to n8n webhooks

Troubleshooting¶

Permission Issues¶

Problem: Services fail to start with permission denied errors

Solution: Use the permission fix script:

/srv/dockerdata/monitoring/fix-permissions.sh

Details: The monitoring stack runs with specific user IDs: - Prometheus: User 65534 (nobody) - Grafana: User 472 - Loki: User 10001 - AlertManager: User 65534 (nobody)

These user IDs must own their respective Docker volume data directories. The fix script automatically corrects ownership and permissions.

Prevention: Watchtower is excluded from auto-updating monitoring containers to prevent permission issues:

# Each monitoring service includes:
labels:
  - "com.centurylinklabs.watchtower.enable=false"

AlertManager Configuration Issues¶

Problem: AlertManager fails to start with "time interval crossing midnight" errors

Root Cause: Complex time interval configurations can cause parsing issues when intervals cross midnight boundaries.

Solution: Use simplified, minimal configuration: - Avoid complex time-based routing - Use simple repeat intervals (1h, 30m, etc.) - Test configuration with: docker exec alertmanager amtool config check /etc/alertmanager/config.yml

Current Configuration: Simplified to basic webhook routing with n8n integration - Default receiver: Basic webhook placeholder - N8N webhook: Routes all severity levels to n8n automation workflows - Simple inhibition rules prevent alert spam

Metrics Not Appearing¶

Check exporter health:
```
docker ps | grep exporter
```
Verify Prometheus targets:
Visit https://prometheus.rbnk.uk/targets
All targets should show "UP"

Test exporter directly:

# Node exporter
curl http://localhost:9100/metrics

# cAdvisor
curl http://localhost:8080/metrics

Grafana Issues¶

Dashboard not loading:
Check datasource configuration
Verify Prometheus is accessible
Review browser console for errors
Verify Grafana permissions (user 472)
No data points:
Verify time range selection
Check query syntax
Ensure metrics exist in Prometheus
Permission denied on startup:
Run permission fix script: /srv/dockerdata/monitoring/fix-permissions.sh
Check volume ownership: ls -la /srv/dockerdata/docker-root/volumes/monitoring_grafana_data/_data

Alert Not Firing¶

Check alert configuration:

docker exec prometheus promtool check rules /etc/prometheus/alerts.yml

Verify alert state in Prometheus:
Visit https://prometheus.rbnk.uk/alerts
Check AlertManager:
```
docker logs alertmanager
```
AlertManager configuration errors:
Check for time interval issues
Validate config: docker exec alertmanager amtool config check
Review webhook connectivity to n8n

High Resource Usage¶

Prometheus:
Reduce retention period
Increase scrape intervals
Remove unused metrics
Grafana:
Limit dashboard refresh rates
Optimize queries
Cache dashboard results
Loki:
Reduce log retention
Filter unnecessary logs
Compress old data

Security Considerations¶

Authentication:
Grafana uses its own authentication
Prometheus protected by Traefik auth
Change default passwords
Network Isolation:
Monitoring network separate from application networks
Only Grafana and Prometheus exposed externally
Data Sensitivity:
Metrics may contain sensitive information
Logs should be filtered for secrets
Use appropriate retention policies
Container Security:
Monitoring containers run with specific non-root user IDs
Watchtower auto-updates disabled to prevent permission issues
Volume permissions strictly controlled via fix script
Alert Security:
AlertManager webhooks use HTTPS for n8n integration
Webhook tokens should be configured for production
Alert content filtered to avoid sensitive data exposure

Integration Guide¶

Adding New Service Metrics¶

For services with Prometheus support:

# In service's docker-compose.yml
services:
  myservice:
    labels:
      - "prometheus.io/scrape=true"
      - "prometheus.io/port=8080"
      - "prometheus.io/path=/metrics"

For custom metrics:

# In prometheus/prometheus.yml
scrape_configs:
  - job_name: 'myservice'
    static_configs:
      - targets: ['myservice:8080']

Creating Service-Specific Dashboards¶

Design dashboard in Grafana UI
Export as JSON
Save to: monitoring/grafana/dashboards/
Restart Grafana to auto-import

Adding Custom Alerts¶

Edit monitoring/prometheus/alerts.yml

Add alert rule:

- alert: MyServiceDown
  expr: up{job="myservice"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "MyService is down"

Restart Prometheus

Resource Requirements¶

Total Stack: ~2-3GB RAM
Disk Space: ~5-10GB for 30-day retention
CPU: Minimal during normal operation

Per-Component Requirements¶

Component	RAM	CPU	Disk
Prometheus	1GB	Low	2-5GB
Grafana	200MB	Low	100MB
Loki	500MB	Low	2-5GB
AlertManager	50MB	Minimal	50MB
Exporters	50-100MB each	Minimal	Minimal

Backup and Recovery¶

What to Backup¶

Grafana:
Dashboards (auto-provisioned)
Data sources (auto-provisioned)
User preferences (in volume)
Prometheus:
Metric data (optional, can be regenerated)
Alert rules (in config files)
Configuration:
All YAML files in monitoring/
Environment variables (.env)

Backup Script¶

The monitoring stack is included in the infrastructure backup:

/srv/dockerdata/_scripts/monitoring-backup.sh

Recovery Process¶

Restore configuration files
Restore Docker volumes (if needed)
Restart monitoring stack
Verify all exporters are connected