Grafana Documentation¶

Overview¶

Grafana is the visualization platform for the monitoring stack, providing dashboards and alerting capabilities. It connects to Prometheus and Loki data sources to create comprehensive monitoring dashboards.

Configuration¶

URL: https://grafana.rbnk.uk
Internal Port: 3000
User ID: 472
Data Directory: /srv/dockerdata/docker-root/volumes/monitoring_grafana_data/_data
Default Credentials: admin / (configured in .env)

Current State¶

The Grafana installation is fully operational with permission fixes applied and comprehensive dashboard coverage.

Data Sources¶

Prometheus: Primary metrics data source (http://prometheus:9090)
Loki: Log aggregation data source (http://loki:3100)
PostgreSQL: Direct database connection for Supabase metrics

Dashboard Categories¶

Infrastructure Overview: System resource utilization and health
Service Health: Application-specific monitoring and availability
Resource Optimization: Performance analysis and capacity planning
Security Monitoring: Security events and access patterns
PostgreSQL Database: Comprehensive database monitoring
Rclone Backup Monitoring: Backup job status and storage metrics

Permission Fix Applied¶

Issue Resolved: Grafana was failing to start due to permission denied errors accessing its data directory.

Solution Implemented: - Created permission fix script at /srv/dockerdata/monitoring/fix-permissions.sh - Grafana runs as user 472 (grafana user in the container) - Data directory ownership corrected: chown -R 472:472 /var/lib/grafana - Permissions set to 755 for proper read/write access

Prevention: - Watchtower auto-updates disabled for monitoring stack - Label added: com.centurylinklabs.watchtower.enable=false - Permission fix script available for future use if needed

Dashboard Management¶

Pre-configured Dashboards¶

Infrastructure Overview: - System resource utilization (CPU, memory, disk, network) - Container health status and resource usage - Network traffic overview and patterns - Disk usage trends and forecasting

Service Health: - Service availability status across all infrastructure - Response time percentiles (p50, p95, p99) - Error rates by service with trending - SSL certificate status and expiry tracking

PostgreSQL Database (Dashboard ID: 9628): - Database connections and activity monitoring - Query performance analysis and slow query identification - Cache hit ratios and buffer statistics - Database size growth and storage utilization - Transaction rates (commits/rollbacks) and deadlock detection - Lock statistics and contention analysis - Checkpoint performance and WAL activity - Background writer statistics and temporary file usage

Rclone Backup Monitoring: - Infrastructure and cloud backup status - R2 storage usage with visual gauges - Storage distribution across cloud providers - Backup error tracking and trend analysis - Storage growth trends over time

Dashboard Provisioning¶

Dashboards are automatically provisioned from:

/srv/dockerdata/monitoring/grafana/provisioning/dashboards/

Adding New Dashboards: 1. Create dashboard in Grafana UI 2. Export as JSON 3. Save to provisioning directory 4. Restart Grafana to auto-import

Custom Dashboard Creation¶

Best Practices: - Use templating for dynamic service selection - Implement proper time range controls - Add appropriate units and thresholds - Include helpful descriptions and documentation - Set reasonable refresh intervals (30s-5m)

Common Panel Types: - Stat Panels: Current values with sparklines - Time Series: Trend analysis over time - Gauge Panels: Visual threshold monitoring - Heatmaps: Distribution analysis - Tables: Detailed metric breakdowns

Alerting¶

Grafana alerting is integrated with the broader notification stack:

Alert Rules¶

Database Connectivity: Monitors PostgreSQL connection health
Service Availability: Tracks HTTP endpoint response codes
Resource Thresholds: CPU, memory, and disk usage limits
Backup Status: Monitors backup job completion and errors

Notification Channels¶

Primary: Routes through AlertManager to n8n workflows
Secondary: Direct integration with Apprise notification stack
Mobile: Push notifications via ntfy service
Email: SMTP integration for critical alerts

Usage¶

Accessing Grafana¶

Navigate to https://grafana.rbnk.uk
Login with configured credentials (admin / password from .env)
Default landing page shows Infrastructure Overview

Dashboards: Browse by folder (Infrastructure, Applications, etc.)
Explore: Ad-hoc querying of Prometheus and Loki data
Alerting: Manage alert rules and notification channels
Administration: User management, data sources, plugins

Log Analysis with Loki¶

Access via Explore section:

# View all logs from a specific container
{container_name="supa-db"}

# Filter for error messages across all services  
{job="containerlogs"} |= "ERROR"

# Search for specific patterns in the last hour
{container_name="traefik"} |= "404" | __timestamp__ > (now() - 1h)

# Analyze log patterns with metrics
rate({job="containerlogs"} |= "error" [5m])

Custom Queries¶

Prometheus (PromQL):

# Container memory usage by service
container_memory_usage_bytes{name!~"POD|k8s_.*"}

# HTTP request rate with status codes
rate(traefik_service_requests_total[5m])

# Database connection utilization
pg_stat_database_numbackends / pg_settings_max_connections * 100

# Backup status monitoring  
time() - rclone_infrastructure_backup_last_success_timestamp

Troubleshooting¶

Permission Issues¶

If Grafana fails to start with permission errors:

# Run the permission fix script
/srv/dockerdata/monitoring/fix-permissions.sh

# Verify ownership
ls -la /srv/dockerdata/docker-root/volumes/monitoring_grafana_data/_data

# Check container logs
docker logs grafana

Dashboard Issues¶

Dashboard Not Loading: 1. Check data source connectivity in Administration → Data Sources 2. Verify Prometheus is accessible: Test & Save data source 3. Review browser console for JavaScript errors 4. Check query syntax in problematic panels

No Data Points: 1. Verify time range selection (top-right corner) 2. Check that metrics exist in Prometheus 3. Validate PromQL query syntax 4. Ensure proper label selectors

Slow Dashboard Performance: 1. Optimize queries with appropriate aggregations 2. Reduce dashboard refresh frequency 3. Limit time range for complex queries 4. Use recording rules for frequently-used complex queries

Authentication Issues¶

# Reset admin password
docker exec -it grafana grafana-cli admin reset-admin-password newpassword

# Check authentication logs
docker logs grafana | grep -i auth

# Verify environment variables
docker exec grafana env | grep GF_SECURITY

Integration Points¶

Prometheus Integration¶

Primary Data Source: All metrics visualization
Query Builder: Visual PromQL query construction
Alert Rules: Prometheus-based alerting with Grafana evaluation

Loki Integration¶

Log Aggregation: Centralized logging from all containers
LogQL Queries: Advanced log filtering and analysis
Correlation: Link metrics with related log entries

n8n Workflow Integration¶

Webhook Alerts: Grafana alerts trigger n8n workflows
Dashboard Embedding: n8n can include Grafana panels
API Integration: n8n can query Grafana APIs for automation

Maintenance¶

Regular Tasks¶

Daily: Review active alerts and dashboard health
Weekly: Update dashboards based on operational needs
Monthly: Clean up unused dashboards and optimize performance

Backup Strategy¶

Configuration: Auto-provisioned from files (no manual backup needed)
Custom Dashboards: Export important custom dashboards as JSON
User Data: Included in infrastructure backup via Docker volumes

Updates and Changes¶

Dashboard Updates: Modify provisioning files and restart Grafana
Data Source Changes: Update via UI or provisioning files
Plugin Installation: Use Grafana CLI within container
Configuration Changes: Update environment variables and restart

Performance Optimization¶

Query Optimization¶

Use appropriate time ranges for queries
Implement proper aggregations (avg, max, rate)
Leverage recording rules for complex calculations
Cache frequently-used dashboard results

Resource Management¶

Monitor Grafana's own resource usage
Limit concurrent dashboard refreshes
Optimize dashboard refresh intervals
Use template variables for dynamic dashboards

Security Considerations¶

Authentication: Strong passwords and consider LDAP/OAuth integration
Authorization: Role-based access control for different user types
Network Security: Internal data source connections only
Data Sensitivity: Be cautious with metrics that might contain sensitive data
Session Management: Configure appropriate session timeouts

Future Enhancements¶

Advanced Templating: More dynamic dashboard variables
Custom Plugins: Develop organization-specific visualization plugins
Enterprise Features: Evaluate need for Grafana Enterprise features
Integration Expansion: Additional data sources (CloudWatch, InfluxDB)
Mobile Access: Optimize dashboards for mobile viewing