Skip to content

Operations Guide

This guide covers day-to-day operational procedures for managing the rbnk.uk infrastructure.

Overview

The operations documentation provides detailed procedures for maintaining, monitoring, and managing the self-hosted infrastructure. All services are containerized using Docker and managed through docker-compose configurations.

Key Operational Areas

DNS Management

Manage DNS records through Cloudflare, including: - Creating and updating DNS records - SSL certificate management - Domain routing configuration

Backup Strategy

Comprehensive backup procedures including: - Automated daily backups to Cloudflare R2 - Cloud storage synchronization with Rclone - Database backup procedures - Recovery workflows

Monitoring

Infrastructure monitoring with: - Prometheus metrics collection - Grafana dashboards - Loki log aggregation - AlertManager notifications

Security

Security procedures including: - Access control management - Secret rotation - Security updates - Audit procedures

Maintenance

Regular maintenance tasks: - Docker image updates - Disk space management - Performance optimization - Health checks

Proxmox Port Forwarding

Managing external access through Proxmox: - Port forwarding configuration - NAT rules management - Service exposure

Tailscale Access

Secure VPN access configuration: - Tailscale setup and management - MagicDNS configuration - Access control policies

Quick Commands

Check Service Status

# View all running services
docker ps

# Check specific service
docker compose -f SERVICE/docker-compose.yml ps

# View service logs
docker compose -f SERVICE/docker-compose.yml logs -f

Restart Services

# Restart specific service
docker compose -f SERVICE/docker-compose.yml restart

# Recreate service (pull latest image)
docker compose -f SERVICE/docker-compose.yml up -d --force-recreate

Backup Operations

# Manual backup
/srv/dockerdata/rclone/backup-infrastructure.sh

# Check backup status
docker exec rclone rclone ls cloudflare:infrastructure-backups

Monitoring

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq

# View active alerts
curl http://localhost:9090/api/v1/alerts | jq

Emergency Procedures

Service Recovery

  1. Check container status: docker ps -a
  2. View logs: docker logs CONTAINER_NAME
  3. Restart if needed: docker restart CONTAINER_NAME
  4. Check Traefik routing if external access issues

Disk Space Issues

  1. Check usage: df -h
  2. Clean Docker: docker system prune -a
  3. Check logs: du -sh /srv/dockerdata/*/logs
  4. Review backups: ls -lah /srv/dockerdata/_backup/

Network Issues

  1. Check Docker networks: docker network ls
  2. Verify Traefik: docker logs traefik
  3. Test internal connectivity: docker exec CONTAINER ping OTHER_CONTAINER
  4. Check Proxmox port forwarding if external access issues

Best Practices

  1. Always backup before major changes
  2. Use the backup scripts in _scripts/

  3. Verify backups are successful

  4. Monitor resource usage

  5. Check Grafana dashboards regularly

  6. Set up alerts for critical thresholds

  7. Document changes

  8. Update relevant documentation

  9. Commit configuration changes to Git

  10. Test in isolation

  11. Use separate test stacks when possible

  12. Verify changes before applying to production

  13. Security first

  14. Never expose services without authentication
  15. Rotate credentials regularly
  16. Keep services updated

Support Resources

  • Logs: Check service logs first for troubleshooting
  • Monitoring: Use Grafana dashboards for performance insights
  • Documentation: Refer to service-specific docs in /srv/dockerdata/docs/services/
  • Backups: Always available in Cloudflare R2 for disaster recovery