# rbnk.uk Documentation > Generated: 2026-05-23T22:55:32.121395Z > Source: https://docs.rbnk.uk ## README ### Docker Infrastructure Documentation
Binding
Set

Application error: a client-side exception has occurred while loading gymplan.rbnk.uk (see the browser console for more information).

Welcome to the comprehensive documentation for the self-hosted Docker infrastructure on `rbnk.uk`. > **Auto-rebuild enabled**: Documentation automatically rebuilds when changes are detected. ## 🆕 What's New ### December 2025 - **CodeScribe** - Code documentation generator with REST API, MCP server, and CLI for AI context generation (\1) - **Browser Use** - LLM-powered browser automation service with REST API, Gradio UI, and MCP integration (\1) - **Claude Code Safety Hooks** - Comprehensive safety infrastructure with pre/post tool hooks, audit logging, and dangerous command blocking (\1) - **Backup Infrastructure Overhaul** - Enhanced dual backup system with Prometheus metrics, Grafana dashboards, and improved R2 integration - **Supabase Enhancements** - Added Realtime, Edge Functions, Analytics, Vector (logging), and ImgProxy services ### November 2025 - **Zurg + Jellyfin** - Media streaming stack with Real-Debrid integration and hardware transcoding - **Docs Auto-Rebuild** - Automatic documentation rebuilding on git push with llms.txt generation ### August 2025 - **Iodine DNS Tunnel** - DNS tunneling service optimized for captive portal bypass (port 53 with AdGuard forwarding) - **Coolify-Traefik Bridge** - Automated routing of Coolify services through main Traefik with DNS record creation - **Authentik SSO** - Single sign-on with dual-route authentication strategy - **StackWiz MCP** - Model Context Protocol server enabling AI-powered Docker stack management - **Monitoring Overhaul** - Comprehensive Prometheus, Grafana, Loki, and AlertManager improvements ## 📚 Documentation Index ### Architecture - \1 - High-level architecture and component relationships - \1 - Docker networks, Traefik routing, SSL setup - \1 - Dual-disk configuration and data management - \1 - Architectural decisions and their rationale ### Services #### Core Infrastructure - \1 - Reverse proxy and SSL termination - \1 - Prometheus, Grafana, Loki, and AlertManager - \1 - Apprise, ntfy, Mailrise, and Uptime Kuma - \1 - Network-wide DNS filtering and ad blocking - \1 - DNS tunneling service for bypassing restrictive firewalls #### Data & Development - \1 - PostgreSQL-based backend-as-a-service - \1 - Lightweight SQLite alternative - \1 - Self-hosted Git service with GitHub mirroring #### AI & Automation - \1 - LLM-powered browser automation with REST API and Gradio UI - \1 - Code documentation generator for AI context with MCP server - \1 - Local LLM service with dual-disk setup - \1 - AI chat interface - \1 - Unified LLM API proxy - \1 - Virtual keys implementation for secure LLM access - \1 - Workflow automation platform with visual programming interface - \1 - Model Context Protocol server for Docker stack management #### Storage & Backup - \1 - Cloud storage backup and synchronization service with infrastructure backups to Cloudflare R2 - \1 - AI-powered document management with intelligent search #### Media & Streaming - \1 - Media streaming stack with Real-Debrid integration - \1 - Unified streaming addon management - \1 - Web-based YouTube downloader for video and audio content #### Utilities - \1 - Automated container updates with service exclusions ### Operations - \1 - Backup procedures and disaster recovery - \1 - Cloudflare DNS configuration and management - \1 - Health checks, logs, and metrics - \1 - Permissions, secrets management, hardening - \1 - Claude Code safety infrastructure and audit logging - \1 - Updates, disk management, troubleshooting - \1 - Port forwarding configuration and automation - \1 - VPN access configuration ### Development - \1 - Get started with development quickly - \1 - Using the stack creation wizard - \1 - How to add new services to the infrastructure - \1 - AI assistant skills, agents, commands, and hooks - \1 - Common issues and solutions ### Troubleshooting - \1 - Diagnosing and fixing SSL/TLS problems ### Security - \1 - Dual repository security model, secret filtering, and sync workflows ### Version Control - \1 - Development workflow with Gitea primary and GitHub mirror - \1 - Automated secure synchronization process ### Reference - \1 - Complete service port mapping - \1 - All configuration variables - \1 - Common commands cheatsheet ## 🚀 Quick Start 1. **Deploy a new service**: Use `stackwiz` to create a new service stack 2. **Access services**: All services available at `https://*.rbnk.uk` 3. **View logs**: `docker compose -f SERVICE/docker-compose.yml logs -f` 4. **Backup data**: Run `/srv/dockerdata/_scripts/SERVICE-backup.sh` 5. **Access Git**: Browse code at `https://gitea.rbnk.uk` or clone via SSH/HTTPS 6. **AI Automation**: Use StackWiz MCP with Claude to create and manage Docker stacks automatically ## 🏗️ Infrastructure Overview ``` ┌─────────────────────────────────────────────────────────────┐ │ Internet │ └────────────────────┬────────────────────────────────────────┘ │ HTTPS (*.rbnk.uk) ┌─────────┴──────────┐ │ Cloudflare DNS │ │ (SSL Certificates)│ └─────────┬──────────┘ │ ┌─────────┴──────────┐ │ Traefik │ ← Reverse Proxy │ (Docker Labels) │ Auto-SSL └─────────┬──────────┘ │ ┌────────────┴────────────┬─────────────┬──────────────┐ │ │ │ │ ┌───────┴────────┐ ┌────────┴───────┐ ┌──┴────────┐ ┌──┴────────┐ │ Supabase │ │ Open WebUI │ │ LiteLLM │ │ Gitea │ │ (PostgreSQL) │ │ (AI Chat UI) │ │ (LLM API) │ │ (Git) │ └────────────────┘ └────────────────┘ └───────────┘ └───────────┘ ┌────────────────┐ ┌─────────────────────┐ ┌────────────────┐ │ Rclone │ │ Monitoring Stack │ │ n8n + MCP │ │ (Cloud Backup) │ │ (Prometheus/Grafana)│ │ (Automation) │ │ & R2 Storage │ └─────────────────────┘ └────────────────┘ └────────────────┘ ┌─────────────────────────────────────────────────────────────────┐ │ Notification Stack │ │ (Apprise + ntfy + Mailrise + Uptime Kuma) │ └─────────────────────────────────────────────────────────────────┘ ``` ## 💾 Storage Layout - **System Disk** (`/dev/sda` - 100GB): OS and system files (29% used) - **Data Disk** (`/dev/sdb` - 500GB): `/srv/dockerdata` - All Docker data (39% used) ## 🔐 Version Control The infrastructure uses a **dual repository strategy** for secure code management: - **Private Repository** (Gitea): `https://gitea.rbnk.uk` - Contains all files including sensitive configuration - **Public Repository** (GitHub): Public mirror with sensitive files automatically filtered out - **Automated Sync**: Changes pushed to Gitea are automatically synced to GitHub with security filtering - **Access**: Gitea provides full Git functionality with web UI for code browsing and management ## 🔑 Key Principles 1. **Service Isolation**: Each service in its own directory with dedicated `docker-compose.yml` 2. **Automatic SSL**: Traefik + Cloudflare handles all SSL certificates 3. **Network Security**: Internal services (databases) not exposed publicly 4. **Standardized Structure**: Consistent patterns across all services 5. **Infrastructure as Code**: Everything configured through files, not UIs 6. **Secure Version Control**: Private Gitea instance with automated public GitHub sync ## ⚠️ Important Notes - **Disk Space**: System disk (100GB, 29% used), Data disk (500GB, 39% used) - healthy capacity - **Backups**: Automated dual backup system - cloud backups at 2 AM, infrastructure backups to R2 at 4 AM - **Secrets**: Stored in `.env` files - ensure proper permissions (640) - **Container Updates**: Watchtower handles automatic updates but critical services (databases, infrastructure) are protected - **Git Security**: Never commit sensitive files to public repos - use Gitea for private code ## 📝 Contributing to Docs When adding documentation: 1. Include the "why" not just the "how" 2. Add examples where helpful 3. Cross-reference related documentation 4. Update this index when adding new guides --- *Last Updated: December 2025 - Added CodeScribe code documentation service with MCP server* --- ## architecture > decisions ### Architecture Decision Records (ADR) Last Updated: 2025-12-02 ## Overview This document logs key architectural decisions made for the infrastructure, including context, rationale, trade-offs, and potential migration paths. --- ## ADR-001: Use Docker Compose for Service Orchestration **Date**: 2024-01-01 **Status**: Accepted **Context**: Need to deploy multiple services with complex interdependencies on a single VM. ### Decision Use Docker Compose instead of Kubernetes or Docker Swarm for service orchestration. ### Rationale - Single VM deployment doesn't justify Kubernetes complexity - Docker Compose provides sufficient orchestration for our scale - Easier to debug and maintain for small teams - Native support for environment variables and networking - Simple backup and restore procedures ### Trade-offs - **Pros**: Simple, maintainable, good documentation, fast deployment - **Cons**: No built-in HA, limited scaling options, manual updates ### Alternatives Considered 1. **Kubernetes**: Too complex for single-node deployment 2. **Docker Swarm**: Better for multi-node, adds unnecessary complexity 3. **Systemd units**: Less portable, harder dependency management ### Migration Path If scaling beyond single node: 1. Convert compose files to Kubernetes manifests using Kompose 2. Implement external configuration management (Helm/Kustomize) 3. Move to managed Kubernetes service ### Lessons Learned - Compose v2 syntax provides all needed features - Network isolation works well for security - Volume management requires careful planning --- ## ADR-002: Use Traefik as Reverse Proxy **Date**: 2024-01-15 **Status**: Accepted **Context**: Need reverse proxy with automatic SSL and service discovery. ### Decision Use Traefik v3 instead of Nginx, Caddy, or HAProxy. ### Rationale - Native Docker integration with label-based configuration - Automatic SSL with Let's Encrypt and Cloudflare DNS - Built-in service discovery - Good observability with metrics and access logs - Active development and community ### Trade-offs - **Pros**: Auto-configuration, great Docker support, modern features - **Cons**: Learning curve, more complex than Nginx for simple cases ### Alternatives Considered 1. **Nginx**: Requires manual configuration, no auto-discovery 2. **Caddy**: Simpler but less flexible for complex routing 3. **HAProxy**: Powerful but lacks modern convenience features ### Migration Path To migrate away from Traefik: 1. Export routing rules from Traefik API 2. Convert to target proxy configuration 3. Implement service discovery mechanism 4. Test thoroughly with canary deployments ### Lessons Learned - Label-based configuration prevents configuration drift - Middleware chains are powerful for auth/redirects - DNS challenge works better than HTTP for wildcard certs --- ## ADR-003: Self-Host Supabase **Date**: 2024-02-01 **Status**: Accepted **Context**: Need PostgreSQL-based backend with auth, storage, and realtime features. ### Decision Self-host Supabase instead of using managed service or building custom. ### Rationale - Full control over data and infrastructure - Cost-effective for our scale - All features available without restrictions - Can customize and extend as needed - Good architectural patterns to follow ### Trade-offs - **Pros**: No vendor lock-in, full control, cost savings, customizable - **Cons**: Maintenance burden, security responsibility, no SLA ### Alternatives Considered 1. **Managed Supabase**: Monthly costs, data sovereignty concerns 2. **Custom stack**: Too much development effort 3. **Firebase**: Vendor lock-in, not PostgreSQL 4. **Hasura + Auth0**: More complex, higher cost ### Migration Path To managed Supabase: 1. Use Supabase migration tools 2. Update connection strings 3. Verify auth flows work correctly To custom stack: 1. PostgREST can be replaced with custom API 2. GoTrue can be replaced with other auth 3. Database remains PostgreSQL ### Lessons Learned - Bootstrap SQL pattern works well for initialization - JWT secret management is critical - Storage API needs careful permission setup - Backup strategy essential from day one --- ## ADR-004: Use Cloudflare for DNS and SSL **Date**: 2024-02-15 **Status**: Accepted **Context**: Need reliable DNS with API access for automatic SSL certificates. ### Decision Use Cloudflare for DNS management and SSL certificate generation. ### Rationale - Free tier sufficient for our needs - Excellent API for DNS challenges - DDoS protection included - DNS analytics helpful for debugging - Good Traefik integration ### Trade-offs - **Pros**: Free, reliable, good API, security features - **Cons**: Dependency on external service, potential API limits ### Alternatives Considered 1. **Route53**: Costs money, AWS lock-in 2. **Self-hosted DNS**: Complexity, reliability concerns 3. **HTTP challenges**: Doesn't work for wildcard certs 4. **Manual certs**: Operational burden ### Migration Path To different DNS provider: 1. Update Traefik configuration for new provider 2. Migrate DNS records 3. Update API credentials 4. Test certificate renewal ### Lessons Learned - API token scoping important for security - Wildcard certificates simplify subdomain management - DNS challenge more reliable than HTTP --- ## ADR-005: Separate Networks for Security **Date**: 2024-03-01 **Status**: Accepted **Context**: Need network isolation between external-facing and internal services. ### Decision Use separate Docker networks: `traefik_proxy` for external, `supabase_internal` for database access, plus `cache_redis_network` and `session_redis_network` for Redis services. ### Rationale - Clear security boundary - Prevents accidental exposure - Easy to audit and understand - Native Docker feature - No performance penalty ### Trade-offs - **Pros**: Better security, clear boundaries, easy to implement - **Cons**: Services need multiple network attachments, some complexity ### Alternatives Considered 1. **Single network**: Security risk 2. **Network policies**: Not available in Docker 3. **Firewall rules**: More complex, less portable 4. **VLANs**: Overkill for single host ### Migration Path Already using best practice, but if needed: 1. Can add more networks for finer segmentation 2. Can implement service mesh if needed 3. Can add network policies with Kubernetes migration ### Lessons Learned - Explicit network membership prevents accidents - Internal services stay internal - Cross-network communication needs explicit configuration --- ## ADR-006: Systemd Timers for Scheduled Tasks **Date**: 2024-03-15 **Status**: Accepted **Context**: Need reliable scheduling for backups and maintenance tasks. ### Decision Use systemd timers instead of cron or container-based scheduling. ### Rationale - Better logging and monitoring - Dependency management - No additional containers needed - Handles system boot properly - Built into the OS ### Trade-offs - **Pros**: Reliable, good logs, dependency handling, no overhead - **Cons**: Linux-specific, requires systemd knowledge ### Alternatives Considered 1. **Cron**: Less visibility, basic logging 2. **Container scheduler**: Additional complexity 3. **n8n/Airflow**: Overkill for simple tasks 4. **Kubernetes CronJobs**: Not applicable ### Migration Path To container-based scheduling: 1. Convert timer units to CronJob specs 2. Implement job containers 3. Handle logging/monitoring 4. Test failure scenarios ### Lessons Learned - OnCalendar syntax intuitive for schedules - Persistent=true handles missed runs - Service units provide good isolation --- ## ADR-007: Git for Configuration Management **Date**: 2024-04-01 **Status**: Accepted **Context**: Need version control for infrastructure configuration. ### Decision Use Git repository for all configuration except secrets. ### Rationale - Version history for debugging - Easy rollback capabilities - Collaboration features - Can implement GitOps later - Industry standard practice ### Trade-offs - **Pros**: Version control, collaboration, audit trail, rollback - **Cons**: Need to carefully exclude secrets, requires discipline ### Alternatives Considered 1. **No version control**: Risky, no history 2. **Config management tool**: Overkill for our scale 3. **Database storage**: Less flexible, custom tooling ### Migration Path Already using best practice. For GitOps: 1. Implement CI/CD pipeline 2. Add automated testing 3. Implement automatic deployment 4. Add approval workflows ### Lessons Learned - .gitignore critical for security - .env.example files valuable for documentation - Commit messages should explain why - Regular commits prevent large disasters --- ## ADR-008: Bootstrap SQL for Database Initialization **Date**: 2024-04-15 **Status**: Accepted **Context**: Need reproducible database initialization for Supabase. ### Decision Use bootstrap SQL files that run on container initialization. ### Rationale - Declarative and reproducible - Version controlled with Git - No manual steps required - Can include custom functions/triggers - Survives container recreation ### Trade-offs - **Pros**: Reproducible, versioned, automated, documented - **Cons**: Only runs on first init, migrations need different approach ### Alternatives Considered 1. **Manual setup**: Error-prone, not reproducible 2. **Migration tool**: Complexity for initial setup 3. **Backup restore**: Includes data, less flexible ### Migration Path For schema changes after initialization: 1. Implement migration tool (Flyway, Liquibase) 2. Convert bootstrap SQL to migrations 3. Version all schema changes 4. Test rollback procedures ### Lessons Learned - Alphabetical ordering provides control - Idempotent scripts are better - Include test data cautiously - Document each script's purpose --- ## ADR-009: 14-Day Backup Retention **Date**: 2024-05-01 **Status**: Accepted **Context**: Balance between storage costs and recovery needs. ### Decision Implement 14-day retention for automated backups. ### Rationale - Two weeks covers most recovery scenarios - Storage costs remain reasonable - Allows recovery from weekend issues - Simple rotation logic - Can restore before issues discovered ### Trade-offs - **Pros**: Good balance, reasonable cost, simple implementation - **Cons**: No long-term archives, might miss slow issues ### Alternatives Considered 1. **7 days**: Too short for some scenarios 2. **30 days**: Higher storage costs 3. **Incremental forever**: Complex management 4. **No backups**: Unacceptable risk ### Migration Path To longer retention: 1. Implement tiered storage (daily/weekly/monthly) 2. Use object storage for archives 3. Implement backup testing automation 4. Consider incremental backups ### Lessons Learned - Test restores regularly - Monitor backup sizes - Alert on backup failures - Document restore procedures --- ## Decision Template ```markdown ## ADR-XXX: [Decision Title] **Date**: YYYY-MM-DD **Status**: Proposed/Accepted/Deprecated/Superseded **Context**: [Why this decision needed to be made] ### Decision [What was decided] ### Rationale [Why this decision was made] ### Trade-offs - **Pros**: [Positive outcomes] - **Cons**: [Negative outcomes] ### Alternatives Considered 1. **Option 1**: [Why not chosen] 2. **Option 2**: [Why not chosen] ### Migration Path [How to move away from this decision if needed] ### Lessons Learned [What we learned from implementing this] ``` ## Review Schedule Architectural decisions should be reviewed: - Quarterly for active development - Annually for stable systems - When major changes are proposed - When issues arise from decisions ## Change Process 1. Propose new ADR with status "Proposed" 2. Discuss with team 3. Update status to "Accepted" or "Rejected" 4. If replacing existing decision, mark old as "Superseded" 5. Document lessons learned after implementation --- ## architecture > network-topology ### Network Topology ## Physical Infrastructure The infrastructure runs inside a Proxmox virtualized environment with multiple network layers between the internet and Docker containers. ## Tailscale Network Layer Tailscale provides an overlay network that bypasses traditional port forwarding requirements and enables secure access to services. ### Tailscale Architecture ``` ┌─────────────────────────────────────────────────────────────────────────────────┐ │ Tailscale Network │ │ │ │ ┌─────────────────┐ ┌──────────────────┐ │ │ │ Your Device │ │ Docker VM │ │ │ │ 100.x.x.x │ ←── WireGuard Encrypted Tunnel ──→ │ dockerhost │ │ │ │ │ │ 100.98.20.43 │ │ │ └─────────────────┘ └──────────────────┘ │ │ │ │ Direct P2P Connection (when possible) or via DERP Relay │ └─────────────────────────────────────────────────────────────────────────────────┘ ``` ### Key Benefits 1. **No Port Forwarding Required**: Access services without configuring Proxmox iptables 2. **End-to-End Encryption**: All traffic encrypted via WireGuard protocol 3. **Simplified Access**: Use Tailscale IPs or MagicDNS names instead of public IPs 4. **NAT Traversal**: Works behind firewalls and complex network setups 5. **Zero-Trust Security**: Only authorized devices can connect ### Access Patterns #### Via Tailscale (Recommended for Admin Access) ``` Your Device → Tailscale VPN → dockerhost (100.98.20.43) → Docker Services ↓ No Proxmox port forwarding needed! ``` #### Traditional Public Access (For End Users) ``` User Browser → Cloudflare → 157.180.63.15 → Proxmox NAT → 10.10.10.10 → Traefik ``` ### Service Access Examples 1. **SSH Access**: ```bash # Via Tailscale (no port forwarding needed) ssh user@100.98.20.43 # or ssh user@dockerhost.tail92c8e3.ts.net # Traditional (requires port 22 forwarding) ssh user@157.180.63.15 ``` 2. **Traefik Dashboard**: ```bash # Via Tailscale (direct access) http://100.98.20.43:8080 # Traditional (blocked without port forward) # Would need port 8080 forwarded, which is not recommended ``` 3. **Direct Database Access**: ```bash # Via Tailscale (secure, no public exposure) psql -h 100.98.20.43 -p 5432 -U postgres # Traditional (requires port 5432 forward - NOT recommended) ``` ### Tailscale Integration Points ``` ┌──────────────────────────────────────────────────────────────────────────────┐ │ Docker VM with Tailscale │ │ │ │ Network Interfaces: │ │ ├── eth0: 10.10.10.10/24 (Proxmox NAT) │ │ ├── tailscale0: 100.98.20.43 (Tailscale VPN) │ │ └── docker0: 172.17.0.1/16 (Docker Bridge) │ │ │ │ Access Methods: │ │ 1. Public Internet → Proxmox NAT → eth0 (requires port forwarding) │ │ 2. Tailscale VPN → tailscale0 (no port forwarding needed) │ │ │ │ Services accessible via both: │ │ - HTTP/HTTPS (via Traefik) │ │ - SSH │ │ - Any Docker exposed port │ └──────────────────────────────────────────────────────────────────────────────┘ ``` ### DNS Considerations When using Tailscale with AdGuard: 1. **MagicDNS**: Tailscale provides automatic DNS names (e.g., dockerhost.tail92c8e3.ts.net) 2. **Custom DNS**: You can configure Tailscale to use your AdGuard instance 3. **Split DNS**: Define custom domains that resolve only within your Tailnet ### Security Benefits 1. **No Public Port Exposure**: Admin interfaces remain completely private 2. **Device Authorization**: Only approved devices can join the Tailnet 3. **ACLs**: Fine-grained access control between Tailscale nodes 4. **Audit Logs**: Track all connections and access attempts 5. **Key Rotation**: Automatic key management and rotation ### Complete Network Flow ``` ┌─────────────────────────────────────────────────────────────────────────────────┐ │ Internet (Public) │ │ │ │ │ ↓ │ │ Hetzner Public IP: 157.180.63.15 │ └──────────────────────────────────┬─────────────────────────────────────────────┘ │ ┌──────────────────────────────────┴─────────────────────────────────────────────┐ │ Proxmox Host (Hetzner Server) │ │ │ │ Network Interfaces: │ │ ├── enp7s0: 157.180.63.15/26 (Public) │ │ └── vmbr0: 10.10.10.1/24 (Internal Bridge) │ │ │ │ NAT & Port Forwarding Rules: │ │ ├── 157.180.63.15:80 → 10.10.10.10:80 (HTTP) │ │ ├── 157.180.63.15:443 → 10.10.10.10:443 (HTTPS) │ │ ├── 157.180.63.15:22 → 10.10.10.10:22 (SSH) │ │ └── 157.180.63.15:51820 → 10.10.10.10:51820 (WireGuard/Optional) │ │ │ └──────────────────────────────────┬─────────────────────────────────────────────┘ │ NAT ↓ ┌──────────────────────────────────┴─────────────────────────────────────────────┐ │ Docker VM (10.10.10.10) - Ubuntu │ │ │ │ Network Configuration: │ │ ├── eth0: 10.10.10.10/24 (From Proxmox) │ │ ├── docker0: 172.17.0.1/16 (Docker Default Bridge) │ │ ├── traefik_proxy: 172.18.0.0/16 (External Services) │ │ └── supabase_internal: 172.19.0.0/16 (Database Network) │ │ │ │ Services Running: │ │ ├── Traefik (Reverse Proxy) - Ports 80, 443 │ │ ├── Docker Daemon │ │ └── SSH Server │ └────────────────────────────────────────────────────────────────────────────────┘ ``` ### Network Hierarchy ``` Internet Traffic Flow: ══════════════════════ 1. User → DNS (*.rbnk.uk) → 157.180.63.15 2. Hetzner Server → Proxmox iptables NAT → 10.10.10.10 3. Docker VM → Traefik (ports 80/443) → Docker Networks 4. Docker Networks → Containers (by hostname/labels) Example: https://supabase.rbnk.uk ──────────────────────────────── User Browser ↓ (DNS resolves to 157.180.63.15) Hetzner Public IP:443 ↓ (iptables DNAT) Proxmox NAT → 10.10.10.10:443 ↓ (VM receives traffic) Traefik Container ↓ (reads Host header) ↓ (routes based on rules) Supabase Studio Container ``` ## Port Forwarding Requirements ### Essential Ports (Always Required) | Service | External Port | Internal Port | Protocol | Purpose | |---------|--------------|---------------|----------|---------| | HTTP | 80 | 80 | TCP | HTTP traffic (redirects to HTTPS) | | HTTPS | 443 | 443 | TCP | All web services via Traefik | | SSH | 22 | 22 | TCP | Remote administration | ### Optional Ports (Service-Specific) | Service | External Port | Internal Port | Protocol | Purpose | |---------|--------------|---------------|----------|---------| | WireGuard | 51820 | 51820 | UDP | VPN access (if needed) | | PostgreSQL | 5432 | 5432 | TCP | Direct DB access (NOT recommended) | | Traefik Dashboard | 8080 | 8080 | TCP | Internal only (no forward) | ### Proxmox NAT Configuration Add to `/etc/network/interfaces` on Proxmox host: ```bash # NAT for Docker VM post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp7s0 -j MASQUERADE # Port forwarding for Docker VM post-up iptables -t nat -A PREROUTING -i enp7s0 -p tcp --dport 80 -j DNAT --to 10.10.10.10:80 post-up iptables -t nat -A PREROUTING -i enp7s0 -p tcp --dport 443 -j DNAT --to 10.10.10.10:443 post-up iptables -t nat -A PREROUTING -i enp7s0 -p tcp --dport 22 -j DNAT --to 10.10.10.10:22 # Cleanup rules on down post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp7s0 -j MASQUERADE post-down iptables -t nat -D PREROUTING -i enp7s0 -p tcp --dport 80 -j DNAT --to 10.10.10.10:80 post-down iptables -t nat -D PREROUTING -i enp7s0 -p tcp --dport 443 -j DNAT --to 10.10.10.10:443 post-down iptables -t nat -D PREROUTING -i enp7s0 -p tcp --dport 22 -j DNAT --to 10.10.10.10:22 ``` ## Docker Networks Overview The infrastructure uses Docker's bridge networks to isolate and connect services appropriately. All Docker networks operate within the VM (10.10.10.10). ``` ┌─────────────────────────────────────────────────────────────────┐ │ Docker VM (10.10.10.10) - Network Context │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Docker Host Network (172.17.0.1) │ │ │ │ │ │ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ │ │ traefik_proxy (172.18.0.0/16) - External │ │ │ │ │ │ Connected to: eth0 for outbound NAT │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌──────────────┐ │ │ │ │ │ │ │ Traefik │ │ LiteLLM │ │ n8n │ │ Open WebUI │ │ │ │ │ │ │ │ .18.2 │ │ .18.3 │ │ .18.4 │ │ .18.5 │ │ │ │ │ │ │ └─────────┘ └──────────┘ └─────────┘ └──────────────┘ │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ │ │ │ │ Gitea │ │ Rclone │ │Prometheus│ │ Grafana │ │ │ │ │ │ │ │ .18.6 │ │ .18.7 │ │ .18.8 │ │ .18.9 │ │ │ │ │ │ │ └─────────┘ └─────┬────┘ └──────────┘ └──────────┘ │ │ │ │ │ │ │ │ │ │ │ │ │ └──→ Cloudflare R2 (External) │ │ │ │ │ │ ↑ ↑ ↑ ↑ │ │ │ │ │ └───────┼────────────┼───────────┼─────────────┼────────┘ │ │ │ │ │ │ │ │ │ │ │ │ ┌───────┼────────────┼───────────┼─────────────┼────────┐ │ │ │ │ │ ↓ ↓ ↓ ↓ │ │ │ │ │ │ ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌──────────────┐ │ │ │ │ │ │ │ Auth │ │ Rest │ │ Storage │ │ Studio │ │ │ │ │ │ │ │ .19.3 │ │ .19.4 │ │ .19.5 │ │ .19.6 │ │ │ │ │ │ │ └────┬────┘ └────┬─────┘ └────┬────┘ └──────┬───────┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────┴────────────┴──────────────┘ │ │ │ │ │ │ ↓ │ │ │ │ │ │ ┌─────────────┐ │ │ │ │ │ │ │ PostgreSQL │ │ │ │ │ │ │ │ .19.2 │ │ │ │ │ │ │ └─────────────┘ │ │ │ │ │ │ │ │ │ │ │ │ supabase_internal (172.19.0.0/16) - Internal │ │ │ │ │ │ Isolated: No direct internet access │ │ │ │ │ └───────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ │ │ cache_redis_network (172.20.0.0/16) - Redis │ │ │ │ │ │ Cache layer for ephemeral data │ │ │ │ │ │ │ │ │ │ │ │ ┌─────────────┐ ┌─────────┐ ┌─────────────┐ │ │ │ │ │ │ │ Cache Redis │───│ Gitea │───│ Stremthru │ │ │ │ │ │ │ │ .20.2 │ │ .20.10 │ │ .20.11 │ │ │ │ │ │ │ └─────────────┘ └─────────┘ └─────────────┘ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ ┌───────────────────────────────────────────────────┐ │ │ │ │ │ session_redis_network (172.21.0.0/16) - Redis │ │ │ │ │ │ Session storage for persistent user data │ │ │ │ │ │ │ │ │ │ │ │ ┌──────────────┐ ┌─────────┐ │ │ │ │ │ │ │Session Redis │───│ Gitea │ │ │ │ │ │ │ │ .21.2 │ │ .21.10 │ │ │ │ │ │ │ └──────────────┘ └─────────┘ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────────────────────┘ │ │ │ │ │ │ │ │ VM Host Network Services: │ │ │ │ └── Ollama (0.0.0.0:11434) ← Via 172.17.0.1 │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ External Access: All traffic enters via Traefik on ports │ │ 80/443, forwarded from Proxmox NAT (157.180.63.15) │ └─────────────────────────────────────────────────────────────────┘ ``` ## Network Definitions ### traefik_proxy (External Network) **Purpose**: Services that need public internet access **Configuration**: ```yaml networks: traefik_proxy: external: true ``` **IP Range**: 172.18.0.0/16 - Gateway: 172.18.0.1 - Traefik: 172.18.0.2 - Services: 172.18.0.3+ **Connected Services**: - Traefik (reverse proxy) - All public-facing services - Supabase services (dual-homed) - Rclone (web GUI, cloud sync, and R2 infrastructure backups) - Prometheus and Grafana (monitoring stack) ### supabase_internal (Internal Network) **Purpose**: Database isolation and internal communication **Configuration**: ```yaml networks: internal: driver: bridge internal: true # No external routing ``` **IP Range**: 172.19.0.0/16 - Gateway: 172.19.0.1 - PostgreSQL: 172.19.0.2 - Internal services: 172.19.0.3+ **Connected Services**: - PostgreSQL (database) - All Supabase components - No external access ### cache_redis_network (Cache Network) **Purpose**: Shared Redis cache for ephemeral data across services **Configuration**: ```yaml networks: cache_redis_network: name: cache_redis_network driver: bridge ``` **IP Range**: 172.20.0.0/16 - Gateway: 172.20.0.1 - Cache Redis: 172.20.0.2 - Connected services: 172.20.0.3+ **Connected Services**: - cache-redis (Redis 7.4 instance) - Gitea (Git object caching) - Stremthru (API response caching) - Any service needing temporary data storage **Cache Characteristics**: - **Persistence**: Disabled (no disk writes) - **Memory Policy**: `allkeys-lru` (evict least recently used) - **Max Memory**: 4GB (configurable) - **Use Cases**: API responses, computed results, temporary data ### session_redis_network (Session Network) **Purpose**: Shared Redis storage for persistent session data **Configuration**: ```yaml networks: session_redis_network: name: session_redis_network driver: bridge ``` **IP Range**: 172.21.0.0/16 - Gateway: 172.21.0.1 - Session Redis: 172.21.0.2 - Connected services: 172.21.0.3+ **Connected Services**: - session-redis (Redis 7.4 instance) - Gitea (web session storage) - Future web applications requiring session management **Session Characteristics**: - **Persistence**: Enabled (AOF + RDB snapshots) - **Memory Policy**: `volatile-lru` (evict expired keys first) - **Max Memory**: 1GB (configurable) - **Use Cases**: User sessions, authentication tokens, persistent state ## Service Communication Patterns ### 1. External User → Service ``` User Request → Cloudflare → 157.180.63.15:443 → Proxmox NAT → 10.10.10.10:443 → Traefik → Service ``` **Example**: Accessing Open WebUI ``` https://openwebui.rbnk.uk ↓ (Cloudflare DNS) 157.180.63.15:443 ↓ (Proxmox iptables DNAT) 10.10.10.10:443 ↓ (Docker VM receives) Traefik container reads Host header ↓ (Docker network routing) Routes to open-webui container on port 8080 ``` ### 2. Service → Database ``` Service → supabase_internal network → PostgreSQL:5432 ``` **Example**: Supabase Studio accessing database ``` Studio (172.19.0.6) → PostgreSQL (172.19.0.2:5432) ``` ### 3. Container → VM Host Service ``` Container → Docker Gateway (172.17.0.1) → VM Host Service ``` **Example**: Open WebUI accessing Ollama (running on VM, not in container) ``` Open WebUI container → http://172.17.0.1:11434 → Ollama API (VM process) ``` **Note**: This is within the VM (10.10.10.10), not the Proxmox host ### 4. Service → External API ``` Service → traefik_proxy network → Docker NAT → VM eth0 → Proxmox NAT → Internet ``` **Example**: LiteLLM calling OpenAI ``` LiteLLM container → Docker bridge → 10.10.10.10 eth0 → Proxmox vmbr0 → 157.180.63.15 → api.openai.com ``` ### 5. Service → Redis Cache ``` Service → cache_redis_network → Cache Redis:6379 ``` **Example**: Gitea caching Git objects ``` Gitea (172.20.0.10) → cache_redis_network → cache-redis (172.20.0.2:6379) ``` **Configuration**: ```yaml # In service docker-compose.yml environment: - REDIS_CACHE_URL=redis://:${REDIS_PASSWORD}@cache-redis:6379/0 networks: - cache_redis_network ``` ### 6. Service → Redis Sessions ``` Service → session_redis_network → Session Redis:6379 ``` **Example**: Gitea storing user sessions ``` Gitea (172.21.0.10) → session_redis_network → session-redis (172.21.0.2:6379) ``` **Configuration**: ```yaml # In service docker-compose.yml environment: - REDIS_SESSION_URL=redis://:${REDIS_PASSWORD}@session-redis:6379/1 networks: - session_redis_network ``` ### 7. Metrics Collection Flow ``` Services → Node Exporter → Prometheus → Grafana ``` **Example**: Rclone backup metrics ``` Rclone script → writes to /var/lib/node_exporter/textfile_collector/rclone_backup.prom Node Exporter → reads metrics files → exposes on :9100/metrics Prometheus → scrapes :9100/metrics every 15s Grafana → queries Prometheus → displays on dashboards ``` ## Traefik Routing Rules ### Host-Based Routing Each service registers itself with Traefik using Docker labels: ```yaml labels: - "traefik.enable=true" - "traefik.http.routers.SERVICE.rule=Host(`subdomain.rbnk.uk`)" - "traefik.http.services.SERVICE.loadbalancer.server.port=8080" ``` ### Path-Based Routing (Supabase) Supabase uses path prefixes to route to different services: ``` supabase.rbnk.uk/ ├── /auth/v1/* → Auth Service (port 9999) ├── /rest/v1/* → PostgREST API (port 3000) ├── /storage/v1/* → Storage API (port 5000) └── /* → Studio UI (port 3000) ``` **Implementation**: ```yaml # Auth service - "traefik.http.routers.supa-auth.rule=Host(`supabase.rbnk.uk`) && PathPrefix(`/auth/v1`)" - "traefik.http.middlewares.supa-auth-stripprefix.stripprefix.prefixes=/auth/v1" # Studio catch-all (must be last) - "traefik.http.routers.supa-studio.rule=Host(`supabase.rbnk.uk`)" - "traefik.http.routers.supa-studio.priority=1" ``` ## SSL/TLS Configuration ### Cloudflare DNS Challenge ``` Traefik → Cloudflare API → DNS TXT Record → Let's Encrypt → Certificate ``` **Configuration**: ```yaml # In Traefik config certificatesResolvers: cf: acme: dnsChallenge: provider: cloudflare ``` **Why DNS Challenge?** - No need to open port 80 - Supports wildcard certificates - Works behind NAT/firewall ### Certificate Storage ``` /srv/dockerdata/traefik/ └── letsencrypt/ └── acme.json # Contains all certificates ``` ## Port Forwarding by Service Type ### Web Services (via Traefik) Most services only need ports 80/443 forwarded to the VM. Traefik handles internal routing. **Required forwards**: 80, 443 **Services**: Supabase, LiteLLM, n8n, Open WebUI, any web app ### Direct Database Access **WARNING**: Not recommended for production. Use Supabase REST API instead. **Required forwards**: 5432 **Configuration**: ```bash # On Proxmox (NOT recommended) iptables -t nat -A PREROUTING -i enp7s0 -p tcp --dport 5432 -j DNAT --to 10.10.10.10:5432 ``` ### VPN Services For secure remote access to internal networks. **Required forwards**: 51820 (UDP for WireGuard) **Configuration**: ```bash # On Proxmox iptables -t nat -A PREROUTING -i enp7s0 -p udp --dport 51820 -j DNAT --to 10.10.10.10:51820 ``` ### Custom TCP Services For services that can't use HTTP/HTTPS. **Example**: Custom game server on port 25565 ```bash # On Proxmox iptables -t nat -A PREROUTING -i enp7s0 -p tcp --dport 25565 -j DNAT --to 10.10.10.10:25565 ``` ### Port Forwarding Best Practices 1. **Minimize Exposure**: Only forward essential ports 2. **Use Traefik**: Prefer HTTP/HTTPS routing over direct port exposure 3. **Document Changes**: Keep track of all forwarded ports 4. **Firewall Rules**: Add iptables rules to Proxmox network config for persistence 5. **Security**: Never expose database or admin ports directly ## Network Security ### 1. Network Isolation ```yaml # Internal network - no internet access networks: internal: internal: true ``` ### 2. Port Exposure **Never expose ports directly**: ```yaml # BAD - Exposes to internet ports: - "5432:5432" # GOOD - Only Traefik exposed # No ports: section, use Traefik labels ``` ### 3. Service Discovery Services find each other by name within same network: ```yaml # In app config DATABASE_URL=postgresql://user:pass@supa-db:5432/dbname # ^^^^^^^ # Service name = hostname ``` ## Common Networking Issues ### 1. Container Can't Reach Internet **Symptom**: API calls fail from container **Check**: ```bash # Is container on external network? docker inspect CONTAINER | jq '.[0].NetworkSettings.Networks' ``` **Fix**: Add to traefik_proxy network ### 2. Service Can't Connect to Database **Symptom**: Connection refused to PostgreSQL **Check**: ```bash # Are both on same network? docker network inspect supabase_internal ``` **Fix**: Ensure service is on internal network ### 3. Traefik Can't Route to Service **Symptom**: 502 Bad Gateway **Check**: ```bash # Is service on traefik_proxy network? docker inspect SERVICE | grep traefik_proxy # Check Traefik can see service curl http://localhost:8080/api/http/services ``` **Fix**: Add traefik.docker.network label: ```yaml - "traefik.docker.network=traefik_proxy" ``` ## Network Debugging Commands ### VM Network Status ```bash # Check VM network interfaces ip addr show # Check routing table ip route # Test connectivity to Proxmox gateway ping 10.10.10.1 # Check NAT status sudo iptables -t nat -L -n -v ``` ### Docker Network Configuration ```bash # List all networks docker network ls # Inspect network details docker network inspect traefik_proxy # See connected containers docker network inspect traefik_proxy | jq '.[0].Containers' ``` ### Test Connectivity Chain ```bash # 1. From container to another container docker exec CONTAINER ping OTHER_CONTAINER # 2. From container to VM host docker exec CONTAINER ping 172.17.0.1 # 3. From container to internet (tests full NAT chain) docker exec CONTAINER ping 8.8.8.8 # 4. Check specific port accessibility docker exec CONTAINER nc -zv SERVICE PORT # 5. Test from VM to Proxmox ping 10.10.10.1 # 6. Test from VM to internet curl -I https://example.com ``` ### DNS Resolution Debugging ```bash # Check DNS inside container docker exec CONTAINER nslookup OTHER_SERVICE # Check external DNS from container docker exec CONTAINER nslookup google.com # Check DNS from VM nslookup google.com # Test specific DNS server docker exec CONTAINER nslookup example.com 8.8.8.8 ``` ### Proxmox-Specific Debugging ```bash # On Proxmox host (via SSH to 157.180.63.15) # Check VM network status qm list qm config VMID | grep net # Check bridge status brctl show # Monitor traffic to VM tcpdump -i vmbr0 host 10.10.10.10 ``` ## Best Practices ### 1. Network Naming - Use descriptive names - Suffix with purpose: `_proxy`, `_internal` - Keep consistent across services ### 2. IP Assignment - Let Docker assign IPs automatically - Use service names, not IPs - Document IP ranges for troubleshooting ### 3. Security - Minimize services on external network - Use internal networks for databases - Never expose management ports ### 4. Documentation - Document custom networks in compose files - Comment non-obvious routing rules - Keep network diagram updated ## Future Improvements 1. **Network Policies**: Implement iptables rules for finer control 2. **Service Mesh**: Consider Consul/Linkerd for complex routing 3. **Monitoring**: Add network metrics to Prometheus 4. **IPv6**: Enable dual-stack networking --- ## architecture > overview ### System Architecture Overview ## High-Level Architecture This infrastructure runs on a single Proxmox VM with a dual-disk setup, hosting multiple containerized services behind a Traefik reverse proxy with automatic SSL via Cloudflare. ``` ┌─────────────────────────────────────────────────────────────────────┐ │ INTERNET │ │ │ │ Users → https://*.rbnk.uk → Cloudflare DNS → Your Server (Hetzner) │ └──────────────────────────────────┬──────────────────────────────────┘ │ ┌──────────────┴──────────────┐ │ Proxmox Host │ │ (EX130-R: 24c/48t, 256GB) │ │ Helsinki, Finland │ └──────────────┬──────────────┘ │ ┌──────────────┴──────────────┐ │ Ubuntu 24.04 LTS VM │ │ (8 vCPU, 16GB RAM) │ │ │ │ ┌───────────┬───────────┐ │ │ │ /dev/sda │ /dev/sdb │ │ │ │ 60GB OS │ 200GB Data│ │ │ └───────────┴───────────┘ │ └──────────────┬──────────────┘ │ ``` ## Container Architecture ``` Docker Host ┌────────────────────────────────────────────────────────────────┐ │ │ │ ┌─────────────┐ traefik_proxy Network (172.18.0.0/16) │ │ │ Traefik │ ←─────────────────────────────────────────┐ │ │ │ (Router) │ │ │ │ └──────┬──────┘ │ │ │ │ │ │ │ ├──────→ [Kong Gateway] ←────┐ │ │ │ │ ↓ │ │ │ │ │ ┌─────┴─────┐ │ supabase_internal │ │ │ │ ├→ [Studio] │ │ Network │ │ │ │ ├→ [Auth] │ │ (172.19.0.0/16) │ │ │ │ ├→ [Rest] │ │ │ │ │ │ ├→ [Storage]│ │ │ │ │ │ ├→ [Meta] ├←────────┤ ↑ │ │ │ │ ├→ [Realtime] │ │ │ │ │ │ ├→ [Functions] │ [PostgreSQL] │ │ │ │ ├→ [Analytics] │ (Isolated) │ │ │ │ ├→ [Imgproxy] │ │ │ │ │ └→ [Vector] ─→ logs │ │ │ │ │ │ │ │ ├──────→ [Open WebUI] ←────────────────────────────┘ │ │ ├──────→ [LiteLLM] │ │ ├──────→ [n8n] │ │ ├──────→ [Gitea] ←─────── Git repositories & web UI │ │ │ ↓ │ │ │ ┌─────┴─────┐ Redis Networks │ │ │ │ Cache │ cache_redis_network (172.20.0.0) │ │ │ │ Sessions │ session_redis_network (172.21.0.0)│ │ │ └───────────┘ │ │ │ ↑ │ │ │ [Cache Redis] ←─── Ephemeral data caching │ │ │ [Session Redis] ←─ Persistent session storage │ │ │ │ │ ├──────→ [Rclone] ←────── Cloud sync & R2 backups │ │ ├──────→ [Prometheus/Grafana] ← Monitoring stack │ │ ├──────→ [StackWiz MCP] ←───── AI-powered stack mgmt │ │ ├──────→ [Apprise/ntfy] ←──── Notification stack │ │ ├──────→ [Metube] ←────────── YouTube downloader │ │ ├──────→ [Paperless-AI] ←──── AI document management │ │ ├──────→ [Watchtower] ←────── Automated updates │ │ ├──────→ [Authentik] ←─────── Identity/SSO management │ │ ├──────→ [Coolify] ←───────── Application deployment │ │ ├──────→ [Jellyfin] ←──────── Media streaming │ │ ├──────→ [Zurg/Rclone] ←───── Media mount/streaming │ │ ├──────→ [Zipline] ←───────── File sharing │ │ └──────→ [Other Services] │ │ │ │ Host Services: │ │ └── Ollama (Port 11434) ← Accessed by containers via gateway │ └────────────────────────────────────────────────────────────────┘ ``` ## Why This Architecture? ### 1. **Traefik as Reverse Proxy** **Decision**: Use Traefik instead of Nginx/Apache **Reasons**: - **Automatic SSL**: Integrates with Cloudflare DNS for wildcard certificates - **Docker Native**: Reads container labels for dynamic configuration - **No Manual Config**: Services self-register via Docker labels - **Path-Based Routing**: Single domain can serve multiple services **Alternative Considered**: Nginx Proxy Manager - Rejected because: Requires manual configuration for each service ### 2. **Multi-Network Architecture** **Decision**: Separate networks for different service types: `traefik_proxy`, `supabase_internal`, `cache_redis_network`, `session_redis_network` **Reasons**: - **Security**: Database never exposed to public network - **Performance**: Dedicated Redis networks for caching/sessions - **Flexibility**: Services can selectively connect to needed networks - **Debugging**: Clear network boundaries for troubleshooting ```yaml # Example: Service using multiple networks networks: - traefik_proxy # For public access - supabase_internal # For database access - cache_redis_network # For ephemeral caching - session_redis_network # For persistent sessions ``` ### 3. **Redis Consolidation Strategy** **Decision**: Two shared Redis instances instead of per-service Redis **Reasons**: - **Resource Efficiency**: Shared memory pool reduces overall RAM usage - **Operational Simplicity**: Two instances vs many individual Redis containers - **Performance**: Purpose-built configurations (cache vs sessions) - **Consistency**: Unified Redis management and monitoring **Implementation**: ```yaml # Cache Redis - Ephemeral data cache-redis: command: redis-server --save "" --appendonly no --maxmemory 4gb --maxmemory-policy allkeys-lru # Session Redis - Persistent data session-redis: command: redis-server --save 300 100 --appendonly yes --maxmemory 1gb --maxmemory-policy volatile-lru ``` ### 4. **Cloudflare DNS Challenge** **Decision**: Use DNS-01 challenge for SSL certificates **Reasons**: - **Wildcard Support**: One certificate for `*.rbnk.uk` - **No Port Exposure**: Works without opening port 80/443 - **Behind NAT**: Works even if server is behind firewall - **Automation**: Traefik handles renewal automatically **Trade-off**: Requires Cloudflare API credentials ### 5. **Storage Architecture** **Decision**: Separate OS and Data disks ``` /dev/sda (60GB) - OS Disk /dev/sdb (200GB) - Data Disk ├── / (System files) └── /srv/dockerdata ├── /boot ├── SERVICE_NAME/ ├── /usr │ ├── docker-compose.yml └── /var │ ├── .env │ └── data/ └── _scripts/ ``` **Reasons**: - **Isolation**: OS disk filling won't affect services - **Performance**: Data I/O doesn't compete with system - **Backup**: Easy to backup entire `/srv/dockerdata` - **Growth**: Data disk can be expanded independently ### 6. **Service Isolation Pattern** **Decision**: Each service in its own directory with dedicated compose file **Structure**: ``` /srv/dockerdata/SERVICE/ ├── docker-compose.yml # Service definition ├── .env # Environment variables (640 permissions) ├── data/ # Persistent data └── config/ # Configuration files ``` **Reasons**: - **Independence**: Services can be managed separately - **Portability**: Easy to move services between hosts - **Clarity**: Clear boundaries between services - **Backup**: Simple per-service backup strategy ## Network Flow ### External Request Flow ``` 1. User → https://app.rbnk.uk 2. Cloudflare DNS → Server IP 3. Traefik (Port 443) → Checks routing rules 4. Traefik → Routes to correct container 5. Container → Processes request 6. Response → Traefik → User ``` ### Internal Service Communication ``` 1. Service A → needs database 2. Service A → connects via supabase_internal network 3. PostgreSQL → only accessible on internal network 4. Response → back through internal network ``` ## Security Layers ### 1. **Network Level** - Internal services not exposed to public - Docker's built-in firewall rules - Host firewall (though not fully configured) ### 2. **Application Level** - JWT tokens for API authentication - Row-level security in PostgreSQL - Service-specific authentication ### 3. **Infrastructure Level** - SSL/TLS for all public endpoints - Environment variables for secrets - File permissions (though needs improvement) ## Scaling Considerations ### Current Limitations 1. **Single Host**: All services on one VM 2. **Vertical Scaling Only**: Add more RAM/CPU to VM 3. **Storage**: Fixed disk sizes (need to plan ahead) 4. **No Redundancy**: Single point of failure ### Future Growth Path 1. **Add More VMs**: Distribute services across multiple hosts 2. **External Database**: Move PostgreSQL to dedicated VM 3. **Object Storage**: Use S3 for large files 4. **Load Balancing**: Multiple instances behind Traefik ## Git Repository Architecture ### Gitea Integration **Decision**: Deploy Gitea as the primary Git service **Reasons**: - **Self-hosted**: Complete control over repositories and data - **Lightweight**: Low resource requirements (< 512MB RAM) - **Feature-rich**: Issues, pull requests, CI/CD integration - **API-driven**: Enables automation and integrations **Architecture**: ``` ┌─────────────────────────────────────────────────────────────┐ │ Git Repository Flow │ │ │ │ Developer ──push──→ Gitea (Private) │ │ │ │ │ ├── Webhook triggers │ │ │ │ │ ├── Backup System (2 AM daily) │ │ │ │ │ └── Sync to GitHub (automated) │ │ │ │ │ ↓ │ │ GitHub (Public Mirror) │ │ │ │ Security: .giteaignore filters sensitive files │ └─────────────────────────────────────────────────────────────┘ ``` ### Repository Sync Strategy **Decision**: Automated one-way sync from Gitea to GitHub **Implementation**: - Systemd timer runs every 15 minutes - Security filter prevents sensitive file exposure - Maintains commit history and authorship - Handles force pushes and branch deletions **Security Filtering**: ```bash # Files never synced to GitHub .env *.key *.pem *-secret* /private/ /credentials/ ``` ## Key Architectural Patterns ### 1. **Configuration as Code** Everything defined in files: - `docker-compose.yml` for services - `.env` for configuration - No clicking in UIs ### 2. **Consistent Structure** Every service follows same pattern: - Same directory layout - Same Docker label format - Same backup approach ### 3. **Automation First** - `stackwiz` for service creation - Backup scripts for data protection - Systemd timers for scheduling (needs implementation) ### 4. **Observability** Fully implemented monitoring stack: - **Prometheus**: Collects metrics from all services including Rclone backup metrics - **Grafana**: Visualization dashboards including dedicated Rclone Backup Monitoring dashboard - **Loki**: Log aggregation for centralized logging - **AlertManager**: Notifications for backup failures and system issues - **Node Exporter**: System metrics collection with custom Rclone textfile metrics - **Watchtower**: Monitors container updates with smart exclusions for databases and infrastructure ### 5. **Unified Notifications** Comprehensive notification system: - **Apprise**: Universal notification hub supporting 100+ services - **ntfy**: Push notification server with authentication and topics - **Mailrise**: SMTP gateway converting emails to push notifications - **Uptime Kuma**: Service monitoring with integrated alerting - **Integration**: All services can send notifications through multiple channels ### 6. **Content Management & Updates** Modern content and update management: - **Metube**: Web-based YouTube downloader with queue management and format selection - **Paperless-AI**: AI-powered document management with intelligent search and organization - **Watchtower**: Automated container updates with intelligent exclusions for critical services ## Architecture Decisions Summary | Decision | Rationale | Trade-off | |----------|-----------|-----------| | Traefik | Auto-SSL, Docker native | More complex than Nginx | | Multi Networks | Security isolation + performance | More complex networking | | Redis Consolidation | Resource efficiency, unified management | Shared dependency risk | | Cloudflare DNS | Wildcard SSL | Vendor lock-in | | Dual Disk | Separation of concerns | Fixed partition sizes | | Service Isolation | Independence | Some resource duplication | | Gitea | Self-hosted Git control | Additional service to maintain | | Dual Repository | Security + visibility | Sync complexity | | Unified Notifications | Central alert management | Multiple services to maintain | | Metube | Local media downloading | Storage space requirements | | Paperless-AI | AI document organization | AI processing overhead | | Watchtower | Automated updates | Requires careful exclusion management | ## Watchtower Update Strategy **Automated Updates with Protection**: Watchtower handles daily container updates while protecting critical services through exclusion labels: ```yaml # Critical services excluded from auto-updates labels: - "com.centurylinklabs.watchtower.enable=false" ``` **Protected Service Categories**: - **Database Services**: PostgreSQL, Redis instances require manual migration planning - **Infrastructure**: Traefik, monitoring stack need coordinated updates - **Security Services**: Authentication and DNS services require validation - **Self-Protection**: Watchtower excludes itself to prevent update conflicts **Benefits**: - Keeps non-critical services updated automatically - Prevents data corruption from unplanned database updates - Maintains system stability through infrastructure protection - Provides email notifications for all update activities ## Next Steps 1. **Expand Content Management**: Add more AI-powered content processing services 2. **Enhance Monitoring**: Integrate Watchtower metrics into Grafana dashboards 3. **Improve Automation**: Add more intelligent update policies 4. **Security Hardening**: Implement secrets management This architecture provides a solid foundation for a self-hosted infrastructure while maintaining flexibility for future growth and automated management. --- ## architecture > service-integrations ### Service Integration Patterns This document describes how new services integrate with the existing infrastructure components, showcasing patterns for authentication, monitoring, notifications, and data flow. ## Table of Contents 1. \1 2. \1 3. \1 4. \1 5. \1 6. \1 7. \1 ## Integration Overview The infrastructure follows a layered integration approach where services connect through: - **Docker Networks**: Container communication via bridge networks - **Traefik Proxy**: Centralized routing and SSL termination - **Notification Stack**: Unified alerting via Apprise/ntfy - **Monitoring Stack**: Metrics collection through Prometheus - **Backup System**: Automated data protection via Rclone ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Integration Layer Architecture │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Services │ ←→ │ Traefik │ ←→ │ Cloudflare │ │ │ │ │ │ (Proxy) │ │ (CDN) │ │ │ └──────┬───────┘ └──────────────┘ └──────────────┘ │ │ │ │ │ ↓ │ │ ┌──────┴───────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Notification │ │ Monitoring │ │ Backup │ │ │ │ Stack │ │ Stack │ │ System │ │ │ │ │ │ │ │ │ │ │ │ • Apprise │ │ • Prometheus │ │ • Rclone │ │ │ │ • ntfy │ │ • Grafana │ │ • R2 Storage │ │ │ │ • Mailrise │ │ • Loki │ │ • Cloud Sync │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ └─────────────────────────────────────────────────────────────────────────┘ ``` ## Core Integration Patterns ### 1. Docker Network Connectivity All services use a dual-network approach for secure isolation: ```yaml networks: default: name: traefik_proxy external: true # Optional internal network for database access internal: name: service_internal driver: bridge internal: true ``` **Pattern Benefits**: - External services accessible via Traefik - Database access isolated to internal networks - Clear separation of concerns ### 2. Traefik Integration Pattern Standard Traefik labels for HTTP services: ```yaml labels: - "traefik.enable=true" - "traefik.docker.network=traefik_proxy" # HTTP redirect - "traefik.http.routers.service-http.rule=Host(`service.rbnk.uk`)" - "traefik.http.routers.service-http.entrypoints=web" - "traefik.http.routers.service-http.middlewares=redirect-to-https@file" # HTTPS - "traefik.http.routers.service.rule=Host(`service.rbnk.uk`)" - "traefik.http.routers.service.entrypoints=websecure" - "traefik.http.routers.service.tls.certresolver=cf" - "traefik.http.services.service.loadbalancer.server.port=8080" ``` ### 3. Monitoring Integration Pattern Services expose metrics for Prometheus collection: ```yaml labels: # Enable Prometheus scraping - "prometheus.io/scrape=true" - "prometheus.io/port=9090" - "prometheus.io/path=/metrics" # Custom job label for organization - "prometheus_job=service-metrics" ``` ### 4. Redis Integration Pattern Services integrate with consolidated Redis instances for caching and session management: #### A. Cache Redis Integration (Ephemeral Data) ```yaml # Service connecting to cache Redis environment: - REDIS_CACHE_URL=redis://:${REDIS_PASSWORD}@cache-redis:6379/0 - CACHE_BACKEND=redis networks: - cache_redis_network # Connect to cache Redis network ``` **Cache Redis Configuration**: - **Purpose**: Temporary data, API responses, computed results - **Persistence**: Disabled (no disk writes) - **Memory Policy**: `allkeys-lru` (evict least recently used) - **Max Memory**: 4GB (configurable) - **Port**: 6380 (host) → 6379 (container) #### B. Session Redis Integration (Persistent Data) ```yaml # Service connecting to session Redis environment: - REDIS_SESSION_URL=redis://:${REDIS_PASSWORD}@session-redis:6379/0 - SESSION_STORE=redis networks: - session_redis_network # Connect to session Redis network ``` **Session Redis Configuration**: - **Purpose**: User sessions, authentication tokens, persistent state - **Persistence**: Enabled (AOF + RDB snapshots) - **Memory Policy**: `volatile-lru` (evict expired keys first) - **Max Memory**: 1GB (configurable) - **Port**: 6381 (host) → 6379 (container) #### C. Multi-Redis Service Pattern ```yaml # Service using both Redis instances environment: - REDIS_CACHE_URL=redis://:${REDIS_PASSWORD}@cache-redis:6379/0 - REDIS_SESSION_URL=redis://:${REDIS_PASSWORD}@session-redis:6379/0 networks: - cache_redis_network - session_redis_network ``` **Use Cases**: - **Gitea**: Cache for Git operations, sessions for web UI - **Web Applications**: API response caching + user session storage - **Background Jobs**: Task queues (cache) + job status (session) ### 5. Notification Integration Pattern Services integrate with notification stack via multiple methods: #### A. Direct API Integration ```bash # Send notification via Apprise curl -X POST https://apprise.rbnk.uk/notify/apprise \ -H "Content-Type: application/json" \ -d '{"title": "Service Alert", "body": "Status update"}' ``` #### B. ntfy Topic Publishing ```bash # Publish to ntfy topic with authentication curl -X POST https://ntfy.rbnk.uk/service-warning-disk \ -H "Authorization: Bearer $NTFY_TOKEN" \ -d "Disk usage at 85%" ``` #### C. SMTP Gateway via Mailrise ```bash # Send email that converts to push notification echo "Service notification" | mail -s "Alert" service@localhost:8025 ``` ## Service-Specific Integrations ### Gitea + Redis Integration **Purpose**: Performance optimization through caching and session management ```yaml # Gitea with dual Redis connections services: gitea: environment: # Cache Redis for Git operations - GITEA__cache__ENABLED=true - GITEA__cache__ADAPTER=redis - GITEA__cache__HOST=redis://:${REDIS_PASSWORD}@cache-redis:6379/0 # Session Redis for web sessions - GITEA__session__PROVIDER=redis - GITEA__session__PROVIDER_CONFIG=redis://:${REDIS_PASSWORD}@session-redis:6379/1 networks: - traefik_proxy - cache_redis_network # For cache operations - session_redis_network # For session storage ``` **Performance Benefits**: ```bash # Cache Redis handles: # - Git object caching # - Repository metadata # - API response caching # Session Redis handles: # - User login sessions # - CSRF tokens # - Web UI state ``` **Monitoring Integration**: ```bash # Redis metrics available via Prometheus redis_connected_clients{instance="cache-redis"} redis_memory_used_bytes{instance="session-redis"} redis_keyspace_hits_total{instance="cache-redis"} ``` ### Metube + Rclone Integration **Purpose**: Automated backup of downloaded media files ```yaml # Metube volumes shared with Rclone volumes: - ./downloads:/downloads - ./config:/config # Rclone access to Metube downloads volumes: - /srv/dockerdata/metube/downloads:/backup/metube:ro ``` **Backup Automation**: ```bash #!/bin/bash # Automated sync of Metube downloads to cloud storage docker exec rclone rclone sync /backup/metube gdrive:media/metube \ --include "*.mp4" \ --include "*.mkv" \ --include "*.avi" \ --transfers 2 \ --bwlimit 10M ``` ### Watchtower + Notification Integration **Configuration**: Watchtower sends update notifications via notification stack ```yaml environment: # Email notifications via Mailrise SMTP gateway - WATCHTOWER_NOTIFICATIONS=email - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=mailrise:8025 - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@rbnk.uk - WATCHTOWER_NOTIFICATION_EMAIL_TO=admin@localhost ``` **Enhanced Notification Flow**: ``` Watchtower Update → SMTP (Mailrise:8025) → Apprise API → ntfy Topic → Push Notification ``` ### Paperless-AI + n8n Workflow Integration **Document Processing Automation**: 1. **Webhook Trigger**: Paperless-AI sends webhook on document upload 2. **n8n Processing**: Workflow extracts metadata and processes content 3. **Notification**: Results sent via notification stack ```yaml # Paperless-AI webhook configuration environment: - WEBHOOK_URL=https://n8n.rbnk.uk/webhook/paperless-process - POST_PROCESSING_ENABLED=true ``` **n8n Workflow Pattern**: ```json { "nodes": [ { "name": "Webhook", "type": "n8n-nodes-base.webhook", "parameters": { "path": "paperless-process" } }, { "name": "Process Document", "type": "n8n-nodes-base.function", "parameters": { "functionCode": "// Extract document metadata\nreturn items.map(item => ({\n json: {\n documentId: item.json.id,\n title: item.json.title,\n processed: true\n }\n}));" } }, { "name": "Send Notification", "type": "n8n-nodes-base.httpRequest", "parameters": { "url": "https://apprise.rbnk.uk/notify/apprise", "method": "POST", "jsonParameters": true, "options": { "bodyParametersUi": { "parameter": [ { "name": "title", "value": "Document Processed" }, { "name": "body", "value": "Document {{$node.Process Document.json.title}} has been processed" } ] } } } } ] } ``` ## Network Architecture ### Docker Network Topology ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Docker Host (10.10.10.10) │ │ │ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ │ traefik_proxy (172.18.0.0/16) │ │ │ │ │ │ │ │ ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌──────────────────┐ │ │ │ │ │ Traefik │ │ Metube │ │ Rclone │ │ Paperless-AI │ │ │ │ │ │ .18.2 │ │ .18.10 │ │ .18.11 │ │ .18.12 │ │ │ │ │ └─────────┘ └──────────┘ └─────────┘ └──────────────────┘ │ │ │ │ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌─────────┐ ┌──────────────────┐ │ │ │ │ │Watchtower│ │ n8n │ │Apprise │ │ Monitoring │ │ │ │ │ │ .18.13 │ │ .18.4 │ │ .18.14 │ │ Stack │ │ │ │ │ └──────────┘ └──────────┘ └─────────┘ └──────────────────┘ │ │ │ └──────────────────────────────────────────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────────────────────────┐ │ │ │ supabase_internal (172.19.0.0/16) │ │ │ │ (No Internet Access) │ │ │ │ │ │ │ │ ┌─────────────┐ ┌──────────┐ ┌──────────┐ │ │ │ │ │ PostgreSQL │ │ Auth │ │ REST │ │ │ │ │ │ .19.2 │ │ .19.3 │ │ .19.4 │ │ │ │ │ └─────────────┘ └──────────┘ └──────────┘ │ │ │ └──────────────────────────────────────────────────────────────────┘ │ │ │ │ Host Services: │ │ ├── Docker Daemon (unix:///var/run/docker.sock) │ │ ├── Ollama (127.0.0.1:11434) │ │ └── Node Exporter (127.0.0.1:9100) │ └─────────────────────────────────────────────────────────────────────────┘ ``` ### Service Discovery Patterns #### 1. Container-to-Container Communication ```yaml # Service A connecting to Service B environment: - SERVICE_B_URL=http://service-b:8080 # Uses Docker's built-in DNS resolution ``` #### 2. Container-to-Host Service ```yaml # Container connecting to host service (Ollama) environment: - OLLAMA_URL=http://172.17.0.1:11434 # 172.17.0.1 is Docker's default gateway to host ``` #### 3. Container-to-Database ```yaml # Connection via internal network environment: - DATABASE_URL=postgresql://user:pass@supa-db:5432/dbname networks: - supabase_internal # Required for database access ``` ## API Integration Examples ### 1. Rclone Remote Control API **Purpose**: Enable other services to trigger backup operations ```bash # Start backup via Rclone RC API curl -X POST http://rclone:5572/sync/sync \ -H "Content-Type: application/json" \ -d '{ "srcFs": "/data/metube", "dstFs": "gdrive:backups/metube", "_async": true }' ``` **Integration in n8n**: ```json { "name": "Trigger Backup", "type": "n8n-nodes-base.httpRequest", "parameters": { "url": "http://rclone:5572/sync/sync", "method": "POST", "jsonParameters": true, "options": { "bodyParametersUi": { "parameter": [ { "name": "srcFs", "value": "/data/{{$node.Previous.json.folder}}" }, { "name": "dstFs", "value": "cloudflare:backups/{{$node.Previous.json.folder}}" }, { "name": "_async", "value": true } ] } } } } ``` ### 2. Supabase API Integration **Authentication Flow**: ```javascript // Service authentication with Supabase const response = await fetch('https://supabase.rbnk.uk/auth/v1/token', { method: 'POST', headers: { 'Content-Type': 'application/json', 'apikey': process.env.SUPABASE_ANON_KEY }, body: JSON.stringify({ email: 'service@example.com', password: process.env.SERVICE_PASSWORD }) }); ``` **Data Operations**: ```javascript // Insert service data const data = await fetch('https://supabase.rbnk.uk/rest/v1/service_logs', { method: 'POST', headers: { 'Content-Type': 'application/json', 'apikey': process.env.SUPABASE_ANON_KEY, 'Authorization': `Bearer ${accessToken}`, 'Prefer': 'return=representation' }, body: JSON.stringify({ service: 'metube', action: 'download_complete', metadata: { filename: 'video.mp4', size: 1024000 } }) }); ``` ### 3. Notification API Integration **Unified Notification Function**: ```bash #!/bin/bash # notification.sh - Unified notification script notify() { local title="$1" local message="$2" local priority="${3:-normal}" local service="${4:-general}" # Send to Apprise curl -s -X POST https://apprise.rbnk.uk/notify/apprise \ -H "Content-Type: application/json" \ -d "{\"title\": \"$title\", \"body\": \"$message\"}" # Send to ntfy topic local topic="${service}-${priority}-alert" curl -s -X POST "https://ntfy.rbnk.uk/$topic" \ -H "Authorization: Bearer $NTFY_TOKEN" \ -d "$message" } # Usage examples notify "Backup Complete" "Metube backup finished successfully" "info" "backup" notify "Disk Warning" "Disk usage at 85%" "warning" "monitoring" notify "Service Error" "Paperless-AI processing failed" "error" "paperless" ``` ## Monitoring & Observability ### Metrics Collection Flow ``` ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐ │ Service │───▶│ Metrics │───▶│ Prometheus │───▶│ Grafana │ │ (Container) │ │ Endpoint │ │ (Scraper) │ │ (Dashboard) │ └─────────────┘ └──────────────┘ └─────────────┘ └──────────────┘ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ Log Output │ │ AlertManager│ │ (Docker) │ │ (Alerts) │ └─────────────┘ └─────────────┘ │ │ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ Loki │ │Notification │ │ (Log Store) │ │ Stack │ └─────────────┘ └─────────────┘ ``` ### Service Metrics Examples #### Metube Metrics ```yaml # Custom metrics via textfile collector - name: metube_downloads_total type: counter help: Total number of downloads - name: metube_download_size_bytes type: histogram help: Size of downloaded files - name: metube_queue_length type: gauge help: Current download queue length ``` #### Rclone Backup Metrics ```yaml # Generated by backup scripts - name: rclone_backup_duration_seconds type: histogram help: Time taken for backup operations - name: rclone_backup_bytes_transferred type: counter help: Total bytes transferred - name: rclone_backup_success type: gauge help: Last backup success status (1=success, 0=failure) ``` ### Log Aggregation Patterns **Structured Logging Configuration**: ```yaml # Docker logging driver logging: driver: "json-file" options: max-size: "10m" max-file: "3" labels: "service,environment" ``` **Loki Labels**: ```yaml # Promtail config for service discovery - job_name: containers docker_sd_configs: - host: unix:///var/run/docker.sock relabel_configs: - source_labels: ['__meta_docker_container_name'] target_label: 'container' - source_labels: ['__meta_docker_container_label_com_docker_compose_service'] target_label: 'service' ``` ## Configuration Examples ### Complete Service Integration Template ```yaml # docker-compose.yml for new service services: my-service: image: myservice:latest container_name: my-service restart: unless-stopped # Environment configuration environment: - SERVICE_PORT=8080 - DATABASE_URL=postgresql://user:pass@supa-db:5432/mydb - NOTIFICATION_URL=https://apprise.rbnk.uk/notify/apprise - BACKUP_ENABLED=true # Redis integration (optional) - REDIS_CACHE_URL=redis://:${REDIS_PASSWORD}@cache-redis:6379/0 - REDIS_SESSION_URL=redis://:${REDIS_PASSWORD}@session-redis:6379/0 # Network connectivity networks: - traefik_proxy # External access - supabase_internal # Database access - cache_redis_network # Cache Redis (optional) - session_redis_network # Session Redis (optional) # Volume mounts volumes: - ./data:/app/data - ./config:/app/config # Traefik integration labels: - "traefik.enable=true" - "traefik.docker.network=traefik_proxy" - "traefik.http.routers.myservice-http.rule=Host(`myservice.rbnk.uk`)" - "traefik.http.routers.myservice-http.entrypoints=web" - "traefik.http.routers.myservice-http.middlewares=redirect-to-https@file" - "traefik.http.routers.myservice.rule=Host(`myservice.rbnk.uk`)" - "traefik.http.routers.myservice.entrypoints=websecure" - "traefik.http.routers.myservice.tls.certresolver=cf" - "traefik.http.services.myservice.loadbalancer.server.port=8080" # Monitoring integration - "prometheus.io/scrape=true" - "prometheus.io/port=9090" - "prometheus.io/path=/metrics" - "prometheus_job=myservice-metrics" # Backup exclusion (if needed) - "backup.exclude=true" networks: traefik_proxy: external: true supabase_internal: external: true cache_redis_network: external: true session_redis_network: external: true ``` ### Backup Integration Script ```bash #!/bin/bash # backup-myservice.sh set -euo pipefail SERVICE_NAME="my-service" BACKUP_DATE=$(date +%Y%m%d-%H%M%S) REMOTE="cloudflare" log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" } # Backup service data log "Backing up $SERVICE_NAME data..." docker exec rclone rclone sync "/backup/dockerdata/$SERVICE_NAME/data" \ "$REMOTE:service-backups/$SERVICE_NAME/$BACKUP_DATE/" \ --exclude "*/tmp/**" \ --exclude "*/cache/**" \ --transfers 4 \ --checkers 8 # Send success notification curl -s -X POST https://apprise.rbnk.uk/notify/apprise \ -H "Content-Type: application/json" \ -d "{\"title\": \"$SERVICE_NAME Backup Complete\", \"body\": \"Backup completed successfully at $BACKUP_DATE\"}" log "$SERVICE_NAME backup completed" ``` ### Health Check Integration ```yaml # Health check configuration healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/health"] interval: 30s timeout: 10s retries: 3 start_period: 40s # Custom health check script # healthcheck.sh #!/bin/bash # Check service health and send alerts if ! curl -sf http://localhost:8080/health > /dev/null; then curl -X POST https://ntfy.rbnk.uk/myservice-error-health \ -H "Authorization: Bearer $NTFY_TOKEN" \ -d "Service health check failed" exit 1 fi ``` ### Alerting Rules Integration ```yaml # prometheus-alerts.yml groups: - name: myservice rules: - alert: MyServiceDown expr: up{job="myservice-metrics"} == 0 for: 2m labels: severity: critical service: myservice annotations: summary: "MyService is down" description: "MyService has been down for more than 2 minutes" - alert: MyServiceHighErrorRate expr: rate(myservice_errors_total[5m]) > 0.1 for: 5m labels: severity: warning service: myservice annotations: summary: "High error rate in MyService" description: "Error rate is {{ $value }} errors per second" ``` This comprehensive integration guide ensures new services follow established patterns while maintaining security, observability, and reliability standards across the infrastructure. --- ## architecture > storage-strategy ### Storage Strategy - Dual Disk Architecture ## Current Storage Layout ``` ┌─────────────────────────────────────────────────────────────┐ │ Proxmox VM: dockerhost │ │ │ │ ┌─────────────────────┐ ┌─────────────────────────┐ │ │ │ /dev/sda (100GB) │ │ /dev/sdb (500GB) │ │ │ │ System Disk │ │ Data Disk │ │ │ ├─────────────────────┤ ├─────────────────────────┤ │ │ │ sda1: /boot/efi 1GB │ │ sdb1: /srv/dockerdata │ │ │ │ sda2: /boot 2GB │ │ 492GB formatted │ │ │ │ sda3: LVM 97GB │ │ 288GB available │ │ │ │ └── / (96GB) │ │ │ │ │ │ 29% used │ │ 39% used │ │ │ └─────────────────────┘ └─────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ ``` ## Why This Dual-Disk Design? ### Original Setup Intent Based on the Proxmox configuration: ``` Disks: • scsi0 (OS): 60GB on vmdata • scsi1 (Data): 200GB on vmdata (thin provisioned) ``` **Design Rationale**: 1. **Separation of Concerns**: OS vs Application Data 2. **Independent Growth**: Data disk can expand without OS reinstall 3. **Performance**: Separate I/O paths for system and data 4. **Backup Strategy**: Different backup policies for OS vs data ### Current Reality **System Disk (100GB)**: - **96GB root partition**: 29% used (~66GB free) - **Status**: Healthy with ample headroom - **Contains**: OS, system packages, logs, Docker engine **Data Disk (500GB)**: - **492GB usable**: ~39% used (288GB available) - **Mounted at**: `/srv/dockerdata` - **Contains**: All container data, configs, backups ## Storage Organization ### Directory Structure ``` /srv/dockerdata/ ├── SERVICE_NAME/ # Each service isolated │ ├── docker-compose.yml # Service definition │ ├── .env # Configuration (640 perms) │ ├── data/ # Persistent data │ │ ├── database/ # Database files │ │ ├── uploads/ # User uploads │ │ └── cache/ # Temporary data │ └── config/ # Configuration files ├── _scripts/ # Operational scripts ├── _templates/ # Service templates ├── _backup/ # Backup storage └── ollama/ # Special case - models └── models/ # Large LLM files ``` ### Storage Patterns by Service Type **Database Services** (PostgreSQL): ``` supabase/data/ ├── postgres/ # ~500MB-5GB typical ├── wal_archive/ # Transaction logs └── backups/ # Local dumps ``` **Application Services**: ``` open-webui/data/ ├── backend/ # SQLite DB, session data └── uploads/ # User uploaded files ``` **Large File Services**: ``` ollama/models/ ├── llama2:7b/ # 3.8GB ├── mistral:latest/ # 4.1GB └── llama2:70b/ # 40GB (careful!) ``` ## Storage Management ### Regular Maintenance Keep storage healthy with periodic cleanup: ```bash # Check what's using space sudo du -h --max-depth=1 / | sort -h du -sh /srv/dockerdata/* | sort -h # Clean Docker artifacts (unused images, containers, volumes) docker system prune -a --volumes # Clean package cache sudo apt clean sudo apt autoremove # Rotate/compress logs sudo journalctl --vacuum-time=14d ``` ### Optional: Move Docker Root to Data Disk If system disk becomes constrained again, move Docker data: ```bash # Stop Docker sudo systemctl stop docker # Move Docker data sudo mv /var/lib/docker /srv/dockerdata/docker-root # Configure Docker to use new location sudo tee /etc/docker/daemon.json < 85%: Critical, act immediately - Data disk > 70%: Plan expansion - Any partition > 90%: Emergency ### How to Expand Data Disk ```bash # On Proxmox host qm resize 100 scsi1 +100G # In VM sudo growpart /dev/sdb 1 sudo resize2fs /dev/sdb1 ``` ### Migration Path When disks can't grow: 1. **Add third disk**: Dedicated for specific service 2. **External storage**: NFS/S3 for large files 3. **New VM**: Distribute services across VMs ## Best Practices ### 1. **Separation Rules** - OS disk: System only - Data disk: All application data - External: Backups and archives ### 2. **Monitoring** - Daily disk usage checks - Weekly growth analysis - Monthly capacity planning ### 3. **Cleanup Routine** ```bash # Weekly cleanup script #!/bin/bash docker system prune -f sudo journalctl --vacuum-time=14d find /srv/dockerdata -name "*.log" -mtime +30 -delete ``` ### 4. **Documentation** - Track which services use most space - Document growth patterns - Plan before hitting limits ## Recommendations ### Ongoing Maintenance 1. Monitor disk usage weekly via Grafana dashboards 2. Run cleanup scripts monthly to remove unused Docker artifacts 3. Review storage growth trends quarterly ### Best Practices 1. Implement automated cleanup scripts 2. Configure log rotation for all services 3. Maintain external backup destination (Cloudflare R2) ### Future Considerations 1. Consider moving Docker root to data disk if system disk usage increases 2. Continue monitoring storage metrics in Grafana 3. Maintain disaster recovery plan with regular backup testing ## Storage Decision Matrix | What to Store | System Disk | Data Disk | External | |---------------|-------------|-----------|----------| | OS Files | ✓ | | | | Docker Engine | ✓ | | | | Container Images | | ✓ | | | Application Data | | ✓ | | | Database Files | | ✓ | | | Backups | | ✓ | ✓ (R2) | | Large Models (LLM) | | ✓ | | | Logs (archived) | | | ✓ | The dual-disk strategy provides a solid foundation with ample capacity for growth. --- ## coolify-traefik-bridge ### Coolify-Traefik Bridge Automated solution for routing Coolify services through the main Traefik proxy with automatic DNS record creation. ## Overview The Coolify-Traefik Bridge script automatically: 1. Detects all Coolify-managed containers 2. Connects them to the main Traefik network 3. Generates Traefik routing configurations 4. Creates DNS records in Cloudflare ## How It Works 1. **Service Detection**: Finds containers with `coolify.managed=true` label 2. **Network Connection**: Ensures containers are on `traefik_proxy` network 3. **Config Generation**: Creates Traefik dynamic configs in `/srv/dockerdata/traefik/dynamic/` 4. **DNS Creation**: Automatically creates A records pointing to server IP ## Usage ### Manual Run (for testing) ```bash /srv/dockerdata/_scripts/coolify-traefik-bridge.sh true ``` ### Enable as Service ```bash sudo systemctl daemon-reload sudo systemctl enable coolify-traefik-bridge sudo systemctl start coolify-traefik-bridge ``` ### Check Service Status ```bash sudo systemctl status coolify-traefik-bridge journalctl -u coolify-traefik-bridge -f ``` ## Configuration - **Watch Interval**: 30 seconds (configurable in script) - **Config Directory**: `/srv/dockerdata/traefik/dynamic/` - **DNS Script**: `/srv/dockerdata/_scripts/cloudflare-dns-create.sh` ## Generated Files For each Coolify service, creates: - `/srv/dockerdata/traefik/dynamic/coolify-.yml` ## Troubleshooting ### Service Not Accessible 1. Check DNS propagation: `dig +short .rbnk.uk` 2. Verify Traefik config: `docker exec traefik cat /etc/traefik/dynamic/coolify-.yml` 3. Check container is on network: `docker inspect | jq '.[0].NetworkSettings.Networks'` ### Long Domain Names Some Coolify services generate long UUIDs in domains. The script handles these automatically but they may look unusual (e.g., `audiobookshelf-xosc0wsscooo40cgwco44ggw.rbnk.uk`). ## Benefits - **Zero Manual Configuration**: Just deploy in Coolify, bridge handles the rest - **Automatic DNS**: No need to manually create DNS records - **SSL Certificates**: Automatically handled by main Traefik - **Unified Routing**: All services go through main Traefik proxy ## Security Notes - Services are automatically connected to `traefik_proxy` network - SSL termination handled by main Traefik with Cloudflare certificates - No ports exposed directly from containers --- ## development > adding-services ### Adding Services Guide This guide covers the process of adding new services to the infrastructure, from initial setup to production deployment. ## Table of Contents - \1 - \1 - \1 - \1 - \1 - \1 - \1 - \1 - \1 ## When to Use stackwiz vs Manual Setup ### Use stackwiz when: - Service follows standard patterns (web app with database) - You need Traefik integration out of the box - Creating Pocketbase-based applications - You want automatic DNS record creation - You want consistent directory structure ```bash # Example: Adding a new web application stackwiz mywebapp # Or with Pocketbase backend stackwiz --pocketbase mywebapp # Legacy simple template (basic cases only) ./_templates/mkstack.sh mywebapp ``` ### Manual setup when: - Complex multi-service stacks (like Supabase) - Special networking requirements - Custom initialization scripts needed - Non-standard port configurations ## Docker Compose Best Practices ### 1. Service Naming Conventions Use clear, descriptive names that indicate the service purpose: ```yaml services: myapp-web: # Main application myapp-db: # Database myapp-redis: # Cache layer myapp-worker: # Background jobs ``` ### 2. Version and Structure Always use Compose file version 3.8+ for full feature support: ```yaml version: '3.8' services: # Services defined here networks: # Network configuration volumes: # Named volumes ``` ### 3. Health Checks Implement health checks for critical services: ```yaml services: myapp-web: image: myapp:latest healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3000/health"] interval: 30s timeout: 10s retries: 3 start_period: 40s ``` ### 4. Resource Limits Set appropriate resource constraints: ```yaml services: myapp-web: deploy: resources: limits: cpus: '2.0' memory: 2G reservations: cpus: '0.5' memory: 512M ``` ## Traefik Integration Patterns ### Basic Web Service Standard HTTP service with automatic SSL: ```yaml services: myapp: image: myapp:latest networks: - traefik_proxy labels: - "traefik.enable=true" - "traefik.docker.network=traefik_proxy" - "traefik.http.routers.myapp.rule=Host(`myapp.rbnk.uk`)" - "traefik.http.routers.myapp.entrypoints=websecure" - "traefik.http.routers.myapp.tls=true" - "traefik.http.routers.myapp.tls.certresolver=cloudflare" - "traefik.http.services.myapp.loadbalancer.server.port=3000" ``` ### Service with Multiple Routes Example from Supabase with path-based routing: ```yaml labels: # Main studio interface - "traefik.http.routers.supabase-studio.rule=Host(`supabase.rbnk.uk`)" # API endpoints - "traefik.http.routers.supabase-rest.rule=Host(`supabase.rbnk.uk`) && PathPrefix(`/rest/v1`)" - "traefik.http.routers.supabase-auth.rule=Host(`supabase.rbnk.uk`) && PathPrefix(`/auth/v1`)" # Different backend services - "traefik.http.services.supabase-rest.loadbalancer.server.port=3000" - "traefik.http.services.supabase-auth.loadbalancer.server.port=9999" ``` ### WebSocket Support For services requiring WebSocket connections: ```yaml labels: - "traefik.http.routers.myapp-ws.rule=Host(`myapp.rbnk.uk`) && PathPrefix(`/ws`)" - "traefik.http.routers.myapp-ws.service=myapp-ws" - "traefik.http.services.myapp-ws.loadbalancer.server.port=3000" # Enable sticky sessions for WebSockets - "traefik.http.services.myapp-ws.loadbalancer.sticky=true" - "traefik.http.services.myapp-ws.loadbalancer.sticky.cookie.name=myapp_ws" ``` ## Network Configuration Decisions ### 1. External vs Internal Networks **External (traefik_proxy)**: For services that need public access ```yaml networks: traefik_proxy: external: true ``` **Internal**: For service-to-service communication ```yaml networks: internal: driver: bridge internal: true ``` ### 2. Multi-Network Services Services can connect to multiple networks: ```yaml services: myapp-api: networks: - traefik_proxy # Public access - supabase_default # Database access - internal # Internal services ``` ### 3. Network Aliases Use aliases for service discovery: ```yaml services: myapp-db: networks: internal: aliases: - database - postgres ``` ## Volume and Data Management ### 1. Named Volumes vs Bind Mounts **Named Volumes** (Preferred for data): ```yaml volumes: myapp-data: driver: local services: myapp: volumes: - myapp-data:/var/lib/myapp ``` **Bind Mounts** (For config files): ```yaml services: myapp: volumes: - ./config/myapp.conf:/etc/myapp/myapp.conf:ro - ./data:/data ``` ### 2. Backup Considerations Structure volumes for easy backup: ```yaml services: myapp-db: volumes: # All data in one parent directory - ./data/postgres:/var/lib/postgresql/data - ./data/backups:/backups ``` ### 3. Init Scripts Pattern From Supabase bootstrap example: ```yaml volumes: - ./bootstrap-sql:/docker-entrypoint-initdb.d:ro ``` Files execute alphabetically: ``` bootstrap-sql/ ├── 00-initial-schema.sql ├── 01-core-tables.sql ├── 02-auth-setup.sql └── 03-app-tables.sql ``` ## Environment Variable Management ### 1. Environment File Structure ```bash # Stack-specific .env file touch myapp/.env chmod 640 myapp/.env ``` Example .env: ```env # Database POSTGRES_USER=myapp POSTGRES_PASSWORD=secure_password_here POSTGRES_DB=myapp # Application APP_SECRET_KEY=generate_with_openssl_rand API_KEY=another_secure_key # External Services REDIS_URL=redis://myapp-redis:6379 DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@myapp-db:5432/${POSTGRES_DB} ``` ### 2. Sharing Secrets Between Services Use common environment variables: ```yaml services: myapp-web: env_file: .env environment: - DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@myapp-db:5432/${POSTGRES_DB} myapp-worker: env_file: .env environment: - DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@myapp-db:5432/${POSTGRES_DB} ``` ### 3. Secret Generation ```bash # Generate secure passwords openssl rand -base64 32 # Generate JWT secrets openssl rand -hex 64 # Generate API keys uuidgen | tr -d '-' # For LiteLLM services, use virtual keys instead of master key # See: LiteLLM Virtual Keys documentation ``` ### 4. LiteLLM Integration If your service needs LLM access, use virtual keys instead of the master key: ```bash # Create virtual key via API (see LiteLLM Virtual Keys guide) curl -X POST "https://llm.rbnk.uk/key/generate" \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ -H "Content-Type: application/json" \ -d '{ "budget_id": "service-budget-id", "metadata": {"service": "myapp"} }' ``` For complete virtual key implementation details, see: \1 ## Testing New Services ### 1. Local Testing First ```bash # Test without Traefik docker compose -f myapp/docker-compose.yml up # Check service is accessible curl http://localhost:3000/health # Check logs docker compose -f myapp/docker-compose.yml logs -f ``` ### 2. Network Testing ```bash # Verify network connectivity docker compose -f myapp/docker-compose.yml exec myapp-web ping myapp-db # Test database connection docker compose -f myapp/docker-compose.yml exec myapp-web psql $DATABASE_URL -c "SELECT 1" ``` ### 3. Traefik Integration Testing ```bash # Add to docker-compose.yml and restart docker compose -f myapp/docker-compose.yml up -d # Check Traefik picked up the service curl http://localhost:8080/api/http/routers | jq '.[] | select(.rule | contains("myapp"))' # Test SSL endpoint curl -v https://myapp.rbnk.uk/health ``` ### 4. Load Testing ```bash # Simple load test ab -n 1000 -c 10 https://myapp.rbnk.uk/ # Monitor resources during test docker stats myapp-myapp-web-1 ``` ## Documentation Requirements ### 1. Service README Create `myapp/README.md`: ```markdown # MyApp Service ## Overview Brief description of what this service does ## Architecture - Main application: Node.js/Python/etc - Database: PostgreSQL - Cache: Redis (if applicable) ## Configuration Key environment variables: - `DATABASE_URL`: PostgreSQL connection string - `REDIS_URL`: Redis connection string - `API_KEY`: External API authentication ## Endpoints - `https://myapp.rbnk.uk` - Main web interface - `https://myapp.rbnk.uk/api` - REST API ## Maintenance ### Backup Automated via `_scripts/backup-myapp.sh` ### Updates ```bash docker compose -f myapp/docker-compose.yml pull docker compose -f myapp/docker-compose.yml up -d ``` ## Troubleshooting Common issues and solutions ``` ### 2. Update Main Documentation Add to `/srv/dockerdata/README.md`: - Service entry in services table - URL in endpoints section - Any special considerations ## Git Workflow for Infrastructure Changes ### 1. Initial Repository Setup ```bash # Configure dual remotes (one-time setup) cd /srv/dockerdata /srv/dockerdata/_scripts/setup-git-remotes.sh # This creates: # - origin → Gitea (primary) # - gitea → Gitea (explicit) # - github → GitHub (optional mirror) ``` ### 2. Feature Branch Workflow ```bash # Create feature branch git checkout -b add-myapp-service # Make changes vim myapp/docker-compose.yml vim myapp/.env.example # Never commit real .env # Test thoroughly docker compose -f myapp/docker-compose.yml up -d # Commit with clear message git add myapp/ git commit -m "Add MyApp service with PostgreSQL and Redis - Web interface at myapp.rbnk.uk - Includes automated backups - Integrated with Traefik for SSL" # Push to Gitea (primary) git push -u origin add-myapp-service # Or push to all remotes git push-all ``` ### 3. Environment File Handling ```bash # Create example file cp myapp/.env myapp/.env.example # Edit to remove sensitive values vim myapp/.env.example # Add to .gitignore echo "myapp/.env" >> .gitignore ``` ### 4. Pre-deployment Checklist - [ ] Service starts successfully - [ ] Health checks pass - [ ] Traefik routing works - [ ] SSL certificate obtained - [ ] Backups configured - [ ] Monitoring/alerts set up - [ ] Documentation complete - [ ] .env.example created - [ ] No secrets in commits ### 5. Deployment Process ```bash # On production server git pull origin main # Copy environment file cp myapp/.env.example myapp/.env vim myapp/.env # Add real values # Deploy docker compose -f myapp/docker-compose.yml up -d # Verify docker compose -f myapp/docker-compose.yml ps curl https://myapp.rbnk.uk/health ``` ## Common Patterns and Examples ### Database-Backed Web App See `_templates/stack-template/` for the standard pattern ### Microservices Stack See `supabase/` for complex multi-service example ### Simple Static Site ```yaml version: '3.8' services: static-site: image: nginx:alpine volumes: - ./public:/usr/share/nginx/html:ro - ./nginx.conf:/etc/nginx/nginx.conf:ro networks: - traefik_proxy labels: - "traefik.enable=true" - "traefik.docker.network=traefik_proxy" - "traefik.http.routers.static.rule=Host(`static.rbnk.uk`)" - "traefik.http.routers.static.entrypoints=websecure" - "traefik.http.routers.static.tls=true" - "traefik.http.routers.static.tls.certresolver=cloudflare" ``` ### API with Rate Limiting ```yaml labels: # Standard routing - "traefik.http.routers.api.rule=Host(`api.rbnk.uk`)" # Rate limiting middleware - "traefik.http.middlewares.api-ratelimit.ratelimit.average=100" - "traefik.http.middlewares.api-ratelimit.ratelimit.burst=200" - "traefik.http.routers.api.middlewares=api-ratelimit" ``` Remember: Always test thoroughly in development before deploying to production! --- ## development > claude-code ### Claude Code Configuration This infrastructure includes comprehensive Claude Code enhancements for AI-assisted operations. ## Overview Claude Code is configured with custom skills, agents, commands, and safety hooks specific to this infrastructure. These enhancements enable: - **Context-aware assistance** - Claude understands this specific infrastructure - **Safety guardrails** - Dangerous commands are blocked before execution - **Specialized agents** - Domain-specific experts for different tasks - **Quick commands** - Slash commands for common operations ## Components Summary | Type | Count | Location | |------|-------|----------| | Skills | 12 | `.claude/skills/` | | Custom Agents | 4 | `.claude/agents/` | | Quick Commands | 4 | `.claude/commands/infra/` | | Safety Hooks | 3 | `.claude/hooks/` | ## Skills Skills are model-invoked capabilities that activate automatically based on context. ### Infrastructure-Specific Skills | Skill | Activates When | Purpose | |-------|---------------|---------| | `monitoring-operations` | Discussing Prometheus, Grafana, Loki | PromQL queries, LogQL, dashboards | | `notification-management` | Working with ntfy, Apprise | Push notifications, SMTP gateway | | `security-auditing` | Security reviews, SSL checks | Certificate verification, secret detection | | `git-operations` | Gitea, GitHub sync | Repository management, BFG filtering | | `redis-management` | Redis operations | cache-redis, session-redis management | | `media-services` | Jellyfin, Metube | Media streaming and downloads | ### Core Skills (Pre-existing) | Skill | Purpose | |-------|---------| | `backup-restore` | Backup procedures and disaster recovery | | `database-operations` | PostgreSQL and Supabase database ops | | `docker-stack-manager` | Docker Compose stack management | | `infrastructure-diagnostics` | Troubleshooting and health checks | | `n8n-workflow-builder` | n8n automation workflows | | `service-deployment` | New service deployment with Traefik | ## Custom Agents Specialized agents with infrastructure-specific knowledge. ### infrastructure-specialist **Model**: Sonnet | **Tools**: Read, Grep, Glob, Bash, WebFetch Context-aware expert on this specific infrastructure. Knows: - Service topology and dependencies - Network architecture (traefik_proxy, supabase_default) - Configuration file locations - Critical service paths ### media-manager **Model**: Haiku | **Tools**: Read, Bash, Glob Fast operations for media services: - Jellyfin library management - Metube download troubleshooting - Stremthru and zurg configuration ### security-reviewer **Model**: Sonnet | **Tools**: Read, Grep, Glob (read-only) Security auditing without modification capability: - Configuration review - Secret detection - SSL/TLS verification - Container security analysis ### log-analyzer **Model**: Haiku | **Tools**: Read, Grep, Glob, Bash Fast log analysis and correlation: - Error pattern detection - Cross-service correlation - Incident timeline building ## Quick Commands Slash commands for common infrastructure operations. ``` /infra/quick-status - Fast infrastructure health overview /infra/service-health - Detailed service diagnostics /infra/emergency-stop - Safe shutdown procedures /infra/backup-now - Trigger immediate backup ``` ### Using Commands Commands are invoked with the slash prefix: ``` /infra/quick-status ``` Some commands accept arguments: ``` /infra/service-health grafana /infra/emergency-stop non-critical /infra/backup-now infrastructure ``` ## Safety Hooks Hooks intercept tool execution to enforce safety policies. ### PreToolUse Hooks **dangerous-command-blocker.py** Blocks catastrophic commands before execution: - `rm -rf /` and similar patterns - `docker system prune -a` - `DROP DATABASE`, `TRUNCATE TABLE` - Writing to system directories (`/etc`, `/usr`) **production-service-guard.py** Requires confirmation for critical service operations: - Protected services: supa-db, traefik, prometheus, grafana, alertmanager, loki, authentik, adguard, gitea, cache-redis, session-redis - Protected operations: stop, remove, kill, restart, down ### PostToolUse Hook **audit-logger.py** Logs all tool executions for audit trail: - Location: `.claude/logs/audit.log` - Includes: timestamp, session ID, tool name, input summary, result status ## Configuration Files | File | Purpose | |------|---------| | `.claude/settings.local.json` | Hook registration, permissions, MCP servers | | `.claude/skills/*/SKILL.md` | Individual skill definitions | | `.claude/agents/*.md` | Agent configurations | | `.claude/commands/infra/*.md` | Command definitions | ## Viewing Audit Logs ```bash # View recent audit entries tail -100 /srv/dockerdata/.claude/logs/audit.log # Search for specific tool usage grep "Bash" /srv/dockerdata/.claude/logs/audit.log | tail -20 # View today's activity grep "$(date +%Y-%m-%d)" /srv/dockerdata/.claude/logs/audit.log ``` ## Extending the Configuration ### Adding a New Skill 1. Create directory: `.claude/skills/my-skill/` 2. Create `SKILL.md` with YAML frontmatter: ```yaml --- name: my-skill description: Description of when this skill should activate --- # Skill Name ## Purpose [What this skill enables] ## Operations [Commands and procedures] ``` ### Adding a New Agent Create `.claude/agents/my-agent.md`: ```yaml --- name: my-agent description: What this agent specializes in tools: Read, Grep, Glob model: haiku --- # Agent Name ## Your Responsibilities [What this agent should focus on] ## Key Knowledge [Infrastructure-specific context] ``` ### Adding a New Command Create `.claude/commands/category/my-command.md`: ```yaml --- description: What this command does arguments: - name: arg1 description: Optional argument required: false --- # Command Name [Instructions for Claude to follow when command is invoked] ``` ## Related Documentation - \1 - Detailed hook documentation - \1 - Automation integration - \1 - AI-powered stack management - \1 - Code documentation generator with MCP integration --- ## development > development-guide ### Development Guide This guide covers development workflows, best practices, and procedures for working with the rbnk.uk infrastructure. ## Overview The development documentation provides guidelines for adding new services, modifying existing configurations, and following best practices for infrastructure as code. ## Development Workflow ### 1. \1 Get up and running quickly with: - Environment setup - Basic commands - Common tasks ### 2. \1 Step-by-step guide for deploying new services: - Using StackWiz for quick deployment - Manual service configuration - Traefik integration - DNS setup ### 3. \1 Using the stack creation wizard: - Creating generic service stacks - Deploying Pocketbase applications - Automatic DNS configuration - Template customization ### 4. \1 Version control best practices: - Branching strategy - Commit guidelines - Secret management - Sync procedures ### 5. \1 Setting up local development environment: - Docker Desktop configuration - Local testing procedures - Debugging techniques ### 6. \1 Integrating with infrastructure APIs: - Supabase REST API - LiteLLM endpoints - n8n webhooks - Monitoring APIs ## Development Tools ### StackWiz Quick stack creation tool: ```bash # Create generic service stackwiz myservice # Create Pocketbase app stackwiz --pocketbase myapp # With DNS creation stackwiz --create-dns myservice ``` ### Docker Commands Essential Docker operations: ```bash # Build and start services docker compose up -d --build # View logs docker compose logs -f SERVICE_NAME # Execute commands in container docker exec -it CONTAINER_NAME /bin/sh # Clean up docker system prune -a ``` ### Git Operations Version control commands: ```bash # Standard workflow git add . git commit -m "feat: add new service" git push origin main # Check for secrets git diff --staged | grep -iE 'password|secret|token|key' ``` ## Best Practices ### 1. Service Configuration - Use environment variables for configuration - Store secrets in `.env` files (never commit) - Follow existing naming conventions - Document all configuration options ### 2. Network Architecture - Add services to appropriate Docker networks - Use internal DNS names for service communication - Configure Traefik labels for external access - Document port mappings ### 3. Security - Never hardcode credentials - Use strong, unique passwords - Enable authentication on all services - Regular security updates ### 4. Documentation - Update documentation with changes - Include configuration examples - Document troubleshooting steps - Keep README files current ### 5. Testing - Test locally when possible - Verify Traefik routing - Check DNS resolution - Monitor logs during deployment ## Common Tasks ### Adding a New Service 1. **Create stack**: `stackwiz servicename` 2. **Configure service**: Edit `docker-compose.yml` 3. **Add Traefik labels**: ```yaml labels: - "traefik.enable=true" - "traefik.http.routers.servicename.rule=Host(`servicename.rbnk.uk`)" - "traefik.http.routers.servicename.tls.certresolver=cf" ``` 4. **Create DNS record**: `./_scripts/cloudflare-dns-create.sh servicename` 5. **Start service**: `docker compose up -d` 6. **Verify**: Access at `https://servicename.rbnk.uk` ### Updating Services 1. **Pull latest images**: ```bash docker compose pull docker compose up -d ``` 2. **Force recreate**: ```bash docker compose up -d --force-recreate ``` 3. **With build**: ```bash docker compose up -d --build ``` ### Debugging Issues 1. **Check logs**: `docker compose logs -f` 2. **Inspect container**: `docker inspect CONTAINER_NAME` 3. **Network debugging**: `docker exec CONTAINER ping TARGET` 4. **Traefik routing**: Check Traefik dashboard 5. **DNS issues**: Verify with `dig servicename.rbnk.uk` ## Integration Examples ### Supabase Integration ```javascript const { createClient } = require('@supabase/supabase-js'); const supabase = createClient( 'https://supabase.rbnk.uk', process.env.ANON_KEY ); ``` ### n8n Webhook ```bash curl -X POST https://n8n.rbnk.uk/webhook/WEBHOOK_ID \ -H "Content-Type: application/json" \ -d '{"event": "deployment", "service": "myapp"}' ``` ### Notification Integration ```bash # Send notification via helper script notify "Deployment Complete" "Service deployed successfully" normal system # Direct Apprise API curl -X POST https://apprise.rbnk.uk/notify/apprise \ -F "title=Deployment" \ -F "body=Service deployed" \ -F "tag=development" ``` ## Troubleshooting For common issues and solutions, see: - \1 - \1 - \1 - \1 ## Resources - **Templates**: `/srv/dockerdata/_templates/` - **Scripts**: `/srv/dockerdata/_scripts/` - **Documentation**: `/srv/dockerdata/docs/` - **Examples**: Check existing service configurations --- ## development > git-workflow ### Git Workflow for Dual Remote Setup ## Table of Contents 1. \1 2. \1 3. \1 4. \1 5. \1 6. \1 7. \1 8. \1 ## Overview This guide explains how to work effectively with our dual-remote Git setup, where Gitea serves as the primary repository and GitHub acts as an optional public mirror. ### Architecture Overview ``` ┌─────────────┐ Push/Pull ┌──────────────┐ │ Developer │ ←────────────────→ │ Gitea │ │ Machine │ │ (Primary) │ └─────────────┘ └──────┬───────┘ │ Auto-sync │ (hourly + │ on push) ↓ ┌──────────────┐ │ GitHub │ │ (Mirror) │ └──────────────┘ ``` ### Benefits of Dual Remote Setup 1. **Data Sovereignty**: Primary code stays on self-hosted infrastructure 2. **Public Collaboration**: Optional GitHub mirror for open-source projects 3. **Redundancy**: Multiple locations for code backup 4. **Flexibility**: Choose where to push based on project needs 5. **Security**: Sensitive projects stay private on Gitea ## Initial Setup ### Step 1: SSH Key Configuration First, ensure you have SSH keys set up for both Gitea and GitHub: ```bash # Generate SSH key if needed ssh-keygen -t ed25519 -C "your-email@example.com" # Add to SSH agent eval "$(ssh-agent -s)" ssh-add ~/.ssh/id_ed25519 # Test Gitea connection ssh -T git@gitea.rbnk.uk # Test GitHub connection ssh -T git@github.com ``` ### Step 2: Configure SSH for Multiple Hosts Create or edit `~/.ssh/config`: ``` # Gitea Host gitea.rbnk.uk HostName gitea.rbnk.uk User git IdentityFile ~/.ssh/id_ed25519 Port 22 # GitHub Host github.com HostName github.com User git IdentityFile ~/.ssh/id_ed25519 Port 22 ``` ### Step 3: Clone and Configure Repository #### Option A: Using Setup Script (Recommended) ```bash # Clone from Gitea git clone git@gitea.rbnk.uk:username/project.git cd project # Run setup script curl -sSL https://gitea.rbnk.uk/admin/scripts/raw/branch/main/setup-git-remotes.sh | bash ``` #### Option B: Manual Configuration ```bash # Clone from Gitea git clone git@gitea.rbnk.uk:username/project.git cd project # GitHub remote should already be configured # If not, add it manually: git remote add github git@github.com:username/project.git # Verify remotes (standard naming as of 2025-07-27) git remote -v # origin = Gitea (primary) # github = GitHub (auto-synced mirror) ``` ### Step 4: Configure Git Aliases Add these helpful aliases to your `.gitconfig`: ```bash git config --global alias.push-all '!git push origin && git push github' git config --global alias.sync-github '!git fetch github && git merge github/$(git branch --show-current) && git push origin' git config --global alias.sync-gitea '!git fetch origin && git merge origin/$(git branch --show-current) && git push github' git config --global alias.remotes 'remote -v' git config --global alias.branches 'branch -avv' ``` ## Daily Development Workflow ### Standard Development Flow 1. **Start with a fresh branch**: ```bash # Ensure you're up to date git fetch origin git checkout main git pull origin main # Create feature branch git checkout -b feature/awesome-feature ``` 2. **Make your changes**: ```bash # Edit files vim src/main.py # Stage changes git add src/main.py # Commit with descriptive message git commit -m "feat: add user authentication module - Implement JWT-based auth - Add login/logout endpoints - Include rate limiting Closes #42" ``` 3. **Push to Gitea (primary)**: ```bash git push -u origin feature/awesome-feature ``` 4. **Create Pull Request**: - Navigate to https://gitea.rbnk.uk/username/project - Click "New Pull Request" - Select your feature branch - Add reviewers and description 5. **Automatic GitHub Sync**: - When PR is merged to main: GitHub sync triggers automatically - Push notification appears showing sync status - Manual sync if needed: `sudo systemctl start gitea-sync.service` ### Commit Message Format Follow conventional commits for consistency: ``` ():