Service Integration Patterns¶

This document describes how new services integrate with the existing infrastructure components, showcasing patterns for authentication, monitoring, notifications, and data flow.

Table of Contents¶

Integration Overview
Core Integration Patterns
Service-Specific Integrations
Network Architecture
API Integration Examples
Monitoring & Observability
Configuration Examples

Integration Overview¶

The infrastructure follows a layered integration approach where services connect through:

Docker Networks: Container communication via bridge networks
Traefik Proxy: Centralized routing and SSL termination
Notification Stack: Unified alerting via Apprise/ntfy
Monitoring Stack: Metrics collection through Prometheus
Backup System: Automated data protection via Rclone

┌─────────────────────────────────────────────────────────────────────────┐
│                        Integration Layer Architecture                     │
│                                                                          │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐              │
│  │   Services   │ ←→ │    Traefik   │ ←→ │  Cloudflare  │              │
│  │              │    │   (Proxy)    │    │    (CDN)     │              │
│  └──────┬───────┘    └──────────────┘    └──────────────┘              │
│         │                                                               │
│         ↓                                                               │
│  ┌──────┴───────┐    ┌──────────────┐    ┌──────────────┐              │
│  │ Notification │    │  Monitoring  │    │    Backup    │              │
│  │    Stack     │    │    Stack     │    │    System    │              │
│  │              │    │              │    │              │              │
│  │ • Apprise    │    │ • Prometheus │    │ • Rclone     │              │
│  │ • ntfy       │    │ • Grafana    │    │ • R2 Storage │              │
│  │ • Mailrise   │    │ • Loki       │    │ • Cloud Sync │              │
│  └──────────────┘    └──────────────┘    └──────────────┘              │
└─────────────────────────────────────────────────────────────────────────┘

Core Integration Patterns¶

1. Docker Network Connectivity¶

All services use a dual-network approach for secure isolation:

networks:
  default:
    name: traefik_proxy
    external: true
  # Optional internal network for database access
  internal:
    name: service_internal
    driver: bridge
    internal: true

Pattern Benefits: - External services accessible via Traefik - Database access isolated to internal networks - Clear separation of concerns

2. Traefik Integration Pattern¶

Standard Traefik labels for HTTP services:

labels:
  - "traefik.enable=true"
  - "traefik.docker.network=traefik_proxy"
  # HTTP redirect
  - "traefik.http.routers.service-http.rule=Host(`service.rbnk.uk`)"
  - "traefik.http.routers.service-http.entrypoints=web"
  - "traefik.http.routers.service-http.middlewares=redirect-to-https@file"
  # HTTPS
  - "traefik.http.routers.service.rule=Host(`service.rbnk.uk`)"
  - "traefik.http.routers.service.entrypoints=websecure"
  - "traefik.http.routers.service.tls.certresolver=cf"
  - "traefik.http.services.service.loadbalancer.server.port=8080"

3. Monitoring Integration Pattern¶

Services expose metrics for Prometheus collection:

labels:
  # Enable Prometheus scraping
  - "prometheus.io/scrape=true"
  - "prometheus.io/port=9090"
  - "prometheus.io/path=/metrics"
  # Custom job label for organization
  - "prometheus_job=service-metrics"

4. Redis Integration Pattern¶

Services integrate with consolidated Redis instances for caching and session management:

A. Cache Redis Integration (Ephemeral Data)¶

# Service connecting to cache Redis
environment:
  - REDIS_CACHE_URL=redis://:${REDIS_PASSWORD}@cache-redis:6379/0
  - CACHE_BACKEND=redis
networks:
  - cache_redis_network  # Connect to cache Redis network

Cache Redis Configuration: - Purpose: Temporary data, API responses, computed results - Persistence: Disabled (no disk writes) - Memory Policy: allkeys-lru (evict least recently used) - Max Memory: 4GB (configurable) - Port: 6380 (host) → 6379 (container)

B. Session Redis Integration (Persistent Data)¶

# Service connecting to session Redis
environment:
  - REDIS_SESSION_URL=redis://:${REDIS_PASSWORD}@session-redis:6379/0
  - SESSION_STORE=redis
networks:
  - session_redis_network  # Connect to session Redis network

Session Redis Configuration: - Purpose: User sessions, authentication tokens, persistent state - Persistence: Enabled (AOF + RDB snapshots) - Memory Policy: volatile-lru (evict expired keys first) - Max Memory: 1GB (configurable) - Port: 6381 (host) → 6379 (container)

C. Multi-Redis Service Pattern¶

# Service using both Redis instances
environment:
  - REDIS_CACHE_URL=redis://:${REDIS_PASSWORD}@cache-redis:6379/0
  - REDIS_SESSION_URL=redis://:${REDIS_PASSWORD}@session-redis:6379/0
networks:
  - cache_redis_network
  - session_redis_network

Use Cases: - Gitea: Cache for Git operations, sessions for web UI - Web Applications: API response caching + user session storage - Background Jobs: Task queues (cache) + job status (session)

5. Notification Integration Pattern¶

Services integrate with notification stack via multiple methods:

A. Direct API Integration¶

# Send notification via Apprise
curl -X POST https://apprise.rbnk.uk/notify/apprise \
  -H "Content-Type: application/json" \
  -d '{"title": "Service Alert", "body": "Status update"}'

B. ntfy Topic Publishing¶

# Publish to ntfy topic with authentication
curl -X POST https://ntfy.rbnk.uk/service-warning-disk \
  -H "Authorization: Bearer $NTFY_TOKEN" \
  -d "Disk usage at 85%"

C. SMTP Gateway via Mailrise¶

# Send email that converts to push notification
echo "Service notification" | mail -s "Alert" service@localhost:8025

Service-Specific Integrations¶

Gitea + Redis Integration¶

Purpose: Performance optimization through caching and session management

# Gitea with dual Redis connections
services:
  gitea:
    environment:
      # Cache Redis for Git operations
      - GITEA__cache__ENABLED=true
      - GITEA__cache__ADAPTER=redis
      - GITEA__cache__HOST=redis://:${REDIS_PASSWORD}@cache-redis:6379/0

      # Session Redis for web sessions
      - GITEA__session__PROVIDER=redis
      - GITEA__session__PROVIDER_CONFIG=redis://:${REDIS_PASSWORD}@session-redis:6379/1

    networks:
      - traefik_proxy
      - cache_redis_network    # For cache operations
      - session_redis_network  # For session storage

Performance Benefits:

# Cache Redis handles:
# - Git object caching
# - Repository metadata
# - API response caching

# Session Redis handles:
# - User login sessions
# - CSRF tokens
# - Web UI state

Monitoring Integration:

# Redis metrics available via Prometheus
redis_connected_clients{instance="cache-redis"}
redis_memory_used_bytes{instance="session-redis"}
redis_keyspace_hits_total{instance="cache-redis"}

Metube + Rclone Integration¶

Purpose: Automated backup of downloaded media files

# Metube volumes shared with Rclone
volumes:
  - ./downloads:/downloads
  - ./config:/config

# Rclone access to Metube downloads
volumes:
  - /srv/dockerdata/metube/downloads:/backup/metube:ro

Backup Automation:

#!/bin/bash
# Automated sync of Metube downloads to cloud storage
docker exec rclone rclone sync /backup/metube gdrive:media/metube \
  --include "*.mp4" \
  --include "*.mkv" \
  --include "*.avi" \
  --transfers 2 \
  --bwlimit 10M

Watchtower + Notification Integration¶

Configuration: Watchtower sends update notifications via notification stack

environment:
  # Email notifications via Mailrise SMTP gateway
  - WATCHTOWER_NOTIFICATIONS=email
  - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=mailrise:8025
  - [email protected]
  - WATCHTOWER_NOTIFICATION_EMAIL_TO=admin@localhost

Enhanced Notification Flow:

Watchtower Update → SMTP (Mailrise:8025) → Apprise API → ntfy Topic → Push Notification

Paperless-AI + n8n Workflow Integration¶

Document Processing Automation:

Webhook Trigger: Paperless-AI sends webhook on document upload
n8n Processing: Workflow extracts metadata and processes content
Notification: Results sent via notification stack

# Paperless-AI webhook configuration
environment:
  - WEBHOOK_URL=https://n8n.rbnk.uk/webhook/paperless-process
  - POST_PROCESSING_ENABLED=true

n8n Workflow Pattern:

href="#__codelineno-18-1">{ "nodes": [ { "name": "Webhook", "type": "n8n-nodes-base.webhook", "parameters": { "path": "paperless-process" } }, { "name": "Process Document", "type": "n8n-nodes-base.function", "parameters": { "functionCode": "// Extract document metadata\nreturn items.map(item => ({\n json: {\n documentId: item.json.id,\n title: item.json.title,\n processed: true\n }\n}));" } }, { "name": "Send Notification", "type": "n8n-nodes-base.httpRequest", "parameters": { "url": "https://apprise.rbnk.uk/notify/apprise", "method": "POST", "jsonParameters": true, "options": { "bodyParametersUi": { "parameter": [ { "name": "title", "value": "Document Processed" }, { "name": "body", "value": "Document {{$node.Process Document.json.title}} has been processed" } ] } } } } ] }

Network Architecture¶

Docker Network Topology¶

┌─────────────────────────────────────────────────────────────────────────┐
│                        Docker Host (10.10.10.10)                         │
│                                                                          │
│  ┌──────────────────────────────────────────────────────────────────┐   │
│  │                    traefik_proxy (172.18.0.0/16)                  │   │
│  │                                                                    │   │
│  │  ┌─────────┐  ┌──────────┐  ┌─────────┐  ┌──────────────────┐  │   │
│  │  │ Traefik │  │  Metube  │  │ Rclone  │  │   Paperless-AI   │  │   │
│  │  │ .18.2   │  │  .18.10  │  │ .18.11  │  │      .18.12      │  │   │
│  │  └─────────┘  └──────────┘  └─────────┘  └──────────────────┘  │   │
│  │                                                                    │   │
│  │  ┌──────────┐  ┌──────────┐  ┌─────────┐  ┌──────────────────┐  │   │
│  │  │Watchtower│  │   n8n    │  │Apprise  │  │  Monitoring      │  │   │
│  │  │  .18.13  │  │  .18.4   │  │ .18.14  │  │   Stack          │  │   │
│  │  └──────────┘  └──────────┘  └─────────┘  └──────────────────┘  │   │
│  └──────────────────────────────────────────────────────────────────┘   │
│                                                                          │
│  ┌──────────────────────────────────────────────────────────────────┐   │
│  │                   supabase_internal (172.19.0.0/16)              │   │
│  │                          (No Internet Access)                     │   │
│  │                                                                    │   │
│  │        ┌─────────────┐    ┌──────────┐    ┌──────────┐          │   │
│  │        │ PostgreSQL  │    │   Auth   │    │   REST   │          │   │
│  │        │   .19.2     │    │  .19.3   │    │  .19.4   │          │   │
│  │        └─────────────┘    └──────────┘    └──────────┘          │   │
│  └──────────────────────────────────────────────────────────────────┘   │
│                                                                          │
│  Host Services:                                                          │
│  ├── Docker Daemon (unix:///var/run/docker.sock)                       │
│  ├── Ollama (127.0.0.1:11434)                                          │
│  └── Node Exporter (127.0.0.1:9100)                                    │
└─────────────────────────────────────────────────────────────────────────┘

Service Discovery Patterns¶

1. Container-to-Container Communication¶

# Service A connecting to Service B
environment:
  - SERVICE_B_URL=http://service-b:8080
  # Uses Docker's built-in DNS resolution

2. Container-to-Host Service¶

# Container connecting to host service (Ollama)
environment:
  - OLLAMA_URL=http://172.17.0.1:11434
  # 172.17.0.1 is Docker's default gateway to host

3. Container-to-Database¶

# Connection via internal network
environment:
  - DATABASE_URL=postgresql://user:pass@supa-db:5432/dbname
networks:
  - supabase_internal  # Required for database access

API Integration Examples¶

1. Rclone Remote Control API¶

Purpose: Enable other services to trigger backup operations

# Start backup via Rclone RC API
curl -X POST http://rclone:5572/sync/sync \
  -H "Content-Type: application/json" \
  -d '{
    "srcFs": "/data/metube",
    "dstFs": "gdrive:backups/metube",
    "_async": true
  }'

Integration in n8n:

{
  "name": "Trigger Backup",
  "type": "n8n-nodes-base.httpRequest",
  "parameters": {
    "url": "http://rclone:5572/sync/sync",
    "method": "POST",
    "jsonParameters": true,
    "options": {
      "bodyParametersUi": {
        "parameter": [
          {
            "name": "srcFs",
            "value": "/data/{{$node.Previous.json.folder}}"
          },
          {
            "name": "dstFs", 
            "value": "cloudflare:backups/{{$node.Previous.json.folder}}"
          },
          {
            "name": "_async",
            "value": true
          }
        ]
      }
    }
  }
}

2. Supabase API Integration¶

Authentication Flow:

// Service authentication with Supabase
const response = await fetch('https://supabase.rbnk.uk/auth/v1/token', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'apikey': process.env.SUPABASE_ANON_KEY
  },
  body: JSON.stringify({
    email: '[email protected]',
    password: process.env.SERVICE_PASSWORD
  })
});

Data Operations:

// Insert service data
const data = await fetch('https://supabase.rbnk.uk/rest/v1/service_logs', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'apikey': process.env.SUPABASE_ANON_KEY,
    'Authorization': `Bearer ${accessToken}`,
    'Prefer': 'return=representation'
  },
  body: JSON.stringify({
    service: 'metube',
    action: 'download_complete',
    metadata: { filename: 'video.mp4', size: 1024000 }
  })
});

3. Notification API Integration¶

Unified Notification Function:

#!/bin/bash
# notification.sh - Unified notification script
notify() {
  local title="$1"
  local message="$2"
  local priority="${3:-normal}"
  local service="${4:-general}"

  # Send to Apprise
  curl -s -X POST https://apprise.rbnk.uk/notify/apprise \
    -H "Content-Type: application/json" \
    -d "{\"title\": \"$title\", \"body\": \"$message\"}"

  # Send to ntfy topic
  local topic="${service}-${priority}-alert"
  curl -s -X POST "https://ntfy.rbnk.uk/$topic" \
    -H "Authorization: Bearer $NTFY_TOKEN" \
    -d "$message"
}

# Usage examples
notify "Backup Complete" "Metube backup finished successfully" "info" "backup"
notify "Disk Warning" "Disk usage at 85%" "warning" "monitoring"
notify "Service Error" "Paperless-AI processing failed" "error" "paperless"

Monitoring & Observability¶

Metrics Collection Flow¶

┌─────────────┐    ┌──────────────┐    ┌─────────────┐    ┌──────────────┐
│   Service   │───▶│    Metrics   │───▶│ Prometheus  │───▶│   Grafana    │
│ (Container) │    │  Endpoint    │    │  (Scraper)  │    │ (Dashboard)  │
└─────────────┘    └──────────────┘    └─────────────┘    └──────────────┘
       │                                       │
       ▼                                       ▼
┌─────────────┐                        ┌─────────────┐
│ Log Output  │                        │ AlertManager│
│   (Docker)  │                        │  (Alerts)   │
└─────────────┘                        └─────────────┘
       │                                       │
       ▼                                       ▼
┌─────────────┐                        ┌─────────────┐
│    Loki     │                        │Notification │
│ (Log Store) │                        │   Stack     │
└─────────────┘                        └─────────────┘

Service Metrics Examples¶

Metube Metrics¶

# Custom metrics via textfile collector
- name: metube_downloads_total
  type: counter
  help: Total number of downloads

- name: metube_download_size_bytes
  type: histogram
  help: Size of downloaded files

- name: metube_queue_length
  type: gauge
  help: Current download queue length

Rclone Backup Metrics¶

# Generated by backup scripts
- name: rclone_backup_duration_seconds
  type: histogram
  help: Time taken for backup operations

- name: rclone_backup_bytes_transferred
  type: counter
  help: Total bytes transferred

- name: rclone_backup_success
  type: gauge
  help: Last backup success status (1=success, 0=failure)

Log Aggregation Patterns¶

Structured Logging Configuration:

# Docker logging driver
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
    labels: "service,environment"

Loki Labels:

# Promtail config for service discovery
- job_name: containers
  docker_sd_configs:
    - host: unix:///var/run/docker.sock
  relabel_configs:
    - source_labels: ['__meta_docker_container_name']
      target_label: 'container'
    - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
      target_label: 'service'

Configuration Examples¶

Complete Service Integration Template¶

# docker-compose.yml for new service
services:
  my-service:
    image: myservice:latest
    container_name: my-service
    restart: unless-stopped

    # Environment configuration
    environment:
      - SERVICE_PORT=8080
      - DATABASE_URL=postgresql://user:pass@supa-db:5432/mydb
      - NOTIFICATION_URL=https://apprise.rbnk.uk/notify/apprise
      - BACKUP_ENABLED=true
      # Redis integration (optional)
      - REDIS_CACHE_URL=redis://:${REDIS_PASSWORD}@cache-redis:6379/0
      - REDIS_SESSION_URL=redis://:${REDIS_PASSWORD}@session-redis:6379/0

    # Network connectivity
    networks:
      - traefik_proxy      # External access
      - supabase_internal  # Database access
      - cache_redis_network    # Cache Redis (optional)
      - session_redis_network  # Session Redis (optional)

    # Volume mounts
    volumes:
      - ./data:/app/data
      - ./config:/app/config

    # Traefik integration
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik_proxy"
      - "traefik.http.routers.myservice-http.rule=Host(`myservice.rbnk.uk`)"
      - "traefik.http.routers.myservice-http.entrypoints=web"
      - "traefik.http.routers.myservice-http.middlewares=redirect-to-https@file"
      - "traefik.http.routers.myservice.rule=Host(`myservice.rbnk.uk`)"
      - "traefik.http.routers.myservice.entrypoints=websecure"
      - "traefik.http.routers.myservice.tls.certresolver=cf"
      - "traefik.http.services.myservice.loadbalancer.server.port=8080"

      # Monitoring integration
      - "prometheus.io/scrape=true"
      - "prometheus.io/port=9090"
      - "prometheus.io/path=/metrics"
      - "prometheus_job=myservice-metrics"

      # Backup exclusion (if needed)
      - "backup.exclude=true"

networks:
  traefik_proxy:
    external: true
  supabase_internal:
    external: true
  cache_redis_network:
    external: true
  session_redis_network:
    external: true

Backup Integration Script¶

#!/bin/bash
# backup-myservice.sh
set -euo pipefail

SERVICE_NAME="my-service"
BACKUP_DATE=$(date +%Y%m%d-%H%M%S)
REMOTE="cloudflare"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}

# Backup service data
log "Backing up $SERVICE_NAME data..."
docker exec rclone rclone sync "/backup/dockerdata/$SERVICE_NAME/data" \
    "$REMOTE:service-backups/$SERVICE_NAME/$BACKUP_DATE/" \
    --exclude "*/tmp/**" \
    --exclude "*/cache/**" \
    --transfers 4 \
    --checkers 8

# Send success notification
curl -s -X POST https://apprise.rbnk.uk/notify/apprise \
    -H "Content-Type: application/json" \
    -d "{\"title\": \"$SERVICE_NAME Backup Complete\", \"body\": \"Backup completed successfully at $BACKUP_DATE\"}"

log "$SERVICE_NAME backup completed"

Health Check Integration¶

# Health check configuration
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s

# Custom health check script
# healthcheck.sh
#!/bin/bash
# Check service health and send alerts
if ! curl -sf http://localhost:8080/health > /dev/null; then
    curl -X POST https://ntfy.rbnk.uk/myservice-error-health \
        -H "Authorization: Bearer $NTFY_TOKEN" \
        -d "Service health check failed"
    exit 1
fi

Alerting Rules Integration¶

# prometheus-alerts.yml
groups:
  - name: myservice
    rules:
      - alert: MyServiceDown
        expr: up{job="myservice-metrics"} == 0
        for: 2m
        labels:
          severity: critical
          service: myservice
        annotations:
          summary: "MyService is down"
          description: "MyService has been down for more than 2 minutes"

      - alert: MyServiceHighErrorRate
        expr: rate(myservice_errors_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
          service: myservice
        annotations:
          summary: "High error rate in MyService"
          description: "Error rate is {{ $value }} errors per second"

This comprehensive integration guide ensures new services follow established patterns while maintaining security, observability, and reliability standards across the infrastructure.