Skip to content

Notification Stack Documentation

The notification stack provides a unified, multi-channel notification system for the entire infrastructure, enabling alerts, monitoring, and communication through various channels and protocols.

Overview

The notification stack consists of four integrated components:

  1. Apprise - Universal notification aggregator supporting 100+ services
  2. ntfy - Push notification server with authentication and topics
  3. Mailrise - SMTP to push notification gateway
  4. Uptime Kuma - Service monitoring with integrated notifications

Architecture

graph LR
    subgraph Sources
        A[Services]
        B[Scripts]
        C[n8n Workflows]
        D[Prometheus Alerts]
        E[Legacy Apps]
        F[Monitoring]
    end

    subgraph Notification Stack
        G[Apprise API]
        H[ntfy Server]
        I[Mailrise SMTP]
        J[Uptime Kuma]
    end

    subgraph Destinations
        K[WhatsApp]
        L[Telegram]
        M[Email]
        N[Discord]
        O[Mobile Push]
        P[Desktop Notifications]
    end

    A --> G
    B --> G
    B --> H
    C --> G
    C --> H
    D --> G
    E --> I
    I --> G
    F --> J
    J --> G

    G --> K
    G --> L
    G --> M
    G --> N
    G --> H
    H --> O
    H --> P

Components

Apprise

URL: https://apprise.rbnk.uk
Purpose: Universal notification aggregator that can send to 100+ different notification services

Key Features: - REST API for easy integration - Tag-based notification routing - Persistent configuration storage - Support for multiple notification formats (text, markdown, HTML) - Batch notifications to multiple services

Configuration: - Stores notification URLs in persistent configuration - Supports tagging for selective notifications - Can chain multiple services together

API Usage: 

# Send notification to all configured services
curl -X POST https://apprise.rbnk.uk/notify/apprise \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Alert Title",
    "body": "Alert message body",
    "type": "info"
  }'

# Send to specific tags
curl -X POST https://apprise.rbnk.uk/notify/apprise \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Critical Alert",
    "body": "System failure detected",
    "tag": "critical,oncall"
  }'

ntfy

URL: https://ntfy.rbnk.uk
Purpose: Self-hosted push notification server with authentication

Key Features: - Real-time push notifications to mobile and desktop - Authentication required for publishing and subscribing - Topic-based message routing - Priority levels (1-5) - Attachment support - Scheduled notifications

Authentication: Service accounts are created for different components: - admin - Administrative access - prometheus - Monitoring alerts - backup - Backup job notifications - n8n - Workflow automation - monitoring - General monitoring alerts

Topic Structure: Topics follow the pattern: service-severity-category

Examples: - backup-error-daily - Daily backup errors - monitoring-warning-disk - Disk space warnings - deployment-info-production - Production deployment notifications - security-critical-breach - Security incidents

Publishing Messages: 

# Using curl with authentication
curl -u USERNAME:PASSWORD \
  -d "Backup completed successfully" \
  -H "Title: Backup Success" \
  -H "Priority: 3" \
  -H "Tags: white_check_mark" \
  https://ntfy.rbnk.uk/backup-info-daily

# Using the notify helper script
notify "Backup Complete" "All backups finished successfully" 3 backup-info-daily

Mailrise

Port: 8025 (SMTP)
Purpose: SMTP gateway that converts emails to push notifications

Key Features: - SMTP server interface for legacy applications - Converts emails to Apprise notifications - No authentication required (internal use only) - Supports plain text and HTML emails - Configurable recipient-to-notification mapping

Configuration (/srv/dockerdata/mailrise/mailrise.conf): 

# Example configuration mapping email addresses to notification URLs
configs:
  # Default notifications
  [email protected]:
    urls:
      - "apprise://apprise.rbnk.uk/notify/apprise"

  # Critical alerts with specific routing
  [email protected]:
    urls:
      - "apprise://apprise.rbnk.uk/notify/apprise?tag=critical"
      - "ntfy://ntfy.rbnk.uk/monitoring-critical-system"

Usage Example: 

# Send email that converts to notification
echo "Test notification" | mail -s "Test Subject" [email protected]

Uptime Kuma

URL: https://uptime.rbnk.uk
Purpose: Service monitoring and uptime tracking

Key Features: - Real-time service monitoring - Multiple monitor types (HTTP, TCP, Docker, etc.) - Incident tracking and status pages - Integrated notification support via Apprise - Historical uptime data - Maintenance windows

Monitor Configuration: - HTTP/HTTPS monitors for all public services - Docker container health checks - Database connectivity tests - Certificate expiration monitoring

Notification Integration: Uptime Kuma sends notifications through Apprise for: - Service down/up events - Certificate expiration warnings - Response time degradation - Maintenance reminders

Helper Scripts

notify Command

A convenience script installed at /usr/local/bin/notify for sending notifications:

#!/bin/bash
# Usage: notify "title" "message" [priority] [channel]

TITLE="${1:-Notification}"
MESSAGE="${2:-No message provided}"
PRIORITY="${3:-3}"
CHANNEL="${4:-general}"

# Send to Apprise
curl -s -X POST https://apprise.rbnk.uk/notify/apprise \
  -H "Content-Type: application/json" \
  -d "{
    \"title\": \"$TITLE\",
    \"body\": \"$MESSAGE\",
    \"type\": \"info\"
  }"

# Also send to ntfy if high priority
if [ "$PRIORITY" -ge 4 ]; then
  curl -s -u "$NTFY_USER:$NTFY_PASS" \
    -d "$MESSAGE" \
    -H "Title: $TITLE" \
    -H "Priority: $PRIORITY" \
    https://ntfy.rbnk.uk/$CHANNEL
fi

Usage Examples: 

# Simple notification
notify "Deployment Complete" "Application v1.2.3 deployed successfully"

# High priority alert
notify "Disk Space Warning" "Server disk usage at 79%" 4 monitoring-warning-disk

# Backup notification
notify "Backup Status" "Daily backup completed: 5.2GB transferred" 3 backup-info-daily

Integration Patterns

1. Direct API Integration

Services call the Apprise or ntfy API directly:

import requests

def send_notification(title, message, tags=None):
    """Send notification via Apprise API"""
    data = {
        "title": title,
        "body": message,
        "type": "info"
    }
    if tags:
        data["tag"] = tags

    response = requests.post(
        "https://apprise.rbnk.uk/notify/apprise",
        json=data
    )
    return response.status_code == 200

2. ntfy Topic Publishing

Services publish to specific ntfy topics:

const axios = require('axios');

async function publishToNtfy(topic, title, message, priority = 3) {
    const auth = Buffer.from(`${process.env.NTFY_USER}:${process.env.NTFY_PASS}`).toString('base64');

    await axios.post(
        `https://ntfy.rbnk.uk/${topic}`,
        message,
        {
            headers: {
                'Title': title,
                'Priority': priority.toString(),
                'Authorization': `Basic ${auth}`
            }
        }
    );
}

// Usage
await publishToNtfy('backup-info-daily', 'Backup Complete', 'All systems backed up successfully');

3. SMTP Gateway (Legacy Apps)

Applications that only support email notifications:

# Application configuration
email:
  smtp_host: mailrise
  smtp_port: 8025
  from_address: [email protected]
  to_address: [email protected]  # Configured in Mailrise

4. Webhook Integration via n8n

n8n workflows handle complex notification logic:

// n8n workflow node example
{
  "nodes": [
    {
      "name": "HTTP Request",
      "type": "n8n-nodes-base.httpRequest",
      "parameters": {
        "url": "https://apprise.rbnk.uk/notify/apprise",
        "method": "POST",
        "jsonParameters": true,
        "bodyParametersJson": {
          "title": "={{$json.alert_title}}",
          "body": "={{$json.alert_message}}",
          "tag": "={{$json.severity}}"
        }
      }
    }
  ]
}

5. Prometheus AlertManager

AlertManager routes to Apprise for alert notifications:

# alertmanager.yml
receivers:
  - name: 'apprise'
    webhook_configs:
      - url: 'https://apprise.rbnk.uk/notify/apprise'
        send_resolved: true

Best Practices

1. Topic Naming Convention

Use consistent topic naming for ntfy: - Format: service-severity-category - Severities: info, warning, error, critical - Categories: backup, monitoring, deployment, security, maintenance

2. Priority Levels

Use appropriate priority levels: - 1-2: Low priority, silent notifications - 3: Default, normal notifications - 4: High priority, override quiet hours - 5: Urgent, may trigger loud alerts

3. Notification Grouping

Use tags in Apprise to group related notifications: - critical: On-call alerts - daily: Daily reports - backup: Backup job status - monitoring: System monitoring - deployment: Deployment notifications

4. Rate Limiting

Implement rate limiting to prevent notification spam: 

# Example: Max 10 notifications per minute per service
if [ $(redis-cli GET "notify:$SERVICE:count") -gt 10 ]; then
    echo "Rate limit exceeded for $SERVICE"
    exit 1
fi

5. Testing Notifications

Test notification channels regularly: 

# Test all channels
notify "Test Alert" "Testing notification system at $(date)" 3 test

# Test specific severity
notify "Critical Test" "Testing critical alerts" 5 monitoring-critical-test

# Test Apprise tags
curl -X POST https://apprise.rbnk.uk/notify/apprise \
  -H "Content-Type: application/json" \
  -d '{"title": "Tag Test", "body": "Testing tag routing", "tag": "critical,oncall"}'

Troubleshooting

Common Issues

  1. Authentication Failures: 

    # Check ntfy credentials
curl -u USERNAME:PASSWORD https://ntfy.rbnk.uk/health

  2. Message Not Delivered:

  3. Check Apprise logs: docker compose -f apprise/docker-compose.yml logs -f
  4. Verify ntfy topic exists and user has access

  5. Check network connectivity

  6. SMTP Gateway Issues:

  7. Verify Mailrise is running: docker ps | grep mailrise
  8. Check configuration mapping in mailrise.conf

  9. Test SMTP connection: telnet localhost 8025

  10. Rate Limiting:

  11. Check for rate limit errors in logs
  12. Implement exponential backoff for retries
  13. Consider batching notifications

Debug Commands

# Check service status
docker compose -f apprise/docker-compose.yml ps
docker compose -f ntfy/docker-compose.yml ps

# View recent notifications (ntfy)
curl -u admin:PASSWORD https://ntfy.rbnk.uk/backup-info-daily/json?poll=1

# Test Apprise configuration
docker exec apprise apprise -t "Test" -b "Test message" --config /config/apprise.yml --dry-run

# Monitor notification flow
docker compose -f apprise/docker-compose.yml logs -f &
docker compose -f ntfy/docker-compose.yml logs -f &

Security Considerations

  1. Authentication:
  2. ntfy requires authentication for all operations
  3. Each service has unique credentials

  4. Credentials stored in service-specific .env files

  5. Network Security:

  6. All services behind Traefik with SSL
  7. Mailrise only accessible internally

  8. No public SMTP exposure

  9. Data Privacy:

  10. Notifications may contain sensitive information
  11. Use encryption for sensitive notification channels

  12. Implement audit logging for critical notifications

  13. Access Control:

  14. Limit ntfy topic access by service
  15. Use Apprise tags for routing control
  16. Regular credential rotation

Monitoring the Notification Stack

Grafana dashboards available for: - Notification delivery success rate - Message queue lengths - Service uptime status - Notification volume by channel

Prometheus metrics: - apprise_notifications_sent_total - ntfy_messages_published_total - mailrise_emails_processed_total - uptimekuma_monitors_up

Integration Examples

Docker Service Health Notifications

# docker-compose.yml
services:
  myapp:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    labels:
      - "autoheal=true"  # Auto-restart unhealthy containers
      - "notify.unhealthy=monitoring-error-container"  # ntfy topic for alerts

Backup Job Notifications

#!/bin/bash
# backup-with-notifications.sh

BACKUP_START=$(date +%s)
notify "Backup Starting" "Beginning daily backup job" 2 backup-info-daily

if perform_backup; then
    BACKUP_END=$(date +%s)
    DURATION=$((BACKUP_END - BACKUP_START))
    notify "Backup Success" "Backup completed in ${DURATION}s" 3 backup-info-daily
else
    notify "Backup Failed" "Backup job failed with error: $?" 5 backup-error-daily
    exit 1
fi

Application Error Notifications

# Python application example
import logging
import requests
from functools import wraps

class NotificationHandler(logging.Handler):
    def emit(self, record):
        if record.levelno >= logging.ERROR:
            requests.post(
                "https://apprise.rbnk.uk/notify/apprise",
                json={
                    "title": f"Application Error: {record.name}",
                    "body": record.getMessage(),
                    "tag": "error,application"
                }
            )

# Add to logger
logger = logging.getLogger(__name__)
logger.addHandler(NotificationHandler())

Maintenance

Regular Tasks

  1. Clean Up Old Notifications (Monthly): 

    # ntfy cleanup (if using file storage)
docker exec ntfy ntfy user prune --older-than 30d

  2. Test Notification Channels (Weekly): 

    # Run notification test script
/srv/dockerdata/_scripts/test-notifications.sh

  3. Update Notification URLs (As needed):

  4. Update Apprise configuration through web UI

  5. Restart services after configuration changes

  6. Monitor Resource Usage: 

    # Check disk usage for notification storage
du -sh /srv/dockerdata/*/data/

# Monitor memory usage
docker stats apprise ntfy mailrise uptime-kuma

Real-World Integration Examples

Rclone Backup Notifications

The Rclone backup scripts use notifications to report status:

# From backup-infrastructure.sh
notify_backup_status() {
    local status=$1
    local message=$2
    local priority=3

    if [ "$status" = "error" ]; then
        priority=5
        topic="backup-error-infrastructure"
    else
        topic="backup-info-infrastructure"
    fi

    notify "Infrastructure Backup: $status" "$message" $priority $topic
}

# Usage in script
notify_backup_status "started" "Beginning infrastructure backup to R2"
if rclone sync /srv/dockerdata cloudflare:infrastructure-backups; then
    notify_backup_status "success" "Backup completed: ${size} transferred"
else
    notify_backup_status "error" "Backup failed with exit code $?"
fi

Prometheus Alert Integration

AlertManager sends notifications through Apprise:

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'apprise'
  routes:
    - match:
        severity: critical
      receiver: 'apprise-critical'

receivers:
  - name: 'apprise'
    webhook_configs:
      - url: 'http://apprise:8000/notify/apprise'
        send_resolved: true

  - name: 'apprise-critical'
    webhook_configs:
      - url: 'http://apprise:8000/notify/apprise?tag=critical,oncall'
        send_resolved: true

n8n Workflow Notifications

n8n workflows use multiple notification methods:

// Infrastructure Health Check Workflow
{
  "nodes": [{
    "name": "Check Services",
    "type": "n8n-nodes-base.httpRequest",
    "parameters": {
      "url": "http://prometheus:9090/api/v1/query",
      "qs": {
        "query": "up{job=~'node|docker'}"
      }
    }
  }, {
    "name": "Process Results",
    "type": "n8n-nodes-base.function",
    "parameters": {
      "code": `
        const results = $input.all();
        const downServices = results[0].json.data.result
          .filter(r => r.value[1] === "0")
          .map(r => r.metric.instance);

        if (downServices.length > 0) {
          return [{
            json: {
              status: "error",
              services: downServices,
              priority: 5,
              topic: "monitoring-error-services"
            }
          }];
        }

        return [{
          json: {
            status: "ok",
            priority: 1,
            topic: "monitoring-info-services"
          }
        }];
      `
    }
  }, {
    "name": "Send Notification",
    "type": "n8n-nodes-base.httpRequest",
    "parameters": {
      "url": "https://ntfy.rbnk.uk/{{ $json.topic }}",
      "method": "POST",
      "authentication": "basicAuth",
      "credentials": {
        "name": "ntfy"
      },
      "body": "{{ $json.status === 'error' ? 'Services DOWN: ' + $json.services.join(', ') : 'All services operational' }}",
      "headers": {
        "Title": "Infrastructure Health Check",
        "Priority": "{{ $json.priority }}"
      }
    }
  }]
}

Docker Health Check Integration

Services can trigger notifications on health status changes:

# docker-compose.yml with health notifications
services:
  critical-app:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    labels:
      - "autoheal=true"

  autoheal:
    image: willfarrell/autoheal
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
      - WEBHOOK_URL=http://apprise:8000/notify/apprise
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Git Push Notifications

Gitea webhooks trigger notifications on repository events:

# Git post-receive hook
#!/bin/bash
while read oldrev newrev refname; do
    branch=$(git rev-parse --symbolic --abbrev-ref $refname)
    if [ "$branch" = "main" ] || [ "$branch" = "production" ]; then
        author=$(git log -1 --pretty=format:'%an' $newrev)
        message=$(git log -1 --pretty=format:'%s' $newrev)

        curl -X POST https://apprise.rbnk.uk/notify/apprise \
          -H "Content-Type: application/json" \
          -d "{
            \"title\": \"Git Push: $branch\",
            \"body\": \"$author: $message\",
            \"tag\": \"deployment\"
          }"
    fi
done

Future Enhancements

Planned improvements for the notification stack:

  1. Notification Templates: Standardized message formats
  2. Delivery Confirmation: Track notification acknowledgment
  3. Escalation Policies: Automatic escalation for unacknowledged alerts
  4. Analytics Dashboard: Notification metrics and trends
  5. Mobile App: Native app for ntfy notifications
  6. Webhook Retries: Automatic retry with exponential backoff