New Services Troubleshooting Guide

This guide provides comprehensive troubleshooting procedures for Metube, Watchtower, and Paperless-AI services in the rbnk.uk infrastructure.

Quick Reference

Service	URL	Main Function	Common Issues
Metube	https://metube.rbnk.uk	YouTube download	Download failures, format issues, storage
Watchtower	N/A (background)	Container updates	Update failures, notification issues, exclusions
Paperless-AI	https://paperless-ai.rbnk.uk	Document AI chat	OCR failures, RAG connection, processing stuck

---

Metube Troubleshooting

Common Issues

1. Download Failures

Symptoms: - Downloads fail to start - Error messages in UI - Files not appearing in downloads folder

Debug Steps:

# Check Metube logs
docker logs metube --tail 100

# Check disk space in downloads directory
df -h /srv/dockerdata/metube/downloads

# Verify permissions
ls -la /srv/dockerdata/metube/downloads

Common Solutions:

# 1. Update yt-dlp (fixes most issues)
docker exec metube yt-dlp --update

# 2. Clear download queue if stuck
docker exec metube rm -rf /downloads/.tmp/*

# 3. Fix permissions
sudo chown -R 1000:1000 /srv/dockerdata/metube/downloads
sudo chown -R 1000:1000 /srv/dockerdata/metube/config

Notification Setup:

# Add to notification monitoring
source /srv/dockerdata/ntfy/.env
curl -H "Authorization: Bearer $MONITORING_TOKEN" \
     -H "Title: Metube Download Failed" \
     -H "Priority: 3" \
     -d "Download failed for URL: $URL. Check logs." \
     http://ntfy:80/metube-errors

2. yt-dlp Version Issues

Symptoms: - "This video is unavailable" for working videos - Format extraction errors - Age-restricted content failures

Solutions:

# Manual yt-dlp update
docker exec metube yt-dlp --update

# Check current version
docker exec metube yt-dlp --version

# Test specific URL
docker exec metube yt-dlp --dry-run "https://youtube.com/watch?v=VIDEO_ID"

# Force format selection if needed
docker exec metube yt-dlp -f 'best[height<=720]' "VIDEO_URL"

3. Storage and Format Issues

Symptoms: - Downloads fail with disk space errors - Specific formats not available - Large files causing issues

Debug Commands:

# Check available space
df -h /srv/dockerdata/metube/downloads

# Monitor disk usage during download
watch "df -h /srv/dockerdata/metube/downloads"

# Check current downloads
docker exec metube ls -la /downloads/.tmp/

# View download queue
docker exec metube cat /config/queue.json 2>/dev/null || echo "No queue file"

Solutions:

# Clean old downloads (older than 30 days)
find /srv/dockerdata/metube/downloads -name "*.mp4" -mtime +30 -delete
find /srv/dockerdata/metube/downloads -name "*.webm" -mtime +30 -delete

# Set download limits in environment
# Edit docker-compose.yml to add:
# - MAX_FILE_SIZE=2G
# - PREFERRED_QUALITY=720p

# Restart with new settings
cd /srv/dockerdata/metube
docker compose down && docker compose up -d

4. Network and Proxy Issues

Symptoms: - Timeouts on certain videos - Regional blocking messages - Slow download speeds

Solutions:

# Test network connectivity from container
docker exec metube ping google.com
docker exec metube curl -I https://youtube.com

# Check if using IPv6 causes issues
docker exec metube yt-dlp --force-ipv4 "VIDEO_URL"

# Use proxy if needed (add to docker-compose.yml)
# environment:
#   - HTTP_PROXY=http://proxy:port
#   - HTTPS_PROXY=http://proxy:port

---

Watchtower Troubleshooting

Common Issues

1. Update Failures

Symptoms: - Services not updating despite new images available - Containers restarting unexpectedly - Email notifications about failed updates

Debug Steps:

# Check Watchtower logs
docker logs watchtower --tail 100

# See what containers Watchtower is monitoring
docker logs watchtower | grep "Found new"

# Check excluded containers
docker ps --format "table {{.Names}}\t{{.Labels}}" | grep watchtower.enable

Solutions:

# 1. Restart Watchtower
cd /srv/dockerdata/watchtower
docker compose restart

# 2. Force update check (run manually)
docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  containrrr/watchtower:latest \
  --run-once --debug

# 3. Check specific container exclusion
docker inspect CONTAINER_NAME | grep -A 5 Labels | grep watchtower

2. Notification Issues

Symptoms: - Not receiving update emails - SMTP errors in logs - Failed notifications

Debug Steps:

# Check email configuration
docker logs watchtower | grep -i "email\|smtp\|notification"

# Test SMTP connectivity
docker exec watchtower nc -zv smtp.gmail.com 587

Enhanced Notification Setup:

# Add to watchtower docker-compose.yml environment section:
# - WATCHTOWER_NOTIFICATIONS=email,ntfy
# - WATCHTOWER_NOTIFICATION_NTFY_URL=http://ntfy:80/watchtower
# - WATCHTOWER_NOTIFICATION_NTFY_TOKEN=${WATCHTOWER_TOKEN}

# Create dedicated watchtower token
source /srv/dockerdata/ntfy/.env
echo "WATCHTOWER_TOKEN=$MONITORING_TOKEN" >> /srv/dockerdata/watchtower/.env

Setup script for enhanced notifications:

#!/bin/bash
# /srv/dockerdata/watchtower/setup-notifications.sh

source /srv/dockerdata/ntfy/.env

# Create notification function
notify_watchtower() {
    local status="$1"
    local message="$2"
    local priority="${3:-3}"

    # Send to ntfy
    curl -H "Authorization: Bearer $MONITORING_TOKEN" \
         -H "Priority: $priority" \
         -H "Title: Watchtower: $status" \
         -H "Tags: watchtower,updates" \
         -d "$message" \
         http://ntfy:80/watchtower

    # Log to system journal
    logger "WATCHTOWER: $status - $message"
}

# Example usage in scripts
# notify_watchtower "Update Failed" "Container myapp failed to update" 4

3. Excluded Containers Not Being Honored

Symptoms: - Containers with exclusion labels still getting updated - Critical services restarting unexpectedly

Debug Commands:

# List all containers and their watchtower labels
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Labels}}" | grep -E "watchtower|Names"

# Check specific container labels
docker inspect CONTAINER_NAME | jq '.[0].Config.Labels'

# Verify exclusion is working
docker logs watchtower | grep "Skipping"

Fix Exclusions:

# Ensure labels are properly set (example for database containers)
# Add to docker-compose.yml labels section:
# - "com.centurylinklabs.watchtower.enable=false"

# Restart containers to apply labels
cd /srv/dockerdata/SERVICE_NAME
docker compose down && docker compose up -d

# Verify exclusion applied
docker inspect CONTAINER_NAME | grep -A 3 "watchtower.enable"

4. Resource Usage Issues

Symptoms: - High CPU usage during updates - Memory exhaustion - Slow update process

Monitor and Fix:

# Monitor Watchtower resource usage
docker stats watchtower

# Add resource limits to docker-compose.yml
# deploy:
#   resources:
#     limits:
#       memory: 512M
#       cpus: '0.5'

# Schedule updates during low-usage hours
# environment:
#   - WATCHTOWER_SCHEDULE=0 2 * * * # 2 AM daily

---

Paperless-AI Troubleshooting

Common Issues

1. OCR Processing Failures

Symptoms: - Documents uploaded but not processed - OCR status shows "failed" - Text extraction incomplete

Debug Steps:

# Check Paperless-AI logs
docker logs paperless-ai --tail 100

# Check if RAG service is running
docker ps | grep rag

# Test document processing
curl -X POST https://paperless-ai.rbnk.uk/api/documents/test \
  -H "Content-Type: application/json" \
  -d '{"test": true}'

Solutions:

# 1. Restart OCR processing
docker exec paperless-ai supervisorctl restart ocr

# 2. Clear stuck processing queue
docker exec paperless-ai rm -rf /app/data/processing/*

# 3. Check OCR engine status
docker exec paperless-ai tesseract --version
docker exec paperless-ai tesseract --list-langs

# 4. Fix permissions
sudo chown -R 1000:1000 /srv/dockerdata/paperless-ai/data

2. RAG Service Connection Issues

Symptoms: - Chat functionality not working - "RAG service unavailable" errors - Connection timeouts

Debug Steps:

# Check if RAG service URL is accessible
docker exec paperless-ai curl -f http://localhost:8000/health || echo "RAG service down"

# Check network connectivity
docker exec paperless-ai nc -zv localhost 8000

# View RAG-specific logs
docker logs paperless-ai | grep -i "rag\|8000"

Solutions:

# 1. Start RAG service if not running
cd /srv/dockerdata/paperless-ai
# Check if RAG service is defined in docker-compose.yml

# 2. Fix RAG service URL configuration
# Edit environment in docker-compose.yml:
# - RAG_SERVICE_URL=http://paperless-ai-rag:8000  # If running as separate service
# or
# - RAG_SERVICE_URL=http://localhost:8000  # If running in same container

# 3. Restart services
docker compose down && docker compose up -d

# 4. Test RAG connection
docker exec paperless-ai curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "test", "documents": []}'

3. Document Processing Stuck

Symptoms: - Upload completes but processing never finishes - Queue shows pending documents indefinitely - High CPU usage without progress

Debug Commands:

# Check processing queue
docker exec paperless-ai ls -la /app/data/processing/

# Monitor CPU and memory usage
docker stats paperless-ai

# Check document processing logs
docker logs paperless-ai | grep -i "processing\|document\|queue"

# View current processing status
curl -s https://paperless-ai.rbnk.uk/api/status | jq '.'

Solutions:

# 1. Clear processing queue
docker exec paperless-ai rm -rf /app/data/processing/*
docker exec paperless-ai rm -rf /app/data/tmp/*

# 2. Restart processing services
docker exec paperless-ai supervisorctl restart all

# 3. Check for corrupted documents
docker exec paperless-ai find /app/data -name "*.pdf" -exec file {} \; | grep -v "PDF document"

# 4. Restart with clean state
cd /srv/dockerdata/paperless-ai
docker compose down
docker volume rm paperless-ai_data  # WARNING: This removes all data
docker compose up -d

4. Authentication and API Issues

Symptoms: - Cannot access web interface - API calls returning 401/403 errors - Login page not loading

Debug Steps:

# Check if service is accessible
curl -I https://paperless-ai.rbnk.uk

# Check Traefik routing
docker logs traefik | grep paperless-ai

# Test direct container access
curl -I http://localhost:3003  # or whatever port is mapped

Solutions:

# 1. Check Traefik labels in docker-compose.yml
docker inspect paperless-ai | grep -A 10 Labels

# 2. Verify port configuration
docker port paperless-ai

# 3. Reset authentication if needed
docker exec paperless-ai rm -f /app/data/auth.db
docker compose restart

# 4. Check environment variables
docker exec paperless-ai env | grep -E "PORT|AUTH|API"

---

Integration with Notification Stack

Setup Monitoring Alerts

Create monitoring script for all three services:

#!/bin/bash
# /srv/dockerdata/_scripts/monitor-new-services.sh

source /srv/dockerdata/ntfy/.env

check_metube() {
    if ! curl -sf https://metube.rbnk.uk >/dev/null; then
        curl -H "Authorization: Bearer $MONITORING_TOKEN" \
             -H "Priority: 4" \
             -H "Title: Metube Service Down" \
             -d "Metube is not responding at https://metube.rbnk.uk" \
             http://ntfy:80/service-alerts
    fi
}

check_paperless_ai() {
    if ! curl -sf https://paperless-ai.rbnk.uk >/dev/null; then
        curl -H "Authorization: Bearer $MONITORING_TOKEN" \
             -H "Priority: 4" \
             -H "Title: Paperless-AI Service Down" \
             -d "Paperless-AI is not responding at https://paperless-ai.rbnk.uk" \
             http://ntfy:80/service-alerts
    fi

    # Check RAG service specifically
    if ! docker exec paperless-ai curl -sf http://localhost:8000/health >/dev/null 2>&1; then
        curl -H "Authorization: Bearer $MONITORING_TOKEN" \
             -H "Priority: 3" \
             -H "Title: Paperless-AI RAG Service Issue" \
             -d "RAG service not responding on port 8000" \
             http://ntfy:80/service-alerts
    fi
}

check_watchtower() {
    # Check if Watchtower is running
    if ! docker ps | grep -q watchtower; then
        curl -H "Authorization: Bearer $MONITORING_TOKEN" \
             -H "Priority: 4" \
             -H "Title: Watchtower Container Down" \
             -d "Watchtower container is not running" \
             http://ntfy:80/service-alerts
    fi

    # Check for recent update failures (last 24 hours)
    if docker logs watchtower --since 24h | grep -i "failed\|error" >/dev/null; then
        curl -H "Authorization: Bearer $MONITORING_TOKEN" \
             -H "Priority: 3" \
             -H "Title: Watchtower Update Failures" \
             -d "Watchtower reported update failures in the last 24 hours" \
             http://ntfy:80/service-alerts
    fi
}

# Run checks
check_metube
check_paperless_ai
check_watchtower

Setup Cron for Regular Monitoring

# Add to crontab: crontab -e
# Check services every 5 minutes
*/5 * * * * /srv/dockerdata/_scripts/monitor-new-services.sh >/dev/null 2>&1

# Daily health report
0 8 * * * /srv/dockerdata/_scripts/daily-health-report.sh

ntfy Topic Configuration

Subscribe to these topics in the ntfy mobile app:

• service-alerts - Service availability issues
• watchtower - Container update notifications
• metube-errors - Download failures and issues
• paperless-ai-status - Document processing status

Uptime Kuma Integration

Add monitors in Uptime Kuma for:

1. Metube HTTP Monitor:
2. URL: https://metube.rbnk.uk
3. Method: GET
4. Interval: 60 seconds

5. Keyword: "metube" (to verify page loads correctly)

6. Paperless-AI HTTP Monitor:

7. URL: https://paperless-ai.rbnk.uk
8. Method: GET

9. Interval: 60 seconds

10. Watchtower Docker Monitor:

11. Type: Docker Container
12. Container: watchtower
13. Docker Host: unix:///var/run/docker.sock

Grafana Dashboard Queries

Add these Prometheus queries to monitor the services:

# Container status
up{job="cadvisor", name=~"metube|paperless-ai|watchtower"}

# Memory usage
container_memory_usage_bytes{name=~"metube|paperless-ai|watchtower"}

# CPU usage
rate(container_cpu_usage_seconds_total{name=~"metube|paperless-ai|watchtower"}[5m])

# Disk usage for downloads
node_filesystem_free_bytes{mountpoint="/srv/dockerdata/metube/downloads"}

---

Emergency Recovery Procedures

Service Recovery Priority

1. Watchtower (Low priority - can be down temporarily)
2. Metube (Medium priority - user-facing service)
3. Paperless-AI (High priority - data processing service)

Quick Recovery Commands

# Emergency restart all three services
docker restart watchtower metube paperless-ai

# Full reset with data preservation
cd /srv/dockerdata/metube && docker compose down && docker compose up -d
cd /srv/dockerdata/watchtower && docker compose down && docker compose up -d  
cd /srv/dockerdata/paperless-ai && docker compose down && docker compose up -d

# Check all services status
docker ps | grep -E "metube|watchtower|paperless-ai"

Backup Important Data

# Backup Metube downloads
tar -czf /srv/dockerdata/_backup/metube-downloads-$(date +%Y%m%d).tar.gz \
  /srv/dockerdata/metube/downloads

# Backup Paperless-AI data
docker run --rm -v paperless-ai_data:/data -v /srv/dockerdata/_backup:/backup \
  alpine tar -czf /backup/paperless-ai-data-$(date +%Y%m%d).tar.gz -C /data .

# Backup Watchtower config
cp /srv/dockerdata/watchtower/docker-compose.yml \
  /srv/dockerdata/_backup/watchtower-config-$(date +%Y%m%d).yml

---

Prevention and Maintenance

Regular Maintenance Tasks

Weekly:

# Clean old Metube downloads
find /srv/dockerdata/metube/downloads -name "*.mp4" -mtime +7 -delete

# Check Watchtower exclusions are still applied
/srv/dockerdata/watchtower/apply-exclusions.sh --verify

# Restart Paperless-AI to clear memory leaks
cd /srv/dockerdata/paperless-ai && docker compose restart

Monthly:

# Update yt-dlp in Metube
docker exec metube yt-dlp --update

# Review Watchtower logs for patterns
docker logs watchtower --since 720h | grep -i "error\|failed" > /tmp/watchtower-issues.log

# Optimize Paperless-AI database
docker exec paperless-ai python manage.py optimize_db

Resource Monitoring

Set up alerts for: - Disk space in /srv/dockerdata/metube/downloads < 5GB - Paperless-AI memory usage > 1GB - Watchtower taking > 30 minutes for updates

Log Rotation

# Add to /etc/logrotate.d/docker-services
/var/lib/docker/containers/*/*-json.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0644 root root
    postrotate
        systemctl reload docker
    endscript
}

This comprehensive guide covers all major troubleshooting scenarios for the three new services. Each section provides practical commands and solutions based on the actual infrastructure setup and notification stack integration.