Storage Strategy - Dual Disk Architecture¶

Current Storage Layout¶

┌─────────────────────────────────────────────────────────────┐
│                     Proxmox VM: dockerhost                   │
│                                                              │
│  ┌─────────────────────┐        ┌─────────────────────────┐ │
│  │   /dev/sda (100GB)  │        │    /dev/sdb (500GB)     │ │
│  │   System Disk       │        │    Data Disk            │ │
│  ├─────────────────────┤        ├─────────────────────────┤ │
│  │ sda1: /boot/efi 1GB │        │ sdb1: /srv/dockerdata   │ │
│  │ sda2: /boot     2GB │        │       492GB formatted    │ │
│  │ sda3: LVM     97GB  │        │       288GB available    │ │
│  │  └── / (96GB)       │        │                         │ │
│  │      29% used        │        │       39% used          │ │
│  └─────────────────────┘        └─────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Why This Dual-Disk Design?¶

Original Setup Intent¶

Based on the Proxmox configuration:

Disks:
• scsi0 (OS): 60GB on vmdata
• scsi1 (Data): 200GB on vmdata (thin provisioned)

Design Rationale: 1. Separation of Concerns: OS vs Application Data 2. Independent Growth: Data disk can expand without OS reinstall 3. Performance: Separate I/O paths for system and data 4. Backup Strategy: Different backup policies for OS vs data

Current Reality¶

System Disk (100GB): - 96GB root partition: 29% used (~66GB free) - Status: Healthy with ample headroom - Contains: OS, system packages, logs, Docker engine

Data Disk (500GB): - 492GB usable: ~39% used (288GB available) - Mounted at: /srv/dockerdata - Contains: All container data, configs, backups

Storage Organization¶

Directory Structure¶

/srv/dockerdata/
├── SERVICE_NAME/              # Each service isolated
│   ├── docker-compose.yml     # Service definition
│   ├── .env                   # Configuration (640 perms)
│   ├── data/                  # Persistent data
│   │   ├── database/          # Database files
│   │   ├── uploads/           # User uploads
│   │   └── cache/             # Temporary data
│   └── config/                # Configuration files
├── _scripts/                  # Operational scripts
├── _templates/                # Service templates
├── _backup/                   # Backup storage
└── ollama/                    # Special case - models
    └── models/                # Large LLM files

Storage Patterns by Service Type¶

Database Services (PostgreSQL):

supabase/data/
├── postgres/          # ~500MB-5GB typical
├── wal_archive/       # Transaction logs
└── backups/           # Local dumps

Application Services:

open-webui/data/
├── backend/           # SQLite DB, session data
└── uploads/           # User uploaded files

Large File Services:

ollama/models/
├── llama2:7b/         # 3.8GB
├── mistral:latest/    # 4.1GB
└── llama2:70b/        # 40GB (careful!)

Storage Management¶

Regular Maintenance¶

Keep storage healthy with periodic cleanup:

# Check what's using space
sudo du -h --max-depth=1 / | sort -h
du -sh /srv/dockerdata/* | sort -h

# Clean Docker artifacts (unused images, containers, volumes)
docker system prune -a --volumes

# Clean package cache
sudo apt clean
sudo apt autoremove

# Rotate/compress logs
sudo journalctl --vacuum-time=14d

Optional: Move Docker Root to Data Disk¶

If system disk becomes constrained again, move Docker data:

# Stop Docker
sudo systemctl stop docker

# Move Docker data
sudo mv /var/lib/docker /srv/dockerdata/docker-root

# Configure Docker to use new location
sudo tee /etc/docker/daemon.json <<EOF
{
  "data-root": "/srv/dockerdata/docker-root"
}
EOF

# Start Docker
sudo systemctl start docker

Backup Storage Considerations¶

Current Backup Location¶

/srv/dockerdata/_backup/
├── supabase/
│   └── supabase-20250718.sql.gz    # ~50-200MB per backup
└── pocketbase/
    └── app-20250718.tar.gz          # ~10-50MB per backup

Issue: Backups on same disk as data (no redundancy)

Recommended Backup Strategy¶

External Backup Target:

# Mount external backup location
/mnt/backups/              # NFS, S3FS, or external disk

# Modify backup scripts to use external location
BACKUP_DIR="/mnt/backups/dockerdata"

Backup Rotation Policy:

# Current: 14-day retention
# Recommended: 7-30-365 (weekly, monthly, yearly)

Storage Monitoring¶

Key Metrics to Track¶

# Disk usage
df -h /srv/dockerdata
df -h /

# Inode usage (small files can exhaust inodes)
df -i

# Directory sizes
du -sh /srv/dockerdata/*

# Growth rate
# Compare weekly to predict when disks fill

Automated Alerts¶

Create monitoring script:

#!/bin/bash
# /srv/dockerdata/_scripts/storage-monitor.sh

THRESHOLD=80
DISK_USAGE=$(df / | awk 'NR==2 {print $5}' | sed 's/%//')

if [ $DISK_USAGE -gt $THRESHOLD ]; then
    echo "WARNING: System disk at ${DISK_USAGE}%"
    # Send alert (email, webhook, etc)
fi

Storage Optimization¶

1. Container Image Management¶

# Remove unused images
docker image prune -a

# Limit image retention
docker image ls --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

# Use Alpine-based images when possible
nginx:alpine     # 40MB vs 190MB for debian
python:alpine    # 50MB vs 900MB for standard

2. Log Management¶

# In docker-compose.yml
services:
  app:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

3. Volume Optimization¶

# Use named volumes for better management
volumes:
  postgres_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /srv/dockerdata/postgres/data

Future Growth Planning¶

When to Expand¶

Monitor these thresholds: - System disk > 85%: Critical, act immediately - Data disk > 70%: Plan expansion - Any partition > 90%: Emergency

How to Expand Data Disk¶

# On Proxmox host
qm resize 100 scsi1 +100G

# In VM
sudo growpart /dev/sdb 1
sudo resize2fs /dev/sdb1

Migration Path¶

When disks can't grow: 1. Add third disk: Dedicated for specific service 2. External storage: NFS/S3 for large files 3. New VM: Distribute services across VMs

Best Practices¶

1. Separation Rules¶

OS disk: System only
Data disk: All application data
External: Backups and archives

2. Monitoring¶

Daily disk usage checks
Weekly growth analysis
Monthly capacity planning

3. Cleanup Routine¶

# Weekly cleanup script
#!/bin/bash
docker system prune -f
sudo journalctl --vacuum-time=14d
find /srv/dockerdata -name "*.log" -mtime +30 -delete

4. Documentation¶

Track which services use most space
Document growth patterns
Plan before hitting limits

Recommendations¶

Ongoing Maintenance¶

Monitor disk usage weekly via Grafana dashboards
Run cleanup scripts monthly to remove unused Docker artifacts
Review storage growth trends quarterly

Best Practices¶

Implement automated cleanup scripts
Configure log rotation for all services
Maintain external backup destination (Cloudflare R2)

Future Considerations¶

Consider moving Docker root to data disk if system disk usage increases
Continue monitoring storage metrics in Grafana
Maintain disaster recovery plan with regular backup testing

Storage Decision Matrix¶

What to Store	System Disk	Data Disk	External
OS Files	✓
Docker Engine	✓
Container Images		✓
Application Data		✓
Database Files		✓
Backups		✓	✓ (R2)
Large Models (LLM)		✓
Logs (archived)			✓

The dual-disk strategy provides a solid foundation with ample capacity for growth.