LiteLLM Service Documentation

Overview

LiteLLM is a unified Large Language Model (LLM) API proxy that provides a single interface to interact with 100+ LLM providers including OpenAI, Anthropic, Google Gemini, Groq, and many more. It acts as a gateway that standardizes API calls across different providers while offering advanced features like load balancing, fallbacks, cost tracking, and rate limiting.

In our infrastructure, LiteLLM serves as the central AI/LLM proxy, enabling applications like Open WebUI to access multiple AI providers through a single endpoint.

Why LiteLLM?

1. Unified Interface

• Single API format (OpenAI-compatible) for all providers
• No need to integrate multiple SDKs or handle different API formats
• Seamless provider switching without code changes

2. Cost Tracking & Management

• Real-time cost tracking across all providers
• Budget limits and alerts
• Usage analytics per model, user, or team

3. Reliability Features

• Automatic fallbacks between providers
• Retry logic with exponential backoff
• Load balancing across multiple API keys
• Circuit breakers for failing endpoints

4. Developer Experience

• OpenAI-compatible API format
• Comprehensive logging and debugging
• API key management and rotation
• Model aliasing for user-friendly names

Architecture

Service Components

┌─────────────────┐      ┌─────────────────┐
│   Open WebUI    │      │  Other Apps     │
└────────┬────────┘      └────────┬────────┘
         │                         │
         └───────────┬─────────────┘
                     │
              ┌──────▼──────┐
              │   LiteLLM   │  (llm.rbnk.uk)
              │   Proxy     │
              └──────┬──────┘
                     │
        ┌────────────┼────────────┐
        │            │            │
   ┌────▼────┐  ┌───▼────┐  ┌───▼────┐
   │ OpenAI  │  │Anthropic│  │ Gemini │
   └─────────┘  └─────────┘  └─────────┘

Network Configuration

• Container: litellm
• Networks:
• traefik_proxy (external) - for web access
• supabase_internal (external) - for database access
• Internal Port: 4000
• External URLs:
• Production: https://llm.rbnk.uk
• Test: https://llm-test.rbnk.uk (legacy)
• Database: Local Supabase (supa-db:5432/postgres)

Important: The traefik.docker.network: traefik_proxy label is required when connecting to multiple networks, otherwise Traefik may route to the wrong network interface.

Services Using LiteLLM

Service	Configuration
yt-journal	OPENAI_BASE_URL=https://llm.rbnk.uk/
karakeep	OPENAI_BASE_URL=https://llm.rbnk.uk
paperless-ai	CUSTOM_BASE_URL=https://llm.rbnk.uk
glance	Dashboard links

Configuration Details

Docker Compose Configuration

The service is defined in /srv/dockerdata/litellm-test/docker-compose.yml:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: unless-stopped
    env_file: ./.env
    volumes:
      - ./litellm-config.yaml:/app/litellm-config.yaml:ro
      - ./data:/data
    command: ["--config", "/app/litellm-config.yaml", "--port", "4000"]
    networks:
      - traefik_proxy
      - supabase_internal
    labels:
      traefik.enable: "true"
      traefik.docker.network: traefik_proxy  # Required for multi-network
      traefik.http.routers.litellm.rule: Host(`llm.rbnk.uk`)
      traefik.http.routers.litellm.entrypoints: websecure
      traefik.http.routers.litellm.tls.certresolver: cf

networks:
  traefik_proxy:
    external: true
  supabase_internal:
    external: true

Model Routing Configuration

Models are configured in /srv/dockerdata/litellm-test/litellm-config.yaml:

model_list:
  - model_name: openai-gpt4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: anthropic-claude-4
    litellm_params:
      model: anthropic/claude-sonnet-4-latest
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: gemini-2-5-pro
    litellm_params:
      model: gemini/gemini-2.5-pro-latest
      api_key: os.environ/GEMINI_API_KEY

  - model_name: groq-llama-3-70b
    litellm_params:
      model: groq/llama-3-70b-8192
      api_key: os.environ/GROQ_API_KEY

Environment Variables

Required environment variables in .env:

# Provider API Keys
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=...
GROQ_API_KEY=gsk_...

# LiteLLM Configuration
LITELLM_MASTER_KEY=sk-...  # Admin API key
LITELLM_SALT_KEY=sk-...    # For hashing
DATABASE_URL=postgresql://postgres:PASSWORD@supa-db:5432/postgres  # Local Supabase

# Optional: Cost tracking
LITELLM_REDACT_MESSAGES_IN_LOGS=true
LITELLM_LOG_LEVEL=INFO

Provider API Keys

Each provider requires its own API key:

1. OpenAI: Get from https://platform.openai.com/api-keys
2. Anthropic: Get from https://console.anthropic.com/
3. Google Gemini: Get from https://aistudio.google.com/apikey
4. Groq: Get from https://console.groq.com/keys

Load Balancing and Fallbacks

Configure multiple deployments for load balancing:

model_list:
  # Primary deployment
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY_1

  # Fallback deployment
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY_2

router_settings:
  routing_strategy: "least-busy"  # or "round-robin", "latency-based"
  fallbacks:
    gpt-4o: ["claude-3-sonnet"]  # Fallback to Claude if GPT-4 fails

Cost Tracking Setup

Enable cost tracking with database persistence:

general_settings:
  database_url: "postgresql://litellm:password@supa-db:5432/litellm"
  master_key: os.environ/LITELLM_MASTER_KEY

  # Cost tracking
  track_cost_callback: true
  max_budget: 1000  # Monthly budget in USD
  budget_duration: "monthly"

  # Alerts
  alerting_threshold: 0.8  # Alert at 80% budget usage
  alerting_email: "[email protected]"

Integration with Open WebUI

Open WebUI connects to LiteLLM as an OpenAI-compatible endpoint:

1. Configure Open WebUI Environment

In Open WebUI's .env:

# Use LiteLLM as OpenAI API
OPENAI_API_BASE_URL=http://litellm-test:4000/v1
OPENAI_API_KEY=sk-litellm-master-key  # Your LITELLM_MASTER_KEY

2. Model Selection in Open WebUI

Models configured in LiteLLM appear automatically in Open WebUI's model selector: - openai-gpt4o → GPT-4o - anthropic-claude-4 → Claude Sonnet 4 - gemini-2-5-pro → Gemini 2.5 Pro - groq-llama-3-70b → Llama 3 70B (via Groq)

3. Advanced Integration Features

# Enable streaming
litellm_settings:
  stream: true
  stream_timeout: 60

# Custom headers for Open WebUI
litellm_settings:
  default_headers:
    "X-Source": "open-webui"

Adding New Models/Providers

1. Basic Model Addition

Add to litellm-config.yaml:

- model_name: mistral-large
  litellm_params:
    model: mistral/mistral-large-latest
    api_key: os.environ/MISTRAL_API_KEY

2. Custom Endpoint Providers

For self-hosted models (e.g., Ollama):

- model_name: local-llama3
  litellm_params:
    model: openai/llama3:8b
    api_base: http://ollama:11434/v1
    api_key: "dummy"  # Required but not used
    custom_llm_provider: "openai"  # Use OpenAI format

3. Model Aliases

Create user-friendly names:

- model_name: fast-chat
  litellm_params:
    model: groq/llama-3-70b-8192
    api_key: os.environ/GROQ_API_KEY

- model_name: smart-chat
  litellm_params:
    model: anthropic/claude-3-5-sonnet-latest
    api_key: os.environ/ANTHROPIC_API_KEY

4. Model-Specific Settings

Configure per-model parameters:

- model_name: gpt-4o-structured
  litellm_params:
    model: openai/gpt-4o
    api_key: os.environ/OPENAI_API_KEY
    response_format: {"type": "json_object"}
    max_tokens: 4096
    temperature: 0.7

Monitoring Usage and Costs

1. Built-in Admin UI

Access at https://llm.rbnk.uk/ui (requires LITELLM_MASTER_KEY):

• Real-time usage statistics
• Cost breakdown by model/user
• API key management
• Model performance metrics

2. API Endpoints for Monitoring

# Get current spend
curl https://llm.rbnk.uk/spend \
  -H "Authorization: Bearer sk-litellm-master-key"

# Get model stats
curl https://llm.rbnk.uk/model/info \
  -H "Authorization: Bearer sk-litellm-master-key"

# Health check
curl https://llm.rbnk.uk/health

3. Prometheus Metrics

Enable metrics export:

general_settings:
  enable_prometheus_metrics: true
  prometheus_port: 9090

4. Custom Callbacks

Log to external systems:

litellm_settings:
  success_callback: ["supabase"]  # Log to Supabase
  failure_callback: ["logger"]

  # Supabase logging config
  supabase_url: "https://supabase.rbnk.uk"
  supabase_key: os.environ/SUPABASE_SERVICE_ROLE_KEY

Performance Optimization

1. Connection Pooling

litellm_settings:
  max_parallel_requests: 100
  request_timeout: 600
  connection_pool_size: 10

2. Caching

Enable response caching:

litellm_settings:
  cache: true
  cache_params:
    type: "redis"  # or "in-memory"
    host: "redis"
    port: 6379
    ttl: 3600  # 1 hour

3. Rate Limiting

Protect against abuse:

general_settings:
  max_requests_per_minute: 60
  max_tokens_per_minute: 150000

  # Per-user limits
  user_max_requests_per_minute: 20
  user_max_tokens_per_minute: 50000

4. Request Optimization

router_settings:
  retry_after: 5  # Seconds between retries
  allowed_fails: 3  # Failures before switching providers
  cooldown_time: 60  # Seconds before retrying failed provider

  # Latency-based routing
  routing_strategy: "latency-based"
  track_latency: true

Security Considerations

1. API Key Management

• Master Key: Admin access to LiteLLM UI and management APIs
• User Keys: Generated via admin UI or API
• Provider Keys: Stored in environment variables, never in config

2. Network Security

• Internal communication only on Docker networks
• SSL/TLS termination at Traefik
• No direct external access to container

3. Access Control

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

  # Enable user authentication
  auth:
    enable: true
    default_team_id: "default-team"

  # IP allowlist
  allowed_ips: ["10.0.0.0/8", "172.16.0.0/12"]

4. Data Privacy

litellm_settings:
  # Redact sensitive data in logs
  redact_messages_in_logs: true
  redact_user_api_keys: true

  # Disable telemetry
  disable_telemetry: true

  # Custom PII detection
  pii_masking: true
  pii_types: ["email", "phone", "ssn"]

5. Audit Logging

general_settings:
  # Enable audit logs
  audit_logs: true
  audit_log_path: "/data/audit.log"

  # What to log
  log_request_body: false  # Privacy
  log_response_body: false
  log_headers: ["X-User-ID", "X-Source"]

Troubleshooting Common Issues

1. Provider Authentication Errors

Symptom: "Invalid API key" or "Authentication failed"

Solutions:

# Check environment variables
docker exec litellm-test env | grep API_KEY

# Test API key directly
curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY"

# Restart with updated env
docker compose -f litellm-test/docker-compose.yml restart

2. Model Not Found

Symptom: "Model not found" errors

Solutions:

# List available models
curl https://llm.rbnk.uk/models \
  -H "Authorization: Bearer sk-litellm-master-key"

# Check model config
docker exec litellm-test cat /app/litellm-config.yaml

# Validate model name format
# Correct: openai/gpt-4o, anthropic/claude-3-sonnet
# Wrong: gpt-4o, claude-3-sonnet

3. Rate Limiting Issues

Symptom: 429 errors or "Rate limit exceeded"

Solutions:

# Increase limits in config
general_settings:
  max_requests_per_minute: 120

# Add more API keys for load balancing
- model_name: gpt-4o
  litellm_params:
    model: openai/gpt-4o
    api_key: os.environ/OPENAI_API_KEY_2

4. High Latency

Symptom: Slow response times

Debug steps:

# Check container resources
docker stats litellm-test

# View detailed logs
docker logs litellm-test --tail 100

# Enable debug mode
docker exec litellm-test litellm --debug

5. Memory Issues

Symptom: Container restarts or OOM errors

Solutions:

# In docker-compose.yml
services:
  litellm:
    deploy:
      resources:
        limits:
          memory: 1G
        reservations:
          memory: 512M

6. Connection Errors

Symptom: "Connection refused" or timeout errors

Check:

# Test internal connectivity
docker exec litellm-test curl http://localhost:4000/health

# Check network
docker network inspect traefik_proxy

# Verify Traefik routing
curl -H "Host: llm.rbnk.uk" http://localhost/health

Common Log Locations

• Container logs: docker logs litellm-test
• Application logs: /srv/dockerdata/litellm-test/data/litellm.log
• Audit logs: /srv/dockerdata/litellm-test/data/audit.log

Debug Commands

# Full debug mode
docker compose -f litellm-test/docker-compose.yml down
docker compose -f litellm-test/docker-compose.yml up

# Test specific model
curl -X POST https://llm.rbnk.uk/v1/chat/completions \
  -H "Authorization: Bearer sk-litellm-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai-gpt4o",
    "messages": [{"role": "user", "content": "Test"}],
    "max_tokens": 10
  }'

# Check provider status
curl https://llm.rbnk.uk/health/providers \
  -H "Authorization: Bearer sk-litellm-master-key"

Best Practices

1. Always use environment variables for API keys
2. Configure fallbacks for critical applications
3. Monitor costs regularly through the admin UI
4. Set budget limits to prevent unexpected charges
5. Use model aliases for easier provider switching
6. Enable caching for frequently repeated queries
7. Implement rate limiting to prevent abuse
8. Regular backups of configuration and usage data
9. Test failover scenarios in development
10. Keep provider SDKs updated via image updates

Virtual Keys Implementation

As of December 2025, our LiteLLM deployment uses Virtual Keys for enhanced security, cost tracking, and service isolation. Virtual keys replace master key usage in production services and provide:

• Individual budget limits per service ($5-$20/month)
• Real-time cost tracking and monitoring
• Security isolation between services
• Automated alerting for budget thresholds

Current Virtual Keys: - n8n Workflows: sk-7UauGn... ($20/month) - Open WebUI: sk-ddZoN7... ($15/month) - Browser-use: sk-3EnNwo... ($10/month) - Testing/Dev: sk-hOsuTU... ($5/month)

For complete virtual keys documentation, see: LiteLLM Virtual Keys Implementation

Related Documentation

• LiteLLM Virtual Keys Implementation - Virtual keys setup and management
• Traefik Configuration - Reverse proxy setup
• Open WebUI - AI chat interface
• Architecture Overview - System design
• Adding Services - Integration guide