Git Security Best Practices¶

This guide covers security practices for working with our dual-repository Git setup, ensuring sensitive data never reaches public repositories.

Table of Contents¶

Overview
Files That Should Never Be Committed
Security Filtering in Sync Process
Pre-Commit Security Checks
Secure Development Practices
Handling Sensitive Data
Emergency Procedures
Security Audit Checklist

Overview¶

Our Git security strategy consists of multiple layers: 1. Prevention: .gitignore patterns and pre-commit hooks 2. Detection: Automated scanning for secrets 3. Filtering: Sync process that removes sensitive files 4. Response: Procedures for handling exposed secrets

Dual Repository Architecture¶

We maintain a secure dual repository approach:

Gitea (Primary/Private): Self-hosted repository that maintains full history including .env files
Contains complete version control history with all configuration files
Allows team members to track changes to sensitive configurations
Provides full rollback capabilities for all infrastructure changes
Accessible only through secure VPN or authenticated access
GitHub (Public Mirror): Public repository with all sensitive files removed from entire history
Automatically synced from Gitea with comprehensive security filtering
All sensitive files are removed from the complete Git history, not just current commits
Uses git-filter-repo to rewrite history, ensuring secrets never reach GitHub
Safe for public collaboration and open-source contributions

This approach provides the best of both worlds: - Internal: Full version control and history for operational needs - External: Clean, secure codebase for public sharing

Files That Should Never Be Committed¶

Critical Files to Exclude¶

# Environment files
*.env
*.env.*
!*.env.example
!*.env.template

# SSL/TLS Certificates
*.pem
*.key
*.crt
*.cer
*.pfx
*.p12
acme.json

# SSH Keys
id_rsa*
id_dsa*
id_ecdsa*
id_ed25519*
*.ppk
known_hosts
authorized_keys

# Credentials and Secrets
**/secrets/
**/credentials/
*.secret
*_secret
*-secret
secrets.yml
secrets.yaml
credentials.json

# API Keys and Tokens
.api_key
.api_keys
*_api_key
*_token
.token
tokens.json

# Database Files
*.sqlite
*.sqlite3
*.db
*.sql.gz
*-dump.sql

# Backup Files
*.bak
*.backup
*.dump
*.tar.gz
_backup/
backups/

# Logs with Potential Sensitive Data
*.log
logs/
**/logs/
audit.log
access.log

# Docker Volumes with Sensitive Data
**/volumes/data/
**/postgres-data/
**/mysql-data/
**/redis-data/

# Application-Specific
supabase/.env
supabase/kong.yml
traefik/letsencrypt/
gitea/data/
pocketbase/pb_data/

# Temporary Files
*.tmp
*.temp
*.swp
.DS_Store

Example .gitignore for Infrastructure¶

# Infrastructure-specific ignores
# This should be in /srv/dockerdata/.gitignore

# Environment files - CRITICAL
**/*.env
!**/*.env.example
!**/*.env.template

# SSL Certificates - CRITICAL
traefik/letsencrypt/
**/*.pem
**/*.key
**/acme.json

# Service Data Directories
gitea/data/
supabase/volumes/
pocketbase/pb_data/
n8n/data/
**/postgres-data/
**/mysql-data/

# Backup directories
_backup/
**/_backup/
*.backup
*.bak

# Logs
**/*.log
**/logs/
sync-logs/

# Temporary files
*.tmp
*.swp
.DS_Store
*~

# IDE files
.idea/
.vscode/
*.iml

# Monitoring data
prometheus/data/
grafana/data/

Security Filtering in Sync Process¶

How the Sync Filter Works¶

Our Gitea-to-GitHub sync process includes comprehensive security filtering that removes sensitive files from the entire Git history:

Key Components¶

Git History Rewriting Tools:
git-filter-repo (preferred): Modern Python-based tool for efficient history rewriting
BFG Repo-Cleaner: Java-based tool specialized in removing passwords and large files
git filter-branch (fallback): Native Git tool (deprecated but still functional)
Security Verification Script: verify-no-secrets.sh performs comprehensive scanning using industry-standard secret detection patterns from:
Gitleaks: Popular open-source secret scanner by Zachary Rice
TruffleHog: High-entropy string and pattern-based secret detection by Truffle Security
detect-secrets: Yelp's tool for preventing secrets in code
Combined patterns from AWS git-secrets, GitHub secret scanning, and GitLab secret detection
Configuration-based Filtering: Patterns defined in git-sync-config.yml control what gets filtered

Why Complete History Rewriting is Essential¶

When syncing to a public repository, removing files from just the current commit is insufficient because: - Git history is immutable - anyone can checkout old commits - Deleted files remain in Git history unless explicitly rewritten - Secret scanning tools check entire repository history - Once exposed, secrets must be considered compromised

Our sync process ensures that sensitive files never existed in the GitHub repository's history.

Complete History Filtering¶

The sync process doesn't just exclude files from the current commit - it removes them from the entire Git history:

# git-sync-config.yml
global_exclude_patterns:
  # Environment files - removed from ALL commits
  - ".env"
  - "*.env"
  - "**/.env*"
  - ".envrc"

  # Private keys - completely erased from history
  - "*.pem"
  - "*.key"
  - "*.pfx"
  - "id_rsa*"
  - "id_dsa*"

  # Credentials - pattern matching across all commits
  - "**/secrets/*"
  - "**/credentials/*"
  - "*password*"
  - "*token*"
  - "*secret*"

Sync Security Process Flow¶

Clone Full Repository

# Full clone from Gitea (not shallow)
git clone https://gitea.rbnk.uk/user/repo.git

History Filtering

# Using git-filter-repo (example)
git filter-repo --paths-from-file exclude-patterns.txt --invert-paths --force

# Using BFG (example)
java -jar bfg.jar --delete-files '*.env' --no-blob-protection repo.git

# Result: Complete removal of sensitive files from all commits

Security Verification

# Run comprehensive security scan
/srv/dockerdata/_scripts/verify-no-secrets.sh /path/to/filtered-repo

# Checks for:
# - Environment files (*.env, .envrc)
# - Private keys (*.pem, *.key, id_rsa*)
# - Cloud credentials (.aws/*, gcp-credentials.json)
# - API keys and tokens (pattern matching)
# - Database dumps (*.sql, *.dump)

Push to GitHub

# Only after verification passes
git push github main --force

Understanding Force Pushes in Our Security Model¶

Force pushes (--force) are necessary when cleaning Git history but are handled safely in our architecture:

Why Force Pushes are Required¶

History rewriting changes commit SHAs
GitHub mirror must exactly match the filtered history
Normal pushes would be rejected due to divergent histories
Complete secret removal requires rewriting past commits

Safety Measures¶

Gitea remains unchanged: Primary repository keeps full history
Automated process: Reduces human error
Pre-sync verification: Ensures only clean history is pushed
One-way sync: GitHub never pushes back to Gitea
Clear ownership: GitHub is explicitly a read-only mirror

Impact on Collaborators¶

External contributors should fork from GitHub (clean history)
Pull requests are managed through GitHub's UI
Internal team uses Gitea for direct development
Force pushes only affect the GitHub mirror, not active development

Security Verification Details¶

The verify-no-secrets.sh script performs industry-standard secret detection:

Pattern-based Detection¶

AWS Access Keys: AKIA[0-9A-Z]{16}
GitHub Tokens: ghp_[0-9a-zA-Z]{36}
Private Key Headers: -----BEGIN (RSA |DSA |EC |OPENSSH )?PRIVATE KEY-----
JWT Tokens: eyJ[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}
API Keys: Various service-specific patterns (Stripe, Twilio, SendGrid, etc.)

File-based Detection¶

Environment files (.env, *.env)
Private key files (*.pem, *.key)
Cloud provider credentials
Database dumps and backups
Terraform state files

Manual Verification¶

Before any sync, you can manually verify that secrets will be filtered:

# Dry run to see what will be filtered
/srv/dockerdata/_scripts/gitea-github-sync.sh -d -r https://gitea.rbnk.uk/user/repo.git

# Security verification only
/srv/dockerdata/_scripts/gitea-github-sync.sh -v -r https://gitea.rbnk.uk/user/repo.git

# Check sync logs for filtering details
tail -f /srv/dockerdata/gitea/sync-logs/sync-*.log

Pre-Commit Security Checks¶

Installing Pre-Commit Hooks¶

# Install the security pre-commit hook
cat > /srv/dockerdata/.git/hooks/pre-commit << 'EOF'
#!/bin/bash
# Pre-commit hook to prevent committing secrets

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'

# Check for common secret patterns
check_secrets() {
    local file=$1

    # Skip binary files
    if file "$file" | grep -q "binary"; then
        return 0
    fi

    # Patterns that indicate secrets
    local patterns=(
        "password.*=.*['\"].*['\"]"
        "api[_-]?key.*=.*['\"].*['\"]"
        "secret.*=.*['\"].*['\"]"
        "token.*=.*['\"].*['\"]"
        "private[_-]?key"
        "-----BEGIN.*PRIVATE KEY-----"
        "-----BEGIN RSA PRIVATE KEY-----"
        "-----BEGIN EC PRIVATE KEY-----"
        "-----BEGIN OPENSSH PRIVATE KEY-----"
        "aws_access_key_id"
        "aws_secret_access_key"
        "AKIA[0-9A-Z]{16}"
    )

    for pattern in "${patterns[@]}"; do
        if grep -iE "$pattern" "$file" > /dev/null 2>&1; then
            echo -e "${RED}ERROR: Potential secret found in $file${NC}"
            echo -e "${YELLOW}Pattern matched: $pattern${NC}"
            return 1
        fi
    done

    return 0
}

# Check file extensions that should never be committed
check_extensions() {
    local file=$1
    local forbidden_extensions=(
        "env"
        "key"
        "pem"
        "pfx"
        "p12"
    )

    for ext in "${forbidden_extensions[@]}"; do
        if [[ "$file" == *".$ext" ]]; then
            echo -e "${RED}ERROR: Forbidden file type: $file${NC}"
            echo -e "${YELLOW}Files with .$ext extension should not be committed${NC}"
            return 1
        fi
    done

    return 0
}

# Main check
failed=0
for file in $(git diff --cached --name-only); do
    if [ -f "$file" ]; then
        check_extensions "$file" || failed=1
        check_secrets "$file" || failed=1
    fi
done

if [ $failed -eq 1 ]; then
    echo -e "${RED}Commit aborted due to security concerns${NC}"
    echo -e "${YELLOW}If this is intentional, use --no-verify flag${NC}"
    exit 1
fi

echo -e "${GREEN}Security checks passed${NC}"
exit 0
EOF

chmod +x /srv/dockerdata/.git/hooks/pre-commit

Using Secret Scanning Tools¶

# Install and use gitleaks
wget https://github.com/zricethezav/gitleaks/releases/latest/download/gitleaks_linux_amd64
chmod +x gitleaks_linux_amd64
sudo mv gitleaks_linux_amd64 /usr/local/bin/gitleaks

# Scan repository
cd /srv/dockerdata
gitleaks detect --source . -v

# Scan before commit
gitleaks protect --staged -v

Secure Development Practices¶

1. Environment Variable Management¶

DO:

# Create .env.example with placeholders
cat > myservice/.env.example << 'EOF'
# Database Configuration
DB_HOST=localhost
DB_PORT=5432
DB_NAME=myapp
DB_USER=myapp_user
DB_PASSWORD=CHANGE_ME

# API Keys
EXTERNAL_API_KEY=YOUR_API_KEY_HERE
JWT_SECRET=GENERATE_A_SECURE_SECRET

# Feature Flags
ENABLE_DEBUG=false
EOF

# Add real values only to .env (never committed)
cp myservice/.env.example myservice/.env
vim myservice/.env  # Add real values

DON'T:

# Never hardcode secrets
API_KEY = "sk-1234567890abcdef"  # NEVER DO THIS

# Never commit .env files
git add myservice/.env  # WRONG!

2. Secret Generation¶

# Generate secure passwords
openssl rand -base64 32

# Generate API keys
python3 -c "import secrets; print(secrets.token_urlsafe(32))"

# Generate JWT secrets
openssl rand -hex 64

# UUID for unique identifiers
uuidgen | tr -d '-'

3. Using Docker Secrets¶

# docker-compose.yml with secrets
version: '3.8'

services:
  app:
    image: myapp:latest
    secrets:
      - db_password
      - api_key
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
      API_KEY_FILE: /run/secrets/api_key

secrets:
  db_password:
    file: ./secrets/db_password.txt
  api_key:
    file: ./secrets/api_key.txt

4. Configuration Templates¶

# Create configuration template
cat > myservice/config/app.conf.template << 'EOF'
server {
    listen 8080;
    api_endpoint = "${API_ENDPOINT}";
    api_key = "${API_KEY}";

    database {
        host = "${DB_HOST:localhost}";
        port = ${DB_PORT:5432};
        name = "${DB_NAME}";
        user = "${DB_USER}";
        password = "${DB_PASSWORD}";
    }
}
EOF

# Generate actual config at runtime
envsubst < app.conf.template > app.conf

Handling Sensitive Data¶

Temporary Secret Storage¶

# Use environment variables for one-time operations
export TEMP_SECRET=$(openssl rand -base64 32)
docker run -e SECRET=$TEMP_SECRET myapp:latest
unset TEMP_SECRET

# Use stdin for secrets
echo -n "your-secret" | docker secret create my_secret -

Encrypted Configuration¶

# Encrypt sensitive configuration
openssl enc -aes-256-cbc -salt -in config.json -out config.json.enc

# Decrypt at runtime
openssl enc -aes-256-cbc -d -in config.json.enc -out config.json

# Use age for modern encryption
age-keygen -o key.txt
age -r age1... -o config.age config.json
age -d -i key.txt config.age > config.json

Secure File Permissions¶

# Set restrictive permissions on sensitive files
chmod 600 .env
chmod 600 secrets/*
chmod 700 secrets/

# Ensure proper ownership
chown root:root .env
chown -R root:docker secrets/

Emergency Procedures¶

If Secrets Are Committed¶

Immediate Actions

# DO NOT PUSH!
# If not pushed, remove from history
git reset --soft HEAD~1
git reset HEAD secret-file
rm secret-file
git commit -m "Remove accidentally staged secret"

If Already Pushed to Gitea

# Remove from history
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch path/to/secret-file" \
  --prune-empty --tag-name-filter cat -- --all

# Force push (coordinate with team)
git push --force --all origin
git push --force --tags origin

If Pushed to GitHub

# Immediately rotate the exposed secret
# Then contact GitHub support to purge from cache

# Use BFG Repo-Cleaner for easier cleanup
java -jar bfg.jar --delete-files secret-file
git reflog expire --expire=now --all
git gc --prune=now --aggressive

Secret Rotation Procedure¶

# 1. Generate new secrets
./scripts/rotate-secrets.sh

# 2. Update all services
for service in */; do
    if [ -f "$service/.env" ]; then
        echo "Updating secrets for $service"
        # Update .env file with new values
    fi
done

# 3. Restart affected services
docker compose down
docker compose up -d

# 4. Verify services are running
docker ps

Security Audit Checklist¶

Pre-Commit Checklist¶

[ ] No .env files in staging area
[ ] No private keys or certificates
[ ] No hardcoded passwords or API keys
[ ] All secrets use environment variables
[ ] .gitignore properly configured
[ ] Pre-commit hooks installed and working

Weekly Security Review¶

[ ] Audit git history for secrets: git log -p | grep -iE 'password|secret|key'
[ ] Check file permissions: find . -name "*.env" -exec ls -la {} \;
[ ] Verify .gitignore entries: git check-ignore **/*.env
[ ] Review GitHub repository for accidental pushes
[ ] Scan with secret detection tools

Monthly Security Tasks¶

[ ] Rotate all API keys and tokens
[ ] Update secret scanning rules
[ ] Review and update .gitignore patterns
[ ] Audit access logs for suspicious activity
[ ] Test secret rotation procedures

Additional Security Tools¶

1. Git-secrets by AWS¶

# Install
git clone https://github.com/awslabs/git-secrets
cd git-secrets && make install

# Configure
git secrets --install
git secrets --register-aws
git secrets --add-provider -- cat /srv/dockerdata/.git-secrets-patterns

2. Detect-secrets¶

# Install
pip install detect-secrets

# Create baseline
detect-secrets scan > .secrets.baseline

# Scan for new secrets
detect-secrets scan --baseline .secrets.baseline

3. TruffleHog¶

# Run TruffleHog
docker run --rm -v "$PWD:/pwd" trufflesecurity/trufflehog:latest \
  git file:///pwd --since-commit HEAD~10

Best Practices Summary¶

Never commit secrets - Use environment variables and .env.example files
Use .gitignore - Maintain comprehensive patterns for sensitive files
Enable pre-commit hooks - Automated checking prevents accidents
Separate configuration from code - Keep secrets in environment
Rotate regularly - Change secrets periodically
Audit continuously - Regular scans for exposed secrets
Encrypt when necessary - Use encryption for sensitive configs
Limit access - Restrict who can access production secrets
Document security procedures - Ensure team knows the protocols
Respond quickly - Have procedures ready for security incidents

Remember: When in doubt, don't commit it!