Gitea to GitHub Sync Documentation¶

Table of Contents¶

Overview
Architecture
Security Features
Configuration
Usage
Filter Tools
Verification Process
Troubleshooting
Best Practices

Overview¶

The Gitea to GitHub sync functionality provides a secure way to mirror repositories from your self-hosted Gitea instance to GitHub while ensuring that sensitive data never reaches the public repository. This is achieved through comprehensive Git history rewriting that removes secrets from ALL commits, not just the latest ones.

Key Benefits¶

Complete Security: Sensitive files are removed from entire Git history
Automated Process: Hourly sync with manual trigger options
Flexible Configuration: Define custom patterns for filtering
Verification Built-in: Automated scanning ensures no secrets remain
Multiple Tool Support: Uses best available tool for history rewriting
Audit Trail: Comprehensive logging of all operations

Architecture¶

Components¶

Sync Script (/srv/dockerdata/_scripts/gitea-github-sync.sh)
Main orchestrator for the sync process
Handles repository cloning, filtering, and pushing
Supports multiple filter tools
Verification Script (/srv/dockerdata/_scripts/verify-no-secrets.sh)
Performs comprehensive security scanning
Uses industry-standard patterns from Gitleaks and TruffleHog
Ensures no secrets remain after filtering
Configuration (/srv/dockerdata/gitea/sync-config/git-sync-config.yml)
Defines what to sync and what to filter
Repository-specific settings
Global exclude patterns
Systemd Timer (gitea-sync.timer)
Runs sync hourly
Can be triggered manually

Process Flow¶

┌─────────────┐     1. Clone      ┌──────────────┐
│    Gitea    │ ─────────────────→ │ Temp Working │
│ Repository  │                    │  Directory   │
└─────────────┘                    └──────┬───────┘
                                          │ 2. Filter History
                                          ↓
                                   ┌──────────────┐
                                   │  Git Filter  │
                                   │    Tools     │
                                   └──────┬───────┘
                                          │ 3. Verify Security
                                          ↓
                                   ┌──────────────┐
                                   │   Security   │
                                   │ Verification │
                                   └──────┬───────┘
                                          │ 4. Push if Clean
                                          ↓
                                   ┌──────────────┐
                                   │    GitHub    │
                                   │  Repository  │
                                   └──────────────┘

Security Features¶

Complete History Filtering¶

Unlike simple .gitignore or sparse-checkout approaches, the sync process:

Rewrites Git History: Removes sensitive files from ALL commits
Updates All References: Ensures consistency across branches and tags
Cleans Repository: Removes unreferenced objects

Pattern-Based Filtering¶

The sync supports multiple pattern types:

Literal Paths: config/database.yml
Wildcards: *.env, **/*.key
Directory Patterns: **/secrets/*
Complex Globs: **/*password*

Security Verification¶

After filtering, the verification script checks for:

File-based Detection¶

Environment files (.env, *.env, .envrc)
Private keys (*.pem, *.key, id_rsa*)
Cloud credentials (.aws/*, gcp-credentials.json)
Database dumps (*.sql, *.dump)
Certificates (*.pfx, *.p12)

Content-based Detection¶

AWS Access Keys: AKIA[0-9A-Z]{16}
GitHub Tokens: ghp_[0-9a-zA-Z]{36}
Private Key Headers: -----BEGIN PRIVATE KEY-----
JWT Tokens: eyJ[a-zA-Z0-9_-]{10,}\.
API Keys for various services (Stripe, Twilio, etc.)

Configuration¶

Main Configuration File¶

Located at /srv/dockerdata/gitea/sync-config/git-sync-config.yml:

# GitHub target configuration
organization: your-github-org
default_visibility: private

# Global sync settings
sync_settings:
  branches_to_sync:
    - main
    - master
    - develop
    - feature/*
    - release/*

# Files to NEVER sync (removed from entire history)
global_exclude_patterns:
  # Environment files
  - ".env"
  - "*.env"
  - "**/.env*"
  - ".envrc"

  # Private keys and certificates
  - "*.pem"
  - "*.key"
  - "*.pfx"
  - "*.p12"
  - "id_rsa*"
  - "id_dsa*"
  - "id_ecdsa*"
  - "id_ed25519*"

  # Cloud provider credentials
  - ".aws/*"
  - "aws-credentials"
  - ".gcp/*"
  - "gcp-credentials.json"
  - ".azure/*"

  # Database dumps and backups
  - "*.sql"
  - "*.dump"
  - "*.bak"
  - "*.backup"
  - "_backup/*"

  # Secrets and credentials
  - "**/secrets/*"
  - "**/credentials/*"
  - "*password*"
  - "*token*"
  - "*secret*"
  - "*private*"

  # Docker and Terraform
  - "docker-compose.override.yml"
  - "*.tfstate"
  - "*.tfstate.*"
  - "terraform.tfvars"

  # Logs and temporary files
  - "*.log"
  - "logs/*"
  - "tmp/*"
  - "temp/*"

# Repository-specific configurations
repositories:
  my-project:
    enabled: true
    branches_to_sync:
      - main
      - production
    additional_exclude:
      - "internal/*"
      - "proprietary/*"

Environment Variables¶

The sync script respects these environment variables:

GITHUB_TOKEN: For API access (optional)
GIT_FILTER_TOOL: Force specific tool (git-filter-repo, bfg, git-filter-branch)
SYNC_DRY_RUN: Always run in dry-run mode
SYNC_LOG_LEVEL: Set logging verbosity (debug, info, warn, error)

Usage¶

Manual Sync¶

# Sync a specific repository
/srv/dockerdata/_scripts/gitea-github-sync.sh -r https://gitea.rbnk.uk/user/repo.git

# Dry run - preview what will be filtered
/srv/dockerdata/_scripts/gitea-github-sync.sh -d -r https://gitea.rbnk.uk/user/repo.git

# Security verification only
/srv/dockerdata/_scripts/gitea-github-sync.sh -v -r https://gitea.rbnk.uk/user/repo.git

# Help and options
/srv/dockerdata/_scripts/gitea-github-sync.sh -h

Automated Sync¶

# Check sync timer status
systemctl status gitea-sync.timer

# View next scheduled run
systemctl list-timers gitea-sync.timer

# Trigger immediate sync
sudo systemctl start gitea-sync.service

# View sync logs
journalctl -u gitea-sync.service -f

# Check detailed logs
tail -f /srv/dockerdata/gitea/sync-logs/sync-*.log

Monitoring Sync Status¶

# Check last successful sync
grep "Successfully synced" /srv/dockerdata/gitea/sync-logs/sync-*.log | tail -1

# Monitor for errors
grep -i error /srv/dockerdata/gitea/sync-logs/sync-*.log

# View filtering details
grep "Filtering with" /srv/dockerdata/gitea/sync-logs/sync-*.log

Filter Tools¶

git-filter-repo (Preferred)¶

Installation:

pip3 install git-filter-repo

Advantages: - Fast and memory-efficient - Modern Python-based tool - Excellent documentation - Active development

How it works: - Creates a fresh repository - Replays commits without sensitive files - Updates all references automatically

BFG Repo-Cleaner¶

Installation:

wget https://repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar
sudo mv bfg-1.14.0.jar /usr/local/bin/bfg.jar

Advantages: - Specifically designed for removing secrets - Very fast for large repositories - Simple pattern matching

How it works: - Works on a mirror clone - Removes files matching patterns - Aggressively cleans history

git filter-branch (Fallback)¶

Advantages: - Built into Git - No additional installation needed - Well-documented

Disadvantages: - Deprecated by Git team - Slower than alternatives - More complex usage

Verification Process¶

What Gets Checked¶

The verification script (verify-no-secrets.sh) performs:

File System Scan
Looks for files with sensitive extensions
Checks for known credential file names
Identifies large database dumps
Content Analysis
Scans file contents for secret patterns
Uses regex patterns from industry tools
Context-aware detection
Pattern Matching
AWS Access Keys
API tokens (GitHub, Stripe, Twilio, etc.)
Private key headers
JWT tokens
Generic password patterns

Running Verification Manually¶

# Verify a repository
/srv/dockerdata/_scripts/verify-no-secrets.sh /path/to/repo

# Check specific patterns
grep -r "password.*=" /path/to/repo
grep -r "BEGIN.*PRIVATE KEY" /path/to/repo

Troubleshooting¶

Common Issues¶

1. Filter Tool Not Found¶

Error: git-filter-repo: command not found

Solution:

# Install git-filter-repo
pip3 install git-filter-repo

# Or use BFG as alternative
wget https://repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar

2. Large Repository Timeout¶

Error: fatal: the remote end hung up unexpectedly

Solution:

# Increase buffer size
git config --global http.postBuffer 524288000

# Use pack limits
git config --global pack.windowMemory "100m"
git config --global pack.packSizeLimit "100m"

3. BFG Memory Issues¶

Error: java.lang.OutOfMemoryError

Solution:

# Run with more memory
java -Xmx4g -jar /usr/local/bin/bfg.jar --delete-files '*.env'

4. Verification Failures¶

Error: Security verification failed! Repository contains sensitive files.

Solution: 1. Check the specific files detected 2. Add patterns to git-sync-config.yml 3. Run sync again

5. Push Failures¶

Error: Updates were rejected because the remote contains work

Solution:

# Force push is expected after filtering
git push --force github main

# The sync script handles this automatically

Debug Mode¶

Enable detailed logging:

# Run with bash debug mode
bash -x /srv/dockerdata/_scripts/gitea-github-sync.sh -r REPO_URL

# Check verbose logs
tail -f /srv/dockerdata/gitea/sync-logs/sync-*.log

Best Practices¶

1. Configuration Management¶

Regular Reviews: Audit exclude patterns monthly
Test New Patterns: Use dry-run mode before adding patterns
Document Changes: Comment why patterns were added

2. Security¶

Never Disable Verification: Always let security checks run
Update Patterns: Add new secret patterns as needed
Monitor Logs: Check for verification failures

3. Performance¶

Use git-filter-repo: It's the fastest option
Sync Specific Branches: Don't sync everything
Schedule Wisely: Avoid peak hours for large repos

4. Repository Management¶

Clean History First: Remove large files before enabling sync
Use Git LFS: For large binary files
Archive Old Branches: Reduce sync overhead

5. Monitoring¶

# Set up alerts for sync failures
# Add to monitoring system:
journalctl -u gitea-sync.service --since "1 hour ago" | grep -i error

# Track sync metrics
# - Time taken
# - Files filtered
# - Repository size

6. Disaster Recovery¶

Keep Gitea Primary: It has the complete history
Document Filtered Files: Know what's removed
Test Recovery: Ensure you can restore if needed

Examples¶

Example 1: Sync Single Repository¶

# Simple sync
/srv/dockerdata/_scripts/gitea-github-sync.sh \
  -r https://gitea.rbnk.uk/myorg/myproject.git

Example 2: Dry Run with Verification¶

# Preview what will happen
/srv/dockerdata/_scripts/gitea-github-sync.sh \
  -d -r https://gitea.rbnk.uk/myorg/myproject.git

# Verify security only
/srv/dockerdata/_scripts/gitea-github-sync.sh \
  -v -r https://gitea.rbnk.uk/myorg/myproject.git

Example 3: Custom Configuration¶

# Custom sync for specific project
repositories:
  sensitive-project:
    enabled: true
    branches_to_sync:
      - main  # Only sync main branch
    additional_exclude:
      - "internal-docs/*"
      - "customer-data/*"
      - "*.xlsx"  # Exclude Excel files
    github_name: "public-project"  # Different name on GitHub

Example 4: Emergency Secret Removal¶

If a secret was accidentally committed:

# 1. Stop automated sync
sudo systemctl stop gitea-sync.timer

# 2. Add pattern to config
echo '  - "path/to/secret/file"' >> \
  /srv/dockerdata/gitea/sync-config/git-sync-config.yml

# 3. Run sync manually
/srv/dockerdata/_scripts/gitea-github-sync.sh \
  -r https://gitea.rbnk.uk/myorg/affected-repo.git

# 4. Re-enable automated sync
sudo systemctl start gitea-sync.timer

Conclusion¶

The Gitea to GitHub sync provides a robust solution for maintaining public mirrors of private repositories without compromising security. By using Git history rewriting tools and comprehensive verification, it ensures that sensitive data never reaches public repositories while maintaining the full development history in your private Gitea instance.

Key takeaways: - Always verify sync configuration before enabling - Monitor sync logs for issues - Keep patterns updated as your project evolves - Use dry-run mode when testing changes - Remember that Gitea remains the source of truth