Git Workflow for Dual Remote Setup¶

Table of Contents¶

Overview
Initial Setup
Daily Development Workflow
Branch Management Strategies
Using Sync Scripts
Security Best Practices
Troubleshooting Common Issues
Advanced Workflows

Overview¶

This guide explains how to work effectively with our dual-remote Git setup, where Gitea serves as the primary repository and GitHub acts as an optional public mirror.

Architecture Overview¶

┌─────────────┐     Push/Pull      ┌──────────────┐
│  Developer  │ ←────────────────→ │    Gitea     │
│   Machine   │                    │ (Primary)    │
└─────────────┘                    └──────┬───────┘
                                          │ Auto-sync
                                          │ (hourly +
                                          │  on push)
                                          ↓
                                   ┌──────────────┐
                                   │    GitHub    │
                                   │  (Mirror)    │
                                   └──────────────┘

Benefits of Dual Remote Setup¶

Data Sovereignty: Primary code stays on self-hosted infrastructure
Public Collaboration: Optional GitHub mirror for open-source projects
Redundancy: Multiple locations for code backup
Flexibility: Choose where to push based on project needs
Security: Sensitive projects stay private on Gitea

Initial Setup¶

Step 1: SSH Key Configuration¶

First, ensure you have SSH keys set up for both Gitea and GitHub:

# Generate SSH key if needed
ssh-keygen -t ed25519 -C "[email protected]"

# Add to SSH agent
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

# Test Gitea connection
ssh -T [email protected]

# Test GitHub connection
ssh -T [email protected]

Step 2: Configure SSH for Multiple Hosts¶

Create or edit ~/.ssh/config:

# Gitea
Host gitea.rbnk.uk
    HostName gitea.rbnk.uk
    User git
    IdentityFile ~/.ssh/id_ed25519
    Port 22

# GitHub
Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519
    Port 22

Step 3: Clone and Configure Repository¶

Option A: Using Setup Script (Recommended)¶

# Clone from Gitea
git clone [email protected]:username/project.git
cd project

# Run setup script
curl -sSL https://gitea.rbnk.uk/admin/scripts/raw/branch/main/setup-git-remotes.sh | bash

Option B: Manual Configuration¶

# Clone from Gitea
git clone [email protected]:username/project.git
cd project

# GitHub remote should already be configured
# If not, add it manually:
git remote add github [email protected]:username/project.git

# Verify remotes (standard naming as of 2025-07-27)
git remote -v
# origin = Gitea (primary)
# github = GitHub (auto-synced mirror)

Step 4: Configure Git Aliases¶

Add these helpful aliases to your .gitconfig:

git config --global alias.push-all '!git push origin && git push github'
git config --global alias.sync-github '!git fetch github && git merge github/$(git branch --show-current) && git push origin'
git config --global alias.sync-gitea '!git fetch origin && git merge origin/$(git branch --show-current) && git push github'
git config --global alias.remotes 'remote -v'
git config --global alias.branches 'branch -avv'

Daily Development Workflow¶

Standard Development Flow¶

Start with a fresh branch:

# Ensure you're up to date
git fetch origin
git checkout main
git pull origin main

# Create feature branch
git checkout -b feature/awesome-feature

Make your changes:

# Edit files
vim src/main.py

# Stage changes
git add src/main.py

# Commit with descriptive message
git commit -m "feat: add user authentication module

- Implement JWT-based auth
- Add login/logout endpoints
- Include rate limiting

Closes #42"

Push to Gitea (primary):

git push -u origin feature/awesome-feature

Create Pull Request:
Navigate to https://gitea.rbnk.uk/username/project
Click "New Pull Request"
Select your feature branch
Add reviewers and description
Automatic GitHub Sync:
When PR is merged to main: GitHub sync triggers automatically
Push notification appears showing sync status
Manual sync if needed: sudo systemctl start gitea-sync.service

Commit Message Format¶

Follow conventional commits for consistency:

<type>(<scope>): <subject>

<body>

<footer>

Types: - feat: New feature - fix: Bug fix - docs: Documentation only - style: Code style (formatting, semicolons, etc) - refactor: Code change that neither fixes a bug nor adds a feature - perf: Performance improvement - test: Adding or modifying tests - build: Changes to build system or dependencies - ci: CI configuration changes - chore: Other changes that don't modify src or test files - revert: Reverts a previous commit

Examples:

# Simple commit
git commit -m "fix: correct user validation logic"

# Detailed commit
git commit -m "feat(auth): add OAuth2 provider support

- Add Google OAuth2 integration
- Add GitHub OAuth2 integration  
- Update user model for external auth
- Add configuration for OAuth providers

BREAKING CHANGE: auth config structure changed
Closes #123, #124"

Branch Management Strategies¶

Git Flow Strategy¶

For larger projects with releases:

main ────────────────────────────────→ (stable releases)
  ↑                                ↑
  └── release/1.2 ────────────────┘
       ↑
develop ─────────────────────────────→ (integration branch)
  ↑    ↑                        ↑
  │    └── feature/oauth ───────┘
  └────── feature/api-v2 ───────┘

Workflow:

# Create feature from develop
git checkout develop
git checkout -b feature/oauth

# Work and commit
git add .
git commit -m "feat: implement OAuth"

# Merge back to develop
git checkout develop
git merge feature/oauth
git push origin develop

# Create release
git checkout -b release/1.2 develop
# ... testing and fixes ...
git checkout main
git merge release/1.2
git tag -a v1.2.0 -m "Release version 1.2.0"
git push origin main --tags

GitHub Flow (Simplified)¶

For simpler projects:

main ────────────────────────────────→
  ↑         ↑              ↑
  └── fix/─┘               │
  └── feature/─────────────┘

Workflow:

# Create feature from main
git checkout main
git checkout -b feature/new-api

# Work, commit, and push
git push -u origin feature/new-api

# Create PR and merge via UI
# Then update local main
git checkout main
git pull origin main

Environment Branches¶

For deployment workflows:

production ──────────────────────────→ (live site)
  ↑
staging ─────────────────────────────→ (testing)
  ↑
main ────────────────────────────────→ (development)

Using Sync Scripts¶

Overview of Sync Process¶

The Gitea to GitHub sync uses advanced Git history rewriting to create a sanitized mirror of your repository. This ensures that sensitive data never reaches GitHub, not even in the Git history.

Key Features:¶

Complete History Filtering: Removes secrets from ALL commits, not just the latest
Multiple Filter Tools: Automatically selects the best available tool (git-filter-repo, BFG, or git filter-branch)
Security Verification: Built-in scanning to ensure no secrets remain
Selective Syncing: Choose which repositories and branches to sync
Audit Trail: Comprehensive logging of all sync operations

Manual Repository Sync¶

The sync script handles secure mirroring between Gitea and GitHub:

# Sync specific repository (removes secrets from entire history)
/srv/dockerdata/_scripts/gitea-github-sync.sh -r https://gitea.rbnk.uk/user/repo.git

# Dry run to preview changes (shows what will be filtered)
/srv/dockerdata/_scripts/gitea-github-sync.sh -d -r https://gitea.rbnk.uk/user/repo.git

# Verify security only (runs secret detection without syncing)
/srv/dockerdata/_scripts/gitea-github-sync.sh -v -r https://gitea.rbnk.uk/user/repo.git

What Happens During Sync:¶

Full Clone: Creates a complete copy of the repository from Gitea
History Rewriting: Removes all instances of sensitive files from every commit
Security Scan: Verifies no secrets remain using industry-standard patterns
Filtered Push: Pushes the cleaned repository to GitHub

Understanding History Filtering¶

The sync process uses one of three tools to rewrite Git history:

1. git-filter-repo (Preferred)¶

# Install if needed
pip3 install git-filter-repo

# How it works internally
git filter-repo --paths-from-file patterns.txt --invert-paths --force

2. BFG Repo-Cleaner¶

# Download if needed
wget https://repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar

# How it works internally
java -jar bfg.jar --delete-files '*.env' --no-blob-protection

3. git filter-branch (Fallback)¶

# Native Git tool (deprecated but functional)
git filter-branch --index-filter 'git rm --cached --ignore-unmatch *.env' --all

Automated Sync Schedule¶

Current sync configuration (as of 2025-07-27): - Hourly sync: Systemd timer runs every hour at :00 - Push notifications: main/master/production branches show sync status - Manual trigger: sudo systemctl start gitea-sync.service - Security filtering: Always applied, regardless of trigger method

Check sync status:

# View timer schedule
systemctl list-timers gitea-sync.timer

# Check last sync results
journalctl -u gitea-sync.service -n 50

# View detailed sync logs
tail -f /srv/dockerdata/gitea/sync-logs/sync-*.log

# Monitor sync in real-time
journalctl -u gitea-sync.service -f

Sync Configuration¶

Edit /srv/dockerdata/gitea/sync-config/git-sync-config.yml:

# GitHub organization
organization: your-org

# Repositories to sync
repositories:
  - source: https://gitea.rbnk.uk/team/project1.git
    target: project1
    sync_branches: [main, develop]

  - source: https://gitea.rbnk.uk/team/project2.git  
    target: project2
    sync_branches: [main]
    create_target: true  # Auto-create GitHub repo

# Security filters (removed from entire history)
global_exclude_patterns:
  # Environment files
  - ".env"
  - "*.env"
  - "**/.env*"
  - ".envrc"

  # Private keys
  - "*.pem"
  - "*.key"
  - "*.pfx"
  - "id_rsa*"
  - "id_dsa*"

  # Credentials and secrets
  - "**/secrets/*"
  - "**/credentials/*"
  - "*password*"
  - "*token*"
  - "*secret*"

  # Database dumps
  - "*.sql"
  - "*.dump"
  - "*.bak"

  # Cloud credentials
  - ".aws/*"
  - "gcp-credentials.json"
  - "azure-credentials.json"

# Additional patterns for specific repos
exclude_patterns:
  - "internal/*"
  - "proprietary/*"

Security Verification¶

The sync process includes comprehensive security verification using patterns from: - Gitleaks: Industry-standard secret scanner - TruffleHog: Deep secret detection - Custom patterns: Specific to your infrastructure

Verification checks for: - Environment files (.env, *.env) - Private keys (*.pem, *.key, id_rsa*) - API keys (AWS, GitHub, Stripe, etc.) - Cloud credentials - Database dumps - Password patterns in code

Troubleshooting Sync Issues¶

Common Problems and Solutions:¶

Sync Timeout on Large Repositories

# Increase Git buffer size
git config --global http.postBuffer 524288000

# Use packfile limits
git config --global pack.windowMemory "100m"
git config --global pack.packSizeLimit "100m"

BFG Memory Issues

# Run with more memory
java -Xmx4g -jar /usr/local/bin/bfg.jar ...

Verification Failures

# Check what's being detected
/srv/dockerdata/_scripts/verify-no-secrets.sh /path/to/repo

# Update exclude patterns
vim /srv/dockerdata/gitea/sync-config/git-sync-config.yml

Filter Tool Not Found

# Install git-filter-repo
pip3 install git-filter-repo

# Or download BFG
wget https://repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar
sudo mv bfg-1.14.0.jar /usr/local/bin/bfg.jar

Sync Webhook Integration¶

Configure Gitea webhook to trigger sync on push:

Go to repository settings in Gitea
Add webhook:
URL: http://localhost:8080/sync-webhook
Secret: Generate secure secret
Events: Push events

Create webhook handler:

#!/bin/bash
# /srv/dockerdata/gitea/hooks/sync-webhook.sh

REPO_URL="$1"
/srv/dockerdata/_scripts/gitea-github-sync.sh -r "$REPO_URL"

Security Best Practices¶

Understanding Our Dual Repository Security Model¶

Our infrastructure uses a sophisticated dual-repository approach that provides both operational flexibility and public security:

Gitea (Primary/Internal)
Maintains complete history including .env files for operational needs
Allows tracking configuration changes and rollbacks
Accessible only via VPN or authenticated access
Essential for debugging and maintaining infrastructure
GitHub (Public Mirror)
Automatically filtered to remove ALL sensitive data from entire history
Uses industry-standard secret detection (Gitleaks, TruffleHog, detect-secrets patterns)
Safe for public collaboration and open-source contributions
Force pushes are normal and expected due to history rewriting

This approach ensures that: - Internal teams have full visibility for operations - Public code is completely clean and secure - Secrets are never exposed, even in Git history - Configuration changes are still tracked (internally)

1. Never Commit Secrets to GitHub-Bound Repositories¶

Use .gitignore:

# Environment files
.env
.env.*
!.env.example

# Credentials
**/credentials/
**/secrets/
*.pem
*.key
*private*

# Local configs
config.local.*
settings.local.*

Pre-commit Hook:

#!/bin/bash
# .git/hooks/pre-commit

# Check for common secret patterns
if git diff --cached --name-only | xargs grep -E "(password|secret|key|token)\s*=\s*[\"'][^\"']+[\"']" 2>/dev/null; then
    echo "ERROR: Possible secrets detected in commit"
    echo "Please remove sensitive data before committing"
    exit 1
fi

2. Use Environment Variables¶

Good:

import os
API_KEY = os.environ.get('API_KEY')

Bad:

API_KEY = 'sk-1234567890abcdef'  # Never do this!

3. Secure Branch Protection¶

Configure in Gitea repository settings: - Protect main branch - Require pull request reviews - Dismiss stale reviews - Require status checks - Include administrators in restrictions

4. Sign Your Commits¶

# Configure GPG signing
git config --global user.signingkey YOUR_GPG_KEY_ID
git config --global commit.gpgsign true

# Sign a specific commit
git commit -S -m "feat: add secure feature"

# Verify signatures
git log --show-signature

5. Audit Access Regularly¶

# Check SSH keys in Gitea
curl -H "Authorization: token YOUR_TOKEN" \
  https://gitea.rbnk.uk/api/v1/user/keys

# Review repository collaborators
curl -H "Authorization: token YOUR_TOKEN" \
  https://gitea.rbnk.uk/api/v1/repos/owner/repo/collaborators

Troubleshooting Common Issues¶

Issue 1: Push Rejected - Non-Fast-Forward¶

Problem: Your branch is behind the remote

! [rejected]        main -> main (non-fast-forward)

Solution:

# Fetch and merge
git fetch origin
git merge origin/main

# Or rebase for cleaner history
git pull --rebase origin main

# Force push only if you're sure (feature branches only!)
git push --force-with-lease origin feature/my-branch

Issue 2: SSH Permission Denied¶

Problem: Cannot authenticate with SSH

Permission denied (publickey).

Solution:

# Check SSH agent
ssh-add -l

# Add key if missing
ssh-add ~/.ssh/id_ed25519

# Test connection
ssh -vT [email protected]

# Check SSH config
cat ~/.ssh/config

Issue 3: Merge Conflicts¶

Problem: Conflicting changes between branches

Solution:

# Start merge
git merge origin/main

# See conflicted files
git status

# Edit conflicts manually
vim conflicted-file.py

# Mark as resolved
git add conflicted-file.py

# Complete merge
git commit -m "merge: resolve conflicts with main"

Issue 4: Large File Rejection¶

Problem: Pushing large files fails

remote: error: File large.bin is 104.32 MB; this exceeds GitHub's file size limit of 100.00 MB

Solution:

# Use Git LFS
git lfs track "*.bin"
git add .gitattributes
git add large.bin
git commit -m "add: large binary with LFS"
git push

Issue 5: Sync Script Failures¶

Problem: Sync script reports errors

Debug Steps:

# Check sync logs
tail -f /srv/dockerdata/gitea/sync-logs/sync-*.log

# Run in debug mode
bash -x /srv/dockerdata/_scripts/gitea-github-sync.sh -r REPO_URL

# Verify configuration
cat /srv/dockerdata/gitea/sync-config/git-sync-config.yml

Advanced Workflows¶

Cherry-Pick Between Remotes¶

When you need specific commits from GitHub:

# Add commit from GitHub to Gitea
git fetch github
git cherry-pick abc123def
git push origin main

Maintaining Forks¶

Keep your fork synchronized:

# Add upstream remote
git remote add upstream [email protected]:original/project.git

# Sync fork
git fetch upstream
git checkout main
git merge upstream/main
git push origin main

Interactive Rebase for Clean History¶

Before pushing to public:

# Rebase last 5 commits
git rebase -i HEAD~5

# In editor, you can:
# - pick: use commit
# - reword: change message
# - squash: combine with previous
# - drop: remove commit

# Force push to feature branch
git push --force-with-lease origin feature/clean-history

Bisect for Bug Finding¶

Find when a bug was introduced:

# Start bisect
git bisect start
git bisect bad                    # Current commit is bad
git bisect good v1.0              # v1.0 was good

# Git will checkout commits to test
# After testing each:
git bisect good  # or
git bisect bad

# When done
git bisect reset

Stash Management¶

Save work temporarily:

# Save current changes
git stash save "WIP: feature implementation"

# List stashes
git stash list

# Apply latest stash
git stash pop

# Apply specific stash
git stash apply stash@{2}

# Create branch from stash
git stash branch feature/from-stash stash@{1}

Submodules for Dependencies¶

Manage external dependencies:

# Add submodule
git submodule add https://github.com/lib/library.git libs/library

# Clone with submodules
git clone --recursive [email protected]:user/project.git

# Update submodules
git submodule update --remote --merge

Quick Reference Card¶

Essential Commands¶

Action	Command
Clone from Gitea	`git clone [email protected]:user/repo.git`
Add GitHub remote	`git remote add github [email protected]:user/repo.git`
Push to all remotes	`git push-all`
Sync from GitHub	`git sync-github`
Sync to GitHub	`git sync-gitea`
Create feature branch	`git checkout -b feature/name`
Update branch from main	`git rebase main`
Interactive rebase	`git rebase -i HEAD~n`
Cherry-pick commit	`git cherry-pick SHA`
Stash changes	`git stash save "message"`
Sign commit	`git commit -S -m "message"`
Amend last commit	`git commit --amend`
Reset to remote	`git reset --hard origin/main`
Clean untracked files	`git clean -fd`

Configuration Files¶

File	Purpose
`.gitignore`	Exclude files from Git
`.gitattributes`	File handling rules
`.gitmessage`	Commit message template
`.gitconfig`	User configuration
`.git/hooks/`	Git hooks directory

Useful Aliases¶

# Add to ~/.gitconfig
[alias]
    # Shortcuts
    co = checkout
    br = branch
    ci = commit
    st = status

    # Useful commands
    last = log -1 HEAD
    unstage = reset HEAD --
    visual = !gitk

    # Workflow helpers
    push-all = !git push origin && git push github
    sync-github = !git fetch github && git merge github/$(git branch --show-current) && git push origin
    sync-gitea = !git fetch origin && git merge origin/$(git branch --show-current) && git push github

    # History viewing
    hist = log --pretty=format:'%h %ad | %s%d [%an]' --graph --date=short
    today = log --since=midnight --author='$(git config user.name)' --oneline

    # Branch management  
    branches = branch -avv
    remotes = remote -v
    cleanup = !git branch --merged | grep -v '\\*\\|main\\|develop' | xargs -n 1 git branch -d

Repository Size Management¶

Preventing Large Files in Git¶

Large files can bloat your repository and slow down operations. Here's how to prevent and handle them:

Pre-commit Size Checks¶

Add this pre-commit hook to prevent large files:

#!/bin/bash
# Save as .git/hooks/pre-commit and make executable

MAX_FILE_SIZE=10485760  # 10MB in bytes

# Check staged files
for file in $(git diff --cached --name-only); do
    if [ -f "$file" ]; then
        size=$(stat -c%s "$file" 2>/dev/null || stat -f%z "$file" 2>/dev/null)
        if [ "$size" -gt "$MAX_FILE_SIZE" ]; then
            echo "Error: $file is larger than 10MB ($(numfmt --to=iec $size))"
            echo "Consider using Git LFS or excluding this file"
            exit 1
        fi
    fi
done

# Warn about potential problem files
git diff --cached --name-only | grep -E '\.(db|sqlite|log|dump|cache)$' | while read file; do
    echo "Warning: $file might grow large over time. Ensure it's in .gitignore"
done

Repository Cleanup with git-filter-repo¶

If large files were accidentally committed, use git-filter-repo to clean history:

# Install the tool
pip3 install git-filter-repo

# Analyze repository for large files
git filter-repo --analyze
cat .git/filter-repo/analysis/path-sizes.txt | head -20

# Remove specific paths from all history
git filter-repo --path path/to/large/file --invert-paths --force

# Remove files by pattern
git filter-repo --path-glob '*.db' --invert-paths --force

# After cleanup, force push to remotes
git remote add origin <your-remote-url>
git push --force origin main

Common Files to Exclude¶

Always add these to .gitignore:

# Database files
*.db
*.sqlite
*.sql
*.dump

# Logs and caches
*.log
logs/
cache/
tmp/
temp/

# Model and data files
*.model
*.pkl
*.h5
*.weights
embeddings/
models/

# Media files (unless needed)
*.mp4
*.mov
*.avi
*.zip
*.tar.gz

# IDE and OS files
.DS_Store
Thumbs.db
.idea/
.vscode/
*.swp

Using Git LFS for Large Files¶

When large files are necessary:

# Install Git LFS
git lfs install

# Track large file types
git lfs track "*.psd"
git lfs track "*.model"
git lfs track "models/**"

# Add the .gitattributes file
git add .gitattributes
git commit -m "Add Git LFS tracking"

# Add and commit large files normally
git add large-file.psd
git commit -m "Add design file via LFS"

Conclusion¶

The dual-remote Git workflow provides flexibility and security for your development process. Key takeaways:

Always push to Gitea first - It's your primary source of truth
Use GitHub for public collaboration - When you want community involvement
Automate with sync scripts - Reduce manual work and errors
Follow security practices - Never commit secrets or sensitive data
Maintain clean history - Use appropriate branching strategies
Monitor repository size - Prevent large files from bloating the repository

With these practices, you can efficiently manage code across both private and public repositories while maintaining security and collaboration flexibility.