CodeScribe - Code Documentation Generator¶

Overview¶

CodeScribe is a self-hosted code documentation service that scans codebases and generates comprehensive markdown documentation optimized for AI context windows. It provides REST API, MCP (Model Context Protocol) server, and CLI interfaces for flexible integration with development workflows and AI assistants.

Quick Links¶

REST API: https://codescribe.rbnk.uk
Health Check: https://codescribe.rbnk.uk/health
MCP Documentation: codescribe/mcp.md
Source Code: /srv/dockerdata/codescribe/

Features¶

Directory Scanning: Recursive scanning with .gitignore support and custom ignore patterns
Markdown Generation: Clean, syntax-highlighted documentation with tree views
Security Scanning: Detection of secrets, API keys, and sensitive data
Token Counting: Multi-model token estimation (GPT-4o, GPT-4, Claude 3.5)
Remote Repository Processing: Clone and document GitHub/GitLab repositories
AI-Optimized Output: Context-friendly formatting for LLM consumption
Multiple Interfaces: REST API, MCP server, and CLI tool

Architecture¶

graph TD
    A[Claude Code] -->|MCP Protocol| B[CodeScribe MCP Server]
    C[REST Clients] -->|HTTP/HTTPS| D[CodeScribe REST API]
    E[CLI Users] -->|Terminal| F[CodeScribe CLI]

    B --> G[Core Engine]
    D --> G
    F --> G

    subgraph "Core Engine"
        G --> H[Scanner]
        G --> I[Generator]
        G --> J[Security]
        G --> K[Tokens]
        G --> L[Remote]
    end

    subgraph "Data Sources"
        H --> M[Local Directories]
        L --> N[Remote Repositories]
    end

    subgraph "Output"
        I --> O[Markdown Docs]
        J --> P[Security Reports]
        K --> Q[Token Counts]
    end

    style B fill:#f96,stroke:#333,stroke-width:2px
    style D fill:#9f9,stroke:#333,stroke-width:2px
    style F fill:#99f,stroke:#333,stroke-width:2px

Installation¶

CodeScribe is deployed as a Docker container:

# docker-compose.yml
services:
  codescribe:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: codescribe
    restart: unless-stopped
    environment:
      - PORT=3000
      - CORS_ORIGINS=*
      - NODE_ENV=production
    volumes:
      - /srv/dockerdata:/data/dockerdata:ro
      - /home:/data/home:ro
      - codescribe-temp:/tmp/codescribe
    networks:
      - traefik_proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.codescribe.rule=Host(`codescribe.rbnk.uk`)"
      - "traefik.http.services.codescribe.loadbalancer.server.port=3000"

Build and Deploy¶

cd /srv/dockerdata/codescribe
docker compose build
docker compose up -d

REST API¶

Endpoints¶

Endpoint	Method	Description
`/health`	GET	Health check
`/api/status`	GET	API status and capabilities
`/api/scan`	POST	Scan directory structure
`/api/generate`	POST	Generate documentation
`/api/remote`	POST	Process remote repository
`/api/security`	POST	Run security scan
`/api/tokens`	POST	Count tokens in content

Usage Examples¶

Health Check¶

curl https://codescribe.rbnk.uk/health
# {"status":"ok","service":"codescribe","timestamp":"..."}

Scan Directory¶

curl -X POST https://codescribe.rbnk.uk/api/scan \
  -H "Content-Type: application/json" \
  -d '{"path": "/data/dockerdata/codescribe/src"}'

Generate Documentation¶

curl -X POST https://codescribe.rbnk.uk/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "path": "/data/dockerdata/my-project",
    "options": {
      "includeTreeView": true,
      "aiOptimized": true,
      "securityCheck": true
    }
  }'

Process Remote Repository¶

curl -X POST https://codescribe.rbnk.uk/api/remote \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/owner/repo"}'

Count Tokens¶

curl -X POST https://codescribe.rbnk.uk/api/tokens \
  -H "Content-Type: application/json" \
  -d '{"content": "Your code or documentation here..."}'

MCP Server¶

The MCP server enables AI assistants like Claude Code to interact with CodeScribe through natural language.

Configuration¶

Add to .mcp.json:

{
  "mcpServers": {
    "codescribe": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "--network", "traefik_proxy",
        "--volume", "/srv/dockerdata:/data/dockerdata:ro",
        "--volume", "/home:/data/home:ro",
        "codescribe:latest",
        "node", "src/mcp/server.js"
      ]
    }
  }
}

Available Tools¶

Tool	Description
`scan_directory`	Scan directory and list files with structure
`generate_documentation`	Generate markdown documentation from path
`process_remote_repository`	Clone and document remote repository
`security_scan`	Scan files for secrets and sensitive data
`count_tokens`	Count tokens for multiple AI models
`get_repository_info`	Get repository metadata and statistics

For detailed MCP documentation, see codescribe/mcp.md.

CLI Tool¶

The CLI tool provides command-line access to all CodeScribe features.

Commands¶

# Generate documentation
codescribe generate ./my-project -o docs.md

# With AI optimization
codescribe generate ./src --ai -o context.md

# Process remote repository
codescribe remote owner/repo --ai

# Scan directory structure
codescribe scan ./src --format json

# Security scan
codescribe security ./project

# Count tokens in file
codescribe tokens ./large-file.ts

Options¶

Option	Description
`--output, -o`	Output file (default: stdout)
`--format, -f`	Output format: markdown, json
`--no-security`	Skip security scanning
`--no-tree`	Skip directory tree in output
`--ai`	Optimize output for AI consumption
`--ignore`	Additional ignore patterns (comma-separated)

Core Components¶

Scanner (`src/core/scanner.js`)¶

Recursively scans directories with intelligent filtering:

Respects .gitignore patterns
Built-in ignore list for common non-essential files
Tracks file metadata (size, extension, type)
Generates tree structure for visualization

Default Ignored Patterns: - node_modules/, .git/, __pycache__/ - Binary files, images, fonts - Lock files, build artifacts - IDE configurations

Generator (`src/core/generator.js`)¶

Creates well-formatted markdown documentation:

Syntax highlighting based on file extension
Tree view of directory structure
Metadata headers with statistics
AI-optimized formatting with clear delimiters

Security Scanner (`src/core/security.js`)¶

Detects sensitive information:

Pattern	Detection
AWS Keys	`AKIA[0-9A-Z]{16}`
GitHub Tokens	`ghp_`, `gho_`, `github_pat_`
Anthropic Keys	`sk-ant-`
OpenAI Keys	`sk-[a-zA-Z0-9]{48}`
Private Keys	`-----BEGIN.*PRIVATE KEY-----`
Generic Secrets	High-entropy strings in sensitive contexts

Token Counter (`src/core/tokens.js`)¶

Accurate token counting using js-tiktoken:

Model	Encoding	Cost/1K Input	Cost/1K Output
GPT-4o	o200k_base	$0.0025	$0.01
GPT-4	cl100k_base	$0.03	$0.06
Claude 3.5	estimate	$0.003	$0.015

Remote Processor (`src/core/remote.js`)¶

Handles remote repository operations:

Supports GitHub, GitLab, Bitbucket URLs
Supports shorthand format (owner/repo)
Clones to temporary directory
Automatic cleanup after processing

Integration Points¶

With Claude Code¶

CodeScribe integrates seamlessly as an MCP server:

User: "Generate documentation for the n8n service"
Claude: [Uses codescribe MCP to scan and generate docs]

With n8n Workflows¶

Trigger documentation generation via webhook:

// n8n HTTP Request node
{
  "method": "POST",
  "url": "https://codescribe.rbnk.uk/api/generate",
  "body": {
    "path": "/data/dockerdata/{{ $json.service }}",
    "options": { "aiOptimized": true }
  }
}

With CI/CD Pipelines¶

Generate docs on code changes:

# GitHub Actions example
- name: Generate Documentation
  run: |
    curl -X POST https://codescribe.rbnk.uk/api/remote \
      -H "Content-Type: application/json" \
      -d '{"url": "${{ github.repository }}"}'

Security Considerations¶

Read-Only Mounts: Host directories mounted as read-only (:ro)
Non-Root User: Container runs as codescribe user (UID 1001)
Secret Detection: Built-in scanning prevents accidental secret exposure
Network Isolation: Only accessible via Traefik proxy
No Persistent State: Stateless service with temporary clone directory

Configuration¶

Environment Variables¶

Variable	Description	Default
`PORT`	API server port	`3000`
`CORS_ORIGINS`	Allowed CORS origins	`*`
`NODE_ENV`	Runtime environment	`production`

Volume Mounts¶

Host Path	Container Path	Mode	Purpose
`/srv/dockerdata`	`/data/dockerdata`	ro	Infrastructure scanning
`/home`	`/data/home`	ro	Home directory scanning
`codescribe-temp`	`/tmp/codescribe`	rw	Temporary clone storage

Monitoring¶

Health Check¶

# Docker health check
docker inspect codescribe --format='{{.State.Health.Status}}'

# Manual health check
curl https://codescribe.rbnk.uk/health

Container Logs¶

docker compose -f /srv/dockerdata/codescribe/docker-compose.yml logs -f

Metrics¶

Monitor through standard Docker metrics: - Container CPU/memory usage - Network I/O - Health check status

Troubleshooting¶

Common Issues¶

404 on Endpoints
Verify Traefik labels in docker-compose.yml
Check cert resolver is cf (not cloudflare)
Ensure middleware is security-headers@file
Permission Denied in Container
Rebuild image to ensure proper chown in Dockerfile
Verify source files copied before changing ownership
Remote Repository Timeout
Large repositories may take time to clone
Consider using sparse checkout for huge repos
Security Scan False Positives
Use --no-security flag for known-safe content
Review detected patterns in security report

Debug Mode¶

# Run with verbose logging
docker run --rm -it \
  -v /srv/dockerdata:/data/dockerdata:ro \
  codescribe:latest \
  node src/index.js

Best Practices¶

Use AI-Optimized Output: The --ai flag or aiOptimized option formats documentation specifically for LLM context windows
Review Security Scans: Always review security reports before sharing generated documentation
Respect Token Limits: Use token counting to ensure documentation fits within model context limits
Cache Results: Store generated documentation rather than regenerating frequently
Incremental Updates: For large codebases, consider scanning specific subdirectories

Technical Details¶

Runtime: Node.js 20 (Alpine)
Framework: Express.js for REST API
MCP SDK: @modelcontextprotocol/sdk
Token Counting: js-tiktoken with official OpenAI encodings
Git Operations: simple-git library
Pattern Matching: ignore package for gitignore support

Future Enhancements¶

Incremental Scanning: Track changes and update only modified files
Custom Templates: User-defined documentation templates
Multi-Language Support: Better detection of language-specific patterns
Streaming Output: Real-time streaming for large repositories
Webhook Notifications: Callback on completion for async processing

CodeScribe - Code Documentation Generator¶

Overview¶

Quick Links¶

Features¶

Architecture¶

Installation¶

Build and Deploy¶

REST API¶

Endpoints¶

Usage Examples¶

Health Check¶

Scan Directory¶

Generate Documentation¶

Process Remote Repository¶

Count Tokens¶

MCP Server¶

Configuration¶

Available Tools¶

CLI Tool¶

Commands¶

Options¶

Core Components¶

Scanner (src/core/scanner.js)¶

Generator (src/core/generator.js)¶

Security Scanner (src/core/security.js)¶

Token Counter (src/core/tokens.js)¶

Remote Processor (src/core/remote.js)¶

Integration Points¶

With Claude Code¶

With n8n Workflows¶

With CI/CD Pipelines¶

Security Considerations¶

Configuration¶

Environment Variables¶

Volume Mounts¶

Monitoring¶

Health Check¶

Container Logs¶

Metrics¶

Troubleshooting¶

Common Issues¶

Debug Mode¶

Best Practices¶

Technical Details¶

Future Enhancements¶

Scanner (`src/core/scanner.js`)¶

Generator (`src/core/generator.js`)¶

Security Scanner (`src/core/security.js`)¶

Token Counter (`src/core/tokens.js`)¶

Remote Processor (`src/core/remote.js`)¶