Review Bot Automator

Universal AI-powered automation for GitHub code review bots
Intelligent suggestion application and conflict resolution for CodeRabbit, GitHub Copilot, and custom review bots

📋 Table of Contents

Problem Statement
Quick Start
Features
Architecture
Use Cases
Environment Variables
Documentation
Contributing
Project Status
License

🎯 Problem Statement

When multiple PR review comments suggest overlapping changes to the same file, traditional automation tools either:

Skip all conflicting changes (losing valuable suggestions)
Apply changes sequentially without conflict awareness (potentially breaking code)
Require tedious manual resolution for every conflict

Review Bot Automator provides intelligent, semantic-aware conflict resolution that:

✅ Understands code structure (JSON, YAML, TOML, Python, TypeScript)
✅ Uses priority-based resolution (user selections, security fixes, syntax errors)
✅ Supports semantic merging (combining non-conflicting changes automatically)
✅ Learns from your decisions to improve over time
✅ Provides detailed conflict analysis and actionable suggestions

🚀 Quick Start

Installation

pip install pr-conflict-resolver

Basic Usage

# Set your GitHub token (required)
export GITHUB_PERSONAL_ACCESS_TOKEN="your_token_here"

# Analyze conflicts in a PR
pr-resolve analyze --owner VirtualAgentics --repo my-repo --pr 123

# Apply suggestions with conflict resolution
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 --strategy priority

# Apply only conflicting changes
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 --mode conflicts-only

# Simulate without applying changes (dry-run mode)
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 --mode dry-run

# Use parallel processing for large PRs
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 --parallel --max-workers 8

# Load configuration from file
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 --config config.yaml

LLM Provider Setup (Optional)

Enable AI-powered features with your choice of LLM provider using zero-config presets:

# ✨ NEW: Zero-config presets for instant setup

# Option 1: Codex CLI (free with GitHub Copilot subscription)
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 \
  --llm-preset codex-cli-free

# Option 2: Local Ollama (free, private, offline) - EASIEST SETUP
./scripts/setup_ollama.sh          # One-time install
./scripts/download_ollama_models.sh  # Download model
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 \
  --llm-preset ollama-local
# See docs/ollama-setup.md for detailed guide

# Option 3: Claude CLI (requires Claude subscription)
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 \
  --llm-preset claude-cli-sonnet

# Option 4: OpenAI API (pay-per-use, ~$0.01 per PR)
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 \
  --llm-preset openai-api-mini \
  --llm-api-key sk-...

# Option 5: Anthropic API (balanced cost/performance)
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123 \
  --llm-preset anthropic-api-balanced \
  --llm-api-key sk-ant-...

Available presets: codex-cli-free, ollama-local, claude-cli-sonnet, openai-api-mini, anthropic-api-balanced

Alternative: Use environment variables

# Anthropic (recommended - 50-90% cost savings with caching)
export CR_LLM_ENABLED="true"
export CR_LLM_PROVIDER="anthropic"
export CR_LLM_API_KEY="sk-ant-..."  # Get from https://console.anthropic.com/

# OpenAI
export CR_LLM_ENABLED="true"
export CR_LLM_PROVIDER="openai"
export CR_LLM_API_KEY="sk-..."  # Get from https://platform.openai.com/api-keys

# Then use as normal
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123

See LLM Configuration Guide for all provider options and detailed setup.

Python API

from pr_conflict_resolver import ConflictResolver
from pr_conflict_resolver.config import PresetConfig

resolver = ConflictResolver(config=PresetConfig.BALANCED)
results = resolver.resolve_pr_conflicts(
    owner="VirtualAgentics",
    repo="my-repo",
    pr_number=123
)

print(f"Applied: {results.applied_count}")
print(f"Conflicts: {results.conflict_count}")
print(f"Success rate: {results.success_rate}%")

🎨 Features

Intelligent Conflict Analysis

Semantic Understanding: Analyzes JSON, YAML, TOML structure, not just text
Conflict Categorization: Exact, major, partial, minor, disjoint-keys, semantic-duplicate
Impact Assessment: Evaluates scope, risk level, and criticality of changes
Actionable Suggestions: Provides specific guidance for each conflict

Smart Resolution Strategies

Priority-Based: User selections > Security fixes > Syntax errors > Regular suggestions
Semantic Merging: Combines non-conflicting changes in structured files
Sequential Application: Applies compatible changes in optimal order
Defer to User: Escalates complex conflicts for manual review

File-Type Handlers

JSON: Duplicate key detection, key-level merging
YAML: Comment preservation, structure-aware merging
TOML: Section merging, format preservation
Python/TypeScript: AST-aware analysis (planned)

Multi-Provider LLM Support

5 Provider Types: OpenAI, Anthropic, Claude CLI, Codex CLI, Ollama
Cost Optimization: Prompt caching reduces Anthropic costs by 50-90%
Flexible Deployment: API-based, CLI-based, or local inference
Provider Selection: Choose based on cost, privacy, or performance needs
Health Checks: Automatic provider validation before use

Learning & Optimization

ML-Assisted Priority: Learns from your resolution decisions
Metrics Tracking: Monitors success rates, resolution times, strategy effectiveness
Conflict Caching: Reuses analysis for similar conflicts
Performance: Parallel processing for large PRs

Configuration & Presets

Conservative: Skip all conflicts, manual review required
Balanced: Priority system + semantic merging (default)
Aggressive: Maximize automation, user selections always win
Semantic: Focus on structure-aware merging for config files

Application Modes

all: Apply both conflicting and non-conflicting changes (default)
conflicts-only: Apply only changes that have conflicts
non-conflicts-only: Apply only changes without conflicts
dry-run: Analyze and report without applying any changes

Rollback & Safety Features

Automatic Rollback: Git-based checkpointing with automatic rollback on failure
Pre-Application Validation: Validates changes before applying (optional)
File Integrity Checks: Verifies file safety and containment
Detailed Logging: Comprehensive logging for debugging and audit trails

Runtime Configuration

Configure via multiple sources with precedence chain: CLI flags > Environment variables > Config file > Defaults

Configuration Files: Load settings from YAML or TOML files
Environment Variables: Set options using CR_* prefix variables
CLI Overrides: Override any setting via command-line flags

See .env.example for available environment variables.

📖 Documentation

User Guides

Getting Started Guide - Installation, setup, and first steps
Configuration Reference - Complete configuration options
LLM Configuration Guide - LLM providers, presets, and advanced configuration
Ollama Setup Guide - Comprehensive Ollama installation and setup
Rollback System - Automatic rollback and recovery
Parallel Processing - Performance tuning guide
Migration Guide - Upgrading from earlier versions
Troubleshooting - Common issues and solutions

Reference Documentation

API Reference - Python API documentation
Conflict Types Explained - Understanding conflict categories
Resolution Strategies - Strategy selection guide

Architecture & Development

Architecture Overview - System design and components
Contributing Guide - How to contribute

Security

Security Policy - Vulnerability reporting, security features
Security Architecture - Design principles, threat model
Threat Model - STRIDE analysis, risk assessment
Incident Response - Security incident procedures
Compliance - GDPR, OWASP, SOC2, OpenSSF
Security Testing - Testing guide, fuzzing, SAST

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                    GitHub PR Comments                       │
│                   (CodeRabbit, Review Bot)                  │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│              Comment Parser & Extractor                     │
│   (Suggestions, Diffs, Codemods, Multi-Options)            │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│              Conflict Detection Engine                      │
│  • Fingerprinting  • Overlap Analysis  • Semantic Check    │
└────────────────────┬────────────────────────────────────────┘
                     │
         ┌───────────┴──────────┐
         ▼                      ▼
┌──────────────────┐   ┌──────────────────┐
│  File Handlers   │   │  Priority System │
│  • JSON          │   │  • User Selected │
│  • YAML          │   │  • Security Fix  │
│  • TOML          │   │  • Syntax Error  │
│  • Python        │   │  • Regular       │
└─────────┬────────┘   └────────┬─────────┘
          │                     │
          └──────────┬──────────┘
                     ▼
┌─────────────────────────────────────────────────────────────┐
│           Resolution Strategy Selector                      │
│  • Skip  • Override  • Merge  • Sequential  • Defer        │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│              Application Engine                             │
│  • Backup  • Apply  • Validate  • Rollback                 │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────────────────────────┐
│        Reporting & Metrics                                  │
│  • Conflict Summary  • Visual Diff  • Success Rate         │
└─────────────────────────────────────────────────────────────┘

🔧 Use Cases

1. CodeRabbit Multi-Option Selections

Problem: User selects "Option 2" but it conflicts with another suggestion Solution: Priority system ensures user selections override lower-priority changes

2. Overlapping Configuration Changes

Problem: Two suggestions modify different keys in package.json Solution: Semantic merging combines both changes automatically

3. Security Fix vs. Formatting

Problem: Security fix conflicts with formatting suggestion Solution: Priority system applies security fix, skips formatting

4. Large PR with 50+ Comments

Problem: Manual conflict resolution is time-consuming Solution: Parallel processing + caching resolves conflicts in seconds

🔧 Environment Variables

Configure the tool using environment variables (see .env.example for all options):

Variable	Description	Default
`GITHUB_PERSONAL_ACCESS_TOKEN`	GitHub API token (required)	None
`CR_MODE`	Application mode (`all`, `conflicts-only`, `non-conflicts-only`, `dry-run`)	`all`
`CR_ENABLE_ROLLBACK`	Enable automatic rollback on failure	`true`
`CR_VALIDATE`	Enable pre-application validation	`true`
`CR_PARALLEL`	Enable parallel processing	`false`
`CR_MAX_WORKERS`	Number of parallel workers	`4`
`CR_LOG_LEVEL`	Logging level (`DEBUG`, `INFO`, `WARNING`, `ERROR`)	`INFO`
`CR_LOG_FILE`	Log file path (optional)	None

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Development Setup

git clone https://github.com/VirtualAgentics/review-bot-automator.git
cd review-bot-automator
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pre-commit install

Running Tests

This project uses pytest 9.0 with native subtests support for comprehensive testing. We maintain >80% test coverage with 1318+ tests including unit, integration, security, and property-based fuzzing tests.

# Run standard tests with coverage
pytest tests/ --cov=src --cov-report=html

# Run property-based fuzzing tests
make test-fuzz              # Dev profile: 50 examples
make test-fuzz-ci           # CI profile: 100 examples
make test-fuzz-extended     # Extended: 1000 examples

# Run all tests (standard + fuzzing)
make test-all

For more details, see:

Testing Guide - Comprehensive testing documentation
Subtests Guide - Writing tests with subtests
CONTRIBUTING.md - Contribution guidelines including testing practices

📜 License

MIT License - see LICENSE for details.

🙏 Acknowledgments

Inspired by the sophisticated code review capabilities of CodeRabbit AI
Built with experience from ContextForge Memory project
Community feedback and contributions

📊 Project Status

Current Version: 0.1.0 (Alpha)

Roadmap:

✅ Phase 0: Security Foundation (COMPLETE)
- ✅ 0.1: Security Architecture Design
- ✅ 0.2: Input Validation & Sanitization
- ✅ 0.3: Secure File Handling
- ✅ 0.4: Secret Detection (14+ patterns)
- ✅ 0.5: Security Testing Suite (95%+ coverage)
- ✅ 0.6: Security Configuration
- ✅ 0.7: CI/CD Security Scanning (7+ tools)
- ✅ 0.8: Security Documentation
✅ Phase 1: Core Features (COMPLETE)
- ✅ Core conflict detection and analysis
- ✅ File handlers (JSON, YAML, TOML)
- ✅ Priority system
- ✅ Rollback system with git-based checkpointing
✅ Phase 2: CLI & Configuration (COMPLETE)
- ✅ CLI with comprehensive options
- ✅ Runtime configuration system
- ✅ Application modes (all, conflicts-only, non-conflicts-only, dry-run)
- ✅ Parallel processing support
- ✅ Multiple configuration sources (file, env, CLI)
🔄 Phase 3: Documentation & Examples (IN PROGRESS)
- 🔄 Comprehensive documentation updates
- 📅 Example configurations and use cases
✅ V2.0 Phase 0: LLM Foundation (COMPLETE) - PR #121
- ✅ Core LLM data models and infrastructure
- ✅ Universal comment parser with LLM + regex fallback
- ✅ LLM provider protocol for polymorphic support
- ✅ Structured prompt engineering system
- ✅ Confidence threshold filtering
✅ V2.0 Phase 1: LLM-Powered Parsing (COMPLETE) - PR #122
- ✅ OpenAI API provider implementation
- ✅ Automatic retry logic with exponential backoff
- ✅ Token counting and cost tracking
- ✅ Comprehensive error handling
- ✅ Integration with ConflictResolver
🔄 V2.0 Phase 2-6 (IN PROGRESS) - 29% complete
- 📅 Multi-provider support (Anthropic, Claude CLI, Codex, Ollama)
- 📅 CLI integration polish and preset system
- 📅 Production hardening (retry logic, cost controls)
- 📅 Comprehensive documentation and migration guides

Security Highlights

ClusterFuzzLite: Continuous fuzzing (3 fuzz targets, ASan + UBSan)
Test Coverage: 82.35% overall, 95%+ for security modules
Security Scanning: CodeQL, Trivy, TruffleHog, Bandit, pip-audit, OpenSSF Scorecard
Secret Detection: 14+ pattern types (GitHub tokens, AWS keys, API keys, etc.)
Documentation: Comprehensive security documentation (threat model, incident response, compliance)

🚀 Upcoming Features (v2.0 - LLM-First Architecture)

Coming Soon: Major architecture upgrade to parse 95%+ of CodeRabbit comments (up from 20%)

The Problem We're Solving

Current system only parses ```suggestion blocks, missing:

❌ Diff blocks (```diff) - 60% of CodeRabbit comments
❌ Natural language suggestions - 20% of comments
❌ Multi-option suggestions
❌ Multiple diff blocks per comment

Result: Only 1 out of 5 CodeRabbit comments are currently parsed.

The Solution: LLM-First Parsing

┌─────────────────────────────────────────────────────────┐
│           LLM Parser (Primary - All Formats)            │
│  • Diff blocks        • Suggestion blocks              │
│  • Natural language   • Multi-options                   │
│  • 95%+ coverage      • Intelligent understanding       │
└──────────────────────────┬──────────────────────────────┘
                           │
                  ┌────────┴────────┐
                  │  Fallback if    │
                  │  LLM fails      │
                  └────────┬────────┘
                           ▼
┌─────────────────────────────────────────────────────────┐
│       Regex Parser (Fallback - Suggestion Blocks)       │
│  • 100% reliable      • Zero cost                       │
│  • Legacy support     • Always available                │
└─────────────────────────────────────────────────────────┘

Multi-Provider Support (User Choice)

Choose your preferred LLM provider:

Provider	Cost Model	Best For	Est. Cost (1000 comments)
Claude CLI	Subscription ($20/mo)	Best quality + zero marginal cost	$0 (covered)
Codex CLI	Subscription ($20/mo)	Cost-effective, OpenAI quality	$0 (covered)
Ollama	Free (local)	Privacy, offline, no API costs	$0
OpenAI API	Pay-per-token	Pay-as-you-go, low volume	$0.07 (with caching)
Anthropic API	Pay-per-token	Best quality, willing to pay	$0.22 (with caching)

Quick Preview

# Current (v1.x) - regex-only
pr-resolve apply --owner VirtualAgentics --repo my-repo --pr 123
# Parses: 1/5 comments (20%)

# v2.0 - LLM-powered (opt-in)
pr-resolve apply --llm --llm-provider claude-cli --owner VirtualAgentics --repo my-repo --pr 123
# Parses: 5/5 comments (100%)

# Use presets for quick config
pr-resolve apply --llm-preset claude-cli-sonnet --owner VirtualAgentics --repo my-repo --pr 123
pr-resolve apply --llm-preset ollama-local --owner VirtualAgentics --repo my-repo --pr 123  # Privacy-first

Backward Compatibility Guarantee

✅ Zero Breaking Changes - All v1.x code works unchanged in v2.0

LLM parsing disabled by default (opt-in via --llm flag)
Automatic fallback to regex if LLM fails
v1.x CLI commands work identically
v1.x Python API unchanged

Enhanced Change Metadata

# v2.0: Changes include AI-powered insights
change = Change(
    path="src/module.py",
    start_line=10,
    end_line=12,
    content="new code",
    # NEW in v2.0 (optional fields)
    llm_confidence=0.95,  # How confident the LLM is
    llm_provider="claude-cli",  # Which provider parsed it
    parsing_method="llm",  # "llm" or "regex"
    change_rationale="Improves error handling",  # Why change was suggested
    risk_level="low"  # "low", "medium", "high"
)

Documentation

Comprehensive planning documentation available:

LLM Refactor Roadmap (15K words) - Full implementation plan
LLM Architecture (8K words) - Technical specification
Migration Guide (3K words) - v1.x → v2.0 upgrade path

Timeline

Phase 0-6: 10-12 weeks implementation
Estimated Release: Q2 2025
GitHub Milestone: v2.0 - LLM-First Architecture
GitHub Issues: #114-#120 (Phases 0-6)

🔗 Related Projects

ContextForge Memory - Original implementation
CodeRabbit AI - AI-powered code review

Made with ❤️ by VirtualAgentics

Name		Name	Last commit message	Last commit date
Latest commit History 116 Commits
.clusterfuzzlite		.clusterfuzzlite
.cursor/rules		.cursor/rules
.github		.github
docs		docs
examples		examples
fuzz		fuzz
scripts		scripts
src/pr_conflict_resolver		src/pr_conflict_resolver
tests		tests
.coderabbit.yaml		.coderabbit.yaml
.cursorignore		.cursorignore
.cursorrules		.cursorrules
.editorconfig		.editorconfig
.env.example		.env.example
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.truffleignore		.truffleignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
pyproject.toml		pyproject.toml
requirements-dev.in		requirements-dev.in
requirements-dev.txt		requirements-dev.txt
requirements-ml.in		requirements-ml.in
requirements-ml.txt		requirements-ml.txt
requirements.in		requirements.in
requirements.txt		requirements.txt
setup-dev.sh		setup-dev.sh
uv.lock		uv.lock

License

VirtualAgentics/review-bot-automator

Folders and files

Latest commit

History

Repository files navigation

Review Bot Automator

📋 Table of Contents

🎯 Problem Statement

🚀 Quick Start

Installation

Basic Usage

LLM Provider Setup (Optional)

Alternative: Use environment variables

Python API

🎨 Features

Intelligent Conflict Analysis

Smart Resolution Strategies

File-Type Handlers

Multi-Provider LLM Support

Learning & Optimization

Configuration & Presets

Application Modes

Rollback & Safety Features

Runtime Configuration

📖 Documentation

User Guides

Reference Documentation

Architecture & Development

Security

🏗️ Architecture

🔧 Use Cases

1. CodeRabbit Multi-Option Selections

2. Overlapping Configuration Changes

3. Security Fix vs. Formatting

4. Large PR with 50+ Comments

🔧 Environment Variables

🤝 Contributing

Development Setup

Running Tests

📜 License

🙏 Acknowledgments

📊 Project Status

Security Highlights

🚀 Upcoming Features (v2.0 - LLM-First Architecture)

The Problem We're Solving

The Solution: LLM-First Parsing

Multi-Provider Support (User Choice)

Quick Preview

Backward Compatibility Guarantee

Enhanced Change Metadata

Documentation

Timeline

🔗 Related Projects

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 5

Uh oh!

Languages

Packages