Hiddenden/openrabbit
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects 1 Releases Wiki Activity

Feature Request: @codebot review-again Command #7

New Issue
Closed
opened 2025-12-28 17:52:16 +00:00 by Latte · 0 comments
Latte commented 2025-12-28 17:52:16 +00:00
Owner
Copy Link

Feature Request: @codebot review-again Command

Summary

Add a @codebot review-again command that re-runs the AI code review on a pull request without requiring new commits, enabling faster iteration cycles and re-evaluation after comment-only changes.

What problem does this solve?

Current Problem:
The AI review workflow only triggers automatically when:

  • PR is opened (pull_request.opened)
  • New commits are pushed (pull_request.synchronize)

Scenarios where developers need manual re-review:

  1. Comment-only changes - Developer addresses feedback in comments but doesn't change code
  2. Clarification provided - Developer explains implementation, wants AI to reconsider severity
  3. False positive fixes - Security scanner flagged something incorrectly, developer wants re-scan
  4. Configuration changes - Updated .ai-review.yml and want to see new results
  5. Testing AI improvements - After updating AI review system, want to re-test PRs

Current Workaround (Painful):

# Force a new commit just to trigger review
git commit --allow-empty -m "Trigger AI review"
git push

User Pain:

  • Cluttered git history with empty "trigger review" commits
  • Waiting for full CI pipeline just for review
  • No way to re-run review after changing config
  • Can't test if AI feedback changed after clarification

User story / Use case

As a developer working on a pull request,
I want to manually trigger an AI re-review without creating new commits,
so that I can get updated feedback after addressing comments, fixing false positives, or updating configuration.

Example scenario 1: False Positive

  1. AI flags hardcoded API key: API_URL = "https://api.example.com"
  2. Developer comments: "This is a public API URL, not a secret"
  3. Developer types: @codebot review-again
  4. AI re-scans, confirms it's not a secret, updates review

Example scenario 2: Configuration Change

  1. AI review finds 10 issues
  2. Developer updates .ai-review.yml to adjust security rules
  3. Types: @codebot review-again
  4. AI re-reviews with new config, finds only 2 real issues

Example scenario 3: Comment-Only Response

  1. AI requests clarification on architecture decision
  2. Developer explains in PR comments (no code changes needed)
  3. Types: @codebot review-again
  4. AI considers explanation, updates severity assessment

Proposed solution

Implementation

Add a @codebot review-again command that:

  1. Triggers on PR comment containing the command
  2. Re-runs PR review using current HEAD commit
  3. Updates existing review comment instead of creating new one
  4. Shows diff from previous review (what changed in findings)
  5. Respects current config (uses latest .ai-review.yml)

Example Output

@username

**🔄 Re-review Requested**

Re-analyzing PR #123 at commit `abc1234`...

---

## AI Code Review (Updated)

**Previous Review:** 5 issues (2 HIGH, 3 MEDIUM)  
**Current Review:** 2 issues (0 HIGH, 2 MEDIUM)

### Changes from Previous Review

**✅ Resolved (3):**
- **[HIGH]** `src/auth.py:45` - Hardcoded API key (confirmed as public URL, not a secret)
- **[HIGH]** `src/db.py:12` - SQL injection risk (false positive - using parameterized query)
- **[MEDIUM]** `src/utils.py:30` - Missing docstring (added in comments)

**Still Present (2):**
- **[MEDIUM]** `src/api.py:67` - Error handling could be more specific
- **[MEDIUM]** `src/config.py:12` - Consider using environment variables

### Summary

| Severity | Count | Change |
|----------|-------|--------|
| HIGH | 0 | -2 ✅ |
| MEDIUM | 2 | -1 ✅ |
| LOW | 0 | 0 |

---

**Overall Severity:** `MEDIUM` (was `HIGH`)  
**AI Recommendation:** Approved ✅ (was Changes Required)

**Status Update:**
- Removed label: `ai-changes-required`
- Added label: `ai-approved`

Files to Modify

1. tools/ai-review/config.yml

Add command to allowed list:

interaction:
  commands:
    - help
    - explain
    - suggest
    - triage
    - review-again    # New command for PRs

2. tools/ai-review/agents/pr_agent.py

Add review-again functionality:

def execute(self, context: AgentContext) -> AgentResult:
    """Execute PR review agent."""
    
    # Check if this is a review-again command
    if context.event_type == "issue_comment" and context.event_data.get("action") == "created":
        comment_body = context.event_data.get("comment", {}).get("body", "")
        if "@codebot review-again" in comment_body or "@bartender review-again" in comment_body:
            return self._handle_review_again(context)
    
    # Normal PR review flow
    return self._handle_pr_review(context)

def _handle_review_again(self, context: AgentContext) -> AgentResult:
    """Re-run PR review on current state."""
    
    # Get PR number from issue
    pr_number = context.event_data.get("issue", {}).get("number")
    
    # Get previous review comment
    previous_review = self.find_ai_comment(context.owner, context.repo, pr_number)
    
    # Run new review
    new_review_result = self._run_pr_review(context.owner, context.repo, pr_number)
    
    # Compare with previous review
    diff = self._compare_reviews(previous_review, new_review_result)
    
    # Format updated review
    updated_review = self._format_review_update(new_review_result, diff)
    
    # Update existing comment
    self.upsert_comment(
        context.owner,
        context.repo,
        pr_number,
        self.format_with_disclaimer(updated_review)
    )
    
    # Update PR labels based on new severity
    self._update_pr_labels(context.owner, context.repo, pr_number, new_review_result)
    
    return AgentResult(
        success=True,
        message=f"Re-reviewed PR #{pr_number}",
        actions_taken=["re-ran pr review", "updated review comment", "updated labels"],
        data={"findings": len(new_review_result.get("findings", []))}
    )

def _compare_reviews(self, previous_review_text: str, new_review: dict) -> dict:
    """Compare previous and new review to show what changed.
    
    Returns:
        {
            "resolved": [...],     # Issues that disappeared
            "new": [...],          # New issues found
            "still_present": [...], # Issues that remain
            "severity_changed": {...} # OLD severity -> NEW severity
        }
    """
    # Parse previous review comment to extract findings
    previous_findings = self._parse_review_comment(previous_review_text)
    new_findings = new_review.get("findings", [])
    
    # Create lookup keys (file:line:rule_id)
    prev_keys = {self._finding_key(f): f for f in previous_findings}
    new_keys = {self._finding_key(f): f for f in new_findings}
    
    resolved = [
        prev_keys[key] for key in prev_keys 
        if key not in new_keys
    ]
    
    new = [
        new_keys[key] for key in new_keys 
        if key not in prev_keys
    ]
    
    still_present = [
        new_keys[key] for key in new_keys 
        if key in prev_keys
    ]
    
    severity_changed = {}
    for key in prev_keys:
        if key in new_keys:
            prev_severity = prev_keys[key].get("severity")
            new_severity = new_keys[key].get("severity")
            if prev_severity != new_severity:
                severity_changed[key] = {
                    "old": prev_severity,
                    "new": new_severity,
                    "finding": new_keys[key]
                }
    
    return {
        "resolved": resolved,
        "new": new,
        "still_present": still_present,
        "severity_changed": severity_changed
    }

def _format_review_update(self, review: dict, diff: dict) -> str:
    """Format review with comparison to previous run."""
    
    lines = ["**🔄 Re-review Requested**\n"]
    lines.append(f"Re-analyzing PR at commit `{review['commit'][:7]}`...\n")
    lines.append("---\n")
    lines.append("## AI Code Review (Updated)\n")
    
    # Summary of changes
    prev_total = len(diff["resolved"]) + len(diff["still_present"])
    curr_total = len(diff["new"]) + len(diff["still_present"])
    
    lines.append(f"**Previous Review:** {prev_total} issues")
    lines.append(f"**Current Review:** {curr_total} issues\n")
    
    # Changes section
    lines.append("### Changes from Previous Review\n")
    
    if diff["resolved"]:
        lines.append(f"**✅ Resolved ({len(diff['resolved'])}):**")
        for finding in diff["resolved"][:5]:  # Show max 5
            lines.append(f"- **[{finding['severity']}]** `{finding['file']}:{finding['line']}` - {finding['description']}")
        if len(diff["resolved"]) > 5:
            lines.append(f"- ... and {len(diff['resolved']) - 5} more")
        lines.append("")
    
    if diff["new"]:
        lines.append(f"**⚠️ New Issues ({len(diff['new'])}):**")
        for finding in diff["new"][:5]:
            lines.append(f"- **[{finding['severity']}]** `{finding['file']}:{finding['line']}` - {finding['description']}")
        if len(diff["new"]) > 5:
            lines.append(f"- ... and {len(diff['new']) - 5} more")
        lines.append("")
    
    if diff["severity_changed"]:
        lines.append(f"**🔄 Severity Changed ({len(diff['severity_changed'])}):**")
        for key, change in list(diff["severity_changed"].items())[:5]:
            finding = change["finding"]
            lines.append(f"- `{finding['file']}:{finding['line']}` - {change['old']} → {change['new']}")
        lines.append("")
    
    if diff["still_present"]:
        lines.append(f"**Still Present ({len(diff['still_present'])}):**")
        for finding in diff["still_present"][:3]:
            lines.append(f"- **[{finding['severity']}]** `{finding['file']}:{finding['line']}` - {finding['description']}")
        if len(diff["still_present"]) > 3:
            lines.append(f"- ... and {len(diff['still_present']) - 3} more")
        lines.append("")
    
    # Overall summary table
    severity_counts = self._count_by_severity(review["findings"])
    
    lines.append("### Summary\n")
    lines.append("| Severity | Count | Change |")
    lines.append("|----------|-------|--------|")
    
    for severity in ["HIGH", "MEDIUM", "LOW"]:
        curr = severity_counts.get(severity, 0)
        # Calculate change (simplified - could be more accurate)
        change_symbol = ""
        lines.append(f"| {severity} | {curr} | {change_symbol} |")
    
    lines.append("")
    
    # Final verdict
    overall_severity = review.get("overall_severity", "MEDIUM")
    recommendation = "Approved ✅" if overall_severity in ["LOW", "NONE"] else "Changes Requested"
    
    lines.append(f"**Overall Severity:** `{overall_severity}`")
    lines.append(f"**AI Recommendation:** {recommendation}")
    
    return "\n".join(lines)

def _finding_key(self, finding: dict) -> str:
    """Create unique key for a finding."""
    return f"{finding.get('file')}:{finding.get('line')}:{finding.get('rule_id', finding.get('category'))}"

def _parse_review_comment(self, comment_text: str) -> list[dict]:
    """Parse previous review comment to extract findings."""
    # Simple regex-based parsing of markdown tables and lists
    # Returns list of findings with file, line, severity, description
    findings = []
    
    if not comment_text:
        return findings
    
    # Look for patterns like: **[HIGH]** `src/file.py:45` - Description
    import re
    pattern = r'\*\*\[(\w+)\]\*\*\s+`([^:]+):(\d+)`\s+-\s+(.+)'
    
    for match in re.finditer(pattern, comment_text):
        findings.append({
            "severity": match.group(1),
            "file": match.group(2),
            "line": int(match.group(3)),
            "description": match.group(4).strip()
        })
    
    return findings

def _count_by_severity(self, findings: list[dict]) -> dict:
    """Count findings by severity level."""
    counts = {"HIGH": 0, "MEDIUM": 0, "LOW": 0}
    for finding in findings:
        severity = finding.get("severity", "MEDIUM")
        counts[severity] = counts.get(severity, 0) + 1
    return counts

def _update_pr_labels(self, owner: str, repo: str, pr_number: int, review: dict):
    """Update PR labels based on review severity."""
    overall_severity = review.get("overall_severity", "MEDIUM")
    
    # Remove old AI labels
    self.gitea.remove_issue_label(owner, repo, pr_number, "ai-approved")
    self.gitea.remove_issue_label(owner, repo, pr_number, "ai-changes-required")
    
    # Add new label
    if overall_severity in ["LOW", "NONE"]:
        self.gitea.add_issue_labels(owner, repo, pr_number, ["ai-approved"])
    else:
        self.gitea.add_issue_labels(owner, repo, pr_number, ["ai-changes-required"])

3. .gitea/workflows/ai-comment-reply.yml

Update workflow to handle PR comments:

name: AI Comment Reply

on:
  issue_comment:
    types: [created]

jobs:
  ai-reply:
    runs-on: ubuntu-latest
    if: contains(github.event.comment.body, '@codebot') || contains(github.event.comment.body, '@bartender')
    steps:
      - uses: actions/checkout@v4

      - uses: actions/checkout@v4
        with:
          repository: YourOrg/OpenRabbit
          path: .ai-review
          token: ${{ secrets.AI_REVIEW_TOKEN }}

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - run: pip install requests pyyaml

      - name: Run AI Comment Response
        env:
          AI_REVIEW_TOKEN: ${{ secrets.AI_REVIEW_TOKEN }}
          AI_REVIEW_REPO: ${{ gitea.repository }}
          AI_REVIEW_API_URL: https://your-gitea.example.com/api/v1
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          SEARXNG_URL: ${{ secrets.SEARXNG_URL }}
        run: |
          cd .ai-review/tools/ai-review
          
          # Determine if this is an issue or PR
          if [ "${{ github.event.issue.pull_request }}" != "" ]; then
            # This is a PR comment
            python main.py pr-comment ${{ gitea.repository }} ${{ github.event.issue.number }} \
              "${{ github.event.comment.body }}"
          else
            # This is an issue comment
            python main.py comment ${{ gitea.repository }} ${{ github.event.issue.number }} \
              "${{ github.event.comment.body }}"
          fi

Alternatives considered

Alternative 1: Automatic Re-review on Comment

  • Rejected: Too noisy - would re-run on every comment
  • Current manual trigger is more intentional

Alternative 2: Time-based Re-review

Example: Re-review PRs every 24 hours

  • Rejected: Wastes LLM tokens on inactive PRs
  • Manual trigger is more cost-effective

Alternative 3: Webhook-based Trigger

External service triggers review via API

  • Rejected: Adds complexity, requires infrastructure
  • Comment-based trigger is simpler

Alternative 4: GitHub Actions Manual Dispatch

Use workflow_dispatch event

  • Considered for future: Good for advanced users
  • Comment-based trigger is more accessible

Alternative 5: Always Create New Comment

Instead of updating existing review

  • Rejected: Clutters PR discussion
  • Updating comment keeps review consolidated

Acceptance criteria

  • @codebot review-again command triggers on PR comments
  • Re-runs full PR review with current config
  • Updates existing AI review comment (doesn't create new)
  • Shows diff from previous review (resolved, new, still present)
  • Updates PR labels (ai-approved, ai-changes-required)
  • Works with security scanner (re-scans for vulnerabilities)
  • Respects latest .ai-review.yml configuration
  • Shows commit hash being reviewed
  • Response time < 30 seconds for typical PR
  • Handles case where no previous review exists
  • Works for both issue comments and PR review comments
  • Command is documented in README and @codebot help
  • Unit tests for review comparison logic
  • Integration test with actual PR

Backwards compatibility:

  • ✅ No breaking changes - purely additive
  • ✅ Normal automatic reviews still work
  • ✅ Existing workflows unchanged

Impact

Complexity: LOW-MEDIUM

  • Estimated effort: 1-2 hours
  • Files modified: pr_agent.py, config.yml, workflow files
  • Reuses existing review logic
  • New logic: review comparison and diff formatting

User Impact: HIGH

  • Faster feedback loops
  • No need for empty commits
  • Better developer experience
  • Enables configuration testing

Technical Impact:

  • Minimal - reuses 90% of existing PR review code
  • New: previous review parsing and comparison
  • Safe operation (idempotent)

Cost Impact:

  • Additional LLM API calls when command is used
  • Mitigation: Only runs on explicit request (not automatic)
  • Users control when to re-review

Mockups / examples

Example 1: Standard Re-review

**Pull Request #45: Add user authentication**

Developer comment:
> The hardcoded API key at line 45 is actually a public API URL, not a secret. 
> @codebot review-again

Bot response:
> @developer
> 
> **🔄 Re-review Requested**
> 
> Re-analyzing PR #45 at commit `abc1234`...
> 
> ---
> 
> ## AI Code Review (Updated)
> 
> **Previous Review:** 3 issues (1 HIGH, 2 MEDIUM)
> **Current Review:** 2 issues (0 HIGH, 2 MEDIUM)
> 
> ### Changes from Previous Review
> 
> **✅ Resolved (1):**
> - **[HIGH]** `src/auth.py:45` - Hardcoded API key detected
>   (Confirmed: public API URL, not a secret)
> 
> **Still Present (2):**
> - **[MEDIUM]** `src/api.py:67` - Consider more specific error handling
> - **[MEDIUM]** `src/config.py:12` - Use environment variables
> 
> ### Summary
> 
> | Severity | Count | Change |
> |----------|-------|--------|
> | HIGH | 0 | -1 ✅ |
> | MEDIUM | 2 | 0 |
> | LOW | 0 | 0 |
> 
> **Overall Severity:** `MEDIUM` (was `HIGH`)
> **AI Recommendation:** Approved ✅
> 
> **Status Update:**
> - Removed label: `ai-changes-required`
> - Added label: `ai-approved`

Example 2: After Config Change

Developer comment:
> Updated .ai-review.yml to disable MD5 detection (we use it for checksums only)
> @codebot review-again

Bot response:
> **🔄 Re-review Requested**
> 
> Re-analyzing with updated configuration...
> 
> ### Changes from Previous Review
> 
> **✅ Resolved (2):**
> - **[MEDIUM]** `src/utils.py:23` - Weak hash algorithm (MD5)
> - **[MEDIUM]** `src/checksum.py:45` - Weak hash algorithm (MD5)
> 
> **Configuration Changes Detected:**
> - Disabled rule: SEC003 (Weak hash algorithms)
> 
> **Current Review:** 0 issues
> 
> **Overall Severity:** `NONE`
> **AI Recommendation:** Approved ✅

Example 3: New Issues Found

Developer comment:
> Fixed the SQL injection. @codebot review-again

Bot response:
> **🔄 Re-review Requested**
> 
> ### Changes from Previous Review
> 
> **✅ Resolved (1):**
> - **[HIGH]** `src/db.py:12` - SQL injection via string formatting
> 
> **⚠️ New Issues (1):**
> - **[MEDIUM]** `src/db.py:15` - Missing input validation on user_id parameter
> 
> **Still Present (1):**
> - **[LOW]** `src/utils.py:30` - Missing docstring
> 
> ### Summary
> 
> | Severity | Count | Change |
> |----------|-------|--------|
> | HIGH | 0 | -1 ✅ |
> | MEDIUM | 1 | +1 ⚠️ |
> | LOW | 1 | 0 |
> 
> **Overall Severity:** `MEDIUM`
> **AI Recommendation:** Changes Requested

Example 4: No Previous Review

Developer comment:
> @codebot review-again

Bot response:
> **🔄 Re-review Requested**
> 
> No previous review found. Running initial review...
> 
> [Standard review output]

Example 5: No Changes

Developer comment:
> @codebot review-again

Bot response:
> **🔄 Re-review Requested**
> 
> Re-analyzed PR #45 at commit `abc1234`
> 
> **No changes from previous review.**
> 
> All findings remain the same:
> - 1 HIGH severity issue
> - 2 MEDIUM severity issues
> 
> **Overall Severity:** `HIGH`
> **AI Recommendation:** Changes Required

Implementation Plan

Phase 1: Core Implementation (1 hour)

  1. Add Command Handler (30 min)

    • Detect review-again in PR comments
    • Add to PRAgent.execute() routing
    • Call _handle_review_again()

  2. Review Comparison Logic (30 min)

    • _parse_review_comment() - Extract previous findings
    • _compare_reviews() - Diff old vs new
    • _finding_key() - Create unique identifiers

Phase 2: Formatting & Labels (30 min)

  1. Format Review Update (15 min)

    • _format_review_update() - Template with diff
    • Show resolved, new, still present
    • Summary table with changes

  2. Label Management (15 min)

    • _update_pr_labels() - Remove old, add new
    • Update based on new severity

Phase 3: Testing & Documentation (30 min)

  1. Testing (20 min)

    • Unit test review comparison
    • Test with/without previous review
    • Test label updates

  2. Documentation (10 min)

    • Update README.md
    • Add to @codebot help
    • Update workflow examples

Dependencies

Required:

  • Existing PR review system (already implemented)
  • Comment parsing regex

Optional:

  • None

Related Issues/PRs

  • Related to: #XXX (@codebot help command)
  • Part of: Milestone 2 - UX & Discoverability
  • Enables: Faster PR iteration cycles
  • Foundation for: Configuration testing workflows

Success Metrics

Week 1 after deployment:

  • Used on 20%+ of PRs
  • Reduces empty "trigger review" commits by 90%
  • Average time to re-review < 30 seconds

Month 1 after deployment:

  • 50%+ of developers aware of command
  • Positive feedback on iteration speed
  • Reduced PR cycle time by 15%

Long-term:

  • Standard part of PR workflow
  • Enables config experimentation
  • Better AI review accuracy over time

Future Enhancements

v2.0 Features:

  • @codebot review-again --full - Re-review entire codebase context
  • @codebot review-again --security-only - Only re-run security scan
  • Scheduled re-reviews for long-lived PRs
  • Compare review with main branch baseline

Checklist

  • I searched existing feature requests and discussions
  • This request includes user story and acceptance criteria
  • Impact assessment completed (LOW-MEDIUM complexity, HIGH value)
  • Implementation plan outlined
  • Acceptance criteria defined
  • Success metrics identified
  • Example scenarios documented

Priority: ⭐ MEDIUM-HIGH - Quick win for developer experience
Milestone: Milestone 2 - UX & Discoverability
Estimated Effort: 1-2 hours
Value: HIGH
Complexity: LOW-MEDIUM
Risk: LOW

Key Features:

  • 🔄 Re-run review without new commits
  • 📊 Show diff from previous review
  • 🏷️ Auto-update labels
  • ⚡ Fast iteration cycles
  • 🔧 Respects config changes
# Feature Request: @codebot review-again Command ## Summary Add a `@codebot review-again` command that re-runs the AI code review on a pull request without requiring new commits, enabling faster iteration cycles and re-evaluation after comment-only changes. ## What problem does this solve? **Current Problem:** The AI review workflow only triggers automatically when: - PR is opened (`pull_request.opened`) - New commits are pushed (`pull_request.synchronize`) **Scenarios where developers need manual re-review:** 1. **Comment-only changes** - Developer addresses feedback in comments but doesn't change code 2. **Clarification provided** - Developer explains implementation, wants AI to reconsider severity 3. **False positive fixes** - Security scanner flagged something incorrectly, developer wants re-scan 4. **Configuration changes** - Updated `.ai-review.yml` and want to see new results 5. **Testing AI improvements** - After updating AI review system, want to re-test PRs **Current Workaround (Painful):** ```bash # Force a new commit just to trigger review git commit --allow-empty -m "Trigger AI review" git push ``` **User Pain:** - Cluttered git history with empty "trigger review" commits - Waiting for full CI pipeline just for review - No way to re-run review after changing config - Can't test if AI feedback changed after clarification ## User story / Use case **As a** developer working on a pull request, **I want** to manually trigger an AI re-review without creating new commits, **so that** I can get updated feedback after addressing comments, fixing false positives, or updating configuration. **Example scenario 1: False Positive** 1. AI flags hardcoded API key: `API_URL = "https://api.example.com"` 2. Developer comments: "This is a public API URL, not a secret" 3. Developer types: `@codebot review-again` 4. AI re-scans, confirms it's not a secret, updates review **Example scenario 2: Configuration Change** 1. AI review finds 10 issues 2. Developer updates `.ai-review.yml` to adjust security rules 3. Types: `@codebot review-again` 4. AI re-reviews with new config, finds only 2 real issues **Example scenario 3: Comment-Only Response** 1. AI requests clarification on architecture decision 2. Developer explains in PR comments (no code changes needed) 3. Types: `@codebot review-again` 4. AI considers explanation, updates severity assessment ## Proposed solution ### Implementation Add a `@codebot review-again` command that: 1. **Triggers on PR comment** containing the command 2. **Re-runs PR review** using current HEAD commit 3. **Updates existing review comment** instead of creating new one 4. **Shows diff** from previous review (what changed in findings) 5. **Respects current config** (uses latest `.ai-review.yml`) ### Example Output ```markdown @username **🔄 Re-review Requested** Re-analyzing PR #123 at commit `abc1234`... --- ## AI Code Review (Updated) **Previous Review:** 5 issues (2 HIGH, 3 MEDIUM) **Current Review:** 2 issues (0 HIGH, 2 MEDIUM) ### Changes from Previous Review **✅ Resolved (3):** - **[HIGH]** `src/auth.py:45` - Hardcoded API key (confirmed as public URL, not a secret) - **[HIGH]** `src/db.py:12` - SQL injection risk (false positive - using parameterized query) - **[MEDIUM]** `src/utils.py:30` - Missing docstring (added in comments) **Still Present (2):** - **[MEDIUM]** `src/api.py:67` - Error handling could be more specific - **[MEDIUM]** `src/config.py:12` - Consider using environment variables ### Summary | Severity | Count | Change | |----------|-------|--------| | HIGH | 0 | -2 ✅ | | MEDIUM | 2 | -1 ✅ | | LOW | 0 | 0 | --- **Overall Severity:** `MEDIUM` (was `HIGH`) **AI Recommendation:** Approved ✅ (was Changes Required) **Status Update:** - Removed label: `ai-changes-required` - Added label: `ai-approved` ``` ### Files to Modify #### 1. **`tools/ai-review/config.yml`** Add command to allowed list: ```yaml interaction: commands: - help - explain - suggest - triage - review-again # New command for PRs ``` #### 2. **`tools/ai-review/agents/pr_agent.py`** Add review-again functionality: ```python def execute(self, context: AgentContext) -> AgentResult: """Execute PR review agent.""" # Check if this is a review-again command if context.event_type == "issue_comment" and context.event_data.get("action") == "created": comment_body = context.event_data.get("comment", {}).get("body", "") if "@codebot review-again" in comment_body or "@bartender review-again" in comment_body: return self._handle_review_again(context) # Normal PR review flow return self._handle_pr_review(context) def _handle_review_again(self, context: AgentContext) -> AgentResult: """Re-run PR review on current state.""" # Get PR number from issue pr_number = context.event_data.get("issue", {}).get("number") # Get previous review comment previous_review = self.find_ai_comment(context.owner, context.repo, pr_number) # Run new review new_review_result = self._run_pr_review(context.owner, context.repo, pr_number) # Compare with previous review diff = self._compare_reviews(previous_review, new_review_result) # Format updated review updated_review = self._format_review_update(new_review_result, diff) # Update existing comment self.upsert_comment( context.owner, context.repo, pr_number, self.format_with_disclaimer(updated_review) ) # Update PR labels based on new severity self._update_pr_labels(context.owner, context.repo, pr_number, new_review_result) return AgentResult( success=True, message=f"Re-reviewed PR #{pr_number}", actions_taken=["re-ran pr review", "updated review comment", "updated labels"], data={"findings": len(new_review_result.get("findings", []))} ) def _compare_reviews(self, previous_review_text: str, new_review: dict) -> dict: """Compare previous and new review to show what changed. Returns: { "resolved": [...], # Issues that disappeared "new": [...], # New issues found "still_present": [...], # Issues that remain "severity_changed": {...} # OLD severity -> NEW severity } """ # Parse previous review comment to extract findings previous_findings = self._parse_review_comment(previous_review_text) new_findings = new_review.get("findings", []) # Create lookup keys (file:line:rule_id) prev_keys = {self._finding_key(f): f for f in previous_findings} new_keys = {self._finding_key(f): f for f in new_findings} resolved = [ prev_keys[key] for key in prev_keys if key not in new_keys ] new = [ new_keys[key] for key in new_keys if key not in prev_keys ] still_present = [ new_keys[key] for key in new_keys if key in prev_keys ] severity_changed = {} for key in prev_keys: if key in new_keys: prev_severity = prev_keys[key].get("severity") new_severity = new_keys[key].get("severity") if prev_severity != new_severity: severity_changed[key] = { "old": prev_severity, "new": new_severity, "finding": new_keys[key] } return { "resolved": resolved, "new": new, "still_present": still_present, "severity_changed": severity_changed } def _format_review_update(self, review: dict, diff: dict) -> str: """Format review with comparison to previous run.""" lines = ["**🔄 Re-review Requested**\n"] lines.append(f"Re-analyzing PR at commit `{review['commit'][:7]}`...\n") lines.append("---\n") lines.append("## AI Code Review (Updated)\n") # Summary of changes prev_total = len(diff["resolved"]) + len(diff["still_present"]) curr_total = len(diff["new"]) + len(diff["still_present"]) lines.append(f"**Previous Review:** {prev_total} issues") lines.append(f"**Current Review:** {curr_total} issues\n") # Changes section lines.append("### Changes from Previous Review\n") if diff["resolved"]: lines.append(f"**✅ Resolved ({len(diff['resolved'])}):**") for finding in diff["resolved"][:5]: # Show max 5 lines.append(f"- **[{finding['severity']}]** `{finding['file']}:{finding['line']}` - {finding['description']}") if len(diff["resolved"]) > 5: lines.append(f"- ... and {len(diff['resolved']) - 5} more") lines.append("") if diff["new"]: lines.append(f"**⚠️ New Issues ({len(diff['new'])}):**") for finding in diff["new"][:5]: lines.append(f"- **[{finding['severity']}]** `{finding['file']}:{finding['line']}` - {finding['description']}") if len(diff["new"]) > 5: lines.append(f"- ... and {len(diff['new']) - 5} more") lines.append("") if diff["severity_changed"]: lines.append(f"**🔄 Severity Changed ({len(diff['severity_changed'])}):**") for key, change in list(diff["severity_changed"].items())[:5]: finding = change["finding"] lines.append(f"- `{finding['file']}:{finding['line']}` - {change['old']} → {change['new']}") lines.append("") if diff["still_present"]: lines.append(f"**Still Present ({len(diff['still_present'])}):**") for finding in diff["still_present"][:3]: lines.append(f"- **[{finding['severity']}]** `{finding['file']}:{finding['line']}` - {finding['description']}") if len(diff["still_present"]) > 3: lines.append(f"- ... and {len(diff['still_present']) - 3} more") lines.append("") # Overall summary table severity_counts = self._count_by_severity(review["findings"]) lines.append("### Summary\n") lines.append("| Severity | Count | Change |") lines.append("|----------|-------|--------|") for severity in ["HIGH", "MEDIUM", "LOW"]: curr = severity_counts.get(severity, 0) # Calculate change (simplified - could be more accurate) change_symbol = "" lines.append(f"| {severity} | {curr} | {change_symbol} |") lines.append("") # Final verdict overall_severity = review.get("overall_severity", "MEDIUM") recommendation = "Approved ✅" if overall_severity in ["LOW", "NONE"] else "Changes Requested" lines.append(f"**Overall Severity:** `{overall_severity}`") lines.append(f"**AI Recommendation:** {recommendation}") return "\n".join(lines) def _finding_key(self, finding: dict) -> str: """Create unique key for a finding.""" return f"{finding.get('file')}:{finding.get('line')}:{finding.get('rule_id', finding.get('category'))}" def _parse_review_comment(self, comment_text: str) -> list[dict]: """Parse previous review comment to extract findings.""" # Simple regex-based parsing of markdown tables and lists # Returns list of findings with file, line, severity, description findings = [] if not comment_text: return findings # Look for patterns like: **[HIGH]** `src/file.py:45` - Description import re pattern = r'\*\*\[(\w+)\]\*\*\s+`([^:]+):(\d+)`\s+-\s+(.+)' for match in re.finditer(pattern, comment_text): findings.append({ "severity": match.group(1), "file": match.group(2), "line": int(match.group(3)), "description": match.group(4).strip() }) return findings def _count_by_severity(self, findings: list[dict]) -> dict: """Count findings by severity level.""" counts = {"HIGH": 0, "MEDIUM": 0, "LOW": 0} for finding in findings: severity = finding.get("severity", "MEDIUM") counts[severity] = counts.get(severity, 0) + 1 return counts def _update_pr_labels(self, owner: str, repo: str, pr_number: int, review: dict): """Update PR labels based on review severity.""" overall_severity = review.get("overall_severity", "MEDIUM") # Remove old AI labels self.gitea.remove_issue_label(owner, repo, pr_number, "ai-approved") self.gitea.remove_issue_label(owner, repo, pr_number, "ai-changes-required") # Add new label if overall_severity in ["LOW", "NONE"]: self.gitea.add_issue_labels(owner, repo, pr_number, ["ai-approved"]) else: self.gitea.add_issue_labels(owner, repo, pr_number, ["ai-changes-required"]) ``` #### 3. **`.gitea/workflows/ai-comment-reply.yml`** Update workflow to handle PR comments: ```yaml name: AI Comment Reply on: issue_comment: types: [created] jobs: ai-reply: runs-on: ubuntu-latest if: contains(github.event.comment.body, '@codebot') || contains(github.event.comment.body, '@bartender') steps: - uses: actions/checkout@v4 - uses: actions/checkout@v4 with: repository: YourOrg/OpenRabbit path: .ai-review token: ${{ secrets.AI_REVIEW_TOKEN }} - uses: actions/setup-python@v5 with: python-version: "3.11" - run: pip install requests pyyaml - name: Run AI Comment Response env: AI_REVIEW_TOKEN: ${{ secrets.AI_REVIEW_TOKEN }} AI_REVIEW_REPO: ${{ gitea.repository }} AI_REVIEW_API_URL: https://your-gitea.example.com/api/v1 OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} SEARXNG_URL: ${{ secrets.SEARXNG_URL }} run: | cd .ai-review/tools/ai-review # Determine if this is an issue or PR if [ "${{ github.event.issue.pull_request }}" != "" ]; then # This is a PR comment python main.py pr-comment ${{ gitea.repository }} ${{ github.event.issue.number }} \ "${{ github.event.comment.body }}" else # This is an issue comment python main.py comment ${{ gitea.repository }} ${{ github.event.issue.number }} \ "${{ github.event.comment.body }}" fi ``` ## Alternatives considered ### Alternative 1: Automatic Re-review on Comment - **Rejected:** Too noisy - would re-run on every comment - Current manual trigger is more intentional ### Alternative 2: Time-based Re-review Example: Re-review PRs every 24 hours - **Rejected:** Wastes LLM tokens on inactive PRs - Manual trigger is more cost-effective ### Alternative 3: Webhook-based Trigger External service triggers review via API - **Rejected:** Adds complexity, requires infrastructure - Comment-based trigger is simpler ### Alternative 4: GitHub Actions Manual Dispatch Use `workflow_dispatch` event - **Considered for future:** Good for advanced users - Comment-based trigger is more accessible ### Alternative 5: Always Create New Comment Instead of updating existing review - **Rejected:** Clutters PR discussion - Updating comment keeps review consolidated ## Acceptance criteria - [ ] `@codebot review-again` command triggers on PR comments - [ ] Re-runs full PR review with current config - [ ] Updates existing AI review comment (doesn't create new) - [ ] Shows diff from previous review (resolved, new, still present) - [ ] Updates PR labels (`ai-approved`, `ai-changes-required`) - [ ] Works with security scanner (re-scans for vulnerabilities) - [ ] Respects latest `.ai-review.yml` configuration - [ ] Shows commit hash being reviewed - [ ] Response time < 30 seconds for typical PR - [ ] Handles case where no previous review exists - [ ] Works for both issue comments and PR review comments - [ ] Command is documented in README and `@codebot help` - [ ] Unit tests for review comparison logic - [ ] Integration test with actual PR **Backwards compatibility:** - ✅ No breaking changes - purely additive - ✅ Normal automatic reviews still work - ✅ Existing workflows unchanged ## Impact **Complexity:** **LOW-MEDIUM** - Estimated effort: 1-2 hours - Files modified: `pr_agent.py`, `config.yml`, workflow files - Reuses existing review logic - New logic: review comparison and diff formatting **User Impact:** **HIGH** - Faster feedback loops - No need for empty commits - Better developer experience - Enables configuration testing **Technical Impact:** - Minimal - reuses 90% of existing PR review code - New: previous review parsing and comparison - Safe operation (idempotent) **Cost Impact:** - Additional LLM API calls when command is used - **Mitigation:** Only runs on explicit request (not automatic) - Users control when to re-review ## Mockups / examples ### Example 1: Standard Re-review ```markdown **Pull Request #45: Add user authentication** Developer comment: > The hardcoded API key at line 45 is actually a public API URL, not a secret. > @codebot review-again Bot response: > @developer > > **🔄 Re-review Requested** > > Re-analyzing PR #45 at commit `abc1234`... > > --- > > ## AI Code Review (Updated) > > **Previous Review:** 3 issues (1 HIGH, 2 MEDIUM) > **Current Review:** 2 issues (0 HIGH, 2 MEDIUM) > > ### Changes from Previous Review > > **✅ Resolved (1):** > - **[HIGH]** `src/auth.py:45` - Hardcoded API key detected > (Confirmed: public API URL, not a secret) > > **Still Present (2):** > - **[MEDIUM]** `src/api.py:67` - Consider more specific error handling > - **[MEDIUM]** `src/config.py:12` - Use environment variables > > ### Summary > > | Severity | Count | Change | > |----------|-------|--------| > | HIGH | 0 | -1 ✅ | > | MEDIUM | 2 | 0 | > | LOW | 0 | 0 | > > **Overall Severity:** `MEDIUM` (was `HIGH`) > **AI Recommendation:** Approved ✅ > > **Status Update:** > - Removed label: `ai-changes-required` > - Added label: `ai-approved` ``` ### Example 2: After Config Change ```markdown Developer comment: > Updated .ai-review.yml to disable MD5 detection (we use it for checksums only) > @codebot review-again Bot response: > **🔄 Re-review Requested** > > Re-analyzing with updated configuration... > > ### Changes from Previous Review > > **✅ Resolved (2):** > - **[MEDIUM]** `src/utils.py:23` - Weak hash algorithm (MD5) > - **[MEDIUM]** `src/checksum.py:45` - Weak hash algorithm (MD5) > > **Configuration Changes Detected:** > - Disabled rule: SEC003 (Weak hash algorithms) > > **Current Review:** 0 issues > > **Overall Severity:** `NONE` > **AI Recommendation:** Approved ✅ ``` ### Example 3: New Issues Found ```markdown Developer comment: > Fixed the SQL injection. @codebot review-again Bot response: > **🔄 Re-review Requested** > > ### Changes from Previous Review > > **✅ Resolved (1):** > - **[HIGH]** `src/db.py:12` - SQL injection via string formatting > > **⚠️ New Issues (1):** > - **[MEDIUM]** `src/db.py:15` - Missing input validation on user_id parameter > > **Still Present (1):** > - **[LOW]** `src/utils.py:30` - Missing docstring > > ### Summary > > | Severity | Count | Change | > |----------|-------|--------| > | HIGH | 0 | -1 ✅ | > | MEDIUM | 1 | +1 ⚠️ | > | LOW | 1 | 0 | > > **Overall Severity:** `MEDIUM` > **AI Recommendation:** Changes Requested ``` ### Example 4: No Previous Review ```markdown Developer comment: > @codebot review-again Bot response: > **🔄 Re-review Requested** > > No previous review found. Running initial review... > > [Standard review output] ``` ### Example 5: No Changes ```markdown Developer comment: > @codebot review-again Bot response: > **🔄 Re-review Requested** > > Re-analyzed PR #45 at commit `abc1234` > > **No changes from previous review.** > > All findings remain the same: > - 1 HIGH severity issue > - 2 MEDIUM severity issues > > **Overall Severity:** `HIGH` > **AI Recommendation:** Changes Required ``` ## Implementation Plan ### Phase 1: Core Implementation (1 hour) 1. **Add Command Handler** (30 min) - Detect `review-again` in PR comments - Add to `PRAgent.execute()` routing - Call `_handle_review_again()` 2. **Review Comparison Logic** (30 min) - `_parse_review_comment()` - Extract previous findings - `_compare_reviews()` - Diff old vs new - `_finding_key()` - Create unique identifiers ### Phase 2: Formatting & Labels (30 min) 1. **Format Review Update** (15 min) - `_format_review_update()` - Template with diff - Show resolved, new, still present - Summary table with changes 2. **Label Management** (15 min) - `_update_pr_labels()` - Remove old, add new - Update based on new severity ### Phase 3: Testing & Documentation (30 min) 1. **Testing** (20 min) - Unit test review comparison - Test with/without previous review - Test label updates 2. **Documentation** (10 min) - Update README.md - Add to `@codebot help` - Update workflow examples ## Dependencies **Required:** - Existing PR review system (already implemented) - Comment parsing regex **Optional:** - None ## Related Issues/PRs - Related to: #XXX (@codebot help command) - Part of: Milestone 2 - UX & Discoverability - Enables: Faster PR iteration cycles - Foundation for: Configuration testing workflows ## Success Metrics **Week 1 after deployment:** - [ ] Used on 20%+ of PRs - [ ] Reduces empty "trigger review" commits by 90% - [ ] Average time to re-review < 30 seconds **Month 1 after deployment:** - [ ] 50%+ of developers aware of command - [ ] Positive feedback on iteration speed - [ ] Reduced PR cycle time by 15% **Long-term:** - [ ] Standard part of PR workflow - [ ] Enables config experimentation - [ ] Better AI review accuracy over time ## Future Enhancements **v2.0 Features:** - `@codebot review-again --full` - Re-review entire codebase context - `@codebot review-again --security-only` - Only re-run security scan - Scheduled re-reviews for long-lived PRs - Compare review with main branch baseline ## Checklist - [x] I searched existing feature requests and discussions - [x] This request includes user story and acceptance criteria - [x] Impact assessment completed (LOW-MEDIUM complexity, HIGH value) - [x] Implementation plan outlined - [x] Acceptance criteria defined - [x] Success metrics identified - [x] Example scenarios documented --- **Priority:** ⭐ **MEDIUM-HIGH** - Quick win for developer experience **Milestone:** Milestone 2 - UX & Discoverability **Estimated Effort:** 1-2 hours **Value:** HIGH **Complexity:** LOW-MEDIUM **Risk:** LOW **Key Features:** - 🔄 Re-run review without new commits - 📊 Show diff from previous review - 🏷️ Auto-update labels - ⚡ Fast iteration cycles - 🔧 Respects config changes
Latte added this to the Milestone 1: Core Features (Quick Wins) milestone 2025-12-28 17:52:16 +00:00
Latte added the Kind/Feature
Priority
High
2
 labels 2025-12-28 17:52:16 +00:00
Latte self-assigned this 2025-12-28 17:52:16 +00:00
Latte added this to the Development Roadmap project 2025-12-28 17:52:16 +00:00
Latte moved this to To Do in Development Roadmap on 2025-12-28 18:54:52 +00:00
Latte moved this to In Progress in Development Roadmap on 2025-12-28 19:02:44 +00:00
Latte added reference dev 2025-12-28 19:09:40 +00:00
Latte changed reference from dev to feature/review-again-command 2025-12-28 19:13:19 +00:00
Latte moved this to Done in Development Roadmap on 2025-12-28 19:15:43 +00:00
Latte closed this issue 2025-12-28 19:15:55 +00:00
Sign in to join this conversation.
feature/review-again-command
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Question
Further information is requested
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status/Ai Approved
AI review approved this PR
 Status/Ai Changes Required
AI review found issues requiring changes
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No Label Kind/Feature
Priority
High
2
Milestone
No items
No Milestone Milestone 1: Core Features (Quick Wins)
Projects
Clear projects
No project Development Roadmap
Assignees
Clear assignees
Bartender Latte
No Assignees Latte
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Hiddenden/openrabbit#7
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?