docs: competitive gap analysis for CodeRabbit-class replacement - #10
docs: competitive gap analysis for CodeRabbit-class replacement#10jaibhasin wants to merge 10 commits into
Conversation
Capture codebase and market research on what PRBot should improve next to build replacement confidence: evals, multi-pass recall, walkthrough UX, and operational gaps versus CodeRabbit and Bugbot. Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
|
Warning Review limit reached
Next review available in: 45 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (35)
📝 WalkthroughWalkthroughThe PR adds research and planning documentation covering PRBot’s competitive gaps, prioritized improvements, implementation sequencing, evaluation gates, review-depth scaling, multi-pass review, permission policy, retries, walkthroughs, and resolution-rate tracking. ChangesReview strategy and implementation roadmap
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Map each priority item onto concrete files, algorithms, config knobs, tests, and small-commit sequencing against the current codebase. Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/research/competitive_gap_analysis.md`:
- Around line 69-81: The comparison claims in
docs/research/competitive_gap_analysis.md lines 69-81 need direct source links
for the 2026 industry comparisons; add authoritative links supporting each
applicable row, or explicitly label unsupported table content as author
synthesis. In docs/research/future_checklist.md lines 33-40, add direct sources
for the CodeRabbit capability claims and keep the existing bullet scope aligned
with those linked sources if broader support is unavailable.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7cb14bbe-c942-46fb-8b4e-38b287ea7944
📒 Files selected for processing (3)
docs/research/competitive_gap_analysis.mddocs/research/future_checklist.mddocs/research/implementation_plan_do_now.md
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/research/implementation_plan_do_now.md`:
- Around line 363-367: Update the “Diversified passes (deterministic)” heading
to describe only reproducible pass ordering, since passes 1 and 2 use nonzero
temperatures; alternatively, set all listed pass temperatures to 0.0 if
deterministic model output is required.
- Around line 379-388: Update the merge quorum rule in the pass-merging logic so
majority_k represents a true majority, computing it as floor(primary_passes / 2)
+ 1 rather than defaulting to 2 for two or more passes. Preserve the existing
singleton, representative-selection, candidate-cap, and failure-policy behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a9b20ad2-79f5-43d6-9913-778d494f4273
📒 Files selected for processing (3)
docs/research/competitive_gap_analysis.mddocs/research/future_checklist.mddocs/research/implementation_plan_do_now.md
🚧 Files skipped from review as they are similar to previous changes (1)
- docs/research/future_checklist.md
| Diversified passes (deterministic): | ||
|
|
||
| 1. Pass 0: current order, temperature `0.0`, full review | ||
| 2. Pass 1: reversed file order, correctness/reliability lens, temperature `0.1` | ||
| 3. Pass 2: high-risk bundles first, security/concurrency/API lens, temperature `0.2` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not call nonzero-temperature passes deterministic.
Passes 1 and 2 use temperatures 0.1 and 0.2, so their model outputs remain stochastic. Rename this to “reproducible pass ordering” or set all temperatures to 0.0 if deterministic output is required.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/research/implementation_plan_do_now.md` around lines 363 - 367, Update
the “Diversified passes (deterministic)” heading to describe only reproducible
pass ordering, since passes 1 and 2 use nonzero temperatures; alternatively, set
all listed pass temperatures to 0.0 if deterministic model output is required.
| Merge rules: | ||
|
|
||
| - Keep if support >= `majority_k` (default 2 when passes >= 2) | ||
| - Or singleton with confidence >= ~0.92 and priority in `{P0,P1}` | ||
| - Representative = highest confidence, then highest priority, then richest body/evidence | ||
| - Cap merged candidates before verifier (for example `max_comments * 3`) | ||
|
|
||
| Then call existing `verifier::verify_findings` unchanged. | ||
|
|
||
| Failure policy: a failed pass contributes empty findings; only mark primary failed if all passes fail. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Define a true majority quorum.
When primary_passes = 2, majority_k = 2 requires unanimity rather than a majority. Either compute the quorum as floor(passes / 2) + 1 or rename the setting to clarify that two-pass mode intentionally requires unanimous support.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/research/implementation_plan_do_now.md` around lines 379 - 388, Update
the merge quorum rule in the pass-merging logic so majority_k represents a true
majority, computing it as floor(primary_passes / 2) + 1 rather than defaulting
to 2 for two or more passes. Preserve the existing singleton,
representative-selection, candidate-cap, and failure-policy behavior.
Clarify that the shipped product uses one primary reviewer plus an independent verifier. Specialist routing remains eval/future work. Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
Mirror GitHub client retry behavior for transient provider failures. Reserve budget once per logical completion and release the semaphore between attempts so multipass work can proceed under rate limits. Also plumb AgentCall.temperature for upcoming diversified passes. Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
Allow admin, maintain, or write as the review/command permission floor. Default remains admin for backward compatibility. Also add primary_passes and related ReviewConfig knobs used by upcoming depth/multipass work. Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
Plan review effort from bundle risk, diversify optional primary passes, and majority-merge candidates before the existing verifier. Default primary_passes remains 1 so behavior stays conservative until evals justify enabling more passes. Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
Generate a soft-fail human walkthrough for the summary comment and track finding resolution across incremental fingerprint invalidation. Summary state moves to version 4 with published/resolved fingerprint sets. Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
Preserve adjudicated catalog rows when regenerating stubs, add a draft --eval-json batch runner, and tighten CI checks around the fixture flow. Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
Co-authored-by: Jai Bhasin <jaibhasin@users.noreply.github.com>
|
@codex please analyze the codechanges made in the repo, check if they were needed or not , see if u can improve them. Also check for other structures of the repo as u are reviewing. Make sure docs are updates, concise, not ambigious, can be easily understood and updated |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fdd7aca0ba
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| cluster.candidates.iter().any(|finding| { | ||
| finding.confidence >= keep_high_confidence_singleton | ||
| && matches!(finding.priority, Priority::P0 | Priority::P1) | ||
| }) |
There was a problem hiding this comment.
With the shipped defaults primary_passes=1 and majority_k=2, every cluster has support 1, so this condition retains only P0/P1 findings with confidence at least 0.92. All P2/P3 findings and lower-confidence P0/P1 findings are discarded before independent verification, allowing the default review path to report clean despite actionable candidates; accept all candidates when only one pass executed, or derive the threshold from the actual pass count.
AGENTS.md reference: AGENTS.md:L25-L28
Useful? React with 👍 / 👎.
| let min_permission = CollaboratorPermission::parse(&args.min_permission)?; | ||
| if !eval_json && !github.has_min_permission(&actor, min_permission).await? { |
There was a problem hiding this comment.
When a repository uses the documented .prbot.toml setting min_permission = "write", this check still uses the CLI/Action argument, which defaults to admin, because the trusted repository configuration is loaded later by prepare_snapshot. Write and maintain collaborators are therefore rejected before that configuration can be read, while a stricter repository value likewise cannot tighten a permissive Action input; load the trusted permission setting before authorizing or remove repository-level support.
Useful? React with 👍 / 👎.
| # Eval payload is the last JSON object printed. | ||
| lines = [line for line in completed.stdout.splitlines() if line.strip().startswith("{")] | ||
| if not lines: | ||
| raise SystemExit(f"no JSON payload for {case['case_id']}: {completed.stdout}") | ||
| return json.loads(lines[-1]) |
There was a problem hiding this comment.
--eval-json prints its payload with serde_json::to_string_pretty, so even an empty result spans multiple lines and this filter normally captures only the standalone opening {; with findings, it can instead select a nested object's opening {. Consequently json.loads(lines[-1]) raises JSONDecodeError for every ready fixture, preventing this runner from producing release-gate drafts; decode the complete trailing JSON document rather than a single line.
AGENTS.md reference: AGENTS.md:L45-L45
Useful? React with 👍 / 👎.
| #[serde(default)] | ||
| pub published_fingerprints: BTreeSet<String>, | ||
| #[serde(default)] | ||
| pub resolved_fingerprints: BTreeSet<String>, |
There was a problem hiding this comment.
Existing v3 summary comments deserialize these new sets as empty even though fingerprints contains previously published active findings. If the first incremental v4 review fixes one of those findings, forget_paths returns its fingerprint but resolve_forgotten refuses to count it because it is absent from published_fingerprints, so pre-upgrade findings and their resolutions permanently disappear from the reported rate; migrate v3 state by seeding publication history from the active fingerprints.
Useful? React with 👍 / 👎.
| state: &SummaryState, | ||
| review_model: &str, | ||
| verification_model: &str, | ||
| walkthrough: Option<&str>, |
There was a problem hiding this comment.
After adding this required argument, the immediately preceding Rustdoc example still calls render_summary with seven arguments and the parameter list does not describe walkthrough. Because the example is marked ignore, documentation tests will not expose the stale invocation, and readers copying it receive code that does not compile; add the argument and document its behavior.
Useful? React with 👍 / 👎.
|
@codex please commit the changes |
Summary
Checks
|
|
@codex Please implement the changes you have suggested. |
Summary
Testing
|
2 participants
Summary
Implements the do-now improvements from the competitive gap analysis, plus the research docs that scoped the work.
Shipped
min_permission(admindefault;maintain/writesupported via Action/CLI/.prbot.toml)DepthPlanfor passes/steps)primary_passes=1)SummaryStatev4run_fixture_batch.py(human adjudication still required for hard gate)Defaults stay conservative
primary_passes=1min_permission=adminTest plan
cargo testcargo clippy --all-targets -- -D warningspython3 scripts/evaluate.py evals/sample.jsonl --allow-small-samplepython3 scripts/run_fixture_batch.py --helpSummary by CodeRabbit