feat(agent): add in-pipeline pre-PR self-review phase#263
Open
nizar-lahlali wants to merge 4 commits into
Open
feat(agent): add in-pipeline pre-PR self-review phase#263nizar-lahlali wants to merge 4 commits into
nizar-lahlali wants to merge 4 commits into
Conversation
Contributor
|
I think this is fine, just should be exposed using the new workflow system, a.k.a: add a method to do self review, and make it configurable as a step in the coding task workflow yaml file |
added 3 commits
June 22, 2026 14:34
Adds an optional self-review phase between agent execution and post-hooks where the LLM critiques its own cumulative diff before the PR is created. This improves first-pass PR quality by catching bugs, style issues, and test gaps before human review. - New self_review.py orchestration module with run_self_review() - New prompts/self_review.py with focused review prompt template - TaskConfig extended with self_review_enabled and self_review_max_turns - Fields threaded through build_config, get_config, server, pipeline - Fail-open: self-review errors never block PR creation - Uses remaining turns/budget from original allocation (capped) - Feature is opt-in (disabled by default)
After the self-review phase completes, the agent writes a structured summary of findings to `.self-review-summary.md`. The pipeline reads this file, posts it as a PR comment via `gh pr comment`, then deletes it so it never appears in the PR diff. Fail-open: if the file is missing or the comment fails to post, the pipeline continues normally.
Addresses review feedback: rather than a hardcoded inline pipeline phase
gated by a per-task API flag, self-review is now a first-class workflow
StepKind, configurable as a step in the coding workflow YAML.
- Add `self_review` StepKind to workflow.schema.json (with a `max_turns`
step field, 1-20, default 5) + models.Step; excluded from repo-less
workflows (schema conditional + validator rule 3) since it diffs/re-runs
the agent against the cloned tree.
- Register `_handle_self_review` in the runner's STEP_HANDLERS; it wraps
run_self_review and accumulates the review loop's turns/cost onto the
shared agent_result. Non-side-effecting, so it may use on_failure:
continue (advisory / fail-open).
- Drive the step from pipeline.run_task via `_execute_self_review_step`
(only_kinds={"self_review"}), mirroring _execute_agent_step. Runs after
the cancel short-circuit and before post-hooks; posts the summary PR
comment after ensure_pr when the step ran.
- Declare the step in coding/new-task-v1.yaml (opt-in by presence;
on_failure: continue). Remove it to disable; tune via max_turns.
- Drop the now-redundant self_review_enabled / self_review_max_turns
fields from TaskConfig, build_config, get_config, server params, and
run_task — configurability lives in the workflow YAML.
- Add golden validator-corpus fixtures (valid self_review step; repo-less
+ self_review rejected) to lock parity.
Agent quality green: ruff, ty (0 errors), 1129 tests, 79.87% coverage.
nizar-lahlali
force-pushed
the
feat/262-pre-pr-self-review
branch
from
19b763c to
3211230
Compare
Contributor
|
It might be semantics but should this be called a self review or should it be explicitly described as a isolated sub agent critic (aka adversarial review)? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
self_review.pyorchestration module withrun_self_review()that generates the diff, invokesrun_agent()a second time with a review-focused prompt, and accumulates turns/costprompts/self_review.pywith a focused review prompt template (correctness, bugs, security, style, test gaps checklist)TaskConfigextended withself_review_enabled(defaultFalse) andself_review_max_turns(default5)build_config(),get_config(),server.py_extract_invocation_params(), andpipeline.pyrun_task().self-review-summary.mdwith structured findings, the pipeline posts it viagh pr comment, then deletes the file so it never appears in the diffDesign decisions:
max_turnsallocation (capped atself_review_max_turns)run_agent()call gives the model fresh context and a clean review perspectivepr_review(read-only), empty diff, no remaining turns/budgetCloses #262
Test plan
tests/test_self_review.py) — includes 13 new tests for summary reader and PR comment postingSELF_REVIEW_ENABLED=truein local batch mode to verify the phase executes and summary is postedself_review_enabled: truein the orchestrator payload