Skip to content

ci(ai-review): move a PR back to draft after three failing rounds - #427

Merged
swkeever merged 1 commit into
mainfrom
skeever/ai-review-hand-off
Nov 13, 2026
Merged

swkeever merged 1 commit into
mainfrom
skeever/ai-review-hand-off

Conversation

@swkeever

Copy link
Copy Markdown
Collaborator

Why

After three failing AI review rounds in a row, the AI review is supposed to move the PR back to draft, so the author re-assesses the approach instead of pushing another fix. Until now it could not. GitHub converts a PR to draft only for a GITHUB_TOKEN with contents: write, and the review's verdict job has contents: read. The conversion failed with "Resource not accessible by integration", and the review kept reviewing each push. Kong/volcano-hosting#3396 hit this.

A probe confirmed the permission: contents: read + pull-requests: write is refused (FORBIDDEN), and contents: write + pull-requests: write converts the PR (run).

What

Opts this repository in to the hand-off that Kong/volcano-actions#49 added, as its README's caller stub shows, and as Kong/volcano-hosting#3407 did:

  • ai-review passes hand_off: true, so the verdict job leaves the hand-off to the new job.
  • A new hand-off job calls ai-review-hand-off.yml, and only when the review's hand-off output is 'true'. It is the only job with contents: write (plus pull-requests: write and statuses: read). It never checks out the PR, and runs only the action's handoff step. That step acts only on a PR that is open, ready, at the reviewed head, and failing in the current set of rounds. Fork PRs never reach it, because the review skips them.
  • If GitHub still refuses the conversion, the job posts the re-assess comment and fails, naming the permissions it needs.

Marking the PR ready for review again starts a new set of three rounds.

Verification

actionlint on the changed workflow. A PR's own review runs the base branch's stub, so the new job first runs after merge, on the next PR that fails three rounds in a row.

🤖 Generated with Claude Code

@swkeever
swkeever requested a review from a team as a code owner November 13, 2026 17:04
hand-off:
needs: ai-review
if: ${{ !cancelled() && needs.ai-review.outputs.hand-off == 'true' }}
uses: Kong/volcano-actions/.github/workflows/ai-review-hand-off.yml@main
@github-actions

Copy link
Copy Markdown

AI review: no P0-P2 findings

Reviewed 420412d. Run log

Opts the AI Review workflow in to the shared hand-off: ai-review now passes hand_off: true, and a new hand-off job calls Kong/volcano-actions/.github/workflows/ai-review-hand-off.yml@​main with contents: write, pull-requests: write, and statuses: read. That job runs only when the review reports that a PR should go back to draft. The diff is small and matches the description. The main risk is the new contents: write token in a pull_request_target workflow, and that token goes to code at another repository's mutable @​main ref.

⚪ P3

  • .github/workflows/ai-review-retarget-override.yml:13 Retarget/override caller of ai-review.yml still runs without the hand-off

Not checked: The AI review read the diff, the code, and the PR's text; it ran no tests or builds and read nothing outside this repository.

  • Whether Kong/volcano-actions ai-review.yml@​main accepts a hand_off input and exposes a hand-off output. Without them, the caller fails validation or the hand-off job silently never runs.
  • Whether Kong/volcano-actions ai-review-hand-off.yml@​main needs only the three permissions granted here and no secrets, and whether it never checks out or runs PR code while holding the write token.
  • Whether the draft conversion works end to end. It first runs after merge, on a later PR that fails three rounds in a row.
  • Verify that the Kong/volcano-actions reusable workflows provide the matching hand_off input and hand-off output and enforce the stated PR eligibility checks.

Human review: needed

  • Should the hand-off job in .github/workflows/ai-review.yml give a contents: write token on pull_request_target to Kong/volcano-actions code at the mutable @​main ref?
  • Should .github/workflows/ai-review.yml grant contents: write to the Kong/volcano-actions hand-off workflow so it can return repeatedly failing PRs to draft?

The AI review found no P0-P2 problems at 420412d, so the code needs no second review for bugs. Decide the points above: approve to accept them, or request changes.

Any collaborator with write access other than the author can approve.

@shawnwu-kong shawnwu-kong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guided review at 420412d, posted by an agent on Shawn's delegation, following the brief's recommendations. The AI review's two human-review reasons ask the same question, and it is accepted.

  1. Grant contents: write to the hand-off job at @main (marker reasons 1 and 2) — Accept. No job with a write token checks out or runs PR code. The hand-off workflow runs only the SHA-pinned action's handoff step and gets no secrets (volcano-actions ai-review-hand-off.yml:25-37 @ f87f40c). Probe run 43455029391 shows contents: write is the least GitHub accepts for convertPullRequestToDraft. @main matches this repo's four existing AI review references, which already have open CodeQL unpinned-tag alerts #7, #8, #11 and #12, so pinning belongs in a repo-wide change. The main ruleset keeps the token from pushing to main. On volcano-hosting the same code moved #3275 to draft after three failing rounds, and marking it ready restarted the count.

Not checked:

  • the hand-off job's first real run in this repo (it runs only after merge; the same code already moved PRs to draft on volcano-hosting);
  • whether a narrower credential than contents: write, such as a GitHub App token, could move a PR to draft.

Generated by the guided-review skill. Install it with one command:

sh -c 'd=$(mktemp -d) && gh repo clone Kong/volcano-conventions "$d" -- --depth 1 -q && make -C "$d" install-skills guided-review pr-review explain-pr; s=$?; rm -rf "$d"; exit $s'

Then ask your coding agent for a guided review of a PR or issue URL. Run the command again to update.

@swkeever
swkeever added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit 7205a2e Nov 13, 2026
100 checks passed
@swkeever
swkeever deleted the skeever/ai-review-hand-off branch November 13, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants