Skip to content

Replace retired GitHub Models triage with agentic workflows - #3629

Open
JoannaaKL wants to merge 9 commits into
mainfrom
joannaakl-ai-triage-investigation
Open

JoannaaKL wants to merge 9 commits into
mainfrom
joannaakl-ai-triage-investigation

Conversation

@JoannaaKL

@JoannaaKL JoannaaKL commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Replaces the issue triage and spam moderation workflows, which stopped working when GitHub Models was retired, with GitHub Agentic Workflows that triage new issues and external pull requests.

Why

GitHub Models was retired on 2026-07-30. models.github.ai now answers every request with a plain 200 OK, so the old workflows fail silently while their runs show green. Even before that, the request ai review label only took effect when someone added another label by hand, since labels added with GITHUB_TOKEN don't trigger other workflows.

What changed

  • Removed issue-labeler.yml, ai-issue-assessment.yml, moderator.yml and .github/prompts/. They're already disabled in the repo, and the request ai review label is deleted.
  • Added issue-triage.md: on each new or reopened issue, a job checks in code whether the template was filled in, and the agent posts one of three fixed assessment comments and can add only one of waiting-for-reply, invalid or spam. Issues that don't use a template and have a body are left alone.
  • Added pr-triage.md and pr-triage-dispatch.yml: fork pull request runs can't use Copilot in a public repo, so an hourly job starts one triage run when an external pull request is opened or marked ready, at most 20 a day. A job checks the pull request against the template in code, and the agent labels and comments only when a check fails.
  • Both use safe-outputs.staged: true, so they only preview their output in the run summary. Remove it once the previews look right.

MCP impact

  • No tool or API changes
    Only GitHub Actions workflows changed.
  • Tool schema or behavior changed
  • New tool added

Prompts tested (tool changes only)

  • N/A

Security / limits

  • No security or limits impact
  • Auth / permissions considered
    Issue triage runs for issues from anyone (roles: all), but neither agent sees issue or pull request content: jobs check it in code and pass only the result. The agents have no shell and no GitHub tools, post fixed comment text, and can write only allowlisted labels and one comment. Pull request writes are pinned to the dispatched pull request. Copilot is billed to the org through copilot-requests: write, with no PAT.
  • Data exposure, filtering, or token/size limits considered
    Each run is capped at 100 AI credits, plus 100 for threat detection. Failure reports stay in the Actions logs instead of public issues.

Tool renaming

  • I am renaming tools as part of this PR (e.g. a part of a consolidation effort)
    • I have added the new tool aliases in deprecated_tool_aliases.go
  • I am not renaming tools as part of this PR

Note: if you're renaming tools, you must add the tool aliases. For more information on how to do so, please refer to the official docs.

Lint & tests

  • Linted locally with ./script/lint
  • Tested locally with ./script/test

Not run, since no Go code changed. Workflow checks:

  • gh aw compile issue-triage pr-triage (gh-aw v1.0.1, strict): passed, 0 warnings
  • gh aw compile --dry-run: gate passed, shellcheck clean
  • zizmor --offline --min-severity medium on the 3 workflows: no findings
  • actionlint on the 3 workflows: only false positives from its outdated schema (copilot-requests, concurrency.queue, job.workflow_ref)
  • Dispatcher script run against this repo with gh workflow run stubbed: picked the expected pull requests and stopped at the daily limit

Docs

  • Not needed
    Workflow-only change.
  • Updated (README / docs / examples)

JoannaaKL and others added 3 commits October 30, 2026 09:32
GitHub Models was retired on 2026-07-30. models.github.ai now returns a
plain 200 OK for every request, so the issue assessment and AI moderator
workflows fail silently while their runs report success. The moderator
action is also archived.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Triage new and reopened issues with GitHub Agentic Workflows. The agent
checks whether the issue template was filled in, posts one assessment
comment, and can only add bug, enhancement, waiting-for-reply, invalid
or spam. Safe outputs start staged, so runs only preview their output.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Fork pull request runs can't use Copilot in a public repository, so an
hourly job starts one triage run per new pull request from an external
contributor, at most 20 a day. Each run can only label and comment on
its own pull request. Safe outputs start staged.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI auto review requested due to automatic review settings October 30, 2026 08:33
@JoannaaKL
JoannaaKL requested a review from a team as a code owner October 30, 2026 08:33
Comment thread .github/workflows/pr-triage.md Resolved
Comment thread .github/workflows/issue-triage.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Pull requests marked ready more than seven days after creation are never triaged because the draft filter excludes them beforehand and the creation-date filter excludes them afterward.

1 open issue
What changed in this PR

Replaces retired GitHub Models automation with agentic workflows for issue and external pull request triage in the GitHub MCP Server repository. No MCP tools or APIs change.

Changes:

  • Removes legacy assessment, labeling, moderation, and prompt files.
  • Adds issue and PR triage, with hourly external-PR dispatch and daily/retry limits.
  • Keeps outputs in preview mode, with allowlisted labels, read-only agent access, and AI credit limits.
File Description
.github/​workflows/​pr-triage.md Defines PR template checks and spam assessment.
.github/​workflows/​pr-triage.lock.yml Adds the compiled, pinned PR triage workflow.
.github/​workflows/​pr-triage-dispatch.yml Dispatches eligible external PRs within run limits.
.github/​workflows/​moderator.yml Removes retired moderation automation.
.github/​workflows/​issue-triage.md Defines issue assessment, duplicate searches, and labeling.
.github/​workflows/​issue-labeler.yml Removes the legacy AI-review label workflow.
.github/​workflows/​ai-issue-assessment.yml Removes Models-based issue assessment.
.github/​prompts/​default-issue-review.prompt.yml Removes the legacy general-issue prompt.
.github/​prompts/​bug-report-review.prompt.yml Removes the legacy bug-report prompt.
.github/​aw/​actions-lock.json Pins the agentic workflow setup action.
.gitattributes Marks compiled workflow locks as generated files.

Review effort: Max (auto)

JoannaaKL and others added 2 commits October 30, 2026 11:44
Pull requests opened as drafts and marked ready more than seven days
later were never triaged. Find pull requests by when they became ready
instead of when they were created, and only count triage runs started
since then, so a pull request marked ready again is triaged again.

Only consider fork pull requests. Members with private org membership
can look external to the workflow token, which would have dispatched
triage for internal pull requests.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Issue and pull request content can try to steer the agent. Decide from
template structure, placeholder text, and other observable signals,
ignore instructions in the content, and post fixed comment text that
never repeats it. Pull request triage only acts when a check fails, and
no longer flags spam, which needs reading the content's meaning.

The agents also lose shell access and are limited to the GitHub tools
they need: issue_read and search_issues, or pull_request_read.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment thread .github/workflows/issue-triage.md Resolved
Comment thread .github/workflows/pr-triage.md Resolved
JoannaaKL and others added 2 commits October 30, 2026 12:10
The template checks no longer need judgment, so a job now runs them in
code and passes only the result and the failed section names to the
agent. The agent never reads pull request content and has no GitHub
tools, and it only runs when a check fails.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The agent now reads only the issue's title and body, adds at most one of
waiting-for-reply, invalid or spam, and posts one of three fixed comments
word for word. It no longer adds type labels, which the templates already
apply, or searches for duplicates, which fed it other issues' content.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment thread .github/workflows/issue-triage.md Resolved
Comment thread .github/workflows/issue-triage.md Resolved
JoannaaKL and others added 2 commits October 30, 2026 12:41
The issue triage rules relied on the agent following instructions, and
sent issue content, including anything sensitive, to the model. A job
now compares the issue with the templates in code, and the agent only
posts the result. Issues that partly use a template, or don't use one
and have a body, are left alone.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Fence lines counted as section content, so a section with only an empty
code block passed, and any fence line toggled code mode. Skip fence
lines and only close a fence with a matching one.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants