Automate public API diff reports for in-development .NET releases#10469
Open
jeffhandley wants to merge 8 commits into
Open
Automate public API diff reports for in-development .NET releases#10469jeffhandley wants to merge 8 commits into
jeffhandley wants to merge 8 commits into
Conversation
- Rename dispatcher agent .github/agents/agentic-workflows.agent.md to agentic-workflows.md and re-add the repo-specific PAT pool, schedule seed, and frontmatter ordering instructions - Add dispatcher skill .github/skills/agentic-workflows/SKILL.md - Recompile release-notes with v0.80.9 and add the generated agentics-maintenance.yml - Apply safe-outputs codemod in release-notes.md: required-title-prefix and required-labels for push-to-pull-request-branch - Bump pinned actions in .github/aw/actions-lock.json: actions/checkout v7.0.0, actions/setup-dotnet v5.4.0, github/gh-aw-actions/setup v0.80.9 - Update copilot-setup-steps.yml gh-aw setup-cli to v0.80.9 - Update validate-pat-pool.yml github/gh-aw-actions/setup pin to v0.80.9 (8c7d04e) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Make the api-diff skill pure report generation (RunApiDiff.ps1 only) - Add api-diff-validation skill owning the dotnet-inspect verification that confirms diffed APIs actually shipped (moved out of api-diff) - Light-touch update release-notes and generate-changes references to use api-diff for diffs and api-diff-validation for verification - Harden RunApiDiff.ps1 temp-folder cleanup with a bounded-retry recursive delete so a transient "directory not empty" race on Linux no longer aborts a multi-SDK diff run - Accept any "<label>.N" prerelease version in RunApiDiff.ps1 instead of only preview/rc/ga, keeping RC and GA specially cased and treating every other label (alpha, beta, ...) as a plain semantic-version label, so a diff can be generated against any published build. Updates the api-diff skill mapping and RunApiDiff.md accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
8e64de0 to
2032529
Compare
Deterministic discovery (no agent) driven entirely by the dnceng NuGet feeds that
decides which public API diffs to produce for the active, not-yet-shipped majors,
emitting one target per diff.
- api-diff-orchestrator.yml: a plain (non-agentic) orchestration workflow whose discover
job sets up the .NET SDK and runs the discovery app, capturing its JSON from a file
so the "dotnet run" build output on stdout cannot contaminate the result.
- api-diff-discover.cs: a C# file-based app using the NuGet client SDK (NuGet.Protocol
+ NuGet.Versioning) for SemVer-correct feed version discovery. maxGA is the highest GA
major on dotnet-public; only maxGA+1 and maxGA+2 are considered.
- major-to-major (cumulative): current build = newest dotnet{major} build of any
prerelease label; baseline = newest (major-1) build on dotnet-public that is RC-or-GA,
advancing RC1 -> RC2 -> GA. Identity is keyed on the major only and kept canonical
((major-1).0-ga_to_major.0) so one evergreen PR is reused as the head advances. Content
at the major root release-notes/major.0/major.0.0/api-diff.
- incremental (milestone deltas): previous public milestone -> latest public milestone,
both named public milestones (preview.N / rc.N) on dotnet-public. Content under
release-notes/major.0/preview/<milestone>/api-diff.
- Each target pins the exact previous/current package versions and their feeds.
- Status: maxGA+1 is code-complete once maxGA+2 has started publishing builds; otherwise
in-development. maxGA+2 is in-development.
- Retry feed queries so a transient blip is not read as "no build"; the maxGA+2 channel
probe is quiet since it does not exist until main forks to it.
- Fix RunApiDiff.ps1 string-sorting bugs where builds with 3-digit identifiers
(e.g., preview.1.100.1) sorted incorrectly relative to 2-digit ones. The build
suffixes are zero-padded during sort to ensure proper semantic ordering when
picking the latest build from feeds.
(Note: It is OK that we only see the first 100 files via `gh pr list --json files`
as api-diff PRs are focused to the files we're looking for.)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per-target worker (invoked by the orchestrator via workflow_call, or manually) that
generates a milestone's public API diff from real published builds and
opens/maintains its draft pull request. It never merges; humans merge.
- api-diff-orchestrator.yml: add the api-diff job that fans out over the discovered targets
and calls the worker as a reusable workflow (workflow_call) so it runs in the
orchestrator's run context and inherits the orchestrator's actor, satisfying the worker's
gh-aw activation role check. Runs the matrix with max-parallel 1, matching the
worker's shared concurrency group so a queued target is never dropped by
concurrency's pending-run-replacement rule.
- api-diff-generate.sh: host-side setup runs the api-diff skill's
RunApiDiff.ps1 against the target's pinned versions/feeds (current build from
the dotnet{major} channel feed, baseline from dotnet-public; the ApiDiff tool
from dotnet{major}-transport). Relocates major-to-major output into <mm>/<mm>.0
and gates on completeness: a non-zero RunApiDiff exit publishes nothing, and a
build that produced at least one SDK report (NETCore / AspNetCore /
WindowsDesktop) publishes those, since RunApiDiff fails outright when an SDK's
ref packs cannot be downloaded. Emits a per-run step summary covering every
outcome, including no-op.
- Identity + guards: find this diff's PR by matching a visible YAML marker as a
whole comment line in the body (so a cumulative marker cannot hijack an
incremental first-preview PR whose marker contains it as a prefix); resolve
candidates from the api-diff label unioned with any PR already on the target
branch (or collision branches starting with the target branch via search), so a
label a human stripped never causes a duplicate. Stand down when a non-automation
api-diff PR already owns the content_dir (verified via `gh pr view --json files`
to avoid API truncation limits).
- No-op / metadata-only: an unchanged build with the same status and the marker
present is a silent no-op; a status flip (draft->Ready) or a missing marker
refreshes the body without regeneration. Exclusion files are normalized to stay
newline-terminated so appends never concatenate onto the previous entry.
- api-diff-worker.md (compiled to .lock.yml): the agent commits the generated reports
inline (restricted to an api-diff report-tree allowlist, markdownlint-clean via
xargs to avoid ARG_MAX limits) and publishes through gh-aw native safe outputs --
create-pull-request / push-to-pull-request-branch open or append to the draft PR,
update-pull-request refreshes the title and body, add-comment posts one run
summary, mark-pull-request-as-ready-for-review flips it Ready when code-complete,
and noop covers the silent path. Every write is constrained to the automation +
api-diff labels and the file allowlist, and a post-step re-validates the
agent-written state block against the host-authored target.json (marker /
current-version / generated-at) before the run is accepted.
- Positions the working tree on the publish branch (detached at the base commit for
a fresh diff, or the maintained PR branch for an incremental update) over an
authenticated URL so the agent's commits land on the right ref despite the agent
job's single-branch, credential-less checkout.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Let a person hand-start an API diff PR that the automation then takes over and maintains. When no marker PR exists for a target, adopt an open PR with no identity marker yet, resolved in priority: our exact target branch is proof of ownership, so adopt a marker-less PR on it regardless of labels (a human may have stripped them or bootstrapped it there); on any other branch, only adopt a PR a human explicitly opted into automation for (both automation + api-diff) that already introduces reports under this target's content_dir. The worker writes the identity marker into the adopted PR's body on the first refresh and maintains it thereafter, exactly as if it had opened the PR itself. A PR labeled only api-diff (no automation) on another branch stays human-owned and still blocks the automation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend the worker to address reviewer feedback on the maintained draft PR on the next scheduled run, without triggering on review events. The agent collects the PR's submitted reviews and inline review comments itself through GitHub integrity filtering: tools.github sets min-integrity: approved, and features.integrity-reactions lets a maintainer promote an external contributor's comment to approved with an endorsement reaction. Only write-access and endorsed feedback reaches the agent, so it applies no host-side trust handling of its own. The host setup reads the PR body's recorded generated_at as the feedback watermark and captures this run's generated_at (its start time) in target.json, then computes a body-free, author-agnostic wake gate that counts review activity newer than the watermark so a caught-up build still wakes the agent for a metadata-only refresh when new review activity exists. The agent acts on feedback created after the watermark, folds in-scope exclusion requests into the same report update, rejects out-of-scope requests, resolves contradictions by recency, and advances generated_at to this run's start time so feedback is processed exactly once across runs. Because generated_at is captured before generation begins, review activity that arrives while a run executes carries a later timestamp and is picked up by the next run rather than skipped. Scope the recorded-state reads to a delimited block so human prose in the PR body cannot shadow them. current_version, generated_at, and status are read from a fenced yaml block the agent writes last in the body, delimited by the visible comment lines '# api-diff:state:begin' and '# api-diff:state:end'. tracking_block in generate.sh, and the publish guardrail, take the last begin..end range and match each delimiter as a whole comment line, so a quoted or bulleted field line, an older state block a maintainer pastes in as a note, or a '+ field:' line inside a diff report above the block no longer wins the first-match extraction and cannot force a false caught-up no-op or a bogus feedback watermark. The agent emits both delimiter lines and the guardrail requires them, scoping its current_version/generated_at validation to the block. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2032529 to
7f25209
Compare
Set the executable bit (100644 -> 100755) so super-linter's BASH_EXEC check passes. The API Diff Worker workflow still invokes it via `bash <path>`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an "API Diff" automation that keeps public API diff report PRs continuously
up to date for the active, not-yet-shipped .NET majors. A scheduled orchestrator
deterministically discovers which diffs should exist from the dnceng NuGet feeds
and dispatches a per-target worker that generates the reports from real published
builds and opens/maintains a draft PR for each one. The automation never
merges -- humans review and merge.
This PR adds only the automation and tooling. No API diff report content is
included; the reports are produced as draft PRs by the workflows after this
merges.
How it works
Discovery (deterministic, no agent).
api-diff-orchestrator.ymlis a plainworkflow whose discover job runs a C# file-based app,
api-diff-discover.cs,that uses the NuGet client SDK (NuGet.Protocol + NuGet.Versioning) for
SemVer-correct feed queries.
maxGAis the highest GA major on dotnet-public;only
maxGA+1andmaxGA+2are considered. It emits one target per diff, eachpinning the exact previous/current package versions and their feeds, and fans out
to the API Diff Worker workflow via
workflow_call(max-parallel 1, matching theworker's shared concurrency group so a queued target is never dropped).
Two tracks of diffs are produced:
dotnet{major}dev-feed buildof any prerelease label; baseline = newest
(major-1)RC-or-GA build ondotnet-public, advancing RC1 -> RC2 -> GA. Identity is keyed on the major and
kept canonical (
(major-1).0-ga_to_major.0) so one evergreen PR is reused asthe head advances. Content at
release-notes/<major>.0/<major>.0.0/api-diff.milestone's draft PR is produced as soon as its build exists (to merge
before the milestone releases); the baseline is pinned to the previous
milestone's shipped public build, so a milestone's diff holds off until its
baseline has shipped. Content under
release-notes/<major>.0/preview/<milestone>/api-diff.Generation + publishing (agentic worker). The API Diff Worker workflow,
api-diff-worker.md(compiled toapi-diff-worker.lock.yml), is the per-targetworker. Its host generation script,
api-diff-generate.sh, positions the workingtree on the publish branch and runs the api-diff skill's
RunApiDiff.ps1againstthe target's pinned versions/feeds, gating on completeness (a non-zero generation
exit publishes nothing). The read-only agent then commits the generated
reports and publishes entirely through gh-aw native safe outputs:
create-pull-request/push-to-pull-request-branchopen or append to the draftPR,
update-pull-requestrefreshes the title and body,add-commentposts a runsummary,
mark-pull-request-as-ready-for-reviewflips it Ready when the milestoneis code-complete, and
noopcovers the silent path. A post-step re-validates theagent-written state block against the host-authored target before anything is
published.
Safety and robustness
separate, permission-controlled safe-output job constrained to the
automationapi-difflabels and anallowed-filesallowlist scoped to the api-diffreport tree plus the two global exclusion files (
release-notes/ApiDiff*ToExclude.txt).Nothing outside that allowlist can be written; the default protected-file check
is kept, excluding only the per-folder
api-diff/README.mdindex the reportslegitimately produce.
runs them only via manual
workflow_dispatch, so forks never open spurious PRs.visible YAML state marker in the PR body so runs never create duplicates.
branch, or an opted-in
automation+api-diffPR) and the automation adoptsand maintains it thereafter. A PR labeled only
api-diffstays human-owned andblocks the automation for that content.
scheduled run using GitHub integrity filtering (min-integrity: approved), so
only write-access or maintainer-endorsed feedback reaches the agent; it never
triggers on review events and applies no trust handling of its own.
Commit walkthrough
dispatcher agent, adds its skill, and recompiles existing workflows, so this
commit also regenerates unrelated lock files (
release-notes.lock.yml,agentics-maintenance.yml, action pins). Tooling-only churn.becomes pure report generation (
RunApiDiff.ps1); a new api-diff-validationskill owns the dotnet-inspect verification.
RunApiDiff.ps1accepts any<label>.Nprerelease build and hardens temp-folder cleanup.(
api-diff-orchestrator.yml+api-diff-discover.cs) and the two-track model.api-diff-worker.md/
api-diff-worker.lock.yml) and its host generation script(
api-diff-generate.sh), publishing through gh-aw native safe outputs.PRs.
feedback handling and the delimited state block.
Two trailing commits satisfy super-linter on the host shell script: marking
api-diff-generate.shexecutable (bash-exec) and anshfmtformatting fix.Notes for reviewers
copilot-pat-poolenvironment per theexisting PAT pool mechanism.
from an empty net11 state it produces the major-to-major draft PR plus the full
preview.1 through preview.6 incremental draft PRs, correctly holding off
preview.7 until its baseline ships. The linked pull requests show the generated
draft PRs and the commit-by-commit progression as each build advanced.