Add AI-powered build failure analysis with NuGet MCP for VMR insertion PRs

[YuliiaKovalova]

Summary

Adds AI-powered build failure analysis for the dotnet/dotnet VMR using a GitHub Actions agentic workflow — the same architecture as microsoft/testfx.

When a Maestro insertion PR fails the build, the AzDO pipeline automatically dispatches a GitHub Actions workflow where an AI agent (claude-opus-4.6) downloads the binlog, diagnoses the root cause, and posts a summary comment with inline suggestion blocks. The agent also has access to the NuGet MCP Server for resolving package version conflicts.

How it runs

Maestro insertion PR (darc-* branch)
  |
  v
AzDO: VMR build stages (existing)
  |
  +-- Succeeds -> nothing happens
  |
  +-- Fails -> TriggerBuildFailureAnalysis stage (new)
                |
                v  curl -> GitHub API workflow_dispatch
                |    (pr-number + azdo-build-id)
                v
              GitHub Actions: build-failure-analysis.md
                +-- Download failed job's BuildLogs from AzDO API
                +-- Select top-level VMR orchestration binlog
                +-- Dump errors via C# MCP client (ModelContextProtocol SDK)
                +-- Delegate to build-failure-analyst agent
                     +-- Root cause analysis (VMR-specific)
                     +-- NuGet conflict resolution via NuGet MCP Server
                     +-- PR comment with error table
                     +-- Inline suggestion blocks on diff lines

MCP Servers available to the agent

Server	Tools	Use case
binlog-mcp	binlog_overview, binlog_errors, binlog_warnings	Extract build errors from MSBuild binary logs
NuGet MCP Server	nuget_fix_vulnerable_packages, nuget_update-package, nuget_get-latest-package-version	Resolve NU1605/NU1608 version conflicts with concrete remediation plans

Trigger conditions

The AzDO stage runs only when:

• Public project (System.PullRequest.PullRequestNumber = GitHub PR number)
• PR trigger with darc-* or release-pr-* branch
• At least one build stage failed

Also available manually: gh workflow run build-failure-analysis.md -f pr-number=1234 -f azdo-build-id=5678

VMR-specific agent expertise

• Maestro/Darc insertion patterns (eng/Version.Details.xml, eng/Versions.props)
• Multi-pass builds (runtime pass 1/2, aspnetcore pass 1/2)
• Source-build compatibility, TFM mismatches
• NuGet conflict resolution via MCP (not just error reporting)
• When fixes belong in the upstream component repo vs dotnet/dotnet

Key design decisions

Decision	Rationale
AzDO -> GitHub dispatch	VMR builds in AzDO; analysis runs in GitHub Actions where gh-aw agents are available
Download binlogs	Reuses existing AzDO build artifacts instead of re-building the VMR
Failed job detection	Queries AzDO timeline API to find the failed job and download only its artifact
Top-level binlog	artifacts/log/Release/Build.binlog aggregates all errors across repos
C# MCP client	Uses ModelContextProtocol NuGet package natively (no Node.js)
NuGet MCP Server	Agent can resolve version conflicts, not just report them
Agentic workflow	Multi-turn reasoning with source file access and tool use

Files

File	Purpose
eng/pipelines/pr.yml	New TriggerBuildFailureAnalysis stage
.github/workflows/build-failure-analysis.md	Agentic workflow: download binlogs, dump, delegate
.github/workflows/shared/build-failure-analysis-shared.md	Shared agent delegation
.github/agents/build-failure-analyst.agent.md	VMR-specific agent with NuGet MCP integration
.github/workflows/scripts/DumpBinlog/	C# MCP client (ModelContextProtocol SDK)

Prerequisites

• GitHubCommentToken secret variable in public AzDO pipeline with issues:write, pull-requests:write, actions:write

Validated end-to-end

Tested on fork simulation PR using real AzDO build artifacts from failed builds. The agent successfully downloaded binlogs, analyzed errors, and posted diagnostic comments.

Prior art

• microsoft/testfx PR #8326 (original), agentic refactor PR #8389
• microsoft/testfx PR #8473 — NuGet MCP Server integration
• microsoft/testfx PR #8456 — feed update to dotnet-tools

[Copilot]

Pull request overview

Adds an Azure DevOps “Build Failure Analysis” stage intended to run on failed Maestro/Darc insertion PR builds, extract/aggregate MSBuild errors (preferably from binlogs), optionally analyze them with GitHub Models, and post a consolidated diagnostic comment plus inline fix suggestions back to the corresponding GitHub PR.

Changes:

• Introduces a new failure-only pipeline stage + step templates to download build logs/binlogs, extract/dedupe errors, run AI analysis, and post PR feedback.
• Adds Node-based helper scripts (MCP client, merge/dedupe, AI prompt/report generator, GitHub comment + review suggestion posters) plus a small unit-test script.
• Adds a reusable template for publishing binlogs/logs as pipeline artifacts and a GitHub Actions “simulation” workflow for end-to-end validation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
eng/pipelines/pr.yml	Wires the new Build Failure Analysis stage into the PR pipeline.
eng/pipelines/templates/stages/build-failure-analysis.yml	Defines the analysis stage, artifact download, and GitHub PR resolution.
eng/pipelines/templates/steps/build-failure-analysis.yml	Implements binlog discovery, extraction, fallback parsing, AI analysis, and PR posting steps.
eng/pipelines/templates/steps/publish-binlogs.yml	Adds a reusable step template for publishing binlogs/logs as pipeline artifacts.
eng/build-failure-analysis/scripts/package.json	Defines Node dependencies for the analysis scripts.
eng/build-failure-analysis/scripts/package-lock.json	Locks transitive dependencies for reproducible installs.
eng/build-failure-analysis/scripts/.gitignore	Ignores node_modules for the scripts directory.
eng/build-failure-analysis/scripts/extract-binlog-errors.js	MCP client to extract overview/errors/warnings from a binlog.
eng/build-failure-analysis/scripts/merge-errors.js	Merges and deduplicates errors across multiple binlog extractions.
eng/build-failure-analysis/scripts/analyze-errors.js	Builds the prompt, calls GitHub Models (with fallback), and generates a markdown report.
eng/build-failure-analysis/scripts/post-pr-comment.js	Posts/updates a single diagnostic PR comment (marker-based update-in-place).
eng/build-failure-analysis/scripts/post-suggestions-azdo.js	Posts inline fix suggestions via GitHub Reviews API using PR diff-line mapping.
eng/build-failure-analysis/scripts/test-utils.js	Simple unit tests for shared path/linkification + merge behavior.
.github/workflows/build-failure-analysis-simulation.yml	Adds a GitHub Actions workflow to simulate a failure and exercise the scripts.

Files not reviewed (1)

• eng/build-failure-analysis/scripts/package-lock.json: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[YuliiaKovalova]

the fresh output result: YuliiaKovalova#3 (comment)