Add build failure analysis workflow using binlog-mcp#8326
Merged
Conversation
Adds a GitHub Actions workflow that: - Triggers on PRs to build with binary logging - On failure, installs binlog-mcp from dotnet-eng feed - Extracts errors via MCP protocol (JSON-RPC over stdio) - Uses GitHub Models API for AI-powered root cause analysis - Posts diagnostic comment on the PR Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a GitHub Actions workflow to automatically extract MSBuild binlog errors on PR build failures, run an AI-assisted analysis, and post a single up-to-date diagnostic comment on the PR.
Changes:
- Added PR-triggered workflow to build with
--binaryLog, extract binlog errors viabinlog-mcp, and comment analysis on failures - Added a Node.js MCP stdio (JSON-RPC) client script to query
binlog-mcpfor overview/errors/warnings - Added logic to delete prior analysis comments to avoid PR comment spam
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
.github/workflows/build-failure-analysis.yml |
New workflow to run build + binlog extraction + AI analysis and post PR comment on failure |
.github/workflows/scripts/extract-binlog-errors.js |
New script implementing MCP Content-Length framing client to extract errors from binlog |
This will be reverted after verifying the workflow.
- set -o pipefail so build failure propagates through tee/tail pipe - Use file-based approach for passing analysis to comment step - Fix PR number reference for pull_request events Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The repo's NuGet.config has package-source-mapping which blocks --add-source. Use a temp nuget.config with just dotnet-eng + nuget.org. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add fallback error extraction from build output when MCP fails - Fix nuget.config for tool install (package source mapping conflict) - Improve Node.js MCP client with proper request tracking, timeouts - Don't pre-load binlog at startup to avoid timeout - Always post PR comment on failure, even without AI analysis - Verify tool can start before attempting MCP communication Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The bash sed approach produced invalid JSON with unescaped control characters from the build log. Use Node.js JSON.stringify instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace raw JSON-RPC with @modelcontextprotocol/sdk for reliable MCP protocol handling (fixes initialize timeout) - Try multiple AI endpoints (models.github.ai, models.inference.ai.azure.com) - Improve fallback: group errors by file with structured markdown - Remove broken --help check (tool is MCP server, not CLI) - Re-introduce build error for testing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The binlog-mcp tool returns errors as JSON array with file, line, code, message fields. Parse this properly and format as grouped markdown with emojis, deduplication, and collapsible warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Extract source context from binlog via binlog_files MCP tool - Generate fix suggestions based on error code (CS0246, CS0234, etc.) - Show source context with expandable details block - Include source context in AI prompt for better AI-generated fixes - Re-introduce build error for testing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Binlogs don't embed .cs source files. Read them directly from the checked-out workspace to provide source context for fix suggestions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Commit package.json + lockfile, use npm ci for deterministic installs - Move tool version to env var (BINLOG_MCP_VERSION) for easy updates - Fix basename collision in link post-processing (escape all regex metacharacters) - Skip workflow entirely for fork PRs (read-only token cannot post comments) - Unify toRelPath implementations (both now handle Windows + Linux paths) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Move dynamic imports inside try/catch in extract script - Only post review suggestions on lines in the PR diff (avoids 422) - Clean up package.json (remove failing test script, add private flag) - Guard client.close() against null client Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AITools.BinlogMcp is only on the dotnet-eng feed and the package name is not prefix-reserved on nuget.org. Including nuget.org as a source creates a dependency confusion attack vector. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 5 changed files in this pull request and generated 3 comments.
Files not reviewed (1)
- .github/workflows/scripts/package-lock.json: Language not supported
Comments suppressed due to low confidence (1)
.github/workflows/build-failure-analysis.yml:156
Analyze errors with AIassumes/tmp/binlog-analysis.jsonis always valid JSON (JSON.parse(fs.readFileSync(...))) and will throw if the file is missing or malformed, which would skip posting any diagnostic comment even though the build failed. Wrapping this parse in a try/catch and falling back to a minimal message (or reusing/tmp/build-output.log) would make the workflow more resilient.
const fs = require('fs');
const path = require('path');
const data = JSON.parse(fs.readFileSync(process.env.BINLOG_DATA, 'utf8'));
const workspace = process.env.GITHUB_WORKSPACE_PATH;
const headSha = process.env.PR_HEAD_SHA;
- Regenerate package-lock.json to match renamed package.json - Validate MCP output is valid JSON before marking as extracted - Fallback now outputs errors as JSON array (matching MCP format) so source context and inline suggestions work in fallback path Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of a switch/case on error codes, ask the LLM to generate exact replacement lines for each error. The LLM sees the source context and error details, and returns a JSON array of fixes that are posted as GitHub suggestion blocks with one-click commit. Also handles declaration-side fixes: when callers break due to a signature change in the PR, the LLM suggests fixes on the declaration line (e.g., making new params optional). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Simplify the workflow from 820 lines to 181 by extracting logic: - analyze-errors.js: AI analysis, source context, fallback formatting - post-suggestions.js: LLM-generated inline fix suggestions - extract-binlog-errors.js: MCP client (unchanged) The workflow YAML is now pure orchestration — no inline JavaScript. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
🔍 Build Failure Analysis1. Summary of FailureThe build failed due to multiple instances of the same error: a required argument for the 2. Root Causes
3. Concrete Code FixTo resolve the missing parameter issue, update the method call in --- Engine/ThreadPoolTestNodeRunner.cs
+++ Engine/ThreadPoolTestNodeRunner.cs
@@ -204,7 +204,7 @@
timesheet.RecordStart();
// Replace with the actual test display name variable
- timesheet.RecordStart();
+ timesheet.RecordStart(testNode.DisplayName); // Assuming testNode has a DisplayName property
PlatformTestNode platformTestNode = testNode.ToPlatformTestNode();Make sure to replace 🤖 Generated using binlog-mcp · commit f44116a · workflow run |
- Only delete comments from github-actions[bot] (not human comments) - Ensure extract script always outputs valid JSON array for errors - Revert test error Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Evangelink
approved these changes
May 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a GitHub Actions workflow that automatically analyzes build failures on PRs using AITools.BinlogMcp and posts diagnostic comments with suggested fixes.
How it works
./build.sh --binaryLogto produce an MSBuild binlogbinlog-mcpvia MCP protocol to get structured error data (overview, errors, warnings)Files
build-failure-analysis.ymlscripts/extract-binlog-errors.jsscripts/analyze-errors.jsscripts/post-suggestions.jsscripts/package.json+ lockfile@modelcontextprotocol/sdkdependencyFeatures
GITHUB_TOKENcleared during build step, skipped for fork PRs