Add a Vally-based evaluation harness for repository-owned Copilot customizations. - #39002
Draft
AndriySvyryd wants to merge 1 commit into
Draft
AndriySvyryd wants to merge 1 commit into
AndriySvyryd wants to merge 1 commit into
Conversation
…tomizations. Discovers and validates instructions, skills, agents, prompts, and agentic workflows by convention. Runs paired treatment and unskilled-control evaluations with token grading and Vally comparison. Selectively evaluates changed components and publishes progressive PR results. Restricts secret-backed runs to trusted PR authors; later commits require manual reruns. Replace repository MCP configuration and duplicated skills with plugins. Update skill guidance and add matching repository-grounded evaluations. Add Dependabot coverage for Vally dependencies.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Secret-bearing PR-head execution and control-isolation defects must be addressed before approval.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds a Vally-based harness to validate and compare repository-owned Copilot customizations.
Changes:
- Adds discovery, validation, preparation, evaluation, and comparison tooling.
- Adds CI workflows and evaluations for repository skills and instructions.
- Replaces local MCP configuration and duplicated testing guidance with plugins.
File summaries
| File | Description |
|---|---|
eng/harness-evaluation/test/harness.test.mjs |
Tests harness behavior. |
eng/harness-evaluation/src/harness.mjs |
Implements harness logic. |
eng/harness-evaluation/src/cli.mjs |
Adds harness CLI. |
eng/harness-evaluation/README.md |
Documents the harness. |
eng/harness-evaluation/package.json |
Defines Node dependencies and scripts. |
eng/harness-evaluation/package-lock.json |
Locks evaluation dependencies. |
eng/harness-evaluation/instructions/copilot-instructions/eval.yaml |
Evaluates repository instructions. |
eng/harness-evaluation/skills/change-tracking/eval.yaml |
Evaluates change-tracking guidance. |
eng/harness-evaluation/skills/cosmos-provider/eval.yaml |
Evaluates Cosmos guidance. |
eng/harness-evaluation/skills/make-custom-agent/eval.yaml |
Evaluates agent authoring. |
eng/harness-evaluation/skills/make-github-actions-workflow/eval.yaml |
Evaluates workflow authoring. |
eng/harness-evaluation/skills/make-instructions/eval.yaml |
Evaluates instruction authoring. |
eng/harness-evaluation/skills/make-skill/eval.yaml |
Evaluates skill authoring. |
eng/harness-evaluation/skills/migrations/eval.yaml |
Evaluates migrations guidance. |
eng/harness-evaluation/skills/model-building/eval.yaml |
Evaluates model-building guidance. |
eng/harness-evaluation/skills/query-pipeline/eval.yaml |
Evaluates query guidance. |
eng/harness-evaluation/skills/run-apichief/eval.yaml |
Evaluates ApiChief guidance. |
eng/harness-evaluation/skills/scaffolding/eval.yaml |
Evaluates scaffolding guidance. |
eng/harness-evaluation/skills/servicing-pr/eval.yaml |
Evaluates servicing guidance. |
eng/harness-evaluation/skills/sqlite-adonet/eval.yaml |
Evaluates SQLite guidance. |
eng/harness-evaluation/skills/tooling/eval.yaml |
Evaluates tooling guidance. |
eng/harness-evaluation/skills/triage/eval.yaml |
Evaluates issue triage. |
eng/harness-evaluation/skills/update-pipeline/eval.yaml |
Evaluates update-pipeline guidance. |
.github/workflows/agent-harness-evaluation.yml |
Runs behavioral evaluations. |
.github/workflows/agent-harness-coverage.yml |
Validates harness coverage. |
.github/dependabot.yml |
Adds npm update coverage. |
.github/copilot/settings.json |
Enables Copilot plugins. |
.github/copilot-instructions.md |
Centralizes testing guidance. |
.claude/settings.json |
Enables corresponding Claude plugins. |
.vscode/mcp.json |
Removes duplicated MCP configuration. |
.agents/skills/testing/SKILL.md |
Removes duplicated testing skill. |
.agents/skills/sqlite-adonet/SKILL.md |
Clarifies command lifecycle. |
.agents/skills/servicing-pr/SKILL.md |
Clarifies quirk recommendations. |
.agents/skills/run-apichief/SKILL.md |
Updates repository references. |
.agents/skills/migrations/SKILL.md |
Expands migration guidance. |
.agents/skills/make-skill/SKILL.md |
Adds paired-evaluation workflow. |
.agents/skills/make-skill/references/testing-patterns.md |
Replaces Waza with Vally guidance. |
.agents/skills/make-instructions/SKILL.md |
Adds instruction evaluation guidance. |
.agents/skills/make-custom-agent/SKILL.md |
Updates related-skill references. |
.agents/skills/change-tracking/SKILL.md |
Refines change-tracking guidance. |
Review details
Files not reviewed (1)
- eng/harness-evaluation/package-lock.json: Generated file
- Files reviewed: 39/40 changed files
- Comments generated: 6
- Review effort level: Balanced
Comment on lines
+167
to
+170
| - name: Check out pull request | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 | ||
| with: | ||
| ref: ${{ needs.authorize.outputs.head_sha }} |
| "private": true, | ||
| "type": "module", | ||
| "engines": { | ||
| "node": ">=22" |
Comment on lines
+17
to
+20
| test/EFCore.Relational.Tests/Update/UpdateSqlGeneratorTest.cs. Write analysis.md for a regression where a dependent row | ||
| is inserted before its principal and an UPDATE omits its concurrency predicate. Identify the owning stages and focused | ||
| tests in these staged files, including the expected `DbUpdateConcurrencyException`. Explain how the fix preserves | ||
| transaction boundaries and store-generated-value propagation. Do not edit source. |
Comment on lines
+347
to
+352
| const environment = spec.agent_environment ?? {}; | ||
| const files = Array.isArray(environment.files) ? [...environment.files] : []; | ||
|
|
||
| for (const stimulus of spec.stimuli ?? []) { | ||
| const stimulusEnvironment = stimulus.agent_environment ?? {}; | ||
| const stimulusFiles = Array.isArray(stimulusEnvironment.files) ? [...stimulusEnvironment.files] : []; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace repository MCP configuration and duplicated skills with plugins.
Update skill guidance and add matching repository-grounded evaluations.
Add Dependabot coverage for Vally dependencies.