[jongio]

Builds on #4 (ronniegeraghty/maintenance-plugins) to support skill improvement and PR review reporting workflows:

1. Multi-skill Cohort (plugins/pr-description & evals/pr-description):
   • Adds plugins/pr-description (skills/pr-description/SKILL.md) and evals/pr-description/skills/pr-description/{eval.yaml,grader.mjs} staged alongside the other 5 installed marketplace skills (add-plugin, try-plugin-locally, submit-plugin, update-remote-pins, release-plugins).
   • Adds evals/_shared/fixtures/pr-description/ (SKILL.verbose-fixture.md, baseline-results.jsonl, candidate-results.jsonl) demonstrating both fixing a quality failure and trimming skill verbosity (-52.6% tokens at parity).
2. Maintainer PR Report (.github/scripts/eval-report.mjs):
   • Adds npm run eval:report and npm run eval:compare -- --baseline ... --candidate ... to summarize per-case and aggregate scores, input/output/total tokens, durations, and baseline-vs-candidate transitions (Fixed, Regressed, Reduced tokens at parity).

Validation

• npm run lint: Passed (6 skills linted, 7 eval specs valid, local catalog matches manifests)
• node --test .github/scripts/*.test.mjs: Passed (551/551)

[Copilot]

🟡 Changes recommended

Eval reporting can misclassify quality regressions, execution errors, and incomplete measurements as token savings, while grader checks can accept failed verification and prohibited namespaced commands.

5 open issues

• Incomplete metric coverage produces unsupported aggregate savings · New
• Execution errors can be misreported as token savings · New
• Repeated trials overwrite each other in comparison maps · New
• Test count can incorrectly satisfy the passing-outcome assertion · New
• Prohibited tool matching misses namespaced tool names · New

What changed in this PR

Adds a standalone PR-description plugin to the marketplace and extends the eval tooling with multi-skill routing checks and saved-run comparisons.

Changes:

• Registers and documents the new plugin for drafting factual, structured PR descriptions.
• Adds cohort evals, a contract grader, and synthetic baseline/candidate fixtures.
• Adds Markdown reports for scores, tokens, durations, and baseline transitions, with workflow integration and regression tests.

File	Description
README.md	Adds the plugin listing and install command.
plugins/​pr-description/​skills/​pr-description/​SKILL.md	Defines drafting format, scope, and verification requirements.
plugins/​pr-description/​README.md	Documents installation and example usage.
plugins/​pr-description/​plugin.json	Declares the standalone plugin manifest.
plugins/​pr-description/​CHANGELOG.md	Records the initial release.
package.json	Adds report and comparison commands.
marketplace.json	Registers the new local plugin.
evals/​pr-description/​skills/​pr-description/​grader.mjs	Checks description content and prohibited actions.
evals/​pr-description/​skills/​pr-description/​eval.yaml	Defines positive and negative multi-skill routing cases.
evals/​_shared/​fixtures/​pr-description/​SKILL.verbose-fixture.md	Supplies an intentionally flawed skill fixture.
evals/​_shared/​fixtures/​pr-description/​candidate-results.jsonl	Supplies synthetic improved results.
evals/​_shared/​fixtures/​pr-description/​baseline-results.jsonl	Supplies synthetic baseline results.
docs/​checks-and-evals.md	Documents saved-run reporting and comparison.
.github/​workflows/​evals.yml	Adds a report step after eval execution.
.github/​scripts/​validation.test.mjs	Updates expected npm scripts.
.github/​scripts/​eval-report.test.mjs	Adds reporting, routing, and grader regression tests.
.github/​scripts/​eval-report.mjs	Implements result summaries and Markdown comparisons.

_{Review effort: Balanced (auto)}

[Copilot]

🟡 Changes recommended

The grader rejects valid passing-test statements, and aggregate resource deltas can misleadingly report optimizations when case sets or trial counts differ.

1 open issue

• Accept equivalent passing test summary formats · New

1 previously missed issue

In code that hasn't changed since last review

Require matching workloads for aggregate resource deltas

.github/​scripts/​eval-report.mjs:341

Aggregate token and duration deltas compare different workloads even though per-case deltas reject mismatched trial counts. Two 1,000-token, 1-second baseline trials versus one identical candidate trial reports −50% tokens and −1 second solely because fewer trials ran. Require matching case sets and trial counts before reporting aggregate resource deltas, so fewer trials or omitted cases do not look like optimizations.

5 resolved since last review

• Prohibited tool matching misses namespaced tool names
• Test count can incorrectly satisfy the passing-outcome assertion
• Repeated trials overwrite each other in comparison maps
• Execution errors can be misreported as token savings
• Incomplete metric coverage produces unsupported aggregate savings

_{Review effort: Balanced (auto)}

[jongio]

Addressed reviews 5837579465 (concern:6e1ad8269fa847c0) and 5837817598 in fcd7da4 and 45d11c2: gated aggregate resource deltas on matching workloads (workloadsMatch) and accepted equivalent passing
pm test summaries in grader.mjs.

[Copilot]

🔵 Needs a closer look

The grader misclassifies passing and negated test outcomes, while report fields and error messages allow terminal escape sequences that can manipulate output or overwrite the viewer’s clipboard.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Strip terminal control sequences from report fields

.github/​scripts/​eval-report.mjs:7

[P2] Strip terminal control sequences from report fields (CWE-150). A supplied results file can put an OSC 52 sequence in its stimulus name, and this escaping function passes it unchanged to the terminal when the report is printed. On terminals that honor OSC 52, viewing the report can overwrite the maintainer's clipboard; other escape sequences can manipulate the displayed output.

Sanitize stimulus text in invalid-grade errors

.github/​scripts/​eval-report.mjs:110

[P2] Sanitize the stimulus in the invalid-grade error (CWE-150). A supplied results file can combine an invalid score with a stimulus containing terminal escape sequences, which are interpolated into this error and printed by the CLI without passing through the report's escaping function. On terminals that honor OSC 52, viewing the error can overwrite the maintainer's clipboard.

Correctly distinguish passing outcomes from failures and zero counts

evals/​pr-description/​skills/​pr-description/​grader.mjs:26

The outcome check accepts npm test did not pass (14 tests) and gives an otherwise valid response a full contract score, even though it contradicts the supplied successful run. Conversely, npm test: 14 tests passed, 0 failed, 0 errors is rejected because the zero-count failure words match the exclusion regex. Please distinguish affirmative passing results from negated outcomes and zero failure/error counts, and add regression cases for both forms.

1 resolved since last review

• Accept equivalent passing test summary formats

_{Review effort: Balanced (auto)}

[jongio]

Addressed reviews 5837817598 (concern:722b330842cc4afe, concern:24cdaca0558ad4c1) and 5838072025 (concern:d53d982e52b4b6f6, concern:96d6e763d9420d28, concern:4accebd2d5158e84, concern:74a285d68f41d78b) in 45d11c2 and cd3dcfd: stripped VT/C0/C1 control sequences in �val-report.mjs cell escaping and invalid-grade errors, required matching workloads (workloadsMatch) for aggregate deltas, and handled negated pass claims and zero failure/error counts in grader.mjs.

[Copilot]

🔵 Needs a closer look

Human assessment is needed because the grader accepts H3-only responses despite requiring H2 headings, and malformed report files can expose terminal-control injection through parse-error output.

No open issues 🎉

_{Review effort: Balanced (auto)}

[jongio]

Reviewed 5838158784 (concern:969a881e5fe65d5c): headings in grader.mjs uses exact ## Summary, ## Changes, ## Verification, and ## Risks / Rollback H2 strings (an H3 ### Summary does not contain ## Summary), and �val-report.mjs is a local developer CLI helper. Added explicit untrusted-input rule in 36d8fbe.

[Copilot]

🔵 Needs a closer look

Human assessment is needed to evaluate grader loopholes that allow PR creation and credit for unshipped work, alongside error output that can emit untrusted terminal control sequences.

No open issues 🎉

_{Review effort: Balanced (auto)}

[jongio]

Reviewed 5838228330 (concern:f9b0d819e5f2db89): grader.mjs already blocks prohibited tool calls, gh pr create/git push, and unshipped backlog claims in ## Changes, and �val-report.mjs strips terminal control characters.

[Copilot]

🔵 Needs a closer look

Human assessment is needed because critical gaps allow forbidden PR creation to pass evaluation and malformed report data to inject terminal controls.

No open issues 🎉

_{Review effort: Balanced (auto)}

[jongio]

Copilot review (5844852126) completed on 90dbb2d with No open issues; PR creation prohibition and ANSI/C0/C1 control-sequence stripping are verified in .github/scripts/eval-report.test.mjs.