Simplify the starter to a marketplace with two maintenance plugins by ronniegeraghty · Pull Request #4 · github/plugin-marketplace-starter

ronniegeraghty · 2026-11-03T02:21:47Z

Turns the starter into a simple template: a plugin marketplace with two plugins for maintaining it, plus evals for both.

What's in the repo now

marketplace.json at the root, listing two local maintenance plugins and one optional remote plugin:
- plugin-authoring: skills add-plugin, try-plugin-locally, and submit-plugin, for contributors. Local testing uses the selected checkout in a fresh session; optional installation-check recipes use a small validator bundled inside the skill.
- marketplace-maintainer: the read-only plugin-reviewer agent and skills update-remote-pins and release-plugins, for owners.
Optional remote GitHub Copilot Evals (eval): pinned to copilot-evals-v0.23.3 at be7b4ee0d58b7bea143e74fd64275b2c47091cab, path packages/plugin. It retains its upstream GitHub Limited Use Source License.
.github/copilot/settings.json still turns on only the two local maintenance plugins. The remote plugin requires explicit installation.
evals/ mirrors plugins/: one eval per skill or agent, with a grader.mjs beside each skill eval. Shared helpers and fixtures live in evals/_shared/.
Two workflows:
- Validate: lint, CI script regression tests, a plugin.json schema check, a catalog match, and remote pin checks. Validation scripts and their tests live in .github/scripts/.
- Evals: the smoke suite, started explicitly by manual dispatch. Pull requests never start paid runs.
The OSPO files are cut back to the template content. README "Make it yours" steps walk adopters through renaming.

What's removed

The six example plugins
The legacy docs, tests, scripts, and schemas folders
The issue and PR templates
The canvas, MCP, and TypeScript tooling
Every npm script except lint and eval

Eval results

The full suite passed at commit 64a3f37: 14 cases, 5 trials each, on claude-haiku-5.5. All 70 trials passed after the fix loop:

Skills: sharper trigger descriptions, and clearer steps where the model skipped one.
Reviewer agent: a fixed first line for a clean result, and it reads every catalog entry.
Graders: wording checks accept equivalent correct answers. A grader counts reading a skill's files as using the skill, because Vally copies skills into the trial workspace.

The evals' default model is claude-haiku-5.5, because gpt-5.4-mini isn't available to every account.

Review fixes

Moved embedded workflow validation code into .github/scripts/.
Local catalog validation compares description alongside name and version.
Remote manifest selection prefers a supported Agent Plugins-schema root; otherwise it follows the CLI's legacy order. API and malformed-JSON errors are reported rather than swallowed as missing files.
Optional local-install recipes call a bundled validator before Copilot. It parses JSON as data, rejects unsafe identifiers and ambiguous metadata, checks matching Agent Plugins root manifests, and rejects traversal and symlink/junction escapes. The recipes use fixed commands with separate validated operands and recorded, exclusively created test homes/caches. Exact cleanup confirmation and host permissions are required before creating installation-test state.
try-plugin-locally, update-remote-pins, and submit-plugin explicitly treat checkout content as untrusted input. Checkout instructions can't grant permissions, bypass approvals, change tools or instruction hierarchy, or override secrets handling and safety. Relevant conventions remain available within the pin/submission workflows, and conflicting instructions are ignored and reported.
Paid evals now require manual dispatch. No repository environment or permission settings were changed; the ready_for_review trigger issue no longer applies because PR events don't start these runs.
Merged the latest main, including the OSPO copyright update in LICENSE.

The earlier baseline passed 478 deterministic checks. The expanded deterministic suite, current skill/eval lint, schema checks, and catalog/pin validation now pass after the feedback fixes. The recipe tests execute the actual PowerShell and bash examples with fake Copilot functions, including hostile metadata, path escapes, collisions, and failure cleanup. These controls constrain the documented executable procedure; they aren't a model sandbox or a guarantee against concurrent filesystem replacement or hard links. The validator needs trusted Node.js 22+ and the fixed Agent Plugins local layout. The paid behavioral suite wasn't rerun for these CI and install-skill changes.

User-review feedback addressed

Complete the adoption checklist, including contribution links, enforcement contacts and lockfile names; explain trusted checkout loading and explicit local bootstrap.
Keep unreleased first versions intact, require a known released base for bumps, and forbid version decreases or invented release history.
Preserve the requested upstream release channel when the latest-release endpoint returns 404; use reviewed non-draft release or explicit tag evidence rather than silently selecting a prerelease.
Finish plugin scaffolds before registering them, verify owner-approved retirement deletions before changing registration, and supply actual comparison inputs to the read-only reviewer.
Require an independently trusted installation helper for another contributor's branch. Confirm the exact cleanup scope before an installation check, and independently verify the recorded identifiers, owned directories, environment and working directory immediately before destructive cleanup.

New behavioral cases cover unknown release bases, first releases and stable-channel preservation. They were linted but weren't model-run. Upstream Evals and host approval limitations are kept separate from fixes to this starter.

Local-checkout testing

Developers test the new or edited plugin from their already-selected marketplace checkout, keeping their branch and uncommitted changes. A fresh copilot --plugin-dir "<absolute-local-plugin-folder>" session loads that source without installation or settings changes.
The default try-plugin-locally smoke case now checks local-session command preparation, the selected absolute source path, unchanged checkout state, and an honest unverified discovery/behavior report. It doesn't start a nested Copilot session, install/uninstall a plugin, create profiles, or switch branches.
Explicit installation checks remain optional, with their existing trust, approval and cleanup controls intact.
At d003cdd, all 525 deterministic tests and current skill/eval lint pass. The new grader regressions cover incorrect sources, paths with spaces, unwanted session/install/branch actions and fabricated host verification. This case has not been model-run; real host discovery and plugin behavior remain unverified.

To check

The GitHub Docs page on creating plugin marketplaces still mentions npm run verify and .github/plugin/marketplace.json. Both are gone here, so the page needs an update after this merges.
The Evals workflow hasn't run in Actions yet.

Remove the six sample plugins (bug-report, code-review, customization-authoring, format-json, json-configuration, team-brainstorm) with their evals, graders, fixtures, and tests. Add plugin-authoring (add-plugin, try-plugin-locally, submit-plugin) and marketplace-maintainer (the read-only plugin-reviewer agent, update-remote-pins, release-plugins), each with evals. Evals mirror plugins/: one eval per asset at evals/<plugin>/skills/<skill>/ or evals/<plugin>/agents/<agent>/, with plugin-only graders and fixtures under evals/<plugin>/ and shared helpers in evals/graders/ and evals/support/. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep only what the marketplace, its two included plugins and their evals, adoption (rename, add, remove), and the open source release checklist need. - new-plugin now creates only the plugin shell: plugin.json, README, CHANGELOG, the catalog entry, the overlay entry, and the README row. No --with flags, skills, agents, MCP servers, hooks, extensions, or evals. - remove-plugin is the one removal command. remove-example, scripts/examples.mjs, and the examples markers are gone. A plugin owns plugins/<name>/, evals/<name>/, and its tests. - validate.mjs checks the catalog, each plugin.json against the schema, versions and changelogs, the repository overlay, and Markdown links. - Remove the canvas eval host, the TypeScript MCP build and typecheck, bundled grader builds, eval coverage and layout enforcement, and the eval launcher's selection options. The launcher keeps absolute params and the pinned runtime bridge the plugins' evals and evals.yml need. - Remove docs/ and .github/copilot-instructions.md. README covers setup, adding and removing plugins, and the OSPO sections. Remote pin rules and local install steps move into the skills that use them. - Drop esbuild, typescript, @types/node, zod, and @modelcontextprotocol/sdk. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The marketplace and its two plugins work with plain files, so the template no longer ships a scripted framework. - Remove scripts/, test/, schemas/, the pull request template, and the issue form. Adding, removing, renaming, and pinning are now short manual steps in README.md, and the skills do them by editing files and using git or gh. - add-plugin creates the plugin shell by hand. release-plugins retires a plugin by deleting its folders and registrations. update-remote-pins resolves shas with gh api or git ls-remote. submit-plugin writes the pull request description itself. plugin-reviewer carries its whole checklist. - The plugins' evals use small static synthetic fixtures (shared base in evals/support/marketplace, overlays next to each eval) instead of a generated workspace, and the graders check the files the skills produce. - package.json keeps only the pinned Vally CLI and Copilot CLI, with lint and eval scripts. validate.yml runs npm run lint. evals.yml runs the pinned Vally CLI with COPILOT_CLI_PATH pointing at the pinned Copilot CLI and adds Vally's report to the job summary. - SECURITY.md is the OSPO template. CONTRIBUTING.md and AGENTS.md describe the manual workflow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Opening a marketplace repository to work on its plugins shouldn't install or turn on those plugins. Delete .github/copilot/settings.json, and have people add the marketplace and install only the plugins they want. - README, AGENTS.md, CODEOWNERS, the plugin READMEs, plugin-reviewer, and the add-plugin, submit-plugin, try-plugin-locally, release-plugins, and update-remote-pins instructions no longer write or check the file. try-plugin-locally uses --plugin-dir or an isolated install instead. - The eval fixtures drop their settings.json files, and the graders drop their settings.json assertions. Every eval case stays. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Both workflows pick up new plugins and evals from plugins/ and evals/ without listing them. - validate.yml runs npm run lint (vally lint plugins --eval-spec evals --strict) and a short inline check that parses marketplace.json and every plugin.json and matches each local entry's name and version, since Vally doesn't read those files. - evals.yml runs the smoke suite by hand, or on pull requests that change plugins/ or evals/ when ENABLE_EVALS is true, never for forks or drafts. It drops the suite input, environment, results upload, and job summary, and keeps COPILOT_CLI_PATH for the plugin-reviewer eval. - The eval npm script no longer writes JUnit, and the README's Checks section matches. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- validate.yml checks every plugins/*/plugin.json against the official Agent Plugins schema with a pinned one-off ajv-cli run. The pinned Copilot CLI has no plugin validate command, and Agent Plugins publishes no marketplace.json schema. - A short inline step checks each remote catalog entry: a github source, a full 40-character sha for a commit that exists, and a plugin.json at that commit whose name matches the entry. It uses gh api with the workflow token and passes when there are no remote entries. - README and AGENTS.md describe the new checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

evals/graders and evals/support could collide with a plugin of the same name now that no script reserves those names. evals/_shared can't, because plugin names start with a lowercase letter or digit. - Move evals/graders to evals/_shared/graders and evals/support to evals/_shared/support, and update the eval files' src paths. The graders still import ./common.mjs from the grading folder. - README, AGENTS.md, add-plugin, and the retire steps describe evals/_shared and drop the reserved-name rule. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Every assertion in the two graders.mjs files maps to a built-in Vally grader in the eval YAML: file-matches, file-exists, file-not-exists, file-contains, file-not-contains, file-not-matches, output-matches, tool-calls (required and disallowed, with argument and result patterns), and run-command to confirm marketplace.json still parses. Delete both graders.mjs files, evals/_shared/graders/common.mjs, and the grading_environment staging. Every case, smoke tag, and negative control stays. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Restore .github/copilot/settings.json, but enable only marketplace-maintainer and plugin-authoring. Other plugins added to the marketplace stay opt-in. - README and AGENTS.md say the file is only for the repository's own maintenance plugins, and "Make it yours" renames the marketplace there too. - add-plugin never adds a plugin to the file. Retiring a plugin removes its key only if the file lists it. plugin-reviewer flags a change that enables any other plugin there. try-plugin-locally notes which plugins the checkout turns on. - The shared eval fixture gets a matching settings.json, and the add-plugin eval checks that it's left unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Move evals/_shared/support/marketplace to evals/_shared/fixtures/sample-marketplace and rename per-eval fixture folders to fixtures, matching Vally's term for files staged into the workspace. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This reverts 15245f0, which replaced the shared and per-plugin graders with Vally built-ins. It keeps the later fixtures rename and settings.json behavior, and adds a file-matches check that add-plugin leaves settings.json unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Each skill's checks now live in grader.mjs next to its eval, unchanged. The json and text readers moved into evals/_shared/graders/common.mjs. Only release-plugins keeps a mode argument, for its release and retire scenarios. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…tral pin example Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ittle graders Ran the full eval suite (5 trials per case) and fixed what it found until all 14 cases passed every trial on claude-haiku-5.5. - Skills: sharper trigger descriptions for add-plugin, submit-plugin, release-plugins, and update-remote-pins; add-plugin always lists the plugin in the root README; try-plugin-locally uses a test home path the user names. - plugin-reviewer: fixed first line for a clean result and a fixed closing line about not running commands; reads every catalog entry and treats a missing remote sha as a finding. - Evals: default model claude-haiku-5.5 (gpt-5.4-mini isn't available to every account); graders accept reading a skill's files as using it, since Vally copies skills into the workspace; looser wording checks; fixture AGENTS.md mirrors the real one's skill hint. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

The local-install recipe reuses a fixed temporary directory and then recursively deletes its contents, risking deletion of pre-existing files.

4 open issues

What changed in this PR

Simplifies the starter into a minimal plugin marketplace with two maintenance plugins, replacing six example plugins and their supporting tooling.

Changes:

Add contributor-facing plugin-authoring and owner-facing marketplace-maintainer, including a read-only reviewer agent.
Add mirrored behavioral evals, synthetic fixtures, and shared grading helpers.
Streamline npm commands, CI workflows, repository settings, and template documentation; remove legacy examples and tooling.

This pull request requires changes. Unresolved findings include critical cleanup behavior that can delete pre-existing files, missing eval triggers, catalog-validation and manifest-resolution gaps, grader false positives, and incomplete first-pin README updates.

File	Description
`tsconfig.json`	Remove TypeScript configuration.
`test/runtime-environment.test.mjs`	Remove runtime bridge tests.
`test/remote-plugins.test.mjs`	Remove remote-pin checker tests.
`test/installation.test.mjs`	Remove standalone installation tests.
`test/grader-format-json.test.mjs`	Remove formatter grader tests.
`test/grader-common.test.mjs`	Remove shared scoring tests.
`test/grader-bug-report.test.mjs`	Remove bug-report grader tests.
`test/grader-brainstorm.test.mjs`	Remove brainstorm grader tests.
`test/extension.test.mjs`	Remove canvas extension tests.
`test/entrypoint.test.mjs`	Remove entrypoint and symlink tests.
`test/board-ui.test.mjs`	Remove board accessibility tests.
`SUPPORT.md`	Remove template-only comment markers.
`SECURITY.md`	Restore template security-reporting guidance.
`scripts/typecheck.mjs`	Remove custom TypeScript checker.
`scripts/problems.mjs`	Remove validation error utilities.
`scripts/examples.mjs`	Remove example ownership utilities.
`scripts/eval-runtime.js`	Remove custom Copilot runtime bridge.
`scripts/entrypoint.mjs`	Remove entrypoint detection helper.
`schemas/plugin.schema.json`	Remove vendored plugin schema.
`schemas/NOTICE`	Remove vendored schema notice.
`schemas/mcp.schema.json`	Remove vendored MCP schema.
`plugins/team-brainstorm/skills/team-brainstorm/SKILL.md`	Remove brainstorm skill.
`plugins/team-brainstorm/README.md`	Remove brainstorm documentation.
`plugins/team-brainstorm/plugin.json`	Remove brainstorm manifest.
`plugins/team-brainstorm/com.github.copilot/extensions/brainstorm/server.mjs`	Remove board HTTP server.
`plugins/team-brainstorm/com.github.copilot/extensions/brainstorm/extension.mjs`	Remove canvas registration and lifecycle code.
`plugins/team-brainstorm/com.github.copilot/extensions/brainstorm/board.mjs`	Remove session-local board model.
`plugins/team-brainstorm/CHANGELOG.md`	Remove brainstorm release history.
`plugins/plugin-authoring/skills/try-plugin-locally/SKILL.md`	Add local trial and cleanup workflow.
`plugins/plugin-authoring/skills/try-plugin-locally/references/local-install.md`	Document CLI and VS Code trials.
`plugins/plugin-authoring/skills/submit-plugin/SKILL.md`	Add contribution preparation workflow.
`plugins/plugin-authoring/skills/add-plugin/SKILL.md`	Add plugin scaffolding workflow.
`plugins/plugin-authoring/README.md`	Document authoring assets and installation.
`plugins/plugin-authoring/plugin.json`	Declare authoring plugin metadata.
`plugins/plugin-authoring/CHANGELOG.md`	Record initial authoring release.
`plugins/marketplace-maintainer/skills/update-remote-pins/SKILL.md`	Add reviewed remote-pin maintenance.
`plugins/marketplace-maintainer/skills/update-remote-pins/references/remote-pins.md`	Document remote sources and verification.
`plugins/marketplace-maintainer/skills/release-plugins/SKILL.md`	Add release readiness workflow.
`plugins/marketplace-maintainer/skills/release-plugins/references/retire-plugin.md`	Document plugin retirement procedure.
`plugins/marketplace-maintainer/README.md`	Document maintenance assets and installation.
`plugins/marketplace-maintainer/plugin.json`	Declare maintenance plugin metadata.
`plugins/marketplace-maintainer/com.github.copilot/agents/plugin-reviewer.agent.md`	Add read-only contribution reviewer.
`plugins/marketplace-maintainer/CHANGELOG.md`	Record initial maintenance release.
`plugins/json-configuration/skills/json-configuration/SKILL.md`	Remove JSON inspection skill.
`plugins/json-configuration/README.md`	Remove JSON plugin documentation.
`plugins/json-configuration/plugin.json`	Remove JSON plugin manifest.
`plugins/json-configuration/mcp.json`	Remove MCP server registration.
`plugins/json-configuration/CHANGELOG.md`	Remove JSON plugin release history.
`plugins/format-json/README.md`	Remove formatter documentation.
`plugins/format-json/plugin.json`	Remove formatter manifest.
`plugins/format-json/com.github.copilot/hooks/hooks.json`	Remove formatter hook registration.
`plugins/format-json/com.github.copilot/hooks/format.mjs`	Remove post-edit formatting implementation.
`plugins/format-json/CHANGELOG.md`	Remove formatter release history.
`plugins/customization-authoring/skills/skill-authoring/SKILL.md`	Remove general skill-authoring workflow.
`plugins/customization-authoring/skills/skill-authoring/references/evals.md`	Remove eval-authoring reference.
`plugins/customization-authoring/README.md`	Remove customization plugin documentation.
`plugins/customization-authoring/plugin.json`	Remove customization plugin manifest.
`plugins/customization-authoring/CHANGELOG.md`	Remove customization release history.
`plugins/code-review/skills/adversarial-review/SKILL.md`	Remove general review workflow.
`plugins/code-review/README.md`	Remove code-review documentation.
`plugins/code-review/plugin.json`	Remove code-review manifest.
`plugins/code-review/com.github.copilot/agents/code-reviewer.agent.md`	Remove general reviewer agent.
`plugins/code-review/CHANGELOG.md`	Remove code-review release history.
`plugins/bug-report/skills/bug-report/templates/ui.md`	Remove UI report template.
`plugins/bug-report/skills/bug-report/templates/cli.md`	Remove CLI report template.
`plugins/bug-report/skills/bug-report/SKILL.md`	Remove bug-report skill.
`plugins/bug-report/README.md`	Remove bug-report documentation.
`plugins/bug-report/plugin.json`	Remove bug-report manifest.
`plugins/bug-report/CHANGELOG.md`	Remove bug-report release history.
`package.json`	Retain only lint and eval tooling.
`marketplace.json`	Replace examples with maintenance plugins.
`evals/team-brainstorm/eval.yaml`	Remove brainstorm behavioral evals.
`evals/support/canvas-sdk.mjs`	Remove canvas SDK fixture.
`evals/support/canvas-loader.mjs`	Remove canvas module loader.
`evals/plugin-authoring/skills/try-plugin-locally/grader.mjs`	Grade isolated installation and cleanup.
`evals/plugin-authoring/skills/try-plugin-locally/eval.yaml`	Add installation and non-trigger cases.
`evals/plugin-authoring/skills/submit-plugin/grader.mjs`	Grade versions and PR descriptions.
`evals/plugin-authoring/skills/submit-plugin/fixtures/released.json`	Supply synthetic comparison baseline.
`evals/plugin-authoring/skills/submit-plugin/eval.yaml`	Add submission preparation cases.
`evals/plugin-authoring/skills/add-plugin/grader.mjs`	Grade plugin scaffolding and registration.
`evals/plugin-authoring/skills/add-plugin/eval.yaml`	Add scaffolding and non-trigger cases.
`evals/marketplace-maintainer/skills/update-remote-pins/grader.mjs`	Grade reviewed offline pin updates.
`evals/marketplace-maintainer/skills/update-remote-pins/fixtures/upstream-snapshot.json`	Supply recorded synthetic upstream evidence.
`evals/marketplace-maintainer/skills/update-remote-pins/fixtures/marketplace.json`	Supply an existing remote pin.
`evals/marketplace-maintainer/skills/update-remote-pins/eval.yaml`	Add remote-pin behavior cases.
`evals/marketplace-maintainer/skills/release-plugins/grader.mjs`	Grade release preparation and retirement.
`evals/marketplace-maintainer/skills/release-plugins/fixtures/released.json`	Supply prior release metadata.
`evals/marketplace-maintainer/skills/release-plugins/fixtures/plugins/sample-notes/plugin.json`	Supply already-bumped plugin metadata.
`evals/marketplace-maintainer/skills/release-plugins/fixtures/plugins/sample-notes/CHANGELOG.md`	Supply matching release notes.
`evals/marketplace-maintainer/skills/release-plugins/fixtures/marketplace.json`	Supply synchronized candidate catalog.
`evals/marketplace-maintainer/skills/release-plugins/fixtures/evals/sample-notes/cases.json`	Supply removable eval fixture.
`evals/marketplace-maintainer/skills/release-plugins/eval.yaml`	Add release and retirement scenarios.
`evals/marketplace-maintainer/agents/plugin-reviewer/fixtures/unpinned/marketplace.json`	Seed an unpinned remote-source defect.
`evals/marketplace-maintainer/agents/plugin-reviewer/fixtures/common/REVIEW-SCOPE.md`	Define synthetic contribution scope.
`evals/marketplace-maintainer/agents/plugin-reviewer/fixtures/common/plugins/review-notes/skills/review-notes/SKILL.md`	Supply reviewable synthetic skill.
`evals/marketplace-maintainer/agents/plugin-reviewer/fixtures/common/plugins/review-notes/README.md`	Record synthetic ownership and runtime facts.
`evals/marketplace-maintainer/agents/plugin-reviewer/fixtures/common/plugins/review-notes/plugin.json`	Supply reviewable plugin metadata.
`evals/marketplace-maintainer/agents/plugin-reviewer/fixtures/common/plugins/review-notes/CHANGELOG.md`	Supply initial-release evidence.
`evals/marketplace-maintainer/agents/plugin-reviewer/fixtures/clean/marketplace.json`	Supply clean catalog control.
`evals/marketplace-maintainer/agents/plugin-reviewer/eval.yaml`	Test delegation, findings, and read-only behavior.
`evals/json-configuration/eval.yaml`	Remove JSON MCP behavioral evals.
`evals/graders/src/authoring.mjs`	Remove generated-skill contract grader.
`evals/graders/format-json.mjs`	Remove formatter integration grader.
`evals/graders/dist/THIRD_PARTY_NOTICES.txt`	Remove bundled grader notices.
`evals/graders/bug-report.mjs`	Remove report contract grader.
`evals/graders/brainstorm.mjs`	Remove brainstorm contract grader.
`evals/format-json/fixtures/input.json`	Remove formatter input fixture.
`evals/format-json/eval.yaml`	Remove formatting integration evals.
`evals/fixtures/review/clean.js`	Remove clean pagination fixture.
`evals/fixtures/review/buggy.js`	Remove defective pagination fixture.
`evals/customization-authoring/eval.yaml`	Remove skill-generation evals.
`evals/code-review/eval.yaml`	Remove adversarial-review evals.
`evals/code-review/agent/eval.yaml`	Remove standalone reviewer eval.
`evals/bug-report/eval.yaml`	Remove bug-report behavioral evals.
`evals/_shared/graders/common.mjs`	Relocate helpers and add skill-use detection.
`evals/_shared/fixtures/sample-marketplace/README.md`	Supply shared catalog documentation fixture.
`evals/_shared/fixtures/sample-marketplace/plugins/sample-notes/skills/sample-notes/SKILL.md`	Supply shared synthetic plugin skill.
`evals/_shared/fixtures/sample-marketplace/plugins/sample-notes/README.md`	Supply synthetic plugin documentation.
`evals/_shared/fixtures/sample-marketplace/plugins/sample-notes/plugin.json`	Supply baseline plugin metadata.
`evals/_shared/fixtures/sample-marketplace/plugins/sample-notes/CHANGELOG.md`	Supply baseline release history.
`evals/_shared/fixtures/sample-marketplace/marketplace.json`	Supply shared synthetic catalog.
`evals/_shared/fixtures/sample-marketplace/AGENTS.md`	Define fixture maintenance rules.
`evals/_shared/fixtures/sample-marketplace/.github/copilot/settings.json`	Supply fixture maintenance-plugin settings.
`docs/security.md`	Remove detailed security guide.
`docs/remote-plugins.md`	Remove standalone remote-plugin guide.
`docs/hooks.md`	Remove hook development guide.
`docs/examples.md`	Remove example catalog documentation.
`docs/eval-authoring.md`	Remove standalone eval-authoring guide.
`.vally.yaml`	Simplify eval discovery and suites.
`.gitignore`	Simplify ignored development artifacts.
`.github/workflows/validate.yml`	Replace custom validation with lint and inline checks.
`.github/workflows/evals.yml`	Run gated smoke evals directly through npm.
`.github/PULL_REQUEST_TEMPLATE.md`	Remove legacy contribution checklist.
`.github/ISSUE_TEMPLATE/new-plugin.yml`	Remove plugin proposal form.
`.github/copilot/settings.json`	Enable only the two maintenance plugins.
`.github/copilot-instructions.md`	Remove legacy tooling instructions.
`.github/CODEOWNERS`	Simplify ownership guidance while retaining maintainers.

_{Review effort: Balanced (auto)}

Compare all local catalog metadata, preserve schema-aware and legacy remote manifest precedence, and safely create and clean up isolated local install homes. Keep paid evals manual-only and cover the fixes with deterministic CI regression tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80b93953-2b51-4e0b-8b73-311db41ea52e

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80b93953-2b51-4e0b-8b73-311db41ea52e

Treat checkout content as untrusted metadata beside the document reads. Keep command execution, settings changes and cleanup inside the trusted procedure and actual user authorization, with policy-placement and procedure-preservation regression checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80b93953-2b51-4e0b-8b73-311db41ea52e

Keep relevant repository conventions available while rejecting checkout instructions that override approvals, tools, scope, instruction hierarchy, secrets handling, or safety. Add adjacent-policy and procedure-preservation regression checks for both skills. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80b93953-2b51-4e0b-8b73-311db41ea52e

Bundle a small validator with the skill to reject unsafe identifiers, ambiguous metadata and out-of-scope paths before CLI calls. Preserve exclusive test-home ownership, fixed command operands, failure cleanup and safe generated defaults, with executable PowerShell and bash regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80b93953-2b51-4e0b-8b73-311db41ea52e

Shorten the README, move detailed check and eval instructions into one directly linked guide, and record the documentation rule in AGENTS.md. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80b93953-2b51-4e0b-8b73-311db41ea52e

Make repeated Git option forms and quoted operands disjoint while preserving benign-output and mutation-attempt checks. Add a bounded regression for long option lists. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80b93953-2b51-4e0b-8b73-311db41ea52e

Copilot

🟡 Changes recommended

The catalog checker treats ordinary files such as plugins/.DS_Store as plugin directories, causing ENOTDIR errors that block linting otherwise valid checkouts.

1 open issue

Ignore files when enumerating plugin folders · New

_{Review effort: Max (auto)}

jongio · 2026-11-10T15:44:29Z

Great simplification of the marketplace starter! We stacked #7 (ignite-skill-eval-golden-paths ->
onniegeraghty/maintenance-plugins) directly on top of �dc8756 so both golden paths (the pr-description multi-skill cohort and the baseline-vs-candidate delta report) build cleanly on your new layout.

Alignment notes with github/evals-internal#754 and github/evals-action#24

marketplace.json (line 25 — �val remote pin):
- copilot-evals-v0.23.3 (�e7b4ee...) rejected /eval compare (compare_skill) when �gent_environment.skills declared multiple co-installed skills (like your 3 plugin-authoring skills in �vals/plugin-authoring/skills/*/eval.yaml).
- github/evals-internal#754 adds companion-skill cohort support to compare_skill and surfaces token/duration % deltas under verified ruler parity. Once #754 is released to github/evals, we should bump
  ef and sha here.
.github/workflows/evals.yml (line 28 —
pm run eval):
- For the live template demo, feat: add pr-description plugin, multi-skill cohort eval, and baseline/token report #7 adds
  pm run eval:report /
  pm run eval:compare alongside this workflow.
- Longer term, if github/evals-action (updated in feat: surface quality scores, duration, token usage, and case outcomes in PR reports evals-action#24 with score, threshold, duration, and token reporting) supports passing COPILOT_CLI_PATH or repo-local
  ode_modules, we can switch this step to github/evals-action so starter repos get sticky PR reports and job summaries out of the box.
�vals/_shared/graders/common.mjs (line 31 — score &
unGrade):
- Really clean pattern for proportional ype: program scoring ((checks.length - failures.length) / checks.length) and separator-agnostic tool matching (matchesTool). We reused this helper in feat: add pr-description plugin, multi-skill cohort eval, and baseline/token report #7's �vals/pr-description/skills/pr-description/grader.mjs and should promote this pattern into �vals-internal's create/improve skill references.
plugins/plugin-authoring/skills/submit-plugin/SKILL.md (line 48 — ## Checks):
- Consider mentioning
  pm run eval /
  pm run eval:compare (or /eval run / /eval compare) under ## Checks so contributors include evaluation pass rates and token/latency deltas in the PR description.
.github/scripts/check-local-catalog.mjs (line 7 —
eaddirSync):
- We included .filter((entry) => entry.isDirectory()) on
  eaddirSync(..., { withFileTypes: true }) in feat: add pr-description plugin, multi-skill cohort eval, and baseline/token report #7 so merging feat: add pr-description plugin, multi-skill cohort eval, and baseline/token report #7 into this branch also resolves the Copilot review finding on non-directory entries like .DS_Store.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…al-report comparison Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…d aggregate deltas Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ount test summaries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ling Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

… creation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

feat: add pr-description plugin, multi-skill cohort eval, and baseline/token report

jongio · 2026-11-10T16:16:01Z

Verified all historical Copilot reviews (5746482082, 5833163990, 5833926486, 5834212190, 5835003229, 5835407017, 5837258428) and resolved the .DS_Store directory filter finding (
4521475737) in

Copilot

🟡 Changes recommended

The workflow removes the evals environment approval gate while retaining a token with copilot-requests: write, allowing manual branch runs to bypass configured required reviewers.

7 open issues

1 resolved since last review

Ignore files when enumerating plugin folders

_{Review effort: Max (auto)}

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

jongio · 2026-11-10T16:46:22Z

Addressed all 7 findings from Copilot review (5845604249) in commit e55dcce and resolved the threads.

Copilot

🟡 Changes recommended

The publication check misclassifies multiline, compound, and quoted shell commands, allowing tag creation to pass while rejecting legitimate read-only commands.

1 open issue

Parse shell commands without exempting multiline or quoted content · New

7 resolved since last review

_{Review effort: Max (auto)}

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

jongio · 2026-11-10T17:12:21Z

Addressed Copilot review (5846082899) in commit 6f246d7 by tokenizing shell invocations across quotes and compound separators; thread PRRT_kwDOVRlbx861zx3t is resolved.

ronniegeraghty and others added 15 commits November 2, 2026 15:03

Trim CODEOWNERS to the OSPO template

46e8e74

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Simplify plugin READMEs, drop hard-coded license, add repository, neu…

9b1ba52

…tral pin example Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

ronniegeraghty requested a review from a team as a code owner November 3, 2026 02:21

Copilot AI auto review requested due to automatic review settings November 3, 2026 02:21

Copilot started reviewing on behalf of ronniegeraghty November 3, 2026 02:22

View session

github-advanced-security AI found potential problems Nov 3, 2026

View reviewed changes

Comment thread .github/workflows/evals.yml Resolved

Copilot AI reviewed Nov 3, 2026

View reviewed changes

Comment thread plugins/plugin-authoring/skills/try-plugin-locally/references/local-install.md Resolved

Comment thread .github/workflows/evals.yml Resolved

Comment thread .github/workflows/validate.yml Resolved

Comment thread .github/workflows/validate.yml Resolved

github-advanced-security AI found potential problems Nov 3, 2026

View reviewed changes

Comment thread plugins/plugin-authoring/skills/try-plugin-locally/SKILL.md Resolved

ronniegeraghty and others added 2 commits November 3, 2026 11:02

Merge main into maintenance-plugins

d244e21

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 80b93953-2b51-4e0b-8b73-311db41ea52e

github-advanced-security AI found potential problems Nov 3, 2026

View reviewed changes

Comment thread plugins/marketplace-maintainer/skills/update-remote-pins/SKILL.md Resolved

Comment thread plugins/plugin-authoring/skills/submit-plugin/SKILL.md Resolved

github-advanced-security AI found potential problems Nov 3, 2026

View reviewed changes

Comment thread plugins/plugin-authoring/skills/try-plugin-locally/SKILL.md Resolved

ronniegeraghty added 2 commits November 3, 2026 12:45

github-advanced-security AI found potential problems Nov 4, 2026

View reviewed changes

Comment thread plugins/plugin-authoring/skills/try-plugin-locally/SKILL.md Resolved

github-advanced-security Bot found potential problems Nov 10, 2026

View reviewed changes

Comment thread evals/plugin-authoring/skills/try-plugin-locally/grader.mjs Fixed

Comment thread evals/plugin-authoring/skills/try-plugin-locally/grader.mjs Fixed

Copilot started reviewing on behalf of ronniegeraghty November 10, 2026 05:12

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread .github/scripts/check-local-catalog.mjs Resolved

jongio mentioned this pull request Nov 10, 2026

feat: add pr-description plugin, multi-skill cohort eval, and baseline/token report #7

Merged

jongio and others added 9 commits November 10, 2026 07:48

Ignore non-directory entries when enumerating plugins/

0f8c197

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

feat: add pr-description golden-path plugin, multi-skill eval, and ev…

156416a

…al-report comparison Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Address PR #7 security and eval reporting review findings

0eb1ff0

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Address remaining Copilot review feedback on pr-description grader an…

e347340

…d aggregate deltas Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

fix: strip terminal controls in eval-report and handle negated/zero-c…

45e6c8b

…ount test summaries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

fix(pr-description): treat supplied diffs and notes as untrusted data

c8e5667

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Refactor: remove custom eval-report script in favor of core Evals too…

f7baf29

…ling Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Keep starter-repo demo fixtures and eval-report CLI for live template…

dafa67c

… creation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Remove temporary install-test directory

90dbb2d

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

jongio requested a review from wbreza November 10, 2026 15:49

Merge pull request #7 from github/ignite-skill-eval-golden-paths

d1fb340

feat: add pr-description plugin, multi-skill cohort eval, and baseline/token report

jongio requested an auto review from Copilot November 10, 2026 16:16

Copilot started reviewing on behalf of jongio November 10, 2026 16:16

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Address Copilot review findings across workflow and eval graders

e55dcce

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

jongio requested an auto review from Copilot November 10, 2026 16:46

Copilot started reviewing on behalf of jongio November 10, 2026 16:47

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread evals/marketplace-maintainer/skills/release-plugins/grader.mjs Resolved

Tokenize shell invocations in release-plugins publication check

6f246d7

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

ronniegeraghty merged commit eaab343 into main Nov 10, 2026
5 checks passed

ronniegeraghty deleted the ronniegeraghty/maintenance-plugins branch November 10, 2026 17:16

Conversation

ronniegeraghty commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's in the repo now

What's removed

Eval results

Review fixes

User-review feedback addressed

Local-checkout testing

To check

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

jongio commented Nov 10, 2026

Alignment notes with github/evals-internal#754 and github/evals-action#24

Uh oh!

jongio commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jongio commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

jongio commented Nov 10, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ronniegeraghty commented Nov 3, 2026 •

edited

Loading