[ronniegeraghty]

Changes

Relates to #11.

Define one environments.marketplace in .vally.yaml and have the six skill evals reference agent_environment: marketplace. Vally stages six local skills from their source directories. Native setup commands fetch the catalog-pinned remote Evals repository, copy its six skills into the trial's .github/skills for Copilot workspace discovery, and remove the temporary source clone. The checkout uses both the catalog ref and immutable SHA.

No separate preparation script, cache, npm hook, manual source checkout, custom runtime grader, or new test file. The host needs Git/network access and ordinary Git read authentication to github/evals. Only skills are staged; this does not install plugins or enable their agents, hooks, MCP servers, or extensions.

Reduced from 17 changed files to 8, with 26 additions and 44 deletions: the shared environment, six eval references, and the existing cohort test's environment lookup. Case fixtures, invocation graders, negative-case protections, and the independent agent eval are unchanged. The earlier preparation/cache machinery, custom inventory grader, added tests, package/helper/lint changes, and expanded docs were unnecessary scope expansion and have been removed.

Native tooling contracts

The published reference documents named environments, local skill sources and setup commands, and native trial-workspace skill discovery.

Verified against installed Vally 0.19.0 source: planning passes the eval directory as baseDir, and skill sources resolve relative to it, not .vally.yaml. The four ../ segments are intentional. environment.skills takes literal existing directories, not glob patterns or remote catalog entries. Native git.type: clone populates the trial root and has no separate source-directory destination, so it is not a drop-in replacement for fetching only remote skills while preserving fixture workspaces. Setup runs automatically inside each disposable trial, without global configuration changes.

Verification

• 47 retained model-free regression tests, npm run lint, and git diff --check passed.
• Executed the exact native setup for all 16 cases without a model: all 12 actual skill sources parsed, temporary clones removed, and final overlaid fixtures preserved byte-for-byte.
• Then ran the six unchanged evals through the installed Vally 0.19.0 CLI with explicit approval: 16 real Copilot case trials, observed model claude-haiku-5.5, one worker, one trial each, zero trial retries. CLI run ID: ca30d729-fe3d-4ebe-bfd9-4c9eebdcf338; source commit: 698e510ad0b4c6f19b06b25617f012f9bc0567ce.
• Every one of the 16 original runtime logs contains a system.message whose <available_skills> inventory advertises all 12 marketplace skills: six local skills with location: custom, and the six remote Evals skills with location: project. Two builtin skills were also advertised. This proves actual discovery and availability for routing, not activation of every skill or loading every full skill body. Target activation remains separately graded.
• trajectory.metadata.skillsLoaded was empty and no session.skills_loaded event was recorded; that field is not the evidence used. The proof is the actual runtime system-context inventory in each saved events.jsonl, not a configuration list or synthetic executor. Original CLI JSONL, reports, trajectories and runtime logs are retained locally and are not committed.

Recorded eval outcomes

Eval	Cases passing all checks	All 12 advertised in actual runtime context
add-plugin	2/2	2/2
submit-plugin	2/2	2/2
try-plugin-locally	1/2	2/2
release-plugins	3/5	5/5
update-remote-pins	3/3	3/3
pr-description	2/2	2/2

The suite is not fully green: 13/16 cases pass all checks, and 4/6 evals pass their thresholds. No execution errors occurred. The three recorded failures are the existing local-session program checks, the release unknown-base program check, and missing release-skill activation for the unreleased-first-version case. The unknown-base answer explicitly requested evidence in a multiline list, which the existing sentence-based check rejected; the recorded failure is retained, not relabeled. No checks, prompts, skills, or thresholds were changed to obtain these results, and no causal baseline comparison was run.

No new CI/Action work, scheduled updates, plugin installation, global configuration changes, full plugin-runtime activation, or merge is included.

[Copilot]

🟡 Changes recommended

Cache preparation can follow symlinks and write outside its intended directory, while validation incorrectly requires eval files for skills where they are optional.

2 open issues

• Reject symlinked cache paths that escape the cache · New
• Allow cataloged skills without eval files · New

What changed in this PR

Moves all six skill evals to a shared marketplace environment so agents can choose among local and pinned remote skills, while leaving the independent agent eval unchanged.

Changes:

• Defines the shared skill inventory and adds automatic remote-source preparation before evals.
• Adds catalog consistency checks, loaded-skill evidence grading, and model-free regression tests.
• Documents preparation, authentication, maintenance, and the skills-only capability boundary. Live loading still requires a separately approved paid run.

File	Description
README.md	Introduces shared eval preparation and retirement updates.
package.json	Adds preparation and pre-eval scripts.
evals/​pr-description/​skills/​pr-description/​eval.yaml	Uses the shared environment and checks loaded skills.
evals/​plugin-authoring/​skills/​try-plugin-locally/​eval.yaml	Uses the shared environment.
evals/​plugin-authoring/​skills/​submit-plugin/​eval.yaml	Uses the shared environment.
evals/​plugin-authoring/​skills/​add-plugin/​eval.yaml	Uses the shared environment.
evals/​marketplace-maintainer/​skills/​update-remote-pins/​eval.yaml	Uses the shared environment.
evals/​marketplace-maintainer/​skills/​release-plugins/​eval.yaml	Uses the shared environment.
evals/​_shared/​graders/​marketplace-context.mjs	Checks the native SDK’s loaded-skill inventory.
docs/​checks-and-evals.md	Explains setup, maintenance, and runtime limitations.
.vally.yaml	Defines the twelve-skill marketplace environment.
.github/​scripts/​validation.test.mjs	Updates expected npm scripts.
.github/​scripts/​prepare-eval-context.mjs	Fetches pinned sources and validates cache contents and inventory.
.github/​scripts/​marketplace-context.test.mjs	Covers preparation, staging, environment coverage, and grading.
.github/​scripts/​lint.mjs	Adds model-free environment consistency checks.
.github/​scripts/​eval-report.test.mjs	Reads skill coverage from the shared environment.
.github/​scripts/​check-remote-pins.mjs	Supports configurable API response formats for blob fetching.

_{Review effort: Balanced (auto)}

[Copilot]

🔵 Needs a closer look

Reported runs fail 3 of 16 cases, including required skill activation, and lack baseline comparisons, requiring maintainer review to determine whether shared-skill routing introduced regressions.

No open issues 🎉

2 resolved since last review

• Reject symlinked cache paths that escape the cache
• Allow cataloged skills without eval files

_{Review effort: Balanced (auto)}