Repository navigation
feat: add the skill eval runner - #78
Open
AntTheLimey wants to merge 11 commits into
Open
AntTheLimey wants to merge 11 commits into
AntTheLimey wants to merge 11 commits into
Conversation
Measures whether the agent skills in plugins/pgedge change outcomes on real pgEdge Starfleet Managed tasks: the same tasks run with the skills, without them and against the raw API, are graded against live API data, and produce a scorecard per harness and model (correct, safe, cheaper, lift). Runs locally on macOS from a config naming the tester's pgedge config and profile, harnesses, models, tasks, arms and runs. A release mode scores the commit a tag would point to and writes a summary for committing that names no tenant, host or other database. It reports; it never decides a release. make test-evals covers grading, scoring and publishing offline.
A failed run's tokens now count against its side, so a cheap wrong answer no longer makes the CLI without skills look cheaper. CI runs make test-evals, which needs no agents and no tokens.
codex exec runs under a generated permissions profile with a private CODEX_HOME per run. The tester's HOME and the login copy are unreadable, writes are limited to the run's own paths, and network goes through Codex's proxy and the arm's allowlist. Negative controls run through codex sandbox, because the model refuses to try a denied path. Only a ChatGPT login is accepted, Codex runs go one at a time, and the runner refuses to start with under three days left on the access token. Cursor is limited to grok-4.7-medium and composer-2.5, and Claude defaults to Sonnet.
Codex accepts only gpt-6.1-sol and gpt-6-luna. The self-report prompt asks for names only, since a sentence on the MCP line failed the cross-arm check.
A lane can be stopped with SIGINT and resumed alone with --resume; the others carry on, and --combine merges the lanes' results. Codex runs take a file lock, so they stay one at a time across processes, and shared files are written by rename. Cursor's token is unset in the agent's shell, which a new control checks.
A stopped Codex lane no longer waits on another lane's lock, --budget is split across lanes, a lane that writes no results still counts as planned, --combine refuses a repeated run, and --resume refuses rows from other skills or another binary.
The lock guarded only against two runs refreshing the login at once, which the three-day expiry guard already makes unlikely, at the cost of running every Codex lane one run at a time.
The grader failed a run that searched the llms reference for the rotate-password section as if it had rotated a password.
--plugin-ref loads the skills and plugin manifests from a ref, such as an open skills PR, and --redo-arm reruns one arm on resume. The skills check on resume covers only kept rows that loaded skills.
With the skills it deleted unasked in every chance it had, so it is not a lane we test or recommend.
Resume compares the binary's source, not its bytes, which change with every build. A lane that reuses the binary still writes its config and shim, and --lane refuses --release. Rotation is judged per command segment, so reading the reference is not a rotation and chaining a read before one does not hide it. --plugin-ref prefers origin's branch, and lanes get the sha the launcher resolved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Adds
evals/, a local runner that measures whether the skills inplugins/pgedgemake an agent better at real pgEdge Starfleet Managed tasks. Each task runs in three setups (skills + CLI, CLI only, raw API) on three agent harnesses, and every answer is graded against the live API.cursor-agent(Grok 4.7 Medium only), and the Codex CLI (GPT-6.1-Sol and GPT-6-Luna, ChatGPT login only). Any other model is refused.--resume,--redoand--redo-armrerun only what is missing, and--combinemerges lanes.--plugin-reftests the skills from another ref, such as an open skills PR.--releasescores the commit a release would tag and writes a scrubbed summary.evals/tasks/; tasks 6 to 10 are the release suite.make test-evalsruns the offline unit tests, and a new CI job runs it. Nothing in CI starts an agent.How it was measured
On 2026-10-23: 225 graded runs (5 lanes × 5 tasks × 3 setups × 3 runs), then the skills arm again with the skills from #75. Every fixture database the runner created was deleted. Killing one lane mid-run left the others running, and resuming it ran only its missing runs. One lane in that run, Cursor's Composer 2.5, has since been dropped.
Gates
make test: 41 packages ok, coverage 90.6% (gate 90.0%)make lint: 0 issuesmake lint-docs: 0 errors in 678 filesmake test-evals: 36 tests okFor the reviewer
cursor-agentand the ChatGPT app's Codex, and a tenant where creating and deleting databases is safe. Seeevals/README.md.cursor.com, whatever its deny rules say; this is listed under Known limits.~/.codex/auth.json. The runner refuses to start with under three days left on the token.