Repository navigation
test(smoke): drive real omp turns through a scripted model - #37
Merged
Merged
Conversation
The smoke ran omp with --no-tools and failed on any agent_start, so the pinned reminder and the pstackpolicy refusal never ran in a real omp. omp-port/scripted-model.mjs is a loopback OpenAI-completions provider that answers `tool:<name> <json>` prompts with that tool call and keeps every request body. Each smoke leg now runs in a scratch HOME with an env -i style environment and a logging gh shim first on PATH. It checks that an unpinned system prompt lacks skill://poteto-mode and a pinned one has it, that bash reaches the gh shim, and that a bash `gh pr merge` comes back to the model as `pstackpolicy: refused` with no gh call. The checks run twice, once with -e and --plugin-dir and once after `omp plugin link plugins/pstack` with no -e.
The smoke covered -e and omp plugin link but not the README's user install, /marketplace add then /marketplace install pstack@pstack-omp. A third leg runs the CLI equivalents in the scratch HOME against the checkout root. omp 18.14.2 classifies an absolute path as a local marketplace source and copies the plugin from the working tree at install time, so the leg tests this tree and never reaches GitHub. It then runs the same assertions as the other legs. Each shape now carries its setup commands as data, and the optional argument is the checkout root rather than the plugin directory, since the catalog lives at the root. A temp copy whose catalog source names ./plugins/pstack-missing passes -e and link and fails the marketplace leg with omp's "Plugin source directory does not exist". A temp copy with an extra uncommitted skill reports 61/61 on all three.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The smoke ran omp with
--no-toolsand failed on anyagent_start. So the two behaviors a user feels most never ran in a real omp. One is the pinned poteto-mode reminder reaching the model. The other ispstackpolicyrefusing a merge. The smoke also loaded the plugin only through-e, which is not a documented install shape.What changed
omp-port/scripted-model.mjsis a loopback OpenAI-completions provider, copied from the verify skill in feat: add verify-pstack-omp verification skill #34 and trimmed to run in process. Atool:<name> <json>prompt makes omp issue that tool call, and every request body is kept for assertions.ghshim first on PATH. An unpinned turn's system prompt must lackskill://poteto-modeand a pinned turn's must contain it.gh --versionthrough bash must reach the shim. A bashgh pr merge 1 --repo pstack-verify/no-such-repo --squashmust come back to the model aspstackpolicy: refused, and the shim must log no call for it.--plugin-dirand-e. The other runsomp plugin link plugins/pstackin the scratch HOME and starts omp with no-e.Scope
pstackpolicygate and the install shape. The marketplace install shape is still untested.gate.ymlsmoke comment (lines 49-50) lists only the old checks. The lane that ownsgate.ymlcan extend it to name the reminder, the merge refusal and the link leg.Tradeoffs
gh --versioncall are controls. Without them, an omp that listsskill://URIs in every system prompt, or a bash tool that skips the shim, would make the two main checks pass without proving anything.Blast Radius
Only the smoke job changes. It now spends about 5 seconds on two legs of four model turns each, all against loopback with no credentials.
Verification
OMP_SMOKE_REQUIRED=1 bun omp-port/smoke-plugin.mjspasses both legs on omp 18.14.1, and on the CI pin v18.12.6 with its digest checked. With no omp on PATH it prints SKIPPED and exits 0.plugins/pstack, passed as the plugin-dir argument. DroppingREMINDERfrom thebefore_agent_startreturn fails both legs with "the pinned turn's system prompt lacks the poteto-mode reminder". Renaming thepstackpolicymerge rule fails both legs because the model getsgh-shim: pr merge 1 ...back.bash omp-port/check-port.shpasses. No process with a smoke HOME and nopstack-smoke-*dir is left after the runs.