Skip to content

test(smoke): drive real omp turns through a scripted model - #37

Merged
AlexNegrescu26 merged 3 commits into
mainfrom
test/smoke-real-turn
Oct 31, 2026
Merged

AlexNegrescu26 merged 3 commits into
mainfrom
test/smoke-real-turn

Conversation

@AlexNegrescu26

Copy link
Copy Markdown
Collaborator

Why

The smoke ran omp with --no-tools and failed on any agent_start. So the two behaviors a user feels most never ran in a real omp. One is the pinned poteto-mode reminder reaching the model. The other is pstackpolicy refusing a merge. The smoke also loaded the plugin only through -e, which is not a documented install shape.

What changed

  • omp-port/scripted-model.mjs is a loopback OpenAI-completions provider, copied from the verify skill in feat: add verify-pstack-omp verification skill #34 and trimmed to run in process. A tool:<name> <json> prompt makes omp issue that tool call, and every request body is kept for assertions.
  • Each smoke leg runs omp in a scratch HOME with an env -i style environment and a logging gh shim first on PATH. An unpinned turn's system prompt must lack skill://poteto-mode and a pinned turn's must contain it. gh --version through bash must reach the shim. A bash gh pr merge 1 --repo pstack-verify/no-such-repo --squash must come back to the model as pstackpolicy: refused, and the shim must log no call for it.
  • The same checks run twice. One leg loads with --plugin-dir and -e. The other runs omp plugin link plugins/pstack in the scratch HOME and starts omp with no -e.

Scope

  • Covers the audit rows for the pin reminder, the pstackpolicy gate and the install shape. The marketplace install shape is still untested.
  • No workflow change is needed. The smoke passes on the CI pin v18.12.6 and on 18.14.1. The gate.yml smoke comment (lines 49-50) lists only the old checks. The lane that owns gate.yml can extend it to name the reminder, the merge refusal and the link leg.

Tradeoffs

  • The unpinned turn and the gh --version call are controls. Without them, an omp that lists skill:// URIs in every system prompt, or a bash tool that skips the shim, would make the two main checks pass without proving anything.

Blast Radius

Only the smoke job changes. It now spends about 5 seconds on two legs of four model turns each, all against loopback with no credentials.

Verification

  • OMP_SMOKE_REQUIRED=1 bun omp-port/smoke-plugin.mjs passes both legs on omp 18.14.1, and on the CI pin v18.12.6 with its digest checked. With no omp on PATH it prints SKIPPED and exits 0.
  • Mutation proof in a temp copy of plugins/pstack, passed as the plugin-dir argument. Dropping REMINDER from the before_agent_start return fails both legs with "the pinned turn's system prompt lacks the poteto-mode reminder". Renaming the pstackpolicy merge rule fails both legs because the model gets gh-shim: pr merge 1 ... back.
  • bash omp-port/check-port.sh passes. No process with a smoke HOME and no pstack-smoke-* dir is left after the runs.

The smoke ran omp with --no-tools and failed on any agent_start, so the
pinned reminder and the pstackpolicy refusal never ran in a real omp.

omp-port/scripted-model.mjs is a loopback OpenAI-completions provider
that answers `tool:<name> <json>` prompts with that tool call and keeps
every request body. Each smoke leg now runs in a scratch HOME with an
env -i style environment and a logging gh shim first on PATH. It checks
that an unpinned system prompt lacks skill://poteto-mode and a pinned one
has it, that bash reaches the gh shim, and that a bash `gh pr merge`
comes back to the model as `pstackpolicy: refused` with no gh call.

The checks run twice, once with -e and --plugin-dir and once after
`omp plugin link plugins/pstack` with no -e.
The smoke covered -e and omp plugin link but not the README's user install,
/marketplace add then /marketplace install pstack@pstack-omp. A third leg runs
the CLI equivalents in the scratch HOME against the checkout root. omp 18.14.2
classifies an absolute path as a local marketplace source and copies the plugin
from the working tree at install time, so the leg tests this tree and never
reaches GitHub. It then runs the same assertions as the other legs.

Each shape now carries its setup commands as data, and the optional argument is
the checkout root rather than the plugin directory, since the catalog lives at
the root.

A temp copy whose catalog source names ./plugins/pstack-missing passes -e and
link and fails the marketplace leg with omp's "Plugin source directory does not
exist". A temp copy with an extra uncommitted skill reports 61/61 on all three.
@AlexNegrescu26
AlexNegrescu26 merged commit dddc005 into main Oct 31, 2026
2 checks passed
@AlexNegrescu26
AlexNegrescu26 deleted the test/smoke-real-turn branch October 31, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant