clarify scenario 005's no-approval-turn assertion - #87
Merged
Conversation
A prompt between resume and invocation has three possible causes, and the assertion named none of them. Two walks on 2026-08-16 disagreed as a result: one saw an approval menu and recorded FAIL, the other lost its scrollback and inferred PASS from claude_session_id alone. The three causes are the classification bug (what this scenario tests), the trust-folder gate, and the Phase 5.5 permission-mode precheck, which has run for every task since v0.111.1 moved it out of the code-task heuristic. - name all three causes at the assertion, so a walker identifies which one fired before recording FAIL - add a dedicated assertion that the 'Permission mode:' line must NOT appear: this fixture's only command is /vault-cli:next-task, so Phase 5.5's own rule requires it to emit nothing. Seeing it is a Phase 5.5 trigger defect to file separately, not a classification failure - require the trust precondition to be proven in the environment the walk runs in; a fresh tmux, isolated HOME or sub-agent shell does not inherit the trust decision and hits a gate that mimics the bug - mark claude_session_id necessary-but-not-sufficient, and require a re-run when the scrollback is lost rather than a PASS - require PLUGIN_VER in the result; this flow's behavior moved twice in one day (v0.110.0, v0.111.1)
There was a problem hiding this comment.
{
"verdict": "approve",
"summary": "Documentation-only PR that clarifies scenario 005's preconditions and assertions. The CHANGELOG entry is correctly formatted with ## Unreleased at top, version alignment passes (all four version strings = 0.111.1), and the scenario changes are internally consistent clarifications that do not alter functional behavior.",
"comments": [],
"concerns_addressed": [
"tests/correctness: scenario 005's no-approval-turn assertion now names all three possible sources of a prompt between resume and invocation, includes a dedicated Phase 5.5 permission-mode assertion, and hardens the trust-folder and session-id preconditions — addressed by CHANGELOG.md and scenarios/005-work-on-resume-auto-invokes-subtask.md changes"
]
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two sub-agent walks of scenario 005 on 2026-08-16 reached opposite verdicts, and neither could explain the disagreement. Both were unreliable for reasons the scenario itself made possible:
claude_session_idalone — one of six assertions, and the only one that cannot distinguish invoked from printedThe root problem is that
No AskUserQuestion option list and no permission prompt appearshas three possible causes and named none of them:What
Scenario-only change, no code:
🔐 Permission mode:must not appear. This fixture's only command is/vault-cli:next-task, and Phase 5.5's own rule is "If none are present, emit nothing — do not warn on read-only or docs-only tasks." So seeing it is a Phase 5.5 trigger defect to file separately, not a classification failure — the scenario now says so rather than leaving the walker to guessHOME, or sub-agent shell does not inherit the trust decision from your normal terminal, and the resulting gate mimics the bug exactlyclaude_session_idnecessary but not sufficient; a lost scrollback makes the walk inconclusive and requires a re-run, not a PASSPLUGIN_VERin the recorded result — this flow's behavior moved twice in one day, so a result without it cannot be compared against another walkTesting
Documentation only; no code paths touched. The claims about Phase 5.5 were verified against the installed plugin at
~/.claude/plugins/cache/vault-cli/vault-cli/0.111.1/agents/work-on-task-assistant.md("Phase 5.5: Permission-mode precheck (ALL tasks — never gated on Phase 5)") and against v0.111.1's own CHANGELOG entry.Context
Surfaced while shipping #83 (
task complete --force). Scenario 005 shares zero files with that change — it exercisescommands/execute-task.mdand the work-on flow, while #83 touchedpkg/ops/complete.go,pkg/cli/cli.goandcommands/complete-task.md— so the contested failure never had anything to do with it.