Skip to content

clarify scenario 005's no-approval-turn assertion - #87

Merged
bborbe merged 1 commit into
masterfrom
docs/scenario-005-phase55
Aug 16, 2026
Merged

clarify scenario 005's no-approval-turn assertion#87
bborbe merged 1 commit into
masterfrom
docs/scenario-005-phase55

Conversation

@bborbe

@bborbe bborbe commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Why

Two sub-agent walks of scenario 005 on 2026-08-16 reached opposite verdicts, and neither could explain the disagreement. Both were unreliable for reasons the scenario itself made possible:

  • one saw an approval-menu interrupt and recorded FAIL, without establishing whether the vault folder was trusted in its environment
  • the other lost its tmux scrollback and inferred PASS from claude_session_id alone — one of six assertions, and the only one that cannot distinguish invoked from printed

The root problem is that No AskUserQuestion option list and no permission prompt appears has three possible causes and named none of them:

  1. the classification bug — the only thing this scenario tests
  2. the trust-folder gate — a precondition listed but never required to be proven
  3. the Phase 5.5 permission-mode precheck — new. v0.110.0 added it inside Phase 5 (gated on the code-task heuristic); v0.111.1 moved it to its own phase that runs for every task. That is a new prompt-shaped output in this scenario's exact flow, shipped the same day the walks ran.

What

Scenario-only change, no code:

  • Name all three causes at the assertion, so a walker identifies which fired before recording FAIL
  • Add a dedicated assertion that 🔐 Permission mode: must not appear. This fixture's only command is /vault-cli:next-task, and Phase 5.5's own rule is "If none are present, emit nothing — do not warn on read-only or docs-only tasks." So seeing it is a Phase 5.5 trigger defect to file separately, not a classification failure — the scenario now says so rather than leaving the walker to guess
  • Require the trust precondition to be proven in the environment the walk runs in. A fresh tmux session, isolated HOME, or sub-agent shell does not inherit the trust decision from your normal terminal, and the resulting gate mimics the bug exactly
  • Mark claude_session_id necessary but not sufficient; a lost scrollback makes the walk inconclusive and requires a re-run, not a PASS
  • Require PLUGIN_VER in the recorded result — this flow's behavior moved twice in one day, so a result without it cannot be compared against another walk

Testing

Documentation only; no code paths touched. The claims about Phase 5.5 were verified against the installed plugin at ~/.claude/plugins/cache/vault-cli/vault-cli/0.111.1/agents/work-on-task-assistant.md ("Phase 5.5: Permission-mode precheck (ALL tasks — never gated on Phase 5)") and against v0.111.1's own CHANGELOG entry.

Context

Surfaced while shipping #83 (task complete --force). Scenario 005 shares zero files with that change — it exercises commands/execute-task.md and the work-on flow, while #83 touched pkg/ops/complete.go, pkg/cli/cli.go and commands/complete-task.md — so the contested failure never had anything to do with it.

A prompt between resume and invocation has three possible causes, and the
assertion named none of them. Two walks on 2026-08-16 disagreed as a
result: one saw an approval menu and recorded FAIL, the other lost its
scrollback and inferred PASS from claude_session_id alone.

The three causes are the classification bug (what this scenario tests),
the trust-folder gate, and the Phase 5.5 permission-mode precheck, which
has run for every task since v0.111.1 moved it out of the code-task
heuristic.

- name all three causes at the assertion, so a walker identifies which
  one fired before recording FAIL
- add a dedicated assertion that the 'Permission mode:' line must NOT
  appear: this fixture's only command is /vault-cli:next-task, so Phase
  5.5's own rule requires it to emit nothing. Seeing it is a Phase 5.5
  trigger defect to file separately, not a classification failure
- require the trust precondition to be proven in the environment the walk
  runs in; a fresh tmux, isolated HOME or sub-agent shell does not inherit
  the trust decision and hits a gate that mimics the bug
- mark claude_session_id necessary-but-not-sufficient, and require a
  re-run when the scrollback is lost rather than a PASS
- require PLUGIN_VER in the result; this flow's behavior moved twice in
  one day (v0.110.0, v0.111.1)

@ben-s-pull-request-reviewer ben-s-pull-request-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{
  "verdict": "approve",
  "summary": "Documentation-only PR that clarifies scenario 005's preconditions and assertions. The CHANGELOG entry is correctly formatted with ## Unreleased at top, version alignment passes (all four version strings = 0.111.1), and the scenario changes are internally consistent clarifications that do not alter functional behavior.",
  "comments": [],
  "concerns_addressed": [
    "tests/correctness: scenario 005's no-approval-turn assertion now names all three possible sources of a prompt between resume and invocation, includes a dedicated Phase 5.5 permission-mode assertion, and hardens the trust-folder and session-id preconditions — addressed by CHANGELOG.md and scenarios/005-work-on-resume-auto-invokes-subtask.md changes"
  ]
}

@bborbe
bborbe merged commit 083ecda into master Aug 16, 2026
1 check passed
@bborbe
bborbe deleted the docs/scenario-005-phase55 branch August 16, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant