Observation-first architecture: eliminate agent lockout from blocking wait tools

[tony]

1. Problems

Agents get trapped when a pane wait hides the terminal behind a boolean. In a live dev loop, the agent is often tailing a server, watching tests, checking several panes, or deciding whether to interrupt. A blocking wait_for_text call gives it only found or timeout, so the agent cannot see that the server picked a different port, printed an error, is waiting for input, already finished, or that another pane now needs attention. This is the lockout reported in #59.

The deeper problem is tool mismatch. wait_for_text is useful for known future output from a process the agent does not control. It is a poor default for command completion, current-screen inspection, fuzzy diagnosis, or multi-pane monitoring. Those cases need different primitives:

Agent intent	Better primitive	Common trap
"Did the command I sent finish?"	wait_for_channel with tmux wait-for -S	Waiting for a prompt or regex in wait_for_text
"What is visible right now?"	snapshot_pane / capture_pane	Calling a future-only wait
"Which pane mentions this?"	search_panes	Listing pane metadata and missing terminal text
"What changed since I last looked?"	Proposed capture_since (#60)	Repeated full snapshots or a long blocking wait
"Did this exact third-party line appear?"	wait_for_text	Treating it as a generic completion primitive
"Did anything change?"	wait_for_content_change, then inspect	Treating changed=True as semantic success

The filed wait-family issues are concrete instances of that mismatch:

• wait_for_text matches stale scrollback immediately — needs baseline anchor #45 fixed stale scrollback instant matches by redefining wait_for_text as "new text after the call begins." That makes search_panes / snapshot_pane the correct answer for "is this already on screen?"
• wait_for_text's delays (90 seconds) is can lock agentic flows - blocking #59 covers the 90-second agent lockout when the expected pattern is wrong, misspelled, already scrolled away, or semantically too narrow.
• wait_for_text: optional include_baseline_row=True for in-place row writes #51 covers same-row output: printf, carriage-return progress bars, spinners, and status redraws can stay on the baseline row that wait_for_text deliberately skips to avoid stale matches.
• search_panes: visual-row capture misses wrap-spanning patterns #55 covers wrap-spanning search misses: tmux's window_pane_search() searches visual rows, so long build/test messages can split across the wrap boundary and silently miss on the fast path.
• wait_for_text: two-call state/capture race within a single poll tick #50 covers the two-call state/capture race: wait_for_text samples pane state, then calls capture-pane; tmux capture math is evaluated against live history in cmd-capture-pane, while history can move through grid_collect_history().
• wait_for_text: adaptive backoff to reduce subprocess pressure on parallel waits #52 covers subprocess pressure under parallel waits: a 50 ms poll interval can become many tmux subprocesses per second across panes and agents.
• wait_for_content_change: surface pane death and respawn as ToolError for parity with wait_for_text #53 covers lifecycle blindness in wait_for_content_change: pane death, respawn, and pane_pid changes are observation facts, not just content-diff edge cases.
• wait_for_text: expose risk_band_warned: bool in WaitForTextResult for clients that don't subscribe to log notifications #54 covers notification-only risk signals: ctx.warning() may not reach the model or programmatic caller, so best-effort correctness must be exposed in structured tool results.
• wait_for_channel and signal_channel block the FastMCP event loop (sync subprocess.run) #18 fixed the server-side version of this class for wait_for_channel: blocking subprocess.run() on the FastMCP event loop could stall the server. That is separate from agent lockout, where the server may stay healthy but the agent is still trapped inside an uninformative call.

The upstream substrates matter:

• tmux wait-for is the right zero-poll primitive for authored command synchronization; see cmd-wait-for.c.
• tmux grid/capture APIs are snapshots, not a durable log stream. History rollover, resize, alternate screen, and wrap behavior all affect what can be reconstructed from capture-pane, grid_collect_history(), and alternate-screen handling in screen.c.
• FastMCP can report progress through Context.report_progress(), and background tasks can send task status notifications, but UI/client notifications are not a substitute for agent-visible tool results.
• Python cancellation and timeouts need explicit async subprocess care. CPython documents asyncio.create_subprocess_exec() and asyncio.wait_for() as the async path, while subprocess.run(timeout=...) kills only the child it is managing after timeout; see asyncio-subprocess.rst and subprocess.py.
• Shell composition is portable only at the simple cmd; tmux wait-for -S channel level. zsh documents list/separator and status behavior in grammar.yo and params.yo. fish has different status variables, events, and prompt markers; see fish_for_bash_users.rst, language.rst events, and mark-prompt. Status-preserving recipes should be shell-specific, not implied as universal.

2. Avenues of investigation

Primary investment: build capture_since (#60) as the observation-first primitive. It should be non-blocking, cursor-based, and bounded by max_lines / max_bytes, returning actual pane output plus cursor and correctness metadata. The first call should be immediately useful by returning visible screen content and establishing a cursor; later calls should return only deltas. This lets the agent tail dev servers, watch tests, inspect multiple panes, and self-correct without betting on one regex.

Keep wait_for_channel as the deterministic authored-command primitive. When the agent sends the command, compose a signal into the command text and wait on tmux's channel:

$ command; tmux wait-for -S libtmux_mcp_done

That path is zero-poll inside tmux and avoids the prompt/regex/stale-output trap. The docs should keep the simple recipe portable, then add shell-specific status-preserving variants only after they are verified for zsh, fish, and POSIX-like shells.

Investigate FastMCP task=True as an optional enhancement, not the foundation. FastMCP supports task-enabled async tools through TaskConfig, server-side task docs (docs/servers/tasks.mdx), and client-side task=True calls (docs/clients/tasks.mdx). This may reduce client/server lockout for long operations, but it does not by itself solve the core problem: the agent still needs model-visible pane output while deciding what to do.

Investigate event-driven pane streams separately. tmux has real streaming surfaces: pipe-pane and control-mode output via control_write_output(). These may support a future stream_pane, but they introduce lifecycle management, cleanup, raw PTY bytes, ANSI/control-sequence handling, buffering, safety-tier concerns, and stateful subscriptions. They should not block the simpler capture_since path.

Tighten the existing wait/search tools while the observation primitive is explored. #50-#55 still reduce real traps: race accounting, same-row behavior, adaptive backoff, lifecycle parity, structured risk fields, and wrap-aware search fallback all improve the current surface even if the recommended workflow shifts away from blocking waits.

Rewrite docs and server instructions around scenarios rather than tool names:

Scenario	Recommended path
I launched a command and need completion	send_keys("cmd; tmux wait-for -S chan") + wait_for_channel("chan")
I need to inspect an already-running pane	snapshot_pane / capture_pane first
I need to tail or monitor output	capture_since once available
I need to search visible/current pane text	search_panes; use slow path where wrap-spanning text matters
I need a known future line from third-party output	wait_for_text with bounded timeout
I only need to know something changed	wait_for_content_change, followed by inspection

Recommended ordering:

1. Ship documentation improvements immediately so agents stop reaching for wait_for_text as a generic completion tool.
2. Implement capture_since as the main new primitive (feat: capture_since — non-blocking, cursor-based delta capture for agentic flows #60).
3. Continue the targeted correctness tickets (wait_for_text: two-call state/capture race within a single poll tick #50-search_panes: visual-row capture misses wrap-spanning patterns #55) where they remain valuable.
4. Add optional FastMCP task support only after client behavior and dependency cost are understood.
5. Treat pipe-pane / control-mode streaming as a separate future stream_pane design, not a retrofit of blocking wait tools.

This keeps the architecture simple: make observation cheap, explicit, and model-visible; reserve blocking waits for cases where blocking is actually the right abstraction.

[tony]

Triage pass (2026-05-24): keep this as an umbrella/tracker, not as a second implementation track competing with #60.

Current-code check: the repo already steers authored-command completion toward wait_for_channel in server instructions and docs, so the documentation action here should be narrowed to scenario polish after #60 lands rather than treated as wholly missing guidance. Also, #59 is closed/superseded and should be cited as provenance for the lockout problem, not as another live sibling ticket.

Link corrections for future body edits: use tmux capture math at cmd-capture-pane.c#L150-L160, wait-for at cmd-wait-for.c#L121-L190, zsh list/separator semantics at grammar.yo#L82-L90, zsh status variables at params.yo#L767-L785, and FastMCP links pinned to the project-relevant v3.3.1 tag such as Context.report_progress() and TaskConfig.

Overlap/value: completing #60 should close or substantially shrink this issue. Keep it open only while it provides architectural grouping for #50-#55/#60; after capture_since and the scenario docs land, this should either close as resolved or become a short checklist of remaining wait-family cleanups.

[tony]

Post-v0.1.0a9 audit (2026-05-24): current live issue set is #17, #49, #50, #51, #52, #60, and this umbrella #61.

PR #62 / v0.1.0a9 closed the targeted correctness tickets #53, #54, and #55, so they should be treated as resolved provenance rather than live sibling work. The remaining wait-family issues are now architectural or deferred:

• feat: capture_since — non-blocking, cursor-based delta capture for agentic flows #60 is the next primary action. A non-blocking capture_since will likely shrink or obsolete much of wait_for_text: two-call state/capture race within a single poll tick #50, wait_for_text: optional include_baseline_row=True for in-place row writes #51, and wait_for_text: adaptive backoff to reduce subprocess pressure on parallel waits #52 by moving exploratory observation away from blocking wait_for_text polling.
• wait_for_text: two-call state/capture race within a single poll tick #50, wait_for_text: optional include_baseline_row=True for in-place row writes #51, and wait_for_text: adaptive backoff to reduce subprocess pressure on parallel waits #52 should stay deferred until after feat: capture_since — non-blocking, cursor-based delta capture for agentic flows #60 lands or a real trace/stress test proves wait_for_text still needs those exact refinements.
• Compose / bulk operations: run multiple actions in one MCP tool call (start with send_keys) #49 is valuable and still relevant, but keep it homogeneous (send_keys_batch) and do not let it expand into heterogeneous send_keys -> wait -> capture orchestration that duplicates feat: capture_since — non-blocking, cursor-based delta capture for agentic flows #60/Observation-first architecture: eliminate agent lockout from blocking wait tools #61.
• MCP tool calls fail: invalid JSON when passing session_name / session_id (Cursor agent) #17 remains a Cursor/Composer compatibility record, not a libtmux-mcp handler bug. Its value is diagnostic/docs/upstream-routing, not server-side repair.

Source-link audit is still valid with pinned refs/tags already noted in issue comments: tmux 134ba6c, CPython 2b6a137, zsh 3e72a52, fish fb29c85, FastMCP v3.3.1, and libtmux 0.58.0/f70cb62 for the wrapper surface.