Typed operations, engines, workspace, query & MCP (#689)

.agents/skills/benchmarking-engine-builds/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: benchmarking-engine-builds
+description: Use when measuring or profiling how fast libtmux.experimental engines build tmux workspaces — comparing classic vs subprocess/control_mode/imsg/mock/pipelined, chasing a build-latency regression, reading percentile grids, or finding where a control-mode build spends its time (cProfile). Runs scripts/bench_engines.py hermetically on throwaway sockets.
+---
+
+# Benchmarking engine builds
+
+## Overview
+
+`scripts/bench_engines.py` times how long each experimental engine takes to
+build a tmux session structure (`W` windows × `P` panes-per-window), sweeping
+shapes × engines × wait-modes and reporting min/avg/median/p90/p95/p99/max.
+
+**Hermetic and safe to run beside a live tmux session:** every server gets its
+own socket under a throwaway `mkdtemp` dir, `TMUX` is unset before libtmux is
+imported, and an `atexit` hook kills every spawned server. The default tmux
+server is never contacted.
+
+It is a PEP 723 script — **always launch it with `uv run`**, never `python`, or
+its inline deps (`rich`, `typer`, editable `libtmux`) won't resolve.
+
+## When to use
+
+- Comparing engine build cost (which engine is fastest for a given shape).
+- Checking whether a change to the ops/plan/engine layer moved build latency.
+- Reading percentile spread (is p99 blowing out?) rather than a single number.
+- Locating the hot path inside one engine's build (`profile` → cProfile cumtime).
+
+## Quick reference
+
+Run from the repo root.
+
+| Command | What it does |
+|---|---|
+| `uv run scripts/bench_engines.py run` | full engine grid (the clean signal) |
+| `uv run scripts/bench_engines.py matrix --shapes 1x4,3x3,5x4` | 4-axis factorial: which choice drives build cost |
+| `uv run scripts/bench_engines.py concurrency --transport control_mode --k 4` | K builds sync-serial vs async-`gather` |
+| `uv run scripts/bench_engines.py contract` | mock-parity ops-language check only (for CI) |
+| `uv run scripts/bench_engines.py profile --engine control_mode --shape 8x4` | cProfile one engine, print slowest by cumtime |
+| `uv run scripts/bench_engines.py cell control_mode 8x4` | one isolated build (for wrapping in hyperfine) |
+
+`run` flags: `--shapes 1x1,1x4,3x3,5x4,8x4`, `--engines classic,subprocess,control_mode,imsg,mock,pipelined`,
+`--wait` (ALSO measure with shell-readiness wait), `--runs 20`, `--warmup 3`,
+`--json-out grid.json`. Shape is `windows x panes-per-window`.
+
+`matrix` sweeps five expression layers (`imperative`, `plan-seq`, `plan-fold`,
+`ws-seq`, `ws-fold`) × transport {subprocess, control_mode} × mode {sync, async}
+against a `classic` reference. `mock` is the offline correctness **oracle**, not
+a results row: `matrix --check` (default on) and `contract` assert every layer ×
+mode renders identical tmux argv to it, so the benchmark doubles as an
+ops-language contract test.
+
+Engines: `classic` (Server/Session/Window/Pane API) · `subprocess` (one fork
+per op) · `control_mode` (one persistent `tmux -C`) · `imsg` (AF_UNIX one-shot) ·
+`mock` (offline, in-memory Python floor) · `pipelined` (prototype: batch
+independent creates via `run_batch`).
+
+## Reading the results
+
+- **`control_mode` is the fastest shipped engine** (~21× classic at 32 panes)
+  because it avoids a per-op `tmux` fork. `pipelined` edges it (~1.4×) by
+  batching independent creates into ~3 round-trips.
+- **Builds are tmux-server-bound, not round-trip-bound** — one shell fork per
+  pane dominates, so cutting round-trips helps less than the count implies.
+  `mock` (~1–2 ms) is the pure-Python floor: the plan/compile layer is
+  negligible; the time is tmux.
+- **`profile` shows ~68% in `select.epoll.poll`** inside `_read_blocks`: each
+  created id is read back before the next op targets it. Latency-bound.
+
+## Common mistakes
+
+- Running with `python` instead of `uv run` — PEP 723 deps don't resolve.
+- **Comparing `--wait` against no-wait across engines.** Shell startup
+  (~0.8–2.1 s) dwarfs a fast build, so the ~20× engine win collapses to ~1.5×
+  once both sides wait. Compare engines with matching readiness policies.
+- Trusting hyperfine whole-process wall time over the in-process grid — Python
+  startup + import dwarfs a 3 ms build and understates the builder. The
+  in-process `run` grid is the clean signal.
+- Expecting `mock` under `--wait`: it has no real panes and is skipped.
+
+## Results & reproduction
+
+Committed results live in `scripts/bench-results/`: `RESULTS.md` (narrative +
+tables), `grid.json` (no-wait grid), `wait.json` (wait comparison). Regenerate
+the raw JSON with `--json-out`.

.agents/skills/testing-mcp-with-cli-agents/SKILL.md

@@ -0,0 +1,216 @@
+---
+name: testing-mcp-with-cli-agents
+description: >-
+  Test an MCP server by driving real CLI agents (Claude, Codex, Cursor, Gemini,
+  Grok, agy) against it — isolating each CLI's config and the server's own
+  backend state instead of trusting unit tests alone. Use this whenever
+  verifying MCP-server behavior end-to-end, checking that a local branch or
+  checkout works across installed agent CLIs, comparing trunk-vs-branch
+  behavior, driving an interactive agent TUI to exercise approval flows or
+  cancellation, or reproducing a bug through a live client. Reach for it even
+  when the user only says "test the MCP", "does the branch work in the agents",
+  "drive the CLI to call the tool", or "check it across Codex/Gemini/Cursor".
+---
+
+# Testing an MCP server through real CLI agents
+
+Unit tests prove the server's internals; they don't prove a real agent can
+discover a tool, clear its approval gate, call it, and survive cancelling it
+mid-flight. This skill exercises that whole path by pointing installed CLI
+agents at a checkout and driving them. Here the server is libtmux's
+tmux-control MCP (`libtmux-engine-mcp`, registered under the `libtmux` slug),
+and its backend-isolation lever is a scratch tmux socket
+(`LIBTMUX_SOCKET=<scratch>` → an isolated `tmux -L <scratch>` server) — the
+thing that scratches every side effect a tool call would otherwise make.
+
+## The core idea: isolate two things, never zero
+
+Driving an MCP server through a real CLI mutates two things you don't want
+touched. Isolate both and the whole exercise is safe and observable:
+
+1. **The CLI's config.** Use a throwaway config-home or project config so the
+   real `~/.codex`, `~/.claude.json`, etc. are never written. Each CLI's lever
+   is in `references/cli-matrix.md`.
+2. **The server's backend / side effects.** Point the server at a *scratch*
+   backend via its own env var or flag, so tool calls never touch real state
+   and you can assert against that scratch backend as independent ground truth.
+
+What "scratch backend" means depends on the server:
+
+| Server kind | Scratch-backend lever | Ground-truth check |
+|---|---|---|
+| tmux control (libtmux-mcp) | `LIBTMUX_SOCKET=<scratch>` → an isolated `tmux -L <scratch>` server | `tmux -L <scratch> list-windows` |
+| search / index (agentgrep) | a scratch index/store dir via the server's data-dir env/flag | inspect the scratch index, not the real store |
+| filesystem | a temp working root | check the temp tree |
+| external API | a sandbox/base-URL override or a recording | the sandbox's own state |
+
+The principle is identical everywhere: the server writes only to scratch, and
+you verify against scratch — so "the agent said it worked" is separated from
+"the tool actually did it," and a destructive tool can't harm anything real.
+
+## Climb only as high as the question needs — three fidelity layers
+
+### Layer 0 — Direct MCP smoke, no CLI at all
+
+Fastest and most deterministic. Drive the server over stdio from a tiny FastMCP
+client against a scratch backend and assert the wire contract directly: the tool
+list, a couple of representative calls, an error path. Use this to answer "is
+the tool surface and result shape correct?" before spending a CLI on it.
+Normalize result shapes before asserting — `structuredContent` is often
+`{"result": [...]}`, and single-value returns can arrive as a bare string.
+
+### Layer 1 — Headless CLI one-shot
+
+Proves the real client can discover and call the tools, scriptably, with no
+send-keys. Every CLI has a non-interactive mode. Run a cheap discovery proof
+first (does the client *see* the server?) — but the cheapest proof differs
+sharply per CLI: grok's `mcp doctor` does a real handshake, codex's `mcp get`
+only parses config, some CLIs have nothing short of a model call.
+`references/cli-matrix.md` has the verified per-CLI invocation, isolation lever,
+and approval-bypass flag. Two recurring surprises: some `mcp list`/`list-tools`
+subcommands read the *ambient* config and ignore your isolated one, and a
+mutating tool call needs a per-CLI approval-bypass flag or it hangs on a no-TTY
+prompt.
+
+### Layer 2 — Interactive, driven by tmux send-keys
+
+The high-fidelity path, and the only one that exercises approval flows, live
+streaming, multi-turn, and cancellation. Run the agent's TUI in a **harness**
+tmux socket (`tmux -L cli-harness`, separate from any socket the server itself
+uses) and drive it. Create a wide harness so the TUI does not wrap:
+
+```console
+$ tmux -L cli-harness new-session -d -s agent -x 220 -y 50
+```
+
+Launch the CLI with its isolated backend:
+
+```console
+$ tmux -L cli-harness send-keys -t agent 'cd /repo && <cli launch with backend isolation>' Enter
+```
+
+Poll until the prompt renders:
+
+```console
+$ tmux -L cli-harness capture-pane -p -t agent | tail -5
+```
+
+Type the task:
+
+```console
+$ tmux -L cli-harness send-keys -t agent 'Use the libtmux MCP to <do a thing>'
+```
+
+Submit it as a separate event:
+
+```console
+$ tmux -L cli-harness send-keys -t agent Enter
+```
+
+If prompted, answer the approval gate:
+
+```console
+$ tmux -L cli-harness send-keys -t agent 'y' Enter
+```
+
+Capture what the agent rendered:
+
+```console
+$ tmux -L cli-harness capture-pane -p -t agent | tail -30
+```
+
+Finally, assert ground truth against the scratch backend, not the transcript.
+Layers 0 and 1 can be fooled by a hallucinated success line; the scratch backend
+cannot.
+
+## Two failure modes that waste the most time
+
+**Approval gates hang naive harnesses.** The first tool use pops an approval
+dialog. A driver that types the prompt and immediately waits for output waits
+forever. Pre-approve with the CLI's trust/approval flags (see the matrix), or
+detect the prompt via `capture-pane` and answer its keystroke before waiting.
+
+**Sleeping instead of waiting is flaky, and blind typing doesn't submit.** Poll
+`capture-pane` for a stable completion marker rather than `sleep N`. Send the
+prompt text and `Enter` as **separate** `send-keys` events — then one Enter
+submits; batching text+Enter in one call is what leaves the prompt unsent. And a
+CLI launched inside a `-L` harness pane runs in a non-login shell that lacks your
+mise/node/uv shims, so `export` the needed bin dirs before launching it.
+
+## High-value test: cancellation / teardown
+
+Cancellation is invisible to the tool list and only reachable through Layer 2.
+With a long-running tool (a wait, a big scan): start it, then while the TUI shows
+"working / esc to interrupt" send `Escape` to that pane. `Esc` during the working
+phase cancels the in-flight tool call while keeping the MCP server subprocess
+alive — the exact client-cancellation a server's teardown path must survive;
+`Esc` after a turn finishes just enters edit-previous mode. Then assert the
+scratch backend is clean and no child process leaked.
+
+## Comparing two versions (trunk vs a branch)
+
+Two worktrees, two scratch backends, same prompt. Diff three things: the **tool
+surface** (a Layer-0 `tools/list` dump or `mcp list-tools`, diffed), the
+**rendered agent behavior** for the same prompt (capture-pane transcripts), and
+the **scratch-backend state** afterward.
+
+## Wiring a checkout into the CLIs: mcp_swap
+
+`scripts/mcp_swap.py` rewrites each CLI's config to run a local checkout, with
+backup/revert. Detect installed CLIs:
+
+```console
+$ uv run scripts/mcp_swap.py detect
+```
+
+Inspect the effective environment and configuration hazards:
+
+```console
+$ uv run scripts/mcp_swap.py doctor --server libtmux-engine
+```
+
+Check the current swap state:
+
+```console
+$ uv run scripts/mcp_swap.py status --server libtmux-engine
+```
+
+Preview a local swap:
+
+```console
+$ uv run scripts/mcp_swap.py use-local --server libtmux-engine --env KEY=VALUE --dry-run
+```
+
+Apply it:
+
+```console
+$ uv run scripts/mcp_swap.py use-local --server libtmux-engine --env KEY=VALUE
+```
+
+Revert the latest swap:
+
+```console
+$ uv run scripts/mcp_swap.py revert
+```
+
+Run `doctor` first — it reports which server name each CLI points at (and warns
+when the repo is registered under a name other than the derived default),
+un-reverted swaps and orphaned backups, missing backups (revert would fail), and
+auth-overriding env vars like `OPENAI_API_KEY`. Use `--env` to inject the
+backend-isolation var (e.g. an isolated socket or data dir) at swap time.
+
+**Prefer zero-mutation isolation for a test.** mcp_swap is for a swap you *want*
+to persist. To just exercise a checkout, use each CLI's throwaway
+config-home / project-config lever (`references/cli-matrix.md`) — all were
+verified to drive the server with the real config confirmed byte-identical
+afterward, and no swap state touched. `use-local` mutates real configs, so
+dry-run first and always `revert` at the end; and the machine may already carry
+an un-reverted swap, so `revert` returns you to *that* state, not a pristine one
+(check `doctor` first).
+
+## When NOT to reach for the full harness
+
+If the question is purely "is the tool surface correct?" stay at Layer 0 —
+booting six CLIs to answer a wire-contract question is wasted effort. Escalate to
+Layers 1 and 2 only when the client's discovery, approval, streaming, or
+cancellation behavior is what's actually in doubt.