tower + vscode: terminal freeze from oversized-replay reconnect storm (unbounded no-newline buffers + client backpressure loop)

[amrmelsayed]

✅ CONFIRMED ROOT CAUSE (2026-06-15)

Captured data (PTY disk logs + a 42,143-line VSCode terminal log + code trace) confirms the mechanism end to end. The original hypotheses below (Tower-wide WS pump degradation, broadcast-list leak, socket backpressure) are superseded — the real cause is an oversized-replay reconnect storm caused by two independent defects that chain:

Tower/shellper side — the root cause

Two terminal buffers bound themselves by newline count, so a full-screen TUI that redraws in place (alternate screen buffer, cursor-addressing + \r, almost no \n) makes them grow without limit:

• RingBuffer.partial (packages/codev/src/terminal/ring-buffer.ts:36-51): combined = partial + data; combined.split('\n') every frame, keeping the trailing fragment. No \n → partial grows unbounded and every frame re-scans the whole thing (O(partial)/frame → O(n²)/session). No byte cap.
• ShellperReplayBuffer (packages/codev/src/terminal/shellper-replay-buffer.ts:45,58, fed at shellper-process.ts:143): evicts only while lineCount > maxLines, so with zero newlines it never evicts.

On every (re)connect Tower sends the buffer as one frame: ws.send(encodeData(getAll().join('\n'))) (tower-websocket.ts:62-65) → a multi-MB replay payload.

VSCode side — the amplifier

The terminal adapter's backpressure guard trips on the replay frame itself and reconnects with no backoff:

• handleData (packages/vscode/src/terminal-adapter.ts:300-308): queuedBytes += payload.length; if (queuedBytes > MAX_QUEUE /*1 MB*/) reconnect() — fires before rendering anything.
• reconnect() (terminal-adapter.ts:281-296) calls backoff.reset() and reconnects instantly → Tower re-sends the same oversized replay → infinite loop.

Why it presents the way it does

• CPU ~93% (one core), all terminals freeze at once: Tower re-serializes a multi-MB replay thousands of times on its single event loop, starving every other terminal. Input stays responsive (tiny), output is frozen — matching the stray-input-char screenshot.
• Memory barely moves: the cost is CPU + GC churn, not retained memory.
• Restart is NOT reliable (corrects the original title): ShellperReplayBuffer is also unbounded, so after restart the shellper replays the multi-MB blob, Tower re-seeds an oversized partial, and the storm resumes — unless the offending session has gone quiet/ended.

Captured evidence

• Storming terminal f2dc55d1 disk log: 1.9 MB, 0 newlines → replay 1.9 MB > 1 MB MAX_QUEUE.
• Incident-window log f02bedcb: 15 MB, 0 newlines (1.5M ESC, 164K CR, 0 LF — a Claude alt-screen redraw stream). Normal sessions: ~20 bytes/line.
• VSCode log (~58 min): 14,026 "WebSocket connected" / 14,015 "Backpressure exceeded 1 MB" / 14,026 "Connecting to" (1:1:1); 14,017 of the reconnects target the single terminal f2dc55d1.
• Threshold is low: only ~1 MB of no-newline output triggers the storm.

Fix (see codev/plans/1047-... on builder/pir-1047)

Three coordinated fixes: (A) RingBuffer.pushData scans only new data + byte-caps partial (< MAX_QUEUE); (B) ShellperReplayBuffer gains a byte cap; (C) the VSCode backpressure path must not infinite-loop on an oversized replay (backoff + re-trip guard, ideally excluding the replay burst from the live-backpressure budget). A and B remove the oversized replay at the source; C makes the client structurally immune. Scope now spans the terminal core and the VSCode extension (re-labeled area/cross-cutting).

---

Original report (investigation history — hypotheses superseded by the confirmed cause above)

Symptom

Over time, architect AND builder PTY terminals stop rendering output and accepting input that propagates back to the visible UI. The symptom affects ALL terminals at once, not individual sessions. The only known recovery is afx tower stop && afx tower start (the default non-force variant).

Recent screenshot of an affected architect terminal: only Ran 1 shell command remained rendered, plus a stray input character (the user's e) at the bottom-left corner where the prompt frame should be. The TUI box-drawing of the prompt panel was gone, but the typed character + cursor block still appeared, suggesting the input side still propagates but the render-back side is broken.

What the diagnosis rules out

Per #1030 / PR #1031 (just landed), the default afx tower stop does NOT kill shellpers. Shellpers and the AI CLI processes they host survive the restart. Yet restart restores responsiveness, so:

• The AI CLI is not hung (it would still be hung after a non-force restart).
• The shellper process is not crashed (it survives the restart too).
• The PTY itself is not dead.

What actually changes during the default-stop / start cycle:

1. Tower's WebSocket connections drop.
2. The VSCode extension's terminal client detects disconnect and reconnects to the new Tower process.
3. New Tower runs reconcileTerminalSessions, reattaches to the still-alive shellpers, and resumes streaming.

So the bug lives in the Tower-to-client streaming layer or the per-terminal client-list state that Tower maintains, not in the AI CLI / shellper / PTY.

Tower-side log behavior

Tower itself stays "alive" in the process sense during the hang: SSE heartbeats keep firing every 30s, cron tasks keep running, the team scanner keeps iterating. So this is not a process-wide event-loop deadlock. It's specifically the WS-to-terminal pump that's wedged.

Top hypotheses (ordered by fit with the all-terminals-at-once pattern)

1. Tower-wide WS send-pump degradation. Memory growth, GC stalls, or an unbounded handler/listener leak that gradually starves the WS output loop. Affects all terminals because they share the same Tower process. Possibly worsened by repeated errors that throw past their handler scope without cleaning up (the cron evaluator ReferenceError filed separately is a candidate contributor).
2. Leaked closed-connection entry in the per-terminal broadcast list. Tower iterates its client list to fan out PTY output; a dead WS still in the list blocks the broadcast for everyone. Restart drops the list.
3. Shellper-to-Tower unix-socket backpressure. Shellper streams output to Tower over its unix socket; if Tower stops draining fast enough, shellper's pipe fills and stops producing. Restart drops the connection, shellper opens a fresh one on reattach.

Diagnostic plan for the next occurrence

1. While hung, observe afx tower log -f — does the SSE heartbeat keep firing every 30s? If yes, event loop is alive; the WS pump is the culprit (H1 or H2). If no, deeper event-loop starvation.
2. ps -p <tower-pid> -o pid,vsz,rss,pcpu while hung. RSS growth into multi-GB territory points at H1.
3. lsof -p <tower-pid> | wc -l while hung. A climbing fd count points at leaked connections (H2-adjacent).
4. From a side terminal: afx send architect "echo ping-from-side". Watch the Tower log for the send event AND watch the affected architect tab for the echoed output. If Tower logs the send but the tab never shows ping-from-side, it's pinned to the Tower-to-VSCode WS direction (not the input side).
5. Open the same architect session in the Tower dashboard at http://localhost:4100. If the dashboard renders fresh output while the VSCode tab stays frozen, the VSCode-side xterm.js renderer is in the loop too (downgrades H1/H2 to "VSCode-extension-specific WS handling" instead of "Tower-side").

Why this is PIR, not BUGFIX

Root cause is not yet known — three plausible mechanisms with overlapping symptoms. Investigation phase (capture diagnostics, narrow to a single mechanism, then design the fix) is the right shape. Pre-PR running-on-real-workload verification is load-bearing because the bug only manifests after time-in-use.

Out of scope

• The unrelated ReferenceError: exitCode is not defined cron evaluator bug visible in the same Tower log window. Filed separately.
• Any change to default tower stop shellper-killing behavior. Shellpers must continue surviving restart per Architect terminal should survive Tower restarts (shellper persistence) #274 / Multi-architect conversation resume: disambiguate via per-architect session UUID #832 / Terminals survive a Tower restart: preserve terminal id + stop afx tower stop killing port clients (#991) #999 / terminal: stale tab on a pre-restart terminal id can't self-recover without a manual state re-fetch #991.

[amrmelsayed]

Captured incident snapshot (2026-06-14, ~10.5h Tower uptime)

Symptoms in flight when measurements were taken:

• SSE clients dropping from 4 → 2 over the previous ~5 minutes
• [INFO] SSE client removed (dead): <id> lines appearing (Tower keepalive detecting unresponsive clients)
• Followed by reconnection cycles seconds later

Metric	Baseline (Tower 3m26s old)	Incident (Tower 10h32m old)	Change
CPU	0.6%	93.0%	+92.4 pp
RSS	157 MB	233 MB	+76 MB
FDs	60	69	+9
Active SSE clients	4	2 (was 4 at 21:46)	-2, churning

Refines the hypothesis from the issue body

The 93% CPU number argues AGAINST H1 (memory-leak → GC stalls) and toward a different mechanism: a tight CPU-bound loop in some handler is starving the WS / SSE pumps. Tower's event loop is NOT deadlocked — SSE heartbeats still fire on schedule every 30s — but the CPU is so saturated that message delivery to terminal clients can't keep up. Clients eventually fail their keepalive, get marked dead, and reconnect. From the architect's seat, that reads as a frozen terminal until restart.

Memory growth of 76 MB over 10.5 hours is real but minor and probably orthogonal. FD growth of 9 over 10.5 hours is glacially slow and probably orthogonal. The CPU is the load-bearing signal.

Candidates for the CPU hot loop, ordered by fit

1. The cron evaluateCondition throw cycle. The Service Health Check task throws ReferenceError: exitCode is not defined on every tick (see tower: cron condition evaluator throws ReferenceError: exitCode is not defined #1048). If the retry-on-error path in tower-cron.ts near line 249 doesn't apply backoff after a throw, every workspace's copy of the broken cron (~13 workspaces register it given the team-scanner output) is hot-firing repeatedly without delay. CPU climb correlates with uptime, which fits — the longer Tower runs, the more times the error path is hit. Worth checking the cron-retry loop semantics first.
2. SSE broadcast loop with a leaked serializer. If broadcast iterates clients and serializes a growing in-memory buffer per client, CPU climbs as the buffer grows. Would be visible by inspecting per-client buffer sizes.
3. Team scanner doing synchronous recursive walking that isn't gitignore-aware. The team-update.yaml skip-warns in the log are cheap; the scan walking would only be visible by profiling.

Visible incident log lines (verbatim)

[2026-06-14T21:49:48.585Z] [INFO] SSE heartbeat: 2 active client(s)
[2026-06-14T21:50:04.053Z] [INFO] SSE client disconnected: 666e61640bcf990c
[2026-06-14T21:50:18.586Z] [INFO] SSE client removed (dead): 263532f30d12ba70
[2026-06-14T21:50:18.586Z] [INFO] SSE client disconnected: 263532f30d12ba70
[2026-06-14T21:50:19.607Z] [INFO] SSE client connected: 1c907396e2bdb8f5
[2026-06-14T21:50:28.739Z] [INFO] SSE client connected: 560bf5d3ffd21cbe

First investigation step for whoever picks this up

Inspect packages/codev/src/agent-farm/servers/tower-cron.ts around line 249 (shouldNotify = evaluateCondition(task.condition, output.trim())). Specifically:

• Is the throw caught in a way that allows the next tick to fire without backoff?
• Is there an interval set up per task, per workspace, that survives the throw?
• Does fixing tower: cron condition evaluator throws ReferenceError: exitCode is not defined #1048 (the ReferenceError) also fix this issue, or is the retry-loop pathology a separate concern?

If #1048's fix happens to suppress the throws and CPU stays low after 10+ hours, then evaluateCondition failures were just the manifestation of a deeper retry-loop bug that should still be addressed defensively. If CPU still climbs after #1048 lands, candidates #2 and #3 above become the next investigation targets.

[amrmelsayed]

Regression-suspect window: v3.1.7 → v3.1.8

This may be a regression from a recent merge. Symptom onset coincides with the v3.1.7 / v3.1.8 release window.

Release timeline (Tower-runtime-affecting changes)

Release	Date	Tower-runtime changes
v3.1.5	2026-05-30	(pre-symptom baseline candidate)
v3.1.6	2026-05-31	minor
v3.1.7	2026-06-03	PIR #991 (terminal reconnect: close-on-stop / reopen-on-reconnect / successor-session resolution); PIR #997 (startup readiness barrier gating WS upgrades); PIR #983 (/api/version endpoint + TowerClient.getVersion probe); PIR #971 (4404 close code handling)
v3.1.8	2026-06-08	PIR #859 (markdown preview + core review-marker codec); PIR #907 (ResolvedEnrichmentCache); PIR #787 (PR reviewRequests + isDraft fields in OverviewPR mapping)
v3.1.9	2026-06-08	hotfix republish of v3.1.8 (no code changes)

Candidates ranked by leak-shape match

The symptom is CPU climbing linearly with Tower uptime (0.6% at t=3m → 93% at t=10h), SSE clients getting marked dead by Tower's own keepalive, restart resolves. That fits a per-message hot path doing work proportional to a growing internal list. Candidates from the release window, ordered by how well each matches that shape:

1. PIR terminal: stale tab on a pre-restart terminal id can't self-recover without a manual state re-fetch #991 (terminal reconnect) — added WS reconnect logic and resolveSuccessorTerminalId. Reconnect cycles are the classic listener-leak surface: if every disconnect → reconnect adds a new listener without removing the old, the hot WS-message path inflates linearly with uptime. Strongest fit.
2. PIR tower: reconcile terminal sessions before serving requests (kill the restart successor-lookup race) #997 (startup readiness barrier) — gated WS upgrades on a barrier. If the gating accumulates state per upgrade, this inflates the upgrade hot path with uptime.
3. PIR vscode: builder briefly appears under 'Uncategorized' area group after cleanup before disappearing #907 (ResolvedEnrichmentCache) — explicit cache with a prune method (packages/codev/src/agent-farm/servers/resolved-enrichment-cache.ts:69-73) that the inline comment claims prevents unbounded growth. Worth verifying whether prune is actually called by the overview aggregator on every refresh — if it isn't, the map grows.
4. PIR vscode: add review comments from the markdown preview pane (hover-+ per block, no editor mode-switch) #859 (markdown preview) — adds a new CustomTextEditor + core codec; less obviously Tower-side but introduces new WS-adjacent surface area.
5. PIR vscode + tower: detect installed-vs-running Tower version divergence; preflight needs an in-memory version probe, not just codev --version #983 (/api/version probe) — VSCode preflight polls this; if each probe accumulates state in Tower, that's constant-time pressure.

What's already ruled out (don't re-investigate)

• Cron registry growth. Captured 6 consecutive bursts at ~10h uptime: 15 firings per burst, constant across bursts. The cron registry is stable; firings are bursty and bounded (15 firings in 1-4 seconds, then nothing for 14+ minutes). Sustained 93% CPU is NOT explained by cron. The cron ReferenceError filed in tower: cron condition evaluator throws ReferenceError: exitCode is not defined #1048 is real but a separate concern — fixing it won't fix this issue.
• SSE pump itself. The code at tower-server.ts:192-262 is well-defended against leaks (removeDeadSseClients splices dead entries; both broadcastNotification and the heartbeat detect client.res.destroyed || writableEnded and remove; 5-minute max-age eviction per Bugfix [Bug] SSE connections leak — server never cleans up disconnected clients #580; heartbeat interval is .unref()'d). The leak is somewhere else.
• Memory-leak → GC stall mechanism. RSS grew only +76 MB over 10h (157 MB → 233 MB), not multi-GB. CPU climb without memory bloat rules out the GC-stall hypothesis.

Quickest validation move

Install v3.1.5 globally and leave Tower running 12+ hours; watch CPU.

npm install -g @cluesmith/codev@3.1.5
afx tower stop && afx tower start
# leave running 12+ hours, observe ps -p $(pgrep -f tower-server.js) -o pcpu

If CPU stays low on v3.1.5 but climbs on v3.1.7+, the regression is confirmed to be in git log v3.1.5..v3.1.7 -- packages/codev/src/agent-farm/servers/ packages/codev/src/terminal/. Bisecting that range narrows it to the specific PIR.

Sampling note

A sample $PID 30 -file /tmp/tower-sample-hot.txt of Tower at the high-CPU state would pin the hot function. macOS sample is built-in and doesn't require a restart or --inspect. Catch it at the next occurrence before restarting.

[amrmelsayed]

Conceptual sibling: #1038

Worth reading alongside this issue. Different observable symptoms, but they share thematic territory and likely the same architectural surface.

	This issue (#1047)	#1038
Symptom	Existing terminals freeze, restart resolves	New terminal spawns fail (posix_spawnp failed)
State that leaks	In-memory JS bookkeeping in Tower	OS-level shellper processes
Lifetime of the leak	Within one Tower's lifetime	Across many Tower restart cycles

Both trace back to the persistence-across-restart design line (#274 / #832 / PR #991 / PR #999), which deliberately made shellpers detach from Tower and survive its restart. That design introduced two related bookkeeping responsibilities:

• Tower tracks "shellpers I know about" — if that bookkeeping leaks in-memory, you get this issue's symptom.
• Tower decides "which shellpers to reap on next start" — if that decision is too conservative, you get Orphan-reap responsive-but-unclaimed shellpers at startup to fix PTY pool exhaustion #1038's symptom.

PIR #991 is the strongest leak-shape candidate for this issue per the earlier comment, and PR #1031 (recently merged) specifically referenced the #991 context when discussing the persistence contract. #1038 was filed as the follow-up to that same line. The fingerprints of #991, #1031, #1038, and this issue all converge on the same persistence design surface.

Practical implication for whoever picks this up: if the in-memory leak turns out to be in shellper tracking specifically, the fix may need to coordinate with the orphan-reap semantics #1038 proposes. And if #1038 lands first and changes how Tower decides which shellpers to keep tracking, that could shift the surface where this issue's leak lives.

[amrmelsayed]

On it! Working on this with the PIR protocol (plan + dev-approval gates before PR).

[amrmelsayed]

Log

vscode-log.txt