[orchestrator-v2] fix(orchestrator): Harden Grok v2 settlement, background task lifecycle, steer visibility, and images#3578
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
size:L
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| () => new Set(serverProjection?.messages.map((message) => message.id) ?? []), | ||
| [serverProjection], | ||
| () => deriveCommittedServerUserMessageIds(serverVisibleTurnItems), | ||
| [serverVisibleTurnItems], |
There was a problem hiding this comment.
User-only count breaks env locking
Medium Severity
committedServerMessageIds now holds only visible user_message turn item ids, but activeMessageCount still uses that set’s size for envLocked and canOverrideServerThreadEnvMode. During the projection-vs-turn-item gap (and vs all projection.messages), the thread can look message-empty and allow env overrides that used to be blocked.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 61f4adb. Configure here.
There was a problem hiding this comment.
Out of scope — steer fix intentionally keys off visible user turn items. Brief projection gap only. Out of scope in PR description.
ApprovabilityVerdict: Needs human review 7 blocking correctness issues found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR. You can customize Macroscope's approvability policy. Learn more. |
| * `_x.ai/session/prompt_complete` notifications (sessionId-matched). Subagent | ||
| * completions on foreign session ids are ignored by the pending-entry gate. | ||
| */ | ||
| export const makeXAiPromptCompletionRuntime = Effect.fn("makeXAiPromptCompletionRuntime")( |
There was a problem hiding this comment.
🟡 Medium acp/XAiAcpExtension.ts:455
The dedup guard in resolveXAiPromptCompletionFallback only checks completedPromptIdsRef when notification.promptId is defined. When a _x.ai/session/prompt_complete notification arrives without a promptId, the guard is skipped and the code resolves the first pending entry matching by sessionId alone. Because makeXAiPromptCompletionRuntime registers a pending entry for every new prompt before the RPC completes, a late duplicate notification from a previous turn (without promptId) will match the new prompt's pending entry and resolve it with the stale notification's stopReason and agentResult instead of waiting for the new prompt's own response. Consider tracking completed sessions alongside completed prompt IDs so that promptId-less notifications can be deduplicated per session.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/provider/acp/XAiAcpExtension.ts around line 455:
The dedup guard in `resolveXAiPromptCompletionFallback` only checks `completedPromptIdsRef` when `notification.promptId` is defined. When a `_x.ai/session/prompt_complete` notification arrives without a `promptId`, the guard is skipped and the code resolves the first pending entry matching by `sessionId` alone. Because `makeXAiPromptCompletionRuntime` registers a pending entry for every new prompt before the RPC completes, a late duplicate notification from a previous turn (without `promptId`) will match the new prompt's pending entry and resolve it with the stale notification's `stopReason` and `agentResult` instead of waiting for the new prompt's own response. Consider tracking completed sessions alongside completed prompt IDs so that `promptId`-less notifications can be deduplicated per session.
There was a problem hiding this comment.
Out of scope — serialized root prompts limit blast radius today. Out of scope in PR description.
There was a problem hiding this comment.
Sorry, I'm unable to act on this request because you do not have permissions within this repository.
4c4d0dd to
8f6ec37
Compare
github-actions
Bot
added
size:XL
ddfedee to
e84f666
Compare
ba852af to
0583e10
Compare
Mark background tools terminalized mid-turn as handled so late CLI re-reports and residual agent chatter do not open synthetic "Background task completed." continuation runs after the root already reported the monitor result in-turn.
Only mark completed (post-normalize) background tools as in-turn handled, not interrupted or failed. When residual agent chatter follows handled work, skip synthetic continuation offers without clearing the wake buffer so other tasks can still drain.
Completing a monitor after STARTED under deferred finalize must not mark the task handled, or post-settle TaskOutput cannot open a continuation (grok-post-settle-continuation-poll). Keep no-wake marking when the tool completes before the root settles.
Before prompt settle, treat failed and interrupted background tools like completed for handled-set purposes so residual CLI chatter does not open synthetic post-settle wakes. Still skip marking after STARTED so deferred finalize TaskOutput continuations keep working.
A steering interrupt on a turn whose native prompt already settled used to hard-kill the Grok process group, destroying still-running fire-and-forget subagents. Live ACP experiments showed the Grok CLI accepts a concurrent session/prompt in that state and that session/cancel kills background subagents, so the settled steer now skips the process-group kill, the ACP cancel, and the runtime respawn behind a new preserveRuntimeOnSettledInterrupt flavor flag (enabled only for Grok). User Stop and mid-prompt interrupts keep the hard teardown path.
…nded tasks Grok session/cancel is detach-and-continue: the CLI re-runs cancelled foreground work as a background task no client RPC can kill (E3 harness, 2026-07-18). User Stop therefore keeps the hard process-group kill and respawn, but mid-prompt steering and restart_active now stay soft: drop restartRuntimeOnEveryInterrupt so the session survives in the same process, and let the model itself decide the fate of backgrounded work via its native kill_command_or_subagent tool. Track that backgrounded work as first-class background tasks: subscribe to _x.ai/task_backgrounded and _x.ai/task_completed through a new applyBackgroundTaskMutation extension-context hook (gated to the root session), and tombstone tasks the model kills, since Grok emits no task_completed for killed tasks.
Pre-push review fixes for the soft non-Stop interrupt change: - Gate applyBackgroundTaskMutation on stoppedRunQuarantine so residual task lifecycle from a stopped run cannot mutate wake machinery, and on the active session id so a cancelled subagent's re-run in its child session cannot gate root wake state. Post-Stop the activeSessionId still points at the stopped session until the next turn respawns, so the quarantine is the only guard in that window. - Fix the stale preserveRuntimeOnSettledInterrupt contract comment to describe the current interrupt matrix. - Add tests: direct Stop drops late background task mutations from the stopped run, and the production Grok flag set (no restartRuntimeOnEveryInterrupt) still hard-kills and respawns on user Stop, proven by session/load and runtime ordinal 2 on the next turn.
| // untouched for the replacement turn. | ||
| if (!settledSoftInterrupt) { | ||
| yield* runtime.cancel; | ||
| } |
There was a problem hiding this comment.
Settled-steer cancel race
Medium Severity
settledSoftInterrupt snapshots promptSettled once, but cancel is gated on that stale flag while finalize later reads live context.promptSettled. Soft interrupts do not hold runtimeCallbackPermit, so the prompt-completion fiber can mark the turn settled (or return while interrupted blocks the defer path) before cancel runs. session/cancel then still fires and can kill the background subagents settled-steer is meant to preserve.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit cc13ec5. Configure here.
| return undefined; | ||
| } | ||
| } | ||
| const context = interruptContext; |
There was a problem hiding this comment.
Stop no-ops after soft steer
High Severity
After a soft mid-prompt steer clears activeTurn without killing the process, a queued user Stop for the same turn hits the already-interrupted early return and skips hard teardown. runtimeRestartRequired and stoppedRunQuarantine never arm, so Stop fails to contain cancel-detached Grok work.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit cc13ec5. Configure here.
Runtime teardown paths that poll real kernel state (cgroup emptiness, process-tree reaping, TERM grace) paced their retries with Effect.sleep on the ambient Clock. Under @effect/vitest it.effect the TestClock never advances, so a scope close that raced a still-populated cgroup slept forever, hanging every GrokTextGeneration test for its full 120s timeout and pushing the serial server suite past the 10-minute CI job budget. Provide the default live Clock to terminateLinuxCgroupLease, terminatePosixOwnedProcessTree, observePosixOwnershipLedgerContinuously, and terminateOwnedProcessGroupImpl. Virtual time is never correct for these sleeps: they pace observations of external OS state.
A soft steering interrupt can clear the active turn while intentionally leaving the runtime process alive. A user Stop arriving afterward hit the cleared-turn early return and no-oped, leaving the orphan process running. Stop now quarantines the transport, tears down the process group, and forces a runtime respawn on the next turn. Also read prompt settlement under the runtime callback permit after draining admitted native responses, so a prompt that returned on the wire but whose completion callback has not yet run still classifies as settled and skips session/cancel.
lease.remove() failures during teardown are swallowed best-effort, so empty t3-acp-<pid>-<uuid> directories can accumulate under the parent cgroup across app restarts. Before creating a new lease, sweep sibling lease directories whose embedded creator pid is dead and whose cgroup.events reports populated 0. All sweep errors are ignored and never delay or fail lease creation.
…op teardown Three follow-ups on the orphan-containment and settled-steer code: - Add a wire-settlement signal completed the moment session/prompt resolves, before the completion callback contends for the callback permit. Settled-soft classification ORs it with promptSettled, so an interrupt that wins the permit ahead of the completion fiber no longer sends a redundant session/cancel that would kill background subagents. - Cancel pending approval/elicitation requests in the orphan Stop path, matching the main hard-restart path, so a request raised by the orphaned agent cannot leave the composer approval-locked forever. - Terminalize carried-over subagents in the orphan Stop path with interrupted node/subagent/turn_item events, so soft-steered subagent cards do not stay running in the transcript after the process is killed.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
There are 6 total unresolved issues (including 4 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6add8c0. Configure here.
| childThreadId: subagent.childThreadId, | ||
| prompt: subagent.task.prompt, | ||
| result, | ||
| }, |
There was a problem hiding this comment.
Carryover Stop drops subagent text
Medium Severity
When terminalizeCarryoverSubagents processes a subagent stop after a soft steer, it only uses subagent.task.result for the final output. This omits streamed partial results, causing the parent subagent card to lose previously projected output.
Reviewed by Cursor Bugbot for commit 6add8c0. Configure here.
| const providerTurn = turns.get(String(subagent.providerTurnId)); | ||
| const parentProviderThreadId = | ||
| providerTurn?.providerThreadId ?? subagent.task.providerThreadId; | ||
| if (parentProviderThreadId === null) continue; |
There was a problem hiding this comment.
Null parent thread skips interrupt
Medium Severity
When resolving the parent providerThreadId, terminalizeCarryoverSubagents falls back to subagent.task.providerThreadId and continues if that value is null. Unbound child sessions keep task.providerThreadId null, and unlike emitSubagent there is no stored parent thread id on the carryover record. Any missed providerTurns lookup therefore skips emission and leaves the subagent running after Stop—the failure mode this change is trying to close.
Reviewed by Cursor Bugbot for commit 6add8c0. Configure here.
1 participant
Fixes #3580
Summary
Harden Grok orchestration v2 so root turns complete correctly, steered user
messages stay visible, Stop then continue recovers from a poisoned ACP child,
async
spawn_subagent/monitorproject real lifecycle (not start ACKs), latepost-settle Grok traffic can land as a continuation run instead of being
dropped, screenshot / image attachments actually reach Grok instead of failing
turn start, and interrupt cleanup actually contains and reaps native Grok work
(cgroup kill / owned process ledger) instead of leaving zombie shells.
Follow-up commits soften non-Stop interrupts: steer / restart_active now cancels
and re-prompts the same live session instead of killing the CLI, a steer on a
settled held turn preserves the runtime (running subagents survive), and user
Stop keeps the unchanged hard kill + respawn.
Primary commits on
fix/grok-v2abovet3code/codex-turn-mapping(thefeature story reviewers should start with):
Follow-up commits may land on top of those two. Treat them as reviewable
delta fixes, not a rewrite of the primary story. Integration snapshots should
still merge the whole branch, not tip-only cherry-pick.
The shared provider-continuation plumbing (C0) already landed in the base branch
alongside the Claude and Codex consumers.
What you'll notice
Steered messages stay visible
When you steer an in-flight Grok run, your message no longer flashes away
between server events. The timeline keeps the optimistic user row until the
committed
user_messageturn item lands.Steering also works while a run is held open for background work: pre-fix,
steering a settled turn that deferred finalize was holding (a subagent still
running) froze the thread at Working, the steer text never reached Grok, and
later messages queued forever.
Grok multi-tool turns finish with tools and final text
Grok often emits a short assistant preamble, then tools, then more text.
Speculative idle settlement used to complete the turn after a short quiet and
call
session/cancel, which froze T3 projection while the CLI kept working.This PR disables Grok idle settle and terminalizes from real provider signals.
Stop then continue recovers instead of
task_already_runningAfter you Stop a Grok run mid-turn, the next message restarts the Grok ACP child
before the recovery prompt, clearing zombie shell state.
Interrupt cleanup contains native work
Stop is not only an ACP cancel. On Linux with cgroup v2, delegated Grok launches
join a unique child cgroup and interrupt uses
cgroup.killas the authoritativecleanup. When cgroup containment is unavailable, the owned-process ledger is the
fallback (Darwin process groups / Windows
taskkillretained). ACP cancel isgeneration-aware; unexpected transport death, double-fork descendants, and
teardown races are covered by focused tests.
Steers no longer kill the Grok process
The interrupt matrix after the follow-up commits:
session/loadon the next turnsession/cancel+ same-session re-prompt; native context survives without asession/loadreplayGrok's
session/cancelis detach-and-continue: the CLI reaps the foregroundshell but re-runs the interrupted command as a background task, and only the
model can kill it (via
kill_command_or_subagent). Product call: the steeredmodel, not the client, decides whether that detached work dies, matching the
CLI's own behavior. The adapter tracks
_x.ai/task_backgrounded/task_completedinto background-task mutations and tombstones model-killedtasks so wake machinery stays truthful, and a direct Stop quarantines any late
lifecycle noise from the stopped run.
spawn_subagent and monitor project real work
Async
spawn_subagentno longer completes on the background ACK alone. T3 bindsthe child id, keeps the subagent running, shows durable
subagent.resultwhenavailable, and hydrates from TaskOutput /
get_command_or_subagent_outputwhenneeded.
monitorstays running through the start ACK, streams as commandexecution, and completes from Bash exit codes or monitor-end text.
Late “will report when finished” can wake again
After the root turn settles (including when a subagent card already has a
result), Grok may still stream a late root follow-up (tool fetch + confirmation)
without a new
session/promptfrom T3. That traffic is buffered and attached toa provider continuation run so the UI can go idle, then show Working again
with the follow-up, instead of dropping frames after finalize.
Long-running monitors initially made this path spam: Grok narrates every
monitor event (~3s apart, wider than the ~2s finalize quiet window), and each
burst opened its own continuation run with its own synthetic "Background task
completed." user row (4+ observed live). Continuation offers are now fully
suppressed while a tracked background task is still running: per-event agent
commentary AND tool re-reports buffer (Grok re-reports a running monitor as
Bash frames that already carry
exit_code: 0mid-stream, so a terminal-lookingtool status is not evidence the task ended), and exactly one continuation run
opens once the monitor actually ends, draining the buffered commentary.
In-turn monitors do not open a redundant wake
When Grok waits on a monitor and reports the listing in the same root turn
(no post-settle wait), late CLI re-reports and residual agent chatter must not
open synthetic "Background task completed." continuation runs. The follow-up
commit marks those tools handled mid-turn and suppresses residual agent-message
offers once handled work is done and nothing is still running.
Settled turns still get the injected monitor report
When Grok settles first ("I'll report when the monitor ends") and the CLI later
injects a monitor-event turn, the report could vanish: injected turns emit no
turn_completed, and the deferred-finalize debounce could fire while Grok wasstill generating the report, dropping the chunk into an already finalized turn
(thread
a8e8b0a9run 5). The run now holds until the report streams (or a 25ssafety elapses), so the listing lands as a normal assistant message.
Screenshots and image attachments work on Grok
Attaching a screenshot to a Grok turn no longer fails immediately with a generic
provider turn start error. Grok’s ACP
initializestill reportspromptCapabilities.image: false, but the agent accepts and vision-processesimage content blocks. The Grok flavor now opts in via
supportsImagePromptssoattachments are sent as ACP image blocks instead of being rejected before
session/prompt.Problem and Fix
user_messageturn item landssession/cancelfor Grok; terminalize only on real provider signalssession/promptstayed open without a reliable complete pathturn_completed(and legacy prompt-complete) for the same prompt idtask_already_runningspawn_subagentshowed completed in ~1ms on the background ACK; child work never boundmonitorstayed spinning forever or never streamed outputpersistent: truemonitors from deferred-finalize holds; they continue post-settleimage: falserestartRuntimeOnEveryInterrupt), discarding native session context and any running subagents even when the user was only redirectingsession/cancel+ same-session re-prompt; a steer on a settled held turn skips the cancel entirely and preserves the runtimesession/canceldetaches and re-runs the interrupted command as a background task; nothing tracked that work, so wake machinery could misread pending state_x.ai/task_backgrounded/task_completedinto background-task mutations; tombstone tasks the model kills viakill_command_or_subagent(notask_completedis emitted for kills)activeSessionIdstill points at the stopped session until the next turn respawns)stoppedRunQuarantineis set, and ignore child-session mutations that must not gate root wake stateDefensive Fixes
provider-turn.restartcould swallow a failed replacement startcontinuationRequestedafter a failed or skipped dispatch blocked further offers and pinned idlestartTurnactiveTurnwas null and in-turn-handled task ids remained<monitor-event>chunks dropped later progress / end notices<monitor>block, and end notice in the chunktask_idsTaskOutput hydrated only the first idExit Code: -1read as successrunningcleared the hydration hold earlyCommits
fix(orchestrator): Grok v2 settlement, spawn_subagent, monitor, and imagesfix(orchestrator): Harden Grok interrupt cleanupfix(orchestrator): Preserve Grok runtime on settled steering interruptfix(orchestrator): Soften non-Stop Grok interrupts and track backgrounded tasksrestartRuntimeOnEveryInterrupt; mid-prompt steer / restart_active becomes soft cancel + same-session re-prompt;_x.aibackground-task lifecycle tracking + kill tombstonesfix(orchestrator): Quarantine post-Stop background task mutationsThe shared C0 plumbing this builds on (
feat(orchestrator): Add shared provider continuation and background item plumbing) is already int3code/codex-turn-mapping, so this PR no longer carries it and integrationstacks pick up C0 exactly once from the base.
Root-matched terminal signal (live evidence)
Current Grok CLI emits completion as:
{ "method": "_x.ai/session/update", "params": { "sessionId": "<root session>", "update": { "sessionUpdate": "turn_completed", "prompt_id": "t3-xai-prompt-N", "stop_reason": "end_turn" } } }Matching rules:
sessionIdequals the active ACP sessionprompt_idequals the pending T3-injected id (t3-xai-prompt-N)task-completed-*prompt ids (background tools)_x.ai/session/prompt_completefor older buildsDo not treat idle silence or subagent completions as root terminalization.
Server mechanics
settleRootTurnWhenIdle: false); speculative idle complete stays disabledsession/promptRPC result or root-matchedturn_completed/ legacyprompt_completestartTurnspawns a fresh Grok process. Non-Stop interrupts no longer restart the runtime (restartRuntimeOnEveryInterruptdropped from the Grok flavor)running+ binds child; TaskOutput / get_command hydrates; durablesubagent.resultwhen availableactiveTurnis null; offer only on completion-like evidence (terminal tool status after normalization, or agent text with no tracked background task still running); monitor-ended reminders andTaskOutputcompletion frames are the genuine end signals that clear tracked task ids (ended ids are tombstoned against straggler re-adds); attach mode skipssession/promptand drains the buffersupportsImagePrompts: trueoverrides ACPpromptCapabilities.image:falseso screenshots are sent as image content blocks_x.ai/task_backgrounded/task_completedbecome background-task mutations;kill_command_or_subagenttombstones the killed task; mutations are dropped while the Stop quarantine is active or when they belong to a child sessionDesign decisions (interrupt complexity; safe to simplify later)
Two choices below are deliberate and reviewable. If maintainers want less
surface area, each has a known downgrade path and a concrete fidelity trade-off.
1. Linux cgroup join wrapper (production spawn path)
Current landing: before
execof the real Grok/ACP binary on Linux, spawnis rewritten through
process.execPath+ a small inline-escript(
wrapCommandForLinuxCgroup). The shim writes its PID into the leasedcgroup.procs, verifies/proc/self/cgroup, strips wrapper env, thenprocess.execves into the real command so membership sticks to the same PID.This is only used when cgroup v2 containment is available; otherwise spawn is
unwrapped and interrupt falls back to the owned-process ledger (reduced
guarantee). Mac/Windows never use this path (process group /
taskkill).Why not PATH
node: the wrapper uses the running Electron/Node binary(
process.execPath+ELECTRON_RUN_AS_NODEunder desktop), so packagedAppImages do not require a separate system Node install.
Preferred cleanup if inline
-eis too opaque: replace the string with asmall checked-in
acp-cgroup-enter.mjsnext to the server (sameexecPathinvocation, no temp files, no C toolchain for production). Same semantics;
easier to review and diff.
Heavier alternative: checked-in C helper binary shipped in the desktop
artifact. Clearer argv contract and no
execvedependency on the ElectronNode build, but adds a real packaging/build matrix. Not chosen for this PR.
Complexity you can drop: if cgroup pre-exec join is too much, ship only the
ledger / process-group fallback on Linux too. Trade-off: double-fork /
reparented descendants can escape kill (the portable ledger comment already
calls this out). Headless interrupt packs would get flakier on adversarial
shell shapes.
2. Pthread spawn test helper (
.c, test-only)Current landing: keep
apps/server/scripts/acp-thread-spawn-helper.c. OneLinux live test compiles it with
cc -pthreadto fork a child from anon-leader pthread, then asserts
childrenOf(parent)still finds thatchild. That exercises the production scan of every
/proc/<pid>/task/<tid>/childrenentry (main-thread-only scans miss this).Local: if
ccis missing, the test soft-skips (suite stays green).CI: Blacksmith Ubuntu runners mirror GitHub’s Ubuntu image and often
already have
gcc, but that is not a repo contract. The Test job thereforeinstalls
build-essentialsoccis present and the fixture runs insteadof silently no-oping:
Simpler alternative (drop complexity): replace the helper with a plain
Node/
.mjsspawn("sleep", …)and rename the test to “finds a spawned child.”No compiler, always runs. Fidelity loss: a regression that only walks the
main task’s children file (not every thread) can still pass a normal spawn
fixture and only fail the pthread case. Other double-fork / cgroup-kill tests
remain; only this multi-thread discovery edge case goes soft.
Not a good substitute without rethinking the assertion: pure JS cannot
faithfully create “non-leader pthread then fork” without native code.
Validation
vp check: format clean on changed files; monorepo lint warnings only (knownVite+ stdout panic can flake the full wrapper after lint)
vp run typecheck: cleanvp test apps/web/src/components/ChatView.logic.test.ts: passvp test apps/server/src/orchestration-v2/Adapters/GrokAdapterV2.test.ts:pass (includes image capability override)
vp test apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.test.ts: pass(wake-evidence filters, continuation offer gating, the monitor start ACK
normalization case, steer-during-hold interrupt finalize, subagent lineage
carry-over across a steering restart, injected-report hold, interrupt
cleanup fixtures, soft mid-prompt cancel + same-session re-prompt, direct
Stop dropping late background task mutations, and the production Grok flag
set still hard-killing and respawning on user Stop)
vp test apps/server/src/provider/acp/AcpSessionRuntime.processTree.test.ts:pass (cgroup / process-tree interrupt evidence; pthread fixture needs
cc,provided in CI via
build-essentialon the Test job; soft-skips locally ifccis absent)vp test apps/server/src/orchestration-v2/ProviderSessionManager.test.ts:pass (idle pin with C0)
vp test apps/server/src/provider/acp/XAiAcpExtension.test.ts: pass (asyncspawn, TaskOutput, Monitor variant cases)
live-scenarios, private serve against thisbranch / restacked AppImage):
grok-interrupt-fast: direct Stop +restart_active(the restart_activepack rewritten for soft semantics: process retained, one initialize, no
session/load, cancel + re-prompt, codeword carry-forward provingsame-session context; detach handling recorded as evidence, with a
branch-conditional
-detachvariant)grok-interrupt-soak: held subagent, mid-prompt steer during a subagenthold (new pack sampling the cancel-kills-subagent edge), duplicate Stop,
leader exit, queued follow-up (scheduler gate); re-run green on the C4 tip
Grok thread titled from scenario 1 "Live-test direct Stop…", 2026-07-17 on
orchestrator-v2 AppImage @ current snapshot; projection checked in userdata
DB for thread
6c9994b7-bb04-4fb7-a9a8-e95688a6b7d7):INTERRUPTED_OK: passRESTART_ACTIVE_OK: passQUEUE_CONTRAST_OK: pass(one late-queue practice attempt finished the sleep without a queue; retry
with queue while Working passed)
DUPLICATE_STOP_OK: passwhile queued; headless
grok-interrupt-queued-followupremains theautomated contract)
hides Stop while the card is open)
HOLD_FOLLOW_OK: pass (subagentinterrupted, no
HOLD_SUB_DONE/ forbidden shell marker)scenario numbering; 4–5 were blocked on desktop Stop UI as above.
16092557-…, 2026-07-17, scenarios 1 and 7 only):INTERRUPTED_OK: pass (sleep interrupted; noSHOULD_NOT_FINISH_CMDoutput)HOLD_FOLLOW_OK: pass (subagentinterrupted; no
HOLD_SUB_DONE/ forbidden shell marker)trees/orchestrator-v2):steer-during-hold, screenshot attach as previously reported on this PR
Out of Scope
Client presentation and desktop chrome that are not part of this Grok
settlement / interrupt PR. Called out so maintainers can track separate UI work
if desired; projection and run status still settle correctly.
+N previous tool calls(interrupted tools, Ask, etc.). Mobile keeps neutral tool rows in the work group; desktop filters neutrals out. Presentation difference only.run_interrupt_request/run_interrupt_resultfor Stops from desktop (verified in userdata). On mobile, those Interrupted rows are not shown for desktop-initiated Stops—not merely collapsed under+N previous tool calls(expanding the fold still does not surface them the way a live mobile Stop does). Live mobile Stops can show anX Interruptedwork-log row and aYou stopped after …fold. Incomplete multi-client / mobile lifecycle parity; not a missing server interrupt.--- Run interrupted ---dividerrun_interrupt_resultas a full-width system divider and leaves interrupted turns unfolded. Mobile never uses that divider chrome.workEntryPreviewprefersdetailover file_change filenamespromptId/ low-risk Grok session edge casesinProgressuntil native outputKnown limitations
After a real terminal signal, the hung
session/promptfiber may still logInterruptwhen cancelled locally; runs complete with full content (lognoise).
Builds that emit neither prompt RPC completion nor
turn_completed/prompt_completecan still under-settle (no silent idle cancel fallback).After Stop, a tool in the old shell may finish briefly before process kill
(
interruptPromptOnCancel: false); recovery still succeeds on the freshchild.
Subagent child-session tool calls (tools inside the child) are not fully
projected as nested tool items; assistant text + final hydration / result card
are covered.
Subagent wall time is provider-variable. A trivial “sleep ~5s then print a
token” general-purpose spawn is often ~10–20s end-to-end, but the child agent
can occasionally take much longer (minute-scale outliers observed). A long
Working state usually reflects child runtime, not the ~2s deferred-finalize
quiet window.
The root model may poll
get_command_or_subagent_outputwith longtimeouts, so the root run’s Working state can cover the entire child wait even
when the subagent card is the durable success signal.
Non-persistent background work still holds Working. Deferred finalize
keeps the root run open while a spawned subagent or a non-persistent monitor
tool row is still running. Persistent monitors (
persistent: true) nolonger hold that gate; they settle the root and continue via post-settle
continuation. Claude-style “go idle then reopen when background finishes” for
non-persistent work is still future work. Content that arrives during a hold
(CLI-injected monitor report) is delivered by the injected-report hold, so
this is about run status staying Working, not about lost output.
Continuation dispatch is by
threadId(the request also carriesdriverand
providerThreadIddetail for future scoping).Buffered per-event commentary drains into the continuation run as several
small assistant messages (one per burst). That is faithful to what the
provider emitted; coalescing the display would be UI-side follow-up work.
Soft-interrupt detach tracking is best-effort. Grok's
session/canceldetaches and re-runs the interrupted foreground command to full duration as
a background task; no client-side ACP method kills it (only the model can,
via
kill_command_or_subagent)._x.ai/task_backgroundedcan arrive late(~16s observed) or not at all (the model's poll-and-report shape emits
none), so lifecycle tracking begins only when a notification arrives.
Residual frames in that gap are accepted best-effort noise per the product
call above; live packs record detach handling as evidence rather than
pinning it.
Fast steers can trigger a model-side no-op flood. Steering within a
couple of seconds of a reply while background work is still held (send a
short message right after the settle reply with a subagent still running)
can push Grok into emitting no-op
run_terminal_command(true) callsabout once per second until interrupted; the same loop shows up without
steering after monitor-event injections (154 no-ops observed in one live
thread, 207 in another). The steer text and background completions still
land once the loop breaks or the run is stopped. This is xAI model
behavior, not orchestrator state; recovery is Stop.
Desktop UI: no Stop while a message is queued or an approval/question card
is open. See Out of Scope (mobile can still run those paths; headless
covers queue-then-Stop).
Mobile vs desktop interrupt presentation. Trailing work-log rows after a
mobile Stop, missing
--- Run interrupted ---dividers, and no Interruptedlifecycle chrome on mobile for desktop-initiated Stops (not just collapsed)
are client parity gaps only; server still projects interrupt items. See
Out of Scope.
Grok exact-reply EOS token leakage (model, not T3). On short
“reply with exactly TOKEN, no tools” recovery prompts, Grok sometimes
returns
TOKENplus a literal end-of-sequence marker. Observed 2026-07-16on desktop AppImage after subagent-hold Stop:
grok-4.5.general-purposesubagent that runssleep 30 && echo HOLD_SHOULD_NOT_MATTERand should returnHOLD_SUB_DONE; root waits for the subagent.Reply with exactly HOLD_FOLLOW_OK. No tools.HOLD_FOLLOW_OK<|eos|>(ASCII<|eos|>appended). Projection stored that string verbatim; subagent was
correctly
interruptedwith no forbidden shell marker.Other exact-reply markers in the same session (
INTERRUPTED_OK,RESTART_ACTIVE_OK,DUPLICATE_STOP_OK) were clean. Treat as xAI/Grokmodel leakage suitable for upstream Grok team follow-up if desired; not a
T3 interrupt/projection bug.
Stuck Working after hard interrupt on a poisoned session is reduced, not
fully gone. This PR hardens the common Stop races (idempotent interrupt,
finalize-before-kill, missing-session restart, sticky continuation flag,
process-group reap). Residual risk remains if a held turn keeps a
background/lineage entry that can no longer receive a terminal signal after
restart (no overall force-finalize cap on that path). Fresh threads and the
headless steer pack are the reliable recovery; candidate follow-ups are a
bounded hold cap and reconciling carried subagent / background-tool state on
interrupt.
Note
Harden Grok v2 prompt settlement, background task lifecycle, process containment, and steer visibility
makeXAiPromptCompletionRuntimenow settles from both_x.ai/session/updateand_x.ai/session/prompt_completenotifications; prompt fibers are raced against deferred completion and returncancelledon interrupt-only causes.makeGrokAcpAdapterFlavorintroduces soft/hard interrupt strategies (interruptPromptOnCancel=false,restartRuntimeAfterInterrupt=true,terminateRuntimeProcessGroupOnInterrupt=true), deferred finalize for background work, post-settle continuation, andsupportsImagePrompts=true.AcpSessionRuntimecan now own detached process groups and, on Linux, contain ACP descendants in a cgroup v2 lease with grace-period kill;grokAcpRuntimeProcessOwnershipselects the appropriate strategy per host platform.extractXAiAcpSubagentUpdateandapplyBackgroundTaskMutationtrack running/completed/failed background tasks and suppress duplicate tool projections forget_command_or_subagent_outputcalls.deriveCommittedServerUserMessageIdsnow derives committed user message IDs from visible turn items instead of projection messages, preventing steer rows from being evicted when message updates arrive one event before matching turn-item updates.isNonRetryableProviderTurnControlFailuremarks terminalprovider-turn.interruptfailures as succeeded in the outbox, stopping retry loops when the session is already gone.ProviderSessionManagerV2idle-release logic now checks runtime identity before continuing a pin, preventing stale pins from applying to replacement sessions.onTermination; intentionalserverProtocol.enddoes not signal a transport failure.Macroscope summarized 6add8c0.
Note
Low Risk
Changes are confined to CI and test/mock ACP tooling; production orchestrator paths are not modified in this diff.
Overview
CI now installs
build-essentialon the Test job so ACP process-tree live tests can compile the pthread fixture withccinstead of soft-skipping when no compiler is present.A new
acp-thread-spawn-helper.cforks a long-running child from a non-leader pthread and writes thread/child PIDs to a path, soAcpSessionRuntime.processTreetests can assert discovery across/proc/.../task/*/children, not only the main thread.acp-mock-agent.tsgains manyT3_ACP_*flags to simulate Grok-style behavior for integration tests: long-running terminal tools (with optional detached sessions and PID files), cancel →task_backgrounded/task_completed, residual session updates and late permission/elicitation callbacks, form/URL elicitation, empty successful Bash (including duplicate updates), post-settle monitor + TaskOutput hydration (with optional injected-report trigger), in-turn monitor output followed by a late duplicate TaskOutput, and optional exit on cancel or after launching a running command.Reviewed by Cursor Bugbot for commit 6add8c0. Bugbot is set up for automated code reviews on this repo. Configure here.