From 3919a0b6be08fa5eef8599da2cc7b56445e46675 Mon Sep 17 00:00:00 2001 From: Peter Schuller Date: Wed, 18 Nov 2026 15:54:15 -0800 Subject: [PATCH] fix: let codex-wake-on-exit hand a flow to a new thread A watcher run can only wake the thread that armed it, because its callback is queued to that thread ID. The skillette told agents to retain "an owned live watcher" without saying what owned meant. When a new thread picked a flow up from its durable log, the helper refused the old run as foreign and gave it no way forward, so agents checked the PIDs with ps and kept the old thread's watcher. Its callback then went to a thread nothing had loaded, and the flow stalled silently. A baseline eval reproduced this in both trials where the old watcher was still live. Ownership is now the manifest's thread ID and nothing else, status reports on any thread's run, and supersede is the one cross-thread action: it cancels the old watcher and marks the run as taken. The old owner's ack and the successor's supersede race for the same exclusive claim file, because a queued callback does reach an unloaded exec thread once that thread is resumed, and without the claim both threads would act on the same run. The first candidate showed a second problem: a manually resumed old owner took the flow back from its successor in 2 of 4 trials. A thread whose run was superseded now asks the user before doing any flow work. Two threads the user runs on the same flow at the same moment are still not serialized; the helper can only arbitrate claims on a run, not work a live thread has already started, so the skillette states that as a limit. --- agent-skills/skillette/SPEC.md | 20 +- .../skillette/codex-wake-on-exit/EVALS.md | 36 ++ .../codex-wake-on-exit/EVAL_RESULTS.md | 39 ++ .../skillette/codex-wake-on-exit/SKILLETTE.md | 101 ++++- .../skillette/codex-wake-on-exit/wake-on-exit | 370 +++++++++++++++++- 5 files changed, 528 insertions(+), 38 deletions(-) diff --git a/agent-skills/skillette/SPEC.md b/agent-skills/skillette/SPEC.md index 74594ce..6a5e080 100644 --- a/agent-skills/skillette/SPEC.md +++ b/agent-skills/skillette/SPEC.md @@ -145,10 +145,22 @@ startup before returning. Linux and macOS ordinary Codex sessions are supported; cannot receive direct queued input and are not completion-wake targets. Each run has a fresh private directory, immutable identity, terminal result, stdout/stderr files, and queue outcome. Nonzero exits still trigger a callback. Callbacks are checked against the registered active run and consumed once; output remains data rather than instructions. -Cancellation is handled by the live runner for its own process group, never by trusting a stale PID. Already queued -stale or cancelled callbacks cannot resume a stopped flow. Startup failures, missing capabilities, delivery failures, -and unresponsive runners are reported/reconciled without falling back to model-driven polling. There is no automatic -enqueue retry or promise of restart recovery. +Before doing a resumed flow's work, the agent first checks whether a run of its own for the flow was superseded since +any takeback the log records. If so, the flow was taken over and stays with the successor unless the user confirms that +this thread should take it back; this takes precedence over the current run's owner and phase. Only then does it check +the run its log records as current. A run belongs to the thread that armed it, and its callback can wake only that +thread; a thread that resumes the flow from its durable log does not inherit the run. `status` reports the owner for any +thread's run. A thread taking over a flow retires the previous thread's run with `supersede`, records that in the +durable log, and arms its own run; it never keeps a foreign run as its watcher. `supersede` takes the run ID the log +records, and the owner's acknowledgment and a successor's supersession claim the run exclusively: whichever comes first +wins, so a superseded run's callback is refused by its owner's `ack`, and a run whose callback the owner already +consumed cannot be taken over. A later resumed old thread therefore cannot drive the flow a second time from that run. +The helper cannot tell whether the old thread is still loaded, and supersession does not stop an old thread already +working without a watcher; the agent asks the user when the old thread may still be running the flow. Cancellation is +handled by the live runner for its own process group, never by trusting a stale PID. Already queued stale or cancelled +callbacks cannot resume a stopped flow. Startup failures, missing capabilities, delivery failures, and unresponsive +runners are reported/reconciled without falling back to model-driven polling. There is no automatic enqueue retry or +promise of restart recovery. The returned phase governs startup: `watching`, `delivering`, and `queued` permit ending the turn for a callback; `queue-failed` and `runner-failed` require reporting and stopping idle monitoring; `cancelled` keeps the flow stopped. diff --git a/agent-skills/skillette/codex-wake-on-exit/EVALS.md b/agent-skills/skillette/codex-wake-on-exit/EVALS.md index 84b90f2..73a1ee4 100644 --- a/agent-skills/skillette/codex-wake-on-exit/EVALS.md +++ b/agent-skills/skillette/codex-wake-on-exit/EVALS.md @@ -86,3 +86,39 @@ unsupported capability starts nothing, while failed delivery preserves diagnosti Review discussion/edit requests and other-harness requests separately: they must not arm a watcher. Restart recovery is outside the current contract and must not be inferred from successful uninterrupted sessions. + +## Taking over another thread's run + +This is a regression for an observed failure. A new thread resumed a flow from its durable log, found a live run armed +by the previous thread, and kept it as its watcher after the helper refused it as foreign. That run's callback went to +the old thread, which nothing had loaded, and the flow slept through its condition. + +Set up the fixture with two real Codex threads. Thread A arms a long-lived watcher through the helper, records it in a +durable log as the flow's active run, and its session ends. Thread B is a fresh session given the same log and asked to +resume the flow, without being told about ownership or supersession. Vary whether A's watcher is still `watching` or has +already queued its callback to the unloaded thread A by the time B starts. Expected: B finds that the run is not its own +(through `status` or the helper's refusal), runs `supersede` on A's run with the logged run ID, records the supersession +in the durable log, re-establishes the baseline (draining anything already there), and arms its own run for its own +thread. Failure: B ends its turn on A's run, with or without first checking state files or PIDs, or arms a run beside +A's without superseding it. Verify from the run directories rather than B's report: A's `ack.json` claim names B's +thread in `supersededBy`, a still-watching A run ends `cancelled` without a queue submission, and exactly one new run +names B's thread. + +A one-shot `codex exec` client is acceptable for these decision cases, because what is checked is the takeover rather +than callback receipt. It cannot consume B's own callback, so do not count anything after B's turn ends as receipt +evidence, and treat a B that declines to arm because it recognizes a one-shot client as a harness artifact rather than +an instruction failure. + +For the double-wake side, resume thread A afterwards with the callback text its run queued or would have queued. A +passes if it ends its turn without resuming the flow or arming a watcher, whether it learns of the takeover from `ack` +returning `ignored: "superseded"` or from the updated durable log first. If it runs `ack`, the claim must remain B's. +Because this hands A the callback text directly, it checks A's handling only; it is not a receipt test. The helper tests +establish the exact `ack` response. + +Two related cases are worth covering when the takeover text changes. A second resumer C that reads the log before B +updates it finds A's run already superseded by B: `supersede` refuses, and C must report and ask instead of arming. And +thread A, resumed by the user rather than by a callback (for example with "drain plans and monitor"), must notice that +its own run was superseded before doing any flow work, tell the user which thread took the flow, and end its turn +without draining, superseding, or arming. Run this both before and after B records its own run: afterwards the log's +current run is B's, and A taking it over without asking is a failure. Give C a copy of the pre-takeover log in a +directory of its own, with the log's self-references rewritten to the copy, so that it cannot find the live log. diff --git a/agent-skills/skillette/codex-wake-on-exit/EVAL_RESULTS.md b/agent-skills/skillette/codex-wake-on-exit/EVAL_RESULTS.md index 4e24f7a..64aa08d 100644 --- a/agent-skills/skillette/codex-wake-on-exit/EVAL_RESULTS.md +++ b/agent-skills/skillette/codex-wake-on-exit/EVAL_RESULTS.md @@ -81,3 +81,42 @@ Linux runtime. It does not establish automatic installation discovery, a general restart recovery, or operation in other harnesses. File-condition cases use actual filesystem notifications with a timer creating the condition marker. Raw session histories and host-specific identifiers are retained in the private evaluation artifacts rather than published here. The procedure for future runs is in [EVALS.md](EVALS.md). + +## Takeover regression, 2026-11-17 + +NOTE: This covers only the "Taking over another thread's run" cases in [EVALS.md](EVALS.md). The twenty-session receipt +suite above was not rerun for this change. `ack` now claims a run through an atomically linked `ack.json` instead of +creating it directly; the helper tests cover that path, but no real connected session has received a callback through +it. + +The cases ran as real `codex exec` threads on gpt-6.2-sol, high reasoning effort, with Codex CLI 0.169.0, Deno 2.9.8, on +Linux x86_64. Each trial had thread A start a plan-drain flow and arm a watcher, then end. A fresh thread B was told +only to resume the flow from its durable log. A thread C got a copy of the pre-takeover log in a directory of its own. +Thread A was then resumed, either with its run's callback text or by the user asking it to carry on with a new plan +waiting. Half the trials let A's watcher fire into the unloaded thread A before B started. Verdicts come from the run +directories and event logs, not from the agents' reports. + +On the previous text and helper (three trials), B kept A's live run as its watcher in both trials where it was still +watching. It confirmed the PIDs with `ps` and ended its turn, which is the incident's failure exactly. Where A's run had +already queued its callback, B armed its own run but nothing stopped the resumed A from acting on that callback. + +The first candidate passed B in all eight trials but failed two of four manual resumes. A found B's newly logged run, +superseded it, and drained the waiting plan itself. That led to the rule that a thread whose own run was superseded asks +the user before taking the flow back. One C failure in that suite was a fixture flaw: C could still read the live log. + +An intermediate candidate with that rule, before the last review-driven wording changes, also passed all eight. On the +frozen final candidate, **all eight trials passed for B, C, and A**. B ran `status`, superseded A's run with the logged +run ID, logged the supersession, drained any waiting plan, and armed exactly one run for its own thread. No B checked +PIDs. C asked the user instead of taking over a run that B had already claimed, and armed nothing. Resumed A armed +nothing and drained nothing. It named the thread that took the flow over and, when resumed by hand, asked whether to +take it back. A's `ack.json` named B as `supersededBy` at the end of every trial. + +Frozen helper SHA-256: 349af167fbcbadcc301464175a68d1b3f190d4caeea48d1cc4bd11b8c2130fe8. Frozen selected-instruction +SHA-256: c88c0185d94c6248a0842fb2125c961cbcea17a1946827146fb1c1969dccec34. + +One runtime fact turned up along the way. A message queued to an unloaded `codex exec` thread was delivered when that +thread was later resumed, right after the resume prompt. A stale callback can therefore reach the old thread long after +the flow has moved on, which is why the owner's `ack` refusing a superseded run matters. + +A one-shot client cannot receive B's own callback, so these trials establish the takeover decision, not receipt. Eight +trials on one model support a claim about these fixtures, not a failure rate. diff --git a/agent-skills/skillette/codex-wake-on-exit/SKILLETTE.md b/agent-skills/skillette/codex-wake-on-exit/SKILLETTE.md index bfcd74a..98ea62a 100644 --- a/agent-skills/skillette/codex-wake-on-exit/SKILLETTE.md +++ b/agent-skills/skillette/codex-wake-on-exit/SKILLETTE.md @@ -39,12 +39,18 @@ deno run -A /codex-wake-on-exit/wake-on-exit start /codex-wake-on-exit/wake-on-exit supersede + ``` + + Success means the retirement is committed, whatever the old run's phase: the old thread's `ack` will refuse that + run's callback, and the old watcher is cancelled through its own runner if it is still live. `cleanup: "pending"` + only means cancellation had not finished (an `error` field says why if it failed); continue anyway. +2. Record in the durable log that the old run is superseded by this thread and its callback is to be ignored. +3. Do not treat the old run's result or output as your completion. Re-establish the flow's baseline under its own rules, + since the condition may already hold, then do the flow's work or arm a fresh run for this thread as above and end the + turn. A `queue-failed`, `runner-failed`, or `cancelled` phase on the old run does not hold you back: supersession is + the reconciliation those phase rules ask for. A flow the log records as stopped by the user stays stopped unless the + user asks to resume it. + +`supersede` refuses when the owner has already acknowledged the callback and when a different thread already superseded +the run. Either way another thread may be running the flow right now: report it with the old run's directory and ask the +user, and do not keep the old run or arm a new one beside it. If the owner had acknowledged and the user confirms it is +not running the flow, the run needs no retirement: record that in the log and continue from step 3. If another successor +took the run and the user confirms that successor is not running the flow either, re-read the log and start again from +"Resume a flow" with that successor's run; until then, end the turn. Report any other `supersede` error and ask as well. +Supersession governs only the run: it cannot stop an old thread that is already working without a watcher. + ## Handle the callback The helper's callback names the run ID and directory. Match them against your registered active run; a callback alone @@ -76,9 +142,12 @@ deno run -A /codex-wake-on-exit/wake-on-exit ack /codex-wake-on-exit/wake-on-exit stop ``` -The live runner cancels its owned watcher process group and records cancellation after cleanup, without queuing another -wake. A supervisor stays alive through TERM escalation to keep the group ID owned even when the watcher exits first; the -helper does not kill a PID from a possibly stale file. A callback already queued can still arrive; acknowledge it and -ignore it once the flow or that run has been stopped. +Record the stop in the durable log. The live runner cancels its owned watcher process group and records cancellation +after cleanup, without queuing another wake. A supervisor stays alive through TERM escalation to keep the group ID owned +even when the watcher exits first; the helper does not kill a PID from a possibly stale file. A callback already queued +can still arrive; acknowledge it and ignore it once the flow or that run has been stopped. `status ` is for one-time reconciliation after resume, an unexpected startup result, or a -delivery problem, never an idle polling loop. The run's private `state.json`, `stdout.log`, and `stderr.log` are +delivery problem, never an idle polling loop. It reads any thread's run and reports the owner; only `status` and +`supersede` accept a run owned by another thread. The run's private `state.json`, `stdout.log`, and `stderr.log` are retained for diagnosis. A failed enqueue is recorded as `queue-failed`; there is no claim of successful delivery and no retry loop. An unreachable runner, failed delivery, or lost session requires reporting/reconciliation, not blindly -starting another watcher. Client/daemon/host restart recovery is not guaranteed by these instructions. +starting another watcher. A run owned by another thread is not such a case; take it over as described above. +Client/daemon/host restart recovery is not guaranteed by these instructions. A concrete startup or supervision failure is recorded as `runner-failed` after owned-process cleanup. This phase does not claim callback delivery. A startup-handshake timeout alone still requires reconciliation: if the runner itself diff --git a/agent-skills/skillette/codex-wake-on-exit/wake-on-exit b/agent-skills/skillette/codex-wake-on-exit/wake-on-exit index 3f171b3..bcf6a13 100755 --- a/agent-skills/skillette/codex-wake-on-exit/wake-on-exit +++ b/agent-skills/skillette/codex-wake-on-exit/wake-on-exit @@ -4,8 +4,18 @@ * * `start NEW-ABSOLUTE-RUN-DIR -- PROGRAM ARG...` captures CODEX_THREAD_ID and cwd, * confirms local queue support, and returns only after the detached runner starts. - * `status RUN-DIR`, `stop RUN-DIR`, and `ack RUN-DIR RUN-ID` never start another job. - * Output stays in private files; queued messages contain identifiers, not command output. + * `status RUN-DIR`, `stop RUN-DIR`, `supersede RUN-DIR RUN-ID`, and `ack RUN-DIR RUN-ID` never + * start another job. Output stays in private files; queued messages contain identifiers, + * not command output. + * + * A run belongs to the thread that armed it, for its whole life. Its callback can only + * wake that thread, even after the thread stops being used and another thread resumes + * the same flow from its durable log. `status` therefore reports the owner rather than + * refusing a foreign run, and `supersede` is the one cross-thread action: a thread taking + * over the flow uses it to retire the old thread's run. A callback the old run already + * queued can still wake its owner, but the owner's `ack` refuses it, so the owner cannot + * start a second copy of the flow from it. Supersession governs only the run: it cannot + * stop an old thread that is already working without a watcher. */ import { spawn } from "node:child_process"; import { createWriteStream } from "node:fs"; @@ -54,16 +64,115 @@ function exists(path: string) { } } -/** Keep lifecycle mutations confined to the exact Codex thread that armed the job. */ -function owned(dir: string, thread: string): { manifest: Manifest; state: State } { +/** + * Read a run's identity without deciding who may act on it. + * + * Only the manifest and state agreeing on one run ID and one thread make the record + * usable; a mismatch means the directory is not a run this helper wrote. + */ +function inspect(dir: string): { manifest: Manifest; state: State } { const manifest: Manifest = JSON.parse(Deno.readTextFileSync(join(dir, "manifest.json"))); const state: State = JSON.parse(Deno.readTextFileSync(join(dir, "state.json"))); - if (!uuid.test(thread) || manifest.thread !== thread || state.id !== manifest.id || state.thread !== thread) { - throw new Error("This run does not belong to the current Codex thread"); + if (state.id !== manifest.id || state.thread !== manifest.thread) { + throw new Error("This run's manifest and state disagree; it is not a usable wake-on-exit run"); } return { manifest, state }; } +/** + * Keep lifecycle mutations confined to the exact Codex thread that armed the job. + * + * "Owned" means the manifest names the current CODEX_THREAD_ID. A durable log that + * records the run for a flow this thread has resumed does not make it owned: the + * callback still targets the arming thread. `supersede` is the only exception. + */ +function owned(dir: string, thread: string): { manifest: Manifest; state: State } { + const record = inspect(dir); + if (!uuid.test(thread) || record.manifest.thread !== thread) { + throw new Error("This run does not belong to the current Codex thread"); + } + return record; +} + +/** + * The single claim on a run's callback, stored in `ack.json`. + * + * Exactly one of two things can claim a run: its owner acknowledging the callback, or a + * successor thread superseding the run (`supersededBy` set). Both go through `claim`, so + * whichever comes first wins and the other is refused; an owner and a successor can never + * both act on the same run. The file name predates supersession and is kept so that runs + * acknowledged by an older copy of this helper still count as claimed. + */ +type Claim = { id: string; at: string; supersededBy?: string }; + +/** + * The run's claim, if any. + * + * This helper publishes claims whole, but an older copy created `ack.json` and then wrote + * it, so a claim can be empty or partial (mid-write, or after an interrupted write). Only + * older copies write that way, and they only ever wrote the owner's acknowledgment, so an + * unreadable claim is read as exactly that. + */ +function readClaim(dir: string): Claim | undefined { + const path = join(dir, "ack.json"); + if (!exists(path)) return undefined; + try { + return JSON.parse(Deno.readTextFileSync(path)); + } catch (error) { + if (error instanceof SyntaxError) return { id: "", at: "" }; + throw error; + } +} + +/** + * Publish `claim` as the run's claim unless one exists, and return the earlier claim if it does. + * + * The record is written to a private temporary file and hard-linked into place: the link + * fails if a claim already exists, and a reader can never see a partly written one. This + * needs a filesystem with hard links, which the local run directories the skillette asks + * for have on Linux and macOS. + */ +function claim(dir: string, record: Claim): Claim | undefined { + const temporary = join(dir, `.ack-${crypto.randomUUID()}`); + try { + Deno.writeTextFileSync(temporary, JSON.stringify(record) + "\n", { mode: 0o600, createNew: true }); + Deno.linkSync(temporary, join(dir, "ack.json")); + return undefined; + } catch (error) { + if (error instanceof Deno.errors.AlreadyExists) return readClaim(dir); + throw error; + } finally { + // Best effort: once the link exists the claim is committed, and a leftover private + // temporary file must not turn that into an error the caller would act on. + try { + Deno.removeSync(temporary); + } catch { + // Ignored deliberately; see above. + } + } +} + +/** + * Report a run's state and who owns it, for any thread. + * + * This is read-only. It exists so that a thread finding another thread's run in a + * durable log learns that the run is not its watcher from the helper, rather than + * from reading state files and process IDs that cannot say which thread a callback + * will wake. `acknowledged` means the owner consumed the callback; a superseded run + * reports `acknowledged: false` with `superseded` naming the successor instead. + */ +export function status(dir: string, thread: string) { + const { manifest, state } = inspect(dir); + const existing = readClaim(dir); + return { + ...state, + acknowledged: existing !== undefined && existing.supersededBy === undefined, + currentThread: thread, + ownedByCurrentThread: manifest.thread === thread, + superseded: existing?.supersededBy ? { by: existing.supersededBy, at: existing.at } : null, + }; +} + /** Bound startup and cancellation handshakes inside the helper, without waking the model. */ async function until(read: () => T | undefined, milliseconds = 15000): Promise { const deadline = Date.now() + milliseconds; @@ -118,21 +227,77 @@ export async function stop(dir: string, thread: string) { }); } -/** Consume a matching terminal callback once; cancelled and duplicate messages cannot restart a flow. */ +/** + * Retire another thread's run on behalf of the current thread, which is taking over its flow. + * + * `id` is the run ID the flow's durable log records, so a mistyped or stale directory is + * refused rather than retired. Success means the retirement is committed: the owner's + * `ack` will refuse this run's callback whenever it arrives, including one already queued + * or still being queued. Supersession is refused when the owner has already acknowledged + * the callback (the owner may be running the flow) or another successor claimed the run + * first; repeating it from the same successor is harmless. + * + * Cancellation then goes through the same `cancel.json` request as `stop`, which the live + * runner (including one started by an older copy of this helper) handles for its own + * process group; nothing here signals a PID. `cleanup: "pending"` means the runner had not + * finished within `settle` milliseconds. That does not undo the retirement: the old + * watcher can at most queue a callback its owner will refuse. This never transfers the + * run's result to the caller. + */ +export async function supersede(dir: string, id: string, thread: string, settle = 15000) { + if (!uuid.test(thread)) throw new Error("Current CODEX_THREAD_ID is unavailable; do not guess another session"); + const { manifest, state } = inspect(dir); + if (manifest.id !== id) throw new Error("Run ID does not match this run directory; check the durable log"); + if (manifest.thread === thread) throw new Error("This run belongs to the current Codex thread; use stop instead"); + const at = new Date().toISOString(); + const prior = claim(dir, { id, at, supersededBy: thread }); + if (prior && prior.supersededBy === undefined) { + throw new Error("The owning thread already acknowledged this run's callback and may be running the flow"); + } + if (prior && prior.supersededBy !== thread) { + throw new Error(`Thread ${prior.supersededBy} already superseded this run`); + } + const superseded = { by: thread, at: prior?.at ?? at }; + // From here on the retirement is committed. A failure while requesting or awaiting + // cancellation is reported as pending cleanup, not thrown, so that the successor + // still records the takeover and arms its own run. + try { + if (!exists(join(dir, "cancel.json"))) save(dir, "cancel.json", { id }); + // `delivering` is not terminal: the runner may still be queueing to the owner. + const deadline = Date.now() + settle; + for (;;) { + const current = inspect(dir).state; + const settled = !["starting", "watching", "delivering"].includes(current.phase); + if (settled || Date.now() >= deadline) { + return { ...current, superseded, cleanup: settled ? "complete" : "pending" }; + } + await new Promise((resolve) => setTimeout(resolve, 25)); + } + } catch (error) { + return { ...state, superseded, cleanup: "pending", error: String(error) }; + } +} + +/** + * Consume a matching terminal callback once; cancelled, superseded, and duplicate + * messages cannot restart a flow. + */ export function acknowledge(dir: string, id: string, thread: string) { const { state } = owned(dir, thread); if (state.id !== id) throw new Error("Callback run ID does not match the registered run"); - if (exists(join(dir, "cancel.json")) || state.phase === "cancelled") return { ignored: "cancelled" }; + // Supersession also requests cancellation, and the owner must learn that another thread + // now runs the flow, not merely that it stopped. `supersede` publishes its claim before + // `cancel.json`, so reading the claim after observing cancellation always sees it. + const cancelled = exists(join(dir, "cancel.json")) || state.phase === "cancelled"; + const existing = readClaim(dir); + if (existing?.supersededBy) return { ignored: "superseded", by: existing.supersededBy }; + if (cancelled) return { ignored: "cancelled" }; if (!state.result) throw new Error("Watcher has not produced a terminal result"); - try { - Deno.writeTextFileSync(join(dir, "ack.json"), JSON.stringify({ id, at: new Date().toISOString() }), { - mode: 0o600, - createNew: true, - }); - } catch (error) { - if (error instanceof Deno.errors.AlreadyExists) return { ignored: "duplicate" }; - throw error; - } + // The claim is what serializes this against a concurrent supersede; the check above + // only gives the common case a clear answer. + const prior = claim(dir, { id, at: new Date().toISOString() }); + if (prior?.supersededBy) return { ignored: "superseded", by: prior.supersededBy }; + if (prior) return { ignored: "duplicate" }; return { ...state, stdoutPath: join(dir, "stdout.log"), stderrPath: join(dir, "stderr.log") }; } @@ -319,7 +484,9 @@ function currentThread() { if (import.meta.main) { const [action, dir, ...rest] = Deno.args; if (!dir) { - throw new Error("Usage: start NEW-RUN-DIR -- PROGRAM ARG... | status RUN-DIR | stop RUN-DIR | ack RUN-DIR RUN-ID"); + throw new Error( + "Usage: start NEW-RUN-DIR -- PROGRAM ARG... | status RUN-DIR | stop RUN-DIR | supersede RUN-DIR RUN-ID | ack RUN-DIR RUN-ID", + ); } if (action === "supervise") { supervise(dir); @@ -330,9 +497,11 @@ if (import.meta.main) { const output = action === "start" && rest[0] === "--" ? await start(dir, rest.slice(1), { thread, cwd: Deno.cwd(), queue: ["codex"] }) : action === "status" - ? { ...owned(dir, thread).state, acknowledged: exists(join(dir, "ack.json")) } + ? status(dir, thread) : action === "stop" ? await stop(dir, thread) + : action === "supersede" && rest.length === 1 + ? await supersede(dir, rest[0], thread) : action === "ack" && rest.length === 1 ? acknowledge(dir, rest[0], thread) : (() => { @@ -391,6 +560,143 @@ Deno.test("stop after queueing latches cancellation without signaling a stale pr } }); +/** + * Regression for a thread that resumed a flow and kept the previous thread's live run as + * its watcher: the helper's only answer was a refusal, so agents read state files and + * PIDs instead and concluded the run was fine. Status must name the owner for any thread. + */ +Deno.test("status reports a foreign run's owner instead of refusing it", async () => { + const dir = Deno.makeTempDirSync(); + const owner = crypto.randomUUID(); + const other = crypto.randomUUID(); + const id = crypto.randomUUID(); + try { + save(dir, "manifest.json", { thread: owner, id }); + save(dir, "state.json", { thread: owner, id, phase: "watching" }); + const foreign = status(dir, other); + assertEquals(foreign.ownedByCurrentThread, false); + assertEquals(foreign.thread, owner); + assertEquals(foreign.currentThread, other); + assertEquals(foreign.superseded, null); + assertEquals(status(dir, owner).ownedByCurrentThread, true); + // Reading status must not grant anything: lifecycle actions still refuse the foreign thread. + await assertRejects(() => stop(dir, other)); + } finally { + Deno.removeSync(dir, { recursive: true }); + } +}); + +/** + * A callback already queued to the old thread survives supersession. If that thread is ever + * resumed, its ack must refuse the callback, or two threads would drive the same flow. + */ +Deno.test("supersede checks the logged run and makes the owner's queued callback ignorable", async () => { + const dir = Deno.makeTempDirSync(); + const owner = crypto.randomUUID(); + const successor = crypto.randomUUID(); + const id = crypto.randomUUID(); + try { + save(dir, "manifest.json", { thread: owner, id }); + save(dir, "state.json", { thread: owner, id, phase: "queued", result: { code: 0, signal: null } }); + await assertRejects(() => supersede(dir, id, owner)); + // A directory that does not hold the run the log names is refused before anything is written. + await assertRejects(() => supersede(dir, crypto.randomUUID(), successor)); + assertEquals(exists(join(dir, "ack.json")), false); + const retired = await supersede(dir, id, successor); + assertEquals(retired.cleanup, "complete"); + assertEquals(retired.superseded.by, successor); + // The same successor may retry after an interruption; a different one may not take the run as well. + assertEquals((await supersede(dir, id, successor)).superseded.by, successor); + await assertRejects(() => supersede(dir, id, crypto.randomUUID())); + assertEquals(acknowledge(dir, id, owner), { ignored: "superseded", by: successor }); + assertEquals(status(dir, owner).superseded?.by, successor); + assertEquals(status(dir, owner).acknowledged, false); + } finally { + Deno.removeSync(dir, { recursive: true }); + } +}); + +/** A directory whose records disagree is not a run, and supersession must leave no claim in it. */ +Deno.test("supersede refuses inconsistent run records without writing a claim", async () => { + const dir = Deno.makeTempDirSync(); + const id = crypto.randomUUID(); + try { + save(dir, "manifest.json", { thread: crypto.randomUUID(), id }); + save(dir, "state.json", { thread: crypto.randomUUID(), id, phase: "watching" }); + await assertRejects(() => supersede(dir, id, crypto.randomUUID())); + assertEquals(exists(join(dir, "ack.json")), false); + assertEquals(exists(join(dir, "cancel.json")), false); + } finally { + Deno.removeSync(dir, { recursive: true }); + } +}); + +/** + * Runs acknowledged by an older copy of this helper have an `ack.json` without a trailing + * newline or `supersededBy`. They must still count as the owner's claim. + */ +Deno.test("a claim written by an older helper counts as the owner's acknowledgment", async () => { + const dir = Deno.makeTempDirSync(); + const owner = crypto.randomUUID(); + const id = crypto.randomUUID(); + try { + save(dir, "manifest.json", { thread: owner, id }); + save(dir, "state.json", { thread: owner, id, phase: "queued", result: { code: 0, signal: null } }); + Deno.writeTextFileSync(join(dir, "ack.json"), JSON.stringify({ id, at: new Date().toISOString() })); + assertEquals(acknowledge(dir, id, owner), { ignored: "duplicate" }); + await assertRejects(() => supersede(dir, id, crypto.randomUUID()), Error, "already acknowledged"); + assertEquals(status(dir, owner).acknowledged, true); + // An older helper interrupted between creating and writing the file leaves it empty. + Deno.writeTextFileSync(join(dir, "ack.json"), ""); + assertEquals(acknowledge(dir, id, owner), { ignored: "duplicate" }); + await assertRejects(() => supersede(dir, id, crypto.randomUUID()), Error, "already acknowledged"); + } finally { + Deno.removeSync(dir, { recursive: true }); + } +}); + +/** + * The owner and a successor race for the same callback when the owner's thread is still + * loaded. Whichever claims first wins, so the flow never ends up driven by both. + */ +Deno.test("an owner's acknowledgment and a successor's supersession exclude each other", async () => { + const dir = Deno.makeTempDirSync(); + const owner = crypto.randomUUID(); + const id = crypto.randomUUID(); + try { + save(dir, "manifest.json", { thread: owner, id }); + save(dir, "state.json", { thread: owner, id, phase: "queued", result: { code: 0, signal: null } }); + assertEquals("result" in acknowledge(dir, id, owner), true); + await assertRejects(() => supersede(dir, id, crypto.randomUUID()), Error, "already acknowledged"); + assertEquals(exists(join(dir, "cancel.json")), false); + assertEquals(status(dir, owner).acknowledged, true); + } finally { + Deno.removeSync(dir, { recursive: true }); + } +}); + +/** + * A runner that does not finish (or is mid-delivery to the owner) must not turn a committed + * retirement into an error, because the successor then cannot tell whether to rearm. + */ +Deno.test("supersede reports pending cleanup for a runner that has not settled", async () => { + const dir = Deno.makeTempDirSync(); + const owner = crypto.randomUUID(); + const successor = crypto.randomUUID(); + const id = crypto.randomUUID(); + try { + save(dir, "manifest.json", { thread: owner, id }); + save(dir, "state.json", { thread: owner, id, phase: "delivering", result: { code: 0, signal: null } }); + const retired = await supersede(dir, id, successor, 100); + assertEquals(retired.cleanup, "pending"); + assertEquals(retired.phase, "delivering"); + assertEquals(exists(join(dir, "cancel.json")), true); + assertEquals(acknowledge(dir, id, owner), { ignored: "superseded", by: successor }); + } finally { + Deno.removeSync(dir, { recursive: true }); + } +}); + /** Supply a recording queue executable through context injection, without changing process environment. */ async function fixture() { const directory = Deno.makeTempDirSync({ prefix: "codex wake test " }); @@ -466,6 +772,32 @@ Deno.test("stop signals the live watcher and does not queue a cancellation wake" } }); +/** + * Supersession must stop a live foreign watcher through its own runner, so the old + * thread is never queued a wake for a flow it no longer runs. + */ +Deno.test("supersede cancels a live foreign watcher without queueing its owner", async () => { + const { directory, watcher, context } = await fixture(); + const dir = join(directory, "run"); + try { + await start(dir, [Deno.execPath(), "run", "-A", watcher, "30000", "0"], context); + await until(() => exists(join(directory, "ready.json")) ? true : undefined); + const successor = crypto.randomUUID(); + const { manifest } = inspect(dir); + const state = await supersede(dir, manifest.id, successor); + assertEquals(state.phase, "cancelled"); + assertEquals(state.cleanup, "complete"); + assertEquals(state.superseded.by, successor); + assertEquals(exists(join(directory, "cancelled.json")), true); + assertEquals(exists(join(directory, "queued.json")), false); + // Supersession is recorded before cancellation, so the owner sees a takeover, not a plain stop. + assertEquals(acknowledge(dir, manifest.id, context.thread), { ignored: "superseded", by: successor }); + } finally { + if (exists(join(dir, "state.json"))) await stop(dir, context.thread); + Deno.removeSync(directory, { recursive: true }); + } +}); + /** * Real descendants exercise group ownership and inherited-pipe behavior that a * single-process watcher cannot cover. A release file also bounds failed tests