Implementation Markdown journey intermittently exceeds 120s in CI #1143

The Implementation Markdown journey exceeds its existing 120-second launcher deadline in a local four-process Deno batch, then passes alone at the same source revision without a source fix.

Command: deno task test packages/cli/tests/adversarial-implementation.test.ts. The batch uses xargs -P 4 to run each of ten wrapper files in its own process. The Deno version observed immediately after this local run is 2.9.7, aarch64-apple-darwin (V8 15.0.245.2-rusty, TypeScript 6.0.3).

The failing output identifies runs the Implementation Markdown journeys without native models:

timed out after 120000ms (load average 8.5, 6.6, 5.2)
--- stdout before the deadline: nothing ---
--- stderr before the deadline: nothing ---
FAILED | 0 passed | 1 failed (1 step) (2m0s)

Both run-start head.txt files name the same revision and both worktree.patch files are empty (SHA256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). This establishes source-identical intermittence under different concurrency/load conditions. Contention, nested startup cost and a stalled inner operation remain possible causes; no cause fix is established. The deadline is unchanged.

This was a local execution, so there is no GitHub Actions run URL. Run location: /private/tmp/xmd-combined-workflow-pilot-20261105/ticket/attempts/0003/runs/{deno-ten,deno-implementation-alone}. Preserved first output, numeric exit ledgers, both worktree snapshots and their hashes are under /private/tmp/xmd-continuation-call0003-20261106/execution-call0003/observed-first-failures; the durable copy is under /Users/tarasmankovski/.codex/visualizations/2026/11/04/01a1a4bf-bfa9-7801-905d-e364192bc859/continuation-call0003-20261106. Codex execution thread: 01a1a4bf-bfa9-7801-905d-e364192bc859.

Keep this issue open until the cause is fixed and focused regression evidence is recorded. A passing rerun does not close it. Related: #1119 (Friction journey timeout).

The same journey also times out in combined delivery CI42213983733 attempt1, Node5/17, at signed commit ede14f092352542b08b84e5d4e1470978d194b4f. The unchanged120000ms wrapper deadline expires with empty captured stdout/stderr; reported load averages1.3,0.9,0.4. Toolchain: Node22.23.4, Deno2.9.5 and Bun1.4.0 on ubuntu-latest. Complete first shard log: 493290 bytes, SHA256 c614e5787f2b6c8af035cef74c406249dde52c9470d000764cd5746c684b2b25, retained before any rerun under /private/tmp/xmd-combined-delivery-20261106/ci-first-failures/node-5-attempt1/ and its durable copy. All first CI logs are also preserved in a complete archive.

A bounded Linux diagnostic42216703321 checks out that exact source with the recorded CI toolchain, keeps the actual wrapper cap unchanged, and instruments the direct Node CLI instead. It completes the Implementation path in99.416 seconds; journal observations place about96.652 seconds in the nested Deno command. The executed outer Node profile is mostly idle. This successful instrumented sample does not identify the cause of the original timeout or count as an unchanged-wrapper rerun. Its240s diagnostic safety cap is separate from the120s test limit. A failed-job CI retry is running on unchanged source; no timeout increase, source correction or exclusion is made. Keep the issue open pending a cause fix and regression evidence.

Delivery diagnosis and bounded correction

The failed-job-only retry also fails: CI42213983733 attempt2, Node5/17 again exceeds120000ms. Complete first retry log:493314bytes, SHA256 8cf52ce159e169d93218c7e07cdfe7195ede79d5cbc2215c05dbaad9ef76a9e5. No third unchanged-source CI retry is planned.

A second bounded Linux comparison42219724791 completes the unchanged Implementation wrapper in82.483s at the same source. It invokes root-imported tsx4.21.0 rather than proving the runtime driver's prepared .bin/tsx4.23.15 parent. This successful diagnostic does not erase either CI failure.

Independent Astra traced an actual deadline overlap: Implementation's outer runCli120s matches the full allowance of its inner Test/child, leaving no startup or cleanup headroom. Under the user's continued delivery authority, a narrow helper correction is being implemented to give only Implementation and Friction outer180s headroom. Inner guards, assertions, execution paths and all other limits remain. This corrects enclosing deadline structure; it does not establish a production cause for the failed observations. Focused evidence, exact-commit independent review and fresh hosted verification remain required. Keep this issue open.

The narrow outer-headroom correction is signed at 0c6b488f4af7dea23953a38181b127d72d9e7cac in PR #1144 and passes exact-commit independent Astra review. Corrected CI42222261409 attempt1 is fully green (60 applicable jobs succeed, expected composability skip); the real Node Implementation wrapper passes in81.276s. Four focused corrected wrappers also pass on local Node22.23.4/Bun1.4.0/Deno. All inner deadlines and assertions remain. These successful observations complete current CI recovery, but their durations remain below the former120s bound, so they do not establish the original production cause. Keep this flake open. Complete coherent corrected-source measurement42222287812 and measured shard calibration are still running/pending; no partial timing record is imported.

Complete coherent measurement evidence

Measurement42222287812 attempt1 completes all three runtimes and assembly at unchanged corrected source 0c6b488f4af7dea23953a38181b127d72d9e7cac; official artifact digests, exact corpus/provenance and values are verified and imported byte-for-byte in PR #1144 metadata5a2b. The actual Node Implementation case completes in121571.756343ms (wholefile122007ms), above the former120000ms outer limit and below the current180000ms limit, with inner guards and assertions unchanged. This shows completed work using the added enclosing headroom in this observation; it is not a universal historical cause fix or a discriminating regression by itself. The complete original Node measurement log and exact case/file receipts are retained in post-pilot-headroom-measurement-evidence.json in the coordinator’s durable bundle.

PR #1144’s final required CI is green after a separately recorded unchanged-head TestAgent recovery (#1146). Its seven-minute/five-run shard timing calibration is explicitly maintainer-deferred for that delivery only, not passed. This issue stays open for the user-authorized cause/regression reconciliation through the workflow accepted by the actual NEW #922 pilot. No new diagnostic or source repair starts from this evidence update.

Additional unchanged-source observation, 2026-11-07

At PR #1163 head 4320ec168a501d1abd13375598b213eaba46367f, CI 42385177050 attempt 1, Deno 18/22 fails packages/cli/tests/adversarial-implementation.test.ts: the inner “Implementation fixture journeys” Test reaches its unchanged 120-second deadline. The whole wrapper reports about 2m1s. This is a timeout observation, with no demonstrated new causal defect in the candidate.

The identical signed revision and clean worktree passes the focused file locally in 1m41s (within a two-file Deno invocation; Preparation follows it). Command: deno task test packages/cli/tests/adversarial-implementation.test.ts packages/cli/tests/adversarial-preparation.test.ts; final exit 0. The first complete CI log is 624,834 bytes, SHA256 53de767ca2196608d1c523607dc3bb70f4bd7bc4192066717a67cd77d6942df5, saved before retry. Local output SHA256 c72644a7cf71694a0ff8c2d6948bb8c60e38b7bf27d21a10e614b3d3178d6310. Full logs, source checks and runtime versions are retained under /Users/tarasmankovski/.codex/visualizations/2026/11/04/01a1a4bf-bfa9-7801-905d-e364192bc859/1163-ci432-timeout-evidence (manifest.json).

The local comparison preserves source and deadlines but uses Darwin arm64, Deno 2.9.7 and Node 26.5.1; the CI failures are on hosted Ubuntu (Deno 2.9.5; Node job 22.23.4). This is qualified same-source passing evidence under a different host/toolchain, not proof of a passing same-job CI rerun or a shared cause. One failed-jobs-only CI retry has been requested; its result is pending. Source, deadlines, exclusions and correction allowances are unchanged. Keep this issue open until a cause fix and regression evidence exist.

Same-head CI retry result, 2026-11-07

On unchanged commit 4320ec168a501d1abd13375598b213eaba46367f, the single failed-jobs retry completed successfully for test-deno (18/22). These are attempt 2 of the same CI run; the first failing output remains preserved as recorded above. This supplies CI intermittence evidence in addition to the qualified local pass. It does not establish a common cause or repair the timeout. The issue remains open. The run's remaining failures are Main's health gate and its aggregate, separate from these product checks.

Additional Bun observation, 2026-11-07

CI 2514, Bun shard 7/10 at 0b78b818adfc4f03ec4b29532c069f6f8126f058 failed the same named Implementation fixture journeys inner 120-second deadline (121,876.24 ms wrapper). All 46 shard files continued to run. The complete first log was saved before any rerun at /private/tmp/xmd-1163-try-delivery-20261107/ci0b78-first-bun-7-10.log. This is a Bun observation, not proof that its cause matches the historical Deno timeout. No retry or timeout/source change was made for this observation; independent review separately requested two consumer corrections. The issue stays open.

The same 0b78b818 CI run also failed Node shard 13/18 at the inner Implementation fixture journeys 120-second deadline. All 42 files ran. Its complete first output is retained at /private/tmp/xmd-1163-try-delivery-20261107/ci0b78-first-node-13-18.log. This adds a Node observation alongside Bun; neither supplies a causal diagnosis or identical-source retry pass. No retry of this source was requested while the separate independent-review correction was being completed.

Node recurrence on PR #1190, 2026-11-08

Node shard 16/18, commit 818cacaafd5101d0b890a94a5d4cd7f7bf58ebd7, failed packages/cli/tests/adversarial-implementation.test.ts: Implementation fixture journeys reached its unchanged 120-second deadline at 18:05:39 UTC. The Node wrapper reported 123866.275398 ms. All 30 assigned files still ran; this was the shard's first failure. No rerun or source/deadline change was made for this observation.

The complete first log is retained locally (695718 bytes, SHA256 b2ca74b2a8d331de9c1facb57a607774cf666698dcc178c8e9e56c03f92297a2). The uploaded artifact, test-journals-node-16-attempt-1, was downloaded and inspected: 67894-byte ZIP, SHA256 660544d32f5501bbf5ee43aa5d1ef768ef18d9231207910d758489ab3d6186ef. It contains only child-9.jsonl and child-10.jsonl under an invocation beginning 2026-11-08T18-05-48.108Z, after the Implementation failure; neither contains the named Test or its 120-second timeout. Its existence therefore does not supply the timed-out execution's trace. Implementation still takes the legacy xmd run carrier path, outside automatic xmd test failure-journal retention.

This adds another Node observation to the existing Deno, Bun and Node history. It does not identify the timed-out inner operation or prove that the causes match. Keep this flake issue open; current main's separate green run is not an exact-revision fix or retry for this old PR head.

Bun recurrence in PR #1203

At signed head ea21243cb186ceb821ceb42eab581098d5ac78fb, CI run 42553057791, Bun shard 5/10 fails packages/cli/tests/adversarial-implementation.test.ts: the inner Implementation fixture journeys Test reaches its unchanged 120-second deadline. The wrapper reports 121,945.11 ms. Command: bun run test:bun 5/10; Linux, Bun 1.4.0 with Deno 2.9.5 and Node 22.23.4. All 46 assigned files still ran.

The complete first raw job log was preserved before further execution: 479,663 bytes, SHA-256 0c08bb5541e7d88dbc6a0346f80d0faf1b0bcdd21f8e3fe9188f5948868cbd39, retained at /private/tmp/xmd-staged-submission-evidence-20261108/ci/repair-bun5-first.log. The workflow artifact query returned no artifacts. The legacy carrier still supplies no interrupted inner-operation attribution.

A single focused local command, bun test --timeout=300000 packages/cli/tests/adversarial-implementation.test.ts, passes in 105.019 seconds on successor head 606845dc597101b804168c3006fb13a1e07eb5d1, macOS, Bun 1.3.14. That successor adds only a literal to the separate expected-wrapper inventory; it changes no Implementation path, assertion or deadline. Local log: 287 bytes, SHA-256 2895b8cc2f19aaf2648c3b4b5bcdc4892cd25dcbd4e3c9e112b1b872bf3815e6. This different-host, different-head pass is qualified evidence, not an exact-head CI retry or causal repair.

No unchanged-head retry of this failure was requested. The older run's unfinished jobs were cancelled after preserving completed failures, because the successor corrects a separate explicit-inventory failure and has its own fresh CI. On successor head 606845dc597101b804168c3006fb13a1e07eb5d1, fresh CI job 128954521507 passes all 46/46 files. Implementation takes 100,258.30 ms and ProductionRole also passes. Complete passing log: 472,358 bytes, SHA-256 fc8d7fc48ec7f258c2c7642ce1be07781c98c65b31e6d708e7b2e735bfaac429, retained as latest-bun5.log. This different-head CI pass changes only the separate expected inventory; it supplies recovery evidence without establishing an exact-head retry or a cause fix. Preserve the existing cause/regression contract, deadlines and assertions; keep this issue open.

Main CI recurrence on 220e54f

On unchanged Main commit 220e54f86dd3698311ee57e660afd00b3e3bc2e1, Linux Deno 2.9.5 command deno run --allow-all --frozen scripts/runtime-tests.ts deno 22/22 failed in attempt 1, job 129563085730. The output identifies Implementation fixture journeys failing with test timed out after 120 seconds.

One failed-jobs-only retry on the same commit passed: attempt 2, job 129626415508. The Implementation step reports ok (1m58s) and the complete shard succeeds. No source, assertion or deadline changed. This is exact-revision CI intermittence; it does not establish a cause fix or undo the earlier enclosing-deadline distinction.

Complete first failure was saved before the retry: 679,112 bytes, SHA-256 80e1d020c9f8e85148194ae5f129d03b209c1be8ad86c019c934ad7ae7e2119f. Complete passing output: 900,882 bytes, SHA-256 2076f35a2a490b9389cbc9c494c499df6b369a78155fd528df164dec0b5a13e7. Both are retained under /private/tmp/xmd-repl-history-capacity-20261109/ as main-original-failure-0.log and main-retry-deno.log; main-health-recovery.json records the exact jobs and source identity. Main CI is green after this single retry; keep this flake issue open pending a cause correction and regression evidence.

Main recurrence on 2bc5ef2

Main CI 2581 attempt 1, Node shard 14/18 failed at exact main commit 2bc5ef29b2966af27c7ba292dd42a759f6f69f79. Implementation fixture journeys exceeded its existing 120-second deadline; all 33 assigned files ran. No assertion or deadline changed. Complete first output was saved before retry at /private/tmp/xmd-staged-followup-evidence-20261110/main-first-node14-129634518227.log: 3,020,577 bytes, SHA-256 f79769972c5bd2e390c4f448513ab17a4fb1595a556ef8378be2f47a23477cea.

One failed-jobs-only retry is selected to restore main health if this known intermittent failure passes. Result is pending; no cause fix or health recovery is claimed. Main failure is tracked by auto-closing #1224 and blocks PR #1203's main-health gate. Keep this flake issue open.

The selected single retry also failed the same deadline at unchanged Main: attempt 2, job 129642870209, again Implementation fixture journeys / test timed out after 120 seconds; all 33 shard files ran. Complete retry output was preserved: 3,020,508 bytes, SHA-256 5fa2bc738bb114f3b3a7fdd78ae12b11f415a36185244d8e7ad872fe3773dec3, retained at /private/tmp/xmd-staged-followup-evidence-20261110/main-attempt2-node14-129642870209.log.

Main is still red at 2bc5ef29b2966af27c7ba292dd42a759f6f69f79; #1224 and dependent main-health gates remain open/failed. The same retry recovered the separate DecisionResolution job, as recorded in #1200. No third retry, deadline increase, assertion change, source fix or merge was performed. These repeated failures do not identify a cause correction; keep #1143 open for its focused repair.

Bun recurrence on corrected #1183

At exact signed 5a27f27f8864b8020f60d52231152f58282fbb00, CI 42800396618 attempt 1, Bun shard 8/10 failed Implementation fixture journeys at the unchanged 120-second inner deadline. Command: bun run test:bun 8/10, Linux Bun 1.4.0; all 47 assigned files ran. The #1183 R6 correction changes only Sidekick recovery provenance and its spec qualification, not this workflow journey or its deadline.

Complete first output was saved before any retry: 503,403 bytes, SHA-256 9aa8e2d52beb343f9a85d5254d120cbdc2fef889a546229b806bd46feb358e7a, at /private/tmp/xmd-staged-pilots-20261110/1183/ci-correction-first-job-129776093060.log. This matches the named deadline condition, without establishing a common cause. One unchanged-head retry of this failed branch job is selected after the current run finishes; no deadline, assertion or source change. Main's separate repeated failure and #1224 remain unchanged. Keep this issue open.

The single selected retry on unchanged 5a27f27f8864b8020f60d52231152f58282fbb00 passed: attempt 2, Bun 8/10 job 129787111309. Implementation completed in 95,925.87 ms and all 47 shard files passed. No source, assertion or deadline changed. Complete passing log: 494,827 bytes, SHA-256 6b83849fc98963a9ad362adb9dc877e12be8b2aa28a123ea3f24fe8c960aa9c8, retained at /private/tmp/xmd-staged-pilots-20261110/1183/ci-correction-retry-job-129787111309.log. All branch technical checks pass; the aggregate still fails Main health under #1224. This establishes same-revision intermittence for this occurrence, not a cause fix. Keep #1143 open; no further retry is selected.

Active causal repair from current Main c900ebf

The Main-health owner has started a fresh causal repair at exact c900ebfdaf2484cfa61b540dd0b1c454448055ae, isolated branch codex/fix-implementation-timeout-1143. The delivered XMD workflow is running from that same merged revision; it does not resume held #1203 or replay completed #1158. Its first assignment is read-only causal diagnosis and a finite plan, followed by a fresh Opus Implementor and independent Fable review with documented Astra fallback. Existing deadlines, meaningful assertions and process coverage remain required.

Both complete failed logs from 2bc5ef2 were copied with their recorded hashes under /private/tmp/xmd-main-1143-repair-20261110/; ownership.json, task.json and the selected evidence record the source, owners and scope. No third blind retry is selected. Local timestamped traces have separated the five inner groups and process lifetimes, but have not yet established the causal correction. No source implementation or passing cause-fix claim is recorded.

Current Main's separate Bun DR6 failure is captured under #1200. It makes that condition blocking again and is kept distinct from this 120-second timeout unless causal evidence joins them. A green run of the Implementation carrier alone does not complete this requested repair.

PR #1228, Node shard 13/18

On commit 92c27cd3c9ff69092a17ed536d24f39918ae9e7e, pnpm test:node 13/18 timed out in Implementation fixture journeys after 120 seconds on CI attempt 1. The first complete job log has SHA-256 1eabeaf8ba4eeea968d23d4207065467242d1b2b5a6ebed363e7924e6b67258b. The failed-jobs-only retry passed the same Linux/Node shard on that identical commit. The cause remains open.

Matched Linux diagnostic and current draft #1238

Draft #1238 at signed a40e68b903e7a322395cd2d7130657397ccf4c92 overlaps the existing staged and unstaged checkout reads with Markdown All/Spawn. The initial TS helper introduced loader and replay defects. Fable reproduced both; the correction removes the helper and adds a deterministic replay regression. The corrected source passes the focused Deno/Node checks and both actual REPL files on pinned Bun 1.4.0 (16 tests). Planner feedback reports no remaining source defect and holds causal acceptance as missing evidence.

The single completed matched Linux observation, on one ubuntu-latest x86_64 worker with Node 22.23.4, Bun 1.4.0 and Deno 2.9.5, ran the actual Node carrier at original serial 321eac2a07e01ca14830c275117b4e94e26bd4c9 then corrected a40e68b9. Both passed with the original cases and 120-second and 180000ms guards. Elapsed time was 93,058.3 ms then 90,693.6 ms. Each made 92 checkout checks, 92 staged and 92 unstaged reads, 14 commit comparisons and 690 process calls. Checkout inclusive time was 8,948 ms then 8,300 ms. Both traces have zero open elements, queries or exec calls, no capture errors and no open telemetry connections. The 648 ms checkout reduction is much smaller than the macOS comparison. Fixed order, shared caches and diagnostic overhead limit the comparison.

The baseline did not time out, so the observation does not identify the work active at the historical deadline or establish a causal fix. The PR remains a draft and this issue remains open under its original acceptance. No further diagnostic pair is selected.

The isolated diagnostic branch contains only temporary execution tooling and is separate from the PR. Its first run, 42912104413, failed diagnostic preflight when the normal launcher filtered the collector-port environment variable; both carriers stopped before fixture work and captured zero events. Both original channels were retained. Embedding the invocation port in the scratch observer corrected that known setup error before the completed pair. The original source and deadlines were unchanged.

All decoded first outcomes and the complete completed-pair traces are retained under /private/tmp/xmd-main-1143-repair-20261110/. work/recovery/linux-pair-comparison.json records the counts, exact source receipts, per-group spans and progress hashes; work/recovery/attempts/0006-planner-correction-feedback/result.json records the unresolved causal gate. Current Main's #1231 timeout is separate.

Original Bun 9/10 shard with a retained trace

The bounded Linux diagnostic ran the original 46 files in the same order at 345c53ec81645eed8025592cd79b4a3d19d9115f, the actual merge commit checked out by the failed #1240 job. The runner image and Bun 1.4.0 and Deno 2.9.5 pins matched. It used the original cases and 120-second deadline. All files passed. The Implementation fixture took 70,056.9 ms, with inner group times of 25.578, 5.054, 2.058, 2.004 and 33.191 seconds. The 10,419-event trace had no open spans, capture errors or unclosed trace connections. The worker conditions and diagnostic instrumentation differ from the original failure, so this comparison leaves the operation at the deadline unknown.

The first diagnostic stopped before tests because its preflight rejected the setup-generated bun.lock change. Both job outputs show that the original Bun install saves the lockfile. The corrected diagnostic retained the patch and verified that it stayed stable through the shard. Both first outcomes, full trace and receipt are preserved in /private/tmp/xmd-main-1143-repair-20261110/. The completion checkpoint is work/recovery/bun9-1240-trace-checkpoint.md. #1238 remains a draft. A failing hosted trace with active-operation attribution is still needed to establish a cause.