Repository navigation
Fix intermittent decision-resolution assertions in Node CI #1200
Description
DR6 in the decision-resolution acceptance suite can fail and then pass on the same commit in Node CI. Find and fix the cause while preserving its repeat, concurrency and interrupted-publication guarantees.
On commit 7dfe3e919f9e5ece4245e19d0fafaa8bc97a71dd, pnpm test:node 1/18 failed once and passed on a single rerun with no source change. Both jobs used Linux and Node 22.23.4. The original complete output was saved before any rerun.
- First failure, job 128859642450: DR6 “A5 repeats, contenders and interrupted publication never duplicate or adopt effects” returned
false !== true; suite result was pass 0, fail 1, exit 1. - Unchanged-commit pass, job 128869915750: same command and revision, pass 1, fail 0, exit 0.
The failing output did not identify which of four boolean assertions failed or show its compared values. The run exposes no downloadable journal artifact. The cause is unestablished; neither a product race nor a test-only defect has been demonstrated.
PR #1198 adds diagnostics at signed commit c6f995bc26cfe8aa1e3cedc350ca9deef6fc89ab: named contender failures include both returned envelopes and stderr; receipt and held-continuation comparisons show the changed text. It preserves all four acceptance conditions and changes no production behavior. Local Deno, Node 26 and verified Node 22.23.2 checks passed. Its Node CI shard also passed. These passes do not establish a cause fix.
To investigate, use packages/cli/tests/adversarial-decision-resolution.test.ts and its executable DR6 fixture, examples/adversarial-implementation/tests/components/Fixture/DecisionInterruption.md. Preserve the first diagnostic failure before rerunning. Keep all accepted A5 conditions: one confirmed successor, losing contenders retain the original hold/reservation, repeats do not duplicate dispatch, and uncertain publication is never adopted from a matching file.
Close this issue only after a reproduced or directly traced cause is corrected and discriminating regression evidence is recorded. An unchanged green rerun restores CI health but does not close this issue.
Related feature: #1165. Delivery PR: #1198.
Additional Bun DR6 observation on 220e54f
On unchanged Main commit 220e54f86dd3698311ee57e660afd00b3e3bc2e1, Linux Bun 1.4.0 (34cbb9a40), command bun run test:bun 2/10 failed in attempt 1, job 129563086045. DR6 A5 repeats, contenders and interrupted publication never duplicate or adopt effects reported Unconfirmed contenders must retain the held reservation. The new diagnostic exposes the first envelope as running / confirmed, while the second is hold / held with DecisionRefused: Snapshot target already exists or is unavailable; both captured stderr strings are empty. This identifies the assertion, not its cause.
One failed-jobs-only retry on the same commit passed: attempt 2, job 129626414165, with the same enabled decision-resolution wrapper and complete shard successful. No source or acceptance assertion changed. This extends the observed runtime evidence to Bun without establishing that the Node and Bun observations share a cause.
Complete first output was saved before retry: 508,680 bytes, SHA-256 e931bcbf23520dc1793be6a0bf28a00c536d3f6317801553a9d2b96f2c28e938. Complete passing output: 627,280 bytes, SHA-256 4404f56d9e6c65808f9fb9c862450b30ba25d74d3b675dc9b365f7255dc106f2. Retained under /private/tmp/xmd-repl-history-capacity-20261109/ as main-original-failure-1.log and main-retry-bun.log, with exact-source/job metadata in main-health-recovery.json. Main is healthy after this single retry; keep the issue open pending a demonstrated cause fix and regression evidence.
Main recurrence on 2bc5ef2
Main CI 2581 attempt 1, Node shard 2/18 failed at exact main commit 2bc5ef29b2966af27c7ba292dd42a759f6f69f79. DR6 reported Unconfirmed contenders must retain the held reservation: the first envelope is hold / held with DecisionRefused: Snapshot target already exists or is unavailable, while the second is running / confirmed; both stderr strings are empty. All 21 assigned files ran. This identifies the failed condition, not the underlying cause.
Complete first output was saved before retry at /private/tmp/xmd-staged-followup-evidence-20261110/main-first-node2-129634518812.log: 329,837 bytes, SHA-256 a2f09b4b74f82bd6dfb746d6941ed95edec768be5488cbae47a83904d4fd7f1a. One failed-jobs-only retry is selected; result is pending. No source, assertion or allowance change is made. Main failure is tracked by auto-closing #1224 and blocks PR #1203's main-health gate. Keep this flake issue open.
The same-revision failed-jobs-only retry passed this shard: attempt 2, job 129642870624, node shard 2/18: 21 of 21 files ran, all passed, completed 2026-11-10 05:10:53 UTC. No source, assertion or diagnostic overlay changed. Complete passing output: 318,209 bytes, SHA-256 67841b29fab0ace820ad6fc50e97051f17e192dd09b16ebbc4614150faf2985a, retained beside the first output as main-attempt2-node2-129642870624.log. This is unchanged-main intermittence, not a cause fix. Main's separate Implementation shard and aggregate are still pending at this observation; keep #1200 open.
Deno recurrence on workflow 8c0cab54
CI 2586 attempt 1, Deno shard 2/22 failed at 8c0cab5487dea32358a956d6cd8011d88bbc2e63. DR6 reported Unconfirmed contenders must retain the held reservation. This matches the named condition in the prior Node and Bun observations; its underlying cause remains unestablished. No retry or assertion change preceded capture.
Complete first output: 739,984 bytes, SHA-256 bdadab55f5b804224d9cc8c3bdd63a90cb09aa0f50ece2bdf97576d6a932dd48, retained at /private/tmp/xmd-staged-followup-evidence-20261110/ci-first-8c-job-129764688587.log. Command deno run --allow-all --frozen scripts/runtime-tests.ts deno 2/22, Linux Deno 2.9.5. Keep this issue open; this observation alone establishes neither an unchanged-revision pass nor a cause fix.
Recurrence on final guide-timing head 6f69a4e4
CI 42806096637 attempt 1, Deno 2/22 job 129795481359 reports the same DR6 named condition, Unconfirmed contenders must retain the held reservation, at exact signed 6f69a4e408372628ab9f0f60ec8a0edc7dfee62e. All 26 assigned files ran. Command: deno run --allow-all --frozen scripts/runtime-tests.ts deno 2/22, Linux Deno 2.9.5.
Complete first output was captured before any retry: 739,556 bytes, SHA-256 0a6ab58d9590708bcc656687ffc9ee498ed31db08e229efe9a7fb75aee0687a3, retained at /private/tmp/xmd-staged-followup-evidence-20261110/ci-first-6f-job-129795481359.log. One unchanged-head retry of this failed branch job is selected after the remaining current job finishes. No source, assertion, deadline or allowance change; no cause fix or common-cause proof is claimed. Keep this issue open.
The selected single retry passed on unchanged 6f69a4e408372628ab9f0f60ec8a0edc7dfee62e: attempt 2, Deno 2/22 job 129809702276. DecisionResolution completed successfully in 1m55s; all 26 assigned files passed. No source, assertion or deadline changed. This establishes same-head CI intermittence for this Deno occurrence, without diagnosing a shared cause.
Complete passing output: 727,696 bytes, SHA-256 ea4ad7f5a089731cce6e79367fc540d294c9ab8a3f772f7bf018ed18f52afcae, at /private/tmp/xmd-staged-followup-evidence-20261110/ci-6f-dr6-retry-job-129809702276.log. All final workflow technical checks pass; Main health and its dependent aggregate remain red under #1224. Keep #1200 open; no additional retry is selected.
Current Main Bun recurrence on c900ebf
Main CI 42817294816, Bun 2/10 job 129833504061 failed at exact current main c900ebfdaf2484cfa61b540dd0b1c454448055ae. DR6 reports Unconfirmed contenders must retain the held reservation. The first contender returned status=running, publication=confirmed; the second returned status=hold, publication=held, but failed the retained-reservation assertion. All 44 shard files ran. This is distinct from the Implementation120s timeout; no common cause is established.
Complete first output is retained at /private/tmp/xmd-main-1143-repair-20261110/current-main-first-bun2-129833504061.log: 686319 bytes, SHA256 528b98096bd8e5c5eadc98d94eead1a7c5c2251a40d5525f2f3b3209a2c38a91. No retry or source/assertion change preceded this capture, and no retry is selected. Current-main failure makes this a blocking recurrence alongside the separately requested causal #1143 repair. Its evidence is recorded separately; the held #1203 workflow is not resumed.
Activity
taras commentedon Nov 9, 2026
Bun occurrence on PR #1204 at signed head
3ff8deaad7add3933159890b1599e2fd62c47e60: CI 42612041600, Bun shard 1/10 failspackages/cli/tests/adversarial-decision-resolution.test.ts, DR6 A5 “repeats, contenders and interrupted publication never duplicate or adopt effects.” Command:bun run test:bun 1/10; Linux, Bun 1.4.0, Deno 2.9.5. All 46 assigned files ran. This is an assertion failure, not a timeout.The diagnostics added in #1198 now expose the differing envelopes. One contender confirmed the single successor. The other retained the original continuation but returned a generic DecisionRefused hold with
Snapshot target already exists or is unavailable, rather than the held reservation expected by the fixture. The complete first job log was preserved before any retry: 635393 UTF-8 bytes, SHA256b08f3b84fd3790a8af08c488de8e36870f6a09618fd6965a64be31995904a956, retained with the first diagnostic and receipt under/private/tmp/xmd-adapter-repairs-01a1be06/ci-3ff8-bun1-first-*.A focused unchanged-head local run,
bun test --timeout=300000 packages/cli/tests/adversarial-decision-resolution.test.ts, passes (1 test, 134.40s). Local macOS arm64/Bun 1.3.14/Deno 2.9.7 differ from CI; the inner contenders execute through Deno. This is qualified same-source evidence, not an identical-CI retry pass.Read-only tracing found no changed #1202 path into this fixture: it never resolves an embedded adapter. A lead for this issue's investigation is Ticket.Documents after the exclusive reservation mkdir in ResolveDecision.md:98. Snapshot's unused() check conflates unavailable with existing, and Entry.ResolveDecision's “without changing the managed ticket” message can follow reservation creation. The envelopes alone do not establish the exact interleaving or a causal repair. Capture per-contender reservation/snapshot state before changing production or the fixture.
One failed-jobs-only CI retry will be requested once the current run completes. No source, assertion, timeout or exclusion changed; keep #1200 open.
taras commentedon Nov 9, 2026
The single failed-jobs retry passed at unchanged
3ff8deaad7add3933159890b1599e2fd62c47e60: Bun shard 1/10, job 129176681527. The full required CI aggregate is green. No source, assertion, timeout or exclusion changed between the failed and passing attempts. This supplies same-revision CI intermittence evidence and restores delivery health; it does not fix the DR6 cause. The first complete failure and read-only diagnosis recorded above remain preserved. Keep #1200 open.taras commentedon Nov 9, 2026
DR6 recurrence on both post-merge main and the separate Claude reconnect PR:
ccd70ca3421df8ee725036a97697204a62344566, main CI 42623737223, Bun shard 1/10, attempt 1,bun run test:bun 1/10, Linux/Bun 1.4.0. Complete first output: 634,013 bytes, SHA2567aea6bb77c9e8edaaf3da39a99f298bf1bbf8d293edc6ec8dbef826fa23a9977.1ff203b261ca32b9041c8e6659729e39e66750bc, PR #1214 CI 42643905855, Deno shard 2/22, attempt 1,deno run --allow-all --frozen scripts/runtime-tests.ts deno 2/22, Linux/Deno 2.9.5. Complete first output: 522,892 bytes, SHA256a381f7b87c67f455c238940d1b86c194b361a65aa9c800dc4c64831fc6364001.Both fail “Unconfirmed contenders must retain the held reservation.” The captured envelopes include one held/unconfirmed result and another confirmed/running successor. This records the observed assertion and returned states; it does not establish the cause. Original outputs are retained under
/private/tmp/xmd-adapter-repairs-01a1be06/asmain-ccd70-129182094255-first-full.logandci-1214-deno2-first-full.logbefore any rerun. No source or acceptance assertion changed. One failed-jobs-only retry of main is planned; the PR run is still completing. Keep #1200 open after any green retry.taras commentedon Nov 9, 2026
The single failed-jobs retry of unchanged main
ccd70ca3421df8ee725036a97697204a62344566passed Bun shard 1/10, job 129261976242: all 46 assigned files passed under Bun 1.4.0. The complete passing log is 616,548 bytes, SHA2565a0b7cb3f42abd2bca970cad0b147ef14b29c6d6d053e8ef023fb2463b3e712a, retained as/private/tmp/xmd-adapter-repairs-01a1be06/main-ccd70-129261976242-retry-full.log. No source, assertion, deadline or exclusion changed between attempts. This supplies another same-revision fail/pass pair; it does not identify or fix the DR6 cause. Keep #1200 open. The separate #1214 Deno recurrence is still awaiting its delivery retry.taras commentedon Nov 9, 2026
PR #1214's single failed-jobs retry passed at unchanged
1ff203b261ca32b9041c8e6659729e39e66750bc: Deno shard 2/22, job 129272398377, all 24 files passed. The required aggregate is green. The complete passing log is 511,008 bytes, SHA256e588ed1c8b8dedb0a1c5cea079fa976af131245a147ba719f7952e170d12c631, retained as/private/tmp/xmd-adapter-repairs-01a1be06/ci-1214-deno2-retry-full.log. The first Deno failure above and this pass use the identical source revision; no assertion, deadline, runtime version or exclusion changed. This establishes same-revision intermittence under Deno as well. Keep #1200 open until the cause is fixed and regression evidence is recorded.taras commentedon Nov 10, 2026
On workflow head
40d6cf0874afa5186ce8f4db6453d23f84450818, Node 2/18 job 129821464222 fails the same named DR6 condition:Unconfirmed contenders must retain the held reservation. The first envelope is hold/held with Snapshot target already exists; the second is running/confirmed; both stderr strings are empty. This identifies the condition, without establishing a common cause.Complete first output was captured before any rerun: 354,466 bytes, SHA-256
19730e4c7f5d8037bc73555a955834d23ab4dec32e45b5609482f5ff54df2d07, retained at/private/tmp/xmd-staged-followup-evidence-20261110/ci-40d-node2-first.log. A single unchanged-head retry of this failed job is selected after the remaining run finishes; no assertion, deadline, source or allowance change. Main health remains a separate failure under #1224. Keep this issue open.taras commentedon Nov 10, 2026
A separate occurrence is now retained on Syntax correction
50e26a423efdca71d42deaf57ddcc3a47a98a8b8: Deno 2/22 job 129827082500, CI42815398360. Commanddeno run --allow-all --frozen scripts/runtime-tests.ts deno 2/22runs all25 assigned files and fails DR6Unconfirmed contenders must retain the held reservation. First envelope is running/confirmed; the other holds. This matches the named condition without proving a common cause.Complete first output, saved before any rerun: 523,910 bytes, SHA-256
ccbe3984648b3dec286bc1ce77a5d73dcedba418124df2cdc42a846a10d7ddda, at/private/tmp/xmd-staged-pilots-20261110/1178/ci-50e-deno2-first.log. One unchanged-head retry of this failed job is selected after the original run settles. No source, assertion, deadline or allowance change; Main health is separate under #1224. Keep this issue open.taras commentedon Nov 10, 2026
The selected single retry passed on unchanged workflow head
40d6cf0874afa5186ce8f4db6453d23f84450818: Node 2/18 job 129833897206, attempt 2. All 24 assigned files passed. Commandpnpm test:node 2/18, Linux Node 22.23.4; no source, assertion or deadline changed.Complete passing output: 342,142 bytes, SHA-256
462acdb063aa131338742d44c081fc00b9971d6106755a993ffa251e3fc4bbe2, retained at/private/tmp/xmd-staged-followup-evidence-20261110/ci-40d-node2-retry-pass.log. This establishes same-head intermittence for this occurrence, without diagnosing a cause. All workflow technical checks now pass; Main health and its aggregate remain red under #1224. No further retry is selected. The separate Syntax-head Deno occurrence above still awaits its selected retry.taras commentedon Nov 10, 2026
The selected source retry passed on unchanged
50e26a423efdca71d42deaf57ddcc3a47a98a8b8: Deno 2/22 job 129838905555, attempt 2. All 25 assigned files passed. Commanddeno run --allow-all --frozen scripts/runtime-tests.ts deno 2/22, Linux Deno 2.9.5; no source, assertion or deadline changed.Complete passing output: 511,806 bytes, SHA-256
7c86f0fb7d6db576b8133e6bf34526d09dca1130c8578720e3458005211fa494, retained at/private/tmp/xmd-staged-pilots-20261110/1178/ci-50e-deno2-retry-pass.log. This establishes same-head intermittence for this occurrence, without establishing a cause or common cause with the other runtime observations. No further DR6 retry is selected. Keep #1200 open.taras commentedon Nov 10, 2026
Current Main advanced to
c900ebfdaf2484cfa61b540dd0b1c454448055aethrough #1230. Main CI42817294816, Bun 2/10 job 129833504061, attempt 1, fails the same named DR6 condition:Unconfirmed contenders must retain the held reservation. Commandbun run test:bun 2/10; all 44 files ran. All other technical Main jobs pass; #1224 remains open.Complete first output was saved before any rerun: 686,318 bytes, SHA-256
7e43eb6ee02fb9416408f0f848ad21a5ceb56f9e88679ceaf8fcec99e7b11e9d, at/private/tmp/xmd-staged-followup-evidence-20261110/main-c900-bun2-first.log. Current run is completed at attempt 1 with no retry active. One unchanged-head retry of this single failed job is selected to recover the shared health gate, without source, assertion or deadline changes. This is the new Main revision's first retry; no further retry of old Main2bc5ef29is selected. The shared condition does not establish a common cause or cause fix.taras commentedon Nov 10, 2026
The selected current Main retry passed on unchanged
c900ebfdaf2484cfa61b540dd0b1c454448055ae: Bun 2/10 job 129846213608, attempt 2. All 44 assigned files passed. Complete passing output: 669,854 bytes, SHA-256e1e6991f95ab451d7f3608d9b0af6dd775df8251feb374a075369773b0378568, retained at/private/tmp/xmd-staged-followup-evidence-20261110/main-c900-bun2-retry-pass.log.Main's aggregate CI is now successful at this exact current head, and #1224 closed automatically. No source, assertion or deadline changed, and no further Main retry is selected. This is same-head intermittence for this occurrence, not a cause fix or common-cause proof. #1200 remains open. The three already technically passing draft PRs are refreshing only their Main-health gates.