Skip to content

test(engine): stop the listen-session harness from dropping its own socket - #864

Merged
choraria merged 1 commit into
mainfrom
fix/listen-session-alarm-flake
Aug 1, 2026
Merged

test(engine): stop the listen-session harness from dropping its own socket#864
choraria merged 1 commit into
mainfrom
fix/listen-session-alarm-flake

Conversation

@choraria

@choraria choraria commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Reproduces and fixes the last unexplained flake from the local-dev-parity lane.

The failure

ListenSession — fail-safe alarm + idle > never throws on a poll error: re-arms the alarm failed
once in CI on 2026-07-29 with expected null not to be null, and never reproduced in 11 local runs.
test is a required check with bypass_actors: [], so a live flake here blocks every PR.

Two earlier hypotheses were falsified: a wall-clock race against the 2s re-arm, and reauthReason
failing closed on a missing boundAtMs.

Root cause — the harness, not the poll loop

connect returns a Response carrying the client end of the WebSocketPair. Three of the six
expectArmed call sites destructured only { stub }, dropping that end unreferenced. workerd is then
free to collect it, which tears down the pair, fires webSocketClose on the DO, and stopIfIdle
deletes the alarm — so getAlarm() reads null for a reason unrelated to the code under test.

In production a real remote peer holds that end for the life of the session, so this is a condition
only the test could create. It never reproduced because it is a GC race: an idle laptop running one
file does not collect.

Re-deriving from source, the assertion can only see null three ways — alarm() early-returning on an
empty socket set, terminate() on a revoked binding (the falsified one), or stopIfIdle. Both
survivors require the socket to be gone
, which turned the question from "why is the alarm null" into
"can the socket vanish first".

What settled it was the socket count added to alarmState last week: every null came with
sockets: 0 before the alarm ran, eliminating everything inside alarm().

Measured

Connecting and firing the alarm under heap churn:

client end runs nulls sockets=0 before alarm
dropped (current code) 200 25 (12.5%) 24
Response retained 300 0 0
retained + accepted 300 0 0

The fix

connect retains the Response, released per-test. That is the one choke point every session passes
through, so a future test that destructures carelessly cannot reintroduce this — it was never one
flaky test, it was a latent class with three exposed call sites.

Two guards keep the retention honest, because otherwise it is an invisible crutch a future edit
deletes silently:

  • deterministic — fails immediately on any machine if the retention is unwired
  • churn sweep — 40 iterations, catches a regression ~99.5% of the time; built so a workerd that
    collects less eagerly makes it greener, never spuriously red

Verified by mutation: removing the retention fails both, the sweep in 18ms.

Test plan

  • pnpm --filter @webhook-co/engine test460 passed / 35 files (was 458; +2 guards)
  • pnpm lint997 pass / 0 fail
  • typecheck 33/33 via the commit hook
  • mutation check: retention removed → 2 failed / 33 passed; restored → 35 passed

Caveat

This is a reproduction of a mechanism that produces exactly that message and symptom, not a recording
of the original CI run. If the same assertion fails again with sockets: 1, that is a genuinely
different bug in the poll loop — and the message now says so.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BRmGUnxeYsQoG9c8BCZcae

…ocket

The "re-arms the alarm" assertion failed once in CI with `expected null not to
be null` and never reproduced in 11 local runs. Reproduced and fixed: the cause
is the harness, not the poll loop.

`connect` returns a Response carrying the CLIENT end of the WebSocketPair. Three
of the six `expectArmed` call sites destructured only `{ stub }`, dropping that
end unreferenced. workerd is then free to collect it, which tears down the pair,
fires `webSocketClose` on the DO, and `stopIfIdle` deletes the alarm — so
`getAlarm()` reads null for a reason unrelated to the code under test. In
production a real remote peer holds that end for the life of the session, so
this is a condition only the test could create.

It never reproduced because it is a GC race and an idle laptop does not collect.
Under heap churn, connecting and firing the alarm:

    client end dropped   → 25 nulls in 200 runs (12.5%)
    Response retained    →  0 nulls in 300 runs
    retained + accepted  →  0 nulls in 300 runs

What identified it was the socket count added to `alarmState` last week: every
null came with `sockets: 0` BEFORE the alarm ran, which eliminated everything
inside `alarm()` and pointed at the socket vanishing beforehand.

`connect` now retains the Response, released per-test — the one choke point every
session passes through, so a future test that destructures carelessly cannot
reintroduce this. Two guards keep the retention honest: a deterministic one that
fails immediately on any machine if it is unwired, and a churn sweep that fails
~99.5% of the time if it stops working. Verified by removing the retention: both
fail, the sweep in 18ms. The sweep is built so a workerd that collects less
eagerly makes it greener, never spuriously red.

Engine suite: 460 passed / 35 files (was 458; +2 guards).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRmGUnxeYsQoG9c8BCZcae
@choraria
choraria merged commit fc3c077 into main Aug 1, 2026
32 checks passed
@choraria
choraria deleted the fix/listen-session-alarm-flake branch August 1, 2026 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant