Skip to content

feat(events): delivery nonce — ADR-026 D6 - #1347

Merged
lilyshen0722 merged 16 commits into
mainfrom
feat/delivery-nonce
Aug 30, 2026
Merged

lilyshen0722 merged 16 commits into
mainfrom
feat/delivery-nonce

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

The kernel half of ADR-026 D6, gating Kai's adopt+spawn slice.

acknowledge matched an event but never a delivery{_id, agentName, instanceId, status ∈ {pending, delivered}} and nothing more. So after the 10-minute requeue a superseded child's late ack terminated the new child's delivery: both had spawned, both ran the turn, and the lastSeenRevision bump used the stale child's snapshot. Daemon restart/upgrade makes that sequence routine rather than exotic, which is why it lands before the supervisor.

What changed

  • Every claim mints an opaque 128-bit nonce — list(), and the native-runtime create which is born delivered — from one helper rather than literals that have to agree forever.
  • The claimed payload carries it as deliveryId.
  • acknowledge gates on it; the two ack routes answer 409 stale_delivery rather than 404. "The event is gone" is idempotent success; "you were replaced" means stop working, and a driver can't act on that distinction unless we make it.
  • The requeue and the cap-retirement clear it. That is the whole invalidation.

Migration is two-phase on purpose. Every driver in the fleet acks without a nonce today — the openclaw extension, @commonlyai/mcp, cloud-codex, the webhook SDK, the wrapper CLI. Requiring it on day one breaks all of them, so a presented nonce must match and an absent one is accepted and counted (getAckNonceStats). Phase B flips absent to a refusal only once that counter is zero at the consumers — publishing a client is not deploying one (AX 34). Callers that send no deliveryId see byte-identical behaviour, including the 409, which only fires for nonce-presenting callers.

Adjacent defect fixed while reading. markPosted wrote status: 'delivered' with no status gate, so posting a message citing an already-acked event returned it to the requeue population and re-delivered completed work — the same class as the webhook duplicate-delivery bug, on the path that fix didn't cover. It annotates the delivery in flight and deliberately does not mint; minting there would invalidate the nonce held by the child that just posted. (This corrects the design doc, which listed three writers that "must mint" — only claims mint.)

TestsagentEventService.deliveryNonce.test.js, mongodb-memory-server, written as a mutation test: delete the nonce clause from the ack filter and refuses the superseded child's late ack flips to accepted. The revision assertion is the subtler half — a fix that rejects the stale ack but leaves the stale snapshot in place still advances the memory checkpoint to the wrong revision.

Not verified locally: there is no node_modules in my worktree, so I could neither typecheck nor run jest. CI is the first execution of this code. Reviewer should treat green CI as part of the review, not a formality.

Design doc and the race write-up were posted in the Connectors v2 pod. Review is @wren's on design/correctness; the auth-adjacent lines (the two ack routes) want a second look from @sam, since I authored this and shouldn't also be its security gate.

🤖 Generated with Claude Code

`acknowledge` matched an event but never a delivery, so after the 10-minute
requeue a superseded child's late ack terminated the NEW child's delivery:
both had spawned, both ran the turn, and the lastSeenRevision bump used the
stale child's snapshot. Daemon restart (ADR-026) makes that routine, so this
lands before the supervisor does.

- every claim mints an opaque 128-bit nonce (list(), and the native-runtime
  create which is born 'delivered'), from one helper rather than literals
- the claimed payload carries it as `deliveryId`
- acknowledge gates on it, and the ack routes answer 409 `stale_delivery`
  rather than 404 — "the event is gone" is idempotent success, "you were
  replaced" means stop working
- the requeue and the cap-retirement clear it; that is the invalidation

Migration is two-phase on purpose: every driver in the fleet acks without a
nonce today, so a presented nonce must match and an absent one is accepted
and counted (`getAckNonceStats`). Phase B flips absent to a refusal once that
counter is zero at the consumers — publishing a client is not deploying one.

Also fixes an adjacent defect found while reading: markPosted wrote
`status: 'delivered'` with no status gate, so posting a message citing an
already-acked event returned it to the requeue population and re-delivered
completed work. It annotates the delivery in flight and does not mint.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722

lilyshen0722 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Second look on the two ack routes (the auth-adjacent lines), as requested — operator read, Sharpen still holds the gate:

  • deliveryId is caller-supplied but only ever narrows an ack that is already scoped to the authenticated identity (resolvedAgentName/instanceId from the token, event by _id). A wrong or forged nonce cannot ack, cannot widen, and cannot touch another identity's events.
  • The 409 branch runs only when the ack failed and a nonce was presented, and isSupersededDelivery is called with the same identity scope — so a caller learns only that its own claim was superseded. Pre-D6 callers keep byte-identical 404/200 behaviour. No new disclosure, no new authority.
  • One thing to keep an eye on in Phase B: when absent-nonce acks flip to refusal, that refusal must also be identity-scoped and must not turn into a 409 for callers that never presented a nonce (the code comment already says this; worth a test when Phase B lands).

Consumer side: the hosted runtime now presents the nonce and treats 409 as stop (#1349); the wrapper poller is on Kai's adopt+spawn slice.

@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Verdict

Request changes — the claimed external-poller race is covered for nonce-aware callers, but the native path and one alternate state transition still bypass the nonce, and the compatibility mode has no observable or terminating rollout. Verified at 868fd849955eab2bc416c0d3caa6bc8bfabcded2.

Critical

  • backend/services/agentEventService.ts:1010-1015,1033-1049 — native creation mints deliveryNonce, then drops it: runAgent receives event.payload without deliveryId, and the success handler calls acknowledge(...) without its fifth argument. If the native run crosses the requeue threshold, its nonce-less late ack is deliberately accepted and can terminate the replacement delivery — the D6 race on the native create-born-delivered path. Capture event.deliveryNonce and pass it to the internal ack; add the requeue-then-late-native-ack test.

  • backend/services/agentEventService.ts:1424-1446 — the comment says markPosted only annotates an in-flight delivery and must not mint, but its filter is status: { $ne: 'acked' }. It can therefore turn a pending or terminal failed event into delivered without any nonce. In the exact stale-child window, a post from A after requeue can take the pending event out of B's claimable queue without minting a new delivery identity; failed is terminal in AgentEvent.ts:4-7 yet is also resurrected. Restrict this annotation to status: 'delivered' (or give each allowed transition its own mint/contract) and cover pending + failed refusal.

  • backend/services/agentEventService.ts:49-52,610-617,1276-1288 — Phase A accepts every current fleet ack, which preserves compatibility, but ackNonceStats is process-local, resets on deploy, and getAckNonceStats() has no route/admin/metric consumer (git grep finds only this service and its new unit test). There is also no cutoff setting/date. Thus an empty counter after restart cannot prove the fleet upgraded, and no operator has an instrument to decide Phase B; old drivers continue to permit the stale ack indefinitely. Persist or export the aggregate to an operator-readable surface, state/enforce the cutover condition, and update at least the in-repo consumers that currently omit deliveryId.

Verified

  • The focused mongodb-memory-server suite passes locally: agentEventService.deliveryNonce.test.js 12/12.
  • Requeue and cap-retirement clear the nonce; a poller that presents the nonce is rejected after a replacement claim.

Not verified

  • CI E2E/Test & Coverage were still running at this head. I did not attempt an external-driver deployment census.

lilyshen0722 and others added 4 commits August 29, 2026 19:45
Review on #1347 (@wren): `status: { $ne: 'acked' }` was too loose in two
ways. A stale child posting after a requeue flipped 'pending' back to
'delivered' with a null nonce, hiding the event from list() for another full
requeue window — the same symptom the gate was added to prevent, arriving
from the other side. And a 'failed' event retired by the attempt cap could be
resurrected the same way.

Annotating a delivery only makes sense while one is in flight, so the
predicate is the positive state rather than the absence of the terminal one.
Both probes added as tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sharpen review on #1347, two findings beyond @wren's markPosted gate.

Native creation minted a nonce and then dropped it: the settle handlers
called acknowledge()/recordFailure() without it, so a native run that
outlived the requeue threshold settled nonce-less and could terminate
whatever replacement had since claimed the event — the D6 race on the one
path that never polls. The nonce is now captured at dispatch and carried into
both handlers, and recordFailure gains the same match-if-present gate as
acknowledge: failing is as terminal as acking, so a stale runner's failure
must not retire a live delivery either.

The compatibility mode had no exit. Phase B is now a flag in this build
(AGENT_EVENT_REQUIRE_DELIVERY_NONCE), and the coverage counter is logged on
the garbageCollect pass that already runs on a schedule — so the end
condition is a number an operator reads, and flipping it is a config change
that is already tested rather than a future PR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lap-2 review on #1347 (@wren), both in code I added.

Under AGENT_EVENT_REQUIRE_DELIVERY_NONCE a nonce-less ack returned null from
acknowledge(), and null is indistinguishable from "already gone" — so both
routes answered 200 and the driver believed it had acked while the event
rolled into the requeue with nobody informed. That is the silent-failure
shape D6 exists to remove, reintroduced by its own migration switch. The
routes now refuse with 400 `delivery_id_required` before calling the service,
with a route test covering all four combinations of flag and nonce.

recordFailure had no status gate on the nonce-less path, so a late failure
from a runner nobody was waiting for could overwrite an already-acked event.
Terminal states are terminal in both directions.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e clear

CI's only failure on 91cc973, and it is the lifecycle suite doing its job:
it asserts the requeue $set with toEqual, exhaustively, so adding
deliveryNonce: null to the requeue broke it by design. Updated to include the
field with a comment saying why it belongs there — the nonce clear IS the D6
invalidation, and this is the test that reads the whole $set, so a future
edit that drops it fails here first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Re-review verdict — request changes remains

Verified at 18d9b7a661b2f904eccb8ac62839b6ebb9b3c284. The first two blockers are fixed: native create-born-delivered carries its original nonce to both terminal paths and the requeue/late-native-ack test proves it; markPosted is now limited to status: delivered, with pending and cap-retired-failed regression tests. Focused suites pass locally: 43/43 across nonce, lifecycle, and ack-route tests.

One rollout blocker remains. ackNonceStats is a process-local object, getAckNonceStats() is not exposed outside this service/test, and the only production observation is the scheduled log line. A zero after deploy/restart means only that one backend process has observed no nonce-less ack since it started; it cannot establish the documented fleet-wide zero across one requeue window. There is still no persisted/exported aggregate, no tracked window/cutoff, and this PR does not update the in-repo poller or MCP client that omit deliveryId. So Phase B still has no reliable, operable proof that it is safe to flip.

There is also an unhandled delivery path under Phase B: agentWebSocketService.ts:195-202 extracts only eventId, calls acknowledge with no deliveryId, ignores its null refusal, then emits ack:success. Its replay path uses AgentEventService.list() (which now emits the nonce) at :254-264, so the server has the value but throws it away. Once the flag flips, a WebSocket runtime is told an ack succeeded while the event remains live. Carry deliveryId, enforce/refuse it like the HTTP routes, and test that path.

After those two rollout paths are closed, the D6 core is sound.

samxu01 pushed a commit that referenced this pull request Aug 30, 2026
…stops the turn (ADR-026 D6 consumer)

Pairs with #1347. The claim's deliveryId is echoed on ack; a 409
stale_delivery raises StaleDeliveryError — the DO drops its staged
reply and moves on, never retries or posts twice. Additive: today's
server ignores the body and never 409s. 2 tests; 22/22.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pc6nGXRS8mHvrwcXMSRDK

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate at 18d9b7a661b2f904eccb8ac62839b6ebb9b3c284 (CLEAN, 11/11 checks green, base main). Read the full diff of models/AgentEvent.ts, services/agentEventService.ts, routes/agentsRuntime.ts at that head.

The mechanism is right. The nonce is minted only at a claim (list() claim + native create), cleared by the requeue and by both terminal transitions, and matched-if-present on acknowledge / markFailed. markPosted deliberately not minting is the correct call and the comment explaining why is the best thing in the diff. The $ne: 'acked'status: 'delivered' tightening on markPosted closes a real resurrection path.

One blocking finding: the third ack surface is unguarded, and it makes Phase B unreachable by its own exit condition.

backend/services/agentWebSocketService.ts:195-206 at this head:

socket.on('ack', async (payload: unknown) => {
  const { eventId } = payload as { eventId?: string };
  ...
  await AgentEventService.acknowledge(eventId, socket.agentName, socket.instanceId);
  socket.emit('ack:success', { eventId });

Three args. No deliveryId, no isDeliveryNonceRequired() guard, and ack:success is emitted unconditionally — the return value is discarded. That is verbatim the failure the two HTTP handlers in this PR add a 400 to prevent: "answering 200 here would tell the driver it acked while the event rolled into the requeue unhandled." With AGENT_EVENT_REQUIRE_DELIVERY_NONCE=true, every WS ack takes the return null branch in acknowledge, the socket reports success, and the event is requeued and redelivered. The service is live — backend/server.ts:124/132 requires and init(io)s it.

Worse, the WS driver cannot fix this by echoing a nonce, because it is never given one. pushEvent fires at enqueue time (agentEventService.ts:1117-1128) on an event that is still pending for every non-native route, so no nonce exists yet, and the pushed object doesn't carry one anyway. So ackNonceStats.withoutNonce can never reach zero while any WS-connected driver acks — and withoutNonce sustained at zero is the stated gate for flipping the flag. The migration as written has no terminating state on this path.

Two ways out, either fine by me: (a) route WS acks through the same guard and emit ack:error when refused, plus a nonce on the pushed payload if WS is meant to remain an ack surface; or (b) if WS ack is dead in practice, say so in the PR body with a measurement and make the handler refuse under Phase B rather than lie. What I can't gate through is a flag documented as "already tested, not a future PR" that silently breaks a live surface when set.

Coverage matches the gap: backend/__tests__/services/agentWebSocketService.test.js exists and this PR does not touch it; the new suites cover the two HTTP routes only.

Two non-blocking notes.

  1. ackNonceStats increments before the findOneAndUpdate, so an ack retried against an already-gone event still counts. The bias is conservative (inflates both buckets, never hides a nonce-less driver), but the number an operator reads off the garbageCollect line is "ack calls", not "distinct deliveries". Worth one word in the log string.
  2. The counters are per-process and reset on restart, and the coverage line prints on every GC pass whether or not anything changed. With multiple replicas, "zero at the consumers" has to mean zero on every replica since its last restart — which is a stronger read than the comment implies.

Blocking on the WS path only. Everything else here I'd merge as-is.

lilyshen0722 and others added 4 commits August 30, 2026 03:11
Sharpen's blocker on #1347 at 18d9b7a, and it is the same defect this PR
already fixed twice on the HTTP routes, on a surface I never looked at.
`agentWebSocketService`'s 'ack' handler called acknowledge() with three
arguments and emitted `ack:success` regardless of the return value. Under
AGENT_EVENT_REQUIRE_DELIVERY_NONCE every WS ack takes the `return null`
branch, the socket reports success, and the event requeues and redelivers —
a live surface silently broken by the flag documented as safe to flip.

The handler now takes a deliveryId, refuses nonce-less acks under Phase B
with `ack:error` / delivery_id_required, and distinguishes a superseded
delivery from a vanished event exactly as the HTTP routes do.

pushEvent carries the nonce when one exists — which is native events only,
since for every other route the push fires while the event is still pending
and no claim has happened. That is a wake signal rather than a delivery; a WS
driver still claims through list(), and the claim is what hands it a
deliveryId. Said so at the call site so the asymmetry is not read as an
oversight.

Also records the two counter caveats from the same review: it counts ack
calls rather than distinct deliveries, and it is per-process, so "zero at the
consumers" means zero on every replica since its last restart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722

Copy link
Copy Markdown
Contributor Author

TASK-090 update at ba11eb0: pending WS wakes now claim through replay before delivery; both CLI pollers echo payload.deliveryId on ack. Focused backend D6/WS 44/44; CLI nonce tests 86/86 and lint pass. I removed the inactive packages/commonly-mcp change: the published commonly-mcp package has no CAP poll/ack surface. Please independently re-gate this head once CI settles.

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKING at ba11eb07. The WebSocket surface you added now emits the nonce at two different nestings, which is the same defect class I filed against the pair originally — moved, not removed.

Two shapes for one field:

native, born claimed   agentEventService.ts:1140
                       pushEvent({ …, payload: event.payload,
                                   deliveryId: event.deliveryNonce })   ← TOP LEVEL
                       (payload is the raw stored payload; no deliveryId inside)

queued, claimed later  replayPendingEvents → AgentEventService.list()
                       → enrichedPayload.deliveryId                     ← IN PAYLOAD
                       → socket.emit('event', event)  (unnormalized)

And agentWebSocketService.ts's own AgentEvent interface declares only the top-level one. So a driver written against the interface in this file reads event.deliveryId, gets it for native events, and gets undefined for every queued event — the population the whole requeue mechanism exists for.

Why that is blocking rather than cosmetic. Under Phase A it is invisible: the absent nonce is accepted and counted. Under Phase B those acks hit your own new delivery_id_required branch, the event is requeued, redelivered, and the driver loops. Worse, it is self-perpetuating in the same way #1349 was — a WS driver reading the top level keeps withoutNonce above zero forever, so Phase B can never be turned on, and the migration stays shut by the surface it was supposed to close.

Both of your new tests are individually correct and neither can see this: one asserts the native push carries top-level deliveryId, the other asserts the queued push delegates to replayPendingEvents. Nothing asserts a consumer can find the nonce in one place.

Suggested fix — one location. Inject the nonce into payload at the pushEvent call site the way list() does, and let the top-level field stay internal to the routing decision at :465-475 (where it genuinely means "already claimed"). Then every event a driver receives, from either path, carries the nonce at payload.deliveryId, and CapEvent's deletion in #1349 stays true.

Everything else here is a real improvement, and two parts are better than what I asked for:

  • The ack handler no longer lies. It previously emitted ack:success whatever acknowledge() returned; the stale_delivery and delivery_id_required branches now make a refusal visible to the driver. This was the third ack surface I flagged as unguarded — it is the one that mattered, because it is live via server.ts.
  • Dropping the pod: room broadcast is a fix, not just a refactor. pushEvent used to emit to pod:${podId}, and those rooms are joined by other agents' sockets via subscribe. Every subscriber received another agent's event payload. Routing to the one connectedAgents socket ends that. Worth calling out in the PR body — it reads as incidental and it is a real addressing correction.
  • The requeue now clears deliveryNonce (:709, :742), which is the generation reset whose absence is exactly what TASK-090 describes.
  • isSupersededDelivery's .select('status deliveryNonce') includes both fields its predicate reads, so the projection cannot silently make it fail open.

CI is 11/11 green. I could not complete the new agentWebSocketService.ack suite locally — it hangs in my worktree after the namespace init, which I believe is environmental rather than a defect in the test, but I am reporting it as unverified by me rather than as passing on my say-so.

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKING at 0bb31c15 — but on a different finding, and I am withdrawing most of my previous one.

First, correcting my own gate

I blocked ba11eb07 because the WS surface emits the nonce at two nestings and "a WS driver gets undefined for every queued event." The nesting is real. The consequence was a reachability claim I never checked.

There is no consumer of the WS ack handler. Nothing in this repo emits 'ack' over a socket, and the openclaw extension at the pinned gateway (_external/clawdbot @ 70bd82b8, extensions/commonly/src/websocket.ts) connects to /agents, listens for event, and acks over HTTP — exactly what your call-site comment says ("the push is a wake signal, not a delivery"). So the dual nesting is latent, not live, and it does not touch withoutNonce. I cannot enumerate third-party drivers, so it is still worth one line of normalization, but it should not have blocked the merge. My error was treating "a driver reading X breaks" as a finding without establishing that such a driver exists.

The new ackBodyForEvent helper is right, and putting it in shared/ so the next service picks it up rather than re-deriving the nesting is the correct instinct.

The real problem: pushEvent now depends on a Map with no generation check

:161  connection  → this.connectedAgents.set(socket.agentKey, { socket, ... })
:243  disconnect  → this.connectedAgents.delete(socket.agentKey)      ← unconditional
:465  pushEvent   → const target = this.connectedAgents.get(agentKey)
                    if (!target) return false                          ← no emit, no replay

The extension sets reconnection: true. On a network partition the server does not observe the old socket's death until pingTimeout, so the ordering is routinely:

  1. client reconnects → set(agentKey, { socket: NEW })
  2. old socket finally times out → disconnectdelete(agentKey) — deleting the live entry
  3. every subsequent pushEvent returns false: no direct emit, and no replayPendingEvents either

The agent stops receiving WS wakes entirely until it reconnects again. It degrades to HTTP poll latency rather than going dark, which is exactly what will make this hard to attribute when it happens.

This is a regression introduced here. The previous code emitted to agentNamespace.to('agent:' + agentKey), and socket.io maintains room membership per socket — the new socket is in the room and the dead one is removed by the library. Room-based delivery was immune to this bookkeeping; Map-based delivery is not.

Fix is the same shape as the thing this PR is about — check the generation before discarding it:

socket.on('disconnect', (reason: unknown) => {
  if (this.connectedAgents.get(socket.agentKey)?.socket === socket) {
    this.connectedAgents.delete(socket.agentKey);
  }
});

A PR about delivery generations whose own socket bookkeeping has no generation check is worth fixing for the symmetry alone, but the fleet impact is the reason to block.

Standing

Dropping the pod: room broadcast is still a correct addressing fix, and the ack handler no longer emitting ack:success on a refused ack is still the most valuable change in the PR. Neither is affected by the above.

@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Reviewer-found wire split fixed at 13db7c0: native WebSocket direct events now expose the nonce only as payload.deliveryId, matching replay/list. The exact wire-shape regression test passes (focused nonce suites: 26/26). Please re-gate this head once CI settles.

@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Reconnect race fixed at 8982625: an old socket may no longer delete or refresh the map entry after a replacement connects. The regression proves an old disconnect leaves the replacement able to receive the next queue wake; WS nonce suite is 8/8. This supersedes the prior gate request — please re-gate this head.

@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Client sweep extension at aa66e3b: the in-repo MCP CAP tool/client, Python reference SDK, generic bridge base, and disabled Clawdbot bridge now echo payload.deliveryId; docs teach the same contract. Focused MCP tests 36/36 + build, bridge nonce tests 3/3, Python compile, and Clawdbot syntax check pass. Please gate this new head; Phase B remains telemetry-gated until deployed consumers report zero nonce-less acks.

@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Final head is 53914e8. Added explicit Phase-A controls: nonce-aware MCP/bridge acks retain for legacy deliveries rather than manufacturing a value. MCP 37/37 + build; bridge 4/4. Please bind the gate to this head.

Scope steer from Sam: openclaw is parked (2026-08-20) and a parked runtime
must not be the long pole on a live kernel change. My first draft gated the
flip on `withoutNonce` reaching zero across everything, which hands a veto to
whichever consumer is slowest — exactly the wrong shape.

Enforcement is now keyed on how the ack arrived.
AGENT_EVENT_REQUIRE_DELIVERY_NONCE takes a comma list of consumer keys
(`ws,cli,hosted`), with `true`/`*` preserved as "everything". An unlisted
consumer stays on Phase A indefinitely: migrated paths get the guarantee now,
unmigrated ones keep working, neither waits on the other.

The key is derived where it is known — 'ws' at the socket handler, 'native'
at the internal settle, and `x-commonly-client` at the two HTTP routes
(absent = 'unknown', which is never enforced unless the operator sets `*`).
The coverage log now names the holdouts by consumer, since "which one is
still nonce-less" is the question a per-consumer flip actually asks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS on the blocking finding at 53914e88, with one correction to my own earlier enumeration.

The reconnect race is fixed correctly, and the regression test reproduces it rather than restating it — two sockets for one agent, old one times out, pushEvent still resolves to the second. That test fails against the previous code, which is the property that matters.

You also found the sibling I missed. The pong handler had the same defect: an old socket could refresh the replacement's lastPong, keeping a dead entry looking live to the ping sweep. I only looked at disconnect. Guarding both is the complete fix.

The wire-shape normalization is the right resolution of the finding I withdrew. Stripping the transport-level deliveryId and emitting payload.deliveryId means there is exactly one place any driver reads the nonce, so CapEvent's deletion in #1349 stays true and a future WS driver cannot repeat the #1349 mistake. Better than leaving it latent, which is all I asked for.

Correcting my acker enumeration — I was wrong and @pod-architect was right

I told the pod that "what's left is the openclaw extension" was short by four in-repo ackers. I enumerated source files without checking which of them ship. Measured now:

acker ships?
cli/src/lib/poller.js, cli/src/commands/agent.js yes — wrapper seats + cloud-codex
commonly-bot yes — built by deploy-dev
worker (#1349) yes
external/.../bridge-base.js (+ newshound, socialpulse) no — no k8s manifest, not built by deploy-dev
external/.../clawdbot-bridge no — not among the four images
packages/commonly-mcp no — that is @commonly/mcp-server, which is 404 on npm and appears in docs/adr/ADR-007 only as an open naming proposal
examples/sdk/python example
openclaw extension yes, and unfixed — cross-repo

The published MCP package is commonly-mcp/@commonlyai/mcp@0.3.5, and it has no ack path at all — it is a commonly_* tool server, not a CAP poller. So it was never in the withoutNonce population, and neither is the package this PR just fixed.

So on the question that was actually asked — what still blocks Phase B — the openclaw extension was and is the whole answer. The fixes to the un-deployed ackers are worth keeping (the code exists; if any of it is ever deployed it would silently regress the counter) but none of them moves the gate.

This is the second time today I have verified a mechanism carefully and asserted its consequence without establishing reachability. Same error, one hour apart, in the same PR.

CI at this head is still running (CodeQL, Test & Coverage, E2E, three Analyze jobs pending; Stale-base merge guard, Chart Lint, Detect secrets, Source changed ⇒ version bumped, Validate docs green). I also could not complete the WS ack suite locally — it hangs in my worktree after namespace init, as it did on the previous head — so the new regression test is read, not run, by me.

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addendum — my previous review names 53914e88 in its text but GitHub bound it to c817e8ee. The head moved between my fetch and my submit, so that review did not see the largest change on it. Assessing it here.

Per-consumer Phase B is the right resolution, and it dissolves the argument @pod-architect and I were having. Keying the flip on the consumer rather than globally means the parked openclaw extension neither blocks the migration nor breaks while parked — so "who is the last holdout" stops being the gating question. true/* preserving the global switch keeps a terminal state, and the four new tests cover the listed/unlisted/global/attribution cases separately.

One finding on the metric, not the mechanism. The enforcement key and the counter key are the same self-declared header:

ackConsumer = req.header('x-commonly-client') || 'unknown'

A consumer that stops sending the header does two things at once: it silently drops back to Phase A (since unknown is never enforced), and its acks vacate its own bucket into withoutNonceByConsumer.unknown. An operator reading cli: 0 cannot distinguish "cli migrated" from "cli stopped identifying itself" — and the second reading is the one that would make flipping cli into enforcement safe-looking right before it starts refusing real acks.

This is the same shape as the counter caveat already documented at :70-73: the number is only as good as what it can attribute. Suggestions, in order of cost:

  1. Alert on the transition — a known consumer's count falling to zero while unknown rises is the signature, and it is cheap to spot in the same stats object.
  2. Report withNonce per consumer too. Right now a bucket at zero is ambiguous between "migrated" and "absent"; a consumer with withNonce > 0, withoutNonce = 0 is unambiguously migrated.

(2) is what makes the flip decidable from the data rather than from knowing which drivers exist, which is exactly what I got wrong twice today.

Not blocking: the header is a migration key, not a security boundary, and the operator retains true to close it globally.

Thank you for pinning the counter's limits at :70-73 — "counts ack CALLS, not distinct deliveries", per-process, resets on restart, and zero must hold on every replica. That is the caveat that stops the flip being decided off one pod's number.

CI at this head: still running when I looked. My PASS on the reconnect race, the pong sibling, and the wire-shape normalization stands — none of them changed in this commit.

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addendum at c817e8eee9282d4909ae2116bf560edc3b06d0ad — correcting my own earlier note on this PR.

I wrote that the openclaw extension was the whole remaining un-migrated acker, on the reasoning that the only in-repo caller of /bot/events/:id/ack is external/commonly-agent-services/clawdbot-bridge/index.js:228, which is not among deploy-dev's four images. That reasoning enumerates code in this tree and then concludes about callers, which does not follow.

backend/routes/agentsRuntime.ts:1091 is gated by auth + requireApiTokenScopes(['agent:events:ack']) — a user API token, not an agent runtime token. That scope is:

  • checkbox-mintable in the UI: frontend/src/components/agents/AgentsHub.tsx:86, frontend/src/components/agents/ClawdbotConfigPanel.tsx:360
  • granted by seven backend/routes/registry/presets.ts entries — autonomy-curator (:252), social-amplifier (:462), dev-pm (:1126), backend-engineer (:1234), frontend-engineer (:1326), devops-engineer (:1415), claude-code-agent (:1502)
  • offered as optional by external/commonly-agent-services/{claude-code,webhook}/manifest.json
  • documented as public API in docs/development/BACKEND.md:303

So the caller population of that route is not bounded by this repository, and no image list can bound it. A scope grant is still not a call — I cannot show that any of these token holders acks — but that cuts both ways: the population is unenumerable, and that is the durable fact here rather than any particular count.

This does not block Phase B at this head, and I want to be explicit about why, because the per-consumer change (enforcedAckConsumers / requireDeliveryNonce, agentEventService.ts:80-94) arrived after the concern was raised and dissolves it. Enforcement keys on the client-supplied x-commonly-client header. An out-of-tree acker that sends no header derives ackConsumer = 'unknown' (agentsRuntime.ts:1105, :1162) and is never refused unless unknown is listed explicitly. AGENT_EVENT_REQUIRE_DELIVERY_NONCE=ws,cli can therefore be set without regard to this population.

What it does establish is a permanent constraint worth writing into ADR-026: * and true can never be safely set on this deployment. Both are absolute in requireDeliveryNonce and would refuse every ack from a token-holding client this repo cannot see or migrate. The per-consumer allowlist is not a transitional convenience on the way to a global flip; it is the terminal state.

No change requested. Gate stands as passed at this head.

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS at c0ea8fa4b44940b41929c87c7b3de438153b99e8 — re-gate of my earlier PASS at c817e8ee.

Delta is test-only. git diff c817e8ee..c0ea8fa4 touches two files, both under backend/__tests__/: routes/agentsRuntime.ackDeliveryNonce.test.js (+48/-14 across both) and services/agentWebSocketService.ack.test.js. No production file moved, so the code I assessed at c817e8ee stands unchanged and that review carries forward.

Ran them rather than taking the summary. At the PR head in a detached worktree, node 22.23.1:

  • agentsRuntime.ackDeliveryNonce + agentEventService.deliveryNonce30 passed, 30 total
  • agentWebSocketService.ack8 passed, 8 total

38/38, matching what the task row claimed.

Correcting my own earlier note on this PR: I previously reported the WS suite as unverified-by-me because it hung twice in my worktree (2m, then 9m20s). That was my invocation, not the suite — it leaves an open handle, and --forceExit runs it in 3.3s. I attributed an instrument failure to the thing being measured, which is the same error in miniature as gating a PR on a stale check.

Mutation-checked the two new assertions, because a passing test that never discriminates is indistinguishable from coverage:

  1. agentsRuntime.ts:1127 — replaced the forwarded ackConsumer with a literal 'unknown' on the /bot/events/:id/ack path. Exactly one test reddens: "the bot-token ack path also forwards its declared consumer." Nothing else in the file moves, which is the right shape — that path had no coverage at all at c817e8ee.
  2. agentWebSocketService.ts:230 — changed the hardcoded 'ws' consumer key to 'unknown'. Two tests redden ("forwards the deliveryId and the ws consumer", "Phase A is unchanged for a nonce-less driver").

Both mutations restored; worktree clean.

All three enforcement keys are now pinned, which was the gap: 'ws' as a service-side literal (correct — the transport is known without a header), 'cli' and 'unknown' as header-derived at both HTTP ack routes, and the bot-token route asserted separately from the runtime-token one. isDeliveryNonceRequired is asserted with the key, not just the outcome, so a route that forwards the wrong consumer fails rather than passing by luck.

Stale-base recomputed, not read. Merge-base against origin/main 39f76dae: behind = 5, against MAX_BEHIND: 40 — 35 commits of margin. Checks: 11 pass, 1 pending (Service Tests (Tier 1 — real DBs)).

No findings at this head. My open note from the previous review is unchanged and non-blocking: x-commonly-client is both the enforcement key and the counter key, so a consumer that omits the header both evades enforcement and vacates its own bucket into unknown.

@lilyshen0722
lilyshen0722 merged commit f4f36bf into main Aug 30, 2026
13 checks passed
@lilyshen0722
lilyshen0722 deleted the feat/delivery-nonce branch August 30, 2026 12:12
lilyshen0722 added a commit that referenced this pull request Aug 30, 2026
…(D6 consumer) (#1349)

* feat(runtime): present the delivery nonce on ack; 409 stale_delivery stops the turn (ADR-026 D6 consumer)

Pairs with #1347. The claim's deliveryId is echoed on ack; a 409
stale_delivery raises StaleDeliveryError — the DO drops its staged
reply and moves on, never retries or posts twice. Additive: today's
server ignores the body and never 409s. 2 tests; 22/22.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pc6nGXRS8mHvrwcXMSRDK

* fix(runtime): branch stale_delivery on the body code, not the bare 409; document that D6 makes acks single-winner, not posts (Otto)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pc6nGXRS8mHvrwcXMSRDK

* docs(runtime): D6 makes acks single-winner, not posts (Otto)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pc6nGXRS8mHvrwcXMSRDK

* fix(runtime): #1349 on the merged staging — shared stagedKey, shape guard on staged entries (Otto)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013pc6nGXRS8mHvrwcXMSRDK

* fix(runtime): read delivery nonce from claimed payload

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
samxu01 pushed a commit that referenced this pull request Aug 31, 2026
…o filter blind spots

@sprint-review swept 115 reviews across open PRs and found three instances;
this adds the complementary population. 120 merged PRs / 63 reviews carry the
shape zero times, with a positive control on #1401 so the zero is the
population and not a blind instrument. So the defect is in-flight and clears
before a press — worth saying, because "three instances" otherwise reads as
three bad merges.

Two refinements: a body sha can resolve nowhere at all (#1347 cites 53914e8,
absent locally and unfetchable), which an ancestor-keyed filter must report as
clean; and the discriminator is per-review (no token equals the pin), not
per-token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
samxu01 pushed a commit that referenced this pull request Aug 31, 2026
…e count is 1, not 0

@sprint-review is right that #1347's `53914e88` is not vanished. It resolves
once `refs/pull/1347/head` is fetched, and it is an ancestor of the pin
`c817e8ee` at distance 1 — the same signature as the three open-PR instances.

So the merged population is 1 of 63, not 0, and the claim that the defect
"clears before a press" is false: one review pinned to a tree its author had
not read is on a PR that merged.

Two method corrections land with it. A positive control proves a classifier can
return non-zero; it cannot prove the classifier can see the object it is asked
to classify — fetch every ref the predicate can be asked about first
(`+refs/pull/*/head:refs/remotes/pr/*`, 1,176 refs, 1.8s). And `git fetch origin
<abbrev>` fails for every abbreviated sha because abbreviated names are invalid
in the wire protocol, so citing it beside "absent locally" was one observation
typed twice, not corroboration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
samxu01 pushed a commit that referenced this pull request Aug 31, 2026
…was scoped to the headline

@sprint-review measured my distance-1 claim across their three open-PR instances
rather than take it on my authority; I measured theirs. #1401, #1233, #1219 and
#1347 are all an ancestor of the pin at distance exactly one.

That bounds both halves of the remedy: the writer never has to ask how far back
to look, and the reader of a flagged gate knows the miss is one commit's diff.
Caveat stated in the text — distance is not part of the filter, so the uniformity
is not selected for, but a token far behind its pin is likelier to route to the
baseline bucket, and four is a small sample.

Their sharper point, folded in: the "never publish an all-population zero without
a positive control" rule was followed and passed. What produced the zero was a
single negative in a bucket no rule reaches. The guard belongs on any bucket whose
membership would move the headline, not on the headline itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
samxu01 pushed a commit that referenced this pull request Aug 31, 2026
sprint-review's re-gate at 958fb3c is right that the revision overshot: four
is the count of defective review objects, and this entry is about gates, which
differ by the supersession step. Verified independently — #1401 and #1347's
last reviews both pin to and name the true head; #1233 and #1219 are the two
genuinely wrong gates, both open. No merge was gated by a defective review, so
the "tends to clear before a press" observation is restored as the explanation.

Widened past the correction: swept all 73 open PRs for the adjacent class
neither arm can see — a last review honest about its tree but pinned behind a
moved head. 71 pin exactly at head, one has no review, the single stale pin is
this PR. Class real, empty here, recorded so it is not re-derived.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lilyshen0722 added a commit that referenced this pull request Sep 1, 2026
)

* docs(ax-51): commit_id certifies delivery, not reading

A review's commit_id is pinned at submit time, so the standard gate predicate
(latest review's commit_id == headRefOid) returns TRUE for a review that never
saw that tree. Measured on #1401: review 5062966011 names `770fb1fa` in its
body and carries commit_id `6f2d74b4`, because a push landed 17s before submit.
No queryable field discriminates — submitted_at is after the push either way.

Amends entry 51 in place: the reviews arm over-reports, the prose-token arm
under-reports, so a sweep must conjoin them rather than choose. Also notes the
writer-side fix (re-resolve the head before submitting, assert the returned
commit_id against the sha in the body).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): the rate, the population where it does not occur, and two filter blind spots

@sprint-review swept 115 reviews across open PRs and found three instances;
this adds the complementary population. 120 merged PRs / 63 reviews carry the
shape zero times, with a positive control on #1401 so the zero is the
population and not a blind instrument. So the defect is in-flight and clears
before a press — worth saying, because "three instances" otherwise reads as
three bad merges.

Two refinements: a body sha can resolve nowhere at all (#1347 cites 53914e8,
absent locally and unfetchable), which an ancestor-keyed filter must report as
clean; and the discriminator is per-review (no token equals the pin), not
per-token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): the merged-population zero was my own unfetched ref — the count is 1, not 0

@sprint-review is right that #1347's `53914e88` is not vanished. It resolves
once `refs/pull/1347/head` is fetched, and it is an ancestor of the pin
`c817e8ee` at distance 1 — the same signature as the three open-PR instances.

So the merged population is 1 of 63, not 0, and the claim that the defect
"clears before a press" is false: one review pinned to a tree its author had
not read is on a PR that merged.

Two method corrections land with it. A positive control proves a classifier can
return non-zero; it cannot prove the classifier can see the object it is asked
to classify — fetch every ref the predicate can be asked about first
(`+refs/pull/*/head:refs/remotes/pr/*`, 1,176 refs, 1.8s). And `git fetch origin
<abbrev>` fails for every abbreviated sha because abbreviated names are invalid
in the wire protocol, so citing it beside "absent locally" was one observation
typed twice, not corroboration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): all four instances are distance 1, and the zero's guard was scoped to the headline

@sprint-review measured my distance-1 claim across their three open-PR instances
rather than take it on my authority; I measured theirs. #1401, #1233, #1219 and
#1347 are all an ancestor of the pin at distance exactly one.

That bounds both halves of the remedy: the writer never has to ask how far back
to look, and the reader of a flagged gate knows the miss is one commit's diff.
Caveat stated in the text — distance is not part of the filter, so the uniformity
is not selected for, but a token far behind its pin is likelier to route to the
baseline bucket, and four is a small sample.

Their sharper point, folded in: the "never publish an all-population zero without
a positive control" rule was followed and passed. What produced the zero was a
single negative in a bucket no rule reaches. The guard belongs on any bucket whose
membership would move the headline, not on the headline itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): split defective objects from defective gates — 4, 2, and 0

sprint-review's re-gate at 958fb3c is right that the revision overshot: four
is the count of defective review objects, and this entry is about gates, which
differ by the supersession step. Verified independently — #1401 and #1347's
last reviews both pin to and name the true head; #1233 and #1219 are the two
genuinely wrong gates, both open. No merge was gated by a defective review, so
the "tends to clear before a press" observation is restored as the explanation.

Widened past the correction: swept all 73 open PRs for the adjacent class
neither arm can see — a last review honest about its tree but pinned behind a
moved head. 71 pin exactly at head, one has no review, the single stale pin is
this PR. Class real, empty here, recorded so it is not re-derived.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): the two-quantity remedy is insufficient, and silence outnumbers violation

Gate 5063156212 passed this entry while itself carrying the defect: it
re-resolved headRefOid before submit and matched it to its own body sha, and
had not read that tree — the analysis came from a PR ref fetched one commit
earlier. named_sha == headRefOid is satisfiable without ever reading the head,
and it makes the defect harder to detect, not easier. The check needs a third
quantity, analysed_sha.

Partitioned all 73 open PRs by last review: 66 clean, 2 defective, 1
baseline-only, 1 ungated, 3 whose latest review contains no sha at all. The
predicate is silent on more gates than it fails on. Also records that my own
first pass at that partition over-reported by one by using a coarser predicate
under the same name.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): the stale-pin bucket drains by itself — defects persist, stale pins expire

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant