perf(think): broadcast transcript deltas at turn boundaries - #2235
perf(think): broadcast transcript deltas at turn boundaries#2235mattzcarey wants to merge 1 commit into
Conversation
Think's _broadcastMessages() serialised the whole cached transcript (up to the 32 MiB hydration budget) to every socket at least twice per turn: after the incoming persist and after the assistant cutover. Add a cf_agent_chat_messages_delta frame to the shared chat protocol that carries only the messages a turn boundary persisted, plus the epoch of the snapshot it applies to. cf_agent_chat_messages now carries that epoch too. The client (useAgentChat) upserts deltas by id (append for unknown ids) and drops any delta whose epoch differs from the last snapshot it applied, so a delta can never land on the wrong base. Server side, the cache epoch bumps whenever the live cache is wholesale replaced (hydration, branch/regeneration resync, compaction, clear, repair) and every connection records the epoch of the last snapshot it was sent. A delta goes out only when every connection is aligned; otherwise the full snapshot is sent and re-aligns them. Still full: connect/resume, transcript repair, orphan materialisation, stalled/errored partial re-anchors, the post-overflow-retry broadcast (streamed chunks were discarded; only a snapshot reconciles them), and any boundary where an id is not in the cache. Bench (think tests/delta-broadcasts): a turn on a 40-message transcript costs two ~2 KiB deltas instead of two ~86 KiB snapshots.
|
🟡 agents import sizesMeasured 336 runtime imports as minified bundles. The primary size is gzip; raw minified size is included for diagnosis. An existing import growing by more than 10% is marked red. This report is informational.
Compared Changed imports (101)
All 336 current runtime imports
Reported by agent-think[bot]. |
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
Problem
Think._broadcastMessages()serialises the whole cached transcript (up to the 32 MiB hydration budget) to every connected socket, and runs at least twice per turn: after the incoming persist and after the assistant cutover. On a long transcript every turn costs two full snapshots per client.Change
Protocol (shared,
packages/agents/src/chat) — this is a wire change:cf_agent_chat_messages_delta{ epoch, messages }: only the messages a turn boundary persisted (upsert by id; unknown ids append in order).cf_agent_chat_messagesgains an optionalepochnaming the snapshot.Client (
useAgentChat) — stores the epoch of the last snapshot applied and drops any delta whose epoch differs, so a delta arriving before its snapshot, or after the server re-derived its transcript, is never applied out of order. Delta upsert keeps the protected streaming assistant and the cross-tab observed accumulator exactly as the snapshot path does.Server (
think.ts) —_cacheEpochbumps in_replaceCachedMessages(hydration, branch/regeneration resync, compaction, clear, repair). Each connection records the epoch of the last snapshot it was sent (connect, dropped-submit rollback, or broadcast)._broadcastMessagesDelta(ids)sends a delta only when every connection is aligned to the current epoch; otherwise it sends the full frame, which re-aligns everyone.Now deltas: incoming persist (
_handleChatRequest,chat()RPC, programmatic turns),addMessages, and the happy-path assistant cutover (WS and RPC).Still a full snapshot, and why:
_transcriptNeedsSnapshotflag at both comment sites)ai-chat is untouched server-side (still snapshots) but shares the client, so it can adopt the frame later.
Numbers
packages/think/src/tests/delta-broadcasts.test.ts(fails onmain): 40-message transcript, one observer.cf_agent_chat_messagesper turn, ~86 KiB eachcf_agent_chat_messages_delta, one row each (bench asserts < 1/20 of a snapshot)Tests
packages/think/src/react-tests/delta-broadcasts.test.tsx): delta before snapshot dropped; wrong epoch / other instance dropped; matching delta upserts and appends; newer snapshot re-keys the guard; epoch-less snapshot accepts no deltas.think-session,assistant-agent, and the e2ewaitForMessagesBroadcastbarrier to accept delta frames.Suites: think workers 897/897, think react 7/7, ai-chat workers 653/653, agents chat 545/545, agents react 136/136.