Wire the ai-bot to-device streaming channel end-to-end (ai-bot side)

[lukemelia]

Summary

• CS-12261 — adds APP_BOXEL_RESPONSE_STREAM_EVENT_TYPE = 'app.boxel.response-stream' and the AppBoxelResponseStreamContent interface (fields: roomId, parentEventId, sequence, body, reasoning, toolRequests) to packages/runtime-common/matrix-constants.ts.
• CS-12262 — adds APP_BOXEL_ORIGINATING_DEVICE_ID_KEY and stamps the sending device id onto every boxel user-prompt event (sendMessage in packages/host/app/services/matrix-service.ts). Also adds the optional key to the CardMessageContent interface in packages/base/matrix-event.gts.
• CS-12263 — implements AI_BOT_STREAMING_MODE=to-device in the ai-bot's Responder: throttled state changes route to matrixClient.sendToDevice() targeted at the (userId, deviceId) that composed the prompt, carrying an app.boxel.response-stream payload. The final consolidated event still lands as a room edit (routed via flush → sendMessageEvent, not the throttled internal) so durable state is always in the room — to-device previews are ephemeral by design.
• Also merged in CS-12260 (off mode) as a base commit, since CS-12263 sits on top of the streamingMode gate that CS-12260 introduced. ai-bot: add AI_BOT_STREAMING_MODE=off to skip mid-stream Matrix edits #5556 remains open with the standalone CS-12260 patch; whichever of the two lands first, the other's diff auto-shrinks.

Together this is the full ai-bot side of the Improve Synapse Performance by Killing Streaming project's to-device path. The host receiver (CS-12264) is the last remaining wire — until it lands, running with AI_BOT_STREAMING_MODE=to-device produces to-device previews the browser doesn't yet consume, and the UX falls back to "thinking → complete response arrives at once", identical to off mode.

Design notes

• Final always lands as a room event. The throttled internal checks responseState.isStreamingFinished and routes to sendMessageEvent (room edit) whenever true, regardless of mode. flush in finalize uses sendMessageEvent directly. So the leading-edge case where the throttled internal fires during finalize still produces a room edit, not a to-device preview.
• Older-client fallback. When mode is to-device but the prompt lacks the originating device id (older client without CS-12262), the responder falls back to off behavior — no mid-stream sends, one final room edit. Deployment can enable to-device mode without waiting for every user's session to refresh.
• Preview payload shape. Full accumulated body / reasoning / toolRequests, with a monotonic per-turn sequence for last-writer-wins reconciliation on the client. Not deltas — gaps and reordering are tolerated by design.
• Encryption. to-device previews use the plaintext client.sendToDevice() path today. Acceptable for Cardstack's own Synapse (the operator can see the traffic either way) and for previews that will also land in the room as the final. For federated deployments or a stronger threat model, swap to client.encryptAndSendToDevice() as follow-up.
• toolRequests shape. Previews run toolCalls through the same toCommandRequest() the room event uses, so both channels carry { id, name, arguments: <object> }; arguments is {} until the streamed JSON completes. Typed unknown[] on the shared interface (the receiver, CS-12264, validates at the boundary), but the runtime shape now matches the room event.
• No isFinal field. The final state always lands as a room edit, never a to-device preview, so an isFinal preview flag would never be set — the room edit replacing the thinking placeholder is already the terminal signal. Dropped from the schema; reintroduce if the receiver ever needs an explicit "discard preview state now" signal.
• Multi-step turns keep streaming. The originating device id is taken from the triggering event, or — when a continuation is triggered by a tool / code-patch result that carries none — from the most recent stamped user-prompt event in the turn's history. So a turn that continues after a tool call still previews to the device that started it, rather than dropping to off behavior.
• sendToDevice takes a Map, not a plain object. matrix-js-sdk's sendToDevice(eventType, contentMap) requires Map<userId, Map<deviceId, content>> and iterates it internally (recursiveMapToObject); a plain nested object throws TypeError: contentMap is not iterable before anything reaches the wire — swallowed by the preview send's try/catch, so previews would have failed silently. The Responder builds the real Map. FakeMatrixClient.sendToDevice now rejects a non-Map argument so a regression fails the unit suite the way it fails in production.

Test plan

• Ai-bot: 22 Responding tests pass, including two new ones — \to-device` streaming mode: streams previews to the target device and lands one final consolidated room event, and `to-device` streaming mode without a target device falls back to `off` behavior. The CS-12260 off` mode test also still passes.
• All 5 Responder Cancellation tests pass — finalize({isCanceled:true}) still routes the final send to the room-edit path.
• Typecheck: no errors touching any of the modified files (the ../base/... / @cardstack/boxel-ui/* errors are the pre-existing local-dist issue).
• Verified the FakeMatrixClient Map guard bites: temporarily reverting the Responder to the plain-object payload flips the to-device streaming unit test to failing (not ok), because the swallowed TypeError leaves no preview sent. Restored, back to green.
• Matrix e2e (packages/matrix/tests/response-stream-to-device.spec.ts): drives a real matrix-js-sdk client against the dockerized Synapse — delivers a response-stream preview to the target device and asserts the real SDK rejects a plain-object contentMap. Typechecks + lints locally; first real run to be watched in CI (needs the full Synapse + realm-server stack, not run locally).
• Staging: set AI_BOT_STREAMING_MODE=to-device on the ai-bot task after CS-12264 lands. Verify (a) the browser renders the streaming preview, (b) only 2 room events land per turn, (c) other devices in the same account see the final response but no live preview.

Commits

• f85e59c65e — CS-12261: app.boxel.response-stream schema
• 7dd96110a8 — CS-12262: stamp originating device id on prompts
• 91eb30c3de — merge CS-12260 (off mode) as base
• 0ff3c38c15 — CS-12263: to-device streaming mode
• c5ad13b90b — fix: send previews as a Map; harden the fake + add a real-SDK matrix e2e boundary test
• 16619683b9 — review findings: normalize preview toolRequests via toCommandRequest; drop isFinal; thread the originating device id through multi-step turns

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files      1 suites   2h 58m 2s ⏱️
3 563 tests 3 548 ✅ 15 💤 0 ❌
3 582 runs  3 567 ✅ 15 💤 0 ❌

Results for commit 1661968.

Realm Server Test Results

    1 files  ±0      1 suites  ±0   12m 0s ⏱️ +44s
1 913 tests ±0  1 913 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 992 runs  ±0  1 992 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 1661968. ± Comparison against earlier commit c5ad13b.