Virtualizing a streaming agent chat: gaps between TanStack Virtual and an AI rendering layer

[codevibesmatter]

Summary

We run a production agent chat UI (long-lived Claude Code sessions, server-authoritative transcripts) built on @tanstack/react-virtual. In wiring Virtual behind a streaming agent chat we hit a recurring class of problems that a generic virtualizer doesn't solve and that a chat-state hook (useChat) doesn't touch either — they live in the seam between the two.

Since TanStack AI is positioned to own that seam, this is a writeup of the gaps we hand-built, sorted by which surface (Virtual / AI / the adapter between them) would most naturally own each. Hoping it's useful input for a Virtual-into-AI rendering story. Happy to co-design or contribute.

Why agent chat ≠ traditional chat (Slack)

One axis flips everything: in traditional chat a row is write-once; in agent chat a row is write-many.

• Rows grow after insert. A streaming turn is inserted at a low estimate, then grows over hundreds of token deltas while pinned at the bottom. Every measure-once-cache-forever shortcut that's legal in Slack is illegal here.
• Mutation is high-frequency and single-row. Token-paced deltas hammer one live row, not human-paced terminal messages.
• Height variance is extreme. Code blocks, tables, reasoning, tool widgets, gates — cold-mount estimate error is huge, so first-frame total-size accuracy is load-bearing.
• Landing is semantic. You want to land on the part that needs the user (a pending tool/permission gate, or the top of the latest answer), not "last read line."
• The transcript is a projection. Edit-resubmit / fork / rewind mean the rendered rows can change identity and order; message identity must survive that.

Slack's hard problem is scale over immutable rows; agent chat's is mutation of live rows. They stress different parts of the stack. Virtual nails the first axis; the second is the gap.

The gaps (and who should own each)

Virtual already owns / nearly owns

1. Persistent measurement seed. A pluggable store that seeds estimateSize() from previously-measured heights so getTotalSize() is accurate on frame 1 of a cold remount of a long history. We back ours with localStorage keyed by (itemId, contentSig). Critical constraint: it must be a seed, never a measurement bypass — useCachedMeasurements: true skips DOM measurement and causes growing rows to paint over the rows below (overlapping bubbles). A streaming list must keep measuring every frame.
2. Content-keyed measurement invalidation. A row that grows under a stable item key serves a stale measured height. We attach a content version (a hash of seq:parts:textLen:partsHash) so a delta orphans the old height and forces a re-measure. A per-item "content version" input to Virtual would generalize this.
3. Single scroll authority (a documented contract). anchorTo: 'end' + followOnAppend already let virtual-core absorb streaming-growth deltas into the offset — instant for mid-row growth, smooth for whole-message append. The footgun is that nothing stops app code from also writing scrollTop, which produces yank-and-jerk. We tried springs and manual tweens and reverted both. Worth making the "only the virtualizer writes scrollTop" rule explicit.

TanStack AI should own (chat-semantic, no place in a generic virtualizer)

4. Semantic boundary anchor. Choose the landing target by content/status: pending gate → align end; else latest answer → align start; else bottom. "Scroll to the thing blocking progress" is a chat concept.
5. One-shot, intent-driven landing controller. Fire exactly one scrollToIndex per mount/transition; surrender on user scroll; guard with a "landed" latch + timeout fallback so it never loops or hangs. Everyone reinvents this badly (effect loops, re-pin wars).
6. Hydration completeness gate. Land only when history is complete enough — we gate on a cold → hydrating → live machine so you never anchor against partial/replaying history, with a fast path for warm cache revisits. useChat exposes status but no "safe to land now" signal.
7. Write-many message identity. Stable keys across resubmit/branch, optimistic→echo dedup, server-minted rows that don't collide with client-stamped ones. A client-id-keyed, write-once identity model breaks under branching and server-authored messages.

The adapter between them (the real integration surface)

8. Chunk → row assembly. The virtualizer counts assembled messages, but growth arrives as fragment appends underneath. The assembly layer is what produces stable getItemKey, the content version for (2), and optimistic dedup for (7). This seam is currently unowned, and if it's wrong everyone rebuilds 4–7 by hand.

What we'd suggest

• Virtual: a persistent estimate-provider hook + a per-item content-version input, and a docs note on why useCachedMeasurements is unsafe for growing rows.
• AI: a small, transport-agnostic scroll/landing controller (semantic anchor + one-shot landing + completeness gate) that drives a Virtual instance.
• Adapter: a documented chunk→row assembly contract as the integration point.

We're independently extracting our own version of this into a pure state-machine core + swappable reactive shell + animation engine, so we'd be glad to share that design or contribute toward whatever shape the maintainers prefer. Thanks for the great libraries.