feat(runtime): persist inbound media for cross-turn replay (#255 Phase 4) - #535
Conversation
…e 4) Inbound images/PDFs were fed to the model inline in the turn they arrived, but lost afterward: session history stores llm.ChatMessage with MediaRef.Bytes tagged json:"-", and Phase 2/3 set no URI, so a multi-turn conversation dropped earlier media. Phase 4 persists it so it replays. - persistInboundMedia (new): writes each inbound media part's bytes to <FilesDir>/inbound/<sha256>.<ext> and records the path as MediaRef.URI, while keeping Bytes inline for the current turn. Content-addressed (dedup + idempotent); best-effort (a failure never fails the turn); no-op without a files dir. Files also land on disk for tools. - Wired into a2aMessageToLLM's two projection sites (new message + replayed task.History) in Execute. - RehydrateMedia (Phase 1) wired in before each provider request: reloads Bytes from URI for parts replayed from a persisted session (URI-only), skipping the fresh turn's already-inline parts. A file that can't be reloaded is left byteless and skipped by the provider serializers (degrade to text, don't fail). Net: history persists URI-only (never base64), and RehydrateMedia refills bytes each turn, so multi-turn image/PDF conversations keep their media without bloating the session file. Tests: persist→session-marshal(Bytes dropped, URI kept)→rehydrate round-trip; content-addressed dedup + idempotent (existing URI untouched); no-files-dir no-op. Docs: runtime-engine persistence/replay note + FilesDir inbound/ pointer, forge.md skill (synced).
initializ-mk
left a comment
There was a problem hiding this comment.
Self-review — verified against source ✅
Phase-4 persistence; posting as a COMMENT (own PR) with one follow-up.
Secure & correct:
- Write side implements the #528 item-8 recommendation —
<sha256-hex>generated name +extForMIME(6 hardcoded extensions; malicious mime →.bin, never a path component) into a dedicatedinbound/subdir,0700/0600. No attacker-controlled path component; content-addressing prevents targeting a path via bytes. - Read side confined + symlink-safe (
RehydrateMedia, #528), wired right before the provider request; skip-if-populated leaves the fresh turn's inline bytes alone and only reloads URI-only replayed parts. - Best-effort throughout (all
_ =) — never fails the turn; history persists URI-only, rehydrated per turn. Round-trip test pins the invariant.
One follow-up inline; adding it to the #255 checklist (item 3, storage).
| } | ||
| sum := sha256.Sum256(m.Bytes) | ||
| path := filepath.Join(inboundDir, hex.EncodeToString(sum[:])+extForMIME(m.MimeType)) | ||
| if err := os.WriteFile(path, m.Bytes, 0o600); err != nil { |
There was a problem hiding this comment.
Follow-up — LOW/MEDIUM (not blocking): no cleanup/GC for inbound/. Content-addressed writes accumulate indefinitely — there's no TTL, size cap, or refcount, and content-addressing makes session-scoped cleanup non-trivial (files are shared across sessions/turns, so deleting on one session's end can break another). Per-request DoS is bounded (body cap + per-part limits, #532), but a long-running agent receiving media grows inbound/ without bound → eventual disk exhaustion. This is the realization of #255 checklist item 3 (storage). The PR's scope-boundaries flag remote replay + tool-path surfacing but not retention — recommend tracking a retention policy (size cap / TTL / GC) as a follow-up.
|
Good |
Part of #255. Phase 4 — persist inbound media. Builds on Phase 1 (
MediaRef.URI+RehydrateMedia) and Phase 2/3 (image + PDF input).The gap
Phase 2/3 fed images/PDFs to the model inline in the turn they arrived, but lost them afterward: session history stores
llm.ChatMessagewithMediaRef.Bytestaggedjson:"-", and noURIwas ever set — so a multi-turn conversation dropped earlier media (turn 2 couldn't "see" the image from turn 1).What this does
persistInboundMedia(new) — writes each inbound media part's bytes to<FilesDir>/inbound/<sha256>.<ext>and records the path asMediaRef.URI, while keepingBytesinline for the current turn. Content-addressed (identical uploads dedup; re-writes idempotent), best-effort (a failure never fails the turn), no-op without a files dir. The files also land on disk for tools.a2aMessageToLLM's two projection sites (new message + replayedtask.History) inExecute.RehydrateMedia(from Phase 1) wired in before each provider request — reloadsBytesfromURIfor parts replayed from a persisted session (URI-only), skipping the fresh turn's already-inline parts. A file that can't be reloaded is left byteless and skipped by the provider serializers (degrades to text, doesn't fail the turn).Net: history persists URI-only (never base64), and
RehydrateMediarefills bytes each turn, so multi-turn image/PDF conversations keep their media without bloating the session file — the invariant Phase 1'sjson:"-"was designed for, now fully realized.Tests
json.Marshal(asserts Bytes dropped, base64 absent, URI kept) →RehydrateMediareloads the exact bytes.go test(core runtime + llm + cli runtime),golangci-lint0 issues, nogo.work.sumchurn.Docs
runtime-engine.md(persistence & cross-turn replay note +FilesDirinbound/pointer),forge.mdskill (synced).Scope boundaries (flagged)
RemoteSessionStorewould need the bytes in the remote store or a shared volume — a follow-up.inbound/, but the agent isn't yet told the path (accepted types are fed inline, so the model sees them directly). Injecting a path reference for tool access rides with the non-inline/extraction follow-up.#255 remaining
Phase 5 model-generated image output · document follow-ups (OpenAI Responses
input_file/ Gemini / extraction) · remote-session media replay.