Skip to content

feat(runtime): persist inbound media for cross-turn replay (#255 Phase 4) - #535

Merged
initializ-mk merged 1 commit into
mainfrom
feat/multimodal-persist-inbound
Sep 28, 2026
Merged

initializ-mk merged 1 commit into
mainfrom
feat/multimodal-persist-inbound

Conversation

@initializ-mk

Copy link
Copy Markdown
Contributor

Part of #255. Phase 4 — persist inbound media. Builds on Phase 1 (MediaRef.URI + RehydrateMedia) and Phase 2/3 (image + PDF input).

The gap

Phase 2/3 fed images/PDFs to the model inline in the turn they arrived, but lost them afterward: session history stores llm.ChatMessage with MediaRef.Bytes tagged json:"-", and no URI was ever set — so a multi-turn conversation dropped earlier media (turn 2 couldn't "see" the image from turn 1).

What this does

  • persistInboundMedia (new) — writes each inbound media part's bytes to <FilesDir>/inbound/<sha256>.<ext> and records the path as MediaRef.URI, while keeping Bytes inline for the current turn. Content-addressed (identical uploads dedup; re-writes idempotent), best-effort (a failure never fails the turn), no-op without a files dir. The files also land on disk for tools.
  • Wired into a2aMessageToLLM's two projection sites (new message + replayed task.History) in Execute.
  • RehydrateMedia (from Phase 1) wired in before each provider request — reloads Bytes from URI for parts replayed from a persisted session (URI-only), skipping the fresh turn's already-inline parts. A file that can't be reloaded is left byteless and skipped by the provider serializers (degrades to text, doesn't fail the turn).

Net: history persists URI-only (never base64), and RehydrateMedia refills bytes each turn, so multi-turn image/PDF conversations keep their media without bloating the session file — the invariant Phase 1's json:"-" was designed for, now fully realized.

Tests

  • Round-trip (the core guarantee): persist → json.Marshal (asserts Bytes dropped, base64 absent, URI kept) → RehydrateMedia reloads the exact bytes.
  • Content-addressed dedup (same bytes → same path) + idempotent (existing URI untouched).
  • No-files-dir → no-op, no error.
  • gofmt, go test (core runtime + llm + cli runtime), golangci-lint 0 issues, no go.work.sum churn.

Docs

runtime-engine.md (persistence & cross-turn replay note + FilesDir inbound/ pointer), forge.md skill (synced).

Scope boundaries (flagged)

  • Remote/distributed session replay: the URI is a host-local path, so replay works within a runtime/host (file session store). Cross-pod replay via RemoteSessionStore would need the bytes in the remote store or a shared volume — a follow-up.
  • Surfacing paths to agent tools: files are on disk under inbound/, but the agent isn't yet told the path (accepted types are fed inline, so the model sees them directly). Injecting a path reference for tool access rides with the non-inline/extraction follow-up.

#255 remaining

Phase 5 model-generated image output · document follow-ups (OpenAI Responses input_file / Gemini / extraction) · remote-session media replay.

…e 4)

Inbound images/PDFs were fed to the model inline in the turn they arrived, but
lost afterward: session history stores llm.ChatMessage with MediaRef.Bytes
tagged json:"-", and Phase 2/3 set no URI, so a multi-turn conversation dropped
earlier media. Phase 4 persists it so it replays.

- persistInboundMedia (new): writes each inbound media part's bytes to
  <FilesDir>/inbound/<sha256>.<ext> and records the path as MediaRef.URI, while
  keeping Bytes inline for the current turn. Content-addressed (dedup +
  idempotent); best-effort (a failure never fails the turn); no-op without a
  files dir. Files also land on disk for tools.
- Wired into a2aMessageToLLM's two projection sites (new message + replayed
  task.History) in Execute.
- RehydrateMedia (Phase 1) wired in before each provider request: reloads Bytes
  from URI for parts replayed from a persisted session (URI-only), skipping the
  fresh turn's already-inline parts. A file that can't be reloaded is left
  byteless and skipped by the provider serializers (degrade to text, don't fail).

Net: history persists URI-only (never base64), and RehydrateMedia refills bytes
each turn, so multi-turn image/PDF conversations keep their media without
bloating the session file.

Tests: persist→session-marshal(Bytes dropped, URI kept)→rehydrate round-trip;
content-addressed dedup + idempotent (existing URI untouched); no-files-dir
no-op. Docs: runtime-engine persistence/replay note + FilesDir inbound/ pointer,
forge.md skill (synced).

@initializ-mk initializ-mk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review — verified against source ✅

Phase-4 persistence; posting as a COMMENT (own PR) with one follow-up.

Secure & correct:

  • Write side implements the #528 item-8 recommendation — <sha256-hex> generated name + extForMIME (6 hardcoded extensions; malicious mime → .bin, never a path component) into a dedicated inbound/ subdir, 0700/0600. No attacker-controlled path component; content-addressing prevents targeting a path via bytes.
  • Read side confined + symlink-safe (RehydrateMedia, #528), wired right before the provider request; skip-if-populated leaves the fresh turn's inline bytes alone and only reloads URI-only replayed parts.
  • Best-effort throughout (all _ =) — never fails the turn; history persists URI-only, rehydrated per turn. Round-trip test pins the invariant.

One follow-up inline; adding it to the #255 checklist (item 3, storage).

}
sum := sha256.Sum256(m.Bytes)
path := filepath.Join(inboundDir, hex.EncodeToString(sum[:])+extForMIME(m.MimeType))
if err := os.WriteFile(path, m.Bytes, 0o600); err != nil {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up — LOW/MEDIUM (not blocking): no cleanup/GC for inbound/. Content-addressed writes accumulate indefinitely — there's no TTL, size cap, or refcount, and content-addressing makes session-scoped cleanup non-trivial (files are shared across sessions/turns, so deleting on one session's end can break another). Per-request DoS is bounded (body cap + per-part limits, #532), but a long-running agent receiving media grows inbound/ without bound → eventual disk exhaustion. This is the realization of #255 checklist item 3 (storage). The PR's scope-boundaries flag remote replay + tool-path surfacing but not retention — recommend tracking a retention policy (size cap / TTL / GC) as a follow-up.

@initializ-mk
initializ-mk merged commit f3d5186 into main Sep 28, 2026
15 checks passed
@wwwtarunjatt0987-cell

Copy link
Copy Markdown

Good

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants