Skip to content

feat(llm): ChatMessage.Parts multimodal scaffolding + history references (#255 Phase 1) - #528

Merged
initializ-mk merged 2 commits into
mainfrom
feat/multimodal-chatmessage-parts
Sep 25, 2026
Merged

initializ-mk merged 2 commits into
mainfrom
feat/multimodal-chatmessage-parts

Conversation

@initializ-mk

Copy link
Copy Markdown
Contributor

Part of #255 (multimodal input/output). Phase 1 — additive scaffolding. No provider serializes Parts and nothing populates them yet, so text-only behavior is byte-identical. This lands the types and invariants that Phase 2 (image input) builds on, kept small and reviewable.

What this adds

  • llm.ContentPart / llm.MediaRef + ChatMessage.Parts (json:",omitempty"). Content remains the flattened text-of-record (authoritative for compression, truncation, dedup, and the security scanners); Parts is the additive multimodal representation.
  • MediaRef.Bytes is json:"-" — the history-bloat guard. Persisted history and the session store carry only the URI reference, never inline base64, so media is not rewritten to the session file and re-sent to the model on every replayed turn.
  • runtime.RehydrateMedia reloads Bytes from the URI, confined to the files dir via the existing confinedFilesPath — the read side Phase 2 calls before building a provider ChatRequest. Skips already-populated bytes; rejects a URI that escapes the files dir.
  • memory.totalChars charges a flat mediaCharWeight per media part, so an image-heavy history still trims (media carries no Content chars — without this, trim would think an all-images conversation is tiny while it blows the real token budget).

Invariants locked by tests

  • Back-compat: a text-only ChatMessage marshals byte-identically to today — no "parts" key (TestChatMessage_TextOnlyWireIsUnchanged).
  • No bloat: inline Bytes never appear in the serialized message (raw or base64); the URI round-trips and Bytes come back empty (TestMediaRef_BytesNeverSerialized).
  • Rehydrate-from-URI, already-populated skip, and path-escape rejection; media budget accounting; HasMedia.

Not in this PR (later phases)

  • Provider serialization (Anthropic image source blocks, OpenAI image_url) — Phase 2.
  • Projecting A2A file parts into ChatMessage.Parts + flipping the Phase 0 reject gate to inline for vision models — Phase 2.
  • Persisting inbound files to .forge/files — Phase 4.

No user-facing behavior yet, so no doc changes — the multimodal docs land with Phase 2/3 when images actually reach the model.

Verification

gofmt, go build (core + cli), go test ./llm/... ./runtime/, golangci-lint (0 issues) all green. No go.work.sum churn.

…erences (#255)

Phase 1 of the multimodal work: additive plumbing only — no provider serializes
Parts and nothing populates them yet, so text-only behavior is byte-identical.
This lands the types + invariants that Phase 2 (image input) builds on.

- llm.ContentPart / llm.MediaRef + ChatMessage.Parts (json omitempty). Content
  stays the flattened text-of-record for compression/dedup/scanners; Parts is
  the additive multimodal representation.
- MediaRef.Bytes is json:"-" — the history-bloat guard: persisted history and
  the session store carry only the URI reference, never inline base64, so media
  isn't rewritten to disk and re-sent to the model every replayed turn.
- runtime.RehydrateMedia reloads Bytes from the URI (confined to the files dir
  via the existing confinedFilesPath) — the read side Phase 2 calls before
  building a provider ChatRequest.
- memory.totalChars charges a flat mediaCharWeight per media part so an
  image-heavy history still trims (media carries no Content chars).

Tests: text-only wire byte-identical (no "parts" key); Bytes never serialized
(inline nor base64) while URI round-trips; HasMedia; rehydrate-from-URI +
already-populated skip + path-escape rejection; media budget accounting.

@initializ-mk initializ-mk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review — verified against source ✅

Phase-1 scaffolding; text-only behavior byte-identical. Posting as a COMMENT (own PR) with one forward-looking note.

Verified good:

  • Parts is omitempty → text-only marshals byte-identically (no parts key); Content stays the text-of-record. Tested.
  • MediaRef.Bytes is json:"-" → only URI persists (history-bloat guard). Tested (BytesNeverSerialized).
  • RehydrateMedia confines every URI via confinedFilesPath (rejects ../absolute/out-of-dir), skips populated/empty, errors name the part. Path-escape rejection tested.
  • mediaCharWeight accounting: per media part, no double-count (text lives in Content). Tested.

One forward-looking security note inline (Phase 4, not this PR).

Nit: commit authored as MK vs initializ-mk.

if media == nil || len(media.Bytes) > 0 || media.URI == "" {
continue
}
path, ok := confinedFilesPath(ctx, media.URI)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forward-looking note — LOW (Phase 4, not this PR): confinedFilesPath is lexical (no symlink resolution). It's sound against ..//absolute traversal (Abs+Rel), but it doesn't EvalSymlinks — so a symlink inside the files dir pointing outward would be followed by the os.ReadFile below. Not exploitable here (Phase 1 populates no URIs from untrusted input, and this is the pre-existing shared file-tool confinement), but when Phase 4 persists inbound (untrusted) files to .forge/files, write them with generated names that can't be symlinks — or add EvalSymlinks to the confinement. Added to the #255 checklist as a Phase-4 item.

…review)

confinedFilesPath is lexical (Abs+Rel) — it blocks ../ and absolute traversal
but does not resolve symlinks, so a symlink placed INSIDE the files dir pointing
outward would be followed by the read. Not exploitable in Phase 1 (no untrusted
URIs), but rather than defer to Phase 4 (persist inbound untrusted files), close
it now in the reader: resolveWithinFilesDir EvalSymlinks both the target and the
files dir (the dir itself may sit under a symlink, e.g. macOS /tmp) and
re-confines before reading. Localized to RehydrateMedia — the shared
confinedFilesPath used by other file tools is untouched. Adds a symlink-escape
rejection test.
@initializ-mk

Copy link
Copy Markdown
Contributor Author

Addressed the symlink note in 2786950 — closed it now rather than deferring to Phase 4.

Rather than modify the shared confinedFilesPath (which other file tools depend on, and where a blanket EvalSymlinks would break legitimately symlinked files dirs — e.g. macOS /tmp → /private/tmp), I hardened the reader where the actual os.ReadFile happens: RehydrateMedia now calls resolveWithinFilesDir, which EvalSymlinks both the target and the files dir (handling the symlinked-dir case) and re-confines with a Rel check before reading. So a symlink placed inside .forge/files pointing outward is caught regardless of how a future phase names files — the confinement lives with the reader, not with the file-naming convention.

Added TestRehydrateMedia_RejectsSymlinkEscape (creates a symlink inside the files dir → outside; asserts rejection + no bytes populated; skips cleanly on platforms without symlink support). The shared confinedFilesPath is untouched, so no behavior change for existing file tools.

Full sweep green: build (core), go test ./runtime/ ./llm/, golangci-lint 0 issues.

Nit (author MK vs initializ-mk): local git identity; noted.

@initializ-mk initializ-mk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closing re-review — commit 2786950a ✅ LGTM

Verified against source. The forward-looking symlink note is addressed now (not deferred).

  • Symlink-safe confinement. resolveWithinFilesDir runs after the lexical confinedFilesPath (defense in depth): EvalSymlinks on both the target AND the files dir, then a Rel check. Resolving the dir too is the subtle part done right — it handles the macOS /tmp → /private/tmp / t.TempDir() case so a legit file under a symlinked temp dir isn't a false escape. A symlink inside the dir pointing outward is caught; a non-existent path errors cleanly; it reads the resolved real path (also shrinks TOCTOU).
  • Test genuine: TestRehydrateMedia_RejectsSymlinkEscape (in-dir symlink → out-of-dir secret ⇒ rejected + Bytes empty; skips where symlinks unsupported). Existing load-from-URI test still passes.

Everything on the PR checks out:

  • Parts omitempty → text-only byte-identical; Content remains the text-of-record.
  • MediaRef.Bytes json:"-" → only URI persists (history-bloat guard).
  • RehydrateMedia confinement is now lexical + symlink-resolved; skips populated/empty; errors name the part.
  • mediaCharWeight accounting: per media part, no double-count.

The #255 checklist item 8 read-side is now done; the write-side (generated names / no attacker symlinks when Phase 4 persists inbound files) remains captured there.

Verdict: LGTM. Needs a second reviewer to merge (own PR). Nit: commit authored as MK vs initializ-mk.

@initializ-mk
initializ-mk merged commit c6343b6 into main Sep 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant