feat(llm): ChatMessage.Parts multimodal scaffolding + history references (#255 Phase 1) - #528
Conversation
…erences (#255) Phase 1 of the multimodal work: additive plumbing only — no provider serializes Parts and nothing populates them yet, so text-only behavior is byte-identical. This lands the types + invariants that Phase 2 (image input) builds on. - llm.ContentPart / llm.MediaRef + ChatMessage.Parts (json omitempty). Content stays the flattened text-of-record for compression/dedup/scanners; Parts is the additive multimodal representation. - MediaRef.Bytes is json:"-" — the history-bloat guard: persisted history and the session store carry only the URI reference, never inline base64, so media isn't rewritten to disk and re-sent to the model every replayed turn. - runtime.RehydrateMedia reloads Bytes from the URI (confined to the files dir via the existing confinedFilesPath) — the read side Phase 2 calls before building a provider ChatRequest. - memory.totalChars charges a flat mediaCharWeight per media part so an image-heavy history still trims (media carries no Content chars). Tests: text-only wire byte-identical (no "parts" key); Bytes never serialized (inline nor base64) while URI round-trips; HasMedia; rehydrate-from-URI + already-populated skip + path-escape rejection; media budget accounting.
initializ-mk
left a comment
There was a problem hiding this comment.
Self-review — verified against source ✅
Phase-1 scaffolding; text-only behavior byte-identical. Posting as a COMMENT (own PR) with one forward-looking note.
Verified good:
Partsisomitempty→ text-only marshals byte-identically (nopartskey);Contentstays the text-of-record. Tested.MediaRef.Bytesisjson:"-"→ onlyURIpersists (history-bloat guard). Tested (BytesNeverSerialized).RehydrateMediaconfines every URI viaconfinedFilesPath(rejects../absolute/out-of-dir), skips populated/empty, errors name the part. Path-escape rejection tested.mediaCharWeightaccounting: per media part, no double-count (text lives inContent). Tested.
One forward-looking security note inline (Phase 4, not this PR).
Nit: commit authored as MK vs initializ-mk.
| if media == nil || len(media.Bytes) > 0 || media.URI == "" { | ||
| continue | ||
| } | ||
| path, ok := confinedFilesPath(ctx, media.URI) |
There was a problem hiding this comment.
Forward-looking note — LOW (Phase 4, not this PR): confinedFilesPath is lexical (no symlink resolution). It's sound against ..//absolute traversal (Abs+Rel), but it doesn't EvalSymlinks — so a symlink inside the files dir pointing outward would be followed by the os.ReadFile below. Not exploitable here (Phase 1 populates no URIs from untrusted input, and this is the pre-existing shared file-tool confinement), but when Phase 4 persists inbound (untrusted) files to .forge/files, write them with generated names that can't be symlinks — or add EvalSymlinks to the confinement. Added to the #255 checklist as a Phase-4 item.
…review) confinedFilesPath is lexical (Abs+Rel) — it blocks ../ and absolute traversal but does not resolve symlinks, so a symlink placed INSIDE the files dir pointing outward would be followed by the read. Not exploitable in Phase 1 (no untrusted URIs), but rather than defer to Phase 4 (persist inbound untrusted files), close it now in the reader: resolveWithinFilesDir EvalSymlinks both the target and the files dir (the dir itself may sit under a symlink, e.g. macOS /tmp) and re-confines before reading. Localized to RehydrateMedia — the shared confinedFilesPath used by other file tools is untouched. Adds a symlink-escape rejection test.
|
Addressed the symlink note in Rather than modify the shared Added Full sweep green: build (core), Nit (author |
initializ-mk
left a comment
There was a problem hiding this comment.
Closing re-review — commit 2786950a ✅ LGTM
Verified against source. The forward-looking symlink note is addressed now (not deferred).
- Symlink-safe confinement.
resolveWithinFilesDirruns after the lexicalconfinedFilesPath(defense in depth):EvalSymlinkson both the target AND the files dir, then aRelcheck. Resolving the dir too is the subtle part done right — it handles the macOS/tmp → /private/tmp/t.TempDir()case so a legit file under a symlinked temp dir isn't a false escape. A symlink inside the dir pointing outward is caught; a non-existent path errors cleanly; it reads the resolved real path (also shrinks TOCTOU). - Test genuine:
TestRehydrateMedia_RejectsSymlinkEscape(in-dir symlink → out-of-dir secret ⇒ rejected +Bytesempty; skips where symlinks unsupported). Existing load-from-URI test still passes.
Everything on the PR checks out:
Partsomitempty→ text-only byte-identical;Contentremains the text-of-record.MediaRef.Bytesjson:"-"→ onlyURIpersists (history-bloat guard).RehydrateMediaconfinement is now lexical + symlink-resolved; skips populated/empty; errors name the part.mediaCharWeightaccounting: per media part, no double-count.
The #255 checklist item 8 read-side is now done; the write-side (generated names / no attacker symlinks when Phase 4 persists inbound files) remains captured there.
Verdict: LGTM. Needs a second reviewer to merge (own PR). Nit: commit authored as MK vs initializ-mk.
Part of #255 (multimodal input/output). Phase 1 — additive scaffolding. No provider serializes
Partsand nothing populates them yet, so text-only behavior is byte-identical. This lands the types and invariants that Phase 2 (image input) builds on, kept small and reviewable.What this adds
llm.ContentPart/llm.MediaRef+ChatMessage.Parts(json:",omitempty").Contentremains the flattened text-of-record (authoritative for compression, truncation, dedup, and the security scanners);Partsis the additive multimodal representation.MediaRef.Bytesisjson:"-"— the history-bloat guard. Persisted history and the session store carry only theURIreference, never inline base64, so media is not rewritten to the session file and re-sent to the model on every replayed turn.runtime.RehydrateMediareloadsBytesfrom theURI, confined to the files dir via the existingconfinedFilesPath— the read side Phase 2 calls before building a providerChatRequest. Skips already-populated bytes; rejects a URI that escapes the files dir.memory.totalCharscharges a flatmediaCharWeightper media part, so an image-heavy history still trims (media carries noContentchars — without this,trimwould think an all-images conversation is tiny while it blows the real token budget).Invariants locked by tests
ChatMessagemarshals byte-identically to today — no"parts"key (TestChatMessage_TextOnlyWireIsUnchanged).Bytesnever appear in the serialized message (raw or base64); theURIround-trips andBytescome back empty (TestMediaRef_BytesNeverSerialized).HasMedia.Not in this PR (later phases)
imagesource blocks, OpenAIimage_url) — Phase 2.fileparts intoChatMessage.Parts+ flipping the Phase 0 reject gate to inline for vision models — Phase 2..forge/files— Phase 4.No user-facing behavior yet, so no doc changes — the multimodal docs land with Phase 2/3 when images actually reach the model.
Verification
gofmt,
go build(core + cli),go test ./llm/... ./runtime/,golangci-lint(0 issues) all green. Nogo.work.sumchurn.