feat(llm): image input to vision models (#255 Phase 2)

.claude/skills/forge.md

@@ -1200,7 +1200,7 @@ when OTel tracing is enabled (OTel v1 / Phase 4 / #105). Both use
 | `AuditScheduleModify` | `schedule_modify` | Schedule mutated at runtime |
 | `EventAuthVerify` | `auth_verify` | Inbound request authenticated (`provider`, `user_id`, `org_id`, `token_kind`; `email` when the identity carries one). **Channel invoker:** for a channel-originated request the transport credential is the loopback token (`provider:internal`/`user_id:forge-internal`, recorded truthfully) and the human sender is stamped as `channel`/`channel_user`/`channel_email` from the `X-Forge-Channel*` headers — honored only for the runtime-internal identity (same trust gate as `applyChannelOnBehalfOf`). Slack/Teams resolve `channel_email`; Telegram (numeric id) & WhatsApp (msisdn) carry `channel_user` only |
 | `EventAuthFail` | `auth_fail` | Inbound request rejected (`reason`, `token_kind`) |
-| `AuditInputMediaRejected` | `input_media_rejected` | Inbound message carried `file` parts the runtime can't forward to the model → rejected 4xx instead of silently dropped (#255). Fields: `dropped` (`["file:<mime>"]`), `count`, `reason`. Gate: `Runner.checkInboundMedia` at the four send handlers; `a2a.Message.FileParts()` |
+| `AuditInputMediaRejected` | `input_media_rejected` | Inbound `file` parts the runtime can't forward to the model → rejected 4xx, not silently dropped (#255). Fields: `dropped` (`["file:<mime>"]`), `count`, `reason` (`model_not_vision_capable` \| `unsupported_media_type`). Gate: `Runner.checkInboundMedia`. **Images** (png/jpeg/gif/webp) on a **vision model** (`coreruntime.ModelSupportsVision`) are NOT rejected — `a2aMessageToLLM` projects them into `llm.ChatMessage.Parts` → Anthropic `image` source blocks / OpenAI `image_url` data URLs. Docs/video still rejected (Phase 3+) |
 | `EventMCPServerStarted` | `mcp_server_started` | MCP server handshake succeeded |
 | `EventMCPServerFailed` | `mcp_server_failed` | MCP server dial / handshake failed |
 | `EventMCPServerDegraded` | `mcp_server_degraded` | MCP server in soft-fail |

docs/core-concepts/runtime-engine.md

@@ -29,10 +29,16 @@ An inbound A2A message is a list of typed parts. `a2a.Message.PromptText()` proj
 
 - **Text parts** are included verbatim (joined by newlines).
 - **Data parts** (`{"kind":"data","data":{…}}`) are appended as a fenced ` ```json ` block of the part's fields. This means structured input — e.g. a workflow step's output dispatched as a data part with no text part — still reaches the model instead of producing an empty prompt (a message with only a data part must never execute as empty).
-- **File parts** are not projected into prompt text (their bytes/URIs aren't text).
+- **File parts** are not projected into prompt *text* (their bytes/URIs aren't text). Image file parts are handled separately as multimodal input (below).
 
 The same projection feeds the inbound guardrail and intent-alignment scanners, so the security checks see exactly what the model sees — a payload carried in a data part can't reach the LLM while bypassing them. Each data part's projected block is capped (~16KiB, rune-safe) — the cap applies identically to the scanners and the prompt, so truncation can't open a divergence.
 
+#### Image input (multimodal)
+
+An image `file` part (`image/png`, `image/jpeg`, `image/gif`, `image/webp`) is forwarded to the model as native vision input when the resolved model is **vision-capable** (`runtime.ModelSupportsVision` — OpenAI `gpt-4o`/`gpt-4.1`/`gpt-5`/`o1`/`o3`/`o4`, Anthropic Claude 3+, Gemini 1.5/2). `a2aMessageToLLM` projects such parts into `llm.ChatMessage.Parts` (the flattened text stays in `Content` as the text-of-record for the scanners), and each provider serializes them natively — Anthropic `image` source blocks, OpenAI/Gemini `image_url` data URLs. A text-only message keeps `Parts` empty and marshals byte-identically to before.
+
+Media the model can't consume is **rejected loudly, never silently dropped** (the `checkInboundMedia` ingest gate): an image on a text-only model, or a document/video part (not yet supported), returns a 4xx and emits the `input_media_rejected` audit event. Note the image **bytes** themselves are not text-scannable, so guardrail/intent scanning still applies only to the text/data projection; this is an accepted limitation.
+
 **Note for guardrail pattern authors:** parts join with **newlines** (matching what the model sees). A pattern intended to match content that may span a part boundary should use `\s+` rather than a literal space — a payload split across two text parts joins as `…end\nstart…`.
 
 ### Session Recovery Deduplication

docs/security/audit-logging.md

@@ -32,7 +32,7 @@ All runtime security events are emitted as structured NDJSON to stderr with corr
 | `mcp_auth_resolved` | The parked call's consent arrived (or the wait was canceled) and it resumed (#330). Carries `server`, `subject`, `wait_ms`. Emitted **once**, attributed to the parked invocation (#366). |
 | `mcp_auth_timeout` | No consent within the window; the parked MCP call fails `no_token` (#330). Carries `server`, `subject`, `wait_ms`, `decision`. |
 | `auth_fail` | Inbound request rejected (with `reason`, `token_kind`). No `task_id` (none is ever created), but carries `workflow_execution_id` when the request had the execution header — so a rejected request is still attributable to its workflow run (#278). |
-| `input_media_rejected` | An inbound `tasks/send`/`sendSubscribe` message carried one or more `file` parts (image/document/…) the runtime cannot forward to the model, so the request was rejected with a 4xx (JSON-RPC invalid-params / HTTP 400) instead of silently dropping the attachment and returning a plausible answer that ignored it (#255). Carries `fields.dropped` (`["file:<mimeType>", …]`), `fields.count`, and `fields.reason` (`file_input_unsupported`). Multimodal input arrives in later slices; until then this makes the accepted-but-dropped case loud. |
+| `input_media_rejected` | An inbound `tasks/send`/`sendSubscribe` message carried `file` parts the runtime cannot forward to the model, so the request was rejected with a 4xx (JSON-RPC invalid-params / HTTP 400) instead of silently dropping the attachment and returning a plausible answer that ignored it (#255). Carries `fields.dropped` (`["file:<mimeType>", …]`), `fields.count`, and `fields.reason` — `model_not_vision_capable` (an image sent to a text-only model) or `unsupported_media_type` (documents/video — not yet accepted). **Images** (`png`/`jpeg`/`gif`/`webp`) sent to a **vision-capable model** are NOT rejected: they are projected into the model request as inline vision input and do not emit this event. |
 | `agent_card_published` | Agent Card finalized at startup or hot-reload (with `name`, `version`, `protocol_version`, `url`, `skill_count`, `capabilities`, `security_schemes`, `card_size_bytes`, `card_sha256`). See [Agent Card reference](../reference/a2a-agent-card.md). |
 | `policy_loaded` | One per non-empty policy layer at startup (system / user / workspace). Carries `fields.layer`, `source` (file path), deny-list size counts, and max bounds. See [Platform Policy](platform-policy.md). |
 | `policy_violation_at_build_time` | One per violation when `forge.yaml` conflicts with any policy layer. Agent refuses to start. Carries `fields.violation_kind` / `offending_value` / `forge_yaml_field` plus `layer` + `source` identifying the enforcing file. See [Platform Policy](platform-policy.md). |

forge-cli/internal/surface/knowledge/forge.md

@@ -1200,7 +1200,7 @@ when OTel tracing is enabled (OTel v1 / Phase 4 / #105). Both use
 | `AuditScheduleModify` | `schedule_modify` | Schedule mutated at runtime |
 | `EventAuthVerify` | `auth_verify` | Inbound request authenticated (`provider`, `user_id`, `org_id`, `token_kind`; `email` when the identity carries one). **Channel invoker:** for a channel-originated request the transport credential is the loopback token (`provider:internal`/`user_id:forge-internal`, recorded truthfully) and the human sender is stamped as `channel`/`channel_user`/`channel_email` from the `X-Forge-Channel*` headers — honored only for the runtime-internal identity (same trust gate as `applyChannelOnBehalfOf`). Slack/Teams resolve `channel_email`; Telegram (numeric id) & WhatsApp (msisdn) carry `channel_user` only |
 | `EventAuthFail` | `auth_fail` | Inbound request rejected (`reason`, `token_kind`) |
-| `AuditInputMediaRejected` | `input_media_rejected` | Inbound message carried `file` parts the runtime can't forward to the model → rejected 4xx instead of silently dropped (#255). Fields: `dropped` (`["file:<mime>"]`), `count`, `reason`. Gate: `Runner.checkInboundMedia` at the four send handlers; `a2a.Message.FileParts()` |
+| `AuditInputMediaRejected` | `input_media_rejected` | Inbound `file` parts the runtime can't forward to the model → rejected 4xx, not silently dropped (#255). Fields: `dropped` (`["file:<mime>"]`), `count`, `reason` (`model_not_vision_capable` \| `unsupported_media_type`). Gate: `Runner.checkInboundMedia`. **Images** (png/jpeg/gif/webp) on a **vision model** (`coreruntime.ModelSupportsVision`) are NOT rejected — `a2aMessageToLLM` projects them into `llm.ChatMessage.Parts` → Anthropic `image` source blocks / OpenAI `image_url` data URLs. Docs/video still rejected (Phase 3+) |
 | `EventMCPServerStarted` | `mcp_server_started` | MCP server handshake succeeded |
 | `EventMCPServerFailed` | `mcp_server_failed` | MCP server dial / handshake failed |
 | `EventMCPServerDegraded` | `mcp_server_degraded` | MCP server in soft-fail |

forge-cli/runtime/media_gate_test.go

@@ -0,0 +1,75 @@
+package runtime
+
+import (
+	"context"
+	"strings"
+	"testing"
+
+	"github.com/initializ/forge/forge-core/a2a"
+	"github.com/initializ/forge/forge-core/llm"
+	coreruntime "github.com/initializ/forge/forge-core/runtime"
+)
+
+func runnerWithModel(model string) *Runner {
+	return &Runner{modelConfig: &coreruntime.ModelConfig{Client: llm.ClientConfig{Model: model}}}
+}
+
+func imagePartMsg(mime string) a2a.Message {
+	return a2a.Message{
+		Role: a2a.MessageRoleUser,
+		Parts: []a2a.Part{
+			a2a.NewTextPart("look"),
+			a2a.NewFilePart(a2a.FileContent{MimeType: mime, Bytes: []byte{1, 2, 3}}),
+		},
+	}
+}
+
+// TestCheckInboundMedia_Phase2 covers the flipped gate: images pass on a
+// vision-capable model, and everything else is still rejected loudly (#255).
+func TestCheckInboundMedia_Phase2(t *testing.T) {
+	ctx := context.Background()
+
+	t.Run("image accepted on vision model", func(t *testing.T) {
+		r := runnerWithModel("gpt-4o")
+		if got := r.checkInboundMedia(ctx, imagePartMsg("image/png"), nil); got != "" {
+			t.Errorf("image on a vision model must be accepted, got reject: %q", got)
+		}
+	})
+
+	t.Run("image rejected on non-vision model", func(t *testing.T) {
+		r := runnerWithModel("gpt-3.5-turbo")
+		got := r.checkInboundMedia(ctx, imagePartMsg("image/png"), nil)
+		if got == "" {
+			t.Fatal("image on a non-vision model must be rejected")
+		}
+		if !strings.Contains(got, "image/png") || !strings.Contains(got, "does not support image") {
+			t.Errorf("reject message should name the mime and the non-vision reason; got %q", got)
+		}
+	})
+
+	t.Run("document rejected even on vision model", func(t *testing.T) {
+		r := runnerWithModel("gpt-4o")
+		got := r.checkInboundMedia(ctx, imagePartMsg("application/pdf"), nil)
+		if got == "" {
+			t.Fatal("a document part must still be rejected in Phase 2")
+		}
+		if !strings.Contains(got, "application/pdf") {
+			t.Errorf("reject message should name the pdf mime; got %q", got)
+		}
+	})
+
+	t.Run("text-only accepted", func(t *testing.T) {
+		r := runnerWithModel("gpt-3.5-turbo")
+		msg := a2a.Message{Role: a2a.MessageRoleUser, Parts: []a2a.Part{a2a.NewTextPart("hi")}}
+		if got := r.checkInboundMedia(ctx, msg, nil); got != "" {
+			t.Errorf("text-only must never be gated, got %q", got)
+		}
+	})
+
+	t.Run("nil modelConfig treats as non-vision", func(t *testing.T) {
+		r := &Runner{}
+		if got := r.checkInboundMedia(ctx, imagePartMsg("image/png"), nil); got == "" {
+			t.Error("with no resolved model, image input must be rejected (fail closed)")
+		}
+	})
+}

forge-cli/runtime/runner.go

@@ -2317,27 +2317,53 @@ func (r *Runner) checkInboundMedia(ctx context.Context, msg a2a.Message, auditLo
 	if len(files) == 0 {
 		return ""
 	}
-	dropped := make([]string, 0, len(files))
+	// Image parts are forwarded to the model as inline vision input when the
+	// resolved model is vision-capable (#255 Phase 2). Everything else — images
+	// on a non-vision model, and documents/video (later phases) — is still
+	// rejected loudly rather than silently dropped.
+	visionCapable := r.modelConfig != nil && coreruntime.ModelSupportsVision(r.modelConfig.Client.Model)
+
+	var dropped []string
+	reason := "unsupported_media_type"
 	for _, f := range files {
 		mt := f.MimeType
 		if mt == "" {
 			mt = "application/octet-stream"
 		}
+		if coreruntime.IsImageMIME(mt) {
+			if visionCapable {
+				continue // accepted → projected to the model as vision input
+			}
+			reason = "model_not_vision_capable"
+		}
 		dropped = append(dropped, "file:"+mt)
 	}
+	if len(dropped) == 0 {
+		return "" // all file parts are images the model can consume
+	}
 	if auditLogger != nil {
 		auditLogger.EmitFromContext(ctx, coreruntime.AuditEvent{
 			Event: coreruntime.AuditInputMediaRejected,
 			Fields: map[string]any{
 				"dropped": dropped,
-				"count":   len(files),
-				"reason":  "file_input_unsupported",
+				"count":   len(dropped),
+				"reason":  reason,
 			},
 		})
 	}
+	var detail string
+	if reason == "model_not_vision_capable" {
+		model := ""
+		if r.modelConfig != nil {
+			model = r.modelConfig.Client.Model
+		}
+		detail = fmt.Sprintf("the configured model %q does not support image input", model)
+	} else {
+		detail = "only image input (png/jpeg/gif/webp) is accepted; documents and video are not yet supported"
+	}
 	return fmt.Sprintf(
-		"this agent does not accept file/image input: %d file part(s) rejected (%s). Send text or data parts instead.",
-		len(files), strings.Join(dropped, ", "),
+		"%d media part(s) rejected (%s): %s.",
+		len(dropped), strings.Join(dropped, ", "), detail,
 	)
 }
 

forge-core/llm/providers/anthropic.go

@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"bytes"
 	"context"
+	"encoding/base64"
 	"encoding/json"
 	"fmt"
 	"io"
@@ -230,13 +231,22 @@ type anthropicMessage struct {
 }
 
 type anthropicContentBlock struct {
-	Type      string          `json:"type"`
-	Text      string          `json:"text,omitempty"`
-	ID        string          `json:"id,omitempty"`
-	Name      string          `json:"name,omitempty"`
-	Input     json.RawMessage `json:"input,omitempty"`
-	ToolUseID string          `json:"tool_use_id,omitempty"`
-	Content   string          `json:"content,omitempty"`
+	Type      string                `json:"type"`
+	Text      string                `json:"text,omitempty"`
+	ID        string                `json:"id,omitempty"`
+	Name      string                `json:"name,omitempty"`
+	Input     json.RawMessage       `json:"input,omitempty"`
+	ToolUseID string                `json:"tool_use_id,omitempty"`
+	Content   string                `json:"content,omitempty"`
+	Source    *anthropicImageSource `json:"source,omitempty"` // type=="image" (#255)
+}
+
+// anthropicImageSource is the source of an image content block:
+// {"type":"image","source":{"type":"base64","media_type":"image/png","data":"…"}}.
+type anthropicImageSource struct {
+	Type      string `json:"type"`       // "base64"
+	MediaType string `json:"media_type"` // e.g. image/png
+	Data      string `json:"data"`       // base64-encoded bytes
 }
 
 type anthropicTool struct {
@@ -343,11 +353,44 @@ func (c *AnthropicClient) convertMessage(m llm.ChatMessage) anthropicMessage {
 		return anthropicMessage{Role: "assistant", Content: data}
 	}
 
+	// Multimodal message: serialize content-parts as a block array (#255).
+	if len(m.Parts) > 0 {
+		data, _ := json.Marshal(anthropicBlocksFromParts(m.Parts))
+		return anthropicMessage{Role: role, Content: data}
+	}
+
 	// Simple text message
 	data, _ := json.Marshal(m.Content)
 	return anthropicMessage{Role: role, Content: data}
 }
 
+// anthropicBlocksFromParts maps provider-agnostic content parts to Anthropic
+// content blocks: text → {type:text}, image → {type:image, source:{base64}}.
+// A media part with no inline bytes is skipped (rehydration is the caller's
+// responsibility before the request is built).
+func anthropicBlocksFromParts(parts []llm.ContentPart) []anthropicContentBlock {
+	blocks := make([]anthropicContentBlock, 0, len(parts))
+	for _, p := range parts {
+		switch p.Type {
+		case llm.ContentPartImage:
+			if p.Media == nil || len(p.Media.Bytes) == 0 {
+				continue
+			}
+			blocks = append(blocks, anthropicContentBlock{
+				Type: "image",
+				Source: &anthropicImageSource{
+					Type:      "base64",
+					MediaType: p.Media.MimeType,
+					Data:      base64.StdEncoding.EncodeToString(p.Media.Bytes),
+				},
+			})
+		default: // text
+			blocks = append(blocks, anthropicContentBlock{Type: "text", Text: p.Text})
+		}
+	}
+	return blocks
+}
+
 // Anthropic-specific response types.
 type anthropicResponse struct {
 	ID         string                  `json:"id"`