Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/skills/forge.md
Original file line number Diff line number Diff line change
Expand Up @@ -1200,7 +1200,7 @@ when OTel tracing is enabled (OTel v1 / Phase 4 / #105). Both use
| `AuditScheduleModify` | `schedule_modify` | Schedule mutated at runtime |
| `EventAuthVerify` | `auth_verify` | Inbound request authenticated (`provider`, `user_id`, `org_id`, `token_kind`; `email` when the identity carries one). **Channel invoker:** for a channel-originated request the transport credential is the loopback token (`provider:internal`/`user_id:forge-internal`, recorded truthfully) and the human sender is stamped as `channel`/`channel_user`/`channel_email` from the `X-Forge-Channel*` headers — honored only for the runtime-internal identity (same trust gate as `applyChannelOnBehalfOf`). Slack/Teams resolve `channel_email`; Telegram (numeric id) & WhatsApp (msisdn) carry `channel_user` only |
| `EventAuthFail` | `auth_fail` | Inbound request rejected (`reason`, `token_kind`) |
| `AuditInputMediaRejected` | `input_media_rejected` | Inbound `file` parts the runtime can't forward to the model → rejected 4xx, not silently dropped (#255). Fields: `dropped` (`["file:<mime>"]`), `count`, `reason` (`model_not_vision_capable` \| `unsupported_media_type`). Gate: `Runner.checkInboundMedia`. **Images** (png/jpeg/gif/webp) on a **vision model** (`coreruntime.ModelSupportsVision`) are NOT rejected — `a2aMessageToLLM` projects them into `llm.ChatMessage.Parts` → Anthropic `image` source blocks / OpenAI `image_url` data URLs. Docs/video still rejected (Phase 3+) |
| `AuditInputMediaRejected` | `input_media_rejected` | Inbound `file` parts the runtime can't forward to the model → rejected 4xx, not silently dropped (#255). Fields: `dropped` (`["file:<mime>"]`), `count`, `reason` (`model_not_vision_capable` \| `unsupported_media_type` \| `too_many_image_parts` \| `image_limit_exceeded`). Gate: `Runner.checkInboundMedia`. **Images** (png/jpeg/gif/webp) on a **vision model** (`coreruntime.ModelSupportsVision`), within the DoS bounds, are NOT rejected — `a2aMessageToLLM` projects them into `llm.ChatMessage.Parts` → Anthropic `image` source blocks / OpenAI `image_url` data URLs. Docs/video still rejected (Phase 3+). **DoS controls** (`media_limits.go` + `mediaSem`): body cap 32 MiB both transports; per-image ≤5 MiB & ≤50 MP (`CheckImageLimits`, header-only decode defuses bombs); ≤20 images/msg; ≤4 concurrent media requests (excess shed with 429/unavailable) |
| `EventMCPServerStarted` | `mcp_server_started` | MCP server handshake succeeded |
| `EventMCPServerFailed` | `mcp_server_failed` | MCP server dial / handshake failed |
| `EventMCPServerDegraded` | `mcp_server_degraded` | MCP server in soft-fail |
Expand Down
9 changes: 9 additions & 0 deletions docs/core-concepts/runtime-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,15 @@ An image `file` part (`image/png`, `image/jpeg`, `image/gif`, `image/webp`) is f

Media the model can't consume is **rejected loudly, never silently dropped** (the `checkInboundMedia` ingest gate): an image on a text-only model, or a document/video part (not yet supported), returns a 4xx and emits the `input_media_rejected` audit event. Note the image **bytes** themselves are not text-scannable, so guardrail/intent scanning still applies only to the text/data projection; this is an accepted limitation.

**DoS bounds.** Because inline images raise the inbound-body cap to 32 MiB (both transports), the gate also enforces per-image and per-message limits, and a concurrency semaphore bounds how many media-bearing requests run at once — a flat body cap alone is not media DoS protection:

| Bound | Limit | On breach |
|-------|-------|-----------|
| Per-image bytes | 5 MiB (`MaxImagePartBytes`) | 4xx `image_limit_exceeded` |
| Decoded dimensions | 100 000 px per side **and** 50 MP total (`MaxImagePixels`, png/jpeg/gif via header-only `DecodeConfig`; webp bounded by bytes) | 4xx `image_limit_exceeded` (defuses decompression bombs; the per-side bound also keeps the pixel product from overflowing int64) |
| Images per message | 20 (`MaxImagePartsPerMessage`) | 4xx `too_many_image_parts` |
| Concurrent media requests | 4 (`maxConcurrentMediaRequests`) | `429`/unavailable — request is shed, not queued |

**Note for guardrail pattern authors:** parts join with **newlines** (matching what the model sees). A pattern intended to match content that may span a part boundary should use `\s+` rather than a literal space — a payload split across two text parts joins as `…end\nstart…`.

### Session Recovery Deduplication
Expand Down
2 changes: 1 addition & 1 deletion docs/security/audit-logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ All runtime security events are emitted as structured NDJSON to stderr with corr
| `mcp_auth_resolved` | The parked call's consent arrived (or the wait was canceled) and it resumed (#330). Carries `server`, `subject`, `wait_ms`. Emitted **once**, attributed to the parked invocation (#366). |
| `mcp_auth_timeout` | No consent within the window; the parked MCP call fails `no_token` (#330). Carries `server`, `subject`, `wait_ms`, `decision`. |
| `auth_fail` | Inbound request rejected (with `reason`, `token_kind`). No `task_id` (none is ever created), but carries `workflow_execution_id` when the request had the execution header — so a rejected request is still attributable to its workflow run (#278). |
| `input_media_rejected` | An inbound `tasks/send`/`sendSubscribe` message carried `file` parts the runtime cannot forward to the model, so the request was rejected with a 4xx (JSON-RPC invalid-params / HTTP 400) instead of silently dropping the attachment and returning a plausible answer that ignored it (#255). Carries `fields.dropped` (`["file:<mimeType>", …]`), `fields.count`, and `fields.reason` — `model_not_vision_capable` (an image sent to a text-only model) or `unsupported_media_type` (documents/video — not yet accepted). **Images** (`png`/`jpeg`/`gif`/`webp`) sent to a **vision-capable model** are NOT rejected: they are projected into the model request as inline vision input and do not emit this event. |
| `input_media_rejected` | An inbound `tasks/send`/`sendSubscribe` message carried `file` parts the runtime cannot forward to the model, so the request was rejected with a 4xx (JSON-RPC invalid-params / HTTP 400) instead of silently dropping the attachment and returning a plausible answer that ignored it (#255). Carries `fields.dropped` (`["file:<mimeType>", …]`), `fields.count`, and `fields.reason` — `model_not_vision_capable` (image sent to a text-only model), `unsupported_media_type` (documents/video — not yet accepted), `too_many_image_parts` (over the per-message image limit), or `image_limit_exceeded` (an image over the per-image byte or pixel bound). **Images** (`png`/`jpeg`/`gif`/`webp`) within those bounds, sent to a **vision-capable model**, are NOT rejected: they are projected into the model request as inline vision input and do not emit this event. A separate load-shedding path returns an unavailable/`429` (no audit event) when the runner is already at its concurrent-media-request cap. |
| `agent_card_published` | Agent Card finalized at startup or hot-reload (with `name`, `version`, `protocol_version`, `url`, `skill_count`, `capabilities`, `security_schemes`, `card_size_bytes`, `card_sha256`). See [Agent Card reference](../reference/a2a-agent-card.md). |
| `policy_loaded` | One per non-empty policy layer at startup (system / user / workspace). Carries `fields.layer`, `source` (file path), deny-list size counts, and max bounds. See [Platform Policy](platform-policy.md). |
| `policy_violation_at_build_time` | One per violation when `forge.yaml` conflicts with any policy layer. Agent refuses to start. Carries `fields.violation_kind` / `offending_value` / `forge_yaml_field` plus `layer` + `source` identifying the enforcing file. See [Platform Policy](platform-policy.md). |
Expand Down
2 changes: 1 addition & 1 deletion forge-cli/internal/surface/knowledge/forge.md
Original file line number Diff line number Diff line change
Expand Up @@ -1200,7 +1200,7 @@ when OTel tracing is enabled (OTel v1 / Phase 4 / #105). Both use
| `AuditScheduleModify` | `schedule_modify` | Schedule mutated at runtime |
| `EventAuthVerify` | `auth_verify` | Inbound request authenticated (`provider`, `user_id`, `org_id`, `token_kind`; `email` when the identity carries one). **Channel invoker:** for a channel-originated request the transport credential is the loopback token (`provider:internal`/`user_id:forge-internal`, recorded truthfully) and the human sender is stamped as `channel`/`channel_user`/`channel_email` from the `X-Forge-Channel*` headers — honored only for the runtime-internal identity (same trust gate as `applyChannelOnBehalfOf`). Slack/Teams resolve `channel_email`; Telegram (numeric id) & WhatsApp (msisdn) carry `channel_user` only |
| `EventAuthFail` | `auth_fail` | Inbound request rejected (`reason`, `token_kind`) |
| `AuditInputMediaRejected` | `input_media_rejected` | Inbound `file` parts the runtime can't forward to the model → rejected 4xx, not silently dropped (#255). Fields: `dropped` (`["file:<mime>"]`), `count`, `reason` (`model_not_vision_capable` \| `unsupported_media_type`). Gate: `Runner.checkInboundMedia`. **Images** (png/jpeg/gif/webp) on a **vision model** (`coreruntime.ModelSupportsVision`) are NOT rejected — `a2aMessageToLLM` projects them into `llm.ChatMessage.Parts` → Anthropic `image` source blocks / OpenAI `image_url` data URLs. Docs/video still rejected (Phase 3+) |
| `AuditInputMediaRejected` | `input_media_rejected` | Inbound `file` parts the runtime can't forward to the model → rejected 4xx, not silently dropped (#255). Fields: `dropped` (`["file:<mime>"]`), `count`, `reason` (`model_not_vision_capable` \| `unsupported_media_type` \| `too_many_image_parts` \| `image_limit_exceeded`). Gate: `Runner.checkInboundMedia`. **Images** (png/jpeg/gif/webp) on a **vision model** (`coreruntime.ModelSupportsVision`), within the DoS bounds, are NOT rejected — `a2aMessageToLLM` projects them into `llm.ChatMessage.Parts` → Anthropic `image` source blocks / OpenAI `image_url` data URLs. Docs/video still rejected (Phase 3+). **DoS controls** (`media_limits.go` + `mediaSem`): body cap 32 MiB both transports; per-image ≤5 MiB & ≤50 MP (`CheckImageLimits`, header-only decode defuses bombs); ≤20 images/msg; ≤4 concurrent media requests (excess shed with 429/unavailable) |
| `EventMCPServerStarted` | `mcp_server_started` | MCP server handshake succeeded |
| `EventMCPServerFailed` | `mcp_server_failed` | MCP server dial / handshake failed |
| `EventMCPServerDegraded` | `mcp_server_degraded` | MCP server in soft-fail |
Expand Down
99 changes: 90 additions & 9 deletions forge-cli/runtime/media_gate_test.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
package runtime

import (
"bytes"
"context"
"encoding/binary"
"hash/crc32"
"strings"
"testing"

Expand All @@ -14,12 +17,28 @@ func runnerWithModel(model string) *Runner {
return &Runner{modelConfig: &coreruntime.ModelConfig{Client: llm.ClientConfig{Model: model}}}
}

func imagePartMsg(mime string) a2a.Message {
// validPNG builds a minimal decodable PNG (signature + IHDR) with the given
// dimensions — enough for image.DecodeConfig, without a real pixel buffer.
func validPNG(w, h uint32) []byte {
var buf bytes.Buffer
buf.Write([]byte{0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a})
ihdr := make([]byte, 0, 13)
ihdr = binary.BigEndian.AppendUint32(ihdr, w)
ihdr = binary.BigEndian.AppendUint32(ihdr, h)
ihdr = append(ihdr, 8, 2, 0, 0, 0)
_ = binary.Write(&buf, binary.BigEndian, uint32(len(ihdr)))
buf.WriteString("IHDR")
buf.Write(ihdr)
_ = binary.Write(&buf, binary.BigEndian, crc32.ChecksumIEEE(append([]byte("IHDR"), ihdr...)))
return buf.Bytes()
}

func fileMsg(mime string, data []byte) a2a.Message {
return a2a.Message{
Role: a2a.MessageRoleUser,
Parts: []a2a.Part{
a2a.NewTextPart("look"),
a2a.NewFilePart(a2a.FileContent{MimeType: mime, Bytes: []byte{1, 2, 3}}),
a2a.NewFilePart(a2a.FileContent{MimeType: mime, Bytes: data}),
},
}
}
Expand All @@ -31,14 +50,14 @@ func TestCheckInboundMedia_Phase2(t *testing.T) {

t.Run("image accepted on vision model", func(t *testing.T) {
r := runnerWithModel("gpt-4o")
if got := r.checkInboundMedia(ctx, imagePartMsg("image/png"), nil); got != "" {
if got := r.checkInboundMedia(ctx, fileMsg("image/png", validPNG(64, 64)), nil); got != "" {
t.Errorf("image on a vision model must be accepted, got reject: %q", got)
}
})

t.Run("image rejected on non-vision model", func(t *testing.T) {
r := runnerWithModel("gpt-3.5-turbo")
got := r.checkInboundMedia(ctx, imagePartMsg("image/png"), nil)
got := r.checkInboundMedia(ctx, fileMsg("image/png", validPNG(64, 64)), nil)
if got == "" {
t.Fatal("image on a non-vision model must be rejected")
}
Expand All @@ -49,12 +68,37 @@ func TestCheckInboundMedia_Phase2(t *testing.T) {

t.Run("document rejected even on vision model", func(t *testing.T) {
r := runnerWithModel("gpt-4o")
got := r.checkInboundMedia(ctx, imagePartMsg("application/pdf"), nil)
got := r.checkInboundMedia(ctx, fileMsg("application/pdf", []byte("%PDF-1.7")), nil)
if got == "" || !strings.Contains(got, "application/pdf") {
t.Errorf("a document part must still be rejected in Phase 2; got %q", got)
}
})

t.Run("oversized image rejected", func(t *testing.T) {
r := runnerWithModel("gpt-4o")
got := r.checkInboundMedia(ctx, fileMsg("image/png", make([]byte, coreruntime.MaxImagePartBytes+1)), nil)
if got == "" || !strings.Contains(got, "limit") {
t.Errorf("oversized image must be rejected with a size reason; got %q", got)
}
})

t.Run("decompression-bomb dimensions rejected", func(t *testing.T) {
r := runnerWithModel("gpt-4o")
got := r.checkInboundMedia(ctx, fileMsg("image/png", validPNG(100_000, 100_000)), nil)
if got == "" {
t.Fatal("a document part must still be rejected in Phase 2")
t.Error("a gigapixel image must be rejected")
}
if !strings.Contains(got, "application/pdf") {
t.Errorf("reject message should name the pdf mime; got %q", got)
})

t.Run("too many image parts rejected", func(t *testing.T) {
r := runnerWithModel("gpt-4o")
parts := []a2a.Part{a2a.NewTextPart("many")}
for range coreruntime.MaxImagePartsPerMessage + 1 {
parts = append(parts, a2a.NewFilePart(a2a.FileContent{MimeType: "image/png", Bytes: validPNG(16, 16)}))
}
got := r.checkInboundMedia(ctx, a2a.Message{Role: a2a.MessageRoleUser, Parts: parts}, nil)
if got == "" || !strings.Contains(got, "image") {
t.Errorf("more than %d images must be rejected; got %q", coreruntime.MaxImagePartsPerMessage, got)
}
})

Expand All @@ -68,8 +112,45 @@ func TestCheckInboundMedia_Phase2(t *testing.T) {

t.Run("nil modelConfig treats as non-vision", func(t *testing.T) {
r := &Runner{}
if got := r.checkInboundMedia(ctx, imagePartMsg("image/png"), nil); got == "" {
if got := r.checkInboundMedia(ctx, fileMsg("image/png", validPNG(16, 16)), nil); got == "" {
t.Error("with no resolved model, image input must be rejected (fail closed)")
}
})
}

// TestAcquireMediaSlot verifies the concurrency semaphore bounds media requests
// and never gates non-media requests (#255 DoS control).
func TestAcquireMediaSlot(t *testing.T) {
r := &Runner{mediaSem: make(chan struct{}, 2)}

// Non-media requests are never bounded.
for range 5 {
if _, ok := r.acquireMediaSlot(false); !ok {
t.Fatal("non-media request must never be shed")
}
}

// Media requests fill the 2 slots, then the 3rd is shed.
rel1, ok1 := r.acquireMediaSlot(true)
rel2, ok2 := r.acquireMediaSlot(true)
if !ok1 || !ok2 {
t.Fatal("first two media slots must be granted")
}
if _, ok3 := r.acquireMediaSlot(true); ok3 {
t.Fatal("third media slot must be shed when at capacity")
}
// Releasing one frees a slot.
rel1()
rel3, ok := r.acquireMediaSlot(true)
if !ok {
t.Fatal("a slot must be available after release")
}
rel2()
rel3()

// A nil semaphore disables the bound (never sheds).
rNil := &Runner{}
if _, ok := rNil.acquireMediaSlot(true); !ok {
t.Fatal("nil mediaSem must disable the bound")
}
}
Loading
Loading