Skip to content

FIX: Keep media context descriptions when the modality router forwards media - #3357

Queued
Lanre Shittu (Shizoqua) wants to merge 2 commits into
microsoft:mainfrom
Shizoqua:claude/wizardly-brahmagupta-5obhxm
Queued

Lanre Shittu (Shizoqua) wants to merge 2 commits into
microsoft:mainfrom
Shizoqua:claude/wizardly-brahmagupta-5obhxm

Conversation

@Shizoqua

Copy link
Copy Markdown

Description

Fixes #3346.

_ModalityFeedbackRouter._build_multimodal_message rebuilt each forwarded media piece from its converted value and data type but left prompt_metadata empty. A seed or response image that carried a context_description lost it on the way to the adversarial chat or into the next objective request, so format_message_piece_for_context fell back to a bare [Image_path] placeholder.

The copy now keeps the media-scoped keys listed in _FORWARDED_MEDIA_METADATA_KEYS, which is currently just context_description. Anything else on the source piece is request or response state and is left behind on purpose. For example, carrying video_id forward would switch OpenAIVideoTarget into remix mode on the next turn. Each copy gets its own metadata dict, so editing it never touches the source piece. IDs and the forwarded converted payload work as before.

This only touches _build_multimodal_message. The pass-through branch of fill_adversarial_placeholders is handled separately in #3302.

Tests and Documentation

Added tests in tests/unit/executor/attack/component/test_modality_router.py:

  • Seed to adversarial, response to adversarial, and response to objective forwarding all keep context_description, drop video_id and inner_target_identifier, format as [Image_path - A blue square], and get fresh piece and conversation IDs.
  • Changing a copy's metadata leaves the source piece unchanged.
  • A source piece with no metadata still produces an empty dict and the plain placeholder.

No docs or notebooks changed, so JupyText was not needed.

@Shizoqua
Lanre Shittu (Shizoqua) force-pushed the claude/wizardly-brahmagupta-5obhxm branch from 5af8061 to cf57df9 Compare October 19, 2026 22:28
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 20, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 20, 2026
@romanlutz
Roman Lutz (romanlutz) added this pull request to the merge queue Oct 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: multimodal router drops media metadata when forwarding seeds and responses

2 participants