fix(integrations): langchain add multimodal content transformation functions for images, audio, and files #5278
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…eport-binary-data
Semver Impact of This PR🟢 Patch (bug fixes) 📋 Changelog PreviewThis is how your changes will appear in the changelog. New Features ✨
Bug Fixes 🐛
Documentation 📚
Internal Changes 🔧Release
Other
🤖 This preview updates automatically when you update the PR. |
…tive content formats
…eport-binary-data
…ats and use common function for data URI parsing
![sentry[bot]](https://avatars.githubusercontent.com/u/1396951?s=60&v=4){kind=small}
| return { | ||
| "type": "blob", |
There was a problem hiding this comment.
Bug: The code hardcodes "modality": "image" for Google-style inline_data or file_data when a type field is absent, ignoring the mime_type which could indicate audio or video.
Severity: HIGH
Suggested Fix
Infer the modality from the mime_type when the type field is not present in a Google-style content block. Create a helper function that maps MIME types (e.g., "audio/mp3", "video/mp4") to the correct modality ("audio", "video", etc.). Use "image" as a default only if the MIME type is missing or unrecognized.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: sentry_sdk/integrations/langchain.py#L263-L264
Potential issue: When processing Google-style content blocks (`inline_data` or
`file_data`) that lack a `type` field, the function `_format_content_block` hardcodes
the modality as `"image"`. This occurs even when the `mime_type` field indicates other
content types like audio, video, or documents, which are supported by Google's Gemini
API. The `mime_type` is extracted but not used to infer the correct modality. This will
lead to incorrect data categorization in Sentry, where non-image content from the
LangChain integration will be mislabeled as an image.
Did we get this right? 👍 / 👎 to inform future reviews.
| return { | ||
| "type": "blob", | ||
| "modality": "image", | ||
| "mime_type": inline_data.get("mime_type", ""), | ||
| "content": inline_data.get("data", ""), | ||
| } | ||
|
|
There was a problem hiding this comment.
Bug: Google-style content blocks (inline_data, file_data) without an explicit type are always assigned modality: "image", ignoring the actual mime_type for audio or video.
Severity: HIGH
Suggested Fix
Instead of hardcoding modality: "image", derive the modality from the mime_type present in the inline_data or file_data dictionary. A helper function could map MIME type prefixes (e.g., 'audio/', 'video/') to the correct modality ('audio', 'video'), with 'image' as a default.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: sentry_sdk/integrations/langchain.py#L263-L269
Potential issue: The fallback handlers for Google-style content blocks, specifically for
`inline_data` and `file_data`, incorrectly hardcode the `modality` as `"image"`.
According to Google Gemini API documentation, content blocks may not have a `type` field
and rely solely on `mime_type` to determine the content. This means if a block with a
`mime_type` like `"audio/mpeg"` is processed, it will fall through to the handler at
line 260 and be incorrectly categorized with `modality: "image"`. This leads to
incorrect data categorization in Sentry for non-image content like audio or video.
Did we get this right? 👍 / 👎 to inform future reviews.
Description
Add more support on message types for
gen_ai.request.messagesIssues
Closes: https://linear.app/getsentry/issue/TET-1637/redact-images-langchain