fix: preserve input image geometry under CUA sandbox and warn on oversized images - #10071
Conversation
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="astrbot/core/pipeline/process_stage/method/agent_sub_stages/internal.py" line_range="264" />
<code_context>
+ # since lossy re-encoding would shift colors. Format
+ # normalization still applies to other formats, and oversized
+ # passthrough images warn below.
+ max_size = 1_000_000
quality = (
options.get("quality") if isinstance(options, dict) else None
</code_context>
<issue_to_address>
**issue (broader_impact):** CUA mode sets the conversion limit to 1,000,000, but the image encoder's existing PNG fallback also re-encodes PNGs larger than 1 MiB as JPEG. Therefore a large, otherwise compliant PNG is not sent byte-exact and its colors/content change, contradicting the CUA behavior documented here and the oversized-image test's `original` assertion.
**Triggers:** When CUA receives a PNG whose encoded size exceeds the PNG fallback threshold.
**Suggested fix:** Add an explicit CUA passthrough path that validates JPEG/PNG metadata and returns the original bytes before the generic PNG fallback, while retaining conversion for non-compliant formats.
</issue_to_address>Sourcery assessment
Approval pending. 1 finding to address first.
Blocking findings: astrbot/core/pipeline/process_stage/method/agent_sub_stages/internal.py:264
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d0ee979de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
f870314 to
7970b4b
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
@zouyonghe cua相关修复你看看呢 |
|
@sourcery-ai review |
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="astrbot/core/pipeline/process_stage/method/agent_sub_stages/internal.py" line_range="266" />
<code_context>
+ # Montages are never used for coordinates and keep the
+ # configured cap, which bounds the 3x3 canvas. Oversized
+ # passthrough images warn below.
+ max_size = 1_000_000
quality = (
options.get("quality") if isinstance(options, dict) else None
</code_context>
<issue_to_address>
**issue (bug_risk):** CUA still images whose longest edge exceeds 1,000,000 pixels are still resized by `_convert_image_bytes_sync`, so their original geometry and 1:1 coordinate mapping are not preserved.
**Triggers:** When a CUA input image has a longest edge greater than 1,000,000 pixels.
**Suggested fix:** Use an explicit no-resize path for CUA stills, or otherwise validate that the configured sentinel cannot be reached by real inputs before claiming geometry is always preserved.
</issue_to_address>Sourcery assessment
Approval pending. 1 finding to address first.
Blocking findings: astrbot/core/pipeline/process_stage/method/agent_sub_stages/internal.py:266
| # Montages are never used for coordinates and keep the | ||
| # configured cap, which bounds the 3x3 canvas. Oversized | ||
| # passthrough images warn below. | ||
| max_size = 1_000_000 |
There was a problem hiding this comment.
issue (bug_risk): CUA still images whose longest edge exceeds 1,000,000 pixels are still resized by _convert_image_bytes_sync, so their original geometry and 1:1 coordinate mapping are not preserved.
Triggers: When a CUA input image has a longest edge greater than 1,000,000 pixels.
Suggested fix: Use an explicit no-resize path for CUA stills, or otherwise validate that the configured sentinel cannot be reached by real inputs before claiming geometry is always preserved.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7970b4b6cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if cua_pixel_mode: | ||
| # CUA pixel tools read coordinates 1:1 on stills, so the | ||
| # still-image resize is lifted; compliant images pass through | ||
| # byte-exact since lossy re-encoding would shift colors. | ||
| # Montages are never used for coordinates and keep the | ||
| # configured cap, which bounds the 3x3 canvas. Oversized | ||
| # passthrough images warn below. | ||
| max_size = 1_000_000 |
There was a problem hiding this comment.
When a CUA sandbox is configured but the selected chat provider lacks image support, this lifts the resize cap before _ensure_img_caption or the quoted-image caption branch sends the image to the dedicated caption provider. Those images are never used by coordinate-based CUA tools, so a typical multi-megabyte phone photo that was previously reduced to image_compress_options.max_size is now uploaded at full resolution and may exceed the caption provider's limits or make captioning fail. Apply the uncapped size only to images that will reach the image-capable CUA chat provider, while retaining the configured cap for caption-only calls.
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
2 participants
电脑使用（CUA 沙箱）场景下，
astrbot_cua_mouse_click等像素坐标工具按模型在图片上读到的坐标 1:1 点击沙箱桌面。此前本次输入图片会按image_compress_options.max_size（默认 1280）缩放，模型读到的是缩放后的坐标，直接点击会漂移——1920×1080 截图缩至 1280×720 时，点击偏差约 (−534, −180) 像素（对角线 25.6%）。本 PR 在 CUA 沙箱运行时保留输入图片的原始宽高与内容：合规 JPEG/PNG 字节级原样发送，色彩与坐标均保持不变；其余格式在原尺寸下重编码，压缩质量等处理照常。处理后的图片超过约 5 MB 时（Anthropic 单图上限；OpenAI/Gemini 约 20 MB），日志给出可能超出服务商上传限制的警告。Modifications / 改动点
请求准备子阶段（
agent_sub_stages/internal.py）：provider_settings.computer_use_runtime为sandbox且sandbox.booter为cua时，静图的最长边上限提升为 1_000_000（等效不缩放），保持像素坐标 1:1；合规 JPEG/PNG 验证后原样复用，其余格式在原尺寸下重编码。动图 montage 与坐标无关，仍按配置上限生成（3×3 画布有界）。请求准备完成后统计本次图片体积，超过 5 MB 时记录一条警告日志。图片准备（
media_utils.py、agent_sub_stages/image_input.py）：prepare_model_image与prepare_request_images新增montage_max_size参数，montage 分支独立使用该上限，默认回退max_size。测试（
tests/test_process_stage_images.py）：新增test_cua_runtime_keeps_input_image_geometry——CUA 沙箱下 200×100 PNG 原样复用（字节一致）、WebP 原尺寸重编码为 JPEG，shipyard_neo/local 运行时仍按配置缩放；新增test_cua_oversize_image_warns——CUA 沙箱下超过 5 MB 的大图触发警告且内容不变，小图与其他运行时不产生警告；新增test_cua_montage_keeps_configured_cap——CUA 沙箱下动图 montage 仍按配置上限缩放。文档与配置提示：
docs/en|zh/providers/image-formats.md新增 TIP，说明 CUA 沙箱下输入图片保留原始宽高及大图体积风险；WebUI「最大边长」配置 hint 同步。This is NOT a breaking change. / 这不是一个破坏性变更。
Screenshots or Test Results / 运行截图或测试结果
Checklist / 检查清单
😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
/ 如果 PR 中有新加入的功能，已经通过 Issue / 邮件等方式和作者讨论过。
👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
/ 我的更改经过了良好的测试，并已在上方提供了“验证步骤”和“运行截图”。
🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in
requirements.txtandpyproject.toml./ 我确保没有引入新依赖库，或者引入了新依赖库的同时将其添加到
requirements.txt和pyproject.toml文件相应位置。😮 My changes do not introduce malicious code.
/ 我的更改没有引入恶意代码。
Summary by Sourcery
Preserve input image geometry for CUA sandbox interactions while retaining bounded animation montages and warning about oversized uploads.
New Features:
Bug Fixes:
Enhancements:
Documentation:
Tests: