Skip to content

feat(models): port Hunyuan-VL (ViT + perceive merger + XD-RoPE decoder)#663

Merged
inureyes merged 1 commit into
mainfrom
feat/issue-536-hunyuan-vl
Jul 5, 2026
Merged

feat(models): port Hunyuan-VL (ViT + perceive merger + XD-RoPE decoder)#663
inureyes merged 1 commit into
mainfrom
feat/issue-536-hunyuan-vl

Conversation

@inureyes

@inureyes inureyes commented Jul 5, 2026

Copy link
Copy Markdown
Member

Summary

Adds Hunyuan-VL (hunyuan_vl, e.g. HunyuanOCR), Tencent's vision-language family.

  • Vision tower (src/vision/encoders/hunyuan_vl.rs): a ViT over raster-ordered 768-wide patch rows. The per-patch conv embedding is folded into a linear at load; a learned position-embedding table (CLS row skipped) is bilinearly interpolated to each image's grid with the upstream truncation-toward-zero corner math; blocks run full attention over the packed sequence (matching upstream, which attends across images) with exact-GELU MLPs. The perceive PatchMerger per image applies RMSNorm, a stride-2 conv pair over the raster grid, a learned image_newline column, a linear to the decoder width, and learned image_begin / image_end rows: mh * (mw + 1) + 2 output rows per image.
  • Placeholder contract: the merger's row count IS the prompt placeholder count, so the runtime reuses the count-based DeepSeek-OCR placeholder expansion (insert_deepseekocr_image_tokens) with no framing tokens; the framing is carried by the features.
  • Decoder (src/models/hunyuan_vl.rs): the Hunyuan dense stack (per-head Q/K RMSNorm applied after the rotation, DynamicNTK-alpha rope base theta * alpha^(d/(d-2))) with XD-RoPE at prefill: 4D [P, T, H, W] position ids split across the head_dim / 2 frequency dims per xdrope_section ([16, 16, 16, 16]). P stays sequential everywhere; inside an image run W cycles 0..=mw per row (the newline slot takes w = mw), H repeats per row, T is the image index. A unit test proves the text-only degeneration to the standard concat-half rotation (fast_rope, traditional = false); decode uses sequential positions (rope_deltas = 0) through the shared MRopeState plumbing.
  • Processor (src/vision/processors/hunyuan_vl.rs): smart resize to multiples of 32 in [min_pixels, max_pixels] (defaults 512^2 / 2048^2, preprocessor_config.json overrides honored), bilinear, CLIP normalization, plain raster patch rows.

Validation

Real checkpoint hadeseus/HunyuanOCR-mlx-4bit (~1B; text 4-bit, vision tower bf16):

  • Text-only: "The capital of France is Paris." (120 tok/s)
  • Character-exact OCR of a rendered image: "MLXCEL RUNTIME Invoice No. 2026-0705"
  • Shapes image: blue square upper right, green triangle lower, colors and positions correct.

8 co-located unit tests (XD-RoPE degeneration, alpha base scaling, processor grid/rows/count) and 4 real-model parity tests pass. Release build, cargo fmt, and cargo clippy --all-targets -- -D warnings are clean.

Out of scope

Video inputs, multi-image XD-RoPE beyond the per-image grid pattern above, and server per-sequence MRoPE binding beyond the PaddleOCR-VL fallback-slot scope.

Closes #536

Add Hunyuan-VL (hunyuan_vl, e.g. HunyuanOCR). The vision tower (src/vision/encoders/hunyuan_vl.rs) is a ViT over raster-ordered 768-wide patch rows: a per-patch conv embedding folded into a linear, a learned position-embedding table bilinearly interpolated to each image's grid (upstream truncation-toward-zero corner math), full attention over the packed sequence, and a perceive PatchMerger per image (RMSNorm, stride-2 conv pair, a learned image_newline column, a linear to the decoder width, and learned image_begin/image_end rows) whose mh*(mw+1)+2 output rows match the prompt placeholder count exactly, so the runtime reuses the count-based DeepSeek-OCR placeholder expansion with no framing tokens. The decoder (src/models/hunyuan_vl.rs) is the Hunyuan dense stack (per-head Q/K RMSNorm applied after the rotation, DynamicNTK-alpha rope base alpha^(d/(d-2))) with XD-RoPE at prefill: 4D [P, T, H, W] position ids split across the head_dim/2 frequency dims per xdrope_section, with P sequential everywhere and the image run carrying row/column/image-index grids; a unit test proves the text-only degeneration to the standard concat-half rotation, and decode uses sequential positions (rope_deltas = 0) via the shared MRopeState plumbing. Validated against hadeseus/HunyuanOCR-mlx-4bit (text 4-bit, vision bf16): text generation ("The capital of France is Paris."), character-exact OCR of a rendered image ("MLXCEL RUNTIME Invoice No. 2026-0705"), and a shapes image described with correct colors and positions. 8 unit tests and 4 real-model parity tests pass.
@inureyes inureyes added type:enhancement New features, capabilities, or significant additions area:models Model architectures, weights, loading, metadata priority:low Low priority labels Jul 5, 2026
@inureyes inureyes merged commit 328fc79 into main Jul 5, 2026
5 checks passed
@inureyes inureyes deleted the feat/issue-536-hunyuan-vl branch July 5, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata priority:low Low priority type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(models): port Hunyuan-VL (ViT + Hunyuan text)

1 participant