feat(models): port Hunyuan-VL (ViT + perceive merger + XD-RoPE decoder)#663
Merged
Conversation
Add Hunyuan-VL (hunyuan_vl, e.g. HunyuanOCR). The vision tower (src/vision/encoders/hunyuan_vl.rs) is a ViT over raster-ordered 768-wide patch rows: a per-patch conv embedding folded into a linear, a learned position-embedding table bilinearly interpolated to each image's grid (upstream truncation-toward-zero corner math), full attention over the packed sequence, and a perceive PatchMerger per image (RMSNorm, stride-2 conv pair, a learned image_newline column, a linear to the decoder width, and learned image_begin/image_end rows) whose mh*(mw+1)+2 output rows match the prompt placeholder count exactly, so the runtime reuses the count-based DeepSeek-OCR placeholder expansion with no framing tokens. The decoder (src/models/hunyuan_vl.rs) is the Hunyuan dense stack (per-head Q/K RMSNorm applied after the rotation, DynamicNTK-alpha rope base alpha^(d/(d-2))) with XD-RoPE at prefill: 4D [P, T, H, W] position ids split across the head_dim/2 frequency dims per xdrope_section, with P sequential everywhere and the image run carrying row/column/image-index grids; a unit test proves the text-only degeneration to the standard concat-half rotation, and decode uses sequential positions (rope_deltas = 0) via the shared MRopeState plumbing. Validated against hadeseus/HunyuanOCR-mlx-4bit (text 4-bit, vision bf16): text generation ("The capital of France is Paris."), character-exact OCR of a rendered image ("MLXCEL RUNTIME Invoice No. 2026-0705"), and a shapes image described with correct colors and positions. 8 unit tests and 4 real-model parity tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Hunyuan-VL (
hunyuan_vl, e.g. HunyuanOCR), Tencent's vision-language family.src/vision/encoders/hunyuan_vl.rs): a ViT over raster-ordered 768-wide patch rows. The per-patch conv embedding is folded into a linear at load; a learned position-embedding table (CLS row skipped) is bilinearly interpolated to each image's grid with the upstream truncation-toward-zero corner math; blocks run full attention over the packed sequence (matching upstream, which attends across images) with exact-GELU MLPs. TheperceivePatchMerger per image applies RMSNorm, a stride-2 conv pair over the raster grid, a learnedimage_newlinecolumn, a linear to the decoder width, and learnedimage_begin/image_endrows:mh * (mw + 1) + 2output rows per image.insert_deepseekocr_image_tokens) with no framing tokens; the framing is carried by the features.src/models/hunyuan_vl.rs): the Hunyuan dense stack (per-head Q/K RMSNorm applied after the rotation, DynamicNTK-alpha rope basetheta * alpha^(d/(d-2))) with XD-RoPE at prefill: 4D[P, T, H, W]position ids split across thehead_dim / 2frequency dims perxdrope_section([16, 16, 16, 16]).Pstays sequential everywhere; inside an image runWcycles0..=mwper row (the newline slot takesw = mw),Hrepeats per row,Tis the image index. A unit test proves the text-only degeneration to the standard concat-half rotation (fast_rope,traditional = false); decode uses sequential positions (rope_deltas = 0) through the sharedMRopeStateplumbing.src/vision/processors/hunyuan_vl.rs): smart resize to multiples of 32 in[min_pixels, max_pixels](defaults 512^2 / 2048^2,preprocessor_config.jsonoverrides honored), bilinear, CLIP normalization, plain raster patch rows.Validation
Real checkpoint
hadeseus/HunyuanOCR-mlx-4bit(~1B; text 4-bit, vision tower bf16):8 co-located unit tests (XD-RoPE degeneration, alpha base scaling, processor grid/rows/count) and 4 real-model parity tests pass. Release build,
cargo fmt, andcargo clippy --all-targets -- -D warningsare clean.Out of scope
Video inputs, multi-image XD-RoPE beyond the per-image grid pattern above, and server per-sequence MRoPE binding beyond the PaddleOCR-VL fallback-slot scope.
Closes #536