feat(models): complete DeepSeek-VL2 VLM port#660
Merged
Conversation
…ncoder) Partial: DeepSeekV2Model.forward_from_embeds/embed_tokens_forward/num_layers (additive) + src/vision/encoders/deepseek_vl2.rs SigLIP tower. NOT compiled/validated. Remaining: downsample_mlp_gelu projector + tile-mosaic assembly wrapper, candidate-resolution processor, loader, runtime wiring, real-checkpoint validation. Scoping in memory project_deepseek_vl2_port_533.
Add DeepSeek-VL2 (deepseek_vl_v2): a SigLIP-style ViT tower (fused-qkv full attention, learned position embedding, GELU-tanh MLP, trailing LayerNorm), the downsample_mlp_gelu projector (2x2 space-to-depth then a two-layer GELU MLP), a candidate-resolution tiling processor (LLaVA-style best-fit selection, mean-color padding, global thumbnail plus local tiles), and the 2D tile mosaic (per-row image_newline columns and a view_separator, ordered [global, view_separator, local]) that scatters into the <image> placeholder positions. The text backbone reuses deepseek_v2.rs unchanged: config.json omits the MLA head dims, so the ModelArgs serde defaults match DeepSeek-V2 exactly, and the checkpoint ships pre-fused switch_mlp experts. The <image> id is 100003 and each placeholder expands to the per-image mosaic token count. Validated on mlx-community/deepseek-vl2-small-4bit: coherent text generation and correct image description on both the single-tile (421 tokens) and 2x2 grid (1023 tokens) paths, with 9 unit tests and 4 real-model parity tests passing. Also clears two pre-existing clippy findings surfaced by a newer local toolchain: a needless return in build.rs (the CUDA recipe below it is cfg(not(macos)), so nothing runs after on macOS) and a runtime assertion on constants in switch_layers.rs (moved into a const block).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the DeepSeek-VL2 (
deepseek_vl_v2) port, building on the WIP SigLIP tower and thedeepseek_v2embeds-forward helpers.src/vision/encoders/deepseek_vl2.rs): SigLIP-style no-CLS ViT (patch conv folded to a linear, learnedpos_embed, fused-qkv full attention, GELU-tanh MLP, trailing LayerNorm). Pixel activations are cast to the tower weight dtype so a bf16 pixel batch meets the f16 tower.src/vision/connectors/deepseek_vl2.rs):downsample_mlp_gelu(2x2 space-to-depth with channel-outermost feature order, then Linear, GELU, Linear).src/vision/processors/deepseek_vl2.rs): candidate-resolution best-fit tiling (max effective area, min-wasted tie-break), mean-color padding, a global thumbnail plus local tiles.src/vision/deepseek_vl2.rs): the 2D tile mosaic (image_newlinecolumns,view_separator, order[global, view_separator, local]) scattered into the<image>positions, over thedeepseek_v2MoE decoder reused unchanged.vlm_deepseek_vl2.rs, detection, runtime dispatch, prompt expansion, model registration, and docs.The text backbone reuses
deepseek_v2.rswith no changes:config.jsonomits the MLA head dims, so theModelArgsserde defaults match DeepSeek-V2, and the checkpoint ships pre-fusedswitch_mlpexperts. The<image>id is 100003 (the tokenizer's<image>; the config carries noimage_token_index).Validation
Real checkpoint
mlx-community/deepseek-vl2-small-4bit:9 co-located unit tests and 4 real-model parity tests (
tests/deepseek_vl_v2_parity.rs) pass. Release build (metal,accelerate),cargo fmt, andcargo clippy --all-targets -- -D warningsare clean.Drive-by
Clears two pre-existing clippy findings surfaced by a newer local toolchain: a needless
returninbuild.rs(the CUDA recipe after it iscfg(not(macos)), so nothing runs after on macOS) and a runtime assertion on constants inswitch_layers.rs(moved into aconstblock).Closes #533