Skip to content

feat(models): complete DeepSeek-VL2 VLM port#660

Merged
inureyes merged 2 commits into
mainfrom
feat/issue-533-deepseek-vl2
Jul 5, 2026
Merged

feat(models): complete DeepSeek-VL2 VLM port#660
inureyes merged 2 commits into
mainfrom
feat/issue-533-deepseek-vl2

Conversation

@inureyes

@inureyes inureyes commented Jul 5, 2026

Copy link
Copy Markdown
Member

Summary

Completes the DeepSeek-VL2 (deepseek_vl_v2) port, building on the WIP SigLIP tower and the deepseek_v2 embeds-forward helpers.

  • Vision tower (src/vision/encoders/deepseek_vl2.rs): SigLIP-style no-CLS ViT (patch conv folded to a linear, learned pos_embed, fused-qkv full attention, GELU-tanh MLP, trailing LayerNorm). Pixel activations are cast to the tower weight dtype so a bf16 pixel batch meets the f16 tower.
  • Projector (src/vision/connectors/deepseek_vl2.rs): downsample_mlp_gelu (2x2 space-to-depth with channel-outermost feature order, then Linear, GELU, Linear).
  • Processor (src/vision/processors/deepseek_vl2.rs): candidate-resolution best-fit tiling (max effective area, min-wasted tie-break), mean-color padding, a global thumbnail plus local tiles.
  • Wrapper (src/vision/deepseek_vl2.rs): the 2D tile mosaic (image_newline columns, view_separator, order [global, view_separator, local]) scattered into the <image> positions, over the deepseek_v2 MoE decoder reused unchanged.
  • Loader / wiring: vlm_deepseek_vl2.rs, detection, runtime dispatch, prompt expansion, model registration, and docs.

The text backbone reuses deepseek_v2.rs with no changes: config.json omits the MLA head dims, so the ModelArgs serde defaults match DeepSeek-V2, and the checkpoint ships pre-fused switch_mlp experts. The <image> id is 100003 (the tokenizer's <image>; the config carries no image_token_index).

Validation

Real checkpoint mlx-community/deepseek-vl2-small-4bit:

  • Text-only: "What is the capital of France?" produced "The capital of France is Paris."
  • Solid image: "What is the main color?" produced "The main color of this image is orange."
  • Shapes image (448px, 2x2 grid, 1023 image tokens): produced "three geometric shapes: a red circle, a blue square, and a green triangle."

9 co-located unit tests and 4 real-model parity tests (tests/deepseek_vl_v2_parity.rs) pass. Release build (metal,accelerate), cargo fmt, and cargo clippy --all-targets -- -D warnings are clean.

Drive-by

Clears two pre-existing clippy findings surfaced by a newer local toolchain: a needless return in build.rs (the CUDA recipe after it is cfg(not(macos)), so nothing runs after on macOS) and a runtime assertion on constants in switch_layers.rs (moved into a const block).

Closes #533

inureyes added 2 commits July 5, 2026 00:27
…ncoder)

Partial: DeepSeekV2Model.forward_from_embeds/embed_tokens_forward/num_layers (additive) + src/vision/encoders/deepseek_vl2.rs SigLIP tower. NOT compiled/validated. Remaining: downsample_mlp_gelu projector + tile-mosaic assembly wrapper, candidate-resolution processor, loader, runtime wiring, real-checkpoint validation. Scoping in memory project_deepseek_vl2_port_533.
Add DeepSeek-VL2 (deepseek_vl_v2): a SigLIP-style ViT tower (fused-qkv full attention, learned position embedding, GELU-tanh MLP, trailing LayerNorm), the downsample_mlp_gelu projector (2x2 space-to-depth then a two-layer GELU MLP), a candidate-resolution tiling processor (LLaVA-style best-fit selection, mean-color padding, global thumbnail plus local tiles), and the 2D tile mosaic (per-row image_newline columns and a view_separator, ordered [global, view_separator, local]) that scatters into the <image> placeholder positions. The text backbone reuses deepseek_v2.rs unchanged: config.json omits the MLA head dims, so the ModelArgs serde defaults match DeepSeek-V2 exactly, and the checkpoint ships pre-fused switch_mlp experts. The <image> id is 100003 and each placeholder expands to the per-image mosaic token count. Validated on mlx-community/deepseek-vl2-small-4bit: coherent text generation and correct image description on both the single-tile (421 tokens) and 2x2 grid (1023 tokens) paths, with 9 unit tests and 4 real-model parity tests passing. Also clears two pre-existing clippy findings surfaced by a newer local toolchain: a needless return in build.rs (the CUDA recipe below it is cfg(not(macos)), so nothing runs after on macOS) and a runtime assertion on constants in switch_layers.rs (moved into a const block).
@inureyes inureyes added type:enhancement New features, capabilities, or significant additions area:models Model architectures, weights, loading, metadata priority:medium Medium priority labels Jul 5, 2026
@inureyes inureyes merged commit c257af7 into main Jul 5, 2026
5 checks passed
@inureyes inureyes deleted the feat/issue-533-deepseek-vl2 branch July 5, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata priority:medium Medium priority type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(models): port DeepSeek-VL2 (MoE text + SigLIP/SAM vision)

1 participant