[Abhishek Kumar Singh (abhishek-singh591)]

Weight-Free Offline Prefill Blocking

Summary

This branch adds and validates offline weight-free prefill blocking across
GPT-OSS, Gemma4, Qwen3-VL-MoE, and Qwen3.5-MoE. It covers the prefill_q,
prefill_kv, and prefill_qkv modes through export, compile, QAIC runtime,
and HF token-parity tests.

Branch Changes

• Add offline prefill-blocking configuration, dispatch, export, and QAIC
  parity coverage for supported causal-LM and VLM paths.
• Preserve retained-state handoff across blocked prefill and decode QPCs.
• Correct offline KV/QKV blocking behavior, including head/query chunking and
  blocked-cache access required by the compiled graph.
• Add weight-free export, transform, dispatch, reproducer-config, and QAIC
  regression coverage for the new modes.
• Update the GPT-OSS and Qwen3-VL-MoE disaggregated examples and add their
  usage documentation.
• Fix MDP intersection alias pruning and CI/Jenkins stage gating for the
  weight-free blocking test suite.

Gemma4 Correction

The Gemma4 follow-up restores the layer-specific bidirectional vision-mask
behavior:

• construct the bidirectional vision mask only for sliding_attention;
• use ctx_len as its target length only for prefill_online;
• preserve future KV blocks with skip_kv=False for that mask path; and
• keep dummy retained KV-cache allocation at effective export seq_len, not
  ctx_len, including the non-degenerate decode trace.

Focused tests cover prompt and decode dummy-cache shapes, and the existing
weight-free Gemma4 decode export test verifies the contract when ctx_len is
provided.

Validation

The complete QAIC parity matrix passes after retrying one transient
QAIC_ERROR_NSP_ALLOC_FAILED device-allocation failure:

Model	prefill_q	prefill_kv	prefill_qkv
GPT-OSS	Pass — 40 tokens, two prompts	Pass — 40 tokens, two prompts	Pass — 40 tokens, two prompts
Gemma4	Pass — 10 tokens	Pass — 10 tokens	Pass — 10 tokens
Qwen3-VL-MoE	Pass — 10 tokens	Pass — 10 tokens	Pass — 10 tokens
Qwen3.5-MoE	Pass — 10 tokens	Pass — 10 tokens	Pass — 10 tokens

[Onkar Chougule (ochougul)]

The new softmax argument ordering introduces a runtime/export regression in existing positional callers. CI is still pending; git diff --check passes.

[Ann Kuruvilla (quic-akuruvil)]

please check the failure on unit tests

[Ann Kuruvilla (quic-akuruvil)]

how many additional tests have been added in WF stage?

[Onkar Chougule (ochougul)]

The attention, cache handoff, and dispatch changes line up on the pinned head. The offline KV/QKV export assertions are unreachable, as noted inline. GitHub reports only DCO as passing; I did not run the QAIC matrix locally.

[Abhishek Kumar Singh (abhishek-singh591)]

Follow-up after syncing upstream main in commit c93530e2b35123e661c4c92d141b2924c4034e37:

• Full local unit suite: 2,115 passed, 7 skipped.
• Added WF parameterized coverage: 12 export-structure cases (4 model families × 3 modes), 18 VLM QAIC parity cases (3 families × 3 modes × weight-free/legacy), and GPT-OSS disaggregated parity expanded from 1 case to 6 (+5).
• Added 2 regression cases ensuring offline prefill_kv/prefill_qkv graphs require both KV read and write ops. Net: 37 additional collected cases across these touched suites.
• Focused merged-tree checks: 13 passed, 3 skipped; Ruff and formatting hooks pass.

All inline review threads are resolved. The older formal review still leaves GitHub at CHANGES_REQUESTED; please re-review the updated head. Jenkins/internal review checks are still pending.

[Abhishek Kumar Singh (abhishek-singh591)]

Blocking-Mode Generation Results

Full-model QAIC runs for Qwen3-VL-MoE 30B, Qwen3.5-MoE 35B-A3B, GPT-OSS 20B, and Gemma 4 E2B. This summarizes the retained result JSON/QEff logs as of November 5, 2026. Missing or cleaned artifacts are called out rather than treated as passing inference.

Configuration at a Glance

Model	HF model ID	Devices / cores	Weight-free	MXFP6	ONNX subfunctions	Context / prefill chunk	Generation
Qwen3-VL-MoE 30B	Qwen/Qwen3-VL-30B-A3B-Instruct	2 / 4	Yes	Yes	Yes	4096 / 128	50-token prefill_q replay; historical modes 20
Qwen3.5-MoE 35B-A3B	Qwen/Qwen3.5-35B-A3B	2 / 4	Yes	Yes	Yes	512 / 64	20 (all blocking modes)
GPT-OSS 20B	openai/gpt-oss-20b	2 / 16	Yes	Yes	Yes	2048 / 128	Script default: 10
Gemma 4 E2B	google/gemma-4-E2B-it	2 / 4	Yes	Yes	Yes	512 / 64	20

Blocking parameters used by the VLM runs:

Mode	Parameters
prefill_q	num_q_blocks: 2
prefill_kv	num_kv_blocks: 2, headpar_split: 1
prefill_qkv	num_q_blocks: 2, num_kv_blocks: 2, headpar_split: 1

The GPT-OSS parity script uses the same Q/K/KV block settings, with expert_parallel_chunk_size: 256; the VLM MoE runs use expert chunk size 16 where applicable.

Results

Qwen3-VL-MoE 30B

• Image: https://picsum.photos/id/237/536/354
• Prompt: Describe all the colors seen in the image.
• prefill_q: 50-token generation passed using retained vision/prefill QPCs and a newly recovered decode QPC. No export or compile was repeated for this generation; decode-only recovery compile took 3110.635 seconds. Earlier export/component-compile timings: vision 1236.350 / 11.439 s; prefill 2588.998 / 1165.794 s; decode export 156.987 s. A separate inference-latency value was not recorded.
• prefill_kv, prefill_qkv: Earlier report records show successful 20-token runs with the same initial text as prefill_q. Their per-mode raw result files are no longer retained, so those runs were not revalidated here.
• Generation (prefill_q, 50 tokens):

Based on the image provided, the following colors are visible:

-   **Black:** This is the dominant color, seen in the fur of the puppy.
-   **Brown:** The wooden planks in the background are a warm, medium brown,

• Artifact: qwen3vlmoe30b_wf_mxfp6_subfunc_2dev_blocking_20261105_101537/run/qwen3-vl-moe-30b/prefill_q/generation_50_tokens.json (local artifact root: /home/sabhis/blocking/qeff_artifacts).

Qwen3.5-MoE 35B-A3B

• Image: https://picsum.photos/id/237/536/354
• Prompt: Descibe the image in details.
• Configuration: 2 devices, 4 cores, weight-free, MXFP6, ONNX subfunctions; same prompt and image for all three blocking modes.
• 20-token generation: All modes passed and generated identical token IDs.

The user wants a detailed description of the image.

1.  **Subject:** A black puppy

• Original stage times (seconds; export / compiler component; inference):

Mode	Vision	Prefill	Decode	Inference
prefill_q	272.404 / 10.109	2933.993 / 1821.984	464.459 / 1945.623	98.311
prefill_kv	561.277 / 33.028	2968.903 / 1853.869	452.359 / 549.544	67.630
prefill_qkv	74.779 / 10.227	2771.784 / 1868.661	446.954 / 141.327	66.498

• Artifacts: Original 20-token results and QPCs: /home/sabhis/blocking/qeff_artifacts/qwen35b_a3b_wf_mxfp6_subfunc_2dev_blocking_20261105_120706/run/qwen3.5-35b-a3b.

GPT-OSS 20B

• Text-only prompt (parity script default): The largest planet in our solar system is
• Configuration: 2 devices, 2 MDP partitions, 16 cores, context 2048, prefill chunk 128, generation length 10, weight-free, MXFP6, MXINT8 KV cache, ONNX subfunctions, user-tiled compile.
• Mode settings: prefill_q uses 2 Q blocks; prefill_kv uses 2 KV blocks and headpar_split: 1; prefill_qkv uses 2 Q blocks, 2 KV blocks, and headpar_split: 1.
• Generated text: No output is included because no retained generation artifact or log verifies the generated text. The saved QEff log confirms model/compile activity but records Text Generation: 0.000; the original run artifact directory has been cleaned.

Gemma 4 E2B

• Image: Golden Gate Bridge photo at file:///home/sabhis/blocking/qeff_artifacts/gemma4e2b_wf_mxfp6_subfunc_2dev_20261105_101051/run/gemma-4-e2b/input_image.jpg
• Prompt: Can you Describe this image in detail?
• Configuration: 2 devices, 4 cores, weight-free, MXFP6, ONNX subfunctions.
• Generation: prefill_q, prefill_kv, and prefill_qkv all passed at 20 tokens and produced identical text:

This is a stunning, long-exposure photograph of the **Golden Gate Bridge** at twilight or dawn

• Stage times for the original runs (seconds; export / compiler component / inference):

Mode	Vision	Prefill	Decode	Inference
prefill_q	67.270 / 8.244	568.269 / 499.356	151.697 / 87.532	29.752
prefill_kv	65.674 / 8.426	162.480 / 504.662	147.290 / 88.978	29.685
prefill_qkv	72.636 / 8.278	200.738 / 149.783	137.333 / 149.308	45.398

[Rishin Raj (quic-rishinr)]

CI-Ready