Skip to content

compile: output_shapes overrides for Slice/Scan/SDPA (unblocks shapeless whole-step compile) - #4296

Open
jonathan308 wants to merge 1 commit into
ml-explore:mainfrom
jonathan308:pr/compile-output-shapes
Open

compile: output_shapes overrides for Slice/Scan/SDPA (unblocks shapeless whole-step compile)#4296
jonathan308 wants to merge 1 commit into
ml-explore:mainfrom
jonathan308:pr/compile-output-shapes

Conversation

@jonathan308

Copy link
Copy Markdown
Contributor

mx.compile(f, shapeless=True) replaces shape-specialized retracing with shape-generic replay, but compile_replace needs primitive().output_shapes(real_inputs) per tape node and several primitives throw — the first being Slice ([Primitive::output_shapes] Slice cannot infer output shapes.). This blocks compiling an LLM decode step as one unit (the KV cache slice changes shape every token → per-token retrace today).

This PR adds the missing overrides needed by a decode-step graph:

  • Slice::output_shapes — replicates the normalize_slice arithmetic from ops.cpp exactly (normalized start, unnormalized end, normalized strides); exact for cache-growth replays (shape only grows, positive starts). Documented limitation: slices created with negative start replay with the trace-time-baked value (pre-existing storage design; out of scope). Negative-strided slices with positive starts recompute exactly.
  • Scan::output_shapes — shape-preserving (cumsum & co.); needed by the decode-mask pattern.
  • ScaledDotProductAttention::output_shapes — was DEFINE_INPUT_OUTPUT_SHAPE() (= q shape; wrong when head_dim(v) != head_dim(qk), and missing the logsumexp second output). Now q.shape[:-1] + v.shape[-1:] plus the [..., 1] LSE output when enabled — matches the call-site construction in fast.cpp.

SliceUpdate, RMSNorm, RoPE already have correct overrides on main — verified by auditing all 128 primitive classes; no change.

Acceptance (Qwen3-0.6B-4bit decode step: offset-as-array, slice_update writes, full-buffer attention + cumsum mask, inputs=/outputs= state threading; 230-token prefill + 40 decode steps crossing the 256→512 cache-growth boundary)

  • shapeless=True: 1 trace total, zero recompiles across growth, 40/40 tokens bit-identical to eager. (Pre-patch: throws on first compiled call.)
  • shapeless=False: 2 traces (1 per growth), 40/40 bit-identical — unchanged behavior.

Two latent trace-time shape-baking traps were found and documented during acceptance (omitted-stop slices bake the trace-time axis size; broadcast_to of an under-sized attention mask bakes a Broadcast node that silently corrupts post-growth shapes in fused regions) — the acceptance recipe avoids both; fixing the defaults is out of scope (needs sentinel defaults in indexing.cpp).

Safety

The patch only adds shape-inference code paths used during compile(shapeless=True) replay; no eval kernel, op semantics, or eager path is touched. python/tests/test_compile.py + test_fast.py + test_ops.py: 254 passed, 327 subtests passed, 0 failed; 6 new tests added (slice arithmetic, negative strides, cumsum, decode-mask, sdpa incl. mismatched v head-dim).

Scope note

This is deliberately an enabler with a correctness contract, not a perf claim: on M3 Ultra, compiled decode is currently speed-neutral (GPU-bound), but shapeless compile removes per-256-token retraces and is a prerequisite for future graph-replay work. Follow-ups (not in this PR): Pad, Scatter, Split, AsStrided, View, Depends overrides; VJP primitives remain unsupported under shapeless.

@zcbenz zcbenz added the await verification This pull request is non-trivial and requires a human expert to verify its correctness. label Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

await verification This pull request is non-trivial and requires a human expert to verify its correctness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants