compile: output_shapes overrides for Slice/Scan/SDPA (unblocks shapeless whole-step compile) - #4296
Open
jonathan308 wants to merge 1 commit into
Open
compile: output_shapes overrides for Slice/Scan/SDPA (unblocks shapeless whole-step compile)#4296jonathan308 wants to merge 1 commit into
jonathan308 wants to merge 1 commit into
Conversation
…s shapeless whole-step compile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mx.compile(f, shapeless=True)replaces shape-specialized retracing with shape-generic replay, butcompile_replaceneedsprimitive().output_shapes(real_inputs)per tape node and several primitives throw — the first beingSlice([Primitive::output_shapes] Slice cannot infer output shapes.). This blocks compiling an LLM decode step as one unit (the KV cache slice changes shape every token → per-token retrace today).This PR adds the missing overrides needed by a decode-step graph:
Slice::output_shapes— replicates thenormalize_slicearithmetic from ops.cpp exactly (normalized start, unnormalized end, normalized strides); exact for cache-growth replays (shape only grows, positive starts). Documented limitation: slices created with negative start replay with the trace-time-baked value (pre-existing storage design; out of scope). Negative-strided slices with positive starts recompute exactly.Scan::output_shapes— shape-preserving (cumsum & co.); needed by the decode-mask pattern.ScaledDotProductAttention::output_shapes— wasDEFINE_INPUT_OUTPUT_SHAPE()(= q shape; wrong whenhead_dim(v) != head_dim(qk), and missing the logsumexp second output). Nowq.shape[:-1] + v.shape[-1:]plus the[..., 1]LSE output when enabled — matches the call-site construction infast.cpp.SliceUpdate,RMSNorm,RoPEalready have correct overrides on main — verified by auditing all 128 primitive classes; no change.Acceptance (Qwen3-0.6B-4bit decode step: offset-as-array,
slice_updatewrites, full-buffer attention + cumsum mask,inputs=/outputs=state threading; 230-token prefill + 40 decode steps crossing the 256→512 cache-growth boundary)shapeless=True: 1 trace total, zero recompiles across growth, 40/40 tokens bit-identical to eager. (Pre-patch: throws on first compiled call.)shapeless=False: 2 traces (1 per growth), 40/40 bit-identical — unchanged behavior.Two latent trace-time shape-baking traps were found and documented during acceptance (omitted-stop slices bake the trace-time axis size;
broadcast_toof an under-sized attention mask bakes aBroadcastnode that silently corrupts post-growth shapes in fused regions) — the acceptance recipe avoids both; fixing the defaults is out of scope (needs sentinel defaults in indexing.cpp).Safety
The patch only adds shape-inference code paths used during
compile(shapeless=True)replay; no eval kernel, op semantics, or eager path is touched.python/tests/test_compile.py+test_fast.py+test_ops.py: 254 passed, 327 subtests passed, 0 failed; 6 new tests added (slice arithmetic, negative strides, cumsum, decode-mask, sdpa incl. mismatched v head-dim).Scope note
This is deliberately an enabler with a correctness contract, not a perf claim: on M3 Ultra, compiled decode is currently speed-neutral (GPU-bound), but shapeless compile removes per-256-token retraces and is a prerequisite for future graph-replay work. Follow-ups (not in this PR):
Pad,Scatter,Split,AsStrided,View,Dependsoverrides; VJP primitives remain unsupported under shapeless.