perf(whisper): remove long-form decode overhead

[unohee]

Summary

• vectorize timestamp-rule masking on MLX while preserving current output semantics
• replace the SciPy median-filter hot path with an exact MLX implementation
• cache Numba DTW compilation and merge alignment evaluation sync points
• reuse encoded audio features across temperature fallback attempts

Exact-output benchmark

Fixed MLX seed, 71-minute production recording, word timestamps enabled:

• base: 127.0028s
• candidate: 110.0909s
• improvement: 13.31% lower latency, 1.154x speedup
• output: 1,755 segments, 7,205 words, 26,343 chars
• exact JSON SHA-256 on both sides: a6b25f7a526e47193c77324b45a4bb8cfe1e717cffe63c1c852a17df2dbd3a26

The benchmark used MLX_METAL_FAST_SYNCH=1; the code-only fallback baseline was 127.0028s and the candidate retained full JSON parity.

Rejected candidates

• compiled timestamp-rule masks: catastrophic output collapse
• eager completion sync: exact but slower
• 256-token chunked KV cache: 71-minute regression
• selected-token-only logprob work: no measurable gain
• token-value timestamp lower bound: changes output and is split into KT-642

Review

OpenSwarm Codex reviewer, 600,000ms timeout: APPROVE. The reviewer-found filter_width=1 boundary regression was fixed before approval.

Tracking

• KT-641
• KT-642
• stacked on the fallback feature-reuse branch from upstream PR #1432

---

Note

Medium Risk
Touches autoregressive timestamp logit rules and word-alignment numerics on a hot path; changes are tuned for exact JSON parity but regressions would affect segment/word timestamps and fallback decoding behavior.

Overview
Cuts long-form MLX Whisper latency (~13% in the PR benchmark) while keeping bit-identical transcript JSON on a 71-minute word-timestamp run.

ApplyTimestampRules stops per-batch Python/tolist() work and builds timestamp suppression with vectorized MLX boolean masks (cached vocab indices). Legacy position-based timestamp lower-bound behavior is explicitly preserved for parity; token-value semantics are deferred to KT-642.

Word-alignment path: median_filter drops SciPy for an MLX reflect-pad + partition median; dtw_cpu gets Numba cache=True; find_alignment batches GPU sync with mx.eval before NumPy conversion.

Temperature fallback in decode_with_fallback reuses encoded audio_features from the first model.decode instead of re-running the encoder on each temperature retry.

^{Reviewed by Cursor Bugbot for commit 4d1120c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[unohee]

Final ceiling audit:

• 71-minute fixed-seed exact result: 127.0028s -> 110.0909s (13.31% lower)
• Metal trace wrapper: 8.0652s, exact SHA preserved
• GPU active union/span: 5.6798s / 6.4250s (88.40%)
• 7,107 sub-0.1ms dependency gaps total 214.39ms, only 2.66% of traced runtime even at a zero-cost theoretical bound
• OpenSwarm Codex review: APPROVE
• production integration merged in https://github.com/unohee/whisper-diarization/pull/3

KT-641 is complete; correctness-changing timestamp semantics remain isolated in KT-642.