--stream-experts only achieves ~5% of available SSD bandwidth on M1 Ultra (300 MB/s observed vs 5-7 GB/s capable)

[ericjlake]

Summary

Running Qwen3.5-122B-A10B-4bit with --stream-experts --ssd-prefetch on an M1 Ultra 64GB produces only 0.6 tok/s generation speed. Profiling shows SSD I/O is only ~300 MB/s — roughly 5% of the M1 Ultra's internal NVMe capacity (5–7 GB/s). The drive is not the bottleneck; something in the expert streaming pipeline is.

Hardware

• Machine: Apple M1 Ultra, 64GB unified memory, macOS
• SSD: Internal NVMe — confirmed 5–7 GB/s sequential read via dd / fio
• SwiftLM version: b253

Model

• Model: Qwen3.5-122B-A10B-4bit (MoE, ~10B activated params per token)
• Weight files: 69.6 GB (4-bit quantized)
• Source: mlx-community on HuggingFace, converted with mlx-vlm 0.3.12

SwiftLM --info output

Strategy:     ⚠️  SWAP-ASSISTED
Overcommit:   1.32× (model is 32% larger than RAM)
GPU layers:   24/32
Est. speed:   ~3 tok/s

Launch command

SwiftLM \
  --model /path/to/Qwen3.5-122B-A10B-4bit \
  --port 8000 \
  --stream-experts \
  --ssd-prefetch

Observed metrics during generation

Metric	Value
Generation speed	0.6 tok/s
Disk I/O (iostat disk0)	~300 MB/s
System-wide memory free %	78–80%
Swap activity (swapins)	0 — no swap at all
CPU utilization	~55%

The fact that swap is zero and 78% RAM is free confirms --stream-experts is working correctly — the model is streaming from SSD rather than hitting macOS swap. However, the SSD is barely being used.

The bandwidth gap

Available SSD bandwidth:    5,000–7,000 MB/s
Observed during generation:       ~300 MB/s
Utilization:                        ~5%

Per-token math for a 10B-activated-param MoE at 4-bit:

• ~5 GB of expert weights need to be accessed per token
• At full SSD speed (5 GB/s): theoretical ~1 tok/s (ignoring prefetch overlap)
• With good prefetch overlap: 3–5 tok/s should be achievable
• Actual: 0.6 tok/s

Even reaching 2–3 GB/s of SSD utilization (well under max) should yield 3–4 tok/s based on linear scaling from our observed baseline.

What we tried

• ulimit -n 4096 before launch — no change in I/O throughput or tok/s
• Both with and without --ssd-prefetch — marginal difference

Hypothesis

The 16-worker prefetch pool (--ssd-prefetch) may be serializing at the MLX memory allocator level rather than issuing parallel disk reads. Each worker allocates GPU/unified memory for the incoming expert shard, and if that allocation path holds a global lock, the reads effectively become sequential regardless of worker count. The SSD is capable of much higher throughput but never gets a chance to saturate because the consumer side is bottlenecked.

Evidence: CPU is only at 55% during generation. A properly pipelined SSD→GPU streaming path should drive CPU closer to 80–90% as the prefetch workers stay busy.

Request

1. Can you confirm whether the prefetch workers are issuing reads concurrently or effectively sequentially?
2. Is there a flag to increase prefetch depth or worker count beyond the current 16?
3. Is this a known limitation of the MLX tensor allocation path on Apple Silicon, and if so, is there a fix planned?

This hardware is otherwise well-suited for this model — plenty of SSD bandwidth, no memory pressure, no swap. Getting even 3–4 tok/s would make 122B viable for async/background inference use cases.

Environment

• macOS (Apple Silicon)
• SwiftLM b253
• MLX (version bundled with SwiftLM b253)
• Model: mlx-community/Qwen3.5-122B-A10B-4bit

[ericjlake]

Follow-up: b259 tested on M1 Ultra 64GB — still 0.6 tok/s

Hardware: M1 Ultra 64GB, 1TB SSD, macOS 26.3 (Tahoe)
SwiftLM version: b259 (re-signed locally — see note below)
Result: 0.6 tok/s, ~300 MB/s SSD — identical to original report

New diagnostic: GPU is NOT at 100%

This is the key finding from b259 testing. The GPU sits idle most of the time waiting for expert weights to arrive. Classic prefetch starvation — not a GPU compute bottleneck.

With only 10B active params per token at 4-bit, the GPU finishes its computation in microseconds then blocks waiting for the next expert batch from SSD. The prefetch pipeline is not keeping the GPU fed.

Why the b259 fix didn't transfer to M1 Ultra

The b107/b259 fix (tensor_name cache key) solves cache collisions in the prefetch offset cache — a correctness fix. The issue on M1 Ultra appears to be I/O concurrency, not cache correctness.

If the 16-worker prefetch pool issues blocking pread syscalls sequentially rather than concurrently, the NVMe controller hardware queue (32+ deep) sits mostly empty. You get ~300 MB/s latency-bound throughput instead of 5+ GB/s bandwidth-bound.

M1 Ultra UltraFusion hypothesis

M1 Ultra is two M1 Max dies connected via UltraFusion. Each die has its own memory controller for its 32GB slice. If MLX allocates all expert weight buffers in one die's address space, all SSD DMA serializes through that die's single memory controller — potentially capping throughput at 2-3 GB/s even if the SSD hardware can do 5-7 GB/s.

The M5 Pro benchmarks in the README are single-die — this architectural split doesn't exist there. This may explain why the fix works on M5 Pro but not M1 Ultra.

What a real fix likely needs

1. Concurrent async I/O: Replace blocking pread with GCD-based async I/O (dispatch_io_create with DISPATCH_IO_RANDOM) to issue multiple NVMe commands concurrently without blocking worker threads
2. Memory allocation striping: Distribute expert weight buffers across both M1 Ultra dies to use both memory controllers
3. Metal-shared buffer DMA: Allocate expert weight buffers as MTLStorageModeShared so DMA goes directly to GPU-accessible memory, eliminating one copy

macOS 26 (Tahoe) note

b259 binary gets SIGKILL on macOS 26 — Gatekeeper rejects it without an approval dialog. Workaround: codesign --force --sign - /path/to/SwiftLM after downloading. Worth noting in the README for Tahoe users.

Happy to test debug builds or patches targeting the I/O concurrency issue.

[solderzzc]

@ericjlake , thanks for your information, I'm currently working on the following:

SSD Expert Streaming Bottleneck Analysis

Qwen3.5-122B-A10B-4bit on M1 Ultra 64GB — 0.6 tok/s Investigation

---

1. Executive Summary

The user reports 0.6 tok/s generation speed with only ~300 MB/s SSD utilization (~5% of the M1 Ultra's internal NVMe capacity of 5–7 GB/s). After a thorough code audit and literature review, the bottleneck is not a single chokepoint — it is a cascade of five serialization barriers that compound to throttle the entire pipeline.

Important

The core insight: SwiftLM currently processes experts one at a time, fully synchronously, with a GPU pipeline flush after every single layer. Even with infinite SSD bandwidth, this architecture would still be slow because the pipeline is never overlapping I/O with compute.

---

2. The Qwen3.5-122B-A10B Architecture

Property	Value
Total Parameters	122B
Activated Parameters	~10B per token
Layers	48
Experts per Layer	256
Routed Experts per Token	8 + 1 shared
Hidden Dimension	3072
Architecture	Hybrid: 3×(Gated DeltaNet → MoE) → 1×(Gated Attention → MoE)
Weight Files	69.6 GB (4-bit quantized)

Per-Token I/O Math

Each MoE layer activates 8 routed experts. Each expert has 3 weight matrices (gate_proj, up_proj, down_proj). At 4-bit quantization with 256 experts, each expert's share of a single weight tensor is:

bytes_per_expert ≈ total_tensor_bytes / 256

For a typical MoE layer in this model:

• ~3 weight matrices × 8 experts = 24 pread() calls per MoE layer
• ~36 MoE layers × 24 calls = ~864 pread() calls per token
• Total data per token ≈ 5 GB (consistent with the user's estimate)

---

3. The Five Serialization Bottlenecks

Bottleneck #1: Sequential Expert Loop (CRITICAL)

File: SwitchLayers.swift:212-286

The hot path is a while startIdx < cpuIndices.count loop that processes experts one at a time:

while startIdx < cpuIndices.count {
    let currentExpert = Int(cpuIndices[startIdx])
    // ... resolve expert range ...
    
    // BLOCKING: allocate, read from SSD, wait for completion
    let w = MLXArray.zeros([1, self.weight.dim(1), self.weight.dim(2)])
    MLX.eval(w)                          // ← GPU sync #1: allocate
    MLXFast.preadInto(w, ...)            // ← SSD read (blocking pread())
    
    // BLOCKING: compute matmul for this single expert
    var expertOutput = MLX.gatherQuantizedMM(rangeX, expertWeight, ...)
    MLX.eval(expertOutput)               // ← GPU sync #2: compute
    Stream.gpu.synchronize()             // ← GPU sync #3: hard barrier
    
    expertResults.append(expertOutput)
    startIdx = endIdx                    // → next expert (sequential!)
}

Impact: For 8 experts per layer, this loop executes 8 iterations of:
allocate → pread → eval → synchronize.
No overlap between I/O and compute. No parallelism between experts.

Bottleneck #2: Per-Layer GPU Synchronization Barrier (CRITICAL)

File: LayerPartitioning.swift:130-139

After every single transformer layer (not just MoE layers), the pipeline forces:

if isSsdStream, let array = result as? MLXArray {
    eval(array)
    Stream.gpu.synchronize()   // ← HARD BARRIER: CPU waits for GPU to drain
    return result
}

This exists to prevent Apple's 5-second GPU Watchdog timeout (kIOGPUCommandBufferCallbackErrorTimeout). The Gated DeltaNet custom Metal kernels on consecutive linear-attention layers can accumulate into oversized command buffers.

Impact: This prevents MLX's lazy graph evaluator from batching work across layers. Each of the 48 layers is individually flushed, creating 48 synchronization points per token.

Bottleneck #3: Synchronous pread() with No Prefetching (HIGH)

File: ssd_streamer.mm:41-57

The SSDStreamer::load_sync() function is a bare, blocking POSIX pread():

ssize_t result = pread(fd_, dst_ptr, length, byte_offset);

• No F_NOCACHE (reads go through macOS page cache — beneficial for hot experts, but adds kernel overhead)
• No preadv() scatter/gather (each expert matrix is a separate syscall)
• No asynchronous dispatch (despite having a dispatch_queue_t io_queue_ member that is never used for I/O)
• No posix_fadvise() equivalent (rdadvise on macOS) to hint the kernel about upcoming reads

Impact: Each of the ~864 pread() calls per token is a blocking syscall. The NVMe controller's internal parallelism (typically 64+ queue depth) is never exploited because we issue one read at a time.

Bottleneck #4: MLX Allocator Mutex Contention (MEDIUM)

File: mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/allocator.cpp

Every expert read requires a fresh allocator::malloc() for the destination buffer:

o.set_data(allocator::malloc(matrix_bytes));  // inside LoadSSDExpert::eval_impl

The MLX Metal allocator holds a global mutex (std::unique_lock lk(mutex_)) during allocation. While the lock duration is short (microseconds), it becomes a contention point if we ever move to concurrent expert loading, since all 16 PAPPS workers would serialize on this lock.

Impact: Currently low because experts are loaded sequentially. Becomes critical if we implement concurrent loading.

Bottleneck #5: PAPPS Prefetch Pool is Dormant (INFORMATIONAL)

File: SwitchLayers.swift:294-303

The predictive N+1 prefetch heuristic was correctly disabled after our PAPPS research showed it fights MoE router entropy:

// PAPPS Heuristic: Prefetch exactly these experts so they are in cache for the N+1 token.
if let info = ssdInfo {
    let uniqueIndices = Set(cpuIndices)
    for _ in uniqueIndices {
        // MLXFast.pappsPrefetch(...)  ← COMMENTED OUT
    }
}

The 16-thread PAPPSQueue in fast.cpp is allocated but receives zero work items. The infrastructure exists but is unused.

---

4. Literature Review: State of the Art

4.1 Apple's "LLM in a Flash" (arXiv:2312.11514)

Apple's own research paper introduces two key techniques for flash/SSD inference:

Technique	Description	Applicability to SwiftLM
Windowing	Reuse previously activated neurons across adjacent tokens to reduce I/O volume	✅ Directly applicable — MoE routing shows temporal locality (~60-70% expert reuse between adjacent tokens for many models)
Row-Column Bundling	Store up_proj row + down_proj column together so one pread() fetches both	⚠️ Requires safetensors relayout — not applicable without a custom converter

Tip

Key insight from Apple's paper: The paper explicitly states that reading data in "larger, more contiguous chunks" is critical. SwiftLM currently issues separate pread() calls for gate_proj, up_proj, and down_proj of each expert — 3 syscalls where 1 bundled read would suffice.

4.2 Pre-gated MoE (ISCA 2024)

The "Pre-gated MoE" paper from Hwang et al. proposes decoupling expert selection from expert execution:

• Run the router gate for Layer N+1 while Layer N is still computing
• Use the predicted expert indices to prefetch from SSD/DRAM
• This hides I/O latency behind compute latency

Applicability: This is exactly what PAPPS attempted with N+1 prediction, but the key difference is that Pre-gated MoE uses the actual router weights for prediction (not a heuristic guess). Since MoE routing is deterministic given the input, running the router one layer ahead is a valid prediction with 100% accuracy (not the random guessing PAPPS did).

4.3 llama.cpp Two-Tier Expert Cache (PR #20757)

The llama.cpp community is implementing:

Tier	Location	Policy	Purpose
Tier 1	GPU VRAM	SLRU (Segmented LRU)	Keep hot experts resident, avoid re-reading
Tier 2	Pinned CPU RAM	LRU	Fast backing store for cache misses
Tier 3	SSD/mmap	Demand-paged	Cold storage

Key optimization: Uses POSIX_MADV_WILLNEED for readahead and POSIX_MADV_DONTNEED for releasing evicted pages.

4.4 SliceMoE: Bit-Sliced Expert Caching (arXiv:2512.12990)

Proposes partial caching — keeping the most significant bits of hot experts in GPU memory while fetching lower-precision "residual" bits from SSD. This reduces per-miss I/O volume by ~50%.

4.5 MoE Expert Routing Locality

Research consistently shows that MoE routing exhibits strong temporal locality:

• Adjacent tokens frequently route to the same experts (60-70% overlap)
• Certain "universal" experts handle disproportionate traffic (Zipfian distribution)
• This means a hot expert cache would have high hit rates even with modest size

---

5. The Pipeline Today (Visual)

sequenceDiagram
    participant Router as MoE Router
    participant Swift as Swift Loop
    participant Alloc as MLX Allocator
    participant SSD as NVMe SSD
    participant GPU as Metal GPU

    Note over Router,GPU: Per MoE Layer (×36 layers per token)
    
    Router->>Swift: Selected experts [e1, e2, ..., e8]
    
    loop For each expert (sequential!)
        Swift->>Alloc: malloc(expert_bytes)
        Alloc-->>Swift: buffer_ptr
        Swift->>SSD: pread(fd, buffer_ptr, size, offset)
        Note over SSD: BLOCKING — CPU waits
        SSD-->>Swift: data ready
        Swift->>GPU: gatherQuantizedMM(x, expert_weight)
        Swift->>GPU: eval(result)
        Swift->>GPU: synchronize() ← HARD BARRIER
        Note over GPU: GPU IDLE during pread()
        Note over SSD: SSD IDLE during GPU compute
    end
    
    Swift->>GPU: eval(layer_output)
    Swift->>GPU: synchronize() ← ANOTHER HARD BARRIER

Loading

The fundamental problem is visible: SSD and GPU are never active simultaneously. They take turns.

---

6. Proposed Optimization Tiers

Tier 1: Hot Expert Cache (Estimated: 0.6 → 1.5-2.0 tok/s)

Important

Lowest engineering risk, highest immediate impact.

Exploit MoE routing locality by caching recently-used expert weights in a unified-memory LRU cache. Research shows 60-70% expert reuse between adjacent tokens.

Implementation:

• Maintain an LRU cache of [String: MLXArray] keyed by tensorName + expertIndex
• Cache size: ~2-4 GB (configurable, fits easily in the 50+ GB of free RAM the user reports)
• On hit: skip pread() entirely — direct pointer reuse
• On miss: pread() + insert into cache, evict coldest entry

Why this works for Qwen3.5-122B:

• 256 experts per layer, but only 8+1 activated per token
• With 48 layers, many experts will be reused across consecutive tokens
• Even a 30% hit rate eliminates ~260 of the ~864 pread() calls per token

Tier 2: Parallel Expert Loading (Estimated: 1.5 → 3.0-4.0 tok/s)

Once the router selects the 8 experts for a layer, all 8 can be loaded concurrently:

Implementation:

// Instead of sequential loop:
let group = DispatchGroup()
var expertWeights = [Int: MLXArray]()
let lock = NSLock()

for expertIdx in selectedExperts {
    group.enter()
    prefetchQueue.async {
        let w = // allocate + pread for expertIdx
        lock.withLock { expertWeights[expertIdx] = w }
        group.leave()
    }
}
group.wait()  // All 8 experts loaded in parallel

Challenges:

• MLX allocator mutex: needs a pool allocator or pre-allocated ring buffer
• Must use F_NOCACHE via fcntl(fd, F_NOCACHE, 1) to prevent kernel page cache contention when 8 threads hit the same file
• NVMe queue depth: M1 Ultra controller supports 64+ concurrent requests — 8 parallel reads should saturate beautifully

Tier 3: Pipelined Layer-Ahead Routing (Estimated: 3.0 → 4.0-5.0 tok/s)

Implement the Pre-gated MoE technique — run Layer N+1's router while Layer N's experts are still computing on GPU:

Implementation:

1. After Layer N's router outputs expert indices for Layer N, begin GPU compute for Layer N
2. Simultaneously, evaluate Layer N+1's router gate (small matmul) on CPU
3. Use Layer N+1's predicted expert indices to begin prefetching from SSD
4. By the time Layer N finishes on GPU, Layer N+1's experts are already in memory

This is fundamentally different from PAPPS N+1 heuristic because:

• PAPPS guessed which experts would be needed (random accuracy)
• Pre-gated MoE computes which experts will be needed (100% accuracy, just one layer early)

Challenge: Requires refactoring partitionedLayerCall to support overlapped execution, and relaxing the per-layer synchronize() barrier (replacing it with Metal shared events for fine-grained synchronization).

---

7. Why --ssd-prefetch Shows No Improvement

The --ssd-prefetch flag enables the PAPPS 16-worker thread pool (PAPPSQueue), but the submit path is commented out in SwitchLayers.swift:294-303. Even if it were active:

1. The N+1 prediction heuristic has ~3% accuracy for 256-expert models (essentially random)
2. Wrong predictions cause I/O contention that slows down correct reads
3. The try_take() consumer does a memcpy() from the prefetch buffer to the MLX buffer — defeating zero-copy

The flag currently has zero functional effect on inference performance.

---

8. Why CPU Is at 55% (Not Higher)

The user observes only 55% CPU utilization. This is consistent with the serialization analysis:

Phase	CPU Active?	SSD Active?	GPU Active?
pread() syscall	✅ Waiting in kernel	✅ Reading	❌ Idle
allocator::malloc()	✅ Brief	❌ Idle	❌ Idle
eval() + synchronize()	✅ Dispatching commands	❌ Idle	✅ Computing
Between layers	✅ Swift control flow	❌ Idle	❌ Draining

CPU is at 55% because it spends ~45% of its time blocked — either waiting for pread() to return or waiting for GPU synchronize() to complete.

---

9. Theoretical Throughput Ceiling

Scenario	SSD Utilization	Expected tok/s
Current (sequential, no cache)	~300 MB/s	0.6
+ Hot Expert Cache (30% hit rate)	~300 MB/s but fewer reads	1.0-1.5
+ Parallel Loading (8 concurrent pread)	~2-3 GB/s	2.5-3.5
+ Pipelined Routing (overlap I/O + compute)	~3-4 GB/s	3.5-5.0
Theoretical maximum (perfect overlap, full BW)	~5-7 GB/s	~5.0

Warning

The absolute ceiling of ~5 tok/s assumes perfect I/O-compute overlap with zero overhead. In practice, 3-4 tok/s is a realistic target for Tier 2+3 combined.

---

10. Recommended Next Steps

1. Instrument first: Add timing breakdowns to QuantizedSwitchLinear.callAsFunction to measure exact time spent in pread() vs. eval() vs. synchronize() per layer. This will validate the bottleneck distribution.

2. Implement Tier 1 (Hot Expert Cache): Lowest risk, immediate impact. Can be done entirely in Swift without touching C++ code.

3. Prototype Tier 2 (Parallel Loading): Requires careful work around the MLX allocator mutex, but the PAPPSQueue infrastructure already exists and can be repurposed.

4. Research Tier 3 (Pipelined Routing): Requires deeper architectural changes to the layer evaluation loop. Consider as a medium-term project.

---

References

• Apple Inc. "LLM in a Flash: Efficient Large Language Model Inference with Limited Memory" (arXiv:2312.11514)
• Hwang et al. "Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference" (ISCA 2024)
• SliceMoE: "Bit-Sliced Expert Caching under Miss-Rate Constraints" (arXiv:2512.12990)
• MoE-SpeQ: "Speculative Quantized Decoding with Proactive Expert Prefetching" (arXiv:2511.14102)
• llama.cpp PR #20757: Two-Tier Expert Cache
• SwiftLM codebase audit: SwitchLayers.swift, moe_stream_op.cpp, ssd_streamer.mm, LayerPartitioning.swift

[solderzzc]

@ericjlake ,
Above was wrong assumptions, the real token/s boost is not as it mentioned. It's around 30% performance improvement after the following changes:

https://github.com/SharpAI/SwiftLM/blob/feature/swiftbuddy-mempalace-v1/docs/moe_ssd_streaming_architecture.md

Please check the feature branch currently I'm working on before I have them merged if you want.

[ericjlake]

I will today… thanks 🙏

…

On Thu, Apr 9, 2026 at 11:45 PM Simba ***@***.***> wrote: *solderzzc* left a comment (SharpAI/SwiftLM#24) <#24 (comment)> @ericjlake <https://github.com/ericjlake> , Above was wrong assumptions, the real token/s boost is not as it mentioned. It's around 30% performance improvement after the following changes: https://github.com/SharpAI/SwiftLM/blob/feature/swiftbuddy-mempalace-v1/docs/moe_ssd_streaming_architecture.md Please check the feature branch currently I'm working on before I have them merged if you want. — Reply to this email directly, view it on GitHub <#24 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B5TD3GOJKXTBR2WF5G6CQRL4VCKANAVCNFSM6AAAAACXTD3NCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMRRG4ZTAMZXGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ericjlake]

Update: Profiling results + 3× improvement found

Hey @solderzzc — I spent the last couple days profiling the expert streaming path in detail and found the primary bottleneck. Sharing everything here in case it's useful.

The bottleneck: MLX.eval() call overhead

Instrumented QuantizedSwitchLinear.callAsFunction() in SwitchLayers.swift and found that MLX.eval() calls consume ~77% of per-token wall time:

Phase	% of wall time
evalOutput (per-expert)	52%
evalZeros (buffer alloc)	25%
pread (actual disk I/O)	21%
Other	2%

Each MLX.eval() acquires a global mutex, creates a GPU command buffer, submits it, and waits for completion. With Qwen3.5-122B's 256 routed experts (top-8 routing, 3 projections × ~88 active experts per token), that's roughly 1,400 eval calls per token — the GPU spends most of its time in setup/teardown rather than compute.

The SSD is barely utilized not because reads are slow, but because the pipeline stalls on eval overhead between reads.

The fix: batch eval

Restructured the expert loading loop into three phases:

1. Batch-allocate all weight buffers upfront + single MLX.eval() to materialize them
2. Sequential pread into each fresh buffer (no eval between reads)
3. Lazy gatherQuantizedMM for all experts + single MLX.eval(expertResults) at the end

This collapses ~1,400 eval calls down to ~2 per projection pass.

Result: 0.58 → 1.78–2.02 tok/s (3–3.5× improvement)

Things that didn't work (so you don't have to try them)

Approach	Result	Why
Reusable single buffer	0.51 tok/s (regression)	preadInto overwrites buffer memory that lazy eval graph still references
Buffer pool + deferred eval	0.41 tok/s	Same stale-reference problem across pool slots
F_NOCACHE on fd	0.42–0.48 tok/s	Bypasses page cache but doesn't address eval overhead
Parallel I/O (GCD dispatch)	0.42–0.48 tok/s	Apple Silicon shares memory controller between SSD DMA and GPU — parallel I/O causes GPU latency spikes
F_RDADVISE (prefetch hints)	Regression	Confirmed by Flash-MoE research — prefetch hurts on Apple Silicon

Key insight from the Flash-MoE paper: on Apple Silicon, SSD DMA and GPU compute share the same memory controller and can't be profitably overlapped. Serial pipeline is hardware-optimal — the win comes from reducing per-expert overhead, not parallelizing I/O.

Hardware

• M1 Ultra 64GB, internal NVMe (5–7 GB/s capable)
• Qwen3.5-122B-A10B-4bit
• SwiftLM b253

Code

The changes are in two files in mlx-swift-lm:

• SwitchLayers.swift — restructured expert loading loop into batch-allocate → sequential pread → lazy gatherQuantizedMM → single batch eval
• LayerPartitioning.swift — removed per-expert Stream.gpu.synchronize() from the SSD stream path

Happy to open a PR against SharpAI/mlx-swift-lm, share the diff here, or whatever works best for you. The changes are pretty contained — just the callAsFunction() method and one line in the layer sync.

Still think there's more headroom (GPU utilization is up but not saturated), but 3× felt worth sharing now rather than waiting.

[solderzzc]

Happy to open a PR against SharpAI/mlx-swift-lm, share the diff here, or whatever works best for you. The changes are pretty contained — just the callAsFunction() method and one line in the layer sync.

@ericjlake I'm very excited to see you found a solution to unblock the bottleneck, pls going forward for a PR.
You can also PR to SwiftLM with documentation for this great found.

[ericjlake]

This comment is intentionally blank — used to check existing thread context.

[solderzzc]

I'll be continue working on README / CI ( Github Action ). So let's keep this opened.