Kernel panic (IOGPUMemory.cpp:550) on M4 Max with large context prefill (~173K tokens)

[kotono-amaha]

Apple Feedback ID: FB22091885

This issue has been filed with Apple and is cross-referenced here for the MLX community. A fix may come from either side.

Kernel panic in IOGPUMemory.cpp:550 triggered by large Metal GPU memory allocation during MLX inference on M4 Max.

PANIC STRING:
"completeMemory() prepare count underflow" @IOGPUMemory.cpp:550

SYSTEM:

• Hardware: Apple M4 Max (36GB unified memory)
• macOS: 26.3 (25D125)
• Kernel: Darwin 25.3.0 xnu-12377.81.4~5/RELEASE_ARM64_T6041

REPRODUCIBLE: Yes — confirmed twice with identical call stacks.

REPRODUCTION STEPS:

1. Install MLX and mlx-lm via pip on Python 3.14 ARM64
2. Load a large quantized LLM (Qwen3.5-27B Q5_K_M) via mlx_vlm.load()
3. Construct a prompt consisting of 147 concatenated model outputs totalling approximately 173,000 tokens
4. Call mlx_vlm.generate() with this prompt — prefill phase begins processing the full context
5. Kernel panics during prefill, consistently at IOGPUMemory.cpp:550

ROOT COMPONENT:
com.apple.iokit.IOGPUFamily (129.3.2)

NOTES:

• Panic does not occur with smaller prompts (under ~10,000 tokens)
• Memory capacity is not the issue — system has 36GB and model occupies ~26GB, leaving sufficient headroom
• Issue appears to be a GPU memory accounting state corruption triggered
  by a single contiguous Metal allocation for a very large attention computation, not an out-of-memory condition
• Two panic logs attached with identical backtraces confirming deterministic reproducibility

Suggested mitigation for MLX:
Add a prefill token count guard in mlx_lm before the Metal allocator is called. If the prompt exceeds a safe threshold (empirically somewhere
below 173K tokens on M4 Max with 36GB), either raise a clear exception with guidance to chunk the prompt, or automatically split the prefill
into safe-sized segments. This would prevent the IOGPUFamily kernel panic without requiring a macOS fix.

[AlchemyHAN]

I’ve run into the same issue on a MacBook M5 with 32GB RAM using the 5-bit MLX version of Qwen 3 Coder. Once the token count gets large enough, it consistently kernel panics and the system reboots to a black screen. It doesn’t look like this is limited to the M4 Max.

[Harperbot]

Adding a production data point. We saw 9 kernel panics in 6 days on M1 Ultra (64GB) with the completeMemory() prepare count underflow signature, which is closely related to this fPendingMemorySet race.

Through crash forensics (breadcrumb logs written with fsync before each Metal operation), we identified two trigger paths:

1. Daemon thread race — mlx_lm.generate() in a thread holding GPU buffers + mx.clear_cache() on main thread before thread finishes
2. Cross-process Metal contention — two Python processes both calling MLX generate simultaneously

We built MetalGuard (v0.4.0) as an application-level safety layer. It doesn't fix the driver bug, but it eliminates the trigger paths:

• Thread registry + wait_for_threads() — blocks cleanup until GPU threads finish
• Cross-process lock — prevents concurrent MLX workloads across processes
• Breadcrumb forensics — metal_guard.breadcrumb("LOAD: model-x START") writes fsync'd lines so after a panic you can see exactly which operation triggered it
• is_metal_oom() now detects the fPendingMemorySet signature specifically

@yoyaku155 — if you're still seeing this on M3 Ultra, MetalGuard may help. After deploying it on our M1 Ultra pipeline (90+ model load/unload cycles per batch), we went from 9 panics/week to zero.

pip install metal-guard

Single file, MIT licensed: https://github.com/Harperbot/metal-guard

[ronm92130]

Adding a corroborating data point from a new hardware class — Mac mini M4 base (32 GB unified memory) — with a much smaller prefill size than the original report.

System

• Hardware: Mac mini M4 base — 10 cores (4P + 6E), 32 GB unified memory
• macOS: 26.4.1 (Build 25E253)
• Kernel: Darwin 25.4.0 (xnu-12377.101.15~1/RELEASE_ARM64_T8132)
• IOGPUFamily: 130.13
• mlx: 0.31.2
• mlx-lm: 0.31.3
• Model: mlx-community/Qwen3.6-35B-A3B-4bit (hybrid 3:1 GatedDeltaNet+Attention, ~19 GB weights)

Panic signature — exact match

panic(cpu 8 caller 0xfffffe00513bbe28): "completeMemory() prepare count underflow" @IOGPUMemory.cpp:550

Panicked task 0xfffffe1e4174a418: 367197 pages, 25 threads: pid 50218: Python
  Kernel Extensions in backtrace:
    com.apple.iokit.IOGPUFamily(130.13)[1CC2AF3D-E6EF-3DC2-B32B-8DBD8CAA09B7]

Full panic log available on request (contains no sensitive data but I didn't want to paste ~2k lines here).

Workload — ordinary throughput bench, not stress

mlx_lm.server launched with:

mlx_lm.server \
  --model /path/to/qwen3.6-35b-a3b-mlx4 \
  --host 127.0.0.1 --port 8080 \
  --max-tokens 8192 \
  --prompt-cache-size 65536 \
  --prompt-cache-bytes 8589934592 \
  --temp 0.7 --top-p 0.80 --top-k 20 \
  --chat-template-args '{"enable_thinking": false}'

Then a throughput bench that issued three sequential requests via /v1/chat/completions:

Request	Prompt tokens	Max-tokens	Result
1	~1,500	2	OK, ~3.9 s wall
2	~11,600	2	OK, ~31.9 s wall
3	~23,200	2	OK, ~75.4 s wall
(followed by 3× isolated-TG runs @ 200 max-tokens)	~25	200	OK, ~8 s each

Kernel panic occurred approximately 8 min 16 s after mlx_lm.server started, somewhere during or after the isolated-TG runs — not during the larger 23 K prefill. The process was healthy throughout, resident memory ~20 GB (well under the 32 GB ceiling).

What's new vs the existing reports in this thread

1. Substantially lower token threshold. The original report notes "Panic does not occur with smaller prompts (under ~10,000 tokens)" at M4 Max. Our panic followed a 23 K prefill that completed successfully, then triggered later during short-prompt generation. So the 10 K floor on M4 Max does not hold on M4 base — the bug can be triggered after brief larger-context exposure even when current request size is tiny.

2. New hardware class: M4 base (previously reported: M4 Max, M5 MacBook, M1 Ultra, M3 Ultra). The M4 base shares the same T8132 SoC family as higher M4s but has only 32 GB unified memory and fewer GPU cores; this rules out "only happens on high-core-count parts" as a hypothesis.

3. New macOS build: 26.4.1 (25E253) — the M4 Max report was 26.3 (25D125) and M3 Ultra reports were 26.4 (25E246). We're on the latest 26.4.1 point release, so the bug persists through the 26.4.x line.

4. Qwen3.6 hybrid architecture (3× GatedDeltaNet + 1× Attention per block ×10 blocks = 40 layers). KV cache pattern differs meaningfully from the dense Qwen3.5-27B and Qwen3-Coder-30B-A3B models in other reports — yet the same panic triggers. Suggests the bug isn't specific to attention-layer buffer patterns.

Consistent with @Harperbot's two-trigger-path hypothesis

Our panic occurred after a mix of large-prefill requests followed by rapid short-prompt generation, which is consistent with a race in Metal buffer lifecycle management rather than a single-allocation bug. No concurrent threads were calling generate() — all requests were sequential through the HTTP server, but mlx_lm.server internally spawns worker threads.

Mitigation tried — --prompt-cache-bytes did not prevent the panic

Ran with --prompt-cache-bytes 8589934592 (8 GiB cap) specifically hoping this would prevent unbounded KV growth per mlx-lm#615. The cap did not prevent the kernel panic. Suggests the underlying Apple-side bug is independent of prompt-cache size.

Net operational impact

We evaluated MLX against llama.cpp for our production serving stack. Measured throughput advantage was substantial (MLX +30-55 % on PE, +40 % on TG). However, on M4 base hardware under normal workload, MLX produced a kernel panic within 10 minutes of real traffic. llama.cpp on the same hardware, same model family, has run for weeks without incident. We are adopting llama.cpp as a result. Posting this data point in case it helps Apple / the MLX team narrow the trigger — this was not a stress test.

Happy to share the full panic log file, redacted if needed.

[Harperbot]

Adding a user-space mitigation data point for anyone landing here from search.

We open-sourced our production workaround as Harperbot/metal-guard (MIT) after hitting this twice on gemma-4-31b-it-8bit / M1 Ultra 64GB. It does not fix the underlying kext bug — that's on Apple — but it provides:

• Subprocess inference guard with cross-model cadence + post-generation flush to narrow the race window
• Panic cooldown gate reading DiagnosticReports/*.panic to refuse new workers above a 72h rolling threshold (avoids "panic → respawn → panic" KeepAlive cycles)
• KNOWN_PANIC_MODELS registry with model → error_class → tier (panic / abort / degradation) so callers can choose to subprocess-isolate or hard-skip

Install (no PyPI yet):

pip install "git+https://github.com/Harperbot/metal-guard.git@v0.11.5"

Repro evidence + mitigation rationale documented in repo. Posting in case it helps anyone here while this issue sits.