[joelteply]

The M5's 35B-A3B aborted its serving engine on its first /train (continuum
card 1287f154): the MoE router normalizes expert weights by
clamp(weights_sum, eps, inf) (llama-graph.cpp, node
ffn_moe_weights_sum_clamped), CLAMP had no backward, and
ggml_build_backward_expand GGML_ABORTs on it, taking every resident's
engine with it.

• CLAMP backward: grad * step(x - min) * step(max - x); an infinite bound
  never clamps.
• ggml_backward_mark is the build's marking walk, extracted; it returns the
  first node the loss's gradient cannot pass through, with why: no
  backward, an in-place write, a non-float source, or a source the
  backward asserts it gives no gradient (a MoE's experts, a softmax mask,
  rope freq factors, loss labels; a non-split GLU and a gathered
  MUL_MAT_ID are refused as structure). ggml_graph_backward_refusal
  exposes it; ggml_build_backward_expand still aborts on what it refuses.
• ggml_opt_build runs it on the forward graph before building anything
  and refuses through opt_ctx->refusal, which opt_fail already carries to
  /train's error and /learn's outcome; static graphs built in
  ggml_opt_init keep the abort, with the reason.
• test-backend-ops: the grad mode's own copy of the walk is replaced by
  ggml_graph_backward_refusal. mean_abs_asymm counted (0-0)/(0+0) = NaN,
  and a NaN mean passed every tolerance, so any case with a zero gradient
  anywhere passed whatever its backward computed (a CLAMP backward that
  let clamped elements through passed). Equal gradients now count zero
  error; the cases this exposes are card 05834b7c. CLAMP's case sets its
  param (it never ran) and gains the router's infinite-bound shape.
• test-opt-dynamic-accum: an in-place clamp is refused by name, then the
  same context trains a differentiable graph.

Measured on IntelMac CPU (Release, GGML_METAL=OFF): a toy qwen35moe from
test-llama-archs trains its attention weights (loss 4.85 over 1024
tokens); with CLAMP's backward removed the same run refuses naming
ffn_moe_weights_sum_clamped-0 (the M5's abort, now a refusal); training
every weight refuses naming the experts' MUL_MAT_ID. CLAMP grad passes,
and a mutant dropping the upper bound fails it (MAA 0.27). test-opt and
test-opt-dynamic-accum pass. Metal and CUDA unbuilt here: the backward
uses STEP, SCALE (with bias) and MUL, which both backends run.

Co-Authored-By: Claude Opus 5.5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01FRQWzgSfo79JtnwZywHKrE

[joelteply]

PASS at 5d7986b (Fable), by reading. Verified: (1) ggml_graph_backward_refusal and ggml_build_backward_expand share ggml_backward_mark, so the refused set IS the abort set by construction (no second list to drift); (2) the CLAMP backward is grad·step(x−min)·step(max−x), 0 at a clamped value and 1 for an infinite bound (the router's clamp(sum, eps, inf)); (3) on Metal both GGML_UNARY_OP_STEP and GGML_OP_SCALE are supported (ggml-metal-device.m:1457/1435), so the backward stays on the M5's GPU; (4) ggml_opt_alloc returns false with refusal set and /train, /learn surface opt_ctx->refusal, while ggml_opt_init (static graphs, no request to refuse) keeps its abort. Nit, non-blocking: ggml_graph_backward_refusal's calloc result is unchecked. I did not build it (M5 is serving); after the pin bump I run the 35B-A3B /train probe on the live lane in a serving-down window and post the result.