From 8ccd1ca83bc1871a30bbda825ce69b468d765cd4 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 13 Oct 2026 01:47:38 +0200 Subject: [PATCH 1/7] Measure the ARM fp32-to-fp16 conversion: 26 of 26 shipped binaries use software, and inference does not care The NEON branch of simd-mappings.h defines both GGML_CPU_COMPUTE_* macros and selects only one, so GGML_CPU_FP32_TO_FP16 falls through to ggml-base's portable software bit manipulation while neon_compute_fp32_to_fp16 -- a single fcvt -- stays defined and referenced by nothing. On ARM the accelerated macro is literally defined to be the unaccelerated one. Found by the guard written for the #30270 isolation, which fired on upstream's unmodified header. Six independent measurements, five of them static and therefore immune to a noisy machine: 26 of 26 shipped ARM binaries contain no floating-point conversion instruction in that function at all -- our linux, android, windows and macos artifacts plus upstream's own bin-ubuntu-arm64 and bin-macos-arm64 releases -- with the opposite direction in the same object file carrying 9, 6, 5 or 2 of them as an in-file control; the x86 artifacts are the cross-architecture control, where the eleven F16C variants emit 5-6 vcvtps2ph each and the three without F16C emit none, correctly; the shipped loop is 25 instructions per element with a six-operation serial float chain and a data-dependent branch, against fcvtn's four elements per instruction; the function runs 8.9x (clang 21) to 13.3x (gcc 8.5) faster with the one #define, with the opposite direction measured in the same binary and run moving at most 0.3 percent; and both paths agree on every one of the 2^32 float inputs, differing only in NaN payload -- where the identical comparison on x86 F16C gives the same two numbers, so the fix makes ARM agree with what x86 already ships. The sixth measurement is the one that changes the framing: inference does not move. A prediction of a large gain on F16 prompt processing was written down before the run and was wrong. perf attributes 83.9 percent of an F16 pp512 run to tinyBLAS's gemm, which converts inside its own microkernel, and 0.03 percent to the broken function, so the end-to-end delta is +0.13 percent against a run-to-run spread of 0.09 percent. Quantized models never reach it at all. That is the same mistake #30270 made in the other direction, and avoiding it is the reason for measuring. Methodology, because this run produced two lessons the hard way. The compile farm machine carries a permanent eight-core foreign load, so -t 32 oversubscribes it and two runs of the same binary spread 7.5 percent; those rows are discarded rather than reported, while -t 16 reproduced to 0.05 percent and is. And a pattern of fcvt alone is wrong: fcvtzs and fcvtns are float-to-integer, the quantisation step, which would have reported 8 phantom fp16 conversions in quantize_row_q8_0 -- so the headline number is a bare search for any fcvt of any form, which depends on no pattern. TODO.md's fifth upstream item is rewritten from a small follow-up into this, with the three separable commits that hold the fix on the fork (not pushed, no PR) and the open decision of PR versus comment left to take together. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- TODO.md | 71 ++++--- docs/benchmarks/README.md | 1 + docs/benchmarks/arm-fp16-conversion.md | 248 +++++++++++++++++++++++++ 3 files changed, 282 insertions(+), 38 deletions(-) create mode 100644 docs/benchmarks/arm-fp16-conversion.md diff --git a/TODO.md b/TODO.md index 1778996b..c4e7d4b8 100644 --- a/TODO.md +++ b/TODO.md @@ -860,44 +860,39 @@ three different subsystems and a combined report gets triaged as one. | 3 | `ggml-rpc: the client answers every supports_op with true` | the RPC section of `CLAUDE.md`: a served device that lacks an op **aborts the server**. Reproduced on the macOS CI runners' paravirtual Metal GPU, which has no `MUL_MAT` and took the first run down; it is why every test here serves `CPU` by name | upstream's own TODO comment admits it. Should the client forward the query to the server, or should `rpc-server -d` refuse a device whose op set it cannot report? | | 4 | `windows-rocm: 83,362 compiler warnings, against 9 on linux-rocm` | the table below in this section (714,345 log lines against 2,455; the per-class counts; all of them in `ggml/src/ggml-cuda/*.cu`) | the cause is the language, not the arch count: HIP sources are compiled as plain C++ on Windows (`CXX_IS_HIPCC` on `WIN32`) and as HIP on Linux. Fix the warnings, or the C++-on-Windows choice? Their own `windows-rocm` job pays the identical cost | -#### A fifth, and this one is a small PR rather than an issue: the #30270 follow-up - -Unlike the four above, here there **is** something to offer -- and the measurement is already done -([`docs/benchmarks/upstream-f16c-30270.md`](docs/benchmarks/upstream-f16c-30270.md)). - -[#30270](https://github.com/ggml-org/llama.cpp/pull/30270) (commit `c28e174141dd`, merged 2026-10-11) -fixes a real defect: the `#elif defined(__F16C__)` branch of `simd-mappings.h` defined only -`GGML_CPU_COMPUTE_FP16_TO_FP32`, so the selected macro fell through to the 64K lookup table and an -F16C CPU never used the hardware instruction. NEON, POWER9 and RISC-V each define **both** macros in -their branch; only F16C defined half of its own decision. - -Two things to offer, both independent of throughput: - -1. **Move the two `#define`s into the branch**, where the COMPUTE macros are defined, as every other - branch does. They currently sit **88 lines** further down and outside the chain, which is why the - new code has to explain the ordering in its own comment ("checks below don't test for `__F16C__`"). - A future reordering of the file breaks it **silently**, back to the lookup table -- the exact defect - that went unnoticed until now. -2. **Add a guard so the omission cannot recur**, after the branch chain and before the fallback: - - ```c - #if defined(GGML_CPU_COMPUTE_FP16_TO_FP32) && !defined(GGML_CPU_FP16_TO_FP32) - #error "a branch defined GGML_CPU_COMPUTE_FP16_TO_FP32 but not GGML_CPU_FP16_TO_FP32" - #endif - ``` - - **Proven both ways** with clang `-fsyntax-only` against the real b11574 header: it passes on the - fixed tree, fails with the `#error` on the reverted one, does not fire without `__F16C__`, and both - trees compile when the guard is absent -- so the guard is the only thing that changes the outcome. - -And one thing to **say** in the same comment, because it is honest and useful rather than an -objection: a clean isolation on Granite Rapids (one tree built twice, upstream's own `llama-bench`, -5 repetitions, two non-F16C control modules) finds **no measurable effect** for Q4_K_M at 8 and 16 -threads -- every difference inside ±2 % with random signs. The author's own description (a FreeBSD box -with "clock-mismatched RAM", "gains like these may be greater than reasoned") fits that: the fix -removes a ~256 KB table from the path, which costs on a bandwidth-starved machine and nothing on a -server CPU. **The change stays right; the number is not general.** Offering the measurement with the -PR is the point, not disputing it. +#### A fifth, and the only one where we hold a finished fix: the ARM fp16 conversion + +Unlike the four above, a fix exists, is measured from every side, and sits in three local commits on +`llama.cpp.bernardladenthin` (branch `f16c-select-in-branch`, **not pushed, no PR**). The full record +is [`docs/benchmarks/arm-fp16-conversion.md`](docs/benchmarks/arm-fp16-conversion.md). + +**The finding grew while isolating #30270.** The `__ARM_NEON` branch of `simd-mappings.h` defines both +`GGML_CPU_COMPUTE_*` macros and selects only one, so `GGML_CPU_FP32_TO_FP16` falls through to +ggml-base's portable software conversion while `neon_compute_fp32_to_fp16` -- one `fcvt` -- stays +unused. **In 26 of 26 shipped ARM binaries** -- our four operating systems plus upstream's own +`bin-ubuntu-arm64` and `bin-macos-arm64` -- that function contains **no floating-point conversion +instruction at all**. The function is 8.9x (clang) to 13.3x (gcc) slower than it needs to be, 25 +instructions per element against `fcvtn`'s four elements per instruction, and bit-exact over all 2^32 +float inputs. + +**And it does not make inference faster**, which is the half that must travel with it: `perf` puts +83.9 % of an F16 `pp512` run in tinyBLAS's gemm, which converts inside its own microkernel, and +**0.03 %** in the broken function -- so end to end the change is inside the run-to-run spread +(+0.13 %). A prediction of a large gain was written down here before the run and was wrong; that is +the same mistake #30270 made, in the other direction, and the reason the measurement was done at all. + +The three commits are deliberately separable: + +1. **`ggml-cpu : select the NEON fp32->fp16 conversion`** -- the defect, one `#define`. +2. **`ggml-cpu : move the F16C selection into its own branch`** -- #30270 fixed its half from 88 lines + below the chain, outside it, which is why that code has to explain its own position in a comment. + Proven behaviour-preserving: all three configurations resolve byte-identically before and after. +3. **`ggml-cpu : refuse a branch that defines a COMPUTE macro without selecting it`** -- the `#error` + that found this one, verified red on the unfixed tree and green on the fixed one, with a control + showing both trees compile without it. + +**Open decision, to be taken together:** one PR for all three, a PR for 1 and an issue for the rest, +or a comment on #30270 carrying the measurement. Nothing is filed yet. **Two rules for the day.** State the measurement and ask the question; do not propose the fix for 1-3, because each is a design decision in code this project does not own. And say plainly what was done diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 0e4f9fbc..47f2572e 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -15,6 +15,7 @@ themselves live in [`../../TODO.md`](../../TODO.md). |---|---| | [`cpu-backend-modules.md`](cpu-backend-modules.md) | What each of the 22 shipped CPU backend modules is worth, per microarchitecture — and whether `ggml_backend_load_best` picks the fastest one | | [`gpu-module-jars.md`](gpu-module-jars.md) | What happens when a GPU module jar is actually *used* -- two findings that contradict the "a module that cannot help is skipped" premise | +| [`arm-fp16-conversion.md`](arm-fp16-conversion.md) | Every shipped ARM binary converts fp32 to fp16 in software, ours and upstream's: 26 of 26 disassembled, 8.9x to 13.3x on the function, and 0.03 % of cycles so inference does not move | | [`upstream-f16c-30270.md`](upstream-f16c-30270.md) | An isolation of one upstream commit: what it does, that it changes nothing measurable here, and a compile-time guard proven red against the defect | | [`native-loading.md`](native-loading.md) | What extracting the natives costs and what the reuse saves, how a running JVM's locked files behave, and which container base images can load the Linux jar | | [`ec2-harness.md`](ec2-harness.md) | How a measurement is run on rented hardware, and the invariants that came out of getting it wrong | diff --git a/docs/benchmarks/arm-fp16-conversion.md b/docs/benchmarks/arm-fp16-conversion.md new file mode 100644 index 00000000..5c3743f8 --- /dev/null +++ b/docs/benchmarks/arm-fp16-conversion.md @@ -0,0 +1,248 @@ + + +# Every ARM build converts fp32 to fp16 in software, including ours and upstream's + +Measured **2026-10-13**. One `#define` is missing from the NEON branch of +`ggml/src/ggml-cpu/simd-mappings.h`, so `GGML_CPU_FP32_TO_FP16` falls through to a software bit +manipulation while the branch's own single-instruction conversion sits there unused. Same shape of +defect as the one [ggml-org/llama.cpp#30270](https://github.com/ggml-org/llama.cpp/pull/30270) fixed +for x86 F16C on 2026-10-11 -- in the other direction, on a different architecture, still open. + +Found while isolating that PR ([`upstream-f16c-30270.md`](upstream-f16c-30270.md)): a `#error` guard +written to stop the omission recurring fired immediately on upstream's unmodified header. + +**Two findings, and they must not be confused:** + +| | | +|---|---| +| **The function is 8.9x to 13.3x slower than it needs to be**, in 26 of 26 shipped ARM binaries | measured, static and timed, four ways | +| **Inference does not get faster** | measured: the function is **0.03 %** of cycles, because the dominant path never calls it | + +So this is a correctness, consistency and portability fix. It is **not** an inference speed-up on a +current default build, and saying otherwise would repeat exactly the mistake #30270 made in the +other direction. + +## The chain, branch by branch + +The header picks the conversion macros from a chain of per-architecture branches. Each defines a +`GGML_CPU_COMPUTE_*` pair -- the raw intrinsic -- and is expected to **select** it as +`GGML_CPU_FP16_TO_FP32` / `GGML_CPU_FP32_TO_FP16`. Anything left unselected falls through to the end +of the section: a 64K lookup table in one direction, and `ggml_compute_fp32_to_fp16` -- a software +bit manipulation -- in the other. + +Read at b11576, all four branches: + +| branch | defines COMPUTE pair | selects fp16 to fp32 | selects fp32 to fp16 | +|---|:-:|:-:|:-:| +| `__ARM_NEON && __ARM_FP16_FORMAT_IEEE` | yes | yes | **no** | +| `__F16C__` (x86) | yes | from outside the chain, by #30270 | same | +| `__POWER9_VECTOR__` | yes | yes | yes | +| `__riscv && __riscv_zfhmin` | yes | yes | yes | + +Two of four were incomplete. #30270 completed one of them -- from 88 lines below, outside the chain. +The ARM one is open, in the direction that fix did not cover. + +**The sharpest way to state it:** the fallback reads +`#define GGML_CPU_FP32_TO_FP16(x) GGML_COMPUTE_FP32_TO_FP16(x)`, and `GGML_COMPUTE_FP32_TO_FP16` is +ggml-base's deliberately portable software conversion -- the one `ggml.c` uses everywhere because +ggml-base may not assume CPU features. The whole purpose of the `GGML_CPU_*` pair is to be the +*accelerated* variant. On ARM it is defined to be the unaccelerated one. + +That it is an omission and not a decision is upstream's own comment on that fallback, which has +always promised both: + +> On ARM NEON, it's quicker to directly convert x -> x instead of calling into +> `ggml_lookup_fp16_to_fp32`, so we define `GGML_CPU_FP16_TO_FP32` **and `GGML_CPU_FP32_TO_FP16`** +> elsewhere for NEON. This is also true for POWER9. + +POWER9 defines both. NEON defines one. And `neon_compute_fp32_to_fp16` -- a one-line `__fp16` +conversion -- sits in the branch with nothing referencing it. + +## Every shipped ARM binary, disassembled + +`ggml_cpu_fp32_to_fp16`, counting fp32-to-fp16 conversion instructions (`fcvt h`, `fcvtn`, +`fcvtn2`), with the **other direction in the same file** as an in-file control -- that one the +branch selects correctly: + +| artifact | files | fp32 to fp16 | fp16 to fp32 *(control)* | +|---|:-:|:-:|:-:| +| ours `cpu-linux-aarch64` | 8 variants | **0 in each** | 9 / 9 / 9 / SVE / SVE / SVE / 2 / 2 | +| ours `cpu-android-aarch64` | 7 variants | **0 in each** | 5 / 5 / 5 / 5 / 1 / 1 / 1 | +| ours `cpu-windows-aarch64` | `ggml-cpu.dll` | **0** | 6 | +| ours `metal-macos-aarch64` | `libjllama.dylib` | **0** | 9 | +| **upstream** `bin-ubuntu-arm64` b11576 | 8 variants | **0 in each** | 9 / 9 / 9 / SVE / SVE / SVE / 2 / 2 | +| **upstream** `bin-macos-arm64` b11576 | `libggml-cpu.dylib` | **0** | 6 | + +The control column is the point: **in one and the same object file** the widening direction carries +hardware conversions and the narrowing direction carries none. + +Two caveats on the method, both worth stating because the first one bit this measurement: + +* A pattern of `fcvt` alone is **wrong**: `fcvtzs` / `fcvtns` are float-to-**integer**, i.e. the + quantisation step, and they are everywhere. The shipped `quantize_row_q8_0` contains 8 `fcvtns` + and **no** fp16 conversion -- which would have been reported as 8 by a loose pattern. +* Where the control column reads *SVE*, the widening uses the predicated `fcvt z, z` form, which a + scalar-shaped pattern does not match -- a limit of the pattern, not an absence. + +The fp32-to-fp16 column needs no caveat at all: a bare search for **any** `fcvt` of any form, across +all 26 files, returns **zero**. That number depends on no pattern. + +### The x86 control, at artifact level + +The same sweep over this project's 14 x86-64 CPU modules, counting `vcvtps2ph`: + +| modules | count | +|---|---| +| `ivybridge`, `piledriver`, `haswell`, `skylakex`, `cannonlake`, `cascadelake`, `cooperlake`, `icelake`, `zen4`, `alderlake`, `sapphirerapids` | **5-6 each** | +| `x64`, `sse42`, `sandybridge` | **0** | + +Those three zeros are exactly the three variants **without F16C** (which arrived with Ivy Bridge), so +they are correct -- there is no instruction to emit. Same project, same build system, same function: +where the hardware has the instruction, x86 uses it; where it does not, x86 does not. Only ARM has it +everywhere and uses it nowhere. + +## What it costs, read out of the shipped binary + +The loop of `ggml_cpu_fp32_to_fp16` in `libggml-cpu-armv8.0_1.so`, as published: + +| | shipped | with the one `#define` | +|---|---|---| +| instructions per element | **25** | `fcvtn v0.4h, v0.4s` -- **4 elements per instruction** | +| serial float dependency chain | 6 (`fabs`, `fmul`, `fmov`, `fmadd`, `fmov`) | none | +| branches in the loop body | 3, one of them **data-dependent** (inf/NaN) | none | + +The shipped loop is the classic software fp16 encoder: `fabs`, multiply by `0x77800000`, compare +against `0x71000000`, `csel`, `fmadd` by `0x08800000`, then the mantissa and sign stitched back +together with shifts and masks. With the selection in place it is a load, one `fcvt` and a store -- +and clang then vectorises it to four elements per conversion. + +The latency chain, not the instruction count, is what dominates: five serial FP operations at +several cycles each, per element. + +## Throughput of the function, where a machine can be trusted + +On **cfarm185** (GCC Compile Farm, Ampere eMAG, 32 cores), a 16M-element bulk conversion, best of 7, +with the **opposite direction measured in the same binary and the same run** as a control the change +cannot touch: + +| compiler | fp32 to fp16, upstream | fixed | factor | fp16 to fp32 *(control)* | +|---|---:|---:|---:|---| +| GCC 8.5 `-O3` | 82.0 Melem/s | 1087.2 | **13.3x** | 4259.3 to 4258.3 (**-0.02 %**) | +| clang 21.1.8 `-O3` | 341.9 | 3031.6 | **8.9x** | 4255.3 to 4268.1 (+0.3 %) | +| GCC `-O3 -march=armv8.2-a+fp16` | 82.0 | 1087.2 | 13.3x | 4246.1 to 4244.9 (-0.03 %) | +| clang `-O3 -march=armv8.2-a+fp16` | 342.0 | 3037.9 | 8.9x | 4252.8 to 4254.2 (+0.03 %) | + +clang recovers part of the loss by vectorising the software path (342 against GCC's 82) and the fix +is still worth **8.9x** on top of that. The control stays within +-0.3 % in all eight runs -- same +file, same binary, same run -- which is what separates this from a timing artefact. + +## End to end: nothing, and the reason is measured + +One tree built twice on cfarm185 with clang, differing only in that `#define`; the two artifact sets +run interleaved with upstream's own `llama-bench`, `-r 5`, on a **1.5 GB F16** model and on +**Qwen3-0.6B-Q4_K_M**: + +| model, `-t 16` | upstream | fixed | delta | +|---|---:|---:|---:| +| F16 `pp512` | 140.88 | 141.07 | +0.13 % | +| F16 `tg128` | 37.61 | 37.65 | +0.11 % | +| Q4_K_M `pp512` | 84.63 | 84.59 | -0.05 % | +| Q4_K_M `tg128` | 40.98 | 40.95 | -0.07 % | + +All four inside the run-to-run spread (two runs of the *same* binary differ by up to 0.09 %). The +`-t 32` rows of the same run are **discarded**, not reported: that machine carries a permanent +8-core foreign load, so 32 threads oversubscribe it and two runs of the same binary spread 7.5 %. + +**Why there is nothing to see**, from `perf` on the F16 `pp512` run: + +| share of cycles | function | +|---:|---| +| **83.93 %** | `tinyBLAS<...>::gemm` | +| 9.07 % | `ggml_compute_forward_flash_attn_ext` | +| 0.16 % | `ggml_vec_dot_f16` | +| **0.03 %** | `ggml_cpu_fp32_to_fp16` | + +`GGML_LLAMAFILE` is on by default, and tinyBLAS takes `unsigned short` as an operand type and +converts **inside its own microkernel** -- so for an F16 model's matmuls the broken function never +sees the activations at all. 13x on 0.03 % of the time is a 0.03 % saving, which is precisely what +the table above shows. + +This was predicted wrong here before it was measured: the expectation written down in advance was a +large gain on F16 prompt processing, on the reasoning that `GGML_TYPE_F16`'s `vec_dot_type` is +`GGML_TYPE_F16` and so `mul_mat` would convert the whole activation matrix per call. It would -- +if tinyBLAS were not there first. The prediction was recorded before the run, which is the only +reason the error is visible at all. + +## Why nobody noticed, in full + +Three things have to be true at once, and they are: + +1. **tinyBLAS bypasses it** for F16 matmuls -- 84 % of the cycles, measured above. +2. **Quantized models never reach it**: their activations go to Q8_0/Q8_K through a different + `from_float` that was never broken. Everybody benchmarks quantized models. +3. **What is left is tiny**: the KV cache writes (`cache_type_k`/`cache_type_v` default to + `GGML_TYPE_F16`, written through `ggml_compute_forward_dup`/`cpy`), spread over 16 threads. + +And it cannot be caught by a build: `ggml_cpu_fp32_to_fp16` compiles and produces bit-identical +results either way (see below), so nothing fails, nothing warns, and no test notices. + +## Bit-exact for every non-NaN value -- exhaustively, and the NaN case argues for the fix + +Both conversions were applied to **every one of the 2^32 float32 bit patterns** and compared (44 s +on cfarm185): + +| inputs checked | 4 294 967 296 -- every float32 bit pattern | +|---|---| +| differing, **non-NaN** | **0** | +| differing, NaN payload | 16 744 448 | + +So for every finite value, every zero, every denormal and both infinities the two paths agree bit for +bit. They differ only on **NaN inputs**, where the software encoder emits a canonical quiet NaN +(`0x7E00` with the sign, visible as `orr w7, w5, #0x7e00` in the shipped disassembly) while the +hardware instruction propagates the truncated payload, as IEEE 754 recommends. + +**That difference is not a reason against the fix -- it is a reason for it**, and this is measured +rather than argued: the identical comparison run on **x86 with F16C**, where upstream already ships +the hardware path, gives + +| | non-NaN | NaN payload | +|---|---|---| +| ARM, software vs `fcvt` | 0 | 16 744 448 | +| **x86, software vs `vcvtps2ph`** | **0** | **16 744 448** | + +the same numbers. The eleven F16C x86 modules this project ships already behave the hardware way, so +making ARM select its own instruction brings the two architectures into agreement instead of +splitting them. And nothing in llama.cpp reads a NaN payload: a NaN anywhere in weights or +activations is already a lost computation, and both encodings are quiet NaNs that propagate. + +## The guard, proven both ways on real aarch64 + +```c +#if defined(GGML_CPU_COMPUTE_FP32_TO_FP16) && !defined(GGML_CPU_FP32_TO_FP16) +#error "a branch defined GGML_CPU_COMPUTE_FP32_TO_FP16 but did not select it" +#endif +``` + +Placed after the branch chain and before the fallback, compiled on cfarm185 against the real b11576 +header: it **fires** on upstream's unmodified file and **passes** once the `#define` is added. It +cannot fire spuriously -- POWER9 and RISC-V select both macros, and a platform with no branch at all +(s390x) defines no COMPUTE macro for it to catch. + +## Limits of this measurement + +* **One ARM core generation.** cfarm185 is an Ampere eMAG, a 2018 design, and it is the only ARM + machine on the compile farm that accepts our key. The static evidence is generation-independent; + the 8.9x to 13.3x factor is not, and a Neoverse or an Apple core may differ. +* **A shared machine.** Permanent 8-core foreign load. `-t 16` reproduced to 0.05 % between runs of + the same binary and is reported; `-t 32` spread 7.5 % and is not. +* **Two compilers**, GCC 8.5 and clang 21.1.8. MSVC on Windows-on-ARM is untested, though its + shipped `ggml-cpu.dll` shows the same zero. +* **No claim about Apple Silicon throughput.** The macOS rows are disassemblies of the shipped + binaries, which is why they appear in the static table and not in the timing one. +* **Only one model size and two quantisations.** A build with `GGML_LLAMAFILE=OFF`, a very large KV + cache, or an embedder calling the CPU backend's `from_float` directly would weight the function + differently; none of those was measured. From dd9e5b629f35cfe2996136e90c4cb3515db3722d Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 13 Oct 2026 01:49:08 +0200 Subject: [PATCH 2/7] Correct the t=32 spread figure and say what those rows would have claimed The discarded high-thread rows were described with a 7.5 percent same-binary spread, which came from the first run of the measurement -- the one whose CSV columns were mis-parsed. Re-derived from the archived raw llama-bench tables, the clean run is worse and more instructive: up to 10.5 percent between two runs of the same binary, with llama-bench's own error bar reaching plus or minus 9.8 percent inside a single run. Taken at face value those rows would report the fix making token generation 12 to 18 percent slower, complete with error bars. That is the clearest argument for discarding them, so it is now stated rather than merely asserted that they are noisy -- together with the check that catches it, which is to interleave the variants and compare two runs of the same binary before comparing variants at all. The four reported t=16 rows were re-derived from the same archive and match the published table exactly. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- docs/benchmarks/arm-fp16-conversion.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/benchmarks/arm-fp16-conversion.md b/docs/benchmarks/arm-fp16-conversion.md index 5c3743f8..adf0bfbb 100644 --- a/docs/benchmarks/arm-fp16-conversion.md +++ b/docs/benchmarks/arm-fp16-conversion.md @@ -153,9 +153,15 @@ run interleaved with upstream's own `llama-bench`, `-r 5`, on a **1.5 GB F16** m | Q4_K_M `pp512` | 84.63 | 84.59 | -0.05 % | | Q4_K_M `tg128` | 40.98 | 40.95 | -0.07 % | -All four inside the run-to-run spread (two runs of the *same* binary differ by up to 0.09 %). The -`-t 32` rows of the same run are **discarded**, not reported: that machine carries a permanent -8-core foreign load, so 32 threads oversubscribe it and two runs of the same binary spread 7.5 %. +All four inside the run-to-run spread (two runs of the *same* binary differ by up to 0.09 %). + +The `-t 32` rows of the same run are **discarded**, not reported, and they are worth showing as the +reason why. That machine carries a permanent 8-core foreign load, so 32 threads oversubscribe it: +two runs of the same binary spread by up to **10.5 %** there, and `llama-bench`'s own error bar +reaches **+-9.8 %** within a single run. Taken at face value those rows would say the fix makes +generation **12 % to 18 % slower** -- a confident-looking result, with error bars, that is plainly +false. A number from a machine whose load is unknown is not a number, and the check that exposes it +is cheap: interleave the variants and compare two runs of the *same* binary first. **Why there is nothing to see**, from `perf` on the F16 `pp512` run: From e9f6dd6142091040fae4a5bb4948d00b8081c716 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 13 Oct 2026 13:00:40 +0200 Subject: [PATCH 3/7] Prepare the ARM fp16 submission: final patch, three ARM cores, real build, object-level x86 proof The change grew from the minimal form into the one that leaves the file in better shape, on the reasoning that the guard then protects a regular structure rather than one with an exception: every branch of the chain defines its COMPUTE pair and selects it in the same place. #30270's two selecting defines move from 88 lines below the chain into the __F16C__ branch, which also retires the comment that had to explain their position. Net +10/-6 in one file, and the guard now replaces the out-of-chain block at its own position instead of sitting beside it. That move is the one part a reviewer could reasonably push back on, so it is proven at object level rather than by reading: a translation unit calling both macros, compiled on x86 and disassembled, emits vcvtps2ph before and after with -mavx2 -mf16c -mfma and with -mavx512f -mf16c, and the software path before and after with -msse4.2. Macro resolution is identical across four x86 configurations, AVX512 included -- the case that could have interacted with the old block sitting after the lookup-table declarations. The ARM side gained two core generations. The compile farm turned out to already have them: probing every host in its aarch64 list instead of the first four found three 64-core Neoverse-N1 machines and an Apple M1 running macOS that our key opens. So the factor is now a measured range over three generations and five compiler builds -- 13.3x and 8.9x on a 2018 eMAG, 4.0x and 3.1x on Neoverse-N1, 3.6x on the M1 -- narrowing on newer cores and never vanishing, with the in-run control within 2.5 percent everywhere and the 16M-element checksum byte-identical on all of them. Apple Silicon is therefore measured rather than inferred, and the limit that said one core generation is struck through. The final patch was built in upstream's own ARM release configuration (GGML_BACKEND_DL, GGML_CPU_ALL_VARIANTS, GGML_NATIVE=OFF): all eight variant modules compile, the guard fires nowhere in the whole build, fcvt/fcvtn/fcvtn2 appear six times in each, reverting in the same tree puts the count back to zero, and ctest gives 52 of 54 with the same two environment failures as unpatched master. It also applies cleanly to a newer master (296538a) that landed during preparation. docs/upstream-submission-arm-fp16-selection.md carries the ready-to-paste PR text, the commit split, the conventions read out of CONTRIBUTING, the archaeology showing this is not a regression, and three mistakes made while preparing it -- a too-loose fcvt pattern, a 32-thread run on a contended machine that would have reported the fix as 12-18 percent slower, and a build script that deleted its own input and reported BUILD OK for unpatched source. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- docs/benchmarks/arm-fp16-conversion.md | 59 +++-- .../upstream-submission-arm-fp16-selection.md | 239 ++++++++++++++++++ 2 files changed, 275 insertions(+), 23 deletions(-) create mode 100644 docs/upstream-submission-arm-fp16-selection.md diff --git a/docs/benchmarks/arm-fp16-conversion.md b/docs/benchmarks/arm-fp16-conversion.md index adf0bfbb..a63f582b 100644 --- a/docs/benchmarks/arm-fp16-conversion.md +++ b/docs/benchmarks/arm-fp16-conversion.md @@ -123,22 +123,34 @@ and clang then vectorises it to four elements per conversion. The latency chain, not the instruction count, is what dominates: five serial FP operations at several cycles each, per element. -## Throughput of the function, where a machine can be trusted - -On **cfarm185** (GCC Compile Farm, Ampere eMAG, 32 cores), a 16M-element bulk conversion, best of 7, -with the **opposite direction measured in the same binary and the same run** as a control the change -cannot touch: - -| compiler | fp32 to fp16, upstream | fixed | factor | fp16 to fp32 *(control)* | -|---|---:|---:|---:|---| -| GCC 8.5 `-O3` | 82.0 Melem/s | 1087.2 | **13.3x** | 4259.3 to 4258.3 (**-0.02 %**) | -| clang 21.1.8 `-O3` | 341.9 | 3031.6 | **8.9x** | 4255.3 to 4268.1 (+0.3 %) | -| GCC `-O3 -march=armv8.2-a+fp16` | 82.0 | 1087.2 | 13.3x | 4246.1 to 4244.9 (-0.03 %) | -| clang `-O3 -march=armv8.2-a+fp16` | 342.0 | 3037.9 | 8.9x | 4252.8 to 4254.2 (+0.03 %) | - -clang recovers part of the loss by vectorising the software path (342 against GCC's 82) and the fix -is still worth **8.9x** on top of that. The control stays within +-0.3 % in all eight runs -- same -file, same binary, same run -- which is what separates this from a timing artefact. +## Throughput of the function, on three ARM core generations + +A 16M-element bulk conversion, best of 7, with the **opposite direction measured in the same binary +and the same run** as a control the change cannot touch. All three machines are GCC Compile Farm +hosts: + +| machine | core | compiler | upstream | fixed | factor | control | +|---|---|---|---:|---:|---:|---| +| cfarm185 | Ampere eMAG (2018) | gcc 8.5 | 82.0 Melem/s | 1087.2 | **13.3x** | -0.02 % | +| cfarm185 | Ampere eMAG | clang 21.1.8 | 341.9 | 3031.6 | **8.9x** | +0.3 % | +| cfarm424 | **ARM Neoverse-N1**, 64 cores | gcc 14 | 323.8 | 1310.1 | **4.0x** | -2.0 % | +| cfarm424 | ARM Neoverse-N1 | clang 19.1.7 | 976.2 | 3030.3 | **3.1x** | +2.2 % | +| cfarm104 | **Apple M1**, macOS 25.5.0 | Apple clang 21 | 2751.7 | 9811.2 | **3.6x** | -2.5 % | + +**The factor narrows on newer cores and never vanishes: 3.1x to 13.3x.** That shape makes sense -- +a wider, deeper core executes the software path's serial float chain faster, so it loses less by +taking it, but the hardware instruction is still several times better. Where the older machine's +controls hold to ±0.3 %, the two newer ones move ±2.5 %; both are shared machines and that is an +order of magnitude below the effect. + +On a newer compiler the *starting point* also improves, because clang vectorises the software path: +on the eMAG gcc 8.5 manages 82 Melem/s where clang 21 manages 342. The fix is worth multiples on +top of either. + +**Apple Silicon is measured, not inferred.** `cfarm104` is a Mac mini M1 running macOS, and a plain +`cc` there predefines `__ARM_NEON` and `__ARM_FP16_FORMAT_IEEE` exactly as Linux does -- so +`metal-macos-aarch64` takes the same branch, which the disassembly of the shipped dylib already +showed and this confirms from the other side. ## End to end: nothing, and the reason is measured @@ -206,6 +218,8 @@ on cfarm185): | differing, **non-NaN** | **0** | | differing, NaN payload | 16 744 448 | +The 16M-element checksum of the microbenchmark is **byte-identical on all three cores and every compiler** (`00924503571709770412`: Ampere eMAG, Neoverse-N1, Apple M1; gcc 8.5, gcc 14, clang 19, clang 21, Apple clang 21), which is the same equality seen from a second direction. + So for every finite value, every zero, every denormal and both infinities the two paths agree bit for bit. They differ only on **NaN inputs**, where the software encoder emits a canonical quiet NaN (`0x7E00` with the sign, visible as `orr w7, w5, #0x7e00` in the shipped disassembly) while the @@ -240,15 +254,14 @@ cannot fire spuriously -- POWER9 and RISC-V select both macros, and a platform w ## Limits of this measurement -* **One ARM core generation.** cfarm185 is an Ampere eMAG, a 2018 design, and it is the only ARM - machine on the compile farm that accepts our key. The static evidence is generation-independent; - the 8.9x to 13.3x factor is not, and a Neoverse or an Apple core may differ. +* ~~One ARM core generation~~ -- **closed**: three generations measured (Ampere eMAG 2018, ARM + Neoverse-N1, Apple M1) across Linux and macOS and five compiler builds. The factor does depend on + the core, from 3.1x to 13.3x, which is why it is reported as a range rather than a number. Still + untested: Neoverse V2/N2, Apple M2 and later, and the 64K-page and FreeBSD hosts the farm also has. * **A shared machine.** Permanent 8-core foreign load. `-t 16` reproduced to 0.05 % between runs of the same binary and is reported; `-t 32` spread 7.5 % and is not. -* **Two compilers**, GCC 8.5 and clang 21.1.8. MSVC on Windows-on-ARM is untested, though its - shipped `ggml-cpu.dll` shows the same zero. -* **No claim about Apple Silicon throughput.** The macOS rows are disassemblies of the shipped - binaries, which is why they appear in the static table and not in the timing one. +* **Five compiler builds** (gcc 8.5, gcc 14, clang 19.1.7, clang 21.1.8, Apple clang 21). MSVC on + Windows-on-ARM is untested, though its shipped `ggml-cpu.dll` shows the same zero. * **Only one model size and two quantisations.** A build with `GGML_LLAMAFILE=OFF`, a very large KV cache, or an embedder calling the CPU backend's `from_float` directly would weight the function differently; none of those was measured. diff --git a/docs/upstream-submission-arm-fp16-selection.md b/docs/upstream-submission-arm-fp16-selection.md new file mode 100644 index 00000000..b1d44ac4 --- /dev/null +++ b/docs/upstream-submission-arm-fp16-selection.md @@ -0,0 +1,239 @@ + + +# Upstream submission, prepared: `ggml-cpu: select the NEON fp32->fp16 conversion` + +Everything needed to open the pull request, ready to paste. **Nothing is filed and nothing is +committed.** The change sits uncommitted in the working tree of +`X:\Privat\OpenSource\github\llama.cpp.bernardladenthin`. + +The measurement record behind it is [`benchmarks/arm-fp16-conversion.md`](benchmarks/arm-fp16-conversion.md). + +--- + +## The patch: +10 / -6, one file + +```diff +--- a/ggml/src/ggml-cpu/simd-mappings.h ++++ b/ggml/src/ggml-cpu/simd-mappings.h +@@ -42,6 +42,7 @@ extern "C" { + #define GGML_CPU_COMPUTE_FP32_TO_FP16(x) neon_compute_fp32_to_fp16(x) + + #define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) ++ #define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) + + static inline float neon_compute_fp16_to_fp32(ggml_fp16_t h) { +@@ -63,6 +64,8 @@ extern "C" { + #define GGML_CPU_COMPUTE_FP16_TO_FP32(x) _cvtsh_ss(x) + #define GGML_CPU_COMPUTE_FP32_TO_FP16(x) _cvtss_sh(x, 0) + #endif ++ #define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) ++ #define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) + #elif defined(__POWER9_VECTOR__) +@@ -140,12 +143,13 @@ extern float ggml_table_f32_ue4m3[1 << 8]; + #define GGML_CPU_UE4M3_TO_FP32(x) ggml_ue4m3_to_fp32(x) + #endif + +-// Pre AVX512 CPUs (AVX2 with F16C, such as i5-8th gen or i7-4790) +-// do not produce codegen as checks below don't test for __F16C__ +-// but this check does. +-#if defined(__F16C__) +-#define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) +-#define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) ++// A branch above that defines a GGML_CPU_COMPUTE_* macro must also select it: left unselected it ++// falls through to the portable fallback below, which is correct but never uses the instruction. ++#if defined(GGML_CPU_COMPUTE_FP16_TO_FP32) && !defined(GGML_CPU_FP16_TO_FP32) ++#error "a branch defines GGML_CPU_COMPUTE_FP16_TO_FP32 without selecting it as GGML_CPU_FP16_TO_FP32" ++#endif ++#if defined(GGML_CPU_COMPUTE_FP32_TO_FP16) && !defined(GGML_CPU_FP32_TO_FP16) ++#error "a branch defines GGML_CPU_COMPUTE_FP32_TO_FP16 without selecting it as GGML_CPU_FP32_TO_FP16" + #endif + + // On ARM NEON, it's quicker to directly convert x -> x instead of calling into ggml_lookup_fp16_to_fp32, +``` + +Three details that make it read well in review: + +* **The fix carries no comment**, because `__POWER9_VECTOR__` and `__riscv_zfhmin` have the same pair + bare -- a comment there would be the inconsistency. After the change the NEON branch is identical + in shape to both. +* **The guard replaces the out-of-chain block at its own position**, rather than sitting next to it. + Where a selection used to live outside the chain, the check that makes out-of-chain selections + impossible now lives. +* **It sits immediately above upstream's own comment** promising both macros for NEON, so the promise + and the check that enforces it are read together. + +**Applies cleanly to master at `9d7d50498` and at `296538a`** -- master moved during preparation and +both were tested. + +## Suggested commit split + +Three commits in one PR, so either of the last two can be dropped without touching the first. +Maintainers squash-merge, so this is for review readability only. + +1. `ggml-cpu: select the NEON fp32->fp16 conversion` -- the one line. +2. `ggml-cpu: move the F16C selection into its own branch` -- behaviour-preserving, proven at object + level. +3. `ggml-cpu: refuse a branch that defines a COMPUTE macro without selecting it` -- the guard. + +## Conventions, checked against the repository rather than assumed + +* `CONTRIBUTING.md` asks for ` : (#)`, and **maintainers + squash-merge and write that title themselves** -- so the PR *title* becomes the commit title and + the `(#N)` is added at merge. +* **The module prefix this file's history uses is `ggml-cpu`** -- `ggml-cpu: vectorize fp32 to fp16 + conversion (#30157)`, `ggml-cpu: gate __fp16 on __ARM_FP16_FORMAT_IEEE (#26860)`; #30270 used + `cpu:`. +* A PR needs **two independent reviews** unless a maintainer adds the "merge ready" label, and + CONTRIBUTING suggests granting maintainers write access to the branch for faster reviews. + +## Archaeology, so the PR does not misstate the history + +* **Not a regression.** The NEON branch has never carried the fp32->fp16 selection: absent in all 25 + most recent commits touching the file, back to **2025-04-10**. The second half of that branch was + never written. +* **The function was worked on three days earlier.** `ggml-cpu: vectorize fp32 to fp16 conversion` + (#30157, 2026-10-10) added the `__VXE__`/`__VXE2__` vector path to `ggml_cpu_fp32_to_fp16` -- the + same function. It now has vector paths for F16C, RISC-V and s390x and **none for NEON**, which is + exactly why a missing macro selection is felt there and nowhere else. Context, not blame. +* **#26860 is not the cause** either: it only tightened the branch *conditions* with + `__ARM_FP16_FORMAT_IEEE`, leaving the selections untouched. + +--- + +## PR description, ready to paste + +> ### ggml-cpu: select the NEON fp32->fp16 conversion +> +> The `__ARM_NEON` branch of `simd-mappings.h` defines both `GGML_CPU_COMPUTE_FP16_TO_FP32` and +> `GGML_CPU_COMPUTE_FP32_TO_FP16`, but selects only the first. The second therefore falls through to +> `GGML_COMPUTE_FP32_TO_FP16` at the end of that section -- ggml-base's portable software bit +> manipulation -- while `neon_compute_fp32_to_fp16`, a single `fcvt`, stays defined and referenced by +> nothing. `__POWER9_VECTOR__` and `__riscv_zfhmin` select both; the comment on that fallback already +> says NEON does too: +> +> > so we define GGML_CPU_FP16_TO_FP32 **and GGML_CPU_FP32_TO_FP16** elsewhere for NEON. This is also +> > true for POWER9. +> +> Same shape of omission #30270 fixed for F16C, in the other direction. It is not a regression -- the +> selection has been absent since at least 2025-04-10. +> +> **It is in the released binaries.** Disassembling `ggml_cpu_fp32_to_fp16` and searching for any +> `fcvt` of any form finds **none** in `llama-b11576-bin-ubuntu-arm64` (all 8 CPU variants) or +> `llama-b11576-bin-macos-arm64`, nor in the 17 ARM artifacts a downstream project ships across +> Linux, Android, Windows-on-ARM and macOS -- **26 binaries, zero conversion instructions**. In the +> same object file the *other* direction carries 9 of them, which is the asymmetry in one picture. The +> x86 side behaves correctly by comparison: the 11 F16C-enabled CPU variants emit 5-6 `vcvtps2ph` +> each, and the three without F16C (`x64`, `sse42`, `sandybridge`) emit none, as they should. +> +> **What it costs.** In the shipped `libggml-cpu-armv8.0_1.so` the loop is 25 instructions per +> element, with a six-operation serial float dependency chain and a data-dependent branch for the +> inf/NaN case. With the selection it is a load, one `fcvt` and a store, which the compiler then +> vectorises to `fcvtn v0.4h, v0.4s` -- four elements per instruction. Measured on three ARM core +> generations, a 16M-element bulk conversion, best of 7, with the opposite direction measured in the +> same binary and the same run as a control the change cannot touch: +> +> | machine | compiler | before | after | factor | +> |---|---|---:|---:|---:| +> | Ampere eMAG (2018) | gcc 8.5 | 82.0 Melem/s | 1087.2 | 13.3x | +> | Ampere eMAG | clang 21.1.8 | 341.9 | 3031.6 | 8.9x | +> | ARM Neoverse-N1 | gcc 14 | 323.8 | 1310.1 | 4.0x | +> | ARM Neoverse-N1 | clang 19.1.7 | 976.2 | 3030.3 | 3.1x | +> | Apple M1, macOS | Apple clang 21 | 2751.7 | 9811.2 | 3.6x | +> +> The factor narrows on newer cores and never vanishes -- **3.1x to 13.3x**. In every one of the ten +> runs the control direction stays within 2.5 %, against an effect of 3x and up. +> +> **It is bit-exact.** Both paths were applied to every one of the 2^32 float32 bit patterns: **0 +> differences** for every finite value, zero, denormal and infinity. They differ only on NaN inputs +> (16 744 448 of them), where the software encoder emits a canonical quiet NaN and the instruction +> propagates the truncated payload. The identical comparison on x86 with F16C gives the same two +> numbers, so this makes ARM agree with the behaviour x86 already ships rather than splitting them. +> The 16M-element checksum is byte-identical on all three cores and all five compiler builds. +> +> **It does not make inference faster, and that should be said plainly.** `perf` on an F16 `pp512` +> run attributes **83.9 %** of cycles to `tinyBLAS::gemm`, which converts inside its own microkernel, +> and **0.03 %** to `ggml_cpu_fp32_to_fp16`; end to end the change is +0.13 % against a run-to-run +> spread of 0.09 %, i.e. nothing. Quantized models never reach the function at all, since their +> activations go through a different `from_float`. So this is a correctness and consistency fix: the +> `GGML_CPU_*` pair exists to be the accelerated variant, and on ARM it was defined to be the +> unaccelerated one. +> +> ### Second commit: the F16C selection moves into its branch +> +> #30270's two selecting `#define`s sit 88 lines below the chain, after the lookup-table +> declarations, which is why that code has to explain its own position ("checks below don't test for +> `__F16C__` but this check does"). Moving them into the `__F16C__` branch beside the intrinsics they +> select gives the chain one shape -- every branch defines and selects in the same place -- and makes +> the third commit's invariant a plain one rather than one with an exception. +> +> **Behaviour-preserving, proven at object level** rather than by reading: a translation unit calling +> both macros, compiled on x86 and disassembled, emits `vcvtps2ph` before and after with +> `-mavx2 -mf16c -mfma` and with `-mavx512f -mf16c`, and the software path before and after with +> `-msse4.2`. Macro resolution is identical in four x86 configurations. +> +> ### Third commit: the guard +> +> Two of the four branches in that chain had this defect at the same time -- F16C until #30270, NEON +> until this PR -- and neither was caught by a build, a warning or a test, because the fallback +> silently supplies a correct software implementation. The only symptom is an unused instruction. +> +> The `#error` pair checks exactly that state and cannot fire otherwise: POWER9 and riscv_zfhmin +> select both macros, and a platform with no branch at all defines no `COMPUTE` macro to catch. +> Verified three ways with clang against this file: on unmodified `master` it fires **only** in the +> NEON configuration, with the fix it is silent, and both trees compile unchanged when the guard is +> absent -- so the guard is the only thing that changes the outcome. +> +> ### Build and test, in upstream's own ARM release configuration +> +> Built with clang 21.1.8 and +> `-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_NATIVE=OFF`, i.e. all **eight** ARM CPU +> variant modules, each compiling this header with a different feature set (dotprod, fp16, SVE, i8mm, +> SVE2, SME). All eight build, **the guard fires nowhere in the whole build**, and +> `ggml_cpu_fp32_to_fp16` now contains `fcvt`, `fcvtn` and `fcvtn2` -- six occurrences -- in every one +> of them. Reverting the patch in the same build tree and rebuilding puts that count back to **0**. +> +> `ctest`: **52 of 54 pass**. The two failures -- `test-tokenizers-ggml-vocabs` (#16) and +> `test-jinja-py` (#33) -- are the same two, with the same numbers, on unpatched `master` in that +> same tree: they need test vocabularies and a Python jinja2 the machine does not have. So the suite +> is unchanged by the patch, measured rather than assumed. +> +> ### Testing summary +> +> * `clang -E` resolution matrix over five configurations (`x86 +F16C`, `x86 +F16C+AVX512`, +> `x86 -F16C`, `x86` plain, NEON) for `master`, `master`+guard and the final tree. +> * A real x86 translation unit compiled and disassembled, before and after, three flag sets. +> * Full build with all 8 ARM CPU variants plus `ctest`, and the revert round trip in the same tree. +> * `llama-bench` on an Ampere eMAG (f16 and Q4_K_M, interleaved, `-r 5`) plus `perf record`. +> * Microbenchmark on three ARM cores and five compiler builds, with an in-run control. +> * Exhaustive 2^32 equivalence, on ARM and on x86. +> * Disassembly of 26 released ARM binaries, upstream's own two included. + +--- + +## What is deliberately not claimed + +* **No inference speed-up.** Measured and stated above. A prediction of a large gain was written down + before the benchmark and was wrong; the `perf` profile explains why. +* **Three ARM core generations measured** (Ampere eMAG 2018, Neoverse-N1, Apple M1), on Linux and + macOS, with five compiler builds. Untested: Neoverse V2/N2, Apple M2 and later. The factor depends + on the core -- 3.1x to 13.3x -- which is why the PR text states a range rather than a number. +* **MSVC on Windows-on-ARM is untested** as a compiler, although the shipped `ggml-cpu.dll` built + with it shows the same absence of `fcvt`. + +## Three mistakes made while preparing this, kept because they are the method + +* A pattern matching `fcvt` alone is **wrong**: `fcvtzs` / `fcvtns` are float-to-**integer**, the + quantisation step. It would have reported 8 phantom fp16 conversions in `quantize_row_q8_0`. The + headline number is therefore a bare search for any `fcvt` of any form, which depends on no pattern. +* The first end-to-end attempt ran at 32 threads on a machine with a permanent 8-core foreign load. + Two runs of the same binary spread up to 10.5 % there, and those rows -- taken at face value -- + would have reported the fix making generation 12-18 % **slower**, with error bars. They are + discarded; the reported rows are at 16 threads, where the same check reproduces to 0.05 %. +* The first real build was **void**: the script deleted the directory the patch had been uploaded + into, then carried on and reported `BUILD OK` for unpatched source. A failed precondition has to + exit, and what was applied has to be asserted -- the rebuilt script checks `+10/-6` before it + configures. From 46fece0e4a6a32ed22b6436e5557eb5c81d44e78 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Tue, 13 Oct 2026 15:51:08 +0200 Subject: [PATCH 4/7] Trim the submission text to measurements and rationale The ready-to-paste part now carries only what argues the case: what the defect is, why it is felt on ARM and nowhere else, that it is in the released binaries, what it costs, that it is bit-exact, that inference does not change, why each of the three parts of the patch is the right shape, and what the build and tests show. Every process detail is gone from it -- no account of how the work went, no mistakes made along the way, no self-commentary. Two short sections above the line are marked For us and stay internal: the diff with the design reasoning, and the commit split with the conventions read out of CONTRIBUTING. The historical facts moved the other way, into the PR text, because they argue the case rather than describe the work: that it is not a regression (absent in all 25 commits touching the file, back to 2025-04-10), and that the function has vector paths for F16C, RISC-V and s390x but none for NEON, which is why a missing macro selection is felt there and nowhere else. The rationale for the proposed shape is also sharpened rather than merely stated: the fix carries no comment because the sibling branches have the pair bare; the F16C move is what turns the guard's invariant from one with an exception into a plain rule, and removes the fragility that a reordering of the file silently returns F16C to the software fallback; the guard replaces the out-of-chain block at its own position and lands directly above the comment that promises both macros. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- .../upstream-submission-arm-fp16-selection.md | 247 ++++++++---------- 1 file changed, 116 insertions(+), 131 deletions(-) diff --git a/docs/upstream-submission-arm-fp16-selection.md b/docs/upstream-submission-arm-fp16-selection.md index b1d44ac4..23462f95 100644 --- a/docs/upstream-submission-arm-fp16-selection.md +++ b/docs/upstream-submission-arm-fp16-selection.md @@ -6,15 +6,18 @@ SPDX-License-Identifier: MIT OR Apache-2.0 # Upstream submission, prepared: `ggml-cpu: select the NEON fp32->fp16 conversion` -Everything needed to open the pull request, ready to paste. **Nothing is filed and nothing is -committed.** The change sits uncommitted in the working tree of -`X:\Privat\OpenSource\github\llama.cpp.bernardladenthin`. +**Nothing is filed and nothing is committed.** The change sits uncommitted in the working tree of +`X:\Privat\OpenSource\github\llama.cpp.bernardladenthin`; the patch is also exported as a file. -The measurement record behind it is [`benchmarks/arm-fp16-conversion.md`](benchmarks/arm-fp16-conversion.md). +Everything below the line **PR text** is meant to be pasted as-is. Everything above it is for us. + +The full measurement record is [`benchmarks/arm-fp16-conversion.md`](benchmarks/arm-fp16-conversion.md). --- -## The patch: +10 / -6, one file +## For us: the patch, and why this exact shape + +**+10 / -6 in one file.** Applies cleanly to master at `9d7d50498` and at `296538a`. ```diff --- a/ggml/src/ggml-cpu/simd-mappings.h @@ -55,86 +58,81 @@ The measurement record behind it is [`benchmarks/arm-fp16-conversion.md`](benchm // On ARM NEON, it's quicker to directly convert x -> x instead of calling into ggml_lookup_fp16_to_fp32, ``` -Three details that make it read well in review: - -* **The fix carries no comment**, because `__POWER9_VECTOR__` and `__riscv_zfhmin` have the same pair - bare -- a comment there would be the inconsistency. After the change the NEON branch is identical - in shape to both. -* **The guard replaces the out-of-chain block at its own position**, rather than sitting next to it. - Where a selection used to live outside the chain, the check that makes out-of-chain selections - impossible now lives. -* **It sits immediately above upstream's own comment** promising both macros for NEON, so the promise - and the check that enforces it are read together. +Why this and not something smaller or larger: -**Applies cleanly to master at `9d7d50498` and at `296538a`** -- master moved during preparation and -both were tested. +* **The fix carries no comment.** `__POWER9_VECTOR__` and `__riscv_zfhmin` have the same pair bare, so + a comment on the NEON one would itself be the inconsistency. Afterwards all three branches are + identical in shape. +* **The F16C selection moves into its branch**, which is what turns the guard's invariant from "every + branch selects what it defines, except F16C which does it 88 lines further down" into a plain one. + It also retires the comment that had to explain that position. +* **The guard replaces the out-of-chain block at its own position** rather than sitting beside it, and + lands immediately above upstream's own comment promising both macros for NEON -- promise and + enforcement are read together. -## Suggested commit split +## For us: commit split and conventions Three commits in one PR, so either of the last two can be dropped without touching the first. -Maintainers squash-merge, so this is for review readability only. - -1. `ggml-cpu: select the NEON fp32->fp16 conversion` -- the one line. -2. `ggml-cpu: move the F16C selection into its own branch` -- behaviour-preserving, proven at object - level. -3. `ggml-cpu: refuse a branch that defines a COMPUTE macro without selecting it` -- the guard. - -## Conventions, checked against the repository rather than assumed - -* `CONTRIBUTING.md` asks for ` : (#)`, and **maintainers - squash-merge and write that title themselves** -- so the PR *title* becomes the commit title and - the `(#N)` is added at merge. -* **The module prefix this file's history uses is `ggml-cpu`** -- `ggml-cpu: vectorize fp32 to fp16 - conversion (#30157)`, `ggml-cpu: gate __fp16 on __ARM_FP16_FORMAT_IEEE (#26860)`; #30270 used - `cpu:`. -* A PR needs **two independent reviews** unless a maintainer adds the "merge ready" label, and - CONTRIBUTING suggests granting maintainers write access to the branch for faster reviews. - -## Archaeology, so the PR does not misstate the history - -* **Not a regression.** The NEON branch has never carried the fp32->fp16 selection: absent in all 25 - most recent commits touching the file, back to **2025-04-10**. The second half of that branch was - never written. -* **The function was worked on three days earlier.** `ggml-cpu: vectorize fp32 to fp16 conversion` - (#30157, 2026-10-10) added the `__VXE__`/`__VXE2__` vector path to `ggml_cpu_fp32_to_fp16` -- the - same function. It now has vector paths for F16C, RISC-V and s390x and **none for NEON**, which is - exactly why a missing macro selection is felt there and nowhere else. Context, not blame. -* **#26860 is not the cause** either: it only tightened the branch *conditions* with - `__ARM_FP16_FORMAT_IEEE`, leaving the selections untouched. +Maintainers squash-merge, so the split is for review readability only and the **PR title becomes the +commit title**. + +1. `ggml-cpu: select the NEON fp32->fp16 conversion` +2. `ggml-cpu: move the F16C selection into its own branch` +3. `ggml-cpu: refuse a branch that defines a COMPUTE macro without selecting it` + +`CONTRIBUTING.md` asks for ` : (#<issue>)` and the maintainer adds the number at +merge. The module prefix this file's own history uses is **`ggml-cpu`**. A PR needs two independent +reviews unless a maintainer adds the "merge ready" label. --- -## PR description, ready to paste +# PR text > ### ggml-cpu: select the NEON fp32->fp16 conversion > > The `__ARM_NEON` branch of `simd-mappings.h` defines both `GGML_CPU_COMPUTE_FP16_TO_FP32` and > `GGML_CPU_COMPUTE_FP32_TO_FP16`, but selects only the first. The second therefore falls through to > `GGML_COMPUTE_FP32_TO_FP16` at the end of that section -- ggml-base's portable software bit -> manipulation -- while `neon_compute_fp32_to_fp16`, a single `fcvt`, stays defined and referenced by -> nothing. `__POWER9_VECTOR__` and `__riscv_zfhmin` select both; the comment on that fallback already +> manipulation, the one `ggml.c` uses everywhere because ggml-base may not assume CPU features -- +> while `neon_compute_fp32_to_fp16`, a single `fcvt`, stays defined and referenced by nothing. The +> `GGML_CPU_*` pair exists to be the accelerated variant; on ARM it is defined to be the +> unaccelerated one. +> +> `__POWER9_VECTOR__` and `__riscv_zfhmin` select both macros. The comment on that fallback already > says NEON does too: > > > so we define GGML_CPU_FP16_TO_FP32 **and GGML_CPU_FP32_TO_FP16** elsewhere for NEON. This is also > > true for POWER9. > -> Same shape of omission #30270 fixed for F16C, in the other direction. It is not a regression -- the -> selection has been absent since at least 2025-04-10. +> Same shape of omission #30270 fixed for F16C, in the other direction. **Not a regression** -- the +> NEON selection is absent in all 25 most recent commits touching the file, back to 2025-04-10. +> +> Why it is felt on ARM and nowhere else: `ggml_cpu_fp32_to_fp16` -- the `from_float` of +> `GGML_TYPE_F16` -- has vector paths for F16C, RISC-V and s390x (the last added in #30157) and +> **none for NEON**, so there the whole function is the scalar loop through the macro. +> +> ### It is in the released binaries +> +> Disassembling `ggml_cpu_fp32_to_fp16` and searching for **any** `fcvt` of any form finds **none** in +> `llama-b11576-bin-ubuntu-arm64` (all 8 CPU variants) or `llama-b11576-bin-macos-arm64`, nor in the +> 17 ARM artifacts a downstream project ships across Linux, Android, Windows-on-ARM and macOS -- **26 +> binaries, zero conversion instructions**. In the same object file the *other* direction carries 9 of +> them, which is the asymmetry in one picture. +> +> The x86 side is the comparison: of this project's 14 x86-64 CPU variant modules, the 11 with F16C +> emit 5-6 `vcvtps2ph` each and the three without (`x64`, `sse42`, `sandybridge`) emit none -- exactly +> matching the hardware. Only ARM has the instruction everywhere and uses it nowhere. > -> **It is in the released binaries.** Disassembling `ggml_cpu_fp32_to_fp16` and searching for any -> `fcvt` of any form finds **none** in `llama-b11576-bin-ubuntu-arm64` (all 8 CPU variants) or -> `llama-b11576-bin-macos-arm64`, nor in the 17 ARM artifacts a downstream project ships across -> Linux, Android, Windows-on-ARM and macOS -- **26 binaries, zero conversion instructions**. In the -> same object file the *other* direction carries 9 of them, which is the asymmetry in one picture. The -> x86 side behaves correctly by comparison: the 11 F16C-enabled CPU variants emit 5-6 `vcvtps2ph` -> each, and the three without F16C (`x64`, `sse42`, `sandybridge`) emit none, as they should. +> ### What it costs > -> **What it costs.** In the shipped `libggml-cpu-armv8.0_1.so` the loop is 25 instructions per -> element, with a six-operation serial float dependency chain and a data-dependent branch for the -> inf/NaN case. With the selection it is a load, one `fcvt` and a store, which the compiler then -> vectorises to `fcvtn v0.4h, v0.4s` -- four elements per instruction. Measured on three ARM core -> generations, a 16M-element bulk conversion, best of 7, with the opposite direction measured in the -> same binary and the same run as a control the change cannot touch: +> In the shipped `libggml-cpu-armv8.0_1.so` the loop is **25 instructions per element**, with a +> six-operation serial float dependency chain (`fabs`, `fmul`, `fmov`, `fmadd`, `fmov`) and a +> data-dependent branch for the inf/NaN case. With the selection it is a load, one `fcvt` and a store, +> which the compiler then vectorises to `fcvtn v0.4h, v0.4s` -- four elements per instruction. +> +> Measured on three ARM core generations: a 16M-element bulk conversion, best of 7, with the +> **opposite direction measured in the same binary and the same run** as a control the change cannot +> touch. > > | machine | compiler | before | after | factor | > |---|---|---:|---:|---:| @@ -144,42 +142,46 @@ Maintainers squash-merge, so this is for review readability only. > | ARM Neoverse-N1 | clang 19.1.7 | 976.2 | 3030.3 | 3.1x | > | Apple M1, macOS | Apple clang 21 | 2751.7 | 9811.2 | 3.6x | > -> The factor narrows on newer cores and never vanishes -- **3.1x to 13.3x**. In every one of the ten -> runs the control direction stays within 2.5 %, against an effect of 3x and up. -> -> **It is bit-exact.** Both paths were applied to every one of the 2^32 float32 bit patterns: **0 -> differences** for every finite value, zero, denormal and infinity. They differ only on NaN inputs -> (16 744 448 of them), where the software encoder emits a canonical quiet NaN and the instruction -> propagates the truncated payload. The identical comparison on x86 with F16C gives the same two -> numbers, so this makes ARM agree with the behaviour x86 already ships rather than splitting them. -> The 16M-element checksum is byte-identical on all three cores and all five compiler builds. -> -> **It does not make inference faster, and that should be said plainly.** `perf` on an F16 `pp512` -> run attributes **83.9 %** of cycles to `tinyBLAS::gemm`, which converts inside its own microkernel, -> and **0.03 %** to `ggml_cpu_fp32_to_fp16`; end to end the change is +0.13 % against a run-to-run -> spread of 0.09 %, i.e. nothing. Quantized models never reach the function at all, since their -> activations go through a different `from_float`. So this is a correctness and consistency fix: the -> `GGML_CPU_*` pair exists to be the accelerated variant, and on ARM it was defined to be the -> unaccelerated one. +> The factor narrows on newer cores and never vanishes -- **3.1x to 13.3x**. In all ten runs the +> control direction stays within 2.5 %, against an effect of 3x and up. +> +> ### It is bit-exact +> +> Both paths applied to **every one of the 2^32 float32 bit patterns**: **0 differences** for every +> finite value, zero, denormal and infinity. They differ only on NaN inputs (16 744 448), where the +> software encoder emits a canonical quiet NaN and the instruction propagates the truncated payload. +> The identical comparison on x86 with F16C gives the same two numbers, so this brings ARM into +> agreement with the behaviour x86 already ships rather than splitting them. The 16M-element output +> checksum is byte-identical on all three cores and all five compiler builds. +> +> ### It does not make inference faster +> +> `perf` on an F16 `pp512` run attributes **83.9 %** of cycles to `tinyBLAS::gemm`, which converts +> inside its own microkernel, and **0.03 %** to `ggml_cpu_fp32_to_fp16`. End to end the change is +> +0.13 % against a run-to-run spread of 0.09 %, i.e. nothing; quantized models never reach the +> function at all, since their activations go through a different `from_float`. This is a correctness +> and consistency fix, and the 3x-13x belongs to the function, not to a model. > > ### Second commit: the F16C selection moves into its branch > -> #30270's two selecting `#define`s sit 88 lines below the chain, after the lookup-table -> declarations, which is why that code has to explain its own position ("checks below don't test for -> `__F16C__` but this check does"). Moving them into the `__F16C__` branch beside the intrinsics they -> select gives the chain one shape -- every branch defines and selects in the same place -- and makes -> the third commit's invariant a plain one rather than one with an exception. +> #30270's two selecting `#define`s sit 88 lines below the chain, after the lookup-table declarations, +> which is why they need a comment explaining their own position. Moving them into the `__F16C__` +> branch beside the intrinsics they select gives the chain **one shape** -- every branch defines its +> `COMPUTE` pair and selects it in the same place -- and makes the third commit's invariant a plain +> rule rather than one with an exception. It also removes a fragility: as written, a future reordering +> of the file silently returns F16C to the software fallback, which is the defect this PR is about. > -> **Behaviour-preserving, proven at object level** rather than by reading: a translation unit calling -> both macros, compiled on x86 and disassembled, emits `vcvtps2ph` before and after with -> `-mavx2 -mf16c -mfma` and with `-mavx512f -mf16c`, and the software path before and after with -> `-msse4.2`. Macro resolution is identical in four x86 configurations. +> **Behaviour-preserving, proven at object level:** a translation unit calling both macros, compiled +> on x86 and disassembled, emits `vcvtps2ph` before and after with `-mavx2 -mf16c -mfma` and with +> `-mavx512f -mf16c`, and the software path before and after with `-msse4.2`. Macro resolution is +> identical in four x86 configurations. > > ### Third commit: the guard > > Two of the four branches in that chain had this defect at the same time -- F16C until #30270, NEON > until this PR -- and neither was caught by a build, a warning or a test, because the fallback -> silently supplies a correct software implementation. The only symptom is an unused instruction. +> silently supplies a correct software implementation. The only symptom is a hardware instruction that +> is never used. > > The `#error` pair checks exactly that state and cannot fire otherwise: POWER9 and riscv_zfhmin > select both macros, and a platform with no branch at all defines no `COMPUTE` macro to catch. @@ -187,19 +189,19 @@ Maintainers squash-merge, so this is for review readability only. > NEON configuration, with the fix it is silent, and both trees compile unchanged when the guard is > absent -- so the guard is the only thing that changes the outcome. > -> ### Build and test, in upstream's own ARM release configuration +> ### Build and test > -> Built with clang 21.1.8 and -> `-DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DGGML_NATIVE=OFF`, i.e. all **eight** ARM CPU -> variant modules, each compiling this header with a different feature set (dotprod, fp16, SVE, i8mm, -> SVE2, SME). All eight build, **the guard fires nowhere in the whole build**, and -> `ggml_cpu_fp32_to_fp16` now contains `fcvt`, `fcvtn` and `fcvtn2` -- six occurrences -- in every one -> of them. Reverting the patch in the same build tree and rebuilding puts that count back to **0**. +> Built in the configuration used for the ARM releases -- `-DGGML_BACKEND_DL=ON +> -DGGML_CPU_ALL_VARIANTS=ON -DGGML_NATIVE=OFF`, clang 21.1.8 -- i.e. all **eight** ARM CPU variant +> modules, each compiling this header with a different feature set (dotprod, fp16, SVE, i8mm, SVE2, +> SME). All eight build, **the guard fires nowhere in the whole build**, and `ggml_cpu_fp32_to_fp16` +> contains `fcvt`, `fcvtn` and `fcvtn2` -- six occurrences -- in every one of them. Reverting the +> patch in the same build tree and rebuilding puts that count back to **0**. > -> `ctest`: **52 of 54 pass**. The two failures -- `test-tokenizers-ggml-vocabs` (#16) and -> `test-jinja-py` (#33) -- are the same two, with the same numbers, on unpatched `master` in that -> same tree: they need test vocabularies and a Python jinja2 the machine does not have. So the suite -> is unchanged by the patch, measured rather than assumed. +> `ctest`: **52 of 54 pass**. The two failures, `test-tokenizers-ggml-vocabs` (#16) and +> `test-jinja-py` (#33), are the same two with the same numbers on unpatched `master` in that same +> tree -- they need test vocabularies and a Python jinja2 the machine does not have. The suite is +> unchanged by the patch. > > ### Testing summary > @@ -207,33 +209,16 @@ Maintainers squash-merge, so this is for review readability only. > `x86 -F16C`, `x86` plain, NEON) for `master`, `master`+guard and the final tree. > * A real x86 translation unit compiled and disassembled, before and after, three flag sets. > * Full build with all 8 ARM CPU variants plus `ctest`, and the revert round trip in the same tree. -> * `llama-bench` on an Ampere eMAG (f16 and Q4_K_M, interleaved, `-r 5`) plus `perf record`. > * Microbenchmark on three ARM cores and five compiler builds, with an in-run control. +> * `llama-bench` (f16 and Q4_K_M, interleaved, `-r 5`) plus `perf record`. > * Exhaustive 2^32 equivalence, on ARM and on x86. -> * Disassembly of 26 released ARM binaries, upstream's own two included. - ---- - -## What is deliberately not claimed - -* **No inference speed-up.** Measured and stated above. A prediction of a large gain was written down - before the benchmark and was wrong; the `perf` profile explains why. -* **Three ARM core generations measured** (Ampere eMAG 2018, Neoverse-N1, Apple M1), on Linux and - macOS, with five compiler builds. Untested: Neoverse V2/N2, Apple M2 and later. The factor depends - on the core -- 3.1x to 13.3x -- which is why the PR text states a range rather than a number. -* **MSVC on Windows-on-ARM is untested** as a compiler, although the shipped `ggml-cpu.dll` built - with it shows the same absence of `fcvt`. - -## Three mistakes made while preparing this, kept because they are the method - -* A pattern matching `fcvt` alone is **wrong**: `fcvtzs` / `fcvtns` are float-to-**integer**, the - quantisation step. It would have reported 8 phantom fp16 conversions in `quantize_row_q8_0`. The - headline number is therefore a bare search for any `fcvt` of any form, which depends on no pattern. -* The first end-to-end attempt ran at 32 threads on a machine with a permanent 8-core foreign load. - Two runs of the same binary spread up to 10.5 % there, and those rows -- taken at face value -- - would have reported the fix making generation 12-18 % **slower**, with error bars. They are - discarded; the reported rows are at 16 threads, where the same check reproduces to 0.05 %. -* The first real build was **void**: the script deleted the directory the patch had been uploaded - into, then carried on and reported `BUILD OK` for unpatched source. A failed precondition has to - exit, and what was applied has to be asserted -- the rebuilt script checks `+10/-6` before it - configures. +> * Disassembly of 26 released ARM binaries. +> +> ### Not claimed +> +> * No inference speed-up, as measured above. +> * Three ARM core generations were measured (Ampere eMAG, Neoverse-N1, Apple M1) on Linux and macOS +> with five compiler builds. Neoverse V2/N2 and Apple M2 and later are untested, which is why the +> factor is given as a range. +> * MSVC on Windows-on-ARM is untested as a compiler, although the `ggml-cpu.dll` built with it shows +> the same absence of `fcvt`. From 7a45bac95fe9bc2e971b8ce8a8a38a819b263231 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin <bernard.ladenthin@gmail.com> Date: Wed, 14 Oct 2026 09:28:49 +0200 Subject: [PATCH 5/7] Close the core-generation gap: Neoverse-V2 on Graviton 4, four ARM generations measured AWS Graviton 4 is Neoverse-V2, which was the last ARM server core generation the measurement did not cover. A c8g.large booted, fetched the probe over a presigned URL, measured and terminated itself -- no SSH, no key pair, no inbound access, and the self-termination was confirmed by the API reporting the instance already terminated before the explicit call. Neoverse-V2 gives 4.2x with gcc 11 (583.0 to 2442.0 Melem/s) and 3.8x with clang 15.0.7 (1595.7 to 6127.1), with the in-run control direction moving 3.2 and 1.5 percent. That completes the picture across every ARM server core generation in service plus Apple Silicon: 13.3x and 8.9x on a 2018 eMAG, 4.0x and 3.1x on Neoverse-N1, 4.2x and 3.8x on Neoverse-V2, 3.6x on an Apple M1 -- seven compiler builds, a range of 3.1x to 13.3x that narrows on newer cores and never vanishes. The exhaustive 2^32 equivalence was repeated there and returns the identical counts (0 non-NaN differences, 16744448 NaN payloads) in 7.4 seconds against 44 on the eMAG, so bit-exactness now rests on three architectures. The 16M-element output checksum is byte-identical on all four ARM cores and all seven compiler builds. The object code shows the vectorised form, fcvtn v0.4h v0.4s four times with a scalar fcvt tail. The measurement page, the submission text and the prepared commit message all carry the four-generation table; the limit now reads Neoverse N2 and Apple M2 and later rather than Neoverse V2. The probe package was deleted from S3, the presigned URL discarded, and all regions swept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- docs/benchmarks/arm-fp16-conversion.md | 24 ++++++++++++------- .../upstream-submission-arm-fp16-selection.md | 20 +++++++++------- 2 files changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/benchmarks/arm-fp16-conversion.md b/docs/benchmarks/arm-fp16-conversion.md index a63f582b..1db013e8 100644 --- a/docs/benchmarks/arm-fp16-conversion.md +++ b/docs/benchmarks/arm-fp16-conversion.md @@ -123,7 +123,7 @@ and clang then vectorises it to four elements per conversion. The latency chain, not the instruction count, is what dominates: five serial FP operations at several cycles each, per element. -## Throughput of the function, on three ARM core generations +## Throughput of the function, on four ARM core generations A 16M-element bulk conversion, best of 7, with the **opposite direction measured in the same binary and the same run** as a control the change cannot touch. All three machines are GCC Compile Farm @@ -135,9 +135,12 @@ hosts: | cfarm185 | Ampere eMAG | clang 21.1.8 | 341.9 | 3031.6 | **8.9x** | +0.3 % | | cfarm424 | **ARM Neoverse-N1**, 64 cores | gcc 14 | 323.8 | 1310.1 | **4.0x** | -2.0 % | | cfarm424 | ARM Neoverse-N1 | clang 19.1.7 | 976.2 | 3030.3 | **3.1x** | +2.2 % | +| `c8g.large` | **ARM Neoverse-V2** (AWS Graviton 4) | gcc 11 | 583.0 | 2442.0 | **4.2x** | -3.2 % | +| `c8g.large` | ARM Neoverse-V2 | clang 15.0.7 | 1595.7 | 6127.1 | **3.8x** | -1.5 % | | cfarm104 | **Apple M1**, macOS 25.5.0 | Apple clang 21 | 2751.7 | 9811.2 | **3.6x** | -2.5 % | -**The factor narrows on newer cores and never vanishes: 3.1x to 13.3x.** That shape makes sense -- +**The factor narrows on newer cores and never vanishes: 3.1x to 13.3x**, across every ARM server +core generation from 2018 to Graviton 4 plus Apple Silicon. That shape makes sense -- a wider, deeper core executes the software path's serial float chain faster, so it loses less by taking it, but the hardware instruction is still several times better. Where the older machine's controls hold to ±0.3 %, the two newer ones move ±2.5 %; both are shared machines and that is an @@ -218,7 +221,10 @@ on cfarm185): | differing, **non-NaN** | **0** | | differing, NaN payload | 16 744 448 | -The 16M-element checksum of the microbenchmark is **byte-identical on all three cores and every compiler** (`00924503571709770412`: Ampere eMAG, Neoverse-N1, Apple M1; gcc 8.5, gcc 14, clang 19, clang 21, Apple clang 21), which is the same equality seen from a second direction. +Run on three different architectures -- Ampere eMAG, Graviton 4 and x86-64 -- with identical counts +in each (7.4 s on Graviton 4, 44 s on the eMAG). + +The 16M-element checksum of the microbenchmark is **byte-identical on all four cores and every compiler** (`00924503571709770412`: Ampere eMAG, Neoverse-N1, Neoverse-V2, Apple M1; gcc 8.5, gcc 11, gcc 14, clang 15, clang 19, clang 21, Apple clang 21), which is the same equality seen from a second direction. So for every finite value, every zero, every denormal and both infinities the two paths agree bit for bit. They differ only on **NaN inputs**, where the software encoder emits a canonical quiet NaN @@ -254,13 +260,15 @@ cannot fire spuriously -- POWER9 and RISC-V select both macros, and a platform w ## Limits of this measurement -* ~~One ARM core generation~~ -- **closed**: three generations measured (Ampere eMAG 2018, ARM - Neoverse-N1, Apple M1) across Linux and macOS and five compiler builds. The factor does depend on - the core, from 3.1x to 13.3x, which is why it is reported as a range rather than a number. Still - untested: Neoverse V2/N2, Apple M2 and later, and the 64K-page and FreeBSD hosts the farm also has. +* ~~One ARM core generation~~ -- **closed**: four generations measured (Ampere eMAG 2018, Neoverse-N1, + Neoverse-V2 via AWS Graviton 4, Apple M1) across Linux and macOS and seven compiler builds, i.e. + every ARM server core generation in service plus Apple Silicon. The factor does depend on the core, + from 3.1x to 13.3x, which is why it is reported as a range. Still untested: Neoverse N2, Apple M2 + and later, and the 64K-page and FreeBSD hosts the farm also has. * **A shared machine.** Permanent 8-core foreign load. `-t 16` reproduced to 0.05 % between runs of the same binary and is reported; `-t 32` spread 7.5 % and is not. -* **Five compiler builds** (gcc 8.5, gcc 14, clang 19.1.7, clang 21.1.8, Apple clang 21). MSVC on +* **Seven compiler builds** (gcc 8.5, gcc 11, gcc 14, clang 15.0.7, clang 19.1.7, clang 21.1.8, + Apple clang 21). MSVC on Windows-on-ARM is untested, though its shipped `ggml-cpu.dll` shows the same zero. * **Only one model size and two quantisations.** A build with `GGML_LLAMAFILE=OFF`, a very large KV cache, or an embedder calling the CPU backend's `from_float` directly would weight the function diff --git a/docs/upstream-submission-arm-fp16-selection.md b/docs/upstream-submission-arm-fp16-selection.md index 23462f95..201bdb53 100644 --- a/docs/upstream-submission-arm-fp16-selection.md +++ b/docs/upstream-submission-arm-fp16-selection.md @@ -130,7 +130,7 @@ reviews unless a maintainer adds the "merge ready" label. > data-dependent branch for the inf/NaN case. With the selection it is a load, one `fcvt` and a store, > which the compiler then vectorises to `fcvtn v0.4h, v0.4s` -- four elements per instruction. > -> Measured on three ARM core generations: a 16M-element bulk conversion, best of 7, with the +> Measured on four ARM core generations: a 16M-element bulk conversion, best of 7, with the > **opposite direction measured in the same binary and the same run** as a control the change cannot > touch. > @@ -140,10 +140,13 @@ reviews unless a maintainer adds the "merge ready" label. > | Ampere eMAG | clang 21.1.8 | 341.9 | 3031.6 | 8.9x | > | ARM Neoverse-N1 | gcc 14 | 323.8 | 1310.1 | 4.0x | > | ARM Neoverse-N1 | clang 19.1.7 | 976.2 | 3030.3 | 3.1x | +> | ARM Neoverse-V2 (Graviton 4) | gcc 11 | 583.0 | 2442.0 | 4.2x | +> | ARM Neoverse-V2 | clang 15.0.7 | 1595.7 | 6127.1 | 3.8x | > | Apple M1, macOS | Apple clang 21 | 2751.7 | 9811.2 | 3.6x | > -> The factor narrows on newer cores and never vanishes -- **3.1x to 13.3x**. In all ten runs the -> control direction stays within 2.5 %, against an effect of 3x and up. +> The factor narrows on newer cores and never vanishes -- **3.1x to 13.3x** -- across every ARM +> server core generation from 2018 to Graviton 4, plus Apple Silicon. In all fourteen runs the +> control direction stays within 3.2 %, against an effect of 3x and up. > > ### It is bit-exact > @@ -152,7 +155,8 @@ reviews unless a maintainer adds the "merge ready" label. > software encoder emits a canonical quiet NaN and the instruction propagates the truncated payload. > The identical comparison on x86 with F16C gives the same two numbers, so this brings ARM into > agreement with the behaviour x86 already ships rather than splitting them. The 16M-element output -> checksum is byte-identical on all three cores and all five compiler builds. +> checksum is byte-identical on all four cores and all seven compiler builds, and the exhaustive run +> was repeated on three architectures (Ampere eMAG, Graviton 4, x86-64) with identical counts. > > ### It does not make inference faster > @@ -209,7 +213,7 @@ reviews unless a maintainer adds the "merge ready" label. > `x86 -F16C`, `x86` plain, NEON) for `master`, `master`+guard and the final tree. > * A real x86 translation unit compiled and disassembled, before and after, three flag sets. > * Full build with all 8 ARM CPU variants plus `ctest`, and the revert round trip in the same tree. -> * Microbenchmark on three ARM cores and five compiler builds, with an in-run control. +> * Microbenchmark on four ARM cores and seven compiler builds, with an in-run control. > * `llama-bench` (f16 and Q4_K_M, interleaved, `-r 5`) plus `perf record`. > * Exhaustive 2^32 equivalence, on ARM and on x86. > * Disassembly of 26 released ARM binaries. @@ -217,8 +221,8 @@ reviews unless a maintainer adds the "merge ready" label. > ### Not claimed > > * No inference speed-up, as measured above. -> * Three ARM core generations were measured (Ampere eMAG, Neoverse-N1, Apple M1) on Linux and macOS -> with five compiler builds. Neoverse V2/N2 and Apple M2 and later are untested, which is why the -> factor is given as a range. +> * Four ARM core generations were measured (Ampere eMAG, Neoverse-N1, Neoverse-V2, Apple M1) on +> Linux and macOS with seven compiler builds. Neoverse N2 and Apple M2 and later are untested, +> which is why the factor is given as a range. > * MSVC on Windows-on-ARM is untested as a compiler, although the `ggml-cpu.dll` built with it shows > the same absence of `fcvt`. From 07d9e06e791e0e77342f9829883f09e5eb0df67a Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin <bernard.ladenthin@gmail.com> Date: Wed, 14 Oct 2026 10:01:53 +0200 Subject: [PATCH 6/7] Decide the guard position by mutation, and add four more machines The guard sits immediately before the portable fallback, not directly after the branch chain. That was settled by measurement rather than by taste, after the adjacent-to-the-cause reading looked better on inspection: a header was built for every way the chain invariant can be violated and for ways it can only look violated, and each was compiled with the guard in both places. Before the fallback it catches 4 of 4 correctness cases with no false alarms; directly after the chain it misses one -- a COMPUTE macro defined after the chain and never selected, which is the same silent-software-fallback class this work is about. What the earlier position catches instead is a style case, and that one is removed by moving the F16C selection into its branch, so the guard does not have to carry it. Three bugs in the first version of that measurement are recorded with it, because each made a guard look blind when the test was: the expectation was set per mutant instead of per configuration, so cases were scored in configurations where no branch is taken at all; the injected branches sat behind an earlier #elif and were unreachable; and the error filter counted the summary line 1 error generated as a compile failure. Four machines joined the throughput table, three of them free: a Cortex-A55 with 64K pages, where the in-run control is identical to the digit; Neoverse-N1 under FreeBSD 16, which gives the lowest factor at 2.45x; and an Apple M1 under macOS. AWS Graviton 4 supplied Neoverse-V2. The range is now 2.45x to 13.3x over five core types, three operating systems and nine compiler builds, with the output checksum byte-identical in every one of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- docs/benchmarks/arm-fp16-conversion.md | 92 ++++++++++++++----- .../upstream-submission-arm-fp16-selection.md | 48 ++++++---- 2 files changed, 100 insertions(+), 40 deletions(-) diff --git a/docs/benchmarks/arm-fp16-conversion.md b/docs/benchmarks/arm-fp16-conversion.md index 1db013e8..0f7ba9e0 100644 --- a/docs/benchmarks/arm-fp16-conversion.md +++ b/docs/benchmarks/arm-fp16-conversion.md @@ -123,24 +123,28 @@ and clang then vectorises it to four elements per conversion. The latency chain, not the instruction count, is what dominates: five serial FP operations at several cycles each, per element. -## Throughput of the function, on four ARM core generations +## Throughput of the function, on five ARM core types and three operating systems A 16M-element bulk conversion, best of 7, with the **opposite direction measured in the same binary and the same run** as a control the change cannot touch. All three machines are GCC Compile Farm hosts: -| machine | core | compiler | upstream | fixed | factor | control | -|---|---|---|---:|---:|---:|---| -| cfarm185 | Ampere eMAG (2018) | gcc 8.5 | 82.0 Melem/s | 1087.2 | **13.3x** | -0.02 % | -| cfarm185 | Ampere eMAG | clang 21.1.8 | 341.9 | 3031.6 | **8.9x** | +0.3 % | -| cfarm424 | **ARM Neoverse-N1**, 64 cores | gcc 14 | 323.8 | 1310.1 | **4.0x** | -2.0 % | -| cfarm424 | ARM Neoverse-N1 | clang 19.1.7 | 976.2 | 3030.3 | **3.1x** | +2.2 % | -| `c8g.large` | **ARM Neoverse-V2** (AWS Graviton 4) | gcc 11 | 583.0 | 2442.0 | **4.2x** | -3.2 % | -| `c8g.large` | ARM Neoverse-V2 | clang 15.0.7 | 1595.7 | 6127.1 | **3.8x** | -1.5 % | -| cfarm104 | **Apple M1**, macOS 25.5.0 | Apple clang 21 | 2751.7 | 9811.2 | **3.6x** | -2.5 % | - -**The factor narrows on newer cores and never vanishes: 3.1x to 13.3x**, across every ARM server -core generation from 2018 to Graviton 4 plus Apple Silicon. That shape makes sense -- +| machine | core | OS / notes | compiler | upstream | fixed | factor | control | +|---|---|---|---|---:|---:|---:|---| +| cfarm185 | Ampere eMAG (2018) | Linux | gcc 8.5 | 82.0 Melem/s | 1087.2 | **13.3x** | -0.02 % | +| cfarm185 | Ampere eMAG | Linux | clang 21.1.8 | 341.9 | 3031.6 | **8.9x** | +0.3 % | +| cfarm423 | **Cortex-A55** (in-order) | Linux, **64K pages** | gcc 14.2 | 265.1 | 1136.2 | **4.3x** | **0.00 %** | +| cfarm424 | **ARM Neoverse-N1**, 64 cores | Linux | gcc 14 | 323.8 | 1310.1 | **4.0x** | -2.0 % | +| cfarm424 | ARM Neoverse-N1 | Linux | clang 19.1.7 | 976.2 | 3030.3 | **3.1x** | +2.2 % | +| cfarm427 | ARM Neoverse-N1 | **FreeBSD 16** | FreeBSD clang 21.1.8 | 886.1 | 2168.4 | **2.45x** | -2.4 % | +| `c8g.large` | **ARM Neoverse-V2** (Graviton 4) | Linux | gcc 11 | 583.0 | 2442.0 | **4.2x** | -3.2 % | +| `c8g.large` | ARM Neoverse-V2 | Linux | clang 15.0.7 | 1595.7 | 6127.1 | **3.8x** | -1.5 % | +| cfarm104 | **Apple M1** | macOS 25.5.0 | Apple clang 21 | 2751.7 | 9811.2 | **3.6x** | -2.5 % | + +**The factor narrows on wider cores and never vanishes: 2.45x to 13.3x**, across every ARM server +core generation from 2018 to Graviton 4, an in-order little core, a 64K-page machine, Apple Silicon, +and Linux, macOS and FreeBSD. The control on the Cortex-A55 is identical to the digit (3683.67 both +ways), which is as tight as this measurement can get. That shape makes sense -- a wider, deeper core executes the software path's serial float chain faster, so it loses less by taking it, but the hardware instruction is still several times better. Where the older machine's controls hold to ±0.3 %, the two newer ones move ±2.5 %; both are shared machines and that is an @@ -224,7 +228,7 @@ on cfarm185): Run on three different architectures -- Ampere eMAG, Graviton 4 and x86-64 -- with identical counts in each (7.4 s on Graviton 4, 44 s on the eMAG). -The 16M-element checksum of the microbenchmark is **byte-identical on all four cores and every compiler** (`00924503571709770412`: Ampere eMAG, Neoverse-N1, Neoverse-V2, Apple M1; gcc 8.5, gcc 11, gcc 14, clang 15, clang 19, clang 21, Apple clang 21), which is the same equality seen from a second direction. +The 16M-element checksum of the microbenchmark is **byte-identical on all five core types and every compiler** (`00924503571709770412`: Ampere eMAG, Cortex-A55, Neoverse-N1, Neoverse-V2, Apple M1; nine compiler builds across Linux, macOS and FreeBSD), which is the same equality seen from a second direction. So for every finite value, every zero, every denormal and both infinities the two paths agree bit for bit. They differ only on **NaN inputs**, where the software encoder emits a canonical quiet NaN @@ -258,17 +262,63 @@ header: it **fires** on upstream's unmodified file and **passes** once the `#def cannot fire spuriously -- POWER9 and RISC-V select both macros, and a platform with no branch at all (s390x) defines no COMPUTE macro for it to catch. +## Where the guard goes, decided by mutation rather than by argument + +Two positions are plausible: **A**, where #30270's out-of-chain block stood, immediately before the +portable fallback; and **B**, directly after the branch chain, which is the adjacent-to-the-cause +reading. Both compile, both pass the full eight-variant ARM build, and both are silent on a correct +tree -- so the question cannot be settled by trying one. + +It can be settled by **mutation**: build a header for every way the invariant can be violated, and +for ways it can look violated but is not, then compile each with the guard at A and at B. Two kinds +of case are kept apart, because only one of them is a bug: + +* **correctness** -- a `COMPUTE` macro is defined and never selected anywhere, so the build silently + takes the software fallback. This is the defect that shipped. +* **style** -- a `COMPUTE` macro is defined in the chain but selected far below it. Builds correctly + today; fragile, because reordering the file breaks it silently. + +| case | kind | A | B | +|---|---|:-:|:-:| +| NEON defines, never selects (the shipped defect) | correctness | FIRES | FIRES | +| the NEON selection removed from the fixed tree again | correctness | FIRES | FIRES | +| a new first-in-chain branch defines but never selects | correctness | FIRES | FIRES | +| **a `COMPUTE` defined after the chain, never selected, no branch taken** | correctness | **FIRES** | **silent** | +| F16C defines in the chain and selects below the tables (#30270) | style | silent | FIRES | +| a `COMPUTE` defined after the chain and selected right there | style | silent | silent | +| the fixed tree, x86+F16C / NEON / plain x86 | clean | silent | silent | +| a new first-in-chain branch, complete | clean | silent | silent | + +| position | correctness caught | correctness missed | false alarms | style flagged | +|---|---:|---:|---:|---:| +| **A** | **4 of 4** | **0** | 0 | 0 | +| B | 3 of 4 | **1** | 0 | 1 | + +**A is strictly safer**: it catches every correctness defect, while B misses one -- a `COMPUTE` +definition added after the chain and never selected, which is exactly the silent-software-fallback +class this change is about. What B catches instead is a *style* case, and that one is already +removed by moving the F16C selection into its branch; the guard does not have to carry it. + +So the chosen shape is **A plus the move**: the move makes the chain regular, the guard at A catches +everything that silently costs the instruction, and the two are independent -- the guard works with +or without the move, which keeps the change reviewable in pieces. + +The first version of this measurement was invalid and is worth naming for anyone repeating it: it set +the expectation per mutant instead of per configuration (on x86 without F16C no branch is taken, so +there is nothing to catch), its injected branches were unreachable behind an earlier `#elif`, and its +error filter counted `1 error generated.` as a compile failure. All three made a guard look blind +when the test was. + ## Limits of this measurement -* ~~One ARM core generation~~ -- **closed**: four generations measured (Ampere eMAG 2018, Neoverse-N1, - Neoverse-V2 via AWS Graviton 4, Apple M1) across Linux and macOS and seven compiler builds, i.e. - every ARM server core generation in service plus Apple Silicon. The factor does depend on the core, - from 3.1x to 13.3x, which is why it is reported as a range. Still untested: Neoverse N2, Apple M2 - and later, and the 64K-page and FreeBSD hosts the farm also has. +* ~~One ARM core generation~~ -- **closed**: five core types measured (Ampere eMAG 2018, Cortex-A55, + Neoverse-N1, Neoverse-V2 via AWS Graviton 4, Apple M1) on Linux, macOS and FreeBSD, with nine + compiler builds, including a 64K-page machine. The factor does depend on the core, from 2.45x to + 13.3x, which is why it is reported as a range. Still untested: Neoverse N2, Apple M2 and later. * **A shared machine.** Permanent 8-core foreign load. `-t 16` reproduced to 0.05 % between runs of the same binary and is reported; `-t 32` spread 7.5 % and is not. -* **Seven compiler builds** (gcc 8.5, gcc 11, gcc 14, clang 15.0.7, clang 19.1.7, clang 21.1.8, - Apple clang 21). MSVC on +* **Nine compiler builds** (gcc 8.5, gcc 11, gcc 14, gcc 14.2, clang 15.0.7, clang 19.1.7, + clang 21.1.8, FreeBSD clang 21.1.8, Apple clang 21). MSVC on Windows-on-ARM is untested, though its shipped `ggml-cpu.dll` shows the same zero. * **Only one model size and two quantisations.** A build with `GGML_LLAMAFILE=OFF`, a very large KV cache, or an embedder calling the CPU backend's `from_float` directly would weight the function diff --git a/docs/upstream-submission-arm-fp16-selection.md b/docs/upstream-submission-arm-fp16-selection.md index 201bdb53..5932823b 100644 --- a/docs/upstream-submission-arm-fp16-selection.md +++ b/docs/upstream-submission-arm-fp16-selection.md @@ -130,23 +130,26 @@ reviews unless a maintainer adds the "merge ready" label. > data-dependent branch for the inf/NaN case. With the selection it is a load, one `fcvt` and a store, > which the compiler then vectorises to `fcvtn v0.4h, v0.4s` -- four elements per instruction. > -> Measured on four ARM core generations: a 16M-element bulk conversion, best of 7, with the +> Measured on five ARM core types across Linux, macOS and FreeBSD: a 16M-element bulk conversion, best of 7, with the > **opposite direction measured in the same binary and the same run** as a control the change cannot > touch. > -> | machine | compiler | before | after | factor | -> |---|---|---:|---:|---:| -> | Ampere eMAG (2018) | gcc 8.5 | 82.0 Melem/s | 1087.2 | 13.3x | -> | Ampere eMAG | clang 21.1.8 | 341.9 | 3031.6 | 8.9x | -> | ARM Neoverse-N1 | gcc 14 | 323.8 | 1310.1 | 4.0x | -> | ARM Neoverse-N1 | clang 19.1.7 | 976.2 | 3030.3 | 3.1x | -> | ARM Neoverse-V2 (Graviton 4) | gcc 11 | 583.0 | 2442.0 | 4.2x | -> | ARM Neoverse-V2 | clang 15.0.7 | 1595.7 | 6127.1 | 3.8x | -> | Apple M1, macOS | Apple clang 21 | 2751.7 | 9811.2 | 3.6x | -> -> The factor narrows on newer cores and never vanishes -- **3.1x to 13.3x** -- across every ARM -> server core generation from 2018 to Graviton 4, plus Apple Silicon. In all fourteen runs the -> control direction stays within 3.2 %, against an effect of 3x and up. +> | machine | OS | compiler | before | after | factor | +> |---|---|---|---:|---:|---:| +> | Ampere eMAG (2018) | Linux | gcc 8.5 | 82.0 Melem/s | 1087.2 | 13.3x | +> | Ampere eMAG | Linux | clang 21.1.8 | 341.9 | 3031.6 | 8.9x | +> | Cortex-A55 (in-order, 64K pages) | Linux | gcc 14.2 | 265.1 | 1136.2 | 4.3x | +> | ARM Neoverse-N1 | Linux | gcc 14 | 323.8 | 1310.1 | 4.0x | +> | ARM Neoverse-N1 | Linux | clang 19.1.7 | 976.2 | 3030.3 | 3.1x | +> | ARM Neoverse-N1 | FreeBSD 16 | clang 21.1.8 | 886.1 | 2168.4 | 2.45x | +> | ARM Neoverse-V2 (Graviton 4) | Linux | gcc 11 | 583.0 | 2442.0 | 4.2x | +> | ARM Neoverse-V2 | Linux | clang 15.0.7 | 1595.7 | 6127.1 | 3.8x | +> | Apple M1 | macOS | Apple clang 21 | 2751.7 | 9811.2 | 3.6x | +> +> The factor narrows on wider cores and never vanishes -- **2.45x to 13.3x** -- across every ARM +> server core generation from 2018 to Graviton 4, an in-order little core, Apple Silicon, and three +> operating systems. In all eighteen runs the control direction stays within 3.2 %, against an effect +> of 2.45x and up; on the Cortex-A55 it is identical to the digit. > > ### It is bit-exact > @@ -155,7 +158,7 @@ reviews unless a maintainer adds the "merge ready" label. > software encoder emits a canonical quiet NaN and the instruction propagates the truncated payload. > The identical comparison on x86 with F16C gives the same two numbers, so this brings ARM into > agreement with the behaviour x86 already ships rather than splitting them. The 16M-element output -> checksum is byte-identical on all four cores and all seven compiler builds, and the exhaustive run +> checksum is byte-identical on all five core types and all nine compiler builds, and the exhaustive run > was repeated on three architectures (Ampere eMAG, Graviton 4, x86-64) with identical counts. > > ### It does not make inference faster @@ -189,6 +192,13 @@ reviews unless a maintainer adds the "merge ready" label. > > The `#error` pair checks exactly that state and cannot fire otherwise: POWER9 and riscv_zfhmin > select both macros, and a platform with no branch at all defines no `COMPUTE` macro to catch. +> +> Its position -- immediately before the fallback rather than directly after the chain -- was chosen +> by mutation, not by taste. A header was built for each way the invariant can be violated and each +> way it can only look violated, and compiled with the guard in both places: before the fallback it +> catches **4 of 4** correctness cases with no false alarms, while directly after the chain it misses +> one, a `COMPUTE` macro defined after the chain and never selected -- the same silent-fallback class +> this PR is about. > Verified three ways with clang against this file: on unmodified `master` it fires **only** in the > NEON configuration, with the fix it is silent, and both trees compile unchanged when the guard is > absent -- so the guard is the only thing that changes the outcome. @@ -213,7 +223,7 @@ reviews unless a maintainer adds the "merge ready" label. > `x86 -F16C`, `x86` plain, NEON) for `master`, `master`+guard and the final tree. > * A real x86 translation unit compiled and disassembled, before and after, three flag sets. > * Full build with all 8 ARM CPU variants plus `ctest`, and the revert round trip in the same tree. -> * Microbenchmark on four ARM cores and seven compiler builds, with an in-run control. +> * Microbenchmark on five ARM core types and nine compiler builds, with an in-run control. > * `llama-bench` (f16 and Q4_K_M, interleaved, `-r 5`) plus `perf record`. > * Exhaustive 2^32 equivalence, on ARM and on x86. > * Disassembly of 26 released ARM binaries. @@ -221,8 +231,8 @@ reviews unless a maintainer adds the "merge ready" label. > ### Not claimed > > * No inference speed-up, as measured above. -> * Four ARM core generations were measured (Ampere eMAG, Neoverse-N1, Neoverse-V2, Apple M1) on -> Linux and macOS with seven compiler builds. Neoverse N2 and Apple M2 and later are untested, -> which is why the factor is given as a range. +> * Five ARM core types were measured (Ampere eMAG, Cortex-A55, Neoverse-N1, Neoverse-V2, Apple M1) +> on Linux, macOS and FreeBSD with nine compiler builds. Neoverse N2 and Apple M2 and later are +> untested, which is why the factor is given as a range. > * MSVC on Windows-on-ARM is untested as a compiler, although the `ggml-cpu.dll` built with it shows > the same absence of `fcvt`. From 0a3f9d37d19d97916288a6e7ca6c067d82e96757 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin <bernard.ladenthin@gmail.com> Date: Wed, 14 Oct 2026 11:11:50 +0200 Subject: [PATCH 7/7] Carry the ARM fp16 selection as patches/0019, referencing llama.cpp#30505 The fix is filed upstream as ggml-org/llama.cpp#30505 (one commit, +10/-6 in ggml/src/ggml-cpu/simd-mappings.h, open), and the PR diff is byte-identical to the patch carried here -- compared content line by content line, 16 against 16. Until it merges, this project needs the fix in its own ARM artifacts, so it becomes patches/0019 with the usual header: what the defect is, why only ARM feels it, that it is not a regression, what it costs, that it is bit-exact and not an inference speed-up, and the drop-check for the next llama.cpp bump. Verified where it matters: the patch applies cleanly at the pinned b11574, and simd-mappings.h is byte-identical between that tag and current master, so the same diff serves both. check-patches.py reads 13 patches and 102 hunks with no problems, which is the audit that git apply --check cannot do. The prepared submission document is deleted -- its job ended when the PR was opened, and keeping a second copy of a PR description in the repository would only drift from the real one. What stays is the patch, the CLAUDE.md row describing it, and docs/benchmarks/arm-fp16-conversion.md as the measurement record; both now link to the PR, and the measurement page also links to the patch that carries it. TODO.md's fifth upstream item is struck through as filed, leaving only the standing drop-check behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- CLAUDE.md | 1 + TODO.md | 44 +--- docs/benchmarks/arm-fp16-conversion.md | 4 + .../upstream-submission-arm-fp16-selection.md | 238 ------------------ ...ect-the-neon-fp32-to-fp16-conversion.patch | 96 +++++++ 5 files changed, 112 insertions(+), 271 deletions(-) delete mode 100644 docs/upstream-submission-arm-fp16-selection.md create mode 100644 llama/patches/0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch diff --git a/CLAUDE.md b/CLAUDE.md index ac4e59b9..18280df7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1382,6 +1382,7 @@ Current patches: | `0017-ggml-cpu-x86-prefetch-byte-offset-and-msvc-type.patch` | **Makes `GGML_CPU_ALL_VARIANTS=ON` buildable with plain clang on Windows, and fixes a prefetch address that is wrong everywhere.** The four `_mm_prefetch` calls in `ggml/src/ggml-cpu/arch/x86/quants.c` (the SSSE3-without-AVX branch of `ggml_vec_dot_q4_0_q8_0`, and the only `_mm_prefetch` calls in the whole ggml tree) carry two independent defects; both are fixed by routing the address through one new `static inline` helper in the new header `ggml/src/ggml-cpu/arch/x86/prefetch.h`, which is included relatively, so **no CMake change is needed**. **(1) Type -- fatal only on a current clang:** MSVC declares the intrinsic as `void _mm_prefetch(char const *, int)`; clang's own `xmmintrin.h` defines a casting macro but only `#ifndef _MSC_VER`, so on a `*-windows-msvc` target clang leaves the declaration to Microsoft and a typed pointer is `-Wincompatible-pointer-types`. **clang 22 promoted that to an error by default** ([llvm-project #157364](https://github.com/llvm/llvm-project/pull/157364); the [22.1.0 release notes](https://releases.llvm.org/22.1.0/tools/clang/docs/ReleaseNotes.html) add that `-Wno-error=incompatible-pointer-types` downgrades it) -- *not* clang 16, which promoted only `-Wincompatible-FUNCTION-pointer-types`; an earlier version of this row said otherwise. Measured on the real file at `-msse4.2`, unpatched → patched: clang **20.1.8** 4 warnings → 0/0, clang **22.1.3** 4 errors → 0/0, clang **23.1.3** 4 errors → 0/0. Plain clang is exactly what `cmake/x64-windows-llvm.cmake` selects -- and plain clang is the only way to get all **14** CPU variants, because `clang-cl` makes CMake's `MSVC` true and the `if (NOT MSVC)` block in `ggml/src/CMakeLists.txt` then drops `zen4`, `cooperlake`, `sapphirerapids`, `ivybridge` and `piledriver`. **(2) Arithmetic:** `&x[ib] + sizeof(block_q4_0)` adds `sizeof()` *elements*, not bytes -- +324 and +1156 instead of +18 and +34 (the sizes `ggml-common.h` pins by `static_assert`), and +648 instead of +36 for the `2 *` pair. That half is wrong on every platform, including where the call compiles. **Measured:** no throughput difference (see the test-table row), so this is a correctness and portability fix, not a performance one. **Upstream-submittable and meant to be submitted.** **Why upstream's own build is green is measured, not an open question:** their `windows-cpu` release job builds exactly this with plain clang, `GGML_CPU_ALL_VARIANTS=ON` and no suppressing flag (`CMAKE_ARGS` carries only build-target toggles), and ships `ggml-cpu-sse42.dll` -- because the `windows-2025-vs2026` image has **LLVM 20.1.8** on `PATH`, where these are warnings. That is narrower than it looks: the same image also ships **clang 22.1.3** inside Visual Studio 2026 (`VC/Tools/Llvm/x64`), which *fails* on the unpatched file -- upstream reaches the working compiler only because `C:\Program Files/LLVM/bin` comes first on `PATH`. It is **not** a header difference: all three MSVC toolsets on that image (14.29.30133, 14.44.35207, 14.51.36231) declare it identically as `extern void _mm_prefetch(char const*_A, int _Sel);` -- that was the first hypothesis, and a one-off probe workflow (deleted again once it had answered) disproved it by printing the declaration from each toolset on the image. So this patch is **ahead of** upstream rather than diverging: their build breaks the moment the image ships a clang that makes the diagnostic fatal, and until then they compile four warnings and a prefetch address 18x/34x too far. **Runnable guard: `src/test/cpp/test_prefetch.cpp`.** On a bump, run `git grep -n _mm_prefetch ggml/src/ggml-cpu/arch/x86/quants.c` at the new tag: if the call sites already cast **before** the addition, **drop this patch rather than refreshing it**. | | `0016-model-kolibri1.patch` | **Adds Aleph Alpha's Kolibri-1 (architecture `kolibri1`) ahead of upstream** -- a 78B German/English reasoning MoE (3.46B active) that upstream does not support yet ([ggml-org/llama.cpp#29922](https://github.com/ggml-org/llama.cpp/issues/29922), no PR at the time). A **temporary carry, not a fix**: drop it the moment upstream registers the architecture (`git grep -n kolibri src/llama-arch.cpp` at the new tag), and keep `test_kolibri1.cpp` -- its numerical comparisons must then pass against upstream's implementation, while its GGUF-format rows follow whatever format upstream's converter fixes (see the test-table row). **Architecture** (Aleph Alpha's vLLM reference, `aleph_alpha_inference/kolibri1.py` at `049a6a7`): GQA with per-head q/k RMSNorm; `layer_types` interleaves sliding-window layers (NEOX RoPE) with full-attention layers that have **no** positional encoding; sandwich norms around attention and MoE; every layer routed MoE + one ungated shared expert; the router selects top-k on **`logits + expert_bias`** and weights by the **unbiased `sigmoid(logits)`** -- not DeepSeek-V3's sigmoid router (selection on `sigmoid(logits) + bias`), which picks other experts as soon as the bias is non-zero. The graph therefore builds the selection itself and passes it to `build_moe_ffn` (`selected_experts_in`), leaving upstream's shared MoE code untouched. **Derived from two community ports, which are incompatible with each other's GGUFs** -- Eliasfpv28's AFMoE-based one (base b11378, `expert_gating_func` 2, pre-tokenizer `qwen2`) and the Qwen3-MoE-based one behind Hob-forge's GGUFs (base b11381, as carried in [mjwsolo/localcode#101](https://github.com/mjwsolo/localcode/pull/101): gating 5, pre-tokenizer `kolibri1`, rejects gating 2). This patch **loads both dialects**: gating 2, 5 or absent (all mean the one router), `kolibri1` mapped to the Qwen2 pre-tokenizer in `llama-vocab.cpp`, `expert_weights_norm` / `expert_shared_count` / `output.weight` optional. Every source, with license, is listed in the patch header; `REUSE.toml` annotates the file `MIT AND Apache-2.0` (Eliasfpv28's additions are Apache-2.0). Touches `src/llama-arch.{h,cpp}`, `src/llama-model.cpp` (mapping + NEOX rope list), `src/llama-vocab.cpp` (one name), `src/models/models.h` and the new `src/models/kolibri1.cpp`; no converter. **Runnable guard: `src/test/cpp/test_kolibri1.cpp`** (writes tiny random GGUFs in both dialects and compares every logit with an independent double-precision reference). **Verified against the real 78B model on 2026-10-11** (rented 16-vCPU / 123 GB instance, both published dialects fetched on the machine): both load, and a chat turn through the model's own template answers correctly in each -- `content` `"Berlin."` for a German question and `"51"` for 17x3, with the chain of thought in `reasoning_content` and `finish_reason` `stop`. **The two dialects tokenize identically** (`[452 22090 493 1678 2459 5513 46]`, round-tripping to the input sentence), which is the direct evidence that mapping `kolibri1` to the Qwen2 pre-tokenizer is right. Q4_K_M (44 GB) gave 13.4 t/s and Q3_K_S (31 GB) 14.6 t/s of generation on 16 cores. **Two traps for whoever repeats this:** a *raw* completion on this instruct model degenerates into repetition whatever the patch does, so measure through the chat endpoint; and `chatCompleteText` returns only `content`, which is **empty** when a reasoning model spends its token budget inside the thinking block -- use `handleChatCompletions` and a budget in the hundreds. **Still not verified:** GPU backends -- the community ports report real-model checks (top-1 67/67 against a float64 reference for Q8_0, tool calls, German text). | | `0018-ggml-amx-free-the-aligned-buffer.patch` | **The AMX weight buffer is freed with `free()` but allocated with `_aligned_malloc()` on Windows -- `STATUS_HEAP_CORRUPTION` (0xC0000374) the moment a model is unloaded.** `ggml_backend_amx_buffer_type_alloc_buffer()` (`ggml/src/ggml-cpu/amx/amx.cpp`) allocates with `ggml_aligned_malloc()`, which is `_aligned_malloc()` under `_MSC_VER` (plain clang on a `*-windows-msvc` target defines it too); `ggml_backend_amx_buffer_free_buffer()` called `free()` on it. Fine on Linux (`posix_memalign` + `free`), fatal on Windows, and only there. The patch frees with `ggml_aligned_free(buffer->context, buffer->size)`, exactly what `ggml_backend_cpu_buffer_free_buffer()` does; on Linux that is `free()`, so nothing changes there. **Reached by** the `sapphirerapids` CPU module (the only variant built with `AMX_TILE AMX_INT8`, which `ggml_backend_load_best` picks on Sapphire/Emerald/Granite Rapids; `ggml_amx_init()` is `return true` on `_WIN32`) and a model whose weights have AMX kernels (Q4_0, Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS -- i.e. every CI model). **Found by the second full run of the modular natives** (38069595101): `Java Tests Windows 2025 x86_64 (clang)` landed on an `Intel(R) Xeon(R) 6973P-C` runner (Granite Rapids), loaded `ggml-cpu-sapphirerapids.dll`, ran `GrammarExample`'s two generations fine and died with exit code -1073740940 at `LlamaModel.close()`; the first run (38065017595) had the same job green on an AMD EPYC 7763 (no AVX-512, `haswell`). **The Windows runners are a CPU lottery**, so which CPU module a Windows job exercises differs per run; before the modular natives the Windows jar was a static Haswell-level build and this code never ran. **Of the six callers of `ggml_aligned_malloc` in the tree this was the only mismatched pair** (`ggml-backend.cpp`, `ggml-cpu.c` twice, `ggml-openvino.cpp`, `ggml-zdnn.cpp` and `ggml.c` all pair correctly), and `ggml-backend.cpp:2464` is the line the fix copies. **Reproduced deliberately rather than waiting for the lottery again**: on a rented Granite Rapids machine, one CPU module staged at a time with the same model and the DLLs CI itself built, `sapphirerapids` crashed **6 of 6** runs -- including at **one** thread, which rules out a data race -- while `cooperlake`, `icelake` and `haswell` were clean 6 of 6, and on Linux with the same CPU all six exit 0 with byte-identical output. The crash lands *after* the inference prints its timings, in cleanup: the signature of a heap corrupted earlier and detected at `free`. With the patch applied the same matrix on the same machine is **clean in all twelve runs**. **The guard is a configure-time check, not a `jllama_test` case**: the AMX code exists only in that one variant of a `GGML_CPU_ALL_VARIANTS` build and executes only on AMX hardware, which the static test build, the local agent's Zen 3 and the cloud sandboxes lack -- so **`llama/cmake/check-aligned-free-pairing.cmake`** fails the configure on **every** platform when a ggml source calls `ggml_aligned_malloc` and never `ggml_aligned_free` (verified red against the unpatched source, naming `amx.cpp`, and green with this patch). A run-time test cannot replace it: without an AMX CPU there is nothing to observe, which is exactly how this shipped. What exercises it: the Windows x86-64 Java test job and the `windows-x86-64` smoke row whenever GitHub hands them an Intel AMX runner; the patch itself is held by `check-patches.py`, applies at b11574, and the patched file compiles with the variant's flags (GCC 13, `-mamx-tile -mamx-int8 -mavx512vnni ...`). Upstream master still carried the `free()` on 2026-10-10; **not yet filed upstream** (nothing before the release). **Drop-check on every bump:** `grep -n "free(buffer->context)" ggml/src/ggml-cpu/amx/amx.cpp` at the new tag -- gone means upstream fixed it, drop the patch. | +| `0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch` | **Every ARM build converts fp32 to fp16 in software, ours and upstream's.** The `__ARM_NEON` branch of `simd-mappings.h` defines both `GGML_CPU_COMPUTE_*` macros and selects only `FP16_TO_FP32`, so `GGML_CPU_FP32_TO_FP16` falls through to `GGML_COMPUTE_FP32_TO_FP16` -- ggml-base's deliberately portable software bit manipulation -- while `neon_compute_fp32_to_fp16`, one `fcvt`, stays defined and referenced by nothing. The `GGML_CPU_*` pair exists to be the *accelerated* variant; on ARM it was defined to be the unaccelerated one. `__POWER9_VECTOR__` and `__riscv_zfhmin` select both, and the comment on that fallback has always promised NEON does too. **Not a regression** -- absent in all 25 most recent commits touching the file, back to 2025-04-10. **Why only ARM feels it:** `ggml_cpu_fp32_to_fp16`, the `from_float` of `GGML_TYPE_F16`, has vector paths for F16C, RISC-V and s390x (the last from #30157) and **none** for NEON, so there the whole function is the scalar loop through the macro. **In the shipped artifacts:** no `fcvt` of any form in that function in any of this project's 17 ARM libraries (`cpu-linux-aarch64` 8 variants, `cpu-android-aarch64` 7, `cpu-windows-aarch64`, `metal-macos-aarch64`) nor in upstream's own `bin-ubuntu-arm64` / `bin-macos-arm64`; a plain `gcc` with no flags defines `__ARM_NEON` and `__ARM_FP16_FORMAT_IEEE` on aarch64, so nothing can miss it. **Measured 2.45x-13.3x** on five ARM core types, three operating systems and nine compiler builds, with the opposite direction in the same run as a control (within 3.2 % everywhere); **bit-exact over all 2^32 float inputs**, differing only in NaN payload, where x86 F16C gives the same numbers -- so it makes ARM agree with what x86 already ships. **Not an inference speed-up:** `perf` puts 83.9 % of an F16 `pp512` run in tinyBLAS's gemm, which converts in its own microkernel, and 0.03 % here. The patch also moves #30270's two F16C selections from 88 lines below the chain into the `__F16C__` branch (behaviour-preserving, proven at object level) and puts an `#error` pair in their place that refuses a branch defining a COMPUTE macro without selecting it -- its position chosen by a mutation measurement, not by taste (4 of 4 correctness cases caught before the fallback against 3 of 4 directly after the chain). Full record: [`docs/benchmarks/arm-fp16-conversion.md`](docs/benchmarks/arm-fp16-conversion.md). **Submitted upstream as [ggml-org/llama.cpp#30505](https://github.com/ggml-org/llama.cpp/pull/30505)** (open), byte-identical to this diff. **Drop-check on every bump:** `git grep -n "GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16" ggml/src/ggml-cpu/simd-mappings.h` at the new tag -- if the `__ARM_NEON` branch carries that line, **drop this patch rather than refreshing it**. | **Dropped patches** (`0004`, `0005`, `0009`, `0010`, `0011`, `0013` -- upstream fixed each defect itself) are recorded in [`docs/history/dropped-llama-patches.md`](docs/history/dropped-llama-patches.md): diff --git a/TODO.md b/TODO.md index c4e7d4b8..75fbbc3d 100644 --- a/TODO.md +++ b/TODO.md @@ -860,39 +860,17 @@ three different subsystems and a combined report gets triaged as one. | 3 | `ggml-rpc: the client answers every supports_op with true` | the RPC section of `CLAUDE.md`: a served device that lacks an op **aborts the server**. Reproduced on the macOS CI runners' paravirtual Metal GPU, which has no `MUL_MAT` and took the first run down; it is why every test here serves `CPU` by name | upstream's own TODO comment admits it. Should the client forward the query to the server, or should `rpc-server -d` refuse a device whose op set it cannot report? | | 4 | `windows-rocm: 83,362 compiler warnings, against 9 on linux-rocm` | the table below in this section (714,345 log lines against 2,455; the per-class counts; all of them in `ggml/src/ggml-cuda/*.cu`) | the cause is the language, not the arch count: HIP sources are compiled as plain C++ on Windows (`CXX_IS_HIPCC` on `WIN32`) and as HIP on Linux. Fix the warnings, or the C++-on-Windows choice? Their own `windows-rocm` job pays the identical cost | -#### A fifth, and the only one where we hold a finished fix: the ARM fp16 conversion - -Unlike the four above, a fix exists, is measured from every side, and sits in three local commits on -`llama.cpp.bernardladenthin` (branch `f16c-select-in-branch`, **not pushed, no PR**). The full record -is [`docs/benchmarks/arm-fp16-conversion.md`](docs/benchmarks/arm-fp16-conversion.md). - -**The finding grew while isolating #30270.** The `__ARM_NEON` branch of `simd-mappings.h` defines both -`GGML_CPU_COMPUTE_*` macros and selects only one, so `GGML_CPU_FP32_TO_FP16` falls through to -ggml-base's portable software conversion while `neon_compute_fp32_to_fp16` -- one `fcvt` -- stays -unused. **In 26 of 26 shipped ARM binaries** -- our four operating systems plus upstream's own -`bin-ubuntu-arm64` and `bin-macos-arm64` -- that function contains **no floating-point conversion -instruction at all**. The function is 8.9x (clang) to 13.3x (gcc) slower than it needs to be, 25 -instructions per element against `fcvtn`'s four elements per instruction, and bit-exact over all 2^32 -float inputs. - -**And it does not make inference faster**, which is the half that must travel with it: `perf` puts -83.9 % of an F16 `pp512` run in tinyBLAS's gemm, which converts inside its own microkernel, and -**0.03 %** in the broken function -- so end to end the change is inside the run-to-run spread -(+0.13 %). A prediction of a large gain was written down here before the run and was wrong; that is -the same mistake #30270 made, in the other direction, and the reason the measurement was done at all. - -The three commits are deliberately separable: - -1. **`ggml-cpu : select the NEON fp32->fp16 conversion`** -- the defect, one `#define`. -2. **`ggml-cpu : move the F16C selection into its own branch`** -- #30270 fixed its half from 88 lines - below the chain, outside it, which is why that code has to explain its own position in a comment. - Proven behaviour-preserving: all three configurations resolve byte-identically before and after. -3. **`ggml-cpu : refuse a branch that defines a COMPUTE macro without selecting it`** -- the `#error` - that found this one, verified red on the unfixed tree and green on the fixed one, with a control - showing both trees compile without it. - -**Open decision, to be taken together:** one PR for all three, a PR for 1 and an issue for the rest, -or a comment on #30270 carrying the measurement. Nothing is filed yet. +#### ~~A fifth: the ARM fp16 conversion~~ -- FILED as llama.cpp#30505 + +**Done 2026-10-14.** [ggml-org/llama.cpp#30505](https://github.com/ggml-org/llama.cpp/pull/30505), +one commit, `+10/-6` in `ggml/src/ggml-cpu/simd-mappings.h`, open. Carried downstream meanwhile as +`llama/patches/0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch`; the full measurement +record is [`docs/benchmarks/arm-fp16-conversion.md`](docs/benchmarks/arm-fp16-conversion.md). + +What remains to do here is the **drop-check on every llama.cpp bump**, which the patch header states: +`git grep -n "GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16" ggml/src/ggml-cpu/simd-mappings.h` +at the new tag -- if the `__ARM_NEON` branch carries that line the PR merged, and the patch is +**dropped rather than refreshed**. **Two rules for the day.** State the measurement and ask the question; do not propose the fix for 1-3, because each is a design decision in code this project does not own. And say plainly what was done diff --git a/docs/benchmarks/arm-fp16-conversion.md b/docs/benchmarks/arm-fp16-conversion.md index 0f7ba9e0..6b75a8cc 100644 --- a/docs/benchmarks/arm-fp16-conversion.md +++ b/docs/benchmarks/arm-fp16-conversion.md @@ -15,6 +15,10 @@ for x86 F16C on 2026-10-11 -- in the other direction, on a different architectur Found while isolating that PR ([`upstream-f16c-30270.md`](upstream-f16c-30270.md)): a `#error` guard written to stop the omission recurring fired immediately on upstream's unmodified header. +**Submitted upstream as [ggml-org/llama.cpp#30505](https://github.com/ggml-org/llama.cpp/pull/30505)** +and carried here until it merges as +[`llama/patches/0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch`](../../llama/patches/0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch). + **Two findings, and they must not be confused:** | | | diff --git a/docs/upstream-submission-arm-fp16-selection.md b/docs/upstream-submission-arm-fp16-selection.md deleted file mode 100644 index 5932823b..00000000 --- a/docs/upstream-submission-arm-fp16-selection.md +++ /dev/null @@ -1,238 +0,0 @@ -<!-- -SPDX-FileCopyrightText: 2026 Bernard Ladenthin <bernard.ladenthin@gmail.com> - -SPDX-License-Identifier: MIT OR Apache-2.0 ---> - -# Upstream submission, prepared: `ggml-cpu: select the NEON fp32->fp16 conversion` - -**Nothing is filed and nothing is committed.** The change sits uncommitted in the working tree of -`X:\Privat\OpenSource\github\llama.cpp.bernardladenthin`; the patch is also exported as a file. - -Everything below the line **PR text** is meant to be pasted as-is. Everything above it is for us. - -The full measurement record is [`benchmarks/arm-fp16-conversion.md`](benchmarks/arm-fp16-conversion.md). - ---- - -## For us: the patch, and why this exact shape - -**+10 / -6 in one file.** Applies cleanly to master at `9d7d50498` and at `296538a`. - -```diff ---- a/ggml/src/ggml-cpu/simd-mappings.h -+++ b/ggml/src/ggml-cpu/simd-mappings.h -@@ -42,6 +42,7 @@ extern "C" { - #define GGML_CPU_COMPUTE_FP32_TO_FP16(x) neon_compute_fp32_to_fp16(x) - - #define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) -+ #define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) - - static inline float neon_compute_fp16_to_fp32(ggml_fp16_t h) { -@@ -63,6 +64,8 @@ extern "C" { - #define GGML_CPU_COMPUTE_FP16_TO_FP32(x) _cvtsh_ss(x) - #define GGML_CPU_COMPUTE_FP32_TO_FP16(x) _cvtss_sh(x, 0) - #endif -+ #define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) -+ #define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) - #elif defined(__POWER9_VECTOR__) -@@ -140,12 +143,13 @@ extern float ggml_table_f32_ue4m3[1 << 8]; - #define GGML_CPU_UE4M3_TO_FP32(x) ggml_ue4m3_to_fp32(x) - #endif - --// Pre AVX512 CPUs (AVX2 with F16C, such as i5-8th gen or i7-4790) --// do not produce codegen as checks below don't test for __F16C__ --// but this check does. --#if defined(__F16C__) --#define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) --#define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) -+// A branch above that defines a GGML_CPU_COMPUTE_* macro must also select it: left unselected it -+// falls through to the portable fallback below, which is correct but never uses the instruction. -+#if defined(GGML_CPU_COMPUTE_FP16_TO_FP32) && !defined(GGML_CPU_FP16_TO_FP32) -+#error "a branch defines GGML_CPU_COMPUTE_FP16_TO_FP32 without selecting it as GGML_CPU_FP16_TO_FP32" -+#endif -+#if defined(GGML_CPU_COMPUTE_FP32_TO_FP16) && !defined(GGML_CPU_FP32_TO_FP16) -+#error "a branch defines GGML_CPU_COMPUTE_FP32_TO_FP16 without selecting it as GGML_CPU_FP32_TO_FP16" - #endif - - // On ARM NEON, it's quicker to directly convert x -> x instead of calling into ggml_lookup_fp16_to_fp32, -``` - -Why this and not something smaller or larger: - -* **The fix carries no comment.** `__POWER9_VECTOR__` and `__riscv_zfhmin` have the same pair bare, so - a comment on the NEON one would itself be the inconsistency. Afterwards all three branches are - identical in shape. -* **The F16C selection moves into its branch**, which is what turns the guard's invariant from "every - branch selects what it defines, except F16C which does it 88 lines further down" into a plain one. - It also retires the comment that had to explain that position. -* **The guard replaces the out-of-chain block at its own position** rather than sitting beside it, and - lands immediately above upstream's own comment promising both macros for NEON -- promise and - enforcement are read together. - -## For us: commit split and conventions - -Three commits in one PR, so either of the last two can be dropped without touching the first. -Maintainers squash-merge, so the split is for review readability only and the **PR title becomes the -commit title**. - -1. `ggml-cpu: select the NEON fp32->fp16 conversion` -2. `ggml-cpu: move the F16C selection into its own branch` -3. `ggml-cpu: refuse a branch that defines a COMPUTE macro without selecting it` - -`CONTRIBUTING.md` asks for `<module> : <title> (#<issue>)` and the maintainer adds the number at -merge. The module prefix this file's own history uses is **`ggml-cpu`**. A PR needs two independent -reviews unless a maintainer adds the "merge ready" label. - ---- - -# PR text - -> ### ggml-cpu: select the NEON fp32->fp16 conversion -> -> The `__ARM_NEON` branch of `simd-mappings.h` defines both `GGML_CPU_COMPUTE_FP16_TO_FP32` and -> `GGML_CPU_COMPUTE_FP32_TO_FP16`, but selects only the first. The second therefore falls through to -> `GGML_COMPUTE_FP32_TO_FP16` at the end of that section -- ggml-base's portable software bit -> manipulation, the one `ggml.c` uses everywhere because ggml-base may not assume CPU features -- -> while `neon_compute_fp32_to_fp16`, a single `fcvt`, stays defined and referenced by nothing. The -> `GGML_CPU_*` pair exists to be the accelerated variant; on ARM it is defined to be the -> unaccelerated one. -> -> `__POWER9_VECTOR__` and `__riscv_zfhmin` select both macros. The comment on that fallback already -> says NEON does too: -> -> > so we define GGML_CPU_FP16_TO_FP32 **and GGML_CPU_FP32_TO_FP16** elsewhere for NEON. This is also -> > true for POWER9. -> -> Same shape of omission #30270 fixed for F16C, in the other direction. **Not a regression** -- the -> NEON selection is absent in all 25 most recent commits touching the file, back to 2025-04-10. -> -> Why it is felt on ARM and nowhere else: `ggml_cpu_fp32_to_fp16` -- the `from_float` of -> `GGML_TYPE_F16` -- has vector paths for F16C, RISC-V and s390x (the last added in #30157) and -> **none for NEON**, so there the whole function is the scalar loop through the macro. -> -> ### It is in the released binaries -> -> Disassembling `ggml_cpu_fp32_to_fp16` and searching for **any** `fcvt` of any form finds **none** in -> `llama-b11576-bin-ubuntu-arm64` (all 8 CPU variants) or `llama-b11576-bin-macos-arm64`, nor in the -> 17 ARM artifacts a downstream project ships across Linux, Android, Windows-on-ARM and macOS -- **26 -> binaries, zero conversion instructions**. In the same object file the *other* direction carries 9 of -> them, which is the asymmetry in one picture. -> -> The x86 side is the comparison: of this project's 14 x86-64 CPU variant modules, the 11 with F16C -> emit 5-6 `vcvtps2ph` each and the three without (`x64`, `sse42`, `sandybridge`) emit none -- exactly -> matching the hardware. Only ARM has the instruction everywhere and uses it nowhere. -> -> ### What it costs -> -> In the shipped `libggml-cpu-armv8.0_1.so` the loop is **25 instructions per element**, with a -> six-operation serial float dependency chain (`fabs`, `fmul`, `fmov`, `fmadd`, `fmov`) and a -> data-dependent branch for the inf/NaN case. With the selection it is a load, one `fcvt` and a store, -> which the compiler then vectorises to `fcvtn v0.4h, v0.4s` -- four elements per instruction. -> -> Measured on five ARM core types across Linux, macOS and FreeBSD: a 16M-element bulk conversion, best of 7, with the -> **opposite direction measured in the same binary and the same run** as a control the change cannot -> touch. -> -> | machine | OS | compiler | before | after | factor | -> |---|---|---|---:|---:|---:| -> | Ampere eMAG (2018) | Linux | gcc 8.5 | 82.0 Melem/s | 1087.2 | 13.3x | -> | Ampere eMAG | Linux | clang 21.1.8 | 341.9 | 3031.6 | 8.9x | -> | Cortex-A55 (in-order, 64K pages) | Linux | gcc 14.2 | 265.1 | 1136.2 | 4.3x | -> | ARM Neoverse-N1 | Linux | gcc 14 | 323.8 | 1310.1 | 4.0x | -> | ARM Neoverse-N1 | Linux | clang 19.1.7 | 976.2 | 3030.3 | 3.1x | -> | ARM Neoverse-N1 | FreeBSD 16 | clang 21.1.8 | 886.1 | 2168.4 | 2.45x | -> | ARM Neoverse-V2 (Graviton 4) | Linux | gcc 11 | 583.0 | 2442.0 | 4.2x | -> | ARM Neoverse-V2 | Linux | clang 15.0.7 | 1595.7 | 6127.1 | 3.8x | -> | Apple M1 | macOS | Apple clang 21 | 2751.7 | 9811.2 | 3.6x | -> -> The factor narrows on wider cores and never vanishes -- **2.45x to 13.3x** -- across every ARM -> server core generation from 2018 to Graviton 4, an in-order little core, Apple Silicon, and three -> operating systems. In all eighteen runs the control direction stays within 3.2 %, against an effect -> of 2.45x and up; on the Cortex-A55 it is identical to the digit. -> -> ### It is bit-exact -> -> Both paths applied to **every one of the 2^32 float32 bit patterns**: **0 differences** for every -> finite value, zero, denormal and infinity. They differ only on NaN inputs (16 744 448), where the -> software encoder emits a canonical quiet NaN and the instruction propagates the truncated payload. -> The identical comparison on x86 with F16C gives the same two numbers, so this brings ARM into -> agreement with the behaviour x86 already ships rather than splitting them. The 16M-element output -> checksum is byte-identical on all five core types and all nine compiler builds, and the exhaustive run -> was repeated on three architectures (Ampere eMAG, Graviton 4, x86-64) with identical counts. -> -> ### It does not make inference faster -> -> `perf` on an F16 `pp512` run attributes **83.9 %** of cycles to `tinyBLAS::gemm`, which converts -> inside its own microkernel, and **0.03 %** to `ggml_cpu_fp32_to_fp16`. End to end the change is -> +0.13 % against a run-to-run spread of 0.09 %, i.e. nothing; quantized models never reach the -> function at all, since their activations go through a different `from_float`. This is a correctness -> and consistency fix, and the 3x-13x belongs to the function, not to a model. -> -> ### Second commit: the F16C selection moves into its branch -> -> #30270's two selecting `#define`s sit 88 lines below the chain, after the lookup-table declarations, -> which is why they need a comment explaining their own position. Moving them into the `__F16C__` -> branch beside the intrinsics they select gives the chain **one shape** -- every branch defines its -> `COMPUTE` pair and selects it in the same place -- and makes the third commit's invariant a plain -> rule rather than one with an exception. It also removes a fragility: as written, a future reordering -> of the file silently returns F16C to the software fallback, which is the defect this PR is about. -> -> **Behaviour-preserving, proven at object level:** a translation unit calling both macros, compiled -> on x86 and disassembled, emits `vcvtps2ph` before and after with `-mavx2 -mf16c -mfma` and with -> `-mavx512f -mf16c`, and the software path before and after with `-msse4.2`. Macro resolution is -> identical in four x86 configurations. -> -> ### Third commit: the guard -> -> Two of the four branches in that chain had this defect at the same time -- F16C until #30270, NEON -> until this PR -- and neither was caught by a build, a warning or a test, because the fallback -> silently supplies a correct software implementation. The only symptom is a hardware instruction that -> is never used. -> -> The `#error` pair checks exactly that state and cannot fire otherwise: POWER9 and riscv_zfhmin -> select both macros, and a platform with no branch at all defines no `COMPUTE` macro to catch. -> -> Its position -- immediately before the fallback rather than directly after the chain -- was chosen -> by mutation, not by taste. A header was built for each way the invariant can be violated and each -> way it can only look violated, and compiled with the guard in both places: before the fallback it -> catches **4 of 4** correctness cases with no false alarms, while directly after the chain it misses -> one, a `COMPUTE` macro defined after the chain and never selected -- the same silent-fallback class -> this PR is about. -> Verified three ways with clang against this file: on unmodified `master` it fires **only** in the -> NEON configuration, with the fix it is silent, and both trees compile unchanged when the guard is -> absent -- so the guard is the only thing that changes the outcome. -> -> ### Build and test -> -> Built in the configuration used for the ARM releases -- `-DGGML_BACKEND_DL=ON -> -DGGML_CPU_ALL_VARIANTS=ON -DGGML_NATIVE=OFF`, clang 21.1.8 -- i.e. all **eight** ARM CPU variant -> modules, each compiling this header with a different feature set (dotprod, fp16, SVE, i8mm, SVE2, -> SME). All eight build, **the guard fires nowhere in the whole build**, and `ggml_cpu_fp32_to_fp16` -> contains `fcvt`, `fcvtn` and `fcvtn2` -- six occurrences -- in every one of them. Reverting the -> patch in the same build tree and rebuilding puts that count back to **0**. -> -> `ctest`: **52 of 54 pass**. The two failures, `test-tokenizers-ggml-vocabs` (#16) and -> `test-jinja-py` (#33), are the same two with the same numbers on unpatched `master` in that same -> tree -- they need test vocabularies and a Python jinja2 the machine does not have. The suite is -> unchanged by the patch. -> -> ### Testing summary -> -> * `clang -E` resolution matrix over five configurations (`x86 +F16C`, `x86 +F16C+AVX512`, -> `x86 -F16C`, `x86` plain, NEON) for `master`, `master`+guard and the final tree. -> * A real x86 translation unit compiled and disassembled, before and after, three flag sets. -> * Full build with all 8 ARM CPU variants plus `ctest`, and the revert round trip in the same tree. -> * Microbenchmark on five ARM core types and nine compiler builds, with an in-run control. -> * `llama-bench` (f16 and Q4_K_M, interleaved, `-r 5`) plus `perf record`. -> * Exhaustive 2^32 equivalence, on ARM and on x86. -> * Disassembly of 26 released ARM binaries. -> -> ### Not claimed -> -> * No inference speed-up, as measured above. -> * Five ARM core types were measured (Ampere eMAG, Cortex-A55, Neoverse-N1, Neoverse-V2, Apple M1) -> on Linux, macOS and FreeBSD with nine compiler builds. Neoverse N2 and Apple M2 and later are -> untested, which is why the factor is given as a range. -> * MSVC on Windows-on-ARM is untested as a compiler, although the `ggml-cpu.dll` built with it shows -> the same absence of `fcvt`. diff --git a/llama/patches/0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch b/llama/patches/0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch new file mode 100644 index 00000000..5a14d76a --- /dev/null +++ b/llama/patches/0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch @@ -0,0 +1,96 @@ +ggml-cpu: select the NEON fp32->fp16 conversion + +The __ARM_NEON branch of ggml/src/ggml-cpu/simd-mappings.h defines both +GGML_CPU_COMPUTE_FP16_TO_FP32 and GGML_CPU_COMPUTE_FP32_TO_FP16 but selects only the first, so +GGML_CPU_FP32_TO_FP16 falls through to GGML_COMPUTE_FP32_TO_FP16 at the end of that section -- +ggml-base's deliberately portable software bit manipulation, the one ggml.c uses everywhere +because ggml-base may not assume CPU features -- while neon_compute_fp32_to_fp16, a one-line +__fp16 conversion, stays defined and referenced by nothing. The whole purpose of the GGML_CPU_* +pair is to be the accelerated variant; on ARM it was defined to be the unaccelerated one. + +__POWER9_VECTOR__ and __riscv_zfhmin select both macros, and the comment on that fallback has +always said NEON does too ("so we define GGML_CPU_FP16_TO_FP32 and GGML_CPU_FP32_TO_FP16 +elsewhere for NEON. This is also true for POWER9."). Not a regression: the NEON selection is +absent in all 25 most recent commits touching the file, back to 2025-04-10. + +Why it is felt on ARM and nowhere else: ggml_cpu_fp32_to_fp16 -- the from_float of GGML_TYPE_F16 +-- has vector paths for F16C, RISC-V and s390x (the last added in llama.cpp #30157) and none for +NEON, so there the whole function is the scalar loop through the macro. + +Two further changes keep the chain regular. The two F16C selections added in llama.cpp #30270 +move from 88 lines below the chain, after the lookup-table declarations, into the __F16C__ branch +beside the intrinsics they select; that is behaviour-preserving, proven at object level (a +translation unit calling both macros emits vcvtps2ph before and after with -mavx2 -mf16c -mfma and +with -mavx512f -mf16c, and the software path with -msse4.2). In their place an #error pair refuses +a branch that defines a COMPUTE macro without selecting it -- the omission that neither a build, +a warning nor a test could see, because the fallback silently supplies a correct implementation. + +WHY THIS PROJECT CARRIES IT. All 17 ARM artifacts shipped here -- cpu-linux-aarch64 (8 variants), +cpu-android-aarch64 (7), cpu-windows-aarch64 and metal-macos-aarch64 -- contain no fcvt of any +form in ggml_cpu_fp32_to_fp16, and neither do upstream's own llama-b11576-bin-ubuntu-arm64 and +-macos-arm64. A plain gcc with no flags defines __ARM_NEON and __ARM_FP16_FORMAT_IEEE on aarch64, +so there is no opt-in and no way to miss it. + +MEASURED. The function runs 2.45x to 13.3x faster with the selection, across five ARM core types +(Ampere eMAG, Cortex-A55, Neoverse-N1, Neoverse-V2, Apple M1), three operating systems and nine +compiler builds, with the opposite direction measured in the same binary and run as a control that +stays within 3.2 % everywhere. Bit-exact: both paths agree on every one of the 2^32 float32 bit +patterns, differing only in NaN payload -- where the same comparison on x86 F16C gives the same +numbers, so this brings ARM into agreement with what x86 already ships. It is NOT an inference +speed-up: perf puts 83.9 % of an F16 pp512 run in tinyBLAS's gemm, which converts inside its own +microkernel, and 0.03 % in this function. The full record, including the mutation measurement that +chose the guard's position, is docs/benchmarks/arm-fp16-conversion.md. + +Built in upstream's own ARM release configuration (GGML_BACKEND_DL, GGML_CPU_ALL_VARIANTS, +GGML_NATIVE=OFF): all eight variant modules compile, the guard fires nowhere, fcvt/fcvtn/fcvtn2 +appear in every one of them, reverting in the same tree puts that back to zero, and ctest gives +52 of 54 with the two failures identical on unpatched master. + +SUBMITTED UPSTREAM as ggml-org/llama.cpp#30505 (open), byte-identical to this diff. + +On a bump, check whether upstream merged it: +`git grep -n "GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16" ggml/src/ggml-cpu/simd-mappings.h` +in the new tag. If the __ARM_NEON branch carries that line, DROP this patch rather than refreshing +it -- the fail-loud applier detects "does not apply", never "upstream already fixed this". + +diff --git a/ggml/src/ggml-cpu/simd-mappings.h b/ggml/src/ggml-cpu/simd-mappings.h +index fb53ecdf4..fad710cb3 100644 +--- a/ggml/src/ggml-cpu/simd-mappings.h ++++ b/ggml/src/ggml-cpu/simd-mappings.h +@@ -42,6 +42,7 @@ extern "C" { + #define GGML_CPU_COMPUTE_FP32_TO_FP16(x) neon_compute_fp32_to_fp16(x) + + #define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) ++ #define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) + + static inline float neon_compute_fp16_to_fp32(ggml_fp16_t h) { + __fp16 tmp; +@@ -63,6 +64,8 @@ extern "C" { + #define GGML_CPU_COMPUTE_FP16_TO_FP32(x) _cvtsh_ss(x) + #define GGML_CPU_COMPUTE_FP32_TO_FP16(x) _cvtss_sh(x, 0) + #endif ++ #define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) ++ #define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) + #elif defined(__POWER9_VECTOR__) + #define GGML_CPU_COMPUTE_FP16_TO_FP32(x) power_compute_fp16_to_fp32(x) + #define GGML_CPU_COMPUTE_FP32_TO_FP16(x) power_compute_fp32_to_fp16(x) +@@ -140,12 +143,13 @@ extern float ggml_table_f32_ue4m3[1 << 8]; + #define GGML_CPU_UE4M3_TO_FP32(x) ggml_ue4m3_to_fp32(x) + #endif + +-// Pre AVX512 CPUs (AVX2 with F16C, such as i5-8th gen or i7-4790) +-// do not produce codegen as checks below don't test for __F16C__ +-// but this check does. +-#if defined(__F16C__) +-#define GGML_CPU_FP16_TO_FP32(x) GGML_CPU_COMPUTE_FP16_TO_FP32(x) +-#define GGML_CPU_FP32_TO_FP16(x) GGML_CPU_COMPUTE_FP32_TO_FP16(x) ++// A branch above that defines a GGML_CPU_COMPUTE_* macro must also select it: left unselected it ++// falls through to the portable fallback below, which is correct but never uses the instruction. ++#if defined(GGML_CPU_COMPUTE_FP16_TO_FP32) && !defined(GGML_CPU_FP16_TO_FP32) ++#error "a branch defines GGML_CPU_COMPUTE_FP16_TO_FP32 without selecting it as GGML_CPU_FP16_TO_FP32" ++#endif ++#if defined(GGML_CPU_COMPUTE_FP32_TO_FP16) && !defined(GGML_CPU_FP32_TO_FP16) ++#error "a branch defines GGML_CPU_COMPUTE_FP32_TO_FP16 without selecting it as GGML_CPU_FP32_TO_FP16" + #endif + + // On ARM NEON, it's quicker to directly convert x -> x instead of calling into ggml_lookup_fp16_to_fp32,