Repository navigation
Carry the ARM fp32-to-fp16 selection as patches/0019, filed upstream as llama.cpp#30505 - #535
Merged
Merged
Conversation
…e software, and inference does not care The NEON branch of simd-mappings.h defines both GGML_CPU_COMPUTE_* macros and selects only one, so GGML_CPU_FP32_TO_FP16 falls through to ggml-base's portable software bit manipulation while neon_compute_fp32_to_fp16 -- a single fcvt -- stays defined and referenced by nothing. On ARM the accelerated macro is literally defined to be the unaccelerated one. Found by the guard written for the #30270 isolation, which fired on upstream's unmodified header. Six independent measurements, five of them static and therefore immune to a noisy machine: 26 of 26 shipped ARM binaries contain no floating-point conversion instruction in that function at all -- our linux, android, windows and macos artifacts plus upstream's own bin-ubuntu-arm64 and bin-macos-arm64 releases -- with the opposite direction in the same object file carrying 9, 6, 5 or 2 of them as an in-file control; the x86 artifacts are the cross-architecture control, where the eleven F16C variants emit 5-6 vcvtps2ph each and the three without F16C emit none, correctly; the shipped loop is 25 instructions per element with a six-operation serial float chain and a data-dependent branch, against fcvtn's four elements per instruction; the function runs 8.9x (clang 21) to 13.3x (gcc 8.5) faster with the one #define, with the opposite direction measured in the same binary and run moving at most 0.3 percent; and both paths agree on every one of the 2^32 float inputs, differing only in NaN payload -- where the identical comparison on x86 F16C gives the same two numbers, so the fix makes ARM agree with what x86 already ships. The sixth measurement is the one that changes the framing: inference does not move. A prediction of a large gain on F16 prompt processing was written down before the run and was wrong. perf attributes 83.9 percent of an F16 pp512 run to tinyBLAS's gemm, which converts inside its own microkernel, and 0.03 percent to the broken function, so the end-to-end delta is +0.13 percent against a run-to-run spread of 0.09 percent. Quantized models never reach it at all. That is the same mistake #30270 made in the other direction, and avoiding it is the reason for measuring. Methodology, because this run produced two lessons the hard way. The compile farm machine carries a permanent eight-core foreign load, so -t 32 oversubscribes it and two runs of the same binary spread 7.5 percent; those rows are discarded rather than reported, while -t 16 reproduced to 0.05 percent and is. And a pattern of fcvt alone is wrong: fcvtzs and fcvtns are float-to-integer, the quantisation step, which would have reported 8 phantom fp16 conversions in quantize_row_q8_0 -- so the headline number is a bare search for any fcvt of any form, which depends on no pattern. TODO.md's fifth upstream item is rewritten from a small follow-up into this, with the three separable commits that hold the fix on the fork (not pushed, no PR) and the open decision of PR versus comment left to take together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
…imed The discarded high-thread rows were described with a 7.5 percent same-binary spread, which came from the first run of the measurement -- the one whose CSV columns were mis-parsed. Re-derived from the archived raw llama-bench tables, the clean run is worse and more instructive: up to 10.5 percent between two runs of the same binary, with llama-bench's own error bar reaching plus or minus 9.8 percent inside a single run. Taken at face value those rows would report the fix making token generation 12 to 18 percent slower, complete with error bars. That is the clearest argument for discarding them, so it is now stated rather than merely asserted that they are noisy -- together with the check that catches it, which is to interleave the variants and compare two runs of the same binary before comparing variants at all. The four reported t=16 rows were re-derived from the same archive and match the published table exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
…uild, object-level x86 proof The change grew from the minimal form into the one that leaves the file in better shape, on the reasoning that the guard then protects a regular structure rather than one with an exception: every branch of the chain defines its COMPUTE pair and selects it in the same place. #30270's two selecting defines move from 88 lines below the chain into the __F16C__ branch, which also retires the comment that had to explain their position. Net +10/-6 in one file, and the guard now replaces the out-of-chain block at its own position instead of sitting beside it. That move is the one part a reviewer could reasonably push back on, so it is proven at object level rather than by reading: a translation unit calling both macros, compiled on x86 and disassembled, emits vcvtps2ph before and after with -mavx2 -mf16c -mfma and with -mavx512f -mf16c, and the software path before and after with -msse4.2. Macro resolution is identical across four x86 configurations, AVX512 included -- the case that could have interacted with the old block sitting after the lookup-table declarations. The ARM side gained two core generations. The compile farm turned out to already have them: probing every host in its aarch64 list instead of the first four found three 64-core Neoverse-N1 machines and an Apple M1 running macOS that our key opens. So the factor is now a measured range over three generations and five compiler builds -- 13.3x and 8.9x on a 2018 eMAG, 4.0x and 3.1x on Neoverse-N1, 3.6x on the M1 -- narrowing on newer cores and never vanishing, with the in-run control within 2.5 percent everywhere and the 16M-element checksum byte-identical on all of them. Apple Silicon is therefore measured rather than inferred, and the limit that said one core generation is struck through. The final patch was built in upstream's own ARM release configuration (GGML_BACKEND_DL, GGML_CPU_ALL_VARIANTS, GGML_NATIVE=OFF): all eight variant modules compile, the guard fires nowhere in the whole build, fcvt/fcvtn/fcvtn2 appear six times in each, reverting in the same tree puts the count back to zero, and ctest gives 52 of 54 with the same two environment failures as unpatched master. It also applies cleanly to a newer master (296538a) that landed during preparation. docs/upstream-submission-arm-fp16-selection.md carries the ready-to-paste PR text, the commit split, the conventions read out of CONTRIBUTING, the archaeology showing this is not a regression, and three mistakes made while preparing it -- a too-loose fcvt pattern, a 32-thread run on a contended machine that would have reported the fix as 12-18 percent slower, and a build script that deleted its own input and reported BUILD OK for unpatched source. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
The ready-to-paste part now carries only what argues the case: what the defect is, why it is felt on ARM and nowhere else, that it is in the released binaries, what it costs, that it is bit-exact, that inference does not change, why each of the three parts of the patch is the right shape, and what the build and tests show. Every process detail is gone from it -- no account of how the work went, no mistakes made along the way, no self-commentary. Two short sections above the line are marked For us and stay internal: the diff with the design reasoning, and the commit split with the conventions read out of CONTRIBUTING. The historical facts moved the other way, into the PR text, because they argue the case rather than describe the work: that it is not a regression (absent in all 25 commits touching the file, back to 2025-04-10), and that the function has vector paths for F16C, RISC-V and s390x but none for NEON, which is why a missing macro selection is felt there and nowhere else. The rationale for the proposed shape is also sharpened rather than merely stated: the fix carries no comment because the sibling branches have the pair bare; the F16C move is what turns the guard's invariant from one with an exception into a plain rule, and removes the fragility that a reordering of the file silently returns F16C to the software fallback; the guard replaces the out-of-chain block at its own position and lands directly above the comment that promises both macros. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
…nerations measured AWS Graviton 4 is Neoverse-V2, which was the last ARM server core generation the measurement did not cover. A c8g.large booted, fetched the probe over a presigned URL, measured and terminated itself -- no SSH, no key pair, no inbound access, and the self-termination was confirmed by the API reporting the instance already terminated before the explicit call. Neoverse-V2 gives 4.2x with gcc 11 (583.0 to 2442.0 Melem/s) and 3.8x with clang 15.0.7 (1595.7 to 6127.1), with the in-run control direction moving 3.2 and 1.5 percent. That completes the picture across every ARM server core generation in service plus Apple Silicon: 13.3x and 8.9x on a 2018 eMAG, 4.0x and 3.1x on Neoverse-N1, 4.2x and 3.8x on Neoverse-V2, 3.6x on an Apple M1 -- seven compiler builds, a range of 3.1x to 13.3x that narrows on newer cores and never vanishes. The exhaustive 2^32 equivalence was repeated there and returns the identical counts (0 non-NaN differences, 16744448 NaN payloads) in 7.4 seconds against 44 on the eMAG, so bit-exactness now rests on three architectures. The 16M-element output checksum is byte-identical on all four ARM cores and all seven compiler builds. The object code shows the vectorised form, fcvtn v0.4h v0.4s four times with a scalar fcvt tail. The measurement page, the submission text and the prepared commit message all carry the four-generation table; the limit now reads Neoverse N2 and Apple M2 and later rather than Neoverse V2. The probe package was deleted from S3, the presigned URL discarded, and all regions swept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
The guard sits immediately before the portable fallback, not directly after the branch chain. That was settled by measurement rather than by taste, after the adjacent-to-the-cause reading looked better on inspection: a header was built for every way the chain invariant can be violated and for ways it can only look violated, and each was compiled with the guard in both places. Before the fallback it catches 4 of 4 correctness cases with no false alarms; directly after the chain it misses one -- a COMPUTE macro defined after the chain and never selected, which is the same silent-software-fallback class this work is about. What the earlier position catches instead is a style case, and that one is removed by moving the F16C selection into its branch, so the guard does not have to carry it. Three bugs in the first version of that measurement are recorded with it, because each made a guard look blind when the test was: the expectation was set per mutant instead of per configuration, so cases were scored in configurations where no branch is taken at all; the injected branches sat behind an earlier #elif and were unreachable; and the error filter counted the summary line 1 error generated as a compile failure. Four machines joined the throughput table, three of them free: a Cortex-A55 with 64K pages, where the in-run control is identical to the digit; Neoverse-N1 under FreeBSD 16, which gives the lowest factor at 2.45x; and an Apple M1 under macOS. AWS Graviton 4 supplied Neoverse-V2. The range is now 2.45x to 13.3x over five core types, three operating systems and nine compiler builds, with the output checksum byte-identical in every one of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
…0505 The fix is filed upstream as ggml-org/llama.cpp#30505 (one commit, +10/-6 in ggml/src/ggml-cpu/simd-mappings.h, open), and the PR diff is byte-identical to the patch carried here -- compared content line by content line, 16 against 16. Until it merges, this project needs the fix in its own ARM artifacts, so it becomes patches/0019 with the usual header: what the defect is, why only ARM feels it, that it is not a regression, what it costs, that it is bit-exact and not an inference speed-up, and the drop-check for the next llama.cpp bump. Verified where it matters: the patch applies cleanly at the pinned b11574, and simd-mappings.h is byte-identical between that tag and current master, so the same diff serves both. check-patches.py reads 13 patches and 102 hunks with no problems, which is the audit that git apply --check cannot do. The prepared submission document is deleted -- its job ended when the PR was opened, and keeping a second copy of a PR description in the repository would only drift from the real one. What stays is the patch, the CLAUDE.md row describing it, and docs/benchmarks/arm-fp16-conversion.md as the measurement record; both now link to the PR, and the measurement page also links to the patch that carries it. TODO.md's fifth upstream item is struck through as filed, leaving only the standing drop-check behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G
bernardladenthin
had a problem deploying
to
maven-central
October 14, 2026 09:15 — with 
GitHub Actions
Failure
GitHub Actions
Failure
bernardladenthin
had a problem deploying
to
maven-central
October 14, 2026 09:15 — with
GitHub Actions
Failure
GitHub Actions
Failure
bernardladenthin
had a problem deploying
to
startgate
October 14, 2026 09:15 — with
GitHub Actions
Error
GitHub Actions
Error
|
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every ARM build of llama.cpp — ours and upstream's — converts fp32 to fp16 in software, because
the
__ARM_NEONbranch ofsimd-mappings.hdefines bothGGML_CPU_COMPUTE_*macros and selectsonly one.
GGML_CPU_FP32_TO_FP16therefore falls through to ggml-base's deliberately portable bitmanipulation, while
neon_compute_fp32_to_fp16— a singlefcvt— sits defined and referenced bynothing. The
GGML_CPU_*pair exists to be the accelerated variant; on ARM it was defined to bethe unaccelerated one.
Filed upstream as ggml-org/llama.cpp#30505
(open, one commit,
+10/-6). This PR carries it downstream until that merges.The evidence, in one table
fcvtof any form inggml_cpu_fp32_to_fp16— our 17 ARM libraries across Linux, Android, Windows-on-ARM and macOS, plus upstream's ownbin-ubuntu-arm64andbin-macos-arm64vcvtps2pheach, the three without F16C emit none — matching the hardware exactlyfcvtndoing four elements per instructionfcvtin every one, revert round trip → 0,ctest52/54 identical to unpatched masterAnd it is not an inference speed-up, which the patch header and the PR both say plainly:
perfputs 83.9 % of an F16
pp512run in tinyBLAS's gemm, which converts inside its own microkernel,and 0.03 % in this function. Quantized models never reach it at all. The 3×–13× belongs to the
function, not to a model.
Why only ARM feels it
ggml_cpu_fp32_to_fp16is thefrom_floatofGGML_TYPE_F16, and it has vector paths for F16C,RISC-V and s390x — the last added three days before this work in upstream #30157 — and none for
NEON. So on ARM the whole function is the scalar loop through the macro, and on every other
architecture it is vectorised before the macro ever matters.
It is not a regression: the NEON selection is absent in all 25 most recent commits touching the
file, back to 2025-04-10. The second half of that branch was never written.
What is in this PR
llama/patches/0019-ggml-cpu-select-the-neon-fp32-to-fp16-conversion.patch— the carry, withthe usual header and the drop-check for the next bump. Verified to apply cleanly at the pinned
b11574;
simd-mappings.his byte-identical between that tag and current master, so one diffserves both. The patch body is byte-identical to the upstream PR, compared content line by
content line.
docs/benchmarks/arm-fp16-conversion.md— the measurement record, including the mutationmeasurement that chose the guard's position (4 of 4 correctness cases caught before the fallback,
against 3 of 4 directly after the chain) and the three ways the first version of that measurement
was invalid.
CLAUDE.md— the patches-table row, naming the drop-check.TODO.md— the fifth upstream item struck through as filed; only the standing drop-checkremains.
Drop-check on every llama.cpp bump
If the
__ARM_NEONbranch carries that line the upstream PR merged, and the patch is droppedrather than refreshed — the fail-loud applier detects "does not apply", never "upstream already
fixed this".
Verification
check-patches.py13 patches / 102 hunks / 0 problems ·check-natives.py0 disagreements ·122 buildcheck tests green · no dead links after removing the prepared submission document, whose
job ended when the PR was opened.
🤖 Generated with Claude Code
https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G