Conversation
Seven of the fourteen iai_algos benchmarks report on the order of a hundred instructions for searches that execute hundreds of thousands of basic blocks. Callgrind only counts while collection is toggled on around the benchmark function, and a search that is inlined into the harness and folded away falls outside that window. Because it depends on inlining, the affected set moves with unrelated edits, so the base-versus-PR comparison in the workflow can report an enormous change for a PR that did nothing of the sort. Run each search in an #[inline(never)] helper and return its result, so the optimiser cannot see through the call and cannot discard the result. All fourteen now measure, and they keep measuring across changes to the algorithms themselves. The workloads are unchanged. Fixes evenfurther#833
This was referenced Sep 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #833.
Seven of the fourteen
iai_algosbenchmarks do not measure the work they run. Callgrind only counts while collection is toggled on around the benchmark function, and a search that gets inlined into the harness and folded away falls outside that window.What changes
Each search moves into an
#[inline(never)]helper, and the benchmark returns its result. The optimiser cannot see through the call, and cannot discard the value. The workloads themselves are unchanged.Before and after
Instructions collected, against basic blocks the process actually executed:
no_path_astarno_path_fringeno_path_bfs_bidirectionalcorner_to_corner_astarcorner_to_corner_fringecorner_to_corner_dfscorner_to_corner_bfsThe seven that already worked are essentially unmoved, for instance
corner_to_corner_dijkstra3,077,484 -> 3,063,305 andcorner_to_corner_iddfs3,712,137 -> 3,790,713.corner_to_corner_astarandcorner_to_corner_fringeare genuinely small, at around 150k and 120k instructions, because the Manhattan heuristic is exact on an open grid so they expand very few nodes. That is now visible rather than hidden behind a broken count.That it stays fixed
The point is not only that the numbers are right today, but that they stay right when the algorithms change, since that is what made the old ones flip. I applied these benchmarks on top of a branch that edits the inner loop of every cost-based search (#834) and re-ran: 0 of 14 stopped measuring.
Why this matters beyond the numbers
.github/workflows/iai-callgrind.ymldiffs these counts between the base branch and the PR branch and posts the result. Because which benchmarks break depends on inlining, that comparison can report an enormous change for a PR that did nothing of the sort.This is not hypothetical. Measured against the broken benchmarks, the overflow check in #834 looked like an 18.6% instruction regression on
corner_to_corner_dijkstra. Measured against these, it is 0.7%, with five of the six benchmarks that use no costs at exactly 0.0%. The 18.6% was the harness, not the change. I have corrected the figures on that PR.Rejected alternatives, in case they come up:
#[inline(never)]on the benchmark functions themselves does not compile, because thelibrary_benchmarkmacro accepts onlybenchandbenchesattributes. Addingblack_boxaround the inputs alone does not help either; the result has to be consumed as well.