Skip to content

bench: make the iai benchmarks measure the work they run - #835

Open
tachsin wants to merge 1 commit into
evenfurther:mainfrom
tachsin:fix/iai-benchmarks
Open

tachsin wants to merge 1 commit into
evenfurther:mainfrom
tachsin:fix/iai-benchmarks

Conversation

@tachsin

@tachsin tachsin commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Fixes #833.

Seven of the fourteen iai_algos benchmarks do not measure the work they run. Callgrind only counts while collection is toggled on around the benchmark function, and a search that gets inlined into the harness and folded away falls outside that window.

What changes

Each search moves into an #[inline(never)] helper, and the benchmark returns its result. The optimiser cannot see through the call, and cannot discard the value. The workloads themselves are unchanged.

Before and after

Instructions collected, against basic blocks the process actually executed:

benchmark before after basic blocks executed
no_path_astar 14 3,521,771 665,378
no_path_fringe 14 2,932,278 631,293
no_path_bfs_bidirectional 16 2,662,440 574,024
corner_to_corner_astar 103 152,879 91,105
corner_to_corner_fringe 103 119,670 89,850
corner_to_corner_dfs 181 2,677,555 698,052
corner_to_corner_bfs 248 2,636,515 603,804

The seven that already worked are essentially unmoved, for instance corner_to_corner_dijkstra 3,077,484 -> 3,063,305 and corner_to_corner_iddfs 3,712,137 -> 3,790,713.

corner_to_corner_astar and corner_to_corner_fringe are genuinely small, at around 150k and 120k instructions, because the Manhattan heuristic is exact on an open grid so they expand very few nodes. That is now visible rather than hidden behind a broken count.

That it stays fixed

The point is not only that the numbers are right today, but that they stay right when the algorithms change, since that is what made the old ones flip. I applied these benchmarks on top of a branch that edits the inner loop of every cost-based search (#834) and re-ran: 0 of 14 stopped measuring.

Why this matters beyond the numbers

.github/workflows/iai-callgrind.yml diffs these counts between the base branch and the PR branch and posts the result. Because which benchmarks break depends on inlining, that comparison can report an enormous change for a PR that did nothing of the sort.

This is not hypothetical. Measured against the broken benchmarks, the overflow check in #834 looked like an 18.6% instruction regression on corner_to_corner_dijkstra. Measured against these, it is 0.7%, with five of the six benchmarks that use no costs at exactly 0.0%. The 18.6% was the harness, not the change. I have corrected the figures on that PR.

Rejected alternatives, in case they come up: #[inline(never)] on the benchmark functions themselves does not compile, because the library_benchmark macro accepts only bench and benches attributes. Adding black_box around the inputs alone does not help either; the result has to be consumed as well.

Seven of the fourteen iai_algos benchmarks report on the order of a
hundred instructions for searches that execute hundreds of thousands of
basic blocks. Callgrind only counts while collection is toggled on around
the benchmark function, and a search that is inlined into the harness and
folded away falls outside that window.

Because it depends on inlining, the affected set moves with unrelated
edits, so the base-versus-PR comparison in the workflow can report an
enormous change for a PR that did nothing of the sort.

Run each search in an #[inline(never)] helper and return its result, so
the optimiser cannot see through the call and cannot discard the result.
All fourteen now measure, and they keep measuring across changes to the
algorithms themselves.

The workloads are unchanged.

Fixes evenfurther#833
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Most iai_algos benchmarks do not measure the work they run

1 participant