Skip to content

feat(callgrind): support Go binaries out of the box - #57

Draft
not-matthias wants to merge 2 commits into
masterfrom
cod-4081-support-go-binaries-in-valgrind-codspeed
Draft

not-matthias wants to merge 2 commits into
masterfrom
cod-4081-support-go-binaries-in-valgrind-codspeed

Conversation

@not-matthias

@not-matthias not-matthias commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Make Callgrind work on Go binaries out of the box: no GODEBUG=asyncpreemptoff, no GOMAXPROCS, no cgo.

Before this, Go programs had two problems under Callgrind:

  1. Crash on the first preemption signal. Go runs every signal handler on a per-thread sigaltstack. That stack is mapped above the heap-allocated goroutine stacks, so the SP-based unwind popped through the handler's call_stack_bottom. Callgrind read that as a longjmp out of the handler. The handler's real rt_sigreturn then aborted with threads.c: Assertion 'sigNum == CLG_(current_state).sig' failed. Go raises SIGURG for async preemption and GC, so this hit every non-trivial program.
  2. Unusable call graphs. Goroutines share OS threads and switch stacks without call/return pairs (runtime.gogo, runtime.mcall, runtime.systemstack, runtime.morestack, vDSO and cgo calls on g0). The single SP-based shadow stack mixed them. Examples: runtime.newobject "called" runtime.main, runtime.mcall "called" main.work.func1, and there were fake '2 recursion clones everywhere.

Changes

  • fix(callgrind) (all architectures): stop unwinding at the bottom of a signal handler that runs on the sigaltstack while SP is still inside that altstack.
  • feat(callgrind) (amd64 only): new callgrind/gostack.c keeps one shadow call stack per goroutine, keyed by the g register (R14). Switch functions are found by name, so this only runs once a Go runtime is seen.
    • gogo and mcall swap shadow stacks. A goroutine resumed on another thread takes its stack from there.
    • Frames run on g0 for a goroutine (systemstack, morestack, vDSO, asmcgocall) stay under the caller. They are tagged so SP comparisons never pop frames of the other stack.
    • Time a goroutine is suspended is removed from its frames' inclusive cost. Recursion counts follow the running goroutine. Shadow frames move with the goroutine stack when the runtime grows or shrinks it.
    • The jump back to a function's entry after morestack no longer counts as the function calling itself.

setup_bbcc gets one UNLIKELY(CLG_(go_runtime)) branch. Non-Go programs and non-amd64 builds take none of the new paths.

Verification

Tested on amd64 only. Fixture: 4 goroutines doing fib(27) plus 20k 1 KiB allocations, which also sends itself SIGURG. It is built with plain go build (static, Go 1.26).

Build Result
3.26.0.codspeed9 (release) 3/3 runs abort with the post_signal assertion
This branch runs complete, worker stacks are goexit;main.work.gowrap1;main.work.func1;main.fib;main.fib'2…
  • callgrind/tests regtests: the same failure set as master (bug497723, which already fails).
  • The existing C, Rust and Python call-graph fixtures give the same results as master.

Known limitations

  • About 30-45 call edges inside Go runtime GC code (gcMarkDone, mallocgc, semrelease1, …) still show a '2 recursion clone. The count changes with GC timing. User code stacks are not affected.
  • Goroutine tracking is amd64 only. On arm64 the altstack crash fix applies, but goroutine stacks are still mixed.
  • Stack switches are recognized by runtime function names. They are stable from Go 1.17 (.abi0 suffixes) and the older names are included, but a future runtime that renames these functions would need an update.
  • cgo: asmcgocall is handled. Callbacks from C into Go (cgocallback) are not tracked specially.

Closes COD-4081

Callgrind pops shadow call stack frames whose saved SP is below the
current SP. When a signal handler runs on a sigaltstack mapped above the
interrupted stack, the first BB of the handler already has a higher SP than
every interrupted frame, so the unwind popped through the handler's
call_stack_bottom. That looks like a longjmp out of the handler, so
post_signal ran early, and the handler's real rt_sigreturn then failed
`Assertion 'sigNum == CLG_(current_state).sig'`.

Go runs every signal handler on a per-thread sigaltstack above its
heap-allocated goroutine stacks, so any Go program aborted under Callgrind
on its first preemption signal (SIGURG) or GC.

Stop unwinding at the bottom of a handler that runs on the sigaltstack
while SP is still inside that altstack. A longjmp that leaves the altstack
still unwinds as before.

Refs COD-4081
@not-matthias
not-matthias force-pushed the cod-4081-support-go-binaries-in-valgrind-codspeed branch from 7ceecde to a77fbd1 Compare October 30, 2026 19:45
Go runs many goroutines on one OS thread, each on its own heap-allocated
stack, and moves between them without call/return pairs: runtime.gogo
jumps into another goroutine, runtime.mcall leaves one for the g0 stack,
and runtime.systemstack, runtime.morestack and the vDSO/cgo calls run code
on g0 and come back. Callgrind's SP-based shadow stack mixed these stacks,
so frames of one goroutine ended up under another, scheduler code under
user code, and call graphs of Go programs were unusable.

On amd64, track Go stacks explicitly once a Go runtime switch function is
seen:

- Each goroutine gets its own shadow call stack, keyed by its g pointer
  (R14). gogo and mcall swap stacks; a goroutine resumed on another thread
  takes its stack from there.
- Frames that run on g0 for a goroutine stay under its caller and are
  tagged, so the SP comparison never pops frames of the other stack.
- Time a goroutine spends suspended is removed from its frames' inclusive
  cost, active-function counts follow the running goroutine, and shadow
  frames follow the runtime when it moves a goroutine's stack.
- The jump back to a function's entry after morestack is not a call.
- Underflow frames on goroutine stacks start at the current cost instead
  of the last dump.

Programs without a Go runtime, and other architectures, take none of these
paths. Some call edges inside the Go GC still carry a timing-dependent
recursion level.

Refs COD-4081
@not-matthias
not-matthias force-pushed the cod-4081-support-go-binaries-in-valgrind-codspeed branch from a77fbd1 to 756bb49 Compare October 30, 2026 19:50
@codspeed

codspeed Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 84 untouched benchmarks
⏩ 60 skipped benchmarks1


Comparing cod-4081-support-go-binaries-in-valgrind-codspeed (756bb49) with master (f3f2fda)

Open in CodSpeed

Footnotes

  1. 60 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant