Skip to content

Guard the shutdown log wakeup against an unallocated notify array - #13541

Draft
bryancall wants to merge 2 commits into
apache:masterfrom
bryancall:fix/guard-preproc-notify-on-shutdown
Draft

Guard the shutdown log wakeup against an unallocated notify array#13541
bryancall wants to merge 2 commits into
apache:masterfrom
bryancall:fix/guard-preproc-notify-on-shutdown

Conversation

@bryancall

@bryancall bryancall commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Log::preproc_notify is allocated in Log::create_threads(), which runs only once logging is fully enabled. Until then the pointer is null, while Log::preproc_threads has already been set to a nonzero value by Log::init(). AutoStopCont::mainEvent walked the array anyway:

// Wake preproc threads to drain remaining log buffers before exit.
for (int i = 0; i < Log::preproc_threads; i++) {
  Log::preproc_notify[i].signal();
}

The result is a SIGSEGV at address zero.

Two distinct ways to get there, which is why the guard is phrased around the array rather than around a race:

  • Startup window. Log::init() runs at traffic_server.cc:2382; Log::load_config() does not run until :2407. In between sit api_init() and plugin_yaml_init(), so the window spans every plugin's TSPluginInit. SignalContinuation is already scheduled by then, and proxy.config.stop.shutdown_timeout defaults to 0, so a SIGTERM during a slow plugin init lands straight in AutoStopCont.
  • Permanently. Log::load_config() returns without calling init_when_enabled() when config_flags & LOGCAT is set, so for the log-only tools the array is null for the life of the process.

#13472 added exactly this guard to the two call sites in LogObject.cc and described the same failure, but did not cover this third copy in traffic_server.cc, which was introduced separately by #13065.

Nothing is lost by skipping the signal. preproc_notify is assigned in exactly one place and never reset, and the preproc threads are spawned in that same function immediately after the allocation, so a null array means no preproc thread exists and the loop could not have drained anything.

Scope

The fourth use, in PeriodicWakeup::wakeup, is deliberately left alone: it is scheduled on the line immediately after create_threads() returns, so the array is always allocated by then. This PR closes the last exposed site.

Worth noting as possible follow-up rather than part of this change: allocating preproc_notify in Log::init() once preproc_threads is final would establish the invariant and let all three guards go away.

Impact

Startup window only. start_HttpProxyServer() runs well after Log::load_config(), so a process that reaches this path never served traffic. The cost is a crash and a core file where a clean exit belonged, not a traffic impact.

Testing

Built with the dev-asan preset (-fsanitize=address,undefined) on Fedora 44 / gcc 16, -Werror clean.

  • unit (ctest): 174/174 passed
  • in-binary regression (traffic_server -R 1): 67 passed, REGRESSION_TEST DONE: PASSED
  • autest: 476 passed / 65 failed / 40 skipped, over 561 tests (remap_acl and remap_acl_yaml excluded for runtime)

The 65 autest failures are pre-existing environment failures on this box, not regressions. I built unmodified master at the same base commit (8aebe2c706) and ran the identical 561-test selection: it produces the same 476/65/40 and the same 61 distinct failing tests, so the failing set matches the baseline exactly. They are concentrated in TLS, QUIC/HTTP3, and config-reload tests.

Log::init() sets Log::preproc_threads to 1 immediately, but
Log::preproc_notify is not allocated until Log::create_threads().
A shutdown signal arriving between those two points reaches
AutoStopCont::mainEvent, which walks preproc_threads entries of a
null array and crashes with a SIGSEGV at address zero.

A previous change added this same guard to the two call sites in
LogObject.cc but did not cover the copy in traffic_server.cc, which
was introduced separately. Production cores show the unguarded site
still firing on builds that already carry the LogObject.cc guards.
The array is also null for the whole life of the log-only tools, where
Log::load_config returns without ever creating the log threads. Saying
only that a shutdown can arrive mid startup invites a later reader to
prove that race unreachable and drop a guard that is still load bearing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant