Guard the shutdown log wakeup against an unallocated notify array - #13541
Draft
bryancall wants to merge 2 commits into
Draft
Guard the shutdown log wakeup against an unallocated notify array#13541bryancall wants to merge 2 commits into
bryancall wants to merge 2 commits into
Conversation
Log::init() sets Log::preproc_threads to 1 immediately, but Log::preproc_notify is not allocated until Log::create_threads(). A shutdown signal arriving between those two points reaches AutoStopCont::mainEvent, which walks preproc_threads entries of a null array and crashes with a SIGSEGV at address zero. A previous change added this same guard to the two call sites in LogObject.cc but did not cover the copy in traffic_server.cc, which was introduced separately. Production cores show the unguarded site still firing on builds that already carry the LogObject.cc guards.
The array is also null for the whole life of the log-only tools, where Log::load_config returns without ever creating the log threads. Saying only that a shutdown can arrive mid startup invites a later reader to prove that race unreachable and drop a guard that is still load bearing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Log::preproc_notifyis allocated inLog::create_threads(), which runs only once logging is fully enabled. Until then the pointer is null, whileLog::preproc_threadshas already been set to a nonzero value byLog::init().AutoStopCont::mainEventwalked the array anyway:The result is a SIGSEGV at address zero.
Two distinct ways to get there, which is why the guard is phrased around the array rather than around a race:
Log::init()runs attraffic_server.cc:2382;Log::load_config()does not run until:2407. In between sitapi_init()andplugin_yaml_init(), so the window spans every plugin'sTSPluginInit.SignalContinuationis already scheduled by then, andproxy.config.stop.shutdown_timeoutdefaults to 0, so a SIGTERM during a slow plugin init lands straight inAutoStopCont.Log::load_config()returns without callinginit_when_enabled()whenconfig_flags & LOGCATis set, so for the log-only tools the array is null for the life of the process.#13472 added exactly this guard to the two call sites in
LogObject.ccand described the same failure, but did not cover this third copy intraffic_server.cc, which was introduced separately by #13065.Nothing is lost by skipping the signal.
preproc_notifyis assigned in exactly one place and never reset, and the preproc threads are spawned in that same function immediately after the allocation, so a null array means no preproc thread exists and the loop could not have drained anything.Scope
The fourth use, in
PeriodicWakeup::wakeup, is deliberately left alone: it is scheduled on the line immediately aftercreate_threads()returns, so the array is always allocated by then. This PR closes the last exposed site.Worth noting as possible follow-up rather than part of this change: allocating
preproc_notifyinLog::init()oncepreproc_threadsis final would establish the invariant and let all three guards go away.Impact
Startup window only.
start_HttpProxyServer()runs well afterLog::load_config(), so a process that reaches this path never served traffic. The cost is a crash and a core file where a clean exit belonged, not a traffic impact.Testing
Built with the
dev-asanpreset (-fsanitize=address,undefined) on Fedora 44 / gcc 16,-Werrorclean.traffic_server -R 1): 67 passed,REGRESSION_TEST DONE: PASSEDremap_aclandremap_acl_yamlexcluded for runtime)The 65 autest failures are pre-existing environment failures on this box, not regressions. I built unmodified master at the same base commit (
8aebe2c706) and ran the identical 561-test selection: it produces the same 476/65/40 and the same 61 distinct failing tests, so the failing set matches the baseline exactly. They are concentrated in TLS, QUIC/HTTP3, and config-reload tests.