Skip to content

Fix orphaned dev processes and make Ctrl-C immediate - #3

Open
kevinrenskers wants to merge 4 commits into
mainfrom
fix/dev-shutdown-lifecycle
Open

Fix orphaned dev processes and make Ctrl-C immediate#3
kevinrenskers wants to merge 4 commits into
mainfrom
fix/dev-shutdown-lifecycle

Conversation

@kevinrenskers

Copy link
Copy Markdown
Member

Supersedes #2, whose commit is included here unchanged.

What changed

saga dev could leave processes running after Ctrl-C:

  • Killing the CLI mid-recompile orphaned the swift build child, which kept running after the CLI exited.
  • Ctrl-C arriving between terminating the old site process and launching its replacement left the replacement running and still watching the site.

Rather than serializing the recompile and shutdown handlers on a shared queue, this locks the two pairs of operations that actually have to be atomic: setting the shutdown flag while reading the running processes, and checking that flag while installing a replacement. Shutdown then waits on nothing, and a recompile can never install a process shutdown would miss.

Also here:

  • Ctrl-C interrupts an in-flight swift build instead of letting it outlive the CLI.
  • The SIGUSR1 handler is installed after startup completes, so a source change during the initial build can't race the first launch.
  • Termination is bounded and escalates to SIGKILL, so a child that blocks SIGTERM can't hang the CLI with SIGINT already ignored.

Validation

Throwaway Saga site, Ctrl-C sent while a recompile was in flight:

exit time orphaned swift build
main 0.02s yes
#2 alone 2.78s (blocks for the build) no
this branch 0.05s no

Plain Ctrl-C with nothing in flight passes on all three.

The narrow race #2 targets — SIGINT landing between terminating the old process and launching the replacement — is not covered by that test. It is a microsecond window I could not reproduce on any version. The lock makes it unreachable by construction, but that is reasoning rather than a measurement.

deverman and others added 2 commits August 4, 2026 15:32
Instead of serializing the recompile and shutdown handlers on a shared
queue, lock the two pairs of operations that actually have to be atomic:
setting the shutdown flag while reading the running processes, and
checking that flag while installing a replacement. Shutdown then waits on
nothing, and a recompile can never install a process it would miss.

Ctrl-C also interrupts an in-flight `swift build` rather than leaving it
running after we exit.

Install the SIGUSR1 handler after startup completes, so a source change
during the initial build can't race the first launch.

Bound process termination and escalate to SIGKILL, so a child that blocks
SIGTERM can't hang the CLI with SIGINT already ignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@deverman

deverman commented Aug 5, 2026

Copy link
Copy Markdown

@kevinrenskers I've installed in and first try worked will monitor for a day or two and then respond.

@kevinrenskers

Copy link
Copy Markdown
Member Author

@deverman I guess it's safe to assume you haven't found any issues? :)

@deverman

Copy link
Copy Markdown

I just got to play around yesterday and last night and so far no issues so it should be good.

@deverman

deverman commented Aug 12, 2026

Copy link
Copy Markdown

@kevinrenskers sorry I found a stray process and asked codex to verify it and so maybe it is not fixed. I didn't have much time to look but here is the summary the AI gave me sorry I don't have more time to look at it.

I need to correct my earlier UAT comment: I found a reproducible startup-window orphan in the exact PR head caff21eab0e76f243284af60238313219b922e32. I also verified that the installed UAT binary matched the release artifact built from that checkout.

Reproduction

  1. Use a minimal fixture executable that writes .build/saga-config.json, then delays its initial SIGUSR2 for 30 seconds.
  2. Run this PR build with saga dev.
  3. Press Ctrl-C after the fixture child launches but while the CLI is blocked in initialBuild.wait().
  4. The CLI exits immediately, while the fixture child remains alive with PPID 1.

Before Ctrl-C, the CLI was the terminal foreground process group and the site child had its own process group. After Ctrl-C, only the CLI exited; the child was reparented to launchd. This also matches the real-world orphan I found, whose stack was still actively inside Saga.watchAndRebuild() rather than being a zombie.

Root cause

start() launches and records the initial site process, then waits for the initial build. The SIGINT dispatch source is not installed until after that wait, the server startup, and browser opening. Ctrl-C during that startup window therefore uses the default SIGINT action, bypassing all lifecycle cleanup.

There is also a smaller race if the handler is only moved earlier: SIGINT could land after Process.run() but before the returned process is assigned to lifecycle.siteProcess.

Suggested change

Install the SIGINT source before launching any child, extract the existing cleanup into one idempotent method, and launch plus register the initial child while holding the lifecycle lock:

func start() throws {
  signal(SIGINT, SIG_IGN)
  let sigintSrc = DispatchSource.makeSignalSource(
    signal: SIGINT,
    queue: DispatchQueue(label: "Saga.Shutdown")
  )
  sigintSrc.setEventHandler { [weak self] in
    print("\nShutting down...")
    guard let self else { Foundation.exit(0) }
    self.shutdown()
  }
  sigintSrc.resume()

  // Existing SIGUSR2 and SIGUSR1 setup...

  let siteProcess = lifecycle.withLock { state -> Process? in
    guard !state.shuttingDown else { return nil }
    let process = launchSiteProcess(productName: productName, cachePath: cachePath)
    state.siteProcess = process
    return process
  }

  guard let siteProcess else {
    log("Failed to launch site process.")
    throw ExitCode.failure
  }

  // Existing initial-build wait and server setup...

  withExtendedLifetime((sigusr1Source, sigusr2Source, sigintSrc)) {
    dispatchMain()
  }
}

private func shutdown() {
  let processes = lifecycle.withLock { state -> (Process?, Process?)? in
    guard !state.shuttingDown else { return nil }
    state.shuttingDown = true
    return (state.siteProcess, state.buildProcess)
  }
  guard let (site, build) = processes else { return }

  terminate(build)
  terminate(site)
  server?.stop()
  Foundation.exit(0)
}

I tested this exact structure in a disposable clone of this PR:

  • swift build passed
  • git diff --check passed
  • the same delayed-initial-SIGUSR2 fixture printed Shutting down...
  • both the patched CLI and fixture child exited
  • no PPID-1 process remained

Handling SIGHUP and SIGTERM through the same method may be useful follow-up hardening, but neither is needed to reproduce this SIGINT startup bug.

kevinrenskers and others added 2 commits August 12, 2026 10:38
Children get their own process group, so a Ctrl-C the CLI doesn't handle
itself kills only the CLI and leaves them running, reparented to launchd.
The SIGINT source was installed after the product lookup, the initial
build, the site launch and the wait for the first render, so a Ctrl-C
anywhere in that stretch orphaned whatever was already running.

Install it first, before anything is spawned, and route both the product
lookup and the initial build through the coordinator so their processes
are tracked and can be interrupted. Registering the initial site process
happens under the lifecycle lock, closing the gap between spawning it and
recording it.

Shutdown moves into one idempotent method shared by every path.

Reported-by: Brent Deverman <brent@deverman.org>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kevinrenskers

Copy link
Copy Markdown
Member Author

Thank you for the review and the suggestion. I have to admit that I am a bit out of my depth with saga-cli, with its processes and all that, so this help is greatly appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants