Skip to content

fix(worker): stop leaking infernet daemons (326 procs / 22 GB on dev2) - #43

Merged
ralyodio merged 1 commit into
masterfrom
fix/infernet-daemon-leak
Oct 14, 2026
Merged

ralyodio merged 1 commit into
masterfrom
fix/infernet-daemon-leak

Conversation

@ralyodio

Copy link
Copy Markdown
Contributor

dev2's c0mpute-worker.service had 326 infernet start --foreground daemons (~22 GB), which pushed the box under earlyoom's threshold and got unrelated processes killed.

Cause: the 5-minute plugin refresh ran infernet update every tick. infernet update has no version check — it re-runs the installer and infernet setup, which restarts the daemon over IPC. The old daemon sometimes releases its socket/pidfile but never exits (its offline heartbeat has no timeout), so another daemon accumulated every few ticks.

Fix

  • Only run infernet update when npm's latest @infernetprotocol/cli differs from the installed version (unknown = skip, so offline nodes stop churning).
  • reap_stale_infernet_daemons() at worker start and on every tick: stops any infernet … start --foreground of the same uid that isn't the pid in daemon.pid (SIGTERM, SIGKILL after 5 s); never acts without a live daemon of record.
  • Tests: cmdline matching + a real-process reaper test (only the unrecorded daemon dies). cargo test -p c0mpute-cli green.

A matching shutdown-deadline fix goes to infernet itself.

🤖 Generated with Claude Code

The 5-minute plugin refresh ran `infernet update` unconditionally. That
command has no version check: it reinstalls and runs `infernet setup`,
which restarts the daemon over IPC. A restart sometimes leaves the old
daemon alive (it drops its socket and pidfile, then hangs on its
offline heartbeat), so every few ticks another one piled up.

- refresh_plugins: run `infernet update` only when npm's latest
  @infernetprotocol/cli differs from the installed version; unknown on
  either side (offline) means skip.
- reap_stale_infernet_daemons (Linux): at worker start and every tick,
  SIGTERM (then SIGKILL after 5 s) any `infernet ... start --foreground`
  owned by this uid that is not the pid in daemon.pid; does nothing
  without a live daemon of record.
- Tests: cmdline matching, and a real-process test that only the
  unrecorded daemon dies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

ThreatCrush Security Scan

8 finding(s)

MEDIUM: 8

Severity Rule Location
MEDIUM js-unescaped-html-sink apps/web/src/app/blog/[slug]/page.tsx:53
MEDIUM js-unescaped-html-sink apps/web/src/app/layout.tsx:95
MEDIUM sh-remote-script-execution scripts/dev-setup.sh:25
MEDIUM sh-eval-expansion scripts/dev-setup.sh:35
MEDIUM sh-remote-script-execution scripts/install.sh:159
MEDIUM sh-remote-script-execution scripts/install.sh:184
MEDIUM sh-remote-script-execution scripts/install.sh:277
MEDIUM sh-remote-script-execution scripts/install.sh:294

Snippets are redacted; ThreatCrush never prints matched credential material.

@ralyodio
ralyodio merged commit 7322d86 into master Oct 14, 2026
12 checks passed
@ralyodio
ralyodio deleted the fix/infernet-daemon-leak branch October 14, 2026 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant