Repository navigation
feat(worker): launcher worker-mode groundwork (config, status, credential renewal, process ownership) - #12705
Draft
mudler-agent wants to merge 16 commits into
Draft
mudler-agent wants to merge 16 commits into
mudler-agent wants to merge 16 commits into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
gosec found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Run native Linux launcher, worker registry, worker and nodes suites in Ubuntu with strict Ginkgo evidence checks. Preserve failures through logging, enforce scratch space preflight and test apt-compatible temporary directory permissions. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A truncated report could pass with only one of its scheduled specs. Require integer counts and reconcile total, scheduled and reported It specs per suite. Keep hook reports outside the It count. Add regressions for malformed counts and incomplete reports under normal and optimized Python. Assisted-by: nib:unknown Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Preserve standalone settings and startup tri-state while preparing separate worker connection and NATS storage. Keep legacy start_on_boot inert, reject unsafe configuration files, and atomically persist owner-only settings before applying them. Remove configuration and environment value dumps. Add migration and filesystem-security specs, config-only isolated verification, and user-facing storage documentation. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Keep worker startup fail-closed pending runtime and ownership integration. Share standalone command construction, isolate worker authentication in the child environment, and bound/redact child log sinks. Add CLI-tag compatibility coverage and a strict focused/full launch verification gate. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Resolve NATS CA, client certificate and key paths against the launcher working directory before validation so the child opens the same paths from its data directory. Preserve saved settings and document resolution semantics. Cover missing and conflicting child files, all three TLS paths and immutable launch snapshots. Worker startup remains fail-closed. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add opt-in atomic owner-only schema-versioned snapshots with session identity, sequence freshness, approval, credential, authenticated NATS, subscription, armed readiness and heartbeat evidence. Reject non-2xx heartbeats and expose an exact bounded runtime capability probe without enabling launcher worker startup. Exercise real JWT NATS lifecycle subscriptions and reconnect with HTTP registration fixtures. Extend status verification with genuine CLI/UI prerequisites, strict full-suite evidence and race runs. Document readiness limits and the remaining lifecycle and integration gates. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Reject expired approval, stale unseen publications, inconsistent timestamps and snapshots not bound to the owned child PID. Keep accepted freshness on the reader's monotonic clock. Observe controller approval independently of static or expiring credentials through a token-protected read-only query. Keep managed credential renewal from restoring offline nodes or changing registration telemetry. Older controllers fail closed; launcher startup remains disabled. Exercise shared production approval and composite readiness components, non-expiring refresh-loop behavior, static revocation, backend failure, permission recovery and publication cleanup. Extend verification to controller handler and authentication suites. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Keep renewal failures separate from current approval and retained credentials. Heartbeat and approval success cannot clear this evidence. Share production renewal wiring with expiring-JWT tests. Cover rejected renewals, missing credentials, recovery, and authenticated reconnect after the original JWT expires. Retain independent non-expiring approval tests. Document renewal evidence and regenerate Swagger for approval queries and credential renewal with make swagger using swag v1.16.6. Assisted-by: nib:gpt-6-astra Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Dynamic credential providers can recover after a JWT expires during a renewal outage. nats.go closes the connection permanently after two identical authentication errors unless IgnoreAuthErrorAbort is set, so the worker could never reconnect after a successful renewal. Set IgnoreAuthErrorAbort only when a credential provider is configured. Static credentials keep the default terminal authentication behavior. Test the scope with a real authenticated broker. The renewal test keeps the same broker running through expiry and the 30 second renewal retry, proves the original JWT is rejected, and verifies that the worker reconnects on the same broker process once renewal succeeds. Document the dynamic-provider retry behavior and the unchanged static behavior in the distributed mode and launcher configuration pages. Assisted-by: Claude:claude-haiku-5-5 go test golangci-lint
The process ledger forgot a backend as soon as its leader stopped, even when descendants of the group still ran, ignored write failures, could not identify a process on macOS, and silently dropped records it could not resolve. A restarted worker could then double-book a port or GPU memory held by an orphan. Release a record only after the whole process group is gone: a group that outlives its leader is killed and waited for up to five seconds, and stays recorded if it cannot be emptied. Refuse to run a backend whose record cannot be written. Read start times from ps where /proc does not exist, and store the launch generation with each record. At startup, refuse to serve when a record is unresolved (unverifiable identity, a group that survives the kill, an unreadable ledger). Keep the record and signal nothing unverified. Assisted-by: Claude:claude-sonnet-5-5 go test golangci-lint
Read the generation from the status publisher instead of the process environment, as the repository lint rules require, and run ps with a fixed locale and path so start times compare equal across processes. Assisted-by: Claude:claude-sonnet-5-5 go test golangci-lint
Compare the owner uid of a stat result as int64 instead of converting os.Geteuid() to uint32, which gosec reports as G115 integer overflow. Both sides widen, so the comparison keeps its meaning. Replace the os.Lstat probe in NewPublisher, which gosec reports as G703 path traversal because the path comes from the environment. The probe now goes through the validated, owner-only session directory descriptor and fstatat with AT_SYMLINK_NOFOLLOW, so a fresh launch still refuses any existing status.json, including a dangling symlink. No suppression comments are added. Assisted-by: Claude:claude-sonnet-5-5 go gosec golangci-lint
mudler-agent
force-pushed
the
feat/launcher-worker-resume
branch
from
October 16, 2026 07:15
f56cfbb to
4fe1947
Compare
| dir := os.NewFile(uintptr(fd), "launcher settings directory") | ||
| var st unix.Stat_t | ||
| if err := unix.Fstat(fd, &st); err != nil || int64(st.Uid) != int64(os.Geteuid()) || st.Mode&0022 != 0 { | ||
| dir.Close() |
| f := os.NewFile(uintptr(fd), "launcher.json") | ||
| var st unix.Stat_t | ||
| if err = unix.Fstat(fd, &st); err != nil || st.Mode&unix.S_IFMT != unix.S_IFREG || int64(st.Uid) != int64(os.Geteuid()) || st.Nlink != 1 { | ||
| f.Close() |
| // Never follow a replaced parent/snapshot to invalidate it. | ||
| if dir, e := directory(p.path); e == nil { | ||
| _ = removeSnapshot(dir) | ||
| dir.Close() |
| } | ||
| b := make([]byte, 32) | ||
| if _, e = rand.Read(b); e != nil { | ||
| os.Remove(dir) |
| f := os.NewFile(uintptr(fd), "worker status") | ||
| var st unix.Stat_t | ||
| if unix.Fstat(fd, &st) != nil || int64(st.Uid) != int64(os.Geteuid()) || st.Mode&unix.S_IFMT != unix.S_IFREG || st.Mode&07777 != 0600 || st.Nlink != 1 || st.Size > MaxSize { | ||
| f.Close() |
| return ErrUnavailable | ||
| } | ||
| if old != nil { | ||
| old.Close() |
A reader that opened status.json just before the writer replaced it holds an inode with no links left. The link-count check required exactly one link, so that read failed as "status unavailable". This was the rare failure of the atomicity spec, and it would show a brief false connection problem on a launcher. An unlinked file that was complete and owner-only is still valid. Refuse only a snapshot with extra hard links. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
… (LW06) The launcher now runs its runtime through one lifecycle controller instead of tracking a bare child process. - One operation at a time: start, stop and quit are refused while another is in progress. Results tagged with an older generation are dropped. - A per-user lock file (owner-only, close-on-exec, dropped by the kernel when the holder dies) keeps a second launcher from starting an overlapping role. - Before a start, the launcher reads the worker's process ledger. It kills and verifies every group that provably belongs to an earlier run and drops records whose group is gone or whose pid was recycled. A record it cannot resolve blocks the start and nothing unverified is signalled. - Stop sends SIGTERM, waits 30 s, kills the runtime group, waits 5 s more, then kills every recorded backend group. It reports stopped only after the runtime group and all recorded groups are verified empty. Otherwise it reports "shutdown incomplete", keeps the lock, blocks every start and leaves stop available to retry. - An unexpected runtime exit runs the same cleanup. - A worker starts only if its runtime prints the status capability. Its state comes from the private status snapshot, never from log text, and liveness alone shows "Connection problem", not "Ready". The process identity rules move to pkg/workerledger so the worker and the launcher judge a recorded group the same way. The worker behaviour and its specs are unchanged. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The repo lint rejects testing.T assertions. Port the two small test files added by the lifecycle change and the link-count fix to Ginkgo and Gomega. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…ns (LW07) The settings window now starts with a mode selector: Run locally or Join a distributed server. Each mode keeps its draft when you switch. - The join form edits the frontend URL, registration token, advertised address and node name. Advanced settings hold the NATS URL and TLS files, the maximum gRPC port, optional JWT and seed overrides, runtime environment overrides and a per-session plaintext acknowledgement. The window shows the reverse HTTP port and gRPC range the frontend must reach. - Secret fields are masked and never filled from the file. A blank field keeps the saved value; replacing or clearing a secret is explicit. - Save persists both profiles. A change that affects a running role offers Stop and apply or Cancel. Stop and apply commits only after the runtime and its backends are confirmed gone and never starts the replacement. Start, save and update are refused while it runs. - Environment and log level edits are drafts until Save, so a change cannot reach the saved profile without the stop prompt. - Worker mode names the controls Start worker and Stop worker. The tray adds Open deployment UI, which opens the saved frontend URL without credentials. Stop stays available until shutdown is confirmed. - Start selected mode when launcher opens keeps the legacy default for standalone. A worker starts at launch only when the owner chose it. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
| func psStartTime(pid int) string { | ||
| ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second) | ||
| defer cancel() | ||
| cmd := exec.CommandContext(ctx, "ps", "-o", "lstart=", "-p", strconv.Itoa(pid)) |
| return err | ||
| } | ||
| tmp := path + ".tmp" | ||
| f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600) |
| // ledger; an unreadable or corrupt one is an error, because ownership of the | ||
| // earlier processes is then unknown. | ||
| func ReadFile(path string) ([]Entry, error) { | ||
| data, err := os.ReadFile(path) |
| } | ||
| } | ||
|
|
||
| cmd := exec.Command(spec.Binary, spec.Args...) |
| f := os.NewFile(uintptr(fd), "worker status directory") | ||
| var st unix.Stat_t | ||
| if unix.Fstat(fd, &st) != nil || int64(st.Uid) != int64(os.Geteuid()) || st.Mode&07777 != 0700 { | ||
| f.Close() |
| return nil, fmt.Errorf("cannot restrict the launcher lock: %w", err) | ||
| } | ||
| if err := unix.Flock(fd, unix.LOCK_EX|unix.LOCK_NB); err != nil { | ||
| f.Close() |
| return nil, errors.New("the launcher lock is not an owned regular file") | ||
| } | ||
| if err := f.Chmod(0o600); err != nil { | ||
| f.Close() |
| f := os.NewFile(uintptr(fd), "launcher lock") | ||
| var st unix.Stat_t | ||
| if unix.Fstat(fd, &st) != nil || st.Mode&unix.S_IFMT != unix.S_IFREG || int64(st.Uid) != int64(os.Geteuid()) || st.Nlink != 1 { | ||
| f.Close() |
| if err != nil { | ||
| return err | ||
| } | ||
| d.Close() |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the worker-side groundwork for a generic distributed worker mode in the
launcher. This branch carries plan tasks LW01 to LW07. The end-to-end acceptance run
(LW08) and the platform acceptance (LW09) are not done (see "Plan status").
mode-specific validated launches and child-only secret injection (LW01 to LW03).
local-ai workercan publish an opt-in, owner-only status snapshot that alauncher polls. Readiness requires current approval, credentials, NATS,
lifecycle subscriptions, armed local checks and an accepted heartbeat (LW04).
Approval is observed through a read-only, registration-token-protected
GET /api/node/:id/statusroute; credential renewal uses arenew_node_idrequest field so renewal does not restore offline nodes.
during a renewal outage (
IgnoreAuthErrorAbort, only when a provider isconfigured). Static credentials keep NATS's default behavior (LW04).
that outlives its leader is killed and waited for (5 s), and stays recorded
if it cannot be emptied;
/proc, or frompswhere/procdoes not exist;unreadable ledger) makes the worker refuse to start; nothing unverified is
signalled and the record is kept.
operation runs; results of older generations are dropped;
starting an overlapping role;
earlier run left, and refuses to start over a record it cannot resolve;
then every recorded backend group; "Stopped" is shown only after all groups
are verified empty, otherwise "shutdown incomplete" keeps starts blocked and
stop available;
comes from the status snapshot, never from log text;
pkg/workerledger, shared by the workerand the launcher.
settings, masked secrets that are kept unless replaced or cleared, derived
reverse-connectivity ports, Stop and apply / Cancel when a change affects a
running role, a per-session plaintext acknowledgement, and the tray actions
Start worker, Stop worker and Open deployment UI.
status.jsonjust before it was replaced saw an inodewith no links and got "status unavailable". This was the rare failure of the
atomicity spec. Only an extra hard link is refused now.
Why
A launcher that starts and stops a worker must be able to prove that no backend
process of an earlier run is still alive before it starts another role. The old
ledger forgot a backend when its leader exited, ignored write errors, could not
identify processes on macOS and dropped records it could not resolve.
How it was tested
Linux, cgo on,
-tags auth:go testoncmd/launcher/internal,pkg/workerledger,pkg/workerstatus,core/services/worker,core/services/messaging,core/cli/workerregistrypass.
go test -racepasses forcmd/launcher/internalandpkg/workerstatus.that ignores SIGTERM, backend groups that outlive the runtime, a leader that
exits before its group, a recycled pid, an unresolvable or corrupt ledger, a
kill that never lands, concurrent starts, stale generations, and two launchers
racing for the lock. Mutation checks: skipping the backend group kill and
skipping the runtime force kill each fail the specs; dropping the stop prompt
in apply fails the apply specs.
drafts, masked secrets, labels per mode, derived ports) and test the settings
apply logic and the tray entries directly.
go vetclean.golangci-lint(Go 1.26 toolchain) reports no finding in thenew packages or in
core/services/worker.make test-launcher-appimagepasses, andmake build-launcher-appimageproduces an AppImage whose extracted tree has the expected layout, no missing
libraries, and the new UI.
Known limits
TLS, a backend and a model. The build host has no PostgreSQL and no Docker
access. A mocked state test is not an acceptance pass. Nothing here proves
registration, approval, credential issuance, backend install, model load or
inference through a real frontend with a launcher-managed worker.
start-up smoke did not run (no
xvfb-runordesktop-file-validateon thehost). macOS did not run at all: the
ps -o lstart=identity path is coveredby a fallback spec that simulates a system without
/proc, and the macOSlauncher, tray and descendant tests have no native run.
pkg/workerstatus,the worker package and the launcher use unix-only calls and do not build for
Windows.
session. There is no screenshot from a real window.
and clean it; the launcher now does this at the next start or stop, and keeps
blocking when it cannot. The worker binary itself still refuses to start over
such a record.
Plan status
Done: LW01 baseline, LW02 secure configuration migration, LW03 validated
launches and secrets, LW04 evidence-backed worker status, LW05 durable process
ownership, LW06 serialized lifecycle and safe role switches, LW07 settings and
tray.
Not done: LW08 authenticated distributed integration run (could not run here),
LW09 packaging and platform acceptance (Linux native and AppImage build done;
AppImage smoke and macOS not run).
Known issues on this branch
golangci-lintreports 10 issues inpkg/workerstatus(9 errcheck, 1forbidigo) from the status snapshot code. They are not fixed.
cmd/launcheris excluded from the repo lint target, so the launcher code ischecked by
go vetand tests only.🤖 Generated with Claude Code