fix(distributed): make backend.stop report what it stopped

[localai-org-maint-bot]

Description

backend.stop was the one lifecycle subject a worker never answered. Every other one — install, upgrade, delete, list, models-running, model-unload, model-stop — uses SubscribeReply; this one used a bare Subscribe, and the controller side was a bare Publish that returned nil as soon as the local publish succeeded.

So a stop that killed nothing, and a stop that failed outright, were indistinguishable from one that worked.

POST /api/nodes/:id/models/unload calls model.unload (acknowledged) and then StopBackend (not). I hit this on a live deployment: the endpoint returned {"message":"model unloaded"} / HTTP 200 twice while the backend process kept running and held 27 GB of VRAM. The endpoint's own "model unloaded but backend stop failed" branch could never fire, and the worker's xlog.Error("Failed to stop backend process") went to a log nobody was reading.

This gives the subject a reply. The worker enumerates the process keys it terminated and reports any per-process error, so StopBackend fails when the stop failed.

Deliberately not errors:

• Nothing matched — reported as success with an empty list, logged as backend.stop matched no running process. Stopping a backend that is not running leaves the caller in the state it asked for, and eviction and cleanup paths stop already-gone models routinely. Making it an error would break them. The empty list is what says nothing matched; ReportsStoppedProcesses is what makes that emptiness trustworthy, mirroring how BackendDeleteReply already solves the same problem.
• No answer — a worker built before this reply still receives the request and still stops the backend, it just stays silent. A timeout degrades to the old assumption rather than failing every stop on a fleet mid-upgrade.

Only silence degrades. A transport error is still reported, because UnloadRemoteModel skips its registry cleanup for a node it could not reach and needs to keep hearing about that. That distinction is what keeps the existing "continues when one node fails" contract intact.

Notes for Reviewers

Compatibility runs both ways and neither direction breaks:

	old controller	new controller
old worker	unchanged	stop happens, no reply → 15s wait, degrade, logged
new worker	SubscribeReply drops the response when msg.Reply == ""	acknowledged

The one cost is that on a mixed fleet each stop against an un-upgraded worker waits out backendStopAckTimeout (15s) before falling back. That bound is sized off the worker's own 5s workerBackendFreeTimeout plus the kill. It's noted in the docs.

Verify:

go test ./core/services/nodes/ ./core/services/worker/ ./core/services/messaging/
make lint LINT_NEW_FROM=origin/master

Five specs in unloader_test.go asserted the old fire-and-forget contract and now assert the new one; the rest of that suite (586 specs) was green throughout. New specs cover the reported failure, the matched-nothing success, the silent-worker degrade and the transport-error report. make lint reports 0 issues.

One unrelated flake to flag honestly: backend_logs_test.go:163 (a WebSocket streaming test) failed once during this work. It passes 3/3 on this branch and passes on clean master; I don't touch websockets or backend logs.

This came out of the same incident as #11860, which fixes the abandoned generations that made the node look wedged in the first place. The two are independent — this one is about being told the truth when you try to clean up afterwards.

Still outstanding from that incident, not addressed here: sweepLeakedInFlight cannot fire while a replica keeps receiving traffic, because IncrementInFlight stamps last_used at request start and the sweeper requires last_used to be 30 minutes stale. A leaked counter causes retries, and the retries keep refreshing last_used, so the sweeper is disabled by exactly the condition it exists to correct. That one needs a design call on the discriminator rather than a patch.

Signed commits

• Yes, I signed my commits.
• Documentation updated (docs/content/) for user-facing changes, or not applicable