fix(llama-cpp): stop a generation whose stream is gone

[localai-org-maint-bot]

Description

grpc::ServerWriter::Write() returns false once the peer is gone, and PredictStream ignored that result at every call site. The handler kept pulling decoded tokens and writing them into a dead stream, so the llama.cpp slot stayed busy until the generation ended on its own terms.

A model configured with max_tokens: 0 and a large context_size ends on its own terms only at the context limit. On a 35B model at ~41 t/s a 120k context is about fifty minutes, and a slot held that long is a slot every other request for that model queues behind.

This came out of a live incident on a distributed deployment. Two abandoned generations had each produced 18,000+ tokens and were still decoding, holding both slots, while node_models.in_flight read 0 — the frontend had long since given up. The node had 41 GB VRAM free, answered its NATS control plane instantly, and served nothing. Every new request timed out waiting for a slot, and each timeout abandoned another generation, so the node fell further behind the longer it ran. Restarting the worker cleared it; the next request answered in 1.1s.

The fix tracks the peer instead. The first failed write retires it for good, since a stream never recovers, and the RPC's own cancellation flag folds into the same predicate so the loop has one condition to test. Returning early is what frees the slot: ~server_response_reader() posts SERVER_TASK_TYPE_CANCEL for whatever is still decoding.

TTSStream already checked Write() (lines 3240, 3261); this brings PredictStream in line. Cancellation stays cooperative and is checked between decoded results, so a batch already in flight may finish before the request stops.

Same class of bug, and same remedy, as #11822 (fix(ds4): cancel abandoned inference) — llama-cpp never got the equivalent treatment.

Notes for Reviewers

The decision is extracted into stream_peer.h so it is unit-testable without a full backend build, following the thread_params.h / request_lifecycle.h pattern. stream_peer_test.cpp is picked up automatically by backend/cpp/run-unit-tests.sh — no CI edit needed.

Verify:

bash backend/cpp/run-unit-tests.sh    # 19 files, exit 0

The sticky-retirement assertions are the ones that matter: I confirmed they fail if observe_write ignores its argument, which is the exact bug being fixed.

Worth knowing, because this fix does not cover it: cancellation only helps once a client has actually gone away. A client that waits still receives a full context worth of tokens, so a generation cap on the model config remains the real guard. The docs note this, and the incident config had max_tokens: 0 with repeat_penalty: 1 — no cap and no brake on a repetition loop.

Not addressed here, found in the same incident and worth separate issues:

• node_models.in_flight is persisted and never reconciled, so counters survive an OOM-killed frontend as permanent phantom load.
• POST /api/nodes/:id/models/unload returned {"message":"model unloaded"} / HTTP 200 twice while the backend process kept running and holding 27 GB — StopBackend fails silently.

Signed commits

• Yes, I signed my commits.
• Documentation updated (docs/content/) for user-facing changes, or not applicable