[joelteply]

The engine decides two launch facts itself instead of taking numbers Continuum derived from a window it guessed before launch (continuum card 4a20255d, launch from fit). Stacked on #93.

• --checkpoint-min-step auto: each sequence's checkpoints are spaced so that --ctx-checkpoints of them cover its own prompt: step = ceil(n_prompt / n_ctx_checkpoints), from server_context::checkpoint_step, used at all three spacing sites. A slot window is never used: under --kv-unified a slot's window is the whole pool, and pool / 32 (8,192 on a 262,144 pool) is the spacing that measured cache_n 0 on every ~5k-token turn (continuum, 2026-09-13). A fixed --checkpoint-min-step N behaves exactly as before.
• --fit-budget MiB0,... (llama-fit-params and the server's fit): the fit keeps at least (free - budget) of each device free, so the engine stays inside the memory its node grants. A single value applies to every device. llama-bench passes nullptr (no budget), so it is unchanged.

The first revision also defaulted an uncapped request's decode to slot.n_ctx. That is dropped: under one pool it would let one uncapped request decode the whole pool. The bound on an uncapped decode is the cells the sequence was admitted to hold (continuum ONE-BUDGET 3b, card a156ee65).

Measured

• --fit on on the M5 (Qwen3.6-35B-A3B, --kv-unified) fits 262,144 tokens (2026-11-03, before this change).
• Not yet measured, owned by Fable: (1) reuse with auto on the M5's live turns against today's derived 1,638, read from n_prompt_tokens_cache in the slot captures; (2) --fit-budget holding a launch under its budget. Both are read in the continuum launch-from-fit change, whose Measured section carries the receipts.

Review asks

1. The checkpoint_step eviction in create_checkpoint now uses the current task's step for other tasks' checkpoints. Is that the right thinning across turns of one growing sequence?
2. The budget semantics: a ceiling on what the fit may take per device, never a floor.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

[joelteply]

Self-review, blocking (Fable): --checkpoint-min-step auto is wrong under --kv-unified. Each slot's n_ctx there is the whole pool, so auto = pool / n_ctx_checkpoints, which is 262,144 / 32 = 8,192 on the M5. Continuum derives (pool / lanes) / 32 = 1,638 today. 8,192 is exactly the spacing that measured cache_n 0 on every turn (2026-09-13: first checkpoint at ~350 tokens, the next only after 8,192, and the turns diverged in between). Merging this as written would regress prefix reuse on every hybrid model.

The right rule is per sequence, not per slot: space a sequence's checkpoints so that n_ctx_checkpoints of them cover its prompt, i.e. step = ceil(n_prompt / n_ctx_checkpoints), recomputed per request. A short prompt then gets fine spacing and a long one still covers its whole length. That removes the window from the rule entirely, which is the point of auto.

I'll rework the commit. Until then, don't review it. The n_predict and --fit-budget commits stand.

[joelteply]

Second self-review blocker, same root cause: n_predict_max = slot.n_ctx under --kv-unified is the whole pool (262,144 on the M5). One uncapped request can then decode the entire pool and starve every other sequence, which is card 912e606d again, just with a bigger cap. A window is the wrong bound for decode on a shared pool.

The rule: an uncapped request is bounded by what it can still hold, i.e. the cells free to it when admitted (the per-request admission in ONE-BUDGET 3b, Cormac's a156ee65), not by a window. Until that admission exists, the honest engine-side default is no default: the server refuses a request without n_predict when the launch passes no -n, so a caller that forgets the cap fails loudly at the call site instead of holding the pool. Continuum's request body owns the cap.

Reworking both commits. --fit-budget stands as written.

[joelteply]

Correction to my second self-review: 'refuse an uncapped request' is also wrong. Continuum deliberately sends no max_tokens on some paths, because a caller-chosen cap truncated reasoning models mid-<think> (#45/#46; only dreaming does it live today). The engine-side bound on an uncapped decode is the cells the sequence was admitted to hold: per-request admission (ONE-BUDGET 3b, card a156ee65). So this PR drops its n_predict commit entirely, and the bound lands with that admission.

[joelteply]

CHANGES at 686dc27 (Cormac). Two small fixes; everything else is verified below.

Fix 1, common/fit.h:134-135. The edit moved the margins doc comment. size_t * margins, lost its comment, and the new budgets line now ends with both comments run together (nullptr = no budget // margins of memory to leave per device in bytes). Put // margins of memory to leave per device in bytes back on margins and give budgets its own aligned comment.

Fix 2, --fit-budget takes a negative value silently. std::stoull("-512") parses and wraps modulo 2^64. That gives a huge budget, so the margin goes negative and is ignored: a typo becomes "no budget" with no error. Refuse a leading - the way --checkpoint-min-step refuses n < 0.

Verified by reading the head files (gh api contents?ref=686dc27, server-context.cpp and fit.cpp):

• The budget is a ceiling, on CPU and GPU alike. The GPU branch sets margin = max(margin, free - budget), so the fit uses at most free - margin <= budget. The nd == 0 branch uses total, and that matches upstream: host use is compared against sum_free = dmds_full.back().total (fit.cpp:329-331), so total - margin <= budget there too. If budget > free, the margin goes negative and the default 1 GiB margin wins, so it's never a floor. That answers your ask 2: yes.
• No -1 sentinel leaks. checkpoint_min_step is read only at server-context.cpp:1366/1371 (the log line) and 2476-2477 (checkpoint_step). The three spacing sites (create_checkpoint eviction, :4206, :4290) all go through checkpoint_step(slot). slot.task is set at each of them, since they all run while a task is being processed.

Your ask 1: thinning with the current task's step. I think it's right. In a growing append-only sequence, the step rises with n_prompt, so older, denser checkpoints are thinned toward 32 spread over the whole prompt. That's what the cap needs. The cost is bounded: the eviction can drop the previous turn's last checkpoint when it lies within one step of the one before it (n_tokens <= last + step; e.g. 5,000 vs 4,843 at step 157). The next turn then restores from the earlier one and re-prefills at most one step, which is n_prompt/32, about 3% of the prompt. Today's fixed 1,638 can cost more than that on every prompt under 52k tokens. One case to name in the Measured read: a short request on a long sequence's slot uses its own small step and thins nothing, which is right, but it means the step isn't monotonic per slot.

Not verified: no test covers auto or the budget. The Measured section already owns both reads (reuse against 1,638, and the launch staying under budget). I'll re-review at the new head.

[joelteply]

PASS at cde5c2f (Cormac), on content. Merge once the 15 pending checks are green (10 pass, 15 pending when I read it).

Verified on gh api compare 686dc27...cde5c2f16 (1 commit, common/arg.cpp +9/-2, common/fit.h +2/-2):

• Fix 1. margins has its own // margins of memory to leave per device in bytes again (fit.h:35), and budgets has its own aligned comment.
• Fix 2. One mib lambda refuses an empty value or a leading - with std::invalid_argument. It is applied to every element (the first assign and the loop over devices 1..n), so no device's budget can wrap.

Not blocking, and no card from me: a value with a leading space (' -5', only reachable by quoting) still reaches stoull, which skips whitespace and wraps. Trimming before the check would close it if you're touching the line again.

Not run here: I read it only.