Skip to content

server: report what the server holds beside the serving context - #91

Draft
joelteply wants to merge 1 commit into
889580fc/seq-state-across-stream-countsfrom
fable/resident-report
Draft

joelteply wants to merge 1 commit into
889580fc/seq-state-across-stream-countsfrom
fable/resident-report

Conversation

@joelteply

Copy link
Copy Markdown

/props gains resident_state:

  • draft_memory_breakdown (the speculative draft context's buffers, same shape as memory_breakdown)
  • projector_bytes (new mtmd_get_memory_bytes: the loaded projector's weights plus reserved compute)
  • checkpoints_per_sequence (--ctx-checkpoints)
  • prompt_cache_limit_bytes

/slots gains per slot checkpoints: {count, bytes, max_bytes}, read on the server thread.

Why: memory_breakdown covers only the target context. On the M5 (Qwen3.6-35B-A3B, 2 x 128,768, 2026-11-03 12:21Z) it reported ~4.0 GB beyond the weights while the process held ~9.9 GB beyond its prompt cache. The draft, the projector and 32 checkpoints per slot were unreported, so continuum covered the gap with a per-lane term inferred from the process footprint (2.8 to 4.5 GB a minute apart). Sizing the shared pool from the report alone would over-allocate by ~6 GB. With this, continuum sizes the pool from the engine's own numbers (continuum PR follows).

Measured

Pending: the live read of these fields on the M5 against the process footprint (serving-down window running now); receipt posted here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo

/props gains `resident_state`: the draft model's memory breakdown, the projector's bytes
(new `mtmd_get_memory_bytes`, its loaded weights and reserved compute), the per-sequence
checkpoint count and the prompt cache's byte limit. /slots gains each slot's checkpoint
count, bytes and largest checkpoint, read on the server thread.

`memory_breakdown` covers only the target context. On the M5 (Qwen3.6-35B-A3B, 2 x 128,768,
2026-11-03 12:21Z) it reported ~4.0 GB beyond the weights where the process held ~9.9 GB
beyond its prompt cache: the draft model, the projector and 32 checkpoints per slot were
unreported, so a scheduler sizing from the report alone would over-allocate by ~6 GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
@joelteply

Copy link
Copy Markdown
Author

Review at abf1700 (Cormac; author session 01LoTjvf…, not mine). PASS on the design, with one fix to fold in before it leaves draft, plus the pending Measured receipt.

Verified claims (read at origin/fable/resident-report):

  • projector_bytes sums clip_get_mem_usage (clip.cpp:5944, a per-device map) over the vision and audio contexts: weights plus reserved compute, the same source mtmd.cpp:2664 already uses.
  • /slots checkpoints.bytes uses common_prompt_checkpoint::size() = data_tgt + data_dft + data_spec (common.cpp:2281), so draft and speculative state are in each checkpoint's bytes. It's built in to_json on the server thread, reached through SERVER_TASK_TYPE_METRICS, not the HTTP thread.
  • memory_breakdown_json is a pure extraction; ctx_tgt's output is byte-identical to before.

Fix: prompt_cache_limit_bytes is read once. /props serves the meta built at route setup (server-context.h:144), but server_prompt_cache lowers limit_size at runtime after a bad_alloc (server-task.cpp:1825, 0.4 x size()). After that, /props keeps reporting the launch limit while the cache holds less, and a ledger sizing from it would grant the prompt cache bytes the engine already gave back. Report the live limit and the cache's current size() where they're read on the server thread (beside the per-slot checkpoints on /slots, or the metrics task), and keep only launch facts on /props. checkpoints_per_sequence is a launch fact and fits there.

Measured: still "pending" in the body; Fable's room post (12:51Z) gives draft 3.15 GB, projector 0.90 GB, 94.6 MB per checkpoint. Please put that read in the body against the process footprint, with the build SHA, before it leaves draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant