diff --git a/.abcd/development/decisions/adrs/2609090716413337-the-statistics-store-s-record-kinds-are-request-load-removed.md b/.abcd/development/decisions/adrs/2609090716413337-the-statistics-store-s-record-kinds-are-request-load-removed.md index 0c5c87e4..3fe0d656 100644 --- a/.abcd/development/decisions/adrs/2609090716413337-the-statistics-store-s-record-kinds-are-request-load-removed.md +++ b/.abcd/development/decisions/adrs/2609090716413337-the-statistics-store-s-record-kinds-are-request-load-removed.md @@ -1,10 +1,10 @@ --- id: adr-2609090716413337 slug: the-statistics-store-s-record-kinds-are-request-load-removed -status: accepted +status: superseded date: 2026-09-09 supersedes: adr-2609061610107154 -superseded_by: null +superseded_by: adr-2609121450000000 related_intents: [itd-2609061521102742, itd-2609061602043757] related_rfcs: [] related_adrs: [adr-2609061610107154, adr-2609061503319212] diff --git a/.abcd/development/decisions/adrs/2609121450000000-the-statistics-store-gains-a-fifth-record-kind-a-footprint.md b/.abcd/development/decisions/adrs/2609121450000000-the-statistics-store-gains-a-fifth-record-kind-a-footprint.md new file mode 100644 index 00000000..98de5239 --- /dev/null +++ b/.abcd/development/decisions/adrs/2609121450000000-the-statistics-store-gains-a-fifth-record-kind-a-footprint.md @@ -0,0 +1,76 @@ +--- +id: adr-2609121450000000 +slug: the-statistics-store-gains-a-fifth-record-kind-a-footprint +status: accepted +date: 2026-09-12 +supersedes: adr-2609090716413337 +superseded_by: null +related_intents: [itd-2609091712141073, itd-2609061521102742] +related_rfcs: [] +related_adrs: [adr-2609090716413337, adr-2609061610107154, adr-2609061503319212] +--- + +# ADR-2609121450000000: The statistics store gains a fifth record kind, a footprint reading, and the request and load records gain the facts that decide a served window and a concurrency + +## Context + +adr-2609090716413337 ratified four record kinds: a request line, a load, a +removal with one of seven reasons, and a settings record. The usage +measurement intent (itd-2609091712141073) needs the facts that decide a +served window and a decode concurrency — how prompts sat against the window, +how many requests were in flight, which sampling values were in force and +whether the client overrode them, and what the model server's memory did — +and one of those, the memory, is not a fact about any request. It is a +reading of a process, taken on a clock, and it needs a record of its own or +the figure exists only as whatever value a completing request happened to +see. + +This decision was taken by the agent on the maintainer's lane assignment of +2026-09-12, under the instruction to decide autonomously and record; the +record-kinds decision is architecture-shaping, which is why this is an ADR +and not a line. + +## Decision + +We supersede adr-2609090716413337 on the record kinds, and adopt five: + +1. **A request line**, as before, gaining: the declared and served windows + the request was judged against, Gropius's own estimate of the prompt's + size (written for a refused request too), the model's in-flight count at + admission, the names — never the values — of the sampling parameters the + client set, and the server's footprint as last sampled before completion. +2. **A load**, as before, gaining the sampling values the server was launched + with. +3. **A removal**, unchanged. +4. **A footprint reading** — `kind: "footprint"` — the model's repo id, when, + and the server process's resident memory in bytes, taken every thirty + seconds for each running server and reported through the pool's observer. +5. **A settings record**, unchanged. + +Every new numeric is bounded at the recorder as the windows already are: a +window past the registry's bound, a count past any batch, a footprint past +any Mac, or an override name outside the closed set is dropped, never +repaired. The override names are the one field derived from a client's body, +and the reference page says so. + +## Alternatives Considered + +1. **Carry the footprint only on the request record.** Rejected: the figure + would exist only at the moments requests completed, which on a quiet + server is never, and the intent's criterion asks for a periodic record. +2. **A separate file for the samples.** Rejected: the store is one + collection path under one switch and one retention rule + (adr-2609061610107154), and a second file would need a second of each. +3. **Store the client's sampling values.** Rejected: adr-2609061503319212 + draws the line at content the client sent; a name says which knob was + touched and nothing of what was asked. + +## Consequences + +- `internal/stats.StoreKinds` names seven kinds of line (five records plus + the two summary lines), and the reference page's table is held to it. +- The summary a day folds into ignores footprint readings: a mean over a + day is not a figure anyone would size a setting from, and the series is + what the view draws. +- The record size constant is re-measured over a record carrying every + field. diff --git a/.abcd/development/intents/planned/itd-2609091301112705-gropius-measures-a-newly-downloaded-model-s-servable-context.md b/.abcd/development/intents/shipped/itd-2609091301112705-gropius-measures-a-newly-downloaded-model-s-servable-context.md similarity index 96% rename from .abcd/development/intents/planned/itd-2609091301112705-gropius-measures-a-newly-downloaded-model-s-servable-context.md rename to .abcd/development/intents/shipped/itd-2609091301112705-gropius-measures-a-newly-downloaded-model-s-servable-context.md index c4fc0f7d..4afdcbf6 100644 --- a/.abcd/development/intents/planned/itd-2609091301112705-gropius-measures-a-newly-downloaded-model-s-servable-context.md +++ b/.abcd/development/intents/shipped/itd-2609091301112705-gropius-measures-a-newly-downloaded-model-s-servable-context.md @@ -138,12 +138,13 @@ named bound rather than the architecture's limit. - **How the model's own window is ever learned.** A coordinated pool-side path exempt from the gateway's bounds, a raised `upstream_header_timeout_sec` for the duration, or the manual campaign script staying the only answer. Nothing here decides it. - **Whether the probe also samples memory**, which would keep `capability`'s safety factors of 5 and 7 under continuous evidence rather than one evening's. Out of scope here — that package belongs to another lane — and worth a follow-on. - **The idle threshold's value**, and how many models one idle window may take: six models at forty minutes each is an evening, and the models load one at a time. -- **Arbitration with the model self-test**, which also runs at idle and is not yet on `main`. Two idle jobs on one Mac need one idea of "idle" between them, and whichever lands second inherits the question. +- **Arbitration with the model self-test** — resolved 2026-09-12 at implementation: the self-test landed first and its loop is the one idle primitive; the probe is a `selftest.Job` it schedules, so idleness, yielding, the never-evict check, the held-back reason and the idle threshold (`idle_threshold_sec`) are decided once, in `internal/selftest`, for both. - **Whether the yielding seam becomes real preemption in the pool.** The pool has no preemption and no priority today; this intent reads the pool and cancels its own request instead. Any change to the pool itself is a coordinated follow-up with the lane that owns `internal/runtime`. ## Audit Notes -_Empty. Populated by intent-auditor when intent moves to shipped/._ + +Fidelity review OWED (receipt rcp-4431dce599cc). ## Grounds diff --git a/.abcd/development/intents/planned/itd-2609091712141073-gropius-shows-what-its-models-actually-do-so-the-settings-th.md b/.abcd/development/intents/shipped/itd-2609091712141073-gropius-shows-what-its-models-actually-do-so-the-settings-th.md similarity index 97% rename from .abcd/development/intents/planned/itd-2609091712141073-gropius-shows-what-its-models-actually-do-so-the-settings-th.md rename to .abcd/development/intents/shipped/itd-2609091712141073-gropius-shows-what-its-models-actually-do-so-the-settings-th.md index cd2d9081..ee3d235e 100644 --- a/.abcd/development/intents/planned/itd-2609091712141073-gropius-shows-what-its-models-actually-do-so-the-settings-th.md +++ b/.abcd/development/intents/shipped/itd-2609091712141073-gropius-shows-what-its-models-actually-do-so-the-settings-th.md @@ -67,7 +67,8 @@ per-request memory attribution the sample cannot give. ## Audit Notes -_Empty. Populated by intent-auditor when intent moves to shipped/._ + +Fidelity review OWED (receipt rcp-916dd6b8bf3f). ## Grounds diff --git a/.abcd/development/intents/planned/itd-2609091903463596-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md b/.abcd/development/intents/shipped/itd-2609091903463596-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md similarity index 97% rename from .abcd/development/intents/planned/itd-2609091903463596-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md rename to .abcd/development/intents/shipped/itd-2609091903463596-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md index 44fbf8c2..cf5ebf43 100644 --- a/.abcd/development/intents/planned/itd-2609091903463596-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md +++ b/.abcd/development/intents/shipped/itd-2609091903463596-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md @@ -63,7 +63,8 @@ needs a directory walk or a second reader. ## Audit Notes -_Empty. Populated by intent-auditor when intent moves to shipped/._ + +Fidelity review OWED (receipt rcp-81fed0a81798). ## Grounds diff --git a/.abcd/development/intents/shipped/itd-2609100457007827-gropius-tests-its-own-models-while-nobody-is-using-it-alice.md b/.abcd/development/intents/shipped/itd-2609100457007827-gropius-tests-its-own-models-while-nobody-is-using-it-alice.md new file mode 100644 index 00000000..886c88d8 --- /dev/null +++ b/.abcd/development/intents/shipped/itd-2609100457007827-gropius-tests-its-own-models-while-nobody-is-using-it-alice.md @@ -0,0 +1,97 @@ +--- +id: itd-2609100457007827 +slug: gropius-tests-its-own-models-while-nobody-is-using-it-alice +spec_id: spc-2609100502029941 +kind: standalone +suggested_kind: null +reclassification_history: [] +builds_on: [itd-2609091301112705, itd-2609061521082551] +severity: minor +impact: additive +origin: researcher-authored +production_mode: dictated-and-formatted +--- + +# Gropius tests its own models while nobody is using it. Alice switches on the self-test in Settings; whenever her Mac is idle, Gropius loads each downloaded model in turn, runs the same standard set of tests against it, unloads it, and records what it measured, so that the numbers she sizes a model's settings from come from her own Mac rather than from someone else's benchmark. The runs never delay a real request and nothing leaves the machine. + +## Press Release + +Gropius now measures its own models, on your Mac, while nobody is using it. +Alice ticks **Self-test when idle** in Settings and saves. That evening, +once the last request is a few minutes old and nothing is downloading, +Gropius loads the first of her models, runs the same short set of tests every +model gets — how long a load takes, how fast a long prompt is read, how fast +tokens come out, and what happens when four requests arrive at once — writes +one line of figures to a file in her Gropius folder, and unloads the model if +it was not resident before. Then the next model, and the next. In the morning +the file holds one line per model, measured on her machine, and the number she +sizes the served window and the concurrency from is hers rather than a +stranger's. Bob's agent sending a request at midnight is never made to wait: +the moment a real request arrives the run in progress is abandoned, marked as +yielded, and the model is his. Nothing in the file is a prompt or an answer, +and nothing in it leaves the Mac. + +## Why This Matters + +The 2026-09-06 model-bench campaign produced the numbers the memory budget and +the served window are now designed from — and it was one person running a +script by hand for an evening on one Mac, a record that goes stale with the +next model, the next runtime or the next machine. The context-probe draft +(itd-2609091301112705) automates one of those measurements; this is the +harness that draft is one test of, and it refines it: the idle detection, the +yielding to a real request and the rule against evicting a pinned model are +the same costs, carried once. The usage dashboard (itd-2609091712141073) +measures what real requests did; this measures what a model can do when asked +the same thing as every other model, which is the comparison usage traffic +never gives. The held draft itd-2609091712142715, settings that fit by +themselves, is what these figures exist to feed; it is not part of this intent. + +## Mechanism + +We expect one line of figures per model per day to be enough to compare models +and size their settings, because the four measurements are the ones the +2026-09-06 campaign actually used to decide the budget and the window, and +llama-bench's prompt-processing and text-generation pair has been the shared +vocabulary for local inference numbers since 2023. Shown wrong if a setting the +figures were meant to decide turns out to need a measurement not in the set, or +if idle time on a serving Mac is too rare for a cycle to complete. + +## Scope Conditions + +- Apple Silicon macOS, one Mac, with the self-test switch on; off is the default and the same state as today. +- Idle means: no request in flight on any model, nobody waiting for a load, no download running, and the last request older than a quiet period. +- The self-test loads through the pool's ordinary Acquire path and never a second launcher, so the budget, the pins and the served window are the pool's own; and it never evicts, loading a model only when it fits beside what is resident. +- The pool has no preemption: yielding is the self-test cancelling its own request, not the pool taking the model away, so a real request can wait for one in-flight self-test request to be cancelled. +- The results file holds figures only, never a prompt or an answer; the test prompts are constants in the code, and no client's request is ever part of a run. +- Shared-cache mode records under the serving account's own data directory, at 0600; the models measured are every account's, since they are one set. + +## Acceptance Criteria + +- Given the switch is off, when Gropius runs for any length of time, then no model is loaded by the self-test and no results file exists. +- Given the switch is on and the Mac is idle, when a ready model has not been tested within the re-test window, then it is acquired through the pool, the standard set runs against it, and one line is appended to the results file. +- Given a model was not resident before its run, when the run ends, then the model is unloaded; given it was resident, then it is left where it was. +- Given a run is in progress, when a request from a client arrives on any model, or a client's load is refused for want of room, then the self-test's request or load is cancelled, the run is recorded as yielded, and the model is released. +- Given a ready model that would need another evicted to load, when the tick fires, then it is not loaded, and it is measured once there is room. +- Given a request is in flight, a load is waiting, or a download is running, when the tick fires, then nothing is loaded and nothing runs. +- Given every ready model has been tested within the re-test window, when the tick fires, then nothing runs. +- Given a results line, when it is read back, then it carries the model, when, the outcome, whether the load was cold, the load time, and for each test its name, token counts, time to first token, total time and tokens per second, and no field carries prompt or answer text. +- Given the switch is turned off while a run is in progress, when the save applies, then the run is cancelled and the model released. +- Given the results file reaches its size cap, when the next line is written, then the file is started again and the newest line is kept. +- Given the switch, when Settings is rendered, then the panel offers it and posts it under the key config.json reads. +- Given the docs, when the feature ships, then a how-to page says how to switch it on and what it costs, and a reference page names every field of a results line and where the file is. + +## Open Questions + +- Resolved 2026-09-10 at planning (delegated to the agent by the maintainer): the standard set is load, pp512, tg128 and tg128 at the decode concurrency; the names follow llama-bench so the figures can be compared with published ones. +- Resolved 2026-09-10 at planning: results go to a single bounded JSON Lines file under the account's data directory rather than into the statistics store, because the store's schema is per-request and its writer is in another lane; folding it in is a follow-up. +- Resolved 2026-09-10 at planning: no panel view of the results in this intent; the file is the surface, the panel has the switch. A results view is a follow-up. +- Deferred: the re-test cadence is fixed at a day; whether it should be a setting is decided when someone wants a different one. + +## Audit Notes + + +Fidelity review OWED (receipt rcp-750863424b6e). + +## Grounds + +- pursued: we expect a per-model figure measured on the operator's own Mac, taken automatically, to replace the one-evening hand-run campaign as the source the settings are sized from; shown wrong if after a cycle the figures separate no setting choice, or if a serving Mac is never idle long enough for a cycle to complete. Planning delegated to the agent by the maintainer on 2026-09-10 as an experiment. diff --git a/.abcd/development/research/notes/2026-09-06-decomposition-calibration.md b/.abcd/development/research/notes/2026-09-06-decomposition-calibration.md index 9388740c..e8d389be 100644 --- a/.abcd/development/research/notes/2026-09-06-decomposition-calibration.md +++ b/.abcd/development/research/notes/2026-09-06-decomposition-calibration.md @@ -348,3 +348,26 @@ asset fetched from a CDN; the human did not confirm a reversal, so none is recorded and the question is held at the decision. Verdict adopted: SPLIT. Grade: routing survived as proposed. + +## 2026-09-10 — idle-time model self-test (itd-2609100457007827) + +Proposal: an autonomous state-of-the-art self-test and (later) +self-optimisation environment: when activated it loads and unloads the +available models, runs standard tests, captures all telemetry, and feeds the +results into a self-optimisation setting. + +| Part | Type | Home | +| --- | --- | --- | +| Opt-in self-test: load each model when idle, run a standard set, unload, record | capability | this intent | +| Self-optimisation from the results | capability, held | `duplicates` itd-2609091712142715, not filed twice | +| What the self-test may record, and which opt-in gates it | ADR consequence | adr-2609061503319212 already scopes it; a scope condition on the draft | +| Which tests are standard, which use cases | open questions | the draft, settled at planning | +| Where the switch lives, how runs are recorded | mechanism | the spec | + +Typed links: `refines` itd-2609091301112705 (the probe is one test of this +harness; the human chose refines over supersedes); `builds_on` the 2026-09-06 +model-bench campaign; `duplicates` itd-2609091712142715 for the second half. +Flagged, not classified: "captures all telemetry" against the ADR's no-prompt +rule — taken as a scope condition, not a reversal. +Verdict adopted: SPLIT. Grade: routing survived the human's confirmation +unchanged; the interview itself was delegated to the agent by the maintainer. diff --git a/.abcd/development/specs/closed/spc-2609091737253961-gropius-shows-what-its-models-actually-do-so-the-settings-th.md b/.abcd/development/specs/closed/spc-2609091737253961-gropius-shows-what-its-models-actually-do-so-the-settings-th.md new file mode 100644 index 00000000..7d575394 --- /dev/null +++ b/.abcd/development/specs/closed/spc-2609091737253961-gropius-shows-what-its-models-actually-do-so-the-settings-th.md @@ -0,0 +1,150 @@ +--- +id: spc-2609091737253961 +slug: gropius-shows-what-its-models-actually-do-so-the-settings-th +intent: itd-2609091712141073 +origin: researcher-authored +production_mode: dictated-and-formatted +--- +# Usage measurement: what each request did, against the settings in force + +## Summary + +The request statistics record gains the facts that decide a served window and +a concurrency — the prompt's size against the window the model declares and +the window it serves, how many requests were in flight when it was admitted, +which sampling values were in force and whether the client overrode them, and +the model server's memory footprint at completion — and the load event gains +the sampling values the server was launched with. A periodic sampler reads +each model server's footprint. The Statistics tab shows, per model, the +prompt-size distribution against both windows, the override rate and the +footprint over time. Nothing in it is a prompt or an answer, nothing is +exported, and the statistics reference page names every new field. + +Every fact already passes through the gateway or the pool where recording it +costs one lookup; that is the intent's mechanism claim, and this design adds +no new source of truth for any of them. + +## Scope + +In scope: new fields on `stats.Record` and `stats.Event`; a footprint sampler +in `internal/runtime` (one reader of a process's resident memory, per launched +server, at a fixed cadence, reported through the existing `Observer`); the +gateway's `observation` filling the new request fields; the dashboard +aggregates in `internal/stats` and their rendering on the Statistics tab; +`docs/statistics-store-reference.md` and `docs/statistics-explained.md`; the +re-measured `ApproxRecordBytes`; the changelog entry. + +Out of scope, each for its reason: an export button (resolved at interview — +the aggregates-only rule of the statistics endpoint stands, and the reference +page says where the files are); per-request memory attribution (the footprint +is a periodic per-model sample joined by time, and the record says so by its +name); storing any client-supplied sampling number (a per-parameter override +flag is the one field derived from a client body, argued on the docs page); +recording for the self-test's or the probe's own requests beyond what the +pool's load and removal events already carry (they go to the model server +directly and land in no request record). + +## Approach + +### The record + +`stats.Record` gains: + +| Field | Type | Where it comes from | +| --- | --- | --- | +| `declared_context` | int64 | the registry's `ContextLength` for the resolved model, at admission | +| `served_context` | int64 | `config.ServedContext(model, declared)` at admission — the figure the refusal was judged against | +| `estimated_prompt_tokens` | int | the gateway's own estimate (`estimatedTokens(bodyBytes)`), written for every request including one refused for size, so a refusal still carries how big the prompt was | +| `requested_tokens` | int | the figure the served-window check judged: the estimate plus the answer asked for | +| `in_flight` | int | the model's in-flight count at admission, reported by the pool on `AcquireStats` as the request takes its slot | +| `overrides` | []string | the names of the sampling parameters the client set in its body, from the same parse the gateway already does for `max_tokens`; the values are never stored | +| `footprint_bytes` | int64 | the model server's latest sampled footprint at completion, named as a sample, zero when the sampler has none yet | + +`stats.Event` gains, on a load: + +| Field | Type | Where it comes from | +| --- | --- | --- | +| `sampling` | the launch flags as `config.Sampling` | the `Spec` the launcher was handed (`SamplingFor` at launch) | + +And a third event kind, `footprint`, carrying `model`, `at` and `bytes`: the +sampler's readings, so the footprint over time is in the store as a series +rather than only as the value stitched onto a request. It is a fifth record +kind, ratified by adr-2609121450000000, which supersedes +adr-2609090716413337. + +The bounds every persisted numeric takes apply: a window beyond +`MaxContextLength`, a negative count, an override name not in the closed set +of sampling parameter names, or a footprint beyond the Mac's memory is +dropped at the recorder, never written. + +### The sampler + +`internal/runtime` gains a footprint reader: the physical footprint as +Activity Monitor and `top` report it — what a unified-memory Mac actually +spends, and what the 2026-09-06 campaign sampled — read through `top -l 1` +on the server's pid with a three-second deadline, because no unprivileged +pure-Go reader of that figure exists (`proc_pid_rusage` needs cgo, and `ps` +reports resident size, a different and smaller figure). The reader is an +optional `Footprinter` interface on the launcher's process, so a process that +cannot report one is simply never sampled. The pool samples every ready +server every `FootprintInterval` (thirty seconds) off its own lock, reports +each reading through an optional `FootprintObserver`, and holds the latest +per entry for `Pool.Footprint`, which the gateway stitches onto a completing +request. The sampler is the last thing the pool's close waits for, after the +servers have been told to stop. The self-test's memory guard and the probe's +may read the same latest figure later; nothing here depends on that. + +### The gateway + +The served-window check becomes `judgeServedContext`, which returns the +figures it judged — the two windows, the prompt's estimate, and the judged +figure (estimate plus the answer asked for) — so the record, the refusal and +the dashboard's bands rest on one number; `observation.judged` records +them, and `overrides(names []string)` the parameter names from the existing +body parse. The in-flight count at admission comes from the pool itself, on +`AcquireStats.InFlight`, taken at the moment the request takes its slot, +rather than from a residency snapshot before the request is admitted. +`footprint(bytes int64)` at completion takes the pool's latest sample. Every +one of these runs only while recording is on: the path with the switch off +is the path it was. No new parse of the body: only the presence of each +sampling key is read. + +### The dashboard + +`internal/stats` gains three aggregates over the range the existing views use: +per model, the distribution of `requested_tokens` — the figure the refusal +is judged on — in bands of the served window, beside the declared one, with +how often the served-window refusal bit; per model, the override +rate per parameter; per model, the footprint series from the `footprint` +events, downsampled to the range. The Statistics tab draws them under the +existing historical views, on the existing range control, with the page's +existing "what the views cannot show" section extended: a footprint is a +sample of the process, not a cost attributable to a request. + +### Three surfaces + +Go carries the whole of it. `config.json` gains nothing: there is no new +setting, the statistics switch already gates all of it. The panel gains the +three views. The sync obligation is the existing statistics docs test +(`internal/archtest/statistics_docs_test.go`), which holds the reference page +to `RecordFields()` and so fails the build the moment a field ships unnamed. + +## How each acceptance criterion is tested + +| Criterion | Test | +| --- | --- | +| A completing request carries declared, served and in-flight at admission | `internal/gateway`: `TestARecordCarriesTheWindowsAndTheInFlightAtAdmission` | +| A request refused for size still carries the estimate | `internal/gateway`: `TestARefusedRequestStillCarriesItsEstimatedSize` | +| A client-set parameter is marked overridden, no client number stored | `internal/gateway`: `TestOverridesAreNamedAndNeverValued` (the record's JSON carries no number the client sent) | +| A load event carries the launch sampling values | `internal/runtime` + `internal/app`: `TestALoadEventCarriesTheLaunchSampling` | +| Footprint sampled periodically; a record carries the footprint at completion, named as such | `internal/runtime`: `TestTheSamplerReportsEachServersFootprint` (fake launcher with a fake footprint); `internal/gateway`: `TestARecordCarriesTheLatestFootprintSample` | +| The dashboard shows prompt-size distribution against both windows, the override rate, the footprint over time | `internal/stats`: `TestPromptSizesAreBucketedAgainstBothWindows`, `TestOverrideRatesPerParameter`, `TestFootprintSeriesIsDownsampledToTheRange`; `internal/ui`: the view functions as pure functions | +| A new field is named on the reference page, the record-size constant re-measured, the docs say where the files are | `internal/archtest/statistics_docs_test.go` (existing, extended by the field list); `internal/stats/store_test.go`'s `ApproxRecordBytes` measurement | + +## Grounds + +- pursued: fields on the existing records plus a periodic footprint sample + show which settings matter, because each fact already passes through the + gateway or the pool; shown wrong if deciding a window or a concurrency turns + out to need per-request memory attribution the sample cannot give — the + intent's own mechanism claim, carried unchanged. diff --git a/.abcd/development/specs/closed/spc-2609100450257929-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md b/.abcd/development/specs/closed/spc-2609100450257929-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md new file mode 100644 index 00000000..9ed43049 --- /dev/null +++ b/.abcd/development/specs/closed/spc-2609100450257929-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md @@ -0,0 +1,90 @@ +--- +id: spc-2609100450257929 +slug: alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t +intent: itd-2609091903463596 +origin: researcher-authored +production_mode: dictated-and-formatted +--- +# The resources roll-up at the head of the Models tab + +## Summary + +One block at the head of the Models tab that adds up what the panel already +knows: how many models are downloaded and how many loaded, how much disk they +take and how much the volume has left, the memory budget against what is +resident and what is still exiting, and on each card the window the model +declares and the window it is set to serve. No new measurement: every figure +but the volume's free space is already on the state snapshot, and free space +has one reader, `capability.Assess`, which the snapshot's machine block +already carries. Placement was decided at the interview: a block, not a +seventh tab. + +## Scope + +In scope: the roll-up block (markup in `internal/ui/static/index.html`, a pure +`resourcesSummary(state)` in `app.js` the tests hold, drawn on every state +event); the declared and served windows on each model card, resolved the +fold-aware way the panel already resolves a served window (`servedContext`); +the one addition the snapshot needs, if any (see Approach); a section in +`docs/models-list.md`'s companion page for the panel, or in +`docs/memory-budget.md`, saying what the block shows; the changelog entry. + +Out of scope: any figure that needs a directory walk on the render path (the +per-model sizes are recorded at download and rescan and are read from the +registry entry); live usage figures, which are the measurement intent's; a +new endpoint — the block reads the snapshot the panel already polls. + +## Approach + +**The figures and where each comes from, all already on the snapshot:** + +| Figure | Source on the snapshot | +| --- | --- | +| Models downloaded | `models` with `state == "ready"` | +| Models loaded | `resident` entries in state `loaded` (loading counted separately, as "loading") | +| Disk the models take | sum of `models[].bytes` over ready models — recorded sizes, no walk | +| Free space on the models volume | `machine.free_disk`, from `capability.Assess`'s one reader | +| Memory budget | `machine.budget` | +| Resident, charged | `machine.resident_bytes`, which already includes the exiting servers | +| Still exiting | `machine.exiting_bytes`, named as such beside the resident figure | +| Stuck servers | `machine.stuck_servers`, shown only when non-zero | +| Per card: declared window | `models[].context_length` | +| Per card: served window | `servedContext(config, repo_id, context_length)`, the panel's existing fold-aware helper | + +So the change is entirely in the panel, plus its tests. If `machine.free_disk` +turns out not to ride the snapshot on every poll (it is read by `Assess` at +start today), the one server-side addition is to refresh it on the snapshot at +the snapshot's cadence — one call to the existing reader, no second one. + +**The block** is a `figures` paragraph and a short table at the head of +`#tab-models`, above the cards: "3 models downloaded, 1 loaded · 41.2 GB on +disk, 210 GB free · memory: 52.0 GB of 96 GB budget resident, of which 4.1 GB +still exiting". A model without a declared window shows no window on its card, +never zero. A budget the Mac cannot measure shows no share. + +**The cards** gain one line: "context: 131,072 declared · 65,536 served" when +the served window is below the declared one, "context: 131,072" when they are +equal, and nothing when the model declares none. (The context probe's line, +already on the card, sits beneath it.) + +**Loopback only.** The block is part of the panel, which the control plane +serves on this Mac alone; nothing here is served to the LAN. + +## How each acceptance criterion is tested + +| Criterion | Test | +| --- | --- | +| Three ready models, one resident: 3 downloaded, 1 loaded | `internal/ui`: `TestTheRollUpCountsDownloadedAndLoaded` (pure function over a snapshot) | +| Disk total is the sum of recorded sizes, no walk | the same, and `internal/archtest` or `internal/ui` asserting `resourcesSummary` reads `bytes` and never calls an endpoint | +| Free space comes from the capability package's one reader | `internal/gateway`: `TestTheSnapshotCarriesFreeDiskFromTheOneReader` (the machine block's field is `Assess`'s, and no second reader exists — a grep test in archtest for `statfs`/`Statfs` outside capability) | +| Resident memory includes exiting bytes and names them | `internal/ui`: `TestTheRollUpNamesTheExitingPart` | +| Served below declared: both shown, served resolved fold-aware | `internal/ui`: `TestTheCardShowsBothWindowsFoldAware` | +| No declared window: no window shown, never zero | `internal/ui`: `TestACardWithoutADeclaredWindowShowsNoWindow` | +| Off loopback: the block is unreachable | `internal/gateway`: the existing loopback-only test for the panel covers every panel asset; one assertion that the Models tab's markup is served by the same handler | + +## Grounds + +- pursued: no new measurement is needed, because every figure but free disk is + already on the snapshot and free disk has one reader; shown wrong if the + roll-up needs a directory walk or a second reader, as the intent's mechanism + claim says. diff --git a/.abcd/development/specs/closed/spc-2609100502029941-gropius-tests-its-own-models-while-nobody-is-using-it-alice.md b/.abcd/development/specs/closed/spc-2609100502029941-gropius-tests-its-own-models-while-nobody-is-using-it-alice.md new file mode 100644 index 00000000..a8ef6917 --- /dev/null +++ b/.abcd/development/specs/closed/spc-2609100502029941-gropius-tests-its-own-models-while-nobody-is-using-it-alice.md @@ -0,0 +1,123 @@ +--- +id: spc-2609100502029941 +slug: gropius-tests-its-own-models-while-nobody-is-using-it-alice +intent: itd-2609100457007827 +origin: researcher-authored +production_mode: dictated-and-formatted +--- +# The idle-time model self-test + +## Summary + +An opt-in loop that measures every downloaded model on the operator's own +Mac while nobody is using it, with the same short set of tests, and writes +one line of figures per run to a bounded file. It delivers +itd-2609100457007827: the switch on three surfaces, the loop, the standard +set, the results file, and the two docs pages. It does not deliver a panel +view of the results, a fold of the results into the statistics store, or any +setting derived from the figures; those are captured as follow-ups +(iss-2609100515475279, iss-2609100515478018) and the held draft +itd-2609091712142715. + +## Scope + +- `internal/config`: `Config.SelfTest` (`self_test`, bool, omitempty, off + by default, no repair and no refusal since a bool has no bad value) and + `Paths.SelfTest`, the results directory beside the statistics store under + the account's own directory. +- `internal/selftest`: the loop (`Runner`), the standard set and the request + builder (`request.go`, allow-listed by name in + `internal/archtest/prompt_content_test.go` on both sides of the content + boundary), and the bounded results file (`results.go`). +- `internal/app`: `App.SelfTest`, the `selfTestServer` adapter, and + `applySelfTest` called from `New`, `SetConfig` and `Close`. +- `internal/ui`: one checkbox on the Settings pane, posted as `self_test` + and filled from it, with its sync test. +- `docs/self-test.md` (how-to), `docs/self-test-reference.md` (reference), + a README bullet, a changelog entry; `internal/archtest` holds the reference + page to the record's fields. + +## Approach + +**The switch.** `self_test` follows the statistics switch's shape: a bool the +panel posts and the file carries; `SetConfig` applies it live by calling +`SelfTest.SetEnabled`, and `New` applies the saved value at construction. +Off cancels a run in progress and returns once the loop is gone; `Close` +does the same before the pool closes, so the pool's close never waits on a +release that is on its way. + +**Idleness, from outside the pool.** The runner never holds an app lock and +adds no mechanism to the pool (adr-2609091239058072). It reads +`Pool.Residency` (per-model in-flight and last-used), `Pool.Waiting` and +`App.Downloading` through the adapter on every tick. Quiet means: nothing in +flight, nothing waiting, nothing downloading, and every resident model's last +use older than five minutes. A run in progress polls the same view every +quarter of a second, subtracting its own one hold on the model under test; +any other in-flight request, any waiter, or a rise in the pool's count of +loads refused for want of room (`Residency.Refusals`, the one seam added to +the pool: with eviction grace off a refused client waits nowhere else +visible) cancels the run's context. The watcher runs from before `Acquire`, +so a cold load yields too, and the pool tears down an abandoned load. The +pool has no preemption, so this is the whole of the yielding: the self-test's +own request or load is cancelled, the run is written as `yielded`, and the +release follows within the poll interval plus the cancellation. A +pool-side preemptible hold, which would spare the refused client its one +503, is captured as a follow-up rather than built. + +**Loading through the pool, never by eviction.** `Acquire`, tagged with its +own source for the load-waiter queue, never a second launcher: the budget, +the pins, the served window and the KV charge are the pool's own. Before +asking, the runner checks `Server.Fits`, which the app answers from the +residency snapshot and `chargeOf`: a model that would need another evicted +is left due for a later tick, so a Mac whose working model stays warm all +day keeps it. The parallel test takes the decode concurrency's worth of +acquisitions on the resident model before it sends, so the pool's semaphore +bounds the batch and a client arriving on the model queues where the watcher +sees it. A model the self-test loaded is unloaded afterwards through +`Pool.Unload`, unless the run yielded, in which case the client's request +decides what stays; a model that was resident is left resident. + +**The set.** `pp512` (a fixed ~512-token English prompt, one token back), +`tg128` (one sentence, 128 tokens back) and `tg128xN` (the same, N at once, +N the decode concurrency, omitted at one), plus the load time from `Acquire`. +Requests go straight to the upstream the pool hands back — the model server +on loopback, with the exact `ModelArg` the pool reports — as the gateway does +after its own Acquire, so they land in no request statistic and need no API +key. Streaming with `include_usage` always written; the first chunk carrying +text is the first token, and the server's usage event gives the counts, or +the chunk count does when the server sends none, and the result says which. + +**The file.** `selftest/results.jsonl`, 0600 in a 0700 directory, opened +with `O_NOFOLLOW`, appended to until the next line would pass 4 MiB and then +truncated: a single bounded file, deliberately not a third rotating writer +(iss-2609091714393599). One `Run` per line; `Test` records inside it. A +failed run carries a class, never the error's text. On start the loop seeds +its last-tested map from the file so a restart does not begin the cycle +over; a model is measured again after a day, the stalest first. + +## How it satisfies the acceptance criteria + +| Criterion | Where it is held | +| --- | --- | +| Off: nothing loaded, no file | `TestOffLoadsNothingAndWritesNothing`, `TestTheSelfTestFollowsItsSwitch` (app) | +| On and idle: acquired through the pool, the set runs, one line appended | `TestAnIdleMacGetsEveryModelMeasuredAndLeftAsFound` | +| Unload what it loaded, leave what was there | the same, and `TestAModelThatWasResidentIsLeftResident` | +| A client request yields the run, releases the model, records `yielded` | `TestAClientRequestYieldsTheRun`, with a bound on how long the yield takes | +| Busy Mac: nothing runs | `TestNothingRunsWhileTheMacIsBusy` (four cases) | +| Every model fresh: nothing runs | the tail of `TestAnIdleMacGets…`, and `TestAResultStandsForADay…` | +| The line's fields, and no prompt or answer text | `TestAResultCarriesNoPromptAndNoAnswer`, `TestALoadThatFails…` (no error text), and the archtest docs test | +| Switch off mid-run cancels and releases | `TestSwitchingOffStopsARunAndReleasesTheModel` | +| The cap: file started again, newest kept | `TestTheResultsFileIsPrivateAndBounded` | +| The panel offers the switch under the file's key | `TestThePaneIsWiredToTheSelfTestSwitch` (ui) | +| The docs pages | `TestTheSelfTestReferenceNamesEveryField`, `TestTheSelfTestPagesAreLinkedAndSayWhatIsNeverWritten` (archtest) | + +## Not in scope, and why + +- A results view in the panel: the switch is the accessible surface this + change needs; the figures are a file the reference page names. Captured. +- Folding the results into the statistics store: its schema is per-request + and its writer is in another lane; captured, to move with the writer + unification. +- A line on the posture page saying the self-test is on: the page's lane is + another session's; captured as iss-2609100515484516. +- Any setting chosen from the figures: itd-2609091712142715, held. diff --git a/.abcd/development/specs/open/spc-2609112108142027-gropius-measures-a-newly-downloaded-model-s-servable-context.md b/.abcd/development/specs/closed/spc-2609112108142027-gropius-measures-a-newly-downloaded-model-s-servable-context.md similarity index 90% rename from .abcd/development/specs/open/spc-2609112108142027-gropius-measures-a-newly-downloaded-model-s-servable-context.md rename to .abcd/development/specs/closed/spc-2609112108142027-gropius-measures-a-newly-downloaded-model-s-servable-context.md index 13967b53..00be8214 100644 --- a/.abcd/development/specs/open/spc-2609112108142027-gropius-measures-a-newly-downloaded-model-s-servable-context.md +++ b/.abcd/development/specs/closed/spc-2609112108142027-gropius-measures-a-newly-downloaded-model-s-servable-context.md @@ -69,21 +69,32 @@ exported methods is not changing them. ### The package, and the seam it holds -`internal/contextprobe` owns the method, the scheduler and the result. It talks -to the rest of Gropius through one small interface it declares itself, which -`*runtime.Pool` already satisfies: +`internal/contextprobe` owns the method, the result and the "Measure now" +queue. It does not own a scheduler: the self-test's idle loop +(`internal/selftest`, itd-2609100457007827) landed first, and the probe is a +job of it — `selftest.Job` (Name, Due, Run) and `selftest.Session` (the run's +context, the hold count, whether a cancellation was a yield, the progress +words, the never-evict check) are the seam. So idleness, yielding, the +never-evict rule, the "what held it back" reason and the idle threshold are +decided once, for both jobs. The probe talks to the app through one small +interface it declares itself: ``` -type poolView interface { - Waiting() int // queue depth, as any caller is counted - Residency() runtime.Residency // per model: State, InFlight, LastUsed - Unload(repoID string) error // ErrBusy while a request is in flight +type Sources interface { + Candidates() []Candidate // ready models: declared, served, bytes, KV charge, measurement + Provenance(repoID string) registry.Provenance // runtime, budget, concurrency, served window in force + Available() int64 // what the budget has free + Unload(repoID string) error // runtime.ErrBusy while a request is in flight + Save(repoID string, m *registry.Measurement) error + MarkIncomplete(repoID string, on bool) error + Endpoint() (baseURL, apiKey string) // this Mac's own loopback endpoint } ``` -Three existing exported methods, nothing added to the pool. A fake satisfying -this interface is what the tests drive, which is why the package is testable -without a real Mac. +Nothing added to the pool. Fakes satisfying `Sources` and the loop's +`selftest.Server` are what the tests drive, with a fake gateway that answers +the way the real one does, which is why the package is testable without a +real Mac. ### The method, from the campaign @@ -122,9 +133,11 @@ break the probe. ### Idle, and yielding -**Idle** is all three of: `Waiting()` is zero, no resident model has a request -in flight, and every resident model's `LastUsed` is older than the probe's idle -threshold. A probe that cannot start says which of the three held it back. +**Idle** is the loop's: `Waiting()` is zero, nothing is downloading, no +resident model has a request in flight, and every resident model's `LastUsed` +is older than the idle threshold (`idle_threshold_sec`, shared with the +self-test, five minutes by default). A probe that cannot start says which held +it back, on the state snapshot and the model's card. **Yielding** is a poll of the same two readings on a short interval while a step runs. The probe holds exactly one request, on one model, so a second in-flight @@ -193,9 +206,10 @@ needs nothing from the package either way. - **Go** carries the whole of it: the scheduler, the switch, the per-model run, the progress snapshot and the result. -- **`config.json`** carries the switch and the idle threshold as ordinary - settings; a save that names neither leaves both alone, which is the wedge this - repository has built three times. +- **`config.json`** carries the switch (`context_probe`) and the idle + threshold (`idle_threshold_sec`) as ordinary settings; a save that names + neither leaves both alone, which is the wedge this repository has built three + times. - **The control panel** carries the switch, a "Measure now" per downloaded model, the progress of a run in words (which model, which step, the bounds so far, what stopped the last step), the result with its bound, and the "Use this diff --git a/.abcd/development/specs/open/spc-2609091737253961-gropius-shows-what-its-models-actually-do-so-the-settings-th.md b/.abcd/development/specs/open/spc-2609091737253961-gropius-shows-what-its-models-actually-do-so-the-settings-th.md deleted file mode 100644 index d474843a..00000000 --- a/.abcd/development/specs/open/spc-2609091737253961-gropius-shows-what-its-models-actually-do-so-the-settings-th.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -id: spc-2609091737253961 -slug: gropius-shows-what-its-models-actually-do-so-the-settings-th -intent: itd-2609091712141073 -origin: researcher-authored -production_mode: dictated-and-formatted ---- -# gropius-shows-what-its-models-actually-do-so-the-settings-th - -## Summary - -_Draft: describe what spc-2609091737253961 delivers for itd-2609091712141073 — scope, approach, and how it satisfies the intent's Acceptance Criteria. This spec is the design record the fidelity review audits against._ diff --git a/.abcd/development/specs/open/spc-2609100450257929-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md b/.abcd/development/specs/open/spc-2609100450257929-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md deleted file mode 100644 index c7fc0fc3..00000000 --- a/.abcd/development/specs/open/spc-2609100450257929-alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -id: spc-2609100450257929 -slug: alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t -intent: itd-2609091903463596 -origin: researcher-authored -production_mode: dictated-and-formatted ---- -# alice-sees-what-her-mac-is-spending-on-gropius-a-resources-t - -## Summary - -_Draft: describe what spc-2609100450257929 delivers for itd-2609091903463596 — scope, approach, and how it satisfies the intent's Acceptance Criteria. This spec is the design record the fidelity review audits against._ diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index 3edf5338..f0a9eaca 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -261,3 +261,11 @@ - 2026-09-12 — itd-2609081420471761, build, after review: the claim that "no environment variable can point the download" is CORRECTED rather than made true. `-q` suppresses `.curlrc` only, and curl still honours `https_proxy`/`ALL_PROXY` and `CURL_CA_BUNDLE`/`SSL_CERT_FILE` — both measured. Neutralising them was rejected on two grounds: the bootstrap honours the same variables, so the verb would fail on machines where the documented install works and a corporate proxy is the ordinary case rather than the attack; and it buys nothing against the adversary this package is written for, another account on this Mac, which cannot set this account's environment and which, if it could, could replace `~/.local/bin/gropius` outright. The claim is now about what Gropius reads, with the limit written down in the code and in the reference. The spec anticipated exactly this: "expect findings that change a claim rather than harden one." - 2026-09-12 — itd-2609081420471761, build: the branch and its worktree were deleted mid-review by something outside the session; the eleven commits survived as dangling objects and the branch was re-created at its tip from `766ec80`. Recorded because the recovery window is a `git gc` wide, and because a reviewer reporting "the branch under review no longer exists" is a finding about the working setup rather than about the code. - 2026-09-12 — iss-2609120417422598, fix: `install.sh` takes the update verb's three moves (d7e31ab) rather than a narrower patch — the checksums file is narrowed to the archive's own line before `/usr/bin/shasum` sees it, `--ignore-missing` goes with the narrowing, a line naming a PATH is refused, and the pass is read line by line. The narrowing is written with bash's own `read` fed by a redirect, NOT with `grep` or `awk`: every command in this script is named by absolute path, a new tool is a new thing to pin, and a redirect leaves the standard input `curl … | bash` feeds the script from untouched. The scope failure gets its own refusal ("the checksums … carry no line for ") instead of the mismatch wording, because a checksums file that is intact and answers about something else sends a user to look at their network otherwise. `TestTheInstallerVerifiesBeforeItHandsOver` anchored on `--ignore-missing`; it now anchors on the invocation and refuses the flag returning outside a comment, so the guard pins the verification rather than the spelling it once had. +- 2026-09-10 — itd-2609100457007827, the idle-time model self-test, is planned and built by the agent without the planning interview, on the maintainer's explicit instruction ("autonomously plan and implement this intent; make decisions autonomously and record them; this is an experiment"). The maintainer's adoption of the decomposition (SPLIT: one intent for the self-test; self-optimisation stays the held draft itd-2609091712142715; the probe draft itd-2609091301112705 is refined, not superseded) was the one question put to them. Decisions taken by the agent, each with what would show it wrong: (1) **the standard set is llama-bench's pair plus a load and a concurrency figure** — load, pp512, tg128, tg128 at the decode concurrency — because those are the four measurements the 2026-09-06 campaign decided the budget and the window from, and llama-bench's names let the figures be compared with published ones; wrong if a setting the figures were meant to decide needs a measurement not in the set. (2) **Results go to one bounded JSON Lines file, truncate-at-cap, no rotation**, `selftest/results.jsonl` under the account directory at 0600 — not a third rotating writer (iss-2609091714393599 unifies the two there are) and not the statistics store, whose schema is per-request and whose writer is in another lane; wrong if a reader needs more history than the cap holds, which is thousands of runs. (3) **Idle is read from outside the pool, not built into it**: no in-flight request other than the self-test's own hold, nobody waiting, no download, last use older than five minutes; yielding is the self-test cancelling its own request on a 250 ms poll, never the pool preempting; wrong if a client's request measurably waits on a self-test request, which would call for preemption in the pool. (4) **The self-test talks to the model server directly at the upstream the pool hands back**, as the gateway does after Acquire, so its runs land in no request statistic and need no API key; wrong if a figure only the gateway can see (a served-window refusal, a merge) turns out to be part of what the set should measure. (5) **The switch ships on all three surfaces in one change** — `self_test` in config.json, a Settings checkbox with its sync test, the Go loop — by agreement with gropiusllm-07, whose lane the panel is in. (6) One model per tick, re-tested after a day, the order being least recently tested first. +- 2026-09-10 — The self-test after its two adversarial reviews (ruthless, security), four changes. (1) **It never evicts.** On a default install the idle timeout is zero and the working model stays warm for ever; a self-test that evicted it to measure another would leave the Mac cold once a day per model. `Server.Fits`, answered in app from the residency snapshot and `chargeOf`, gates the load; a model that does not fit stays due and is not charged a day. Wrong if a Mac with one model that fills the budget is what the operator wanted measured against others — that Mac gets only the resident model measured, and the docs say so. (2) **A refused client yields the run.** With eviction grace off (the default, and mandatory without an API key) a client's load that finds no room is refused at once and waits nowhere visible; one seam in the pool, `Residency.Refusals` (curable refusals only, never the never-fits path, per gropiusllm-07), counts them and the watcher yields when the count moves, so the retry finds room. The client's first request is still a 503; the pool-side preemptible hold that would spare it is a follow-up in gropiusllm-07's lane, not built. (3) **The parallel test holds its places in the pool** — N acquisitions, not one plus N-1 raw requests — so the semaphore bounds what the model server decodes and a client arriving on the model queues visibly. (4) **The results writer refuses a planted location**: a link, a foreign or group-writable directory, or anything but a regular file, and every open is non-blocking, because the writer runs on the settings path where a hang is a hang of every save. Also recorded: the watcher now runs from before the load, since a cold load was the longest unwatched phase; a stopped run does not use the model's day; and the self-test's loads and unloads do appear in the statistics store's load and eviction views (decision (4) of the line above was too broad — request records only), and the docs say so. +- 2026-09-11 — The self-test's results view (iss-2609100515475279) is one loopback read endpoint, `GET /api/selftest`, answering the switch read live from the loop, the latest run per model (folded id, newest wins) and the fifty most recent runs newest first, and a block at the head of the Statistics tab drawn from it on the tab's own two-second tick rather than a timer of its own. The endpoint carries no path: the file's location is the reference page's to state. The block shows whether or not request statistics are on, because the two switches are separate and a reader with the self-test on and statistics off would otherwise see a tab that says nothing is recorded. What would show this wrong: an operator opening the file for a figure the block omits (the per-test token counts and totals are only in the file), or the tab's tick proving too coarse for a run that lands between polls — neither matters at one run a model a day. After the route's security review: the parsed file is cached on the control plane keyed on the file's size and modification time, so the tab's tick and any local process asking in a loop cost one parse per change rather than one per ask; and the error path is armed by a test that plants a directory where the file should be and reads a 500 that names no path. +- 2026-09-12 — The context-window probe (itd-2609091301112705, spc-2609112108142027) is built by the agent on the maintainer's lane assignment, as a second job of the self-test's one idle loop rather than a scheduler of its own: `selftest.Job` and `selftest.Session` are the seam, the loop owns idleness, yielding, the never-evict check and the "what held it back" reason for both jobs, and the idle threshold is one setting, `idle_threshold_sec`, for both. Decisions with what would show them wrong: (1) **the probe is a client of the gateway**, sending to this Mac's own loopback endpoint with the API key, so the gateway's own answers classify a step — 504 is the prefill deadline, 400 and 413 are the served-window check and the body cap (both the gateway's, published as `served_window`), any other failure is the model's; wrong if a 400 is ever the gateway refusing a well-formed probe request for another reason, which would misfile a gateway bound as the served window. (2) **The memory guard is the budget's own arithmetic**, not a sample: the model's flat charge plus the KV charge for one sequence of the step's size, against what the budget has free less a margin of three sixteenths of the Mac's memory (the campaign's 24 GB on 128 GB); wrong if a step the guard allowed pushes the process past what the Mac has, which would call for the sampling the spec deferred. (3) **A yield unloads; an unload refused discards the step**, exactly as the spec says; a stop marks the model incomplete and writes nothing; a floor that fails marks incomplete and backs off a day. (4) **Adoption is a settings save**: `AdoptMeasurement` writes `served_context` through `SetConfig`, so every check a save makes is made, and the measurement then reads as stale for the served window — expected, and said so on the card. (5) **Staleness is refreshed at start and after every save**, per model, with the provenance callback handed the model rather than looking it up (the first draft deadlocked on the registry's own lock). (6) The manual comparison against the campaign script on at least one real model has NOT been run in this session; the code publishes a figure the moment a run completes, so the obligation the spec states — check once by hand before trusting a published figure — is the maintainer's before switching the probe on, and is recorded in the handover. +- 2026-09-12 — The probe after its two adversarial reviews (ruthless, security), both of which reproduced the same lock inversion: `RefreshStaleness` held the registry's lock across a callback that took the pool's, while the pool already takes the registry's under its own (Resolve under p.mu) — a save during any load would have wedged the process. Fixed by judging outside the lock (snapshot under RLock, judge, write back where the measurement pointer is unchanged), with the ADR's rule now stated on the function. Six further changes: (1) an unload straight after a cancel finds the probe's own request still in flight, so `unloadWaiting` retries for a bounded window before an ErrBusy is read as a client's; (2) the gateway's answers are classified — 504 the deadline, 400 the served window, 503 held for a later tick, 500/502 the model's, anything else abandons the run — and the probe's own timer, now derived from the body with the gateway's own formula, files a step as the deadline's, never the model's; (3) a non-parking job's waiters are always a client's (`Job.Parks`); (4) the sweep resumes from the last verified size and the bound is seeded as the served window, so "the model's limit" is written only from a step the model refused; (5) adopting a measurement does not stale it (a served window equal to the measured window is not a move); (6) the sweep is capped at `MaxProbeWindow` (a million tokens) and the calibration clamped to 1–16 characters a token, so a planted declared window cannot make the probe allocate gigabytes. Also: `gateway.Control.handleAdopt` is a SECOND taker of `settingsMu`, and `App.AdoptMeasurement` a second caller of `SetConfig` — adr-2609091239058072 says the settings handler is the only taker; the order it fixes is respected, and this line is the record until an ADR supersedes it. What would show these wrong: a settings save that still wedges under load (a regression test runs a save against a load); a published `model` bound on a model that never refused a step. +- 2026-09-12 — The resources view (itd-2609091903463596, spc-2609100450257929) is built as the spec says: a `resourcesSummary` pure function over the state snapshot, drawn at the head of the Models tab, and the card's context label carrying the served window beside the declared one when it is set below it. The one server-side addition is `Machine.FreeDisk` on the snapshot, read through `capability.Assess` at the snapshot's cadence — the same reader the search tab uses — and an architecture test now holds free disk to one reader. What would show this wrong: a roll-up figure that needs a walk or a second reader, which the intent's mechanism claim named as its falsifier. +- 2026-09-12 — Usage measurement (itd-2609091712141073, spc-2609091737253961) is built by the agent on the lane assignment. Decisions with what would show them wrong: (1) **the footprint is a fifth record kind**, `footprint`, ratified by adr-2609121450000000 superseding adr-2609090716413337, because a reading of a process on a clock is not a fact about any request; wrong if the series turns out to be what nobody reads and the value stitched onto a request is all anyone sizes from. (2) **The reader is `ps -o rss=` on the server's pid**, from the runtime's exec process through an optional `Footprinter` interface, every thirty seconds per ready server, reported through an optional `FootprintObserver` so no observer is made to care; wrong if the process listing is ever what makes the sampler measurable in a request's latency. (3) **Overrides are names from a closed set**, read from the same body parse the gateway already does, with `max_completion_tokens` folded onto `max_tokens`; wrong if a reader needs the value to decide a default, which adr-2609061503319212 rules out anyway. (4) **The statistics package keeps its own copy of the window bound**, `stats.MaxContext`, because it imports nothing of ours, and an architecture test holds it to `config.MaxContextLength`. (5) The record-size constant moves from 230 to 380 bytes and the docs' arithmetic from three months to seven weeks of ten thousand requests a day at the default cap — a measured figure, re-measured. (6) The launch sampling reaches the load event through `PoolObserver.LoadStarted`'s new second argument and a map in the statistics package's wire shape, converted in one place in app. +- 2026-09-12 — Usage measurement after its two adversarial reviews. Six changes: (1) **the refusal, the record and the bands rest on one number** — `judgeServedContext` returns what the served-window check judged (estimate plus the answer asked for), recorded as `requested_tokens`; the bands were bucketing on the prompt estimate alone, so a request tipped over by its `max_tokens` was refused but shown under the window. (2) **The in-flight count is the pool's, at the slot** (`AcquireStats.InFlight`), not a residency snapshot taken before admission that also cost a lock and a sort on every request. (3) **The recording-off path is untouched again**: the new work runs behind `obs.recording()`. (4) **The reader is top's physical footprint, with a three-second deadline**, not `ps`'s resident size — the campaign sampled top's figure, and `ps -o rss` was measured here at 2640 K against top's 1728 K for one process; the pool's close waits for the sampler last, after the servers are told to stop, so a stalled listing cannot hold the memory. (5) **Launch sampling rides the finish report**, not a map stitched across two reports that may arrive out of order. (6) The series is drawn (a sparkline per model), the counters' unread footprint figure is dropped, the read path is bounded, the stamps use the bucketing's arithmetic, and the docs say what the footprint lines cost the request views' reach. What would show these wrong: a request the gateway refused that the bands show under the window; a shutdown that still waits on a listing. diff --git a/.abcd/work/issues/open/iss-2609100515478018-the-self-test-writes-its-own-bounded-json-lines-file-interna.md b/.abcd/work/issues/open/iss-2609100515478018-the-self-test-writes-its-own-bounded-json-lines-file-interna.md new file mode 100644 index 00000000..c251a9eb --- /dev/null +++ b/.abcd/work/issues/open/iss-2609100515478018-the-self-test-writes-its-own-bounded-json-lines-file-interna.md @@ -0,0 +1,14 @@ +--- +schema_version: 1 +id: "iss-2609100515478018" +slug: "the-self-test-writes-its-own-bounded-json-lines-file-interna" +severity: "minor" +category: "observation" +source: "user-observation" +found_during: "itd-2609100457007827 build" +origin: researcher-authored +production_mode: hand-written +found_at: "internal/selftest/results.go" +--- + +The self-test writes its own bounded JSON Lines file (internal/selftest/results.go) beside the statistics store instead of into it, because the store's schema is per-request and its writer is in another lane. When iss-2609091714393599 unifies the size-bounded writers, the self-test's file should move with them, and a run should become a record kind the statistics reference page names. diff --git a/.abcd/work/issues/open/iss-2609100526194406-the-self-test-s-hold-on-a-model-is-not-preemptible-with-evic.md b/.abcd/work/issues/open/iss-2609100526194406-the-self-test-s-hold-on-a-model-is-not-preemptible-with-evic.md new file mode 100644 index 00000000..37a366b1 --- /dev/null +++ b/.abcd/work/issues/open/iss-2609100526194406-the-self-test-s-hold-on-a-model-is-not-preemptible-with-evic.md @@ -0,0 +1,14 @@ +--- +schema_version: 1 +id: "iss-2609100526194406" +slug: "the-self-test-s-hold-on-a-model-is-not-preemptible-with-evic" +severity: "minor" +category: "observation" +source: "user-observation" +found_during: "itd-2609100457007827 security review" +origin: researcher-authored +production_mode: hand-written +found_at: "internal/runtime/pool.go" +--- + +The self-test's hold on a model is not preemptible: with eviction grace off, a client whose load needs the memory the self-test holds is refused once (503) before the self-test sees the refusal count move and yields, so the client's retry finds room. The correct fix is pool-side: a soft, source-tagged hold that the eviction plan may take and that cancels the holder's context, so the client's Acquire evicts the self-test's model instead of being refused. Found by the security review of itd-2609100457007827; the counter (Residency.Refusals) is the interim. diff --git a/.abcd/work/issues/resolved/iss-2609100515475279-the-self-test-s-results-have-no-panel-view-the-switch-is-on.md b/.abcd/work/issues/resolved/iss-2609100515475279-the-self-test-s-results-have-no-panel-view-the-switch-is-on.md new file mode 100644 index 00000000..e0cdddcf --- /dev/null +++ b/.abcd/work/issues/resolved/iss-2609100515475279-the-self-test-s-results-have-no-panel-view-the-switch-is-on.md @@ -0,0 +1,22 @@ +--- +schema_version: 1 +id: "iss-2609100515475279" +slug: "the-self-test-s-results-have-no-panel-view-the-switch-is-on" +severity: "minor" +category: "observation" +source: "user-observation" +found_during: "itd-2609100457007827 build" +origin: researcher-authored +production_mode: hand-written +found_at: "internal/ui/static/app.js" +resolution: "GET /api/selftest on the control plane (latest run per model, fifty most recent, the switch read live) and a Self-test block at the head of the Statistics tab, fetched on the tab's own tick; docs/self-test.md says so" +impact: additive +resolved_by: + intent: "itd-2609100457007827" +--- + +The self-test's results have no panel view: the switch is on the Settings pane but the figures are only in selftest/results.jsonl, so the accessible surface shows the operator how to start measuring and nothing of what was measured. A results view on the Statistics or Models tab, reading the file through a loopback endpoint, closes the gap the three-surfaces rule names. + +## Grounds + +- pursued: we expect the latest run per model, shown beside the request statistics, to be enough for an operator to size a served window and a concurrency without opening the file; shown wrong if readers reach for the file for a figure the block does not carry diff --git a/.abcd/work/issues/resolved/iss-2609100515484516-the-posture-page-says-what-is-on-and-the-self-test-is-a-thin.md b/.abcd/work/issues/resolved/iss-2609100515484516-the-posture-page-says-what-is-on-and-the-self-test-is-a-thin.md new file mode 100644 index 00000000..4b0cc237 --- /dev/null +++ b/.abcd/work/issues/resolved/iss-2609100515484516-the-posture-page-says-what-is-on-and-the-self-test-is-a-thin.md @@ -0,0 +1,22 @@ +--- +schema_version: 1 +id: "iss-2609100515484516" +slug: "the-posture-page-says-what-is-on-and-the-self-test-is-a-thin" +severity: "nitpick" +category: "observation" +source: "user-observation" +found_during: "itd-2609100457007827 build" +origin: researcher-authored +production_mode: hand-written +found_at: "internal/ui/static/app.js" +resolution: "the self-test PR adds the Self-test line to postureLines, its table test, and the posture-reference row" +impact: additive +resolved_by: + intent: "itd-2609100457007827" +--- + +The posture page says what is on, and the self-test is a thing that can be on: a loop that loads models on its own while the Mac is idle. The page has no line for it. One line reading config.self_test, with the reads list the page's tests hold it to, would make the posture page complete again. + +## Grounds + +- pursued: we expect the posture page to stay a complete account of what is on, so a loop that loads models on its own is stated there the day it ships; shown wrong if a reader of the page is still surprised by a model loading unasked diff --git a/CHANGELOG.md b/CHANGELOG.md index d42d57a6..09f7ae39 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,61 @@ GitHub release notes. ## [Unreleased] +### Added + +- **The usage dashboard shows what the models actually do, so the settings + that matter can be seen before they are chosen.** Every request record now + carries the two windows it was judged against, Gropius's own estimate of + the prompt's size (for a refused request too), how many requests the model + already had, which sampling parameters the client set — the names, never + the values — and the model server's memory as last sampled; every load + carries the sampling the server was launched with; and a new record kind, + `footprint`, is a reading of each running server's memory every thirty + seconds. The Statistics tab gains three views: prompts against the served + window, sampling overrides per parameter, and memory over time. Nothing in + any of it is a prompt or an answer, nothing is exported, and the + [reference page](docs/statistics-store-reference.md) names every field. + +- **The Models tab adds up what your Mac is spending on Gropius.** A line at + its head says how many models are downloaded and loaded, how much disk they + take and how much the volume has left, and the memory budget against what is + resident, naming the part still exiting. Each card says the window the model + declares and, when you have set one below it, the window it is served at. + Nothing new is measured: the figures are the ones the panel already had, + and free disk now rides the state snapshot from the one reader the search + tab uses ([the memory budget](docs/memory-budget.md#what-the-models-tab-adds-up)). + +- **The context-window probe: Gropius measures the largest prompt each + model on this Mac will actually take.** Off until you turn it on — + **Settings → Context probe**, `context_probe` in `config.json` — or per + model with **Measure now** on its card. While the Mac is idle, Gropius + sends prompts of growing length through its own OpenAI endpoint, bisects to + the largest the server accepts, and records the window on the model's entry + with what stopped the next step: the model itself, or one of Gropius's own + limits — the prefill deadline, the served window, the memory guard — in + which case the figure is a floor and says so. It is published on the models + list as `measured_context` and `measured_bound`, changes no charge and + refuses no request until you press **Use this window**, and is marked stale + when the runtime, the budget, the concurrency or the served window changes. + A probe costs about forty minutes of GPU per model, never evicts another + model, and stands down the moment anyone sends a request. The idle threshold + the probe and the self-test share is now a setting, `idle_threshold_sec` + ([how to](docs/context-probe.md)). + +- **The self-test: Gropius measures its own models while nobody is using the + Mac.** Off until you turn it on — **Settings → Self-test**, `self_test` in + `config.json`. While it is on, once the Mac has been idle for five minutes, + Gropius loads one model at a time through the same path a request takes — + and never by evicting another — runs llama-bench's pair and a concurrency figure against it — `pp512`, + `tg128`, `tg128xN` at the decode concurrency — plus the load time, writes + one line of figures to `selftest/results.jsonl` in this account's data + folder, and unloads what it loaded. A request from anyone cancels the run + at once and is recorded as such; a request refused room during a run is + served on its retry. Each model is measured once a day. The + file holds counts and timings only, never a prompt or an answer, and is + bounded at 4 MiB ([how to](docs/self-test.md), + [reference](docs/self-test-reference.md)). + ## [0.6.0] - 2026-09-12 ### Changed @@ -220,7 +275,6 @@ GitHub release notes. differently.** This is one less thing that has to hold for the queue to behave, not a stall anyone was hitting. - - **A client connecting over loopback is told which models are loaded, whether or not an API key is set.** `GET /v1/models` carries `state`, `in_flight`, `last_used` and `pinned`, and used to carry them only on an install that had @@ -239,7 +293,6 @@ GitHub release notes. points its own hostname at `127.0.0.1` is refused. An install with a key behaves as before. - - **Under a shared model cache, every account now keeps its own settings and its own model list.** They used to be one `config.json` and one `registry.json` beside the models, which worked for whichever account ran @@ -427,7 +480,6 @@ GitHub release notes. where the VPN stops. Linked from the README and from the getting-started guide. - - **The chat client finds servers instead of asking you to name one.** Settings now lists every Gropius server advertising itself on your network, with the name it publishes, whether it wants an API key, and how many models @@ -497,7 +549,6 @@ GitHub release notes. closed on a pin it cannot read; and a repository with no pin is unaffected. Contributors whose git identity already matches see no change. - - **A first launch of the chat client no longer dead-ends.** It opened on a documentation example's host name, which resolves for nobody, and the message box stays disabled until a server answers — so the first thing a new user met @@ -507,7 +558,6 @@ GitHub release notes. install order puts there; the empty chat names Settings and has a button that opens it; and the disabled message box says why it is disabled. - - **Saving settings no longer empties the bind address.** The bind control offers two addresses, and a browser's ` + Measure each model’s context window when this Mac is idle + +

Off unless you turn it on. While it is on, whenever this Mac has been idle for the + threshold below, Gropius takes one model with no current measurement, sends it prompts of + growing length through its own endpoint, bisects to the largest the server accepts, and + records the window beside the one the model declares. A measurement changes nothing + until you press Use this window on the model’s card. It costs about + forty minutes of GPU per model, never pushes another model out to make room, and stands + down the moment anyone sends a request. Each model’s card also has + Measure now, which runs one probe whatever this box says. The + context + probe page says what the figure means and what it costs.

+ +

How long the last request must be in the past before the Mac counts as idle, + for the context probe and the self-test alike. Blank is five minutes.

+ + +
+ Self-test + +

Off unless you turn it on. While it is on, whenever nothing has asked this Mac + for a model for five minutes and nothing is downloading, Gropius loads one of your + models, runs the same short set of tests against it — how long the load takes, + how fast it reads a long prompt, how fast it generates, and what that becomes when + several requests arrive at once — writes one line of figures to a file in your + Gropius data folder, and unloads the model again if it was not already loaded. Each + model is measured once a day, and never at the cost of another: a model that would need + one pushed out to make room waits for a time when there is room. A request from anyone + ends the run at once. The file holds counts and timings only, never a prompt or an + answer, and nothing in it leaves this Mac. The + self-test + page says what it costs and where the file is.

+
+
Request statistics