Interactivity on 100GB+ datasets: O(change) view sync and ledger writes

agent_config.yaml

@@ -17,7 +17,16 @@ agent:
   opencode_url: http://127.0.0.1:4096
 
   # OpenCode model, "providerID/modelID" (can also be set as env variable
-  # OPENCODE_MODEL). Empty string self-heals to whatever OpenCode's own
-  # config was last set to, or a configured provider default, falling back
-  # to the free-tier "opencode/deepseek-v4-flash-free" if neither resolves.
-  opencode_model: "opencode/deepseek-v4-flash-free"
+  # OPENCODE_MODEL).
+  #
+  # Left EMPTY on purpose. Empty means "follow OpenCode's own config" -- which
+  # is what the studio's model picker (.wl-ag-model) writes on every pick, via
+  # PUT /config -- so choosing a model in the UI also changes the model
+  # weightslab's own queries use, and it is re-checked before every turn.
+  # When OpenCode names no model either, the fallback is "opencode/big-pickle".
+  #
+  # A value here SEEDS the choice: it is used when nothing has been chosen in
+  # OpenCode's config yet (and is published there, so the studio shows it), but
+  # a model picked in the UI afterwards wins. Set OPENCODE_MODEL instead to
+  # PIN a model that nothing can override.
+  opencode_model: ""

docs/agent.rst

@@ -211,16 +211,71 @@ If ``OPENCODE_URL`` is set and reachable, the UI server adopts it directly
 instead of spawning a child; the backend SDK agent reads the same variable
 (``agent.py``'s ``_load_config``) — set it once and both sides talk to the one
 server, so a model you authenticate once is available everywhere.
-``OPENCODE_MODEL`` (or ``agent_config.yaml``'s ``agent.opencode_model``) picks
-the default model for the backend SDK agent, as an OpenCode
-``providerID/modelID`` string (e.g. ``openrouter/anthropic/claude-opus-4.6``).
-Leave it unset to fall back, in order, to: whatever model OpenCode's own
-``/config`` was last set to (the model picker's own pick, e.g. from the
-Weights Studio landing page), and otherwise the free-tier
-``opencode/deepseek-v4-flash-free`` automatically — a provider's own
-reported default used to be tried in between, but that could itself be an
-arbitrary, non-text-reasoning model whenever any provider had credentials
-configured, so it no longer overrides this.
+Which model gets used, and how the two sides agree
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+**OpenCode's own config is the shared source of truth.** ``GET /config``'s
+``model`` field is read by every client of that server -- the Weights Studio
+model picker, the OpenCode CLI, and the backend SDK agent -- and written by the
+picker (``PATCH /global/config``, ``global.config.update``) whenever you choose
+a model. Neither side has to know the other exists; they meet in that one
+field.
+
+The backend SDK agent resolves its model in this order:
+
+1. ``OPENCODE_MODEL`` -- a hard **pin**, for automation that must force a
+   model regardless of what anyone picked. The UI picker cannot move it; if the
+   two disagree, ``agent status`` logs which model this backend is actually
+   using and why.
+2. ``GET /config``'s ``model`` -- the shared choice above. Re-read **before
+   every turn**, so switching models in the UI mid-run takes effect
+   immediately; it is not latched at start-up.
+3. ``agent_config.yaml``'s ``agent.opencode_model`` -- a **seed**, not a pin:
+   which model to use when nothing has been chosen in OpenCode's config yet.
+   It is published there, so the studio shows it; once anything is chosen (in
+   the UI, by ``agent model``, or by the CLI), that choice wins and the seed
+   goes unused. Pinning the yaml value instead meant a run started *after*
+   picking a model in the studio quietly went back to the yaml one.
+4. ``opencode/big-pickle``, the built-in fallback, when nothing above resolves.
+   Published too, so a backend that started before the UI hands the picker the
+   model it is itself using.
+
+A provider's own reported default used to be tried just before the built-in
+fallback, but that could itself be an arbitrary, non-text-reasoning model
+whenever any provider had credentials configured, so it no longer overrides it.
+
+Whoever chooses last wins, and both surfaces follow: picking in the UI moves
+the backend's next query, and ``agent model <providerID/modelID>`` from the
+CLI moves the UI's picker.
+
+Either start order therefore converges on one model:
+
+.. code-block:: text
+
+   Studio first:      pick a model in the UI  ->  PATCH /global/config
+                      ->  weightslab start    ->  GET /config  ->  same model
+                      (agent_config.yaml's seed is not used: something was chosen)
+
+   weightslab first:  nothing chosen anywhere
+                      ->  agent_config.yaml's opencode_model, else
+                          opencode/big-pickle -- and published
+                      ->  Studio starts       ->  GET /config  ->  same model
+
+The start-up banner states which of the three won:
+
+.. code-block:: text
+
+   Agent initialized from configuration C:\Users\you\wl_agent_config.yaml:
+        OpenCode URL=http://127.0.0.1:4096
+        Model=opencode/muse-spark-1.3-contributor-free (from agent_config.yaml's opencode_model (nothing chosen yet), published to OpenCode's config so the studio shows it)
+
+Other values in the parentheses are ``pinned by OPENCODE_MODEL``, ``from
+OpenCode's config, which the studio model picker writes``, ``chosen here and
+published to OpenCode's config`` (an ``agent model`` switch), ``built-in
+default, published to OpenCode's config for the studio``, and ``unresolved --
+OpenCode unreachable, retried on the first query``. It used to print ``Model=(server default)`` whenever nothing was
+pinned, which read as "my pick was ignored" even when the first query would
+have picked it up.
 
 Credentials and provider setup live in OpenCode itself, never in WeightsLab:
 
@@ -335,7 +390,8 @@ configure ``agent_config.yaml`` and/or environment variables.
    # 3. agent_config.yaml
    agent:
      opencode_url: http://127.0.0.1:4096
-     opencode_model: ""   # empty = use OpenCode's own configured default
+     opencode_model: ""   # empty = follow OpenCode's config (the UI picker);
+                          # a value here PINS the model instead
 
 Then check it from the CLI:
 

docs/perf/o_change_register.md

@@ -0,0 +1,170 @@
+# Interactivity for 100GB+ datasets — register of O(data) operations
+
+Goal: every per-step and per-request operation should cost **O(change)** or
+**O(page)**, never **O(dataset)**. Today several do, so cost grows with the
+dataset while the actual work stays constant.
+
+> **Baseline caveat.** The measurements below were taken against a stale
+> in-place copy of weightslab (`~/weightslab_src`, 1.3.3+multiview), which
+> differs from `dev` in 51 files. Two serving costs listed there are **already
+> fixed on dev** (see §B). Serving numbers must be re-measured on this branch
+> before any serving change is attributed an improvement. The storage findings
+> (§A) were re-verified against dev and still hold.
+
+Reference measurements (UltraEdit, 3,959,093 rows, ~19 cols, A10G box):
+
+| | measured |
+|---|---|
+| bare-torch step (no WL) | 1,162 ms → 20.66 samples/s |
+| with WL, no UI client | ~5.5 samples/s (**3.8× slower**) |
+| with WL + image requests | ~1.5 samples/s (**13.8× slower**) |
+| per-step signal write itself | **4 ms (0.34%)** — already fine |
+| grid page latency (64 imgs) | p50 5.2 s, p95 9.6 s |
+
+The signal path is not the problem. Storage write-amplification and the view
+rebuild are.
+
+---
+
+## A. STORAGE — `data/h5_dataframe_store.py`
+
+`upsert()` **receives** only dirty rows but **implements** a full table
+replacement.
+
+| line | operation | cost |
+|---|---|---|
+| 693 | `_create_backup()` — full file copy **before every upsert** | O(file) |
+| 708 | `existing = store.select(key)` — read entire table | O(N) |
+| ~768 | `pd.concat([existing, delta])` | O(N) |
+| ~772 | `existing[~existing.index.duplicated()]` — dedupe all rows | O(N) |
+| ~785 | `_decategorize_for_storage(existing)` | O(N) |
+| 801 | `store.remove(key)` — drop table | O(N) |
+| 804 | `store.append(..., data_columns=True)` — rewrite + index **every** column | O(N·cols) |
+| 846–883 | same read/remove/rewrite in the column-delete path | O(N) |
+
+**Amplification:** ~5 KB of changed signals per flush → ~200 MB written,
+roughly **40,000×**. At `ledger_flush_interval=3.0s` vs ~1.5 s steps, that is a
+full-table rewrite about every 2 steps.
+
+Fix direction: append new rows; modify existing rows in place
+(`select_as_coordinates` + `table.modify_rows`). Backup incrementally, not per
+upsert. No `data_columns=True` — no `store.select()` in this file uses `where=`,
+so those per-column indexes are built and never read.
+
+*(A previous attempt to narrow `data_columns` broke the write path entirely —
+678 upsert failures, zero persisted data. Any change here needs a
+write→read→assert-contents check, not just a timing check.)*
+
+## B. SERVING — `trainer/services/data_service.py`
+
+`_pull_into_all_data_view_df()` (line 938) runs several full-frame passes.
+
+**Already fixed on dev — do not re-report as wins:**
+- the collapse no longer re-enters `get_combined_df()`; the pulled frame is
+  passed in, so the frame is copied once, not twice
+- `array_proxy` no longer does a per-cell `.apply(convert_to_proxy)` (was
+  1,660 ms at 4M rows)
+
+Remaining, to be **re-measured on this branch**:
+
+| line | operation | cost (measured @4M) |
+|---|---|---|
+| 946 | `get_combined_df()` → `dataframe_manager:2140 self._df.copy()` | 101 ms–1.2 s |
+| — | `get_collapse_annotations_to_samples_df(df)` — groupby collapse | 6,326 ms* |
+| — | `safe_reset_index(df)` | 1,912 ms |
+| — | `set_index([origin, sample_id])` | O(N) |
+| 3636 | `updated_df.reindex(target_order)` | 290 ms |
+
+Callers — each one is a full O(N) rebuild: lines **440, 851, 3593, 4605, 4626**,
+reached from `GetDataSamples`, `GetMetaData`, `EditDataSample`, `GetDataSplits`.
+
+Held under `_update_lock`, which the trainer also needs → measured lock holds of
+39–126 s and the 3.7× training penalty while browsing.
+
+**The collapse is provably a no-op when `annotation_id.max() == 0`** (UltraEdit
+is exactly 1:1) yet still costs 6.3 s per rebuild.
+
+Fix direction: serve a page from the source frame by index (O(page)); rebuild
+the full view only for genuinely global operations (histogram, global sort);
+apply deltas rather than rebuilding; never hold the writer lock across a
+rebuild — build off-lock and swap the reference.
+
+## C. OTHER FULL SCANS
+
+| location | note |
+|---|---|
+| `dataframe_manager:1875` `data_snapshot.iterrows()` | input is O(change), but row-wise Python per flush |
+| `dataframe_manager:2400` `.apply(lambda …)` | per cell |
+| `data_service:1200` `_compute_natural_sort_stats` | builds a list of one Series per row (4M objects). Gated off (`compute_natural_sort=False`) — latent |
+| `data_service:538` PreviewCache | bounded by `WL_MAX_PREVIEW_CACHE_SIZE` — OK |
+
+## D. ALREADY O(change) — keep
+
+- `self._pending` dirty-row set (`dataframe_manager:95, 751, 761`)
+- flush work set: `work = list(self._pending)` (`:1827`)
+- `_origin_revisions` per-origin version counters (`:94, 1235`)
+
+The bookkeeping needed for differential updates already exists; the storage and
+view layers just don't use it.
+
+\* measured on the stale copy; re-measure on dev.
+
+## Measurement protocol
+
+Fixed workload: **1,000 train samples** = 41 steps at batch 24. Every change is
+reported as:
+
+1. wall-clock for the 41 steps, vs the bare-torch floor
+2. bytes written to H5 for those steps
+3. grid-page latency (64 images) and training throughput **while** serving
+4. **ledger contents verified** — signal columns present, measured-row count
+
+(4) is not optional: a previous "10× win" was writes silently failing.
+
+---
+
+# E. Triage — which call sites need a full reconstruction
+
+`_slowUpdateInternals()` rebuilds the whole view: `copy → collapse → reset_index
+→ set_index → reindex`. It has 18 call sites, and almost none of them need
+that. Most just want **fresh values for rows the trainer touched**, which is
+`O(change)`.
+
+`_fastUpdateInternals()` applies only dirty rows, via a maintained
+`sample_id → position` map (`_rebuild_view_pos_map`), and returns `False` —
+falling back to the full rebuild — whenever it cannot safely apply:
+
+- no view yet, or no position map (first build)
+- a dirty `sample_id` absent from the map (new rows ⇒ structural change)
+- backlog > `max_dirty` (a rebuild is genuinely cheaper)
+
+So the worst case is today's behaviour, never wrong data.
+
+| site | routing | why |
+|---|---|---|
+| `_bg_view_refresh` | **fast** | exists purely to refresh values after a stale read — the textbook differential case, and the one that holds `_lock` against the trainer |
+| `_process_get_data_samples` | **fast** | grid fetch needs current values, not a new frame |
+| `_compute_custom_signals` | **fast** | writes new signal *values*; schema unchanged |
+| `GetDataSplits` | **fast** | read-only summary |
+| `EditDataSample` ×5 | **fast** | per-sample value edits |
+| `EditDataSample` ×3 (`df.modify`, `df.drop_column`) | **full** | changes the schema — differential cannot add/remove columns |
+| `ApplyDataQuery` `@reset`/`@clear` | **full** | clears `_is_filtered` to restore the full universe; a differential updates values but cannot restore *dropped rows* |
+| `ApplyDataQuery` filter + agent paths | **full** (deferred) | a forced rebuild preserves `_is_filtered` (`:3691`), so swapping in a differential changes which rows the user sees. Rare, user-initiated, low perf value, high blast radius — not worth the risk until the filter semantics are pinned down |
+| `_compute_natural_sort_stats` | **full** | gated off (`compute_natural_sort=False`); latent |
+| `_manual_save_data_state` | **full** | explicit user save; correctness over speed |
+
+**Kill-switch:** `WL_FAST_VIEW=0` disables the differential and the position-map
+build, reproducing prior behaviour exactly. This is what makes a like-for-like
+A/B possible from a single tree.
+
+## Why the no-client benchmark cannot show this
+
+A 41-step run with no UI client attached records **0 rebuild events** — nothing
+calls `_slowUpdateInternals` at all, so the fast path has nothing to improve and
+correctly measures as no change. The rebuild cost only materialises when a
+client is attached, which is the case that measured **3.7× slower** with p50
+grid latency of 5.2 s.
+
+The A/B is therefore run under load: baseline → under-load → recovery phases
+within one training process (`t_imgload.py`), so each arm is normalised against
+its own idle throughput.

tests/conftest.py

@@ -0,0 +1,37 @@
+"""Shared pytest setup.
+
+Keeps the suite out of the developer's own WeightsLab state.
+
+``weightslab.utils.active_experiment`` records the active experiment directory
+in a real per-user file (``~/.weightslab/active_experiment.json``) so a
+training run started in another terminal lands in the experiment the UI
+established. That is deliberate at runtime -- and poison in tests: a run of
+this suite while a ``weightslab start`` was up resolved its root_log_dir into
+that live experiment, found the config and checkpoints of whatever was running
+there, and failed in setUp with an unrelated config (seen for real:
+tests/gRPC/test_grpc_tag_operations.py loading a segmentation example's
+hyperparameters).
+
+So every test session gets its own throwaway state directory. Tests that
+exercise the handoff itself set ``WEIGHTSLAB_STATE_DIR`` to their own temp
+directory anyway; this only changes the default.
+"""
+
+import os
+import tempfile
+
+import pytest
+
+
+@pytest.fixture(scope="session", autouse=True)
+def _isolate_weightslab_state():
+    previous = os.environ.get("WEIGHTSLAB_STATE_DIR")
+    with tempfile.TemporaryDirectory(prefix="wl-test-state-") as state_dir:
+        os.environ["WEIGHTSLAB_STATE_DIR"] = state_dir
+        try:
+            yield state_dir
+        finally:
+            if previous is None:
+                os.environ.pop("WEIGHTSLAB_STATE_DIR", None)
+            else:
+                os.environ["WEIGHTSLAB_STATE_DIR"] = previous