#351: fetch samples_map_lite once, serve every lite query from an in-memory copy

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: "continue with #351" — make the cold boot stop reading samples_map_lite_v3 twice, without giving back the #345 time-to-usable win. What I did: measured the cause, fixed it at the one choke point every query passes through, took it through three Codex review rounds, re-measured, and ran the affected Playwright specs against a local render with production data. Nothing is merged or deployed; this PR is the proposal.

Fixes #351.

Cause (measured, not inferred)

At boot the samples table's refreshAll() fires loadCount and loadPage(0) concurrently (Promise.all, explorer.qmd tableView). Each is a full scan of samples_map_lite_v3.parquet (62.9 MB). With #345's range requests working, each concurrent scan pulls the whole file through DuckDB-WASM's per-read ranged GETs and nothing shares the bytes between them: 132.8 MB in 65 ranged GETs, 54 % overlap, for a 63 MB file. Later consumers (the #300 filtered-cluster aggregation, search-result JOINs) scan it again.

Fix

• New liteFile cell: after facetIndexReady settles (so the 63 MB never competes with Explorer downloads ~74 MB cold: DuckDB-WASM 1.24.0's HEAD+Range probe expects 206; our (correct) 200 defeats range requests #345's ~3.5 MB boot-critical facet chain), fetch() the file once, stream it with a 30 s no-bytes watchdog, check the byte count against Content-Length, registerFileBuffer() it as an in-memory DuckDB file, and validate the parquet footer (parquet_metadata) before declaring it ready. Any failure → the virtual file is dropped and queries keep reading the URL with ranged GETs (today's behaviour).
• The db cell's existing query wrapper (which already tracks in-flight queries for the explorer: filtered cluster/heatmap view at world zoom (don't promote to raw point mode for broad filters) #300 idle gate) resolves every read_parquet('<lite URL>') to the in-memory name. The 24 query sites are untouched; lite_url stays a URL (tests and the Substrate↔UI coherence: release manifest so sibling data files can't silently diverge #334 manifest check read it). A lite query waits at most 120 s for the fetch to settle, then reads the URL for that query — the shared gate is bounded per query (Codex rounds 1–2).
• tests/playwright/measure_parquet_ranges.py: the measurement method behind Explorer downloads ~74 MB cold: DuckDB-WASM 1.24.0's HEAD+Range probe expects 206; our (correct) 200 defeats range requests #345/Explorer reads samples_map_lite_v3 twice via range requests (~120 MB for a 63 MB file) — #345 fixes time-to-usable, not total bytes #351 (every .parquet GET incl. Web-Worker fetches, range coverage/overlap per file, time-to-table, warm reload), previously living in /tmp.

Numbers (local render → production data host, cold cache, headless Chromium, 200 s window, same machine/link)

	baseline (upstream/main)	this PR
samples_map_lite_v3	132.8 MB, 65 ranged GETs, 54 % overlap	62.9 MB, one GET
all parquet bytes	156.1 MB	86.2 MB
samples table shows "Page 1 of …"	122 s	63 s
sample_facets_v4 / masks (range reads kept)	21.3 MB / 0.8 MB	21.3 MB / 0.8 MB
"falling back to full HTTP read"	0	0

The table gets faster, not slower: two scans from memory beat two scans over ranged reads. Caveats, stated plainly: this link ran at ~1–2 MB/s, so the fetch took 46 s; on a much slower link (< ~0.6 MB/s) a lite query issued early can hit the 120 s cap and fall back to ranged reads for that query — no worse than today for it, plus the background fetch. Not claimed: browser caching of the whole file on revisits — the immutable headers allow it, but a warm reload in headless Chromium re-fetched all 63 MB, so the comment says so rather than promising a win.

Memory: steady state one copy in the DuckDB WASM heap (the footprint the pre-#345 "full HTTP read" fallback had); transiently ~2 copies while the bytes are assembled and transferred to the worker.

Review

Codex (gpt-5.4, static, read-only sandbox) × 6 rounds, each asked to refute: R1 changes-required (P1 unbounded wait if the facet chain or fetch never settles; P2 memory comment; P2 measure latency) → R2 changes-required (P1 the cap must be absolute, not phase-scoped; P2 validate a truncated body) → R3 LGTM-with-nits → R4 changes-required (P1 the #300 readiness probe waited for the whole fetch — caught by the Playwright run; fixed with the parquet_schema probe + narrowed literal) → R5 changes-required (P2 a cap expiry could cost a second 120 s wait — fixed with one shared deadline; P1 idle-cap overlap) → R6 LGTM-with-nits: agreed the residual idle-cap overlap is #300's pre-existing best-effort gate, strictly narrowed here (P3, follow-up), nit applied. Its refutation attempts that held: all 23 table-read sites match the exact literal; ?data_base= sanitisation blocks quote injection; no double registration; URL/buffer mixing is consistent under the immutable-file contract.

Playwright (local :5860 render, production data host, --workers=1):

• explorer-smoke 4/4 on the final render.
• filtered-clusters-300 with DATA_BASE=https://data.isamples.org: 2/2 on this branch, 1/2 on main (baseline's readiness probe never flipped inside 120 s on this link). Its count-conservation step queries the lite file through the new in-memory path and matches the rendered clusters. Spec header updated: with the _v3 republish, production carries res4/res6, so it can run against live data.
• explorer-map-overlay: 6/10 on both branch and main — the same four table tests and the base-layer-picker occlusion test fail identically on baseline on this ~1–2 MB/s link (60 s budgets vs a 63 s table); not a regression, no signal.

Not in this PR

• bandwidth_matrix.py host-filter widening (on perf/313-bandwidth-findings).
• Slow-link (3G) emulation runs — the 120 s cap is a judgement call; if a 3G measurement matters before merge, measure_parquet_ranges.py + Playwright's CDP throttling can produce it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LtTxB4jfTZgaTR7CK4zKqy

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: have Codex refute the change before a human looks. What I did: six static review rounds; posting the final verdict verbatim.

Codex round 6 (final) — LGTM-with-nits

Verdict: LGTM-with-nits

Your P1 reasoning is correct. I cannot construct a realistic timeline where this branch creates worse heavy-query concurrency than upstream.

1. [P3, follow-up] The 20-second idle gate remains best-effort, but this branch strictly narrows the existing exposure.

   Upstream timeline:

   • The table starts COUNT and page scans concurrently.
   • The filtered aggregation waits at most 20 seconds.
   • Because cold ranged scans exceed 20 seconds, the aggregation starts while they remain active.

   Branch timeline:

   • Lite queries increment _inFlight before waiting (explorer.qmd).
   • The heavy aggregation waits for the shared lite deadline/buffer first, then starts its idle wait (explorer.qmd).
   • Normally the released in-memory scans finish within the 20-second idle window.
   • If in-memory scans exceed 20 seconds—or the 120-second deadline falls back to slow URL scans—the existing cap can still permit overlap (explorer.qmd).

   That residual is the pre-existing limitation of explorer: filtered cluster/heatmap view at world zoom (don't promote to raw point mode for broad filters) #300’s non-exclusive gate, not a Explorer reads samples_map_lite_v3 twice via range requests (~120 MB for a 63 MB file) — #345 fixes time-to-usable, not total bytes #351 regression. An exclusive barrier belongs in a separate follow-up.

2. [Nit] “This cell always settles” is slightly too absolute.

   The 30-second watchdog bounds a no-progress fetch, but not an indefinitely trickling response or a hung registration/validation operation (explorer.qmd). This does not strand queries because their shared gate expires independently after 120 seconds. I would reword this as “query consumers always proceed,” but it is non-blocking.

The new pre-query gate has no unbounded wait: one deadline is armed by the first lite demand and shared by every caller (explorer.qmd). Repeated calls after expiry resolve immediately to the URL; they do not pay another 120 seconds. The #300 path therefore has at most the remaining shared deadline plus the 20-second idle cap, and its supersession check remains immediately before query dispatch (explorer.qmd). Query execution itself remains uncapped, as before.

There is also no semantic cross-query inconsistency. A query already released to the URL may coexist with a later query redirected to the buffer, but both name the same versioned, immutable object. Count/page queries can therefore use different transports without seeing different data. This depends on the established immutability contract; the patch does not independently enforce a hash.

git diff --check passed. A Quarto inspection could not run because the restricted environment denied Quarto’s sysctl call, not because of a document error.

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: "deploy 351 to rdhyee to check everything is ok first." What I did: dispatched the fork's quarto-pages.yml on this branch (run 33197096254, commit e600da8), then verified the live staging page.

Staging: https://rdhyee.github.io/isamplesorg.github.io/explorer.html

Bandwidth (tests/playwright/measure_parquet_ranges.py, cold, 200 s, headless Chromium — the link was ~0.7 MB/s at the time):

	staging (this branch)
samples_map_lite_v3	62.9 MB, one GET, 0 ranged reads; buffer registered + validated (bytes: 62924115)
all parquet	86.3 MB (fix locally: 86.2; main locally: 156.1)
sample_facets_v4 / masks	21.3 MB / 0.8 MB ranged, unchanged
"falling back to full HTTP read"	0
lite fetch	started 14.9 s (after facetIndexReady), 86 s on that link

Boot milestones with the #300 spec's URL (?material=… at world zoom, later on a fast link): filteredClustersReady 5.1 s on this branch vs 5.2 s on main (identical), facetIndexReady 6.2 s, lite buffer 12.7 s; total parquet 119 MB vs 173 MB on main; table at 32 s vs 52 s.

Playwright against staging (TEST_URL=https://rdhyee.github.io/isamplesorg.github.io, --workers=1):

• explorer-smoke: 4/4.
• filtered-clusters-300 with DATA_BASE=https://data.isamples.org: 2/2 — after fixing the spec itself (6a063ee→HEAD): it built its URL as a bare /explorer.html, which replaces the fork's sub-path, so against staging it had been loading a 404 page (readiness could never flip). helpers/url.js exists for exactly this (PR tests: extract URL helper for sub-path-safe page.goto across the suite #238); the spec now uses it. facet-tree.spec.js and fts-v1.spec.js have the same latent bug — not touched here.

One false alarm, recorded so nobody chases it: a manual Chrome check showed DuckDB not initialising for ~78 s. That tab was document.visibilityState === "hidden" (a background tab; Chrome throttles it) and everything ran the moment it was foregrounded. Headless runs are always visible and show DuckDB up by ~6 s.

Also added: measure_parquet_ranges.py now records boot milestones (6a063ee).

Nothing merged or deployed to production.