Explorer reads samples_map_lite_v3 twice via range requests (~120 MB for a 63 MB file) — #345 fixes time-to-usable, not total bytes

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: ship #345, but with the numbers stated honestly. What I did: measured the canary (shim on) against production (shim off) three ways on 2026-08-27 and found that the headline "74 MB → 3.8 MB" describes bytes until the facets are usable, not the full cold boot. Filing the part the shim does not fix.

Finding

With range requests working (canary), DuckDB-WASM reads isamples_202608_samples_map_lite_v3.parquet (62.9 MB) in full — twice: ~237 ranged GETs, ~120 MB transferred, ~63 MB unique coverage (100 % of the file), ~48 % overlap. It happens in the background over ~40 s after the facets are already on screen.

cold boot, unthrottled, Chromium	production (shim off)	canary (shim on)
bytes until facets usable	~74 MB	~3.5 MB ← the #345 win
sample_facets_v4 (69.4 MB)	69.4 MB whole	9.9 MB ranged (14 %)
sample_facet_masks (10.1 MB)	10.1 MB whole	0.8 MB ranged (8 %)
samples_map_lite_v3 (62.9 MB)	62.9 MB whole, once	~120 MB ranged, read twice
full-boot parquet bytes (60 s)	~144 MB	~131 MB
"falling back to full HTTP read"	9	0

So #345 turns a 74 MB blocking download into a 3.5 MB one (the 3G time-to-facets goes 423 s → 78 s), but the total transferred over a full boot is roughly unchanged, because the map file is scanned end-to-end regardless — and now twice.

Likely cause / fix

Two consumers scan the lite file (globe points and the table/count?), each pulling every row group through DuckDB-WASM's buffer, so ranged reads re-fetch what a single whole-file read got once. Options: (a) load samples_map_lite_v3 once into a DuckDB table at boot and have every consumer read the table; (b) one explicit whole-file fetch for that file only (it is always read entirely), keeping range reads for the facet files; (c) check for duplicate read_parquet of the same URL in explorer.qmd. Fixing this would make the full-boot total ~70 MB lower than today.

Method: Playwright page.on('response') over every .parquet GET incl. Web-Worker fetches, summing Content-Length (206 = range length) and merging Content-Range intervals for unique coverage; harness tests/playwright/bandwidth_matrix.py (on perf/313-bandwidth-findings, host filter widened to workers.dev) reproduces the 3.5 MB-to-milestones figure. Scripts: /tmp/measure_parquet_bytes.py, /tmp/measure_ranges.py (local; will add to the repo).

[rdhyee]

🤖 rbotyee — production numbers after the #345 deploy (2026-08-27 17:50 ET, cold cache, Chromium, 60 s): samples_map_lite_v3 133.9 MB in 258 ranged GETs for a 62.9 MB file; sample_facets_v4 21.3 MB; total 157 MB; 0 fallbacks. So this is now the dominant cost of a cold boot — same shape as the canary measurement above.

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: "continue with #351". What I did: measured the cause (the samples table's boot-time COUNT + page queries scan the file concurrently, each through ranged GETs), fixed it in #352, and re-measured: lite 132.8 MB → 62.9 MB in one GET, all parquet 156 MB → 86 MB, samples table at 63 s instead of 122 s. Six Codex rounds → LGTM-with-nits. Merge/deploy waits for a human.

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: "go for it — merge". What I did: merged #352 (f0f47bf) after staging verification on the fork; production deploys via the Pages workflow on push to main; I am verifying isamples.org after the run and will post the numbers here.

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — What I did: verified production after the #352 deploy (f0f47bf, Pages run 33199843318).

isamples.org/explorer.html, cold cache, headless Chromium, 200 s (tests/playwright/measure_parquet_ranges.py):

	before (#345 only, 08-27)	now
samples_map_lite_v3	133.9 MB in 258 ranged GETs	62.9 MB, one GET, 0 ranged
all parquet, full boot	~157 MB	86.3 MB
"falling back to full HTTP read"	0	0
samples table shows Page 1	—	22 s
filteredClustersReady / facetIndexReady / lite buffer	—	4.2 s / 6.5 s / 16.1 s

Playwright against production: explorer-smoke 4/4; filtered-clusters-300 (with DATA_BASE=https://data.isamples.org) 2/2.

So the cold boot now moves ~70 MB less than yesterday and the facet-first boot from #345 is untouched (facets ready at 6.5 s, lite fetch starts after). Follow-up candidates noted in #352: an exclusive barrier for the #300 heavy query (pre-existing 20 s idle cap), and facet-tree.spec.js / fts-v1.spec.js still build bare /explorer.html URLs.