Skip to content

Add --cache-right: replay the reference's network on repeat runs - #21

Merged
tmashara merged 2 commits into
mainfrom
reference-cache
Oct 24, 2026
Merged

tmashara merged 2 commits into
mainfrom
reference-cache

Conversation

@tmashara

@tmashara tmashara commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Adds --cache-right, an opt-in flag that makes repeat runs against the same reference faster. It replays the right side's network traffic from a HAR that an earlier run recorded, so the run doesn't fetch from the reference again.

Why HAR replay, not a cached screenshot

locate() needs the right page live, holding the job's document. It probes the right page by pixel box (with elementFromPoint occlusion and visibility checks), and it runs document.querySelector with selectors the left page generated. You can't answer those from a snapshot without a DOM, so caching the Shot would mean dropping or degrading dom attribution. Two more problems with caching shots:

  • A cached shot was taken under an older run's pinned clock, so any page that shows a time would differ from the left side.
  • --rerun-failed already reuses shots that came out clean. The shots a loop actually reruns are the ones that differ, which are exactly the ones that need attribution.

With replay, the right page still loads, settles and gets shot under this run's clock. Attribution, the HTTP ≥ 400 refusal, blank detection, failed stylesheets and fonts, unstable pixels and the text diff all come from a live render, as before. Only the round trips to the reference are skipped. Left and right capture at the same time, so the reference's network latency is what sets the wall time. That latency is what replay removes.

Cache key and invalidation

  • File key, <out>/.cache/right-<hash>.har: renderdiff version, browser name and version, right origin, the right storage state's contents, CONTEXT_OPTIONS and blockUrlPatterns. These are what change what the reference serves. Masks, CSS and selectors change only rendering, which still happens live, so they aren't in the key.
  • Per shot: [route.right, width, before-hook source]. A recording is replayed only when it covers every shot in the run. Otherwise the whole right side loads live and is recorded again.
  • Invalidation: the flag is opt-in, and a recording is replayed for at most 1 hour (REPLAY_TTL). ETag and Last-Modified don't help here: SSR HTML rarely has them, and they wouldn't catch asset changes. A request the recording doesn't have goes to the network (notFound: "fallback"). Responses with status ≥ 400 are dropped from the recording and entries are deduped to one per method+URL, so a 500 or a once-failed asset is never replayed; shots that errored or had failed assets aren't marked as covered. A failed save only warns (recording not saved); results and the report are still written. Recording metadata lives in a small <har>.json sidecar written last, so a half-written save is never replayed.
  • Never a silent stale clean: the default behaviour is unchanged. A replayed run prints right: replayed from a recording of <ISO> (--cache-right), and Each result measured against a replay records rightReplayedFrom, and results.json gets meta.rightReplayedFrom (the oldest replay among all results, reused ones included; null otherwise). --rerun-failed reuses a replayed result only when this run also passes --cache-right and the recording is still within the TTL; otherwise that shot reruns live. The report header shows the same thing. SKILL.md tells agents to drop the flag for the final run.
  • When the out dir is wiped, .cache is kept only with --cache-right, and saving a recording deletes older right-* files (they can hold a signed-in session's cookies).
  • A --rerun-failed --cache-right re-record covers only the rerun shots.

Measured

Fixture site, 6 routes × 2 widths, with the right origin behind a proxy that adds a fixed delay to every request:

right latency live replayed
600 ms, --concurrency 2 12.6 s / 12.5 s 6.3 s / 6.3 s
300 ms, default concurrency 4.7 s / 4.6 s 4.1 s / 4.1 s
0 ms 4.0 s 3.9 s

On the replayed runs the proxy received 0 requests, and the results were identical to the live runs. The run that records costs about the same as a live run.

Tests

  • tests/unit/cache.test.ts: how recordings merge (errors and duplicates dropped, older recordings replaced), the TTL and coverage rules, and the key (including blockUrlPatterns).
  • tests/visual/accepted.test.ts: a repeat run loads only the left side and gives the same results, with dom elements still named. A shot the recording lacks makes the run load live and record again. A right route that 500s once isn't replayed as an error. --rerun-failed without the flag after a replayed run reruns those shots live.
  • typecheck, lint, format:check, npm test (226) and npm run test:visual (304) all pass.

🤖 Generated with Claude Code

tmashara and others added 2 commits October 24, 2026 15:29
The right page still loads, settles and is shot live, so element
attribution, blank/failed-asset refusals and unstable detection are
unchanged; only round trips to the reference are skipped. Opt-in, a
recording is replayed for an hour and only when it covers every shot
in the run. The run prints a right: line and records meta.rightCache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A shot measured against a replay is marked rightReplayedFrom and reused
by --rerun-failed only in a run that would replay it; meta carries the
oldest. Recordings drop error responses and repeats, skip shots with
failed assets, replace older recordings, keep their index in a sidecar,
and a failed save no longer loses the report. .cache survives the wipe
only with --cache-right.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tmashara
tmashara merged commit d8bdc67 into main Oct 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant