Report failed stylesheets and fonts; skill hints for upgrades

README.md

@@ -176,6 +176,7 @@ only.
 | `unaligned` | Heights differ but too few rows match to find bands. Use the height delta and regions. |
 | `3 reg` | Regions: connected patches of differing pixels. A missing link barely moves the score, but it is still a region. |
 | `unstable 1,204px` | Pixels that kept changing between shots of the same load (a ticker, a carousel). Excluded from the score, so the route is never clean. |
+| `failed left: font Inter 700` | A stylesheet or font face that side failed to load (blocked URLs aside). Fix that first: a fallback font changes every line of text. |
 | `text -"Pricing"` | A line of visible text only the reference has (`+`: only your change). Up to three, then `(+N more)`. |
 | `dom sel prop a→b` | Element on both sides whose computed `prop` differs, left value first. |
 | `dom +sel` / `-sel` | Element only on the left / only on the right. |
@@ -253,6 +254,7 @@ Everything lands in the out dir (`./renderdiff-out` by default):
         }]
       }],
       "text": { "added": [], "removed": ["Pricing"] },   // added: left only; removed: right only
+      "failed": { "left": [], "right": [] },   // stylesheets and font faces that didn't load: "font Inter 700", "stylesheet https://…"
       "images": { "left": "about.1440.left.png", "right": "about.1440.right.png", "diff": "about.1440.diff.png" },
       "crops": [{
         "path": "about.1440.c1.png", "kind": "band",          // or "region"

docs/how-it-works.md

@@ -23,9 +23,11 @@ honestly. This is what renderdiff does to make sure of that.
   dev toolbars.
 - **Settles the page** — makes native lazy images eager, scrolls every viewport-height
   slice a frame at a time so script lazy loaders fire, waits for every image to decode,
-  `document.fonts.ready` and up to 3s for the network to go 250ms with nothing in flight
+  `document.fonts.ready` (at most 5s) and up to 3s for the network to go 250ms with nothing in flight
   (server-sent events and beacons don't count), then re-injects the stylesheet to
-  catch late-injected chrome.
+  catch late-injected chrome. Stylesheets that failed or answered ≥ 400 (blocked URLs
+  aside) and font faces that failed (blocked font files included) are recorded per side as
+  `failed`, and keep the route off the `clean:` line.
 - **Shoots until stable** — repeats the screenshot until two in a row match (at most 3; 2
   for tiled pages). Pixels still changing between the last two shots of the same load are
   noise from that origin: they're painted magenta on both sides, reported per comparison

docs/output.md

@@ -29,11 +29,12 @@ score and text; the score and region count are then shown as ranges, and `unstab
 largest. Bands are `+` for rows only the left has and `-` for rows only the right has, at
 `y`, with `~` when the widths put it in different places. Up to three text-diff lines
 follow (`-"x"` only on the right, `+"x"` only on the left, `(+N more)` past that). Under a
-row, `dom` names up to three elements behind the changes (`@w` when the row covers several
+row, `failed` lists stylesheets and font faces a side failed to load (fix those first),
+`dom` names up to three elements behind the changes (`@w` when the row covers several
 widths: the first one's), and `crops` lists that row's crop files, relative to the
 directory `results.json` is in. Errors print in full, grouped across the widths that share
-a message. Clean means not one pixel differs and none went unmeasured as unstable; a route
-with unstable pixels gets its own row. With `--threshold`, the verdict line ends
+a message. Clean means not one pixel differs, none went unmeasured as unstable and no
+stylesheet or font failed; a route with unstable pixels or a failed asset gets its own row. With `--threshold`, the verdict line ends
 `, B below 97.00%`. `--progress` gets the per-shot output when piped, `--no-progress` the
 summary on a terminal.
 
@@ -63,6 +64,7 @@ because an unmeasured route is not a verified one.
 | Text | `text: 0 added, 1 removed` | `text -"Pricing"` | `text: { added: [], removed: ["Pricing"] }` |
 | Regions | (on the report card) | `3 reg` | `regions: [...]` |
 | Unstable | `(unstable 1,204px)` | `unstable 1,204px` | `unstablePixels: 1204` |
+| Failed assets | `failed left: font Inter 700` | `failed left: font Inter 700` | `failed: { left: ["font Inter 700"], right: [] }` |
 
 Under each score, `--progress` output and the report card say where to look:
 

skills/renderdiff/SKILL.md

@@ -1,6 +1,7 @@
 ---
 name: renderdiff
 description: Visual-parity check for website migrations. Use when a framework migration, CSS or design-system refactor, template rewrite, CMS move or dependency/build upgrade should not change how pages look, to prove it by screenshot-diffing a local build against production (or any reference) across viewport widths. Also for "did I break the layout", visual regression checks, pixel diffs of two URLs, or running the renderdiff CLI.
+allowed-tools: Bash(npx renderdiff:*), Bash(npx renderdiff@latest:*), Bash(renderdiff:*), Read
 ---
 
 # renderdiff
@@ -21,9 +22,15 @@ legitimate: [reference.md](reference.md).
    with `--accept "differs before the migration"`: what already differs is recorded, not
    yours. Accept only what you looked at, never to get green.
 2. Both origins up, with the same content and data. `--dry-run` shows the URL pairs.
+   Upgrading a dependency or build? The cleanest reference is the pre-upgrade commit in a
+   `git worktree` on another port: same data, fonts and CDN, so every diff is yours. Serve
+   a production build: dev overlays and debug bars exist only on your side (Next's and
+   Astro's are hidden; others go in `hideSelectors`). Routes: one per template or layout,
+   from the sitemap or router; skip search results, pagination and admin.
 3. One route, one width. A row: score, `h±N`/`w±N` height/width delta,
    `band`, `aligned`, `N reg`, `unstable Npx` (moving pixels, unmeasured), `text`.
-4. `text` first: `-"x"` is missing from your change, `+"x"` only in yours.
+4. `failed` first: a stylesheet or font one side didn't load explains the whole row; fix
+   that and rerun. Then `text`: `-"x"` is missing from your change, `+"x"` only in yours.
 5. `band -24@y196`: 24 rows only the reference has. `aligned 100.00%`: the band is the
    whole difference. `unaligned`: see reference.md.
 6. `dom` before crops. `sel prop l→r`: computed value, left first; find the *rule*.

skills/renderdiff/reference.md

@@ -12,6 +12,7 @@ Left is your change, right is the reference: `+` is only/more on the left, `-` o
 | `ERROR HTTP 404 at …` (or any status ≥ 400) | That side answered with an error status. Usually a route that doesn't exist on one origin, or a dev server not ready. |
 | `ERROR page did not render on either side (…)` | Both sides had under 200 chars of text and no visible images, SVG, canvas, video, iframes or form controls, so renderdiff refused to score two blanks as a match. Usually a failed asset or an app that never mounted. One blank side isn't refused; it shows up as a low score. |
 | `ERROR` rows | Unmeasured, not failed. They exit 3 because an unmeasured route isn't a verified one. Report them as unverified. |
+| `failed left: font Inter 700` / `stylesheet …` | That side didn't get the asset (404, CORS, a font CDN that refuses localhost). Fix it before reading anything else on the row: a fallback font reflows every line. |
 | Uniform low score, whole page shifted | Almost always a height or font-loading difference near the top pushing everything down. A `band` names where; compare left and right at that `y`. No band means the shift is horizontal or the content itself changed. |
 | `text -"x" +"y"` | Content differs: `-` lines are on the reference only, `+` on your change only. `(+N more)` means more lines; the report card or results.json lists them all. |
 | `unaligned` | The pages differ in height but share too few identical rows to align, so no bands. Either they are very different, the edits total more than 2,000 rows, or sub-threshold noise touches every row. Use the height delta and regions. |

src/capture.ts

@@ -18,6 +18,8 @@ export interface Shot {
   text: string[];
   /** Page px of the shot's top-left: the element's for a selector route, else 0,0. */
   origin: { x: number; y: number };
+  /** Stylesheets and font faces that failed to load, e.g. `font Inter 700 italic`, `stylesheet https://…`. */
+  failed: string[];
 }
 
 /**
@@ -37,8 +39,8 @@ const TILE_HEIGHT = 16_000;
  */
 const MAX_PAGE_HEIGHT = 40_000;
 
-/** Requests still open per page, and when the last one opened or closed. */
-const network = new WeakMap<Page, { open: Set<Request>; last: number }>();
+/** Requests still open per page, when the last one opened or closed, and stylesheets that failed. */
+const network = new WeakMap<Page, { open: Set<Request>; last: number; failed: Set<string> }>();
 
 /** Never finish, so they would hold network idle open until its timeout. */
 const STREAMS = new Set(["eventsource", "websocket", "ping"]);
@@ -58,7 +60,12 @@ export class HookError extends Error {}
  * page, before any navigation.
  */
 export async function prepare(page: Page, cfg: ResolvedConfig, now: number): Promise<void> {
-  const net = { open: new Set<Request>(), last: Date.now() };
+  const net = { open: new Set<Request>(), last: Date.now(), failed: new Set<string>() };
+  const blocked = new WeakSet<Request>();
+  const sheet = (r: Request) => {
+    if (r.resourceType() === "stylesheet" && !blocked.has(r))
+      net.failed.add(`stylesheet ${r.url()}`);
+  };
   network.set(page, net);
   page.on("request", (r) => {
     if (STREAMS.has(r.resourceType())) return;
@@ -69,15 +76,25 @@ export async function prepare(page: Page, cfg: ResolvedConfig, now: number): Pro
     if (net.open.delete(r)) net.last = Date.now();
   };
   page.on("requestfinished", done);
-  page.on("requestfailed", done);
+  page.on("requestfailed", (r) => {
+    done(r);
+    // Aborted: cancelled by the page or by the next navigation, not a load that failed.
+    if (r.failure()?.errorText !== "net::ERR_ABORTED") sheet(r);
+  });
+  page.on("response", (r) => {
+    if (r.status() >= 400) sheet(r.request());
+  });
   // A request the old document left open never finishes or fails, so it can't hold the next load.
   page.on("framenavigated", (f) => {
     if (f !== page.mainFrame()) return;
     net.open.clear();
     net.last = Date.now();
   });
   for (const pattern of cfg.blockUrlPatterns) {
-    await page.route(pattern, (route) => route.abort());
+    await page.route(pattern, (route) => {
+      blocked.add(route.request());
+      return route.abort();
+    });
   }
   // Time flows, so Date-based debounce and busy-waits work; `capture` rewinds it per load.
   await page.clock.install({ time: now });
@@ -104,6 +121,8 @@ export async function capture(
 ): Promise<Shot> {
   await page.setViewportSize({ width, height: 900 });
   await page.clock.setSystemTime(now);
+  // Here, not on framenavigated, which a pushState during hydration fires too.
+  network.get(page)!.failed.clear();
   const res = await page.goto(url, {
     waitUntil: "domcontentloaded",
     timeout: 45_000,
@@ -145,8 +164,19 @@ export async function capture(
     );
     await Promise.race([decoded, new Promise((r) => setTimeout(r, 5_000))]);
   }, MAX_PAGE_HEIGHT);
-  await page.evaluate(() => document.fonts.ready);
+  // fonts.ready also resolves when a face fails, so the failures are read off it below.
+  const fonts = await page.evaluate(async () => {
+    await Promise.race([document.fonts.ready, new Promise((r) => setTimeout(r, 5_000))]);
+    const failed: string[] = [];
+    document.fonts.forEach((f) => {
+      if (f.status !== "error") return;
+      const variant = [f.weight, f.style].filter((v) => v !== "normal" && v !== "400");
+      failed.push(["font", f.family, ...variant].join(" "));
+    });
+    return failed;
+  });
   await networkIdle(page, 250, 3_000);
+  const failed = [...new Set([...network.get(page)!.failed, ...fonts])];
   await page.addStyleTag({ content: cfg.neutralizeCss }); // anything injected late
 
   const selector = route.selector?.[side];
@@ -254,8 +284,9 @@ export async function capture(
   for (let n = 2; ; n++) {
     await page.waitForTimeout(50);
     const png = await shoot();
-    if (png.equals(prev)) return { png, unstable: null, blank, text, origin };
-    if (n === tries) return { png, unstable: await differing(prev, png), blank, text, origin };
+    if (png.equals(prev)) return { png, unstable: null, blank, text, origin, failed };
+    if (n === tries)
+      return { png, unstable: await differing(prev, png), blank, text, origin, failed };
     prev = png;
   }
 }

src/format.ts

@@ -5,6 +5,11 @@ import type { ElementDiff } from "./locate.ts";
 export const floor4 = (n: number) => Math.floor(n * 1e4 + 1e-7) / 1e4;
 export const pct = (n: number) => `${(floor4(n) * 100).toFixed(2)}%`;
 export const signed = (n: number) => (n > 0 ? `+${n}` : `${n}`);
+/** `left: font Inter 700; right: …`, the sides with assets that failed to load; empty when none did. */
+export const fmtFailed = (f: { left: string[]; right: string[] } | undefined) =>
+  (["left", "right"] as const)
+    .flatMap((s) => (f?.[s].length ? [`${s}: ${f[s].join(", ")}`] : []))
+    .join("; ");
 export const fmtRegion = (r: { x: number; y: number; w: number; h: number }) =>
   `${r.w}×${r.h} at (${r.x},${r.y})`;
 /** Signed like the height delta: + when left has the extra rows, − when right does. */

src/report.ts

@@ -4,7 +4,7 @@ import type { Accepted } from "./accept.ts";
 import type { Band, Box, Region, TextDiff } from "./compare.ts";
 import { CONTEXT_OPTIONS, name, type ResolvedConfig, type Sides } from "./config.ts";
 import type { ElementDiff } from "./locate.ts";
-import { floor4, fmtBand, fmtElements, fmtRegion, pct, signed } from "./format.ts";
+import { floor4, fmtBand, fmtElements, fmtFailed, fmtRegion, pct, signed } from "./format.ts";
 
 interface BaseResult {
   /** The route: `/a`, or `/a::/b` for a pair. */
@@ -43,6 +43,8 @@ export interface ComparedResult extends BaseResult {
   crops: CropResult[];
   /** Every band and region matched an accepted one, and no padding or unstable pixel went unmatched. */
   accepted: boolean;
+  /** Stylesheets and font faces that failed to load per side, e.g. `font Inter 700`. */
+  failed: { left: string[]; right: string[] };
 }
 
 /** A crop of one change: left and right panels, a 1px red box round it, or a 2px red seam. */
@@ -200,6 +202,8 @@ const dom = (els: ElementDiff[] | undefined) =>
 
 function findings(r: ComparedResult): string {
   const items: string[] = [];
+  const failed = fmtFailed(r.failed);
+  if (failed) items.push(`<p class="failed">failed to load: ${esc(failed)}</p>`);
   if (r.regions.length) {
     items.push(
       `<p>${r.regions.length} region(s)</p><ul class="regions">${r.regions

src/run.ts

@@ -18,7 +18,7 @@ import {
   type ResolvedConfig,
   type Route,
 } from "./config.ts";
-import { fmtBand, fmtElements, fmtRegion, pct, signed } from "./format.ts";
+import { fmtBand, fmtElements, fmtFailed, fmtRegion, pct, signed } from "./format.ts";
 import {
   stats,
   writeReport,
@@ -96,8 +96,12 @@ function wipeable(dir: string): boolean {
   return fs.statSync(dir).isDirectory() && fs.readdirSync(dir).every(ours);
 }
 
+/** A stylesheet or font face failed on a side; older results.json files lack the field. */
+const broken = (r: ComparedResult) => !!(r.failed?.left.length || r.failed?.right.length);
+
 const reusable = (r: RouteResult) =>
-  r.status === "compared" && ((r.diffPixels === 0 && r.unstablePixels === 0) || r.accepted);
+  r.status === "compared" &&
+  ((r.diffPixels === 0 && r.unstablePixels === 0 && !broken(r)) || r.accepted);
 
 const files = (r: RouteResult) =>
   r.status === "compared" ? [...Object.values(r.images), ...r.crops.map((c) => c.path)] : [];
@@ -305,11 +309,14 @@ export async function run({
           images,
           crops: cropped,
           accepted: false,
+          failed: { left: leftShot.failed, right: rightShot.failed },
         });
         const { regions, bands, alignedScore, text } = metrics;
         const dom = elementsOf(metrics);
+        const assets = fmtFailed({ left: leftShot.failed, right: rightShot.failed });
         const details = [
           sel,
+          assets ? `failed ${assets}` : "",
           regions.length
             ? `${regions.length} region${regions.length > 1 ? "s" : ""}, largest ${fmtRegion(regions[0]!)}`
             : "",
@@ -550,8 +557,9 @@ export function compact(
   for (const route of routes) {
     const rows = results.filter((r) => name(r) === route);
     const ok = rows.filter((r): r is ComparedResult => r.status === "compared");
-    // Unstable pixels went unmeasured, so a shot with any is not clean, only unchanged.
-    const same = ok.filter((r) => r.diffPixels === 0 && r.unstablePixels === 0);
+    // Unstable pixels went unmeasured and a failed asset rendered a fallback, so a shot with
+    // either is not clean, only unchanged.
+    const same = ok.filter((r) => r.diffPixels === 0 && r.unstablePixels === 0 && !broken(r));
     const acc = ok.filter((r) => r.accepted);
     const changed = ok.filter((r) => !same.includes(r) && !r.accepted);
     if (same.length === rows.length) {
@@ -581,6 +589,11 @@ export function compact(
     }
     for (const g of groups.values()) {
       out.push(`  @${g.map((r) => r.width).join(",")}  ${row(g)}`);
+      const failed = fmtFailed({
+        left: [...new Set(g.flatMap((r) => r.failed?.left ?? []))],
+        right: [...new Set(g.flatMap((r) => r.failed?.right ?? []))],
+      });
+      if (failed) out.push(`    failed ${failed}`);
       const dom = elementsOf(g[0]!);
       if (dom.length)
         out.push(`    dom ${g.length > 1 ? `@${g[0]!.width} ` : ""}${fmtElements(dom)}`);

tests/fixtures/configs/block-asset.mjs

@@ -0,0 +1,4 @@
+/** Aborts one stylesheet the broken-asset fixture links, which must not be reported as failed. */
+export default {
+  blockUrlPatterns: ["**/shared/blocked.css"],
+};

tests/fixtures/pages/README.md

@@ -18,7 +18,7 @@ Three groups:
   alignment (`taller`, `nav`, and `shifted`, which has no band), the text diff (`nav`) and the
   elements each change names (`margin`, `color`, `nav`).
 - **Guard rails** — `blank`, `tall`, `animated`, `lazy`, `lazy-deep`, `lazy-js`, `consent`,
-  `drifting`, `tracker`, `clock`, `ticker`, `unsettled`, `sprite`, `short`, `tall-region`, `reorder`, `stripe`, `frameset`, `menu`, `signed-in`, `fixed`, `inserted`, `labelled`.
+  `drifting`, `tracker`, `clock`, `ticker`, `unsettled`, `sprite`, `short`, `tall-region`, `reorder`, `stripe`, `frameset`, `menu`, `signed-in`, `fixed`, `inserted`, `labelled`, `broken-asset`.
   One per defence in `src/capture.ts` / `src/config.ts`.
 
 ## Authoring rules

tests/fixtures/pages/broken-asset/left.html

@@ -0,0 +1,29 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Broken assets</title>
+<link rel="stylesheet" href="/shared/base.css">
+<link rel="stylesheet" href="/shared/missing.css">
+<link rel="stylesheet" href="/shared/blocked.css">
+<style>
+/* GUARD RAIL for failed assets. Both sides link a stylesheet that 404s, one a config blocks,
+   and a font face that 404s; only the 404s are reported, per side. The block differs so the
+   route has a row to carry them. */
+@font-face { font-family: "Gone"; src: url("/shared/gone.woff2") format("woff2"); }
+h1 { font-family: "Gone", ui-monospace, SFMono-Regular, Menlo, monospace; }
+.box { width: 208px; height: 160px; background: #c62828; }
+</style>
+</head>
+<body>
+<div class="page">
+<h1>Broken assets</h1>
+<p class="lede">This pair is deliberately different. The block below is red on the left and blue on the right, and both sides ask for a stylesheet and a font that are not there.</p>
+<div class="box"></div>
+<section class="notes">
+<h2>Notes</h2>
+<p>A font that never loads falls back silently, and the diff then reads as a typography change. Naming the missing font tells the reader where to look first.</p>
+</section>
+</div>
+</body>
+</html>