dev/viewership-metrics: Probe every redirect rule on the live site

.gitignore

@@ -56,3 +56,6 @@ public/changelog.rss
 .amp/portals/
 
 public/technical-changelog.rss
+
+# script output
+/logs

package.json

@@ -11,6 +11,7 @@
 		"check-filenames": "node dev/check-filenames.mjs",
 		"check-images": "node dev/check-images.mjs",
 		"page-views-report": "node viewership-metrics/page-views-report.mjs",
+		"probe-redirects": "node viewership-metrics/probe-redirects.mjs",
 		"generate-mermaid-logos": "node dev/generate-aws-icons.mjs",
 		"baseai": "baseai",
 		"sync": "npx baseai@latest deploy -m memory-sg-docs-live",

viewership-metrics/README.md

@@ -17,7 +17,8 @@ Cloudflare keeps 90 days of history, so `--days` maxes out at 90.
 
 ## Reports
 
-Written to `reports/` (gitignored). Every page report has the same rows,
+Written to `reports/`, committed so the numbers can be read without a
+token. Every page report has the same rows,
 sorted differently:
 
 | File                         | Sorted by                                |
@@ -33,17 +34,44 @@ is the subset whose referrer is not sourcegraph.com, Cloudflare's page-view
 proxy. 3xx, 404 and 5xx are response counts for the path. Sitemap is `yes`
 when the path is listed in <https://sourcegraph.com/sitemap.xml> (an index
 over `sitemap-main.xml` for blog and changelog, and `docs/sitemap.xml`). A
-`no` on `/docs` is a deleted page or probe that still returns 200; the blog
-sitemap only lists recent posts, so `no` on `/blog` is normal for old posts.
-
-`redirect-rules.md` matches each rule's source against `/docs` 3xx counts.
-Rules are first-match-wins, like `src/middleware.ts`, so a rule whose source
-repeats an earlier one is flagged `shadowed`. The Chain column shows how
-many more redirects a browser follows when a rule's destination is itself
-another rule's source, and where the user finally lands. Its Sitemap column
-says whether the rule's destination is in the sitemap, blank when the
-redirect leaves the site. A `no` with an empty Chain means the rule sends
-people to a soft 404; a `no` with a Chain is an intermediate hop.
+`no` on `/docs` is a deleted page or probe path; the blog sitemap only lists
+recent posts, so `no` on `/blog` is normal for old posts.
+
+`redirect-rules.md` credits `/docs` 3xx counts to rules the way
+`src/middleware.ts` matches them: the request path must equal a source
+exactly, and the first such rule wins. A rule that can never fire says why
+in its Hits column: an earlier rule has the same source, its source has a
+`#fragment` (browsers never send one, so the bare path is matched or the
+page is served), or its source ends in `.md` (rewritten before the lookup).
+The Chain column shows how many more redirects a browser follows when a
+rule's destination is itself another rule's source, and where the user
+finally lands. Sitemap source and Sitemap destination say whether
+`/docs<Source>` and the destination page are in the sitemap; destination is
+blank when the redirect leaves the site. A destination `no` with an empty
+Chain means the rule sends people to a 404; with a Chain it is an
+intermediate hop.
+
+## Redirect probe
+
+`npm run probe-redirects` requests every rule's source on the live site,
+follows the redirects like a browser, and writes `reports/redirect-probe.json`
+with, per rule: whether it `fires` (else `matchedRuleLine`, the rule the
+middleware picks instead, or null when the page is served), `sitemapSource`
+and `sitemapDestination`, every hop, the final URL and status, whether the
+first redirect is the one the rule promises (`outcome`), and the Cloudflare
+rows from `page-views-by-path.md` (run `page-views-report` first) for the
+source, credited only to the firing rule, and for the destination, the page
+the user ends up on. Needs no token; about a minute.
+
+Fragments: browsers never send `#fragment`, so a rule whose source has one
+is probed as its bare path. The browser keeps the user's fragment across
+redirects unless a `Location` header carries its own, so `final.fragment`
+is what the address bar shows, `final.fragmentFrom` says where it came from
+(`request` or `redirect`), and `final.anchorFound` whether the page has an
+element with that id. `summary.byFragmentCase` totals all of this for the
+four source/destination fragment combinations, and `summary.*.alignment`
+sorts rules into works / lands on error / never fires, with or without
+redirect traffic.
 
 ## Filters
 
@@ -65,9 +93,10 @@ people to a soft 404; a `no` with a Chain is an intermediate hop.
 
 ## Known caveats
 
-- The docs site returns 200 for unknown paths, so deleted docs pages and
-  probe paths like `/docs/.zshrc` show up as page views. Blog and changelog
-  return real 404s.
+- Until 2026-09-09 ([#1860](https://github.com/sourcegraph/docs/pull/1860))
+  the docs site returned 200 for unknown paths, so in any window that
+  reaches back before then, deleted docs pages and probe paths like
+  `/docs/.zshrc` count as page views rather than 404s.
 - Counts are adaptive-sampled estimates from `httpRequestsAdaptiveGroups`,
   not exact totals.
 - Blog and changelog pages are served from `github.com/sourcegraph/sourcegraph`
@@ -87,7 +116,8 @@ people to a soft 404; a `no` with a Chain is an intermediate hop.
 From the first 90-day run, September 2026
 ([thread](https://ampcode.com/threads/T-01a08479-a4c8-72ad-8f31-fe7ecc43bf34)):
 
-- **Docs soft-404s.** `/docs/<anything>` returns 200, so deleted pages such as
+- **Docs soft-404s** (fixed in #1860). `/docs/<anything>` returned 200, so
+  deleted pages such as
   `/docs/code_intelligence/tutorials/indexing_go_repo` (830 requests) and
   probes such as `/docs/.zshrc` (660), `/docs/id_dsa` (480) and
   `/docs/__data.json` (990) count as page views and never surface as errors.
@@ -125,4 +155,7 @@ From the first 90-day run, September 2026
   traffic.
 - **Redirect rule added mid-window.** `/changelog/self-hosted/server` shows
   150 requests served as 200 alongside 110 redirects, so the rule likely
-  landed partway through the 90 days.
+  landed partway through the 90 days. Same for
+  `/docs/self-hosted/executors/deploy-executors`: 80 page views before
+  [#1818](https://github.com/sourcegraph/docs/pull/1818) deleted the page
+  and added its rule on 2026-07-22.

viewership-metrics/page-views-report.mjs

@@ -14,14 +14,21 @@
 
 import fs from 'fs';
 import path from 'path';
-import {fileURLToPath} from 'url';
+import {
+	HOST,
+	REPORTS_DIR,
+	SITEMAP_URL,
+	fetchSitemapPaths,
+	landingPath,
+	loadRedirectRules,
+	neverFiresReason,
+	normalizePath,
+	requestPath
+} from './shared.mjs';
 
 const ZONE_TAG =
 	process.env.CLOUDFLARE_ZONE_ID ?? 'a168cb2eefa87d19793824cd9bf83f3a';
-const HOST = 'sourcegraph.com';
 const PATH_PREFIXES = ['/docs', '/changelog', '/blog'];
-// An index pointing at sitemap-main.xml (blog, changelog) and docs/sitemap.xml.
-const SITEMAP_URL = `https://${HOST}/sitemap.xml`;
 const EXCLUDED_COUNTRIES = ['CN'];
 const EXCLUDED_ASN_DESCRIPTIONS = ['Hetzner Online GmbH'];
 
@@ -54,15 +61,6 @@ const PAGE_SIZE = 10000;
 const HOUR_MS = 60 * 60 * 1000;
 const DAY_MS = 24 * HOUR_MS;
 
-const SCRIPT_DIR = path.dirname(fileURLToPath(import.meta.url));
-const REPO_ROOT = path.dirname(SCRIPT_DIR);
-const REPORTS_DIR = path.join(SCRIPT_DIR, 'reports');
-const REDIRECTS_FILE = path.join(REPO_ROOT, 'src', 'data', 'redirects.ts');
-
-// One `{source: '...', destination: '...' | CONSTANT}` entry in redirects.ts.
-const REDIRECT_RULE_PATTERN =
-	/\{\s*source:\s*'([^']*)',\s*destination:\s*(?:'([^']*)'|(\w+))\s*,?\s*\}/g;
-
 const QUERY = `
 query PageViews($zoneTag: string!, $filter: ZoneHttpRequestsAdaptiveGroupsFilter_InputObject!) {
 	viewer {
@@ -208,11 +206,6 @@ async function fetchWindow(token, start, end, rowsByPath) {
 	}
 }
 
-// Merge trailing-slash variants of the same page.
-function normalizePath(pagePath) {
-	return pagePath.length > 1 ? pagePath.replace(/\/+$/, '') : pagePath;
-}
-
 // The site redirects "/page/" to "/page"; that redirect is not worth counting.
 function isTrailingSlashRedirect(row) {
 	const {clientRequestPath, edgeResponseStatus} = row.dimensions;
@@ -223,64 +216,6 @@ function isTrailingSlashRedirect(row) {
 	);
 }
 
-// src/middleware.ts matches rules by exact source path (relative to /docs)
-// and the first match wins, so a repeated source is a dead rule.
-function loadRedirectRules() {
-	const text = fs.readFileSync(REDIRECTS_FILE, 'utf8');
-	const firstLineBySource = new Map();
-	const rules = [];
-	let line = 1;
-	let cursor = 0;
-	for (const match of text.matchAll(REDIRECT_RULE_PATTERN)) {
-		const [, source, destination, constantName] = match;
-		line += text.slice(cursor, match.index).split('\n').length - 1;
-		cursor = match.index;
-		rules.push({
-			line,
-			source,
-			destination: destination ?? constantName,
-			shadowedBy: firstLineBySource.get(source)
-		});
-		if (!firstLineBySource.has(source)) firstLineBySource.set(source, line);
-	}
-	return rules;
-}
-
-// Paths listed in the sitemap, following <sitemapindex> entries. Any page
-// with traffic that is not here is deleted, unlisted or a probe.
-async function fetchSitemapPaths(url = SITEMAP_URL, paths = new Set()) {
-	const response = await fetch(url);
-	if (!response.ok) {
-		throw new Error(`${url}: HTTP ${response.status}`);
-	}
-	const xml = await response.text();
-	const locations = [...xml.matchAll(/<loc>([^<]+)<\/loc>/g)].map(match =>
-		match[1].trim()
-	);
-	if (xml.includes('<sitemapindex')) {
-		for (const location of locations) {
-			await fetchSitemapPaths(location, paths);
-		}
-	} else {
-		for (const location of locations) {
-			paths.add(normalizePath(new URL(location).pathname));
-		}
-	}
-	return paths;
-}
-
-// Where a redirect destination lands on sourcegraph.com, or null when it
-// leaves the site (or is a constant the regex could not resolve).
-function landingPath(destination) {
-	if (destination.startsWith('/')) {
-		return normalizePath(`/docs${destination}`.replace(/[#?].*$/, ''));
-	}
-	if (destination.startsWith(`https://${HOST}/`)) {
-		return normalizePath(new URL(destination).pathname);
-	}
-	return null;
-}
-
 function reportHeader(title, start, end) {
 	return [
 		`# ${title}`,
@@ -307,18 +242,19 @@ function docsDestinationPath(destination) {
 
 // Follow a rule's destination through further rules, as a browser would.
 // Returns the rules hit after this one, and whether they loop back.
-function chainAfter(rule, liveRuleBySource) {
+function chainAfter(rule, firingRuleBySource) {
 	const hops = [];
-	let next = liveRuleBySource.get(docsDestinationPath(rule.destination));
+	let next = firingRuleBySource.get(docsDestinationPath(rule.destination));
 	while (next && next !== rule && !hops.includes(next)) {
 		hops.push(next);
-		next = liveRuleBySource.get(docsDestinationPath(next.destination));
+		next = firingRuleBySource.get(docsDestinationPath(next.destination));
 	}
 	return {hops, loop: Boolean(next)};
 }
 
-// Hits = redirects served on /docs<source>. Live rules sort by hits, then
-// shadowed duplicates; both keep file order within a tie.
+// Hits = redirects served on /docs<source>, credited to the rule the
+// middleware matches (`fires`). Firing rules sort by hits, then the rules
+// that never fire; both keep file order within a tie.
 function formatRedirectRulesReport({
 	title,
 	start,
@@ -328,69 +264,79 @@ function formatRedirectRulesReport({
 	sitemapPaths
 }) {
 	const hitsOf = rule =>
-		rowsByPath.get(`/docs${rule.source}`)?.redirects ?? 0;
-	const live = rules.filter(rule => !rule.shadowedBy);
-	const ruleHits = live.reduce((sum, rule) => sum + hitsOf(rule), 0);
-	const liveRuleBySource = new Map(live.map(rule => [rule.source, rule]));
+		rule.fires
+			? (rowsByPath.get(`/docs${rule.source}`)?.redirects ?? 0)
+			: 0;
+	const firing = rules.filter(rule => rule.fires);
+	const ruleHits = firing.reduce((sum, rule) => sum + hitsOf(rule), 0);
+	const firingRuleBySource = new Map(firing.map(rule => [rule.source, rule]));
 	const chainOf = rule =>
-		rule.shadowedBy
-			? {hops: [], loop: false}
-			: chainAfter(rule, liveRuleBySource);
-	const chained = live.filter(rule => chainOf(rule).hops.length > 0);
+		rule.fires
+			? chainAfter(rule, firingRuleBySource)
+			: {hops: [], loop: false};
+	const chained = firing.filter(rule => chainOf(rule).hops.length > 0);
 	const longestChain = Math.max(
 		0,
 		...chained.map(rule => chainOf(rule).hops.length)
 	);
-	// Sitemap membership of the rule's own destination; blank when off-site.
-	const sitemapOf = rule => {
-		const landing = landingPath(rule.destination);
-		return landing === null ? '' : sitemapPaths.has(landing) ? 'yes' : 'no';
-	};
-	const destinationUnlisted = live.filter(rule => sitemapOf(rule) === 'no');
+	const sitemapCell = pagePath =>
+		pagePath === null ? '' : sitemapPaths.has(pagePath) ? 'yes' : 'no';
+	const sitemapSource = rule =>
+		sitemapCell(normalizePath(`/docs${requestPath(rule.source)}`));
+	// Blank when the destination leaves the site.
+	const sitemapDestination = rule =>
+		sitemapCell(landingPath(rule.destination));
+	const destinationUnlisted = firing.filter(
+		rule => sitemapDestination(rule) === 'no'
+	);
 	const docsRedirects = [...rowsByPath.entries()]
 		.filter(([pagePath]) => pagePath.startsWith('/docs'))
 		.reduce((sum, [, totals]) => sum + totals.redirects, 0);
 	const sorted = [...rules].sort(
 		(a, b) =>
-			(a.shadowedBy ? 1 : 0) - (b.shadowedBy ? 1 : 0) ||
+			(a.fires ? 0 : 1) - (b.fires ? 0 : 1) ||
 			hitsOf(b) - hitsOf(a) ||
 			a.line - b.line
 	);
 	const lines = [
 		...reportHeader(title, start, end),
-		`- Rules: ${rules.length} in ${path.relative(REPO_ROOT, REDIRECTS_FILE)}; ` +
-			`${live.length} live, ${rules.length - live.length} shadowed by an ` +
-			`earlier rule with the same source (never match), ` +
-			`${live.filter(rule => hitsOf(rule) === 0).length} live with zero hits`,
+		`- Rules: ${rules.length} in src/data/redirects.ts; ${firing.length} ` +
+			`can fire, ${rules.length - firing.length} never do (Hits says why: ` +
+			'an earlier rule has the same source, browsers never send the ' +
+			`source's #fragment, or a .md path is rewritten first); ` +
+			`${firing.filter(rule => hitsOf(rule) === 0).length} can fire but ` +
+			'had zero hits',
 		`- Hits: ${ruleHits} redirects matched a rule, of ${docsRedirects} ` +
 			'redirects on /docs paths (the rest are version and other redirects)',
 		'- Hits count 3xx responses on /docs<Source> with the same filters as ' +
 			'the page views reports; adaptive-sampled estimates.',
-		`- Chains: ${chained.length} live rules redirect to another rule's ` +
+		`- Chains: ${chained.length} firing rules redirect to another rule's ` +
 			`source, so the browser follows more redirects (longest chain: ` +
 			`${longestChain} more). Chain shows the extra hops and where the ` +
 			'user ends up.',
-		`- Sitemap: whether the rule's destination is in ${SITEMAP_URL} ` +
-			`(blank when it leaves the site). ${destinationUnlisted.length} ` +
-			`live rules point at an unlisted page, ${destinationUnlisted.reduce(
+		`- Sitemap source / destination: whether /docs<Source> and the ` +
+			`destination page are in ${SITEMAP_URL} (destination blank when ` +
+			`it leaves the site). ${destinationUnlisted.length} firing rules ` +
+			`point at an unlisted page, ${destinationUnlisted.reduce(
 				(sum, rule) => sum + hitsOf(rule),
 				0
-			)} hits; unless Chain shows a further redirect, on /docs that is ` +
-			'likely a soft 404.',
+			)} hits; unless Chain shows a further redirect, that is a 404.`,
 		'',
-		'| Line | Source | Destination | Hits | Chain | Sitemap |',
-		'| ---: | --- | --- | ---: | --- | --- |',
+		'| Line | Source | Destination | Hits | Chain | Sitemap source | Sitemap destination |',
+		'| ---: | --- | --- | ---: | --- | --- | --- |',
 		...sorted.map(rule => {
 			const {hops, loop} = chainOf(rule);
-			const hits = rule.shadowedBy
-				? `shadowed by line ${rule.shadowedBy}`
-				: hitsOf(rule);
+			const hits = rule.fires
+				? hitsOf(rule)
+				: rule.matchedRuleLine !== null
+					? `line ${rule.matchedRuleLine} matches first`
+					: neverFiresReason(rule);
 			const chain = loop
 				? `LOOP after ${hops.length} more`
 				: hops.length
 					? `${hops.length} more → ${hops.at(-1).destination}`
 					: '';
-			return `| ${rule.line} | ${rule.source} | ${rule.destination} | ${hits} | ${chain} | ${sitemapOf(rule)} |`;
+			return `| ${rule.line} | ${rule.source} | ${rule.destination} | ${hits} | ${chain} | ${sitemapSource(rule)} | ${sitemapDestination(rule)} |`;
 		}),
 		''
 	];
@@ -412,8 +358,8 @@ function formatReport({title, start, end, total, rows}) {
 			'All counts are adaptive-sampled estimates.',
 		`- Sitemap: ${total.sitemapPaths} of ${total.paths} paths are in ` +
 			`${SITEMAP_URL}, with ${total.sitemapRequests} of ${total.requests} ` +
-			'requests. The rest are deleted, unlisted or probe paths; on /docs ' +
-			'they still return 200. The blog sitemap lists only recent posts.',
+			'requests. The rest are deleted, unlisted or probe paths. ' +
+			'The blog sitemap lists only recent posts.',
 		'',
 		'| Path | Requests | Visits | 3xx | 404 | 5xx | Sitemap |',
 		'| --- | ---: | ---: | ---: | ---: | ---: | --- |',