Parallelize and speed up link checking.

.github/workflows/checks.yml

@@ -35,49 +35,38 @@ jobs:
       - name: Build with Jekyll
         run: bundle exec jekyll build
         working-directory: ./docs
-      - name: Set up Python for link checks
-        uses: actions/setup-python@v5
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
         with:
-          python-version: '3.14'
-          cache: 'pip'
-      - name: Install Python deps
-        run: pip install -r requirements.txt
-      - name: Check online links (check_links.py)
-        # `--fallback-extensions html` mirrors what GitHub Pages does at request time:
-        # an extensionless URL like `/FAQ` is served as `/FAQ.html`. This workflow's
-        # Jekyll build runs without --baseurl (no Pages prefix), so no --base-path is
-        # needed -- contrast with jekyll-gh-pages.yml.
+          node-version: '22'
+          cache: 'npm'
+          cache-dependency-path: package-lock.json
+      - name: Install Node.js dependencies
+        run: npm ci
+      - name: Check links (check_links.mjs)
+        # Three passes run in parallel via /sep/:
+        #   1. Online (_site/): --fallback-extensions html mirrors GitHub Pages'
+        #      extensionless-URL behaviour.  No --base-path needed -- this
+        #      workflow builds without --baseurl (contrast jekyll-gh-pages.yml).
+        #   2. Offline (_site-offline/): strict, no extension fallback.  --forbid
+        #      catches any surviving https://docs.twinbasic.com/<path> link the
+        #      offlinify rewrite missed (bare root URL is exempt).
+        #   3. Book (_site-pdf/book.html): --no-fail makes failures informational
+        #      (some links are not yet fully resolved).
         run: >-
-          python scripts/check_links.py
+          node scripts/check_links.mjs
           --offline --include-fragments
           --fallback-extensions html
           --index-files 'index.html,.'
           --root-dir docs/_site
           docs/_site
-      - name: Check offline links (check_links.py)
-        run: >-
-          python scripts/check_links.py
+          /sep/
           --offline --include-fragments
           --index-files index.html
+          --forbid 'https://docs.twinbasic.com'
           --root-dir docs/_site-offline
           docs/_site-offline
-      - name: Check for surviving live-site links in offline tree
-        # Flags any https://docs.twinbasic.com/<path> reference left in
-        # _site-offline/ HTML outside <code>/<pre> blocks. After offlinify
-        # strips the jekyll-seo-tag block, anything surviving is a source
-        # link that points at the live site instead of using a relative or
-        # /tB/... permalink that resolves locally. The bare root URL
-        # (https://docs.twinbasic.com[/]) is exempt -- intentional "go to
-        # the live site" links are allowed.
-        run: python scripts/check_offline_live_links.py
-      - name: Check book links (informational)
-        # Failures do not block the build. The book still has absolute
-        # intra-site URLs that the chapter transform has not yet rewritten
-        # and some fragment anchors that are not yet generated. Tracked here
-        # for visibility until those are fixed.
-        continue-on-error: true
-        run: >-
-          python scripts/check_links.py
-          --offline --include-fragments
+          /sep/
+          --offline --no-fail --include-fragments
           --root-dir docs/_site-pdf
-          docs/_site-pdf/book.html
+          docs/_site-pdf/book.html

.github/workflows/jekyll-gh-pages.yml

@@ -57,72 +57,47 @@ jobs:
         env:
           JEKYLL_ENV: production
           PAGES_REPO_NWO: "${{ github.repository }}"
-      - name: Set up Python for link checks
-        uses: actions/setup-python@v5
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
         with:
-          python-version: '3.14'
-          cache: 'pip'
-      - name: Install Python deps
-        run: pip install -r requirements.txt
-      - name: Check online links (check_links.py)
-        # `--fallback-extensions html` mirrors what GitHub Pages does at request time:
-        # an extensionless URL like `/FAQ` is served as `/FAQ.html`. Without the flag
-        # every pretty permalink on the site would look broken.
-        #
-        # `--base-path` strips the Pages baseurl (e.g. `/twinBASIC-docs`) from absolute
-        # URLs before resolving against `--root-dir`. Equivalent to the `--remap` regex
-        # that lychee used in earlier iterations of this step.
+          node-version: '22'
+          cache: 'npm'
+          cache-dependency-path: package-lock.json
+      - name: Install Node.js dependencies and Chromium
+        # Install npm deps first so the link checks (check_links.mjs)
+        # can use htmlparser2. Chromium download runs in the same step
+        # so the cache hit / miss is one decision.
+        run: |
+          npm ci
+          sudo npx puppeteer browsers install chrome --install-deps
+      - name: Check links (check_links.mjs)
+        # Three passes run in parallel via /sep/:
+        #   1. Online (_site/): --fallback-extensions html mirrors GitHub Pages'
+        #      extensionless-URL behaviour.  --base-path strips the Pages baseurl
+        #      (e.g. `/twinBASIC-docs`) from absolute URLs before resolving.
+        #   2. Offline (_site-offline/): strict, no extension fallback.  --forbid
+        #      catches any surviving https://docs.twinbasic.com/<path> link the
+        #      offlinify rewrite missed (bare root URL is exempt).
+        #   3. Book (_site-pdf/book.html): --no-fail makes failures informational
+        #      (some links are not yet fully resolved).
         run: >-
-          python scripts/check_links.py
+          node scripts/check_links.mjs
           --offline --include-fragments
           --fallback-extensions html
           --index-files 'index.html,.'
           --base-path '${{ steps.pages.outputs.base_path }}'
           --root-dir docs/_site
           docs/_site
-      - name: Check offline links (check_links.py)
-        # Strict check on `_site-offline/`: every link must resolve to an actual file
-        # under `file://`, with no extension fallback. Catches relative links in
-        # markdown sources that point at a permalink that doesn't match the rendered
-        # filename (e.g. `[Foo](Foo/)` when Jekyll wrote `Foo.html`, not
-        # `Foo/index.html`) -- the kind of breakage the online check above hides
-        # behind `--fallback-extensions html`.
-        run: >-
-          python scripts/check_links.py
+          /sep/
           --offline --include-fragments
           --index-files index.html
+          --forbid 'https://docs.twinbasic.com'
           --root-dir docs/_site-offline
           docs/_site-offline
-      - name: Check for surviving live-site links in offline tree
-        # Flags any https://docs.twinbasic.com/<path> reference left in
-        # _site-offline/ HTML outside <code>/<pre> blocks. After offlinify
-        # strips the jekyll-seo-tag block, anything surviving is a source
-        # link that points at the live site instead of using a relative or
-        # /tB/... permalink that resolves locally. The bare root URL
-        # (https://docs.twinbasic.com[/]) is exempt -- intentional "go to
-        # the live site" links are allowed.
-        run: python scripts/check_offline_live_links.py
-      - name: Check book links (informational)
-        # Failures do not block the build. The book still has absolute
-        # intra-site URLs that the chapter transform has not yet rewritten
-        # and some fragment anchors that are not yet generated. Tracked here
-        # for visibility until those are fixed.
-        continue-on-error: true
-        run: >-
-          python scripts/check_links.py
-          --offline --include-fragments
+          /sep/
+          --offline --no-fail --include-fragments
           --root-dir docs/_site-pdf
           docs/_site-pdf/book.html
-      - name: Set up Node.js
-        uses: actions/setup-node@v4
-        with:
-          node-version: '22'
-          cache: 'npm'
-          cache-dependency-path: package-lock.json
-      - name: Install Node.js dependencies and Chromium
-        run: |
-          npm ci
-          sudo npx puppeteer browsers install chrome --install-deps
       - name: Render book PDF
         run: |
           mkdir -p _pdf

WIP.md

@@ -430,7 +430,7 @@ From `docs/`:
 
 - `bundle exec jekyll build` (or `build.bat`) — builds three trees in a single Jekyll run: the online copy at `_site/`, a `file://`-browsable copy at `_site-offline/`, and the sparse pagedjs source at `_site-pdf/`. The offline pass (`_plugins/offlinify.rb`, activated by `also_build_offline: true` in `_config.yml`) adds ~3-5s and the PDF pass (`_plugins/pdfify.rb`, activated by `also_build_pdf: true`) adds <1s on top of the normal ~13s build. The PDF plugin captures `book.html`'s rendered output (the concatenated chapter document built via `_layouts/book-combined.html`) at `:pages, :post_render`, drops the page from `site.pages` at `:site, :post_render` so `_site/book.html` is never written, and at `:site, :post_write` writes the captured bytes into `_site-pdf/book.html` along with `assets/css/print.css`, `assets/css/rouge.css`, and every relative `<img src=>` target -- just what pagedjs needs to render the book PDF. The companion `offline_exclude: [..., book.html]` entry in `_config.yml` keeps `offlinify.rb` from copying book.html into `_site-offline/`: offlinify's per-page hook fires before pdfify's `:site, :post_render` (Jekyll fires every per-page hook before any site-level post-render hook), so during offlinify's pass `book.html` is still in `site.pages` and the exclude is what makes it skip writing the offline copy. When `also_build_pdf: false` the exclude does the same job from a different angle -- pdfify never runs, `book.html` renders normally to `_site/`, and the exclude still keeps it out of `_site-offline/`. After Jekyll's WRITE phase, the offline plugin walks `_site/`, copies binary assets verbatim into `_site-offline/`, and for each HTML and CSS file rewrites every root-absolute `href` / `src` / `url()` to a page-relative path with the resolved file extension (`/FAQ` → `../../FAQ.html`, `/Tutorials/CEF/` → `../../Tutorials/CEF/index.html`). It also patches the offline copy of `assets/js/just-the-docs.js` in two places — `navLink()` to match the active nav entry by resolved DOM `link.href` rather than `document.location.pathname` (the upstream pathname-vs-attribute compare returns no match under `file://`, leaving the sidebar with no `.active` class so the nav appears collapsed on every navigation), and `initSearch()` to read the lunr index from `window.SEARCH_DATA` rather than fetching `search-data.json` over `XMLHttpRequest` (XHR to `file://` resources is blocked by browsers; classic `<script src=>` is not). To support that, the plugin (a) generates `_site-offline/assets/js/search-data.js` once per build by wrapping the rendered `search-data.json` in `window.SEARCH_DATA = {...};`, and (b) injects two `<script>` tags per page right before `just-the-docs.js`: one that sets `window.OFFLINE_SITE_ROOT` to the per-page relative prefix to the offline site root, and one that loads `search-data.js`. The patched `initSearch()` rewrites every `doc.url` from a root-absolute permalink (`/tB/Core/Const`) to a page-relative path (`<OFFLINE_SITE_ROOT>tB/Core/Const.html`) so search-result clicks land on the actual file regardless of which page the user is on.
 - `bundle exec jekyll serve` (or `serve.bat`) — local server at `localhost:4000`. Note that `_site-offline/` is also produced on the initial build, but live-reload only updates `_site/`; manual rebuild needed for offline updates.
-- `check.bat` — link check (offline Lychee against `_site/`).
+- `check.bat` — link check (offline `scripts/check_links.mjs` against `_site/` and `_site-offline/`; the offline pass also runs `--forbid 'https://docs.twinbasic.com'` to catch surviving live-site links).
 - `book.bat` — renders the PDF from `_site-pdf/book.html` via `pagedjs-cli` into `_pdf/book.pdf`. Run `build.bat` first to populate `_site-pdf/`.
 
 The HTML whitespace compression that wraps every page's render chain is handled by `_plugins/html-compress.rb` rather than the just-the-docs theme's `vendor/compress.html` Liquid layout — see [_plugins/html-compress.md](docs/_plugins/html-compress.md) for the full writeup. The Liquid layout's per-page cost in the profile was ~2.4s of Liquid filter dispatch (a `split: " " | join: " "` over the outside-of-`<pre>` content, lowering to a per-page Array allocation of every whitespace-delimited token across 837 pages — millions of small `String` objects). The layout is short-circuited via `compress_html.ignore.envs: all` in `_config.yml`; it then outputs a bare `{{ content }}` and the plugin takes over at `:pages, :post_render` / `:documents, :post_render` with `priority :normal`, doing the same pre-block-protected whitespace collapse via `content.split(PRE_BLOCK_RE).each { |s| s.split(" ").join(" ") }` in C-implemented Ruby. The `:normal` priority is the *middle* tier of a three-level convention across the site's `:post_render` hooks: mutators (`book-href-rewrite`) run at `:high`, this cleanup pass at `:normal`, readers (`pdfify`, `offlinify`) at `:low`. The invariant "compress runs after every mutator and before every reader" therefore holds by construction; no downstream plugin has to be whitespace-aware. Pages whose layout chain doesn't reach `vendor/compress` are gated out via a `:site, :pre_render` precompute that walks `site.layouts[name].data["layout"]` for every layout key and marks the entire compress-reaching chain (default → table_wrappers → vendor/compress) -- jekyll-redirect-from stubs, the SCSS-derived CSS pages, and `assets/js/zzzz-search-data.json` all stay un-gated and pass through verbatim. `book.html` (which uses the minimal `book-combined` layout that has no parent) is *also* outside that chain but is explicitly added to the compress-eligible set at the end of the precompute, so the same whitespace collapse runs on it -- saves paged.js's render-time `WhiteSpaceFilter` ~37k DOM mutations (~28k `textContent` overwrites + ~9k `removeChild` calls) at the cost of ~480 ms once per Jekyll build. Output is byte-identical to the layout-based version: a recursive `diff -rq` of `_site/` against a vendor/compress.html baseline reports zero differences across all ~840 HTML pages, 290 redirect stubs, every CSS / JSON / SVG / image asset. The plugin's correctness depended on two non-obvious details that broke an earlier cut -- the layout-chain walk has to compare against the layout *key* (`"vendor/compress"`) rather than `layout.name` (which carries the `.html` extension), and the per-segment `split(" ").join(" ")` strips trailing whitespace that the Liquid layout's *template* re-adds via its trailing-newline source character, so the plugin captures `content.end_with?("\n")` before the split and re-appends a `\n` after the join. Both regressions surfaced as nonzero `diff -rq` counts during development and are flagged in the plugin's header comment and [_plugins/html-compress.md](docs/_plugins/html-compress.md).
@@ -567,7 +567,7 @@ After a batch of changes, verify the site builds clean and all links resolve. Fr
 build.bat && check.bat
 ```
 
-`check.bat` runs Lychee in offline mode against the built `_site/` tree — it catches broken intra-site links, missing pages, and malformed `redirect_from` entries (the most common breakage when adding new pages or moving content between sections). A clean run is the bar for "ready to commit".
+`check.bat` runs [scripts/check_links.mjs](scripts/check_links.mjs) in offline mode against both `_site/` and `_site-offline/` — it catches broken intra-site links, missing pages, malformed `redirect_from` entries (the most common breakage when adding new pages or moving content between sections), and (via `--forbid 'https://docs.twinbasic.com'` on the offline pass) any extracted link that still points at the live docs site after the offlinify rewrite. A clean run is the bar for "ready to commit".
 
 Requires `build.bat` to have produced an up-to-date `_site/`.
 

docs/Miscellaneous/Documentation Development.md

@@ -201,7 +201,7 @@ To check that none of the internal links in the most recent documentation build
 
     check.bat
 
-This runs three checks: `scripts/check_links.py` against `_site/` (the live tree, in offline mode), the same against `_site-offline/` (the file://-browsable mirror), and `scripts/check_offline_live_links.py` over `_site-offline/` that flags any surviving `https://docs.twinbasic.com/<path>` link --- the offline mirror should not navigate back to the live docs site. The same three checks run in CI on every pull request and on every push to `staging`.
+This runs two checks: `scripts/check_links.mjs` against `_site/` (the live tree, in offline mode), and the same against `_site-offline/` (the file://-browsable mirror) with `--forbid 'https://docs.twinbasic.com'` to also flag any surviving live-site link --- the offline mirror should not navigate back to the live docs site. The same two checks run in CI on every pull request and on every push to `staging`.
 
 ### Building and Local Serving