From 952cad89999da8d849979ff3a3b4083a21e1ecac Mon Sep 17 00:00:00 2001 From: Derek Lewis Date: Thu, 10 Sep 2026 23:39:28 +0000 Subject: [PATCH] =?UTF-8?q?=F0=9F=93=96=E2=9C=A8=EF=BC=9Arender=20the=20SD?= =?UTF-8?q?K's=20API=20reference=20here?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SDK generates its API reference as TypeDoc Markdown rather than as TypeDoc's own HTML, so that this portal can render it in the same documentation layout as everything else. This imports it: a versioned artifact under vendor/sdk-api/ becomes pages below /docs/sdk//api/, with a /docs/sdk/ landing page listing the releases and a package index beside every page. An artifact is release output, and is read as something this repository did not write. Its manifest is validated before anything is read from it, and every path it names has to resolve inside its own directory, through symbolic links as well as lexically. A documentation root with no importable pages, or without the README.md its release root links to, is refused rather than published as an empty release. Navigation URLs are derived from the pages actually imported rather than taken from the artifact, so a URL written into that JSON cannot reach a reader. Imported Markdown is rendered with Liquid disabled, and the rendered body passes an element and attribute allowlist before the layout embeds it: script elements, event handlers and unsafe schemes go, while escaped examples, API anchors and tables stay. TypeDoc's link matcher cannot consume another link opener, so malformed artifact text cannot make it retry a whole suffix at every one of them. Releases are ordered by precedence rather than by the name of their directory, which would have put 10.0.0 above 4.2.0 and 3.0.0. Digits are compared as digits: a release part is bounded by the schema at `\d+` and no further, so past 2^53 two distinct versions would round to one Number and sort as equal. Every page is titled by its own first heading except the package index, which TypeDoc heads with the project name: it is titled Packages, rather than repeating the site under a breadcrumb that already says it. Beside each page are the symbols declared alongside it, which is the reach TypeDoc's own theme gives from its sidebar and which a list of ten package roots did not. The package a page belongs to is opened under its name, its symbols grouped as TypeDoc groups them; the other nine say how many symbols they hold. TypeDoc expands that tree with script, and a page here knows its own package from its front matter, so this needs none. Headings are normalized in prose only. A run of hashes inside a fence is part of an example -- a shell comment, a Markdown sample -- and promoting it rewrote what the example said. Nothing repairs raw HTML on the way in. The generated corpus has none: the package READMEs it comes from were rewritten to carry no centered markup, no shields and no logo, so a helper that fixed image dimensions and closed stray tags would have shipped with nothing left to fix. Navigation naming a page that is not imported is refused rather than published. It was the one malformed thing an artifact could carry and still build: the entry lost its URL and rendered as a dead label. The generated pages sit outside the filename, Markdown, spelling, JSON and EditorConfig checks. Their names are TypeDoc's, taken from the symbols they document, and their prose is the SDK's; a correction made here would be overwritten by the next release. Held to the conventions for authored content, one vendored release fails those five between them. The contract note beside them is written here, and stays checked. The build stops at the step that failed. It ran every script whatever any one returned, which did not matter while that list was only Eleventy. With the import ahead of it, a refused artifact left the generated collection emptied and Eleventy rendered the site without an API reference in it, reporting the failure afterwards. Signed-off-by: Derek Lewis Assisted-by: Codex:gpt-5 Assisted-by: Codex:gpt-6 Assisted-by: Claude-Code:claude-opus-5 --- .ecrc.json | 5 +- .gitignore | 2 + .markdownlint-cli2.jsonc | 6 +- .prettierignore | 4 + _assets/styles/_sdk-api.scss | 143 +++++++ _assets/styles/main.scss | 1 + _layouts/sdk-api.liquid | 111 ++++++ build/shared/sdk-docs.mts | 399 ++++++++++++++++++++ build/shared/sdk-docs.test.mts | 336 +++++++++++++++++ build/shared/sdk-import.test.mts | 202 ++++++++++ build/tasks/compile/build-portal.mts | 11 +- build/tasks/compile/import-sdk-api-docs.mts | 221 +++++++++++ build/tasks/verify/verify-filenames.mts | 14 +- build/tasks/verify/verify-json.mts | 10 +- build/tasks/verify/verify-md.mts | 5 + build/tasks/verify/verify-spelling.mts | 6 + collections/_pages/sdk.html | 33 ++ eleventy.config.mjs | 8 + package-scripts.yml | 1 + package.json | 3 + pnpm-lock.yaml | 67 ++++ project-terms.txt | 2 + vendor/sdk-api/README.md | 72 ++++ 23 files changed, 1656 insertions(+), 6 deletions(-) create mode 100644 _assets/styles/_sdk-api.scss create mode 100644 _layouts/sdk-api.liquid create mode 100644 build/shared/sdk-docs.mts create mode 100644 build/shared/sdk-docs.test.mts create mode 100644 build/shared/sdk-import.test.mts create mode 100644 build/tasks/compile/import-sdk-api-docs.mts create mode 100644 collections/_pages/sdk.html create mode 100644 vendor/sdk-api/README.md diff --git a/.ecrc.json b/.ecrc.json index d12afe614..27b82edee 100644 --- a/.ecrc.json +++ b/.ecrc.json @@ -13,7 +13,10 @@ "collections/_docs/contributing.md", "collections/_docs/security.md", "collections/_docs/support.md", - "collections/_pages/vision.md" + "collections/_pages/vision.md", + "vendor/sdk-api/[^/]+/", + "collections/_sdk-api/", + "_data/sdkApi.json" ], "AllowedContentTypes": [], "PassedFiles": [], diff --git a/.gitignore b/.gitignore index f59404f1a..502cc6b08 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,5 @@ node_modules/ # Ignore generated build products. _site/ /assets/ +/collections/_sdk-api/ +/_data/sdkApi.json diff --git a/.markdownlint-cli2.jsonc b/.markdownlint-cli2.jsonc index 25ac8112a..0358323be 100644 --- a/.markdownlint-cli2.jsonc +++ b/.markdownlint-cli2.jsonc @@ -19,7 +19,11 @@ "collections/_docs/contributing.md", "collections/_docs/security.md", "collections/_docs/support.md", - "collections/_pages/vision.md" + "collections/_pages/vision.md", + // The SDK's API reference: TypeDoc's output, vendored as the release + // published it, and the pages the import task derives from that. + "vendor/sdk-api/*/**", + "collections/_sdk-api/" ], "markdownItPlugins": [ diff --git a/.prettierignore b/.prettierignore index 097d10468..4957b3c16 100644 --- a/.prettierignore +++ b/.prettierignore @@ -16,3 +16,7 @@ collections/_docs/contributing.md collections/_docs/security.md collections/_docs/support.md collections/_pages/vision.md + +# Derived from the vendored SDK API artifact by +# compile.importSdkApiDocs, which rewrites them on every build. +collections/_sdk-api/ diff --git a/_assets/styles/_sdk-api.scss b/_assets/styles/_sdk-api.scss new file mode 100644 index 000000000..f0f722849 --- /dev/null +++ b/_assets/styles/_sdk-api.scss @@ -0,0 +1,143 @@ +@use 'sass:color'; + +// Generated API reference pages use the same reading typography as the +// community documentation. Their one additional element is the navigation +// beside them: ten package roots, with the symbols of whichever package the +// page belongs to opened underneath it, so that a function is one click from +// the ones declared beside it. + +$sdk-api-rule: color.mix($tertiary, $body-bg, 35%); + +.sdk-api .sdk-api-kicker { + margin: 1rem 0 0; + font-family: $font-family-monospace; + font-size: 0.75rem; + font-weight: 700; + color: rgb(231 231 214 / 72%); + text-transform: uppercase; + letter-spacing: 0.08em; +} + +.sdk-api .sdk-api-grid { + display: grid; + grid-template-columns: minmax(0, 65ch) minmax(11rem, 15rem); + gap: clamp(2rem, 5vw, 5rem); + justify-content: center; +} + +.sdk-api .doc-prose { + width: 100%; + margin-inline: 0; +} + +.sdk-api .sdk-api-nav { + align-self: start; + padding-inline-start: 1.25rem; + border-inline-start: 1px solid $sdk-api-rule; +} + +.sdk-api .sdk-api-nav-title { + margin: 0; + font-family: $font-family-monospace; + font-size: 0.75rem; + font-weight: 700; + text-transform: uppercase; + letter-spacing: 0.08em; +} + +.sdk-api .sdk-api-nav-list { + padding: 0; + margin: 0.875rem 0 0; + list-style: none; +} + +.sdk-api .sdk-api-nav-package + .sdk-api-nav-package { + margin-top: 0.5rem; +} + +// The package a page belongs to is the one carrying a list, so it reads as a +// heading over that list rather than as another item in the same column. +.sdk-api .sdk-api-nav-package.is-open { + margin-top: 1.25rem; + margin-bottom: 1.25rem; +} + +.sdk-api .sdk-api-nav-package.is-open > .sdk-api-nav-package-name { + font-weight: 700; +} + +.sdk-api .sdk-api-nav-list a { + color: $doc-link; + text-decoration: none; +} + +.sdk-api .sdk-api-nav-list a:hover, +.sdk-api .sdk-api-nav-list a[aria-current='page'] { + text-decoration: underline; +} + +.sdk-api .sdk-api-nav-list a[aria-current='page'] { + font-weight: 700; +} + +// How many symbols a closed package holds, which its name does not say. +.sdk-api .sdk-api-nav-count { + margin-inline-start: 0.5ch; + font-family: $font-family-monospace; + font-size: 0.75rem; + color: $doc-muted; +} + +// What the open package's symbols are grouped by -- Functions, Classes, and +// so on -- set quietly, since the names beneath are what is being scanned. +.sdk-api .sdk-api-nav-group { + margin: 0.75rem 0 0; + font-size: 0.6875rem; + font-weight: 700; + color: $doc-muted; + text-transform: uppercase; + letter-spacing: 0.08em; +} + +// Indented past the package names, against a rule, so that the column reads +// as one package opened rather than as packages interrupted by a list. +.sdk-api .sdk-api-nav-group, +.sdk-api .sdk-api-nav-symbols { + padding-inline-start: 0.75rem; + border-inline-start: 1px solid $doc-hairline; +} + +.sdk-api .sdk-api-nav-symbols { + margin: 0.25rem 0 0; + font-family: $font-family-monospace; + font-size: 0.8125rem; + list-style: none; +} + +.sdk-api .sdk-api-nav-symbols li + li { + margin-top: 0.25rem; +} + +// A package with eighty-odd symbols makes a column far taller than the prose +// beside it, and navigation that scrolls away is navigation you have to go +// back for. Only where there are two columns to begin with: stacked, it is +// the end of the page and scrolls with it. +@media (width >= 960px) { + .sdk-api .sdk-api-nav { + position: sticky; + top: calc(#{$navbar-height} + 1.5rem); + max-height: calc(100vh - #{$navbar-height} - 3rem); + overflow-y: auto; + overscroll-behavior: contain; + } +} + +@media (width < 960px) { + .sdk-api .sdk-api-grid { + display: block; + } + + .sdk-api .sdk-api-nav { + margin-top: 3rem; + } +} diff --git a/_assets/styles/main.scss b/_assets/styles/main.scss index 8876f0899..a11687425 100644 --- a/_assets/styles/main.scss +++ b/_assets/styles/main.scss @@ -27,6 +27,7 @@ @import 'nav'; @import 'news'; @import 'news-item'; +@import 'sdk-api'; // Optional Bootstrap components (core styles). @import '../../node_modules/bootstrap/scss/utilities'; diff --git a/_layouts/sdk-api.liquid b/_layouts/sdk-api.liquid new file mode 100644 index 000000000..30432028d --- /dev/null +++ b/_layouts/sdk-api.liquid @@ -0,0 +1,111 @@ +--- +layout: default +body_classes: + - docs + - doc-page + - sdk-api +editable: false +--- +{%- assign release = sdkApi.versions | where: 'version', api_version | first -%} +
+
+
+
+ +

API reference · {{ api_version }}

+

{{ title | escape }}

+
+
+ +
+
+
{{ content | sanitizeSdkHtml }}
+ + {%- comment -%} + Every package, with the one this page belongs to opened beneath it to + show the symbols declared alongside it. TypeDoc's own theme keeps + that tree in a sidebar and expands it with script; a page here knows + its own package from its front matter, so the same reach costs none. + A closed package says how many symbols it holds, which its name does + not. + {%- endcomment -%} + {%- if release -%} + + {%- endif -%} +
+
+
+
diff --git a/build/shared/sdk-docs.mts b/build/shared/sdk-docs.mts new file mode 100644 index 000000000..254b3a04c --- /dev/null +++ b/build/shared/sdk-docs.mts @@ -0,0 +1,399 @@ +/** + * @file Turns TypeDoc Markdown paths into stable OpenINF portal URLs. + * @author The OpenINF Authors & Friends + * @license MIT OR Apache-2.0 OR BlueOak-1.0.0 + * @module {type ES6Module} build/shared/sdk-docs + */ + +import { realpath } from 'node:fs/promises'; +import { isAbsolute, relative, resolve } from 'node:path'; +import { + dirname as pathDirname, + extname as pathExtname, + normalize as pathNormalize, + relative as pathRelative, +} from 'node:path/posix'; +import sanitizeHtml from 'sanitize-html'; + +const normalizePath = (input: string) => + pathNormalize(input).replace(/^\.\//, ''); + +/** + * Maps one TypeDoc Markdown filename to its portal-relative source filename. + * `README.md` is a directory index in TypeDoc's tree; the portal gives it an + * actual `index.md` source, so Eleventy can make the corresponding clean URL. + * @param {string} sourcePath A slash-separated path beneath the Markdown root. + * @returns The matching generated source path beneath `collections/_sdk-api`. + */ +export function portalPath(sourcePath: string) { + const normalized = normalizePath(sourcePath); + + if (normalized === 'README.md') return 'index.md'; + if (normalized === 'packages.md') return 'packages/index.md'; + + const packageMatch = normalized.match(/^@openinf\/([^/]+)\/(.*)$/); + if (packageMatch === null) { + throw new Error(`Unexpected TypeDoc path: ${sourcePath}`); + } + + const [, packageName, remainder] = packageMatch; + if (remainder === 'README.md') return `packages/${packageName}/index.md`; + + return `packages/${packageName}/${remainder}`; +} + +/** + * The package a generated page belongs to, named as the navigation names it, + * or an empty string for the two pages that belong to no package. The layout + * reads this to find that package in the navigation tree, so that a symbol + * page can list the symbols beside it rather than only the package roots. + * @param {string} sourcePath A slash-separated path beneath the Markdown root. + * @returns The package's navigation title, or an empty string. + */ +export function packageOf(sourcePath: string) { + return normalizePath(sourcePath).match(/^(@openinf\/[^/]+)\//)?.[1] ?? ''; +} + +/** + * Returns the public URL for one TypeDoc Markdown source path. + * @param {string} version The SDK release version that generated the page. + * @param {string} sourcePath A slash-separated path beneath the Markdown root. + * @returns The portal URL at which Eleventy publishes the page. + */ +export function portalUrl(version: string, sourcePath: string) { + const file = portalPath(sourcePath).replace(/\.md$/, ''); + const path = file.endsWith('/index') ? file.slice(0, -'/index'.length) : file; + const suffix = path === 'index' ? '' : `${path}/`; + + return `/docs/sdk/${version}/api/${suffix}`; +} + +/** + * Reads TypeDoc's first heading after removing the product link and generated + * breadcrumb it puts above every page. The portal supplies those itself. + * @param {string} markdown A generated TypeDoc Markdown page. + * @returns The page title and the Markdown that belongs in the portal body. + */ +export function stripTypeDocChrome(markdown: string) { + let content = markdown.replace(/\r\n/g, '\n'); + + // The root has bold text; child pages link that same text home. Both are + // followed by a thematic break. + content = content.replace( + /^(?:\[\*{1,2}OpenINF\*{1,2}\]\([^\n)]*\)|\*{1,2}OpenINF\*{1,2})\n\n\*\*\*\n\n/, + '' + ); + // A child page's generated breadcrumb would duplicate the portal's trail. + content = content.replace(/^\[OpenINF\]\([^\n)]*\)(?:\s*\/[^\n]*)?\n\n/, ''); + + const heading = content.match(/^# ([^\n]+)\n+/); + const htmlHeading = content.match(/^]*)?>([^<]+)<\/h1>\n+/); + + if (heading !== null) { + return { + title: heading[1], + content: content.slice(heading[0].length), + }; + } + + if (htmlHeading !== null) { + return { + title: htmlHeading[1], + content: content.slice(htmlHeading[0].length), + }; + } + + throw new Error('Generated TypeDoc page has no h1'); +} + +/** + * Rewrites relative TypeDoc Markdown links to their portal URLs. TypeDoc's + * links name `.md` files, whereas the portal publishes clean directory URLs. + * @param {string} markdown Generated TypeDoc Markdown. + * @param {string} sourcePath The current page's source path beneath the Markdown root. + * @param {Map} urls Public URLs keyed by TypeDoc source path. + * @returns Markdown whose internal document links point at the portal. + */ +export function rewriteTypeDocLinks( + markdown: string, + sourcePath: string, + urls: Map +) { + // A destination cannot consume another link opener. Otherwise an + // unterminated run of `](` makes the matcher retry the entire suffix at + // every opener, producing quadratic work on malformed artifact text. + const rewrite = (prose: string) => + prose.replace( + /(\]\()(?]+>?)(\))/g, + (whole, before: string, target: string, after: string) => { + const wrapped = target.startsWith('<') && target.endsWith('>'); + const destination = wrapped ? target.slice(1, -1) : target; + const hash = destination.indexOf('#'); + const file = hash === -1 ? destination : destination.slice(0, hash); + const fragment = hash === -1 ? '' : destination.slice(hash); + + if ( + !file.endsWith('.md') || + /^[a-z][a-z\d+.-]*:/i.test(file) || + file.startsWith('/') + ) + return whole; + + const resolved = normalizePath( + pathNormalize(`${pathDirname(sourcePath)}/${file}`) + ); + const url = urls.get(resolved); + if (url === undefined) { + throw new Error( + `${sourcePath} links to generated Markdown that is not imported: ${file}` + ); + } + + return `${before}${url}${fragment}${after}`; + } + ); + + // Generated signatures and examples are fenced; inline examples use one + // backtick. A link-shaped string in either is code, not navigation. + return markdown + .split(/(```[\s\S]*?```|`[^`\n]*`)/) + .map((part, index) => (index % 2 === 0 ? rewrite(part) : part)) + .join(''); +} + +/** + * Sanitizes the rendered SDK body before the portal layout embeds it. + * Running after Markdown rendering also covers Markdown-generated URLs; + * code examples have already been escaped and remain readable text. + * @param {string} html Rendered API documentation, without the portal shell. + * @returns HTML restricted to documentation elements and safe attributes. + */ +export function sanitizeSdkHtml(html: string) { + return sanitizeHtml(html, { + allowedTags: [ + 'a', + 'abbr', + 'b', + 'blockquote', + 'br', + 'code', + 'dd', + 'del', + 'details', + 'div', + 'dl', + 'dt', + 'em', + 'h1', + 'h2', + 'h3', + 'h4', + 'h5', + 'h6', + 'hr', + 'i', + 'img', + 'li', + 'ol', + 'p', + 'pre', + 's', + 'span', + 'strong', + 'sub', + 'summary', + 'sup', + 'table', + 'tbody', + 'td', + 'th', + 'thead', + 'tr', + 'ul', + ], + allowedAttributes: { + '*': ['id', 'class', 'title', 'aria-label', 'aria-describedby'], + a: ['href', 'name', 'rel'], + img: ['src', 'alt', 'width', 'height'], + ol: ['start'], + th: ['colspan', 'rowspan', 'scope', 'align'], + td: ['colspan', 'rowspan', 'align'], + details: ['open'], + }, + allowedSchemes: ['https', 'http', 'mailto'], + allowedSchemesByTag: { img: ['https', 'http'] }, + allowProtocolRelative: false, + }); +} + +/** + * Resolves an artifact member and checks physical as well as lexical bounds. + * @param {string} root Directory containing the trusted artifact boundary. + * @param {string} value Relative member path, possibly containing symlinks. + * @returns The contained real path to use for subsequent reads. + */ +export async function artifactFile(root: string, value: string) { + if (isAbsolute(value)) throw new Error(`${value} must be relative`); + const directory = await realpath(root); + const candidate = resolve(directory, value); + const check = (file: string) => { + const fromRoot = relative(directory, file); + if ( + fromRoot === '..' || + fromRoot.startsWith('../') || + fromRoot.startsWith('..\\') || + isAbsolute(fromRoot) + ) { + throw new Error(`${value} escapes ${directory}`); + } + }; + check(candidate); + const file = await realpath(candidate); + check(file); + return file; +} + +/** + * Keeps generated subsection headings valid below the portal-provided h1. + * A few hand-authored comments begin at h3 before TypeDoc emits an h2. + * @param {string} markdown Generated Markdown body. + * @returns Markdown with leading h3 sections promoted to h2. + */ +export function normalizeHeadingHierarchy(markdown: string) { + let hasH2 = false; + + const promote = (prose: string) => + prose.replace(/^#{2,3} .+$/gm, (heading) => { + if (heading.startsWith('## ')) { + hasH2 = true; + return heading; + } + + return hasH2 ? heading : `## ${heading.slice(4)}`; + }); + + // A run of hashes inside a fence is part of an example -- a shell comment, + // a Markdown sample -- and promoting it would rewrite what the example + // says. Split the same way the link rewriter does, and read only the prose. + return markdown + .split(/(```[\s\S]*?```|`[^`\n]*`)/) + .map((part, index) => (index % 2 === 0 ? promote(part) : part)) + .join(''); +} + +/** + * Rejects an artifact path which would leave its declared documentation root. + * @param {string} root The artifact's documentation root. + * @param {string} candidate A path that claims to sit under it. + * @returns A normalized path beneath `root`. + */ +export function pathInside(root: string, candidate: string) { + const relative = pathRelative(root, candidate); + if (relative === '' || (!relative.startsWith('../') && relative !== '..')) { + return normalizePath(candidate); + } + + throw new Error(`${candidate} is outside ${root}`); +} + +/** + * Compares two runs of digits as numbers, without making them into any. What + * the artifact schema accepts is `\\d+`, which is not bounded, and past 2^53 + * two distinct versions round to one Number and compare as equal. Leading + * zeros go first, since they lengthen a string without changing its value. + * @param {string} a One run of digits. + * @param {string} b Another run of digits. + * @returns A negative number when `a` is the smaller. + */ +function compareDigits(a: string, b: string) { + const left = a.replace(/^0+(?=\d)/, ''); + const right = b.replace(/^0+(?=\d)/, ''); + + if (left.length !== right.length) return left.length - right.length; + + return left < right ? -1 : left > right ? 1 : 0; +} + +/** + * Splits a release version into the parts that decide its precedence. + * Build metadata takes no part in it, so it is dropped before anything else + * is read: left in, `3.1.1+build.1` would have `1+build` for its patch, and + * comparing that to a number answers neither larger nor smaller. + * @param {string} version A release version the artifact schema accepts. + * @returns Its numeric release parts and its prerelease identifiers. + */ +function releaseParts(version: string) { + const precedence = version.split('+', 1)[0] ?? version; + const at = precedence.indexOf('-'); + + return { + release: (at === -1 ? precedence : precedence.slice(0, at)).split('.'), + prerelease: at === -1 ? [] : precedence.slice(at + 1).split('.'), + }; +} + +/** + * Orders release versions newest first, by precedence rather than by name. + * The portal lists releases in the order it imports them, and importing them + * in the order their directory names sort would put 10.0.0 before 3.0.0. + * @param {string} a One release version. + * @param {string} b Another release version. + * @returns A negative number when `a` is the newer release. + */ +export function compareVersions(a: string, b: string) { + const left = releaseParts(a); + const right = releaseParts(b); + + for (const [index, part] of left.release.entries()) { + const difference = compareDigits(right.release[index] ?? '0', part); + if (difference !== 0) return difference; + } + + // A release outranks every prerelease of the same version. + if (left.prerelease.length === 0) { + return right.prerelease.length === 0 ? 0 : -1; + } + if (right.prerelease.length === 0) return 1; + + const numeric = /^\d+$/; + const length = Math.max(left.prerelease.length, right.prerelease.length); + + for (let index = 0; index < length; index += 1) { + const ours = left.prerelease[index]; + const theirs = right.prerelease[index]; + + // A prerelease that runs out of identifiers first is the older one. + if (ours === undefined) return 1; + if (theirs === undefined) return -1; + if (ours === theirs) continue; + + if (numeric.test(ours) && numeric.test(theirs)) { + return compareDigits(theirs, ours); + } + + // Numeric identifiers rank below alphanumeric ones. + if (numeric.test(ours)) return 1; + if (numeric.test(theirs)) return -1; + + return ours < theirs ? 1 : -1; + } + + return 0; +} + +/** + * The title the portal gives one generated page. TypeDoc heads its package + * index with the project name, which on the portal reads as the name of the + * site rather than as what the page is; every other page names itself. + * @param {string} sourcePath A slash-separated path beneath the Markdown root. + * @param {string} heading The page's own first heading. + * @returns The title to publish the page under. + */ +export function portalTitle(sourcePath: string, heading: string) { + return normalizePath(sourcePath) === 'packages.md' ? 'Packages' : heading; +} + +/** Whether a file is one TypeDoc Markdown page the portal should import. */ +export function isApiMarkdown(sourcePath: string) { + return pathExtname(sourcePath) === '.md' && !sourcePath.startsWith('_media/'); +} diff --git a/build/shared/sdk-docs.test.mts b/build/shared/sdk-docs.test.mts new file mode 100644 index 000000000..2e2904b0c --- /dev/null +++ b/build/shared/sdk-docs.test.mts @@ -0,0 +1,336 @@ +/** + * @file Tests for TypeDoc Markdown portal helpers. + * @author The OpenINF Authors & Friends + * @license MIT OR Apache-2.0 OR BlueOak-1.0.0 + * @module {type ES6Module} build/shared/sdk-docs.test + */ + +import { + deepStrictEqual, + doesNotMatch, + match, + throws, +} from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { describe, test } from 'node:test'; +import { + compareVersions, + isApiMarkdown, + normalizeHeadingHierarchy, + packageOf, + pathInside, + portalPath, + portalTitle, + portalUrl, + rewriteTypeDocLinks, + sanitizeSdkHtml, + stripTypeDocChrome, +} from '@openinf/portal/build/sdk-docs'; + +describe('portalPath', () => { + test('turns TypeDoc indexes into portal indexes', () => { + deepStrictEqual(portalPath('README.md'), 'index.md'); + deepStrictEqual(portalPath('packages.md'), 'packages/index.md'); + deepStrictEqual( + portalPath('@openinf/util/README.md'), + 'packages/util/index.md' + ); + }); + + test('keeps a symbol below its package', () => { + deepStrictEqual( + portalPath('@openinf/util/functions/isString.md'), + 'packages/util/functions/isString.md' + ); + }); +}); + +describe('portalUrl', () => { + test('gives indexes clean trailing-slash URLs', () => { + deepStrictEqual(portalUrl('3.0.0', 'README.md'), '/docs/sdk/3.0.0/api/'); + deepStrictEqual( + portalUrl('3.0.0', '@openinf/util/README.md'), + '/docs/sdk/3.0.0/api/packages/util/' + ); + }); + + test('gives a symbol its own URL', () => { + deepStrictEqual( + portalUrl('3.0.0', '@openinf/util/functions/isString.md'), + '/docs/sdk/3.0.0/api/packages/util/functions/isString/' + ); + }); +}); + +describe('stripTypeDocChrome', () => { + test('keeps only a child page’s consumer content', () => { + deepStrictEqual( + stripTypeDocChrome( + '[**OpenINF**](../../README.md)\n\n***\n\n[OpenINF](../../packages.md) / @openinf/util\n\n# @openinf/util\n\nUseful things.\n' + ), + { title: '@openinf/util', content: 'Useful things.\n' } + ); + }); + + test('also handles the root page', () => { + deepStrictEqual( + stripTypeDocChrome('**OpenINF**\n\n***\n\n# OpenINF\n\nWelcome.\n'), + { title: 'OpenINF', content: 'Welcome.\n' } + ); + }); + + test('handles a package readme with an HTML h1', () => { + deepStrictEqual( + stripTypeDocChrome( + '[**OpenINF**](../../README.md)\n\n***\n\n[OpenINF](../../packages.md) / @openinf/util-errors\n\n

@openinf/util-errors

\n\nIntro.\n' + ), + { title: '@openinf/util-errors', content: 'Intro.\n' } + ); + }); +}); + +describe('rewriteTypeDocLinks', () => { + test('maps a relative Markdown link and keeps its fragment', () => { + const urls = new Map([ + ['@openinf/util/README.md', '/docs/sdk/3.0.0/api/packages/util/'], + [ + '@openinf/util/functions/isString.md', + '/docs/sdk/3.0.0/api/packages/util/functions/isString/', + ], + ]); + + deepStrictEqual( + rewriteTypeDocLinks( + 'See [isString](functions/isString.md#examples).', + '@openinf/util/README.md', + urls + ), + 'See [isString](/docs/sdk/3.0.0/api/packages/util/functions/isString/#examples).' + ); + }); + + test('leaves external links and code alone', () => { + const urls = new Map(); + deepStrictEqual( + rewriteTypeDocLinks( + 'See [TypeDoc](https://typedoc.org) and `](README.md)`.', + 'README.md', + urls + ), + 'See [TypeDoc](https://typedoc.org) and `](README.md)`.' + ); + }); + + test('refuses a link to an omitted generated page', () => { + throws( + () => rewriteTypeDocLinks('[no](missing.md)', 'README.md', new Map()), + /not imported/ + ); + }); + + test('leaves absolute Markdown URLs untouched', () => { + const input = '[source](https://example.com/README.md#intro)'; + deepStrictEqual(rewriteTypeDocLinks(input, 'README.md', new Map()), input); + }); + + test('finishes the CodeQL repeated-link input within a bounded process', () => { + // A child process timeout can interrupt a synchronous regexp. A test + // timeout in the same process cannot. The old expression takes minutes. + execFileSync( + process.execPath, + [ + '--input-type=module', + '-e', + ` + import { rewriteTypeDocLinks } from ${JSON.stringify(new URL('./sdk-docs.mts', import.meta.url).href)}; + const input = '](' + '](!'.repeat(300_000); + if (rewriteTypeDocLinks(input, 'README.md', new Map()) !== input) { + throw new Error('Malformed link text changed'); + } + `, + ], + { timeout: 10_000 } + ); + }); +}); + +describe('artifact safety', () => { + test('keeps an artifact path inside its root', () => { + deepStrictEqual( + pathInside( + 'vendor/sdk-api/3.0.0/docs', + 'vendor/sdk-api/3.0.0/docs/README.md' + ), + 'vendor/sdk-api/3.0.0/docs/README.md' + ); + throws(() => + pathInside( + 'vendor/sdk-api/3.0.0/docs', + 'vendor/sdk-api/3.0.0/manifest.json' + ) + ); + }); + + test('does not make generated maintainer documents API pages', () => { + deepStrictEqual(isApiMarkdown('@openinf/util/README.md'), true); + deepStrictEqual(isApiMarkdown('_media/RELEASING.md'), false); + }); + + test('leaves a run of hashes inside an example alone', () => { + // A shell comment in a fence is what the example says, not a section of + // the page, and promoting it would rewrite the example. + const markdown = [ + 'Prose before any heading.', + '', + '```sh', + '### output', + '```', + '', + '## Parameters', + ].join('\n'); + match(normalizeHeadingHierarchy(markdown), /### output/); + }); + + test('promotes an orphaned h3 below the portal page title', () => { + deepStrictEqual( + normalizeHeadingHierarchy( + '### Explanation\n\nBody.\n\n## Parameters\n\n### Value' + ), + '## Explanation\n\nBody.\n\n## Parameters\n\n### Value' + ); + }); +}); + +describe('rendered SDK HTML', () => { + test('removes executable markup and unsafe URL schemes', () => { + const result = sanitizeSdkHtml( + [ + '', + '', + '', + '

Text

', + '', + 'Unsafe', + '', + 'Protocol relative', + ].join('') + ); + doesNotMatch( + result, + /script|style|iframe|svg|onclick|onload|onerror|alert\(1\)|data:|href=/i + ); + match(result, /

Text<\/p>/); + match(result, /src="https:\/\/example.com\/image.png"/); + }); + + test('preserves API anchors, tables, formatting, and escaped examples', () => { + const input = + '

Example

' + + '
<script>example</script>
' + + '' + + '
Type
T
'; + deepStrictEqual(sanitizeSdkHtml(input), input); + }); +}); + +describe('compareVersions', () => { + test('orders releases newest first rather than by name', () => { + deepStrictEqual( + ['3.0.0', '10.0.0', '4.2.0', '3.1.0', '3.0.1'].sort(compareVersions), + ['10.0.0', '4.2.0', '3.1.0', '3.0.1', '3.0.0'] + ); + }); + + test('ranks a release above its own prereleases', () => { + deepStrictEqual( + ['3.1.0-next.0', '3.1.0', '3.0.0', '3.1.0-next.10'].sort(compareVersions), + ['3.1.0', '3.1.0-next.10', '3.1.0-next.0', '3.0.0'] + ); + }); + + test('ranks a longer prerelease above the prefix it extends', () => { + deepStrictEqual( + ['3.1.0-alpha', '3.1.0-alpha.1', '3.1.0-beta'].sort(compareVersions), + ['3.1.0-beta', '3.1.0-alpha.1', '3.1.0-alpha'] + ); + }); + + test('ignores build metadata, which carries no precedence', () => { + deepStrictEqual(['3.1.0', '3.1.1+build.1'].sort(compareVersions), [ + '3.1.1+build.1', + '3.1.0', + ]); + deepStrictEqual(compareVersions('3.1.0+a', '3.1.0+b'), 0); + deepStrictEqual( + ['3.1.0-next.1+build.9', '3.1.0-next.2'].sort(compareVersions), + ['3.1.0-next.2', '3.1.0-next.1+build.9'] + ); + }); + + test('ranks a numeric prerelease identifier below an alphanumeric one', () => { + deepStrictEqual(['3.1.0-1', '3.1.0-alpha'].sort(compareVersions), [ + '3.1.0-alpha', + '3.1.0-1', + ]); + }); +}); + +describe('packageOf', () => { + test('names the package a symbol page belongs to', () => { + deepStrictEqual( + packageOf('@openinf/util/functions/isString.md'), + '@openinf/util' + ); + deepStrictEqual(packageOf('@openinf/util/README.md'), '@openinf/util'); + }); + + test('leaves the pages that belong to no package unnamed', () => { + deepStrictEqual(packageOf('README.md'), ''); + deepStrictEqual(packageOf('packages.md'), ''); + }); +}); + +describe('comparing versions past what a Number holds', () => { + // The artifact schema bounds a release part at `\\d+` and no further, so + // two versions can differ by one and still round to the same Number. + test('keeps two releases apart beyond 2^53', () => { + deepStrictEqual( + ['9007199254740992.0.0', '9007199254740993.0.0'].sort(compareVersions), + ['9007199254740993.0.0', '9007199254740992.0.0'] + ); + }); + + test('keeps two prerelease identifiers apart beyond 2^53', () => { + deepStrictEqual( + ['1.0.0-9007199254740992', '1.0.0-9007199254740993'].sort( + compareVersions + ), + ['1.0.0-9007199254740993', '1.0.0-9007199254740992'] + ); + }); + + test('reads a leading zero as the number it does not change', () => { + deepStrictEqual(compareVersions('1.0.0', '01.0.0'), 0); + deepStrictEqual(compareVersions('01.0.0', '2.0.0') > 0, true); + }); +}); + +describe('portalTitle', () => { + test('names the package index rather than repeating the project', () => { + deepStrictEqual(portalTitle('packages.md', 'OpenINF'), 'Packages'); + }); + + test('leaves every other page titled by its own heading', () => { + deepStrictEqual( + portalTitle( + '@openinf/util/functions/isString.md', + 'Function: isString()' + ), + 'Function: isString()' + ); + deepStrictEqual( + portalTitle('README.md', 'OpenINF API reference'), + 'OpenINF API reference' + ); + }); +}); diff --git a/build/shared/sdk-import.test.mts b/build/shared/sdk-import.test.mts new file mode 100644 index 000000000..9b449f87d --- /dev/null +++ b/build/shared/sdk-import.test.mts @@ -0,0 +1,202 @@ +/** + * @file Regression tests for the SDK artifact import boundary. + * @author The OpenINF Authors & Friends + * @license MIT OR Apache-2.0 OR BlueOak-1.0.0 + * @module {type ES6Module} build/shared/sdk-import.test + */ + +import { deepStrictEqual, match, rejects } from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import { + mkdir, + mkdtemp, + readFile, + rm, + symlink, + writeFile, +} from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { test } from 'node:test'; +import { fileURLToPath } from 'node:url'; +import { promisify } from 'node:util'; + +const execute = promisify(execFile); +const task = fileURLToPath( + new URL('../tasks/compile/import-sdk-api-docs.mts', import.meta.url) +); + +/** + * Creates a disposable artifact and runs the real command in its workspace. + * @returns Fixture paths, runner, and cleanup for one isolated test. + */ +async function fixture() { + const cwd = await mkdtemp(join(tmpdir(), 'sdk-import-test-')); + const root = join(cwd, 'vendor/sdk-api/3.0.0'); + await mkdir(join(root, 'docs'), { recursive: true }); + await mkdir(join(cwd, '_data')); + const manifest = { + schemaVersion: 1, + version: '3.0.0', + commit: '0123456789abcdef', + generatedAt: '2026-09-07T00:00:00.000Z', + docsPath: 'docs', + navigationPath: 'docs/navigation.json', + }; + await writeFile(join(root, 'manifest.json'), JSON.stringify(manifest)); + await writeFile(join(root, 'docs/navigation.json'), '[]'); + await writeFile(join(root, 'docs/README.md'), '# API\n\n{{ page.url }}\n'); + return { + cwd, + root, + manifest, + run: () => execute(process.execPath, [task], { cwd }), + cleanup: () => rm(cwd, { recursive: true, force: true }), + }; +} + +test('imports a valid artifact with a stable URL and no Liquid evaluation', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await f.run(); + const page = await readFile( + join(f.cwd, 'collections/_sdk-api/3.0.0/index.md'), + 'utf8' + ); + match(page, /permalink: "\/docs\/sdk\/3.0.0\/api\/"/); + match(page, /templateEngineOverride: md/); + match(page, /\{\{ page.url \}\}/); + const data = JSON.parse( + await readFile(join(f.cwd, '_data/sdkApi.json'), 'utf8') + ); + deepStrictEqual(data.versions[0].root, '/docs/sdk/3.0.0/api/'); +}); + +test('rejects a navigation symlink outside the artifact without publishing metadata', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await writeFile(join(f.cwd, 'outside.json'), '[]'); + await rm(join(f.root, 'docs/navigation.json')); + await symlink( + join(f.cwd, 'outside.json'), + join(f.root, 'docs/navigation.json') + ); + await rejects(f.run(), /escapes/); + await rejects(readFile(join(f.cwd, '_data/sdkApi.json')), { code: 'ENOENT' }); +}); + +test('rejects an escaping docs directory symlink', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await mkdir(join(f.cwd, 'outside-docs')); + await symlink(join(f.cwd, 'outside-docs'), join(f.root, 'linked-docs')); + await writeFile( + join(f.root, 'manifest.json'), + JSON.stringify({ ...f.manifest, docsPath: 'linked-docs' }) + ); + await rejects(f.run(), /escapes/); +}); + +test('accepts a navigation symlink contained in the artifact', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await writeFile(join(f.root, 'navigation.json'), '[]'); + await rm(join(f.root, 'docs/navigation.json')); + await symlink('../navigation.json', join(f.root, 'docs/navigation.json')); + await f.run(); +}); + +test('does not publish a URL supplied by navigation data', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await writeFile( + join(f.root, 'docs/navigation.json'), + JSON.stringify([ + { title: 'API', url: 'javascript:alert(1)' }, + { title: 'Home', path: 'README.md', url: 'javascript:alert(1)' }, + ]) + ); + await f.run(); + const data = JSON.parse( + await readFile(join(f.cwd, '_data/sdkApi.json'), 'utf8') + ); + deepStrictEqual(data.versions[0].navigation, [ + { title: 'API' }, + { title: 'Home', path: 'README.md', url: '/docs/sdk/3.0.0/api/' }, + ]); +}); + +test('rejects a file used as docsPath', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await writeFile( + join(f.root, 'manifest.json'), + JSON.stringify({ ...f.manifest, docsPath: 'manifest.json' }) + ); + await rejects(f.run(), /must be a directory/); +}); + +test('rejects docs containing only excluded maintainer Markdown', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await rm(join(f.root, 'docs/README.md')); + await mkdir(join(f.root, 'docs/_media')); + await writeFile(join(f.root, 'docs/_media/README.md'), '# Maintainers\n'); + await rejects(f.run(), /no importable API Markdown/); + await rejects(readFile(join(f.cwd, '_data/sdkApi.json')), { code: 'ENOENT' }); +}); + +test('rejects a corpus without the page its release root links to', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await rm(join(f.root, 'docs/README.md')); + await writeFile(join(f.root, 'docs/packages.md'), '# Packages\n'); + await rejects(f.run(), /missing its API root README.md/); +}); + +test('refuses navigation that names a page it is not importing', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await writeFile( + join(f.root, 'docs/navigation.json'), + JSON.stringify([ + { title: 'Home', path: 'README.md' }, + { title: 'Ghost', path: '@openinf/util/functions/doesNotExist.md' }, + ]) + ); + await rejects(f.run(), /not imported/); + await rejects(readFile(join(f.cwd, '_data/sdkApi.json')), { code: 'ENOENT' }); +}); + +test('lists several releases newest first rather than by directory name', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + const second = join(f.cwd, 'vendor/sdk-api/10.0.0'); + await mkdir(join(second, 'docs'), { recursive: true }); + await writeFile( + join(second, 'manifest.json'), + JSON.stringify({ ...f.manifest, version: '10.0.0' }) + ); + await writeFile(join(second, 'docs/navigation.json'), '[]'); + await writeFile(join(second, 'docs/README.md'), '# API\n'); + await f.run(); + const data = JSON.parse( + await readFile(join(f.cwd, '_data/sdkApi.json'), 'utf8') + ); + deepStrictEqual( + data.versions.map((release: { version: string }) => release.version), + ['10.0.0', '3.0.0'] + ); +}); + +test('titles the package index rather than repeating the project name', async (t) => { + const f = await fixture(); + t.after(f.cleanup); + await writeFile(join(f.root, 'docs/packages.md'), '# OpenINF\n'); + await f.run(); + const page = await readFile( + join(f.cwd, 'collections/_sdk-api/3.0.0/packages/index.md'), + 'utf8' + ); + match(page, /title: "Packages"/); +}); diff --git a/build/tasks/compile/build-portal.mts b/build/tasks/compile/build-portal.mts index 72f3f8713..ca21e93f6 100644 --- a/build/tasks/compile/build-portal.mts +++ b/build/tasks/compile/build-portal.mts @@ -15,7 +15,7 @@ process.env.ELEVENTY_ENV = 'production'; // Eleventy compiles the stylesheet and copies the other assets itself, so // there is nothing left to run ahead of it. -const scripts = ['eleventy']; +const scripts = ['nps compile.importSdkApiDocs', 'eleventy']; // Only siteify health files in _this_ task if they're missing. if (!existsSync('collections/_docs/support.md')) { @@ -27,8 +27,15 @@ if (!existsSync('collections/_docs/support.md')) { if (existsSync(PATHS.siteDir)) rmSync(PATHS.siteDir, { recursive: true, force: true }); +// Each script here is the previous one's input: importing the SDK artifact +// writes the collection Eleventy then renders. Carrying on past a failure +// would build the site from whatever the failed step managed to write, and +// still report the failure afterwards -- so the first one ends the build. for (const element of scripts) { exitCode = await exec(element); - if (exitCode !== 0) process.exitCode = exitCode; + if (exitCode !== 0) { + process.exitCode = exitCode; + break; + } } diff --git a/build/tasks/compile/import-sdk-api-docs.mts b/build/tasks/compile/import-sdk-api-docs.mts new file mode 100644 index 000000000..428867cc1 --- /dev/null +++ b/build/tasks/compile/import-sdk-api-docs.mts @@ -0,0 +1,221 @@ +/** + * @file Imports a versioned TypeDoc Markdown artifact into the portal. + * @author The OpenINF Authors & Friends + * @license MIT OR Apache-2.0 OR BlueOak-1.0.0 + * @module {type ES6Module} build/tasks/compile/import-sdk-api-docs + */ + +import { existsSync } from 'node:fs'; +import { + mkdir, + readdir, + readFile, + rm, + stat, + writeFile, +} from 'node:fs/promises'; +import { dirname as pathDirname, join as pathJoin, relative } from 'node:path'; +import { + artifactFile, + compareVersions, + isApiMarkdown, + normalizeHeadingHierarchy, + packageOf, + portalPath, + portalTitle, + portalUrl, + rewriteTypeDocLinks, + stripTypeDocChrome, +} from '@openinf/portal/build/sdk-docs'; +import { glob } from '@openinf/portal/build/utils'; +import { z } from 'zod'; + +const artifactDir = 'vendor/sdk-api'; +const collectionDir = 'collections/_sdk-api'; +const dataPath = '_data/sdkApi.json'; + +const Artifact = z.object({ + schemaVersion: z.literal(1), + version: z.string().regex(/^\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?$/), + commit: z.string().regex(/^[0-9a-f]{7,64}$/), + generatedAt: z.string().datetime(), + docsPath: z.string().min(1), + navigationPath: z.string().min(1), +}); + +type NavigationNode = { + title: string; + kind?: number; + path?: string; + isDeprecated?: boolean; + children?: NavigationNode[]; + url?: string; +}; + +// Drop fields the artifact does not own, especially precomputed public URLs. +const Navigation: z.ZodType = z.array( + z.object({ + title: z.string(), + kind: z.number().optional(), + path: z.string().optional(), + isDeprecated: z.boolean().optional(), + children: z.lazy(() => Navigation).optional(), + }) +); + +const sourcePath = (docsPath: string, file: string) => + relative(docsPath, file).replaceAll('\\', '/'); + +const readArtifact = async (directory: string) => { + const root = await artifactFile(artifactDir, directory); + const manifestPath = await artifactFile(root, 'manifest.json'); + const parsed = Artifact.safeParse( + JSON.parse(await readFile(manifestPath, 'utf8')) + ); + + if (!parsed.success) + throw new Error(`Invalid ${manifestPath}: ${parsed.error}`); + if (parsed.data.version !== directory) { + throw new Error(`${manifestPath} version does not match its directory`); + } + + return { ...parsed.data, directory: root }; +}; + +const withFrontmatter = ( + title: string, + version: string, + source: string, + permalink: string, + content: string +) => + [ + '---', + `title: ${JSON.stringify(title)}`, + 'layout: sdk-api.liquid', + 'templateEngineOverride: md', + `api_version: ${JSON.stringify(version)}`, + `api_package: ${JSON.stringify(packageOf(source))}`, + `api_source: ${JSON.stringify(source)}`, + `permalink: ${JSON.stringify(permalink)}`, + 'editable: false', + '---', + '', + content.trim(), + '', + ].join('\n'); + +const addUrls = ( + nodes: NavigationNode[], + urls: Map, + where: string +): NavigationNode[] => + nodes.map((node) => { + // Every other malformed thing in an artifact is refused rather than + // worked around, and this was the exception: a path naming no imported + // page left `url` undefined, `JSON.stringify` dropped the key, and the + // build went on to publish a navigation entry that goes nowhere. + if (node.path !== undefined && !urls.has(node.path)) { + throw new Error( + `${where} names a page that is not imported: ${node.path}` + ); + } + + return { + ...node, + ...(node.path === undefined ? {} : { url: urls.get(node.path) }), + ...(node.children === undefined + ? {} + : { children: addUrls(node.children, urls, where) }), + }; + }); + +await rm(collectionDir, { recursive: true, force: true }); +await rm(dataPath, { force: true }); + +if (!existsSync(artifactDir)) { + console.info('No SDK API artifacts to import.'); + process.exit(); +} + +const entries = await readdir(artifactDir, { withFileTypes: true }); +const artifactDirectories = entries + .filter((entry) => entry.isDirectory()) + .map((entry) => entry.name) + .sort(compareVersions); + +if (artifactDirectories.length === 0) { + console.info('No SDK API artifacts to import.'); + process.exit(); +} + +const imported = []; + +for (const directory of artifactDirectories) { + const artifact = await readArtifact(directory); + const docsPath = await artifactFile(artifact.directory, artifact.docsPath); + const navigationPath = await artifactFile( + artifact.directory, + artifact.navigationPath + ); + + if (!(await stat(docsPath)).isDirectory()) { + throw new Error(`${docsPath} must be a directory`); + } + + const files = (await glob(`${docsPath}/**/*.md`)) + .map((file) => ({ file, source: sourcePath(docsPath, file) })) + .filter(({ source }) => isApiMarkdown(source)); + if (files.length === 0) { + throw new Error(`${docsPath} contains no importable API Markdown`); + } + if (!files.some(({ source }) => source === 'README.md')) { + throw new Error(`${docsPath} is missing its API root README.md`); + } + const urls = new Map( + files.map(({ source }) => [source, portalUrl(artifact.version, source)]) + ); + + for (const { file, source } of files) { + const resolved = await artifactFile(docsPath, sourcePath(docsPath, file)); + const markdown = await readFile(resolved, 'utf8'); + const { title, content } = stripTypeDocChrome(markdown); + const destination = pathJoin( + collectionDir, + artifact.version, + portalPath(source) + ); + await mkdir(pathDirname(destination), { recursive: true }); + await writeFile( + destination, + withFrontmatter( + portalTitle(source, title), + artifact.version, + source, + portalUrl(artifact.version, source), + normalizeHeadingHierarchy(rewriteTypeDocLinks(content, source, urls)) + ) + ); + } + + const navigation = addUrls( + Navigation.parse(JSON.parse(await readFile(navigationPath, 'utf8'))), + urls, + artifact.navigationPath + ); + imported.push({ + version: artifact.version, + commit: artifact.commit, + generatedAt: artifact.generatedAt, + root: `/docs/sdk/${artifact.version}/api/`, + navigation, + }); +} + +await writeFile( + dataPath, + `${JSON.stringify({ versions: imported }, null, 2)}\n` +); +console.info( + `Imported SDK API documentation for ${imported.length} version(s).` +); diff --git a/build/tasks/verify/verify-filenames.mts b/build/tasks/verify/verify-filenames.mts index b69dc0211..d193a4292 100644 --- a/build/tasks/verify/verify-filenames.mts +++ b/build/tasks/verify/verify-filenames.mts @@ -29,9 +29,21 @@ const EXEMPT = new Set([ // Eleventy takes the global data key from the filename, so kebab-casing // this one would quietly unhook the front matter validator. '_data/eleventyDataSchema.mjs', + // And this one, which the import task writes, is where the SDK layout and + // the SDK landing page read `sdkApi` from. + '_data/sdkApi.json', ]); -const files = await glob(['**/*', '!_site/', '!node_modules/']); +const files = await glob([ + '**/*', + '!_site/', + '!node_modules/', + // The SDK's API reference: TypeDoc's output, vendored as the release + // published it, and the pages the import task derives from that. Every + // name and word in it belongs to the SDK, and nothing here can change one. + '!vendor/sdk-api/*/', + '!collections/_sdk-api/', +]); // This one hands nothing to a tool, but `glob` finding nothing would still // leave it looping zero times and reporting success. diff --git a/build/tasks/verify/verify-json.mts b/build/tasks/verify/verify-json.mts index 1c679c215..b060b4a32 100644 --- a/build/tasks/verify/verify-json.mts +++ b/build/tasks/verify/verify-json.mts @@ -7,7 +7,15 @@ import { exec, glob, matched, quote } from '@openinf/portal/build/utils'; -const EXCLUDED = ['!_site/', '!node_modules/']; +const EXCLUDED = [ + '!_site/', + '!node_modules/', + // The SDK API artifact's manifest and navigation data are generated + // by the SDK, and the import task validates them on the way in. + '!vendor/sdk-api/*/', + '!collections/_sdk-api/', + '!_data/sdkApi.json', +]; const jsonFiles = await glob(['**/*.json', '**/*.jsonc', ...EXCLUDED]); // Biome has no JSON5 parser. Handed one it reports the path as ignored and diff --git a/build/tasks/verify/verify-md.mts b/build/tasks/verify/verify-md.mts index 362e3ac72..77e6387b9 100644 --- a/build/tasks/verify/verify-md.mts +++ b/build/tasks/verify/verify-md.mts @@ -20,6 +20,11 @@ const markdownFiles = await glob([ '!collections/_docs/security.md', '!collections/_docs/support.md', '!collections/_pages/vision.md', + // The SDK's API reference: TypeDoc's output, vendored as the release + // published it, and the pages the import task derives from that. Every + // name and word in it belongs to the SDK, and nothing here can change one. + '!vendor/sdk-api/*/', + '!collections/_sdk-api/', ]); let exitCode = 0; diff --git a/build/tasks/verify/verify-spelling.mts b/build/tasks/verify/verify-spelling.mts index 1af811ee0..56b4245d1 100644 --- a/build/tasks/verify/verify-spelling.mts +++ b/build/tasks/verify/verify-spelling.mts @@ -34,6 +34,12 @@ const files = await glob([ '!collections/_docs/security.md', '!collections/_docs/support.md', '!collections/_pages/vision.md', + // The SDK's API reference: TypeDoc's output, vendored as the release + // published it, and the pages the import task derives from that. Every + // name and word in it belongs to the SDK, and nothing here can change one. + '!vendor/sdk-api/*/', + '!collections/_sdk-api/', + '!_data/sdkApi.json', ]); process.exitCode = matched(files, SPELLCHECKED) diff --git a/collections/_pages/sdk.html b/collections/_pages/sdk.html new file mode 100644 index 000000000..24537d51e --- /dev/null +++ b/collections/_pages/sdk.html @@ -0,0 +1,33 @@ +--- +title: OpenINF SDK +layout: deep-page.liquid +permalink: /docs/sdk/ +description: API reference and package guides for the OpenINF SDK. +body_classes: docs doc-page sdk-overview +editable: false +--- +The OpenINF SDK is the shared foundation for building dependable integrations +with OpenINF. Its API reference is generated from the package source and +rendered here in the same documentation system as the rest of the portal. + +{% if sdkApi and sdkApi.versions.size > 0 %} +

API reference

+ + {% for release in sdkApi.versions %} +

{{ release.version }}

+ +

+ Built from SDK commit {{ release.commit }}. +

+ +

Browse the API reference

+ {% endfor %} + +{% else %} +

API reference

+ +

+ The complete reference for all OpenINF SDK packages will appear here with + the public SDK release. +

+{% endif %} diff --git a/eleventy.config.mjs b/eleventy.config.mjs index aab887652..a5824d9be 100644 --- a/eleventy.config.mjs +++ b/eleventy.config.mjs @@ -4,6 +4,7 @@ import { extname, join, parse as pathParse } from 'node:path'; import { EleventyI18nPlugin } from '@11ty/eleventy'; import { PATHS } from '@openinf/portal/build/constants'; import { hasViewBox, replaceInlineSvg } from '@openinf/portal/build/inline-svg'; +import { sanitizeSdkHtml } from '@openinf/portal/build/sdk-docs'; import autoprefixer from 'autoprefixer'; import cssnano from 'cssnano'; import { minify as minifyHtml } from 'html-minifier-terser'; @@ -18,6 +19,9 @@ import { minify as minifyJs } from 'terser'; // skipcq: JS-0116 export default async function (eleventyConfig) { const isProduction = process.env.ELEVENTY_ENV === 'production'; + // This registers a sanitizer callback, not a value containing HTML. + // eslint-disable-next-line xss/no-mixed-html + eleventyConfig.addFilter('sanitizeSdkHtml', sanitizeSdkHtml); eleventyConfig.amendLibrary('md', (md) => { md.use(markdownItAnchor); @@ -120,6 +124,10 @@ export default async function (eleventyConfig) { // only from being rendered as a page -- passthrough still publishes it // beside the icons it covers. eleventyConfig.ignores.add(`${PATHS.assetsDir}**/*.md`); + // SDK release artifacts are transformed into the generated collection by a + // build task. Rendering their raw Markdown directly would publish every + // source path twice and leave TypeDoc's `.md` links intact. + eleventyConfig.ignores.add('vendor/sdk-api/**'); // The stylesheet is compiled by Eleventy as a template of its own, which is // what puts it in the dependency graph: editing a partial recompiles diff --git a/package-scripts.yml b/package-scripts.yml index c13ef831e..f2407efa2 100644 --- a/package-scripts.yml +++ b/package-scripts.yml @@ -1,6 +1,7 @@ scripts: compile: buildPortal: node build/tasks/compile/build-portal.mts + importSdkApiDocs: node build/tasks/compile/import-sdk-api-docs.mts siteifyHealthFiles: node build/tasks/compile/siteify-health-files.mts verify: # Announces each task and names the ones that failed. Without this the diff --git a/package.json b/package.json index c90f82f85..2c6fb970a 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,7 @@ "./build/inline-svg": "./build/shared/inline-svg.mts", "./build/landing": "./build/shared/landing.mts", "./build/pull-request": "./build/shared/pull-request.mts", + "./build/sdk-docs": "./build/shared/sdk-docs.mts", "./build/utils": "./build/utils.mts" }, "homepage": "https://open.inf.is", @@ -32,6 +33,7 @@ "@tsconfig/node-lts": "24.0.1", "@types/console-log-level": "1.4.5", "@types/node": "24.13.3", + "@types/sanitize-html": "2.16.1", "@yarnpkg/shell": "4.1.3", "autoprefixer": "10.5.4", "bootstrap": "5.3.8", @@ -86,6 +88,7 @@ "retext-simplify": "8.0.0", "retext-syntax-mentions": "4.0.0", "retext-syntax-urls": "4.0.0", + "sanitize-html": "2.17.7", "sass": "1.103.1", "stylelint": "17.14.1", "stylelint-config-recess-order": "7.8.0", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 4ad26e2d4..0309b3434 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -133,6 +133,9 @@ importers: '@types/node': specifier: 24.13.3 version: 24.13.3 + '@types/sanitize-html': + specifier: 2.16.1 + version: 2.16.1 '@yarnpkg/shell': specifier: 4.1.3 version: 4.1.3(typanion@3.14.0) @@ -295,6 +298,9 @@ importers: retext-syntax-urls: specifier: 4.0.0 version: 4.0.0 + sanitize-html: + specifier: 2.17.7 + version: 2.17.7 sass: specifier: 1.103.1 version: 1.103.1 @@ -1097,6 +1103,9 @@ packages: '@types/pluralize@0.0.29': resolution: {integrity: sha512-BYOID+l2Aco2nBik+iYS4SZX0Lf20KPILP5RGmM1IgzdwNdTs0eebiFriOPcej1sX9mLnSoiNte5zcFxssgpGA==} + '@types/sanitize-html@2.16.1': + resolution: {integrity: sha512-n9wjs8bCOTyN/ynwD8s/nTcTreIHB1vf31vhLMGqUPNHaweKC4/fAl4Dj+hUlCTKYgm4P3k83fmiFfzkZ6sgMA==} + '@types/supports-color@8.1.3': resolution: {integrity: sha512-Hy6UMpxhE3j1tLpl27exp1XqHD7n8chAiNPzWfz16LPZoMMoSc4dzLl6w9qijkEb/r5O1ozdu1CWGA2L83ZeZg==} @@ -1641,6 +1650,9 @@ packages: dale-chall@2.0.1: resolution: {integrity: sha512-PcTrFy9XMcnIaGa3qe8KBL+zSxqR224TDi9WZ2un4o9RAutxAjMaCrQ5BQ5QdB0CyFv/SC/Px4d7ORhaJq6T1Q==} + dayjs@1.11.23: + resolution: {integrity: sha512-QDTCU0M0MxR3hQfnlDJfwekQiaanm1ubOD231u73WBckQ/fsamwRLiE2GBz6D3a/xF1NgfiDLJjXBa1hYOYTtQ==} + debug@2.6.9: resolution: {integrity: sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==} peerDependencies: @@ -1665,6 +1677,10 @@ packages: decode-named-character-reference@1.3.0: resolution: {integrity: sha512-GtpQYB283KrPp6nRw50q3U9/VfOutZOe103qlN7BPP6Ad27xYnOIWv4lPzo8HCAL+mMZofJ9KEy30fq6MfaK6Q==} + deepmerge@4.3.1: + resolution: {integrity: sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A==} + engines: {node: '>=0.10.0'} + depd@2.0.0: resolution: {integrity: sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==} engines: {node: '>= 0.8'} @@ -1781,6 +1797,10 @@ packages: resolution: {integrity: sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==} engines: {node: '>=0.12'} + entities@7.0.1: + resolution: {integrity: sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==} + engines: {node: '>=0.12'} + entities@8.0.0: resolution: {integrity: sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA==} engines: {node: '>=20.19.0'} @@ -1828,6 +1848,10 @@ packages: resolution: {integrity: sha512-vbRorB5FUQWvla16U8R/qgaFIya2qGzwDrNmCZuYKrbdSUMG6I1ZCGQRefkRVhuOkIGVne7BQ35DSfo1qvJqFg==} engines: {node: '>=0.8.0'} + escape-string-regexp@4.0.0: + resolution: {integrity: sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==} + engines: {node: '>=10'} + escape-string-regexp@5.0.0: resolution: {integrity: sha512-/veY75JbMK4j1yjvuUxuVsiS/hr/4iHs9FTT6cgTexxdE0Ly/glccBAkloH/DofkjRbZU3bnoj38mOmhkZ0lHw==} engines: {node: '>=12'} @@ -2054,6 +2078,9 @@ packages: resolution: {integrity: sha512-n6l5uca7/y5joxZ3LUePhzmBFUJ+U2YWzhMa8XUTecSeSlQiZdF5XAd/Q3/WUl0VsXgUwWi8I7CNIwdI5WN1SQ==} engines: {node: '>=20.10'} + htmlparser2@10.1.0: + resolution: {integrity: sha512-VTZkM9GWRAtEpveh7MSF6SjjrpNVNNVJfFup7xTY3UpFtm67foy9HDVXneLtFVt4pMz5kZtgNcvCniNFb1hlEQ==} + htmlparser2@12.0.0: resolution: {integrity: sha512-Tz7u1i95/g2x2jz81+x0FBVhBhY5aRTvD3tXXdFaljuNdzDLJ8UGNRrTcj2cgQvAg3iW/h77Fz15nLW0L0CrZw==} engines: {node: '>=20.19.0'} @@ -2263,6 +2290,9 @@ packages: known-css-properties@0.37.0: resolution: {integrity: sha512-JCDrsP4Z1Sb9JwG0aJ8Eo2r7k4Ou5MwmThS/6lcIe1ICyb7UBJKGRIUUdqc2ASdE/42lgz6zFUnzAIhtXnBVrQ==} + launder@1.7.1: + resolution: {integrity: sha512-mU6WRz5EusL9ZZuiZ5SO4Y6C0P9PAUR9iwdb6bzj4KDihm28DiHFw+/yk9DBH4f+Pv1wuzQ4e2jV3oQ7mkIqvw==} + levenshtein-edit-distance@1.0.0: resolution: {integrity: sha512-gpgBvPn7IFIAL32f0o6Nsh2g+5uOvkt4eK9epTfgE4YVxBxwVhJ/p1888lMm/u8mXdu1ETLSi6zeEmkBI+0F3w==} hasBin: true @@ -3344,6 +3374,10 @@ packages: safe-buffer@5.2.1: resolution: {integrity: sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==} + sanitize-html@2.17.7: + resolution: {integrity: sha512-PGtEkc9cbnedU3s9TmzDbpsZ8w086g/0Q8k8/oIO1NLNU3i5k9yn835CrjJSajp1KMmkisbO1qPXxNKO3welAg==} + engines: {node: '>=22.12.0'} + sass@1.103.1: resolution: {integrity: sha512-9icZURbP51S6S0QGoyaeqk9uB06GNWxsFYWfH5RgpFgqK5FA8tJcM3AdVxrZEVJ7dz+L87nG95gBKf4VuaMHGw==} engines: {node: '>=20.19.0'} @@ -4671,6 +4705,10 @@ snapshots: '@types/pluralize@0.0.29': {} + '@types/sanitize-html@2.16.1': + dependencies: + htmlparser2: 10.1.0 + '@types/supports-color@8.1.3': {} '@types/text-table@0.2.5': {} @@ -5212,6 +5250,8 @@ snapshots: dale-chall@2.0.1: {} + dayjs@1.11.23: {} + debug@2.6.9(supports-color@10.2.2): dependencies: ms: 2.0.0 @@ -5230,6 +5270,8 @@ snapshots: dependencies: character-entities: 2.0.2 + deepmerge@4.3.1: {} + depd@2.0.0: {} dependency-graph@1.0.0: {} @@ -5348,6 +5390,8 @@ snapshots: entities@6.0.1: {} + entities@7.0.1: {} + entities@8.0.0: {} env-paths@2.2.1: {} @@ -5397,6 +5441,8 @@ snapshots: escape-string-regexp@1.0.5: {} + escape-string-regexp@4.0.0: {} + escape-string-regexp@5.0.0: {} esm-import-transformer@3.0.5: @@ -5626,6 +5672,13 @@ snapshots: html-tags@5.1.0: {} + htmlparser2@10.1.0: + dependencies: + domelementtype: 2.3.0 + domhandler: 5.0.3 + domutils: 3.2.2 + entities: 7.0.1 + htmlparser2@12.0.0: dependencies: domelementtype: 3.0.0 @@ -5787,6 +5840,10 @@ snapshots: known-css-properties@0.37.0: {} + launder@1.7.1: + dependencies: + dayjs: 1.11.23 + levenshtein-edit-distance@1.0.0: {} line-column@1.0.2: @@ -7620,6 +7677,16 @@ snapshots: safe-buffer@5.2.1: {} + sanitize-html@2.17.7: + dependencies: + deepmerge: 4.3.1 + escape-string-regexp: 4.0.0 + htmlparser2: 12.0.0 + is-plain-object: 5.0.0 + launder: 1.7.1 + parse-srcset: 1.0.2 + postcss: 8.5.26 + sass@1.103.1: dependencies: chokidar: 5.0.0 diff --git a/project-terms.txt b/project-terms.txt index f36483cf7..b0be6aa65 100644 --- a/project-terms.txt +++ b/project-terms.txt @@ -13,6 +13,7 @@ browserslistrc davidanson dbaeumer deepsource +describedby distros Dockerfiles dorny @@ -69,6 +70,7 @@ repoints retarget retargets returntocorp +rowspan rubocop screencap sdcard diff --git a/vendor/sdk-api/README.md b/vendor/sdk-api/README.md new file mode 100644 index 000000000..421ec2581 --- /dev/null +++ b/vendor/sdk-api/README.md @@ -0,0 +1,72 @@ +## SDK API artifact contract + +The [OpenINF SDK](https://github.com/OpenINF/sdk) supplies one directory per +release here. This portal turns its TypeDoc Markdown into pages below +`/docs/sdk//api/`, in the portal's own documentation layout. + +```text +vendor/sdk-api/ + 3.0.0/ + manifest.json + docs/ + README.md + navigation.json + @openinf/ + util/ + README.md + functions/ + isString.md +``` + +`manifest.json` has this shape: + +```json +{ + "schemaVersion": 1, + "version": "3.0.0", + "commit": "0123456789abcdef", + "generatedAt": "2026-10-01T00:00:00.000Z", + "docsPath": "docs", + "navigationPath": "docs/navigation.json" +} +``` + +### Where a directory here comes from + +The SDK's **Release** workflow builds one on the run that publishes, from the +versions that shipped and the commit they shipped from, and attaches it to that +run as **sdk-api-docs**. Adding it here is a deliberate step, described under +"Publishing the API reference" in the SDK's `RELEASING.md`: download the +artifact, unzip it beside the releases already here, and open a pull request. + +A release adds a version rather than replacing one. Every release keeps the +reference it was published with, so a link into an older version keeps +describing that version instead of following the latest source. + +### What it may contain + +The artifact contains generated Markdown only. Do not edit it by hand. The +import task refuses generated maintainer documents; the SDK's API-specific +TypeDoc overview must link to public portal pages rather than to its +repository-local contributor or release instructions. + +Manifest paths must resolve inside their version directory, including through +symbolic links. The Markdown root must be a directory with importable pages +and a `README.md` entry point; an empty artifact cannot advertise a release. +Navigation URLs are derived from imported pages, not accepted from the source +JSON. Imported Markdown is rendered without Liquid evaluation, then its HTML +body is sanitized before the portal layout embeds it. Script elements, event +handlers, and unsafe URL schemes are not allowed; escaped code examples, +API anchors, and tables are retained. + +The SDK applies these same rules to its own output before it builds an +artifact, so a corpus this portal would refuse fails there first. + +### Why the checks skip it + +Names here are TypeDoc's, taken from the symbols they document, and the prose +is the SDK's. Neither is this repository's to correct, and a fix applied here +would be overwritten by the next release. So the version directories are +outside the filename, Markdown, spelling, JSON, and EditorConfig checks, along +with `collections/_sdk-api/`, which the import task derives from them. This +file is not generated, and remains checked.