diff --git a/.gitignore b/.gitignore index 59cc3acb6..1d4da224e 100644 --- a/.gitignore +++ b/.gitignore @@ -12,3 +12,4 @@ __pycache__/ *$py.class *.db *.sqlite +run_e2e_pipeline_test.local.sh diff --git a/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md new file mode 100644 index 000000000..8a8bed28b --- /dev/null +++ b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/README.md @@ -0,0 +1,148 @@ +# Process Kotlin Website JSON + +Scripts for converting a `kotlin-web-site/docs` checkout (JetBrains +Writerside-flavored Markdown) into the JSON block schema this project's +templating engine renders, and for loading that JSON, its navigation tree, +and its media straight into a `documentation.db`-schema SQLite database. + +## Scripts + +| Script | Purpose | +|---|---| +| [`md_to_json.py`](md_to_json.py) | Converts every `topics/**/*.md` page into one JSON file (see schema below). Writes `theme.json` and copies `images/` into the output directory. | +| [`build_nav.py`](build_nav.py) | Builds `nav.json`/`nav.html` sidebar navigation from `kr.tree`, resolving each `` against `md_to_json.py`'s output. | +| [`find_missing_assets.py`](find_missing_assets.py) | QA pass: reports cross-page links, images, and `` targets in the source tree that don't resolve to anything. Reuses `md_to_json.py`'s own resolution logic, so it flags exactly what would end up broken on the rendered site. | +| [`populate_db.py`](populate_db.py) | The database path: converts the docs tree the same way `md_to_json.py` does, builds nav the same way `build_nav.py` does, and inserts pages + nav + images + CSS/JS directly into `documentation.db` (replacing everything under `k/html/` and `assets/`). Supports pruning whole `kr.tree` subtrees via `--blacklisted-element-titles`. | +| [`optimize_media.py`](optimize_media.py) | Standalone media optimizer: downscales/recompresses a directory of images (pngquant, Pillow, Scour/cairosvg for SVG) into a mirrored output directory. | +| [`insert_optimized_media.py`](insert_optimized_media.py) | Runs `optimize_media.py`'s pipeline over a directory of raw media, then replaces the corresponding `k/html/images/*` rows in an existing database, rewriting any page that referenced a renamed file and deleting anything left unreferenced. | + +## Requirements + +- Python 3.10+ +- `pip install markdown-it-py Pillow scour brotli` +- `cairosvg` (only needed if an optimized SVG exceeds `--svg-rasterize-threshold`): `pip install cairosvg` +- `pngquant` on `PATH` (e.g. `apt install pngquant`) — required by `optimize_media.py`/`insert_optimized_media.py`, and by `populate_db.py` for the images it inserts directly from the Writerside export. + +`populate_db.py` also expects, relative to its own location, and already +included in this directory: + +- `templates/page.peb`, `templates/nav.peb` — Pebble templates upserted into the `Templates` table. +- `assets/docs.css`, `assets/tabs.js`, `assets/sidebar.js` — static assets inserted at `assets/`. + +## Inputs you need before starting + +- A checkout of `kotlin-web-site/docs` (the `` argument below) — contains `topics/`, `images/`, `v.list`, and `kr.tree`. +- A config JSON with theming colors, e.g.: + ```json + {"broken-ext-link-color": "#cc0000", "menu-no-link-color": "#999999"} + ``` +- Writerside's own image export zip (e.g. `webHelpImages.zip`, found next to `kr.tree`) if you're using `populate_db.py`. + +## Workflow: generate JSON + nav for a static/templated preview + +Use this to produce standalone JSON pages and nav data (not the database) +for local inspection or a different renderer. + +```bash +# 1. Convert every topic .md into JSON, one file per page +python3 md_to_json.py config.json + +# 2. Build the sidebar nav from kr.tree against that JSON output +python3 build_nav.py + +# 3. (optional) Check for broken links/images/includes in the source tree +python3 find_missing_assets.py missing-assets-report.md +``` + +`` ends up containing: +- `topics/**/*.json` — one page per source `.md` file (schema below) +- `theme.json` — the two theming colors, carried from `config.json` +- `images/` — copied straight from `/images/` +- `nav.json` / `nav.html` — sidebar tree and a pre-rendered static copy + +### Page JSON schema + +```json +{ + "id": "enum-classes", + "sourceFile": "topics/enum-classes.md", + "title": "Enum classes", + "blocks": [ { "type": "heading", "level": 2, "id": "...", "html": "..." }, "..." ] +} +``` + +Block types: `heading`, `paragraph`, `code`, `blockquote`, `list`, `table`, +`image`, `hr`, `tabs`, `note`/`tip`/`warning`, `html` (raw passthrough). See +the module docstring in [`md_to_json.py`](md_to_json.py) for full shapes and +known limitations (nested tabs, `` resolution, variable +substitution). + +## Workflow: generate + insert directly into the documentation database + +This is the path that actually populates `documentation.db`. It performs +the same conversion as `md_to_json.py`/`build_nav.py` internally — you don't +run those scripts first. + +```bash +python3 populate_db.py config.json [db-path] +``` + +- `db-path` defaults to `documentation.db` in the current directory, and must already exist with the expected schema (`Languages`, `ContentTypes`, `Templates` tables populated). +- A timestamped backup (`.backup-`) is written before any changes, via SQLite's `VACUUM INTO`. +- Everything under `k/html/` and `assets/` is deleted and re-inserted in a single transaction (rolled back on error), then the database is `VACUUM`ed. + +### Pruning documentation you don't want (ADFA-4737) + +To leave a whole `kr.tree` subtree out of the database entirely — nav +entry, converted pages, and all — pass `--blacklisted-element-titles` with +the full `toc-title` path from a top-level element down to the one you want +to drop. Levels are joined with `\/` (backslash-slash), not a bare `/`, +since a bare `/` commonly appears inside a real title. The example below is +illustrative only — open `/kr.tree` and copy the actual +`toc-title` chain for whatever section you're dropping (e.g. Kotlin/Wasm): + +```bash +python3 populate_db.py config.json documentation.db \ + --blacklisted-element-titles \ + "\/" +``` + +Any other page's in-content link to a pruned topic renders as a styled +"broken" link (via `broken-ext-link-color`) rather than a dead link with no +indication anything changed. Run with `--blacklisted-element-titles` first +against a scratch copy of the database and check the warnings on stderr for +any path that didn't match — that usually means the toc-title or ancestor +chain was copied wrong. + +## Workflow: optimizing and inserting media + +Two options, depending on whether the database already has pages loaded: + +**Standalone optimization only** (no database involved): + +```bash +python3 optimize_media.py [--max-width 500] [--webp] [...] +``` + +**Optimize and update an existing database's images in place:** + +```bash +python3 insert_optimized_media.py [work-dir] [options] +``` + +This re-runs `optimize_media.py`'s pipeline, backs up the database first, +replaces each `k/html/images/` row with the optimized bytes, rewrites +any page/nav reference to a file that got renamed during optimization (e.g. +`--webp` conversion or SVG rasterization), and deletes any image no page +references anymore. Both scripts share the same tuning flags +(`--max-width`, `--jpeg-quality`, `--webp`, `--webp-quality`, +`--pngquant-speed`, `--svg-precision`, `--svg-rasterize-threshold`, +`--verbose`, `--log-file`), settable via `--config ` instead of the +command line — see either script's module docstring for the full option +reference. + +## Recommended order for a full refresh + +1. `find_missing_assets.py` against the new `` — fix anything broken in the source before converting it. +2. `populate_db.py`, with `--blacklisted-element-titles` for anything you don't want documented (e.g. Kotlin/Wasm per ADFA-4737). +3. `insert_optimized_media.py` against the raw media directory, if you want optimized (resized/compressed) images rather than Writerside's own export as-is. diff --git a/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/assets/docs.css b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/assets/docs.css new file mode 100644 index 000000000..b8eef9286 --- /dev/null +++ b/ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/assets/docs.css @@ -0,0 +1,227 @@ +html, body { + background: #ffffff; + font-family: Arial, Helvetica, sans-serif; +} + +.docs-layout { + display: flex; + align-items: flex-start; +} + +.docs-sidebar { + width: 280px; + flex-shrink: 0; + box-sizing: border-box; + overflow-y: auto; + max-height: 100vh; + position: sticky; + top: 0; +} + +.docs-content { + flex: 1; + min-width: 0; + padding: 0 24px; +} + +/* Source images/videos carry explicit width="..." attributes (from + Writerside's `{width="800"}` sizing hints, see md_to_json.py), which is an + intrinsic pixel size the img/video would otherwise render at even when + that's wider than the viewport. max-width: 100% lets it shrink to fit + instead of overflowing on narrow screens, while height: auto keeps its + aspect ratio as it scales down. */ +.docs-content img, +.docs-content video, +.docs-content iframe { + max-width: 100%; + height: auto; +} + +/* Long code lines can't shrink like an image can without breaking the code's + formatting, so let the block itself scroll horizontally instead - without + this the unconstrained width pushes out past the viewport and the whole + page grows a horizontal scrollbar on narrow screens. */ +.docs-content pre.code-block { + overflow-x: auto; + box-sizing: border-box; + max-width: 100%; +} + +/* Sidebar tree: