ADFA-4739: Pipeline for producing template-based Kotlin documentation

[alexmmiller]

Summary

An end-to-end pipeline for producing templated Kotlin documentation.

Changes

Kotlin website docs

• Adds Python scripts used to convert kotlin-web-site/docs (Writerside Markdown) into JSON, build nav, optimize media, and insert everything into documentation.db, into ProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/. Adds a README documenting each script's purpose, required inputs (kr.tree, webHelpImages.zip, config JSON), and end-to-end usage — including populate_db.py --blacklisted-element-titles, the mechanism this ticket needs for pruning docs sections (e.g. Kotlin/Wasm) we don't want to ship.

Kotlin-stdlib docs

• Adds a sync_kdoc_json_to_db.py script into a new directory scripts/sync_kotlin_stdlib_docs/, for syncing generated kotlin-stdlib docs against the database.

Test plan for Kotlin website docs:

• Reviewer confirms the new scripts match their originals in /home/alex/ADFA/KotlinWebsiteProcess/ and Dokka-plugin-kdoc2json/scripts/ (no edits were made during the copy)
• README instructions are accurate for running populate_db.py with --blacklisted-element-titles against a scratch database

[davidschachterADFA]

Claude says:

Issues found

1. README is out of sync with what's actually in the PR.
   The README says (lines 32–35):

per ADFA-4737, only the .py files from the original working directory were copied here. templates/ and assets/ ... must be placed alongside these scripts before running populate_db.py.

But the PR itself does include templates/{page.peb,nav.peb,nav.html}, assets/{docs.css,tabs.js,sidebar.js}, and config.json (added by the "Config/assets for pipeline test" commit). A reviewer or new user following the README literally would go hunting for files that are already sitting right there. Worth updating that note (or clarifying it only applied to the original copy-over before the pipeline-test commit added them).

2. sync_kdoc_json_to_db.py defaults --db to a personal path.

python
parser.add_argument("--db", default="/home/alex/documentation.db", ...)

This is clearly a leftover from the author's local machine. It's harmless in that a wrong path just fails the os.path.isfile check, but it shouldn't ship as the default in a shared repo — either make it required (no default) or default to documentation.db in cwd, matching populate_db.py's convention.

3. Possible chain-rename bug in insert_optimized_media.py's rewrite_pages.
   rewrite_pages iterates rename_map.items() and does sequential str.replace calls on the same new_text buffer:

python
for old_name, new_name in rename_map.items():
...
new_text = new_text.replace(old_ref, new_ref)

If one rename's new_name happens to equal another rename's old_name (e.g. foo.png → foo.webp from one optimization, and some unrelated foo.webp → foo-2.webp renamed elsewhere), the second substitution could re-match text that the first substitution just wrote, double-rewriting a reference that was never actually pointing at the second file. It's a narrow edge case (requires two renames whose old/new basenames chain), but since build_rename_map only guards against the same old name mapping to two different new names, not this chain scenario, it's worth at least a comment acknowledging the assumption (or building a single combined regex/dict-based single-pass substitution instead of N sequential replaces).

4. Minor: sync_kdoc_json_to_db.py doesn't explicitly BEGIN the way the other two DB-writing scripts do (conn.execute("BEGIN")). Python's sqlite3 module will still implicitly open a transaction before the first DML statement with the default isolation level, so this isn't a correctness bug, but it's an inconsistency with the explicit-transaction style used elsewhere in the same PR — could confuse a future reader into thinking it's unguarded.

[alexmmiller]

e2e_run_output.log
I attached the log from running the updated pipeline in the most recent commit.

Summary:

== Summary ==
Database: /Users/alex/test_db_assets/documentation.test.db
k/html/* rows: 502
k/html/images/* rows: 235
k/html/images/.webp rows: 161
k/kotlin-stdlib/ rows: 3132
k/kotlin-reflect/* rows: 65
k/kotlin-test/* rows: 41

== Blacklist pruning verification ==
Blacklisted toc-element path(s) checked: 3
Development > Web development: matched
Interoperability > JavaScript interop: matched
Interoperability > Swift/Objective-C and C interop: matched
Topic page(s) expected removed: 38
PASS: all 38 blacklisted topic page(s) confirmed absent from /Users/alex/test_db_assets/documentation.test.db.

Done. Backups (populate_db.py, insert_optimized_media.py, and sync_kdoc_json_to_db.py
each make their own) live alongside /Users/alex/test_db_assets/documentation.test.db as documentation.test.db.backup-* and
documentation.test.db.bak.*
The real database at /Users/alex/test_db_assets/documentation.db was never opened for writing.
EXIT_CODE=0