Skip to content

Add portable CHM and PDF Markdown export - #3

Open
johnml1135 wants to merge 1 commit into
sillsdev:developfrom
johnml1135:tools/markdown-export
Open

johnml1135 wants to merge 1 commit into
sillsdev:developfrom
johnml1135:tools/markdown-export

Conversation

@johnml1135

@johnml1135 johnml1135 commented Aug 21, 2026 •

Copy link
Copy Markdown

A help change is currently a 5 MB opaque binary diff. This publishes the same content as Markdown — one file per topic, with images, frontmatter and a full table of contents — on a markdown-export branch that nothing hand-edits. Diffs become readable text, one changed file per edited topic, and the AI bot ingests about 65% fewer tokens (~2.14M → ~759K) with the prose intact.

tools/convert.py builds the corpus; .github/workflows/markdown-export.yml validates every pull request and publishes on pushes to develop and on release tags. Each build also emits author-report.md for RoboHelp and PDF authors, and author-report.json for automation.

Verified: 2 CHMs, 1,630 topics, 13 PDFs, 583 images, 0 fatal — against both the CHM this branch was written against and the current one on develop. 222 tests and ruff run before anything publishes. Validation is green on this repo's own runners; publishing correctly skips on pull_request.

Two findings worth reading before merge: this workflow had never once run, and six links have been broken on the export branch since the first publish. Both are fixed here and explained in the comment below.

@johnml1135

johnml1135 commented Sep 9, 2026 •

Copy link
Copy Markdown
Author

This workflow had never once run. jobs.validate.env referenced ${{ runner.temp }}, a context GitHub refuses in job-level env. That fails at load time, so every run ended in seconds with no jobs and no logs to diagnose it from. The export has never been built by CI.

Fixing that surfaced two more — and the last is not a CI bug at all. Six links have been broken on markdown-export since the first publish. Three Parser topics link to using_tools/texts_&_words_tools/… while those topics live under Using_Tools/Texts_&_Words_tools/…. RoboHelp resolves that case-insensitively, so the links open in the CHM and 404 on GitHub.

They stayed hidden behind Path.resolve(), which rewrites a path to its real on-disk case on Windows: a Windows build passed while Linux failed the same corpus. Link checking is now case-exact on every platform, the exporter republishes each link in the case its file really has, and the drift is reported to authors as a new source_link_case advisory.

Verified end to end on a fork, publish job included — 0 fatal against both the CHM this branch was written against and the current one on develop.

The three fixes
Fix Effect
runner context out of job-level env The workflow parses, so it can run at all
--repo . under uv run --directory src --repo src resolved to src/src, dying before one topic converted
Case-exact link checking, canonicalized output Six live 404s corrected; Windows and Linux now agree

Each has a regression test. The suite is 222 tests plus ruff, both run by validation before anything publishes.

Verification
Build Fatal Advisory
CHM this branch was written against 0 276
CHM at current develop, merged in here 0 255

The recent link repairs on develop clear 21 advisories: missing links 16→0, duplicate titles 4→0, stale TOC 1→0. The six case mismatches remain, and are now reported rather than silently published.

Publishing safety

Publishing is a separate job behind needs: validate, holding the only contents: write permission. It runs on push and non-dry-run dispatch, never on pull_request, and appends to markdown-export rather than force-pushing it. Pandoc is pinned by SHA-256, and every action by commit SHA.

Two things for a maintainer: GITHUB_TOKEN needs contents: write — the job declares it, but if this org restricts default workflow permissions the publish step fails on first run. And markdown-export does not exist here yet; the job creates it as an orphan branch on that first run, which also fixes the README's ../../tree/markdown-export link.

The FieldWorks help ships as a 5 MB CHM: opaque in diffs, and roughly
two thirds RoboHelp markup by token when the AI bot ingests it. Publish
it as Markdown as well -- one file per topic, with images, frontmatter
and a full table of contents -- on a separate markdown-export branch
that nothing hand-edits.

tools/convert.py drives the build: cross-platform CHM extraction that
refuses to run when it would silently truncate filenames, a Pandoc Lua
filter mapping RoboHelp semantics to clean GFM, and PDF conversion
pinned against approved outlines so drift fails the build. Every run
emits author-report.md for RoboHelp and PDF authors, and
author-report.json for automation.

markdown-export.yml validates on pull request and publishes on pushes to
develop and on release tags. Validation lints, tests, converts, and
uploads the corpus; publishing is a separate job holding the only write
permission, and it appends to markdown-export rather than rewriting it.

Corpus link checking is case-exact on every platform. Resolving targets
with Path.resolve() folds a link onto its real case on Windows, so a
Windows build passed while a case-sensitive host 404ed on the same
corpus. Links whose authored case differs from their target are now
republished in the case the file really has and reported back to
authors, because RoboHelp resolves them case-insensitively and cannot
show the author the difference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011wJtRQnGiFNnS4ZoYw61F7
@johnml1135
johnml1135 force-pushed the tools/markdown-export branch from 16437e8 to 439e064 Compare September 9, 2026 19:44
@MarlonH

MarlonH commented Sep 9, 2026 via email

Copy link
Copy Markdown
Collaborator

@johnml1135

Copy link
Copy Markdown
Author

6 links across 3 topics — 2 in each. "Three" was topics, "six" was links; that read ambiguously, sorry.

Your repairs did land. Every Texts_and_Words_overview.htm and interlinear_views_colors.htm link in those three topics is now correct case. The six that remain point at two other files, which is why searching for the T & Wds overview link wouldn't have surfaced them:

Topic (User_Interface/Menus/Parser/) Wrong-case targets
Default_XAmple_parser_overview.htm Analyze_Text_overview.htm, specify_the_word_gloss.htm
Parsing_words_overview.htm the same two
Phonological_Rule_Based_parser_overview.htm the same two

All six spell it using_tools/texts_&amp;_words_tools/…; the files live at Using_Tools/Texts_&_Words_tools/…. Verified against the CHM currently on develop, so with your fixes already in.

Parsing_words_overview.htm shows it most clearly — it links to Analyze_Text_overview.htm twice, once as Using_Tools/Texts_%26_Words_tools/… and once as using_tools/texts_&amp;_words_tools/…. Same source topic, same destination, two different spellings.

Nothing is blocked on this. The exporter republishes all six in the case the files really have, so the Markdown branch is correct as of this PR. The advisory only records that the RoboHelp source still differs — worth tidying whenever it suits, not before merge.

Your other repairs cleared 21 advisories: missing links 16→0, duplicate titles 4→0, stale TOC 1→0.

On the S3 bucket — no interaction. The export reads the CHM committed to this repo, so it tracks what's on develop regardless of when the bucket is refreshed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants