docs: build the witan-context documentation site by blarghmatey · Pull Request #254 · mitodl/agent-kit

blarghmatey · 2026-08-19T15:00:51Z

witan is mature enough that its documentation is now the bottleneck: ~12,000
lines of real prose existed, spread across three package READMEs, nine ADRs,
two user guides, runbooks, and a pile of design specs — none of it navigable,
and none of it reachable by anyone who is not already in the repository.

This stands up a Read the Docs site (project witan-context) covering the
witan trio — witan-council, witan-code, witan-core — organised along Diátaxis
lines: tutorials, how-to, reference, explanation.

The load-bearing decision is that reference is GENERATED, not written. A
handwritten tool reference is wrong by the second release, so bin/gen_docs.py
derives all of it from the thing it documents — the registered FastMCP tool
objects (60 tools, full JSON Schema), the cyclopts command tree, the .pg
schema files — and just docs-check fails CI when a committed page no longer
matches the code. The one hybrid is the environment reference: names are
discovered from source, descriptions live in docs/_data/environment.toml, and
a setting with no entry there is a hard error, so a new env var cannot ship
undocumented.

Package-resident prose (user guides, ADRs, code-graph internals) is MIRRORED
rather than moved, so the copy next to the code stays authoritative for anyone
reading the repo or the PyPI package, while the site gets it without a second
copy to drift. Mirroring rewrites relative links — to site paths where the
target is also mirrored, to GitHub otherwise — which is what took the build
from 69 unresolved links to none.

Two bugs the generator surfaced, both pre-existing and both fixed here:

Discovering env var names by scanning raw source text invents settings that
do not exist. WITAN_BRANCH appears only in a comment saying there is no
such override; WITAN_EMBED_ENABLED and WITAN_REQUEST_TIMEOUT are never
read. Discovery now walks the AST, where comments are absent and docstrings
are skipped, while ordinary string literals are kept so constants like
HTTP_TRANSPORT_ENV_VAR = "WITAN_OMNIGRAPH_HTTP" still count.
clean_desc did not escape |, so a parameter description enumerating its
options had every table cell past the fourth silently DROPPED —
task_update.status rendered as the single word "open". 17 descriptions
contain a raw pipe.

docs/design/ and docs/agent-memory.md move under docs/internals/, flagged
as point-in-time documents rather than current-state, since much of what they
specify changed during implementation. References to them across four packages
are updated accordingly.

Toolchain is Zensical, the Material for MkDocs team's successor generator —
Material entered maintenance mode in early 2026. It is pre-1.0, so it is pinned
identically in CI and .readthedocs.yaml (an unpinned pair would let the
published site diverge from what CI verified), and the source tree uses no
plugins, keeping a fall back to mkdocs-material cheap.

Verified: just docs-check clean, 53 pages build with no unresolved links or
anchors, and every tool parameter renders a real description.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_017oVnpzjRJQVWq9hJGE2Rs7

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

Copilot

Pull request overview

Builds the Zensical-based witan-context documentation site with generated references, mirrored package documentation, and Read the Docs validation.

Changes:

Adds site configuration, publishing, and CI freshness checks.
Generates reference material and organizes documentation using Diátaxis.
Moves historical specifications under docs/internals/ and updates links.

Reviewed changes

Copilot reviewed 56 out of 68 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`zensical.toml`	Configures site navigation and theme.
`.readthedocs.yaml`	Defines Read the Docs publishing.
`.github/workflows/docs.yml`	Adds freshness and build checks.
`.gitignore`	Ignores generated site output.
`justfile`	Adds documentation commands.
`bin/gen_docs.py`	Generates and mirrors documentation.
`docs/_data/environment.toml`	Defines environment-setting descriptions.
`AGENTS.md`	Documents the new site structure.
`configs/pi/README.md`	Updates the internals link.
`mcp/servers/witan/README.md`	Updates moved-document links.
`mcp/servers/witan/docs/USER_GUIDE.md`	Updates links and anchors.
`mcp/servers/witan/docs/CLI_REFERENCE.md`	Updates migration anchor.
`mcp/servers/witan/docs/migration-runbook.md`	Corrects its cutover anchor.
`mcp/servers/witan/docs/deployed-witan-onboarding.md`	Corrects migration links.
`mcp/servers/witan-code/README.md`	Updates the development guide link.
`docs/index.md`	Adds the site landing page.
`docs/getting-started/index.md`	Introduces the tutorial sequence.
`docs/getting-started/installation.md`	Documents installation and setup.
`docs/getting-started/first-memory.md`	Adds the memory tutorial.
`docs/getting-started/tasks-and-projects.md`	Adds coordination guidance.
`docs/getting-started/code-graph.md`	Adds code-indexing guidance.
`docs/guides/index.md`	Adds the guide directory.
`docs/guides/witan-user-guide.md`	Mirrors the witan guide.
`docs/guides/witan-code-user-guide.md`	Mirrors the code-graph guide.
`docs/reference/index.md`	Adds the reference overview.
`docs/reference/environment.md`	Generates environment reference.
`docs/reference/graph-schema.md`	Generates coordination schema reference.
`docs/reference/bridge-schema.md`	Generates bridge schema reference.
`docs/reference/mcp-tools/index.md`	Summarizes the MCP tool catalog.
`docs/reference/mcp-tools/tasks.md`	Generates task-tool reference.
`docs/explanation/index.md`	Adds the explanation directory.
`docs/explanation/architecture.md`	Documents system architecture.
`docs/explanation/memory-model.md`	Explains graph memory semantics.
`docs/explanation/task-coordination.md`	Explains task coordination.
`docs/explanation/code-graph/symbol-format.md`	Mirrors symbol-format documentation.
`docs/explanation/code-graph/symbol-table.md`	Mirrors symbol-table documentation.
`docs/explanation/code-graph/package-map.md`	Mirrors package-map documentation.
`docs/explanation/code-graph/stage2-stitching.md`	Mirrors stitching documentation.
`docs/explanation/code-graph/edge-precision-tiers.md`	Mirrors precision-tier documentation.
`docs/explanation/decisions/0002-witan-cedar-authorization-bundle.md`	Mirrors ADR 0002.
`docs/explanation/decisions/0003-atomic-task-claims-cas.md`	Mirrors ADR 0003.
`docs/explanation/decisions/0006-code-graph-branch-ownership-and-reaping.md`	Mirrors ADR 0006.
`docs/explanation/decisions/0007-local-to-shared-store-migration-transport.md`	Mirrors ADR 0007.
`docs/explanation/decisions/0008-optional-task-phase-tag.md`	Mirrors ADR 0008.
`docs/explanation/decisions/0009-stateless-mcp-protocol-era.md`	Mirrors ADR 0009.
`docs/internals/index.md`	Introduces historical internals.
`docs/internals/agent-memory.md`	Relocates the implementation guide.
`docs/internals/design/graph-structured-memory.md`	Relocates memory design notes.
`docs/internals/design/omnigraph-remote-call-overhead-spike.md`	Relocates performance findings.
`docs/internals/design/witan-workflow-ux-p1-spec.md`	Relocates workflow specification.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

witan is mature enough that its documentation is now the bottleneck: ~12,000 lines of real prose existed, spread across three package READMEs, nine ADRs, two user guides, runbooks, and a pile of design specs — none of it navigable, and none of it reachable by anyone who is not already in the repository. This stands up a Read the Docs site (project `witan-context`) covering the witan trio — witan-council, witan-code, witan-core — organised along Diátaxis lines: tutorials, how-to, reference, explanation. The load-bearing decision is that reference is GENERATED, not written. A handwritten tool reference is wrong by the second release, so `bin/gen_docs.py` derives all of it from the thing it documents — the registered FastMCP tool objects (60 tools, full JSON Schema), the cyclopts command tree, the `.pg` schema files — and `just docs-check` fails CI when a committed page no longer matches the code. The one hybrid is the environment reference: names are discovered from source, descriptions live in `docs/_data/environment.toml`, and a setting with no entry there is a hard error, so a new env var cannot ship undocumented. Package-resident prose (user guides, ADRs, code-graph internals) is MIRRORED rather than moved, so the copy next to the code stays authoritative for anyone reading the repo or the PyPI package, while the site gets it without a second copy to drift. Mirroring rewrites relative links — to site paths where the target is also mirrored, to GitHub otherwise — which is what took the build from 69 unresolved links to none. Two bugs the generator surfaced, both pre-existing and both fixed here: - Discovering env var names by scanning raw source text invents settings that do not exist. `WITAN_BRANCH` appears only in a comment saying there is *no* such override; `WITAN_EMBED_ENABLED` and `WITAN_REQUEST_TIMEOUT` are never read. Discovery now walks the AST, where comments are absent and docstrings are skipped, while ordinary string literals are kept so constants like `HTTP_TRANSPORT_ENV_VAR = "WITAN_OMNIGRAPH_HTTP"` still count. - `clean_desc` did not escape `|`, so a parameter description enumerating its options had every table cell past the fourth silently DROPPED — `task_update.status` rendered as the single word "open". 17 descriptions contain a raw pipe. `docs/design/` and `docs/agent-memory.md` move under `docs/internals/`, flagged as point-in-time documents rather than current-state, since much of what they specify changed during implementation. References to them across four packages are updated accordingly. Toolchain is Zensical, the Material for MkDocs team's successor generator — Material entered maintenance mode in early 2026. It is pre-1.0, so it is pinned identically in CI and `.readthedocs.yaml` (an unpinned pair would let the published site diverge from what CI verified), and the source tree uses no plugins, keeping a fall back to mkdocs-material cheap. Verified: `just docs-check` clean, 53 pages build with no unresolved links or anchors, and every tool parameter renders a real description. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oVnpzjRJQVWq9hJGE2Rs7

…fix the link check Both jobs in the docs workflow failed on their first run. Two independent bugs, neither caught locally. ## The link check matched its own success message Zensical prints "N issues found" when a link is broken and "No issues found" when none are — and the success message CONTAINS the failure substring, so `grep -q "issues found"` matched a perfectly clean build. The job's very first run reported "Build finished in 2.75s / No issues found" and then failed with "Zensical reported unresolved links or anchors." The predicate now requires a leading digit, which is what actually separates the two, and additionally requires the build to have reported completion — a build that dies before printing its summary would otherwise pass a digit-only test by saying nothing at all. Both directions are covered by cases checked against Zensical's real output. ## Generating docs required a downloaded 12MB binary `witan.server` constructs an `OmnigraphClient` at module scope, and `OmnigraphClient.__init__` resolves the omnigraph binary eagerly, so importing the servers to read their registered tool metadata needed a binary that generation never actually uses. That coupled the docs CI job to the binary's availability, and the bill came due on the first run: the upstream `edge` tag moved, the pinned checksum stopped matching, the install step failed, and a docs-only change went red for a reason that had nothing to do with docs. Two changes make generation self-sufficient: - **Pre-create the throwaway store.** `_ensure_graph` branches on whether the store exists: an existing one tolerates a missing binary (it catches the RuntimeError and returns), a missing one must be `omnigraph init`-ed and cannot. Creating the directory takes the tolerant branch. - **Stub the binary when there is no real one.** Only when neither PATH nor `~/.local/bin/omnigraph` has it — so on a machine with witan set up this is inert and behaviour is identical to before. It fires only in a bare environment like CI, which has no business downloading a binary to read docstrings. Verified by reproducing the exact CI condition locally: no omnigraph on PATH *and* a relocated HOME, since `_find_binary` falls back to `~/.local/bin/omnigraph` regardless of PATH — which is why the first two attempts to reproduce this passed misleadingly. Note this does NOT fix the `witan-code (code graph)` job, which fails on the same moved `edge` tag. That is a real upstream problem in the pinned checksum and needs its own change; it is simply no longer able to take the docs with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oVnpzjRJQVWq9hJGE2Rs7

Useful on its own — rebuilding the site after an upstream Zensical release needs no repo change to justify it — and an escape hatch for the situation that prompted adding it. After the previous commit was force-pushed, GitHub attached only CodeQL and the Copilot reviewer to the new head SHA. No `pull_request` run was created for either rebased branch, and no force-push event appears in the PR timeline, so both docs jobs simply had no result on the commit that fixes them. With no `workflow_dispatch` there was no way to ask for one without pushing another commit or closing and reopening the PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oVnpzjRJQVWq9hJGE2Rs7

…Python-dependent `just docs-check` failed in CI on two pages nobody had edited: - | `kind` | `env_var` \| `package` \| `service` \| `endpoint`? | … + | `kind` | `env_var` \| `endpoint` \| `package` \| `service`? | … The JSON Schema FastMCP derives from a `Literal` does not order its `enum` the same way on every Python. `BindingKind` — declared once, in one order — comes out as `env_var, package, service, endpoint` on 3.12 and `env_var, endpoint, package, service` on 3.14. Not hash ordering: the order is stable across `PYTHONHASHSEED` within a version and differs between versions, which is what distinguishes an interpreter difference from set iteration. Confirmed by generating under both. With no `.python-version` in the repo, `uv` resolves whatever it likes per environment — CI got 3.12, the author's machine had 3.14 — so the two disagreed about pages neither had touched. Left alone this is worse than a one-off red build: it makes `--check` fail for any contributor whose Python differs from CI's, on output they did not produce, with a diff that looks like corruption. Pinned in the shebang, so `just docs-gen` and `just docs-check` use the same interpreter for everyone, and matched in the workflow's `uv sync` so the venv agrees. The version is arbitrary — it only has to be fixed. Sorting the enum in the generator was the alternative and was rejected: it would make order interpreter-independent, but at the cost of rendering lifecycle enums alphabetically (`blocked | closed | in_progress | open` instead of `open | in_progress | blocked | closed`), which is worse for a reader than picking an interpreter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oVnpzjRJQVWq9hJGE2Rs7

…rose Review feedback on #254 (Copilot). Every claim was checked against the code before changing anything; all of them were right. **Readiness was wrong in two places.** Both the tutorial and the explanation said a task is ready when it is `open` and its blockers have closed. `readiness.status_pickable` treats `open` and `blocked` alike — a `blocked` task whose blockers all closed is exactly the case that should become pickable — and it also returns an `in_progress` task once its lease has lapsed, which is how work abandoned by a crashed session comes back. Only `closed` is never pickable. As written the docs disagreed with both `task_ready` and the generated reference. **The CI-only default-view rule is a shared-graph rule.** `owns_view` is explicit that "a local store has one user, who is its writer — nothing to arbitrate". The env-var reference and the code-graph tutorial both stated the restriction unconditionally, which told local users their ordinary default-branch reindex would be refused — in the tutorial that is precisely the command being taught. **`witan code stitch` does not build the bridge.** Indexing already writes the bindings; `stitch` prints the Stage-2 join and stores nothing. It was presented as a required build step, which gave readers an unnecessary command and the wrong lifecycle model. **Caller edges are heuristic, not ground truth.** The tutorial promised answers that are "structurally correct". The tools' own docstrings say the opposite — heuristic name resolution, may miss or over-report. The advantage over grep is syntax awareness, not correctness, and the page now says so and links the precision tiers. **The CLI and MCP surfaces are not equivalent.** The orientation page claimed nothing is available to one and not the other, contradicting its own sibling pages: `memory_store` is deliberately MCP-only (the CLI reads memory, it does not write it) and the code queries are tools only. **A mutual-exclusion claim survived the earlier fix.** The landing page's opening still read "what one agent claims, another cannot take" — the exact guarantee ADR 0003 disclaims and the rest of the site is careful about. **Two links pointed at a repository that does not exist.** `mitodl/omnigraph`; every other reference in the tree uses `ModernRelay/omnigraph`. **The design-doc move was incomplete.** `packages/agent-config-kit/README.md` (4), `packages/witan-core/README.md` (2), four source comments and one ADR still linked `docs/design/…`, which now 404s — despite the previous commit claiming cross-package references were updated. `packages/witan-core/CHANGELOG.md` is deliberately left: a changelog records the path as it was at the time. **The local preview used a different renderer from CI.** `docs-build` and `docs-serve` resolved the latest Zensical while CI and Read the Docs pin 0.0.56 — so a contributor could preview output neither CI nor production would produce, which is the drift the pin exists to prevent. Pinned all three. Verified: `just docs-check` clean, site builds with no unresolved links. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oVnpzjRJQVWq9hJGE2Rs7

The `repo` and `task_update.parent` descriptions were corrected on the tool-descriptions branch below this one, after these pages had been generated. The rebase brought the docstrings up without the rendered output, so `just docs-check` flagged both pages — which is exactly the drift the gate exists to catch, working on its first real case rather than a synthetic one. Regenerated, no hand edits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017oVnpzjRJQVWq9hJGE2Rs7

Copilot AI balanced review requested due to automatic review settings August 19, 2026 15:24

Copilot started reviewing on behalf of blarghmatey August 19, 2026 15:24 View session

Copilot AI reviewed Aug 19, 2026

View reviewed changes

blarghmatey force-pushed the witan-context-docs-site branch 2 times, most recently from 2a783a8 to e3d1591 Compare August 19, 2026 16:36

blarghmatey and others added 6 commits August 19, 2026 12:47

blarghmatey force-pushed the witan-context-docs-site branch from e3d1591 to d092fbf Compare August 19, 2026 16:49

blarghmatey force-pushed the witan-context-docs-site branch from d092fbf to 81eee64 Compare August 19, 2026 17:18

blarghmatey force-pushed the witan-context-docs-site branch from 81eee64 to d092fbf Compare August 19, 2026 19:34

Navigation Menu

Uh oh!

docs: build the witan-context documentation site - #254

docs: build the witan-context documentation site#254
blarghmatey wants to merge 6 commits into
witan-adr-renumberfrom
witan-context-docs-site

blarghmatey commented Aug 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

blarghmatey commented Aug 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants