diff --git a/docs/decisions/0001-myst-migration/0001-migrate-to-mystmd.md b/docs/decisions/0001-myst-migration/0001-migrate-to-mystmd.md new file mode 100644 index 0000000..0c18f7a --- /dev/null +++ b/docs/decisions/0001-myst-migration/0001-migrate-to-mystmd.md @@ -0,0 +1,83 @@ +# ADR 0001 — Migrate build system from Hugo to MyST-MD + +Date: 2026-05-11 +Status: Proposed — **open question for maintainers (see below)** +Branch: lb/myst-migration +Issue: scientific-python/scientific-python.org#846 + +## Context + +`learn.scientific-python.org` builds with Hugo via `make html` and deploys via +Netlify (auto-deploy on push to `main`). We want to convert the content files +to MyST syntax. + +Three realistic MyST toolchain options exist. They are not equivalent: +`jupyter-book` is a higher-level tool built on top of `mystmd`; `mystmd` is the +underlying engine and is available as both a Node package (npm) and a Python +package (pip/conda) that bundles Node internally. + +## Decision (proposed) + +Replace Hugo with **`mystmd` Python package** (`pip install mystmd`, +`myst build --html`) as the build tool. + +## Options considered + +| Option | Pros | Cons | +| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Stay on Hugo** | Theme parity with sibling SP sites; no content changes needed | Hugo is not Python; MyST content conversion still desirable long-term | +| **jupyter-book 2.x** | Pure Python (`pip install jupyter-book`); SP ecosystem familiar with JB; conda-forge package; handles notebook execution natively | Wraps `mystmd` under the hood — extra abstraction; config format (`_config.yml`, `_toc.yml`) is not portable to plain `myst.yml` if JB is dropped later; JB 2.x released late 2024 — docs and community experience thin; feature lag vs direct `mystmd` | +| **mystmd — Node CLI** (`npm install mystmd`) | Native runtime; latest npm releases immediately; same `myst.yml` config; active ExecutableBooks development | Requires Node.js in every build environment (Netlify, RTD, CI); unfamiliar to Python contributors | +| **mystmd — Python package** (`pip install mystmd`) | Python-centric install (pip/conda-forge); bundles Node internally — no separate Node needed; same `myst.yml` config as Node CLI (no extra abstraction layer); proven by `tools.scientific-python.org` PR #81; works on all considered deploy platforms | Node bundled internally — slightly opaque; PyPI/conda releases may lag npm by 1–7 days | + +## Rationale for proposed decision + +**Why not jupyter-book:** `learn.scientific-python.org` contains no Jupyter +notebooks; JB's primary value (notebook execution, Sphinx integration) does not +apply here. JB 2.x uses `mystmd` as its build engine, so the team would get +`mystmd` indirectly with an extra config layer on top. The JB config format +(`_config.yml`, `_toc.yml`) is not portable — if JB were dropped later, the +config would need to be rewritten to `myst.yml` from scratch. + +**Why not the Node CLI:** Requires Node.js in every build environment. SP +contributors and maintainers work in Python environments; npm is unfamiliar and +adds friction for new contributors. The Python package provides identical +functionality without any Node setup. + +**Why the Python package:** Fits SP's Python-centric workflow; `conda install +-c conda-forge mystmd` works for conda users. No Node.js needed in Netlify, +RTD, or GitHub Actions (Node is bundled inside the package). The `myst.yml` +config is identical to the Node CLI — switching delivery method later is a +one-line change. + +## Open question for maintainers + +> **Which MyST toolchain should `learn.scientific-python.org` adopt?** +> +> A. `mystmd` Python package — proposed above (`pip install mystmd`) +> B. `jupyter-book 2.x` — if the team prefers a unified JB-based approach +> C. `mystmd` Node CLI — if the team prefers the native Node runtime +> +> This is the foundational decision for the migration. All downstream ADRs +> (0002–0007) assume option A. If maintainers choose B, the `myst.yml` / +> `_toc.yml` structure and config format change significantly. Option C +> requires Node.js toolchain setup in `netlify.toml` and CI. + +## Installation + +For Netlify builds and RTD: `pip install mystmd` (no Node configuration needed +in `netlify.toml`; see ADR 0007). + +Local dev: install via any preferred method (pip, conda, npm — developer's +choice). The repo does not mandate a specific local environment. + +## Consequences + +- `myst build --html` replaces `make html` (Hugo) +- `pip install mystmd` is the chosen delivery method; no Node toolchain required + in CI or `netlify.toml` +- The `scientific-python-hugo-theme` submodule is removed (see ADR 0003) +- `netlify.toml` is updated to use `mystmd` (see PLAN.md Phase 6) +- Nine content files require shortcode conversion (see ADR 0002) +- Footer/quicklinks are not yet supported in MyST default templates (see ADR 0005) +- Other SP repos remain on Hugo until they choose to migrate (see ADR 0006) diff --git a/docs/decisions/0001-myst-migration/0002-shortcode-mapping.md b/docs/decisions/0001-myst-migration/0002-shortcode-mapping.md new file mode 100644 index 0000000..8074f91 --- /dev/null +++ b/docs/decisions/0001-myst-migration/0002-shortcode-mapping.md @@ -0,0 +1,57 @@ +# ADR 0002 — Hugo shortcode → MyST directive mapping + +Date: 2026-05-11 +Status: Proposed +Branch: lb/myst-migration + +## Context + +Nine content files use Hugo shortcodes that MyST does not understand: + +| Shortcode type | Files affected (pre-Phase-2 names) | +| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `{{< grid >}}` / `[[item]]` | `_index.md` (root), `contributors/_index.md`, `documentation/_index.md` | +| `{{< admonition >}}` | `maintainers/_index.md`, `maintainers/interacting-with-new-contributors.md`, `maintainers/managing-conflict.md`, `maintainers/meeting_types.md`, `contributors/first-contribution.md`, `community/onboarding.md` | + +## Decision + +Convert shortcodes in two **type-batched commits** (one per shortcode type, +not one per file), using the following canonical mappings: + +``` +{{< grid columns="1 2 2 3" >}} → ::::{grid} 1 2 2 3 + :gutter: 2 + +[[item]] → :::{card} + type = 'card' :link: <link> + title = 'X' + link = 'y' <body> + body = 'z' ::: + +{{< /grid >}} → :::: + +{{< admonition warning >}}… → :::{warning} +{{< /admonition >}} … + ::: + +{{< admonition note >}}… → :::{note} +{{< /admonition >}} … + ::: +``` + +## Options considered + +- **Custom MyST plugin** to interpret Hugo TOML shortcode syntax — high effort, + no value once content is converted. +- **Per-file commits** — nine files but ten shortcode occurrences (one file + has two grid blocks); reviewers re-read the same mapping repeatedly. +- **Type-batched commits** — two diffs total; reviewers evaluate the mapping + pattern once per type. + +## Consequences + +- Two commits in Phase 3 of the migration (see [PLAN.md](PLAN.md)) +- Reviewers can verify correctness by comparing rendered output against the + Hugo-built site for these nine pages +- `grep -rE '\{\{<' content/` returns zero matches after these commits +- `content/index.md` is the canonical style reference going forward diff --git a/docs/decisions/0001-myst-migration/0003-remove-hugo-theme-submodule.md b/docs/decisions/0001-myst-migration/0003-remove-hugo-theme-submodule.md new file mode 100644 index 0000000..09b3621 --- /dev/null +++ b/docs/decisions/0001-myst-migration/0003-remove-hugo-theme-submodule.md @@ -0,0 +1,42 @@ +# ADR 0003 — Remove scientific-python-hugo-theme submodule + +Date: 2026-05-11 +Status: Proposed +Branch: lb/myst-migration + +## Context + +`themes/scientific-python-hugo-theme` is a git submodule pinned to `v0.21`. +The same theme submodule is also referenced by three sibling repos: +`scientific-python.org`, `blog.scientific-python.org`, and +`tools.scientific-python.org`. Each repo pins the submodule independently in +its own `.gitmodules`; they do not share a checkout. + +MyST-MD does not use Hugo themes. Once Hugo is removed from this repo, the +submodule has no consumer here. + +## Decision + +Remove the submodule from this repo in Phase 7 of the migration. + +```bash +git submodule deinit -f themes/scientific-python-hugo-theme +git rm themes/scientific-python-hugo-theme +rm -rf .git/modules/themes/scientific-python-hugo-theme +``` + +## Options considered + +1. **Remove in this PR** — clean cut; no dead code after Hugo is gone. +2. **Keep until all four SP repos migrate** — delays cleanup by months or + quarters; leaves a submodule that nothing in this repo uses. +3. **Vendor a snapshot** — no benefit; MyST doesn't use it. + +## Consequences + +- The `themes/` directory is deleted from this repo +- The three sibling repos are **unaffected** — they reference the submodule + from their own `.gitmodules` and pin their own SHA +- The upstream `scientific-python-hugo-theme` repo is not affected +- A follow-up PR to the sibling repos removes their copies when they migrate + (see ADR 0006) diff --git a/docs/decisions/0001-myst-migration/0004-defer-cookie-jekyll.md b/docs/decisions/0001-myst-migration/0004-defer-cookie-jekyll.md new file mode 100644 index 0000000..8d3ad33 --- /dev/null +++ b/docs/decisions/0001-myst-migration/0004-defer-cookie-jekyll.md @@ -0,0 +1,42 @@ +# ADR 0004 — Defer migration of external-content/cookie (Jekyll) + +Date: 2026-05-11 +Status: Proposed +Branch: lb/myst-migration + +## Context + +`external-content/cookie` is a Jekyll site (git submodule, pinned to +`2025.10.01`). It is built separately via `make cookie` and its output is +merged into `public/development/` before deploy. It has its own upstream +release cadence and contributors independent of `learn`. + +## Decision + +Leave `external-content/cookie` unchanged in this PR. File a follow-up issue: +_"MyST: migrate external-content/cookie off Jekyll"_. + +The `make cookie`, `make external`, `cookie_ruby_deps`, `cookie_web_prepare`, +and `prepare` Makefile targets are preserved. `make html-all` continues to +build the MyST site then merge the Jekyll output into `public/`. + +## Options considered + +1. **Keep as-is permanently** — MyST output and Jekyll output coexist forever; + `ghp-import` merges them. Lowest risk but leaves a Jekyll dependency + indefinitely. +2. **Convert cookie to MyST** — large, independent effort; distracts from this + PR and would require its own review. +3. **Drop cookie** — would break the `/development/` path; not acceptable + without a replacement. +4. **Defer with follow-up issue** — keep `make cookie` working now; track + conversion separately so it gets its own focused review. + +## Consequences + +- `external-content/cookie` submodule remains at `2025.10.01` +- `public/development/` continues to be produced by Jekyll +- The `html-all` Makefile target reconciles MyST's `_build/html/` output + with cookie's `public/development/` via `mkdir -p public && cp -r _build/html/* public/` + before `make external` overlays `/development/` +- A follow-up issue tracks the eventual Jekyll → MyST conversion diff --git a/docs/decisions/0001-myst-migration/0005-defer-footer-quicklinks.md b/docs/decisions/0001-myst-migration/0005-defer-footer-quicklinks.md new file mode 100644 index 0000000..26e4a2c --- /dev/null +++ b/docs/decisions/0001-myst-migration/0005-defer-footer-quicklinks.md @@ -0,0 +1,49 @@ +# ADR 0005 — Defer footer and quicklinks to follow-up + +Date: 2026-05-11 +Status: Proposed +Branch: lb/myst-migration + +## Context + +Hugo `config.yaml` defines two visual elements that have no direct equivalent +in MyST default templates: + +1. **Footer social icons** — GitHub, YouTube, Mastodon, Discourse, Discord +2. **Quicklinks columns** — three columns of site-wide nav links (About, + Maintainers/SPECs, Press kit) + +MyST's default HTML template renders a minimal footer with no configurable +social links or quicklinks. + +## Decision + +Ship this PR with the MyST default footer. File a follow-up issue: +_"MyST: footer + quicklinks parity with Hugo theme"_. + +Add a comment block at the bottom of `myst.yml` pointing at the follow-up +issue number so the gap is immediately discoverable. + +## Options considered + +1. **Custom MyST theme / template override** — achieves full parity but is a + month of separate work; blocks the migration on a visual detail. +2. **Static HTML injected via template** — fragile; bypasses MyST conventions + and creates a maintenance burden. +3. **`site.parts.footer:` with a `footer.md` file + custom CSS + scienceicons + plugin** — uses MyST's built-in parts mechanism; no custom theme required. + Demonstrated in `tools.scientific-python.org` PR #81 (brianhawthorne, + October 2025). Viable path for the follow-up issue. +4. **Defer with documented issue** — unblocks the migration for SciPy 2026; + footer work is tracked and discoverable. + +## Consequences + +- The deployed site will have a minimal footer until the follow-up is resolved +- `config.yaml`'s `params.footer` and `params.quicklinks` sections are not + ported to `myst.yml` +- The follow-up issue is linked from `myst.yml` and from the PR description +- Option 3 above (`site.parts.footer:` + CSS) is the recommended implementation + path for the follow-up; `tools.scientific-python.org` PR #81 is the reference +- If the SP community decides MyST theming is too limited, this ADR is a + natural decision point to reconsider the tool choice (see ADR 0001) diff --git a/docs/decisions/0001-myst-migration/0006-sibling-repo-migration.md b/docs/decisions/0001-myst-migration/0006-sibling-repo-migration.md new file mode 100644 index 0000000..c0698cb --- /dev/null +++ b/docs/decisions/0001-myst-migration/0006-sibling-repo-migration.md @@ -0,0 +1,52 @@ +# ADR 0006 — Sibling SP repos migrate independently + +Date: 2026-05-11 +Status: Proposed +Branch: lb/myst-migration + +## Context + +The Scientific Python ecosystem has four Hugo-based sites sharing the same +theme submodule: + +| Repo | Domain | +| ----------------------------------------------- | ------------------------------------------- | +| `scientific-python/learn.scientific-python.org` | learn.scientific-python.org ← **this repo** | +| `scientific-python/scientific-python.org` | scientific-python.org | +| `scientific-python/blog.scientific-python.org` | blog.scientific-python.org | +| `scientific-python/tools.scientific-python.org` | tools.scientific-python.org | + +Cross-site nav links are plain absolute URLs (not build-time references). +There is no shared build pipeline coupling the repos. + +`tools.scientific-python.org` already has an open MyST migration PR +([#81](https://github.com/scientific-python/tools.scientific-python.org/pull/81), +brianhawthorne, opened October 2025, stale as of May 2026). It demonstrates +a working shortcode conversion and a footer implementation using +`site.parts.footer:` + custom CSS (see ADR 0005 option 3). + +## Decision + +`learn` migrates first. File one tracking issue per sibling repo after this +PR merges, each linking to this PR as a worked example. Sibling repos adopt +MyST on their own schedule. + +## Options considered + +1. **Migrate all four in lock-step** — synchronizes visual consistency; blocks + `learn` on the slowest-moving repo. +2. **`learn` first; siblings when ready** — proves the pattern; doesn't block + SciPy 2026 deadline. +3. **Wait for MyST theme parity** — defers everything until ADR 0005 follow-up + is resolved; not necessary since content parity is achievable now. + +## Consequences + +- Cross-site nav continues to work: all links are absolute URLs +- Sibling repos remain on Hugo until they choose to migrate; no visual + breakage to end users +- This PR and `tools.scientific-python.org` PR #81 together form the reference + corpus for sibling repos evaluating MyST migration +- Tracking issues are filed in Phase 8 of the migration plan +- The `scientific-python-hugo-theme` submodule removal in ADR 0003 affects + only this repo; the other three repos remove it in their own migration PRs diff --git a/docs/decisions/0001-myst-migration/0007-deploy-strategy.md b/docs/decisions/0001-myst-migration/0007-deploy-strategy.md new file mode 100644 index 0000000..abe76eb --- /dev/null +++ b/docs/decisions/0001-myst-migration/0007-deploy-strategy.md @@ -0,0 +1,62 @@ +# ADR 0007 — Update netlify.toml for MyST build + +Date: 2026-05-11 +Status: Proposed +Branch: lb/myst-migration + +## Context + +Both `scientific-python.org` and `learn.scientific-python.org` deploy via +Netlify, which auto-deploys on push to `main` and generates PR preview deploys. +The build command is defined in `netlify.toml`. The current command builds with +Hugo + Dart Sass. + +GitHub Actions runs only a lint workflow (`lint.yml`). There is no gh-pages +deploy workflow. + +## Decision + +Update `netlify.toml` in Phase 6 of the migration: remove the Dart Sass and +Hugo toolchain setup; add `pip install mystmd` before the existing +`make html-all` call. Build command, publish directory, and +`netlify-plugin-checklinks` are otherwise unchanged. + +## Options considered + +1. **Update `netlify.toml`** — minimal change; keeps Netlify as the deploy + target, PR previews continue to work automatically. +2. **Replace Netlify with gh-pages** — larger change; requires creating new + GitHub Actions deploy workflows, reconfiguring DNS, and losing Netlify PR + previews. Out of scope for this PR. +3. **Keep `netlify.toml` for the checklinks plugin only** — the Netlify + checklinks plugin can be replaced by a `lychee`-based GitHub Actions job + as a separate improvement; not required for this migration. +4. **Drop Netlify entirely; use CircleCI for builds + circleci-artifacts- + redirector-action for PR previews** — demonstrated by + `tools.scientific-python.org` PR #81. Viable but introduces CircleCI + account dependency and is a larger infrastructure change than needed here. +5. **Migrate to Read the Docs** — RTD has first-class MyST/Sphinx support, + built-in PR preview deploys (including for forks), and is already used + widely across the Scientific Python ecosystem. Would replace Netlify + entirely; requires a `.readthedocs.yaml` config and DNS reconfiguration. + Resolves the fork-contributor preview gap (ADR 0007 future work item 1) + as a side effect. Not pursued in this PR — custom domain setup and RTD + account provisioning are out of scope for the migration itself. + +## Consequences + +- Dart Sass and Hugo version pins removed from `netlify.toml` +- `pip install mystmd` added before `make html-all` in the build command +- Build command (`make html-all`), publish dir (`public/`), and + `netlify-plugin-checklinks` are unchanged +- Netlify auto-deploy and PR previews continue unchanged + +## Future work (out of scope) + +Two follow-up improvements; both filed as issues before Phase 4 commit 2 +(see PLAN.md Phase 4 prerequisite): + +- **gh-pages PR preview**: allows contributors working from a fork to + preview builds on their own GitHub Pages without requiring Netlify access. +- **Replace `netlify-plugin-checklinks` with a `lychee`-based GitHub Actions + job**: keeps link checking in CI, removes the Netlify plugin dependency. diff --git a/docs/decisions/0001-myst-migration/PLAN.md b/docs/decisions/0001-myst-migration/PLAN.md new file mode 100644 index 0000000..8780301 --- /dev/null +++ b/docs/decisions/0001-myst-migration/PLAN.md @@ -0,0 +1,404 @@ +# Implementation Plan — Hugo → MyST-MD migration + +Branch: `lb/myst-migration` +Issue: scientific-python/scientific-python.org#846 +Deadline: 2026-07-13 (SciPy 2026) + +This plan is structured commit-by-commit. Every commit is independently +reviewable and the history is intended to read as a narrative for other +Scientific Python maintainers evaluating MyST. + +--- + +## Phase 0 — Pre-work (no code; decisions only) + +Goal: lock the six open questions from REQUIREMENTS.md before writing +production code. Output of this phase is the full planning record in +`docs/decisions/0001-myst-migration/`. + +### Commits + +1. `docs: add ADRs for MyST migration decisions` + - Files: `docs/decisions/README.md` (new), + `docs/decisions/0001-myst-migration/README.md` (new), + `docs/decisions/0001-myst-migration/0001-migrate-to-mystmd.md` (new), + `docs/decisions/0001-myst-migration/0002-shortcode-mapping.md` (new), + `docs/decisions/0001-myst-migration/0003-remove-hugo-theme-submodule.md` (new), + `docs/decisions/0001-myst-migration/0004-defer-cookie-jekyll.md` (new), + `docs/decisions/0001-myst-migration/0005-defer-footer-quicklinks.md` (new), + `docs/decisions/0001-myst-migration/0006-sibling-repo-migration.md` (new), + `docs/decisions/0001-myst-migration/0007-deploy-strategy.md` (new) + - Why: every subsequent commit references a decision in this directory. + Reviewers can read the "why" once and the diffs become small. ADR + format ensures these records survive as an archive even after the + migration is complete. + +2. `docs: add REQUIREMENTS and PLAN to migration ADR directory` + - Files: `docs/decisions/0001-myst-migration/REQUIREMENTS.md` (new), + `docs/decisions/0001-myst-migration/PLAN.md` (new) + - Why: REQUIREMENTS and PLAN are part of the migration record. Keeping + them alongside the ADRs means a reviewer has the full context — + what was needed, what was decided, and how it was sequenced — in + one directory. + +### Verification + +- All 7 ADRs answer their question with Context → Decision → Options + considered → Consequences (Decision-first style — rationale follows). +- No build artifacts touched; `make html` still produces the existing + Hugo site. + +### CI status: green (no code changed). + +### Recommended answers to the six open questions + +| # | Question | Recommendation | One-line rationale | +| --- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | Footer / quicklinks | **Defer**: file follow-up issue, ship MyST default footer | Custom MyST theme work blocks the migration; visual parity is not a SciPy 2026 must-have. | +| 2 | Cookie / Jekyll | **Defer**: keep the submodule and `make cookie` target unchanged in this PR | External content lives at `/development/`; touching Jekyll doubles the PR scope. | +| 3 | Netlify | **Update `netlify.toml`**: remove Hugo/Dart Sass toolchain setup; add `pip install mystmd`; keep `make html-all` as build command | Netlify is the active deploy target for the live site and PR previews. Minimal change; see ADR 0007. | +| 4 | Preview deploy | **No change needed**: Netlify handles PR previews automatically from `netlify.toml` | Updating the build command in Phase 5 is sufficient; no workflow changes required. | +| 5 | Other SP repos | **Tracking issues only**, filed at end of PR | This PR proves the pattern; sibling repos adopt independently. No code coupling. | +| 6 | Theme submodule | **Remove in this PR** | The submodule is consumed only by Hugo, which is being removed. The other three repos vendor their own submodule and are unaffected. | + +--- + +## Phase 1 — Foundation: get `myst build` working + +Goal: a developer who clones the repo and runs `myst build --html` gets a +complete site. Hugo is still present and still builds; the two systems coexist. +Installation of `mystmd` is left to the developer's preference (pip, conda, +npm, etc.). + +### Commits + +1. `build: add myst.yml at repo root` + - Files: `myst.yml` (new) + - Create `myst.yml` at the repository root with `site:` and `project:` + config; set `toc:` file paths prefixed with `content/` using the + current filenames (e.g. `file: content/_index.md`); Phase 2 updates + these entries after the renames + - Set `project.id` to a freshly generated UUID (e.g. via + `python3 -c "import uuid; print(uuid.uuid4())"`) — without this MyST + regenerates the id on each build, breaking caching and cross-references + - Why: `myst.yml` can also live in a subdirectory (build via `cd subdir && +myst build --html`) but root placement with `content/`-prefixed toc paths + is simpler for this repo layout and avoids wrapping every Make target. + - Note: MyST supports separating the toc into a standalone file via the + `extends:` key in `myst.yml` (`extends: _toc.yml`). Docs: + https://mystmd.org/guide/frontmatter#composing-myst-yml — the community + preference (per `tools.scientific-python.org` PR #81 review) is to keep + toc and site config separate once the toc grows. For `learn` the toc is + small enough to inline for now; split it in a follow-up if it grows. + +### Verification + +- `myst build --html` exits 0 and writes to `_build/html/`. + Note: `_index.md` files are explicitly listed in `toc:`, so MyST processes + them even though `_`-prefixed files are excluded from auto-discovery. +- Spot-check `_build/html/index.html` exists and renders content from + `content/_index.md` (file is renamed to `index.md` in Phase 2). +- MyST bundles thebe (for executable notebooks) even when unused; expect + `*thebe*.js` files in `_build/html/`. Harmless — do not add a cleanup + step unless output size becomes a concern. +- `make html` (Hugo) still works — coexistence proven. + +### CI status: still green. + +Only `lint.yml` runs at this point; no deploy workflow exists yet. + +--- + +## Phase 2 — File renames: align with MyST conventions + +Goal: every section index is `index.md`, not `_index.md`. No content +changes. + +### Commits + +1. `content: rename _index.md to index.md (6 files)` + - Files renamed: + - `content/_index.md` → `content/index.md` + - `content/about/_index.md` → `content/about/index.md` + - `content/community/_index.md` → `content/community/index.md` + - `content/contributors/_index.md` → `content/contributors/index.md` + - `content/documentation/_index.md` → `content/documentation/index.md` + - `content/maintainers/_index.md` → `content/maintainers/index.md` + - Update `myst.yml` `toc:` entries to match. + - Why: `_index.md` is a Hugo idiom; MyST excludes `_`-prefixed files + from auto-discovery. Renaming once now means every later commit + references the final filename. + - Use `git mv` so blame is preserved. + +### Verification + +- `myst build --html` still succeeds. +- Hugo build (`make html`) is now broken — acceptable; we are + intentionally cutting over. + +### CI status: **Netlify deploy fails** (Hugo can no longer find section + +indices after the rename). Lint passes. This is the first commit where +the deploy is intentionally broken; it remains broken through Phase 5. +All commits on this PR are merged together — local `myst build --html` +is the test during development. Deploy is restored in Phase 6. + +--- + +## Phase 3 — Shortcode conversion (split by directive type) + +Goal: zero Hugo shortcodes remain in `content/`. Reviewers evaluate the +mapping pattern **once per shortcode type**, not nine times. + +### Commits + +1. `content: convert {{< grid >}} to MyST grid/card directives` + - Files (3): + - `content/index.md` + - `content/contributors/index.md` + - `content/documentation/index.md` + - Mapping (canonical, documented in ADR 0002): + + ``` + {{< grid columns="1 2 2 3" >}} → ::::{grid} 1 2 2 3 + :gutter: 2 + + [[item]] → :::{card} <title> + type = 'card' :link: <link> + title = 'X' + link = 'y' <body> + body = 'z' ::: + + {{< /grid >}} → :::: + ``` + + - Note: `content/contributors/index.md` contains **two** grid blocks; + its diff is larger than the other two files. + - Why: one commit, one pattern. A reviewer reads this diff and learns + the entire grid mapping; the per-file changes are mechanical. + +2. `content: convert {{< admonition >}} to MyST admonitions` + - Files (6): + - `content/maintainers/index.md` + - `content/maintainers/interacting-with-new-contributors.md` + - `content/maintainers/managing-conflict.md` + - `content/maintainers/meeting_types.md` + - `content/contributors/first-contribution.md` + - `content/community/onboarding.md` + - Mapping: + ``` + {{< admonition warning >}} … {{< /admonition >}} → :::{warning} + … + ::: + {{< admonition note >}} … {{< /admonition >}} → :::{note} + ``` + - Why: same rationale — one diff, one pattern. + +### Verification + +- `grep -rE '\{\{<' content/` returns no matches. +- `myst build --html` builds with zero "unknown directive" warnings. +- Visual diff: open `_build/html/contributors/index.html` and confirm + cards render with titles, body text, and working links. + +### CI status: Netlify deploy still failing (same reason as Phase 2). + +--- + +## Phase 4 — MyST config completeness + +Goal: `myst.yml` carries the metadata that used to live in +`config.yaml`, **except** for what we explicitly deferred (footer, +quicklinks). + +### Commits + +1. `config: populate myst.yml site metadata` + - Files: `myst.yml` + - Add `site.title`, `site.domain`, `site.description`, + `site.options.logo`, `site.options.logo_dark`, `site.options.favicon`, + and `nav:` from `config.yaml`'s `params.navbar`. + - Why: feature parity for everything except footer/quicklinks, which + ADR 0005 defers. + +2. `config: document deferred footer/quicklinks` + - Files: add a comment block at the bottom of `myst.yml` with + footer/quicklinks deferred, including the actual follow-up issue + numbers. + - **Prerequisite — file these four follow-up issues before writing + this commit** (so real issue numbers are in the comment from the + start; no follow-up docs commit needed): + 1. "MyST: footer + quicklinks parity with Hugo theme" (ADR 0005) + 2. "MyST: migrate external-content/cookie off Jekyll" (ADR 0004) + 3. "MyST: gh-pages PR preview for fork contributors" (ADR 0007) + 4. "MyST: replace netlify-plugin-checklinks with lychee GH Actions" + (ADR 0007 option 3) + - Why: issue numbers known before the commit is written; comment is + complete from the start. + +### Verification + +- `myst build --html` produces no warnings about unknown config keys. +- Header nav links to scientific-python.org, blog, tools. + +### CI status: Netlify deploy still failing (same reason as Phase 2). + +--- + +## Phase 5 — Makefile: update for MyST + +Goal: `make html` builds with MyST, `make html-all` builds MyST site plus +cookie content into `public/`. Netlify can then call `make html-all` unchanged. + +### Commits + +1. `build: rewrite Makefile for MyST` + - Files: `Makefile` + - New targets: + - `html` → `myst build --html` (output: `_build/html/`) + - `serve` → `myst start` + - `clean` → `rm -rf _build public` + - `html-all` → `make html`, then `mkdir -p public && cp -r _build/html/* public/`, + then `make external` overlays cookie at `public/development/` + - `cookie` / `external` / `cookie_ruby_deps` / `cookie_web_prepare` / + `prepare` / `help` — keep (ADR 0004: cookie deferred; `help` is + `.DEFAULT_GOAL`) + - Update `.PHONY` declaration to list the new and retained targets + - Why: Makefile must be updated before netlify.toml so Phase 6 can + call `make html-all` and produce the full site including cookie. + - Note: `make html-all` does not invoke `prepare` automatically. On a + fresh clone, run `make prepare` first to initialise the cookie + submodule before running `make html-all`. + +### Verification + +- `make html` → site at `_build/html/`. +- `make serve` → live reload at `http://localhost:3000` (MyST default). +- `make html-all` → MyST output plus cookie merged into `public/`. + +**Pre-Phase 6 sign-off (R3):** before updating Netlify, confirm: + +- Every page in the Hugo build has a counterpart in `_build/html/`. + Hugo is broken on this branch since Phase 2; produce the reference list + by running `make html` from a clean checkout of `main` (Hugo still works + there) and listing `public/`. +- Nav links, card grids, and admonitions render correctly in the MyST build. + +Note: automated parity tests were considered and rejected — the site is static +content with no dynamic behaviour; a manual visual comparison is sufficient for +a one-time migration. + +### CI status: Netlify deploy still failing (Hugo still in netlify.toml). + +--- + +## Phase 6 — CI: update netlify.toml for MyST + +Goal: Netlify deploys the full MyST-built site including cookie content. +PR previews resume. This is the cutover — after +this phase the live site is built by MyST. + +### Commits + +1. `ci: update netlify.toml to build with mystmd` + - Files: `netlify.toml` + - Remove `HUGO_VERSION`, `DART_SASS_VERSION`, `DART_SASS_URL` env vars + and the Dart Sass download/install steps from the build command + - Add `pip install mystmd` before `make html-all` + - Keep `make html-all` as the build command (unchanged) + - Keep `publish = "public"` (unchanged) + - Keep `PYTHON_VERSION = "3.13"` (needed for pip) + - Keep `netlify-plugin-checklinks` (link checking preserved) + - Why: the build command already calls `make html-all`; only the + Hugo/Dart Sass toolchain setup needs replacing. `pip install mystmd` + is the chosen install method — see ADR 0001. + +### Verification + +Verify before opening the PR. **Assumes the branch is pushed to the +upstream repo** (`scientific-python/learn.scientific-python.org`), not a +fork — Netlify preview deploys are only triggered for branches in the +upstream repo. A fork-based contributor cannot verify this way before +opening the PR; the Netlify build log on the opened PR is the first +opportunity. The gh-pages PR preview follow-up issue (filed in Phase 4) +covers that gap. + +- Push the branch; Netlify preview build log shows `make html-all` + running and completing with exit 0. +- `netlify-plugin-checklinks` (kept in `netlify.toml`) reports no broken + external links. +- Confirm `/development/` cookie content is present in the preview deploy. +- Confirm the Netlify preview renders pages correctly. + +### CI status: **green** (lint passes; Netlify deploy succeeds). + +--- + +## Phase 7 — Remove Hugo artifacts + +Goal: a fresh clone has no Hugo references. + +### Commits + +1. `chore: remove Hugo site config` + - Files: delete `config.yaml` + - Why: nothing reads it. + +2. `chore: remove scientific-python-hugo-theme submodule` + - Files: `.gitmodules` (edit), `themes/scientific-python-hugo-theme` + (deinit + remove) + - Steps: + ``` + git submodule deinit -f themes/scientific-python-hugo-theme + git rm themes/scientific-python-hugo-theme + rm -rf .git/modules/themes/scientific-python-hugo-theme + ``` + - Why: ADR 0003 — the submodule is only consumed by Hugo. The + other three SP repos vendor their own copy; this removal is + unilateral and safe. + +3. `chore: update .gitignore for MyST output` + - Files: `.gitignore` + - Drop `.hugo*` (Hugo build cache) and `resources/` (Hugo generated + assets); add `_build/`. Keep `public/` (`html-all` writes there). + `*~` and `.DS_Store` remain unchanged. + - Why: clean working tree. + +### Verification + +- `rg -i hugo .` returns matches only in `docs/decisions/` and git history. +- `git submodule status` shows only `external-content/cookie`. +- `myst build --html` still green. +- CI still green. + +### CI status: green. + +--- + +## Phase 8 — PR + tracking issues + +Goal: ship. + +No code commits in this phase. Deferred-work follow-up issues were filed +as a Phase 4 prerequisite and their numbers are already in `myst.yml`. + +### Actions + +1. **Open PR** against `scientific-python/learn.scientific-python.org:main`. + - Title: `Migrate from Hugo to MyST-MD` + - Body: link to `docs/decisions/0001-myst-migration/`, summarise phases, link to issue #846. + - Request review from the maintainers tagged on #846. + +2. **File sibling-repo tracking issues** (one per repo, ADR 0006; filed + after PR opens so each issue can link to it as a worked example): + - `scientific-python/scientific-python.org`: "Evaluate MyST-MD + migration (parallel to learn)" + - `scientific-python/blog.scientific-python.org`: same + - `scientific-python/tools.scientific-python.org`: same + +### Verification + +- PR open, all checks green, myst.yml comment block links to all four + deferred-work issues. + +--- diff --git a/docs/decisions/0001-myst-migration/README.md b/docs/decisions/0001-myst-migration/README.md new file mode 100644 index 0000000..eaa8849 --- /dev/null +++ b/docs/decisions/0001-myst-migration/README.md @@ -0,0 +1,43 @@ +# 0001 — MyST-MD Migration + +Decisions made during the migration of `learn.scientific-python.org` from +Hugo to MyST-MD (`mystmd` CLI). + +Branch: `lb/myst-migration` — Issue: scientific-python/scientific-python.org#846 + +## AI assistance disclosure + +The planning documents and ADRs in this directory were drafted with the +assistance of Claude Code (Anthropic). The author reviewed, tested, and takes +full responsibility for all content. All final editorial decisions, commit +messages, and PR communication are human-authored. + +Scientific Python does not yet have a published AI contribution policy. +This disclosure follows the spirit of [SciPy's AI contribution +policy](https://scipy.github.io/devdocs/dev/conduct/ai_policy.html), the +closest published reference in the ecosystem. + +## Planning docs + +| File | Purpose | +| ---------------------------------- | ---------------------------------------------------------- | +| [REQUIREMENTS.md](REQUIREMENTS.md) | What the migration must achieve; testable success criteria | +| [PLAN.md](PLAN.md) | Commit-by-commit implementation sequence | + +## ADRs + +| # | Title | Status | +| ------------------------------------------- | --------------------------------------------------- | -------- | +| [0001](0001-migrate-to-mystmd.md) | Migrate build system from Hugo to MyST-MD | Proposed | +| [0002](0002-shortcode-mapping.md) | Hugo shortcode → MyST directive mapping | Proposed | +| [0003](0003-remove-hugo-theme-submodule.md) | Remove scientific-python-hugo-theme submodule | Proposed | +| [0004](0004-defer-cookie-jekyll.md) | Defer migration of external-content/cookie (Jekyll) | Proposed | +| [0005](0005-defer-footer-quicklinks.md) | Defer footer and quicklinks to follow-up | Proposed | +| [0006](0006-sibling-repo-migration.md) | Sibling SP repos migrate independently | Proposed | +| [0007](0007-deploy-strategy.md) | Update netlify.toml for MyST build | Proposed | + +**ADR status convention:** `Proposed` means the decision is implemented on +this branch and awaiting maintainer review. `Accepted` means the PR has +merged and the decision stands. The Decision section of each ADR uses +present tense to describe what the branch implements, not to assert a +final verdict. diff --git a/docs/decisions/0001-myst-migration/REQUIREMENTS.md b/docs/decisions/0001-myst-migration/REQUIREMENTS.md new file mode 100644 index 0000000..4219152 --- /dev/null +++ b/docs/decisions/0001-myst-migration/REQUIREMENTS.md @@ -0,0 +1,163 @@ +# MyST Migration — Requirements Document + +> Author: Lundy Bernard +> Date: 2026-05-11 +> Branch: lb/myst-migration +> Issue: https://github.com/scientific-python/scientific-python.org/issues/846 +> Deadline: SciPy 2026 — 2026-07-13 (Minneapolis) +> Plan: [PLAN.md](PLAN.md) + +--- + +## Purpose + +Replace the Hugo-based build system for `learn.scientific-python.org` with +[MyST-MD](https://mystmd.org/) (the `mystmd` CLI). The migration must be +exploratory and reviewable: the git history and accompanying documentation +should tell the story clearly enough that other Scientific Python maintainers +can evaluate the approach, raise concerns, and vote to adopt or reject it. + +--- + +## Context + +### Repository layout + +`learn.scientific-python.org` has three build layers today: + +| Layer | Tool | Status | +| ----------------------- | ----------------------------------------------------- | ----------------------- | +| Main site | Hugo + `scientific-python-hugo-theme` (git submodule) | Active — `make html` | +| External/cookie content | Jekyll (`external-content/cookie` submodule) | Active — `make cookie` | +| CI / deploy | Netlify (auto-deploy on push to main, PR previews) | Active — `netlify.toml` | + +Starting state on branch `lb/myst-migration` (clean branch off updated main): + +- `content/_index.md` and 5 subdirectory index files use Hugo naming + convention (`_index.md`) +- No `myst.yml` exists; Phase 1 creates it new at the repo root +- `docs/decisions/0001-myst-migration/` — ADR set committed; all seven ADRs + `Status: Proposed`, pending maintainer review + +### Hugo artifacts to remove (after MyST is working) + +- `config.yaml` — Hugo site config +- `themes/scientific-python-hugo-theme` — git submodule +- `netlify.toml` — Hugo/Dart Sass toolchain setup removed; file kept and updated +- `Makefile` — targets to be updated; see ADR 0001 and ADR 0004 + +### External content (cookie) + +`external-content/cookie` is a Jekyll site (git submodule). It is built +separately and its output merged into `public/`. See ADR 0004 for the +options considered and proposed decision. + +### Scientific Python ecosystem scope + +`learn.scientific-python.org` is one subdomain of the Scientific Python +ecosystem. Other repos that may need parallel or follow-on treatment: + +| Repo | Domain | Build tool today | +| ------------------------------------------------ | --------------------------- | ---------------- | +| `scientific-python/scientific-python.org` | scientific-python.org | Hugo | +| `scientific-python/blog.scientific-python.org` | blog.scientific-python.org | Hugo | +| `scientific-python/tools.scientific-python.org` | tools.scientific-python.org | Hugo | +| `scientific-python/scientific-python-hugo-theme` | (shared theme submodule) | N/A | + +All four repos share `scientific-python-hugo-theme`. A MyST migration of +`learn` decouples it from the theme; the other repos remain on Hugo until they +migrate independently. Cross-site nav links (`/`, blog, tools) are plain URLs +and do not break. No shared build pipeline couples the repos. + +### CI / deployment + +Current state: Netlify auto-deploys on push to `main` using the build command +in `netlify.toml`. Netlify also handles PR preview deploys. GitHub Actions runs +`lint.yml` (pre-commit checks) only. Local dev: `make serve` → +`localhost:3000`. + +Target: update `netlify.toml` to remove the Hugo and Dart Sass toolchain +setup and add `pip install mystmd`; the build command (`make html-all`), +publish directory, and `netlify-plugin-checklinks` are otherwise unchanged. +See ADR 0007. + +### Footer / quicklinks + +`config.yaml` defines footer social icons and quicklinks columns. MyST has no +built-in equivalent. See ADR 0005 for options considered and proposed decision. + +--- + +## Requirements + +### R1 — Reviewable git history + +Each logical change must be a separate, self-contained commit with a clear +message explaining _why_ not just _what_. The PR must be readable as a +narrative: "here is what we changed, here is why each step was necessary." + +See [PLAN.md](PLAN.md) for the commit-by-commit sequence. + +### R2 — Decision log + +The ADR set in `docs/decisions/0001-myst-migration/` documents each +significant decision. ADRs are currently `Status: Proposed` and become +`Accepted` (or modified / rejected) on maintainer review. Topics covered: + +- Why MyST over alternatives (Hugo, jupyter-book) — ADR 0001 +- How shortcode mapping was chosen — ADR 0002 +- What happens to the Hugo theme submodule (removal timeline) — ADR 0003 +- What happens to the cookie/Jekyll external content — ADR 0004 +- Footer/quicklinks approach — ADR 0005 +- Whether and when other SP repos migrate — ADR 0006 +- Deploy strategy: update Netlify for MyST, gh-pages as future option — ADR 0007 + +### R3 — No regressions in rendered content + +Before removing Hugo artifacts, a rendered-output comparison must be run: + +- Every page that exists in the Hugo build must exist in the MyST build +- Nav links, card grids, admonitions must render correctly +- External links must not break (verified by `netlify-plugin-checklinks` + during the Phase 6 Netlify preview deploy) + +### R4 — CI must be green on the final commit + +Each commit on the PR branch must leave CI in a defined state (pass or +known-failing with documented reason). Netlify deploy is intentionally broken +from Phase 2 (renames break Hugo) through Phase 5 (Makefile rewrite); this is +documented in each phase's CI status note. Lint must pass throughout. The +final commit (Phase 6 netlify.toml update) must have fully green CI. + +### R5 — Other SP repos unblocked, not broken + +The PR must not require simultaneous changes to other SP repos. Cross-site nav +uses plain URLs; no shared pipeline coupling. Document which follow-on issues +to open for the other repos. + +### R6 — External content decision documented + +The cookie/Jekyll submodule decision must be documented before the PR is +opened, even if the decision is "defer." + +### R7 — MyST config complete + +`myst.yml` must cover all metadata currently in `config.yaml`: + +- Site title, domain, nav +- Footer social links (deferred; follow-up issue number recorded in `myst.yml` comment block) +- Quicklinks (deferred; follow-up issue number recorded in `myst.yml` comment block) + +--- + +## Success criteria + +- `myst build --html` produces a complete site with no warnings about missing + directives or broken references +- All shortcodes converted; no Hugo syntax remains in `content/` +- `netlify.toml` updated: Hugo/Dart Sass toolchain removed, `pip install mystmd` added, + `make html-all` retained as build command; Netlify deploys successfully +- Hugo artifacts (`config.yaml`, theme submodule) removed or removal committed + with a clear timeline +- PR is self-contained: a reviewer with no prior context can follow the git + history and understand every decision diff --git a/docs/decisions/README.md b/docs/decisions/README.md new file mode 100644 index 0000000..9b608de --- /dev/null +++ b/docs/decisions/README.md @@ -0,0 +1,22 @@ +# Architecture Decision Records + +Significant decisions made during the development of +`learn.scientific-python.org`. Each ADR captures context, options considered, +and rationale so future contributors understand _why_ the codebase looks the +way it does. + +## Conventions + +- **Grouped changes** get a numbered subdirectory: `NNNN-topic/`. Files inside + restart numbering from `0001`. +- **Standalone decisions** live directly here as `NNNN-title.md`. +- ADRs are immutable once accepted. To reverse a decision, write a new ADR with + `Status: Supersedes NNNN`. + +Statuses: `Proposed → Accepted → Deprecated / Superseded by NNNN` + +## Index + +| # | Title | Status | +| ---------------------------- | ------------------------------- | -------- | +| [0001](0001-myst-migration/) | MyST-MD migration (7 decisions) | Proposed |