Skip to content

feat(directives): generic tutorial directive grammar foundation (openspec: tutorial-directive-grammar) - #3021

Merged
jung-thomas merged 16 commits into
DEVfrom
worktree-directive-grammar
Nov 5, 2026
Merged

jung-thomas merged 16 commits into
DEVfrom
worktree-directive-grammar

Conversation

@jung-thomas

Copy link
Copy Markdown
Contributor

What & why

Implements the tutorial-directive-grammar OpenSpec change (proposed in #3018) — the foundation the other 7 markdown-syntax features build on. Adds one generic CommonMark-style directive grammar to tutorial markdown:

  • inline :name[label]{attrs}, leaf ::name[label]{attrs}, container :::name[label]{attrs} … ::: (nest with more colons)
  • a directive registry (name → kinds, attr schema, renderer, plain-text fallback) with build-time validation
  • a fence-aware pre-pass in compose.ts (after wrapTokens, before convertOptionBlocks) rewriting directives into tut-<name> Hugo shortcodes
  • a shared renderDirectivesAsText() fallback adopted by the MCP step slicer, freshness detector, and AI-authoring read paths
  • a directive-unknown lint warning, and npm run build:directive-schema → schemas/tutorial-directives.schema.json for authoring tools

Ships ZERO concrete directives — the 7 follow-up features register directives on top. Core guarantee: byte-identical output for tutorials that don't use directives, proven by a corpus-regression test.

Security: escaping is safe-by-construction

DirectiveDef.render returns a structured { params, body?, markdown? }; the framework (buildShortcode) HTML-escapes every param + label (incl. backslash) — a directive author cannot emit an unescaped value. This invariant is inherited by all fan-out features.

Type of change

  • New feature (additive, opt-in syntax)

How tested

  • 118 directive-specific unit tests pass (parser, tokenizer, renderer, fallback, compose integration, error surfacing, conformance, corpus regression, lint, schema).
  • openspec validate tutorial-directive-grammar --strict → valid.
  • Pipeline-interaction fixtures prove: variables inside directive labels survive (runs after wrapTokens), directives inside OPTION panels render in the right tab (runs before convertOptionBlocks), directive-looking text inside code fences is untouched, and empty-registry output is byte-identical (toBe).
  • Not run: full npm test in a fresh unbuilt worktree has pre-existing env failures unrelated to this change (missing compiled sap-fundamental.css, missing srv/lib/_shared/{content,core}.bundle.mjs, a parseMarkdownSteps {time=} failure owned by a pre-existing feat(tutorials): per-step time/optional + interactive checklists (openspec: tutorial-step-pacing) #3019 commit). All directive tests are green.

Review notes

  • Went through a full subagent-driven review loop incl. a whole-branch review; two latent defects found + fixed before this PR: (1) a bare unclosed :::foo no longer throws under an empty registry (byte-identity safety); (2) attribute tokenizer now supports key='value' and backslash-escaped quotes, and escapeShortcodeParam escapes backslash.
  • Deferred minors (non-load-bearing, documented in the plan): duplicated test-suite block in directives-render.test.ts, the shared container-close/inline-column fallbacks in the two back-to-front replacers.
  • Known behavioral note for fan-out: param backslashes are entity-encoded (&#92;) — directive families with backslash-bearing attrs should decode in their template if needed.

Deploy scope

  • Backend only (srv/**, scripts/**) + docs — no content rebuild, no admin UI, no Hugo asset change. Parser/renderer only; empty registry = no runtime behavior change.

Checklist

  • Branched from DEV
  • No secrets committed
  • Did not edit generated hugo/content/tutorials/
  • Docs updated (docs/authors/writing-tutorials.md §Directives + gotcha)

…g), C2 injection tests, I1 real recursion, M1 cleanup
Insert expandDirectives(body, {slug, variables}) between the wrapTokens
block and convertOptionBlocks in composeTutorial.  The call is the only
production-code change; with an empty registry it is an identity
transform, so tutorials without directives are unaffected.

Five integration tests in directives-compose.test.ts cover: plain-step
render, variable-in-label (proves AFTER wrapTokens), directive-inside-
OPTION-panel (proves BEFORE convertOptionBlocks), code-fence passthrough,
and empty-registry identity.

Regression: variables.test.ts (34 tests) and v2.test.ts (9 tests) green.
Adopt renderDirectivesAsText (Task 3) at three read sites so models and
MCP clients see readable prose instead of raw :::details / :kbd[ syntax.

Sites adopted:
- packages/mcp/tutorial-step-slicer.js: apply in loadAndParseMarkdown
  before the step entry is built; import follows the existing srv/lib
  pattern (../../scripts/parsers/directives.js → .ts resolved at runtime).
- srv/lib/freshness-detector.js: strip directives from src.markdown before
  passing to extractCodeBlocks / extractTutorialContext; same import pattern.
- scripts/lib/expand-ai-authored.ts: strip directives from stepBody when
  read from stepBodies (step-source path only; video path overwrites below).

All three sites: empty registry → identity transform (no registered
directives yet → safe no-op on current content).

Tests: extended directives-text.test.ts with MCP read-path contract
(:::details[Why] + :kbd[Mod+S] → **Why** + Mod+S, no raw syntax) and
identity check. Slicer and expand-ai-authored unit suites all pass (9/9
and 32/32 respectively).
@jung-thomas
jung-thomas merged commit 225c8c3 into DEV Nov 5, 2026
7 of 8 checks passed
@jung-thomas
jung-thomas deleted the worktree-directive-grammar branch November 5, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant