Challenge-widget: wire freeText grading, spec build-time cache, AI Core cost sizing (activation gates)

[jung-thomas]

Follow-up to the json-render challenge-widget POC (research spike #2362). The POC is merged to main and deployed, but ships dark: DB flag flag.challengeWidget is OFF on PROD and no flag-on content rebuild has baked widgets. Before it can be activated in PROD, three gates from the #2362 writeup must clear.

Blockers to activation

1. Wire freeText grading

The generator writes a <slug>.challenge-answers.json sidecar with freeText reference answers, but no publish-side consumer exists — freeText challenges render and collect input but cannot be graded.

• Add a ValidateAnswerSpecs-style upsert to persist reference answers.
• Add a runtime grade endpoint (mirror the AI-quiz grading path).
• MCQ is already client-graded (answerIndex); this is freeText only.

2. Spec the build-time cache

Unlike the AI-quiz path (loadAiQuizCache), every flag-on rebuild regenerates all specs — cost + per-build nondeterminism.

• Add a per-slug cache for challenge specs (mirror loadAiQuizCache).
• Required before any bulk/repeat PROD rebuild.

3. AI Core cost sizing

Size a full flag-on PROD generation pass: one model call per substantive step across ~1400 tutorials, capped by AI_AUTHOR_BUILD_CAP.

• Do feat: add slug mapping endpoint and getEventProgress fallback #2 (cache) first so the sizing reflects cached-rebuild cost, not cold cost.

Also needed before public (not code)

• Editorial review of AI-authored challenges — no human-in-loop review step exists.
• Recorded greenlight to promote past spike status.

Activation order (once gates clear)

1. Flip DB flag flag.challengeWidget = true on PROD.
2. Flag-on full PROD rebuild: gh workflow run rebuild-content.yml --ref main -f mode=full -f challenge-widget-enabled=true.
3. Re-smoke (expect transient done-button-render red against stale content until re-bake completes).

Ref: docs/developers/reference/generative-ui-research.md. Code seams: srv/lib/ai-challenge-spec.js, hugo-apps/src/challenge-render/, scripts/fetch-tutorials.ts.

[jung-thomas]

Gates 1+2 landed in #2444 (targets DEV). Gate 3 sizing below — analytical estimate.

A live per-call DEV measurement was deferred: cf was targeted at PROD during a concurrent operation and couldn't be repointed. The figures come from the challenge generator's measured prompt/schema sizes + the AI-quiz path's per-call profile (same model, same forced-tool-call shape). Full doc: docs/developers/reference/challenge-widget-cost-sizing.md.

Per-call: ~236 tok system + ~255 tok schema + 400–1000 tok step body ≈ ~1,200 in / ~400 out ≈ 1,600 tok/call (one call per substantive step).

Pass	Calls	Tokens
Cold full (no cache, ~1400 tut × ~5 substantive steps ≈ 7,000)	~7,000 (capped by AI_AUTHOR_BUILD_CAP, default 200/build)	~11.2M
Warm rebuild (gate-2 cache, ~2% step churn)	~140	~224K (~50× less)

Cost = tokens × the AICore Sonnet deployment rate (fill in $/1K).

Before any bulk PROD rebuild: run the live per-call pass on DEV (steps in the doc) to replace the ~1,600 tok working figure and the ~5-substantive-steps/tutorial assumption with real corpus numbers.

Still open (human/ops, per this issue): editorial review of AI-authored challenges + recorded greenlight; then flip flag.challengeWidget + flag-on full rebuild.

[jung-thomas]

Status

Code gates 1 + 2 implemented in #2444 (targets DEV, awaiting review). Gate 3 delivered as an analytical sizing estimate (comment above + docs/developers/reference/challenge-widget-cost-sizing.md); the live DEV measurement is deferred.

Gate	State
1 — freeText grading (+ MCQ wired)	✅ implemented in #2444
2 — build-time spec cache	✅ implemented in #2444
3 — AI Core cost sizing	🟡 analytical estimate only; live DEV pass deferred

Verification: 65 new/extended unit tests green, cds compile clean, full npm test 10,437 pass (20 failing test files are pre-existing fresh-worktree missing-dep issues, none in this change).

The widget still ships dark — this PR does not flip flag.challengeWidget or trigger a rebuild.

Ops / human follow-ups before PROD activation

1. Review + merge feat(challenge-widget): grading loop + spec cache + cost sizing (#2441) #2444 to DEV (normal PR review — not fast-merged).
2. Live per-call cost measurement on DEV — deferred because cf was locked to PROD during a concurrent op. Steps in the sizing doc: cf target -s dev, pause the sap-devs scheduler, cds bind AI Core, cold pass on ~5–10 slugs with a low AI_AUTHOR_BUILD_CAP, then warm pass. Replaces the ~1,600 tok/call working figure and the ~5-substantive-steps/tutorial assumption with real numbers.
3. Editorial review of AI-authored challenges — no human-in-loop review step exists yet.
4. Recorded greenlight to promote past spike status.

Activation order (once 1–4 clear)

1. Flip DB flag flag.challengeWidget = true on PROD.
2. Flag-on full PROD rebuild: gh workflow run rebuild-content.yml --ref main -f mode=full -f challenge-widget-enabled=true (raise AI_AUTHOR_BUILD_CAP for the cold pass, or run capped repeats — gate-2 cache makes reruns cheap).
3. Re-smoke (expect transient done-button-render red against stale content until the re-bake completes).

[jung-thomas]

Post-deploy runbook & recurring maintenance

PR #2444 merged to DEV (merge commit 8c17e8d). The worktree is cleaned up; nothing to deploy from this work session (deploy is a separate, explicitly-authorized step). Recording the operational steps here since these are one-time / on-change, not a standing schedule.

One-time activation (after gates 3-live, editorial review, greenlight)

1. Live cost re-measurement on DEV (replaces the analytical estimate): cf target -s dev, pause the sap-devs scheduler, cds bind AI Core, cold pass on ~5–10 slugs with a low AI_AUTHOR_BUILD_CAP, then a warm pass to confirm ~0 model calls + the [challenge] … spec cache hit(s) log line. Steps in docs/developers/reference/challenge-widget-cost-sizing.md.
2. Flip flag.challengeWidget = true (PROD) via the admin feature-flags UI.
3. Flag-on full rebuild: gh workflow run rebuild-content.yml --ref main -f mode=full -f challenge-widget-enabled=true. Raise AI_AUTHOR_BUILD_CAP for the cold pass, or run capped repeats — the gate-2 cache makes reruns cheap (warm ≈ 0 calls).
4. Re-smoke: expect a transient done-button-render red against stale content until the re-bake completes.

Recurring / on-change maintenance

• On PROMPT_VERSION bump or model swap (srv/lib/ai-challenge-spec.js): the spec cache stepHash folds both in, so every <slug>.challenge-spec-cache.json entry invalidates automatically → the next flag-on rebuild regenerates (full cold cost again). Budget for a cold pass whenever either changes.
• ChallengeAnswers is REPLACE-per-slug: each publish of a slug clears+reinserts its rows. No orphan buildup, no manual purge. A slug that stops emitting freeText nodes drops its rows on next publish.
• No standing cleanup job needed: the spec cache lives in .tutorial-cache/ (rebuilt/ephemeral), and ChallengeAnswers is self-cleaning via REPLACE. Nothing accrues that requires periodic pruning.

Kill switch

Set flag.challengeWidget = false (admin UI) — runtime /api/challenge-grade returns 503 disabled and baked widgets stop grading. To also stop baking new widgets, drop -f challenge-widget-enabled=true from the rebuild dispatch (build-time env gate).