feat(challenge-widget): grading loop + spec cache + cost sizing (#2441) - #2444
Merged
Merged
Conversation
…izing (#2441) Clears the three activation gates from the #2362 challenge-widget POC writeup. The widget still ships dark (flag.challengeWidget OFF on PROD); this PR does NOT flip the flag or trigger a rebuild — those need editorial review + greenlight. Gate 1 — grading loop (mirrors the AI-quiz validate-answer trio): - db: ChallengeAnswers entity (server-only freeText reference answers, keyed tutorial/stepNumber/nodeId; anti-leak — reference never ships to clients). - POST /content/challenge-answers publish endpoint (REPLACE-per-slug, contentAuthMiddleware) + CLI uploader wired into publish-content.ts (non-fatal, channel=qa skipped). - POST /api/challenge-grade runtime endpoint (authenticated, rate-limited, gated on flag.challengeWidget ImsConfig, fail-open). Reuses the validate-answer prompt/schema/redaction and defaultCallModel verbatim. - client: ChallengeRenderer.vue now submits — MCQ client-graded vs public answerIndex (no server call), freeText AI-graded via /api/challenge-grade; slug+stepNumber threaded through main.ts (isPreview never POSTs). Gate 2 — build-time spec cache (mirrors ai-quiz-cache): - scripts/lib/challenge-spec-cache.ts + wiring in fetch-tutorials.ts: a hit reconstructs step.challenge + the reference sidecar with no model call (hashKey over stepBody+PROMPT_VERSION+modelName). Turns repeat flag-on rebuilds from cold-cost into cache-miss-only. Gate 3 — AI Core cost sizing: - docs/developers/reference/challenge-widget-cost-sizing.md — analytical estimate (cold ~11M tokens across ~1400 tutorials; warm rebuild ~50x less). Flagged as estimate pending a live per-call DEV measurement (cf was locked to PROD during a concurrent op). Tests: 65 new/extended unit tests across the 8 touched files, all green; cds compile clean; existing ai-challenge-spec test still passes.
…2441) check-srv-qa-route-drift flagged the new publish route. It is intentionally srv-only — the QA model (com.sap.developers.ims.qa) does not load ChallengeAnswers, srv-qa has no /api/challenge-grade reader, and the publish CLI skips the step for channel=qa. Same rationale as validate-answer-specs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2441. Clears the three activation gates from the #2362 challenge-widget POC writeup. The widget still ships dark — this PR does NOT flip
flag.challengeWidgetor trigger a rebuild; those need the editorial review + recorded greenlight from the issue.The whole subsystem mirrors the existing AI-quiz
validate-answerpath ~1:1.Gate 1 — freeText + MCQ grading
ChallengeAnswersentity — server-only freeText reference answers, keyed(tutorial, stepNumber, nodeId). Anti-leak: the reference never ships to clients (MCQanswerIndexstays public/client-graded).POST /content/challenge-answers(REPLACE-per-slug,contentAuthMiddleware) + a CLI uploader wired non-fatally intopublish-content.ts(skipped forchannel=qa, same as validate-answer validate-answer-specs publish returns HTTP 500 {error:internal} for 23 slugs on QA full rebuild #1375).POST /api/challenge-grade— authenticated, rate-limited (30/hr, 5/5min per step), gated on theflag.challengeWidgetImsConfig kill switch (fail-open). Reusesvalidate-answer-prompt.jsschema/redaction anddefaultCallModelverbatim.ChallengeRenderer.vuenow submits — MCQ client-graded against the publicanswerIndex(no server call); freeText AI-graded via/api/challenge-grade.slug+stepNumberthreaded throughmain.ts;isPreviewnever POSTs.Gate 2 — build-time spec cache
scripts/lib/challenge-spec-cache.ts(mirrorsai-quiz-cache.ts) + wiring infetch-tutorials.ts. A cache hit reconstructsstep.challenge+ the reference sidecar with no model call (hashKeyover stepBody + PROMPT_VERSION + modelName). Turns repeat flag-on rebuilds from cold-cost into cache-miss-only.Gate 3 — AI Core cost sizing
docs/developers/reference/challenge-widget-cost-sizing.md— analytical estimate (cold full pass ~11M tokens across ~1400 tutorials; warm rebuild ~50× less). Flagged as an estimate pending a live per-call DEV measurement — cf was locked to PROD during a concurrent operation, so the live pass was deferred; the doc includes the exact steps to run it on DEV.Verification
cds compile srvclean; existingai-challenge-spec.test.jsstill passes.npm test: 10,437 tests pass. The 20 failing test files are pre-existing fresh-worktree missing-optional-dep issues (sigma/mediapipe graph+selfie islands, srv-qa suites needing a bound model) — none are in this change.Not in scope (human/ops follow-ups)
flag.challengeWidgeton PROD.