Observed on 2026-09-14, generating the Qdrant Edge course spec. A subagent with no memory of the authoring session ran /generate-spec --env=coding-agent-lab from the base guide (8e44ecb) and the new overlay (b0d88d6) alone, forbidden from reading any prior output. It produced an 836-line spec with 7 Decision Ledger rows. The hand-written file it was compared against has 4. Three rows agree; five subjects diverge.
Three defects belonged to the overlay and are fixed in 58ecfb9. The four below belong to the base guide and are not applied — they need a promotion pass with a validated regen property, per fix-and-test-travel-together.
F1 — The surfacing bar does not force one answer (the headline finding)
Two runs, same guide, same materials, opposite Ledgers on three subjects:
| Subject |
Clean-room run |
Hand-written |
| Recognition threshold (0.80) |
design-argued row |
body default + Ask-First lever |
| Freshness re-ranking |
baked in as a business rule |
design-structural row |
| Photo-lane fetch depth |
contradicted row |
body default via the lever rule |
The clean-room run flagged both of its calls as borderline and cited the guide text pulling each way. On the threshold it quoted §12.9's counter-example ("sensitivity alone does NOT qualify") against L5's narrated two-sided trade-off, and chose design-argued because the course hands the choice to the learner. On freshness it noted the design-argued bake-in litmus "points both ways" — meaning-only ranking is both a working stepping stone (L3/L4) and explicitly demonstrated failing (L5).
Implication. §13's Regeneration Test is failing on its own terms: "the built project changes between no-intake runs → the §3 default is ambiguous". Here it is the Ledger's composition, not a default, that moves. Every downstream artifact — the §0 gate's question count, resolved-decisions.md, both evals — is keyed to rows.
No rule proposed yet — logged for a think. The honest options look like (a) tighten the design-argued two-sided litmus so "a baseline the course moves beyond" and "an anti-pattern the course warns against" separate mechanically; (b) add a tie-break that prefers the business rule when both readings survive; (c) accept N-run variance on borderline subjects and require the generator to record which litmus branch it took, so at least the choice is auditable.
F2 — No category for a fixture-inexpressible input
§3 defines live as excluded-from-the-offline-run, for criteria needing network. The clean-room run hit an input a spec cannot author at all: a spoken audio recording. Synthetic silence produces no transcript, so the voice lane's acceptance criterion cannot run on authored fixtures.
It extended the mechanism itself, under a distinct tag, and said so.
Implication. Every course with an audio, video, or sensor-capture modality hits this. The guide's fixture doctrine ("facts you author") has no answer, so each generation invents one.
Proposed rule (§3, beside the live bar) — not applied:
A criterion whose input cannot be authored as a fixture (recorded speech, camera frames, sensor traces) is tagged live-<kind> and excluded from the offline run, exactly as live is. The spec must additionally carry an offline criterion covering the wiring that input feeds — the path is testable even when the payload is not.
F3 — Row-grouping order is ambiguous
§5.5 says the table is grouped learner → design (design-argued, design-structural) → realization → contradicted "with pattern-pipeline order within each group". Unclear whether design is one group (argued and structural interleaved by pipeline position) or two sub-blocks.
The clean-room run chose two sub-blocks as "the more literal reading of the parenthetical" and noted either reading produces a different table.
Proposed fix (§5.5 row order) — not applied: state it explicitly, either way. One sentence closes it.
F4 — §5 and §14 contradict each other on version pinning
- §5 step 5: "Decide the stack: pin modern versions as this build's choice"
- §14: "Version claims match the notebooks' actual install lines; era stated honestly if unpinned"
This course pins every dependency exactly. "Modern" and "matches the notebooks" are the same only by accident, and diverge as soon as a course ages.
Proposed fix — not applied: §5 concedes precedence to §14 where the course itself pinned. Suggested wording: pin the course's own versions when it pinned them, recording the pins as provenance; choose modern versions only where the course left them unpinned; either way §14's honesty requirement governs the claim.
Evidence: clean-room output at /tmp/cleanroom-spec.coding-agent-lab.md (836 lines, 2026-09-14). Overlay defects from the same run fixed in 58ecfb9. Slides-glob finding routed out to its own issue — it is skill tooling, not guide territory.
🤖 Generated with Claude Code
Observed on 2026-09-14, generating the Qdrant Edge course spec. A subagent with no memory of the authoring session ran
/generate-spec --env=coding-agent-labfrom the base guide (8e44ecb) and the new overlay (b0d88d6) alone, forbidden from reading any prior output. It produced an 836-line spec with 7 Decision Ledger rows. The hand-written file it was compared against has 4. Three rows agree; five subjects diverge.Three defects belonged to the overlay and are fixed in
58ecfb9. The four below belong to the base guide and are not applied — they need a promotion pass with a validated regen property, per fix-and-test-travel-together.F1 — The surfacing bar does not force one answer (the headline finding)
Two runs, same guide, same materials, opposite Ledgers on three subjects:
design-arguedrowdesign-structuralrowcontradictedrowThe clean-room run flagged both of its calls as borderline and cited the guide text pulling each way. On the threshold it quoted §12.9's counter-example ("sensitivity alone does NOT qualify") against L5's narrated two-sided trade-off, and chose
design-arguedbecause the course hands the choice to the learner. On freshness it noted thedesign-arguedbake-in litmus "points both ways" — meaning-only ranking is both a working stepping stone (L3/L4) and explicitly demonstrated failing (L5).Implication. §13's Regeneration Test is failing on its own terms: "the built project changes between no-intake runs → the §3 default is ambiguous". Here it is the Ledger's composition, not a default, that moves. Every downstream artifact — the §0 gate's question count,
resolved-decisions.md, both evals — is keyed to rows.No rule proposed yet — logged for a think. The honest options look like (a) tighten the
design-arguedtwo-sided litmus so "a baseline the course moves beyond" and "an anti-pattern the course warns against" separate mechanically; (b) add a tie-break that prefers the business rule when both readings survive; (c) accept N-run variance on borderline subjects and require the generator to record which litmus branch it took, so at least the choice is auditable.F2 — No category for a fixture-inexpressible input
§3 defines
liveas excluded-from-the-offline-run, for criteria needing network. The clean-room run hit an input a spec cannot author at all: a spoken audio recording. Synthetic silence produces no transcript, so the voice lane's acceptance criterion cannot run on authored fixtures.It extended the mechanism itself, under a distinct tag, and said so.
Implication. Every course with an audio, video, or sensor-capture modality hits this. The guide's fixture doctrine ("facts you author") has no answer, so each generation invents one.
Proposed rule (§3, beside the
livebar) — not applied:F3 — Row-grouping order is ambiguous
§5.5 says the table is grouped
learner → design (design-argued, design-structural) → realization → contradicted"with pattern-pipeline order within each group". Unclear whether design is one group (argued and structural interleaved by pipeline position) or two sub-blocks.The clean-room run chose two sub-blocks as "the more literal reading of the parenthetical" and noted either reading produces a different table.
Proposed fix (§5.5 row order) — not applied: state it explicitly, either way. One sentence closes it.
F4 — §5 and §14 contradict each other on version pinning
This course pins every dependency exactly. "Modern" and "matches the notebooks" are the same only by accident, and diverge as soon as a course ages.
Proposed fix — not applied: §5 concedes precedence to §14 where the course itself pinned. Suggested wording: pin the course's own versions when it pinned them, recording the pins as provenance; choose modern versions only where the course left them unpinned; either way §14's honesty requirement governs the claim.
Evidence: clean-room output at
/tmp/cleanroom-spec.coding-agent-lab.md(836 lines, 2026-09-14). Overlay defects from the same run fixed in58ecfb9. Slides-glob finding routed out to its own issue — it is skill tooling, not guide territory.🤖 Generated with Claude Code