Observed 2026-09-14. Two clean-room subagents generated the Qdrant Edge spec through /generate-spec --env=coding-agent-lab, one against overlay b0d88d6 and one against 58ecfb9. Three defects they found are already fixed in 58ecfb9. These six are not applied — they need their own revision plus a regen that shows the fix landed.
Overlay files: .claude/skills/generate-spec/references/spec-generation-guide.coding-agent-lab.md and its .runtime.md companion.
G1 — The overlay never states the deliverable's shape (highest priority)
The second run reported this as its single biggest judgment call. Nothing in the base guide or the overlay says what artifact a lab build produces. The course's own artifact is a Jupyter notebook — which the lab has no surface for. The run inferred a served web app from runtime facts that only describe how the preview works if you serve something:
One server, port 4000, bound to 0.0.0.0, serving both the page and the API.
Every URL inside the page must be relative … belongs in the spec as a numbered business rule with an acceptance criterion, not as a note.
That language presupposes a page. environments/coding-agent-lab.md §1 and §10 do make a served app the environment's default shape — so the inference was right — but the overlay never says so, and a future run could infer a CLI instead with nothing to stop it. Currently harmless by luck.
Proposed fix — not applied: state the deliverable shape in the rules file, as a default with an escape: the lab's default artifact is a single server on port 4000 serving both a page and its API; a course whose pattern has no user-facing surface may deliver a library plus its test suite, and must say so in §1.
G2 — ADD 2 forbids a co-label that some constraints genuinely need
ADD 2: "It must never also carry a course-demonstrated label."
But "release the speech model before loading the embedders" is independently (a) demonstrated by the course's own helper, with a stated memory-budget reason, and (b) required by the runtime facts ("A spec whose pipeline can hold several models resident at once must carry a business rule releasing one before loading the next"). The run wrote it as "course-demonstrated, reinforced [environment]" — arguably the banned co-label — judging that splitting it into two near-duplicate rules was worse.
Proposed fix — not applied: ADD 2 gains a third form for a constraint both sources independently require, e.g. course-demonstrated, reinforced [environment]. The ban's real target is claiming course provenance for an environment-only fact; it should not forbid recording that both are true.
G3 — ADD 3's AC obligation has no branch for an empty Invariant
ADD 3: "Every ANSWERED or CONSTRAINED dimension's Invariant is exercised by ≥1 acceptance criterion at its new home, or the table states why it cannot be."
The project dimension's Invariant is empty by base §3's own rule ("typically empty for project/data/goal/scope-boundary"). There is nothing to exercise — which is not the same as "cannot be". The run wrote the explanation into the table cell.
Proposed fix — not applied: add the third branch — a dimension whose derived Invariant is empty owes no AC; the table says "Invariant empty" and that discharges it.
G4 — The base title template reads wrongly in lab mode
Base §6 pins # Spec: <Project> — **Standalone Takeaway**. The overlay does not override it, so base wins under precedence rule 2 — but in this mode nobody is taking anything away; the lab fixes the assignment and the learner builds it there.
Proposed fix — not applied: either an OVERRIDE 3 on the title line, or an explicit note that the base title stands and why. Right now it is left to each run to notice and live with.
G5 — The overlay does not carry its own commit, but ADD 3 requires pinning it
ADD 3: "The provenance header pins both the base guide's commit and this overlay's." The overlay states only "Written against base guide 8e44ecb" — it has no self-hash. The run derived it from git log and got the right answer, but that is an inference about repo state, not something the document supplies.
Proposed fix — not applied: say how to obtain it — pin the overlay's commit as reported by git log -1 --format=%h -- <this file> — rather than requiring a fact the file does not contain.
G6 — Unclear whether environments/coding-agent-lab.md may be opened
The rules file says "Both are distilled from environments/coding-agent-lab.md", and OUT OF SCOPE forbids reading certain of its sections as instructions — which implies the file may be opened. The second run did not open it, judging the two overlay files self-contained. If the distillation dropped anything load-bearing, that run silently missed it.
Proposed fix — not applied: state it outright. Recommended: the source doc is provenance, not an input — do not read it during generation; if the overlay is missing a fact, that is an overlay bug to file, not a gap to fill by reading around it.
Related: #28 (base-guide findings from the same runs), #29 (the slides glob). Overlay defects already fixed: 58ecfb9.
🤖 Generated with Claude Code
Observed 2026-09-14. Two clean-room subagents generated the Qdrant Edge spec through
/generate-spec --env=coding-agent-lab, one against overlayb0d88d6and one against58ecfb9. Three defects they found are already fixed in58ecfb9. These six are not applied — they need their own revision plus a regen that shows the fix landed.Overlay files:
.claude/skills/generate-spec/references/spec-generation-guide.coding-agent-lab.mdand its.runtime.mdcompanion.G1 — The overlay never states the deliverable's shape (highest priority)
The second run reported this as its single biggest judgment call. Nothing in the base guide or the overlay says what artifact a lab build produces. The course's own artifact is a Jupyter notebook — which the lab has no surface for. The run inferred a served web app from runtime facts that only describe how the preview works if you serve something:
That language presupposes a page.
environments/coding-agent-lab.md§1 and §10 do make a served app the environment's default shape — so the inference was right — but the overlay never says so, and a future run could infer a CLI instead with nothing to stop it. Currently harmless by luck.Proposed fix — not applied: state the deliverable shape in the rules file, as a default with an escape: the lab's default artifact is a single server on port 4000 serving both a page and its API; a course whose pattern has no user-facing surface may deliver a library plus its test suite, and must say so in §1.
G2 — ADD 2 forbids a co-label that some constraints genuinely need
ADD 2: "It must never also carry a course-demonstrated label."
But "release the speech model before loading the embedders" is independently (a) demonstrated by the course's own helper, with a stated memory-budget reason, and (b) required by the runtime facts ("A spec whose pipeline can hold several models resident at once must carry a business rule releasing one before loading the next"). The run wrote it as "course-demonstrated, reinforced
[environment]" — arguably the banned co-label — judging that splitting it into two near-duplicate rules was worse.Proposed fix — not applied: ADD 2 gains a third form for a constraint both sources independently require, e.g. course-demonstrated, reinforced
[environment]. The ban's real target is claiming course provenance for an environment-only fact; it should not forbid recording that both are true.G3 — ADD 3's AC obligation has no branch for an empty Invariant
ADD 3: "Every ANSWERED or CONSTRAINED dimension's Invariant is exercised by ≥1 acceptance criterion at its new home, or the table states why it cannot be."
The
projectdimension's Invariant is empty by base §3's own rule ("typically empty for project/data/goal/scope-boundary"). There is nothing to exercise — which is not the same as "cannot be". The run wrote the explanation into the table cell.Proposed fix — not applied: add the third branch — a dimension whose derived Invariant is empty owes no AC; the table says "Invariant empty" and that discharges it.
G4 — The base title template reads wrongly in lab mode
Base §6 pins
# Spec: <Project> — **Standalone Takeaway**. The overlay does not override it, so base wins under precedence rule 2 — but in this mode nobody is taking anything away; the lab fixes the assignment and the learner builds it there.Proposed fix — not applied: either an OVERRIDE 3 on the title line, or an explicit note that the base title stands and why. Right now it is left to each run to notice and live with.
G5 — The overlay does not carry its own commit, but ADD 3 requires pinning it
ADD 3: "The provenance header pins both the base guide's commit and this overlay's." The overlay states only "Written against base guide
8e44ecb" — it has no self-hash. The run derived it fromgit logand got the right answer, but that is an inference about repo state, not something the document supplies.Proposed fix — not applied: say how to obtain it — pin the overlay's commit as reported by
git log -1 --format=%h -- <this file>— rather than requiring a fact the file does not contain.G6 — Unclear whether
environments/coding-agent-lab.mdmay be openedThe rules file says "Both are distilled from
environments/coding-agent-lab.md", and OUT OF SCOPE forbids reading certain of its sections as instructions — which implies the file may be opened. The second run did not open it, judging the two overlay files self-contained. If the distillation dropped anything load-bearing, that run silently missed it.Proposed fix — not applied: state it outright. Recommended: the source doc is provenance, not an input — do not read it during generation; if the overlay is missing a fact, that is an overlay bug to file, not a gap to fill by reading around it.
Related: #28 (base-guide findings from the same runs), #29 (the slides glob). Overlay defects already fixed:
58ecfb9.🤖 Generated with Claude Code