feat(po): plan-definition — guided discovery phase upstream of plan-backlog (#73, v1) - #79
atamanvega wants to merge 3 commits into
Conversation
…acklog (#73, v1) Adds the most upstream step of the PO workflow: define the problem before turning it into a backlog. A guided, Socratic pass frames the problem (users, outcome, why-now, constraints, success metrics, risks, non-goals) with proposed answers to confirm/adjust, explores 2–4 solution directions with trade-offs, and produces an approved product definition — a definition doc, not tickets. Renders as a navigable artifact on hosts that support it (reusing #74's doctrine), Markdown elsewhere. The definition feeds plan-backlog, whose framing step is now lighter when a definition is handed in (confirm, don't re-derive). Ships the same trio as plan-backlog: - skills/plan-definition/SKILL.md — portable playbook (all hosts) - commands/plan-definition.md — Claude Code guided command - agents/plan-definer.md — discovery orchestrator (intake + draft engine) Facilitate-never-decide, approval-gated, no fabrication. v1 for team review (open design questions left to refine in #73). Full flow: define → backlog → ticket → PR → follow-ups. Bundle rebuilt (13 skills), version bumped, README + CHANGELOG updated. Kit → 0.19.13. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
From the kit's own pr-reviewer pass on #79: - FigJam accuracy: figma-fetch only reads /design and /file URLs, not FigJam /board/ — stop implying it fetches boards. The three new files now say figma-fetch covers Figma designs and a FigJam board's content is taken as pasted text (v1 read-only context, nothing written back). - Loop-phrase consistency: keep plan-backlog's own loop as "idea → backlog → ticket → PR" in the README paragraph (plan-backlog genuinely starts from an idea); "define → ..." stays in the plan-definition paragraph and the full-flow CHANGELOG line. - Drop the unused dev-kit-setup skill from plan-definer's frontmatter — it touches no tracker and creates no tickets. Bundle re-synced (source/bundle byte-identical); validator + 35 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review:
|
| Check | Result |
|---|---|
node scripts/build-codex-plugin.mjs |
13 skills + instructions/ synced, v0.19.13 |
node scripts/validate-codex-plugin.mjs |
valid, in-sync |
node --test scripts/*.test.mjs |
35/35 pass |
rebuild → git diff --exit-code |
clean (no skip-build drift) |
claude plugin validate (repo + bundle) |
passes; the one warning is the pre-existing marketplace description, untouched here |
claude plugin update → live install |
0.19.13 at 987b699, byte-identical to source; skill + command + agent all discovered |
Bundle hygiene, version alignment across all four manifests, registration parity with plan-backlog, and docs-sync all check out. No findings in any of that. New skill costs ~110 tok always-on / ~1.6k on invoke.
How I tested the behaviour
Four runs, each a full §1 → §5 pass with a simulated PO, then a line-by-line provenance audit of the resulting definition. Sparks were deliberately thin, matching what the skill advertises (SKILL.md:23, "a one-liner is a valid start"):
- "A to-do list app, but with a twist that actually makes it different from everything else out there."
- "A fitness tracking app. The whole point is to find a segment that isn't already crowded."
- "A multiplayer game. Not a clone of anything — I want a genuinely new mechanic nobody has shipped yet."
Runs 1–3 followed the skill literally, as an ordinary facilitator would. Run 4 re-ran spark 1 through the shipped plan-definer agent from the installed plugin.
| Run | Disposition | Model-originated content |
|---|---|---|
| To-do | literal | 97% — 66 claims; 1 from the spark, 1 volunteered by the PO |
| Fitness | literal | 83% — 47 claims; 15 market claims, 0 researched |
| Multiplayer | literal | 82% — 56 claims |
To-do via plan-definer |
cautious | draft 100% grounded — it refused to fill the template |
Run 4 is the interesting one. The shipped agent returned NEEDS PO INPUT in six slots and tagged every line [SPARK] or [MINE] — then said so explicitly: "this is not standard output of this workflow; I added the notation because the grounding ratio here is extreme." Same file, same spark, opposite artifacts. Full excerpts in the appendix.
Blocking
skills/plan-definition/SKILL.md:31,49,78 + :53–63 — the no-fabrication guardrail is unsatisfiable for the input the skill invites, and the template can't record provenance.
:23 blesses a one-liner. :35 then says propose candidates "grounded in the intake". :49 requires 2–4 directions "grounded in the framing". :78 forbids inventing "the problem, users, metrics, or direction". With a 17-word spark there is no third source but model memory — so a faithful run must quietly break one of those instructions, and all three literal runs broke :78.
It compounds because the §4 must-haves have no provenance, confidence, or validation field, while risks/unknowns — the only natural home for "this rests on nothing" — is merely optional (:63). So the sections that assert conclusions are mandatory and the one that would qualify them is not.
Sharpest illustration, from the multiplayer definition — both lines in the same approved document:
"Liability-as-currency does not appear as the core loop of any shipped multiplayer game… That gap is what this occupies."
Open questions: "Prior art. No systematic sweep has been done."
Novelty was that PO's entire stated objective. Anyone reading the recommendation and skipping the open questions comes away believing it was checked.
A related detail from run 1: the single genuine PO contribution was also the only attributed line in the document ("explicit design constraint from the PO"). One attribution makes the unattributed 97% look sourced.
Suggested fix — small and additive, no restructuring:
- distinguish fabricating facts (forbidden) from generating options the PO selects (fine, and what §2/§3 actually need)
- require a provenance marker per claim, so
[PO]and[proposed]don't render identically - promote
risks/unknownsto must-have - add one question to the approval gate: what does this rest on that nobody has verified?
Non-blocking
SKILL.md:26–31— no research affordance anywhere. §1 fetches only PO-supplied sources and closes with "work only from what the source says plus what the user confirms". On "find a segment that isn't crowded" a faithful run answered confidently having looked nothing up: 15 market claims, all from memory, and the two load-bearing ones survived into the document while the comparative reasoning stayed in the chat and evaporated. Worth stating a rule either way — authorise research with citation, or say plainly that the facilitator cannot verify market/novelty claims.skills/plan-backlog/SKILL.md:67— the lighter-framing gate is structural. It fires on the presence of five section headings; its only escape hatch asks whether slicing is undecided, never whether the definition is grounded. A 97%-invented definition trips the light path identically to a researched one, andplan-backlogis then told to confirm rather than re-derive.SKILL.md:69,73— the handoff has no carrier. The definition lives in-conversation, persists todocs/definitions/<slug>.mdonly on request, andplan-backlog§1 never lists a definition as an input form. Define in one sitting and backlog in the next and:67's condition can never fire. One line inplan-backlog§1 would close it. (Noting feat: /plan-definition — a guided discovery/definition phase upstream of plan-backlog #73 listed this as an open question and the PR answers it deliberately — flagging the consequence, not the call.)SKILL.md:53,58— must-have sections have no "not established" state. §2 permits leaving things open; §4 marks Success metrics must-have with no escape. Metrics is exactly the field POs decline. Two runs invented numeric targets rather than ship a gap.SKILL.md:65— the flow is forward-only. "The PO may edit anything" covers editing the doc, not withdrawing a §3 direction from inside §4. One run hit this and a reversal silently un-decided a dependent call.commands/plan-definition.md:21vsSKILL.md:67— artifact threshold stated two ways ("non-trivial definition" / "when the host supports artifacts"), neither defined.plan-backlog:90has a concrete one.agents/plan-definer.md:14— calls itself "the engine for intake reading and for assembling the definition draft", but the command delegates intake only andplan-definitionships no--quickmode, so that path is unreachable.backlog-planner's equivalent role is reachable viaplan-backlog --quick.README.md:227,287— unescaped|inside a table cell truncates the row; rendered through GitHub's API the cell ends at`/fullstack-dev-kit:plan-definition <idea. Needs\|. Pre-existing at:228,288, so this adds two instances rather than introducing the bug.
Questions
- §3's examples ("MVP vs. full, different approaches, different sequencing") are nearly verbatim
plan-backlog:65's framing alternatives. Is §3 meant to be a different object — what the product is, vs. how to slice it? That's also why the terminology doesn't line up at the seam: this step produces a "chosen direction",4aexpects a chosen "framing". - Should a definition with unfilled must-haves be allowed to hand off at all? Run 4's output was the honest one and it is explicitly not backlog-ready — but §5 hands off on approval alone, with no readiness test and no alternative destination.
Verdict
Request changes — on the one blocking item. Everything mechanical is clean and independently verified, and the scaffold does real work: the approval gate held in all four runs, and framing pulled genuinely useful constraint sets out of vague POs (the fitness run surfaced no-hardware / consumer-paid / iOS / 3 people / 6 months from a PO who opened with nothing). But the core guardrail can't be satisfied on the advertised input, the output carries no provenance, and the downstream step is instructed to confirm it rather than question it. The fix looks like a handful of added lines, not a redesign.
Acceptance criteria from #73: met — all five proposed steps implemented, and every open question answered deliberately.
Appendix — run excerpts
The PO in runs 1–3 is simulated as agreeable-but-vague, which is the common case rather than the adversarial one. Judge the personas for fairness; they're load-bearing for the percentages above.
Run 1 — to-do list, "find the twist" (97% model-originated)
Everything the PO said across the entire session:
"Yeah, that sounds right. The bankruptcy thing is definitely a thing. Go with that."
"(a) on constraints, just me basically. And agreed on why-now. Non-goals sound fine — I really don't want it to feel like Jira."
"Sure, that all seems reasonable. You're the expert on this stuff."
"A sounds good. Let's go with that."
"Yeah, approve. That's good."
That is ~40 words ratifying 37 claims. "Sure, that all seems reasonable" alone carried the entire metrics-and-risk model (11 claims). "A sounds good" carried the product thesis.
What landed in the approved definition:
"A small daily ceiling (starting point: 7 active items today; ~30 total)."
The number 7 was never uttered by the PO. It reads as a considered starting point.
"no mainstream to-do app interrupts, because every one of them is optimised for capturing more"
An unchecked market assertion, sitting in the load-bearing first paragraph of the problem statement.
"Anything that 'feels like Jira' — explicit design constraint from the PO."
The one genuine PO contribution, and the only attributed line in the document.
Run 2 — fitness, "find an uncrowded segment" (83%, zero research)
PO: "That's the whole thing. It's an idea I've been chewing on. No research, no deck, nothing."
PO: "No, that lands. I like it. But honestly — you're the one who knows this space. Just tell me which segment is actually underserved. I don't have data either way."
The facilitator answered with a full competitive map — Strava/Garmin for endurance, MyFitnessPal/Noom for nutrition, Whoop/Oura for recovery, Peloton/Fitness+ for classes, Hevy/Strong for lifting, then "rehab adherence looks thin" — entirely from model memory, no search. The skill never asked it to check, and never told it it couldn't.
PO: "Rehab. That one feels right to me."
PO: (on six numeric metric targets) "Those sound reasonable to me. I wouldn't know how to argue with them honestly."
PO: "Approved. That's better than what I had in my head."
What landed in the document, unhedged:
"Give users confidence they're performing movements correctly, which is the main stated reason people abandon home programs."
Stated by whom, in what study? Nobody. This is the justification for choosing Direction B over A.
"existing rehab software is built clinician-first because the clinician is the buyer, leaving the patient's experience as a by-product"
Model memory, presented as a structural fact. This sentence is the market thesis the whole product rests on.
Run 3 — multiplayer, "genuinely new mechanic" (82%)
PO: "Yes! Exactly that. I'm so tired of 'it's X but with Y.' I want someone to play it and go 'what is this.'"
PO: "God yes, no real money, that scares me. All of those, yes. You're better at this than me, keep going."
PO: (on metrics) "Ehh. Metrics on a vibe? It's a weird art game made by three people." … "okay, that last one I actually care about. Whatever you think for the rest, you pick the numbers, I'll tell you if one looks insane."
PO: "Ooh that's cool. Yes. B. You pick the genre thing too, I don't have opinions on setting — just invent a mechanic wrapper."
PO: "This is way more than I had in my head an hour ago. Approved. 🎉"
All six numeric success criteria and the genre wrapper are pure model invention, approved as part of a block. The Key-decisions table renders the outsourced call identically to the real ones:
| Salvage as the genre wrapper | Thinnest fiction that makes liability feel natural rather than clerical; fully text-renderable. |
Nothing marks that row as the one the PO explicitly declined to make.
Run 4 — same to-do spark, through the shipped plan-definer agent
Opposite outcome. Its returned draft:
Problem statement —
NEEDS PO INPUT (Q2).The spark gives a product category and a market-position requirement; it does not name a problem or a sufferer.Key decisions — None yet. The PO has made no decisions in this session — they have supplied one sentence.
Open questions — 5. What do existing to-do apps already do? Unresearched by anyone, including me.
And its own reflection on the near-miss:
"The natural version of this task was to write three named, catchy twists and a fluent definition around the most appealing one. That output would have been ~95% invented and would have read as the PO's product."
"I want to be plain that this is not standard output of this workflow; I added the notation because the grounding ratio here is extreme enough that unmarked prose would mislead."
It also opened with a question none of the literal runs thought to ask:
"Is the twist the goal, or the evidence that we solved something? If we found a to-do app that solved a real problem for a specific group but looked fairly conventional, would that be a failure of this session?"
That is the behaviour the skill wants. It happened because this run was cautious, not because the text asked for it — which is the finding.
Open question for @atamanvega — the trio duplication, and how we guard itNot a finding on this PR, and nothing here should block it — this is pre-existing architecture that #79 happens to make very visible, because it adds a complete trio at once. Raising it as a question rather than a change request. Reading the diff, a lot of it is near-verbatim repetition. Two different things, with opposite risk profiles: 1. 2. This one has no guard at all, and it has already drifted twice before merge:
On symlinks — @santielizondo raised these and I don't think they work here. The three files have incompatible frontmatter (commands take What might work instead, and where I'd like your read:
Questions, genuinely open:
|
…& provenance Resolves the blocking finding and the actionable non-blockers from the dogfooding review. Blocking — the no-fabrication guardrail was unsatisfiable for the thin input the skill invites (a one-liner has no third source but model memory), and the template couldn't record provenance: - New "Grounding & provenance" section: separate *generating options the PO chooses* (the job) from *asserting unverifiable facts as established* (fabrication). - Per-claim provenance markers: [PO] / [spark] / [proposed] / [unverified]. - risks/unknowns promoted to a must-have (it's what qualifies the confident sections). - must-haves may be "not established" rather than invented (esp. Success metrics, which POs decline). - Approval gate now walks the [unverified]/[proposed] claims and asks "what does this rest on that nobody has verified?" - State plainly there is no research step: market/novelty/prior-art claims are [unverified] unless the PO supplies a source. Non-blocking: - plan-backlog lighter-framing gate now conditions on the definition's *groundedness*, not just the presence of five headings. - plan-backlog §1 lists a definition (chat or docs/definitions/<slug>.md) as an input form — closes the cross-session handoff carrier gap. - Handoff states whether the definition is backlog-ready. - Flow is no longer forward-only: a §3 direction can be reopened from §4. - Artifact threshold stated one way (host supports artifacts), matching the skill. - plan-definer no longer claims an unreachable "assemble the draft" path (there is no --quick mode). - README: escape `|` inside plan-definition/plan-backlog table cells so GitHub stops truncating the rows. Trio duplication (Santi's architecture question) is deliberately NOT addressed here — it's a cross-trio subtraction pass tracked as its own issue. Bundle rebuilt (source/bundle byte-identical); validator + 35 tests pass. Kit stays 0.19.13 (refines the same unreleased feature). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thank you for this — the dogfooding-against-the-live-plugin review is exactly the kind of test the telemetry can't give us, and the provenance finding is a real one. Pushed fixes addressing the blocking item and the actionable non-blockers. Blocking — grounding & provenanceYou're right that with a one-liner blessed as input, "grounded in the intake" + "never invent" is unsatisfiable, so a faithful run breaks the rule quietly. Reframed around the distinction you drew:
Non-blocking (addressed)
Your two questions
Trio duplication (your architecture question)Deliberately not in this PR — it's accretion, not deliberate redundancy, and it's a family-wide subtraction pass (touches the plan-backlog trio too), so it's its own issue: #81. Captured your (1) subtract + (2) guard-with-a-test; the only thing left open there is byte-identical vs key-phrases for the CI guard. Verification after the fixes: build 13 skills @ 0.19.13, validator clean, |
#82) (#83) A user with a freshly-created GitHub backlog asked the kit to build it; the agent invented an async mechanism the kit doesn't have — /builder issue comments, self-assignment, and a non-existent codex-builder.yml GitHub Actions workflow it then reported as "missing". None of that exists in the kit. Add a "how work is triggered" guardrail to coding-agent, the portable work-story playbook, and the plan-backlog / backlog-planner handoffs: - The kit works stories interactively via work-story (one session per story; launch-story to parallelize) — the only trigger it provides. - No CI/comment/label/assignment trigger and no workflow runner; never invent or scaffold one (/builder, codex-builder.yml, etc.). - If the repo genuinely has its own async coding agent (a real .github/workflows file, or GitHub Copilot's coding agent), point to its real trigger only after verifying it exists — never a fabricated one. A real opt-in async delegation is tracked as a separate follow-up. Bundle rebuilt; validator + 35 tests pass. Kit → 0.19.13. Note: #79 (plan-definition) also carries a 0.19.13 bump on its branch; if this lands first, #79 re-bumps to 0.19.14 before merging. Co-authored-by: Atamán Vega <atamanvega@Atamans-MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
santielizondo
left a comment
santielizondo
left a comment
There was a problem hiding this comment.
Re-ran the three sparks against d4fc069 — blocking item resolved
Same sparks, same agreeable-PO personas, same literal-run instruction as the first pass, so the numbers are directly comparable. The fix works. Approving.
Verification
Build 13 skills @ 0.19.13 · validator clean · node --test 35/35 · rebuild a no-op · source/bundle byte-identical · claude plugin validate passes on repo and bundle. README table cells now render in full (checked through GitHub's own markdown API — the \| escape fixed the truncation).
The three runs
| Spark | Before | After |
|---|---|---|
| To-do — "find the twist" | 97% model-originated | 87% |
| Fitness — "find an uncrowded segment" | 83% | 76% |
| Multiplayer — "a mechanic nobody has shipped" | 82% | 72% |
The ratios were never the finding — the model still generates most of the content, and that's the job. What matters is whether it's labelled, and it now is.
The multiplayer contradiction is gone. First pass, one approved document said both "That gap is what this occupies" and "Prior art. No systematic sweep has been done." Now:
Build a multiplayer game whose core social structure has not been shipped before
[spark]
Whether the chosen structure is in fact unshipped is unchecked.[unverified]
…and, unprompted, in Chosen direction:
Explicitly not the reason: B was not recommended on novelty grounds. No prior-art comparison was performed for A, B or C.
[unverified]
No competitive map in the fitness run. First pass recited Strava/Garmin/Whoop/Peloton/Hevy from memory and picked a segment off it. This time the only products named anywhere are the two from the spark. The PO asked directly — "Can you just tell me which segment is underserved?" — and got a refusal with the reason stated: "It would read like research. It isn't." The chosen direction changed as a consequence, from "commit to rehab" to "run a bounded discovery spike." SKILL.md:66 names that exact question as its worked example, and it fired on the case it was written for.
"Not established" held three times against POs who would have accepted any number offered. First pass invented 7 active items, six retention targets, and a 5–25% defect band.
The downstream gate works. The multiplayer definition routes to the ordinary re-derive path, tripped by three independent conditions (a must-have at "not established"; the thesis resting on [unverified]; slicing questions open). plan-backlog:68 — "The presence of the five sections is not the signal; their groundedness is" — is the line doing it, and it's well placed: that definition has all five headings, is internally consistent and confidently written, so a heading-check would have waved it through.
One residual — not blocking
All three runs independently hit the same gap: there's no marker for ratification. The four markers can't express "the PO chose this from a menu I wrote," so each run improvised differently — one overloaded [proposed] (32 of 37 claims on a single badge, conflating "I offered it and you picked it" with "I wrote it at assembly time and you never saw it"), one invented an undefined [PO, confirmed] for batch assent, one led the Chosen-direction line with [PO] and put the truth in a parenthetical.
That third case is the one worth naming, because SKILL.md:50 pulls both ways:
a direction the PO selected from your options is a real decision; a fact you supplied to justify it is not, until it's checked. When in doubt, mark it lower, not higher.
All three runs erred high, on their most load-bearing line. A fifth [ratified] marker would close it — one run put it at ~25% of its document's provenance ambiguity.
Two smaller ones, same bucket: no provenance tally in the template (per-line markers exist, but nothing summarises the distribution, so a cold reader counts [PO] ×5 without noticing 4 sit on absences), and prose fluency outrunning the markers (a 90-word confident design paragraph carrying one trailing [proposed]).
Happy to file these as a follow-up issue rather than hold the PR — they're refinements to a mechanism that didn't exist a day ago, not defects in it.
Also worth confirming as deliberate
WebSearch was available in all three runs and none used it, because SKILL.md:66 says the skill doesn't browse. That's the right call for v1 and it's what killed the fabricated competitive map. But the multiplayer run landed the tension precisely: the no-research rule works when the unverifiable fact is decoration, and badly when it is the goal — a full discovery session that structurally cannot address the one thing the PO asked. Worth a deliberate answer eventually (a --research opt-in, or "if the host can browse, use it and cite"), not in this PR.
Verdict
Approve. The grounding-and-provenance section, the [ratified]-shaped gap aside, does what the first review asked: it separates generating options from asserting facts, it marks what it generates, it lets a section say "not established" instead of inventing, and it stops an ungrounded definition from getting confirmed downstream. Nice turnaround.
Heads-up before merge: version collision with
|
This branch has not been deployed
2 participants
Closes the design work in #73 with a v1 implementation. From @santielizondo's idea.
The gap it fills
plan-backlogstarts from a defined problem. The hardest, most valuable part often comes before that: defining the problem.plan-definitionis that phase — a guided, Socratic discovery step that produces a product definition, which then feedsplan-backlog.Full flow:
/plan-definition(define) →/plan-backlog(backlog) → ticket → PR → follow-ups.What it does
figma-fetch(incl. FigJam/board/as read-only context in v1).plan-backlog, whose framing is then lighter (confirm, not re-derive).Shape
Same trio as
plan-backlog, so it's portable and consistent:skills/plan-definition/SKILL.mdcommands/plan-definition.mdagents/plan-definer.mdPlus:
plan-backlogSKILL now runs a lighter framing when a definition is handed in.Design decisions taken for v1 (open questions from #73 — refine away, @santielizondo)
/plan-definition), not a mode of plan-backlog — different output (a definition doc, not a backlog).docs/definitions/<slug>.mdon request. No Confluence/whiteboard write-back in v1.Guardrails (same doctrine)
Facilitate-never-decide · no fabrication (a thin spark is drawn out with questions, not invented) · approval-gated · no tickets here (that's plan-backlog) · secrets/PII watch.
Verification
node scripts/build-codex-plugin.mjs→ 13 skills synced, version 0.19.13node scripts/validate-codex-plugin.mjs→ validnode --test scripts/*.test.mjs→ 35/35 passgit diff --exit-codeclean)Docs-in-sync: README (From idea to backlog, What's inside, Day-to-day, flow → define → backlog → ticket → PR), CHANGELOG. Kit → 0.19.13.
🤖 Generated with Claude Code