Skip to content

feat(test-plan): add a form-semantics axis, qualify dedup, define the PR set - #37447

Open
erickgonzalez wants to merge 4 commits into
mainfrom
issue-37225-form-semantics-and-pr-set
Open

feat(test-plan): add a form-semantics axis, qualify dedup, define the PR set#37447
erickgonzalez wants to merge 4 commits into
mainfrom
issue-37225-form-semantics-and-pr-set

Conversation

@erickgonzalez

Copy link
Copy Markdown
Member

Proposed Changes

Three gaps found by reviewing a real run of the skill. Two further findings from the same review are
deliberately not here — see Not in this PR.

1 · A tenth coverage axis: Form & state semantics. The existing nine axes are platform concerns
(permissions, hosts, languages, version state, cache, workflow, push publish, persistence, UI/UX),
and the UI/UX checklist is purely presentational — rendering, keyboard, focus, screen reader, empty
states, responsive, i18n, theme, copy. It asks whether a control looks right; nothing asked whether
the data survives. The new axis prompts for every option of a bounded choice, numeric boundaries,
derived fields, child-entity CRUD inside a parent form, optional-set-vs-unset, which gesture persists
what, leaving with unsaved work, and locked states. On a form-heavy change the matrix previously
never asked.

2 · A mocked unit spec no longer retires a manual case. §5 dropped any axis a PR-added test
covered, with no caveat. PR #37064 added 162 mockReturnValue calls — four mocking
$hasUnsavedChanges — and zero e2e specs
, so the unsaved-changes guard was retired from the manual
plan on the strength of a spec that builds a fake component. A mock proves component logic, not a
persistence round-trip, a real route change, or a status transition.

3 · A run resolves a PR set as well as an issue set. The skill was single-PR by construction:
§5 diffed one <pr> and §2 resolved issues but never PRs. Spec-driven work lands as several PRs
against the same issues, and the skill can legitimately be handed issues alone. §2 now defines the
input contract for all three shapes; coverage runs over the union of the set's diffs; the Summary
names the PRs covered and any deliberately left out; unmerged PRs are excluded and said to be.

Checklist

  • Tests — n/a (skill instructions); every claim below verified against the cited PR/issue
  • Translations — n/a
  • Security Implications Contemplated — no change to tool permissions or input handling

Additional Info

Not in this PR, from the same review:

Unrelated: this also regenerates CATALOG.md, which was already stale on main by one count
line after speckit-docs-converge landed. Not part of this change, but skill-lint gates every PR
touching .claude/skills/**.

SKILL.md is now 445 lines — still under the ~500 convention, but close enough that the next
addition should move into references/.

This PR fixes: #37225

… PR set

Three gaps found by review of a real run, kept separate from the two findings
still under discussion.

Form & state semantics — a tenth coverage axis. The existing nine are platform
concerns and the UI/UX checklist is purely presentational: it asks whether a
control looks right, never whether the data survives. Nothing prompted for
every option of a bounded choice, numeric boundaries, derived fields, child
CRUD inside a parent form, which gesture persists what, or leaving with unsaved
work. On a form-heavy change the matrix simply never asked.

A mocked unit spec no longer retires a manual case. The dedup rule dropped any
axis a PR-added test covered, with no caveat. PR #37064 added 162
mockReturnValue calls, four of them mocking $hasUnsavedChanges, and zero e2e
specs -- so the unsaved-changes guard was retired from the manual plan on the
strength of a spec that builds a fake component. A mock proves component logic,
not a persistence round-trip, a route change or a status transition.

A run now resolves a PR set as well as an issue set. The skill was single-PR by
construction: section 5 diffed one <pr>, section 2 resolved issues and never
PRs. Spec-driven work lands as several PRs against the same issues, and the
skill can be handed issues alone. Coverage now runs over the union of the set's
diffs, the Summary names the PRs covered and any deliberately left out, and
unmerged PRs are excluded and said to be excluded.

Also regenerates CATALOG.md, which was already stale on main by one count line
after speckit-docs-converge landed -- unrelated to this change, but skill-lint
gates every PR touching .claude/skills/**.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@claude

claude Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Claude finished @erickgonzalez's task in 1m 26s —— View job


Review

Documentation-only change to the dot-test-plan skill (three markdown files). I reviewed the committed diff (origin/main...HEAD) and verified the substantive factual claims against the codebase.

Note on environment: the working tree in my checkout carries an uncommitted revert of this PR (16 insertions / 98 deletions — the exact inverse), so Read/grep of the files show the old text. I reviewed the committed state via git diff, which is correct. Flagging only so nobody is confused by a stray diff — it is not part of the PR.

Claims I verified

  • darkModeSelector pin — core-web/libs/ui/src/lib/theme/providers.ts:31 reads darkModeSelector: options?.darkModeSelector ?? false. Accurate.
  • ✅ No prefers-color-scheme / data-theme usage — zero matches across core-web. Accurate.
  • ✅ "The single .dark selector in the repo belongs to the standalone block-editor app config" — exactly one match, core-web/apps/dotcms-block-editor/src/app/app.config.ts. Accurate.
  • ✅ Axis count is internally consistent at ten across all three files (README intro + reference link, coverage-matrix.md header, and the ten table rows: Permissions, Sites/hosts, Languages, Content version state, Cache, Workflow, Push Publish, Persistence, UI/UX, + Form & state semantics).
  • ✅ PR-set contract is consistent between SKILL.md §2 (the table + the "not once per PR, not once per issue" walk rule), SKILL.md §5/§6, coverage-matrix.md, and the README "Multi-issue consolidation" section. The mocked-spec caveat now appears in both SKILL.md §5 and the coverage-matrix.md dedup footer, and they no longer contradict (the drift called out in your own comment is resolved).

New Issues

No issues found.

The change is coherent, the dedup/mocked-spec caveat is now stated identically in both places, and the "does not exist in dotCMS" carve-out is backed by real code references rather than assertion. The self-caught third drift instance (coverage-matrix.md:102) is a good catch — that was the one that mattered, since it was a substantive contradiction on this PR's own change rather than stale prose.

One forward-looking note you already raised in the description stands on its own merit: at 445 lines SKILL.md is close enough to the ~500 convention that the next addition should move into references/. Not a blocker for this PR.

· issue-37225-form-semantics-and-pr-set

Three lines in the file this PR edits still described the state before the
edit:

  - the header said "Nine product-surface axes" while the table below it had
    ten. This is an instruction file a model reads to decide which axes to
    walk, so an undercount risks the new axis being treated as optional --
    exactly what adding it was meant to prevent
  - the walk instruction said "the union of everything the PR changed",
    singular, contradicting SKILL.md section 6's "union of the PR set's diffs"
  - the dedup rule said "a test the PR itself added", also singular, and
    restated the rule without the mocked-spec caveat added to section 5, so
    the two files disagreed on when an axis may be dropped

The first two were raised in review; the third is the same drift one line
further down and would have left the reference file contradicting SKILL.md on
the substantive change in this PR.

Verified consistent across all four files: ten axis rows, ten in the header,
ten in the README, and both the matrix and section 6 walking the PR set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@erickgonzalez

Copy link
Copy Markdown
Member Author

Both valid — fixed in 2488f01, along with a third instance of the same drift that the review didn't catch.

🟡 "Nine product-surface axes" — fixed

Right, and right about why it matters rather than being cosmetic. This is an instruction file a model
reads to decide which axes to walk; a header undercounting its own table invites treating the tenth
as an afterthought, which defeats the point of adding it. Now Ten coverage axes., aligned with the
README's wording.

🟡 "the union of everything the PR changed" — fixed

Now the union of everything the PR set changed — not once per PR, and not once per issue. The
explicit "not once per PR" is there because that is the wrong reading the new multi-PR contract makes
newly available.

A third one, same drift, one line further down

coverage-matrix.md:102 also said "a test the PR itself added" — singular, and it restated the
dedup rule without the mocked-spec caveat this PR adds to SKILL.md §5. So the two files
disagreed on when an axis may be dropped, which is worse than the two you flagged: not stale prose
but a substantive contradiction on this PR's own change. Now:

An axis or mandatory case already covered by a test one of the PRs itself added gets no manual
case … This holds only when that test exercises the same path a person would: a spec that mocks the
store or builds a fake component does not retire the case (SKILL.md §5).

Root cause

All three are the same mistake: I edited this file to add a row and a checklist and never re-read the
prose around them. The three substantive changes were verified against the cited PRs; the file's own
narration was not. Worth noting for anyone reviewing a future skill PR — a reference file's
introduction is easy to leave describing the state before the edit.

Verified consistent across all four files now: ten axis rows, ten in the matrix header, ten in the
README, and both the matrix and SKILL.md §6 walking the PR set. skill-lint clean, fences balanced.

…have

Generated plans have been carrying a "verify light and dark mode" case for a
feature that does not exist. The UI/UX checklist asked for it directly --
"Theme / contrast: light and dark both render correctly" -- so the case was
prompted by the skill, not invented from thin air, and every one of them was a
hallucination a reviewer had to spend attention rejecting.

Verified before encoding the constraint:

  - core-web/libs/ui/src/lib/theme/providers.ts pins PrimeNG's
    darkModeSelector to `options?.darkModeSelector ?? false`
  - no file under core-web/libs or core-web/apps reads prefers-color-scheme
  - no element carries a data-theme attribute, and there is no theme switcher
  - the repository's only `.dark` selector is in the standalone block-editor
    app config, with nothing wired to activate it

Split the checklist item: contrast is real and stays, extended to disabled,
error and selected states. Light/dark is removed and named explicitly in a new
"Does not exist in dotCMS" section, because deleting the prompt is not enough
on its own -- a model generating a UI plan will reach for dark mode from
general web knowledge unless told it is absent, and told why.

The section is scoped so it cannot become stale in the wrong direction: if a
change genuinely introduces one of these, the diff shows it and it is in scope
like any other new behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Skill proposal: dot-test-plan (adds new 'test' domain prefix)

1 participant