Repository navigation
Add rule templates (abstract, extends, semantic.criterion) - #237
mindsocket wants to merge 1 commit into
Conversation
A rule with "abstract": true is a template that never runs. A rule that names it in "extends" takes its fields, then sets its own; semantic fields replace the template's one at a time. The resolved rule is checked as a full rule. semantic.criterion is read after the rubric, so a template's rubric can stay generic. The bundled sctx://core/_semantic_patterns partial holds four templates (children-distinct, children-cover-parent, refines-parent, serves-ancestors), and seven bundled rules now extend them. goal-advances-parent's passing verdict is now "refines". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XphpcQHEAB9KwJ5Ltiomb
mindsocket
left a comment
mindsocket
left a comment
There was a problem hiding this comment.
Interim thoughts - these are more encouragements to keep exploring the solution space and running comparisons and iterating on wording and shape before zeroing in on a final solution.
| | `refines-parent` | The node is one part, cause, case or means of its parent | The child | `parent` | `opportunity-refines-parent`, `goal-advances-parent` | | ||
| | `serves-ancestors` | The node serves the intent of the levels above | The child | `ancestors` | `solution-serves-outcome`, `opportunity-serves-ancestors` | | ||
|
|
||
| ```json |
There was a problem hiding this comment.
Considerations as part of the review of this PR and the one it's stacked on:
- Is criterion needed?
- Is criterion a good name? Is "criterion gets tacked onto rubric" a valid and reasonably understandable statement to a user?
- How does a custom rule (no template, hand-coded in top-level schema) vs a template rule with criterion vs a template rule without criterion compare in evals, for the same intent? That informs where we might focus attention on the shape, composition and wording used in rules.
| | `serves-ancestors` | The node serves the intent of the levels above | The child | `ancestors` | `solution-serves-outcome`, `opportunity-serves-ancestors` | | ||
|
|
||
| ```json | ||
| { "id": "goals-distinct", "extends": "children-distinct", "description": "Sibling goals are distinct", |
There was a problem hiding this comment.
This doesn't feel like a great example yet. We need to prove that criterion is useful vs not having one at all for a given use of a bundled template.
There was a problem hiding this comment.
I think a good example might emerge where the criterion clarifies the interpretation of the language in a rule. eg "refines" means {blah} in this context
| "id": "children-distinct", | ||
| "abstract": true, | ||
| "category": "coherence", | ||
| "description": "A node's children are distinct: achieving or addressing one does not achieve or address another", |
There was a problem hiding this comment.
We'll probably need to experiment with:
- language - 'achieve' or 'address' may be too specific, too vague, orthogonal, or risks misinterpretation by models
- additional "keywords" that might help guide models - eg KAOS, MECE, minto pyramid etc - referencing established research and concepts that would help the model interpret the rubric as intended
- skills (for schema authoring) to steer criterion overlay tips that build on the core rules
| } | ||
| }, | ||
| { | ||
| "id": "serves-ancestors", |
There was a problem hiding this comment.
These rules could be fine, but I wonder if they've wandered from the more rigid and research backed rule ideas from the original analysis in artifact.
| "category": "coherence", | ||
| "description": "A node belongs under its parent: it is one part, cause, case or means of the parent", | ||
| "semantic": { | ||
| "rubric": "Judge only how the item relates to its parent. Something worthwhile in the same area as the parent does not refine the parent unless it passes the criterion's test.", |
There was a problem hiding this comment.
Some rules sound quite alike. That could be fine when the nuance matters, but those nuances might need describing. Hold that thought until we've converged on working evaluations.
| { | ||
| "id": "goals-cover-parent", | ||
| "category": "best-practice", | ||
| "extends": "children-cover-parent", |
There was a problem hiding this comment.
A/B test the original rules vs these new ones.
Also consider an AI improvement loop to hillclimb on this (get my approval on approach first)
🤖 Generated with Claude Code
https://claude.ai/code/session_019XphpcQHEAB9KwJ5Ltiomb
Slice 4 of the semantic rule primitives plan, stacked on #233. This is a draft to inform the dialect decision. The
Rulesurface is not agreed yet.Summary
"abstract": trueis a template, and it never runs. A rule withextends: "<template id>"takes the template's fields, then sets its own. Top-level fields replace the template's. Fields insidesemanticreplace the template's one at a time.extendsnames a rule that isn't abstract;semantic.criterion. The judge reads it after the rubric. It is folded into the judge spec'srubric, so all three provider shapes (chat, Jev, OpenAI decisions) get it with no provider-specific code.sctx://core/_semantic_patterns. This new bundled partial holds four templates. Seven slice 3 rules now extend them:children-distinct:opportunities-distinct,goals-distinctchildren-cover-parent:goals-cover-parentrefines-parent:opportunity-refines-parent,goal-advances-parentserves-ancestors:solution-serves-outcome,opportunity-serves-ancestorssolution-addresses-opportunitystays plain, because means-to-need is a different test from refining a parent. The type-specific rules stay plain too.Rule.semantic.criterionis new.abstractandextendsexist only on the declared form (DeclaredRule, internal), because resolved metadata never contains them.Choices for review
semantic.severityreplaces the template's whole map, not single entries.goal-advances-parentverdict renamed. Its passing verdict is nowrefines, shared with the template, and its eval labels are updated.schemas showchange. It lists resolved rules, which is what runs. It doesn't print rubrics, so templates add nothing to its output.Test plan
bun run test,typecheck,lint,test:hook,test:modbun run eval:semantic --checktests/schema/rule-templates.test.ts: resolution, reuse, unknown and concrete targets, the resolved-rule check, no nested templates, and bundled rulesGenerated by Claude Code