From 37f000d7e46861fd36f702837cad39d48f230d4f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 02:59:48 +0000 Subject: [PATCH] Add rule templates and move slice 3 rules onto them A rule with "abstract": true is a template that never runs. A rule that names it in "extends" takes its fields, then sets its own; semantic fields replace the template's one at a time. The resolved rule is checked as a full rule. semantic.criterion is read after the rubric, so a template's rubric can stay generic. The bundled sctx://core/_semantic_patterns partial holds four templates (children-distinct, children-cover-parent, refines-parent, serves-ancestors), and seven bundled rules now extend them. goal-advances-parent's passing verdict is now "refines". Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019XphpcQHEAB9KwJ5Ltiomb --- docs/concepts.md | 2 + docs/rules.md | 4 + docs/schemas.md | 2 +- docs/semantic-rules.md | 26 +- evals/semantic/goal-advances-parent.json5 | 20 +- plugin/.claude-plugin/plugin.json | 2 +- plugin/hooks/projection.js | 132 +++++ plugin/skills/structured-context/SKILL.md | 1 + .../references/schema-authoring.md | 2 +- schemas/_ost_strict.json | 36 +- schemas/_semantic_patterns.json | 71 +++ schemas/_strategy_general.json | 44 +- .../_structured_context_schema_meta.json | 458 +++++++++++------- src/judge/prompt.ts | 3 +- src/judge/provider/jev.ts | 7 +- src/judge/provider/openai-decisions.ts | 2 +- src/judge/types.ts | 3 +- src/judge/verdicts.ts | 2 +- src/schema/metadata-contract.ts | 136 +++--- src/schema/rule-templates.ts | 46 ++ src/schema/schema.ts | 6 +- tests/judge/helpers.ts | 1 + tests/judge/verdicts.test.ts | 5 +- tests/schema/rule-templates.test.ts | 100 ++++ 24 files changed, 790 insertions(+), 321 deletions(-) create mode 100644 schemas/_semantic_patterns.json create mode 100644 src/schema/rule-templates.ts create mode 100644 tests/schema/rule-templates.test.ts diff --git a/docs/concepts.md b/docs/concepts.md index 652733b..e2d7d56 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -185,6 +185,8 @@ Rules may be: Rules are distinct from schema validation: the schema checks structure; rules check meaning and quality. +A **rule template** is a rule with `abstract: true`. It never runs; a rule that names it in `extends` takes its fields and sets its own. + A rule's `when` is its **applicability condition**: a test that decides whether the rule applies to a node. Its `expression` is the test the node must pass. Each executable rule has a **category** (`validation`, `coherence`, `workflow`, `best-practice`) that supplies its default **severity** (`error`, `warning`, `info`). A rule may override its category's severity, but categories are the preferred grouping. Only `error` severity violations fail validation. diff --git a/docs/rules.md b/docs/rules.md index 218f5c8..6d8ffa6 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -168,6 +168,10 @@ Example override: } ``` +### Rule templates + +A rule can reuse a rule template with `extends`. See [Rule templates](semantic-rules.md#rule-templates). + ## Cascade gating When a schema uses `$metadata.hierarchy`, rule evaluation cascades down the hierarchy, one level at a time from roots and orphans. A node becomes a **gate** when it fails in a way that makes its descendants' rules untrustworthy or wasteful to run. The nodes below a gate are **gated**: their rules are not evaluated. diff --git a/docs/schemas.md b/docs/schemas.md index 4e74052..c5b3f97 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -402,7 +402,7 @@ Every schema `$id` uses the `sctx://` scheme. The shape of the `$id` says where - `$id` values follow the [namespace rules](#schema-id-namespaces) and must not collide across bundled, plugin and local schemas. - `$ref` resolution is transitive across files. - Partials with no `$metadata` should prefer `$schema: "https://json-schema.org/draft/2020-12/schema"` so they validate standalone as plain JSON Schema fragments. -- **Bundled partials as entity libraries**: `_sctx_base.json`, `_strategy_general.json`, `_knowledge_wiki.json`, `_okf.json`, and `_ost_strict.json` provide reusable entity definitions and metadata. Composing schemas can reference their entity types via `$ref` rather than redefining them. +- **Bundled partials as entity libraries**: `_sctx_base.json`, `_strategy_general.json`, `_knowledge_wiki.json`, `_okf.json`, and `_ost_strict.json` provide reusable entity definitions and metadata. Composing schemas can reference their entity types via `$ref` rather than redefining them. `_semantic_patterns.json` provides [rule templates](semantic-rules.md#rule-templates). - **Partials can carry metadata**: Partials may include `$metadata` (hierarchy, aliases, relationships, rules). A schema gets it only by listing the partial in `$metadata.imports`; `$ref` alone brings none. ## Plugin-contributed schemas diff --git a/docs/semantic-rules.md b/docs/semantic-rules.md index 7171fff..820d316 100644 --- a/docs/semantic-rules.md +++ b/docs/semantic-rules.md @@ -44,6 +44,7 @@ A semantic rule has the [common rule fields](rules.md) and a `semantic` object. | Field | Required | Description | |---|---|---| | `semantic.rubric` | no | Tells the model what to assess. Default: `description` | +| `semantic.criterion` | no | Text the model reads after the rubric. A rule that extends a [template](#rule-templates) uses it to make the template's rubric specific | | `semantic.verdicts` | no | The answers the model can give. Default: `pass` and `fail`. See [Verdicts](#verdicts) | | `semantic.severity` | if `semantic.verdicts` is set | The severity of each verdict. See [Severity](#severity) | | `semantic.scope` | no | The related nodes to send with the evaluated node. Default: none. See [Related nodes and content](#related-nodes-and-content) | @@ -146,15 +147,26 @@ Scope sends node data only, never the verdicts of other rules, and [gating](rule A global rule cannot have a `semantic` part, so it has no scope. -The bundled schemas use scope in three ways. We recommend them as models for your own rules: +## Rule templates -| Check | Attach the rule to | Scope | Bundled example | -|---|---|---|---| -| The children are distinct, or cover the node | The parent, with a `when` that needs two or more children | `children` | `opportunities-distinct`, `goals-cover-parent` | -| The node fits its parent | The child | `parent` | `opportunity-refines-parent`, `solution-addresses-opportunity` | -| The node serves the intent of the levels above | The child | `ancestors` | `solution-serves-outcome`, `opportunity-serves-ancestors` | +A **rule template** is a rule with `"abstract": true`, which never runs. A rule that names it in `extends` takes all of its fields, then sets its own. Top-level fields replace the template's. Fields inside `semantic` replace the template's one at a time, so a rule can add a `criterion` and keep the template's rubric, scope and verdicts. -The rules are in the bundled partials, such as `schemas/_ost_strict.json`. +The bundled partial `sctx://core/_semantic_patterns` holds four templates. To use one, add the partial to `$metadata.imports`: + +| Template | Checks | Attach the rule to | Scope | Bundled rules that extend it | +|---|---|---|---|---| +| `children-distinct` | The children are distinct | The parent, with a `when` that needs two or more children | `children` | `opportunities-distinct`, `goals-distinct` | +| `children-cover-parent` | The children cover the parent | The parent, with the same `when` | `children` | `goals-cover-parent` | +| `refines-parent` | The node is one part, cause, case or means of its parent | The child | `parent` | `opportunity-refines-parent`, `goal-advances-parent` | +| `serves-ancestors` | The node serves the intent of the levels above | The child | `ancestors` | `solution-serves-outcome`, `opportunity-serves-ancestors` | + +```json +{ "id": "goals-distinct", "extends": "children-distinct", "description": "Sibling goals are distinct", + "when": "$count(nodes[$$.current.id in resolvedParentIds and resolvedType = 'goal']) >= 2", + "semantic": { "criterion": "Judge the child goals. Each should be a different result." } } +``` + +A rule must extend a template, not another rule, and a template cannot extend a template. The rule is checked as a full rule once it takes the template's fields. ## Input size and trimming diff --git a/evals/semantic/goal-advances-parent.json5 b/evals/semantic/goal-advances-parent.json5 index 850eee6..fe40dfe 100644 --- a/evals/semantic/goal-advances-parent.json5 +++ b/evals/semantic/goal-advances-parent.json5 @@ -2,8 +2,8 @@ space: 'strategy-trees', expected: { 'goal-advances-parent': { - 'Launch online ordering for 100 bookshops.md': 'advances', - 'Help bookshops show live stock on their own websites.md': 'advances', + 'Launch online ordering for 100 bookshops.md': 'refines', + 'Help bookshops show live stock on their own websites.md': 'refines', // About bookshops, but not tools for them. 'Open our own flagship bookshop.md': ['weak-link', 'misplaced'], 'Grow our social media following.md': 'weak-link', @@ -11,23 +11,23 @@ 'Give independent bookshops online tools.md': 'weak-link', // Sells through a rival of the bookshops, not for them. 'Sell our own e-books on Kindle.md': ['weak-link', 'misplaced'], - 'Sign 100 bookshops to the ordering pilot.md': 'advances', - 'Build a checkout that works on phones.md': 'advances', + 'Sign 100 bookshops to the ordering pilot.md': 'refines', + 'Build a checkout that works on phones.md': 'refines', // Restates the parent goal. 'Get 100 bookshops ordering online.md': 'weak-link', 'Run a reading festival.md': 'misplaced', // Support could help the launch, but the goal does not say how. 'Improve our customer support.md': 'weak-link', - 'Cook dinner together three nights a week.md': 'advances', + 'Cook dinner together three nights a week.md': 'refines', 'Get a promotion this year.md': ['weak-link', 'misplaced'], - 'Take the kids hiking every month.md': 'advances', + 'Take the kids hiking every month.md': 'refines', // Restates half of the mission. - 'Be healthier.md': ['advances', 'weak-link'], - 'Build to 80 km weeks by January.md': 'advances', + 'Be healthier.md': ['refines', 'weak-link'], + 'Build to 80 km weeks by January.md': 'refines', // A means to the goal more than a part of it. - 'Buy new trail shoes.md': ['advances', 'weak-link'], + 'Buy new trail shoes.md': ['refines', 'weak-link'], // Supports the race more than it is part of it. - 'Lose 5 kg before race day.md': ['advances', 'weak-link'], + 'Lose 5 kg before race day.md': ['refines', 'weak-link'], }, }, } diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index e695810..ff50a8f 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "structured-context", - "version": "0.23.14", + "version": "0.23.15", "description": "Structured content authoring for Obsidian — automatic validation hooks, slash commands, and agent skills to keep markdown files compliant with your schema.", "homepage": "https://github.com/mindsocket/structured-context", "repository": "https://github.com/mindsocket/structured-context", diff --git a/plugin/hooks/projection.js b/plugin/hooks/projection.js index 580ab0b..53d329d 100644 --- a/plugin/hooks/projection.js +++ b/plugin/hooks/projection.js @@ -125,6 +125,138 @@ var HIERARCHY_LEVEL_SCHEMA = { required: ["type"], additionalProperties: false }; +var SEVERITY_SCHEMA = { enum: ["error", "warning", "info"] }; +var SEMANTIC_SCOPES = ["parent", "siblings", "children", "ancestors", "descendants", "branch"]; +var VERDICT_SEVERITY_SCHEMA = { enum: ["error", "warning", "info", "ignore"] }; +var SEMANTIC_SCHEMA = { + type: "object", + properties: { + /** What the model judges the node against. Falls back to the rule's `description`. */ + rubric: { type: "string", minLength: 1 }, + /** What this rule adds to its rubric: the model reads it after the rubric. Lets a template's rubric stay generic. */ + criterion: { type: "string", minLength: 1 }, + /** + * The closed set of verdicts the model may return. Each is a description (what the verdict means, for the + * model) or `{ description, guidance }`, where `guidance` is shown to the user when it fires. + * Defaults to `pass`/`fail`. + */ + verdicts: { + type: "object", + additionalProperties: { + oneOf: [ + { type: "string", minLength: 1 }, + { + type: "object", + properties: { description: { type: "string", minLength: 1 }, guidance: { type: "string", minLength: 1 } }, + required: ["description"], + additionalProperties: false + } + ] + }, + minProperties: 2 + }, + /** + * Severity per verdict in effect (declared, default or reserved). A verdict mapped to `ignore` is judged but + * not reported, and a verdict left out passes. Required with `verdicts`; with the default verdicts, `fail` + * takes the rule's own `severity` instead. + */ + severity: { + type: "object", + additionalProperties: VERDICT_SEVERITY_SCHEMA, + minProperties: 1 + }, + /** + * Related nodes, along main-hierarchy edges, sent with the evaluated node. Additive; left out or empty means + * the evaluated node alone. + */ + scope: { + type: "array", + items: { enum: SEMANTIC_SCOPES }, + uniqueItems: true + }, + /** + * Which node bodies are sent: `nodeOnly` (the evaluated node's), `fullScope` (it and every related node's), + * or `true` for `fullScope`. Left out: no bodies. + */ + includeContent: { enum: ["nodeOnly", "fullScope", true] } + }, + additionalProperties: false +}; +var RULE_PROPERTIES = { + id: { type: "string", minLength: 1 }, + category: { enum: ["validation", "coherence", "workflow", "best-practice"] }, + /** + * Severity of a failed `expression` and of the default `fail` verdict. Defaults from the category. + */ + severity: SEVERITY_SCHEMA, + description: { type: "string", minLength: 1 }, + /** JSONata test a node must pass. When the rule also has `semantic`, a failing node is reported without being evaluated by the model. */ + expression: { type: "string", minLength: 1 }, + /** Optional JSONata applicability condition, evaluated per node before the rule runs. The rule applies only when it returns true. */ + when: { type: "string", minLength: 1 }, + /** A model evaluates the node against a rubric. `{}` takes every default; `true` means the same. */ + semantic: { oneOf: [{ const: true }, SEMANTIC_SCHEMA] }, + type: { type: "string", minLength: 1 }, + /** Run once for the whole space instead of once per node. */ + global: { type: "boolean" }, + override: { type: "boolean" }, + /** Whether a failure of this rule gates (suppresses) downstream rules on the node's hierarchy subtree. Defaults to true when the failure's severity is `error`. */ + gatesDescendants: { type: "boolean" } +}; +var RULE_CONSTRAINTS = [ + // A rule has an expression, a semantic part, or both. + { anyOf: [{ required: ["expression"] }, { required: ["semantic"] }] }, + // A global rule has no node to apply to or to evaluate. + { + if: { required: ["global"], properties: { global: { const: true } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { not: { anyOf: [{ required: ["when"] }, { required: ["type"] }, { required: ["semantic"] }] } } + }, + // Custom verdicts need a severity for each failing verdict. + { + if: { required: ["semantic"], properties: { semantic: { type: "object", required: ["verdicts"] } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { properties: { semantic: { type: "object", required: ["severity"] } } } + }, + // With the default verdicts, `fail` takes the rule's severity. + { + if: { required: ["semantic"], properties: { semantic: { type: "object", not: { required: ["verdicts"] } } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { + properties: { + semantic: { type: "object", properties: { severity: { type: "object", not: { required: ["fail"] } } } } + } + } + }, + // A rule with custom verdicts and no expression has no failure for the top-level severity to describe. + { + if: { + required: ["semantic"], + properties: { semantic: { type: "object", required: ["verdicts"] } }, + not: { required: ["expression"] } + }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { not: { required: ["severity"] } } + } +]; +var RULE_SCHEMA = { + type: "object", + properties: RULE_PROPERTIES, + required: ["id", "category", "description"], + additionalProperties: false, + allOf: RULE_CONSTRAINTS +}; +var DECLARED_RULE_SCHEMA = { + oneOf: [ + { ...RULE_SCHEMA, properties: { ...RULE_PROPERTIES, abstract: { type: "boolean" } } }, + { + type: "object", + properties: { ...RULE_PROPERTIES, extends: { type: "string", minLength: 1 } }, + required: ["id", "extends"], + additionalProperties: false + } + ] +}; var RELATIONSHIP_SCHEMA = { type: "object", properties: { diff --git a/plugin/skills/structured-context/SKILL.md b/plugin/skills/structured-context/SKILL.md index 12be8e8..e642008 100644 --- a/plugin/skills/structured-context/SKILL.md +++ b/plugin/skills/structured-context/SKILL.md @@ -117,6 +117,7 @@ what the rule actually sees in the `current` object, then adjust the rule in the | `Link target "X" in field "parent" not found` | Broken wikilink | Fix the link, or ensure the target file exists and has that name | | `Link target "X" in field "parent" is ambiguous (matches multiple nodes)` | Several nodes share that name, for example `Overview.md` in different folders | Link with enough of the path to be unique, such as `[[alpha/Overview]]` | | `[rule-id] ... (Rule error: ...)` | A rule's `expression` failed to evaluate (syntax or runtime error) | Verify the expression with `dump` and a JSONata tester | +| `Rule "X" extends "Y", but no rule template ... has that id` | `extends` names a rule that is missing or lacks `"abstract": true` | Import the partial that holds the template (bundled ones are in `sctx://core/_semantic_patterns`), or fix the id | | `[rule-id] ... (Semantic evaluation error: ...)` | The model provider call failed, or input is still over the per-node input cap after trimming | Check `sctx providers` and the API key; narrow `semantic.scope` or drop `includeContent` | ## Plugins diff --git a/plugin/skills/structured-context/references/schema-authoring.md b/plugin/skills/structured-context/references/schema-authoring.md index c0eace4..439d5b1 100644 --- a/plugin/skills/structured-context/references/schema-authoring.md +++ b/plugin/skills/structured-context/references/schema-authoring.md @@ -101,7 +101,7 @@ Other rule fields: - `includeContent` — `nodeOnly` (the evaluated node's body) or `fullScope`/`true` (every body in scope). Without it, no body is sent. - A rule with `scope` or `includeContent` may return `missing-context` (default `warning`) when its input was trimmed to fit. - Semantic rules send the schema's type and field descriptions to the model with the rubric. Descriptions that say what a good node looks like save repeating that detail in each rubric, and let more general rules draw on the schema. -- To check a set of children, a fit with the parent, or the intent of the levels above, copy a bundled rule (`opportunities-distinct`, `opportunity-refines-parent`, `solution-serves-outcome` in `_ost_strict.json`). `sctx docs semantic-rules` shows which scope each uses. +- To check a set of children, a fit with the parent, or the intent of the levels above, import `sctx://core/_semantic_patterns` and extend one of its rule templates: `{ "id": "…", "extends": "children-distinct", "type": "…", "semantic": { "criterion": "…" } }`. `sctx docs semantic-rules` lists the templates. ## Metadata composition semantics diff --git a/schemas/_ost_strict.json b/schemas/_ost_strict.json index 961fe99..aa03d35 100644 --- a/schemas/_ost_strict.json +++ b/schemas/_ost_strict.json @@ -5,6 +5,7 @@ // This schema follows Teresa Torres' canonical 4-level structure as described in // "Continuous Discovery Habits" (2021) and at producttalk.org. "$metadata": { + "imports": ["sctx://core/_semantic_patterns"], "hierarchy": { "levels": ["outcome", { "type": "opportunity", "selfRef": true }, "solution", "assumption_test"] }, @@ -71,42 +72,20 @@ }, { "id": "opportunity-refines-parent", - "category": "coherence", + "extends": "refines-parent", "type": "opportunity", "description": "A top-level opportunity drives its outcome, and a child opportunity is a subset of its parent opportunity", "semantic": { - "rubric": "Judge only how the opportunity relates to its parent. A real need in the same area as the parent does not refine the parent unless it fits the verdict's test.", - "scope": ["parent"], - "verdicts": { - "refines": "Under an outcome, addressing it would drive the outcome. Under an opportunity, it is one part, cause or case of the parent need, narrower than the parent", - "weak-link": { - "description": "Related to the parent, but the link is indirect, or it only restates the parent", - "guidance": "Name the narrower need it covers, or move it next to its parent." - }, - "misplaced": { - "description": "It would not drive the outcome, or it is not part of the parent opportunity", - "guidance": "Move it under the outcome or opportunity it belongs to." - } - }, - "severity": { "weak-link": "info", "misplaced": "warning" } + "criterion": "Under an outcome, addressing the opportunity would drive the outcome. Under an opportunity, it is one part, cause or case of the parent need, narrower than the parent." } }, { "id": "opportunities-distinct", - "category": "coherence", + "extends": "children-distinct", "description": "Sibling opportunities are distinct: the team can address one without addressing another", "when": "$count(nodes[$$.current.id in resolvedParentIds and resolvedType = 'opportunity']) >= 2", "semantic": { - "rubric": "Judge the item's child opportunities as a set, and ignore its other children. They are distinct when each is a different need, so that addressing one does not address another.", - "scope": ["children"], - "verdicts": { - "distinct": "Each child opportunity is a different need", - "overlapping": { - "description": "Two child opportunities state the same need, or partly cover the same need, or one contains another", - "guidance": "Merge them, sharpen the boundary between them, or nest the narrower one under the broader one." - } - }, - "severity": { "overlapping": "warning" } + "criterion": "Judge the child opportunities. Each should be a different customer need, so that addressing one does not address another." } }, { @@ -133,12 +112,11 @@ }, { "id": "solution-serves-outcome", - "category": "coherence", + "extends": "serves-ancestors", "type": "solution", "description": "The solution serves the outcome above it, not only its parent opportunity", "semantic": { - "rubric": "Follow the chain from the solution through its opportunities to the outcome. Pass when the solution, if it worked, would move the outcome's metric in the intended direction. Fail when it would not move the outcome, or would move it the wrong way, even if it addresses its parent opportunity.", - "scope": ["ancestors"] + "criterion": "The top level is an outcome with a metric. The solution serves it when, if it worked, it would move that metric in the intended direction." } }, { diff --git a/schemas/_semantic_patterns.json b/schemas/_semantic_patterns.json new file mode 100644 index 0000000..c13a18c --- /dev/null +++ b/schemas/_semantic_patterns.json @@ -0,0 +1,71 @@ +{ + "$schema": "https://raw.githubusercontent.com/mindsocket/structured-context/main/schemas/generated/latest/_structured_context_schema_meta.json", + "$id": "sctx://core/_semantic_patterns", + // Rule templates for common semantic checks on a hierarchy. A schema imports this partial and declares a rule + // that `extends` a template, adding its own `type`, `when`, `description` and `semantic.criterion`. + "$metadata": { + "rules": [ + { + "id": "children-distinct", + "abstract": true, + "category": "coherence", + "description": "A node's children are distinct: achieving or addressing one does not achieve or address another", + "semantic": { + "rubric": "Judge the item's children as a set. If the criterion names a kind of child, judge only those and ignore the rest. The children are distinct when each is a different need, result or idea, so that achieving or addressing one does not achieve or address another.", + "scope": ["children"], + "verdicts": { + "distinct": "Each child is different", + "overlapping": { + "description": "Two children state the same thing, or partly cover the same ground, or one contains another", + "guidance": "Merge them, sharpen the boundary between them, or nest the narrower one under the broader one." + } + }, + "severity": { "overlapping": "warning" } + } + }, + { + "id": "children-cover-parent", + "abstract": true, + "category": "best-practice", + "description": "Together, a node's children cover what it needs", + "semantic": { + "rubric": "Judge the item's children as a set. If the criterion names a kind of child, judge only those and ignore the rest. Pass when achieving all of them would achieve the item as its title and body state it. Fail when a part that the item names or plainly needs has no child.", + "scope": ["children"], + "includeContent": "fullScope" + } + }, + { + "id": "refines-parent", + "abstract": true, + "category": "coherence", + "description": "A node belongs under its parent: it is one part, cause, case or means of the parent", + "semantic": { + "rubric": "Judge only how the item relates to its parent. Something worthwhile in the same area as the parent does not refine the parent unless it passes the criterion's test.", + "scope": ["parent"], + "verdicts": { + "refines": "It passes the criterion's test: it is one part, cause, case or means of its parent, narrower than the parent", + "weak-link": { + "description": "Related to the parent, but the link is indirect, or it restates the parent in other words instead of naming one part of it", + "guidance": "Say how it serves the parent, or narrow it to one part of the parent." + }, + "misplaced": { + "description": "It does not belong under its parent", + "guidance": "Move it under the node it belongs to." + } + }, + "severity": { "weak-link": "info", "misplaced": "warning" } + } + }, + { + "id": "serves-ancestors", + "abstract": true, + "category": "coherence", + "description": "A node serves every level above it, not only its parent", + "semantic": { + "rubric": "Follow the chain from the item through each ancestor to the top. Pass when the item, if achieved, serves every level as the criterion describes. Fail when it would serve its parent but ignore, or work against, the intent of a level above.", + "scope": ["ancestors"] + } + } + ] + } +} diff --git a/schemas/_strategy_general.json b/schemas/_strategy_general.json index d233422..67c16c4 100644 --- a/schemas/_strategy_general.json +++ b/schemas/_strategy_general.json @@ -5,6 +5,7 @@ // Provides a hierarchy, aliases, relationships, and entity type $defs. // Schemas that import this partial get its metadata; a $ref to its $defs alone brings none. "$metadata": { + "imports": ["sctx://core/_semantic_patterns"], "hierarchy": { "levels": [ "vision", @@ -44,65 +45,40 @@ "rules": [ { "id": "goals-distinct", - "category": "coherence", + "extends": "children-distinct", "description": "Sibling goals are distinct: progress on one is not progress on another", "when": "$count(nodes[$$.current.id in resolvedParentIds and resolvedType = 'goal']) >= 2", "semantic": { - "rubric": "Judge the item's child goals as a set, and ignore its other children. They are distinct when each is a different result, so that achieving one does not achieve another.", - "scope": ["children"], - "verdicts": { - "distinct": "Each child goal is a different result", - "overlapping": { - "description": "Two child goals state the same result, or partly cover the same result, or one contains another", - "guidance": "Merge them, sharpen the boundary between them, or nest the narrower one under the broader one." - } - }, - "severity": { "overlapping": "warning" } + "criterion": "Judge the child goals. Each should be a different result, so that achieving one does not achieve another." } }, { "id": "goals-cover-parent", - "category": "best-practice", + "extends": "children-cover-parent", "description": "Together, a mission's or goal's child goals cover what it needs", "when": "$count(nodes[$$.current.id in resolvedParentIds and resolvedType = 'goal']) >= 2", "semantic": { - "rubric": "Judge the item's child goals as a set, and ignore its other children. Pass when achieving all of them would achieve the item as its title and body state it. Fail when a part that the item names or plainly needs has no goal.", - "scope": ["children"], - "includeContent": "fullScope" + "criterion": "Judge the child goals." } }, { "id": "goal-advances-parent", - "category": "coherence", + "extends": "refines-parent", "type": "goal", "description": "A goal advances its parent mission, or is one part of its parent goal", "when": "$exists(parent)", "semantic": { - "rubric": "Judge only how the goal relates to its parent. A worthwhile goal in the same area as the parent does not advance the parent unless it fits the verdict's test.", - "scope": ["parent"], - "verdicts": { - "advances": "Under a mission, achieving it moves the mission forward. Under a goal, it is one part of the parent goal, narrower than the parent", - "weak-link": { - "description": "Related to the parent, but the link is indirect, or it restates the parent in other words instead of naming one part of it", - "guidance": "Say how it advances the parent, or narrow it to one part of the parent." - }, - "misplaced": { - "description": "Achieving it would not advance the parent", - "guidance": "Move it under the mission or goal it advances." - } - }, - "severity": { "weak-link": "info", "misplaced": "warning" } + "criterion": "Under a mission, achieving the goal moves the mission forward. Under a goal, it is one part of the parent goal, narrower than the parent." } }, { "id": "opportunity-serves-ancestors", - "category": "coherence", + "extends": "serves-ancestors", "type": "opportunity", "description": "Pursuing the opportunity serves the mission and vision above it, not only its parent goal", "semantic": { - "rubric": "Follow the chain from the opportunity through its goals to the mission and vision. Pass when pursuing it serves every level. Fail when it would advance its parent but work against, or ignore, the intent of a level above.", - "scope": ["ancestors"], - "includeContent": "fullScope" + "includeContent": "fullScope", + "criterion": "The levels above are goals, then the mission and vision." } }, { diff --git a/schemas/generated/latest/_structured_context_schema_meta.json b/schemas/generated/latest/_structured_context_schema_meta.json index 5257dab..ae5d15e 100644 --- a/schemas/generated/latest/_structured_context_schema_meta.json +++ b/schemas/generated/latest/_structured_context_schema_meta.json @@ -1,6 +1,6 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://raw.githubusercontent.com/mindsocket/structured-context/main/schemas/generated/0.13.0/_structured_context_schema_meta.json", + "$id": "https://raw.githubusercontent.com/mindsocket/structured-context/main/schemas/generated/0.23.0/_structured_context_schema_meta.json", "title": "structured-context schema dialect", "description": "Extends JSON Schema 2020-12 with top-level $metadata for imports, hierarchy, relationships, aliases and rules.", "type": "object", @@ -187,208 +187,330 @@ "rules": { "type": "array", "items": { - "type": "object", - "properties": { - "id": { - "type": "string", - "minLength": 1 - }, - "category": { - "enum": ["validation", "coherence", "workflow", "best-practice"] - }, - "severity": { - "enum": ["error", "warning", "info"] - }, - "description": { - "type": "string", - "minLength": 1 - }, - "expression": { - "type": "string", - "minLength": 1 - }, - "when": { - "type": "string", - "minLength": 1 - }, - "semantic": { - "oneOf": [ - { - "const": true + "oneOf": [ + { + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 1 }, - { - "type": "object", - "properties": { - "rubric": { - "type": "string", - "minLength": 1 + "category": { + "enum": ["validation", "coherence", "workflow", "best-practice"] + }, + "severity": { + "enum": ["error", "warning", "info"] + }, + "description": { + "type": "string", + "minLength": 1 + }, + "expression": { + "type": "string", + "minLength": 1 + }, + "when": { + "type": "string", + "minLength": 1 + }, + "semantic": { + "oneOf": [ + { + "const": true }, - "verdicts": { + { "type": "object", - "additionalProperties": { - "oneOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "object", - "properties": { - "description": { + "properties": { + "rubric": { + "type": "string", + "minLength": 1 + }, + "criterion": { + "type": "string", + "minLength": 1 + }, + "verdicts": { + "type": "object", + "additionalProperties": { + "oneOf": [ + { "type": "string", "minLength": 1 }, - "guidance": { - "type": "string", - "minLength": 1 + { + "type": "object", + "properties": { + "description": { + "type": "string", + "minLength": 1 + }, + "guidance": { + "type": "string", + "minLength": 1 + } + }, + "required": ["description"], + "additionalProperties": false } - }, - "required": ["description"], - "additionalProperties": false - } - ] - }, - "minProperties": 2 - }, - "severity": { - "type": "object", - "additionalProperties": { - "enum": ["error", "warning", "info", "ignore"] - }, - "minProperties": 1 - }, - "scope": { - "type": "array", - "items": { - "enum": ["parent", "siblings", "children", "ancestors", "descendants", "branch"] + ] + }, + "minProperties": 2 + }, + "severity": { + "type": "object", + "additionalProperties": { + "enum": ["error", "warning", "info", "ignore"] + }, + "minProperties": 1 + }, + "scope": { + "type": "array", + "items": { + "enum": ["parent", "siblings", "children", "ancestors", "descendants", "branch"] + }, + "uniqueItems": true + }, + "includeContent": { + "enum": ["nodeOnly", "fullScope", true] + } }, - "uniqueItems": true - }, - "includeContent": { - "enum": ["nodeOnly", "fullScope", true] + "additionalProperties": false } - }, - "additionalProperties": false - } - ] - }, - "type": { - "type": "string", - "minLength": 1 - }, - "global": { - "type": "boolean" - }, - "override": { - "type": "boolean" - }, - "gatesDescendants": { - "type": "boolean" - } - }, - "required": ["id", "category", "description"], - "additionalProperties": false, - "allOf": [ - { - "anyOf": [ - { - "required": ["expression"] + ] }, - { - "required": ["semantic"] - } - ] - }, - { - "if": { - "required": ["global"], - "properties": { - "global": { - "const": true - } + "type": { + "type": "string", + "minLength": 1 + }, + "global": { + "type": "boolean" + }, + "override": { + "type": "boolean" + }, + "gatesDescendants": { + "type": "boolean" + }, + "abstract": { + "type": "boolean" } }, - "then": { - "not": { + "required": ["id", "category", "description"], + "additionalProperties": false, + "allOf": [ + { "anyOf": [ { - "required": ["when"] - }, - { - "required": ["type"] + "required": ["expression"] }, { "required": ["semantic"] } ] - } - } - }, - { - "if": { - "required": ["semantic"], - "properties": { - "semantic": { - "type": "object", - "required": ["verdicts"] - } - } - }, - "then": { - "properties": { - "semantic": { - "type": "object", - "required": ["severity"] - } - } - } - }, - { - "if": { - "required": ["semantic"], - "properties": { - "semantic": { - "type": "object", + }, + { + "if": { + "required": ["global"], + "properties": { + "global": { + "const": true + } + } + }, + "then": { "not": { - "required": ["verdicts"] + "anyOf": [ + { + "required": ["when"] + }, + { + "required": ["type"] + }, + { + "required": ["semantic"] + } + ] } } - } - }, - "then": { - "properties": { - "semantic": { - "type": "object", + }, + { + "if": { + "required": ["semantic"], + "properties": { + "semantic": { + "type": "object", + "required": ["verdicts"] + } + } + }, + "then": { + "properties": { + "semantic": { + "type": "object", + "required": ["severity"] + } + } + } + }, + { + "if": { + "required": ["semantic"], "properties": { - "severity": { + "semantic": { "type": "object", "not": { - "required": ["fail"] + "required": ["verdicts"] + } + } + } + }, + "then": { + "properties": { + "semantic": { + "type": "object", + "properties": { + "severity": { + "type": "object", + "not": { + "required": ["fail"] + } + } } } } } + }, + { + "if": { + "required": ["semantic"], + "properties": { + "semantic": { + "type": "object", + "required": ["verdicts"] + } + }, + "not": { + "required": ["expression"] + } + }, + "then": { + "not": { + "required": ["severity"] + } + } } - } + ] }, { - "if": { - "required": ["semantic"], - "properties": { - "semantic": { - "type": "object", - "required": ["verdicts"] - } + "type": "object", + "properties": { + "id": { + "type": "string", + "minLength": 1 }, - "not": { - "required": ["expression"] + "category": { + "enum": ["validation", "coherence", "workflow", "best-practice"] + }, + "severity": { + "enum": ["error", "warning", "info"] + }, + "description": { + "type": "string", + "minLength": 1 + }, + "expression": { + "type": "string", + "minLength": 1 + }, + "when": { + "type": "string", + "minLength": 1 + }, + "semantic": { + "oneOf": [ + { + "const": true + }, + { + "type": "object", + "properties": { + "rubric": { + "type": "string", + "minLength": 1 + }, + "criterion": { + "type": "string", + "minLength": 1 + }, + "verdicts": { + "type": "object", + "additionalProperties": { + "oneOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "object", + "properties": { + "description": { + "type": "string", + "minLength": 1 + }, + "guidance": { + "type": "string", + "minLength": 1 + } + }, + "required": ["description"], + "additionalProperties": false + } + ] + }, + "minProperties": 2 + }, + "severity": { + "type": "object", + "additionalProperties": { + "enum": ["error", "warning", "info", "ignore"] + }, + "minProperties": 1 + }, + "scope": { + "type": "array", + "items": { + "enum": ["parent", "siblings", "children", "ancestors", "descendants", "branch"] + }, + "uniqueItems": true + }, + "includeContent": { + "enum": ["nodeOnly", "fullScope", true] + } + }, + "additionalProperties": false + } + ] + }, + "type": { + "type": "string", + "minLength": 1 + }, + "global": { + "type": "boolean" + }, + "override": { + "type": "boolean" + }, + "gatesDescendants": { + "type": "boolean" + }, + "extends": { + "type": "string", + "minLength": 1 } }, - "then": { - "not": { - "required": ["severity"] - } - } + "required": ["id", "extends"], + "additionalProperties": false } ] } diff --git a/src/judge/prompt.ts b/src/judge/prompt.ts index fcdcd4e..46e09ab 100644 --- a/src/judge/prompt.ts +++ b/src/judge/prompt.ts @@ -36,10 +36,9 @@ export function buildJudgeUserPrompt( { node, related = [], schemaDescriptions }: Pick, rule: SemanticRuleSpec, ): string { - const rubric = rule.rubric || rule.description; const lines: string[] = [ ``, - `${rubric}`, + `${rule.rubric}`, '', ...Object.entries(rule.verdicts).map( ([name, description]) => `${description}`, diff --git a/src/judge/provider/jev.ts b/src/judge/provider/jev.ts index 01163a4..4fe2959 100644 --- a/src/judge/provider/jev.ts +++ b/src/judge/provider/jev.ts @@ -56,7 +56,7 @@ export class JevJudgeProvider implements SemanticJudgeProvider { for (let start = 0; start < rules.length; start += JEV_REQUEST_PACKING_LIMIT_QUESTIONS) { const part = rules.slice(start, start + JEV_REQUEST_PACKING_LIMIT_QUESTIONS); const questions: Record = {}; - for (const rule of part) questions[rule.id] = decisionQuestion(rule, rule.rubric || rule.description); + for (const rule of part) questions[rule.id] = decisionQuestion(rule, rule.rubric); const data = await this.post( withDescriptions(nodeState(node, input.related), input.schemaDescriptions), questions, @@ -116,10 +116,7 @@ export class JevJudgeProvider implements SemanticJudgeProvider { nodes[`n${k}`] = nodeState(input.node, input.related); Object.assign(descriptions, input.schemaDescriptions); for (const rule of input.rules) { - questions[`${rule.id}__n${k}`] = decisionQuestion( - rule, - `For node \`nodes.n${k}\`: ${rule.rubric || rule.description}`, - ); + questions[`${rule.id}__n${k}`] = decisionQuestion(rule, `For node \`nodes.n${k}\`: ${rule.rubric}`); } }); const data = await this.post(withDescriptions({ nodes }, descriptions), questions); diff --git a/src/judge/provider/openai-decisions.ts b/src/judge/provider/openai-decisions.ts index d9edced..7306ee3 100644 --- a/src/judge/provider/openai-decisions.ts +++ b/src/judge/provider/openai-decisions.ts @@ -60,7 +60,7 @@ export class OpenAIDecisionsJudgeProvider implements SemanticJudgeProvider { questions: rules.map((rule) => ({ type: 'choice', name: rule.id, - instructions: rule.rubric || rule.description, + instructions: rule.rubric, choices: Object.entries(rule.verdicts).map(([value, description]) => ({ value, description })), })), }), diff --git a/src/judge/types.ts b/src/judge/types.ts index 636c77a..78a5461 100644 --- a/src/judge/types.ts +++ b/src/judge/types.ts @@ -4,7 +4,8 @@ import type { RuleCategory, SpaceNode } from '../types'; export interface SemanticRuleSpec { id: string; description: string; - rubric?: string; + /** What the model judges against: the rule's rubric (or description), then its criterion. */ + rubric: string; category: RuleCategory | string; /** Allowed verdicts, each with a description of when it applies. */ verdicts: Record; diff --git a/src/judge/verdicts.ts b/src/judge/verdicts.ts index 5c26cb6..732cc78 100644 --- a/src/judge/verdicts.ts +++ b/src/judge/verdicts.ts @@ -84,7 +84,7 @@ export function resolveSemanticRule(rule: Rule): ResolvedSemanticRule { spec: { id: rule.id, description: rule.description, - rubric: semantic.rubric, + rubric: [semantic.rubric ?? rule.description, semantic.criterion].filter(Boolean).join('\n'), category: rule.category, verdicts, }, diff --git a/src/schema/metadata-contract.ts b/src/schema/metadata-contract.ts index 5e577b4..1cb9064 100644 --- a/src/schema/metadata-contract.ts +++ b/src/schema/metadata-contract.ts @@ -71,6 +71,8 @@ const SEMANTIC_SCHEMA = { properties: { /** What the model judges the node against. Falls back to the rule's `description`. */ rubric: { type: 'string', minLength: 1 }, + /** What this rule adds to its rubric: the model reads it after the rubric. Lets a template's rubric stay generic. */ + criterion: { type: 'string', minLength: 1 }, /** * The closed set of verdicts the model may return. Each is a description (what the verdict means, for the * model) or `{ description, guidance }`, where `guidance` is shown to the user when it fires. @@ -119,65 +121,86 @@ const SEMANTIC_SCHEMA = { additionalProperties: false, } as const; -const RULE_SCHEMA = { - type: 'object', - properties: { - id: { type: 'string', minLength: 1 }, - category: { enum: ['validation', 'coherence', 'workflow', 'best-practice'] }, - /** - * Severity of a failed `expression` and of the default `fail` verdict. Defaults from the category. - */ - severity: SEVERITY_SCHEMA, - description: { type: 'string', minLength: 1 }, - /** JSONata test a node must pass. When the rule also has `semantic`, a failing node is reported without being evaluated by the model. */ - expression: { type: 'string', minLength: 1 }, - /** Optional JSONata applicability condition, evaluated per node before the rule runs. The rule applies only when it returns true. */ - when: { type: 'string', minLength: 1 }, - /** A model evaluates the node against a rubric. `{}` takes every default; `true` means the same. */ - semantic: { oneOf: [{ const: true }, SEMANTIC_SCHEMA] }, - type: { type: 'string', minLength: 1 }, - /** Run once for the whole space instead of once per node. */ - global: { type: 'boolean' }, - override: { type: 'boolean' }, - /** Whether a failure of this rule gates (suppresses) downstream rules on the node's hierarchy subtree. Defaults to true when the failure's severity is `error`. */ - gatesDescendants: { type: 'boolean' }, +const RULE_PROPERTIES = { + id: { type: 'string', minLength: 1 }, + category: { enum: ['validation', 'coherence', 'workflow', 'best-practice'] }, + /** + * Severity of a failed `expression` and of the default `fail` verdict. Defaults from the category. + */ + severity: SEVERITY_SCHEMA, + description: { type: 'string', minLength: 1 }, + /** JSONata test a node must pass. When the rule also has `semantic`, a failing node is reported without being evaluated by the model. */ + expression: { type: 'string', minLength: 1 }, + /** Optional JSONata applicability condition, evaluated per node before the rule runs. The rule applies only when it returns true. */ + when: { type: 'string', minLength: 1 }, + /** A model evaluates the node against a rubric. `{}` takes every default; `true` means the same. */ + semantic: { oneOf: [{ const: true }, SEMANTIC_SCHEMA] }, + type: { type: 'string', minLength: 1 }, + /** Run once for the whole space instead of once per node. */ + global: { type: 'boolean' }, + override: { type: 'boolean' }, + /** Whether a failure of this rule gates (suppresses) downstream rules on the node's hierarchy subtree. Defaults to true when the failure's severity is `error`. */ + gatesDescendants: { type: 'boolean' }, +} as const; + +const RULE_CONSTRAINTS = [ + // A rule has an expression, a semantic part, or both. + { anyOf: [{ required: ['expression'] }, { required: ['semantic'] }] }, + // A global rule has no node to apply to or to evaluate. + { + if: { required: ['global'], properties: { global: { const: true } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { not: { anyOf: [{ required: ['when'] }, { required: ['type'] }, { required: ['semantic'] }] } }, }, - required: ['id', 'category', 'description'], - additionalProperties: false, - allOf: [ - // A rule has an expression, a semantic part, or both. - { anyOf: [{ required: ['expression'] }, { required: ['semantic'] }] }, - // A global rule has no node to apply to or to evaluate. - { - if: { required: ['global'], properties: { global: { const: true } } }, - // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable - then: { not: { anyOf: [{ required: ['when'] }, { required: ['type'] }, { required: ['semantic'] }] } }, - }, - // Custom verdicts need a severity for each failing verdict. - { - if: { required: ['semantic'], properties: { semantic: { type: 'object', required: ['verdicts'] } } }, - // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable - then: { properties: { semantic: { type: 'object', required: ['severity'] } } }, - }, - // With the default verdicts, `fail` takes the rule's severity. - { - if: { required: ['semantic'], properties: { semantic: { type: 'object', not: { required: ['verdicts'] } } } }, - // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable - then: { - properties: { - semantic: { type: 'object', properties: { severity: { type: 'object', not: { required: ['fail'] } } } }, - }, + // Custom verdicts need a severity for each failing verdict. + { + if: { required: ['semantic'], properties: { semantic: { type: 'object', required: ['verdicts'] } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { properties: { semantic: { type: 'object', required: ['severity'] } } }, + }, + // With the default verdicts, `fail` takes the rule's severity. + { + if: { required: ['semantic'], properties: { semantic: { type: 'object', not: { required: ['verdicts'] } } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { + properties: { + semantic: { type: 'object', properties: { severity: { type: 'object', not: { required: ['fail'] } } } }, }, }, - // A rule with custom verdicts and no expression has no failure for the top-level severity to describe. + }, + // A rule with custom verdicts and no expression has no failure for the top-level severity to describe. + { + if: { + required: ['semantic'], + properties: { semantic: { type: 'object', required: ['verdicts'] } }, + not: { required: ['expression'] }, + }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { not: { required: ['severity'] } }, + }, +] as const; + +/** A rule as it runs: declared in full, or resolved from the template it extends. */ +export const RULE_SCHEMA = { + type: 'object', + properties: RULE_PROPERTIES, + required: ['id', 'category', 'description'], + additionalProperties: false, + allOf: RULE_CONSTRAINTS, +} as const; + +/** + * A rule as a schema declares it. A rule template (`abstract: true`) is a full rule that never runs. A rule with + * `extends` names a template and declares only what it changes; it is checked as a full rule once resolved. + */ +const DECLARED_RULE_SCHEMA = { + oneOf: [ + { ...RULE_SCHEMA, properties: { ...RULE_PROPERTIES, abstract: { type: 'boolean' } } }, { - if: { - required: ['semantic'], - properties: { semantic: { type: 'object', required: ['verdicts'] } }, - not: { required: ['expression'] }, - }, - // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable - then: { not: { required: ['severity'] } }, + type: 'object', + properties: { ...RULE_PROPERTIES, extends: { type: 'string', minLength: 1 } }, + required: ['id', 'extends'], + additionalProperties: false, }, ], } as const; @@ -260,7 +283,7 @@ export const METADATA_SCHEMA = { }, rules: { type: 'array', - items: RULE_SCHEMA, + items: DECLARED_RULE_SCHEMA, }, }, additionalProperties: false, @@ -289,5 +312,6 @@ export type MetadataContractRelationship = FromSchema; export type SharedEmbeddingFields = FromSchema; export type Rule = FromSchema; +export type DeclaredRule = FromSchema; export type RuleSemantic = FromSchema; export type MetadataRef = FromSchema; diff --git a/src/schema/rule-templates.ts b/src/schema/rule-templates.ts new file mode 100644 index 0000000..8bad44f --- /dev/null +++ b/src/schema/rule-templates.ts @@ -0,0 +1,46 @@ +import Ajv2020 from 'ajv/dist/2020'; +import { type DeclaredRule, RULE_SCHEMA, type Rule } from './metadata-contract'; + +const validateRule = new Ajv2020().compile(RULE_SCHEMA); + +type Semantic = Exclude; + +const semanticObject = (semantic: Rule['semantic']): Semantic | undefined => (semantic === true ? {} : semantic); + +/** + * Resolve rule templates in merged rules. A rule with `extends` takes the template's fields, then its own: top-level + * fields replace the template's, and `semantic` fields replace the template's one by one. Templates are dropped. + */ +export function resolveRuleTemplates(rules: DeclaredRule[]): Rule[] { + const templates = new Map(rules.filter((rule) => 'abstract' in rule && rule.abstract).map((rule) => [rule.id, rule])); + const resolved: Rule[] = []; + for (const rule of rules) { + if (templates.has(rule.id)) continue; + if (!('extends' in rule) || rule.extends === undefined) { + const { abstract: _, ...concrete } = rule as DeclaredRule & { abstract?: boolean }; + resolved.push(concrete as Rule); + continue; + } + const template = templates.get(rule.extends); + if (!template) { + throw new Error( + `Rule "${rule.id}" extends "${rule.extends}", but no rule template (a rule with "abstract": true) has that id.`, + ); + } + const { abstract: _, ...base } = template as Rule & { abstract?: boolean }; + const { extends: __, semantic, ...own } = rule; + const baseSemantic = semanticObject(base.semantic); + const ownSemantic = semanticObject(semantic); + const result = { + ...base, + ...own, + ...(baseSemantic || ownSemantic ? { semantic: { ...baseSemantic, ...ownSemantic } } : {}), + }; + if (!validateRule(result)) { + const errors = validateRule.errors?.map((e) => `${e.instancePath || '(root)'} ${e.message}`).join('; '); + throw new Error(`Rule "${rule.id}" is not a valid rule once it extends "${rule.extends}": ${errors}`); + } + resolved.push(result as Rule); + } + return resolved; +} diff --git a/src/schema/schema.ts b/src/schema/schema.ts index 9551894..f3cf4cf 100644 --- a/src/schema/schema.ts +++ b/src/schema/schema.ts @@ -9,6 +9,7 @@ import { shortenPluginName } from '../plugins/util'; import type { Hierarchy, HierarchyLevel, SchemaMetadata, SchemaWithMetadata } from '../types'; import { loadMetaSchemas } from './meta-schemas'; import { + type DeclaredRule, METADATA_SCHEMA, type MetadataContract, type MetadataContractHierarchy, @@ -16,6 +17,7 @@ import { type MetadataRef, type Rule, } from './metadata-contract'; +import { resolveRuleTemplates } from './rule-templates'; import { isObject, mergeVariantProperties, resolveJsonPointer } from './schema-refs'; const packageDir = import.meta.url ? dirname(fileURLToPath(import.meta.url)) : ''; @@ -504,7 +506,7 @@ function extractMetadata(schema: AnySchemaObject, schemaRefRegistry: Map = new Map(); - const mergedRules: Merged = new Map(); + const mergedRules: Merged = new Map(); const mergedRelationships: Merged = new Map(); for (const provider of metadataProviders) { @@ -569,7 +571,7 @@ function extractMetadata(schema: AnySchemaObject, schemaRefRegistry: Map [alias, value])); - const rules = values(mergedRules).map(({ override, ...rule }) => rule); + const rules = resolveRuleTemplates(values(mergedRules).map(({ override, ...rule }) => rule)); const relationships = values(mergedRelationships); // Collect valid type names from the schema's oneOf list for validation diff --git a/tests/judge/helpers.ts b/tests/judge/helpers.ts index aa57077..db6471f 100644 --- a/tests/judge/helpers.ts +++ b/tests/judge/helpers.ts @@ -29,6 +29,7 @@ export const passFailRule: SemanticRuleSpec = { export const graded: SemanticRuleSpec = { id: 'framing', description: 'Framing quality', + rubric: 'Framing quality', category: 'best-practice', verdicts: { good: 'Clear need', weak: 'Vague need', solution: 'A solution in disguise' }, }; diff --git a/tests/judge/verdicts.test.ts b/tests/judge/verdicts.test.ts index a25ddf0..63e6151 100644 --- a/tests/judge/verdicts.test.ts +++ b/tests/judge/verdicts.test.ts @@ -24,9 +24,10 @@ describe('resolveSemanticRule', () => { expect(resolveSemanticRule({ ...base, semantic: true }).severities).toEqual({ fail: 'info' }); }); - it('uses the semantic rubric, falling back to nothing so the prompt can use the description', () => { + it('uses the semantic rubric, falling back to the description, followed by the criterion', () => { expect(resolveSemanticRule({ ...base, semantic: { rubric: 'R' } }).spec.rubric).toBe('R'); - expect(resolveSemanticRule(base).spec.rubric).toBeUndefined(); + expect(resolveSemanticRule(base).spec.rubric).toBe(base.description); + expect(resolveSemanticRule({ ...base, semantic: { rubric: 'R', criterion: 'C' } }).spec.rubric).toBe('R\nC'); }); it('maps declared verdicts to severities; unmapped and ignored verdicts pass', () => { diff --git a/tests/schema/rule-templates.test.ts b/tests/schema/rule-templates.test.ts new file mode 100644 index 0000000..1fe25e5 --- /dev/null +++ b/tests/schema/rule-templates.test.ts @@ -0,0 +1,100 @@ +import { afterAll, beforeAll, describe, expect, it } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { bundledSchemasDir, loadMetadata } from '../../src/schema/schema'; + +const META = + 'https://raw.githubusercontent.com/mindsocket/structured-context/main/schemas/generated/latest/_structured_context_schema_meta.json'; + +const template = { + id: 'children-distinct', + abstract: true, + category: 'coherence', + description: 'Children are distinct', + semantic: { + rubric: 'Judge the children as a set.', + scope: ['children'], + verdicts: { distinct: 'All different', overlapping: 'Two overlap' }, + severity: { overlapping: 'warning' }, + }, +}; + +describe('rule templates', () => { + let dir: string; + beforeAll(() => { + dir = mkdtempSync(join(tmpdir(), 'sctx-rule-templates-')); + }); + afterAll(() => rmSync(dir, { recursive: true, force: true })); + + const load = (name: string, rules: object[]) => { + const path = join(dir, `${name}.json`); + writeFileSync( + path, + JSON.stringify({ $schema: META, $id: `sctx://rule-templates.${name}`, $metadata: { rules }, type: 'object' }), + ); + return loadMetadata(path); + }; + + it('resolves a rule that extends a template, and drops the template', () => { + const { rules } = load('resolve', [ + template, + { + id: 'goals-distinct', + extends: 'children-distinct', + description: 'Goals are distinct', + semantic: { criterion: 'Judge the goals.', severity: { overlapping: 'info' } }, + }, + ]); + expect(rules as unknown).toEqual([ + { + id: 'goals-distinct', + category: 'coherence', + description: 'Goals are distinct', + semantic: { ...template.semantic, criterion: 'Judge the goals.', severity: { overlapping: 'info' } }, + }, + ]); + }); + + it('uses one template for several rules', () => { + const { rules } = load('reuse', [ + template, + { id: 'a', extends: 'children-distinct' }, + { id: 'b', extends: 'children-distinct', category: 'best-practice' }, + ]); + expect(rules?.map((rule) => [rule.id, rule.category])).toEqual([ + ['a', 'coherence'], + ['b', 'best-practice'], + ]); + }); + + it('fails when the template is unknown or is not abstract', () => { + expect(() => load('unknown', [{ id: 'a', extends: 'nope' }])).toThrow( + 'Rule "a" extends "nope", but no rule template (a rule with "abstract": true) has that id.', + ); + const { abstract: _, ...concrete } = template; + expect(() => load('concrete', [concrete, { id: 'a', extends: 'children-distinct' }])).toThrow('no rule template'); + }); + + it('checks the resolved rule as a full rule', () => { + expect(() => load('invalid', [template, { id: 'a', extends: 'children-distinct', severity: 'warning' }])).toThrow( + 'Rule "a" is not a valid rule once it extends "children-distinct"', + ); + }); + + it('rejects a template that extends another template', () => { + expect(() => load('nested', [template, { ...template, id: 'b', extends: 'children-distinct' }])).toThrow( + 'Invalid $metadata', + ); + }); + + it('gives bundled rules their template rubric and their own criterion', () => { + const { rules } = loadMetadata(join(bundledSchemasDir, 'strict_ost.json')); + const ids = rules?.map((rule) => rule.id) ?? []; + expect(ids).not.toContain('children-distinct'); + const rule = rules?.find((r) => r.id === 'opportunities-distinct'); + const semantic = rule?.semantic === true ? {} : rule?.semantic; + expect(semantic?.rubric).toStartWith("Judge the item's children as a set."); + expect(semantic?.criterion).toStartWith('Judge the child opportunities.'); + }); +});