feat(schemas): bundled semantic rules with evals

docs/schemas.md

@@ -33,6 +33,8 @@ Resolution order: space `schema` > global `schema`. Schema resolution fails if n
 
 ## Bundled schemas
 
+Each bundled schema has [semantic rules](semantic-rules.md). `sctx schemas show <schema>` lists them.
+
 ### `strategy_general.json`
 
 Flexible planning schema spanning strategy + OST-like flow.
@@ -57,9 +59,11 @@ Main types:
 
 This schema reuses shared structural defs from partials via `$ref` and imports its hierarchy and rules from `_ost_strict` via `$metadata.imports`.
 
+Its type descriptions and semantic rules follow Teresa Torres' *Continuous Discovery Habits* (2021).
+
 ### `knowledge_wiki.json`
 
-Flat, LLM-maintained knowledge base (after Karpathy's LLM Wiki pattern). No hierarchy or required parent links; wikilinks between any pages are valid. Validates structure and provenance, not meaning.
+Flat, LLM-maintained knowledge base (after Karpathy's LLM Wiki pattern). No hierarchy or required parent links; wikilinks between any pages are valid. Validates structure and provenance.
 
 Main types:
 - `source`

docs/semantic-rules.md

@@ -146,6 +146,16 @@ Scope sends node data only, never the verdicts of other rules, and [gating](rule
 
 A global rule cannot have a `semantic` part, so it has no scope.
 
+The bundled schemas use scope in three ways. We recommend them as models for your own rules:
+
+| Check | Attach the rule to | Scope | Bundled example |
+|---|---|---|---|
+| The children are distinct, or cover the node | The parent, with a `when` that needs two or more children | `children` | `opportunities-distinct`, `goals-cover-parent` |
+| The node fits its parent | The child | `parent` | `opportunity-refines-parent`, `solution-addresses-opportunity` |
+| The node serves the intent of the levels above | The child | `ancestors` | `solution-serves-outcome`, `opportunity-serves-ancestors` |
+
+The rules are in the bundled partials, such as `schemas/_ost_strict.json`.
+
 ## Input size and trimming
 
 The input for one evaluated node is capped at about 32,000 estimated tokens. Over the cap, structured-context cuts node bodies, longest related bodies first, and marks each cut with `[RULE_EVALUATION_CONTENT_TRIMMED n chars]`. If the input still does not fit, the rule reports a semantic evaluation error. A run that trims content says so in its summary.

evals/semantic/assumption-category-fits.json5

@@ -0,0 +1,28 @@
+{
+  space: 'ost-trees',
+  expected: {
+    'assumption-category-fits': {
+      'Test 01: Teams want to see who edited a page.md': 'pass',
+      'Test 02: Storing every revision costs less than $0.10 per seat a month.md': 'pass',
+      'Test 03: We can diff rich-text documents in under 200 ms.md': 'pass',
+      'Test 04: Users can find the history panel without help.md': 'pass',
+      "Test 05: Showing who edited what won't be used to monitor employees.md": 'pass',
+      'Test 06: Our sync engine can merge offline edits without conflicts.md': 'pass',
+      'Test 07: Admins understand the retention settings on first read.md': 'pass',
+      'Test 08: Users can find the restore button without help.md': 'fail',
+      // Cost is viability, not whether the team can build it.
+      'Test 09: Keeping 90 days of revisions costs under $0.10 per seat a month.md': 'fail',
+      'Test 10: Teams want to restore old versions.md': 'fail',
+      'Test 11: Sales can sell version history as an enterprise add-on.md': 'fail',
+      'Test 12: We can build live cursors on our current websocket stack.md': 'fail',
+      "Test 13: Users will notice the 'restored' banner and understand what changed.md": 'pass',
+      // Legal constraints on the business are viability.
+      'Test 14: Legal will approve keeping deleted content for 90 days.md': 'pass',
+      'Test 15: Teams will keep using history after the first month.md': 'pass',
+      "Test 16: Managers won't use edit history to rank their staff.md": 'pass',
+      // Mentions a survey, but the assumption is ethical.
+      "Test 17: Managers won't use edit history to rank their staff, judged by a survey.md": 'fail',
+      'Test 18: Users will pay for longer history.md': 'fail',
+    },
+  },
+}

evals/semantic/concept-or-entity.json5

@@ -0,0 +1,31 @@
+{
+  space: 'knowledge-wiki',
+  expected: {
+    'concept-or-entity': {
+      'Affordance.md': 'pass',
+      'Retrieval-augmented generation.md': 'pass',
+      // A named method is still a concept.
+      'Opportunity solution tree.md': 'pass',
+      'Jobs to be done.md': 'pass',
+      'Anthropic.md': 'pass',
+      'GPT-4.md': 'pass',
+      // One specific book.
+      'Continuous Discovery Habits.md': 'pass',
+      'Teresa Torres.md': 'pass',
+      'Notion.md': 'fail',
+      'Don Norman.md': 'fail',
+      'Technical debt.md': 'fail',
+      // A generic design, despite the proper-sounding name.
+      'Transformer architecture.md': 'fail',
+      'Kanban.md': 'pass',
+      // One company's system, though it inspired a concept.
+      'Toyota Production System.md': 'pass',
+      'Claude.md': 'fail',
+      'Large language model.md': 'fail',
+      // One specific document.
+      'Agile Manifesto.md': 'fail',
+      // A method, though it has a proper name.
+      'Scrum.md': 'fail',
+    },
+  },
+}

evals/semantic/fixtures.json5

@@ -2,5 +2,11 @@
   spaces: [
     // A small product's opportunity solution tree.
     { name: 'strict-ost', path: 'fixtures/strict-ost.jsonl', schema: 'strict_ost.json' },
+    // Trees for the strict_ost rules. A rule that judges a node's children gets one small tree per scenario.
+    { name: 'ost-trees', path: 'fixtures/ost-trees.jsonl', schema: 'strict_ost.json' },
+    // Trees for the strategy_general rules, built the same way.
+    { name: 'strategy-trees', path: 'fixtures/strategy-trees.jsonl', schema: 'strategy_general.json' },
+    { name: 'knowledge-wiki', path: 'fixtures/knowledge-wiki.jsonl', schema: 'knowledge_wiki.json' },
+    { name: 'okf', path: 'fixtures/okf.jsonl', schema: 'okf.json' },
   ],
 }