Semantic eval harness for bundled schemas

AGENTS.md

@@ -45,6 +45,7 @@ There are several places that need reviewing and updating with any new feature o
 |---|---|
 | CLI command, flag or output | `docs/*` for the command; `plugin/skills/structured-context/references/commands.md`; the command list in `plugin/skills/structured-context/SKILL.md`; `plugin/commands/*` |
 | Rule, schema or `$metadata` shape | `docs/schemas.md`, `docs/rules.md`, `docs/semantic-rules.md`; `references/schema-authoring.md`, `references/schema-design.md`; bundled `schemas/` |
+| Bundled semantic rule | its eval in `evals/semantic/` |
 | User-visible error or warning message | the troubleshooting table in `SKILL.md` |
 | Config key or space config | `docs/config.md`; the config example in `references/commands.md` |
 | `validate --json` output | `plugin/scripts/state.ts` (the hooks parse it), `scripts/mod-contract.ts` and `bun run test:hook` |
@@ -99,6 +100,7 @@ For most development only the main unit tests need re-running regularly.
 - `bun run test:package` — builds and packs the package, installs the tarball, and runs the installed CLI. Runs in CI and before every version bump; run it after changing the build, `tsconfig*.json` or `files`.
 - `bun run test:judge` — live judge tests (`tests/judge/live.test.ts`): one real provider call per provider whose API key is set. Costs money; use sparingly when changing judges or semantic rule behaviour.
 - `bun run test:models` — check that every bundled provider's configured model exists (no API keys)
+- `bun run eval:semantic --provider <name>` — semantic evals (`evals/`): how often a provider's verdicts match labelled cases. Costs money (`--check` doesn't). Run it when changing a bundled semantic rule, the schema descriptions it draws on, or a judge.
 
 ## Debugging
 

evals/README.md

@@ -0,0 +1,85 @@
+# Semantic evals
+
+Semantic evals measure how well model providers judge semantic rules. A case is a node, a rule, and the verdict the node should get. An eval holds the cases on the nodes of one space. That space is a fixture, which keeps its nodes in the suite, or a configured space.
+
+## Run the evals
+
+The run calls real providers, so it costs money. Name each provider to run. Config `modelProviders` entries work as well as bundled providers.
+
+```bash
+bun run eval:semantic --provider openrouter-jev --provider anthropic
+```
+
+From outside this repo, run `bun run <path to structured-context>/evals/run.ts` with the same flags.
+
+| Flag | Purpose |
+|---|---|
+| `--provider <name>` | The provider to judge with. Repeat it, or give a comma-separated list, to compare providers |
+| `--from <suite>` | The suite to run. Repeatable. Default: `core`. See [Suites](#suites) |
+| `--eval <name>` | Run only this eval, as `core/opportunity-customer-voice` or `opportunity-customer-voice`. Repeatable |
+| `--runs <n>` | Judge each eval this many times, to measure stability. Default: 3 |
+| `--per-node` | Send one request per node, even to a provider that packs several nodes into one request |
+| `--check` | Check the evals and stop, with no provider calls |
+| `--json` | Print every judgement and score as JSON instead of the report |
+
+The report has one row per eval, provider and rule:
+
+| Column | Meaning |
+|---|---|
+| Agreement | Share of judgements, over every case and run, that match the label |
+| Stable | Share of cases that got the same verdict on every run |
+| p(expected) | Mean probability the provider gave the labelled verdicts. Only decision models report it |
+| Reserved | Judgements with a reserved verdict, such as `missing-context` |
+| Errors | Cases that got an evaluation error, or no judgement, on some run |
+
+Below the table, the report lists each miss, and the agreeing cases with the lowest p(expected). Those are the cases nearest to a miss.
+
+Each run validates through the same pipeline as `sctx validate --semantic`, with only the labelled rules, and evaluates gated nodes too. Only labelled nodes are judged. Other nodes still reach the model as related nodes, when a rule's `scope` includes them.
+
+## Suites
+
+A suite is a directory of evals and the fixtures they use:
+
+```
+evals/semantic/
+  opportunity-customer-voice.json5   an eval
+  fixtures.json5                     the fixtures, as space configs
+  fixtures/strict-ost.jsonl          a fixture's nodes
+```
+
+| `--from` | Directory | Schema names resolve from |
+|---|---|---|
+| `core` | `evals/semantic/` in this repo | the bundled schemas |
+| `<plugin name>` | `evals/semantic/` in the plugin, found as the config finds plugins | the config that finds the plugin |
+| a path | that directory | that directory |
+
+## Add an eval
+
+An eval is a `.json5` file in the suite. It names a `space`: one of the suite's fixtures, or else a space in the config. `expected` maps a rule id to node ids and their verdicts:
+
+```json5
+{
+  space: 'strict-ost',
+  expected: {
+    'opportunity-customer-voice': {
+      'Build a Slack integration.md': 'fail',
+      // A solution in disguise.
+      'Add a progress bar to workspace setup.md': 'fail',
+    },
+  },
+}
+```
+
+A list of verdicts accepts any of them. Write a comment on a case whose label is not obvious. We recommend one eval per rule, so each rule's cases are easy to find and to run alone.
+
+## Add a fixture
+
+A fixture is a space whose nodes are in a JSONL file in the suite. `fixtures.json5` lists the fixtures as a config lists spaces: a `name`, the `path` to the nodes file, a `schema`, and optional `plugins`. Several fixtures can share a schema, and several evals can share a fixture.
+
+The nodes file holds one parsed node per line, as `sctx dump` prints them. To make one from markdown notes, configure them as a space and run `sctx dump <space> | jq -c '.nodes[]'`.
+
+For a rule whose `scope` sends related nodes, such as siblings, give each scenario its own small tree, so that one case's related nodes do not change another's.
+
+## Checks
+
+`--check` finds the problems that would make scores meaningless: a label on a node the rule does not judge, an unknown rule or verdict, a label that accepts every verdict, and a labelled node that fails validation. Every run checks first. `bun run test` checks core's evals.

evals/judge.ts

@@ -0,0 +1,98 @@
+import { FakeJudgeProvider } from '../src/judge/provider/fake';
+import type { SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult } from '../src/judge/types';
+import { resolveSemanticRule } from '../src/judge/verdicts';
+import { validateNodes } from '../src/validate';
+import type { Eval } from './load';
+
+/** One verdict (or error) the judge gave for a node and rule. */
+export interface Judgement {
+  nodeId: string;
+  ruleId: string;
+  result: SemanticJudgeResult;
+}
+
+/** Validate with only the eval's labelled rules, judging gated nodes too, as `sctx validate --semantic` would. */
+const validate = async (evaluation: Eval, provider: SemanticJudgeProvider) => {
+  const read = await evaluation.read();
+  return validateNodes(evaluation.context, read.nodes, read, {
+    includeGated: true,
+    ruleFilter: { id: Object.keys(evaluation.expected) },
+    judge: { provider },
+  });
+};
+
+/**
+ * Problems that would make an eval's scores meaningless, found before any paid call. A labelled node's
+ * validation errors count, because its rule input would not match a real space.
+ */
+export async function checkEval(evaluation: Eval): Promise<string[]> {
+  const problems: string[] = [];
+  const judge = new FakeJudgeProvider();
+  const result = await validate(evaluation, judge);
+  const judged = new Set(judge.callHistory.flatMap((input) => input.rules.map((r) => `${r.id}\0${input.node.id}`)));
+  const labelled = new Set(Object.values(evaluation.expected).flatMap(Object.keys));
+  for (const issue of result.issues) {
+    if (issue.severity === 'error' && issue.nodeId && labelled.has(issue.nodeId)) {
+      problems.push(`${issue.nodeId}: ${issue.message}`);
+    }
+  }
+  const rules = evaluation.context.schema.metadata.rules ?? [];
+  for (const [ruleId, labels] of Object.entries(evaluation.expected)) {
+    const rule = rules.find((r) => r.id === ruleId);
+    if (!rule?.semantic) {
+      problems.push(`rule "${ruleId}": not a semantic rule of the schema`);
+      continue;
+    }
+    const verdicts = Object.keys(resolveSemanticRule(rule).spec.verdicts);
+    for (const [nodeId, expected] of Object.entries(labels)) {
+      const at = `rule "${ruleId}", node "${nodeId}"`;
+      if (!judged.has(`${ruleId}\0${nodeId}`)) problems.push(`${at}: not judged (unknown node, or the rule skips it)`);
+      const unknown = expected.filter((v) => !verdicts.includes(v));
+      if (unknown.length > 0) problems.push(`${at}: ${unknown.join(', ')} not a verdict (${verdicts.join(', ')})`);
+      else if (verdicts.every((v) => expected.includes(v))) problems.push(`${at}: accepts every verdict`);
+    }
+  }
+  return problems.map((p) => `${evaluation.name}: ${p}`);
+}
+
+/**
+ * Judge the eval's labelled cases, recording every result the provider returns. Unlabelled nodes still reach
+ * the input as related nodes but are not judged themselves. `perNode` hides `evaluateNodes`, so a provider that
+ * packs nodes into one request gets one request per node instead.
+ */
+export async function judgeEval(
+  evaluation: Eval,
+  provider: SemanticJudgeProvider,
+  options: { perNode?: boolean } = {},
+): Promise<Judgement[]> {
+  const judgements: Judgement[] = [];
+  const labelledOnly = (input: SemanticJudgeInput): SemanticJudgeInput | undefined => {
+    const rules = input.rules.filter((rule) => evaluation.expected[rule.id]?.[input.node.id] !== undefined);
+    return rules.length > 0 ? { ...input, rules } : undefined;
+  };
+  const record = (input: SemanticJudgeInput, results: Record<string, SemanticJudgeResult>) => {
+    for (const rule of input.rules) {
+      judgements.push({ nodeId: input.node.id, ruleId: rule.id, result: results[rule.id] ?? { error: 'no result' } });
+    }
+    return results;
+  };
+  const recording: SemanticJudgeProvider = {
+    name: provider.name,
+    async evaluate(input) {
+      const sent = labelledOnly(input);
+      return sent ? record(sent, await provider.evaluate(sent)) : {};
+    },
+  };
+  const { evaluateNodes } = provider;
+  if (evaluateNodes && !options.perNode) {
+    recording.evaluateNodes = async (inputs) => {
+      const sent = inputs.map(labelledOnly);
+      const kept = sent.filter((input) => input !== undefined);
+      const results = kept.length > 0 ? await evaluateNodes.call(provider, kept) : [];
+      let k = 0;
+      return sent.map((input) => (input ? record(input, results[k++] ?? {}) : {}));
+    };
+  }
+  await validate(evaluation, recording);
+  return judgements;
+}

evals/load.ts

@@ -0,0 +1,118 @@
+import { existsSync, readdirSync, readFileSync } from 'node:fs';
+import { basename, join, resolve } from 'node:path';
+import JSON5 from 'json5';
+import type { Config, SpaceConfig } from '../src/config';
+import { resolvePluginDir } from '../src/plugins/util';
+import { readSpace } from '../src/read/read-space';
+import { resolveGraphEdges } from '../src/read/resolve-graph-edges';
+import { createSpaceContext } from '../src/space-context';
+import type { BaseNode, SpaceContext } from '../src/types';
+
+/** Where core and each plugin keep their semantic suite. */
+export const SUITE_SUBDIR = join('evals', 'semantic');
+
+/** The suite's fixtures: spaces whose nodes are in a JSONL file. */
+const FIXTURES_FILE = 'fixtures.json5';
+
+const CORE_SUITE_DIR = join(import.meta.dir, 'semantic');
+
+/** A directory of evals and their fixtures, and the directory schema names resolve from. */
+export interface Suite {
+  name: string;
+  dir: string;
+  configDir: string;
+}
+
+/** Rule id → node id → the verdicts that count as agreeing. */
+export type Expected = Record<string, Record<string, string[]>>;
+
+/** Labelled cases on the nodes of one space: a fixture, or a configured space. */
+export interface Eval {
+  /** `<suite>/<file name>`, e.g. `core/opportunity-customer-voice`. */
+  name: string;
+  context: SpaceContext;
+  expected: Expected;
+  read: () => Promise<ReturnType<typeof resolveGraphEdges>>;
+}
+
+/** `core`, a plugin name, or a path to a suite directory. */
+export function resolveSuite(source: string, configDirs: string[]): Suite {
+  if (source === 'core') return { name: 'core', dir: CORE_SUITE_DIR, configDir: CORE_SUITE_DIR };
+  if (source.includes('/') || source.includes('\\') || source.startsWith('.')) {
+    const dir = resolve(source);
+    if (!existsSync(dir)) throw new Error(`Suite "${source}": no directory at ${dir}`);
+    return { name: basename(dir), dir, configDir: dir };
+  }
+  for (const configDir of configDirs) {
+    const pluginDir = resolvePluginDir(source, configDir);
+    if (!pluginDir) continue;
+    const dir = join(pluginDir, SUITE_SUBDIR);
+    if (!existsSync(dir)) throw new Error(`Plugin "${source}" has no ${SUITE_SUBDIR}/ directory (${pluginDir})`);
+    return { name: source, dir, configDir };
+  }
+  throw new Error(`Suite "${source}" is not core, a path, or a plugin found from the config.`);
+}
+
+type EvalFile = { space: string; expected: Record<string, Record<string, string | string[]>> };
+type FixturesFile = { spaces: Pick<SpaceConfig, 'name' | 'path' | 'schema' | 'plugins'>[] };
+type Space = Pick<Eval, 'context' | 'read'>;
+
+/**
+ * Every `*.json5` at the top of each suite, except `fixtures.json5`, is an eval. An eval's `space` names one of
+ * the suite's fixtures, or else a space in `config`.
+ */
+export function loadEvals(suites: Suite[], config: Config): Eval[] {
+  return suites.flatMap((suite) => {
+    const fixtures = loadFixtures(suite);
+    return readdirSync(suite.dir)
+      .filter((file) => file.endsWith('.json5') && file !== FIXTURES_FILE)
+      .sort()
+      .map((file) => {
+        const name = `${suite.name}/${basename(file, '.json5')}`;
+        try {
+          const { space, expected: labels } = JSON5.parse(readFileSync(join(suite.dir, file), 'utf8')) as EvalFile;
+          if (typeof space !== 'string') throw new Error('needs a "space"');
+          const expected: Expected = {};
+          for (const [ruleId, verdicts] of Object.entries(labels)) {
+            expected[ruleId] = Object.fromEntries(Object.entries(verdicts).map(([id, v]) => [id, [v].flat()]));
+          }
+          return { name, expected, ...(fixtures.get(space) ?? configuredSpace(space, config)) };
+        } catch (err) {
+          throw new Error(`Eval ${name}: ${err instanceof Error ? err.message : String(err)}`);
+        }
+      });
+  });
+}
+
+function configuredSpace(name: string, config: Config): Space {
+  const context = createSpaceContext(name, config);
+  return { context, read: () => readSpace(context) };
+}
+
+/** A fixture's nodes file holds one parsed node per line, as `sctx dump` prints them. Resolved fields are dropped, since resolution runs again. */
+function loadFixtures(suite: Suite): Map<string, Space> {
+  const path = join(suite.dir, FIXTURES_FILE);
+  if (!existsSync(path)) return new Map();
+  const { spaces } = JSON5.parse(readFileSync(path, 'utf8')) as FixturesFile;
+  const config: Config = { spaces: spaces.map((s) => ({ ...s, path: resolve(suite.dir, s.path) })) };
+  return new Map(
+    config.spaces.map(({ name, path: nodesPath }) => {
+      const context = createSpaceContext(name, config, { configDir: suite.configDir });
+      const nodes = readNodes(nodesPath);
+      return [name, { context, read: async () => resolveGraphEdges(nodes, context.schema.metadata) }];
+    }),
+  );
+}
+
+function readNodes(path: string): BaseNode[] {
+  return readFileSync(path, 'utf8')
+    .split('\n')
+    .flatMap((line, i) => {
+      if (line.trim() === '') return [];
+      const { id, title, type, schemaData, linkTargets, content, contentLinks } = JSON.parse(line) as BaseNode;
+      if (!id || typeof title !== 'string' || typeof type !== 'string' || !schemaData || !linkTargets) {
+        throw new Error(`${path}:${i + 1}: not a parsed node (needs id, title, type, schemaData and linkTargets)`);
+      }
+      return [{ id, title, type, schemaData, linkTargets, content, contentLinks }];
+    });
+}