[mindsocket]

Requested by Rog · project thread

Slice 2 of reusable semantic rules: a way to measure how well each provider judges semantic rules, before slice 3 adds primitive rules and type descriptions to the bundled schemas. It covers core's bundled schemas, plugins and configured spaces.

What it adds

• Terms. A case is a node, a rule and the verdict the node should get. An eval is a .json5 file of cases on one space. A fixture is a space whose nodes live in the suite. A suite is a directory of evals and fixtures.
• Layout.

  evals/                               harness: load, judge, score, run
  evals/semantic/                      core's suite
    opportunity-customer-voice.json5   { space: 'strict-ost', expected: { <rule id>: { <node id>: verdict } } }
    fixtures.json5                     fixtures as space configs: name, path, schema, plugins
    fixtures/strict-ost.jsonl          parsed nodes, as `sctx dump <space> | jq -c '.nodes[]'` prints them
  

  An eval's space names one of the suite's fixtures, or else a configured space. Fixtures can share a schema, and evals can share a fixture.
• Suites (--from, default core): core's evals/semantic/, a plugin's evals/semantic/ (found as the config finds plugins), or a path.
• Runner: bun run eval:semantic --provider <name>… [--from …] [--eval …] [--runs 3] [--per-node] [--check] [--json]. It reports agreement, stability, mean p(expected), reserved verdicts and errors per eval, provider and rule. It lists each miss, and the agreeing cases with the lowest p(expected). --per-node hides evaluateNodes, to test Jev packing.
• Checks (--check, and before every paid run): a label on a node the rule doesn't judge, an unknown rule or verdict, a label that accepts every verdict, and a labelled node that fails validation. Unlabelled nodes are allowed, as related nodes for scope rules. bun run test checks core's evals.

Design

The harness runs the real pipeline: validateNodes with only the labelled rules and includeGated. It wraps the provider so that only labelled nodes are judged, while the rest still reach the model as related nodes. It needs no change to src/ and adds no exports. Fixtures are read without a parse plugin; a shared core reader can replace that when core #112 gives it a second caller.

Alternatives considered:

• Call judgeNodes directly. It skips when, expressions and the rule filter.
• Labels in node frontmatter. Frontmatter goes to the judge, so the label would leak into the input.
• Fixtures in the space graph data format. Every node would need its resolved edges written by hand.
• A sctx eval command. It adds public surface; plugin authors need a core checkout until then.

Not run live in the cloud container. A live run on core's eval scored 100% for openrouter-jev, jev --per-node and openai, so slice 3 needs harder cases.

🤖 Generated with Claude Code

https://claude.ai/code/session_019XphpdrUMvqf8fuj8uG7JZ

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

[claude]

Code review: correction

My earlier "no issues" comment was wrong. bun run build fails on this PR.

tsconfig.json now includes evals/**/*, and tsconfig.build.json extends it. Its exclude list doesn't mention evals, so the build compiles evals/semantic/*.ts against rootDir: "src" and stops with:

error TS6059: File '.../evals/semantic/run.ts' is not under 'rootDir '.../src'.
  Matched by include pattern 'evals/**/*' in 'tsconfig.build.json'

The fix is to add "evals" to the exclude array in tsconfig.build.json. CLAUDE.md also says to run bun run test:package after changing tsconfig*.json. I haven't run that, so the build is the only check I've made.

I found this from the repo's Stop hook output after the review, not from the review itself.