diff --git a/AGENTS.md b/AGENTS.md index 6f02776..ad2aac8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -45,6 +45,7 @@ There are several places that need reviewing and updating with any new feature o |---|---| | CLI command, flag or output | `docs/*` for the command; `plugin/skills/structured-context/references/commands.md`; the command list in `plugin/skills/structured-context/SKILL.md`; `plugin/commands/*` | | Rule, schema or `$metadata` shape | `docs/schemas.md`, `docs/rules.md`, `docs/semantic-rules.md`; `references/schema-authoring.md`, `references/schema-design.md`; bundled `schemas/` | +| Bundled semantic rule | its eval in `evals/semantic/` | | User-visible error or warning message | the troubleshooting table in `SKILL.md` | | Config key or space config | `docs/config.md`; the config example in `references/commands.md` | | `validate --json` output | `plugin/scripts/state.ts` (the hooks parse it), `scripts/mod-contract.ts` and `bun run test:hook` | @@ -99,6 +100,7 @@ For most development only the main unit tests need re-running regularly. - `bun run test:package` — builds and packs the package, installs the tarball, and runs the installed CLI. Runs in CI and before every version bump; run it after changing the build, `tsconfig*.json` or `files`. - `bun run test:judge` — live judge tests (`tests/judge/live.test.ts`): one real provider call per provider whose API key is set. Costs money; use sparingly when changing judges or semantic rule behaviour. - `bun run test:models` — check that every bundled provider's configured model exists (no API keys) +- `bun run eval:semantic --provider ` — semantic evals (`evals/`): how often a provider's verdicts match labelled cases. Costs money (`--check` doesn't). Run it when changing a bundled semantic rule, the schema descriptions it draws on, or a judge. ## Debugging diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..a85bca1 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,85 @@ +# Semantic evals + +Semantic evals measure how well model providers judge semantic rules. A case is a node, a rule, and the verdict the node should get. An eval holds the cases on the nodes of one space. That space is a fixture, which keeps its nodes in the suite, or a configured space. + +## Run the evals + +The run calls real providers, so it costs money. Name each provider to run. Config `modelProviders` entries work as well as bundled providers. + +```bash +bun run eval:semantic --provider openrouter-jev --provider anthropic +``` + +From outside this repo, run `bun run /evals/run.ts` with the same flags. + +| Flag | Purpose | +|---|---| +| `--provider ` | The provider to judge with. Repeat it, or give a comma-separated list, to compare providers | +| `--from ` | The suite to run. Repeatable. Default: `core`. See [Suites](#suites) | +| `--eval ` | Run only this eval, as `core/opportunity-customer-voice` or `opportunity-customer-voice`. Repeatable | +| `--runs ` | Judge each eval this many times, to measure stability. Default: 3 | +| `--per-node` | Send one request per node, even to a provider that packs several nodes into one request | +| `--check` | Check the evals and stop, with no provider calls | +| `--json` | Print every judgement and score as JSON instead of the report | + +The report has one row per eval, provider and rule: + +| Column | Meaning | +|---|---| +| Agreement | Share of judgements, over every case and run, that match the label | +| Stable | Share of cases that got the same verdict on every run | +| p(expected) | Mean probability the provider gave the labelled verdicts. Only decision models report it | +| Reserved | Judgements with a reserved verdict, such as `missing-context` | +| Errors | Cases that got an evaluation error, or no judgement, on some run | + +Below the table, the report lists each miss, and the agreeing cases with the lowest p(expected). Those are the cases nearest to a miss. + +Each run validates through the same pipeline as `sctx validate --semantic`, with only the labelled rules, and evaluates gated nodes too. Only labelled nodes are judged. Other nodes still reach the model as related nodes, when a rule's `scope` includes them. + +## Suites + +A suite is a directory of evals and the fixtures they use: + +``` +evals/semantic/ + opportunity-customer-voice.json5 an eval + fixtures.json5 the fixtures, as space configs + fixtures/strict-ost.jsonl a fixture's nodes +``` + +| `--from` | Directory | Schema names resolve from | +|---|---|---| +| `core` | `evals/semantic/` in this repo | the bundled schemas | +| `` | `evals/semantic/` in the plugin, found as the config finds plugins | the config that finds the plugin | +| a path | that directory | that directory | + +## Add an eval + +An eval is a `.json5` file in the suite. It names a `space`: one of the suite's fixtures, or else a space in the config. `expected` maps a rule id to node ids and their verdicts: + +```json5 +{ + space: 'strict-ost', + expected: { + 'opportunity-customer-voice': { + 'Build a Slack integration.md': 'fail', + // A solution in disguise. + 'Add a progress bar to workspace setup.md': 'fail', + }, + }, +} +``` + +A list of verdicts accepts any of them. Write a comment on a case whose label is not obvious. We recommend one eval per rule, so each rule's cases are easy to find and to run alone. + +## Add a fixture + +A fixture is a space whose nodes are in a JSONL file in the suite. `fixtures.json5` lists the fixtures as a config lists spaces: a `name`, the `path` to the nodes file, a `schema`, and optional `plugins`. Several fixtures can share a schema, and several evals can share a fixture. + +The nodes file holds one parsed node per line, as `sctx dump` prints them. To make one from markdown notes, configure them as a space and run `sctx dump | jq -c '.nodes[]'`. + +For a rule whose `scope` sends related nodes, such as siblings, give each scenario its own small tree, so that one case's related nodes do not change another's. + +## Checks + +`--check` finds the problems that would make scores meaningless: a label on a node the rule does not judge, an unknown rule or verdict, a label that accepts every verdict, and a labelled node that fails validation. Every run checks first. `bun run test` checks core's evals. diff --git a/evals/judge.ts b/evals/judge.ts new file mode 100644 index 0000000..0f299f0 --- /dev/null +++ b/evals/judge.ts @@ -0,0 +1,98 @@ +import { FakeJudgeProvider } from '../src/judge/provider/fake'; +import type { SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult } from '../src/judge/types'; +import { resolveSemanticRule } from '../src/judge/verdicts'; +import { validateNodes } from '../src/validate'; +import type { Eval } from './load'; + +/** One verdict (or error) the judge gave for a node and rule. */ +export interface Judgement { + nodeId: string; + ruleId: string; + result: SemanticJudgeResult; +} + +/** Validate with only the eval's labelled rules, judging gated nodes too, as `sctx validate --semantic` would. */ +const validate = async (evaluation: Eval, provider: SemanticJudgeProvider) => { + const read = await evaluation.read(); + return validateNodes(evaluation.context, read.nodes, read, { + includeGated: true, + ruleFilter: { id: Object.keys(evaluation.expected) }, + judge: { provider }, + }); +}; + +/** + * Problems that would make an eval's scores meaningless, found before any paid call. A labelled node's + * validation errors count, because its rule input would not match a real space. + */ +export async function checkEval(evaluation: Eval): Promise { + const problems: string[] = []; + const judge = new FakeJudgeProvider(); + const result = await validate(evaluation, judge); + const judged = new Set(judge.callHistory.flatMap((input) => input.rules.map((r) => `${r.id}\0${input.node.id}`))); + const labelled = new Set(Object.values(evaluation.expected).flatMap(Object.keys)); + for (const issue of result.issues) { + if (issue.severity === 'error' && issue.nodeId && labelled.has(issue.nodeId)) { + problems.push(`${issue.nodeId}: ${issue.message}`); + } + } + const rules = evaluation.context.schema.metadata.rules ?? []; + for (const [ruleId, labels] of Object.entries(evaluation.expected)) { + const rule = rules.find((r) => r.id === ruleId); + if (!rule?.semantic) { + problems.push(`rule "${ruleId}": not a semantic rule of the schema`); + continue; + } + const verdicts = Object.keys(resolveSemanticRule(rule).spec.verdicts); + for (const [nodeId, expected] of Object.entries(labels)) { + const at = `rule "${ruleId}", node "${nodeId}"`; + if (!judged.has(`${ruleId}\0${nodeId}`)) problems.push(`${at}: not judged (unknown node, or the rule skips it)`); + const unknown = expected.filter((v) => !verdicts.includes(v)); + if (unknown.length > 0) problems.push(`${at}: ${unknown.join(', ')} not a verdict (${verdicts.join(', ')})`); + else if (verdicts.every((v) => expected.includes(v))) problems.push(`${at}: accepts every verdict`); + } + } + return problems.map((p) => `${evaluation.name}: ${p}`); +} + +/** + * Judge the eval's labelled cases, recording every result the provider returns. Unlabelled nodes still reach + * the input as related nodes but are not judged themselves. `perNode` hides `evaluateNodes`, so a provider that + * packs nodes into one request gets one request per node instead. + */ +export async function judgeEval( + evaluation: Eval, + provider: SemanticJudgeProvider, + options: { perNode?: boolean } = {}, +): Promise { + const judgements: Judgement[] = []; + const labelledOnly = (input: SemanticJudgeInput): SemanticJudgeInput | undefined => { + const rules = input.rules.filter((rule) => evaluation.expected[rule.id]?.[input.node.id] !== undefined); + return rules.length > 0 ? { ...input, rules } : undefined; + }; + const record = (input: SemanticJudgeInput, results: Record) => { + for (const rule of input.rules) { + judgements.push({ nodeId: input.node.id, ruleId: rule.id, result: results[rule.id] ?? { error: 'no result' } }); + } + return results; + }; + const recording: SemanticJudgeProvider = { + name: provider.name, + async evaluate(input) { + const sent = labelledOnly(input); + return sent ? record(sent, await provider.evaluate(sent)) : {}; + }, + }; + const { evaluateNodes } = provider; + if (evaluateNodes && !options.perNode) { + recording.evaluateNodes = async (inputs) => { + const sent = inputs.map(labelledOnly); + const kept = sent.filter((input) => input !== undefined); + const results = kept.length > 0 ? await evaluateNodes.call(provider, kept) : []; + let k = 0; + return sent.map((input) => (input ? record(input, results[k++] ?? {}) : {})); + }; + } + await validate(evaluation, recording); + return judgements; +} diff --git a/evals/load.ts b/evals/load.ts new file mode 100644 index 0000000..4af82c2 --- /dev/null +++ b/evals/load.ts @@ -0,0 +1,118 @@ +import { existsSync, readdirSync, readFileSync } from 'node:fs'; +import { basename, join, resolve } from 'node:path'; +import JSON5 from 'json5'; +import type { Config, SpaceConfig } from '../src/config'; +import { resolvePluginDir } from '../src/plugins/util'; +import { readSpace } from '../src/read/read-space'; +import { resolveGraphEdges } from '../src/read/resolve-graph-edges'; +import { createSpaceContext } from '../src/space-context'; +import type { BaseNode, SpaceContext } from '../src/types'; + +/** Where core and each plugin keep their semantic suite. */ +export const SUITE_SUBDIR = join('evals', 'semantic'); + +/** The suite's fixtures: spaces whose nodes are in a JSONL file. */ +const FIXTURES_FILE = 'fixtures.json5'; + +const CORE_SUITE_DIR = join(import.meta.dir, 'semantic'); + +/** A directory of evals and their fixtures, and the directory schema names resolve from. */ +export interface Suite { + name: string; + dir: string; + configDir: string; +} + +/** Rule id → node id → the verdicts that count as agreeing. */ +export type Expected = Record>; + +/** Labelled cases on the nodes of one space: a fixture, or a configured space. */ +export interface Eval { + /** `/`, e.g. `core/opportunity-customer-voice`. */ + name: string; + context: SpaceContext; + expected: Expected; + read: () => Promise>; +} + +/** `core`, a plugin name, or a path to a suite directory. */ +export function resolveSuite(source: string, configDirs: string[]): Suite { + if (source === 'core') return { name: 'core', dir: CORE_SUITE_DIR, configDir: CORE_SUITE_DIR }; + if (source.includes('/') || source.includes('\\') || source.startsWith('.')) { + const dir = resolve(source); + if (!existsSync(dir)) throw new Error(`Suite "${source}": no directory at ${dir}`); + return { name: basename(dir), dir, configDir: dir }; + } + for (const configDir of configDirs) { + const pluginDir = resolvePluginDir(source, configDir); + if (!pluginDir) continue; + const dir = join(pluginDir, SUITE_SUBDIR); + if (!existsSync(dir)) throw new Error(`Plugin "${source}" has no ${SUITE_SUBDIR}/ directory (${pluginDir})`); + return { name: source, dir, configDir }; + } + throw new Error(`Suite "${source}" is not core, a path, or a plugin found from the config.`); +} + +type EvalFile = { space: string; expected: Record> }; +type FixturesFile = { spaces: Pick[] }; +type Space = Pick; + +/** + * Every `*.json5` at the top of each suite, except `fixtures.json5`, is an eval. An eval's `space` names one of + * the suite's fixtures, or else a space in `config`. + */ +export function loadEvals(suites: Suite[], config: Config): Eval[] { + return suites.flatMap((suite) => { + const fixtures = loadFixtures(suite); + return readdirSync(suite.dir) + .filter((file) => file.endsWith('.json5') && file !== FIXTURES_FILE) + .sort() + .map((file) => { + const name = `${suite.name}/${basename(file, '.json5')}`; + try { + const { space, expected: labels } = JSON5.parse(readFileSync(join(suite.dir, file), 'utf8')) as EvalFile; + if (typeof space !== 'string') throw new Error('needs a "space"'); + const expected: Expected = {}; + for (const [ruleId, verdicts] of Object.entries(labels)) { + expected[ruleId] = Object.fromEntries(Object.entries(verdicts).map(([id, v]) => [id, [v].flat()])); + } + return { name, expected, ...(fixtures.get(space) ?? configuredSpace(space, config)) }; + } catch (err) { + throw new Error(`Eval ${name}: ${err instanceof Error ? err.message : String(err)}`); + } + }); + }); +} + +function configuredSpace(name: string, config: Config): Space { + const context = createSpaceContext(name, config); + return { context, read: () => readSpace(context) }; +} + +/** A fixture's nodes file holds one parsed node per line, as `sctx dump` prints them. Resolved fields are dropped, since resolution runs again. */ +function loadFixtures(suite: Suite): Map { + const path = join(suite.dir, FIXTURES_FILE); + if (!existsSync(path)) return new Map(); + const { spaces } = JSON5.parse(readFileSync(path, 'utf8')) as FixturesFile; + const config: Config = { spaces: spaces.map((s) => ({ ...s, path: resolve(suite.dir, s.path) })) }; + return new Map( + config.spaces.map(({ name, path: nodesPath }) => { + const context = createSpaceContext(name, config, { configDir: suite.configDir }); + const nodes = readNodes(nodesPath); + return [name, { context, read: async () => resolveGraphEdges(nodes, context.schema.metadata) }]; + }), + ); +} + +function readNodes(path: string): BaseNode[] { + return readFileSync(path, 'utf8') + .split('\n') + .flatMap((line, i) => { + if (line.trim() === '') return []; + const { id, title, type, schemaData, linkTargets, content, contentLinks } = JSON.parse(line) as BaseNode; + if (!id || typeof title !== 'string' || typeof type !== 'string' || !schemaData || !linkTargets) { + throw new Error(`${path}:${i + 1}: not a parsed node (needs id, title, type, schemaData and linkTargets)`); + } + return [{ id, title, type, schemaData, linkTargets, content, contentLinks }]; + }); +} diff --git a/evals/run.ts b/evals/run.ts new file mode 100644 index 0000000..3a4581a --- /dev/null +++ b/evals/run.ts @@ -0,0 +1,99 @@ +/** Semantic evals: judge labelled cases with real providers and report agreement. See README.md. */ +import { dirname } from 'node:path'; +import { parseArgs } from 'node:util'; +import { configPath, loadConfigOrEmpty } from '../src/config'; +import { createJudgeProvider } from '../src/judge/factory'; +import { checkEval, type Judgement, judgeEval } from './judge'; +import { loadEvals, resolveSuite } from './load'; +import { type RuleScore, scoreEval } from './score'; + +const { values } = parseArgs({ + options: { + from: { type: 'string', multiple: true }, + provider: { type: 'string', multiple: true }, + eval: { type: 'string', multiple: true }, + runs: { type: 'string', default: '3' }, + 'per-node': { type: 'boolean', default: false }, + json: { type: 'boolean', default: false }, + check: { type: 'boolean', default: false }, + }, +}); + +const fail = (message: string): never => { + console.error(message); + process.exit(1); +}; + +const list = (values: string[] | undefined) => (values ?? []).flatMap((v) => v.split(',')).filter(Boolean); +const providerNames = list(values.provider); +const runs = Number(values.runs); +if (providerNames.length === 0 && !values.check) fail('Name a model provider with --provider , or pass --check.'); +if (!Number.isInteger(runs) || runs < 1) fail(`--runs must be a positive whole number, not "${values.runs}".`); + +// Build every provider and check every eval before the first paid call. +const config = loadConfigOrEmpty(); +// Plugins resolve from the main config file's directory and from each included file's. +const configDirs = [ + ...new Set([dirname(configPath()), ...config.spaces.flatMap((s) => (s.sourceDir ? [s.sourceDir] : []))]), +]; +const providers = providerNames.map((name) => createJudgeProvider(config, {}, name)!); +const suites = (list(values.from).length > 0 ? list(values.from) : ['core']).map((s) => resolveSuite(s, configDirs)); +const only = list(values.eval); +const evals = loadEvals(suites, config).filter( + (e) => only.length === 0 || only.some((name) => e.name === name || e.name.endsWith(`/${name}`)), +); +if (evals.length === 0) fail('No evals found.'); +const problems = (await Promise.all(evals.map(checkEval))).flat(); +if (problems.length > 0) fail(`Evals have problems:\n${problems.join('\n')}`); +if (values.check) { + console.log(`Checked ${evals.length} eval(s): ${evals.map((e) => e.name).join(', ')}`); + process.exit(0); +} + +const results: Array<{ eval: string; provider: string; scores: RuleScore[]; runs: Judgement[][] }> = []; +for (const evaluation of evals) { + for (const [i, provider] of providers.entries()) { + const judged: Judgement[][] = []; + for (let run = 0; run < runs; run++) { + judged.push(await judgeEval(evaluation, provider, { perNode: values['per-node'] })); + } + results.push({ + eval: evaluation.name, + provider: providerNames[i]!, + scores: scoreEval(evaluation, judged), + runs: judged, + }); + } +} + +if (values.json) { + console.log(JSON.stringify(results, null, 2)); + process.exit(0); +} + +const pct = (n: number | undefined) => (n === undefined ? '-' : `${Math.round(n * 100)}%`); +const lines = [ + `Semantic evals: ${runs} run(s) per provider${values['per-node'] ? ', one request per node' : ''}`, + '', + '| Eval | Provider | Rule | Cases | Agreement | Stable | p(expected) | Reserved | Errors |', + '|---|---|---|---|---|---|---|---|---|', +]; +const misses: string[] = []; +const closest: string[] = []; +for (const { eval: name, provider, scores } of results) { + for (const s of scores) { + const stable = runs > 1 ? pct(s.stable) : '-'; + lines.push( + `| ${name} | ${provider} | ${s.ruleId} | ${s.cases} | ${pct(s.agreement)} | ${stable} | ${pct(s.expectedProbability)} | ${s.reserved} | ${s.errors} |`, + ); + const at = `${name} · ${provider} · ${s.ruleId}`; + for (const m of s.misses) { + misses.push(`- ${at} · ${m.nodeId}: expected ${m.expected.join(' or ')}, got ${m.got.join(', ')}`); + if (m.evidence) misses.push(` ${m.evidence}`); + } + for (const c of s.closest) closest.push(`- ${at} · ${c.nodeId}: ${pct(c.probability)}`); + } +} +if (misses.length > 0) lines.push('', 'Misses:', ...misses); +if (closest.length > 0) lines.push('', 'Closest agreeing cases, by p(expected):', ...closest); +console.log(lines.join('\n')); diff --git a/evals/score.ts b/evals/score.ts new file mode 100644 index 0000000..2399900 --- /dev/null +++ b/evals/score.ts @@ -0,0 +1,79 @@ +import { RESERVED_VERDICTS } from '../src/schema/metadata-contract'; +import type { Judgement } from './judge'; +import type { Eval } from './load'; + +/** How well a provider's verdicts on one rule match the labels, over every run. README.md explains each score. */ +export interface RuleScore { + ruleId: string; + cases: number; + runs: number; + agreement: number; + stable: number; + expectedProbability?: number; + reserved: number; + errors: number; + /** Cases where some run disagreed with the label, with the verdict (or error) each run gave. */ + misses: Array<{ nodeId: string; expected: string[]; got: string[]; evidence?: string }>; + /** The agreeing cases with the lowest mean p(expected): the ones nearest to a miss. */ + closest: Array<{ nodeId: string; probability: number }>; +} + +const CLOSEST = 3; +const reserved: readonly string[] = RESERVED_VERDICTS; + +const describe = (result: Judgement['result'] | undefined) => + !result ? 'not judged' : 'error' in result ? `error: ${result.error}` : result.verdict; + +/** Score each labelled rule over the runs of one eval with one provider. */ +export function scoreEval(evaluation: Pick, runs: Judgement[][]): RuleScore[] { + return Object.entries(evaluation.expected).map(([ruleId, labels]) => { + const perRun = runs.map((run) => new Map(run.filter((j) => j.ruleId === ruleId).map((j) => [j.nodeId, j.result]))); + let agreeing = 0; + let stable = 0; + let errors = 0; + let reservedCount = 0; + const probabilities: number[] = []; + const misses: RuleScore['misses'] = []; + const agreed: RuleScore['closest'] = []; + for (const [nodeId, expected] of Object.entries(labels)) { + const results = perRun.map((run) => run.get(nodeId)); + const verdicts = results.map((r) => (r && !('error' in r) ? r.verdict : undefined)); + agreeing += verdicts.filter((v) => v !== undefined && expected.includes(v)).length; + reservedCount += verdicts.filter((v) => v !== undefined && reserved.includes(v)).length; + if (verdicts.includes(undefined)) errors++; + else if (new Set(verdicts).size === 1) stable++; + const caseProbabilities = results.flatMap((r) => + r && !('error' in r) && r.probabilities + ? [expected.reduce((sum, v) => sum + (r.probabilities?.[v] ?? 0), 0)] + : [], + ); + probabilities.push(...caseProbabilities); + if (verdicts.some((v) => v === undefined || !expected.includes(v))) { + const miss = results.find((r) => !r || 'error' in r || !expected.includes(r.verdict)); + misses.push({ + nodeId, + expected, + got: results.map(describe), + ...(miss && !('error' in miss) && miss.evidence !== undefined && { evidence: miss.evidence }), + }); + } else if (caseProbabilities.length > 0) { + agreed.push({ nodeId, probability: mean(caseProbabilities) }); + } + } + const cases = Object.keys(labels).length; + return { + ruleId, + cases, + runs: runs.length, + agreement: cases && runs.length ? agreeing / (cases * runs.length) : 0, + stable: cases ? stable / cases : 0, + ...(probabilities.length > 0 && { expectedProbability: mean(probabilities) }), + reserved: reservedCount, + errors, + misses, + closest: agreed.sort((a, b) => a.probability - b.probability).slice(0, CLOSEST), + }; + }); +} + +const mean = (values: number[]) => values.reduce((a, b) => a + b, 0) / values.length; diff --git a/evals/semantic/fixtures.json5 b/evals/semantic/fixtures.json5 new file mode 100644 index 0000000..5f17d2a --- /dev/null +++ b/evals/semantic/fixtures.json5 @@ -0,0 +1,6 @@ +{ + spaces: [ + // A small product's opportunity solution tree. + { name: 'strict-ost', path: 'fixtures/strict-ost.jsonl', schema: 'strict_ost.json' }, + ], +} diff --git a/evals/semantic/fixtures/strict-ost.jsonl b/evals/semantic/fixtures/strict-ost.jsonl new file mode 100644 index 0000000..18d7996 --- /dev/null +++ b/evals/semantic/fixtures/strict-ost.jsonl @@ -0,0 +1,14 @@ +{"id":"Increase weekly active teams.md","title":"Increase weekly active teams","type":"outcome","schemaData":{"title":"Increase weekly active teams","type":"outcome","status":"identified","metric":"Increase % of trial teams active in their second week","content":"Teams that use the product every week renew."},"linkTargets":["Increase weekly active teams"]} +{"id":"Setting up a workspace takes so long that I give up.md","title":"Setting up a workspace takes so long that I give up","type":"opportunity","schemaData":{"title":"Setting up a workspace takes so long that I give up","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview with Priya, 2026-09-02","content":"Priya abandoned setup twice before her team lead finished it for her."},"linkTargets":["Setting up a workspace takes so long that I give up"]} +{"id":"I can't tell which plan fits a team of my size.md","title":"I can't tell which plan fits a team of my size","type":"opportunity","schemaData":{"title":"I can't tell which plan fits a team of my size","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Sales call notes, 2026-09-10","content":"Prospects asked which tier covers twelve seats with guest access."},"linkTargets":["I can't tell which plan fits a team of my size"]} +{"id":"I want to see what my teammates changed while I was away.md","title":"I want to see what my teammates changed while I was away","type":"opportunity","schemaData":{"title":"I want to see what my teammates changed while I was away","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview with Tomas, 2026-09-15","content":"Tomas reads every page again after a holiday to find what is new."},"linkTargets":["I want to see what my teammates changed while I was away"]} +{"id":"I'm afraid I'll lose my work if my browser crashes.md","title":"I'm afraid I'll lose my work if my browser crashes","type":"opportunity","schemaData":{"title":"I'm afraid I'll lose my work if my browser crashes","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Support tickets, September 2026","content":"Several tickets describe retyping long notes after a crash."},"linkTargets":["I'm afraid I'll lose my work if my browser crashes"]} +{"id":"I don't know whether my changes have been saved.md","title":"I don't know whether my changes have been saved","type":"opportunity","schemaData":{"title":"I don't know whether my changes have been saved","type":"opportunity","status":"identified","parent":"[[I'm afraid I'll lose my work if my browser crashes]]","source":"Usability test round 3","content":"Four of six participants pressed the save shortcut repeatedly."},"linkTargets":["I don't know whether my changes have been saved"]} +{"id":"Show unsaved-change indicator.md","title":"Show unsaved-change indicator","type":"solution","schemaData":{"title":"Show unsaved-change indicator","type":"solution","status":"exploring","parent":"[[I don't know whether my changes have been saved]]","content":"A dot on the tab while changes are pending."},"linkTargets":["Show unsaved-change indicator"]} +{"id":"Add a progress bar to workspace setup.md","title":"Add a progress bar to workspace setup","type":"opportunity","schemaData":{"title":"Add a progress bar to workspace setup","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Product team brainstorm","content":"Show users how many setup steps remain."},"linkTargets":["Add a progress bar to workspace setup"]} +{"id":"Build a Slack integration.md","title":"Build a Slack integration","type":"opportunity","schemaData":{"title":"Build a Slack integration","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Competitor review","content":"Post page updates to a Slack channel."},"linkTargets":["Build a Slack integration"]} +{"id":"Offer a discount on annual plans.md","title":"Offer a discount on annual plans","type":"opportunity","schemaData":{"title":"Offer a discount on annual plans","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Pricing workshop","content":"Ten percent off when paying yearly."},"linkTargets":["Offer a discount on annual plans"]} +{"id":"Increase trial-to-paid conversion by 10%.md","title":"Increase trial-to-paid conversion by 10%","type":"opportunity","schemaData":{"title":"Increase trial-to-paid conversion by 10%","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Quarterly planning","content":"The board wants conversion to rise this quarter."},"linkTargets":["Increase trial-to-paid conversion by 10%"]} +{"id":"Reduce churn among small teams.md","title":"Reduce churn among small teams","type":"opportunity","schemaData":{"title":"Reduce churn among small teams","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Retention dashboard","content":"Teams of fewer than five seats churn twice as often."},"linkTargets":["Reduce churn among small teams"]} +{"id":"Onboarding.md","title":"Onboarding","type":"opportunity","schemaData":{"title":"Onboarding","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Interview synthesis","content":"Everything about how new users get started."},"linkTargets":["Onboarding"]} +{"id":"New users find the editor toolbar confusing.md","title":"New users find the editor toolbar confusing","type":"opportunity","schemaData":{"title":"New users find the editor toolbar confusing","type":"opportunity","status":"identified","parent":"[[Increase weekly active teams]]","source":"Usability test round 2","content":"Participants hovered over icons without knowing what they did."},"linkTargets":["New users find the editor toolbar confusing"]} diff --git a/evals/semantic/opportunity-customer-voice.json5 b/evals/semantic/opportunity-customer-voice.json5 new file mode 100644 index 0000000..9291342 --- /dev/null +++ b/evals/semantic/opportunity-customer-voice.json5 @@ -0,0 +1,21 @@ +{ + space: 'strict-ost', + expected: { + 'opportunity-customer-voice': { + 'Setting up a workspace takes so long that I give up.md': 'pass', + "I can't tell which plan fits a team of my size.md": 'pass', + "I'm afraid I'll lose my work if my browser crashes.md": 'pass', + "I don't know whether my changes have been saved.md": 'pass', + 'I want to see what my teammates changed while I was away.md': 'pass', + // A solution in disguise. + 'Add a progress bar to workspace setup.md': 'fail', + 'Build a Slack integration.md': 'fail', + 'Offer a discount on annual plans.md': 'fail', + // A business metric. + 'Increase trial-to-paid conversion by 10%.md': 'fail', + 'Reduce churn among small teams.md': 'fail', + // A topic, not a need. + 'Onboarding.md': 'fail', + }, + }, +} diff --git a/package.json b/package.json index 73db597..1fc124c 100644 --- a/package.json +++ b/package.json @@ -57,6 +57,7 @@ "test:mod": "claude plugin test plugin", "test:hook:e2e": "bun test hook-test/hooks.test.ts", "test:all": "bun test", + "eval:semantic": "bun run evals/run.ts", "lint": "biome check", "lint:fix": "biome check --write", "preversion": "bun run generate:schema-meta && bun run lint:fix && bun run test && bun run test:package", diff --git a/tests/evals/semantic.test.ts b/tests/evals/semantic.test.ts new file mode 100644 index 0000000..ccdec5b --- /dev/null +++ b/tests/evals/semantic.test.ts @@ -0,0 +1,149 @@ +import { afterAll, describe, expect, test } from 'bun:test'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { basename, join } from 'node:path'; +import { checkEval, type Judgement, judgeEval } from '../../evals/judge'; +import { loadEvals, resolveSuite } from '../../evals/load'; +import { scoreEval } from '../../evals/score'; +import type { Config } from '../../src/config'; +import { FakeJudgeProvider } from '../../src/judge/provider/fake'; +import type { SemanticJudgeInput } from '../../src/judge/types'; + +const coreEvals = loadEvals([resolveSuite('core', [])], { spaces: [] }); + +describe.each(coreEvals.map((e) => [e.name, e] as const))('eval %s', (_name, evaluation) => { + test('passes its checks', async () => { + expect(await checkEval(evaluation)).toEqual([]); + }); +}); + +describe('a suite directory', () => { + const dir = mkdtempSync(join(tmpdir(), 'sctx-evals-')); + afterAll(() => rmSync(dir, { recursive: true, force: true })); + const write = (path: string, value: unknown) => { + mkdirSync(join(dir, path, '..'), { recursive: true }); + writeFileSync(join(dir, path), JSON.stringify(value)); + }; + const node = (title: string, type: string, fields: Record) => + JSON.stringify({ + id: `${title}.md`, + title, + type, + linkTargets: [title], + schemaData: { title, type, status: 'identified', ...fields }, + }); + const opportunity = (title: string) => node(title, 'opportunity', { parent: '[[Grow usage]]', source: 'Interviews' }); + mkdirSync(join(dir, 'nodes')); + writeFileSync( + join(dir, 'nodes', 'tree.jsonl'), + [ + node('Grow usage', 'outcome', { metric: 'Weekly active teams' }), + opportunity('Onboarding'), + opportunity('Build a Slack integration'), + ].join('\n'), + ); + write('fixtures.json5', { spaces: [{ name: 'tree', path: 'nodes/tree.jsonl', schema: 'strict_ost.json' }] }); + write('a.json5', { space: 'tree', expected: { 'opportunity-customer-voice': { 'Onboarding.md': 'fail' } } }); + write('b.json5', { + space: 'tree', + expected: { + 'opportunity-customer-voice': { 'Grow usage.md': 'fail', 'Build a Slack integration.md': ['pass', 'fail'] }, + }, + }); + mkdirSync(join(dir, 'space')); + writeFileSync( + join(dir, 'space', 'Grow usage.md'), + '---\nstatus: identified\ntype: outcome\nmetric: Weekly active teams\n---\n', + ); + writeFileSync( + join(dir, 'space', 'Onboarding.md'), + '---\nstatus: identified\ntype: opportunity\nparent: "[[Grow usage]]"\nsource: Interviews\n---\n', + ); + write('mine.json5', { space: 'work', expected: { 'opportunity-customer-voice': { 'Onboarding.md': 'fail' } } }); + const config: Config = { + spaces: [{ name: 'work', path: join(dir, 'space'), schema: 'strict_ost.json', sourceDir: dir }], + }; + const [a, b, mine] = loadEvals([resolveSuite(dir, [])], config); + + test('loads each eval file, with evals sharing a fixture', () => { + expect([a, b, mine].map((e) => e!.name)).toEqual(['a', 'b', 'mine'].map((n) => `${basename(dir)}/${n}`)); + expect(a!.context).toBe(b!.context); + expect(mine!.context.space.name).toBe('work'); + }); + + test('passes an eval that labels only some judged nodes', async () => { + expect(await checkEval(a!)).toEqual([]); + expect(await checkEval(mine!)).toEqual([]); + }); + + test('reports labels the rule never judges, and labels that accept every verdict', async () => { + expect(await checkEval(b!)).toEqual([ + `${b!.name}: rule "opportunity-customer-voice", node "Grow usage.md": not judged (unknown node, or the rule skips it)`, + `${b!.name}: rule "opportunity-customer-voice", node "Build a Slack integration.md": accepts every verdict`, + ]); + }); + + test('sends only labelled nodes to the provider', async () => { + const judge = new FakeJudgeProvider({ verdict: 'fail' }); + const judged = await judgeEval(a!, judge); + expect(judged.map((j) => j.nodeId)).toEqual(['Onboarding.md']); + expect(judge.callHistory.map((input) => input.node.id as string)).toEqual(['Onboarding.md']); + }); +}); + +describe('judgeEval', () => { + const [evaluation] = coreEvals; + /** A judge that packs nodes, counting its calls of each kind. */ + class PackingJudge extends FakeJudgeProvider { + calls = { evaluate: 0, evaluateNodes: 0 }; + override evaluate(input: SemanticJudgeInput) { + this.calls.evaluate++; + return super.evaluate(input); + } + async evaluateNodes(inputs: SemanticJudgeInput[]) { + this.calls.evaluateNodes++; + return Promise.all(inputs.map((input) => super.evaluate(input))); + } + } + + test('passes packed inputs through, or one input per call with perNode', async () => { + const packed = new PackingJudge(); + const perNode = new PackingJudge(); + const a = await judgeEval(evaluation!, packed); + const b = await judgeEval(evaluation!, perNode, { perNode: true }); + expect(packed.calls).toEqual({ evaluate: 0, evaluateNodes: expect.any(Number) }); + expect(perNode.calls).toEqual({ evaluate: b.length, evaluateNodes: 0 }); + expect(a.length).toBe(b.length); + }); +}); + +describe('scoreEval', () => { + const evaluation = { expected: { r: { a: ['pass'], b: ['fail'], c: ['pass', 'fail'] } } }; + const j = (nodeId: string, verdict: string, probabilities?: Record): Judgement => ({ + nodeId, + ruleId: 'r', + result: { verdict, ...(probabilities && { probabilities }) }, + }); + + test('scores agreement, stability, probability and errors over runs', () => { + const runs = [ + [j('a', 'pass', { pass: 0.9, fail: 0.1 }), j('b', 'pass', { pass: 0.6, fail: 0.4 }), j('c', 'fail')], + [j('a', 'pass', { pass: 0.7, fail: 0.3 }), j('b', 'fail', { pass: 0.2, fail: 0.8 })], + ]; + const [score] = scoreEval(evaluation, runs); + expect(score).toMatchObject({ cases: 3, runs: 2, stable: 1 / 3, errors: 1, reserved: 0 }); + expect(score!.agreement).toBeCloseTo(4 / 6); + expect(score!.expectedProbability).toBeCloseTo((0.9 + 0.4 + 0.7 + 0.8) / 4); + expect(score!.misses).toEqual([ + { nodeId: 'b', expected: ['fail'], got: ['pass', 'fail'] }, + { nodeId: 'c', expected: ['pass', 'fail'], got: ['fail', 'not judged'] }, + ]); + expect(score!.closest).toEqual([{ nodeId: 'a', probability: expect.closeTo(0.8) }]); + }); + + test('counts reserved verdicts and errors', () => { + const runs = [[j('a', 'missing-context'), { nodeId: 'b', ruleId: 'r', result: { error: 'boom' } }, j('c', 'pass')]]; + const [score] = scoreEval(evaluation, runs); + expect(score).toMatchObject({ reserved: 1, errors: 1, agreement: 1 / 3 }); + }); +}); diff --git a/tsconfig.build.json b/tsconfig.build.json index 55948f3..81aa233 100644 --- a/tsconfig.build.json +++ b/tsconfig.build.json @@ -11,6 +11,7 @@ "hook-test", "plugin/scripts", "scripts", + "evals", "**/*.test.ts", "**/*.spec.ts" ] diff --git a/tsconfig.json b/tsconfig.json index 7a7bd17..50ca1dd 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -29,6 +29,14 @@ "noUnusedParameters": false, "noPropertyAccessFromIndexSignature": false }, - "include": ["src/**/*", "tests/**/*", "smoke/**/*", "hook-test/**/*", "plugin/scripts/**/*", "scripts/**/*"], + "include": [ + "src/**/*", + "tests/**/*", + "smoke/**/*", + "hook-test/**/*", + "plugin/scripts/**/*", + "scripts/**/*", + "evals/**/*" + ], "exclude": ["node_modules", "dist"] }