From 6432abc96789bb7e74fdf51bdea1212714cc63e3 Mon Sep 17 00:00:00 2001 From: forrestlinfeng Date: Mon, 26 Oct 2026 14:13:16 +0800 Subject: [PATCH] Count an injected strategy as used, and distill turns that reused one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The applied verdict matched a strategy's steps against the turn's commands, paths, edited code and answer. It failed on text without spaces (a turn that followed every Chinese step read as `unknown`, or `unused` once it ran a command) and would need tuning per language. The model follows what is injected into its turn, so an injection is now a use: every report says `applied: used`, and whether the strategy helped is left to what already judges it — the outcome from the last check or how the turn ended, the next prompt, and whether the edits were kept. A turn that reused an injected strategy no longer skips distillation; with every injection a use, that rule would have stopped nearly every turn from being distilled. The engine's Gene intake drops one that repeats a Gene it already has. applied.js and the per-turn edited-code buffer (up to 200,000 characters) existed only for the verdict and are removed; a turn records which files it edited instead. Co-Authored-By: Claude Opus 5.5 --- README.md | 28 +++++------ README.zh.md | 12 ++--- src/applied.js | 70 -------------------------- src/edited-content.js | 5 -- src/index.js | 9 ++-- src/reuse.js | 20 ++++---- src/session-distill.js | 9 ++-- src/turn-record.js | 15 +++--- test/applied.test.js | 95 ------------------------------------ test/reporting.test.js | 16 +++--- test/reuse.test.js | 21 +++----- test/session-distill.test.js | 16 +++++- 12 files changed, 70 insertions(+), 246 deletions(-) delete mode 100644 src/applied.js delete mode 100644 test/applied.test.js diff --git a/README.md b/README.md index 5e52fc6..16e720c 100644 --- a/README.md +++ b/README.md @@ -104,9 +104,10 @@ counts those reports as unevidenced. ## Session-end distillation -A turn that ended completed, used tools, passed its last check and reused no injected strategy leaves a -candidate in `~/.evolver/state/distill-pending.json`; a later verified turn replaces it, and a next prompt -that reads as a correction drops it. When the session ends, one call to the session's own model writes a +A turn that ended completed, used tools and passed its last check leaves a candidate in +`~/.evolver/state/distill-pending.json`, a turn that reused an injected strategy too (the engine's intake +drops a Gene that repeats one it already has); a later verified turn replaces it, and a next prompt that +reads as a correction drops it. When the session ends, one call to the session's own model writes a Gene from a digest of that turn (task, commands, touched paths, checks, final answer — never the whole history) and sends it to the Proxy's `POST /conversation/distill` with `persist: true`, `publish: false` and its own token cost as `distill_usage`. Keys and tokens are stripped from the candidate first, paths and @@ -123,17 +124,14 @@ weighs each report by its evidence: | Report | When | `task_id` | `evidence` | |---|---|---|---| -| Turn end | the turn ends | `dsh::` | `tier` `L1` when the turn ran a test, build, lint or type check, else `none`; `applied`; `check` `{kind, passed, fail_then_pass}` of the last check; `aborted`; `stage: immediate` | -| Correction | the next prompt says the answer did not hold, or approves it | `…:correction` | `tier: L3`, `applied`, `stage: correction` | -| Retention | the next session in the same project finds the turn's edits kept or reverted | `…:retention` | `tier: L2`, `applied`, `stage: retention` | -| Aborted | the session goes away, or is killed, before the turn ends (a killed session's go out at the next session start, once its dsh process is gone) | `dsh::` | `tier: none`, `applied: partial`, `aborted: true`, `stage: immediate` | - -- **`applied`** compares the strategy's numbered steps with the turn's shell commands, - touched paths, the code its edits wrote (200,000 characters a turn, kept in memory only) - and the final answer: `used` when half the steps show up, `partial` when fewer do, - `unused` when none do. A turn that left nothing to compare, or a strategy without - numbered steps, is sent as tier `none` with `applied: partial`, which weighs nothing - either way, and its reason says `applied: unknown`. +| Turn end | the turn ends | `dsh::` | `tier` `L1` when the turn ran a test, build, lint or type check, else `none`; `applied: used`; `check` `{kind, passed, fail_then_pass}` of the last check; `aborted`; `stage: immediate` | +| Correction | the next prompt says the answer did not hold, or approves it | `…:correction` | `tier: L3`, `applied: used`, `stage: correction` | +| Retention | the next session in the same project finds the turn's edits kept or reverted | `…:retention` | `tier: L2`, `applied: used`, `stage: retention` | +| Aborted | the session goes away, or is killed, before the turn ends (a killed session's go out at the next session start, once its dsh process is gone) | `dsh::` | `tier: none`, `applied: used`, `aborted: true`, `stage: immediate` | + +- **`applied`** is always `used`: the model follows a strategy injected into its turn. Whether the + strategy helped is what the outcome says, judged by the turn's last check (or how the turn ended when it + ran none), the next prompt and whether its edits were kept. - **Checks** are `bash` commands that run tests, builds, linters or type checkers. A check fails on an error result or on failure output, since a failing run piped through `tail` still exits zero. `fail_then_pass` compares commands with output pipes (`| tail -9`) and @@ -148,7 +146,7 @@ weighs each report by its evidence: - `/evolver-status` lists the Hub's verdict (`eligible`, `counted`, `weight`) on recent reports and the reason for each rejected one. Reports with evidence are also written to the local root event log once. -- Reasons name the turn, the applied verdict and the check kind, never the prompt, a +- Reasons name the turn and the check kind, never the prompt, a command, code or a diff, and the prompt loses keys and tokens before recall sends it. ## Requirements diff --git a/README.zh.md b/README.zh.md index db432d2..bd8637f 100644 --- a/README.zh.md +++ b/README.zh.md @@ -94,17 +94,17 @@ dsh plugin --profile dsh-tui add -w @evomap/dsh-evolver@latest | 上报 | 时机 | `task_id` | `evidence` | |---|---|---|---| -| 回合结束 | 回合结束时 | `dsh:<会话>:<回合>` | 本回合跑过测试、构建、lint 或类型检查时 `tier` 为 `L1`，否则为 `none`；`applied`；最后一次检查的 `check` `{kind, passed, fail_then_pass}`；`aborted`；`stage: immediate` | -| 追问补报 | 下一条提示词表示结果不对，或表示认可 | `…:correction` | `tier: L3`、`applied`、`stage: correction` | -| 留存回访 | 同一项目的下一个会话发现本回合的改动被保留或被撤销 | `…:retention` | `tier: L2`、`applied`、`stage: retention` | -| 中止 | 回合结束前会话消失或被强杀（被强杀会话的上报在下次会话开始时发出） | `dsh::` | `tier: none`、`applied: partial`、`aborted: true`、`stage: immediate` | +| 回合结束 | 回合结束时 | `dsh:<会话>:<回合>` | 本回合跑过测试、构建、lint 或类型检查时 `tier` 为 `L1`，否则为 `none`；`applied: used`；最后一次检查的 `check` `{kind, passed, fail_then_pass}`；`aborted`；`stage: immediate` | +| 追问补报 | 下一条提示词表示结果不对，或表示认可 | `…:correction` | `tier: L3`、`applied: used`、`stage: correction` | +| 留存回访 | 同一项目的下一个会话发现本回合的改动被保留或被撤销 | `…:retention` | `tier: L2`、`applied: used`、`stage: retention` | +| 中止 | 回合结束前会话消失或被强杀（被强杀会话的上报在下次会话开始时发出） | `dsh::` | `tier: none`、`applied: used`、`aborted: true`、`stage: immediate` | -- **`applied`** 把策略的编号步骤与本回合的 shell 命令、涉及的路径、编辑写入的代码（每回合最多 20 万字符，只留在内存里）和最终回答比对：半数以上步骤出现记 `used`，少于半数记 `partial`，一条都没有记 `unused`。回合没留下可比对的内容，或策略没有编号步骤时，按 tier `none` 加 `applied: partial` 上报，两边都不计权重，reason 里写 `applied: unknown`。 +- **`applied`** 固定为 `used`：注入到回合里的策略，模型就会照着用。策略有没有起作用看结果：本回合最后一次检查是否通过（没跑检查时看回合如何结束）、下一条提示词，以及改动是否被保留。 - **检查**指运行测试、构建、lint 或类型检查的 `bash` 命令。结果报错或输出里有失败信息都算失败，因为经 `tail` 管道的失败运行退出码仍是 0。判断 `fail_then_pass` 时会先去掉输出管道（`| tail -9`）和重定向（`2>&1`）再比较命令。 - **outcome** 跟随最后一次检查；没有检查时跟随回合的结束方式。 - **投递**：上报先存进 `~/.evolver/state/reports-outbox.json`，在每次回合结束、追问和会话开始时重试，保留 8 天。Hub 只要给出答复就算送达，包括 `duplicate_report` 和不计分的回执（`same_owner`、`daily_cap`）。Hub 判为无效的（`terminal: true`，或旧版 Proxy 返回的 `hub 400` / `hub 422`）记为被拒，不再重发。`hub 404`、`hub 5xx` 和网络失败会重试。 - `/evolver-status` 会列出最近上报的 Hub 判定（`eligible`、`counted`、`weight`）以及每条被拒上报的原因。带证据的上报还会在本地 root event 日志里记一次。 -- reason 只写回合、采用判定和检查类型，不含提示词、命令、代码或 diff；召回前会先去掉提示词里的密钥和令牌。 +- reason 只写回合和检查类型，不含提示词、命令、代码或 diff；召回前会先去掉提示词里的密钥和令牌。 ## 环境要求 diff --git a/src/applied.js b/src/applied.js deleted file mode 100644 index e29181f..0000000 --- a/src/applied.js +++ /dev/null @@ -1,70 +0,0 @@ -// SPDX-License-Identifier: MIT -// Copyright (c) 2026 EvoMap - -const STEP_PREFIX = /^\s*\d+\.\s+/; -const BACKTICKED = /`([^`\n]{2,120})`/g; -const PATH_LIKE = /(?:^|[\s("'])((?:\.{0,2}\/)?[\w@.-]+(?:\/[\w@.-]+)+|[\w-]+\.(?:js|mjs|cjs|ts|tsx|jsx|py|go|rs|java|kt|rb|php|json|ya?ml|toml|md|sh|sql|css|html|vue|svelte))(?=$|[\s)"',.:;])/g; -const COMMAND_LIKE = /\b((?:npm|npx|pnpm|yarn|bun|node|git|go|cargo|pytest|python3?|pip|make|docker|kubectl|tsc|eslint|jest|vitest|dsh|gh)\s+[\w:@./=-]+)/g; -const WORD = /[\p{L}\p{N}_]{4,}/gu; -const STOPWORDS = new Set([ - 'that', 'this', 'with', 'from', 'into', 'then', 'when', 'each', 'only', 'before', 'after', 'step', 'steps', 'make', 'sure', - 'should', 'must', 'never', 'always', 'use', 'using', 'used', 'have', 'has', 'were', 'will', 'your', 'their', 'there', 'these', - 'those', 'what', 'which', 'while', 'until', 'other', 'every', 'more', 'than', 'also', 'apply', 'where', 'fits', 'report', - 'outcome', 'evolver_asset_reuse_result', 'safety', 'boundary', 'optional', 'otherwise', 'first', 'next', 'last', 'same', -]); - -const USED_RATIO = 0.5; -export const UNKNOWN = 'unknown'; -const TERM_COVERAGE = 0.5; -const MIN_STEP_TERMS = 3; - -export function strategyStepsOf(injectedText) { - return String(injectedText ?? '') - .split('\n') - .filter((line) => STEP_PREFIX.test(line)) - .map((line) => line.replace(STEP_PREFIX, '').trim()); -} - -function matchesOf(text, pattern) { - return [...text.matchAll(pattern)].map((match) => match[1].trim().toLowerCase()); -} - -export function concreteFeaturesOf(step) { - const features = [...matchesOf(step, BACKTICKED), ...matchesOf(step, PATH_LIKE), ...matchesOf(step, COMMAND_LIKE)]; - return [...new Set(features.map((feature) => feature.replace(/[.,:;]+$/, '')).filter((feature) => feature.length >= 3))]; -} - -function termsOf(text) { - return [...new Set((String(text).toLowerCase().match(WORD) ?? []).filter((term) => !STOPWORDS.has(term)))]; -} - -function turnHaystack({ commands = [], paths = [], edits = [], answer = '' }) { - return [...commands, ...paths, ...edits, answer].join('\n').toLowerCase(); -} - -function termsWereFollowed(step, haystack) { - const terms = termsOf(step); - if (terms.length < MIN_STEP_TERMS) return false; - return terms.filter((term) => haystack.includes(term)).length / terms.length >= TERM_COVERAGE; -} - -function stepWasFollowed(step, haystack) { - return concreteFeaturesOf(step).some((feature) => haystack.includes(feature)) || termsWereFollowed(step, haystack); -} - -function leftNothingToCompare({ commands = [], paths = [], edits = [] }) { - return commands.length + paths.length + edits.length === 0; -} - -function verdictOf(turn, steps, hits) { - if (hits > 0) return hits / steps >= USED_RATIO ? 'used' : 'partial'; - return steps === 0 || leftNothingToCompare(turn) ? UNKNOWN : 'unused'; -} - -export function appliedVerdictOf({ strategyText, ...turn }) { - const steps = strategyStepsOf(strategyText); - const haystack = turnHaystack(turn); - const hits = steps.filter((step) => stepWasFollowed(step, haystack)).length; - const ratio = steps.length === 0 ? 0 : hits / steps.length; - return { verdict: verdictOf(turn, steps.length, hits), hits, steps: steps.length, ratio: Math.round(ratio * 1000) / 1000 }; -} diff --git a/src/edited-content.js b/src/edited-content.js index 75d9985..22db0f1 100644 --- a/src/edited-content.js +++ b/src/edited-content.js @@ -3,7 +3,6 @@ export const EDIT_TOOL_NAMES = ['write', 'edit', 'str_replace_editor']; -const CONTENT_KEYS = ['content', 'new_string', 'new_str', 'file_text', 'text']; const PATH_KEYS = ['path', 'file_path']; function firstString(args, keys) { @@ -13,10 +12,6 @@ function firstString(args, keys) { return ''; } -export function editedContent(args) { - return firstString(args, CONTENT_KEYS); -} - export function editedPath(args) { return firstString(args, PATH_KEYS); } diff --git a/src/index.js b/src/index.js index fc01373..af3e3a8 100644 --- a/src/index.js +++ b/src/index.js @@ -3,7 +3,6 @@ import { boundContextSummary, createUserMessage } from '@deepseek-ai/dsh-llm'; -import { appliedVerdictOf } from './applied.js'; import { evolverCommands } from './commands.js'; import { Config } from './config.js'; import { createRecallLog, createReportLog, pluginDiagnostics } from './diagnostics.js'; @@ -186,12 +185,11 @@ function reportsOf(assets, { sessionId, turn, reasonKind, record }) { turn: turn ?? asset.turn, reasonKind, checks: record.checks, - applied: appliedVerdictOf({ strategyText: asset.strategyText, ...record }).verdict, })); } // Every injected strategy is reported when its turn ends, with the turn's own evidence; the turn itself -// becomes this session's distillation candidate when it finished verified without reusing one. +// becomes this session's distillation candidate when it finished verified. function reportTurns(ctx, records, deliver, labelable, projectDirOf, prompts) { const closeTurn = (session, { turn, reason }) => { const sessionId = sessionKeyOf(session); @@ -203,8 +201,7 @@ function reportTurns(ctx, records, deliver, labelable, projectDirOf, prompts) { labelable.set(sessionId, reuses); deliver(reports.map((report) => ({ report, sessionId }))).catch(() => {}); rememberForRetention({ sessionId, projectDir: projectDirOf(session), reuses, editedBefore: finished.editedBefore }); - const reusedAsset = reuses.some((reuse) => reuse.applied === 'used'); - if (skipReason(reasonKind, finished, { reusedAsset })) return; + if (skipReason(reasonKind, finished)) return; const model = modelOf(session?.deriveMessages?.() ?? []); rememberCandidate(sessionId, candidateOf({ prompt: prompts.get(sessionId), record: finished, model, turn })); }; @@ -218,7 +215,7 @@ function reportTurns(ctx, records, deliver, labelable, projectDirOf, prompts) { function abortedReports(assets) { return assets.map((asset) => ({ - report: reportsOf([asset], { sessionId: asset.sessionId, reasonKind: 'aborted', record: { checks: [], commands: [], paths: [], edits: [], answer: '' } })[0], + report: reportsOf([asset], { sessionId: asset.sessionId, reasonKind: 'aborted', record: { checks: [] } })[0], sessionId: asset.sessionId, })); } diff --git a/src/reuse.js b/src/reuse.js index 7ea5510..8c5677f 100644 --- a/src/reuse.js +++ b/src/reuse.js @@ -1,7 +1,6 @@ // SPDX-License-Identifier: MIT // Copyright (c) 2026 EvoMap -import { UNKNOWN } from './applied.js'; import { checkEvidenceOf } from './checks.js'; import { REJECTED, RETRY, SENT, isSettled, withAttempt } from './report-outbox.js'; import { outcomeOfReason } from './turn-outcomes.js'; @@ -14,16 +13,15 @@ function turnTaskId(sessionId, turn) { return `dsh:${sessionId || 'unknown'}:${turn ?? 'unknown'}`; } -// The Hub has no `unknown`, and `unused` carries a soft negative there; a -// partial reuse at tier none weighs nothing either way. -const UNKNOWN_APPLIED_AS = 'partial'; +// The model follows a strategy injected into its turn, so an injection is a use; whether it helped is +// what the outcome, the check, the next prompt and the edits' retention tell. +const APPLIED = 'used'; -function immediateEvidenceOf({ checks, applied, reasonKind }) { +function immediateEvidenceOf({ checks, reasonKind }) { const check = checkEvidenceOf(checks); - const isKnown = applied !== UNKNOWN; return { - tier: check && isKnown ? 'L1' : 'none', - applied: isKnown ? applied : UNKNOWN_APPLIED_AS, + tier: check ? 'L1' : 'none', + applied: APPLIED, ...(check && { check }), aborted: reasonKind === 'aborted', stage: 'immediate', @@ -45,13 +43,13 @@ function reportOf({ assetId, taskId, outcome, reason, evidence }) { return { asset_id: assetId, outcome, task_id: taskId, reason, evidence }; } -export function immediateReportOf({ asset, sessionId, turn, reasonKind, checks, applied }) { - const evidence = immediateEvidenceOf({ checks, applied, reasonKind }); +export function immediateReportOf({ asset, sessionId, turn, reasonKind, checks }) { + const evidence = immediateEvidenceOf({ checks, reasonKind }); return reportOf({ assetId: asset.assetId, taskId: turnTaskId(sessionId, turn), outcome: immediateOutcomeOf(evidence, reasonKind), - reason: `Injected by Evolver into dsh turn ${turn}; applied: ${applied}; ${checkPhrase(evidence.check)}.`, + reason: `Injected by Evolver into dsh turn ${turn}; ${checkPhrase(evidence.check)}.`, evidence, }); } diff --git a/src/session-distill.js b/src/session-distill.js index fd2578e..e21395b 100644 --- a/src/session-distill.js +++ b/src/session-distill.js @@ -26,16 +26,15 @@ const MAX_ATTEMPTS = 3; const pendingPath = () => statePath('distill-pending.json'); /** - * Why a finished turn leaves no candidate, or null when it does. A turn that reused an injected strategy - * adds evidence to that asset through its reuse report rather than a near-copy of it; a chat reply without a - * single command has no method to keep. + * Why a finished turn leaves no candidate, or null when it does. A chat reply without a single command has no + * method to keep. A turn that reused an injected strategy is distilled too: the engine's intake drops a Gene + * that repeats one it already has. */ -export function skipReason(reasonKind, record, { reusedAsset = false } = {}) { +export function skipReason(reasonKind, record) { if (reasonKind !== 'completed') return 'not_completed'; if (!record.answer) return 'no_answer'; if (record.checks.at(-1)?.failed) return 'failed_check'; if (record.commands.length === 0 && record.edits.length === 0) return 'no_tool_use'; - if (reusedAsset) return 'asset_reused'; return null; } diff --git a/src/turn-record.js b/src/turn-record.js index 5df9bbb..7402feb 100644 --- a/src/turn-record.js +++ b/src/turn-record.js @@ -3,16 +3,15 @@ import path from 'node:path'; -import { editedContent, editedPath } from './edited-content.js'; +import { editedPath } from './edited-content.js'; import { contentHashOf } from './retention-pending.js'; import { checkCommandOf, verificationFailed } from './verification.js'; -const TURN_EDITED_MAX_CHARS = 200_000; const CHECK_COMMAND_MAX_CHARS = 200; const PATH_KEYS = ['path', 'file_path']; function freshTurnRecord() { - return { checks: [], commands: [], paths: [], edits: [], editedChars: 0, editedBefore: new Map(), answer: '' }; + return { checks: [], commands: [], paths: [], edits: [], editedBefore: new Map(), answer: '' }; } export function createTurnRecords() { @@ -44,11 +43,9 @@ export function noteFileBeforeEdit(record, projectDir, args) { if (!record.editedBefore.has(file)) record.editedBefore.set(file, contentHashOf(file)); } -function noteEditedContent(record, args) { - const kept = editedContent(args).slice(0, TURN_EDITED_MAX_CHARS - record.editedChars); - if (!kept) return; - record.edits.push(kept); - record.editedChars += kept.length; +function noteEditedFile(record, args) { + const edited = editedPath(args); + if (edited) record.edits.push(edited); } export function noteToolResult(record, exec, result, { isEdit }) { @@ -57,7 +54,7 @@ export function noteToolResult(record, exec, result, { isEdit }) { for (const key of PATH_KEYS) if (typeof args[key] === 'string') record.paths.push(args[key]); const check = checkCommandOf(exec); if (check) record.checks.push({ command: check.slice(0, CHECK_COMMAND_MAX_CHARS), failed: verificationFailed(result) }); - if (isEdit && !result?.isError) noteEditedContent(record, args); + if (isEdit && !result?.isError) noteEditedFile(record, args); } export function noteAnswer(record, message) { diff --git a/test/applied.test.js b/test/applied.test.js deleted file mode 100644 index b9f1ef8..0000000 --- a/test/applied.test.js +++ /dev/null @@ -1,95 +0,0 @@ -import assert from 'node:assert/strict'; -import { readFileSync } from 'node:fs'; -import { test } from 'node:test'; - -import { appliedVerdictOf, concreteFeaturesOf, strategyStepsOf } from '../src/applied.js'; - -const STRATEGY = [ - '[Evolution Memory] Fix flaky Redis connection in tests (EvoMap network):', - '1. Set `maxRetriesPerRequest` to null in lib/redis.js', - '2. Run npm test to confirm the suite passes', - '3. Wrap the client in a singleton factory module', - '4. Close the connection in the global teardown hook', - '', - 'Apply them where they fit.', -].join('\n'); - -test('only the numbered steps of an injected strategy are matched', () => { - assert.equal(strategyStepsOf(STRATEGY).length, 4); -}); - -test('backticked names, paths and commands are concrete features', () => { - assert.deepEqual(concreteFeaturesOf('Set `maxRetriesPerRequest` to null in lib/redis.js'), ['maxretriesperrequest', 'lib/redis.js']); - assert.deepEqual(concreteFeaturesOf('Run npm test to confirm'), ['npm test']); -}); - -test('a turn that touched the named file and ran the named command used the strategy', () => { - const verdict = appliedVerdictOf({ - strategyText: STRATEGY, - commands: ['npm test'], - paths: ['/repo/lib/redis.js'], - answer: 'I wrapped the client in a singleton factory module.', - }); - assert.equal(verdict.verdict, 'used'); - assert.equal(verdict.hits, 3); - assert.equal(verdict.steps, 4); -}); - -test('one matching step out of four is partial', () => { - const verdict = appliedVerdictOf({ strategyText: STRATEGY, commands: ['npm test'], paths: [], answer: 'Done.' }); - assert.equal(verdict.verdict, 'partial'); - assert.equal(verdict.ratio, 0.25); -}); - -test('a turn sharing nothing with the strategy left it unused', () => { - const verdict = appliedVerdictOf({ strategyText: STRATEGY, commands: ['ls'], paths: ['/repo/README.md'], answer: 'Renamed the heading.' }); - assert.equal(verdict.verdict, 'unused'); - assert.equal(verdict.hits, 0); -}); - -test('generic words alone do not count as following a prose step', () => { - const prose = '1. Make sure you always check the result before you report the outcome'; - assert.equal(appliedVerdictOf({ strategyText: prose, commands: ['ls'], paths: [], answer: 'make sure always check before report outcome' }).verdict, 'unused'); -}); - -test('a strategy with no steps cannot be judged', () => { - assert.deepEqual(appliedVerdictOf({ strategyText: '', commands: ['npm test'], paths: [], answer: '' }), { verdict: 'unknown', hits: 0, steps: 0, ratio: 0 }); -}); - -const SESSION = JSON.parse(readFileSync(new URL('./fixtures/fair-play-session.json', import.meta.url), 'utf8')); - -function turnOf({ strategyText, tools, answer }) { - const args = tools.map((tool) => tool.arguments); - return { - strategyText, - commands: args.map((input) => input.command).filter((command) => typeof command === 'string'), - paths: args.flatMap((input) => [input.path, input.file_path]).filter((value) => typeof value === 'string'), - edits: tools.filter((tool) => ['write', 'edit'].includes(tool.name)).map((tool) => tool.arguments.content ?? tool.arguments.new_string), - answer, - }; -} - -const BUILD_TURN = turnOf(SESSION.turns[0]); - -test('a strategy carried out in the code the turn wrote is used', () => { - const verdict = appliedVerdictOf(BUILD_TURN); - assert.equal(verdict.verdict, 'used'); - assert.equal(verdict.steps, 4); - assert.ok(verdict.hits >= 3, `hits ${verdict.hits}`); -}); - -test('a prose word that reads like a command does not stop a step from matching on its terms', () => { - const step = '1. On get, move the accessed node to the head of the list to mark it as most recently used'; - const edits = ['moveToHead(node) { if (node === this.head) return } // most recently used']; - assert.equal(appliedVerdictOf({ strategyText: step, commands: [], paths: [], edits, answer: '' }).verdict, 'used'); -}); - -test('work that shares nothing with the strategy is unused', () => { - const verdict = appliedVerdictOf({ ...BUILD_TURN, edits: ['export const greeting = "hello";\n'], commands: ['ls'], paths: ['/tmp/x/greeting.js'], answer: '好了。' }); - assert.equal(verdict.verdict, 'unused'); -}); - -test('a turn that left only an answer, or a strategy without steps, cannot be judged', () => { - assert.equal(appliedVerdictOf(turnOf(SESSION.turns[1])).verdict, 'unknown'); - assert.equal(appliedVerdictOf({ strategyText: 'no numbered steps here', commands: ['npm test'], paths: [], edits: [], answer: '' }).verdict, 'unknown'); -}); diff --git a/test/reporting.test.js b/test/reporting.test.js index f17356d..3e1de3c 100644 --- a/test/reporting.test.js +++ b/test/reporting.test.js @@ -80,15 +80,15 @@ test('a recorded dsh session reports each turn with the evidence it left', async asset_id: 'sha256:872ad8edfa0d86d74140e386528acc4aee7812c946b9e3200c4464468586d0ea', outcome: 'success', task_id: 'dsh:session-fair-play:1', - reason: 'Injected by Evolver into dsh turn 1; applied: used; last test passed.', + reason: 'Injected by Evolver into dsh turn 1; last test passed.', evidence: { tier: 'L1', applied: 'used', check: { kind: 'test', passed: true, fail_then_pass: false }, aborted: false, stage: 'immediate' }, }); assert.deepEqual(explained, { asset_id: 'sha256:ed3b8595561c58c86c0c630e1c77f698a8a5b3e0e5d1aabd6acce1a8fa3a3750', outcome: 'success', task_id: 'dsh:session-fair-play:2', - reason: 'Injected by Evolver into dsh turn 2; applied: unknown; no check ran.', - evidence: { tier: 'none', applied: 'partial', aborted: false, stage: 'immediate' }, + reason: 'Injected by Evolver into dsh turn 2; no check ran.', + evidence: { tier: 'none', applied: 'used', aborted: false, stage: 'immediate' }, }); assert.deepEqual(injected, [], 'edits add nothing to the model context'); @@ -99,7 +99,7 @@ test('a recorded dsh session reports each turn with the evidence it left', async outcome: 'failed', task_id: 'dsh:session-fair-play:2:correction', reason: "The person's next prompt read as correction of the turn that reused this strategy.", - evidence: { tier: 'L3', applied: 'partial', stage: 'correction' }, + evidence: { tier: 'L3', applied: 'used', stage: 'correction' }, }); }); @@ -132,11 +132,11 @@ test('an interrupted turn whose last check failed reports the failure and the ab const [report] = reportsIn(proxy.requests); assert.equal(report.outcome, 'failed'); assert.deepEqual(report.evidence, { - tier: 'L1', applied: 'unused', check: { kind: 'test', passed: false, fail_then_pass: false }, aborted: true, stage: 'immediate', + tier: 'L1', applied: 'used', check: { kind: 'test', passed: false, fail_then_pass: false }, aborted: true, stage: 'immediate', }); }); -test('a turn that did nothing the strategy names reports it unused', async (t) => { +test('an injected strategy is reported as used, whatever the turn did with it', async (t) => { const { proxy, listeners } = await harness(t, 'evolver-unused', oneAsset('sha256:unused', [ 'Configure the Redis client with exponential reconnect backoff', 'Wrap the client in a singleton factory module', 'Close the connection in the global teardown hook', 'Assert reconnection inside the integration suite', ])); @@ -148,7 +148,7 @@ test('a turn that did nothing the strategy names reports it unused', async (t) = await untilRequest(proxy.requests, '/asset/reuse-result'); assert.deepEqual(reportsIn(proxy.requests)[0].evidence, { - tier: 'L1', applied: 'unused', check: { kind: 'test', passed: true, fail_then_pass: false }, aborted: false, stage: 'immediate', + tier: 'L1', applied: 'used', check: { kind: 'test', passed: true, fail_then_pass: false }, aborted: false, stage: 'immediate', }); }); @@ -216,7 +216,7 @@ for (const [verdict, change, outcome] of [ outcome, task_id: `dsh:session-${verdict}-a:1:retention`, reason: `By the next session the files the turn edited were ${verdict}.`, - evidence: { tier: 'L2', applied: 'unused', stage: 'retention' }, + evidence: { tier: 'L2', applied: 'used', stage: 'retention' }, }); listeners.get('agent/session-start')({ agent: fakeAgent({ sessionId: `session-${verdict}-c`, cwd: projectDir }).agent, source: 'startup' }); diff --git a/test/reuse.test.js b/test/reuse.test.js index c56e3f2..09f215d 100644 --- a/test/reuse.test.js +++ b/test/reuse.test.js @@ -38,14 +38,14 @@ function ledgers({ hub = { ok: true, data: {} }, local = true } = {}) { }; } -const immediate = (overrides = {}) => immediateReportOf({ asset: ASSET, sessionId: 's1', turn: 4, reasonKind: 'completed', checks: [], applied: 'used', ...overrides }); +const immediate = (overrides = {}) => immediateReportOf({ asset: ASSET, sessionId: 's1', turn: 4, reasonKind: 'completed', checks: [], ...overrides }); -test('a turn that ran no check still reports its injection, as tier none', () => { +test('a turn that ran no check still reports its injection as used, at tier none', () => { assert.deepEqual(immediate(), { asset_id: 'sha256:a', outcome: 'success', task_id: 'dsh:s1:4', - reason: 'Injected by Evolver into dsh turn 4; applied: used; no check ran.', + reason: 'Injected by Evolver into dsh turn 4; no check ran.', evidence: { tier: 'none', applied: 'used', aborted: false, stage: 'immediate' }, }); assert.equal(immediate({ reasonKind: 'error' }).outcome, 'failed'); @@ -57,10 +57,10 @@ test('a check the turn ran makes the report L1 and decides its outcome', () => { assert.deepEqual(passed.evidence.check, { kind: 'test', passed: true, fail_then_pass: false }); assert.equal(passed.evidence.tier, 'L1'); - const failed = immediate({ checks: [{ command: 'npx tsc -p .', failed: true }], applied: 'partial' }); + const failed = immediate({ checks: [{ command: 'npx tsc -p .', failed: true }] }); assert.equal(failed.outcome, 'failed'); assert.deepEqual(failed.evidence.check, { kind: 'typecheck', passed: false, fail_then_pass: false }); - assert.equal(failed.evidence.applied, 'partial'); + assert.equal(failed.evidence.applied, 'used'); }); test('the same check failing and then passing is recorded as such', () => { @@ -96,11 +96,11 @@ test('a correction or approval follow-up is L3 under the turn task id plus :corr }); test('kept or reverted edits are an L2 follow-up under :retention; anything else sends nothing', () => { - const reuse = reportedReuseOf(immediate({ applied: 'partial' })); + const reuse = reportedReuseOf(immediate()); const kept = retentionReportOf(reuse, 'kept'); assert.equal(kept.task_id, 'dsh:s1:4:retention'); assert.equal(kept.outcome, 'success'); - assert.deepEqual(kept.evidence, { tier: 'L2', applied: 'partial', stage: 'retention' }); + assert.deepEqual(kept.evidence, { tier: 'L2', applied: 'used', stage: 'retention' }); assert.equal(retentionReportOf(reuse, 'reverted').outcome, 'failed'); assert.equal(retentionReportOf(reuse, 'rewritten'), null); assert.equal(retentionReportOf(reuse, 'no_change'), null); @@ -212,13 +212,6 @@ test('the status a newer Proxy passes through is kept with the rejection', () => }); }); -test('a reuse that cannot be judged is exposure only, without the unused penalty', () => { - const report = immediate({ applied: 'unknown' }); - assert.deepEqual(report.evidence, { tier: 'none', applied: 'partial', aborted: false, stage: 'immediate' }); - assert.match(report.reason, /applied: unknown/); - assert.equal(immediate({ applied: 'unknown', checks: [{ command: 'npm test', failed: false }] }).evidence.tier, 'none'); -}); - test('the Hub verdict on a delivered report is kept with it and passed on', async () => { const sinks = ledgers({ hub: { ok: true, data: { recorded: true, eligible: true, counted: true, weight: 0.6, asset_id: 'sha256:a', reuse_stats: {} } } }); const outbox = await deliverPending(sinks, queuedOne(verified()), NOW); diff --git a/test/session-distill.test.js b/test/session-distill.test.js index 8e2fd6a..d8bfcde 100644 --- a/test/session-distill.test.js +++ b/test/session-distill.test.js @@ -43,13 +43,12 @@ function session(id) { return { id, header: {}, deriveMessages: () => [{ role: 'assistant', source: MODEL }] }; } -test('only a verified, completed turn that used tools and reused no strategy becomes a candidate', () => { +test('only a verified, completed turn that used tools becomes a candidate', () => { assert.equal(skipReason('completed', record()), null); assert.equal(skipReason('aborted', record()), 'not_completed'); assert.equal(skipReason('completed', record({ answer: '' })), 'no_answer'); assert.equal(skipReason('completed', record({ checks: [{ command: 'npm test', failed: true }] })), 'failed_check'); assert.equal(skipReason('completed', record({ commands: [] })), 'no_tool_use'); - assert.equal(skipReason('completed', record(), { reusedAsset: true }), 'asset_reused'); }); test('the prompt is a digest of the turn, never its whole history, and asks for no answer values', () => { @@ -236,6 +235,19 @@ test('a verified turn is distilled once, when the session ends, outside the conv assert.equal(proxy.requests.filter((request) => request.path === '/conversation/distill').length, 1); }); +test('a verified turn that reused an injected strategy is distilled too', async (t) => { + const reused = { asset_id: 'sha256:reused', short_title: 'Retry flaky uploads', similarity: 0.8, strategy: ['Run `npm test` to see the failure', 'Add a retry with backoff', 'Run npm test again', 'Say what changed'] }; + const { proxy, listeners, llm } = await harness(t, 'evolver-reused', proxyAnswer([reused])); + const agent = { ...fakeAgent({ sessionId: 'session-reused' }).agent, session: session('session-reused') }; + + await verifiedTurn(listeners, agent, 1); + await untilRequest(proxy.requests, '/asset/reuse-result'); + await listeners.get('session/flush')(agent.session); + await untilRequest(proxy.requests, '/conversation/distill'); + + assert.equal(llm.calls.length, 1); +}); + test('a turn the next prompt corrects is not distilled', async (t) => { const { proxy, listeners, llm } = await harness(t, 'evolver-corrected', proxyAnswer()); const agent = { ...fakeAgent({ sessionId: 'session-corrected' }).agent, session: session('session-corrected') };