diff --git a/.changeset/liveness-stale-evidence-fails-ci.md b/.changeset/liveness-stale-evidence-fails-ci.md new file mode 100644 index 0000000000..9d912099ed --- /dev/null +++ b/.changeset/liveness-stale-evidence-fails-ci.md @@ -0,0 +1,38 @@ +--- +"@objectstack/spec": patch +--- + +fix(spec): `check:liveness` 的 stale-evidence 从 `⚠` 升级为 `✗` 判红,摘要行不再把「声明数」当作「解析成功数」(#5623) + +`live` 判定的语义就是它的 evidence 指针 ——「这个属性有运行时消费者,在这里」。指针指向本仓已不存在的文件时,这条声明就不再可证伪:declared 有,enforced 无。而把它变成这样,只需要一次目录重组或一次改名。 + +实测(`origin/main`,把 `packages/spec/liveness/query.json` 的 5 条 evidence 路径指回迁移前的 `packages/plugins/driver-sql/...`): + +``` +evidence paths: 330 resolved against this checkout, 101 attributed to another repo (…) + +⚠ 5 'live' entr(ies) cite a missing file: + query/fields → packages/plugins/driver-sql/src/sql-driver.ts + … +``` + +退出码 **0**。五条断链逐条点名了,CI 一路绿灯 —— 而且摘要行里的 `330` 一动没动,因为它数的是 local 路径**总数**,不是解析成功数,"resolved" 这个词是失真的。 + +**为什么以前是 `⚠`,以及为什么现在不该是。** 这不是对本仓路径的有意宽容,是解析器时代的遗留:#3857 之前的检查取 `evidence.split(':')[0]` 当文件名,227 条里报 48 条、**全是误报**,那种情况下它当然不能判红。`evidence.mts` 修好了解析(realm marker 归属 + 仓根路径提取),此后这份清单一直是空的 —— 恰恰是这一点让「有命中」重新变成信号。当时被那 48 条噪声埋掉的唯一一条真腐烂(`object.enable.clone`,消费者从 `@objectstack/objectql` 搬到了 `@objectstack/metadata-protocol`),就是宽容的实际代价。 + +同族的旁证:同一目录下的 `check-empty-state.mts` 对 `rotted-evidence` 一直是 `✗` + exit 1,README 也写着它的执行点路径「resolves like `evidence` above, so a pointer that rots is reported rather than trusted」。真正有意的宽容在这个 gate 里都写明了理由(`verifiedAt` 的年龄、undrilled 容器计数、`PENDING_GOVERNANCE` —— 每一条都在代码里明说 "never fails CI"),stale-evidence 那一段一个字都没有。 + +**改动** + +1. **`live` 条目引用本仓缺失文件 → `✗` 判红**,并给出三条修法(仓内搬家就改指针并补 `verifiedAt`;搬去别的仓就加 realm marker;消费者真没了就按 ADR-0049 重新判定,而不是随手指向一个看着像的幸存者)。 + **边界不变**:cross-repo attribution(`objectui:` / `cloud:` / `packages/services/service-ai/…`,当前 101 条)只计数、不解析,永远不判红 —— `checkEvidence` 只对 local 桶做存在性检查,这个分界是结构性的,不是靠约定。 +2. **摘要行改成两个数**,而不是二选一: + ``` + evidence paths: 330 repo-local path(s) declared by 'live' entries, 330 resolved against this checkout; 101 attributed to another repo (…) + ``` + 两个数都留着是有原因的:"declared" 是 #3857 留下的解析健康度信号(有单测防止解析器退化成「什么都提不出来」而假绿),"resolved" 是判定。绿的时候两者相等 —— 这正是当初只印前一个会被读成通过的原因。有断链时行尾追加 `, N MISSING`。 +3. 新增 `--ledger-root=`:让 gate 读 `packages/spec/liveness` 的一份副本。自测据此把真 gate 跑在「只坏了一条指针」的 ledger 副本上,不改动仓库里任何文件。 + +`check-liveness.test.ts`(新增 8 例)直接 spawn 真脚本断言退出码 —— 这个 bug 从来不是「检查看不见」:它把五条全点名了还是 exit 0,所以只测 `checkEvidence` 的单测全程是绿的。 + +脚本本身(`scripts/`)不进 npm 包,但 `packages/spec/liveness/**` **是发布内容**(`package.json` 的 `files` 里有 `liveness`,`npm pack --dry-run` 实测 29 个文件入包,含该目录的 `README.md`)。一个新开始判红的 gate 必须同步它面向作者的文档,否则下一位作者只能靠撞红的 CI 才知道规则变了 —— 所以 README 里那段新的失败语义也随包发出,按 patch 记账,而不是空 frontmatter(空 changeset 是 `changesets/action` 的真实输入,全空集合会静默绿着卡住发布 —— #4898 卡住 17.0.0-rc.2 的就是这个)。当前 `main` 上 330 条本仓路径全部解析成功,该 gate 落地即绿。 diff --git a/packages/spec/liveness/README.md b/packages/spec/liveness/README.md index a34b56f0aa..df1129ff97 100644 --- a/packages/spec/liveness/README.md +++ b/packages/spec/liveness/README.md @@ -74,7 +74,18 @@ checkout. Prose around the paths is fine and encouraged — `packages/spec/src/s (mergeActionsIntoObjects stable-sorts each group)` resolves the path and ignores the parenthetical. -**A path in another repo must say so**, or the gate will report it as rot: +**A repo-local path that does not resolve FAILS CI** (`✗`, exit 1 — since #5623; +it was a non-failing `⚠` before, calibrated for the parser bug described in the +note below rather than for the parser we have). A `live` verdict *is* its +evidence pointer, so a pointer into thin air is a claim nothing can falsify — and +a directory move or a rename is the whole cost of getting there. Repair it, or, +if the consumer is genuinely gone, re-classify the property under ADR-0049 +enforce-or-remove instead of repointing at a plausible survivor. + +**A path in another repo must say so**, or the gate will report it as rot — +attributed paths are counted and never resolved, so they cannot fail this check +(that boundary is deliberate: ~101 entries cite the renderer repo or the closed +cloud runtime, whose files are legitimately absent here): ```jsonc "evidence": "objectui: packages/app-shell/src/views/RecordDetailView.tsx:573" diff --git a/packages/spec/scripts/liveness/check-liveness.mts b/packages/spec/scripts/liveness/check-liveness.mts index 2a0c8c33a1..079f4396ba 100644 --- a/packages/spec/scripts/liveness/check-liveness.mts +++ b/packages/spec/scripts/liveness/check-liveness.mts @@ -41,6 +41,17 @@ // (see proof-registry.mts), a `live` classification MUST carry a valid proof — // the file must exist and declare the `@proof: ` tag. CI fails otherwise. // +// EVIDENCE POINTERS (ADR-0087): a `live` verdict IS its evidence pointer — "this +// property has a runtime consumer, here it is". A cited path that is repo-rooted +// and attributed to THIS repo must resolve against this checkout, or CI fails: +// an unresolvable pointer makes the claim unfalsifiable, and a directory move or +// a rename is all it takes (see evidence.mts). Cross-repo attribution +// (`objectui: …`, `cloud: …`, `packages/services/service-ai/…`) is counted, not +// resolved — those files are legitimately absent here. This was a ⚠ until #5623, +// for one historical reason: the pre-#3857 `evidence.split(':')[0]` parser +// flagged 48 of 227 entries with a ~100% false-positive rate, so failing on it +// would have failed every build. The parse fix is what turned a hit into signal. +// // RE-VERIFICATION CLOCK (`verifiedAt`): a ledger entry is a claim with a // timestamp, and code moves under it in BOTH directions — `flow.status` (#3711) // and `action.undoable` (#3714) were both understated by entries that were @@ -56,6 +67,10 @@ // tsx check-liveness.mts --stale-verification # print the re-verification worklist // tsx check-liveness.mts --stale-verification=90 # ...with a custom staleness threshold // tsx check-liveness.mts --undrilled # print the undrilled-container worklist +// tsx check-liveness.mts --ledger-root= # read the ledgers from instead of +// # packages/spec/liveness — how the self-test +// # runs the REAL gate against a mutated copy +// # of the real ledgers without touching them process.env.OS_EAGER_SCHEMAS = '1'; @@ -93,7 +108,22 @@ import { const here = dirname(fileURLToPath(import.meta.url)); const specRoot = resolve(here, '../..'); // packages/spec const repoRoot = resolve(specRoot, '../..'); -const ledgerRoot = join(specRoot, 'liveness'); + +const args = process.argv.slice(2); + +// `--ledger-root=` points the walk at a COPY of packages/spec/liveness. +// It exists because the evidence gate below now fails the build, and a gate that +// fails is only worth as much as the proof that it fails — which needs a rotted +// pointer to fail ON. Committing one to a shipped ledger is not an option (it +// would fail every other run), and mutating a tracked file mid-test leaves the +// worktree dirty when the test crashes. So the self-test copies the real ledgers +// to a temp dir, breaks exactly one pointer there, and runs THIS script — same +// code path CI runs, no repo state touched. Everything else still resolves +// against the real repoRoot, so the resulting exit 1 has exactly one cause. +const ledgerRootArg = args.find((a) => a.startsWith('--ledger-root=')); +const ledgerRoot = ledgerRootArg + ? resolve(ledgerRootArg.slice('--ledger-root='.length)) + : join(specRoot, 'liveness'); // Governed metadata types, rolled out highest-frequency / highest-risk first. // (`query` is not a metadata type — see SPEC_ONLY_SCHEMAS below.) @@ -262,7 +292,6 @@ function loadLedger(type: string): any { } // ---- dump mode ---- -const args = process.argv.slice(2); const dumpIdx = args.indexOf('--dump'); if (dumpIdx !== -1) { const type = args[dumpIdx + 1]; @@ -300,8 +329,15 @@ const report: any = { deferredContainers: [] as string[], // containers whose subtree IS classified elsewhere — resolved, not believed deferredChildKeys: 0, // how many child keys those resolved deferrals actually cover verification: null as VerificationReport | null, // `verifiedAt` ages — the re-verification worklist - evidenceLocal: 0, // repo-rooted evidence paths actually resolved against this checkout - evidenceForeign: 0, // evidence paths attributed to objectui / cloud — not resolvable here + // The three evidence counters, and the distinction between the first two is + // the whole point: `evidenceLocal` is how many repo-rooted paths `live` entries + // DECLARE, `evidenceMissing` is how many of those do not exist here. The + // summary line used to print `evidenceLocal` under the word "resolved", so + // breaking five pointers left the count at 330 and the run still said + // "330 resolved" (#5623). Count and word now agree. + evidenceLocal: 0, // repo-rooted evidence paths attributed to THIS repo — DECLARED, not yet proven + evidenceMissing: 0, // ...of which this many do not exist here (=== staleEvidence.length) — FAILS the gate + evidenceForeign: 0, // evidence paths attributed to objectui / cloud — not resolvable here, never failed }; // Every classified entry, for the `verifiedAt` fold below. Collected during the @@ -324,6 +360,7 @@ function classify(type: string, path: string, status: string, led: any, cat: any const ev = checkEvidence(led.evidence, (p) => existsSync(join(repoRoot, p))); report.evidenceForeign += ev.foreign.length; report.evidenceLocal += ev.local.length; + report.evidenceMissing += ev.missing.length; for (const miss of ev.missing) report.staleEvidence.push(`${type}/${path} → ${miss}`); } // ── ADR-0054 prove-it-runs ── @@ -507,6 +544,11 @@ const totalProofFailures = report.proofErrors.length + report.proofMissing.lengt const failed = totalUnclassified > 0 || totalProofFailures > 0 || + // A `live` entry whose repo-local evidence path is gone. Red since #5623 — the + // ⚠ it replaces was calibrated for the false-positive era, not for the parser + // that now resolves 330 paths and reports zero. Cross-repo attribution never + // reaches this list: checkEvidence only resolves the LOCAL bucket. + report.staleEvidence.length > 0 || report.orphanEntries.length > 0 || report.verification.errors.length > 0 || report.ungoverned.length > 0 || @@ -524,13 +566,39 @@ if (asJson) { } const boundClasses = HIGH_RISK_CLASSES.filter((c) => c.bound).map((c) => c.label); console.log(`\nprove-it-runs (ADR-0054): proof REQUIRED for bound high-risk classes — ${boundClasses.join(', ') || 'none'}`); + // Two numbers, because one cannot carry both facts. "Declared" is the + // extraction-health signal (#3857's unit test guards it against degrading to + // "extracts nothing"); "resolved" is the verdict. They are equal on a green + // run, which is exactly why printing only the first read as a pass. console.log( - `\nevidence paths: ${report.evidenceLocal} resolved against this checkout, ` + - `${report.evidenceForeign} attributed to another repo (objectui / cloud — not resolvable here).`, + `\nevidence paths: ${report.evidenceLocal} repo-local path(s) declared by 'live' entries, ` + + `${report.evidenceLocal - report.evidenceMissing} resolved against this checkout` + + (report.evidenceMissing ? `, ${report.evidenceMissing} MISSING` : '') + + `; ${report.evidenceForeign} attributed to another repo (objectui / cloud — not resolvable here).`, ); if (report.staleEvidence.length) { - console.log(`\n⚠ ${report.staleEvidence.length} 'live' entr(ies) cite a missing file:`); + console.log(`\n✗ ${report.staleEvidence.length} 'live' entr(ies) cite a file that is missing from THIS repo:`); report.staleEvidence.forEach((s: string) => console.log(` ${s}`)); + console.log( + '\n A `live` verdict IS its evidence pointer — "this property has a runtime consumer,\n' + + ' here it is". When the cited file is gone from this checkout the claim is no longer\n' + + ' falsifiable: declared, but nothing enforces that anything still reads the property,\n' + + ' and a directory move or a rename is the whole cost of getting there.\n\n' + + ' Three repairs, and picking the right one is the work:\n' + + ' • the consumer MOVED inside this repo → repoint the path, and stamp `verifiedAt`\n' + + ' while you have the call graph open;\n' + + ' • the consumer moved to ANOTHER repo → say so with a realm marker\n' + + ' (`objectui: packages/app-shell/…`, `cloud: …`). Attributed paths are counted,\n' + + ' never resolved, and never fail here — that boundary is deliberate;\n' + + ' • the consumer is GONE → the verdict is not `live` any more. Re-classify under\n' + + ' ADR-0049 enforce-or-remove instead of repointing at a plausible survivor.\n\n' + + ' This was a ⚠ until #5623 for one reason: the pre-#3857 parser took\n' + + ' `evidence.split(":")[0]` as the filename, flagged 48 of 227 entries and every one\n' + + ' was a false positive — so it could not fail the build, and the one real rot it was\n' + + ' burying (`object.enable.clone`, whose consumer had moved packages) sat unread.\n' + + ' evidence.mts fixed the parse; the list has been empty since, which is what makes\n' + + ' a hit worth stopping for.', + ); } if (report.orphanProofs.length) { console.log(`\n⚠ ${report.orphanProofs.length} unregistered dogfood proof tag(s) — add to proof-registry.mts:`); @@ -659,7 +727,8 @@ if (asJson) { console.log( '\n✓ every governed-type property at the walk\'s one-level granularity is classified, every ' + 'registered type is governed or explicitly pending, no ledger row outlives its property, ' + - 'every container inheritance is declared, and all bound high-risk proofs resolve.', + 'every container inheritance is declared, every `live` entry\'s repo-local evidence path ' + + 'resolves, and all bound high-risk proofs resolve.', ); if (report.undrilledChildKeys) { console.log( diff --git a/packages/spec/scripts/liveness/check-liveness.test.ts b/packages/spec/scripts/liveness/check-liveness.test.ts new file mode 100644 index 0000000000..4bd5efe381 --- /dev/null +++ b/packages/spec/scripts/liveness/check-liveness.test.ts @@ -0,0 +1,183 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// Self-test for the liveness gate's EVIDENCE guard (#5623). +// +// WHY IT RUNS THE REAL SCRIPT. Everything this file asserts is a property of the +// gate as CI invokes it: which finding classes reach `process.exit(1)`, and what +// the summary line claims. Those two live in `check-liveness.mts` itself, not in +// a helper, and re-implementing the decision in a test would pin the copy rather +// than the gate — which is the exact failure #5623 reports one layer down. The +// bug was never that the check could not SEE the rot: it named all five rotted +// pointers, correctly, and exited 0 anyway. A test of `checkEvidence` (there is +// one, in evidence.test.ts) was therefore green throughout. +// +// So each case spawns `check-liveness.mts` the way `pnpm check:liveness` does and +// reads its exit code. `--ledger-root=` lets a case point the walk at a COPY +// of packages/spec/liveness with one pointer broken, so the run has exactly one +// cause for its verdict and no repo file is ever mutated — a crashed test leaves +// the worktree clean. Precedent for spawning a gate in its own test: +// scripts/check-generated-ledger.test.ts. + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { spawnSync } from 'node:child_process'; +import { createRequire } from 'node:module'; +import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const HERE = path.dirname(fileURLToPath(import.meta.url)); +const SPEC = path.resolve(HERE, '../..'); +const GATE = path.join(HERE, 'check-liveness.mts'); +const LEDGERS = path.join(SPEC, 'liveness'); + +// A repo-rooted path shaped exactly like a real pointer (so `evidence.mts` +// extracts it) that this repo has never contained. +const ROTTED = 'packages/plugins/driver-sql/src/sql-driver.ts'; + +function runGate(ledgerRoot?: string): { status: number | null; output: string } { + const require = createRequire(import.meta.url); + const tsx = require.resolve('tsx/cli'); + const argv = [tsx, GATE, ...(ledgerRoot ? [`--ledger-root=${ledgerRoot}`] : [])]; + const r = spawnSync(process.execPath, argv, { cwd: SPEC, encoding: 'utf8', maxBuffer: 32 * 1024 * 1024 }); + if (r.error) throw r.error; + return { status: r.status, output: `${r.stdout ?? ''}${r.stderr ?? ''}` }; +} + +/** Rewrite one property's `evidence` in a copied ledger. */ +function setEvidence(root: string, type: string, prop: string, evidence: string): void { + const file = path.join(root, `${type}.json`); + const ledger = JSON.parse(readFileSync(file, 'utf8')); + ledger.props[prop].evidence = evidence; + writeFileSync(file, `${JSON.stringify(ledger, null, 2)}\n`); +} + +function summaryLine(output: string): string { + return output.split('\n').find((l) => l.startsWith('evidence paths:')) ?? ''; +} + +describe('check:liveness — evidence pointers (#5623)', () => { + let tmp: string; + + beforeAll(() => { + tmp = mkdtempSync(path.join(tmpdir(), 'os-liveness-')); + cpSync(LEDGERS, path.join(tmp, 'liveness'), { recursive: true }); + }); + afterAll(() => rmSync(tmp, { recursive: true, force: true })); + + // The control. Without it, every "exit 1" below would also be satisfied by the + // copy simply not being readable. + it('is green against a verbatim copy of the shipped ledgers', () => { + const { status, output } = runGate(path.join(tmp, 'liveness')); + expect(status, output).toBe(0); + expect(output).toContain('✓ every governed-type property'); + }); + + it('FAILS when a `live` entry cites a repo-local file that is gone', () => { + const root = path.join(tmp, 'broken-local'); + cpSync(LEDGERS, root, { recursive: true }); + setEvidence(root, 'query', 'limit', `${ROTTED}:1345`); + + const { status, output } = runGate(root); + // The regression this pins: before #5623 this run printed the finding and + // exited 0, so a directory move could rot an ADR-0087 evidence chain with + // nothing in CI to notice. + expect(status, output).toBe(1); + expect(output).toContain("'live' entr(ies) cite a file that is missing from THIS repo"); + expect(output).toContain(`query/limit → ${ROTTED}`); + // ✗, not ⚠ — the grading is the fix, and the two are one character apart. + expect(output).toMatch(/✗ 1 'live' entr\(ies\) cite a file/); + expect(output).not.toMatch(/⚠ \d+ 'live' entr\(ies\)/); + }); + + it('names EVERY rotted pointer, not just the first', () => { + const root = path.join(tmp, 'broken-many'); + cpSync(LEDGERS, root, { recursive: true }); + for (const prop of ['fields', 'where', 'orderBy', 'limit', 'offset']) { + setEvidence(root, 'query', prop, `${ROTTED} (rotted by the self-test)`); + } + const { status, output } = runGate(root); + expect(status, output).toBe(1); + expect(output).toMatch(/✗ 5 'live' entr\(ies\) cite a file/); + for (const prop of ['fields', 'where', 'orderBy', 'limit', 'offset']) { + expect(output).toContain(`query/${prop} → ${ROTTED}`); + } + }); + + // THE BOUNDARY. Tightening the local case must not drag the ~101 cross-repo + // attributions in with it: those files are legitimately absent from this + // checkout, and failing on them would make the gate unsatisfiable for every + // property whose consumer is the renderer or the closed cloud runtime. + it('stays green when the missing path is attributed to ANOTHER repo', () => { + const root = path.join(tmp, 'foreign'); + cpSync(LEDGERS, root, { recursive: true }); + setEvidence(root, 'query', 'limit', 'objectui: packages/app-shell/src/no-such-file.tsx:12'); + setEvidence(root, 'query', 'offset', 'cloud: packages/ee-runtime/src/also-not-here.ts:9'); + // The closed cloud runtime is repo-ROOTED and still foreign — the shape that + // would break first if the boundary were drawn on the path text alone. + setEvidence(root, 'query', 'orderBy', 'packages/services/service-ai/src/nope.ts:1'); + + const { status, output } = runGate(root); + expect(status, output).toBe(0); + expect(output).not.toContain('cite a file that is missing'); + }); + + it('still fails a local path that shares a string with a foreign clause', () => { + // A realm marker's scope ends at the clause boundary. If it did not, one + // `objectui:` anywhere in an entry would silence the whole entry — a + // one-token opt-out of the gate. + const root = path.join(tmp, 'mixed'); + cpSync(LEDGERS, root, { recursive: true }); + setEvidence(root, 'query', 'limit', `objectui: packages/app-shell/src/x.tsx; ${ROTTED}:1345`); + + const { status, output } = runGate(root); + expect(status, output).toBe(1); + expect(output).toContain(`query/limit → ${ROTTED}`); + }); +}); + +describe('check:liveness — the evidence summary line (#5623)', () => { + let tmp: string; + + beforeAll(() => { + tmp = mkdtempSync(path.join(tmpdir(), 'os-liveness-sum-')); + }); + afterAll(() => rmSync(tmp, { recursive: true, force: true })); + + it('prints declared and resolved as separate numbers, equal on a green run', () => { + const { status, output } = runGate(); + expect(status, output).toBe(0); + const line = summaryLine(output); + const m = line.match(/^evidence paths: (\d+) repo-local path\(s\) declared by 'live' entries, (\d+) resolved/); + expect(m, line).not.toBeNull(); + expect(m![1]).toBe(m![2]); + // Guards the same degradation evidence.test.ts guards: a parser that extracts + // nothing would make "declared === resolved" vacuously true. + expect(Number(m![1])).toBeGreaterThan(100); + expect(line).not.toContain('MISSING'); + }); + + it('MOVES the resolved count when a pointer rots — the mis-labelled count of #5623', () => { + const root = path.join(tmp, 'liveness'); + cpSync(LEDGERS, root, { recursive: true }); + const green = summaryLine(runGate(root).output); + const declared = Number(green.match(/: (\d+) repo-local/)![1]); + + setEvidence(root, 'query', 'limit', `${ROTTED}:1345`); + const red = summaryLine(runGate(root).output); + + // `declared` is unchanged — the pointer is still DECLARED, it just does not + // resolve. That is precisely why the old line, which printed this number + // under the word "resolved", read 330 both before and after five pointers + // were broken. + expect(red).toContain(`${declared} repo-local path(s) declared`); + expect(red).toContain(`${declared - 1} resolved against this checkout`); + expect(red).toContain('1 MISSING'); + }); + + it('reports foreign attributions separately and never as missing', () => { + const { output } = runGate(); + const line = summaryLine(output); + expect(line).toMatch(/; \d+ attributed to another repo \(objectui \/ cloud — not resolvable here\)\./); + }); +});