diff --git a/.abcd/development/brief/02-constraints/04-ethics.md b/.abcd/development/brief/02-constraints/04-ethics.md index 381c230..8b642df 100644 --- a/.abcd/development/brief/02-constraints/04-ethics.md +++ b/.abcd/development/brief/02-constraints/04-ethics.md @@ -24,8 +24,14 @@ The sensitive artefacts — a participant's voice and screen — stay on local hardware. ASR is local; raw audio/video never leave the machine. Only derived text (transcript, serialised events, and any keyframes the analyst explicitly -releases) reaches a cloud LLM; a fully local variant (local LLM for analysis) -is the fallback if an ethics protocol requires it. +releases) reaches an LLM at all. Whether that LLM is in the cloud is the +operator's choice, not the tool's: the oracle is host-delegated, so `analyze` +emits a request and something else answers it. Answer it on the same machine and +no session content leaves the machine at any step — the fallback an ethics +protocol requires. Which route a session took is **recorded rather than +assumed**: `analyze -ingest` writes the operator's declared backend and model +into `findings.jsonl` and `report.md` prints it, with the standing caveat that +the CLI cannot verify a claim about a program it never called. For sessions with external participants: diff --git a/.abcd/development/brief/04-surfaces/06-analyze.md b/.abcd/development/brief/04-surfaces/06-analyze.md index 567be87..a84804f 100644 --- a/.abcd/development/brief/04-surfaces/06-analyze.md +++ b/.abcd/development/brief/04-surfaces/06-analyze.md @@ -18,9 +18,15 @@ page ([`../05-internals/02-schemas.md`](../05-internals/02-schemas.md)). | `-session` | (required) | session directory | | `-out` | *(stdout)* | write the emitted request to `FILE` instead of stdout (emit mode) | | `-ingest` | *(off)* | validate the answer JSON at `FILE` (or `-` for stdin) into `findings.jsonl` (ingest mode) | +| `-backend` | *(unrecorded)* | record which backend answered the request: `local` or `cloud` (ingest mode) | +| `-model` | *(not recorded)* | record the model that answered the request; free text, at most 200 characters, refused when it renders as nothing under `session.CodeRendersEmpty` (ingest mode) | `analyze` runs in exactly one mode: emit (no `-ingest`) or ingest (`-ingest`). -`-out` and `-ingest` together is an error. Emit reads `manifest.json` and +`-out` and `-ingest` together is an error. `-backend` and `-model` belong to +ingest alone — emit mutates nothing, so there is nothing to record against — and +either in emit mode is a usage error; so is `-model` without `-backend`, and so +is `-backend unrecorded`, which is what the flag's absence records rather than a +value an operator states. Emit reads `manifest.json` and `timeline.jsonl`; ingest reads `timeline.jsonl` only. Both hint to run `merge` first when the timeline is missing (matching [`report`](04-report.md)). @@ -70,6 +76,28 @@ first when the timeline is missing (matching [`report`](04-report.md)). - An answer with no findings (a bare `[]`, `{"findings":[]}`, or a truncated file) is refused rather than written: the write truncates, so an empty answer would otherwise erase a prior good `findings.jsonl` and report success. +- Every ingest writes one **provenance record** (`kind:"provenance"`) as the + FIRST line of `findings.jsonl`, in the same `session.CommitRecords` call as the + findings: the rubric version (from the package constant, not the answer's + claim), the backend, the model when given, and the date. First position because + ingest replaces the whole file while `review` appends verdicts to its end, so a + last-position record would be overtaken by the first verdict; riding in the + same commit is what makes a re-ingest replace the declaration together with the + findings it describes. The verdict-overwrite guard is untouched and outranks + it — a file holding verdicts refuses a re-ingest whatever the flags say. +- The record is the operator's **declaration**, not a measurement: the CLI never + calls a model and cannot observe where the request ran. With no `-backend` the + record states `unrecorded` and the run announces the intention on stderr, so + the choice is visible in the output of the run that made it rather than + silently absent; the notice says "will record" because it prints before + validation, and a run that then fails writes nothing. +- `Ingest` refuses a `Provenance` that is not one (`Provenance.Valid`: the kind + literal, a backend in the closed set, a non-empty rubric and date) before it + reads a byte, so the package cannot write a first line its own `ParseRecords` + would refuse. The only way to satisfy the check is to have built the record + through `NewProvenance`. + A `findings.jsonl` written before the record existed carries none and reads as + "not recorded" everywhere. ## Deferred diff --git a/.abcd/development/brief/05-internals/02-schemas.md b/.abcd/development/brief/05-internals/02-schemas.md index 91df9d1..c16a2cf 100644 --- a/.abcd/development/brief/05-internals/02-schemas.md +++ b/.abcd/development/brief/05-internals/02-schemas.md @@ -14,7 +14,7 @@ sessions// interactions.jsonl # normalised interaction events (epoch ms) transcript.jsonl # word-aligned utterances (session-relative seconds) timeline.jsonl # merged, session-relative timeline - findings.jsonl # analysis findings + appended verdicts (written by analyze/review) + findings.jsonl # provenance + analysis findings + appended verdicts (written by analyze/review) tests.jsonl # regression-test drafts + appended decisions (written by draft-tests/review) report.md # human-readable session report ``` @@ -93,11 +93,12 @@ file. {"t":129.01,"src":"event","id":"ev-001","payload":{"kind":"click","selector":"[data-testid=save-btn]","text":"Save","route":"/settings"}} ``` -## `findings.jsonl` — findings plus appended verdicts +## `findings.jsonl` — provenance, findings, plus appended verdicts The analysis layer's output, written by [`analyze -ingest`](../04-surfaces/06-analyze.md) -and [`review`](../04-surfaces/07-review.md). Two record kinds share the file, one -per line. A finding line carries no `kind`; a verdict line is discriminated by +and [`review`](../04-surfaces/07-review.md). Three record kinds share the file, +one per line. A provenance line is discriminated by `kind: "provenance"`; a +finding line carries no `kind`; a verdict line is discriminated by `kind: "verdict"`. Verdicts are **appended, never in-place rewrites**, so the finding's birth state and full decision history survive as the precision measure ([note §2](../../research/2026-07-17-architecture-note.md)). Ingest decodes each @@ -105,6 +106,33 @@ finding with unknown fields disallowed — the shape is closed — and is the so validation boundary; every field below is checked, and `status` is forced to `"unverified"` on ingest whatever the answer JSON claims. +**Provenance record** (`analyze.Provenance`): + +The operator's declaration of what answered the analysis request, written by +ingest as the **first** line of the file, in the same `session.CommitRecords` +call as the findings. It is a declaration, not a measurement: the CLI never +calls a model and cannot observe where the request ran. Exactly one per file. + +| Field | Type | Required | Notes | +|---|---|---|---| +| `kind` | string | yes | literal `"provenance"` — the discriminator | +| `rubric` | string | yes | the rubric version ingest enforced, from the package constant — never the answer's claimed rubric, which may be absent entirely (a bare-array answer) | +| `backend` | string | yes | one of `local \| cloud \| unrecorded`; `unrecorded` is written when no `-backend` is given and is not claimable from the flag | +| `model` | string | no | operator free text, at most `analyze.MaxModelLength` (200) runes, non-blank once `session.SafeText` is applied; omitted when not given | +| `at` | string | yes | ISO date `YYYY-MM-DD`, supplied by the caller (`NewProvenance`), never `time.Now()` inside the package | + +```json +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b","at":"2026-09-15"} +``` + +First position is a writer convention, not a reader requirement — `ParseRecords` +reads the record wherever it sits. A record whose `backend` falls outside the +closed set is **ignored**, as an out-of-enum verdict is, so an uninterpretable +claim never reaches the report; two interpretable records is a **hard error** +naming both lines, as a duplicate finding id is, because an ambiguous +attribution would have the report state a producer that may not be the one. A +file carrying no record reads as "not recorded" everywhere. + **Finding record** (`analyze.Finding`): | Field | Type | Required | Notes | diff --git a/.abcd/development/brief/06-delivery/02-verification.md b/.abcd/development/brief/06-delivery/02-verification.md index 3f31226..fa8c17e 100644 --- a/.abcd/development/brief/06-delivery/02-verification.md +++ b/.abcd/development/brief/06-delivery/02-verification.md @@ -45,10 +45,16 @@ go test -race ./... The pipeline smoke test asserts that `timeline.jsonl` and `report.md` are non-empty and that the report renders the sample session's fixed content: the -`## Timeline` and `## Findings` headings, the confirmed `F-001` finding, the -"save button" utterance text, the `save-btn` selector, the exact -`**Utterances:** 10 · **Events:** 10` header count, and one indented event -bullet naming that same selector. The header count is what catches events +`## Timeline` and `## Findings` headings, the `_Provenance` line, the confirmed +`F-001` finding, the "save button" utterance text, the `save-btn` selector, the +exact `**Utterances:** 10 · **Events:** 10` header count, and one indented event +bullet naming that same selector. The provenance assertion pins the line's +presence, not a particular backend: the bundled sample declares none — its +findings were hand-authored for the repository rather than produced by any model, +and a sample claiming `local` would plant a false provenance claim in the one +artefact users copy from — so the grep catches the record being dropped by the +reader or the renderer, which would silently strip the one statement the report +makes about its own origin. The header count is what catches events going missing from the merge: every other assertion up to it still passes with `interactions.jsonl` deleted ("save button" comes from the utterance's own text, and the `save-btn` selector renders from `findings.jsonl` regardless of diff --git a/.abcd/development/intents/drafts/itd-8-local-analysis.md b/.abcd/development/intents/drafts/itd-8-local-analysis.md deleted file mode 100644 index 0cdc656..0000000 --- a/.abcd/development/intents/drafts/itd-8-local-analysis.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -id: itd-8 -slug: local-analysis -spec_id: null -kind: null -suggested_kind: null -reclassification_history: [] -builds_on: [] -severity: major ---- - -# Analysis That Never Leaves the Machine - -## Press Release - -> **Testimony can run its analysis pass entirely on local hardware.** A flag on the analysis surface points the same versioned rubric at a locally hosted model instead of a cloud one, producing the same `findings.jsonl` — same schema, same typing, same `status: unverified` birth state — with no session text leaving the machine. Voice and screen were already local; with this, so is every derived artefact, and the privacy boundary moves from "only derived text leaves" to "nothing leaves". -> -> "The ethics application asks where the data goes, and 'nowhere' is a much shorter answer than a paragraph about derived text," said Carol, session moderator. "Being able to say the whole pipeline runs on one machine in a locked office is what got the protocol through." - -## Why This Matters - -The local-processing boundary is the pipeline's strongest privacy property and, as the ethics constraints record, the strongest card in a research-ethics application. Today that boundary sits between raw media and derived text: audio and video stay put, but transcript and events reach a cloud model. For most sessions that is a defensible line. For a protocol that will not permit it — or a participant population where it cannot be justified — the whole pipeline becomes unusable at the last step, after all the local capture and transcription work is already done. - -Making this a flag on the existing surface rather than a separate mode is deliberate. One analysis code path, one rubric, one findings schema means a local-analysed session is comparable with a cloud-analysed one, and the retained human verdicts measure both against the same bar. The operator chooses per session; the artefacts do not fork. - -## What's In Scope - -- A flag on the analysis surface selecting a locally hosted model backend, with the cloud path remaining the default. -- Identical output contract: the same rubric version, the same findings schema, and `status: unverified` on every machine-generated finding regardless of backend. -- Recording which backend and model produced a session's findings, so the provenance of a verdict is auditable and the two backends' precision can be compared over time. -- A stated, documented quality floor below which a local backend is not fit for the second-coder role. - -## What's Out of Scope - -- Local transcription, which is already local by design and is not part of this choice. -- Shipping, vendoring, or managing a local model — the flag points at a backend the operator has already stood up. -- Guaranteeing parity of finding quality between backends; the retained verdicts measure the gap rather than hiding it. -- A local backend for the codebase-mapping step (itd-3), which is a separate agentic surface. - -## Acceptance Criteria - -- **Given** a merged timeline and a local backend configured, **when** the analysis pass runs with the local flag, **then** `findings.jsonl` is produced under the same rubric and schema as a cloud run, with no session text sent off the machine. -- **Given** a findings file produced by either backend, **when** it is read, **then** the backend and model that produced it are recorded alongside the findings. -- **Given** the local flag is not passed, **when** the analysis pass runs, **then** behaviour is unchanged from today. - -## Open Questions - -- What is the acceptable quality floor for a local model in the second-coder role, and how is it measured — retained verdict precision against a cloud-analysed control set? -- Can a local model sustain the two-pass structure (segment coding then session-level synthesis), or does synthesis degrade first and need a smaller chunk size? -- Does the local path need its own rubric phrasing to survive a smaller context window, and if so, does that break comparability with cloud runs? - -## Audit Notes - -_Empty. Populated by intent-fidelity-reviewer when intent moves to shipped/._ diff --git a/.abcd/development/intents/shipped/itd-8-local-analysis.md b/.abcd/development/intents/shipped/itd-8-local-analysis.md new file mode 100644 index 0000000..39bb488 --- /dev/null +++ b/.abcd/development/intents/shipped/itd-8-local-analysis.md @@ -0,0 +1,175 @@ +--- +id: itd-8 +slug: local-analysis +spec_id: spc-2609150759135349 +kind: standalone +suggested_kind: null +reclassification_history: [] +builds_on: [] +severity: major +--- + +# A Findings File That Says Where It Was Analysed + +## Press Release + +> **Testimony records where a session's analysis happened.** `analyze -ingest` takes `-backend local|cloud` and an optional `-model NAME`, and writes that declaration into the session as the first line of `findings.jsonl`, beside the findings it accompanies — so a findings file always states which rubric coded it, on which side of the machine boundary, with which model, and on what date. `report` prints the declaration under its Findings heading, so the artefact people actually read carries the claim too. And the documentation sets out the fully local route end to end: emit the request, run it against a model hosted on the same machine, ingest the answer with `-backend local` — at which point no session content has left the machine at any step, and the findings file says so. Passing neither flag changes nothing but what is recorded: the file states that the backend was not recorded, and the run says so as it goes. +> +> "The ethics committee asks where the data goes, and 'nowhere' is a much shorter answer than a paragraph about derived text," said Carol, session moderator. "But what got the protocol through was not the promise. It was being able to hand them a findings file that names the model which produced it and says it ran on the machine in our locked office." + +## Why This Matters + +The local-processing boundary is the pipeline's strongest privacy property and, as the ethics constraints record, the strongest card in a research-ethics application. This intent was first drafted as a flag that would point the rubric at a locally hosted model. There is no such flag to build. The repository's oracle is host-delegated by design — the CLI never calls a model, holds no keys, and adds no network dependency — so there is no backend for a flag to select, and a flag pretending to select one would be the first lie the tool ever told about itself. What the pipeline can honestly offer is the other half of the same promise. The operator already chooses where the emitted request runs; what is missing is any trace of that choice in the evidence. + +That absence is the real gap. An analysis run against a model on the operator's own machine and one run against a cloud model produce byte-identical findings files. Six months later, when a supervisor, a co-author, or an ethics reviewer asks which it was, the answer rests on somebody's memory of a shell session. Recording the operator's declaration at the moment of ingest turns it into evidence that travels with the findings: it survives the session directory being archived, copied, or handed to a colleague, and it is on the page of the report rather than in the head of the person who ran it. The record is a declaration, not a measurement — Testimony has no way to observe where the request ran, and says so plainly wherever the claim appears. What it does guarantee is that the claim is written down, at the moment it is made, by the person who made it. + +The original draft also asked for a stated quality floor below which a local backend is not fit for the second-coder role. That is deliberately deferred rather than built. The pipeline already carries the instrument that answers it: every finding is born `unverified`, and `review` retains each `confirmed | rejected | duplicate` verdict as an ongoing precision measure. Once findings files name their backend and model, the verdicts already accumulating on disk become a comparison between backends — measured from real sessions, not asserted in advance. A floor written into the tool today would be a number nobody had measured, and enforcing it would mean refusing a run on the strength of a claim the CLI cannot check. The retained verdicts are the honest instrument; this intent makes their results attributable. + +## What's In Scope + +- `-backend local|cloud` and an optional `-model NAME` on `analyze -ingest`, written into the session as a single provenance record beside the findings the same run validates. +- `report` rendering that declaration under its Findings heading, so the shareable artefact carries it. +- A `findings.jsonl` written before this change — one carrying no provenance record — still loading normally everywhere, and reading as "not recorded" in the report. +- Documentation of the fully local workflow end to end: emit the request, run it against a locally hosted model, ingest with local provenance; and a privacy statement that no session content leaves the machine, stated as conditional on that workflow. +- Unchanged behaviour when neither flag is given, apart from what the run records and announces. + +## What's Out of Scope + +- Any change to how the model work is done. The CLI still never calls a model, holds no keys, and adds no network dependency; no flag selects a backend, because there is no backend in the CLI for a flag to select. +- Verifying the declaration. Testimony records what the operator states and cannot observe where the request actually ran; every surface that renders the record says so. +- Shipping, vendoring, or managing a local model, or advising which one to run. +- A stated quality floor for a local backend, deferred to the retained verdicts as set out above. +- Recording a hostname, address, or machine name. A session directory is an exchange unit; the record carries only what is safe to share. +- The same provenance question for the drafting layer's `tests.jsonl`, which has its own rubric and its own record, and for the codebase-mapping step (itd-3). +- Local transcription, which is already local by design and is not part of this choice. + +## Scope Conditions + +- The operator has already stood up whatever answers the emitted request; Testimony neither installs, configures, nor reaches it. +- `analyze -ingest` is the only writer of the provenance record, so a `findings.jsonl` assembled by hand or by another tool may carry none and is read as "not recorded" rather than refused. +- The declaration is taken on trust: the CLI cannot observe where the request ran, so the record's worth rests on the operator being the person who ran it. +- The session directory remains an exchange unit, so the record holds only a backend word, a model name, a rubric version, and a date. + +## Acceptance Criteria + +- **Given** a session with a merged timeline and a clean answer JSON, **when** `analyze -ingest FILE -backend local -model NAME` runs, **then** the first line of `findings.jsonl` is a provenance record naming the rubric version, the backend `local`, the model `NAME`, and the date, and the finding lines that follow are byte-for-byte the lines a run without those flags would have written. +- **Given** a `findings.jsonl` carrying a provenance record, **when** `report` runs, **then** the Findings section states the backend, the model, the rubric version and the date on one line before the status groups, and notes that the backend is the operator's declaration. +- **Given** a `findings.jsonl` written before this change, carrying no provenance record, **when** `report` and `review` read it, **then** both load it unchanged and the report's Findings section states that the provenance is not recorded. +- **Given** an `analyze -ingest` run with neither `-backend` nor `-model`, **when** it completes, **then** the finding lines written are unchanged from today's, the provenance record states that the backend was not recorded, and the command names that choice on stderr rather than recording it silently. +- **Given** a `findings.jsonl` that already holds verdict records, **when** `analyze -ingest` runs with any combination of the provenance flags, **then** the run is refused with the existing message and neither the verdicts nor the existing provenance record is rewritten. +- **Given** an operator following the documented local route, **when** they read the how-to, **then** it names every command in the route in order — emit, run against the locally hosted model, ingest with `-backend local`, report — and the privacy page states the "no session content leaves the machine" conclusion explicitly as conditional on that route. + +## Open Questions + +- Should the runner that answered the request — an agent CLI, a local serving tool, a colleague — be a field of its own, or does a free-text `-model` carry it well enough? A field named for a host invites an address, which is exactly what a shared session directory must not carry. +- Should `tests.jsonl` carry the same record for the drafting step, and if so does it share this record's shape or keep its own? +- Once several sessions carry a backend and a model, what is the smallest honest comparison the retained verdicts support — a per-backend confirm rate, or something that accounts for the analyst having seen the findings in a different order? +- Should a later revision make `-backend` required when ingesting, once no script depends on today's default, and what deprecation would that need? + +## Audit Notes + + +Fidelity review — receipt rcp-234149889a14 (verifier abcd:intent-auditor claude-opus-5[1m]). + +Provenance: abcd:intent-auditor@claude-opus-5[1m] · rubric_hash sha256:ab4fb122d86a2334f7bd3e84cc059514ada55bf7d42852a6152117a979a9eac6 · prompt_hash sha256:8ae89997220e872a3ec8025bf57338f8b193c79a6b957faaf3694f93cc94e97d +Input attestations: diff:HEAD..working tree (worktree .abcd/.work.local/worktrees/itd-8, branch feat/itd-8-analysis-provenance, HEAD 3142ecc)@sha256:fe12eb00736802f4897ef6b6b5c62e84f27bdfc18ca467a41fcb7a1106976201; + +Acceptance rollup: MET 6 · MET_WITH_CONCERNS 0 · NOT_MET 0 · INCONCLUSIVE 0 + +Per-criterion verdicts: +- ac-1 — MET: commitFindings prepends the marshalled Provenance record ahead of every finding in the same Records slice, the record carries rubric/backend/model/at, and TestIngestWritesProvenanceAsFirstLine asserts both the first-line shape and byte-identical finding lines against a run without the flags + evidence: internal/analyze/ingest.go:185 + evidence: internal/analyze/analyze.go:85 + evidence: internal/analyze/analyze_test.go:1250 + evidence: internal/cli/cli_test.go:1270 +- ac-2 — MET: renderProvenance is called directly after the Findings heading and before the status grouping, emitting one line carrying backend, model, rubric and ingest date prefixed "as declared at ingest"; the report test pins both the exact string and the heading < provenance < first-group ordering + evidence: internal/report/report.go:171 + evidence: internal/report/report.go:267 + evidence: internal/report/report_test.go:1042 + evidence: internal/report/report_test.go:1061 +- ac-3 — MET: ParseRecords returns a nil *Provenance when no provenance line is present rather than erroring, report renders "_Provenance: not recorded._" for that case, and review's walk over a provenance-free fixture is byte-identical to one carrying the record + evidence: internal/analyze/analyze.go:233 + evidence: internal/report/report.go:247 + evidence: internal/report/report_test.go:1078 + evidence: internal/review/review.go:128 + evidence: internal/review/review_test.go:1019 + evidence: internal/analyze/analyze_test.go:1384 +- ac-4 — MET: with no -backend the CLI prints the "recording the provenance as backend not recorded" notice to stderr before reading the answer and NewProvenance writes backend "unrecorded"; the CLI test asserts the notice is on stderr not stdout, that the record says unrecorded and carries no model key, and the analyze parity test shows the finding lines are unchanged + evidence: internal/cli/cli.go:475 + evidence: internal/analyze/analyze.go:152 + evidence: internal/cli/cli_test.go:1315 + evidence: internal/analyze/analyze_test.go:1283 +- ac-5 — MET: the verdict-overwrite guard is unchanged and runs inside commitFindings under the lock, independent of the provenance argument; TestIngestRefusesVerdictFileWithProvenanceFlags re-ingests a verdict-bearing file with different backend and model flags, gets the existing "refusing to overwrite" message, and asserts the file is byte-identical before and after, so the existing provenance line is untouched + evidence: internal/analyze/ingest.go:202 + evidence: internal/analyze/analyze_test.go:1329 +- ac-6 — MET: docs/how-to/analyse-locally.md walks the four steps in order — analyze -out, the local runner, analyze -ingest -backend local, report — and privacy.md now states the "no session content leaves the machine at any step" conclusion with the conditional attached to exactly that route + evidence: docs/how-to/analyse-locally.md:12 + evidence: docs/how-to/analyse-locally.md:19 + evidence: docs/how-to/analyse-locally.md:37 + evidence: docs/how-to/analyse-locally.md:48 + evidence: docs/how-to/analyse-locally.md:78 + evidence: docs/explanation/privacy.md:16 + +Gap audit: +- honoured: + - analyze -ingest takes -backend local|cloud and an optional -model NAME, and writes that declaration into the session as the first line of findings.jsonl + evidence: internal/cli/cli.go:398 + evidence: internal/analyze/ingest.go:185 + - report prints the declaration under its Findings heading, so the artefact people read carries the claim too + evidence: internal/report/report.go:171 + evidence: internal/report/report.go:242 + - the documentation sets out the fully local route end to end: emit the request, run it against a locally hosted model, ingest with -backend local + evidence: docs/how-to/analyse-locally.md:12 + evidence: docs/README.md:4 + - no change to how the model work is done; the CLI still never calls a model, holds no keys, and adds no network dependency, and no flag selects a backend + evidence: internal/analyze/analyze.go:152 + evidence: docs/reference/cli.md:202 + - every surface that renders the record says Testimony cannot verify where the request ran + evidence: internal/report/report.go:267 + evidence: docs/how-to/analyse-locally.md:100 + evidence: docs/reference/session-directory.md:136 + - the record carries only a backend word, a model name, a rubric version, and a date — no hostname, address, or machine name + evidence: internal/analyze/analyze.go:85 + evidence: docs/reference/session-directory.md:140 + - the same provenance question for the drafting layer's tests.jsonl is out of scope and stays out + evidence: internal/drafttests/drafttests.go:420 + evidence: internal/drafttests/drafttests_test.go:401 + - no stated quality floor for a local backend; the retained verdicts are left as the instrument + evidence: docs/how-to/analyse-locally.md:105 +- diverged: + - "Passing neither flag changes nothing but what is recorded" — in fact the ingest success line on stdout changed for every invocation, from "(all unverified)" to "(all unverified; backend not recorded)", so a caller parsing that documented line sees a different string even with no flags passed; the divergence is documented rather than hidden + evidence: internal/cli/cli.go:495 + evidence: internal/cli/cli.go:845 + evidence: docs/reference/cli.md:230 + - "a findings.jsonl assembled by hand or by another tool may carry none and is read as 'not recorded' rather than refused" — tolerance holds for zero records and for an uninterpretable backend, but a hand-assembled file carrying two interpretable provenance records is now a hard read error rather than a tolerated read + evidence: internal/analyze/analyze.go:320 + evidence: internal/analyze/analyze_test.go:1425 + evidence: docs/reference/session-directory.md:152 + - the committed example session's findings.jsonl gained a backend:unrecorded provenance line, so the shipped sample now declares an unrecorded backend rather than carrying no record at all + evidence: examples/sample-session/findings.jsonl:1 +- missing: (none) + +Scope-condition dispositions: +- cond-2609150759135103 — survived: the delivery adds no install, configuration, or network path to a model: the how-to hands request.txt to whatever the operator already runs, and NewProvenance only records a word the operator typed + evidence: docs/how-to/analyse-locally.md:37 + evidence: docs/how-to/analyse-locally.md:105 + evidence: internal/analyze/analyze.go:152 +- cond-2609150759137440 — narrowed: analyze -ingest is indeed the only writer and a file with no record reads as "not recorded", but the reader added a hard refusal for a file carrying two interpretable provenance records, so "read rather than refused" no longer holds for every hand-assembled file + narrowing: holds for a foreign findings.jsonl carrying zero provenance records or one whose backend is outside the closed set (both read as "not recorded"); a file carrying two interpretable provenance records is refused outright by ParseRecords, which fails report, review and draft-tests on that file + evidence: internal/analyze/analyze.go:320 + evidence: internal/analyze/analyze.go:297 + evidence: internal/report/report.go:247 + evidence: internal/analyze/analyze_test.go:1425 +- cond-2609150759138300 — survived: nothing in the delivery attempts to observe or verify the backend, and every rendering surface labels the record a declaration: the report line, the reference pages, the how-to, and the privacy explanation + evidence: internal/report/report.go:267 + evidence: docs/reference/cli.md:225 + evidence: docs/how-to/analyse-locally.md:100 + evidence: docs/explanation/privacy.md:20 +- cond-2609150759134348 — survived: the Provenance struct is closed at kind, rubric, backend, model and at — no host, address, path or machine field — and the model free text is bounded at 200 runes and sanitised at every sink + evidence: internal/analyze/analyze.go:85 + evidence: internal/analyze/analyze.go:47 + evidence: internal/cli/cli.go:845 + evidence: docs/reference/session-directory.md:140 +## Grounds + +- pursued: a findings file that states which backend and model produced it, on which side of the machine boundary, is what an ethics reviewer needs and what makes the local route auditable; what would show it wrong is operators leaving the backend unrecorded on most sessions, which the printed notice is there to surface diff --git a/.abcd/development/specs/closed/spc-2609150759135349-local-analysis.md b/.abcd/development/specs/closed/spc-2609150759135349-local-analysis.md new file mode 100644 index 0000000..458ee60 --- /dev/null +++ b/.abcd/development/specs/closed/spc-2609150759135349-local-analysis.md @@ -0,0 +1,670 @@ +--- +id: spc-2609150759135349 +slug: local-analysis +intent: itd-8 +origin: researcher-authored +production_mode: hand-written +--- +# local-analysis + +## Summary + +`analyze -ingest` gains two provenance flags — `-backend local|cloud` and an +optional `-model NAME` — and writes the operator's declaration into the session +as a **provenance record**: one `kind:"provenance"` line, the **first** line of +`findings.jsonl`, committed by the same `session.CommitRecords` call that writes +the findings it accompanies. `report` renders that line under its Findings +heading. Nothing else about the pipeline changes: the CLI still never calls a +model, holds no keys, and adds no network dependency, and there is no backend in +the CLI for a flag to select. The flags record a claim; they do not route work. + +The flags are **optional**. Omitting them writes a provenance record whose +backend is `unrecorded` and prints one notice on stderr naming that choice, so +the declaration is never *silently* missing and no existing invocation or script +breaks. A `findings.jsonl` written before this change carries no provenance line +at all; every reader tolerates that and `report` says "provenance: not +recorded". + +The record is a **declaration, not a measurement**. Testimony cannot observe +where the emitted request ran. Every surface that renders the record — the +report line, the reference pages, the how-to, the privacy explanation — states +that in so many words, and this spec treats any wording that implies the tool +verified the claim as a defect. + +The third deliverable is documentation: a new goal-keyed how-to, +`docs/how-to/analyse-locally.md`, walks the fully local route end to end, and +`docs/explanation/privacy.md` replaces its current hand-wave ("a fully local +analysis path keeps even the derived text on the machine") with the real +statement, explicitly conditional on that route. + +Every test in this slice is fixture-based and hermetic; stdlib only; no new +dependency, no new file in the session directory, and no new writer. + +## Design + +### The provenance record + +One line of `findings.jsonl`, discriminated by `kind:"provenance"` exactly as a +verdict is discriminated by `kind:"verdict"`: + +```json +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b","at":"2026-09-15"} +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"unrecorded","at":"2026-09-15"} +``` + +```go +// Provenance is the operator's declaration of what answered the analysis +// request this findings file was ingested from. It is a declaration, not a +// measurement: the CLI never calls a model and cannot observe where the request +// ran. +type Provenance struct { + Kind string `json:"kind"` // literal "provenance" + Rubric string `json:"rubric"` // the rubric version enforced at ingest + Backend string `json:"backend"` // local | cloud | unrecorded + Model string `json:"model,omitempty"` // free text, operator-supplied + At string `json:"at"` // YYYY-MM-DD +} +``` + +| Field | Type | Required | Rule | +|---|---|---|---| +| `kind` | string | yes | literal `"provenance"` — the discriminator | +| `rubric` | string | yes | written as the `RubricVersion` package constant, never as the answer's claimed rubric | +| `backend` | string | yes | one of `local \| cloud \| unrecorded`; a closed set | +| `model` | string | no | operator free text; at most 200 runes, non-blank under `session.CodeRendersEmpty` (the predicate report renders it with); omitted when not given | +| `at` | string | yes | ISO date `YYYY-MM-DD`, supplied by the caller | + +`rubric` is written from the package constant rather than from the answer, +because the constant is the scheme ingest actually *enforced* — the answer's +claimed rubric is untrusted input that ingest has already refused if it is not a +known version, and echoing it would let a bare-array answer (which carries no +rubric at all) leave the field empty. The field exists so that a provenance line +copied out of the session directory still says which coding scheme produced the +findings it came from, the same self-description `tests.jsonl`'s `session` field +carries. + +`model` is free text on purpose: model names are not a closed set and never will +be, and a validated list would refuse an honest answer the week after it +shipped. It is bounded (200 runes, the `drafttests` `maxTitle` precedent) and +sanitised at the sink, which is what makes it safe to carry in an artefact +designed to be shared. + +**No `-host` flag, and no host field.** Decided against; see Decisions. + +### Position in the file: first line + +The provenance record is written as `records[0]`, ahead of every finding line. + +- It is written by ingest, and ingest replaces the whole file + (`session.CommitRecords` truncates and rewrites). Verdicts are **appended** by + `review` through `session.AppendRecord`, which always writes at the end. A + provenance line placed last would therefore not stay last: the first verdict + appended after an ingest lands below it, and its position would carry no + meaning at all. `tail -1` would find a verdict; `head -1` finds the + provenance. +- The resulting file reads in the order it was decided: the producer, then what + it produced, then the human decisions in the order they were taken. That is + the append-only ethos the session artefacts already follow — the file grows at + the end, and the one record describing the whole ingest sits at the top. +- Re-ingest semantics fall out for free and satisfy the intent's requirement + that "a re-ingest replaces it with the findings it accompanies": the + provenance line is part of the same `Commit.Records` slice as the findings, so + one `CommitRecords` call replaces both together. A provenance record can never + outlive the findings it describes, and findings can never acquire a provenance + from a different run. +- Verdicts stay protected unchanged. `commitFindings`' `Guard` still calls + `holdsVerdicts`, which scans for raw `kind:"verdict"` lines; a provenance line + is not one, so the guard's behaviour and its message are untouched. A + `findings.jsonl` holding verdicts refuses re-ingest exactly as today — + including a re-ingest whose only intent is to correct the provenance. That is + the right trade: the precision record outranks a correctable claim, and the + operator's route is to re-run review's history intact rather than have ingest + learn to rewrite a file it is forbidden to touch. + +### `analyze.ParseRecords` — tolerating and exposing the record + +`ParseRecords` gains a third branch on the `kind` probe, before the +"missing `t`" refusal: + +1. `kind == "verdict"` — unchanged. +2. `kind == "provenance"` — decode into `Provenance`. + - If `backend` is outside the closed set `local | cloud | unrecorded`, the + record is **ignored** (skipped, not returned, not counted), exactly as an + out-of-enum verdict is ignored today. A backend nobody can interpret must + not reach the report as a privacy claim; the file then reads as "not + recorded", which is the truthful fallback. + - A **second** in-enum provenance record is a **hard error**, naming both + lines: + `%s:%d: duplicate provenance record (first seen at line %d); a findings file records exactly one producer` + This mirrors the duplicate-finding-id refusal and rests on the same + argument: with two conflicting claims a single-valued consumer silently + picks one, and here picking wrong prints a false privacy claim into the + shareable artefact. An ambiguous attribution is worse than none, so it is + refused rather than resolved by a rule nobody can see. +3. otherwise — the existing finding path, unchanged. + +Signature change, in both the reader and its on-disk wrapper: + +```go +func ParseRecords(r io.Reader, name string) (*Provenance, []Finding, []Verdict, error) +func Load(dir string) (*Provenance, []Finding, []Verdict, error) +``` + +`nil` means the file carries no interpretable provenance record — a file written +before this change, or one whose only provenance line was ignored above. + +`ParseRecords` stays a **reader, not a validator**, consistent with how it +treats finding fields: it does not check `model`'s length or `at`'s shape on +read, because a hand-edited or exchanged `findings.jsonl` reaches it directly and +each sink already defends itself. `backend` is the one exception, filtered on +read, because it is the field that *is* the claim and the report must never +print an uninterpretable one. + +The per-line and total-size caps apply unchanged: the provenance line goes +through the same `total += len(raw)+1` accounting and the same +`session.MaxJSONLLine` scanner buffer as every other line. + +**Call sites** (compiler-enforced, five in non-test code): +`internal/report/report.go:153`, `internal/review/review.go:125`, +`internal/review/review.go:408` (`AppendVerdict`'s under-lock re-check — +discards the provenance with `_`), `internal/drafttests/drafttests.go:420`, and +`internal/drafttests/review.go:72` (both discard it with `_`). Plus the test +call sites. + +A second reader (`analyze.LoadProvenance(dir)`) was rejected: it would scan +`findings.jsonl` twice and give the duplicate-and-enum rules two homes that can +drift. One reader, one rule set, and the compiler updates every consumer. + +### `analyze.NewProvenance` — one home for the flag rules + +```go +// NewProvenance validates the operator's declaration and returns the record +// ingest will write. backend is "" when the flag was not given, which records +// "unrecorded". +func NewProvenance(backend, model, at string) (Provenance, error) +``` + +Rules, in this order, each returning the first failure: + +- `backend` must be `""`, `"local"`, or `"cloud"`; anything else → + `invalid -backend %q (want local or cloud)`. `"unrecorded"` is **not** + accepted from the flag: it is what the absence of the flag records, not a + value an operator states. +- `model != ""` with `backend == ""` → + `-backend is required with -model`. +- `model`, when given, must not render as nothing: + `session.CodeRendersEmpty(model)` → + `-model must not be blank (it renders as nothing: whitespace, invisible characters, or backticks alone)`. + The predicate is **the same function report uses** to decide whether the model + is present in its code span, moved into `internal/session` for the purpose. A + local SafeText-only test is not equivalent: report strips backticks when it + renders the span, so a model of backticks alone renders as nothing there, and + two predicates had it accepted at the flag, echoed on the success line, and + then rendered as "not recorded" — the tool contradicting itself about what it + had just stored. +- `model` length: `utf8.RuneCountInString(model) > 200` → + `-model is %d characters, exceeding the limit of 200`. +- `at` must match `^\d{4}-\d{2}-\d{2}$` → `invalid date %q (want YYYY-MM-DD)`. + +The rules live in `internal/analyze`, not in `cli.go`, following the +`review.ParseVerdictFlag` / `review.ParseKindFlag` precedent: the package that +owns the record owns its rules, and `cli.go` wraps the returned error into +`usageErr` (`analyze: %w`). This also bounds the record by construction, so the +line can never approach `session.MaxJSONLLine` and no second size rule is +needed for it beyond the total below. + +`Provenance.Valid()` re-states these as post-conditions (the kind literal, a +backend in the closed set, a non-empty rubric and date) and `Ingest` calls it +before it reads a byte. `Ingest` takes the record by value, so without that guard +a zero-valued `Provenance` commits `{"kind":"","rubric":"","backend":"","at":""}` +as line 1 and reports success — a line that falls through `ParseRecords`' +discriminator into the finding branch and is refused there for its missing `t`, +making `Ingest` a writer that produces a file its own `Load` cannot open. The +error names `analyze.NewProvenance`, since building the record through it is the +only way to satisfy the check. + +The stored `Model` is the **raw** operator string, not the `SafeText` form: +`report` sanitises at the sink like every other untrusted field, and storing the +sanitised form would make the record disagree with what the operator typed. + +### `analyze.Ingest` — writing the record + +```go +func Ingest(dir string, r io.Reader, prov Provenance) ([]Finding, error) +``` + +The date is a parameter, never `time.Now()` inside the package — the +`review.Options.Today` precedent — so ingest stays deterministic and its golden +tests need no clock injection. + +Order of work is unchanged; two points are added: + +- `oversizedFindings` gains the provenance line's encoded length into its + `total`, so the `session.MaxJSONLBytes` pre-flight measures the file that will + actually be written. `CommitRecords` leaves that pre-flight to its callers, so + omitting the provenance line's bytes would let a file land one record past the + cap that every reader then refuses. The per-line check does not apply to it + (bounded by construction, above); the total does. +- `commitFindings(dir, prov, findings)` prepends `json.Marshal(prov)` to + `records`. The `Guard` closure is untouched. + +**One canonical primitive.** The write goes through `session.CommitRecords`, +the existing primitive, with one extra element in `Records`. No new writer, no +new file, no second lock, no change to `session/records.go`. + +### CLI surface + +``` +testimony analyze [-session DIR] -ingest FILE [-backend local|cloud] [-model NAME] +``` + +```go +backend := fs.String("backend", "", "ingest mode: record which backend answered the request: local | cloud") +model := fs.String("model", "", "ingest mode: record the model that answered the request (free text)") +``` + +Guards, in the order the existing gauntlet runs them (all `usageErr`, exit 2, +placed with the other flag checks and **before** `resolveSession`, so a refused +run never first announces an inferred session): + +| Condition | Message | +|---|---| +| `-backend` given empty | `analyze: -backend must not be empty` | +| `-model` given empty | `analyze: -model must not be empty` | +| either given in emit mode (`*ingest == ""`) | `analyze: -backend and -model apply to the ingest mode only` | +| `NewProvenance` rejects the pair | `analyze: ` | + +The two empty-flag guards are the repository's established convention — an +explicitly-empty flag is an unset shell variable spliced into the invocation, +not a value — and they must come first, because without them an empty `-backend` +would fall through `NewProvenance`'s `backend == ""` branch and silently record +`unrecorded` for an operator who believed they had named a backend. The +emit-mode guard is `draft-tests -window`'s precedent: a flag that does nothing in +the mode you are in is refused, never silently ignored. + +**`-backend` is optional, not required.** See Decisions for the reasoning. When +it is absent in ingest mode, one line goes to stderr **before** the answer is +read, matching `resolveSession`'s inferred-session notice (an implicit choice +must at least be visible in the output of the run that made it): + +``` +analyze: no -backend given; the provenance will record "backend not recorded" +``` + +The tense is deliberate: the notice prints before the answer is validated, so it +also prints on runs that go on to fail and write nothing. "will record" states an +intention a later refusal simply overtakes, where "recording" would claim +something the run never did. + +stderr, never stdout, so nothing changes for a caller piping `analyze`'s output. + +The success line grows one clause: + +``` +validated 5 findings → sessions/x/findings.jsonl (all unverified; local backend, model llama3.1:70b) +validated 5 findings → sessions/x/findings.jsonl (all unverified; local backend, model not recorded) +validated 5 findings → sessions/x/findings.jsonl (all unverified; backend not recorded, model not recorded) +``` + +Every branch renders both halves of the declaration rather than returning early +on the backend: the two are independent, so a record carrying a model must say so +whatever its backend reads, and the clause keeps one shape in all three cases. + +The model is printed through `session.SafeText` — it is operator text reaching a +terminal, the same treatment the emitted request gives manifest fields. + +The top-level `usage` string gains the ingest line's new flags, and +`docs/reference/cli.md` gains the rows; `TestUsageListsEveryFlagAndCommand` +gains `-backend local|cloud` to pin it. + +### `report` — one line under the Findings heading + +`renderFindings` prints the provenance line immediately after `## Findings` and +before the first status group, on the two paths where `findings.jsonl` was read +successfully. The absent-file and unreadable-file notices are unchanged and +print no provenance line — there is no file to have a provenance. + +``` +## Findings + +_Provenance (as declared at ingest): local backend · model `llama3.1:70b` · rubric `testimony-analysis/v1` · ingested 2026-09-15._ + +### Confirmed (1) +``` + +Rendering rules, each following the sink-defence pattern the file already uses: + +- `backend` is rendered from a **switch on the closed enum**, never from the + parsed string, so no untrusted byte reaches that position at all: + `local` → `local backend`, `cloud` → `cloud backend`, + `unrecorded` → `backend not recorded`. +- `model` → `mdCode`, with `codeRendersEmpty` falling back to + `model not recorded` (the `findingAnchor` pattern). +- `rubric` → `mdCode`, with `codeRendersEmpty` falling back to + `rubric not recorded`. +- `at` → `mdInline`, with `inlineRendersEmpty` dropping the + `· ingested ` clause entirely (the existing verdict-`at` pattern). +- `prov == nil` → `_Provenance: not recorded._` + +The parenthetical "(as declared at ingest)" is load-bearing, not decoration: the +report is the artefact that travels, and it must not read as though the tool +measured where the analysis ran. + +### `review`, `draft-tests`: no behaviour change + +`review` discards the provenance (`_`) at both call sites; its interactive walk, +its verdict records, and `AppendVerdict`'s under-lock re-check are unchanged. + +`draft-tests` does **not** put the analysis provenance into its emitted drafting +request, and does not write a provenance record of its own. The drafting request +asks a model to reconstruct reproduction steps from a confirmed finding's event +window; which backend coded the finding first changes no instruction in it, and +adding an operator-supplied free-text string to a second prompt buys nothing. A +provenance record for `tests.jsonl` is a separate question with a separate +rubric (`testimony-testdraft/v1`) and is named as an open question on the +intent. + +### Files written before this change + +No migration, no version field, no upgrade step. `Load` returns `nil`, `report` +prints `_Provenance: not recorded._`, `review` and `draft-tests` behave +identically, and the next successful `analyze -ingest` (which is only possible +on a file free of verdicts, per the existing guard) writes one. + +## Acceptance mapping + +Test names below are the ones this slice adds; the package is named with each. + +**AC1 — ingest writes the record as the first line; the findings are unchanged.** +`NewProvenance` validates the pair, `commitFindings` prepends it to the same +`CommitRecords` batch, and the finding encoding is untouched. +- `analyze.TestIngestWritesProvenanceAsFirstLine` — ingest a known-good fixture + with `-backend local -model "llama3.1:70b"`; line 1 decodes to the expected + `Provenance`; lines 2..n are **byte-identical** to the same fixture ingested + without the flags (the parity assertion AC1 names). +- `analyze.TestNewProvenanceRules` — table: unknown backend, `"unrecorded"` + passed explicitly, `-model` without `-backend`, blank-after-SafeText model, + 201-rune model, malformed date; each asserts the exact message. +- `cli.TestAnalyzeIngestRecordsProvenance` — end to end through `Run`, asserting + the success line's new clause. + +**AC2 — `report` states the declaration before the status groups.** +- `report.TestReportRendersProvenanceLine` — golden: the line's exact text, its + position directly under `## Findings` and above `### Confirmed`, and the + "(as declared at ingest)" wording. +- `report.TestReportProvenanceSanitisesModelAndRubric` — a model of + `` `x](http://h/b.png) `` and a rubric carrying ESC and a bidi control render + inert; a model that renders empty falls back to `model not recorded`; an `at` + that renders empty drops its clause. +- `report.TestReportProvenanceRendersBackendFromEnum` — `local`, `cloud`, and + `unrecorded` each render their fixed phrase. + +**AC3 — a pre-change file loads unchanged and reads as "not recorded".** +- `analyze.TestParseRecordsToleratesMissingProvenance` — the existing fixture, + no provenance line, returns `nil` and the same findings and verdicts as today. +- `report.TestReportProvenanceNotRecorded` — golden asserts + `_Provenance: not recorded._`. +- `review.TestReviewIgnoresProvenanceRecord` — a walk and a non-interactive + verdict over a file *with* a provenance first line behave exactly as over one + without, and the appended verdict lands last. + +**AC4 — neither flag: findings unchanged, `unrecorded` recorded, choice announced.** +- `cli.TestAnalyzeIngestWithoutBackendAnnouncesUnrecorded` — asserts the exact + stderr notice, that it is on stderr and not stdout + (the `TestInferenceLineStaysOffStdout` pattern), the success line's + `backend not recorded` clause, exit 0, and the written record's + `"backend":"unrecorded"` with no `model` key. +- Covered jointly by `analyze.TestIngestWritesProvenanceAsFirstLine`'s + byte-parity assertion on the finding lines. + +**AC5 — a file holding verdicts still refuses re-ingest, with any flags.** +- `analyze.TestIngestRefusesVerdictFileWithProvenanceFlags` — the existing + refusal message unchanged, and the file byte-identical afterwards including + its provenance line. + +**AC6 — the documented local route, and the conditional privacy statement.** +Satisfied by the Docs plan below; checked by review rather than by a Go test. +The repository's `abcd docs lint` (link and currency gates) and the +`docs-currency-reviewer` pass are the standing instruments. + +## Decisions + +- **`-backend` is optional, with an announced `unrecorded` default — not + required-when-ingesting.** Required was the starting preference and was + rejected on three grounds. (i) It breaks every existing invocation: + `analyze -ingest answer.json` would begin exiting 2, which is a behaviour + change the intent's own criterion forbids ("unchanged apart from what is + recorded") and which no CHANGELOG `### Added` entry can soften. (ii) It does + not achieve what it was for. A required flag cannot make provenance *true* — + the operator can answer either way and the CLI cannot check — and it does + nothing at all for the files already on disk, which is where the missing + provenance actually is. (iii) The optional form achieves the stated goal + better: **every** `findings.jsonl` written after this change carries a + provenance record, one that says `unrecorded` when nobody said otherwise. The + claim "a findings file always says what produced it" becomes literally true, + including for the operator who did not read the release notes. The stderr + notice is what keeps it from being *silent*, and it follows the repository's + own precedent for an implicit choice (`resolveSession`). Reopening this is an + open question on the intent, for a later revision once no script depends on + today's default. + +- **No `-host` flag and no host field.** A field named for a host invites an + address — `http://192.168.1.4:11434`, a machine name, a colleague's laptop — + and a session directory is an exchange unit, archived and handed to reviewers. + The repository's privacy invariant forbids hostnames in committed and shared + artefacts, and no validation can tell `ollama` from `ollama.internal.corp` + reliably enough to enforce it. The claim that carries the ethics weight is + which side of the machine boundary the request ran on, and `backend` states + exactly that; the reproducibility weight is carried by `model`, which is free + text and can perfectly well read `ollama/llama3.1:70b` — which is how local + runners name models anyway. Two fields, one closed and one free, cover both + questions without inviting the one string the artefact must not carry. + +- **The provenance line is the first line, not the last.** Ingest rewrites the + whole file while verdicts are appended to its end, so a last-position + provenance would be overtaken by the first verdict and its position would mean + nothing. First position makes `head -1` the answer, reads in decision order, + and — because it rides in the same `CommitRecords` batch as the findings — is + replaced with them by a re-ingest and can never describe a different run's + findings. + +- **A second in-enum provenance record is a hard error; an out-of-enum `backend` + is ignored.** The two rules have different jobs. Ignoring an uninterpretable + backend is the out-of-enum-verdict precedent: the file stays readable and the + report falls back to the truthful "not recorded". Refusing two readable but + conflicting claims is the duplicate-finding-id precedent: a single-valued + consumer would otherwise pick one silently, and here picking wrong prints a + false privacy claim into the artefact people share. + +- **`Load`/`ParseRecords` change signature rather than gaining a second + reader.** Four non-test call sites, all internal, all compiler-enforced. A + `LoadProvenance` helper would scan the file a second time and give the + duplicate and enum rules two homes that can disagree — precisely the drift the + one-canonical-primitive rule exists to prevent. + +- **The record is written through `session.CommitRecords` with one extra + element.** No new writer, no new session file, no change to + `internal/session`. The only adjustment is that `oversizedFindings` counts the + provenance line into the `MaxJSONLBytes` total, because `CommitRecords` + delegates that pre-flight to its callers. + +- **`rubric` comes from the package constant, `model` from the operator + verbatim.** The constant is the scheme ingest enforced; the answer's claimed + rubric is untrusted and may be absent entirely (a bare-array answer). The + model string is stored raw and sanitised at each sink, so the record does not + quietly disagree with what the operator typed. + +- **The bundled `examples/sample-session/findings.jsonl` records + `backend: "unrecorded"` with no model, dated on or before its verdicts.** Its + findings were hand-authored for + the repository and were produced by no model at all; a sample asserting `local` + would plant a false provenance claim in the one artefact users copy from. The + `local`/`cloud`/model-present renderings are covered by unit fixtures instead. + The sample still exercises the new code path end to end in CI, since the line + is parsed and rendered (`grep -q "_Provenance"` in the pipeline smoke). Its + date must not post-date the bundled verdicts: ingest is refused once a file + holds verdicts, so a provenance stamped after them is a state the tool cannot + produce, and the sample is the artefact users copy from. + +- **`draft-tests` neither shows nor writes provenance.** Out of scope, with the + symmetric question recorded on the intent. + +## Test plan + +Hermetic, fixture-based, CI-safe on ubuntu with no LLM, network, tool, or TTY. +Stdlib only. + +**`internal/analyze`** +- `TestNewProvenanceRules` — the full rejection table above, each asserting the + exact message, plus the accepting cases (`local`, `cloud`, absent backend, + a 200-rune model, a model containing an inline-Markdown trigger stored raw). +- `TestIngestWritesProvenanceAsFirstLine` — first line decodes to the expected + record; the remaining lines are byte-identical to a no-flag ingest of the same + fixture. +- `TestIngestOmitsModelWhenNotGiven` — the written line has no `model` key + (`omitempty`), and `backend` is `unrecorded`. +- `TestIngestCountsProvenanceInTotalSize` — an answer that fits under + `MaxJSONLBytes` only if the provenance line is uncounted is refused, with the + existing total-size message. +- `TestIngestRefusesVerdictFileWithProvenanceFlags` — existing message; file + byte-identical afterwards. +- `TestIngestRefusesInvalidProvenance` — table over the zero value, a wrong + kind, a backend outside the set, an empty rubric and an empty date; each + asserts the refusal names `analyze.NewProvenance` and that nothing was + written. Positive control: every record `NewProvenance` builds ingests and + loads back unchanged. +- `TestNewProvenanceRefusesBacktickOnlyModel` — a backtick, a run of backticks, + and a whitespace-and-backticks value are each refused with the blank-model + message, while a model that merely *contains* a backtick stays accepted. +- `TestParseRecordsExposesProvenance` — a file with a provenance first line + returns it alongside unchanged findings and verdicts. +- `TestParseRecordsToleratesMissingProvenance` — `nil`, no error. +- `TestParseRecordsIgnoresUnknownBackend` — a provenance line with + `"backend":"loocal"` yields `nil` and no error; the findings still load. +- `TestParseRecordsRefusesDuplicateProvenance` — two in-enum provenance lines; + exact message naming both line numbers. +- `TestParseRecordsProvenanceAnywhereInFile` — a provenance line appearing after + the findings (a hand-edited file) is still returned; position is a writer + convention, not a reader requirement. +- `TestParseRecordsProvenanceCountsTowardTotalCap` — the line participates in + the `MaxJSONLBytes` accounting. + +**`internal/report`** +- `TestReportRendersProvenanceLine`, `TestReportProvenanceNotRecorded`, + `TestReportProvenanceRendersBackendFromEnum`, + `TestReportProvenanceSanitisesModelAndRubric` (as mapped above). +- `TestReportNoFindings` — unchanged: the absent-file notice prints no + provenance line. +- The existing `TestRoundTrip` golden is regenerated to include the line. + +**`internal/review`** +- `TestReviewIgnoresProvenanceRecord` — walk and non-interactive verdict over a + file with a provenance first line; the appended verdict lands last and the + provenance line is byte-unchanged (the append-only property, extended to the + new record). + +**`internal/drafttests`** +- `TestDraftTestsIgnoresProvenanceRecord` — emit, ingest, and render over a + `findings.jsonl` carrying a provenance line produce byte-identical output to + the same session without one. + +**`internal/cli`** +- `TestAnalyzeIngestRecordsProvenance` — success line's new clause. +- `TestAnalyzeIngestWithoutBackendAnnouncesUnrecorded` — the stderr notice, its + absence from stdout, exit 0. +- `TestAnalyzeProvenanceFlagsAreUsageErrors` — table over the exit-2 messages + (empty `-backend`, empty `-model`, unknown backend, `-backend unrecorded`, + `-model` without `-backend`, a backtick-only `-model`, and either flag in emit + mode), each asserting exit 2 and the exact stderr text; extends + `TestInvalidFlagValuesExitTwo`'s family. +- `TestUsageListsEveryFlagAndCommand` — gains `-backend local|cloud`. + +**Sample smoke (CI)** — `merge → report` over `examples/sample-session` renders +the provenance line, pinned by `grep -q "_Provenance"` in +`.github/workflows/ci.yml` and enumerated with the other pinned assertions in +`.abcd/development/brief/06-delivery/02-verification.md`; the existing grep for +the confirmed save-feedback finding is unaffected. + +**Live verification (part of done, not CI).** Run the emitted request against a +model hosted on the maintainer's own machine, ingest with +`-backend local -model `, render the report, and read the +resulting provenance line and the new how-to end to end against what actually +happened. Fix what it exposes before the PR. This is the one step that proves +the documented route is the route, and it is the claim the intent's press +release makes. + +## Docs plan + +- **`docs/reference/cli.md`, `## testimony analyze`** — two rows in the flag + table (`-backend`, `-model`, both marked ingest-mode); the mode sentence gains + "either flag in emit mode is a usage error"; the ingest-behaviour paragraph + gains the record, its position, the `unrecorded` default, the stderr notice, + and the new success-line clause. The exit-2 table needs no change — the new + refusals are the rows it already describes. + +- **`docs/reference/session-directory.md`, `## findings.jsonl`** — the opening + sentence becomes "Three record kinds share the file"; a **Provenance record** + table (the five fields and their rules) and an example line go above the + finding table, with the first-line convention, the one-per-file rule, the + ignored-unknown-backend rule, and the plain statement that the record is the + operator's declaration and is not verified. + +- **New `docs/how-to/analyse-locally.md`** — a how-to, not a section of + `analyse-a-session.md`. Both would be the same Diátaxis type, so the type does + not decide it; the *goal* does. `analyse-a-session.md` answers "how do I + analyse a session"; this answers "how do I keep the analysis on this machine", + a different goal with a different reader (an operator working to an ethics + protocol), and Diátaxis keys how-tos on goals. Folding it in would also mean + conditionals inside two of an already five-step page's steps, and the privacy + explanation needs a concrete link target whose whole subject is the claim it + is making. Precedent: `record-a-terminal-session.md` and + `transcribe-a-recording.md` are goal-keyed pages beside the main flow. The + page walks: what "local" means here (the CLI never calls a model, so the only + thing that has to be local is whatever answers the request) → emit to a file → + run it against the locally hosted model of your choice, offline → ingest with + `-backend local -model NAME` → read the provenance line in `report.md` → what + the record does and does not prove. Personas Alice/Bob/Carol; British English; + present tense. + +- **`docs/how-to/analyse-a-session.md`** — step 3 gains the two flags and one + sentence on what ingest records; a pointer to the new page sits beside it. + +- **`docs/explanation/privacy.md`** — the sentence "If your setting demands it, + a fully local analysis path keeps even the derived text on the machine." + is replaced by the real statement: when the emitted request is answered by a + model running on the same machine and the answer is ingested with + `-backend local`, no session content leaves the machine at any step of the + pipeline — followed immediately by the two honest qualifications (the record + is the operator's declaration, not a measurement; and the demo page's rrweb + CDN request, already disclosed above on that page, is unaffected either way). + Links to the new how-to. + +- **`docs/README.md`** — the new how-to in its index. + +- **`README.md`** — one clause in the analysis paragraph noting that findings + carry the declared backend and model. No roadmap change. + +- **Brief (`.abcd/development/brief/`)** — `04-surfaces/06-analyze.md` gains the + two flags and the ingest-writes-provenance behaviour; + `05-internals/02-schemas.md` gains the provenance record table beside the + finding and verdict tables (the schema-move invariant: schema, sample, and + tests move in the same change); `02-constraints/04-ethics.md`'s "a fully local + variant (local LLM for analysis) is the fallback" sentence is corrected to + describe the recorded declaration rather than a variant of the tool. + +- **`.github/workflows/ci.yml` and + `.abcd/development/brief/06-delivery/02-verification.md`** — the pipeline + smoke greps the rendered report for `_Provenance`, and the brief's enumeration + of pinned assertions names it and says why it pins presence rather than a + particular backend. + +- **`.abcd/work/DECISIONS.md`** — one dated line: the draft's local-backend flag + is refuted against the host-delegated architecture and replaced by a recorded + provenance declaration; `-backend` optional with an announced `unrecorded` + default; no host field. + +- **`CHANGELOG.md`, `## [Unreleased]` → `### Added`** — one entry: the two + flags, the provenance record and its position, the `unrecorded` default and + its notice, the report line, and the new how-to. No `### Changed` entry is + needed for the success line's extra clause; it is named inside the `Added` + entry, since the behaviour it reports is the addition. diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index 5316cb7..1aaaded 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -1688,3 +1688,23 @@ Architecture-shaping decisions graduate to an ADR under falling back to a relative root, which is the scattered-session outcome the fixed default exists to end. A behaviour change, called out in the changelog with `-out sessions` as the one-line migration. +- 2026-09-15 — itd-8 ("local analysis") rebuilt as provenance rather than a + backend flag: the draft promised a flag pointing the rubric at a locally + hosted model, which the host-delegated architecture cannot honour — the CLI + never calls a model, so there is no backend for a flag to select. What ships + instead is the operator's declaration, recorded: `analyze -ingest` takes + `-backend local|cloud` and a free-text `-model NAME` and writes one + `kind:"provenance"` record as the FIRST line of `findings.jsonl`, in the same + `session.CommitRecords` call as the findings (so a re-ingest replaces the + declaration together with what it describes, and the verdict-overwrite guard + is untouched and still outranks it); `report` prints it under the Findings + heading. Both flags are OPTIONAL with an announced `unrecorded` default rather + than required-when-ingesting: required would break every existing invocation + at exit 2, could not make a declaration true, and would do nothing for the + files already on disk, whereas optional makes every findings file written from + now on say something true about its own origin — and the stderr notice (the + `resolveSession` precedent) keeps the choice from being silent. No `-host` + field: a field named for a host invites an address, and a session directory is + an exchange unit. The quality-floor question the draft raised is deferred to + the retained verdicts, which now become a per-backend comparison, rather than + built as an unmeasured number the CLI could not enforce anyway. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bc62a1e..1db5cd4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -138,6 +138,14 @@ jobs: grep -q "## Findings" examples/sample-session/report.md grep -q "### Confirmed (1)" examples/sample-session/report.md grep -q "\*\*F-001\*\* bug" examples/sample-session/report.md + # The provenance record leading findings.jsonl reaches the report. The + # bundled sample declares no backend (its findings were hand-authored + # for the repository, not produced by any model), so this asserts the + # line is rendered at all rather than a particular backend: it catches + # the record being dropped by the reader or the renderer, which would + # silently strip the one statement the report makes about its own + # origin. + grep -q "_Provenance" examples/sample-session/report.md # The event half of the pipeline: every assertion above is satisfiable # from the transcript and findings alone (verified: deleting # interactions.jsonl left them all green) — the header counts assertion diff --git a/CHANGELOG.md b/CHANGELOG.md index 857b585..79e8737 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,33 @@ break an existing invocation is called out in the entry that records it. ### Added +- `testimony analyze -ingest` records what answered the analysis request. + `-backend local|cloud` and a free-text `-model NAME` write one provenance + record — `{"kind":"provenance","rubric":…,"backend":…,"model":…,"at":…}` — as + the **first** line of `findings.jsonl`, in the same write as the findings, and + `report` prints it under the Findings heading, so a findings file and the + report rendered from it always state what produced them. The record is the + operator's **declaration**, not a measurement: the CLI still never calls a + model, holds no keys, and adds no network dependency, so there is no backend + for a flag to select and no way for `analyze` to observe where the request was + answered — every surface that renders the record says so. Both flags are + optional and no existing invocation changes: with neither, the record states + `"backend":"unrecorded"` and the run announces the intention on stderr before + it reads the answer, so the choice is visible rather than silent. The ingest + success line names what was recorded (`… (all unverified; local backend, model + llama3.1:70b)`). A re-ingest replaces the provenance record together with the + findings it accompanies; the verdict-overwrite guard is unchanged and still + outranks it, so a `findings.jsonl` holding verdicts refuses a re-ingest + whatever the flags say. A `findings.jsonl` written before this change carries + no such record, loads unchanged everywhere, and reports `Provenance: not + recorded`. Wrong invocations are refused at exit 2 with the rest of that + family: an explicitly-empty `-backend`/`-model`, an unknown backend, + `-backend unrecorded` (which is what the flag's absence records, not a claim an + operator may state), `-model` without `-backend`, and either flag in emit mode. + A new how-to, [Analyse a session locally](docs/how-to/analyse-locally.md), + walks the fully local route end to end, and the privacy explanation now states + plainly that no session content leaves the machine at any step — conditional on + exactly that route. - `testimony draft-tests` turns a **confirmed** finding into a proposed regression test case, and `testimony review -kind tests` records the human accept / edit / reject pass over each draft. The oracle stays host-delegated, diff --git a/README.md b/README.md index bb048c8..9719223 100644 --- a/README.md +++ b/README.md @@ -120,14 +120,18 @@ stamps the session), `demo` (instrumented capture), `transcribe` (local WhisperX or whisper.cpp), `import` (an asciinema terminal recording joins the session's interaction stream on the shared clock), `merge`, `report`, the first-pass analysis layer — `analyze` (emit an analysis request, then validate the answer -into findings) and `review` (record human verdicts) — and the regression-test +into findings, recording the backend and model you declare answered it) and +`review` (record human verdicts) — and the regression-test drafting layer, `draft-tests` (turn a confirmed finding into a proposed test case, then render the accepted ones as a Markdown test plan) with `review -kind tests` for the accept / edit / reject pass. `record` captures the microphone by default; screen video is opt-in with `-video`. The model work is host-delegated — the CLI never calls a model, holds no keys, and adds no network dependency — every finding is *unverified* until you confirm or reject it, and -every drafted test is a *proposal* until you accept it. +every drafted test is a *proposal* until you accept it. Run the analysis request +against a model on your own machine and no session content leaves it at any +step; `findings.jsonl` and `report.md` carry your declaration that it did +([analyse a session locally](docs/how-to/analyse-locally.md)). Coming next, in user terms: diff --git a/docs/README.md b/docs/README.md index f5261d0..dca435f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,6 @@ # Testimony documentation - **[Tutorials](tutorials/getting-started.md)** — learn by doing: capture, transcribe, and report on your first session in about five minutes. -- **[How-to guides](how-to/)** — recipes for specific tasks: [transcribe a recording](how-to/transcribe-a-recording.md), [record a terminal session](how-to/record-a-terminal-session.md), [instrument your own app](how-to/instrument-your-own-app.md), [analyse a session](how-to/analyse-a-session.md), [draft regression tests](how-to/draft-regression-tests.md). +- **[How-to guides](how-to/)** — recipes for specific tasks: [transcribe a recording](how-to/transcribe-a-recording.md), [record a terminal session](how-to/record-a-terminal-session.md), [instrument your own app](how-to/instrument-your-own-app.md), [analyse a session](how-to/analyse-a-session.md), [analyse a session locally](how-to/analyse-locally.md), [draft regression tests](how-to/draft-regression-tests.md). - **[Reference](reference/)** — exact descriptions of the [command line](reference/cli.md) and the [session directory](reference/session-directory.md). - **[Explanation](explanation/)** — background and reasoning: [how alignment works](explanation/how-alignment-works.md), [privacy](explanation/privacy.md). diff --git a/docs/explanation/privacy.md b/docs/explanation/privacy.md index 1fa68ce..7eba81d 100644 --- a/docs/explanation/privacy.md +++ b/docs/explanation/privacy.md @@ -11,7 +11,13 @@ The rule is simple: **raw recordings stay local; only derived text is ever analy - The capture server listens on your machine and writes to a local session directory. One disclosure sits alongside it: the demo page loads its session-replay recorder (rrweb) from a public CDN, so the participant's browser makes one request to that third party — which thereby sees an IP address and the time of the session, though never any session content. The script tag pins the version and carries a Subresource Integrity hash, so a browser refuses any script whose bytes differ from the one pinned — a tampered or substituted CDN file never runs, and the page degrades to the same no-rrweb case as a blocked CDN. Offline, or with the CDN blocked, the demo still works and still captures the interaction stream; only the archival replay stream stays empty. - What the pipeline produces for analysis is derived text: the transcript, the normalised event stream, the merged timeline, and the report. These are small, readable files you can inspect line by line before sharing them with anyone — or with any analysis tool. -The distinction matters because the derived text is a much narrower disclosure than the recording it came from. A transcript contains what was said; the audio contains a voiceprint. An event stream says a button was clicked; a screen recording shows everything else that was visible at the time. When an analysis layer (local or cloud) enters the picture, it sits on the far side of this boundary: it sees only the text you choose to give it, never the raw audio or video. If your setting demands it, a fully local analysis path keeps even the derived text on the machine. +The distinction matters because the derived text is a much narrower disclosure than the recording it came from. A transcript contains what was said; the audio contains a voiceprint. An event stream says a button was clicked; a screen recording shows everything else that was visible at the time. When an analysis layer (local or cloud) enters the picture, it sits on the far side of this boundary: it sees only the text you choose to give it, never the raw audio or video. + +The boundary can be closed completely. Testimony never calls a model itself: `analyze` emits a request, something else answers it, and `analyze -ingest` validates the answer. So when the thing that answers runs on the same machine — and you ingest with `analyze -ingest ... -backend local` — **no session content leaves the machine at any step**: not the recording, not the transcript, not the timeline, not the findings. That conclusion is conditional on exactly that route, which [Analyse a session locally](../how-to/analyse-locally.md) sets out end to end; it does not hold for a session whose request you pasted into a cloud assistant, however local the rest of the pipeline was. + +Which route a given session took is recorded rather than remembered. Every `analyze -ingest` writes a provenance record as the first line of `findings.jsonl` — the backend, the model, the rubric version, and the date — and `report.md` prints it under the Findings heading, so the claim travels with the evidence to whoever reads it next. Two qualifications belong with it. The record is your **declaration**: Testimony has no way to observe where the request was answered, so it writes down what you state, and the value of the record rests on your being the person who ran it. And it changes nothing about the boundary itself — a session analysed in the cloud is exactly as exposed as before, and now says so. + +The demo page's one disclosure — the session-replay recorder it loads from a public CDN, noted above — is unaffected either way: it belongs to capture, not analysis, and a real session in your own instrumented app does not involve it. A terminal recording sits at the widest point of that boundary. A shell shows far more of the machine than a demo app does: its output routinely carries usernames, hostnames, absolute paths, environment values, and occasionally a secret a tool prints. Keystrokes never reach the derived text — `import` drops every input event a cast holds, so a password typed at a prompt that suppresses echo cannot enter the interaction stream — and the raw `terminal.cast` is local evidence of the same class as `audio.wav`. What does travel outward is the derived text, so a terminal session asks one thing of you that a browser session never had to: read or redact `timeline.jsonl` before running `analyze`. [Record a terminal session](../how-to/record-a-terminal-session.md) sets out the practice. diff --git a/docs/how-to/analyse-a-session.md b/docs/how-to/analyse-a-session.md index 0876184..fd7655a 100644 --- a/docs/how-to/analyse-a-session.md +++ b/docs/how-to/analyse-a-session.md @@ -50,9 +50,17 @@ accepted): Validate the answer against the findings schema and write `findings.jsonl`: ```sh -testimony analyze -session ~/Testimony/sessions/ -ingest answer.json +testimony analyze -session ~/Testimony/sessions/ -ingest answer.json -backend cloud -model ``` +`-backend local|cloud` and `-model NAME` record what answered the request. Ingest +writes that declaration as the first line of `findings.jsonl` and `report` shows +it, so a findings file always says what produced it. Both flags are optional — +leave them off and the record states that the backend was not recorded, and the +run says so on stderr. Testimony records what you state and cannot verify it: +`analyze` never calls a model. To keep the whole analysis on this machine, see +[Analyse a session locally](analyse-locally.md). + Ingest is the validation boundary, and it never trusts the model. It rejects, with a precise message, any finding whose evidence id is not in the timeline, whose quote is not spoken verbatim in a cited utterance, whose `type` or `severity` is @@ -103,7 +111,9 @@ testimony report -session ~/Testimony/sessions/ open ~/Testimony/sessions//report.md ``` -The Findings section lists findings under **Confirmed**, **Unverified**, +The Findings section opens with the provenance line — what you declared at +ingest, or `Provenance: not recorded` for a findings file written before the +record existed — then lists findings under **Confirmed**, **Unverified**, **Duplicate**, and **Rejected**, each with its quote, anchor, and — where you recorded one — the verdict and its date. Change a verdict at any time with `testimony review -session ~/Testimony/sessions/ -finding F-NNN -verdict ` diff --git a/docs/how-to/analyse-locally.md b/docs/how-to/analyse-locally.md new file mode 100644 index 0000000..24f0377 --- /dev/null +++ b/docs/how-to/analyse-locally.md @@ -0,0 +1,126 @@ +# Analyse a session locally + +This guide covers the route that keeps a session's analysis on the machine that +recorded it, and records that it did. Follow it when an ethics protocol, a +participant population, or your own judgement rules out sending even derived text +to a cloud service. + +Prerequisite: a session with a merged `timeline.jsonl`, and a model you can run +on the machine. Testimony neither ships, installs, nor configures one — see +"What Testimony does and does not do" below. + +The route is four steps: **emit** the request to a file, **run** it against your +local model, **ingest** the answer with `-backend local`, and **read** the +provenance line in the report. + +## 1. Emit the request to a file + +```sh +testimony analyze -session sessions/ -out request.txt +``` + +`analyze` reads `manifest.json` and `timeline.jsonl` and writes one +self-contained prompt. Nothing in the session directory changes, and nothing +leaves the machine: `analyze` never calls a model, holds no keys, and adds no +network dependency. + +`request.txt` contains the whole of what the model sees — the rubric, the session +context, and the timeline lines. Read it before you go further if you want to +know exactly what you are about to hand over. + +## 2. Run the request against your local model + +Give `request.txt` to whatever you run on this machine, and save the JSON answer +beside the session: + +```sh +your-local-runner < request.txt > answer.json +``` + +The request asks for JSON only. Any runner will do — the contract is the text in +`request.txt` and the JSON shape it asks for, not a particular tool. This is the +step where "local" is either true or not: it is true if, and only if, the program +you pipe into answers from this machine. + +## 3. Ingest the answer, declaring the backend + +```sh +testimony analyze -session sessions/ -ingest answer.json \ + -backend local -model llama3.1:70b +``` + +Ingest validates the answer against the findings schema exactly as it does for +any other run — see [Analyse a session](analyse-a-session.md) for what it +rejects — and writes `findings.jsonl`. The difference is the first line: + +```json +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b","at":"2026-09-15"} +``` + +`-model` is free text and at most 200 characters. A value that renders as +nothing — whitespace, invisible characters, or backticks alone — is refused +rather than stored, so what the report shows is always what you typed. Use whatever names the model +you actually ran; a runner-qualified name such as `ollama/llama3.1:70b` is fine, +and is often the most useful thing to write. + +The run confirms what it recorded: + +``` +validated 5 findings → sessions//findings.jsonl (all unverified; local backend, model llama3.1:70b) +``` + +Omit `-backend` and the run still succeeds — the flags never break an existing +invocation — but the record then says the backend was not recorded, and the run +tells you so on stderr before it starts. There is no way to claim `unrecorded` on purpose: it is +what the absence of a declaration records. + +## 4. Read the provenance line in the report + +```sh +testimony report -session sessions/ +``` + +The Findings section opens with the declaration: + +``` +_Provenance (as declared at ingest): local backend · model `llama3.1:70b` · rubric `testimony-analysis/v1` · ingested 2026-09-15._ +``` + +That line travels with the report. When Carol hands the session to an ethics +reviewer six months later, the claim is on the page rather than in anybody's +memory of a shell session. + +Review the findings as usual (`testimony review -session sessions/`); your +verdicts are appended below, and the provenance line is never rewritten. + +## What Testimony does and does not do + +**It does** carry your declaration with the evidence: written at the moment you +make it, replaced only when the findings it describes are replaced, and rendered +into the report that gets shared. + +**It does not** verify the declaration. `analyze` never calls a model and has no +way to observe where `request.txt` was answered. The record says what you told +it, which is why it is worth only as much as your being the person who ran the +request. + +**It does not** manage the model. Which model you run, how you run it, and +whether it is good enough for the second-coder role are yours to decide. On that +last question, Testimony's answer is the one it already gives for every backend: +every finding is born `unverified`, and the confirm/reject verdicts you retain +are the running measure of how well the analyser is doing. Now that findings +files name their backend, those verdicts can tell you how a local model compares +with a cloud one on your own sessions, rather than on somebody's benchmark. + +## Where this leaves the privacy boundary + +Voice and screen were already local, and so is transcription. Run this route and +the analysis is too, so no session content leaves the machine at any step. The +[privacy explanation](../explanation/privacy.md) sets out the boundary in full, +including the one disclosure that survives regardless — the demo page's +session-replay recorder, which a real session in your own instrumented app does +not use. + +For every flag, see the [command-line reference](../reference/cli.md); for the +record's exact fields, the +[session directory reference](../reference/session-directory.md#findingsjsonl). diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 5ec0608..abb1478 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -216,7 +216,7 @@ The first-pass analysis layer. `analyze` never calls a model, holds no keys, and ``` testimony analyze [-session DIR] [-out FILE] # emit the request -testimony analyze [-session DIR] -ingest FILE # validate the answer → findings.jsonl +testimony analyze [-session DIR] -ingest FILE [-backend local|cloud] [-model NAME] # validate the answer → findings.jsonl ``` | Flag | Default | Meaning | @@ -224,12 +224,34 @@ testimony analyze [-session DIR] -ingest FILE # validate the answer → find | `-session` | *(inferred)* | session directory; when omitted, the current directory if it holds a Testimony session `manifest.json` (see [session directory inference](#session-directory-inference)) | | `-out` | *(stdout)* | emit mode: write the request to `FILE` instead of stdout | | `-ingest` | *(off)* | ingest mode: validate the answer JSON at `FILE` (or `-` for stdin) into `findings.jsonl` | +| `-backend` | *(unrecorded)* | ingest mode: record which backend answered the request — `local` or `cloud` | +| `-model` | *(not recorded)* | ingest mode: record the model that answered the request; free text, at most 200 characters, and refused when it renders as nothing (whitespace, invisible characters, or backticks alone) | -`analyze` runs in exactly one mode: emit (no `-ingest`) or ingest (`-ingest`). Combining `-out` and `-ingest` is an error. Emit reads `manifest.json` and `timeline.jsonl`; ingest reads `timeline.jsonl` only. Both hint to run `merge` first when the timeline is missing, and both refuse a timeline whose entries carry a `src` other than `speech` or `event`, or a duplicated entry id — findings cite evidence by id, so a reused one cannot be resolved unambiguously (a `merge`-produced timeline never carries either defect: `merge` refuses a transcript whose utterance ids repeat or collide with the `ev-NNN` event ids it synthesises). +`analyze` runs in exactly one mode: emit (no `-ingest`) or ingest (`-ingest`). Combining `-out` and `-ingest` is an error. `-backend` and `-model` belong to ingest alone — emit mutates nothing in the session directory, so there is nothing for them to be recorded against — and passing either in emit mode is a usage error rather than a silently ignored flag. `-model` without `-backend` is a usage error too: the backend is the field that carries the claim, so a model name on its own records nothing about where the request ran. `unrecorded` is not accepted from `-backend`; it is what the absence of the flag records. Emit reads `manifest.json` and `timeline.jsonl`; ingest reads `timeline.jsonl` only. Both hint to run `merge` first when the timeline is missing, and both refuse a timeline whose entries carry a `src` other than `speech` or `event`, or a duplicated entry id — findings cite evidence by id, so a reused one cannot be resolved unambiguously (a `merge`-produced timeline never carries either defect: `merge` refuses a transcript whose utterance ids repeat or collide with the `ev-NNN` event ids it synthesises). Emit behaviour: writes a single self-contained prompt — the rubric version header (`testimony-analysis/v1`), the second-coder stance, two-pass instructions (segment coding, then session synthesis), the rubric body (five `type` definitions, the `1..4` severity scale, the evidence hard-constraints), the session context (app, participant, tasks), the timeline lines inline, and the required output shape with a worked example. Nothing in the session directory is mutated. The timeline is emitted whole (v1 does not chunk by task boundary; the manifest carries no task timestamps). With `-out FILE` the prompt goes to a file and the command prints `wrote `; otherwise it prints to stdout. -Ingest behaviour: reads the answer from `FILE` (or stdin when `-`), accepting a top-level object with a `findings` array (optionally a `rubric`, which must be a known version) or a bare array. Ingest is the sole validation boundary and never trusts the model. Each finding is decoded with unknown fields disallowed, then checked against every schema rule (see [session directory reference](session-directory.md#findingsjsonl)): id format and uniqueness, `t` within the session, the `type`, `severity`, and `mode` enums, non-empty `evidence` of at most 64 ids with every id real and at least one spoken `utt-*` anchor, a `quote` that is a verbatim substring of one *cited* evidence utterance, and any `ui` selector/route matching a real event. Validation is transactional — all errors are reported at once and nothing is written on any failure. On success every finding is forced to `status: unverified`, `findings.jsonl` is written, and the command prints `validated N findings → (all unverified)`. An answer with no findings (a bare `[]`, `{"findings":[]}`, or a truncated file) is refused rather than written, so it cannot erase a prior `findings.jsonl`; an answer whose findings would together push `findings.jsonl` past the session's 16 MiB total-size limit is refused the same way (see [`session-directory.md`](session-directory.md)). Ingest refuses to overwrite a `findings.jsonl` that already holds verdict records — counting any `kind:"verdict"` line, even one whose value is outside the closed enum. +Ingest behaviour: reads the answer from `FILE` (or stdin when `-`), accepting a top-level object with a `findings` array (optionally a `rubric`, which must be a known version) or a bare array. Ingest is the sole validation boundary and never trusts the model. Each finding is decoded with unknown fields disallowed, then checked against every schema rule (see [session directory reference](session-directory.md#findingsjsonl)): id format and uniqueness, `t` within the session, the `type`, `severity`, and `mode` enums, non-empty `evidence` of at most 64 ids with every id real and at least one spoken `utt-*` anchor, a `quote` that is a verbatim substring of one *cited* evidence utterance, and any `ui` selector/route matching a real event. Validation is transactional — all errors are reported at once and nothing is written on any failure. On success every finding is forced to `status: unverified`, `findings.jsonl` is written, and the command prints `validated N findings → `, followed by the birth state and the provenance the run recorded (the exact forms are under **Provenance** below). An answer with no findings (a bare `[]`, `{"findings":[]}`, or a truncated file) is refused rather than written, so it cannot erase a prior `findings.jsonl`; an answer whose findings would together push `findings.jsonl` past the session's 16 MiB total-size limit is refused the same way (see [`session-directory.md`](session-directory.md)). Ingest refuses to overwrite a `findings.jsonl` that already holds verdict records — counting any `kind:"verdict"` line, even one whose value is outside the closed enum. + +**Provenance.** Every ingest writes one [provenance record](session-directory.md#findingsjsonl) as the **first** line of `findings.jsonl`, in the same write as the findings, so a findings file always states what produced it. The record carries the rubric version, the backend, the model when one was given, and the date. It is the operator's **declaration**: `analyze` never calls a model and has no way to observe where the emitted request ran, so it records what you tell it. + +With neither flag the record states `"backend":"unrecorded"` and the run says so on stderr before it reads the answer, so the choice is visible in the output of the run that made it (stated as an intention, because a run that then fails validation writes nothing): + +``` +analyze: no -backend given; the provenance will record "backend not recorded" +``` + +The success line names what was recorded: + +``` +validated 5 findings → sessions/x/findings.jsonl (all unverified; local backend, model llama3.1:70b) +validated 5 findings → sessions/x/findings.jsonl (all unverified; cloud backend, model not recorded) +validated 5 findings → sessions/x/findings.jsonl (all unverified; backend not recorded, model not recorded) +``` + +A re-ingest replaces the provenance record together with the findings it accompanies, so a declaration can never outlive the findings it describes. The verdict guard is unchanged and outranks it: once a `findings.jsonl` holds verdicts, no re-ingest may rewrite it — including one whose only purpose is to correct the provenance. + +For the fully local route end to end, see [Analyse a session locally](../how-to/analyse-locally.md). ## `testimony draft-tests` diff --git a/docs/reference/session-directory.md b/docs/reference/session-directory.md index a6a859a..79f9c41 100644 --- a/docs/reference/session-directory.md +++ b/docs/reference/session-directory.md @@ -131,10 +131,31 @@ Event payload (`src: "event"`): `kind`, plus `selector`, `text`, `value`, and `r ## `findings.jsonl` -The analysis layer's output, written by `testimony analyze -ingest` and appended to by `testimony review`. Two record kinds share the file, one per line: a **finding** line (no `kind` field) and a **verdict** line (`kind: "verdict"`). Verdicts are appended, never written in place, so a finding's original state and the full verdict history are retained. Blank lines are ignored. +The analysis layer's output, written by `testimony analyze -ingest` and appended to by `testimony review`. Three record kinds share the file, one per line: a **provenance** line (`kind: "provenance"`), a **finding** line (no `kind` field), and a **verdict** line (`kind: "verdict"`). Verdicts are appended, never written in place, so a finding's original state and the full verdict history are retained. Blank lines are ignored. Ingest validates every finding against the merged timeline and is the sole validation boundary — it never trusts the model. Unknown fields are rejected (the shape is closed), and `status` is forced to `"unverified"` on ingest regardless of the answer JSON. +**Provenance record** + +The operator's declaration of what answered the analysis request, written by `analyze -ingest` as the **first** line of the file, in the same write as the findings it accompanies. It is a declaration, not a measurement: `analyze` never calls a model and cannot observe where the emitted request ran, so it records what the operator states. Exactly one record per file. + +| Field | Type | Required | Meaning | +|---|---|---|---| +| `kind` | string | yes | literal `"provenance"` (the discriminator) | +| `rubric` | string | yes | the rubric version ingest enforced — always the version `analyze` itself pins, never the version the answer claimed | +| `backend` | string | yes | one of `local`, `cloud`, `unrecorded`; `unrecorded` is what `analyze -ingest` writes when no `-backend` is given | +| `model` | string | no | the model the operator names with `-model`; free text, at most 200 characters, absent when none was given | +| `at` | string | yes | ingest date, ISO `YYYY-MM-DD` | + +```json +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b","at":"2026-09-15"} +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"unrecorded","at":"2026-09-15"} +``` + +First position is the writer's convention, not a reader's requirement: a hand-edited file that puts the record elsewhere is still read. A record whose `backend` falls outside the closed set is **ignored**, exactly as an out-of-enum verdict is, so a claim no reader can interpret never reaches the report; the file then reads as if it carried none. Two interpretable records is a hard error naming both lines — an ambiguous attribution would have `report` state a producer that may not be the one, so it is refused rather than resolved silently. + +A `findings.jsonl` written before this record existed carries none. It loads unchanged everywhere, and `report.md` states `Provenance: not recorded`. A re-ingest replaces the record together with the findings it describes; the verdict guard is unchanged and outranks it, so a file holding verdicts refuses a re-ingest whatever the provenance flags say. + **Finding record** | Field | Type | Required | Meaning | @@ -220,4 +241,4 @@ Human-readable Markdown rendered from the timeline and findings: - a header with session name, app, participant, duration (`MM:SS`, the latest moment on the timeline — the maximum over all entries, taking an utterance's end `t1` and an event's time), and utterance/event counts, plus the task list; - a **Timeline** section: each utterance as `**[MM:SS] :** “”` (curly quotes), with the events joined to it — the first utterance (in time) whose span, widened by the report's join window, contains the event — as indented bullets ``[MM:SS] `` "" value="…" ()`` (straight quotes, selector in its own code span); events matched by no utterance appear as standalone bullets in time order; every `MM:SS` in `report.md` (including the header's duration) carries a leading `-` for a negative time — one preceding `t0` (a recording predating it, see `t`'s note above) — except a time that rounds to zero, which renders `00:00` unsigned; -- a **Findings** section rendering `findings.jsonl` grouped by effective status (Confirmed, Unverified, Duplicate, Rejected), each group headed with a count and each finding line carrying its id, type, severity, clock, quote, anchor, and any verdict and date. When there is no `findings.jsonl` the section is a short notice pointing at `analyze` and `review`; when the file exists but cannot be read, the section instead reports that `findings.jsonl` could not be read, without the underlying error (`report` still exits `0`). +- a **Findings** section opening with the provenance line — the backend, model, rubric version and ingest date the [provenance record](#findingsjsonl) declares, or `Provenance: not recorded` when the file carries none — then rendering `findings.jsonl` grouped by effective status (Confirmed, Unverified, Duplicate, Rejected), each group headed with a count and each finding line carrying its id, type, severity, clock, quote, anchor, and any verdict and date. When there is no `findings.jsonl` the section is a short notice pointing at `analyze` and `review`; when the file exists but cannot be read, the section instead reports that `findings.jsonl` could not be read, without the underlying error (`report` still exits `0`). diff --git a/examples/sample-session/findings.jsonl b/examples/sample-session/findings.jsonl index 835de3d..02fb843 100644 --- a/examples/sample-session/findings.jsonl +++ b/examples/sample-session/findings.jsonl @@ -1,3 +1,4 @@ +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"unrecorded","at":"2026-07-17"} {"id":"F-001","t":22,"type":"bug","severity":3,"mode":"A","quote":"I clicked save and nothing happened","evidence":["utt-004","ev-003","ev-004"],"ui":{"selector":"[data-testid=save-btn]","route":"#general"},"status":"unverified"} {"id":"F-002","t":38,"type":"preference","severity":2,"mode":"A","quote":"I like this dark mode toggle","evidence":["utt-006","ev-006"],"ui":{"selector":"[data-testid=theme-toggle]","route":"#appearance"},"status":"unverified"} {"id":"F-003","t":38,"type":"inconsistency","severity":2,"mode":"A","quote":"This is how the save button should feel","evidence":["utt-006"],"status":"unverified"} diff --git a/internal/analyze/analyze.go b/internal/analyze/analyze.go index 4f793c3..c4a9ef8 100644 --- a/internal/analyze/analyze.go +++ b/internal/analyze/analyze.go @@ -21,6 +21,7 @@ import ( "reflect" "regexp" "strings" + "unicode/utf8" "github.com/REPPL/Testimony/internal/session" ) @@ -29,6 +30,24 @@ import ( // sessions and future rubric revisions are explicit. const RubricVersion = "testimony-analysis/v1" +// The closed backend set a Provenance record may carry. BackendUnrecorded is +// written when the operator gives no -backend: the declaration is then +// explicitly absent rather than silently missing, which is what lets every +// findings.jsonl written by this tool say something true about its own origin. +const ( + BackendLocal = "local" + BackendCloud = "cloud" + BackendUnrecorded = "unrecorded" +) + +// MaxModelLength bounds the operator-supplied model name, in runes. Model names +// are not a closed set and never will be — a validated list would refuse an +// honest answer the week after it shipped — so the field is free text, and the +// bound plus the sink sanitisation are what make it safe to carry in an artefact +// designed to be shared. The limit matches drafttests' maxTitle, the sibling +// bound on operator-supplied free text in a session record. +const MaxModelLength = 200 + // Finding is one candidate finding — one line of findings.jsonl. Finding lines // carry no "kind" field; the schema is closed (ingest decodes with // DisallowUnknownFields). @@ -51,6 +70,26 @@ type UI struct { Route string `json:"route,omitempty"` } +// Provenance is the operator's declaration of what answered the analysis +// request a findings file was ingested from. It is one line of findings.jsonl, +// discriminated by kind:"provenance", written by Ingest as the FIRST line of +// the file — ahead of every finding — because ingest replaces the whole file +// while review appends verdicts to its end: a last-position record would be +// overtaken by the first verdict appended after it, and its position would then +// carry no meaning at all. +// +// It is a declaration, not a measurement. The CLI never calls a model and +// cannot observe where the emitted request ran, so every surface that renders +// this record says as much. What the record guarantees is that the claim was +// written down, at the moment it was made, by the person who made it. +type Provenance struct { + Kind string `json:"kind"` // literal "provenance" + Rubric string `json:"rubric"` // the rubric version enforced at ingest + Backend string `json:"backend"` // local | cloud | unrecorded + Model string `json:"model,omitempty"` // free text, operator-supplied + At string `json:"at"` // YYYY-MM-DD +} + // Verdict is an appended, non-destructive human decision on a finding. It is // discriminated by kind:"verdict"; the last verdict for a finding wins. type Verdict struct { @@ -76,15 +115,129 @@ var ( } verdictSet = map[string]bool{"confirmed": true, "rejected": true, "duplicate": true} knownRubrics = map[string]bool{RubricVersion: true} + // The backend set is closed and includes "unrecorded", the value written when + // the operator names no backend. NewProvenance refuses "unrecorded" from the + // flag — it is what the absence of a declaration records, not a declaration — + // but ParseRecords must accept it on read, because it is a value this tool + // writes. + backendSet = map[string]bool{BackendLocal: true, BackendCloud: true, BackendUnrecorded: true} + isoDateRe = regexp.MustCompile(`^\d{4}-\d{2}-\d{2}$`) ) // IsFindingID reports whether s is a well-formed finding id (F-NNN). func IsFindingID(s string) bool { return findingIDRe.MatchString(s) } -// Load reads findings.jsonl from dir, splitting finding lines from appended -// verdict lines. A missing file returns an error satisfying fs.ErrNotExist so -// callers can render an absence notice. -func Load(dir string) ([]Finding, []Verdict, error) { +// NewProvenance validates the operator's declaration and returns the record +// Ingest will write. backend is "" when the flag was not given, which records +// BackendUnrecorded; model is "" when that flag was not given, and is then +// omitted from the written line. +// +// The rules live here rather than in the CLI, following ParseVerdictFlag and +// ParseKindFlag: the package that owns the record owns its rules, so the +// library and the command cannot drift about what a legal declaration is, and +// a caller that is not the CLI cannot write a record the CLI would have +// refused. The command wraps the returned error into its usage-error path, so +// a bad declaration is a wrong invocation (exit 2) rather than a runtime +// failure. +// +// Validating the length here is also what bounds the written line by +// construction: with model held to MaxModelLength runes and backend confined to +// the closed set, the encoded record cannot approach session.MaxJSONLLine, so +// the write path needs no per-line size rule for it (only the file total, which +// oversizedFindings counts it into). +// +// at is a parameter rather than time.Now() inside this package — the +// review.Options.Today precedent — so ingest stays deterministic and its golden +// tests need no clock injection. +func NewProvenance(backend, model, at string) (Provenance, error) { + switch backend { + case "", BackendLocal, BackendCloud: + default: + // BackendUnrecorded lands here deliberately: it is what the absence of a + // declaration records, not a declaration an operator states, so accepting + // it from the flag would let "unrecorded" be claimed as though it were an + // answer to the question. + return Provenance{}, fmt.Errorf("invalid -backend %q (want %s or %s)", backend, BackendLocal, BackendCloud) + } + // A model without a backend is a half-declaration: the backend is the field + // that carries the privacy claim, so a model name on its own records nothing + // about where the request ran while looking, in the report, as though it + // did. Refused for the same reason review refuses -verdict without -finding. + if model != "" && backend == "" { + return Provenance{}, fmt.Errorf("-backend is required with -model") + } + if model != "" { + // Presence is judged on the rendered form, not the raw one, matching every + // other operator-supplied string that reaches a report or a request: a + // model of invisible-only Unicode is non-empty raw but renders as nothing, + // which would print a blank model where a name belongs. + // + // The predicate is session.CodeRendersEmpty, the same one report uses to + // decide whether the model is present in its code span, and deliberately + // not a local SafeText-only test: report strips backticks when it renders + // the span, so a model of backticks alone renders as nothing there. With + // two predicates it was accepted here, echoed on the success line, and then + // reported as "not recorded" — the tool contradicting itself about what it + // had just stored. One function, one answer. + if session.CodeRendersEmpty(model) { + return Provenance{}, fmt.Errorf("-model must not be blank (it renders as nothing: whitespace, invisible characters, or backticks alone)") + } + if n := utf8.RuneCountInString(model); n > MaxModelLength { + return Provenance{}, fmt.Errorf("-model is %d characters, exceeding the limit of %d", n, MaxModelLength) + } + } + if !isoDateRe.MatchString(at) { + return Provenance{}, fmt.Errorf("invalid date %q (want YYYY-MM-DD)", at) + } + if backend == "" { + backend = BackendUnrecorded + } + return Provenance{ + Kind: "provenance", + Rubric: RubricVersion, + Backend: backend, + // The raw operator string, not its SafeText form: report sanitises at the + // sink like every other untrusted field, and storing the sanitised form + // would make the record quietly disagree with what the operator typed. + Model: model, + At: at, + }, nil +} + +// Valid reports whether p is a record Ingest may write — the shape ParseRecords +// will read back. It exists because Ingest takes a Provenance by value from its +// caller, so nothing but this check stands between a zero-valued struct and a +// findings.jsonl whose first line no reader accepts: a record with no "kind" +// falls through the discriminator to the finding branch and is refused there for +// its missing "t", which would make Ingest a writer that produces a file its own +// Load cannot open, after reporting success. +// +// The rules are exactly NewProvenance's post-conditions, so the only way to +// satisfy them is to have built the record through it. +func (p Provenance) Valid() error { + if p.Kind != "provenance" { + return fmt.Errorf("provenance record has kind %q, want \"provenance\" (build it with analyze.NewProvenance)", p.Kind) + } + if !backendSet[p.Backend] { + return fmt.Errorf("provenance record has backend %q, want %s, %s or %s (build it with analyze.NewProvenance)", + p.Backend, BackendLocal, BackendCloud, BackendUnrecorded) + } + if p.Rubric == "" { + return fmt.Errorf("provenance record has no rubric (build it with analyze.NewProvenance)") + } + if p.At == "" { + return fmt.Errorf("provenance record has no date (build it with analyze.NewProvenance)") + } + return nil +} + +// Load reads findings.jsonl from dir, splitting the provenance record from +// finding lines and appended verdict lines. A missing file returns an error +// satisfying fs.ErrNotExist so callers can render an absence notice. The +// returned provenance is nil when the file carries none — a findings.jsonl +// written before the record existed, or one assembled by hand — which every +// caller renders as "not recorded" rather than treating as a failure. +func Load(dir string) (*Provenance, []Finding, []Verdict, error) { path := filepath.Join(dir, session.FindingsFile) // Route through the read-side no-follow guard, not plain os.Open: findings.jsonl // in an exchanged (attacker-authored) session may be a symlink or a FIFO, and a @@ -92,20 +245,29 @@ func Load(dir string) ([]Finding, []Verdict, error) { // an fs.ErrNotExist-satisfying error, which callers render as an absence notice. f, err := session.OpenFileNoFollowRead(path) if err != nil { - return nil, nil, err + return nil, nil, nil, err } defer f.Close() return ParseRecords(f, path) } -// ParseRecords splits a findings.jsonl stream into finding and verdict records, -// applying the same rules Load documents: blank lines are skipped and a verdict -// carrying an out-of-enum value is ignored rather than applied. name labels -// errors. Load is ParseRecords over the on-disk file opened through the -// no-follow guard; review.AppendVerdict reuses it to re-read the current -// findings through its own already-locked descriptor, so the re-check and the -// append observe the same file under one lock. -func ParseRecords(r io.Reader, name string) ([]Finding, []Verdict, error) { +// ParseRecords splits a findings.jsonl stream into the provenance record, +// finding records, and verdict records, applying the same rules Load documents: +// blank lines are skipped, a verdict carrying an out-of-enum value is ignored +// rather than applied, and so is a provenance record whose backend is outside +// the closed set. name labels errors. Load is ParseRecords over the on-disk file +// opened through the no-follow guard; review.AppendVerdict reuses it to re-read +// the current findings through its own already-locked descriptor, so the +// re-check and the append observe the same file under one lock. +// +// This stays a reader, not a validator, exactly as it is for finding fields: a +// hand-edited or exchanged findings.jsonl reaches it directly and each sink +// already defends itself, so the provenance record's model length and date shape +// are not checked here. backend is the one exception, filtered below, because it +// is the field that *is* the claim. +func ParseRecords(r io.Reader, name string) (*Provenance, []Finding, []Verdict, error) { + var provenance *Provenance + provenanceLine := 0 var findings []Finding var verdicts []Verdict // Finding ids must be unique across the file. Ingest already rejects duplicates @@ -137,7 +299,7 @@ func ParseRecords(r io.Reader, name string) ([]Finding, []Verdict, error) { // guards against for its own callers. total += int64(len(raw)) + 1 if total > session.MaxJSONLBytes { - return nil, nil, fmt.Errorf("%s: exceeds %d bytes across %d lines; refusing to read", name, session.MaxJSONLBytes, line) + return nil, nil, nil, fmt.Errorf("%s: exceeds %d bytes across %d lines; refusing to read", name, session.MaxJSONLBytes, line) } if len(bytes.TrimSpace(raw)) == 0 { continue @@ -147,12 +309,12 @@ func ParseRecords(r io.Reader, name string) ([]Finding, []Verdict, error) { T *float64 `json:"t"` } if err := json.Unmarshal(raw, &probe); err != nil { - return nil, nil, fmt.Errorf("%s:%d: %w", name, line, err) + return nil, nil, nil, fmt.Errorf("%s:%d: %w", name, line, err) } if probe.Kind == "verdict" { var v Verdict if err := json.Unmarshal(raw, &v); err != nil { - return nil, nil, fmt.Errorf("%s:%d: %w", name, line, err) + return nil, nil, nil, fmt.Errorf("%s:%d: %w", name, line, err) } // The verdict enum is closed (confirmed|rejected|duplicate). A verdict // carrying any other value — a typo, an empty string, or a foreign @@ -167,6 +329,35 @@ func ParseRecords(r io.Reader, name string) ([]Finding, []Verdict, error) { verdicts = append(verdicts, v) continue } + if probe.Kind == "provenance" { + var p Provenance + if err := json.Unmarshal(raw, &p); err != nil { + return nil, nil, nil, fmt.Errorf("%s:%d: %w", name, line, err) + } + // The backend set is closed (local|cloud|unrecorded), and backend is the + // field that *is* the claim. A record carrying any other value — a typo, + // an empty string, or a foreign value from a shared or hand-edited + // session — states a privacy claim no reader can interpret, so it is + // ignored rather than surfaced: the file then reads as "not recorded", + // which is the truthful fallback, instead of putting an uninterpretable + // claim on the page of the shareable report. This is the same stance the + // out-of-enum verdict above takes. + if !backendSet[p.Backend] { + continue + } + // Two readable but conflicting claims are refused rather than resolved by + // a rule nobody can see. Ignoring one would make a single-valued consumer + // pick silently, and picking wrong prints a false privacy claim into the + // artefact people share — an ambiguous attribution is worse than none. + // This mirrors the duplicate-finding-id refusal below, and rests on the + // same argument. + if provenance != nil { + return nil, nil, nil, fmt.Errorf("%s:%d: duplicate provenance record (first seen at line %d); a findings file records exactly one producer", name, line, provenanceLine) + } + provenance = &p + provenanceLine = line + continue + } // A line that is JSON null (or {}) decodes cleanly into a value-typed // Finding as its zero value, so a hand-edited or exchanged findings.jsonl // carrying one silently injects a phantom finding — id "", severity 0 — @@ -175,11 +366,11 @@ func ParseRecords(r io.Reader, name string) ([]Finding, []Verdict, error) { // writes carries a real "t", so its absence means the line was never a // finding at all. if probe.T == nil { - return nil, nil, fmt.Errorf("%s:%d: not a finding or verdict record (missing t)", name, line) + return nil, nil, nil, fmt.Errorf("%s:%d: not a finding, verdict, or provenance record (missing t)", name, line) } var fnd Finding if err := json.Unmarshal(raw, &fnd); err != nil { - return nil, nil, fmt.Errorf("%s:%d: %w", name, line, err) + return nil, nil, nil, fmt.Errorf("%s:%d: %w", name, line, err) } id := session.SafeText(fnd.ID) // An empty (or, per the TrimSpace check, whitespace-only) id is @@ -196,18 +387,18 @@ func ParseRecords(r io.Reader, name string) ([]Finding, []Verdict, error) { // fallback in report.md and review's interactive walk, so it must be // refused here rather than treated as present. if strings.TrimSpace(id) == "" { - return nil, nil, fmt.Errorf("%s:%d: finding has no id; every finding must have a unique id", name, line) + return nil, nil, nil, fmt.Errorf("%s:%d: finding has no id; every finding must have a unique id", name, line) } if seenID[id] { - return nil, nil, fmt.Errorf("%s:%d: duplicate finding id %q; each finding must have a unique id", name, line, fnd.ID) + return nil, nil, nil, fmt.Errorf("%s:%d: duplicate finding id %q; each finding must have a unique id", name, line, fnd.ID) } seenID[id] = true findings = append(findings, fnd) } if err := sc.Err(); err != nil { - return nil, nil, fmt.Errorf("%s: %w", name, err) + return nil, nil, nil, fmt.Errorf("%s: %w", name, err) } - return findings, verdicts, nil + return provenance, findings, verdicts, nil } // SameIdentity reports whether a and b are the same finding — equal in every diff --git a/internal/analyze/analyze_test.go b/internal/analyze/analyze_test.go index 3a3d2b9..fa37fa4 100644 --- a/internal/analyze/analyze_test.go +++ b/internal/analyze/analyze_test.go @@ -2,6 +2,7 @@ package analyze import ( "bytes" + "encoding/json" "errors" "fmt" "os" @@ -14,6 +15,20 @@ import ( "github.com/REPPL/Testimony/internal/session" ) +// testProv is the declaration most ingest fixtures are written with: the +// "no -backend given" default, so the provenance line every written +// findings.jsonl now carries is the same one in every test that does not care +// about it. +var testProv = mustProvenance("", "", "2026-09-15") + +func mustProvenance(backend, model, at string) Provenance { + p, err := NewProvenance(backend, model, at) + if err != nil { + panic(err) + } + return p +} + // TestIngestRejectsQuoteThatSanitisesToEmpty is the verbatim-bypass regression. A // quote of only stripped characters (a lone U+202E) is raw-non-empty but SafeText // reduces it to "", and strings.Contains(text, "") is always true, so pre-fix the @@ -22,7 +37,7 @@ import ( func TestIngestRejectsQuoteThatSanitisesToEmpty(t *testing.T) { dir := writeSession(t, timelineFixture) answer := "{\"findings\":[{\"id\":\"F-001\",\"t\":22,\"type\":\"bug\",\"severity\":3,\"quote\":\"‮\",\"evidence\":[\"utt-004\"]}]}" - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil || !strings.Contains(err.Error(), "quote must be non-empty") { t.Fatalf("expected a sanitised-empty quote refusal, got %v", err) } @@ -40,7 +55,7 @@ func TestIngestRejectsQuoteThatSanitisesToEmpty(t *testing.T) { func TestIngestRejectsQuoteThatSanitisesToWhitespace(t *testing.T) { dir := writeSession(t, timelineFixture) answer := "{\"findings\":[{\"id\":\"F-001\",\"t\":22,\"type\":\"bug\",\"severity\":3,\"quote\":\"\\t\",\"evidence\":[\"utt-004\"]}]}" - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil || !strings.Contains(err.Error(), "quote must be non-empty") { t.Fatalf("expected a whitespace-only quote refusal, got %v", err) } @@ -84,7 +99,7 @@ func writeSession(t *testing.T, timeline string) string { func TestIngestGood(t *testing.T) { dir := writeSession(t, timelineFixture) - findings, err := Ingest(dir, strings.NewReader(goodAnswer)) + findings, err := Ingest(dir, strings.NewReader(goodAnswer), testProv) if err != nil { t.Fatalf("Ingest: %v", err) } @@ -96,7 +111,7 @@ func TestIngestGood(t *testing.T) { t.Fatalf("status: got %q, want unverified", findings[0].Status) } // findings.jsonl is written and reloads as unverified. - got, _, err := Load(dir) + _, got, _, err := Load(dir) if err != nil { t.Fatalf("Load: %v", err) } @@ -128,7 +143,7 @@ func TestIngestValidationFailures(t *testing.T) { for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { dir := writeSession(t, timelineFixture) - _, err := Ingest(dir, strings.NewReader(`{"findings":[`+tc.finding+`]}`)) + _, err := Ingest(dir, strings.NewReader(`{"findings":[`+tc.finding+`]}`), testProv) if err == nil { t.Fatalf("expected error containing %q, got nil", tc.want) } @@ -153,7 +168,7 @@ func TestIngestRejectsEmptyEvidenceID(t *testing.T) { tl := timelineFixture + `{"t":30,"src":"event","id":"","payload":{"kind":"click"}}` + "\n" dir := writeSession(t, tl) answer := `{"findings":[{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":["utt-004",""]}]}` - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil || !strings.Contains(err.Error(), `evidence id "" not found in the timeline`) { t.Fatalf("expected an empty-evidence-id refusal, got %v", err) } @@ -185,7 +200,7 @@ func TestIngestRejectsInvisibleOnlySelectorAndRoute(t *testing.T) { answer := fmt.Sprintf( `{"findings":[{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":["utt-004"],"ui":{"selector":%q}}]}`, zeroWidthNoBreak) - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil || !strings.Contains(err.Error(), "not present on any timeline event") { t.Fatalf("expected an invisible-only ui.selector refusal, got %v", err) } @@ -194,7 +209,7 @@ func TestIngestRejectsInvisibleOnlySelectorAndRoute(t *testing.T) { routeAnswer := fmt.Sprintf( `{"findings":[{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":["utt-004"],"ui":{"route":%q}}]}`, softHyphen) - _, err = Ingest(dir, strings.NewReader(routeAnswer)) + _, err = Ingest(dir, strings.NewReader(routeAnswer), testProv) if err == nil || !strings.Contains(err.Error(), "not present on any timeline event") { t.Fatalf("expected an invisible-only ui.route refusal, got %v", err) } @@ -219,7 +234,7 @@ func TestIngestQuoteValidatesAgainstSanitisedUtterance(t *testing.T) { // The agent's quote is the SafeText'd span — no RLM, because the request never // showed it one. answer := `{"findings":[{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"the save button and nothing happened","evidence":["utt-004"]}]}` - findings, err := Ingest(dir, strings.NewReader(answer)) + findings, err := Ingest(dir, strings.NewReader(answer), testProv) if err != nil { t.Fatalf("an honest quote copied from the sanitised request was rejected: %v", err) } @@ -234,7 +249,7 @@ func TestIngestDuplicateID(t *testing.T) { {"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":["utt-004"]}, {"id":"F-001","t":22,"type":"friction","severity":2,"quote":"No message","evidence":["utt-004"]} ]}` - _, err := Ingest(dir, strings.NewReader(dup)) + _, err := Ingest(dir, strings.NewReader(dup), testProv) if err == nil || !strings.Contains(err.Error(), "duplicate id") { t.Fatalf("expected duplicate id error, got %v", err) } @@ -253,7 +268,7 @@ func TestLoadRejectsDuplicateFindingID(t *testing.T) { if err := os.WriteFile(filepath.Join(dir, session.FindingsFile), []byte(dup), 0o644); err != nil { t.Fatalf("write findings: %v", err) } - _, _, err := Load(dir) + _, _, _, err := Load(dir) if err == nil || !strings.Contains(err.Error(), "duplicate finding id") { t.Fatalf("expected a duplicate-finding-id refusal naming line 2, got %v", err) } @@ -275,7 +290,7 @@ func TestLoadRejectsDuplicateFindingIDBySafeText(t *testing.T) { if err := os.WriteFile(filepath.Join(dir, session.FindingsFile), []byte(dup), 0o644); err != nil { t.Fatalf("write findings: %v", err) } - _, _, err := Load(dir) + _, _, _, err := Load(dir) if err == nil || !strings.Contains(err.Error(), "duplicate finding id") { t.Fatalf("expected a duplicate-finding-id refusal naming line 2, got %v", err) } @@ -298,7 +313,7 @@ func TestLoadRejectsEmptyFindingID(t *testing.T) { if err := os.WriteFile(filepath.Join(dir, session.FindingsFile), []byte(noID), 0o644); err != nil { t.Fatalf("write findings: %v", err) } - _, _, err := Load(dir) + _, _, _, err := Load(dir) if err == nil || !strings.Contains(err.Error(), "no id") { t.Fatalf("expected a no-id refusal naming line 1, got %v", err) } @@ -325,7 +340,7 @@ func TestLoadRejectsWhitespaceOnlyFindingID(t *testing.T) { if err := os.WriteFile(filepath.Join(dir, session.FindingsFile), []byte(blankID), 0o644); err != nil { t.Fatalf("write findings: %v", err) } - _, _, err := Load(dir) + _, _, _, err := Load(dir) if err == nil || !strings.Contains(err.Error(), "no id") { t.Fatalf("expected a no-id refusal naming line 1, got %v", err) } @@ -349,7 +364,7 @@ func TestLoadRejectsNullLine(t *testing.T) { if err := os.WriteFile(filepath.Join(dir, session.FindingsFile), []byte(lines), 0o644); err != nil { t.Fatalf("write findings: %v", err) } - _, _, err := Load(dir) + _, _, _, err := Load(dir) if err == nil || !strings.Contains(err.Error(), "missing t") { t.Fatalf("expected a missing-t refusal for the null line, got %v", err) } @@ -378,7 +393,7 @@ func TestLoadRejectsOversizedTotal(t *testing.T) { if err := os.WriteFile(filepath.Join(dir, session.FindingsFile), buf.Bytes(), 0o644); err != nil { t.Fatalf("write findings: %v", err) } - _, _, err := Load(dir) + _, _, _, err := Load(dir) if err == nil || !strings.Contains(err.Error(), "exceeds") { t.Fatalf("expected an oversize-total refusal, got %v", err) } @@ -386,7 +401,7 @@ func TestLoadRejectsOversizedTotal(t *testing.T) { func TestIngestUnknownRubric(t *testing.T) { dir := writeSession(t, timelineFixture) - _, err := Ingest(dir, strings.NewReader(`{"rubric":"testimony-analysis/v99","findings":[]}`)) + _, err := Ingest(dir, strings.NewReader(`{"rubric":"testimony-analysis/v99","findings":[]}`), testProv) if err == nil || !strings.Contains(err.Error(), "unknown rubric") { t.Fatalf("expected unknown rubric error, got %v", err) } @@ -395,7 +410,7 @@ func TestIngestUnknownRubric(t *testing.T) { func TestIngestBareArrayAccepted(t *testing.T) { dir := writeSession(t, timelineFixture) bare := `[{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":["utt-004"]}]` - findings, err := Ingest(dir, strings.NewReader(bare)) + findings, err := Ingest(dir, strings.NewReader(bare), testProv) if err != nil { t.Fatalf("Ingest bare array: %v", err) } @@ -406,7 +421,7 @@ func TestIngestBareArrayAccepted(t *testing.T) { func TestIngestRefusesOverwriteWithVerdicts(t *testing.T) { dir := writeSession(t, timelineFixture) - if _, err := Ingest(dir, strings.NewReader(goodAnswer)); err != nil { + if _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv); err != nil { t.Fatalf("first Ingest: %v", err) } // Append a verdict, then a re-ingest must be refused. @@ -419,7 +434,7 @@ func TestIngestRefusesOverwriteWithVerdicts(t *testing.T) { f.Close() before, _ := os.ReadFile(path) - if _, err := Ingest(dir, strings.NewReader(goodAnswer)); err == nil || !strings.Contains(err.Error(), "refusing to overwrite") { + if _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv); err == nil || !strings.Contains(err.Error(), "refusing to overwrite") { t.Fatalf("expected overwrite refusal, got %v", err) } after, _ := os.ReadFile(path) @@ -445,7 +460,7 @@ func TestIngestAcceptsNegativeAnchoredFinding(t *testing.T) { {"id":"F-001","t":-3.0,"type":"bug","severity":3,"quote":"I clicked save and nothing happened", "evidence":["utt-004","ev-003"],"status":"unverified"} ]}` - findings, err := Ingest(dir, strings.NewReader(answer)) + findings, err := Ingest(dir, strings.NewReader(answer), testProv) if err != nil { t.Fatalf("Ingest of a negative-anchored finding: %v", err) } @@ -468,7 +483,7 @@ func TestIngestRejectsFindingAfterNegativeSessionEnd(t *testing.T) { {"id":"F-001","t":-0.5,"type":"bug","severity":3,"quote":"I clicked save and nothing happened", "evidence":["utt-004","ev-003"],"status":"unverified"} ]}` - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil || !strings.Contains(err.Error(), "outside the session") { t.Fatalf("expected an out-of-range refusal for t after the negative session end, got %v", err) } @@ -479,14 +494,14 @@ func TestIngestRejectsFindingAfterNegativeSessionEnd(t *testing.T) { // empty slice with O_TRUNC and reported success. func TestIngestRefusesEmptyFindings(t *testing.T) { dir := writeSession(t, timelineFixture) - if _, err := Ingest(dir, strings.NewReader(goodAnswer)); err != nil { + if _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv); err != nil { t.Fatalf("first Ingest: %v", err) } path := filepath.Join(dir, session.FindingsFile) before, _ := os.ReadFile(path) for _, empty := range []string{`{"findings":[]}`, `[]`} { - if _, err := Ingest(dir, strings.NewReader(empty)); err == nil || !strings.Contains(err.Error(), "no findings") { + if _, err := Ingest(dir, strings.NewReader(empty), testProv); err == nil || !strings.Contains(err.Error(), "no findings") { t.Fatalf("empty answer %q: expected a no-findings refusal, got %v", empty, err) } after, _ := os.ReadFile(path) @@ -503,7 +518,7 @@ func TestIngestRefusesEmptyFindings(t *testing.T) { // so the guard saw none and the human-decision record was overwritten. func TestIngestRefusesOverwriteWithForeignVerdict(t *testing.T) { dir := writeSession(t, timelineFixture) - if _, err := Ingest(dir, strings.NewReader(goodAnswer)); err != nil { + if _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv); err != nil { t.Fatalf("first Ingest: %v", err) } path := filepath.Join(dir, session.FindingsFile) @@ -516,7 +531,7 @@ func TestIngestRefusesOverwriteWithForeignVerdict(t *testing.T) { f.Close() before, _ := os.ReadFile(path) - if _, err := Ingest(dir, strings.NewReader(goodAnswer)); err == nil || !strings.Contains(err.Error(), "refusing to overwrite") { + if _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv); err == nil || !strings.Contains(err.Error(), "refusing to overwrite") { t.Fatalf("expected overwrite refusal for a foreign-verdict file, got %v", err) } after, _ := os.ReadFile(path) @@ -734,7 +749,7 @@ func TestIngestReportsFindingPositionInAnswer(t *testing.T) { {"id":"F-002","t":22,"type":"bug","severity":3,"quote":"No message","evidence":["utt-004"],"code_refs":["x"]}, {"id":"F-001","t":22,"type":"friction","severity":2,"quote":"No message","evidence":["utt-004"]} ]}` - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil { t.Fatalf("expected a duplicate-id error, got nil") } @@ -760,7 +775,7 @@ func TestIngestLabelsUndecodableNeighbourByAnswerPosition(t *testing.T) { {"id":"F-003","t":22,"type":"bug","severity":3,"quote":"No message","evidence":["utt-004"]}, {"id":"F-4","t":22,"type":"bug","severity":3,"quote":"No message","evidence":["utt-004"]} ]}` - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil { t.Fatalf("expected an id-format error, got nil") } @@ -788,7 +803,7 @@ func (e *endlessReader) Read(p []byte) (int, error) { func TestIngestRejectsOversizedAnswer(t *testing.T) { dir := writeSession(t, timelineFixture) r := &endlessReader{} - _, err := Ingest(dir, r) + _, err := Ingest(dir, r, testProv) if err == nil || !strings.Contains(err.Error(), "refusing to ingest") { t.Fatalf("expected an over-size refusal, got %v", err) } @@ -810,7 +825,7 @@ func TestIngestRejectsOversizedEvidence(t *testing.T) { } finding := `{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":[` + strings.Join(ev, ",") + `]}` - _, err := Ingest(dir, strings.NewReader(`{"findings":[`+finding+`]}`)) + _, err := Ingest(dir, strings.NewReader(`{"findings":[`+finding+`]}`), testProv) if err == nil || !strings.Contains(err.Error(), "exceeding the limit") { t.Fatalf("expected an evidence-cardinality refusal, got %v", err) } @@ -832,7 +847,7 @@ func TestIngestRejectsFindingWithoutT(t *testing.T) { {"id":"F-001","type":"bug","severity":3,"quote":"I clicked save and nothing happened", "evidence":["utt-004","ev-003"]} ]}` - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil || !strings.Contains(err.Error(), "missing t") { t.Fatalf("expected a missing-t refusal, got %v", err) } @@ -854,7 +869,7 @@ func TestIngestAcceptsFindingAtZero(t *testing.T) { {"id":"F-001","t":0,"type":"bug","severity":3,"quote":"I clicked save and nothing happened", "evidence":["utt-004"]} ]}` - findings, err := Ingest(dir, strings.NewReader(answer)) + findings, err := Ingest(dir, strings.NewReader(answer), testProv) if err != nil { t.Fatalf("Ingest of a finding anchored at t=0: %v", err) } @@ -891,7 +906,7 @@ func TestIngestRejectsOversizedFindingLine(t *testing.T) { answer := fmt.Sprintf( `{"findings":[{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":[%q,%q,%q]}]}`, longID, longID, longID) - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil || !strings.Contains(err.Error(), "exceeding the") { t.Fatalf("expected an over-long line refusal, got %v", err) } @@ -911,7 +926,7 @@ func TestIngestOversizedFindingLeavesPriorFileIntact(t *testing.T) { longID := "utt-" + strings.Repeat("x", 2<<20) dir := writeSession(t, timelineFixture+longIDTimeline(23, longID)) - if _, err := Ingest(dir, strings.NewReader(goodAnswer)); err != nil { + if _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv); err != nil { t.Fatalf("first Ingest: %v", err) } path := filepath.Join(dir, session.FindingsFile) @@ -921,7 +936,7 @@ func TestIngestOversizedFindingLeavesPriorFileIntact(t *testing.T) { {"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":["utt-004"]}, {"id":"F-002","t":23,"type":"friction","severity":2,"quote":"No message","evidence":[%q,%q,%q]} ]}`, longID, longID, longID) - if _, err := Ingest(dir, strings.NewReader(answer)); err == nil || !strings.Contains(err.Error(), "F-002") { + if _, err := Ingest(dir, strings.NewReader(answer), testProv); err == nil || !strings.Contains(err.Error(), "F-002") { t.Fatalf("expected an over-long line refusal naming F-002, got %v", err) } after, _ := os.ReadFile(path) @@ -964,7 +979,7 @@ func TestOversizedFindingsRejectsOversizedTotal(t *testing.T) { decoded = append(decoded, positioned{finding: f, at: i}) } - errs := oversizedFindings(findings, decoded) + errs := oversizedFindings(findings, decoded, 0) joined := errors.Join(errs...) if joined == nil || !strings.Contains(joined.Error(), "file limit") { t.Fatalf("expected a total-size refusal naming the file limit, got %v", joined) @@ -1004,7 +1019,7 @@ func TestIngestRejectsOversizedFindingsTotal(t *testing.T) { t.Fatalf("test setup: answer is %d bytes, at or over session.MaxAnswerBytes (%d); the read-side cap would refuse it before this test's own check runs", len(answer), session.MaxAnswerBytes) } - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil || !strings.Contains(err.Error(), "file limit") { t.Fatalf("expected a total-size refusal naming the file limit, got %v", err) } @@ -1044,7 +1059,7 @@ func TestIngestGuardAndWriteAreOneLockedStep(t *testing.T) { done := make(chan error, 1) go func() { - _, err := Ingest(dir, strings.NewReader(goodAnswer)) + _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv) done <- err }() @@ -1120,7 +1135,7 @@ func TestIngestRefusesDuplicateTimelineIDs(t *testing.T) { "evidence":["utt-001","ev-003"]} ]}` dir := writeSession(t, dupTimeline) - _, err := Ingest(dir, strings.NewReader(answer)) + _, err := Ingest(dir, strings.NewReader(answer), testProv) if err == nil { t.Fatal("Ingest accepted a timeline with duplicate utterance ids") } @@ -1141,7 +1156,7 @@ func TestIngestRefusesUnknownTimelineSrc(t *testing.T) { {"t":100,"src":"Event","id":"ev-009","payload":{"kind":"click","selector":"[data-testid=save-btn]"}} ` dir := writeSession(t, badSrc) - _, err := Ingest(dir, strings.NewReader(goodAnswer)) + _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv) if err == nil { t.Fatal("Ingest accepted a timeline entry with unknown src") } @@ -1176,3 +1191,390 @@ func TestEmitRequestOrdersTimelineByTime(t *testing.T) { t.Fatalf("emitted request lists utt-002 (t=50) before utt-001 (t=10); timeline not sorted by time:\n%s", got) } } + +// --- provenance (itd-8 / spc-2609150759135349) --- + +// TestNewProvenanceRules pins every rule of the declaration in one table. The +// rules live in this package rather than in the CLI (the ParseVerdictFlag +// precedent), so this is where they are held: the command only wraps whatever +// comes back into its usage-error path. +func TestNewProvenanceRules(t *testing.T) { + long := strings.Repeat("m", MaxModelLength+1) + for _, tc := range []struct{ name, backend, model, at, want string }{ + {"unknown backend", "loocal", "", "2026-09-15", `invalid -backend "loocal" (want local or cloud)`}, + // "unrecorded" is what the ABSENCE of a declaration records, not a + // declaration an operator states, so it must not be claimable from the flag. + {"unrecorded is not claimable", BackendUnrecorded, "", "2026-09-15", `invalid -backend "unrecorded" (want local or cloud)`}, + {"model without backend", "", "llama3.1:70b", "2026-09-15", "-backend is required with -model"}, + {"model of invisible-only Unicode", BackendLocal, "​⁠", "2026-09-15", "-model must not be blank"}, + {"model of whitespace only", BackendLocal, " \t ", "2026-09-15", "-model must not be blank"}, + {"over-long model", BackendLocal, long, "2026-09-15", fmt.Sprintf("-model is %d characters, exceeding the limit of %d", MaxModelLength+1, MaxModelLength)}, + {"malformed date", BackendLocal, "", "15-09-2026", `invalid date "15-09-2026" (want YYYY-MM-DD)`}, + } { + t.Run(tc.name, func(t *testing.T) { + _, err := NewProvenance(tc.backend, tc.model, tc.at) + if err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("NewProvenance(%q, %q, %q) = %v, want an error containing %q", tc.backend, tc.model, tc.at, err, tc.want) + } + }) + } + + // The accepting cases, including the one the absent flag takes. + for _, tc := range []struct { + name, backend, model string + wantBackend string + }{ + {"local with a model", BackendLocal, "llama3.1:70b", BackendLocal}, + {"cloud without a model", BackendCloud, "", BackendCloud}, + {"no flag records unrecorded", "", "", BackendUnrecorded}, + {"model at exactly the limit", BackendLocal, strings.Repeat("m", MaxModelLength), BackendLocal}, + // Inline-Markdown triggers are stored raw and neutralised at each sink, so + // the record never quietly disagrees with what the operator typed. + {"inline-markdown model stored raw", BackendLocal, "![x](http://h/b.png)", BackendLocal}, + } { + t.Run(tc.name, func(t *testing.T) { + p, err := NewProvenance(tc.backend, tc.model, "2026-09-15") + if err != nil { + t.Fatalf("NewProvenance: %v", err) + } + if p.Kind != "provenance" || p.Rubric != RubricVersion || p.Backend != tc.wantBackend || p.Model != tc.model || p.At != "2026-09-15" { + t.Fatalf("NewProvenance = %+v, want kind provenance, rubric %s, backend %s, model %q", p, RubricVersion, tc.wantBackend, tc.model) + } + }) + } +} + +// TestIngestWritesProvenanceAsFirstLine is AC1: the declaration leads the file, +// and the findings under it are byte-for-byte the lines a run without the flags +// would have written — the provenance record adds a line, it does not alter one. +func TestIngestWritesProvenanceAsFirstLine(t *testing.T) { + withFlags := writeSession(t, timelineFixture) + if _, err := Ingest(withFlags, strings.NewReader(goodAnswer), mustProvenance(BackendLocal, "llama3.1:70b", "2026-09-15")); err != nil { + t.Fatalf("Ingest: %v", err) + } + lines := readLines(t, withFlags) + if len(lines) < 2 { + t.Fatalf("findings.jsonl has %d lines, want a provenance line plus findings", len(lines)) + } + var got Provenance + if err := json.Unmarshal([]byte(lines[0]), &got); err != nil { + t.Fatalf("first line is not a provenance record: %v (%q)", err, lines[0]) + } + want := Provenance{Kind: "provenance", Rubric: RubricVersion, Backend: BackendLocal, Model: "llama3.1:70b", At: "2026-09-15"} + if got != want { + t.Fatalf("first line = %+v, want %+v", got, want) + } + + // The parity assertion: the same answer ingested without the flags yields + // identical finding lines. + withoutFlags := writeSession(t, timelineFixture) + if _, err := Ingest(withoutFlags, strings.NewReader(goodAnswer), testProv); err != nil { + t.Fatalf("Ingest (no flags): %v", err) + } + a, b := readLines(t, withFlags)[1:], readLines(t, withoutFlags)[1:] + if strings.Join(a, "\n") != strings.Join(b, "\n") { + t.Fatalf("finding lines differ with and without the provenance flags:\nwith %q\nwithout %q", a, b) + } +} + +// TestIngestOmitsModelWhenNotGiven pins the omitempty contract: an unrecorded +// model is an absent key, not an empty string, so the written record carries no +// field the operator never filled in. +func TestIngestOmitsModelWhenNotGiven(t *testing.T) { + dir := writeSession(t, timelineFixture) + if _, err := Ingest(dir, strings.NewReader(goodAnswer), testProv); err != nil { + t.Fatalf("Ingest: %v", err) + } + first := readLines(t, dir)[0] + if strings.Contains(first, `"model"`) { + t.Fatalf("provenance line carries a model key when none was given: %q", first) + } + if !strings.Contains(first, `"backend":"`+BackendUnrecorded+`"`) { + t.Fatalf("provenance line does not record an unrecorded backend: %q", first) + } +} + +// TestIngestCountsProvenanceInTotalSize is the pre-flight regression: +// session.CommitRecords leaves the total-size check to its callers, so the +// provenance line's bytes must be counted or a file can land one record past +// the cap every reader then refuses. +func TestIngestCountsProvenanceInTotalSize(t *testing.T) { + var findings []Finding + var decoded []positioned + f := Finding{ + ID: "F-001", T: 22, Type: "bug", Severity: 3, + Quote: "I clicked save and nothing happened", Evidence: []string{"utt-004"}, Status: "unverified", + } + // One finding, and a provenance budget large enough on its own to blow the + // file cap: the check must be reached through the provBytes argument alone. + findings = append(findings, f) + decoded = append(decoded, positioned{finding: f, at: 1}) + + if errs := oversizedFindings(findings, decoded, 0); len(errs) != 0 { + t.Fatalf("one small finding was refused with no provenance budget: %v", errors.Join(errs...)) + } + errs := oversizedFindings(findings, decoded, int(session.MaxJSONLBytes)) + joined := errors.Join(errs...) + if joined == nil || !strings.Contains(joined.Error(), "file limit") { + t.Fatalf("the provenance line's bytes were not counted into the total: %v", joined) + } + if !strings.Contains(joined.Error(), "provenance record") { + t.Fatalf("the total-size refusal does not say the provenance record is included: %v", joined) + } +} + +// TestIngestRefusesVerdictFileWithProvenanceFlags is AC5: the verdict guard +// outranks a correctable declaration. A file holding human decisions refuses a +// re-ingest whatever the provenance flags say, and nothing in it moves. +func TestIngestRefusesVerdictFileWithProvenanceFlags(t *testing.T) { + dir := writeSession(t, timelineFixture) + if _, err := Ingest(dir, strings.NewReader(goodAnswer), mustProvenance(BackendLocal, "llama3.1:70b", "2026-09-15")); err != nil { + t.Fatalf("Ingest: %v", err) + } + path := filepath.Join(dir, session.FindingsFile) + f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644) + if err != nil { + t.Fatalf("open: %v", err) + } + if _, err := f.WriteString(`{"kind":"verdict","finding":"F-001","verdict":"confirmed","at":"2026-09-15"}` + "\n"); err != nil { + t.Fatalf("append verdict: %v", err) + } + f.Close() + before, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + + _, err = Ingest(dir, strings.NewReader(goodAnswer), mustProvenance(BackendCloud, "a-different-model", "2026-09-16")) + if err == nil || !strings.Contains(err.Error(), "refusing to overwrite") { + t.Fatalf("re-ingest over a verdict-bearing file = %v, want the existing overwrite refusal", err) + } + after, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + if !bytes.Equal(before, after) { + t.Fatalf("the refused re-ingest changed the file:\nbefore %q\nafter %q", before, after) + } +} + +// TestParseRecordsExposesProvenance is the reader half of AC1/AC2. +func TestParseRecordsExposesProvenance(t *testing.T) { + in := `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b","at":"2026-09-15"} +{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"q","evidence":["utt-004"],"status":"unverified"} +{"kind":"verdict","finding":"F-001","verdict":"confirmed","at":"2026-09-15"} +` + prov, findings, verdicts, err := ParseRecords(strings.NewReader(in), session.FindingsFile) + if err != nil { + t.Fatalf("ParseRecords: %v", err) + } + if prov == nil { + t.Fatalf("provenance record was not exposed") + } + if prov.Backend != BackendLocal || prov.Model != "llama3.1:70b" || prov.Rubric != RubricVersion || prov.At != "2026-09-15" { + t.Fatalf("provenance = %+v", *prov) + } + if len(findings) != 1 || len(verdicts) != 1 { + t.Fatalf("got %d findings and %d verdicts, want 1 and 1", len(findings), len(verdicts)) + } +} + +// TestParseRecordsToleratesMissingProvenance is AC3: a findings.jsonl written +// before the record existed still loads, and says so by returning nil. +func TestParseRecordsToleratesMissingProvenance(t *testing.T) { + in := `{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"q","evidence":["utt-004"],"status":"unverified"} +{"kind":"verdict","finding":"F-001","verdict":"confirmed","at":"2026-09-15"} +` + prov, findings, verdicts, err := ParseRecords(strings.NewReader(in), session.FindingsFile) + if err != nil { + t.Fatalf("ParseRecords: %v", err) + } + if prov != nil { + t.Fatalf("provenance = %+v, want nil for a file that carries none", *prov) + } + if len(findings) != 1 || len(verdicts) != 1 { + t.Fatalf("got %d findings and %d verdicts, want 1 and 1", len(findings), len(verdicts)) + } +} + +// TestParseRecordsIgnoresUnknownBackend mirrors the out-of-enum verdict rule: a +// claim no reader can interpret must not reach the shareable report, so the +// record is dropped and the file reads as "not recorded" — the truthful +// fallback — rather than failing to load at all. +func TestParseRecordsIgnoresUnknownBackend(t *testing.T) { + for _, backend := range []string{"loocal", "", "on-prem"} { + in := fmt.Sprintf(`{"kind":"provenance","rubric":"testimony-analysis/v1","backend":%q,"at":"2026-09-15"}`, backend) + "\n" + + `{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"q","evidence":["utt-004"],"status":"unverified"}` + "\n" + prov, findings, _, err := ParseRecords(strings.NewReader(in), session.FindingsFile) + if err != nil { + t.Fatalf("backend %q: ParseRecords: %v", backend, err) + } + if prov != nil { + t.Fatalf("backend %q: an uninterpretable backend was surfaced as %+v", backend, *prov) + } + if len(findings) != 1 { + t.Fatalf("backend %q: the findings stopped loading (%d)", backend, len(findings)) + } + } +} + +// TestParseRecordsRefusesDuplicateProvenance mirrors the duplicate-finding-id +// refusal: two readable but conflicting claims would make a single-valued +// consumer pick one silently, and picking wrong prints a false privacy claim +// into the artefact people share. +func TestParseRecordsRefusesDuplicateProvenance(t *testing.T) { + in := `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","at":"2026-09-15"} +{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"q","evidence":["utt-004"],"status":"unverified"} +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"cloud","at":"2026-09-16"} +` + _, _, _, err := ParseRecords(strings.NewReader(in), session.FindingsFile) + if err == nil || !strings.Contains(err.Error(), "duplicate provenance record (first seen at line 1)") { + t.Fatalf("ParseRecords = %v, want a duplicate-provenance refusal naming both lines", err) + } + if !strings.Contains(err.Error(), ":3:") { + t.Fatalf("the refusal does not name the offending line: %v", err) + } + + // An ignored (out-of-enum) record is not a first sighting, so it cannot make + // a single legitimate record look like a duplicate. + ok := `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"nonsense","at":"2026-09-15"} +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","at":"2026-09-16"} +` + prov, _, _, err := ParseRecords(strings.NewReader(ok), session.FindingsFile) + if err != nil { + t.Fatalf("an ignored record was counted as a duplicate: %v", err) + } + if prov == nil || prov.Backend != BackendLocal { + t.Fatalf("provenance = %v, want the one interpretable record", prov) + } +} + +// TestParseRecordsProvenanceAnywhereInFile pins that first-line position is a +// writer convention, not a reader requirement: a hand-edited file that puts the +// record last still has it read. +func TestParseRecordsProvenanceAnywhereInFile(t *testing.T) { + in := `{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"q","evidence":["utt-004"],"status":"unverified"} +{"kind":"verdict","finding":"F-001","verdict":"confirmed","at":"2026-09-15"} +{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"cloud","at":"2026-09-15"} +` + prov, findings, verdicts, err := ParseRecords(strings.NewReader(in), session.FindingsFile) + if err != nil { + t.Fatalf("ParseRecords: %v", err) + } + if prov == nil || prov.Backend != BackendCloud { + t.Fatalf("a trailing provenance record was not read: %v", prov) + } + if len(findings) != 1 || len(verdicts) != 1 { + t.Fatalf("got %d findings and %d verdicts, want 1 and 1", len(findings), len(verdicts)) + } +} + +// TestParseRecordsProvenanceCountsTowardTotalCap: the new record is not exempt +// from the read-side total-size invariant every other line obeys. +func TestParseRecordsProvenanceCountsTowardTotalCap(t *testing.T) { + var b strings.Builder + line := `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"nonsense","model":"` + strings.Repeat("m", 4096) + `","at":"2026-09-15"}` + "\n" + for b.Len() <= int(session.MaxJSONLBytes) { + b.WriteString(line) + } + _, _, _, err := ParseRecords(strings.NewReader(b.String()), session.FindingsFile) + if err == nil || !strings.Contains(err.Error(), "refusing to read") { + t.Fatalf("ParseRecords = %v, want the total-size refusal", err) + } +} + +// readLines returns the non-blank lines of the session's findings.jsonl. +func readLines(t *testing.T, dir string) []string { + t.Helper() + b, err := os.ReadFile(filepath.Join(dir, session.FindingsFile)) + if err != nil { + t.Fatalf("read findings: %v", err) + } + var out []string + for _, l := range strings.Split(string(b), "\n") { + if strings.TrimSpace(l) != "" { + out = append(out, l) + } + } + return out +} + +// TestIngestRefusesInvalidProvenance is the writer-reader agreement regression: +// Ingest takes a Provenance by value, so nothing but its own guard stood between +// a zero-valued struct and a findings.jsonl whose first line ParseRecords +// refuses. Pre-fix, Ingest(dir, r, Provenance{}) committed +// {"kind":"","rubric":"","backend":"","at":""} and reported success, and the +// very next Load of that file failed with "missing t" — a writer producing a +// file no reader accepts. +func TestIngestRefusesInvalidProvenance(t *testing.T) { + good := mustProvenance(BackendLocal, "llama3.1:70b", "2026-09-15") + for _, tc := range []struct { + name string + prov Provenance + want string + }{ + {"zero value", Provenance{}, `kind "", want "provenance"`}, + {"wrong kind", func() Provenance { p := good; p.Kind = "verdict"; return p }(), `kind "verdict"`}, + {"backend outside the set", func() Provenance { p := good; p.Backend = "on-prem"; return p }(), `backend "on-prem"`}, + {"no rubric", func() Provenance { p := good; p.Rubric = ""; return p }(), "has no rubric"}, + {"no date", func() Provenance { p := good; p.At = ""; return p }(), "has no date"}, + } { + t.Run(tc.name, func(t *testing.T) { + dir := writeSession(t, timelineFixture) + _, err := Ingest(dir, strings.NewReader(goodAnswer), tc.prov) + if err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("Ingest = %v, want an error containing %q", err, tc.want) + } + if !strings.Contains(err.Error(), "analyze.NewProvenance") { + t.Fatalf("the refusal does not name the constructor to use: %v", err) + } + // Refused before anything is read or written. + if _, statErr := os.Stat(filepath.Join(dir, session.FindingsFile)); statErr == nil { + t.Fatalf("findings.jsonl was written despite an unwritable provenance record") + } + }) + } + + // And the positive control: every record NewProvenance builds is writable, + // and the file it lands in reads back. + for _, p := range []Provenance{ + mustProvenance(BackendLocal, "llama3.1:70b", "2026-09-15"), + mustProvenance(BackendCloud, "", "2026-09-15"), + mustProvenance("", "", "2026-09-15"), + } { + dir := writeSession(t, timelineFixture) + if _, err := Ingest(dir, strings.NewReader(goodAnswer), p); err != nil { + t.Fatalf("Ingest(%+v): %v", p, err) + } + got, findings, _, err := Load(dir) + if err != nil { + t.Fatalf("Load after Ingest(%+v): %v", p, err) + } + if got == nil || *got != p { + t.Fatalf("Load returned %v, want %+v", got, p) + } + if len(findings) != 1 { + t.Fatalf("got %d findings, want 1", len(findings)) + } + } +} + +// TestNewProvenanceRefusesBacktickOnlyModel pins the one-predicate rule: report +// renders the model inside a code span, which strips backticks, so a model of +// backticks alone renders as nothing there. Pre-fix NewProvenance judged +// emptiness with SafeText only, so `-model '```'` was accepted, echoed on the +// success line, and then reported as "model not recorded" — the tool +// contradicting itself about what it had just stored. +func TestNewProvenanceRefusesBacktickOnlyModel(t *testing.T) { + for _, model := range []string{"`", "```", " `` \t"} { + _, err := NewProvenance(BackendLocal, model, "2026-09-15") + if err == nil || !strings.Contains(err.Error(), "-model must not be blank") { + t.Fatalf("NewProvenance(model %q) = %v, want the blank-model refusal", model, err) + } + } + // A model that merely contains a backtick is still real content and stays + // accepted — the refusal is about rendering as nothing, not about the byte. + if _, err := NewProvenance(BackendLocal, "x`y", "2026-09-15"); err != nil { + t.Fatalf("NewProvenance(model \"x`y\") = %v, want it accepted", err) + } +} diff --git a/internal/analyze/ingest.go b/internal/analyze/ingest.go index 862def3..703195f 100644 --- a/internal/analyze/ingest.go +++ b/internal/analyze/ingest.go @@ -75,7 +75,22 @@ func LoadTimeline(dir string) ([]timeline.Entry, error) { // stray fields are all rejected here, transactionally (all errors reported, // nothing written on any failure). To protect the retained precision record it // refuses to overwrite a findings.jsonl that already holds verdict records. -func Ingest(dir string, r io.Reader) ([]Finding, error) { +// +// prov is the operator's declaration of what answered the request (see +// NewProvenance, which is the only way to build a valid one). It is written as +// the first line of the same file, in the same commit, so a re-ingest replaces +// the provenance record together with the findings it accompanies: a provenance +// line can never outlive the findings it describes, and findings can never +// acquire a provenance from a different run. +func Ingest(dir string, r io.Reader, prov Provenance) ([]Finding, error) { + // Checked before anything is read, let alone written: this file's first line + // is the caller's to supply, and an unwritable one is a fact about the + // argument alone — the same reason AppendRecord pre-flights its record before + // the open. Without it a zero-valued Provenance commits a line ParseRecords + // refuses, so the very next Load of a file Ingest reported writing fails. + if err := prov.Valid(); err != nil { + return nil, err + } entries, err := LoadTimeline(dir) if err != nil { return nil, err @@ -137,13 +152,17 @@ func Ingest(dir string, r io.Reader) ([]Finding, error) { findings[i] = p.finding findings[i].Status = "unverified" } - errs = append(errs, oversizedFindings(findings, decoded)...) + provLine, err := json.Marshal(prov) + if err != nil { + return nil, fmt.Errorf("write %s: %w", session.FindingsFile, err) + } + errs = append(errs, oversizedFindings(findings, decoded, len(provLine)+1)...) if len(errs) > 0 { return nil, errors.Join(errs...) } - if err := commitFindings(dir, findings); err != nil { + if err := commitFindings(dir, provLine, findings); err != nil { return nil, err } return findings, nil @@ -163,9 +182,15 @@ func Ingest(dir string, r io.Reader) ([]Finding, error) { // Each finding is encoded with json.Marshal, the same encoder (HTML escaping on, // Go's default) oversizedFindings measures with, so the bytes written are exactly // the bytes that passed the size check. -func commitFindings(dir string, findings []Finding) error { +func commitFindings(dir string, provLine []byte, findings []Finding) error { path := filepath.Join(dir, session.FindingsFile) - records := make([][]byte, 0, len(findings)) + // The provenance record leads, ahead of every finding: the file then reads in + // the order it was decided — the producer, what it produced, then the human + // verdicts review appends to the end — and `head -1` is the answer to "what + // wrote this?". Riding in this same Records slice is also what makes a + // re-ingest replace the declaration together with the findings it describes. + records := make([][]byte, 0, len(findings)+1) + records = append(records, provLine) for _, f := range findings { b, err := json.Marshal(f) if err != nil { @@ -211,9 +236,15 @@ func commitFindings(dir string, findings []Finding) error { // each finding's answer position for the same reason validate's do; a line // already flagged as over-long is excluded from the total so one oversized // finding cannot also trigger a redundant total-size error. -func oversizedFindings(findings []Finding, decoded []positioned) []error { +func oversizedFindings(findings []Finding, decoded []positioned, provBytes int) []error { var errs []error - var total int64 + // The provenance line is written into the same file by the same commit, so the + // total-size pre-flight must measure it too: CommitRecords delegates that + // pre-flight to its callers, and omitting these bytes would let a file land one + // record past the cap every reader then refuses. It needs no per-line check — + // NewProvenance bounds the model to MaxModelLength runes and the backend to a + // closed set, so the encoded record cannot approach MaxJSONLLine. + total := int64(provBytes) var counted int for i, f := range findings { label := findingLabel(f, decoded[i].at) @@ -231,7 +262,7 @@ func oversizedFindings(findings []Finding, decoded []positioned) []error { counted++ } if total > session.MaxJSONLBytes { - errs = append(errs, fmt.Errorf("findings encode to %d bytes across %d findings, exceeding the %d-byte %s file limit ParseRecords enforces; refusing to write a file report and review could not read back", total, counted, session.MaxJSONLBytes, session.FindingsFile)) + errs = append(errs, fmt.Errorf("%d findings and the provenance record encode to %d bytes, exceeding the %d-byte %s file limit ParseRecords enforces; refusing to write a file report and review could not read back", counted, total, session.MaxJSONLBytes, session.FindingsFile)) } return errs } diff --git a/internal/cli/cli.go b/internal/cli/cli.go index ada88cf..bbf3ba7 100644 --- a/internal/cli/cli.go +++ b/internal/cli/cli.go @@ -41,6 +41,7 @@ Usage: testimony report [-session DIR] [-window 2.5] render timeline.jsonl as a Markdown report testimony analyze [-session DIR] [-out FILE] emit the analysis request (rubric + timeline) on stdout or to FILE testimony analyze [-session DIR] -ingest FILE validate answer JSON (FILE or "-") → findings.jsonl (all findings unverified) + [-backend local|cloud] [-model NAME] record what answered the request, beside the findings testimony draft-tests [-session DIR] [-window 10] [-out FILE] emit the regression-test drafting request (rubric + confirmed findings + event windows) testimony draft-tests [-session DIR] -ingest FILE validate answer JSON (FILE or "-") → tests.jsonl (all drafts proposed) testimony draft-tests [-session DIR] -render [-out FILE] render the accepted drafts as Markdown test cases @@ -421,17 +422,24 @@ func Run(args []string) int { dir := fs.String("session", "", "session directory") out := fs.String("out", "", "write the emitted request to FILE instead of stdout") ingest := fs.String("ingest", "", "validate answer JSON at FILE (or \"-\" for stdin) into findings.jsonl") + backend := fs.String("backend", "", "ingest mode: record which backend answered the request: local | cloud") + model := fs.String("model", "", "ingest mode: record the model that answered the request (free text)") fs.Parse(rest) if err := rejectArgs(fs); err != nil { return usageErr(err) } outSet, ingestSet := false, false + backendSet, modelSet := false, false fs.Visit(func(f *flag.Flag) { switch f.Name { case "out": outSet = true case "ingest": ingestSet = true + case "backend": + backendSet = true + case "model": + modelSet = true } }) // An explicitly-empty -ingest or -out is a wrong invocation (an unset @@ -451,6 +459,34 @@ func Run(args []string) int { if *ingest != "" && *out != "" { return usageErr(fmt.Errorf("analyze: -out and -ingest cannot be combined")) } + // An explicitly-empty -backend or -model is a wrong invocation for the same + // reason -ingest/-out are, and the guards must come before NewProvenance + // below: an empty -backend would otherwise fall through its "flag not given" + // branch and silently record "unrecorded" for an operator who believed they + // had named a backend — the provenance record then states the opposite of + // what they meant it to. + if backendSet && *backend == "" { + return usageErr(fmt.Errorf("analyze: -backend must not be empty")) + } + if modelSet && *model == "" { + return usageErr(fmt.Errorf("analyze: -model must not be empty")) + } + // Both flags record what answered the request, which only ingest has an + // answer to record against; emit mutates nothing in the session directory. + // Silently ignored, they would let an operator who meant to record their + // backend believe they had — the draft-tests -window precedent, which + // refuses rather than ignores a flag that does nothing in the mode you are + // in. + if *ingest == "" && (backendSet || modelSet) { + return usageErr(fmt.Errorf("analyze: -backend and -model apply to the ingest mode only")) + } + // The declaration's rules live in internal/analyze, which owns the record — + // the review.ParseVerdictFlag precedent — so a bad declaration surfaces here + // as a wrong invocation (exit 2) rather than as a runtime failure. + prov, err := analyze.NewProvenance(*backend, *model, time.Now().Format("2006-01-02")) + if err != nil { + return usageErr(fmt.Errorf("analyze: %w", err)) + } // Resolved last of the invocation checks (see report above): a run refused // for another flag must not first announce an inferred session. sess, err := resolveSession(fs, *dir) @@ -458,6 +494,18 @@ func Run(args []string) int { return usageErr(err) } if *ingest != "" { + // An implicit choice must at least be visible in the output of the run + // that made it — resolveSession's inferred-session notice, applied to the + // other implicit choice this command makes. On stderr, so a caller piping + // analyze's output is unaffected. + // + // The tense is deliberate. This prints before the answer is validated, so + // it also prints on runs that go on to fail and write nothing; "will + // record" states an intention that a later refusal simply overtakes, + // where "recording" would claim something the run never did. + if !backendSet { + fmt.Fprintln(os.Stderr, `analyze: no -backend given; the provenance will record "backend not recorded"`) + } in := os.Stdin if *ingest != "-" { // Read the answer file through the no-follow guard, like every other @@ -472,12 +520,12 @@ func Run(args []string) int { defer f.Close() in = f } - findings, err := analyze.Ingest(sess, in) + findings, err := analyze.Ingest(sess, in, prov) if err != nil { return fail(err) } - fmt.Printf("validated %d findings → %s (all unverified)\n", - len(findings), filepath.Join(sess, session.FindingsFile)) + fmt.Printf("validated %d findings → %s (all unverified; %s)\n", + len(findings), filepath.Join(sess, session.FindingsFile), describeProvenance(prov)) return 0 } prompt, err := analyze.EmitRequest(sess) @@ -817,6 +865,35 @@ func Run(args []string) int { } } +// describeProvenance renders the declaration for the ingest success line, so the +// operator sees what was recorded in the output of the run that recorded it +// rather than having to open findings.jsonl to find out. +// +// The backend phrase comes from a switch on the closed enum, never from the +// stored string; the model is operator-supplied text reaching a terminal, so it +// goes through session.SafeText — the same treatment the emitted request gives +// manifest fields — and falls back to the placeholder when it renders as +// nothing. +func describeProvenance(p analyze.Provenance) string { + var backend string + switch p.Backend { + case analyze.BackendLocal: + backend = "local backend" + case analyze.BackendCloud: + backend = "cloud backend" + default: + backend = "backend not recorded" + } + // Every branch falls through to the model clause rather than returning early: + // the two halves of the declaration are independent, so a record that carries + // a model must say so whatever its backend reads, and the line keeps one shape + // the operator can scan for in all three cases. + if session.CodeRendersEmpty(p.Model) { + return backend + ", model not recorded" + } + return backend + ", model " + session.SafeText(p.Model) +} + // rejectArgs refuses leftover positional arguments after flag parsing. Flag // parsing stops at the first non-flag argument, so a stray positional silently // discarded every flag that followed it and the command ran with defaults at diff --git a/internal/cli/cli_test.go b/internal/cli/cli_test.go index 7fe8c25..81076a2 100644 --- a/internal/cli/cli_test.go +++ b/internal/cli/cli_test.go @@ -304,6 +304,7 @@ func TestInvalidFlagValuesExitTwo(t *testing.T) { func TestUsageListsEveryFlagAndCommand(t *testing.T) { for _, want := range []string{"-commit HASH", "testimony help", "testimony draft-tests", "-window 10", "-kind findings|tests", "-decision edited -edit FILE", + "-backend local|cloud", "-model NAME", "transcribe, import, merge, report, analyze, draft-tests, or"} { if !strings.Contains(usage, want) { t.Errorf("usage text does not mention %q", want) @@ -1393,3 +1394,146 @@ func TestUsageShowsTheFixedDefaultRoot(t *testing.T) { t.Error("usage text still advertises the old relative sessions/ default") } } + +// --- provenance flags (itd-8 / spc-2609150759135349) --- + +// miniAnswer writes a one-finding answer valid against miniSession's timeline. +func miniAnswer(t *testing.T, dir string) string { + t.Helper() + path := filepath.Join(dir, "answer.json") + body := `{"rubric":"testimony-analysis/v1","findings":[{"id":"F-001","t":0,"type":"bug","severity":3,"quote":"hi","evidence":["utt-001"]}]}` + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatalf("write answer: %v", err) + } + return path +} + +func firstFindingsLine(t *testing.T, dir string) string { + t.Helper() + b, err := os.ReadFile(filepath.Join(dir, session.FindingsFile)) + if err != nil { + t.Fatalf("read findings: %v", err) + } + return strings.SplitN(strings.TrimRight(string(b), "\n"), "\n", 2)[0] +} + +// TestAnalyzeIngestRecordsProvenance is AC1 through the command: the declaration +// reaches findings.jsonl and the run says what it recorded, so the operator sees +// it without opening the file. +func TestAnalyzeIngestRecordsProvenance(t *testing.T) { + for _, tc := range []struct { + name string + args []string + wantClause string + wantFirstLine string + }{ + { + "local with a model", + []string{"-backend", "local", "-model", "llama3.1:70b"}, + "(all unverified; local backend, model llama3.1:70b)", + `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b",`, + }, + { + "cloud without a model", + []string{"-backend", "cloud"}, + "(all unverified; cloud backend, model not recorded)", + `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"cloud","at":`, + }, + } { + t.Run(tc.name, func(t *testing.T) { + dir := miniSession(t) + answer := miniAnswer(t, dir) + var code int + stdout := captureStdout(t, func() { + code = Run(append([]string{"analyze", "-session", dir, "-ingest", answer}, tc.args...)) + }) + if code != 0 { + t.Fatalf("exit %d, want 0", code) + } + if !strings.Contains(stdout, tc.wantClause) { + t.Fatalf("success line does not carry %q: %q", tc.wantClause, stdout) + } + if first := firstFindingsLine(t, dir); !strings.HasPrefix(first, tc.wantFirstLine) { + t.Fatalf("first line of findings.jsonl = %q, want it to start %q", first, tc.wantFirstLine) + } + }) + } +} + +// TestAnalyzeIngestWithoutBackendAnnouncesUnrecorded is AC4. The flags are +// optional so no existing invocation breaks, but an implicit choice must be +// visible in the output of the run that made it — resolveSession's +// inferred-session notice, applied to the other implicit choice this command +// makes. The notice belongs on stderr so a caller piping analyze is unaffected. +func TestAnalyzeIngestWithoutBackendAnnouncesUnrecorded(t *testing.T) { + dir := miniSession(t) + answer := miniAnswer(t, dir) + + var code int + var stdout string + stderr := captureStderr(t, func() { + stdout = captureStdout(t, func() { + code = Run([]string{"analyze", "-session", dir, "-ingest", answer}) + }) + }) + if code != 0 { + t.Fatalf("exit %d, want 0 — the provenance flags are optional", code) + } + const want = `analyze: no -backend given; the provenance will record "backend not recorded"` + if !strings.Contains(stderr, want) { + t.Fatalf("stderr does not carry the notice %q: %q", want, stderr) + } + if strings.Contains(stdout, "no -backend given") { + t.Fatalf("the notice leaked onto stdout: %q", stdout) + } + if !strings.Contains(stdout, "(all unverified; backend not recorded, model not recorded)") { + t.Fatalf("success line does not report the unrecorded backend: %q", stdout) + } + first := firstFindingsLine(t, dir) + if !strings.Contains(first, `"backend":"unrecorded"`) { + t.Fatalf("first line does not record an unrecorded backend: %q", first) + } + if strings.Contains(first, `"model"`) { + t.Fatalf("first line carries a model key when none was given: %q", first) + } +} + +// TestAnalyzeProvenanceFlagsAreUsageErrors extends the exit-2 family: an +// explicitly-empty flag is an unset shell variable spliced into the invocation, +// and a flag that does nothing in the mode you are in is refused rather than +// silently ignored (the draft-tests -window precedent). The empty-flag guards +// matter most of all: without them an empty -backend would fall through to the +// "not given" branch and record the opposite of what the operator meant. +func TestAnalyzeProvenanceFlagsAreUsageErrors(t *testing.T) { + dir := miniSession(t) + answer := miniAnswer(t, dir) + for _, tc := range []struct { + name string + args []string + want string + }{ + {"empty backend", []string{"-ingest", answer, "-backend", ""}, "analyze: -backend must not be empty"}, + {"empty model", []string{"-ingest", answer, "-model", ""}, "analyze: -model must not be empty"}, + {"unknown backend", []string{"-ingest", answer, "-backend", "loocal"}, `analyze: invalid -backend "loocal" (want local or cloud)`}, + {"unrecorded is not claimable", []string{"-ingest", answer, "-backend", "unrecorded"}, `analyze: invalid -backend "unrecorded" (want local or cloud)`}, + {"model without backend", []string{"-ingest", answer, "-model", "llama"}, "analyze: -backend is required with -model"}, + // A model of backticks alone renders as nothing in report's code span, so + // accepting it here would have the success line and the report disagree. + {"model of backticks only", []string{"-ingest", answer, "-backend", "local", "-model", "```"}, "analyze: -model must not be blank"}, + {"backend in emit mode", []string{"-backend", "local"}, "analyze: -backend and -model apply to the ingest mode only"}, + {"model in emit mode", []string{"-model", "llama"}, "analyze: -backend and -model apply to the ingest mode only"}, + } { + t.Run(tc.name, func(t *testing.T) { + var code int + stderr := captureStderr(t, func() { + code = Run(append([]string{"analyze", "-session", dir}, tc.args...)) + }) + if code != 2 { + t.Fatalf("exit %d, want 2 (usage error)", code) + } + if !strings.Contains(stderr, "testimony: "+tc.want) { + t.Fatalf("stderr = %q, want %q", stderr, tc.want) + } + }) + } +} diff --git a/internal/drafttests/drafttests.go b/internal/drafttests/drafttests.go index 15a7ebb..b0a347b 100644 --- a/internal/drafttests/drafttests.go +++ b/internal/drafttests/drafttests.go @@ -417,7 +417,10 @@ func noAcceptedDrafts(dir string, drafts []Draft, decisions []Decision) error { // first when there is none — every mode of this package needs them, because a // draft is only ever a proposal about a finding a human confirmed. func loadFindings(dir string) ([]analyze.Finding, []analyze.Verdict, error) { - findings, verdicts, err := analyze.Load(dir) + // The analysis provenance is discarded: the drafting layer has its own rubric + // and its own record, and which backend coded a finding changes no instruction + // in the drafting request. + _, findings, verdicts, err := analyze.Load(dir) if err != nil { if errors.Is(err, fs.ErrNotExist) { return nil, nil, fmt.Errorf("no %s (run `testimony analyze -ingest` first)", session.FindingsFile) diff --git a/internal/drafttests/drafttests_test.go b/internal/drafttests/drafttests_test.go index de8d472..807ca37 100644 --- a/internal/drafttests/drafttests_test.go +++ b/internal/drafttests/drafttests_test.go @@ -1,6 +1,7 @@ package drafttests import ( + "bytes" "os" "path/filepath" "strings" @@ -67,7 +68,7 @@ func readEntries(t *testing.T) []timeline.Entry { func loadFixtureFindings(t *testing.T) ([]analyze.Finding, []analyze.Verdict) { t.Helper() - findings, verdicts, err := analyze.Load(writeSession(t)) + _, findings, verdicts, err := analyze.Load(writeSession(t)) if err != nil { t.Fatalf("analyze.Load: %v", err) } @@ -396,3 +397,50 @@ func TestClockRendersNegativeSessionRelativeTimes(t *testing.T) { } } } + +// TestDraftTestsIgnoresProvenanceRecord pins the itd-8 non-goal: the analysis +// provenance is not carried into the drafting request and not written into +// tests.jsonl. Which backend coded a finding changes no instruction in a request +// that asks for reproduction steps, so every mode must produce byte-identical +// output over a findings.jsonl that carries the record and one that does not. +func TestDraftTestsIgnoresProvenanceRecord(t *testing.T) { + const provLine = `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b","at":"2026-09-15"}` + + plain := writeSession(t) + withProv := writeSession(t, session.FindingsFile, provLine+"\n"+string(fixture(t, "findings.jsonl"))) + + reqA, err := EmitRequest(plain, 10) + if err != nil { + t.Fatalf("EmitRequest (plain): %v", err) + } + reqB, err := EmitRequest(withProv, 10) + if err != nil { + t.Fatalf("EmitRequest (with provenance): %v", err) + } + if reqA != reqB { + t.Fatalf("the drafting request differs with a provenance record present") + } + if strings.Contains(reqB, "llama3.1:70b") || strings.Contains(reqB, "provenance") { + t.Fatalf("the drafting request leaked the analysis provenance:\n%s", reqB) + } + + for _, dir := range []string{plain, withProv} { + if _, err := Ingest(dir, strings.NewReader(string(fixture(t, "answer.json")))); err != nil { + t.Fatalf("Ingest: %v", err) + } + } + a, err := os.ReadFile(filepath.Join(plain, session.TestsFile)) + if err != nil { + t.Fatalf("read tests (plain): %v", err) + } + b, err := os.ReadFile(filepath.Join(withProv, session.TestsFile)) + if err != nil { + t.Fatalf("read tests (with provenance): %v", err) + } + if !bytes.Equal(a, b) { + t.Fatalf("tests.jsonl differs with a provenance record present:\n%s\n%s", a, b) + } + if bytes.Contains(b, []byte("provenance")) { + t.Fatalf("tests.jsonl gained a provenance record: %s", b) + } +} diff --git a/internal/drafttests/review.go b/internal/drafttests/review.go index ddab1ff..ce6f70a 100644 --- a/internal/drafttests/review.go +++ b/internal/drafttests/review.go @@ -69,7 +69,7 @@ func Review(opts ReviewOptions) error { // absent findings.jsonl degrades to placeholders rather than blocking a human // decision that is already overdue. func findingsFor(dir string) []analyze.Finding { - findings, _, err := analyze.Load(dir) + _, findings, _, err := analyze.Load(dir) if err != nil { return nil } diff --git a/internal/report/report.go b/internal/report/report.go index 33bd03f..6bd9d25 100644 --- a/internal/report/report.go +++ b/internal/report/report.go @@ -146,11 +146,12 @@ func Render(dir string, window float64) (string, error) { return b.String(), nil } -// renderFindings appends the Findings section, grouping findings.jsonl by -// effective status. When no findings file exists it leaves a short, non-fatal -// notice. Report reads only derived text; it never touches media. +// renderFindings appends the Findings section: the provenance line, then the +// findings grouped by effective status. When no findings file exists it leaves +// a short, non-fatal notice — and no provenance line, since there is no file to +// have a provenance. Report reads only derived text; it never touches media. func renderFindings(b *strings.Builder, dir string) { - findings, verdicts, err := analyze.Load(dir) + prov, findings, verdicts, err := analyze.Load(dir) if err != nil { if errors.Is(err, fs.ErrNotExist) { b.WriteString("_No findings yet — run `testimony analyze` then `testimony review`._\n") @@ -167,6 +168,8 @@ func renderFindings(b *strings.Builder, dir string) { return } + renderProvenance(b, prov) + eff := analyze.EffectiveStatus(findings, verdicts) byStatus := map[string][]analyze.Finding{} for _, f := range findings { @@ -219,6 +222,58 @@ func renderFindings(b *strings.Builder, dir string) { } } +// renderProvenance writes the one line that says what produced these findings, +// directly under the Findings heading and above the first status group. +// +// The wording is load-bearing, not decoration. The report is the artefact that +// travels — to a co-author, an archive, an ethics reviewer — and the record is +// the operator's declaration, which Testimony has no way to verify: the CLI +// never calls a model and cannot observe where the emitted request ran. "(as +// declared at ingest)" is what keeps the line from reading as though the tool +// measured it. +// +// backend is rendered from a switch on the closed enum rather than from the +// parsed string, so no untrusted byte reaches that position at all — analyze's +// reader has already dropped any record whose backend is outside the set, and +// this switch means even a future widening of that set cannot leak raw bytes +// here. Every other field is operator-supplied or hand-editable text and takes +// the same sink defence the rest of this file applies: presence is decided on +// the rendered form, never on raw emptiness. +func renderProvenance(b *strings.Builder, p *analyze.Provenance) { + if p == nil { + // No provenance record: a findings.jsonl written before the record existed, + // one assembled by hand, or one whose only provenance line carried a backend + // no reader can interpret. Saying so plainly is the truthful fallback. + b.WriteString("_Provenance: not recorded._\n\n") + return + } + var backend string + switch p.Backend { + case analyze.BackendLocal: + backend = "local backend" + case analyze.BackendCloud: + backend = "cloud backend" + default: + backend = "backend not recorded" + } + model := "model not recorded" + if !session.CodeRendersEmpty(p.Model) { + model = "model " + mdCode(p.Model) + } + rubric := "rubric not recorded" + if !session.CodeRendersEmpty(p.Rubric) { + rubric = "rubric " + mdCode(p.Rubric) + } + fmt.Fprintf(b, "_Provenance (as declared at ingest): %s · %s · %s", backend, model, rubric) + // The date clause is dropped entirely when it renders to nothing, rather than + // printed blank — the same rule the verdict suffix above follows for its own + // at/of fields. + if !inlineRendersEmpty(p.At) { + fmt.Fprintf(b, " · ingested %s", mdInline(p.At)) + } + b.WriteString("._\n\n") +} + // findingAnchor renders a finding's on-screen anchor: the ui selector (in // backticks) and route when present, else the evidence ids. // @@ -233,7 +288,7 @@ func renderFindings(b *strings.Builder, dir string) { func findingAnchor(f analyze.Finding) string { if f.UI != nil { var parts []string - if !codeRendersEmpty(f.UI.Selector) { + if !session.CodeRendersEmpty(f.UI.Selector) { parts = append(parts, mdCode(f.UI.Selector)) } if !inlineRendersEmpty(f.UI.Route) { @@ -336,19 +391,7 @@ func mdCode(s string) string { return "`" + strings.ReplaceAll(session.SafeText(s), "`", "") + "`" } -// codeRendersEmpty reports whether mdCode(s) would carry no meaningful -// content — s reduces to nothing but whitespace once SafeText and backtick -// removal are applied (invisible-only Unicode, backticks alone, or literal -// whitespace, e.g. a lone tab, which SafeText maps to a space). A caller -// deciding whether to show a code span at all, rather than fall back to -// something more informative, must judge presence on this rendered form — -// judging it on s's raw emptiness lets a value that renders as nothing (or -// as invisible whitespace) through as if it were real content. -func codeRendersEmpty(s string) bool { - return strings.TrimSpace(strings.ReplaceAll(session.SafeText(s), "`", "")) == "" -} - -// inlineRendersEmpty is codeRendersEmpty's mdInline sibling: mdInline escapes +// inlineRendersEmpty is session.CodeRendersEmpty's mdInline sibling: mdInline escapes // a backtick rather than stripping it, so a lone backtick is meaningful, // visible content there, unlike inside a code span. func inlineRendersEmpty(s string) bool { @@ -406,7 +449,7 @@ func eventLine(e timeline.Entry) string { // renders to nothing or to whitespace only (invisible-only Unicode, // backticks alone in a code span, or literal whitespace) must be omitted // rather than appended as an empty or blank fragment. - if sel := raw("selector"); !codeRendersEmpty(sel) { + if sel := raw("selector"); !session.CodeRendersEmpty(sel) { parts = append(parts, mdCode(sel)) } if t := raw("text"); !inlineRendersEmpty(t) { diff --git a/internal/report/report_test.go b/internal/report/report_test.go index ec7b513..63d3411 100644 --- a/internal/report/report_test.go +++ b/internal/report/report_test.go @@ -22,6 +22,17 @@ const answerFixture = `{"rubric":"testimony-analysis/v1","findings":[ {"id":"F-002","t":38,"type":"preference","severity":2,"quote":"I like this dark mode toggle","evidence":["utt-006"]} ]}` +// testProvenance is the declaration the report fixtures are ingested with: a +// local backend and a named model, so the golden exercises the fully rendered +// provenance line rather than only its placeholder form. +func testProvenance() analyze.Provenance { + p, err := analyze.NewProvenance(analyze.BackendLocal, "llama3.1:70b", "2026-07-17") + if err != nil { + panic(err) + } + return p +} + func setupSession(t *testing.T) string { t.Helper() dir := t.TempDir() @@ -39,7 +50,7 @@ func setupSession(t *testing.T) string { func TestRoundTrip(t *testing.T) { dir := setupSession(t) - if _, err := analyze.Ingest(dir, strings.NewReader(answerFixture)); err != nil { + if _, err := analyze.Ingest(dir, strings.NewReader(answerFixture), testProvenance()); err != nil { t.Fatalf("Ingest: %v", err) } findingsBefore := findingLines(t, dir) @@ -343,7 +354,7 @@ func findingLines(t *testing.T, dir string) []string { } var out []string for _, l := range strings.Split(strings.TrimRight(string(b), "\n"), "\n") { - if !strings.Contains(l, `"kind":"verdict"`) { + if !strings.Contains(l, `"kind":"verdict"`) && !strings.Contains(l, `"kind":"provenance"`) { out = append(out, l) } } @@ -692,7 +703,7 @@ func TestReportEventLineOmitsSelectorThatRendersEmpty(t *testing.T) { // TestReportFindingAnchorFallsBackOnWhitespaceOnlyUI is the literal-whitespace // sibling of TestReportFindingAnchorFallsBackOnBlankUI: session.SafeText maps // a tab to a space rather than stripping it, so a selector of "\t" is -// non-empty even after SafeText and backtick removal — codeRendersEmpty must +// non-empty even after SafeText and backtick removal — session.CodeRendersEmpty must // judge it on the TRIMMED form to still fall back to the evidence ids, not // render a code span holding only a space. func TestReportFindingAnchorFallsBackOnWhitespaceOnlyUI(t *testing.T) { @@ -1006,3 +1017,112 @@ func TestReportFindingTypeAndQuotePlaceholderOnEmpty(t *testing.T) { t.Fatalf("report is missing the — placeholder for an empty/invisible-only finding type:\n%s", md) } } + +// --- provenance (itd-8 / spc-2609150759135349) --- + +// renderWithFindings writes a findings.jsonl verbatim and renders the report. +func renderWithFindings(t *testing.T, findings string) string { + t.Helper() + dir := setupSession(t) + if err := os.WriteFile(filepath.Join(dir, session.FindingsFile), []byte(findings), 0o644); err != nil { + t.Fatalf("write findings: %v", err) + } + md, err := Render(dir, 2.5) + if err != nil { + t.Fatalf("Render: %v", err) + } + return md +} + +const findingLine = `{"id":"F-001","t":22,"type":"bug","severity":3,"quote":"I clicked save and nothing happened","evidence":["utt-004"],"status":"unverified"}` + +// TestReportRendersProvenanceLine is AC2: the declaration is stated on one line +// under the Findings heading and above the first status group, and its wording +// says it is a declaration rather than something the tool measured. +func TestReportRendersProvenanceLine(t *testing.T) { + md := renderWithFindings(t, + `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b","at":"2026-09-15"}`+"\n"+findingLine+"\n") + + want := "_Provenance (as declared at ingest): local backend · model `llama3.1:70b` · rubric `testimony-analysis/v1` · ingested 2026-09-15._" + if !strings.Contains(md, want) { + t.Fatalf("report does not carry the provenance line %q:\n%s", want, md) + } + heading := strings.Index(md, "## Findings") + prov := strings.Index(md, "_Provenance") + group := strings.Index(md, "### Confirmed") + if heading < 0 || prov < 0 || group < 0 || !(heading < prov && prov < group) { + t.Fatalf("provenance line is not between the Findings heading and the first status group (heading %d, prov %d, group %d)", heading, prov, group) + } +} + +// TestReportProvenanceRendersBackendFromEnum pins the fixed phrase each backend +// renders as. The phrase comes from a switch on the closed enum, never from the +// stored string, so no untrusted byte can reach that position. +func TestReportProvenanceRendersBackendFromEnum(t *testing.T) { + for backend, want := range map[string]string{ + "local": "local backend", + "cloud": "cloud backend", + "unrecorded": "backend not recorded", + } { + md := renderWithFindings(t, + `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"`+backend+`","at":"2026-09-15"}`+"\n"+findingLine+"\n") + if !strings.Contains(md, "_Provenance (as declared at ingest): "+want+" ·") { + t.Fatalf("backend %q did not render as %q:\n%s", backend, want, md) + } + } +} + +// TestReportProvenanceNotRecorded is AC3's report half: a findings.jsonl with no +// provenance record — one written before the record existed, or one whose only +// record carried an uninterpretable backend — says so rather than implying one. +func TestReportProvenanceNotRecorded(t *testing.T) { + for _, tc := range []struct{ name, findings string }{ + {"no provenance line", findingLine + "\n"}, + {"uninterpretable backend", `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"on-prem","at":"2026-09-15"}` + "\n" + findingLine + "\n"}, + } { + t.Run(tc.name, func(t *testing.T) { + md := renderWithFindings(t, tc.findings) + if !strings.Contains(md, "_Provenance: not recorded._") { + t.Fatalf("report does not say the provenance is not recorded:\n%s", md) + } + if !strings.Contains(md, "**F-001**") { + t.Fatalf("the findings stopped rendering:\n%s", md) + } + }) + } +} + +// TestReportProvenanceSanitisesModelAndRubric: both fields are hand-editable +// text reaching the shareable artefact, so each takes the sink defence the rest +// of this file applies. Both render inside a code span (mdCode, as findingAnchor +// renders a selector), which makes inline Markdown literal — so the defence that +// matters there is that a backtick cannot close the span early and let the tail +// render as active markup, and that the control bytes never reach the file. +func TestReportProvenanceSanitisesModelAndRubric(t *testing.T) { + md := renderWithFindings(t, + `{"kind":"provenance","rubric":"r\u001b[31m","backend":"local","model":"x`+"`"+`![beacon](http://h/b.png)","at":"2026-09-15"}`+"\n"+findingLine+"\n") + if strings.Contains(md, "\x1b") { + t.Fatalf("an ANSI escape survived into the provenance line:\n%s", md) + } + // The backtick is stripped, so the span the model sits in cannot be closed by + // its own content and the image form after it stays literal text. + if !strings.Contains(md, "model `x![beacon](http://h/b.png)`") { + t.Fatalf("the model is not rendered inside an unbroken code span:\n%s", md) + } + + // A model and a rubric that render to nothing fall back to their placeholders + // rather than leaving an empty code span on the page. + md = renderWithFindings(t, + `{"kind":"provenance","rubric":"​","backend":"cloud","model":" ","at":"2026-09-15"}`+"\n"+findingLine+"\n") + if !strings.Contains(md, "model not recorded") || !strings.Contains(md, "rubric not recorded") { + t.Fatalf("blank model/rubric did not fall back to their placeholders:\n%s", md) + } + + // An at that renders to nothing drops its clause entirely, the same rule the + // verdict suffix follows. + md = renderWithFindings(t, + `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"cloud","at":"​"}`+"\n"+findingLine+"\n") + if strings.Contains(md, "ingested") { + t.Fatalf("a blank date left a dangling ingested clause:\n%s", md) + } +} diff --git a/internal/review/review.go b/internal/review/review.go index 7358e61..beef723 100644 --- a/internal/review/review.go +++ b/internal/review/review.go @@ -122,7 +122,10 @@ func Run(opts Options) error { } return fmt.Errorf("session directory: %w", err) } - findings, verdicts, err := analyze.Load(opts.Dir) + // The provenance record is discarded here: review judges findings, and the + // declaration of what produced them changes nothing about the walk or the + // verdicts it appends. + _, findings, verdicts, err := analyze.Load(opts.Dir) if err != nil { if errors.Is(err, fs.ErrNotExist) { return fmt.Errorf("no %s (run `testimony analyze -ingest` first)", session.FindingsFile) @@ -405,7 +408,7 @@ func AppendVerdict(dir string, v analyze.Verdict, expect *analyze.Finding) error // now names a different finding, and for a duplicate verdict if the "of" target // has vanished. func verifyTarget(current io.Reader, v analyze.Verdict, expect analyze.Finding) error { - findings, _, err := analyze.ParseRecords(current, session.FindingsFile) + _, findings, _, err := analyze.ParseRecords(current, session.FindingsFile) if err != nil { return err } diff --git a/internal/review/review_test.go b/internal/review/review_test.go index 9c68b8e..45a3fc1 100644 --- a/internal/review/review_test.go +++ b/internal/review/review_test.go @@ -75,7 +75,7 @@ func TestNonInteractiveConfirm(t *testing.T) { if err != nil { t.Fatalf("Run: %v", err) } - findings, verdicts, err := analyze.Load(dir) + _, findings, verdicts, err := analyze.Load(dir) if err != nil { t.Fatalf("Load: %v", err) } @@ -95,7 +95,7 @@ func TestNonInteractiveDuplicate(t *testing.T) { if err := Run(Options{Dir: dir, Finding: "F-002", Verdict: "duplicate-of-F-001", Out: &out, Today: "2026-07-17"}); err != nil { t.Fatalf("Run: %v", err) } - findings, verdicts, _ := analyze.Load(dir) + _, findings, verdicts, _ := analyze.Load(dir) st := analyze.EffectiveStatus(findings, verdicts)["F-002"] if st.Value != "duplicate" || st.Of != "F-001" { t.Fatalf("F-002 status: %+v, want duplicate of F-001", st) @@ -123,7 +123,7 @@ func TestNonInteractiveConfirmMatchesRenderedID(t *testing.T) { t.Fatalf("Run: %v", err) } - findings, verdicts, err := analyze.Load(dir) + _, findings, verdicts, err := analyze.Load(dir) if err != nil { t.Fatalf("Load: %v", err) } @@ -185,7 +185,7 @@ func TestInteractiveGatedWhenNotTTY(t *testing.T) { if !strings.Contains(out.String(), "not a terminal") { t.Fatalf("expected a TTY-gating notice, got %q", out.String()) } - _, verdicts, _ := analyze.Load(dir) + _, _, verdicts, _ := analyze.Load(dir) if len(verdicts) != 0 { t.Fatalf("gated review wrote %d verdicts, want 0", len(verdicts)) } @@ -202,7 +202,7 @@ func TestInteractiveWalk(t *testing.T) { if err := Run(Options{Dir: dir, In: strings.NewReader(script), Out: &out, IsTTY: true, Today: "2026-07-17"}); err != nil { t.Fatalf("Run: %v", err) } - findings, verdicts, _ := analyze.Load(dir) + _, findings, verdicts, _ := analyze.Load(dir) eff := analyze.EffectiveStatus(findings, verdicts) if eff["F-001"].Value != "confirmed" { t.Fatalf("F-001: %+v", eff["F-001"]) @@ -232,7 +232,7 @@ func TestInteractiveDuplicateTargetMustExist(t *testing.T) { if !strings.Contains(out.String(), "duplicate target F-099 not found") { t.Fatalf("expected a not-found notice for F-099, got %q", out.String()) } - _, verdicts, _ := analyze.Load(dir) + _, _, verdicts, _ := analyze.Load(dir) if len(verdicts) != 0 { t.Fatalf("bad duplicate target wrote %d verdicts, want 0", len(verdicts)) } @@ -263,7 +263,7 @@ func TestInteractiveDuplicateRefusesRenderedSelfMatch(t *testing.T) { if !strings.Contains(out.String(), "duplicate of itself") { t.Fatalf("expected a self-duplicate refusal, got %q", out.String()) } - _, verdicts, _ := analyze.Load(dir) + _, _, verdicts, _ := analyze.Load(dir) if len(verdicts) != 0 { t.Fatalf("self-duplicate target wrote %d verdicts, want 0", len(verdicts)) } @@ -275,7 +275,7 @@ func TestInteractiveQuitStops(t *testing.T) { if err := Run(Options{Dir: dir, In: strings.NewReader("q\n"), Out: &out, IsTTY: true, Today: "2026-07-17"}); err != nil { t.Fatalf("Run: %v", err) } - _, verdicts, _ := analyze.Load(dir) + _, _, verdicts, _ := analyze.Load(dir) if len(verdicts) != 0 { t.Fatalf("quit-first wrote %d verdicts, want 0", len(verdicts)) } @@ -533,7 +533,7 @@ func TestAppendVerdictTerminatesAnUnterminatedLastLine(t *testing.T) { } // The verdict is still readable through the normal loader. - _, verdicts, err := analyze.Load(dir) + _, _, verdicts, err := analyze.Load(dir) if err != nil { t.Fatalf("Load: %v", err) } @@ -912,7 +912,7 @@ func TestAppendVerdictAcceptsUnchangedFinding(t *testing.T) { if err := AppendVerdict(dir, rec, &shown); err != nil { t.Fatalf("AppendVerdict on an unchanged finding: %v", err) } - _, verdicts, err := analyze.Load(dir) + _, _, verdicts, err := analyze.Load(dir) if err != nil { t.Fatalf("Load: %v", err) } @@ -1010,3 +1010,64 @@ func TestRunRefusesCrossFamilyFlags(t *testing.T) { }) } } + +// TestReviewIgnoresProvenanceRecord is AC3's review half (itd-8): a +// findings.jsonl carrying the analysis provenance behaves for review exactly as +// one without it. The record is not a finding, so it never enters the walk or +// the queue, and the append-only property extends to it — a verdict lands after +// it and leaves it byte-unchanged. +func TestReviewIgnoresProvenanceRecord(t *testing.T) { + const provLine = `{"kind":"provenance","rubric":"testimony-analysis/v1","backend":"local","model":"llama3.1:70b","at":"2026-09-15"}` + + withProv := t.TempDir() + if err := os.WriteFile(filepath.Join(withProv, session.FindingsFile), []byte(provLine+"\n"+findingsFixture), 0o644); err != nil { + t.Fatalf("write findings: %v", err) + } + withoutProv := writeSession(t) + + // The interactive walk offers the same findings either way. + var a, b bytes.Buffer + for _, tc := range []struct { + dir string + out *bytes.Buffer + }{{withProv, &a}, {withoutProv, &b}} { + if err := Run(Options{Dir: tc.dir, In: strings.NewReader("q\n"), Out: tc.out, IsTTY: true, Today: "2026-09-15"}); err != nil { + t.Fatalf("Run: %v", err) + } + } + if a.String() != b.String() { + t.Fatalf("the walk differs with and without a provenance record:\nwith %q\nwithout %q", a.String(), b.String()) + } + + // A verdict appends at the end and touches nothing above it. + if err := Run(Options{Dir: withProv, Finding: "F-001", Verdict: "confirmed", Out: io.Discard, Today: "2026-09-15"}); err != nil { + t.Fatalf("Run verdict: %v", err) + } + lines := strings.Split(strings.TrimRight(readFindingsFile(t, withProv), "\n"), "\n") + if lines[0] != provLine { + t.Fatalf("the provenance line changed after a verdict:\n%q", lines[0]) + } + if !strings.Contains(lines[len(lines)-1], `"kind":"verdict"`) { + t.Fatalf("the verdict did not land last: %q", lines[len(lines)-1]) + } + // And the record is still readable as itself. + prov, findings, verdicts, err := analyze.Load(withProv) + if err != nil { + t.Fatalf("Load: %v", err) + } + if prov == nil || prov.Backend != analyze.BackendLocal { + t.Fatalf("provenance = %v after review", prov) + } + if len(findings) != 3 || len(verdicts) != 1 { + t.Fatalf("got %d findings and %d verdicts, want 3 and 1", len(findings), len(verdicts)) + } +} + +func readFindingsFile(t *testing.T, dir string) string { + t.Helper() + b, err := os.ReadFile(filepath.Join(dir, session.FindingsFile)) + if err != nil { + t.Fatalf("read: %v", err) + } + return string(b) +} diff --git a/internal/session/session.go b/internal/session/session.go index f69af32..7b62c0d 100644 --- a/internal/session/session.go +++ b/internal/session/session.go @@ -424,6 +424,30 @@ func SafeInline(s string) string { return b.String() } +// CodeRendersEmpty reports whether s would carry no meaningful content once +// rendered inside a Markdown code span: it reduces to nothing but whitespace +// after SafeText and backtick removal (invisible-only Unicode, backticks alone, +// or literal whitespace — a lone tab, which SafeText maps to a space). +// +// Backticks are part of the predicate because they are part of the rendering: a +// code span strips them (they would otherwise close the span early and let the +// tail render as active markup), so a value made only of backticks renders as +// nothing at all. +// +// This is the one home for that judgement, shared by every surface that decides +// whether such a value is present. A caller deciding whether to show a code span +// at all, rather than fall back to something more informative, must judge +// presence on this rendered form — judging it on s's raw emptiness lets a value +// that renders as nothing through as if it were real content. Just as +// importantly, a surface that ACCEPTS such a value (analyze.NewProvenance +// vetting -model) and a surface that RENDERS it (report's provenance line) must +// agree: with two predicates, a model of backticks alone was accepted at the +// flag, echoed on the success line, and then rendered as "not recorded" in the +// report — the tool contradicting itself about what it had just stored. +func CodeRendersEmpty(s string) bool { + return strings.TrimSpace(strings.ReplaceAll(SafeText(s), "`", "")) == "" +} + // SafeTextLines applies SafeText to s one line at a time, preserving the // newlines SafeText itself would strip (they fall under r < 0x20). A // subprocess's captured output — ffmpeg's multi-line metadata dump, a device