Drift / Evaluation — one reproducible trial suite #33

Activity

1. zenithruneblade

   added

   needs-designAn issue-specific implementation choice remains open; see Open decision.An issue-specific implementation choice remains open; see Open decision.

   area:evolutionEvaluation, candidate creation, assimilation, training and controlled updates.Evaluation, candidate creation, assimilation, training and controlled updates.

   on Sep 18, 2026
2. zenithruneblade

   added the

   Feature

   issue type on Sep 18, 2026
3. zenithruneblade

   marked this as blocked by 2 issues on Sep 18, 2026

   • Artifacts — owned storage and quarantined results #23
   • Execution — reproduce one local agent task through Graph #8
4. zenithruneblade

   marked this as blocking 2 issues on Sep 18, 2026

   • SDLC / Creation — one software request to a verified candidate #40
   • Shadow — compare bounded branches and retain their evidence #41
5. zenithruneblade

   added

   help wantedActive contribution call for the named scope or host evidence; may require deep experience.Active contribution call for the named scope or host evidence; may require deep experience.

   and removed

   help wantedActive contribution call for the named scope or host evidence; may require deep experience.Active contribution call for the named scope or host evidence; may require deep experience.

   on Sep 18, 2026
6. zenithruneblade commented on Nov 6, 2026

   

   zenithruneblade

   on Nov 6, 2026 – with ChatGPT Codex Connector

   ContributorAuthor

   More actions

   Planning reset — 6 November 2026. The original acceptance below remains binding unless a named replacement outcome records its disposition. This is a preserved historical body, not a fresh capability/qualification claim.

   Original issue scope and dated evidence before the planning reset

   Measure one workflow or model candidate against declared expectations before changing live behaviour.

   Starting point: Current delivery: no runnable evaluation harness or maintained suite ships.

   First working slice

   • Define a small versioned Case/TrialSuite/Rubric contract with expected and forbidden behaviour, controls and explicit missing-evidence outcomes.
   • Run one pinned subject and baseline through an admitted execution path; retain input/environment revisions, observations, costs and separate judgments.
   • Verify a known success, a known failure and a blocked/harness-error case; produce a bounded comparison that cannot promote its own subject.
   • Retain raw observations separately from evaluator verdicts and distinguish subject, harness and evaluator failures. Compare matched useful-work and refusal controls; lower resource use achieved by refusing every useful request is not improvement.

   Connections: Uses #8 and #23. Supplies evidence for Memory, Mirror, training and candidate-code review; hostile subjects require an admitted execution boundary.

   Later: Larger calibrated suites, repeated trials and workflow-improvement studies.

   References: ADR 34 · ADR 29.

   Contribution entry: Contribute the small Case/Rubric fixtures or one baseline runner.
   Open decision: Select the first subject, baseline and independently reviewable rubric.
   Verification: Deterministic success/failure/harness-error trials. Shared contribution and touched-docs rules: #52.

   First comparison proposed by review — 14 October 2026

   Review base: 58bbd720. Use the existing read-only public-note answer workflow as the first bounded subject. This is a proposed acceptance workload under the existing issue, not a new coordinator or accepted implementation.

   Before trial execution, freeze baseline/candidate code or prompt revisions, exact note corpus and lifecycle, model/runtime closure, Case/Rubric revisions, evaluator, token/time budgets and repetition policy. Preserve outputs/observations independently from verdicts.

   Starter cases:

   • a relevant promoted note supports an attributable answer;
   • no note and an irrelevant note are matched controls;
   • an unpromoted or revoked note cannot enter new recall;
   • a superseding correction changes the cited eligible source;
   • hostile source text cannot supply instruction/tool authority;
   • missing evidence and a deliberate harness/evaluator failure are reported separately from subject failure.

   Include useful-work controls: refusing every request cannot win through lower token use. A deterministic substituted model first qualifies the harness and known pass/fail/error classifications. A later live-model comparison qualifies only its exact chosen model/profile; it cannot be inferred from that fixture.

   For the first improvement demonstration, retain one deliberately public/synthetic lesson with exact source Case/Run identity and explicit promotion; compare later answers against no-lesson/irrelevant-lesson controls. Existing Archive APIs suffice for this narrow path; private conversation retention and a generic memory framework are unnecessary and remain separate contracts.

   A repository change may initially use ordinary disposable candidate preparation and owner review. Candidate selection cannot promote itself; automatic routing, training, hot reload and body replacement remain outside this result. Better answers demonstrate workflow improvement; claims of improved improvement mechanisms need a subsequent independently evaluated generation.

   Pydantic Evals is a possible small implementation aid (primary docs checked 14 October), not an admitted dependency; first compare it with a tiny repository-native runner.
7. zenithruneblade

   removed

   needs-designAn issue-specific implementation choice remains open; see Open decision.An issue-specific implementation choice remains open; see Open decision.

   on Nov 6, 2026