Skip to content

experiment-tracker: reps carry no evidence pointers (run id / artifact refs), so lineage cannot reach the record #587

Description

@drewstone

Consumer story

discovery-lab keeps a register of scientific results (RESULTS.md) whose every entry must resolve mechanically to runs, oracle grades, and artifacts. The lab built a provenance graph (tools/provenance.mjs) where a register result cites runs and every answer carries source file paths. It wanted to adopt experiment-tracker.ts as the experiment/lineage substrate and could not: the chain from an experiment to the evidence that produced its scores does not exist in the type.

What was read (0.145.3 published package; src/experiment-tracker.ts)

  • ExperimentRep is {rep, score, timestamp, passed?, metrics?} — a bare number with no run id, no result path, no artifact pointer. The only file paths in the module are ExperimentProvenance.changedFiles (git-diff paths).
  • Experiment.parentId gives experiment-to-experiment lineage, but a rep cannot name the run or record it measures, so "which run produced this score" is unanswerable from the tracker alone.
  • Contrast: the package already has the right vocabulary elsewhere — SearchArtifactRef {role, uri, sha256, byteLength} (campaign/search-ledger.ts) and EvidenceRef {kind, uri, excerpt?} (analyst/types.ts). Neither is reachable from ExperimentRep.
  • fileExperimentStore is a whole-file JSON rewrite with no append, chain, or lock, while ledger-core owns exactly those properties.

Required change

ExperimentRep accepts optional evidence: a runId?: string and evidence?: EvidenceRef[] (or SearchArtifactRef[]) that validation, stats, and verdicts preserve unchanged. computeExperimentStats/improvementVerdict ignore them; stores round-trip them.

What this deletes downstream

discovery-lab's register→run join stays in RESULTS.md prose + a local registry file today; with rep-level evidence refs, register results become Experiments whose reps point at runs, and the lab's registerChain query becomes a projection of upstream records. Recorded in discovery docs/15-adoption-decisions.md.

Note on adopted surfaces (working well)

The same lab work adopted supervisor-run (readRuntimeSupervisorRun, analyzeSupervisorRunSources, journal dialect detection) and ledger-core (hashCanonical) directly over real run directories with zero adapters — those fit exactly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions