Found while preparing a controlled graph-vs-ablated experiment in agent-dating, whose run records must carry every condition a third party needs to reproduce a run's inputs.
Problem
GUARDIAN_FEATURES selects which context sections a review gets — full_files, flow, drift, chunked, axes (guardian/collector.py). In the non-chunked path it decides whether full changed files are included and whether a module with no impact graph falls back to its outbound flow graph. That is a large difference in what the model reads.
It is read in scripts/guardian_martian.py:374 as
features=parse_features(os.environ.get("GUARDIAN_FEATURES", "")),and it is:
- not part of
review_fingerprint — guardian/review_fingerprint.py reads no environment, which is right for a digest of code, but it means two reviews with equal fingerprints can have had different prompts;
- not written to the
ReviewRecord — no row in benchmarks/martian-*.jsonl carries it.
Consequence
- Nobody can say from the corpus which sections its reviews contained, so a graph-vs-ablated comparison built on it cannot state what the graph arm actually added.
- Runs are not reproducible from their records: the same fingerprint, models and temperature do not pin down the prompt.
Suggested fix
Record it the way temperature is recorded (#393): write the parsed feature set to the row, sorted, alongside a source (env / default). The fingerprint can stay a digest of code, since features are runtime configuration like temperature, not code. Rows written before the change should read as unknown, never as the empty set — an absent field and "no features" are different claims.
Found while preparing a controlled graph-vs-ablated experiment in agent-dating, whose run records must carry every condition a third party needs to reproduce a run's inputs.
Problem
GUARDIAN_FEATURESselects which context sections a review gets —full_files,flow,drift,chunked,axes(guardian/collector.py). In the non-chunked path it decides whether full changed files are included and whether a module with no impact graph falls back to its outbound flow graph. That is a large difference in what the model reads.It is read in
scripts/guardian_martian.py:374asand it is:
review_fingerprint—guardian/review_fingerprint.pyreads no environment, which is right for a digest of code, but it means two reviews with equal fingerprints can have had different prompts;ReviewRecord— no row inbenchmarks/martian-*.jsonlcarries it.Consequence
Suggested fix
Record it the way
temperatureis recorded (#393): write the parsed feature set to the row, sorted, alongside a source (env/default). The fingerprint can stay a digest of code, since features are runtime configuration like temperature, not code. Rows written before the change should read as unknown, never as the empty set — an absent field and "no features" are different claims.