Skip to content

[Bug]: untrusted metrics baseline under bare --ci silently disables metrics gates (clone/metrics trust asymmetry) #55

Description

@orenlab

Metadata

Field Value
CodeClone version 2.1.0a1 (reproduced on release/2.1.0a2, commit 932d4b73)
Affected area CLI / baseline / cache

What happened?

Baseline trust enforcement under --ci is asymmetric between the clone lane and the metrics lane. An untrusted/missing clone baseline fails closed (exit 2), but an untrusted/missing metrics baseline under bare --ci silently disables all metrics-delta gates and the run exits 0.

Evidence:

  1. --ci forces fail_on_new = True unconditionally (codeclone/surfaces/cli/startup.py:85-91) → any untrusted clone baseline becomes ExitCode.CONTRACT_ERROR (2).
  2. --ci forces fail_on_new_metrics = True only if the metrics baseline actually loaded: if args.ci and state.loaded: args.fail_on_new_metrics = True (codeclone/surfaces/cli/baseline_state.py:247-248). Missing/untrusted metrics section → the flag stays off → no contract error is raised for that lane.
  3. With metrics_diff = None, summarize_metrics_diff(None) returns None and every diff_* gate field defaults to 0 (codeclone/report/gates/evaluator.py:75-77,166) — all delta gates (health regression, typing/docstring regression, new cycles, new dead code, API breaks) evaluate as "no regression".
  4. Bonus dead parameter: evaluate_gates() accepts baseline_status and discards it (evaluator.py:547: _ = baseline_status), so gate evaluation cannot see baseline trust at all.

Net effect: in CI, tampering with or deleting the embedded metrics section (or a schema mismatch after upgrade) does not fail the pipeline — it just switches the metrics gates off. Confused evidence becomes a confident green.

What did you expect instead?

Symmetric fail-closed gating: under --ci/gating, any required lane that is untrusted/missing exits 2 before gate evaluation — same policy for clones and metrics. An untrusted lane must never silently disable its gates; in normal (non-gating) mode it should be reported as unavailable, not as "no regression".

Minimal reproduction

# 1. Create a trusted baseline:
#    codeclone . --update-baseline
# 2. Corrupt only the metrics section (or its metrics_payload_sha256) in
#    codeclone.baseline.json, leaving clones intact.
# 3. Run: codeclone . --ci
# Observed: warning about untrusted metrics baseline, exit code 0,
#           all metrics gates skipped.
# Expected: exit code 2 (contract error), same as a corrupted clone section.

Additional context

  • Explicit per-gate flags (--fail-on-new-metrics, --fail-on-typing-regression, ...) do pre-empt the hole via _metrics_baseline_gate_requested (baseline_state.py:293-350,398-404) — the asymmetry is specifically the bare --ci path.
  • Fix contract: Phase 39 spec specs/phase-39-baseline-container-lanes.md §10.11 + invariant I-16 (gating symmetry; per-lane trust vector; evaluate_gates must consume trust).
  • Related UI symptom: HTML overview uses metrics-diff presence as the "baselined" proxy for clone novelty display (report/html/sections/_overview.py:900-930).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions