Skip to content

--accept-protected-failures refuses when a costlier regression check comes first in the plan #406

Closed

Description

@DanielNetzer

Problem

amend --accept-protected-failures refuses a valid, by-design protected failure with protected regression observation 1 does not match the approved check whenever the plan lists a costly regression check (for example go test ./...) before the cheaper one that failed.

Seen on #403, Molecule mol-403-5e0af62179c7cb56a75b9b2b, v0.4.5:

  • The regression checks in plan order were go-test-all (go test -timeout 20m ./...), go-vet, gofmt, protocol-and-cache-regression (a targeted go test -run …), the hygiene scripts, then the baselines.
  • The protected view ran go-vet, gofmt and protocol-and-cache-regression, which failed by design on TestAdapterInvocationChangesRequireProtocolBump after a protocol bump. It never ran go-test-all; that check is recorded under notRun as "an earlier required check failed".
  • failingProtectedCheck (lab/internal/molecule/accept_protected_failures.go) compares ran with the plan's regression checks position by position, from the start. regression[0] is go-test-all and ran[0] is go-vet, so it refuses.

The comments in executionOrder (lab/internal/verification/order.go) and in runViewDeltas say the protected view keeps plan order and that --accept-protected-failures reads its run as a plan-order prefix. In practice the protected run skipped the costlier earlier check, so its run isn't a prefix.

The workaround was to reorder the plan's regression checks by cost with amend --parent … --plan … (revision 3), then run checks-only, then accept. That works, but it needs a hand-edited plan.

Expected

A by-design protected failure can be accepted whatever order the plan lists its regression checks in, without weakening the rule that the run must have stopped on that check for its own exit.

Find out which side is wrong:

  • if the protected view is meant to keep plan order, it should run go-test-all first (and then stop there);
  • if cost order is intended in the protected view too, failingProtectedCheck should compare ran with the regression checks in the same execution order (executionOrder), and treat checks recorded as notRun after the stopping check as not part of the prefix.

Fix it on one side, and correct the comments that describe the other.

Acceptance

  • A regression test reproduces the Probe the Codex sandbox before coding and declare environment skips (#43, slice 3) #403 order (a full-suite check listed first, a cheaper targeted check that fails by design) and the protected delta is then accepted.
  • The existing refusals still hold: a run that didn't fail on a regression check, a setup failure, a check that failed before the last one recorded without accepted deltas, or a command that doesn't match the plan.
  • No change to which checks a candidate view runs or how it orders them.

Activity

  1. added
    bugSomething isn't working
    Next improvementCapabilities, composition, evidence and usability after the reviewed-PR MVP.
    on Nov 8, 2026
  2. DanielNetzer commented on Nov 8, 2026

    @DanielNetzer
    OwnerAuthor

    Fixed by #410: a covered check left unrun by an earlier stop now keeps CoveredBy, so --accept-protected-failures leaves it out of the plan-order prefix. The protected view keeps plan order; the plan-reorder workaround is no longer needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Next improvementCapabilities, composition, evidence and usability after the reviewed-PR MVP.bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions