Skip to content

ci: bind corpus headline deferral to the executing base - #870

Open
luisleo526 wants to merge 2 commits into
mainfrom
ci/corpus-headline-base-binding
Open

luisleo526 wants to merge 2 commits into
mainfrom
ci/corpus-headline-base-binding

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Bind the corpus headline deferral to the executing base.

The trusted-base bridge is kept. A changed headline is deferred only when both the immutable base workflow and the head workflow contain the complete evidence contract. The admission check requires the hosted GITHUB_SHA to equal the pull request's base commit and reads the canonical repository's full history.

Tests cover rejection on an old base and acceptance when both sides carry the new contract, using real Git fixtures.

This pull request is also the first hosted run of the admission check itself: its own CI shows whether the hosted commit and the base commit agree and whether the full-history fetch succeeds, including after a ready-for-review transition.

🤖 Generated with Claude Code

Preserve the trusted-base bridge and defer changed headlines only when both
immutable base and head workflows contain the complete evidence contract.
Cover OLD-base rejection and both-NEW acceptance with real Git fixtures.
from pathlib import Path
import subprocess
import tempfile
import unittest
@luisleo526
luisleo526 marked this pull request as draft November 14, 2026 22:41
@luisleo526
luisleo526 marked this pull request as ready for review November 14, 2026 22:41

@graphify-labs graphify-labs Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 2 advisory finding(s) below merit a look before merge.

Not checked: tests were not run; formal verification proved 0 of 3 changed function(s) (2 not verified).

Formal verification. 1 change(s) alter behavior, breaking input(s) attached. PR-changed functions: 1/3 verified (0 proven, 0 may-equivalent, 1 distinguished) · 2 not verified (2 vacuous).

Behavior changes: gke\_admit\_workflow\_findings changes behavior, here is the input that shows it.

The verifier found a concrete input on which gke\_admit\_workflow\_findings behaves differently before and after the change. If that change is intended, ship it; if not, this is your bug.

Guarantee: This difference was REPRODUCED, the verifier actually ran both versions on that input and saw them disagree. It is real, not an artifact.

Evidence: On input \{"workflow":"' '","name":"'gke\-pr\.yml'"\}, the old code produced \['gke\-pr\.yml: trigger must be exactly pull\_request\_target', 'gke\-pr\.yml: trigger must restrict branches to main', 'gke\-pr\.yml: trigger types are incomplete or reordered', 'gke\-pr\.yml: run\… but the new code produces \['gke\-pr\.yml result must match one complete OLD or NEW headline contract', 'gke\-pr\.yml: trigger must be exactly pull\_request\_target', 'gke\-pr\.yml: trigger must restrict branches to main', 'g… (outputs first differ at character 15). Paste that input straight into a regression test.

Not verified on this run: check\_commands (vacuous: never exercised), main (vacuous: never exercised).


Graphify review — findings

Adds check_corpus_headline_evidence.py, which runs from the trusted PR base and refuses a change to the EXPECTED_VERIFY headline default unless a recorded native run backs it. It hashes the tracked scientific inputs (pin normalized, docs and CI files excluded), then checks the record's source commit, probe population, zero verifier and native exits, native summary digest, headline bytes, and loader environment, treating candidate objects and PR attachments as data only. Extends ci_preflight.py and its tests so the checker's window, the GKE runner and arm-job policy, the promotion guard, and the prebuilt-image and result-classifier steps are enforced in preflight, with the classifier's Compare API lookup and the attachment handling in main being where the new checks could silently under-cover.

Worth a look

  • Compare API results are silently limited to the first page — scripts/ci_preflight.py:635 · Escalate · high
    • unverified hypothesis — survived the adversarial refute pass; needs human review
  • PR attachment is not revalidated after evidence checks — scripts/check_corpus_headline_evidence.py:457 · Escalate · medium
    • unverified hypothesis — survived the adversarial refute pass; needs human review

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 1153 functions depend on the 1056 functions this change touches.

Health — this change adds coupling hotspots:

  • worse: gke_admit_workflow_findings() — 21 callers, 15 callees
  • worse: main_admission_data_findings() — 10 callers, 3 callees
  • worse: main() — 1 callers, 14 callees
  • new: check_service() — 1 callers, 9 callees
  • new: engine_identity() — 1 callers, 7 callees
  • new: main() — 1 callers, 7 callees

Verification — 1153 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

  • verification_scope: 1120 function(s) in the blast radius were not formally verified this run

Test selection

Test selection

8 of 1436 test file(s) selected (1%) via static blast radius.

  • scripts/test_check_corpus_headline_evidence.py — impact, changed-test
  • scripts/test_check_doc_reverts.py — impact
  • scripts/test_ci_corpus_variant.py — impact, test-dependency
  • scripts/test_ci_preflight.py — impact, changed-test
  • scripts/test_ci_preflight_split.py — test-dependency
  • scripts/test_ci_release_gate.py — impact
  • scripts/test_cr150_source_gate.py — impact
  • tests/test_report_sink_format.cpp — impact

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Risky patterns (advisory)

  • medium no-stderr-to-stdout in scripts/check_corpus_headline_evidence.py:464: error printed to stdout instead of stderr: print("corpus headline evidence refused: " + " ".join(str(error).splitlines()))

Docs that may be stale (advisory)

…and 10 more.

Formal verification

Behavior changes: gke\_admit\_workflow\_findings changes behavior, here is the input that shows it.

The verifier found a concrete input on which gke\_admit\_workflow\_findings behaves differently before and after the change. If that change is intended, ship it; if not, this is your bug.

Guarantee: This difference was REPRODUCED, the verifier actually ran both versions on that input and saw them disagree. It is real, not an artifact.

Evidence: On input \{"workflow":"' '","name":"'gke\-pr\.yml'"\}, the old code produced \['gke\-pr\.yml: trigger must be exactly pull\_request\_target', 'gke\-pr\.yml: trigger must restrict branches to main', 'gke\-pr\.yml: trigger types are incomplete or reordered', 'gke\-pr\.yml: run\… but the new code produces \['gke\-pr\.yml result must match one complete OLD or NEW headline contract', 'gke\-pr\.yml: trigger must be exactly pull\_request\_target', 'gke\-pr\.yml: trigger must restrict branches to main', 'g… (outputs first differ at character 15). Paste that input straight into a regression test.

Could not verify: Could not verify check\_commands.

The verifier did not have enough to check check\_commands, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 90 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly NameError — names the real obstacle, not a sampling gap)

Could not verify: Could not verify main.

The verifier did not have enough to check main, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: not verifiable: all 1 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly NameError — names the real obstacle, not a sampling gap)

· 3 grounded finding(s) anchored inline below; 3 more finding(s) on lines outside this diff (see the check run).

or ("/" not in path and path.endswith(".md")))


def engine_identity(repo, revision):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — engine_identity()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

return (json_object(archive.read(names[0])), archive.read(names[1]), archive.read(names[2]))


def check_service(service, attached, repo, base, head, expected):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — check_service()

fans out to 9 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

return total


def main(argv=None):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Health regression — main()

fans out to 7 callees (efferent coupling).

Grounded coupling-delta finding (deterministic), not an LLM guess.

@luisleo526
luisleo526 marked this pull request as draft November 15, 2026 19:49
@luisleo526
luisleo526 marked this pull request as ready for review November 15, 2026 19:49
Use the validated GITHUB_SHA and matching checkout HEAD as the executing
base for pull_request_target admission. Keep the payload base SHA as
informational data so a stale PR payload cannot reject a valid event.
Preserve canonical fetch, full-history, immutable-object and workflow
contract checks, with real Git regressions for stale and equal bases.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant