From d63de0d8469c060661f14ebca39256bd50313e10 Mon Sep 17 00:00:00 2001 From: KeyffMS <124252104+KeyffMS@users.noreply.github.com> Date: Sun, 6 Sep 2026 20:40:38 +0200 Subject: [PATCH 1/8] Stage hash-verified C09 candidate for complete Git object publication --- .c09-publication/part-1.patch | 103 ++++++++++ .c09-publication/part-2.patch | 218 +++++++++++++++++++++ .c09-publication/part-3.patch | 246 ++++++++++++++++++++++++ .c09-publication/part-4.patch | 202 +++++++++++++++++++ .c09-publication/part-5.patch | 244 +++++++++++++++++++++++ .github/workflows/c09-reviewed-tree.yml | 59 ++++++ 6 files changed, 1072 insertions(+) create mode 100644 .c09-publication/part-1.patch create mode 100644 .c09-publication/part-2.patch create mode 100644 .c09-publication/part-3.patch create mode 100644 .c09-publication/part-4.patch create mode 100644 .c09-publication/part-5.patch create mode 100644 .github/workflows/c09-reviewed-tree.yml diff --git a/.c09-publication/part-1.patch b/.c09-publication/part-1.patch new file mode 100644 index 0000000..4612e71 --- /dev/null +++ b/.c09-publication/part-1.patch @@ -0,0 +1,103 @@ +diff --git a/.github/workflows/plananvil-tests.yml b/.github/workflows/plananvil-tests.yml +index 544e1ee..88d75f7 100644 +--- a/.github/workflows/plananvil-tests.yml ++++ b/.github/workflows/plananvil-tests.yml +@@ -82,3 +82,22 @@ jobs: + run: python tools/build_release.py --output "$RUNNER_TEMP/plananvil-dist" + - name: Check patch whitespace + run: git diff --check ++ ++ codex-conformance: ++ name: Codex 0.153.4 loopback conformance (no model service) ++ runs-on: ubuntu-latest ++ timeout-minutes: 8 ++ steps: ++ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 ++ with: ++ persist-credentials: false ++ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 ++ with: ++ python-version: "3.11" ++ - name: Install pinned official CLI into runner temp ++ run: | ++ python tools/install_ci_codex.py "$RUNNER_TEMP/codex-conformance-bin" ++ echo "$RUNNER_TEMP/codex-conformance-bin" >> "$GITHUB_PATH" ++ echo "PLANANVIL_TEST_CODEX_BIN=$RUNNER_TEMP/codex-conformance-bin/codex" >> "$GITHUB_ENV" ++ - name: Verify real CLI lifecycle against deterministic loopback responses ++ run: python -m unittest discover -s tests -p test_qualification_c09_cli.py -v +diff --git a/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md b/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md +new file mode 100644 +index 0000000..4cf95bc +--- /dev/null ++++ b/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md +@@ -0,0 +1,44 @@ ++# C09 finite recovery audit — 2026-09-06 ++ ++## Scope and evidence boundary ++ ++Reviewed baseline: PlanAnvil `44f95258079d6b3ab5b79c178815bbc085edbe02`, live qualification #23 (`34021584365`), Codex CLI `0.153.4`. C10 and C13 passed in that run and their runtimes are not changed here. No live result is manufactured by offline testing. ++ ++#23 observed 20 successful commands and 20 automatic compaction cycles before the 900-second deadline; no PreCompact stop and no completed turn. Its content-free command labels were `other`. They do not establish which commands ran or prove the complete historical causal chain. ++ ++## Proven fixture defects ++ ++The active call path is v7.run_c09 -> compat._codex0152_compaction -> regression._patched_v4 -> v4._c09_runtime. The adapter overwrote C09's nominal threshold with **200 tokens** and replaced the generic prompt with eight repeated large reads. Bootstrap/checkpoint state still advertised **MAP_INSTRUCTIONS**, whereas the turn requested a separate compaction exercise. There was no checked finite phase protocol reconciling the canonical recovery instruction with the requested workload. ++ ++These are inspectable setup conflicts, not proof of a Codex deadlock. Raising the timeout or accepting partial event counts would not fix them. A very small trigger can repeatedly interrupt ordinary recovery tools; repeated payload reads without a stable explicit completion protocol also make progress fragile. ++ ++## Pinned upstream source review ++ ++All source references below are to `openai/codex`, tag `rust-v0.153.4`: ++ ++- `codex-rs/core/src/session/turn.rs`: mid-turn rollover requires `needs_follow_up` and either a new-window request or exhausted token budget. After successful automatic compaction the turn runs pending SessionStart hooks and continues sampling. A final response without follow-up can end the turn rather than forcing another rollover. ++- `codex-rs/core/src/session/context_window.rs`: body-after-prefix usage is active tokens minus the window prefill baseline; the trigger includes an optional TokenBudget fallback buffer. The model hard context cap remains independent. ++- `codex-rs/core/src/compact.rs`: compaction replaces history; mid-turn initial context is injected before the last user message. Actual PreCompact/PostCompact events bracket successful compaction. It does not make an arbitrary sequence of repeated tool calls finite. ++- `codex-rs/core/src/hook_runtime.rs`: model-visible recovery after compaction uses pending SessionStart(source=compact). PostCompact is not an additional-context delivery channel. ++- `codex-rs/core/src/tools/handlers/unified_exec.rs` and `unified_exec/exec_command.rs`: real exec_command accepts `cmd` and `max_output_tokens`; the original command is used for the canonical Bash hook and shell argv is used for execution. ++- `codex-rs/core/tests/suite/compact.rs`: upstream integration tests drive real tool calls/compaction with controlled Responses/SSE data. This supports a separate loopback conformance test, not calling that test live model qualification. ++ ++Official references: https://developers.openai.com/codex/hooks/ and https://developers.openai.com/codex/config-reference/ (including their official redirects). These define supported configuration and lifecycle behavior; they do not promise model compliance with a specific test prompt. ++ ++## Repair ++ ++C09 now has one finite turn with three explicitly ordered read-only commands: **first -> second -> finish**. Each calls the installed product checkpoint/Git validator and fully reads canonical recovery inputs. The first two produce phase-specific inert high-volume stimuli; finish emits only a small receipt and requires the final structured result. No command calls hooks manually, changes canonical state, performs implementation work or reads qualification telemetry. ++ ++The outer fixture prepares its helper before bootstrap and seeds a hash-checked C09_FINITE_RECOVERY next action before the real checkpoint is created. Recovery and the initial request now describe the same task, not MAP_INSTRUCTIONS. Requested output budgets preserve the stimuli (65536 tokens); the disposable trigger is 8192 body-after-prefix tokens, no longer silently overwritten by the 200-token adapter. This threshold/workload separation leaves room for ordinary recovery and ending the turn. It is not a product configuration change or a claim that every model response is bounded below that threshold. ++ ++The evaluator strengthens, rather than relaxes, the gate: three successful exact commands/receipts; ordered automatic PreCompact -> PostCompact -> SessionStart(compact) cycles between the first/second/finish calls; successful product hook exit codes; real reconciliation after the second compaction; valid before/after checkpoint; unchanged source/planning Git **and file bytes including ignored state**; one completed positive structured C09 result. Timeout, missing context, extra/repeated/reordered/failed tools, unexpected tools, incomplete streams, reader or cleanup failure cannot pass. The 900-second live deadline is unchanged. ++ ++Fixture source and exact prompt are included in the manifest-validated evidence template. Only fixed phase labels, counts, booleans and codes are retained; neither private canonical contents nor raw transcripts are uploaded. ++ ++## Verification plan and limitations ++ ++Executable offline tests run actual installer/start/checkpoint/Git/hooks and the three fixture commands; only the model process is substituted in the offline driver. Negative cases cover stale checkpoint, wrong canonical action, byte changes missed by Git status, missing/manual/reordered lifecycle, repeated/failed/extra tools, incomplete completion and receipt validation. ++ ++A separate hosted conformance job downloads the exact official CLI release and verifies asset hashes. It runs the real CLI, real sandbox, tools and product hooks against a loopback Responses simulator, with no credentials or external model requests. The simulator selects responses and usage; passing demonstrates CLI integration, not autonomous live-model behavior. The normal offline suite does not require downloads or Codex. ++ ++Existing cross-platform product/harness suites, packaging, C10/C13 regression tests, evidence validation and release checks remain mandatory. The full model-backed C01-C16 workflow is the final qualification, not an outcome inferred from CI. No sandbox, approvals, auth ownership, product hook output, C13 fallback or release requirement is weakened. +diff --git a/tests/test_live_codex_qualification_harness_v4.py b/tests/test_live_codex_qualification_harness_v4.py +index 235248d..bdfdde4 100644 +--- a/tests/test_live_codex_qualification_harness_v4.py ++++ b/tests/test_live_codex_qualification_harness_v4.py +@@ -36,7 +36,7 @@ class LiveCodexHarnessV4Tests(unittest.TestCase): + self.assertIn("_create_checkpoint(planning=planning, run_root=run_root)", self.source) + + def test_c09_requires_two_real_compactions_and_continuation(self) -> None: +- self.assertIn("C09_COMPACT_LIMIT = 1000", self.source) ++ self.assertIn("C09_COMPACT_LIMIT = c09.COMPACT_LIMIT", self.source) + self.assertIn("len(pre) >= 2 and len(post) >= 2", self.source) + self.assertIn("_continued_after_second_postcompact", self.source) + self.assertIn("tool_use_after_second_postcompact", self.source) +diff --git a/tests/test_live_codex_qualification_regression.py b/tests/test_live_codex_qualification_regression.py +index ac41b14..b56340c 100644 +--- a/tests/test_live_codex_qualification_regression.py ++++ b/tests/test_live_codex_qualification_regression.py +@@ -30,7 +30,7 @@ class QualificationRegressionAuditTests(unittest.TestCase): + + def test_compaction_repairs_use_low_redundant_triggers(self) -> None: + self.assertIn("v4.C08_COMPACT_LIMIT = 40", self.source) +- self.assertIn("v4.C09_COMPACT_LIMIT = 200", self.source) ++ self.assertNotIn("v4.C09_COMPACT_LIMIT = 200", self.source) + self.assertIn("expanded = [name for name in names for _ in range(2)]", self.source) + self.assertIn("return v4._c08_runtime(**kwargs)", self.source) + self.assertIn("return v4._c09_runtime(**kwargs)", self.source) diff --git a/.c09-publication/part-2.patch b/.c09-publication/part-2.patch new file mode 100644 index 0000000..7e5006b --- /dev/null +++ b/.c09-publication/part-2.patch @@ -0,0 +1,218 @@ +diff --git a/tests/test_qualification_c09.py b/tests/test_qualification_c09.py +new file mode 100644 +index 0000000..5c56eff +--- /dev/null ++++ b/tests/test_qualification_c09.py +@@ -0,0 +1,212 @@ ++from __future__ import annotations ++ ++from contextlib import ExitStack ++import json ++import os ++from pathlib import Path ++import re ++import subprocess ++import sys ++import tempfile ++import unittest ++from unittest import mock ++ ++ROOT = Path(__file__).resolve().parents[1] ++sys.path.insert(0, str(ROOT / "tools")) ++import qualification_c09 as c09 ++import qualification_process as observation ++import live_codex_qualification_harness_v7 as v7 ++from test_qualification_execution_boundaries import execute_hook ++ ++v4 = v7.compat.v4 ++base = v4.base ++ ++ ++def completed_payload(): ++ return {"capability_id": "C09", "trial": c09.TRIAL, "outcome": "PASS", ++ "assertions": [], "observations": ["C09_FINISHED"], "blocker": None} ++ ++ ++def valid_events(): ++ return {"event_types": {"turn.completed": 1}, "completed_command_items": 3, ++ "event_tail": [{"event": "item.completed", "item": "command_execution", ++ "command": "c09_" + phase, "exit_code": 0, "c09_receipt_ok": True} ++ for phase in c09.PHASES]} ++ ++ ++def valid_records(): ++ result = [] ++ for phase in c09.PHASES: ++ result.append({"event": "PreToolUse", "c09_phase": phase, "returncode": 0}) ++ if phase != "finish": ++ result += [{"event": "PreCompact", "trigger": "auto", "returncode": 0}, ++ {"event": "PostCompact", "trigger": "auto", "returncode": 0}, ++ {"event": "SessionStart", "source": "compact", "additional_context": True, "returncode": 0}] ++ return result ++ ++ ++class C09ProtocolTests(unittest.TestCase): ++ def test_active_adapter_cannot_restore_the_old_200_token_threshold(self): ++ with tempfile.TemporaryDirectory() as tmp: ++ old_c08 = v4.C08_COMPACT_LIMIT ++ with v7.compat._codex0152_compaction(Path(tmp), "C09"): ++ self.assertEqual(v4.C09_COMPACT_LIMIT, c09.COMPACT_LIMIT) ++ self.assertEqual(v4.C09_COMPACT_LIMIT, 8192) ++ self.assertEqual(v4.C08_COMPACT_LIMIT, old_c08) ++ ++ def test_file_fingerprint_detects_ignored_state_edits_but_not_git_database(self): ++ with tempfile.TemporaryDirectory() as tmp: ++ repo = Path(tmp) ++ path = repo / ".pursue/state.json" ++ path.parent.mkdir() ++ path.write_text('{"revision": 1}') ++ before = c09.file_fingerprint(repo) ++ (repo / ".git").mkdir() ++ (repo / ".git/index").write_text("database") ++ self.assertEqual(c09.file_fingerprint(repo), before) ++ path.write_text('{"revision": 2}') ++ self.assertNotEqual(c09.file_fingerprint(repo), before) ++ ++ def test_protocol_is_finite_and_explicit_about_completion(self): ++ prompt = c09.prompt() ++ for command in c09.COMMANDS.values(): ++ self.assertEqual(prompt.count(command), 1) ++ self.assertIn("C09_FINISHED", prompt) ++ self.assertIn("outcome PASS", prompt) ++ self.assertIn("max_output_tokens=65536", prompt) ++ ++ def test_full_ordered_proof_passes(self): ++ self.assertTrue(all(c09.protocol_checks(valid_events(), valid_records()).values())) ++ ++ def test_counts_alone_never_pass(self): ++ checks = c09.protocol_checks({"completed_command_items": 20}, valid_records()) ++ self.assertFalse(checks["three_completed_reconciliations"]) ++ self.assertFalse(checks["turn_completed"]) ++ ++ def test_reordered_repeated_missing_and_nonzero_tools_are_rejected(self): ++ for mode in ("repeat", "reorder", "missing", "exit", "receipt", "truncated", "forbidden"): ++ with self.subTest(mode=mode): ++ events = valid_events() ++ rows = events["event_tail"] ++ if mode == "repeat": rows.append(rows[0]) ++ elif mode == "reorder": rows.reverse() ++ elif mode == "missing": rows.pop() ++ elif mode == "exit": rows[1]["exit_code"] = 2 ++ elif mode == "receipt": rows[1]["c09_receipt_ok"] = False ++ elif mode == "truncated": events["event_tail_truncated"] = True ++ elif mode == "forbidden": events["item_types"] = {"mcp_tool_call": 1} ++ self.assertFalse(all(c09.protocol_checks(events, valid_records()).values())) ++ ++ def test_manual_compaction_missing_recovery_and_failed_hook_are_rejected(self): ++ for mode in ("manual", "no_context", "failed", "wrong_order"): ++ rows = valid_records() ++ if mode == "manual": rows[1]["trigger"] = "manual" ++ elif mode == "no_context": rows[3]["additional_context"] = False ++ elif mode == "failed": rows[1]["returncode"] = 1 ++ else: rows[1], rows[2] = rows[2], rows[1] ++ self.assertFalse(all(c09.protocol_checks(valid_events(), rows).values())) ++ ++ def test_cli_command_shapes_and_receipts_are_classified_without_content(self): ++ receipt = {"c09_phase": "first", "checkpoint_ok": True, ++ "canonical_read": True, "git_reconciled": True} ++ command = c09.COMMANDS["first"] ++ self.assertEqual(observation.command_label(command), "c09_first") ++ self.assertEqual(observation.command_label("/bin/bash -lc '" + command + "'"), "c09_first") ++ self.assertEqual(observation.command_label(command + "; echo secret"), "other") ++ collector = observation.StructuralEvents() ++ collector.accept(json.dumps({"type": "item.completed", "item": { ++ "type": "command_execution", "command": command, "exit_code": 0, ++ "aggregated_output": "NEVER_PERSIST_THIS\n" + json.dumps(receipt) + "\nC09_NEXT=second\n", ++ }}).encode()) ++ result = collector.summary() ++ self.assertTrue(result["event_tail"][0]["c09_receipt_ok"]) ++ self.assertNotIn("NEVER_PERSIST_THIS", json.dumps(result)) ++ ++ ++class C09FixtureExecutionTests(unittest.TestCase): ++ def test_real_install_checkpoint_three_readonly_commands_and_product_hooks(self): ++ """Offline lifecycle driver; real product operations, NOT live Codex evidence.""" ++ with tempfile.TemporaryDirectory(prefix="c09 fixture ") as tmp: ++ rt = Path(tmp) ++ schemas = base.write_schemas(rt / "schemas") ++ invoked = [] ++ ++ def observed_driver(args, *, cwd, timeout): ++ self.assertEqual(args[args.index("--sandbox") + 1], "read-only") ++ self.assertIn("model_auto_compact_token_limit=8192", args) ++ self.assertNotIn("--ignore-user-config", args) ++ self.assertFalse(any("trust_level" in arg for arg in args)) ++ source = Path(base.git(cwd, "rev-parse", "--path-format=absolute", "--git-common-dir")).parent ++ configured = base.load_json(source / ".codex/hooks.json")["hooks"] ++ collector = observation.StructuralEvents() ++ ++ def hook(name, tag, phase=None): ++ for group in configured.get(name, []): ++ if group.get("matcher") and not re.search(group["matcher"], tag): ++ continue ++ event = {"hook_event_name": name, "cwd": str(cwd), "source": tag, "trigger": tag} ++ if phase: ++ event.update(tool_name="Bash", tool_input={"command": c09.COMMANDS[phase]}) ++ for handler in group["hooks"]: ++ result = execute_hook(handler["command"], cwd, event) ++ self.assertEqual(result.returncode, 0, result.stderr) ++ payload = json.loads(result.stdout) if result.stdout.strip() else {} ++ self.assertIsNot(payload.get("continue"), False) ++ if name == "SessionStart": ++ self.assertIn("C09_FINITE_RECOVERY", payload["hookSpecificOutput"]["additionalContext"]) ++ self.assertNotIn("MAP_INSTRUCTIONS", payload["hookSpecificOutput"]["additionalContext"]) ++ ++ hook("SessionStart", "startup") ++ for phase in c09.PHASES: ++ hook("PreToolUse", "Bash", phase) ++ command = [sys.executable, "-B", c09.SCRIPT, phase] ++ result = subprocess.run(command, cwd=cwd, text=True, capture_output=True, timeout=30) ++ self.assertEqual(result.returncode, 0, result.stderr) ++ invoked.append(phase) ++ self.assertNotIn(str(cwd), result.stdout) ++ if phase == "finish": self.assertLess(len(result.stdout), 2048) ++ else: self.assertGreater(len(result.stdout), c09.COMPACT_LIMIT * 8) ++ collector.accept(json.dumps({"type": "item.completed", "item": { ++ "type": "command_execution", "command": c09.COMMANDS[phase], ++ "exit_code": result.returncode, "aggregated_output": result.stdout, ++ }}).encode()) ++ if phase != "finish": ++ hook("PreCompact", "auto") ++ hook("PostCompact", "auto") ++ hook("SessionStart", "compact") ++ collector.accept(b'{"type":"turn.completed"}') ++ events = collector.summary() ++ events.update(process_cleanup_ok=True, process_returncode=0, timeout=False) ++ base.json_dump(Path(args[args.index("-o") + 1]), completed_payload()) ++ return observation.ProcessResult(0, False, events) ++ ++ # An empty test home contains no credentials. The live path persists ++ # trust and restores config exactly; only the model process is replaced. ++ with mock.patch.dict(os.environ, {"CODEX_HOME": str(rt / "home")}), mock.patch.object( ++ observation, "run_observed", side_effect=observed_driver ++ ), mock.patch.object(v4, "_write_result", side_effect=lambda **kw: (kw["result"], True)) as writer: ++ result, _ = v7.run_c09(root=ROOT, runtime_root=rt, schemas=schemas, version="codex-cli 0.153.4", ++ os_name="offline", source_commit="b" * 40, date="2026-09-06") ++ self.assertEqual(result, "REPRODUCED", writer.call_args.kwargs) ++ self.assertEqual(invoked, list(c09.PHASES)) ++ self.assertTrue(all(writer.call_args.kwargs["trials"][0]["protocol_checks"].values())) ++ self.assertFalse((rt / "home/config.toml").exists()) ++ # Locate the active planning root from the copied, unchanged helper. ++ helpers = list((rt / "C09").rglob("c09_probe.py")) ++ for helper in helpers: ++ candidate = helper.parents[1] ++ states = list((candidate / ".pursue/runs").glob("*/state.json")) ++ if states: ++ state = json.loads(states[0].read_text()) ++ state["next_action"]["type"] = "MAP_INSTRUCTIONS" ++ states[0].write_text(json.dumps(state)) ++ proc = subprocess.run([sys.executable, "-B", str(helper), "finish"], cwd=candidate, ++ capture_output=True, text=True, timeout=30) ++ self.assertNotEqual(proc.returncode, 0) ++ break ++ else: ++ self.fail("Planning fixture was not available for negative checkpoint test") ++ ++ ++if __name__ == "__main__": ++ unittest.main() diff --git a/.c09-publication/part-3.patch b/.c09-publication/part-3.patch new file mode 100644 index 0000000..a02538b --- /dev/null +++ b/.c09-publication/part-3.patch @@ -0,0 +1,246 @@ +diff --git a/tests/test_qualification_c09_cli.py b/tests/test_qualification_c09_cli.py +new file mode 100644 +index 0000000..5d80377 +--- /dev/null ++++ b/tests/test_qualification_c09_cli.py +@@ -0,0 +1,129 @@ ++"""Real pinned Codex CLI against a loopback Responses simulator, no model service. ++ ++Opt-in hosted CI conformance check. It is not live C09 qualification evidence: ++the simulator chooses tool calls and token usage. Codex itself must perform the ++tools, compaction, product hooks, context injection and structured completion. ++""" ++from __future__ import annotations ++ ++from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer ++import json ++import os ++from pathlib import Path ++import subprocess ++import sys ++import tempfile ++import threading ++import unittest ++from unittest import mock ++ ++ROOT = Path(__file__).resolve().parents[1] ++sys.path.insert(0, str(ROOT / "tools")) ++import live_codex_qualification_harness_v7 as v7 ++import qualification_c09 as c09 ++import qualification_process as process ++from test_qualification_c09 import completed_payload ++ ++ ++@unittest.skipUnless(os.environ.get("PLANANVIL_TEST_CODEX_BIN"), "pinned CLI conformance job only") ++class C09RealCLIConformance(unittest.TestCase): ++ def test_actual_cli_finishes_two_compactions_and_recovery_in_one_turn(self): ++ binary = os.environ["PLANANVIL_TEST_CODEX_BIN"] ++ version = subprocess.check_output([binary, "--version"], text=True).strip() ++ self.assertEqual(version, "codex-cli 0.153.4") ++ state = {"regular": 0, "compact": 0, "context_seen": [], "requests": 0} ++ ++ class Server(BaseHTTPRequestHandler): ++ def log_message(self, *args): ++ pass ++ ++ def do_POST(self): ++ data = self.rfile.read(int(self.headers.get("Content-Length", 0))) ++ try: ++ body = json.loads(data) ++ except ValueError: ++ self.send_error(400, "Expected uncompressed fixture JSON") ++ return ++ state["requests"] += 1 ++ if state["requests"] > 10: ++ self.send_error(400, "Finite fixture request limit exceeded") ++ return ++ compact = not body.get("tools") ++ if compact: ++ state["compact"] += 1 ++ item = {"type": "message", "role": "assistant", "id": "summary-" + str(state["compact"]), ++ "content": [{"type": "output_text", "text": "Completed earlier C09 phase. Continue with next phase, do not repeat."}]} ++ output_tokens = 32 ++ else: ++ index = state["regular"] ++ state["regular"] += 1 ++ state["context_seen"].append("C09_FINITE_RECOVERY" in json.dumps(body.get("input"))) ++ if index < 3: ++ phase = c09.PHASES[index] ++ item = {"type": "function_call", "id": "fc-" + phase, "call_id": "call-" + phase, ++ "name": "exec_command", "arguments": json.dumps({"cmd": c09.COMMANDS[phase], ++ "max_output_tokens": c09.OUTPUT_TOKENS if index < 2 else 2048})} ++ output_tokens = 20000 if index < 2 else 8 ++ else: ++ item = {"type": "message", "role": "assistant", "id": "final", ++ "content": [{"type": "output_text", "text": json.dumps(completed_payload())}]} ++ output_tokens = 64 ++ rid = "fixture-" + str(state["requests"]) ++ events = [ ++ {"type": "response.created", "response": {"id": rid}}, ++ {"type": "response.output_item.done", "output_index": 0, "item": item}, ++ {"type": "response.completed", "response": {"id": rid, ++ "usage": {"input_tokens": 1000, "output_tokens": output_tokens, ++ "total_tokens": 1000 + output_tokens}}}, ++ ] ++ response = "".join("data: " + json.dumps(e) + "\n\n" for e in events).encode() ++ self.send_response(200) ++ self.send_header("Content-Type", "text/event-stream") ++ self.send_header("Content-Length", str(len(response))) ++ self.end_headers() ++ self.wfile.write(response) ++ ++ with ThreadingHTTPServer(("127.0.0.1", 0), Server) as server, tempfile.TemporaryDirectory(prefix="c09 cli ") as tmp: ++ worker = threading.Thread(target=server.serve_forever, daemon=True) ++ worker.start() ++ rt = Path(tmp) ++ base = v7.base ++ original_args = base.common_codex_args ++ actual_observed = process.run_observed ++ ++ def bounded_observed(args, **kwargs): ++ # The offline peer is instant. Bound a broken conformance test; ++ # the live C09 timeout remains 900 seconds. ++ kwargs["timeout"] = 120 ++ return actual_observed(args, **kwargs) ++ ++ def local_args(**kwargs): ++ args = original_args(**kwargs) ++ args[0] = binary ++ args += ["-c", 'model_provider="fixture"', "-c", 'model_providers.fixture.name="Fixture"', ++ "-c", f'model_providers.fixture.base_url="http://127.0.0.1:{server.server_port}/v1"', ++ "-c", 'model_providers.fixture.wire_api="responses"', ++ "-c", 'model_providers.fixture.requires_openai_auth=false', ++ "-c", 'model_providers.fixture.stream_max_retries=0', ++ "-c", 'model_providers.fixture.request_max_retries=0'] ++ return args ++ ++ try: ++ with mock.patch.dict(os.environ, {"CODEX_HOME": str(rt / "home")}), mock.patch.object( ++ base, "common_codex_args", side_effect=local_args ++ ), mock.patch.object(process, "run_observed", side_effect=bounded_observed), mock.patch.object(v7.compat.v4, "_write_result", side_effect=lambda **kw: (kw["result"], True)) as writer: ++ result, _ = v7.run_c09(root=ROOT, runtime_root=rt, schemas=base.write_schemas(rt / "schemas"), ++ version=version, os_name="offline-cli", source_commit="b" * 40, date="2026-09-06") ++ details = writer.call_args.kwargs ++ self.assertEqual(result, "REPRODUCED", details) ++ self.assertEqual(state["compact"], 2, state) ++ self.assertEqual(state["regular"], 4, state) ++ self.assertTrue(all(state["context_seen"]), state) ++ print("CODEX_01534_OFFLINE_CONFORMANCE_OK: 3 real tools, 2 compactions, 2 compact recovery contexts, 1 completed turn") ++ finally: ++ server.shutdown() ++ worker.join(timeout=5) ++ ++ ++if __name__ == "__main__": ++ unittest.main() +diff --git a/tests/test_qualification_execution_boundaries.py b/tests/test_qualification_execution_boundaries.py +index ba94760..59094e4 100644 +--- a/tests/test_qualification_execution_boundaries.py ++++ b/tests/test_qualification_execution_boundaries.py +@@ -78,19 +78,18 @@ class SubagentProcessContractTests(unittest.TestCase): + + class CompactionCompletionTests(unittest.TestCase): + def evaluate(self, payload, events, error, records=None): ++ from test_qualification_c09 import valid_records + with tempfile.TemporaryDirectory() as tmp, ExitStack() as stack: + root = Path(tmp) + if records is None: +- records = [ +- {"event": "PreCompact"}, {"event": "PostCompact"}, +- {"event": "PreCompact"}, {"event": "PostCompact"}, +- {"event": "PreToolUse"}, +- ] ++ records = valid_records() + stack.enter_context(mock.patch.object(v4, "_runtime_paths", return_value=(root,) * 7)) + stack.enter_context(mock.patch.object(base, "ensure_git_repo")) + stack.enter_context(mock.patch.object(base, "git", return_value="a" * 40)) + stack.enter_context(mock.patch.object(base, "git_snapshot", return_value={"head": "a" * 40})) + stack.enter_context(mock.patch.object(v4, "_start_active_run", return_value=(root, ".pursue/runs/test"))) ++ stack.enter_context(mock.patch.object(v4.c09, "seed_state")) ++ stack.enter_context(mock.patch.object(v4, "_create_checkpoint")) + stack.enter_context(mock.patch.object(v4, "_checkpoint_validation", return_value={"ok": True})) + stack.enter_context(mock.patch.object(v4, "_clear_hook_log")) + stack.enter_context(mock.patch.object(v4, "_read_hook_records", return_value=records)) +@@ -111,17 +110,20 @@ class CompactionCompletionTests(unittest.TestCase): + self.assertEqual(result["result"], "BLOCKED") + + def test_completed_positive_trial_still_passes(self) -> None: +- result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None) ++ from test_qualification_c09 import completed_payload, valid_events ++ result = self.evaluate(completed_payload(), valid_events(), None) + self.assertEqual(result["result"], "REPRODUCED") + self.assertTrue(result["expected_met"]) + + def test_second_compaction_still_requires_subsequent_tool_use(self) -> None: +- result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None, ++ from test_qualification_c09 import completed_payload, valid_events ++ result = self.evaluate(completed_payload(), valid_events(), None, + [{"event": "PreCompact"}, {"event": "PostCompact"}] * 2) + self.assertNotEqual(result["result"], "REPRODUCED") + + def test_observed_stop_with_valid_checkpoint_still_fails(self) -> None: +- result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None, ++ from test_qualification_c09 import completed_payload, valid_events ++ result = self.evaluate(completed_payload(), valid_events(), None, + [{"event": "PreCompact", "continue": False}, + {"event": "PostCompact"}] * 2 + [{"event": "PreToolUse"}]) + self.assertEqual(result["result"], "FAILED") +diff --git a/tools/install_ci_codex.py b/tools/install_ci_codex.py +new file mode 100644 +index 0000000..d4a1b37 +--- /dev/null ++++ b/tools/install_ci_codex.py +@@ -0,0 +1,53 @@ ++"""Install an official, hash-verified pinned CLI into a disposable hosted CI dir. ++ ++Only the explicit conformance CI job calls this networked helper. Local product ++and harness unit tests do not need Codex, credentials, network, or installation. ++""" ++from __future__ import annotations ++ ++import hashlib ++import io ++import json ++from pathlib import Path ++import sys ++import tarfile ++from urllib.request import Request, urlopen ++ ++TAG = "rust-v0.153.4" ++ASSETS = ("codex-x86_64-unknown-linux-musl.tar.gz", "bwrap-x86_64-unknown-linux-musl.tar.gz") ++ ++ ++def fetch(url): ++ with urlopen(Request(url, headers={"User-Agent": "PlanAnvil-CI-conformance"}), timeout=120) as response: ++ return response.read() ++ ++ ++def main(): ++ target = Path(sys.argv[1]).resolve() ++ target.mkdir(parents=True, exist_ok=True) ++ release = json.loads(fetch("https://api.github.com/repos/openai/codex/releases/tags/" + TAG)) ++ if release["tag_name"] != TAG or release["draft"] or release["prerelease"]: ++ raise ValueError("Unexpected Codex release metadata") ++ for name in ASSETS: ++ asset = next(a for a in release["assets"] if a["name"] == name) ++ url = "https://github.com/openai/codex/releases/download/" + TAG + "/" + name ++ if asset["browser_download_url"] != url: ++ raise ValueError("Unexpected binary origin") ++ data = fetch(url) ++ if asset["digest"] != "sha256:" + hashlib.sha256(data).hexdigest(): ++ raise ValueError("Codex release asset digest mismatch") ++ with tarfile.open(fileobj=io.BytesIO(data)) as archive: ++ members = [m for m in archive.getmembers() if m.isfile()] ++ if len(members) != 1: ++ raise ValueError("Expected exactly one executable") ++ stream = archive.extractfile(members[0]) ++ if stream is None: ++ raise ValueError("Missing release executable") ++ output = target / ("codex" if name.startswith("codex-") else "bwrap") ++ output.write_bytes(stream.read()) ++ output.chmod(0o755) ++ print(name + " " + asset["digest"]) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/.c09-publication/part-4.patch b/.c09-publication/part-4.patch new file mode 100644 index 0000000..6f98b49 --- /dev/null +++ b/.c09-publication/part-4.patch @@ -0,0 +1,202 @@ +diff --git a/tools/live_codex_qualification_harness_v4.py b/tools/live_codex_qualification_harness_v4.py +index 303651c..1fc55d5 100644 +--- a/tools/live_codex_qualification_harness_v4.py ++++ b/tools/live_codex_qualification_harness_v4.py +@@ -9,6 +9,8 @@ import sys + from pathlib import Path + from typing import Any, Callable + ++import qualification_c09 as c09 ++ + import live_codex_qualification_harness as v1 + import live_codex_qualification_harness_v2 as v2 + import live_codex_qualification_harness_v3 as prior +@@ -19,7 +21,7 @@ TARGET_CAPABILITIES = {"C06", "C08", "C09"} + _ORIGINAL_CAPABILITY_RUNTIME = prior.capability_runtime + + C08_COMPACT_LIMIT = 200 +-C09_COMPACT_LIMIT = 1000 ++C09_COMPACT_LIMIT = c09.COMPACT_LIMIT + COMPACT_SCOPE = "body_after_prefix" + HOOK_LOG_RELATIVE = ".pursue/qualification-hook-events.jsonl" + +@@ -833,20 +835,23 @@ def _c09_runtime( + worktrees=worktrees, + version=version, + compact_limit=C09_COMPACT_LIMIT, +- create_checkpoint=True, +- segments=4, +- segment_bytes=32768, ++ create_checkpoint=False, ++ segments=0, ++ segment_bytes=0, ++ prepare_repo=c09.prepare_repo, ++ hook_proxy_source=c09.proxy_source(_hook_proxy_source()), + ) ++ c09.seed_state(planning, _run_root) ++ _create_checkpoint(planning=planning, run_root=_run_root) + fixture_commit = base.git(repo, "rev-parse", "HEAD") + checkpoint_before = _checkpoint_validation(planning) + _clear_hook_log(planning) ++ source_before = base.git_snapshot(repo) ++ files_before = (c09.file_fingerprint(repo), c09.file_fingerprint(planning)) + before = base.git_snapshot(planning) + payload, events, error = _run_codex_probe( + cwd=planning, +- prompt=_compact_probe_prompt( +- capability_id, +- ["segment-01.txt", "segment-02.txt", "segment-03.txt", "segment-04.txt"], +- ), ++ prompt=c09.prompt(), + schemas=schemas, + results_dir=results_dir, + position=1, +@@ -857,12 +862,18 @@ def _c09_runtime( + observe_process=True, + ) + after = base.git_snapshot(planning) ++ source_after = base.git_snapshot(repo) ++ files_after = (c09.file_fingerprint(repo), c09.file_fingerprint(planning)) + records = _read_hook_records(planning) + pre = _event_records(records, "PreCompact") + post = _event_records(records, "PostCompact") + stops = [item for item in pre if item.get("continue") is False] + checkpoint_after = _checkpoint_validation(planning) + ++ checks = c09.protocol_checks(events, records) ++ checks["source_and_planning_unchanged"] = (source_before == source_after and before == after ++ and files_before == files_after) ++ protocol_ok = all(checks.values()) + two_compactions = len(pre) >= 2 and len(post) >= 2 + continued_after_second = _continued_after_second_postcompact(records) + checkpoint_coherent = bool(checkpoint_before.get("ok")) and bool(checkpoint_after.get("ok")) +@@ -872,6 +883,8 @@ def _c09_runtime( + and not events.get("timeout") + and payload.get("capability_id") == capability_id + and payload.get("outcome") == "PASS" ++ and payload.get("trial") == c09.TRIAL ++ and "C09_FINISHED" in payload.get("observations", []) + ) + completion_blocker = error or ( + None if invocation_completed else "C09 did not return a completed positive structured result." +@@ -883,9 +896,10 @@ def _c09_runtime( + "trial_name": "checkpoint_auto_compact_recover_recompact", + "outcome": ( + "BLOCKED" +- if not invocation_completed or not two_compactions ++ if not invocation_completed or not two_compactions or not protocol_ok + else ("PASS" if checkpoint_coherent and no_stop_loop else "FAIL") + ), ++ "protocol_checks": checks, + "assertions": [ + { + "name": "codex_invocation_completed_without_timeout", +@@ -925,10 +939,10 @@ def _c09_runtime( + f"invocation_error={error or 'none'}", + f"invocation_completed={str(invocation_completed).lower()}", + ], +- "blocker": completion_blocker, ++ "blocker": completion_blocker or (None if protocol_ok else "C09 finite protocol checks incomplete"), + "event_summary": events, + "hook_timeline": [ +- {key: item[key] for key in ("event", "returncode", "continue", "additional_context") ++ {key: item[key] for key in ("event", "returncode", "continue", "additional_context", "source", "trigger", "c09_phase") + if key in item} + for item in records[-128:] + if item.get("event") in {"PreToolUse", "SessionStart", "PreCompact", "PostCompact"} +@@ -945,6 +959,9 @@ def _c09_runtime( + "runtime_cli_override": True, + "project_trust_method": "persisted_user_config", + "process_observation": "bounded_structural_jsonl", ++ "finite_phases": list(c09.PHASES), ++ "tool_max_output_tokens": c09.OUTPUT_TOKENS, ++ "canonical_action": "C09_FINITE_RECOVERY", + }, + } + +@@ -953,6 +970,10 @@ def _c09_runtime( + expected_met = False + blocker = "The deterministic C09 fixture did not begin with a valid checkpoint." + summary = "C09 blocked during deterministic fixture preparation." ++ elif not checks["source_and_planning_unchanged"]: ++ result, expected_met = "FAILED", False ++ blocker = "C09 changed source or planning repository state." ++ summary = "C09 failed repository immutability." + elif not invocation_completed: + result = "BLOCKED" + expected_met = False +@@ -978,6 +999,10 @@ def _c09_runtime( + expected_met = False + blocker = "Two compactions completed, but no subsequent real tool call demonstrated continuation after the second compaction." + summary = "C09 blocked because freedom from a permanent stop loop was not fully exercised." ++ elif not protocol_ok: ++ result, expected_met = "BLOCKED", False ++ blocker = "C09 finite protocol was not verified: " + ", ".join(k for k, ok in checks.items() if not ok) ++ summary = "C09 lacks a completed ordered two-cycle recovery proof." + else: + result = "REPRODUCED" + expected_met = True +diff --git a/tools/live_codex_qualification_regression.py b/tools/live_codex_qualification_regression.py +index 36e860e..d9c77e7 100644 +--- a/tools/live_codex_qualification_regression.py ++++ b/tools/live_codex_qualification_regression.py +@@ -170,7 +170,7 @@ def _patched_v4(cap_runtime: Path, capability_id: str) -> Iterator[None]: + v4._run_codex_probe = run_probe + v4._compact_probe_prompt = _compact_prompt + v4.C08_COMPACT_LIMIT = 40 +- v4.C09_COMPACT_LIMIT = 200 ++ # C09 owns its finite workload and threshold; do not override it here. + try: + yield + finally: +diff --git a/tools/prepare_capabilities.py b/tools/prepare_capabilities.py +index 7a5cea2..b851fb0 100644 +--- a/tools/prepare_capabilities.py ++++ b/tools/prepare_capabilities.py +@@ -9,6 +9,8 @@ import shutil + import tarfile + from pathlib import Path, PurePosixPath + ++import qualification_c09 as c09 ++ + PART_GLOB = 'templates.part*' + + C06_CODEX0152_OVERLAY = { +@@ -131,21 +133,28 @@ python3 tools/live_codex_qualification_harness_v7.py \\ + } + + C09_COMPLETION_OVERLAY = { +- 'fixture/README.md': '''# C09 fixture and completion requirements ++ 'fixture/README.md': """# C09 finite recovery fixture + +-The outer harness installs the actual product, creates the planning worktree and a valid checkpoint, and then exercises genuine automatic compaction. ++The outer harness installs the unmodified product, prepares this fixture BEFORE bootstrap, replaces the bootstrap MAP_INSTRUCTIONS action with the hashed C09_FINITE_RECOVERY scenario, then creates and validates a real product checkpoint. + +-Two real compaction cycles, coherent checkpoint/Git state, and subsequent real tool use remain required. They are not sufficient when Codex times out or fails to return a completed positive structured C09 result. Partial event counts cannot turn an incomplete invocation into REPRODUCED. ++One Codex turn executes exactly three read-only commands: first, second, finish. Every command uses the actual installed checkpoint/Git validator and reads canonical recovery inputs. Only first and second emit bounded inert compaction stimuli. Finish emits a small receipt and ends the workload. No manual hook calls, synthetic live events, canonical writes during the turn, permission changes or repeated initial reads are permitted. + +-The deliberately low fixture threshold is not a product default. This correction does not silently retune it or weaken C08's intentional negative stop trial. Record a remaining timeout as BLOCKED. Retain bounded content-free event and hook sequences, command labels/counts, error categories, and process completion/owned-tree cleanup status. Never persist raw stdout/stderr or use partial progress as proof of successful completion. +-''', +- 'run-command.txt': '''# Existing controlled workflow: main -> recovery for C09/C10/C13. +-# The recovery driver selects the same v7 capability runtime used by full. ++The trigger is 8192 body-after-prefix tokens; requested tool output budget is 65536 tokens for each stimulus. These are disposable fixture settings, not product defaults. The active compatibility layer must not replace this trigger with the old 200-token value. TokenBudget is disabled only in the existing isolated qualification configuration. ++ ++Require two ordered automatic PreCompact -> PostCompact -> SessionStart(source=compact) cycles, three successful canonical/Git reconciliations, real tool use after the second cycle, exact command order, valid checkpoints, unchanged source/planning files and Git, and one completed positive C09 turn. Timeout (still 900 seconds), extra/failed/repeated tools, missing recovery, failed hooks, reader/cleanup failures or incomplete evidence cannot be REPRODUCED. No raw output or private canonical state is persisted. ++ ++The fixed scenario resolves proven conflicting fixture instructions and removes an over-aggressive trigger. Historical #23 labels do not identify its exact commands, so the historical timeout's complete causal chain is not claimed as observed. Live success must still be established by the actual model-backed run. ++""", ++ 'fixture/c09_probe.py': c09.PROBE_SOURCE, ++ 'prompt.txt': c09.prompt(), ++ 'run-command.txt': """# Existing controlled workflow: main -> recovery for C09/C10/C13, or full for C01-C16. ++# Both select the identical C09 runtime; only full can satisfy the release gate. + python3 tools/live_codex_qualification_recovery.py --root --source-commit --run-id --output --allow-c13-non-ephemeral-fallback +-# A targeted pass is not a full C01-C16 release pass. +-''', ++# A loopback CLI conformance pass is offline and cannot count as live capability evidence. ++""", + } + ++ + C10_ISOLATION_OVERLAY = { + 'README.md': """# C10 — Recovery context through SessionStart + diff --git a/.c09-publication/part-5.patch b/.c09-publication/part-5.patch new file mode 100644 index 0000000..73166ff --- /dev/null +++ b/.c09-publication/part-5.patch @@ -0,0 +1,244 @@ +diff --git a/tools/qualification_c09.py b/tools/qualification_c09.py +new file mode 100644 +index 0000000..1d94a4e +--- /dev/null ++++ b/tools/qualification_c09.py +@@ -0,0 +1,205 @@ ++"""Finite C09 workload: real read-only recovery, two large outputs, then finish. ++ ++This module prepares the disposable fixture, not product behavior. It never ++invokes lifecycle hooks or supplies fabricated live events to the evaluator. ++""" ++from __future__ import annotations ++ ++import hashlib ++import json ++import os ++from pathlib import Path ++from typing import Any ++ ++COMPACT_LIMIT = 8192 ++OUTPUT_TOKENS = 65536 ++PHASES = ("first", "second", "finish") ++SCRIPT = "qualification-payload/c09_probe.py" ++TRIAL = "checkpoint_auto_compact_recover_recompact" ++COMMANDS = {phase: f"python3 -B {SCRIPT} {phase}" for phase in PHASES} ++ ++# Shipped only in a disposable qualification repository. All runtime operations ++# are reads. Checkpoint/Git decisions come from the installed product validator. ++PROBE_SOURCE = r'''from __future__ import annotations ++import hashlib, json, sys ++from pathlib import Path ++ ++root = Path(__file__).resolve().parents[1] ++sys.path.insert(0, str(root / ".codex" / "hooks")) ++from plan_anvil_hooklib import active_run_for_event ++from plan_anvil_checkpoint import validate_checkpoint_for_run ++ ++def main(): ++ if len(sys.argv) != 2 or sys.argv[1] not in {"first", "second", "finish"}: ++ return 2 ++ phase = sys.argv[1] ++ active = active_run_for_event({"cwd": str(root)}) ++ if active is None: ++ return 2 ++ check = validate_checkpoint_for_run(active) ++ if not check.ok or check.path is None: ++ print(json.dumps({"c09_phase": phase, "checkpoint_ok": False})) ++ return 2 ++ files = [active.run_root / "manifest.json", active.run_root / "state.json", ++ active.run_root / "local-state.json", check.path] ++ files += sorted((root / ".pursue").glob("SYSTEM_PROFILE*.md")) ++ # Fully read the canonical recovery inputs without printing local-state or ++ # paths. The real validator above reconciles canonical identity with Git. ++ sizes = [len(path.read_bytes()) for path in files] ++ state = json.loads((active.run_root / "state.json").read_text(encoding="utf-8")) ++ if state.get("next_action") != {"type": "C09_FINITE_RECOVERY", "target": "evidence/c09-scenario.json"}: ++ return 2 ++ scenario = json.loads((active.run_root / "evidence/c09-scenario.json").read_text(encoding="utf-8")) ++ if scenario.get("phases") != ["first", "second", "finish"]: ++ return 2 ++ receipt = {"c09_phase": phase, "checkpoint_ok": True, "canonical_read": True, ++ "git_reconciled": True, "canonical_file_count": len(files), ++ "canonical_bytes": sum(sizes)} ++ print(json.dumps(receipt, sort_keys=True)) ++ if phase != "finish": ++ # Deterministic inert high-volume output, not an instruction or a secret. ++ # A phase-specific stimulus avoids eight indistinguishable repeated reads. ++ for i in range(1024): ++ print(hashlib.sha512((phase + ":" + str(i)).encode()).hexdigest()) ++ print(json.dumps(receipt, sort_keys=True)) ++ print("C09_NEXT=" + {"first": "second", "second": "finish", "finish": "RETURN_RESULT"}[phase]) ++ return 0 ++ ++if __name__ == "__main__": ++ try: ++ raise SystemExit(main()) ++ except (OSError, ValueError, KeyError, TypeError): ++ # No raw canonical contents/private paths in errors or evidence. ++ print("C09_FIXTURE_READ_FAILED", file=sys.stderr) ++ raise SystemExit(2) ++''' ++ ++ ++def prepare_repo(repo: Path) -> None: ++ path = repo / SCRIPT ++ path.parent.mkdir(parents=True, exist_ok=True) ++ path.write_text(PROBE_SOURCE, encoding="utf-8") ++ ++ ++def seed_state(planning: Path, run_root: str) -> None: ++ """Replace bootstrap's unrelated MAP_INSTRUCTIONS action before checkpointing.""" ++ run = planning / run_root ++ scenario = {"schema_version": "1.0", "phases": list(PHASES), "commands": COMMANDS, ++ "completion": "Return the final C09 result after finish; never restart first."} ++ data = (json.dumps(scenario, indent=2, sort_keys=True) + "\n").encode() ++ path = run / "evidence/c09-scenario.json" ++ path.parent.mkdir(parents=True, exist_ok=True) ++ path.write_bytes(data) ++ state_path = run / "state.json" ++ state = json.loads(state_path.read_text(encoding="utf-8")) ++ state["revision"] += 1 ++ state["next_action"] = {"type": "C09_FINITE_RECOVERY", "target": "evidence/c09-scenario.json"} ++ state.setdefault("artifact_hashes", {})["evidence/c09-scenario.json"] = "sha256:" + hashlib.sha256(data).hexdigest() ++ state_path.write_text(json.dumps(state, indent=2, sort_keys=True) + "\n", encoding="utf-8") ++ ++ ++def prompt() -> str: ++ return f'''Qualify C09 using this finite, read-only scenario in ONE Codex turn. ++The outer harness has already installed PlanAnvil and created a valid checkpoint. ++Do not start a plan, map instructions, or create/repair any files. The canonical ++next action is C09_FINITE_RECOVERY, not the earlier bootstrap MAP_INSTRUCTIONS. ++ ++Make exactly THREE sequential exec_command calls, each once and in this order: ++1. cmd={COMMANDS["first"]!r}, max_output_tokens={OUTPUT_TOKENS} ++2. cmd={COMMANDS["second"]!r}, max_output_tokens={OUTPUT_TOKENS} ++3. cmd={COMMANDS["finish"]!r}, max_output_tokens=2048 ++Use workdir for the current planning directory; do not prepend cd or combine calls. ++Wait for each command to finish before choosing the next one. The first two outputs ++are inert compaction stimuli: do not analyse, quote, hash, or reread their padding. ++ ++After each automatic compaction continue with the NEXT phase, never restart first. ++Each command fully reads manifest/state/local-state/checkpoint/profiles and calls the ++actual product checkpoint/Git validator. This performs the recovery requested by the ++SessionStart context; do not perform additional recovery reads outside those commands. ++No manual hook calls, telemetry reads, other tools, state changes, or subagents. ++ ++After finish exits 0, make NO further calls. Return capability_id C09, trial {TRIAL}, ++outcome PASS, and observation C09_FINISHED. If a command fails, stop with FAIL/BLOCKED; ++do not retry. The outer evaluator, not your self-report, checks the real compactions. ++''' ++ ++ ++def proxy_source(original: str) -> str: ++ """Extend only C09 telemetry without changing stdout or any product decision.""" ++ addition = '''\nif isinstance(event, dict): ++ source = event.get("source") ++ if source in {"startup", "resume", "clear", "compact"}: ++ record["source"] = source ++ trigger = event.get("trigger") ++ if trigger in {"auto", "manual"}: ++ record["trigger"] = trigger ++ raw_command = event.get("tool_input", {}).get("command") if isinstance(event.get("tool_input"), dict) else None ++ commands = ''' + repr({command: phase for phase, command in COMMANDS.items()}) + ''' ++ if event_name == "PreToolUse": ++ record["c09_phase"] = commands.get(raw_command, "other") ++''' ++ anchor = 'try:\n parsed = json.loads(completed.stdout)' ++ # The active compat adapter has this exact executable contract, covered by ++ # subprocess tests. Reject a future adapter change instead of silently losing evidence. ++ if anchor not in original: ++ raise ValueError("C09 telemetry adapter contract changed") ++ return original.replace(anchor, addition + "\n" + anchor, 1) ++ ++ ++def file_fingerprint(repo: Path) -> str: ++ """Hash all fixture files, including ignored canonical state, without paths in evidence. ++ ++ Git status alone misses edits to an already-untracked state file. Exclude only ++ Git's database/locator and the known legacy telemetry sink (normally external). ++ """ ++ if not repo.is_dir(): ++ raise ValueError("C09 fixture directory is missing") ++ digest = hashlib.sha256() ++ def walk_error(_error: OSError) -> None: ++ raise ValueError("C09 fixture files could not be completely inspected") ++ ++ for directory, dirs, names in os.walk(repo, onerror=walk_error): ++ for name in dirs: ++ if name != ".git" and (Path(directory) / name).is_symlink(): ++ raise ValueError("Unexpected directory symlink in C09 fixture") ++ dirs[:] = sorted(d for d in dirs if d != ".git") ++ for name in sorted(names): ++ path = Path(directory) / name ++ rel = path.relative_to(repo).as_posix() ++ if rel in {".git", ".pursue/qualification-hook-events.jsonl"}: ++ continue ++ if path.is_symlink(): ++ raise ValueError("Unexpected symlink in C09 fixture") ++ data = path.read_bytes() ++ digest.update(rel.encode("utf-8") + b"\0") ++ digest.update(hashlib.sha256(data).digest()) ++ return digest.hexdigest() ++ ++ ++def protocol_checks(events: dict[str, Any], records: list[dict[str, Any]]) -> dict[str, bool]: ++ commands = [r for r in events.get("event_tail", []) ++ if r.get("event") == "item.completed" and r.get("item") == "command_execution"] ++ expected = ["c09_" + phase for phase in PHASES] ++ command_order = ([r.get("command") for r in commands] == expected ++ and all(r.get("exit_code") == 0 and r.get("c09_receipt_ok") is True for r in commands) ++ and events.get("completed_command_items") == 3 ++ and not events.get("event_tail_truncated")) ++ tool_rows = [r for r in records if r.get("event") == "PreToolUse"] ++ hook_order = [r.get("c09_phase") for r in tool_rows] == list(PHASES) ++ positions = {r.get("c09_phase"): i for i, r in enumerate(records) if r.get("event") == "PreToolUse"} ++ cycles = hook_order ++ for start, end in (("first", "second"), ("second", "finish")): ++ between = records[positions[start] + 1:positions[end]] if hook_order else [] ++ lifecycle = [r.get("event") for r in between if r.get("event") in {"PreCompact", "PostCompact", "SessionStart"}] ++ cycles = cycles and lifecycle == ["PreCompact", "PostCompact", "SessionStart"] ++ cycles = cycles and all(r.get("trigger") == "auto" for r in between if r.get("event") in {"PreCompact", "PostCompact"}) ++ cycles = cycles and any(r.get("event") == "SessionStart" and r.get("source") == "compact" ++ and r.get("additional_context") is True for r in between) ++ hooks_ok = bool(records) and all(r.get("returncode") == 0 for r in records) ++ forbidden = sum(events.get("item_types", {}).get(k, 0) for k in ++ ("file_change", "mcp_tool_call", "web_search", "collab_tool_call", "other")) ++ return {"three_completed_reconciliations": command_order, ++ "ordered_automatic_recovery_cycles": bool(cycles), "product_hooks_succeeded": hooks_ok, ++ "no_unexpected_tools": not forbidden, ++ "turn_completed": events.get("event_types", {}).get("turn.completed") == 1 ++ and not events.get("event_types", {}).get("turn.failed")} +diff --git a/tools/qualification_process.py b/tools/qualification_process.py +index 3a53fff..4857b2f 100644 +--- a/tools/qualification_process.py ++++ b/tools/qualification_process.py +@@ -29,6 +29,7 @@ ITEM_TYPES = frozenset({ + }) + STATUSES = frozenset({"in_progress", "completed", "failed", "declined", "cancelled"}) + COMMANDS = { ++ **{f"python3 -B qualification-payload/c09_probe.py {p}": f"c09_{p}" for p in ("first", "second", "finish")}, + **{f"cat qualification-payload/segment-{i:02d}.txt": f"segment_{i:02d}" for i in range(1, 5)}, + "git status --porcelain=v1 --untracked-files=all": "git_status", + "git rev-parse HEAD": "git_head", +@@ -118,6 +119,20 @@ class StructuralEvents: + if item_kind == "command_execution": + label = command_label(item.get("command")) + row["command"] = label ++ if label in {"c09_first", "c09_second", "c09_finish"} and kind == "item.completed": ++ # Fixed scalar receipt only; never persist output or paths. ++ raw_output = item.get("aggregated_output", "") ++ receipt = {} ++ if isinstance(raw_output, str): ++ for line in raw_output.splitlines()[-4:]: ++ try: ++ candidate = json.loads(line) ++ except (ValueError, RecursionError): ++ continue ++ if isinstance(candidate, dict): ++ receipt = candidate ++ row["c09_receipt_ok"] = (receipt.get("c09_phase") == label[4:] ++ and all(receipt.get(k) is True for k in ("checkpoint_ok", "canonical_read", "git_reconciled"))) + if kind == "item.completed": + self.completed_commands += 1 + self.commands[label] += 1 diff --git a/.github/workflows/c09-reviewed-tree.yml b/.github/workflows/c09-reviewed-tree.yml new file mode 100644 index 0000000..31f1219 --- /dev/null +++ b/.github/workflows/c09-reviewed-tree.yml @@ -0,0 +1,59 @@ +name: Stage reviewed C09 Git tree +on: + push: + branches: [fix/c09-finite-recovery] + paths: [.github/workflows/c09-reviewed-tree.yml] +permissions: + contents: read +jobs: + stage: + if: github.repository == 'KeyffMS/PlanAnvil' && github.ref == 'refs/heads/fix/c09-finite-recovery' + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: write + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + with: + fetch-depth: 2 + persist-credentials: false + - name: Reconstruct exact reviewed candidate without executing it + shell: bash + run: | + set -euo pipefail + python3 - <<'PY' + import hashlib, json, os, subprocess + from pathlib import Path + def git(*args): + return subprocess.check_output(['git', *args], text=True).strip() + assert git('rev-parse', 'HEAD^') == '44f95258079d6b3ab5b79c178815bbc085edbe02' + patch = b''.join(Path('.c09-publication/part-' + str(i) + '.patch').read_bytes() for i in range(1, 6)) + assert hashlib.sha256(patch).hexdigest() == '2cdb37a41d13bde3d1165a4ab7bcadca93258a049dc45a55653aa3e2d8174180' + temp = Path(os.environ['RUNNER_TEMP']) + patch_path = temp / 'reviewed-c09.patch' + patch_path.write_bytes(patch) + subprocess.run(['git', 'apply', '--check', str(patch_path)], check=True) + subprocess.run(['git', 'apply', '--index', str(patch_path)], check=True) + names = git('diff', '--cached', '--name-only').splitlines() + assert len(names) == 13 + entries = [{'path': n, 'mode': '100644', 'type': 'blob', 'content': Path(n).read_text(encoding='utf-8')} for n in names] + subprocess.run(['git', 'rm', '-r', '.c09-publication', '.github/workflows/c09-reviewed-tree.yml'], check=True) + subprocess.run(['git', 'diff', '--cached', '--check'], check=True) + assert git('write-tree') == '682a9239c32f356c8f9d7510139bf341a5947487' + (temp / 'c09-tree.json').write_text(json.dumps({'base_tree': '678b85380318f57e3458a4870e3daaae1753f208', 'tree': entries}), encoding='utf-8') + print('Reviewed 13-file candidate reconstructed exactly; temporary staging files excluded.') + PY + - name: Store verified Git tree object only; do not update any branch + shell: bash + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + gh api --method POST repos/KeyffMS/PlanAnvil/git/trees --input "$RUNNER_TEMP/c09-tree.json" > "$RUNNER_TEMP/c09-tree-result.json" + python3 - <<'PY' + import json, os + from pathlib import Path + result = json.loads((Path(os.environ['RUNNER_TEMP']) / 'c09-tree-result.json').read_text()) + assert result['sha'] == '682a9239c32f356c8f9d7510139bf341a5947487' + print('VERIFIED_C09_TREE=' + result['sha']) + PY From 30bb42481e30e0e66b9b881f825083e45fa4f3f5 Mon Sep 17 00:00:00 2001 From: KeyffMS <124252104+KeyffMS@users.noreply.github.com> Date: Sun, 6 Sep 2026 20:44:49 +0200 Subject: [PATCH 2/8] Stage content-addressed blobs only; retain final workflow tree changes in authorized connector --- .github/workflows/c09-reviewed-tree.yml | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/.github/workflows/c09-reviewed-tree.yml b/.github/workflows/c09-reviewed-tree.yml index 31f1219..af4fcac 100644 --- a/.github/workflows/c09-reviewed-tree.yml +++ b/.github/workflows/c09-reviewed-tree.yml @@ -15,7 +15,7 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 with: - fetch-depth: 2 + fetch-depth: 3 persist-credentials: false - name: Reconstruct exact reviewed candidate without executing it shell: bash @@ -26,7 +26,7 @@ jobs: from pathlib import Path def git(*args): return subprocess.check_output(['git', *args], text=True).strip() - assert git('rev-parse', 'HEAD^') == '44f95258079d6b3ab5b79c178815bbc085edbe02' + assert git('rev-parse', 'HEAD~2') == '44f95258079d6b3ab5b79c178815bbc085edbe02' patch = b''.join(Path('.c09-publication/part-' + str(i) + '.patch').read_bytes() for i in range(1, 6)) assert hashlib.sha256(patch).hexdigest() == '2cdb37a41d13bde3d1165a4ab7bcadca93258a049dc45a55653aa3e2d8174180' temp = Path(os.environ['RUNNER_TEMP']) @@ -36,24 +36,28 @@ jobs: subprocess.run(['git', 'apply', '--index', str(patch_path)], check=True) names = git('diff', '--cached', '--name-only').splitlines() assert len(names) == 13 - entries = [{'path': n, 'mode': '100644', 'type': 'blob', 'content': Path(n).read_text(encoding='utf-8')} for n in names] + entries = [{'path': n, 'sha': git('hash-object', n), 'content': Path(n).read_text(encoding='utf-8')} for n in names] subprocess.run(['git', 'rm', '-r', '.c09-publication', '.github/workflows/c09-reviewed-tree.yml'], check=True) subprocess.run(['git', 'diff', '--cached', '--check'], check=True) assert git('write-tree') == '682a9239c32f356c8f9d7510139bf341a5947487' - (temp / 'c09-tree.json').write_text(json.dumps({'base_tree': '678b85380318f57e3458a4870e3daaae1753f208', 'tree': entries}), encoding='utf-8') + (temp / 'c09-blobs.json').write_text(json.dumps(entries), encoding='utf-8') print('Reviewed 13-file candidate reconstructed exactly; temporary staging files excluded.') PY - - name: Store verified Git tree object only; do not update any branch + - name: Store verified content objects only; no tree, commit, ref or workflow changes shell: bash env: GH_TOKEN: ${{ github.token }} run: | set -euo pipefail - gh api --method POST repos/KeyffMS/PlanAnvil/git/trees --input "$RUNNER_TEMP/c09-tree.json" > "$RUNNER_TEMP/c09-tree-result.json" python3 - <<'PY' - import json, os + import json, os, subprocess from pathlib import Path - result = json.loads((Path(os.environ['RUNNER_TEMP']) / 'c09-tree-result.json').read_text()) - assert result['sha'] == '682a9239c32f356c8f9d7510139bf341a5947487' - print('VERIFIED_C09_TREE=' + result['sha']) + entries = json.loads((Path(os.environ['RUNNER_TEMP']) / 'c09-blobs.json').read_text()) + for entry in entries: + payload = json.dumps({'content': entry['content'], 'encoding': 'utf-8'}) + result = subprocess.run(['gh', 'api', '--method', 'POST', 'repos/KeyffMS/PlanAnvil/git/blobs', '--input', '-'], input=payload, text=True, capture_output=True, check=True) + actual = json.loads(result.stdout)['sha'] + assert actual == entry['sha'] + print(entry['path'] + ' ' + actual) + print('VERIFIED_C09_BLOBS=13; no branches changed') PY From 9d3a7017da7f257a3455dc955a38cd51e2f26d15 Mon Sep 17 00:00:00 2001 From: KeyffMS <124252104+KeyffMS@users.noreply.github.com> Date: Sun, 6 Sep 2026 20:51:24 +0200 Subject: [PATCH 3/8] Publish verified finite C09 recovery candidate and remove temporary staging workflow --- .c09-publication/part-1.patch | 103 -------- .c09-publication/part-2.patch | 218 ---------------- .c09-publication/part-3.patch | 246 ------------------ .c09-publication/part-4.patch | 202 -------------- .c09-publication/part-5.patch | 244 ----------------- .github/workflows/c09-reviewed-tree.yml | 63 ----- .github/workflows/plananvil-tests.yml | 19 ++ ...EX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md | 44 ++++ ...est_live_codex_qualification_harness_v4.py | 2 +- ...est_live_codex_qualification_regression.py | 2 +- tests/test_qualification_c09.py | 212 +++++++++++++++ tests/test_qualification_c09_cli.py | 129 +++++++++ ...test_qualification_execution_boundaries.py | 18 +- tools/install_ci_codex.py | 53 ++++ tools/live_codex_qualification_harness_v4.py | 47 +++- tools/live_codex_qualification_regression.py | 2 +- tools/prepare_capabilities.py | 27 +- tools/qualification_c09.py | 205 +++++++++++++++ tools/qualification_process.py | 15 ++ 19 files changed, 744 insertions(+), 1107 deletions(-) delete mode 100644 .c09-publication/part-1.patch delete mode 100644 .c09-publication/part-2.patch delete mode 100644 .c09-publication/part-3.patch delete mode 100644 .c09-publication/part-4.patch delete mode 100644 .c09-publication/part-5.patch delete mode 100644 .github/workflows/c09-reviewed-tree.yml create mode 100644 docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md create mode 100644 tests/test_qualification_c09.py create mode 100644 tests/test_qualification_c09_cli.py create mode 100644 tools/install_ci_codex.py create mode 100644 tools/qualification_c09.py diff --git a/.c09-publication/part-1.patch b/.c09-publication/part-1.patch deleted file mode 100644 index 4612e71..0000000 --- a/.c09-publication/part-1.patch +++ /dev/null @@ -1,103 +0,0 @@ -diff --git a/.github/workflows/plananvil-tests.yml b/.github/workflows/plananvil-tests.yml -index 544e1ee..88d75f7 100644 ---- a/.github/workflows/plananvil-tests.yml -+++ b/.github/workflows/plananvil-tests.yml -@@ -82,3 +82,22 @@ jobs: - run: python tools/build_release.py --output "$RUNNER_TEMP/plananvil-dist" - - name: Check patch whitespace - run: git diff --check -+ -+ codex-conformance: -+ name: Codex 0.153.4 loopback conformance (no model service) -+ runs-on: ubuntu-latest -+ timeout-minutes: 8 -+ steps: -+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 -+ with: -+ persist-credentials: false -+ - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 -+ with: -+ python-version: "3.11" -+ - name: Install pinned official CLI into runner temp -+ run: | -+ python tools/install_ci_codex.py "$RUNNER_TEMP/codex-conformance-bin" -+ echo "$RUNNER_TEMP/codex-conformance-bin" >> "$GITHUB_PATH" -+ echo "PLANANVIL_TEST_CODEX_BIN=$RUNNER_TEMP/codex-conformance-bin/codex" >> "$GITHUB_ENV" -+ - name: Verify real CLI lifecycle against deterministic loopback responses -+ run: python -m unittest discover -s tests -p test_qualification_c09_cli.py -v -diff --git a/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md b/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md -new file mode 100644 -index 0000000..4cf95bc ---- /dev/null -+++ b/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md -@@ -0,0 +1,44 @@ -+# C09 finite recovery audit — 2026-09-06 -+ -+## Scope and evidence boundary -+ -+Reviewed baseline: PlanAnvil `44f95258079d6b3ab5b79c178815bbc085edbe02`, live qualification #23 (`34021584365`), Codex CLI `0.153.4`. C10 and C13 passed in that run and their runtimes are not changed here. No live result is manufactured by offline testing. -+ -+#23 observed 20 successful commands and 20 automatic compaction cycles before the 900-second deadline; no PreCompact stop and no completed turn. Its content-free command labels were `other`. They do not establish which commands ran or prove the complete historical causal chain. -+ -+## Proven fixture defects -+ -+The active call path is v7.run_c09 -> compat._codex0152_compaction -> regression._patched_v4 -> v4._c09_runtime. The adapter overwrote C09's nominal threshold with **200 tokens** and replaced the generic prompt with eight repeated large reads. Bootstrap/checkpoint state still advertised **MAP_INSTRUCTIONS**, whereas the turn requested a separate compaction exercise. There was no checked finite phase protocol reconciling the canonical recovery instruction with the requested workload. -+ -+These are inspectable setup conflicts, not proof of a Codex deadlock. Raising the timeout or accepting partial event counts would not fix them. A very small trigger can repeatedly interrupt ordinary recovery tools; repeated payload reads without a stable explicit completion protocol also make progress fragile. -+ -+## Pinned upstream source review -+ -+All source references below are to `openai/codex`, tag `rust-v0.153.4`: -+ -+- `codex-rs/core/src/session/turn.rs`: mid-turn rollover requires `needs_follow_up` and either a new-window request or exhausted token budget. After successful automatic compaction the turn runs pending SessionStart hooks and continues sampling. A final response without follow-up can end the turn rather than forcing another rollover. -+- `codex-rs/core/src/session/context_window.rs`: body-after-prefix usage is active tokens minus the window prefill baseline; the trigger includes an optional TokenBudget fallback buffer. The model hard context cap remains independent. -+- `codex-rs/core/src/compact.rs`: compaction replaces history; mid-turn initial context is injected before the last user message. Actual PreCompact/PostCompact events bracket successful compaction. It does not make an arbitrary sequence of repeated tool calls finite. -+- `codex-rs/core/src/hook_runtime.rs`: model-visible recovery after compaction uses pending SessionStart(source=compact). PostCompact is not an additional-context delivery channel. -+- `codex-rs/core/src/tools/handlers/unified_exec.rs` and `unified_exec/exec_command.rs`: real exec_command accepts `cmd` and `max_output_tokens`; the original command is used for the canonical Bash hook and shell argv is used for execution. -+- `codex-rs/core/tests/suite/compact.rs`: upstream integration tests drive real tool calls/compaction with controlled Responses/SSE data. This supports a separate loopback conformance test, not calling that test live model qualification. -+ -+Official references: https://developers.openai.com/codex/hooks/ and https://developers.openai.com/codex/config-reference/ (including their official redirects). These define supported configuration and lifecycle behavior; they do not promise model compliance with a specific test prompt. -+ -+## Repair -+ -+C09 now has one finite turn with three explicitly ordered read-only commands: **first -> second -> finish**. Each calls the installed product checkpoint/Git validator and fully reads canonical recovery inputs. The first two produce phase-specific inert high-volume stimuli; finish emits only a small receipt and requires the final structured result. No command calls hooks manually, changes canonical state, performs implementation work or reads qualification telemetry. -+ -+The outer fixture prepares its helper before bootstrap and seeds a hash-checked C09_FINITE_RECOVERY next action before the real checkpoint is created. Recovery and the initial request now describe the same task, not MAP_INSTRUCTIONS. Requested output budgets preserve the stimuli (65536 tokens); the disposable trigger is 8192 body-after-prefix tokens, no longer silently overwritten by the 200-token adapter. This threshold/workload separation leaves room for ordinary recovery and ending the turn. It is not a product configuration change or a claim that every model response is bounded below that threshold. -+ -+The evaluator strengthens, rather than relaxes, the gate: three successful exact commands/receipts; ordered automatic PreCompact -> PostCompact -> SessionStart(compact) cycles between the first/second/finish calls; successful product hook exit codes; real reconciliation after the second compaction; valid before/after checkpoint; unchanged source/planning Git **and file bytes including ignored state**; one completed positive structured C09 result. Timeout, missing context, extra/repeated/reordered/failed tools, unexpected tools, incomplete streams, reader or cleanup failure cannot pass. The 900-second live deadline is unchanged. -+ -+Fixture source and exact prompt are included in the manifest-validated evidence template. Only fixed phase labels, counts, booleans and codes are retained; neither private canonical contents nor raw transcripts are uploaded. -+ -+## Verification plan and limitations -+ -+Executable offline tests run actual installer/start/checkpoint/Git/hooks and the three fixture commands; only the model process is substituted in the offline driver. Negative cases cover stale checkpoint, wrong canonical action, byte changes missed by Git status, missing/manual/reordered lifecycle, repeated/failed/extra tools, incomplete completion and receipt validation. -+ -+A separate hosted conformance job downloads the exact official CLI release and verifies asset hashes. It runs the real CLI, real sandbox, tools and product hooks against a loopback Responses simulator, with no credentials or external model requests. The simulator selects responses and usage; passing demonstrates CLI integration, not autonomous live-model behavior. The normal offline suite does not require downloads or Codex. -+ -+Existing cross-platform product/harness suites, packaging, C10/C13 regression tests, evidence validation and release checks remain mandatory. The full model-backed C01-C16 workflow is the final qualification, not an outcome inferred from CI. No sandbox, approvals, auth ownership, product hook output, C13 fallback or release requirement is weakened. -diff --git a/tests/test_live_codex_qualification_harness_v4.py b/tests/test_live_codex_qualification_harness_v4.py -index 235248d..bdfdde4 100644 ---- a/tests/test_live_codex_qualification_harness_v4.py -+++ b/tests/test_live_codex_qualification_harness_v4.py -@@ -36,7 +36,7 @@ class LiveCodexHarnessV4Tests(unittest.TestCase): - self.assertIn("_create_checkpoint(planning=planning, run_root=run_root)", self.source) - - def test_c09_requires_two_real_compactions_and_continuation(self) -> None: -- self.assertIn("C09_COMPACT_LIMIT = 1000", self.source) -+ self.assertIn("C09_COMPACT_LIMIT = c09.COMPACT_LIMIT", self.source) - self.assertIn("len(pre) >= 2 and len(post) >= 2", self.source) - self.assertIn("_continued_after_second_postcompact", self.source) - self.assertIn("tool_use_after_second_postcompact", self.source) -diff --git a/tests/test_live_codex_qualification_regression.py b/tests/test_live_codex_qualification_regression.py -index ac41b14..b56340c 100644 ---- a/tests/test_live_codex_qualification_regression.py -+++ b/tests/test_live_codex_qualification_regression.py -@@ -30,7 +30,7 @@ class QualificationRegressionAuditTests(unittest.TestCase): - - def test_compaction_repairs_use_low_redundant_triggers(self) -> None: - self.assertIn("v4.C08_COMPACT_LIMIT = 40", self.source) -- self.assertIn("v4.C09_COMPACT_LIMIT = 200", self.source) -+ self.assertNotIn("v4.C09_COMPACT_LIMIT = 200", self.source) - self.assertIn("expanded = [name for name in names for _ in range(2)]", self.source) - self.assertIn("return v4._c08_runtime(**kwargs)", self.source) - self.assertIn("return v4._c09_runtime(**kwargs)", self.source) diff --git a/.c09-publication/part-2.patch b/.c09-publication/part-2.patch deleted file mode 100644 index 7e5006b..0000000 --- a/.c09-publication/part-2.patch +++ /dev/null @@ -1,218 +0,0 @@ -diff --git a/tests/test_qualification_c09.py b/tests/test_qualification_c09.py -new file mode 100644 -index 0000000..5c56eff ---- /dev/null -+++ b/tests/test_qualification_c09.py -@@ -0,0 +1,212 @@ -+from __future__ import annotations -+ -+from contextlib import ExitStack -+import json -+import os -+from pathlib import Path -+import re -+import subprocess -+import sys -+import tempfile -+import unittest -+from unittest import mock -+ -+ROOT = Path(__file__).resolve().parents[1] -+sys.path.insert(0, str(ROOT / "tools")) -+import qualification_c09 as c09 -+import qualification_process as observation -+import live_codex_qualification_harness_v7 as v7 -+from test_qualification_execution_boundaries import execute_hook -+ -+v4 = v7.compat.v4 -+base = v4.base -+ -+ -+def completed_payload(): -+ return {"capability_id": "C09", "trial": c09.TRIAL, "outcome": "PASS", -+ "assertions": [], "observations": ["C09_FINISHED"], "blocker": None} -+ -+ -+def valid_events(): -+ return {"event_types": {"turn.completed": 1}, "completed_command_items": 3, -+ "event_tail": [{"event": "item.completed", "item": "command_execution", -+ "command": "c09_" + phase, "exit_code": 0, "c09_receipt_ok": True} -+ for phase in c09.PHASES]} -+ -+ -+def valid_records(): -+ result = [] -+ for phase in c09.PHASES: -+ result.append({"event": "PreToolUse", "c09_phase": phase, "returncode": 0}) -+ if phase != "finish": -+ result += [{"event": "PreCompact", "trigger": "auto", "returncode": 0}, -+ {"event": "PostCompact", "trigger": "auto", "returncode": 0}, -+ {"event": "SessionStart", "source": "compact", "additional_context": True, "returncode": 0}] -+ return result -+ -+ -+class C09ProtocolTests(unittest.TestCase): -+ def test_active_adapter_cannot_restore_the_old_200_token_threshold(self): -+ with tempfile.TemporaryDirectory() as tmp: -+ old_c08 = v4.C08_COMPACT_LIMIT -+ with v7.compat._codex0152_compaction(Path(tmp), "C09"): -+ self.assertEqual(v4.C09_COMPACT_LIMIT, c09.COMPACT_LIMIT) -+ self.assertEqual(v4.C09_COMPACT_LIMIT, 8192) -+ self.assertEqual(v4.C08_COMPACT_LIMIT, old_c08) -+ -+ def test_file_fingerprint_detects_ignored_state_edits_but_not_git_database(self): -+ with tempfile.TemporaryDirectory() as tmp: -+ repo = Path(tmp) -+ path = repo / ".pursue/state.json" -+ path.parent.mkdir() -+ path.write_text('{"revision": 1}') -+ before = c09.file_fingerprint(repo) -+ (repo / ".git").mkdir() -+ (repo / ".git/index").write_text("database") -+ self.assertEqual(c09.file_fingerprint(repo), before) -+ path.write_text('{"revision": 2}') -+ self.assertNotEqual(c09.file_fingerprint(repo), before) -+ -+ def test_protocol_is_finite_and_explicit_about_completion(self): -+ prompt = c09.prompt() -+ for command in c09.COMMANDS.values(): -+ self.assertEqual(prompt.count(command), 1) -+ self.assertIn("C09_FINISHED", prompt) -+ self.assertIn("outcome PASS", prompt) -+ self.assertIn("max_output_tokens=65536", prompt) -+ -+ def test_full_ordered_proof_passes(self): -+ self.assertTrue(all(c09.protocol_checks(valid_events(), valid_records()).values())) -+ -+ def test_counts_alone_never_pass(self): -+ checks = c09.protocol_checks({"completed_command_items": 20}, valid_records()) -+ self.assertFalse(checks["three_completed_reconciliations"]) -+ self.assertFalse(checks["turn_completed"]) -+ -+ def test_reordered_repeated_missing_and_nonzero_tools_are_rejected(self): -+ for mode in ("repeat", "reorder", "missing", "exit", "receipt", "truncated", "forbidden"): -+ with self.subTest(mode=mode): -+ events = valid_events() -+ rows = events["event_tail"] -+ if mode == "repeat": rows.append(rows[0]) -+ elif mode == "reorder": rows.reverse() -+ elif mode == "missing": rows.pop() -+ elif mode == "exit": rows[1]["exit_code"] = 2 -+ elif mode == "receipt": rows[1]["c09_receipt_ok"] = False -+ elif mode == "truncated": events["event_tail_truncated"] = True -+ elif mode == "forbidden": events["item_types"] = {"mcp_tool_call": 1} -+ self.assertFalse(all(c09.protocol_checks(events, valid_records()).values())) -+ -+ def test_manual_compaction_missing_recovery_and_failed_hook_are_rejected(self): -+ for mode in ("manual", "no_context", "failed", "wrong_order"): -+ rows = valid_records() -+ if mode == "manual": rows[1]["trigger"] = "manual" -+ elif mode == "no_context": rows[3]["additional_context"] = False -+ elif mode == "failed": rows[1]["returncode"] = 1 -+ else: rows[1], rows[2] = rows[2], rows[1] -+ self.assertFalse(all(c09.protocol_checks(valid_events(), rows).values())) -+ -+ def test_cli_command_shapes_and_receipts_are_classified_without_content(self): -+ receipt = {"c09_phase": "first", "checkpoint_ok": True, -+ "canonical_read": True, "git_reconciled": True} -+ command = c09.COMMANDS["first"] -+ self.assertEqual(observation.command_label(command), "c09_first") -+ self.assertEqual(observation.command_label("/bin/bash -lc '" + command + "'"), "c09_first") -+ self.assertEqual(observation.command_label(command + "; echo secret"), "other") -+ collector = observation.StructuralEvents() -+ collector.accept(json.dumps({"type": "item.completed", "item": { -+ "type": "command_execution", "command": command, "exit_code": 0, -+ "aggregated_output": "NEVER_PERSIST_THIS\n" + json.dumps(receipt) + "\nC09_NEXT=second\n", -+ }}).encode()) -+ result = collector.summary() -+ self.assertTrue(result["event_tail"][0]["c09_receipt_ok"]) -+ self.assertNotIn("NEVER_PERSIST_THIS", json.dumps(result)) -+ -+ -+class C09FixtureExecutionTests(unittest.TestCase): -+ def test_real_install_checkpoint_three_readonly_commands_and_product_hooks(self): -+ """Offline lifecycle driver; real product operations, NOT live Codex evidence.""" -+ with tempfile.TemporaryDirectory(prefix="c09 fixture ") as tmp: -+ rt = Path(tmp) -+ schemas = base.write_schemas(rt / "schemas") -+ invoked = [] -+ -+ def observed_driver(args, *, cwd, timeout): -+ self.assertEqual(args[args.index("--sandbox") + 1], "read-only") -+ self.assertIn("model_auto_compact_token_limit=8192", args) -+ self.assertNotIn("--ignore-user-config", args) -+ self.assertFalse(any("trust_level" in arg for arg in args)) -+ source = Path(base.git(cwd, "rev-parse", "--path-format=absolute", "--git-common-dir")).parent -+ configured = base.load_json(source / ".codex/hooks.json")["hooks"] -+ collector = observation.StructuralEvents() -+ -+ def hook(name, tag, phase=None): -+ for group in configured.get(name, []): -+ if group.get("matcher") and not re.search(group["matcher"], tag): -+ continue -+ event = {"hook_event_name": name, "cwd": str(cwd), "source": tag, "trigger": tag} -+ if phase: -+ event.update(tool_name="Bash", tool_input={"command": c09.COMMANDS[phase]}) -+ for handler in group["hooks"]: -+ result = execute_hook(handler["command"], cwd, event) -+ self.assertEqual(result.returncode, 0, result.stderr) -+ payload = json.loads(result.stdout) if result.stdout.strip() else {} -+ self.assertIsNot(payload.get("continue"), False) -+ if name == "SessionStart": -+ self.assertIn("C09_FINITE_RECOVERY", payload["hookSpecificOutput"]["additionalContext"]) -+ self.assertNotIn("MAP_INSTRUCTIONS", payload["hookSpecificOutput"]["additionalContext"]) -+ -+ hook("SessionStart", "startup") -+ for phase in c09.PHASES: -+ hook("PreToolUse", "Bash", phase) -+ command = [sys.executable, "-B", c09.SCRIPT, phase] -+ result = subprocess.run(command, cwd=cwd, text=True, capture_output=True, timeout=30) -+ self.assertEqual(result.returncode, 0, result.stderr) -+ invoked.append(phase) -+ self.assertNotIn(str(cwd), result.stdout) -+ if phase == "finish": self.assertLess(len(result.stdout), 2048) -+ else: self.assertGreater(len(result.stdout), c09.COMPACT_LIMIT * 8) -+ collector.accept(json.dumps({"type": "item.completed", "item": { -+ "type": "command_execution", "command": c09.COMMANDS[phase], -+ "exit_code": result.returncode, "aggregated_output": result.stdout, -+ }}).encode()) -+ if phase != "finish": -+ hook("PreCompact", "auto") -+ hook("PostCompact", "auto") -+ hook("SessionStart", "compact") -+ collector.accept(b'{"type":"turn.completed"}') -+ events = collector.summary() -+ events.update(process_cleanup_ok=True, process_returncode=0, timeout=False) -+ base.json_dump(Path(args[args.index("-o") + 1]), completed_payload()) -+ return observation.ProcessResult(0, False, events) -+ -+ # An empty test home contains no credentials. The live path persists -+ # trust and restores config exactly; only the model process is replaced. -+ with mock.patch.dict(os.environ, {"CODEX_HOME": str(rt / "home")}), mock.patch.object( -+ observation, "run_observed", side_effect=observed_driver -+ ), mock.patch.object(v4, "_write_result", side_effect=lambda **kw: (kw["result"], True)) as writer: -+ result, _ = v7.run_c09(root=ROOT, runtime_root=rt, schemas=schemas, version="codex-cli 0.153.4", -+ os_name="offline", source_commit="b" * 40, date="2026-09-06") -+ self.assertEqual(result, "REPRODUCED", writer.call_args.kwargs) -+ self.assertEqual(invoked, list(c09.PHASES)) -+ self.assertTrue(all(writer.call_args.kwargs["trials"][0]["protocol_checks"].values())) -+ self.assertFalse((rt / "home/config.toml").exists()) -+ # Locate the active planning root from the copied, unchanged helper. -+ helpers = list((rt / "C09").rglob("c09_probe.py")) -+ for helper in helpers: -+ candidate = helper.parents[1] -+ states = list((candidate / ".pursue/runs").glob("*/state.json")) -+ if states: -+ state = json.loads(states[0].read_text()) -+ state["next_action"]["type"] = "MAP_INSTRUCTIONS" -+ states[0].write_text(json.dumps(state)) -+ proc = subprocess.run([sys.executable, "-B", str(helper), "finish"], cwd=candidate, -+ capture_output=True, text=True, timeout=30) -+ self.assertNotEqual(proc.returncode, 0) -+ break -+ else: -+ self.fail("Planning fixture was not available for negative checkpoint test") -+ -+ -+if __name__ == "__main__": -+ unittest.main() diff --git a/.c09-publication/part-3.patch b/.c09-publication/part-3.patch deleted file mode 100644 index a02538b..0000000 --- a/.c09-publication/part-3.patch +++ /dev/null @@ -1,246 +0,0 @@ -diff --git a/tests/test_qualification_c09_cli.py b/tests/test_qualification_c09_cli.py -new file mode 100644 -index 0000000..5d80377 ---- /dev/null -+++ b/tests/test_qualification_c09_cli.py -@@ -0,0 +1,129 @@ -+"""Real pinned Codex CLI against a loopback Responses simulator, no model service. -+ -+Opt-in hosted CI conformance check. It is not live C09 qualification evidence: -+the simulator chooses tool calls and token usage. Codex itself must perform the -+tools, compaction, product hooks, context injection and structured completion. -+""" -+from __future__ import annotations -+ -+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -+import json -+import os -+from pathlib import Path -+import subprocess -+import sys -+import tempfile -+import threading -+import unittest -+from unittest import mock -+ -+ROOT = Path(__file__).resolve().parents[1] -+sys.path.insert(0, str(ROOT / "tools")) -+import live_codex_qualification_harness_v7 as v7 -+import qualification_c09 as c09 -+import qualification_process as process -+from test_qualification_c09 import completed_payload -+ -+ -+@unittest.skipUnless(os.environ.get("PLANANVIL_TEST_CODEX_BIN"), "pinned CLI conformance job only") -+class C09RealCLIConformance(unittest.TestCase): -+ def test_actual_cli_finishes_two_compactions_and_recovery_in_one_turn(self): -+ binary = os.environ["PLANANVIL_TEST_CODEX_BIN"] -+ version = subprocess.check_output([binary, "--version"], text=True).strip() -+ self.assertEqual(version, "codex-cli 0.153.4") -+ state = {"regular": 0, "compact": 0, "context_seen": [], "requests": 0} -+ -+ class Server(BaseHTTPRequestHandler): -+ def log_message(self, *args): -+ pass -+ -+ def do_POST(self): -+ data = self.rfile.read(int(self.headers.get("Content-Length", 0))) -+ try: -+ body = json.loads(data) -+ except ValueError: -+ self.send_error(400, "Expected uncompressed fixture JSON") -+ return -+ state["requests"] += 1 -+ if state["requests"] > 10: -+ self.send_error(400, "Finite fixture request limit exceeded") -+ return -+ compact = not body.get("tools") -+ if compact: -+ state["compact"] += 1 -+ item = {"type": "message", "role": "assistant", "id": "summary-" + str(state["compact"]), -+ "content": [{"type": "output_text", "text": "Completed earlier C09 phase. Continue with next phase, do not repeat."}]} -+ output_tokens = 32 -+ else: -+ index = state["regular"] -+ state["regular"] += 1 -+ state["context_seen"].append("C09_FINITE_RECOVERY" in json.dumps(body.get("input"))) -+ if index < 3: -+ phase = c09.PHASES[index] -+ item = {"type": "function_call", "id": "fc-" + phase, "call_id": "call-" + phase, -+ "name": "exec_command", "arguments": json.dumps({"cmd": c09.COMMANDS[phase], -+ "max_output_tokens": c09.OUTPUT_TOKENS if index < 2 else 2048})} -+ output_tokens = 20000 if index < 2 else 8 -+ else: -+ item = {"type": "message", "role": "assistant", "id": "final", -+ "content": [{"type": "output_text", "text": json.dumps(completed_payload())}]} -+ output_tokens = 64 -+ rid = "fixture-" + str(state["requests"]) -+ events = [ -+ {"type": "response.created", "response": {"id": rid}}, -+ {"type": "response.output_item.done", "output_index": 0, "item": item}, -+ {"type": "response.completed", "response": {"id": rid, -+ "usage": {"input_tokens": 1000, "output_tokens": output_tokens, -+ "total_tokens": 1000 + output_tokens}}}, -+ ] -+ response = "".join("data: " + json.dumps(e) + "\n\n" for e in events).encode() -+ self.send_response(200) -+ self.send_header("Content-Type", "text/event-stream") -+ self.send_header("Content-Length", str(len(response))) -+ self.end_headers() -+ self.wfile.write(response) -+ -+ with ThreadingHTTPServer(("127.0.0.1", 0), Server) as server, tempfile.TemporaryDirectory(prefix="c09 cli ") as tmp: -+ worker = threading.Thread(target=server.serve_forever, daemon=True) -+ worker.start() -+ rt = Path(tmp) -+ base = v7.base -+ original_args = base.common_codex_args -+ actual_observed = process.run_observed -+ -+ def bounded_observed(args, **kwargs): -+ # The offline peer is instant. Bound a broken conformance test; -+ # the live C09 timeout remains 900 seconds. -+ kwargs["timeout"] = 120 -+ return actual_observed(args, **kwargs) -+ -+ def local_args(**kwargs): -+ args = original_args(**kwargs) -+ args[0] = binary -+ args += ["-c", 'model_provider="fixture"', "-c", 'model_providers.fixture.name="Fixture"', -+ "-c", f'model_providers.fixture.base_url="http://127.0.0.1:{server.server_port}/v1"', -+ "-c", 'model_providers.fixture.wire_api="responses"', -+ "-c", 'model_providers.fixture.requires_openai_auth=false', -+ "-c", 'model_providers.fixture.stream_max_retries=0', -+ "-c", 'model_providers.fixture.request_max_retries=0'] -+ return args -+ -+ try: -+ with mock.patch.dict(os.environ, {"CODEX_HOME": str(rt / "home")}), mock.patch.object( -+ base, "common_codex_args", side_effect=local_args -+ ), mock.patch.object(process, "run_observed", side_effect=bounded_observed), mock.patch.object(v7.compat.v4, "_write_result", side_effect=lambda **kw: (kw["result"], True)) as writer: -+ result, _ = v7.run_c09(root=ROOT, runtime_root=rt, schemas=base.write_schemas(rt / "schemas"), -+ version=version, os_name="offline-cli", source_commit="b" * 40, date="2026-09-06") -+ details = writer.call_args.kwargs -+ self.assertEqual(result, "REPRODUCED", details) -+ self.assertEqual(state["compact"], 2, state) -+ self.assertEqual(state["regular"], 4, state) -+ self.assertTrue(all(state["context_seen"]), state) -+ print("CODEX_01534_OFFLINE_CONFORMANCE_OK: 3 real tools, 2 compactions, 2 compact recovery contexts, 1 completed turn") -+ finally: -+ server.shutdown() -+ worker.join(timeout=5) -+ -+ -+if __name__ == "__main__": -+ unittest.main() -diff --git a/tests/test_qualification_execution_boundaries.py b/tests/test_qualification_execution_boundaries.py -index ba94760..59094e4 100644 ---- a/tests/test_qualification_execution_boundaries.py -+++ b/tests/test_qualification_execution_boundaries.py -@@ -78,19 +78,18 @@ class SubagentProcessContractTests(unittest.TestCase): - - class CompactionCompletionTests(unittest.TestCase): - def evaluate(self, payload, events, error, records=None): -+ from test_qualification_c09 import valid_records - with tempfile.TemporaryDirectory() as tmp, ExitStack() as stack: - root = Path(tmp) - if records is None: -- records = [ -- {"event": "PreCompact"}, {"event": "PostCompact"}, -- {"event": "PreCompact"}, {"event": "PostCompact"}, -- {"event": "PreToolUse"}, -- ] -+ records = valid_records() - stack.enter_context(mock.patch.object(v4, "_runtime_paths", return_value=(root,) * 7)) - stack.enter_context(mock.patch.object(base, "ensure_git_repo")) - stack.enter_context(mock.patch.object(base, "git", return_value="a" * 40)) - stack.enter_context(mock.patch.object(base, "git_snapshot", return_value={"head": "a" * 40})) - stack.enter_context(mock.patch.object(v4, "_start_active_run", return_value=(root, ".pursue/runs/test"))) -+ stack.enter_context(mock.patch.object(v4.c09, "seed_state")) -+ stack.enter_context(mock.patch.object(v4, "_create_checkpoint")) - stack.enter_context(mock.patch.object(v4, "_checkpoint_validation", return_value={"ok": True})) - stack.enter_context(mock.patch.object(v4, "_clear_hook_log")) - stack.enter_context(mock.patch.object(v4, "_read_hook_records", return_value=records)) -@@ -111,17 +110,20 @@ class CompactionCompletionTests(unittest.TestCase): - self.assertEqual(result["result"], "BLOCKED") - - def test_completed_positive_trial_still_passes(self) -> None: -- result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None) -+ from test_qualification_c09 import completed_payload, valid_events -+ result = self.evaluate(completed_payload(), valid_events(), None) - self.assertEqual(result["result"], "REPRODUCED") - self.assertTrue(result["expected_met"]) - - def test_second_compaction_still_requires_subsequent_tool_use(self) -> None: -- result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None, -+ from test_qualification_c09 import completed_payload, valid_events -+ result = self.evaluate(completed_payload(), valid_events(), None, - [{"event": "PreCompact"}, {"event": "PostCompact"}] * 2) - self.assertNotEqual(result["result"], "REPRODUCED") - - def test_observed_stop_with_valid_checkpoint_still_fails(self) -> None: -- result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None, -+ from test_qualification_c09 import completed_payload, valid_events -+ result = self.evaluate(completed_payload(), valid_events(), None, - [{"event": "PreCompact", "continue": False}, - {"event": "PostCompact"}] * 2 + [{"event": "PreToolUse"}]) - self.assertEqual(result["result"], "FAILED") -diff --git a/tools/install_ci_codex.py b/tools/install_ci_codex.py -new file mode 100644 -index 0000000..d4a1b37 ---- /dev/null -+++ b/tools/install_ci_codex.py -@@ -0,0 +1,53 @@ -+"""Install an official, hash-verified pinned CLI into a disposable hosted CI dir. -+ -+Only the explicit conformance CI job calls this networked helper. Local product -+and harness unit tests do not need Codex, credentials, network, or installation. -+""" -+from __future__ import annotations -+ -+import hashlib -+import io -+import json -+from pathlib import Path -+import sys -+import tarfile -+from urllib.request import Request, urlopen -+ -+TAG = "rust-v0.153.4" -+ASSETS = ("codex-x86_64-unknown-linux-musl.tar.gz", "bwrap-x86_64-unknown-linux-musl.tar.gz") -+ -+ -+def fetch(url): -+ with urlopen(Request(url, headers={"User-Agent": "PlanAnvil-CI-conformance"}), timeout=120) as response: -+ return response.read() -+ -+ -+def main(): -+ target = Path(sys.argv[1]).resolve() -+ target.mkdir(parents=True, exist_ok=True) -+ release = json.loads(fetch("https://api.github.com/repos/openai/codex/releases/tags/" + TAG)) -+ if release["tag_name"] != TAG or release["draft"] or release["prerelease"]: -+ raise ValueError("Unexpected Codex release metadata") -+ for name in ASSETS: -+ asset = next(a for a in release["assets"] if a["name"] == name) -+ url = "https://github.com/openai/codex/releases/download/" + TAG + "/" + name -+ if asset["browser_download_url"] != url: -+ raise ValueError("Unexpected binary origin") -+ data = fetch(url) -+ if asset["digest"] != "sha256:" + hashlib.sha256(data).hexdigest(): -+ raise ValueError("Codex release asset digest mismatch") -+ with tarfile.open(fileobj=io.BytesIO(data)) as archive: -+ members = [m for m in archive.getmembers() if m.isfile()] -+ if len(members) != 1: -+ raise ValueError("Expected exactly one executable") -+ stream = archive.extractfile(members[0]) -+ if stream is None: -+ raise ValueError("Missing release executable") -+ output = target / ("codex" if name.startswith("codex-") else "bwrap") -+ output.write_bytes(stream.read()) -+ output.chmod(0o755) -+ print(name + " " + asset["digest"]) -+ -+ -+if __name__ == "__main__": -+ main() diff --git a/.c09-publication/part-4.patch b/.c09-publication/part-4.patch deleted file mode 100644 index 6f98b49..0000000 --- a/.c09-publication/part-4.patch +++ /dev/null @@ -1,202 +0,0 @@ -diff --git a/tools/live_codex_qualification_harness_v4.py b/tools/live_codex_qualification_harness_v4.py -index 303651c..1fc55d5 100644 ---- a/tools/live_codex_qualification_harness_v4.py -+++ b/tools/live_codex_qualification_harness_v4.py -@@ -9,6 +9,8 @@ import sys - from pathlib import Path - from typing import Any, Callable - -+import qualification_c09 as c09 -+ - import live_codex_qualification_harness as v1 - import live_codex_qualification_harness_v2 as v2 - import live_codex_qualification_harness_v3 as prior -@@ -19,7 +21,7 @@ TARGET_CAPABILITIES = {"C06", "C08", "C09"} - _ORIGINAL_CAPABILITY_RUNTIME = prior.capability_runtime - - C08_COMPACT_LIMIT = 200 --C09_COMPACT_LIMIT = 1000 -+C09_COMPACT_LIMIT = c09.COMPACT_LIMIT - COMPACT_SCOPE = "body_after_prefix" - HOOK_LOG_RELATIVE = ".pursue/qualification-hook-events.jsonl" - -@@ -833,20 +835,23 @@ def _c09_runtime( - worktrees=worktrees, - version=version, - compact_limit=C09_COMPACT_LIMIT, -- create_checkpoint=True, -- segments=4, -- segment_bytes=32768, -+ create_checkpoint=False, -+ segments=0, -+ segment_bytes=0, -+ prepare_repo=c09.prepare_repo, -+ hook_proxy_source=c09.proxy_source(_hook_proxy_source()), - ) -+ c09.seed_state(planning, _run_root) -+ _create_checkpoint(planning=planning, run_root=_run_root) - fixture_commit = base.git(repo, "rev-parse", "HEAD") - checkpoint_before = _checkpoint_validation(planning) - _clear_hook_log(planning) -+ source_before = base.git_snapshot(repo) -+ files_before = (c09.file_fingerprint(repo), c09.file_fingerprint(planning)) - before = base.git_snapshot(planning) - payload, events, error = _run_codex_probe( - cwd=planning, -- prompt=_compact_probe_prompt( -- capability_id, -- ["segment-01.txt", "segment-02.txt", "segment-03.txt", "segment-04.txt"], -- ), -+ prompt=c09.prompt(), - schemas=schemas, - results_dir=results_dir, - position=1, -@@ -857,12 +862,18 @@ def _c09_runtime( - observe_process=True, - ) - after = base.git_snapshot(planning) -+ source_after = base.git_snapshot(repo) -+ files_after = (c09.file_fingerprint(repo), c09.file_fingerprint(planning)) - records = _read_hook_records(planning) - pre = _event_records(records, "PreCompact") - post = _event_records(records, "PostCompact") - stops = [item for item in pre if item.get("continue") is False] - checkpoint_after = _checkpoint_validation(planning) - -+ checks = c09.protocol_checks(events, records) -+ checks["source_and_planning_unchanged"] = (source_before == source_after and before == after -+ and files_before == files_after) -+ protocol_ok = all(checks.values()) - two_compactions = len(pre) >= 2 and len(post) >= 2 - continued_after_second = _continued_after_second_postcompact(records) - checkpoint_coherent = bool(checkpoint_before.get("ok")) and bool(checkpoint_after.get("ok")) -@@ -872,6 +883,8 @@ def _c09_runtime( - and not events.get("timeout") - and payload.get("capability_id") == capability_id - and payload.get("outcome") == "PASS" -+ and payload.get("trial") == c09.TRIAL -+ and "C09_FINISHED" in payload.get("observations", []) - ) - completion_blocker = error or ( - None if invocation_completed else "C09 did not return a completed positive structured result." -@@ -883,9 +896,10 @@ def _c09_runtime( - "trial_name": "checkpoint_auto_compact_recover_recompact", - "outcome": ( - "BLOCKED" -- if not invocation_completed or not two_compactions -+ if not invocation_completed or not two_compactions or not protocol_ok - else ("PASS" if checkpoint_coherent and no_stop_loop else "FAIL") - ), -+ "protocol_checks": checks, - "assertions": [ - { - "name": "codex_invocation_completed_without_timeout", -@@ -925,10 +939,10 @@ def _c09_runtime( - f"invocation_error={error or 'none'}", - f"invocation_completed={str(invocation_completed).lower()}", - ], -- "blocker": completion_blocker, -+ "blocker": completion_blocker or (None if protocol_ok else "C09 finite protocol checks incomplete"), - "event_summary": events, - "hook_timeline": [ -- {key: item[key] for key in ("event", "returncode", "continue", "additional_context") -+ {key: item[key] for key in ("event", "returncode", "continue", "additional_context", "source", "trigger", "c09_phase") - if key in item} - for item in records[-128:] - if item.get("event") in {"PreToolUse", "SessionStart", "PreCompact", "PostCompact"} -@@ -945,6 +959,9 @@ def _c09_runtime( - "runtime_cli_override": True, - "project_trust_method": "persisted_user_config", - "process_observation": "bounded_structural_jsonl", -+ "finite_phases": list(c09.PHASES), -+ "tool_max_output_tokens": c09.OUTPUT_TOKENS, -+ "canonical_action": "C09_FINITE_RECOVERY", - }, - } - -@@ -953,6 +970,10 @@ def _c09_runtime( - expected_met = False - blocker = "The deterministic C09 fixture did not begin with a valid checkpoint." - summary = "C09 blocked during deterministic fixture preparation." -+ elif not checks["source_and_planning_unchanged"]: -+ result, expected_met = "FAILED", False -+ blocker = "C09 changed source or planning repository state." -+ summary = "C09 failed repository immutability." - elif not invocation_completed: - result = "BLOCKED" - expected_met = False -@@ -978,6 +999,10 @@ def _c09_runtime( - expected_met = False - blocker = "Two compactions completed, but no subsequent real tool call demonstrated continuation after the second compaction." - summary = "C09 blocked because freedom from a permanent stop loop was not fully exercised." -+ elif not protocol_ok: -+ result, expected_met = "BLOCKED", False -+ blocker = "C09 finite protocol was not verified: " + ", ".join(k for k, ok in checks.items() if not ok) -+ summary = "C09 lacks a completed ordered two-cycle recovery proof." - else: - result = "REPRODUCED" - expected_met = True -diff --git a/tools/live_codex_qualification_regression.py b/tools/live_codex_qualification_regression.py -index 36e860e..d9c77e7 100644 ---- a/tools/live_codex_qualification_regression.py -+++ b/tools/live_codex_qualification_regression.py -@@ -170,7 +170,7 @@ def _patched_v4(cap_runtime: Path, capability_id: str) -> Iterator[None]: - v4._run_codex_probe = run_probe - v4._compact_probe_prompt = _compact_prompt - v4.C08_COMPACT_LIMIT = 40 -- v4.C09_COMPACT_LIMIT = 200 -+ # C09 owns its finite workload and threshold; do not override it here. - try: - yield - finally: -diff --git a/tools/prepare_capabilities.py b/tools/prepare_capabilities.py -index 7a5cea2..b851fb0 100644 ---- a/tools/prepare_capabilities.py -+++ b/tools/prepare_capabilities.py -@@ -9,6 +9,8 @@ import shutil - import tarfile - from pathlib import Path, PurePosixPath - -+import qualification_c09 as c09 -+ - PART_GLOB = 'templates.part*' - - C06_CODEX0152_OVERLAY = { -@@ -131,21 +133,28 @@ python3 tools/live_codex_qualification_harness_v7.py \\ - } - - C09_COMPLETION_OVERLAY = { -- 'fixture/README.md': '''# C09 fixture and completion requirements -+ 'fixture/README.md': """# C09 finite recovery fixture - --The outer harness installs the actual product, creates the planning worktree and a valid checkpoint, and then exercises genuine automatic compaction. -+The outer harness installs the unmodified product, prepares this fixture BEFORE bootstrap, replaces the bootstrap MAP_INSTRUCTIONS action with the hashed C09_FINITE_RECOVERY scenario, then creates and validates a real product checkpoint. - --Two real compaction cycles, coherent checkpoint/Git state, and subsequent real tool use remain required. They are not sufficient when Codex times out or fails to return a completed positive structured C09 result. Partial event counts cannot turn an incomplete invocation into REPRODUCED. -+One Codex turn executes exactly three read-only commands: first, second, finish. Every command uses the actual installed checkpoint/Git validator and reads canonical recovery inputs. Only first and second emit bounded inert compaction stimuli. Finish emits a small receipt and ends the workload. No manual hook calls, synthetic live events, canonical writes during the turn, permission changes or repeated initial reads are permitted. - --The deliberately low fixture threshold is not a product default. This correction does not silently retune it or weaken C08's intentional negative stop trial. Record a remaining timeout as BLOCKED. Retain bounded content-free event and hook sequences, command labels/counts, error categories, and process completion/owned-tree cleanup status. Never persist raw stdout/stderr or use partial progress as proof of successful completion. --''', -- 'run-command.txt': '''# Existing controlled workflow: main -> recovery for C09/C10/C13. --# The recovery driver selects the same v7 capability runtime used by full. -+The trigger is 8192 body-after-prefix tokens; requested tool output budget is 65536 tokens for each stimulus. These are disposable fixture settings, not product defaults. The active compatibility layer must not replace this trigger with the old 200-token value. TokenBudget is disabled only in the existing isolated qualification configuration. -+ -+Require two ordered automatic PreCompact -> PostCompact -> SessionStart(source=compact) cycles, three successful canonical/Git reconciliations, real tool use after the second cycle, exact command order, valid checkpoints, unchanged source/planning files and Git, and one completed positive C09 turn. Timeout (still 900 seconds), extra/failed/repeated tools, missing recovery, failed hooks, reader/cleanup failures or incomplete evidence cannot be REPRODUCED. No raw output or private canonical state is persisted. -+ -+The fixed scenario resolves proven conflicting fixture instructions and removes an over-aggressive trigger. Historical #23 labels do not identify its exact commands, so the historical timeout's complete causal chain is not claimed as observed. Live success must still be established by the actual model-backed run. -+""", -+ 'fixture/c09_probe.py': c09.PROBE_SOURCE, -+ 'prompt.txt': c09.prompt(), -+ 'run-command.txt': """# Existing controlled workflow: main -> recovery for C09/C10/C13, or full for C01-C16. -+# Both select the identical C09 runtime; only full can satisfy the release gate. - python3 tools/live_codex_qualification_recovery.py --root --source-commit --run-id --output --allow-c13-non-ephemeral-fallback --# A targeted pass is not a full C01-C16 release pass. --''', -+# A loopback CLI conformance pass is offline and cannot count as live capability evidence. -+""", - } - -+ - C10_ISOLATION_OVERLAY = { - 'README.md': """# C10 — Recovery context through SessionStart - diff --git a/.c09-publication/part-5.patch b/.c09-publication/part-5.patch deleted file mode 100644 index 73166ff..0000000 --- a/.c09-publication/part-5.patch +++ /dev/null @@ -1,244 +0,0 @@ -diff --git a/tools/qualification_c09.py b/tools/qualification_c09.py -new file mode 100644 -index 0000000..1d94a4e ---- /dev/null -+++ b/tools/qualification_c09.py -@@ -0,0 +1,205 @@ -+"""Finite C09 workload: real read-only recovery, two large outputs, then finish. -+ -+This module prepares the disposable fixture, not product behavior. It never -+invokes lifecycle hooks or supplies fabricated live events to the evaluator. -+""" -+from __future__ import annotations -+ -+import hashlib -+import json -+import os -+from pathlib import Path -+from typing import Any -+ -+COMPACT_LIMIT = 8192 -+OUTPUT_TOKENS = 65536 -+PHASES = ("first", "second", "finish") -+SCRIPT = "qualification-payload/c09_probe.py" -+TRIAL = "checkpoint_auto_compact_recover_recompact" -+COMMANDS = {phase: f"python3 -B {SCRIPT} {phase}" for phase in PHASES} -+ -+# Shipped only in a disposable qualification repository. All runtime operations -+# are reads. Checkpoint/Git decisions come from the installed product validator. -+PROBE_SOURCE = r'''from __future__ import annotations -+import hashlib, json, sys -+from pathlib import Path -+ -+root = Path(__file__).resolve().parents[1] -+sys.path.insert(0, str(root / ".codex" / "hooks")) -+from plan_anvil_hooklib import active_run_for_event -+from plan_anvil_checkpoint import validate_checkpoint_for_run -+ -+def main(): -+ if len(sys.argv) != 2 or sys.argv[1] not in {"first", "second", "finish"}: -+ return 2 -+ phase = sys.argv[1] -+ active = active_run_for_event({"cwd": str(root)}) -+ if active is None: -+ return 2 -+ check = validate_checkpoint_for_run(active) -+ if not check.ok or check.path is None: -+ print(json.dumps({"c09_phase": phase, "checkpoint_ok": False})) -+ return 2 -+ files = [active.run_root / "manifest.json", active.run_root / "state.json", -+ active.run_root / "local-state.json", check.path] -+ files += sorted((root / ".pursue").glob("SYSTEM_PROFILE*.md")) -+ # Fully read the canonical recovery inputs without printing local-state or -+ # paths. The real validator above reconciles canonical identity with Git. -+ sizes = [len(path.read_bytes()) for path in files] -+ state = json.loads((active.run_root / "state.json").read_text(encoding="utf-8")) -+ if state.get("next_action") != {"type": "C09_FINITE_RECOVERY", "target": "evidence/c09-scenario.json"}: -+ return 2 -+ scenario = json.loads((active.run_root / "evidence/c09-scenario.json").read_text(encoding="utf-8")) -+ if scenario.get("phases") != ["first", "second", "finish"]: -+ return 2 -+ receipt = {"c09_phase": phase, "checkpoint_ok": True, "canonical_read": True, -+ "git_reconciled": True, "canonical_file_count": len(files), -+ "canonical_bytes": sum(sizes)} -+ print(json.dumps(receipt, sort_keys=True)) -+ if phase != "finish": -+ # Deterministic inert high-volume output, not an instruction or a secret. -+ # A phase-specific stimulus avoids eight indistinguishable repeated reads. -+ for i in range(1024): -+ print(hashlib.sha512((phase + ":" + str(i)).encode()).hexdigest()) -+ print(json.dumps(receipt, sort_keys=True)) -+ print("C09_NEXT=" + {"first": "second", "second": "finish", "finish": "RETURN_RESULT"}[phase]) -+ return 0 -+ -+if __name__ == "__main__": -+ try: -+ raise SystemExit(main()) -+ except (OSError, ValueError, KeyError, TypeError): -+ # No raw canonical contents/private paths in errors or evidence. -+ print("C09_FIXTURE_READ_FAILED", file=sys.stderr) -+ raise SystemExit(2) -+''' -+ -+ -+def prepare_repo(repo: Path) -> None: -+ path = repo / SCRIPT -+ path.parent.mkdir(parents=True, exist_ok=True) -+ path.write_text(PROBE_SOURCE, encoding="utf-8") -+ -+ -+def seed_state(planning: Path, run_root: str) -> None: -+ """Replace bootstrap's unrelated MAP_INSTRUCTIONS action before checkpointing.""" -+ run = planning / run_root -+ scenario = {"schema_version": "1.0", "phases": list(PHASES), "commands": COMMANDS, -+ "completion": "Return the final C09 result after finish; never restart first."} -+ data = (json.dumps(scenario, indent=2, sort_keys=True) + "\n").encode() -+ path = run / "evidence/c09-scenario.json" -+ path.parent.mkdir(parents=True, exist_ok=True) -+ path.write_bytes(data) -+ state_path = run / "state.json" -+ state = json.loads(state_path.read_text(encoding="utf-8")) -+ state["revision"] += 1 -+ state["next_action"] = {"type": "C09_FINITE_RECOVERY", "target": "evidence/c09-scenario.json"} -+ state.setdefault("artifact_hashes", {})["evidence/c09-scenario.json"] = "sha256:" + hashlib.sha256(data).hexdigest() -+ state_path.write_text(json.dumps(state, indent=2, sort_keys=True) + "\n", encoding="utf-8") -+ -+ -+def prompt() -> str: -+ return f'''Qualify C09 using this finite, read-only scenario in ONE Codex turn. -+The outer harness has already installed PlanAnvil and created a valid checkpoint. -+Do not start a plan, map instructions, or create/repair any files. The canonical -+next action is C09_FINITE_RECOVERY, not the earlier bootstrap MAP_INSTRUCTIONS. -+ -+Make exactly THREE sequential exec_command calls, each once and in this order: -+1. cmd={COMMANDS["first"]!r}, max_output_tokens={OUTPUT_TOKENS} -+2. cmd={COMMANDS["second"]!r}, max_output_tokens={OUTPUT_TOKENS} -+3. cmd={COMMANDS["finish"]!r}, max_output_tokens=2048 -+Use workdir for the current planning directory; do not prepend cd or combine calls. -+Wait for each command to finish before choosing the next one. The first two outputs -+are inert compaction stimuli: do not analyse, quote, hash, or reread their padding. -+ -+After each automatic compaction continue with the NEXT phase, never restart first. -+Each command fully reads manifest/state/local-state/checkpoint/profiles and calls the -+actual product checkpoint/Git validator. This performs the recovery requested by the -+SessionStart context; do not perform additional recovery reads outside those commands. -+No manual hook calls, telemetry reads, other tools, state changes, or subagents. -+ -+After finish exits 0, make NO further calls. Return capability_id C09, trial {TRIAL}, -+outcome PASS, and observation C09_FINISHED. If a command fails, stop with FAIL/BLOCKED; -+do not retry. The outer evaluator, not your self-report, checks the real compactions. -+''' -+ -+ -+def proxy_source(original: str) -> str: -+ """Extend only C09 telemetry without changing stdout or any product decision.""" -+ addition = '''\nif isinstance(event, dict): -+ source = event.get("source") -+ if source in {"startup", "resume", "clear", "compact"}: -+ record["source"] = source -+ trigger = event.get("trigger") -+ if trigger in {"auto", "manual"}: -+ record["trigger"] = trigger -+ raw_command = event.get("tool_input", {}).get("command") if isinstance(event.get("tool_input"), dict) else None -+ commands = ''' + repr({command: phase for phase, command in COMMANDS.items()}) + ''' -+ if event_name == "PreToolUse": -+ record["c09_phase"] = commands.get(raw_command, "other") -+''' -+ anchor = 'try:\n parsed = json.loads(completed.stdout)' -+ # The active compat adapter has this exact executable contract, covered by -+ # subprocess tests. Reject a future adapter change instead of silently losing evidence. -+ if anchor not in original: -+ raise ValueError("C09 telemetry adapter contract changed") -+ return original.replace(anchor, addition + "\n" + anchor, 1) -+ -+ -+def file_fingerprint(repo: Path) -> str: -+ """Hash all fixture files, including ignored canonical state, without paths in evidence. -+ -+ Git status alone misses edits to an already-untracked state file. Exclude only -+ Git's database/locator and the known legacy telemetry sink (normally external). -+ """ -+ if not repo.is_dir(): -+ raise ValueError("C09 fixture directory is missing") -+ digest = hashlib.sha256() -+ def walk_error(_error: OSError) -> None: -+ raise ValueError("C09 fixture files could not be completely inspected") -+ -+ for directory, dirs, names in os.walk(repo, onerror=walk_error): -+ for name in dirs: -+ if name != ".git" and (Path(directory) / name).is_symlink(): -+ raise ValueError("Unexpected directory symlink in C09 fixture") -+ dirs[:] = sorted(d for d in dirs if d != ".git") -+ for name in sorted(names): -+ path = Path(directory) / name -+ rel = path.relative_to(repo).as_posix() -+ if rel in {".git", ".pursue/qualification-hook-events.jsonl"}: -+ continue -+ if path.is_symlink(): -+ raise ValueError("Unexpected symlink in C09 fixture") -+ data = path.read_bytes() -+ digest.update(rel.encode("utf-8") + b"\0") -+ digest.update(hashlib.sha256(data).digest()) -+ return digest.hexdigest() -+ -+ -+def protocol_checks(events: dict[str, Any], records: list[dict[str, Any]]) -> dict[str, bool]: -+ commands = [r for r in events.get("event_tail", []) -+ if r.get("event") == "item.completed" and r.get("item") == "command_execution"] -+ expected = ["c09_" + phase for phase in PHASES] -+ command_order = ([r.get("command") for r in commands] == expected -+ and all(r.get("exit_code") == 0 and r.get("c09_receipt_ok") is True for r in commands) -+ and events.get("completed_command_items") == 3 -+ and not events.get("event_tail_truncated")) -+ tool_rows = [r for r in records if r.get("event") == "PreToolUse"] -+ hook_order = [r.get("c09_phase") for r in tool_rows] == list(PHASES) -+ positions = {r.get("c09_phase"): i for i, r in enumerate(records) if r.get("event") == "PreToolUse"} -+ cycles = hook_order -+ for start, end in (("first", "second"), ("second", "finish")): -+ between = records[positions[start] + 1:positions[end]] if hook_order else [] -+ lifecycle = [r.get("event") for r in between if r.get("event") in {"PreCompact", "PostCompact", "SessionStart"}] -+ cycles = cycles and lifecycle == ["PreCompact", "PostCompact", "SessionStart"] -+ cycles = cycles and all(r.get("trigger") == "auto" for r in between if r.get("event") in {"PreCompact", "PostCompact"}) -+ cycles = cycles and any(r.get("event") == "SessionStart" and r.get("source") == "compact" -+ and r.get("additional_context") is True for r in between) -+ hooks_ok = bool(records) and all(r.get("returncode") == 0 for r in records) -+ forbidden = sum(events.get("item_types", {}).get(k, 0) for k in -+ ("file_change", "mcp_tool_call", "web_search", "collab_tool_call", "other")) -+ return {"three_completed_reconciliations": command_order, -+ "ordered_automatic_recovery_cycles": bool(cycles), "product_hooks_succeeded": hooks_ok, -+ "no_unexpected_tools": not forbidden, -+ "turn_completed": events.get("event_types", {}).get("turn.completed") == 1 -+ and not events.get("event_types", {}).get("turn.failed")} -diff --git a/tools/qualification_process.py b/tools/qualification_process.py -index 3a53fff..4857b2f 100644 ---- a/tools/qualification_process.py -+++ b/tools/qualification_process.py -@@ -29,6 +29,7 @@ ITEM_TYPES = frozenset({ - }) - STATUSES = frozenset({"in_progress", "completed", "failed", "declined", "cancelled"}) - COMMANDS = { -+ **{f"python3 -B qualification-payload/c09_probe.py {p}": f"c09_{p}" for p in ("first", "second", "finish")}, - **{f"cat qualification-payload/segment-{i:02d}.txt": f"segment_{i:02d}" for i in range(1, 5)}, - "git status --porcelain=v1 --untracked-files=all": "git_status", - "git rev-parse HEAD": "git_head", -@@ -118,6 +119,20 @@ class StructuralEvents: - if item_kind == "command_execution": - label = command_label(item.get("command")) - row["command"] = label -+ if label in {"c09_first", "c09_second", "c09_finish"} and kind == "item.completed": -+ # Fixed scalar receipt only; never persist output or paths. -+ raw_output = item.get("aggregated_output", "") -+ receipt = {} -+ if isinstance(raw_output, str): -+ for line in raw_output.splitlines()[-4:]: -+ try: -+ candidate = json.loads(line) -+ except (ValueError, RecursionError): -+ continue -+ if isinstance(candidate, dict): -+ receipt = candidate -+ row["c09_receipt_ok"] = (receipt.get("c09_phase") == label[4:] -+ and all(receipt.get(k) is True for k in ("checkpoint_ok", "canonical_read", "git_reconciled"))) - if kind == "item.completed": - self.completed_commands += 1 - self.commands[label] += 1 diff --git a/.github/workflows/c09-reviewed-tree.yml b/.github/workflows/c09-reviewed-tree.yml deleted file mode 100644 index af4fcac..0000000 --- a/.github/workflows/c09-reviewed-tree.yml +++ /dev/null @@ -1,63 +0,0 @@ -name: Stage reviewed C09 Git tree -on: - push: - branches: [fix/c09-finite-recovery] - paths: [.github/workflows/c09-reviewed-tree.yml] -permissions: - contents: read -jobs: - stage: - if: github.repository == 'KeyffMS/PlanAnvil' && github.ref == 'refs/heads/fix/c09-finite-recovery' - runs-on: ubuntu-latest - timeout-minutes: 5 - permissions: - contents: write - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 - with: - fetch-depth: 3 - persist-credentials: false - - name: Reconstruct exact reviewed candidate without executing it - shell: bash - run: | - set -euo pipefail - python3 - <<'PY' - import hashlib, json, os, subprocess - from pathlib import Path - def git(*args): - return subprocess.check_output(['git', *args], text=True).strip() - assert git('rev-parse', 'HEAD~2') == '44f95258079d6b3ab5b79c178815bbc085edbe02' - patch = b''.join(Path('.c09-publication/part-' + str(i) + '.patch').read_bytes() for i in range(1, 6)) - assert hashlib.sha256(patch).hexdigest() == '2cdb37a41d13bde3d1165a4ab7bcadca93258a049dc45a55653aa3e2d8174180' - temp = Path(os.environ['RUNNER_TEMP']) - patch_path = temp / 'reviewed-c09.patch' - patch_path.write_bytes(patch) - subprocess.run(['git', 'apply', '--check', str(patch_path)], check=True) - subprocess.run(['git', 'apply', '--index', str(patch_path)], check=True) - names = git('diff', '--cached', '--name-only').splitlines() - assert len(names) == 13 - entries = [{'path': n, 'sha': git('hash-object', n), 'content': Path(n).read_text(encoding='utf-8')} for n in names] - subprocess.run(['git', 'rm', '-r', '.c09-publication', '.github/workflows/c09-reviewed-tree.yml'], check=True) - subprocess.run(['git', 'diff', '--cached', '--check'], check=True) - assert git('write-tree') == '682a9239c32f356c8f9d7510139bf341a5947487' - (temp / 'c09-blobs.json').write_text(json.dumps(entries), encoding='utf-8') - print('Reviewed 13-file candidate reconstructed exactly; temporary staging files excluded.') - PY - - name: Store verified content objects only; no tree, commit, ref or workflow changes - shell: bash - env: - GH_TOKEN: ${{ github.token }} - run: | - set -euo pipefail - python3 - <<'PY' - import json, os, subprocess - from pathlib import Path - entries = json.loads((Path(os.environ['RUNNER_TEMP']) / 'c09-blobs.json').read_text()) - for entry in entries: - payload = json.dumps({'content': entry['content'], 'encoding': 'utf-8'}) - result = subprocess.run(['gh', 'api', '--method', 'POST', 'repos/KeyffMS/PlanAnvil/git/blobs', '--input', '-'], input=payload, text=True, capture_output=True, check=True) - actual = json.loads(result.stdout)['sha'] - assert actual == entry['sha'] - print(entry['path'] + ' ' + actual) - print('VERIFIED_C09_BLOBS=13; no branches changed') - PY diff --git a/.github/workflows/plananvil-tests.yml b/.github/workflows/plananvil-tests.yml index 544e1ee..88d75f7 100644 --- a/.github/workflows/plananvil-tests.yml +++ b/.github/workflows/plananvil-tests.yml @@ -82,3 +82,22 @@ jobs: run: python tools/build_release.py --output "$RUNNER_TEMP/plananvil-dist" - name: Check patch whitespace run: git diff --check + + codex-conformance: + name: Codex 0.153.4 loopback conformance (no model service) + runs-on: ubuntu-latest + timeout-minutes: 8 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 + with: + python-version: "3.11" + - name: Install pinned official CLI into runner temp + run: | + python tools/install_ci_codex.py "$RUNNER_TEMP/codex-conformance-bin" + echo "$RUNNER_TEMP/codex-conformance-bin" >> "$GITHUB_PATH" + echo "PLANANVIL_TEST_CODEX_BIN=$RUNNER_TEMP/codex-conformance-bin/codex" >> "$GITHUB_ENV" + - name: Verify real CLI lifecycle against deterministic loopback responses + run: python -m unittest discover -s tests -p test_qualification_c09_cli.py -v diff --git a/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md b/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md new file mode 100644 index 0000000..4cf95bc --- /dev/null +++ b/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md @@ -0,0 +1,44 @@ +# C09 finite recovery audit — 2026-09-06 + +## Scope and evidence boundary + +Reviewed baseline: PlanAnvil `44f95258079d6b3ab5b79c178815bbc085edbe02`, live qualification #23 (`34021584365`), Codex CLI `0.153.4`. C10 and C13 passed in that run and their runtimes are not changed here. No live result is manufactured by offline testing. + +#23 observed 20 successful commands and 20 automatic compaction cycles before the 900-second deadline; no PreCompact stop and no completed turn. Its content-free command labels were `other`. They do not establish which commands ran or prove the complete historical causal chain. + +## Proven fixture defects + +The active call path is v7.run_c09 -> compat._codex0152_compaction -> regression._patched_v4 -> v4._c09_runtime. The adapter overwrote C09's nominal threshold with **200 tokens** and replaced the generic prompt with eight repeated large reads. Bootstrap/checkpoint state still advertised **MAP_INSTRUCTIONS**, whereas the turn requested a separate compaction exercise. There was no checked finite phase protocol reconciling the canonical recovery instruction with the requested workload. + +These are inspectable setup conflicts, not proof of a Codex deadlock. Raising the timeout or accepting partial event counts would not fix them. A very small trigger can repeatedly interrupt ordinary recovery tools; repeated payload reads without a stable explicit completion protocol also make progress fragile. + +## Pinned upstream source review + +All source references below are to `openai/codex`, tag `rust-v0.153.4`: + +- `codex-rs/core/src/session/turn.rs`: mid-turn rollover requires `needs_follow_up` and either a new-window request or exhausted token budget. After successful automatic compaction the turn runs pending SessionStart hooks and continues sampling. A final response without follow-up can end the turn rather than forcing another rollover. +- `codex-rs/core/src/session/context_window.rs`: body-after-prefix usage is active tokens minus the window prefill baseline; the trigger includes an optional TokenBudget fallback buffer. The model hard context cap remains independent. +- `codex-rs/core/src/compact.rs`: compaction replaces history; mid-turn initial context is injected before the last user message. Actual PreCompact/PostCompact events bracket successful compaction. It does not make an arbitrary sequence of repeated tool calls finite. +- `codex-rs/core/src/hook_runtime.rs`: model-visible recovery after compaction uses pending SessionStart(source=compact). PostCompact is not an additional-context delivery channel. +- `codex-rs/core/src/tools/handlers/unified_exec.rs` and `unified_exec/exec_command.rs`: real exec_command accepts `cmd` and `max_output_tokens`; the original command is used for the canonical Bash hook and shell argv is used for execution. +- `codex-rs/core/tests/suite/compact.rs`: upstream integration tests drive real tool calls/compaction with controlled Responses/SSE data. This supports a separate loopback conformance test, not calling that test live model qualification. + +Official references: https://developers.openai.com/codex/hooks/ and https://developers.openai.com/codex/config-reference/ (including their official redirects). These define supported configuration and lifecycle behavior; they do not promise model compliance with a specific test prompt. + +## Repair + +C09 now has one finite turn with three explicitly ordered read-only commands: **first -> second -> finish**. Each calls the installed product checkpoint/Git validator and fully reads canonical recovery inputs. The first two produce phase-specific inert high-volume stimuli; finish emits only a small receipt and requires the final structured result. No command calls hooks manually, changes canonical state, performs implementation work or reads qualification telemetry. + +The outer fixture prepares its helper before bootstrap and seeds a hash-checked C09_FINITE_RECOVERY next action before the real checkpoint is created. Recovery and the initial request now describe the same task, not MAP_INSTRUCTIONS. Requested output budgets preserve the stimuli (65536 tokens); the disposable trigger is 8192 body-after-prefix tokens, no longer silently overwritten by the 200-token adapter. This threshold/workload separation leaves room for ordinary recovery and ending the turn. It is not a product configuration change or a claim that every model response is bounded below that threshold. + +The evaluator strengthens, rather than relaxes, the gate: three successful exact commands/receipts; ordered automatic PreCompact -> PostCompact -> SessionStart(compact) cycles between the first/second/finish calls; successful product hook exit codes; real reconciliation after the second compaction; valid before/after checkpoint; unchanged source/planning Git **and file bytes including ignored state**; one completed positive structured C09 result. Timeout, missing context, extra/repeated/reordered/failed tools, unexpected tools, incomplete streams, reader or cleanup failure cannot pass. The 900-second live deadline is unchanged. + +Fixture source and exact prompt are included in the manifest-validated evidence template. Only fixed phase labels, counts, booleans and codes are retained; neither private canonical contents nor raw transcripts are uploaded. + +## Verification plan and limitations + +Executable offline tests run actual installer/start/checkpoint/Git/hooks and the three fixture commands; only the model process is substituted in the offline driver. Negative cases cover stale checkpoint, wrong canonical action, byte changes missed by Git status, missing/manual/reordered lifecycle, repeated/failed/extra tools, incomplete completion and receipt validation. + +A separate hosted conformance job downloads the exact official CLI release and verifies asset hashes. It runs the real CLI, real sandbox, tools and product hooks against a loopback Responses simulator, with no credentials or external model requests. The simulator selects responses and usage; passing demonstrates CLI integration, not autonomous live-model behavior. The normal offline suite does not require downloads or Codex. + +Existing cross-platform product/harness suites, packaging, C10/C13 regression tests, evidence validation and release checks remain mandatory. The full model-backed C01-C16 workflow is the final qualification, not an outcome inferred from CI. No sandbox, approvals, auth ownership, product hook output, C13 fallback or release requirement is weakened. diff --git a/tests/test_live_codex_qualification_harness_v4.py b/tests/test_live_codex_qualification_harness_v4.py index 235248d..bdfdde4 100644 --- a/tests/test_live_codex_qualification_harness_v4.py +++ b/tests/test_live_codex_qualification_harness_v4.py @@ -36,7 +36,7 @@ def test_c08_uses_real_auto_compact_stop_and_repair(self) -> None: self.assertIn("_create_checkpoint(planning=planning, run_root=run_root)", self.source) def test_c09_requires_two_real_compactions_and_continuation(self) -> None: - self.assertIn("C09_COMPACT_LIMIT = 1000", self.source) + self.assertIn("C09_COMPACT_LIMIT = c09.COMPACT_LIMIT", self.source) self.assertIn("len(pre) >= 2 and len(post) >= 2", self.source) self.assertIn("_continued_after_second_postcompact", self.source) self.assertIn("tool_use_after_second_postcompact", self.source) diff --git a/tests/test_live_codex_qualification_regression.py b/tests/test_live_codex_qualification_regression.py index ac41b14..b56340c 100644 --- a/tests/test_live_codex_qualification_regression.py +++ b/tests/test_live_codex_qualification_regression.py @@ -30,7 +30,7 @@ def test_hook_telemetry_is_external_and_fail_open(self) -> None: def test_compaction_repairs_use_low_redundant_triggers(self) -> None: self.assertIn("v4.C08_COMPACT_LIMIT = 40", self.source) - self.assertIn("v4.C09_COMPACT_LIMIT = 200", self.source) + self.assertNotIn("v4.C09_COMPACT_LIMIT = 200", self.source) self.assertIn("expanded = [name for name in names for _ in range(2)]", self.source) self.assertIn("return v4._c08_runtime(**kwargs)", self.source) self.assertIn("return v4._c09_runtime(**kwargs)", self.source) diff --git a/tests/test_qualification_c09.py b/tests/test_qualification_c09.py new file mode 100644 index 0000000..5c56eff --- /dev/null +++ b/tests/test_qualification_c09.py @@ -0,0 +1,212 @@ +from __future__ import annotations + +from contextlib import ExitStack +import json +import os +from pathlib import Path +import re +import subprocess +import sys +import tempfile +import unittest +from unittest import mock + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tools")) +import qualification_c09 as c09 +import qualification_process as observation +import live_codex_qualification_harness_v7 as v7 +from test_qualification_execution_boundaries import execute_hook + +v4 = v7.compat.v4 +base = v4.base + + +def completed_payload(): + return {"capability_id": "C09", "trial": c09.TRIAL, "outcome": "PASS", + "assertions": [], "observations": ["C09_FINISHED"], "blocker": None} + + +def valid_events(): + return {"event_types": {"turn.completed": 1}, "completed_command_items": 3, + "event_tail": [{"event": "item.completed", "item": "command_execution", + "command": "c09_" + phase, "exit_code": 0, "c09_receipt_ok": True} + for phase in c09.PHASES]} + + +def valid_records(): + result = [] + for phase in c09.PHASES: + result.append({"event": "PreToolUse", "c09_phase": phase, "returncode": 0}) + if phase != "finish": + result += [{"event": "PreCompact", "trigger": "auto", "returncode": 0}, + {"event": "PostCompact", "trigger": "auto", "returncode": 0}, + {"event": "SessionStart", "source": "compact", "additional_context": True, "returncode": 0}] + return result + + +class C09ProtocolTests(unittest.TestCase): + def test_active_adapter_cannot_restore_the_old_200_token_threshold(self): + with tempfile.TemporaryDirectory() as tmp: + old_c08 = v4.C08_COMPACT_LIMIT + with v7.compat._codex0152_compaction(Path(tmp), "C09"): + self.assertEqual(v4.C09_COMPACT_LIMIT, c09.COMPACT_LIMIT) + self.assertEqual(v4.C09_COMPACT_LIMIT, 8192) + self.assertEqual(v4.C08_COMPACT_LIMIT, old_c08) + + def test_file_fingerprint_detects_ignored_state_edits_but_not_git_database(self): + with tempfile.TemporaryDirectory() as tmp: + repo = Path(tmp) + path = repo / ".pursue/state.json" + path.parent.mkdir() + path.write_text('{"revision": 1}') + before = c09.file_fingerprint(repo) + (repo / ".git").mkdir() + (repo / ".git/index").write_text("database") + self.assertEqual(c09.file_fingerprint(repo), before) + path.write_text('{"revision": 2}') + self.assertNotEqual(c09.file_fingerprint(repo), before) + + def test_protocol_is_finite_and_explicit_about_completion(self): + prompt = c09.prompt() + for command in c09.COMMANDS.values(): + self.assertEqual(prompt.count(command), 1) + self.assertIn("C09_FINISHED", prompt) + self.assertIn("outcome PASS", prompt) + self.assertIn("max_output_tokens=65536", prompt) + + def test_full_ordered_proof_passes(self): + self.assertTrue(all(c09.protocol_checks(valid_events(), valid_records()).values())) + + def test_counts_alone_never_pass(self): + checks = c09.protocol_checks({"completed_command_items": 20}, valid_records()) + self.assertFalse(checks["three_completed_reconciliations"]) + self.assertFalse(checks["turn_completed"]) + + def test_reordered_repeated_missing_and_nonzero_tools_are_rejected(self): + for mode in ("repeat", "reorder", "missing", "exit", "receipt", "truncated", "forbidden"): + with self.subTest(mode=mode): + events = valid_events() + rows = events["event_tail"] + if mode == "repeat": rows.append(rows[0]) + elif mode == "reorder": rows.reverse() + elif mode == "missing": rows.pop() + elif mode == "exit": rows[1]["exit_code"] = 2 + elif mode == "receipt": rows[1]["c09_receipt_ok"] = False + elif mode == "truncated": events["event_tail_truncated"] = True + elif mode == "forbidden": events["item_types"] = {"mcp_tool_call": 1} + self.assertFalse(all(c09.protocol_checks(events, valid_records()).values())) + + def test_manual_compaction_missing_recovery_and_failed_hook_are_rejected(self): + for mode in ("manual", "no_context", "failed", "wrong_order"): + rows = valid_records() + if mode == "manual": rows[1]["trigger"] = "manual" + elif mode == "no_context": rows[3]["additional_context"] = False + elif mode == "failed": rows[1]["returncode"] = 1 + else: rows[1], rows[2] = rows[2], rows[1] + self.assertFalse(all(c09.protocol_checks(valid_events(), rows).values())) + + def test_cli_command_shapes_and_receipts_are_classified_without_content(self): + receipt = {"c09_phase": "first", "checkpoint_ok": True, + "canonical_read": True, "git_reconciled": True} + command = c09.COMMANDS["first"] + self.assertEqual(observation.command_label(command), "c09_first") + self.assertEqual(observation.command_label("/bin/bash -lc '" + command + "'"), "c09_first") + self.assertEqual(observation.command_label(command + "; echo secret"), "other") + collector = observation.StructuralEvents() + collector.accept(json.dumps({"type": "item.completed", "item": { + "type": "command_execution", "command": command, "exit_code": 0, + "aggregated_output": "NEVER_PERSIST_THIS\n" + json.dumps(receipt) + "\nC09_NEXT=second\n", + }}).encode()) + result = collector.summary() + self.assertTrue(result["event_tail"][0]["c09_receipt_ok"]) + self.assertNotIn("NEVER_PERSIST_THIS", json.dumps(result)) + + +class C09FixtureExecutionTests(unittest.TestCase): + def test_real_install_checkpoint_three_readonly_commands_and_product_hooks(self): + """Offline lifecycle driver; real product operations, NOT live Codex evidence.""" + with tempfile.TemporaryDirectory(prefix="c09 fixture ") as tmp: + rt = Path(tmp) + schemas = base.write_schemas(rt / "schemas") + invoked = [] + + def observed_driver(args, *, cwd, timeout): + self.assertEqual(args[args.index("--sandbox") + 1], "read-only") + self.assertIn("model_auto_compact_token_limit=8192", args) + self.assertNotIn("--ignore-user-config", args) + self.assertFalse(any("trust_level" in arg for arg in args)) + source = Path(base.git(cwd, "rev-parse", "--path-format=absolute", "--git-common-dir")).parent + configured = base.load_json(source / ".codex/hooks.json")["hooks"] + collector = observation.StructuralEvents() + + def hook(name, tag, phase=None): + for group in configured.get(name, []): + if group.get("matcher") and not re.search(group["matcher"], tag): + continue + event = {"hook_event_name": name, "cwd": str(cwd), "source": tag, "trigger": tag} + if phase: + event.update(tool_name="Bash", tool_input={"command": c09.COMMANDS[phase]}) + for handler in group["hooks"]: + result = execute_hook(handler["command"], cwd, event) + self.assertEqual(result.returncode, 0, result.stderr) + payload = json.loads(result.stdout) if result.stdout.strip() else {} + self.assertIsNot(payload.get("continue"), False) + if name == "SessionStart": + self.assertIn("C09_FINITE_RECOVERY", payload["hookSpecificOutput"]["additionalContext"]) + self.assertNotIn("MAP_INSTRUCTIONS", payload["hookSpecificOutput"]["additionalContext"]) + + hook("SessionStart", "startup") + for phase in c09.PHASES: + hook("PreToolUse", "Bash", phase) + command = [sys.executable, "-B", c09.SCRIPT, phase] + result = subprocess.run(command, cwd=cwd, text=True, capture_output=True, timeout=30) + self.assertEqual(result.returncode, 0, result.stderr) + invoked.append(phase) + self.assertNotIn(str(cwd), result.stdout) + if phase == "finish": self.assertLess(len(result.stdout), 2048) + else: self.assertGreater(len(result.stdout), c09.COMPACT_LIMIT * 8) + collector.accept(json.dumps({"type": "item.completed", "item": { + "type": "command_execution", "command": c09.COMMANDS[phase], + "exit_code": result.returncode, "aggregated_output": result.stdout, + }}).encode()) + if phase != "finish": + hook("PreCompact", "auto") + hook("PostCompact", "auto") + hook("SessionStart", "compact") + collector.accept(b'{"type":"turn.completed"}') + events = collector.summary() + events.update(process_cleanup_ok=True, process_returncode=0, timeout=False) + base.json_dump(Path(args[args.index("-o") + 1]), completed_payload()) + return observation.ProcessResult(0, False, events) + + # An empty test home contains no credentials. The live path persists + # trust and restores config exactly; only the model process is replaced. + with mock.patch.dict(os.environ, {"CODEX_HOME": str(rt / "home")}), mock.patch.object( + observation, "run_observed", side_effect=observed_driver + ), mock.patch.object(v4, "_write_result", side_effect=lambda **kw: (kw["result"], True)) as writer: + result, _ = v7.run_c09(root=ROOT, runtime_root=rt, schemas=schemas, version="codex-cli 0.153.4", + os_name="offline", source_commit="b" * 40, date="2026-09-06") + self.assertEqual(result, "REPRODUCED", writer.call_args.kwargs) + self.assertEqual(invoked, list(c09.PHASES)) + self.assertTrue(all(writer.call_args.kwargs["trials"][0]["protocol_checks"].values())) + self.assertFalse((rt / "home/config.toml").exists()) + # Locate the active planning root from the copied, unchanged helper. + helpers = list((rt / "C09").rglob("c09_probe.py")) + for helper in helpers: + candidate = helper.parents[1] + states = list((candidate / ".pursue/runs").glob("*/state.json")) + if states: + state = json.loads(states[0].read_text()) + state["next_action"]["type"] = "MAP_INSTRUCTIONS" + states[0].write_text(json.dumps(state)) + proc = subprocess.run([sys.executable, "-B", str(helper), "finish"], cwd=candidate, + capture_output=True, text=True, timeout=30) + self.assertNotEqual(proc.returncode, 0) + break + else: + self.fail("Planning fixture was not available for negative checkpoint test") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_qualification_c09_cli.py b/tests/test_qualification_c09_cli.py new file mode 100644 index 0000000..5d80377 --- /dev/null +++ b/tests/test_qualification_c09_cli.py @@ -0,0 +1,129 @@ +"""Real pinned Codex CLI against a loopback Responses simulator, no model service. + +Opt-in hosted CI conformance check. It is not live C09 qualification evidence: +the simulator chooses tool calls and token usage. Codex itself must perform the +tools, compaction, product hooks, context injection and structured completion. +""" +from __future__ import annotations + +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import threading +import unittest +from unittest import mock + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tools")) +import live_codex_qualification_harness_v7 as v7 +import qualification_c09 as c09 +import qualification_process as process +from test_qualification_c09 import completed_payload + + +@unittest.skipUnless(os.environ.get("PLANANVIL_TEST_CODEX_BIN"), "pinned CLI conformance job only") +class C09RealCLIConformance(unittest.TestCase): + def test_actual_cli_finishes_two_compactions_and_recovery_in_one_turn(self): + binary = os.environ["PLANANVIL_TEST_CODEX_BIN"] + version = subprocess.check_output([binary, "--version"], text=True).strip() + self.assertEqual(version, "codex-cli 0.153.4") + state = {"regular": 0, "compact": 0, "context_seen": [], "requests": 0} + + class Server(BaseHTTPRequestHandler): + def log_message(self, *args): + pass + + def do_POST(self): + data = self.rfile.read(int(self.headers.get("Content-Length", 0))) + try: + body = json.loads(data) + except ValueError: + self.send_error(400, "Expected uncompressed fixture JSON") + return + state["requests"] += 1 + if state["requests"] > 10: + self.send_error(400, "Finite fixture request limit exceeded") + return + compact = not body.get("tools") + if compact: + state["compact"] += 1 + item = {"type": "message", "role": "assistant", "id": "summary-" + str(state["compact"]), + "content": [{"type": "output_text", "text": "Completed earlier C09 phase. Continue with next phase, do not repeat."}]} + output_tokens = 32 + else: + index = state["regular"] + state["regular"] += 1 + state["context_seen"].append("C09_FINITE_RECOVERY" in json.dumps(body.get("input"))) + if index < 3: + phase = c09.PHASES[index] + item = {"type": "function_call", "id": "fc-" + phase, "call_id": "call-" + phase, + "name": "exec_command", "arguments": json.dumps({"cmd": c09.COMMANDS[phase], + "max_output_tokens": c09.OUTPUT_TOKENS if index < 2 else 2048})} + output_tokens = 20000 if index < 2 else 8 + else: + item = {"type": "message", "role": "assistant", "id": "final", + "content": [{"type": "output_text", "text": json.dumps(completed_payload())}]} + output_tokens = 64 + rid = "fixture-" + str(state["requests"]) + events = [ + {"type": "response.created", "response": {"id": rid}}, + {"type": "response.output_item.done", "output_index": 0, "item": item}, + {"type": "response.completed", "response": {"id": rid, + "usage": {"input_tokens": 1000, "output_tokens": output_tokens, + "total_tokens": 1000 + output_tokens}}}, + ] + response = "".join("data: " + json.dumps(e) + "\n\n" for e in events).encode() + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Content-Length", str(len(response))) + self.end_headers() + self.wfile.write(response) + + with ThreadingHTTPServer(("127.0.0.1", 0), Server) as server, tempfile.TemporaryDirectory(prefix="c09 cli ") as tmp: + worker = threading.Thread(target=server.serve_forever, daemon=True) + worker.start() + rt = Path(tmp) + base = v7.base + original_args = base.common_codex_args + actual_observed = process.run_observed + + def bounded_observed(args, **kwargs): + # The offline peer is instant. Bound a broken conformance test; + # the live C09 timeout remains 900 seconds. + kwargs["timeout"] = 120 + return actual_observed(args, **kwargs) + + def local_args(**kwargs): + args = original_args(**kwargs) + args[0] = binary + args += ["-c", 'model_provider="fixture"', "-c", 'model_providers.fixture.name="Fixture"', + "-c", f'model_providers.fixture.base_url="http://127.0.0.1:{server.server_port}/v1"', + "-c", 'model_providers.fixture.wire_api="responses"', + "-c", 'model_providers.fixture.requires_openai_auth=false', + "-c", 'model_providers.fixture.stream_max_retries=0', + "-c", 'model_providers.fixture.request_max_retries=0'] + return args + + try: + with mock.patch.dict(os.environ, {"CODEX_HOME": str(rt / "home")}), mock.patch.object( + base, "common_codex_args", side_effect=local_args + ), mock.patch.object(process, "run_observed", side_effect=bounded_observed), mock.patch.object(v7.compat.v4, "_write_result", side_effect=lambda **kw: (kw["result"], True)) as writer: + result, _ = v7.run_c09(root=ROOT, runtime_root=rt, schemas=base.write_schemas(rt / "schemas"), + version=version, os_name="offline-cli", source_commit="b" * 40, date="2026-09-06") + details = writer.call_args.kwargs + self.assertEqual(result, "REPRODUCED", details) + self.assertEqual(state["compact"], 2, state) + self.assertEqual(state["regular"], 4, state) + self.assertTrue(all(state["context_seen"]), state) + print("CODEX_01534_OFFLINE_CONFORMANCE_OK: 3 real tools, 2 compactions, 2 compact recovery contexts, 1 completed turn") + finally: + server.shutdown() + worker.join(timeout=5) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_qualification_execution_boundaries.py b/tests/test_qualification_execution_boundaries.py index ba94760..59094e4 100644 --- a/tests/test_qualification_execution_boundaries.py +++ b/tests/test_qualification_execution_boundaries.py @@ -78,19 +78,18 @@ def probe(**_kwargs): class CompactionCompletionTests(unittest.TestCase): def evaluate(self, payload, events, error, records=None): + from test_qualification_c09 import valid_records with tempfile.TemporaryDirectory() as tmp, ExitStack() as stack: root = Path(tmp) if records is None: - records = [ - {"event": "PreCompact"}, {"event": "PostCompact"}, - {"event": "PreCompact"}, {"event": "PostCompact"}, - {"event": "PreToolUse"}, - ] + records = valid_records() stack.enter_context(mock.patch.object(v4, "_runtime_paths", return_value=(root,) * 7)) stack.enter_context(mock.patch.object(base, "ensure_git_repo")) stack.enter_context(mock.patch.object(base, "git", return_value="a" * 40)) stack.enter_context(mock.patch.object(base, "git_snapshot", return_value={"head": "a" * 40})) stack.enter_context(mock.patch.object(v4, "_start_active_run", return_value=(root, ".pursue/runs/test"))) + stack.enter_context(mock.patch.object(v4.c09, "seed_state")) + stack.enter_context(mock.patch.object(v4, "_create_checkpoint")) stack.enter_context(mock.patch.object(v4, "_checkpoint_validation", return_value={"ok": True})) stack.enter_context(mock.patch.object(v4, "_clear_hook_log")) stack.enter_context(mock.patch.object(v4, "_read_hook_records", return_value=records)) @@ -111,17 +110,20 @@ def test_missing_completion_payload_cannot_pass(self) -> None: self.assertEqual(result["result"], "BLOCKED") def test_completed_positive_trial_still_passes(self) -> None: - result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None) + from test_qualification_c09 import completed_payload, valid_events + result = self.evaluate(completed_payload(), valid_events(), None) self.assertEqual(result["result"], "REPRODUCED") self.assertTrue(result["expected_met"]) def test_second_compaction_still_requires_subsequent_tool_use(self) -> None: - result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None, + from test_qualification_c09 import completed_payload, valid_events + result = self.evaluate(completed_payload(), valid_events(), None, [{"event": "PreCompact"}, {"event": "PostCompact"}] * 2) self.assertNotEqual(result["result"], "REPRODUCED") def test_observed_stop_with_valid_checkpoint_still_fails(self) -> None: - result = self.evaluate({"capability_id": "C09", "outcome": "PASS"}, {}, None, + from test_qualification_c09 import completed_payload, valid_events + result = self.evaluate(completed_payload(), valid_events(), None, [{"event": "PreCompact", "continue": False}, {"event": "PostCompact"}] * 2 + [{"event": "PreToolUse"}]) self.assertEqual(result["result"], "FAILED") diff --git a/tools/install_ci_codex.py b/tools/install_ci_codex.py new file mode 100644 index 0000000..d4a1b37 --- /dev/null +++ b/tools/install_ci_codex.py @@ -0,0 +1,53 @@ +"""Install an official, hash-verified pinned CLI into a disposable hosted CI dir. + +Only the explicit conformance CI job calls this networked helper. Local product +and harness unit tests do not need Codex, credentials, network, or installation. +""" +from __future__ import annotations + +import hashlib +import io +import json +from pathlib import Path +import sys +import tarfile +from urllib.request import Request, urlopen + +TAG = "rust-v0.153.4" +ASSETS = ("codex-x86_64-unknown-linux-musl.tar.gz", "bwrap-x86_64-unknown-linux-musl.tar.gz") + + +def fetch(url): + with urlopen(Request(url, headers={"User-Agent": "PlanAnvil-CI-conformance"}), timeout=120) as response: + return response.read() + + +def main(): + target = Path(sys.argv[1]).resolve() + target.mkdir(parents=True, exist_ok=True) + release = json.loads(fetch("https://api.github.com/repos/openai/codex/releases/tags/" + TAG)) + if release["tag_name"] != TAG or release["draft"] or release["prerelease"]: + raise ValueError("Unexpected Codex release metadata") + for name in ASSETS: + asset = next(a for a in release["assets"] if a["name"] == name) + url = "https://github.com/openai/codex/releases/download/" + TAG + "/" + name + if asset["browser_download_url"] != url: + raise ValueError("Unexpected binary origin") + data = fetch(url) + if asset["digest"] != "sha256:" + hashlib.sha256(data).hexdigest(): + raise ValueError("Codex release asset digest mismatch") + with tarfile.open(fileobj=io.BytesIO(data)) as archive: + members = [m for m in archive.getmembers() if m.isfile()] + if len(members) != 1: + raise ValueError("Expected exactly one executable") + stream = archive.extractfile(members[0]) + if stream is None: + raise ValueError("Missing release executable") + output = target / ("codex" if name.startswith("codex-") else "bwrap") + output.write_bytes(stream.read()) + output.chmod(0o755) + print(name + " " + asset["digest"]) + + +if __name__ == "__main__": + main() diff --git a/tools/live_codex_qualification_harness_v4.py b/tools/live_codex_qualification_harness_v4.py index 303651c..1fc55d5 100644 --- a/tools/live_codex_qualification_harness_v4.py +++ b/tools/live_codex_qualification_harness_v4.py @@ -9,6 +9,8 @@ from pathlib import Path from typing import Any, Callable +import qualification_c09 as c09 + import live_codex_qualification_harness as v1 import live_codex_qualification_harness_v2 as v2 import live_codex_qualification_harness_v3 as prior @@ -19,7 +21,7 @@ _ORIGINAL_CAPABILITY_RUNTIME = prior.capability_runtime C08_COMPACT_LIMIT = 200 -C09_COMPACT_LIMIT = 1000 +C09_COMPACT_LIMIT = c09.COMPACT_LIMIT COMPACT_SCOPE = "body_after_prefix" HOOK_LOG_RELATIVE = ".pursue/qualification-hook-events.jsonl" @@ -833,20 +835,23 @@ def _c09_runtime( worktrees=worktrees, version=version, compact_limit=C09_COMPACT_LIMIT, - create_checkpoint=True, - segments=4, - segment_bytes=32768, + create_checkpoint=False, + segments=0, + segment_bytes=0, + prepare_repo=c09.prepare_repo, + hook_proxy_source=c09.proxy_source(_hook_proxy_source()), ) + c09.seed_state(planning, _run_root) + _create_checkpoint(planning=planning, run_root=_run_root) fixture_commit = base.git(repo, "rev-parse", "HEAD") checkpoint_before = _checkpoint_validation(planning) _clear_hook_log(planning) + source_before = base.git_snapshot(repo) + files_before = (c09.file_fingerprint(repo), c09.file_fingerprint(planning)) before = base.git_snapshot(planning) payload, events, error = _run_codex_probe( cwd=planning, - prompt=_compact_probe_prompt( - capability_id, - ["segment-01.txt", "segment-02.txt", "segment-03.txt", "segment-04.txt"], - ), + prompt=c09.prompt(), schemas=schemas, results_dir=results_dir, position=1, @@ -857,12 +862,18 @@ def _c09_runtime( observe_process=True, ) after = base.git_snapshot(planning) + source_after = base.git_snapshot(repo) + files_after = (c09.file_fingerprint(repo), c09.file_fingerprint(planning)) records = _read_hook_records(planning) pre = _event_records(records, "PreCompact") post = _event_records(records, "PostCompact") stops = [item for item in pre if item.get("continue") is False] checkpoint_after = _checkpoint_validation(planning) + checks = c09.protocol_checks(events, records) + checks["source_and_planning_unchanged"] = (source_before == source_after and before == after + and files_before == files_after) + protocol_ok = all(checks.values()) two_compactions = len(pre) >= 2 and len(post) >= 2 continued_after_second = _continued_after_second_postcompact(records) checkpoint_coherent = bool(checkpoint_before.get("ok")) and bool(checkpoint_after.get("ok")) @@ -872,6 +883,8 @@ def _c09_runtime( and not events.get("timeout") and payload.get("capability_id") == capability_id and payload.get("outcome") == "PASS" + and payload.get("trial") == c09.TRIAL + and "C09_FINISHED" in payload.get("observations", []) ) completion_blocker = error or ( None if invocation_completed else "C09 did not return a completed positive structured result." @@ -883,9 +896,10 @@ def _c09_runtime( "trial_name": "checkpoint_auto_compact_recover_recompact", "outcome": ( "BLOCKED" - if not invocation_completed or not two_compactions + if not invocation_completed or not two_compactions or not protocol_ok else ("PASS" if checkpoint_coherent and no_stop_loop else "FAIL") ), + "protocol_checks": checks, "assertions": [ { "name": "codex_invocation_completed_without_timeout", @@ -925,10 +939,10 @@ def _c09_runtime( f"invocation_error={error or 'none'}", f"invocation_completed={str(invocation_completed).lower()}", ], - "blocker": completion_blocker, + "blocker": completion_blocker or (None if protocol_ok else "C09 finite protocol checks incomplete"), "event_summary": events, "hook_timeline": [ - {key: item[key] for key in ("event", "returncode", "continue", "additional_context") + {key: item[key] for key in ("event", "returncode", "continue", "additional_context", "source", "trigger", "c09_phase") if key in item} for item in records[-128:] if item.get("event") in {"PreToolUse", "SessionStart", "PreCompact", "PostCompact"} @@ -945,6 +959,9 @@ def _c09_runtime( "runtime_cli_override": True, "project_trust_method": "persisted_user_config", "process_observation": "bounded_structural_jsonl", + "finite_phases": list(c09.PHASES), + "tool_max_output_tokens": c09.OUTPUT_TOKENS, + "canonical_action": "C09_FINITE_RECOVERY", }, } @@ -953,6 +970,10 @@ def _c09_runtime( expected_met = False blocker = "The deterministic C09 fixture did not begin with a valid checkpoint." summary = "C09 blocked during deterministic fixture preparation." + elif not checks["source_and_planning_unchanged"]: + result, expected_met = "FAILED", False + blocker = "C09 changed source or planning repository state." + summary = "C09 failed repository immutability." elif not invocation_completed: result = "BLOCKED" expected_met = False @@ -978,6 +999,10 @@ def _c09_runtime( expected_met = False blocker = "Two compactions completed, but no subsequent real tool call demonstrated continuation after the second compaction." summary = "C09 blocked because freedom from a permanent stop loop was not fully exercised." + elif not protocol_ok: + result, expected_met = "BLOCKED", False + blocker = "C09 finite protocol was not verified: " + ", ".join(k for k, ok in checks.items() if not ok) + summary = "C09 lacks a completed ordered two-cycle recovery proof." else: result = "REPRODUCED" expected_met = True diff --git a/tools/live_codex_qualification_regression.py b/tools/live_codex_qualification_regression.py index 36e860e..d9c77e7 100644 --- a/tools/live_codex_qualification_regression.py +++ b/tools/live_codex_qualification_regression.py @@ -170,7 +170,7 @@ def run_probe(**kwargs: Any): v4._run_codex_probe = run_probe v4._compact_probe_prompt = _compact_prompt v4.C08_COMPACT_LIMIT = 40 - v4.C09_COMPACT_LIMIT = 200 + # C09 owns its finite workload and threshold; do not override it here. try: yield finally: diff --git a/tools/prepare_capabilities.py b/tools/prepare_capabilities.py index 7a5cea2..b851fb0 100644 --- a/tools/prepare_capabilities.py +++ b/tools/prepare_capabilities.py @@ -9,6 +9,8 @@ import tarfile from pathlib import Path, PurePosixPath +import qualification_c09 as c09 + PART_GLOB = 'templates.part*' C06_CODEX0152_OVERLAY = { @@ -131,21 +133,28 @@ } C09_COMPLETION_OVERLAY = { - 'fixture/README.md': '''# C09 fixture and completion requirements + 'fixture/README.md': """# C09 finite recovery fixture -The outer harness installs the actual product, creates the planning worktree and a valid checkpoint, and then exercises genuine automatic compaction. +The outer harness installs the unmodified product, prepares this fixture BEFORE bootstrap, replaces the bootstrap MAP_INSTRUCTIONS action with the hashed C09_FINITE_RECOVERY scenario, then creates and validates a real product checkpoint. -Two real compaction cycles, coherent checkpoint/Git state, and subsequent real tool use remain required. They are not sufficient when Codex times out or fails to return a completed positive structured C09 result. Partial event counts cannot turn an incomplete invocation into REPRODUCED. +One Codex turn executes exactly three read-only commands: first, second, finish. Every command uses the actual installed checkpoint/Git validator and reads canonical recovery inputs. Only first and second emit bounded inert compaction stimuli. Finish emits a small receipt and ends the workload. No manual hook calls, synthetic live events, canonical writes during the turn, permission changes or repeated initial reads are permitted. -The deliberately low fixture threshold is not a product default. This correction does not silently retune it or weaken C08's intentional negative stop trial. Record a remaining timeout as BLOCKED. Retain bounded content-free event and hook sequences, command labels/counts, error categories, and process completion/owned-tree cleanup status. Never persist raw stdout/stderr or use partial progress as proof of successful completion. -''', - 'run-command.txt': '''# Existing controlled workflow: main -> recovery for C09/C10/C13. -# The recovery driver selects the same v7 capability runtime used by full. +The trigger is 8192 body-after-prefix tokens; requested tool output budget is 65536 tokens for each stimulus. These are disposable fixture settings, not product defaults. The active compatibility layer must not replace this trigger with the old 200-token value. TokenBudget is disabled only in the existing isolated qualification configuration. + +Require two ordered automatic PreCompact -> PostCompact -> SessionStart(source=compact) cycles, three successful canonical/Git reconciliations, real tool use after the second cycle, exact command order, valid checkpoints, unchanged source/planning files and Git, and one completed positive C09 turn. Timeout (still 900 seconds), extra/failed/repeated tools, missing recovery, failed hooks, reader/cleanup failures or incomplete evidence cannot be REPRODUCED. No raw output or private canonical state is persisted. + +The fixed scenario resolves proven conflicting fixture instructions and removes an over-aggressive trigger. Historical #23 labels do not identify its exact commands, so the historical timeout's complete causal chain is not claimed as observed. Live success must still be established by the actual model-backed run. +""", + 'fixture/c09_probe.py': c09.PROBE_SOURCE, + 'prompt.txt': c09.prompt(), + 'run-command.txt': """# Existing controlled workflow: main -> recovery for C09/C10/C13, or full for C01-C16. +# Both select the identical C09 runtime; only full can satisfy the release gate. python3 tools/live_codex_qualification_recovery.py --root --source-commit --run-id --output --allow-c13-non-ephemeral-fallback -# A targeted pass is not a full C01-C16 release pass. -''', +# A loopback CLI conformance pass is offline and cannot count as live capability evidence. +""", } + C10_ISOLATION_OVERLAY = { 'README.md': """# C10 — Recovery context through SessionStart diff --git a/tools/qualification_c09.py b/tools/qualification_c09.py new file mode 100644 index 0000000..1d94a4e --- /dev/null +++ b/tools/qualification_c09.py @@ -0,0 +1,205 @@ +"""Finite C09 workload: real read-only recovery, two large outputs, then finish. + +This module prepares the disposable fixture, not product behavior. It never +invokes lifecycle hooks or supplies fabricated live events to the evaluator. +""" +from __future__ import annotations + +import hashlib +import json +import os +from pathlib import Path +from typing import Any + +COMPACT_LIMIT = 8192 +OUTPUT_TOKENS = 65536 +PHASES = ("first", "second", "finish") +SCRIPT = "qualification-payload/c09_probe.py" +TRIAL = "checkpoint_auto_compact_recover_recompact" +COMMANDS = {phase: f"python3 -B {SCRIPT} {phase}" for phase in PHASES} + +# Shipped only in a disposable qualification repository. All runtime operations +# are reads. Checkpoint/Git decisions come from the installed product validator. +PROBE_SOURCE = r'''from __future__ import annotations +import hashlib, json, sys +from pathlib import Path + +root = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(root / ".codex" / "hooks")) +from plan_anvil_hooklib import active_run_for_event +from plan_anvil_checkpoint import validate_checkpoint_for_run + +def main(): + if len(sys.argv) != 2 or sys.argv[1] not in {"first", "second", "finish"}: + return 2 + phase = sys.argv[1] + active = active_run_for_event({"cwd": str(root)}) + if active is None: + return 2 + check = validate_checkpoint_for_run(active) + if not check.ok or check.path is None: + print(json.dumps({"c09_phase": phase, "checkpoint_ok": False})) + return 2 + files = [active.run_root / "manifest.json", active.run_root / "state.json", + active.run_root / "local-state.json", check.path] + files += sorted((root / ".pursue").glob("SYSTEM_PROFILE*.md")) + # Fully read the canonical recovery inputs without printing local-state or + # paths. The real validator above reconciles canonical identity with Git. + sizes = [len(path.read_bytes()) for path in files] + state = json.loads((active.run_root / "state.json").read_text(encoding="utf-8")) + if state.get("next_action") != {"type": "C09_FINITE_RECOVERY", "target": "evidence/c09-scenario.json"}: + return 2 + scenario = json.loads((active.run_root / "evidence/c09-scenario.json").read_text(encoding="utf-8")) + if scenario.get("phases") != ["first", "second", "finish"]: + return 2 + receipt = {"c09_phase": phase, "checkpoint_ok": True, "canonical_read": True, + "git_reconciled": True, "canonical_file_count": len(files), + "canonical_bytes": sum(sizes)} + print(json.dumps(receipt, sort_keys=True)) + if phase != "finish": + # Deterministic inert high-volume output, not an instruction or a secret. + # A phase-specific stimulus avoids eight indistinguishable repeated reads. + for i in range(1024): + print(hashlib.sha512((phase + ":" + str(i)).encode()).hexdigest()) + print(json.dumps(receipt, sort_keys=True)) + print("C09_NEXT=" + {"first": "second", "second": "finish", "finish": "RETURN_RESULT"}[phase]) + return 0 + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except (OSError, ValueError, KeyError, TypeError): + # No raw canonical contents/private paths in errors or evidence. + print("C09_FIXTURE_READ_FAILED", file=sys.stderr) + raise SystemExit(2) +''' + + +def prepare_repo(repo: Path) -> None: + path = repo / SCRIPT + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(PROBE_SOURCE, encoding="utf-8") + + +def seed_state(planning: Path, run_root: str) -> None: + """Replace bootstrap's unrelated MAP_INSTRUCTIONS action before checkpointing.""" + run = planning / run_root + scenario = {"schema_version": "1.0", "phases": list(PHASES), "commands": COMMANDS, + "completion": "Return the final C09 result after finish; never restart first."} + data = (json.dumps(scenario, indent=2, sort_keys=True) + "\n").encode() + path = run / "evidence/c09-scenario.json" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(data) + state_path = run / "state.json" + state = json.loads(state_path.read_text(encoding="utf-8")) + state["revision"] += 1 + state["next_action"] = {"type": "C09_FINITE_RECOVERY", "target": "evidence/c09-scenario.json"} + state.setdefault("artifact_hashes", {})["evidence/c09-scenario.json"] = "sha256:" + hashlib.sha256(data).hexdigest() + state_path.write_text(json.dumps(state, indent=2, sort_keys=True) + "\n", encoding="utf-8") + + +def prompt() -> str: + return f'''Qualify C09 using this finite, read-only scenario in ONE Codex turn. +The outer harness has already installed PlanAnvil and created a valid checkpoint. +Do not start a plan, map instructions, or create/repair any files. The canonical +next action is C09_FINITE_RECOVERY, not the earlier bootstrap MAP_INSTRUCTIONS. + +Make exactly THREE sequential exec_command calls, each once and in this order: +1. cmd={COMMANDS["first"]!r}, max_output_tokens={OUTPUT_TOKENS} +2. cmd={COMMANDS["second"]!r}, max_output_tokens={OUTPUT_TOKENS} +3. cmd={COMMANDS["finish"]!r}, max_output_tokens=2048 +Use workdir for the current planning directory; do not prepend cd or combine calls. +Wait for each command to finish before choosing the next one. The first two outputs +are inert compaction stimuli: do not analyse, quote, hash, or reread their padding. + +After each automatic compaction continue with the NEXT phase, never restart first. +Each command fully reads manifest/state/local-state/checkpoint/profiles and calls the +actual product checkpoint/Git validator. This performs the recovery requested by the +SessionStart context; do not perform additional recovery reads outside those commands. +No manual hook calls, telemetry reads, other tools, state changes, or subagents. + +After finish exits 0, make NO further calls. Return capability_id C09, trial {TRIAL}, +outcome PASS, and observation C09_FINISHED. If a command fails, stop with FAIL/BLOCKED; +do not retry. The outer evaluator, not your self-report, checks the real compactions. +''' + + +def proxy_source(original: str) -> str: + """Extend only C09 telemetry without changing stdout or any product decision.""" + addition = '''\nif isinstance(event, dict): + source = event.get("source") + if source in {"startup", "resume", "clear", "compact"}: + record["source"] = source + trigger = event.get("trigger") + if trigger in {"auto", "manual"}: + record["trigger"] = trigger + raw_command = event.get("tool_input", {}).get("command") if isinstance(event.get("tool_input"), dict) else None + commands = ''' + repr({command: phase for phase, command in COMMANDS.items()}) + ''' + if event_name == "PreToolUse": + record["c09_phase"] = commands.get(raw_command, "other") +''' + anchor = 'try:\n parsed = json.loads(completed.stdout)' + # The active compat adapter has this exact executable contract, covered by + # subprocess tests. Reject a future adapter change instead of silently losing evidence. + if anchor not in original: + raise ValueError("C09 telemetry adapter contract changed") + return original.replace(anchor, addition + "\n" + anchor, 1) + + +def file_fingerprint(repo: Path) -> str: + """Hash all fixture files, including ignored canonical state, without paths in evidence. + + Git status alone misses edits to an already-untracked state file. Exclude only + Git's database/locator and the known legacy telemetry sink (normally external). + """ + if not repo.is_dir(): + raise ValueError("C09 fixture directory is missing") + digest = hashlib.sha256() + def walk_error(_error: OSError) -> None: + raise ValueError("C09 fixture files could not be completely inspected") + + for directory, dirs, names in os.walk(repo, onerror=walk_error): + for name in dirs: + if name != ".git" and (Path(directory) / name).is_symlink(): + raise ValueError("Unexpected directory symlink in C09 fixture") + dirs[:] = sorted(d for d in dirs if d != ".git") + for name in sorted(names): + path = Path(directory) / name + rel = path.relative_to(repo).as_posix() + if rel in {".git", ".pursue/qualification-hook-events.jsonl"}: + continue + if path.is_symlink(): + raise ValueError("Unexpected symlink in C09 fixture") + data = path.read_bytes() + digest.update(rel.encode("utf-8") + b"\0") + digest.update(hashlib.sha256(data).digest()) + return digest.hexdigest() + + +def protocol_checks(events: dict[str, Any], records: list[dict[str, Any]]) -> dict[str, bool]: + commands = [r for r in events.get("event_tail", []) + if r.get("event") == "item.completed" and r.get("item") == "command_execution"] + expected = ["c09_" + phase for phase in PHASES] + command_order = ([r.get("command") for r in commands] == expected + and all(r.get("exit_code") == 0 and r.get("c09_receipt_ok") is True for r in commands) + and events.get("completed_command_items") == 3 + and not events.get("event_tail_truncated")) + tool_rows = [r for r in records if r.get("event") == "PreToolUse"] + hook_order = [r.get("c09_phase") for r in tool_rows] == list(PHASES) + positions = {r.get("c09_phase"): i for i, r in enumerate(records) if r.get("event") == "PreToolUse"} + cycles = hook_order + for start, end in (("first", "second"), ("second", "finish")): + between = records[positions[start] + 1:positions[end]] if hook_order else [] + lifecycle = [r.get("event") for r in between if r.get("event") in {"PreCompact", "PostCompact", "SessionStart"}] + cycles = cycles and lifecycle == ["PreCompact", "PostCompact", "SessionStart"] + cycles = cycles and all(r.get("trigger") == "auto" for r in between if r.get("event") in {"PreCompact", "PostCompact"}) + cycles = cycles and any(r.get("event") == "SessionStart" and r.get("source") == "compact" + and r.get("additional_context") is True for r in between) + hooks_ok = bool(records) and all(r.get("returncode") == 0 for r in records) + forbidden = sum(events.get("item_types", {}).get(k, 0) for k in + ("file_change", "mcp_tool_call", "web_search", "collab_tool_call", "other")) + return {"three_completed_reconciliations": command_order, + "ordered_automatic_recovery_cycles": bool(cycles), "product_hooks_succeeded": hooks_ok, + "no_unexpected_tools": not forbidden, + "turn_completed": events.get("event_types", {}).get("turn.completed") == 1 + and not events.get("event_types", {}).get("turn.failed")} diff --git a/tools/qualification_process.py b/tools/qualification_process.py index 3a53fff..4857b2f 100644 --- a/tools/qualification_process.py +++ b/tools/qualification_process.py @@ -29,6 +29,7 @@ }) STATUSES = frozenset({"in_progress", "completed", "failed", "declined", "cancelled"}) COMMANDS = { + **{f"python3 -B qualification-payload/c09_probe.py {p}": f"c09_{p}" for p in ("first", "second", "finish")}, **{f"cat qualification-payload/segment-{i:02d}.txt": f"segment_{i:02d}" for i in range(1, 5)}, "git status --porcelain=v1 --untracked-files=all": "git_status", "git rev-parse HEAD": "git_head", @@ -118,6 +119,20 @@ def accept(self, raw: bytes, *, stderr: bool = False) -> None: if item_kind == "command_execution": label = command_label(item.get("command")) row["command"] = label + if label in {"c09_first", "c09_second", "c09_finish"} and kind == "item.completed": + # Fixed scalar receipt only; never persist output or paths. + raw_output = item.get("aggregated_output", "") + receipt = {} + if isinstance(raw_output, str): + for line in raw_output.splitlines()[-4:]: + try: + candidate = json.loads(line) + except (ValueError, RecursionError): + continue + if isinstance(candidate, dict): + receipt = candidate + row["c09_receipt_ok"] = (receipt.get("c09_phase") == label[4:] + and all(receipt.get(k) is True for k in ("checkpoint_ok", "canonical_read", "git_reconciled"))) if kind == "item.completed": self.completed_commands += 1 self.commands[label] += 1 From 8949207291e5f0cbf6ad5cfbdcfbedfc67011a6e Mon Sep 17 00:00:00 2001 From: KeyffMS <124252104+KeyffMS@users.noreply.github.com> Date: Sun, 6 Sep 2026 21:00:37 +0200 Subject: [PATCH 4/8] Identify real Codex compaction from canonical request metadata rather than optional tools --- tests/test_qualification_c09_cli.py | 49 +++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/tests/test_qualification_c09_cli.py b/tests/test_qualification_c09_cli.py index 5d80377..64cddc5 100644 --- a/tests/test_qualification_c09_cli.py +++ b/tests/test_qualification_c09_cli.py @@ -25,13 +25,47 @@ from test_qualification_c09 import completed_payload +def request_kind(body): + """Use pinned Codex request metadata, never the optional tools field. + + Source: rust-v0.153.4/core/src/responses_metadata.rs. Responses Lite may + omit tools on ordinary turns; absence of tools is not a compaction signal. + """ + metadata = body.get("client_metadata") or {} + encoded = metadata.get("x-codex-turn-metadata") + canonical = json.loads(encoded) if isinstance(encoded, str) else {} + kind = canonical.get("request_kind", metadata.get("request_kind")) + if kind not in {"turn", "compaction"}: + raise ValueError("Missing or unexpected Codex request kind") + if metadata.get("request_kind", kind) != kind: + raise ValueError("Conflicting Codex request kind metadata") + return kind + + +class RequestKindTests(unittest.TestCase): + def test_turn_without_tools_is_not_compaction(self): + body = {"client_metadata": {"x-codex-turn-metadata": json.dumps({"request_kind": "turn"})}} + self.assertEqual(request_kind(body), "turn") + body["tools"] = [] + self.assertEqual(request_kind(body), "turn") + + def test_compaction_is_identified_from_canonical_metadata(self): + self.assertEqual(request_kind({"client_metadata": {"request_kind": "compaction"}}), "compaction") + + def test_missing_or_conflicting_metadata_is_rejected(self): + for body in ({"tools": []}, {"client_metadata": {"request_kind": "turn", + "x-codex-turn-metadata": json.dumps({"request_kind": "compaction"})}}): + with self.assertRaises(ValueError): + request_kind(body) + + @unittest.skipUnless(os.environ.get("PLANANVIL_TEST_CODEX_BIN"), "pinned CLI conformance job only") class C09RealCLIConformance(unittest.TestCase): def test_actual_cli_finishes_two_compactions_and_recovery_in_one_turn(self): binary = os.environ["PLANANVIL_TEST_CODEX_BIN"] version = subprocess.check_output([binary, "--version"], text=True).strip() self.assertEqual(version, "codex-cli 0.153.4") - state = {"regular": 0, "compact": 0, "context_seen": [], "requests": 0} + state = {"regular": 0, "compact": 0, "context_seen": [], "requests": 0, "request_kinds": []} class Server(BaseHTTPRequestHandler): def log_message(self, *args): @@ -48,7 +82,14 @@ def do_POST(self): if state["requests"] > 10: self.send_error(400, "Finite fixture request limit exceeded") return - compact = not body.get("tools") + try: + kind = request_kind(body) + except ValueError: + state["metadata_error"] = True + self.send_error(400, "Missing or conflicting Codex request metadata") + return + state["request_kinds"].append(kind) + compact = kind == "compaction" if compact: state["compact"] += 1 item = {"type": "message", "role": "assistant", "id": "summary-" + str(state["compact"]), @@ -115,9 +156,11 @@ def local_args(**kwargs): result, _ = v7.run_c09(root=ROOT, runtime_root=rt, schemas=base.write_schemas(rt / "schemas"), version=version, os_name="offline-cli", source_commit="b" * 40, date="2026-09-06") details = writer.call_args.kwargs - self.assertEqual(result, "REPRODUCED", details) + self.assertEqual(result, "REPRODUCED", {"peer": state, "evaluation": details}) self.assertEqual(state["compact"], 2, state) self.assertEqual(state["regular"], 4, state) + self.assertEqual(state["request_kinds"], + ["turn", "compaction", "turn", "compaction", "turn", "turn"]) self.assertTrue(all(state["context_seen"]), state) print("CODEX_01534_OFFLINE_CONFORMANCE_OK: 3 real tools, 2 compactions, 2 compact recovery contexts, 1 completed turn") finally: From a5c6872a88f9662a26f770c579adf153ac318b71 Mon Sep 17 00:00:00 2001 From: KeyffMS <124252104+KeyffMS@users.noreply.github.com> Date: Sun, 6 Sep 2026 21:04:16 +0200 Subject: [PATCH 5/8] Expose bounded loopback tool failures and distinguish real recovery context from the prompt --- tests/test_qualification_c09_cli.py | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/tests/test_qualification_c09_cli.py b/tests/test_qualification_c09_cli.py index 64cddc5..e735951 100644 --- a/tests/test_qualification_c09_cli.py +++ b/tests/test_qualification_c09_cli.py @@ -65,7 +65,7 @@ def test_actual_cli_finishes_two_compactions_and_recovery_in_one_turn(self): binary = os.environ["PLANANVIL_TEST_CODEX_BIN"] version = subprocess.check_output([binary, "--version"], text=True).strip() self.assertEqual(version, "codex-cli 0.153.4") - state = {"regular": 0, "compact": 0, "context_seen": [], "requests": 0, "request_kinds": []} + state = {"regular": 0, "compact": 0, "context_seen": [], "requests": 0, "request_kinds": [], "tool_results": {}} class Server(BaseHTTPRequestHandler): def log_message(self, *args): @@ -89,6 +89,20 @@ def do_POST(self): self.send_error(400, "Missing or conflicting Codex request metadata") return state["request_kinds"].append(kind) + for entry in body.get("input", []): + if not isinstance(entry, dict) or entry.get("type") != "function_call_output": + continue + phase = str(entry.get("call_id", "")).removeprefix("call-") + if phase not in c09.PHASES: + continue + output = entry.get("output", "") + text = output if isinstance(output, str) else json.dumps(output) + success = '"git_reconciled": true' in text and '"checkpoint_ok": true' in text + state["tool_results"][phase] = {"receipt_seen": success} + if not success: + # Loopback-only failures; no credentials or model service. + # Do not retain successful canonical output or padding. + state["tool_results"][phase]["diagnostic"] = v7.base.sanitize_text(text[:1200]) compact = kind == "compaction" if compact: state["compact"] += 1 @@ -98,7 +112,9 @@ def do_POST(self): else: index = state["regular"] state["regular"] += 1 - state["context_seen"].append("C09_FINITE_RECOVERY" in json.dumps(body.get("input"))) + context = json.dumps(body.get("input")) + state["context_seen"].append("Recover PlanAnvil from files:" in context + and "C09_FINITE_RECOVERY" in context) if index < 3: phase = c09.PHASES[index] item = {"type": "function_call", "id": "fc-" + phase, "call_id": "call-" + phase, From 2b0064b9b55c89cc218b8fe77d2b38924d9f4923 Mon Sep 17 00:00:00 2001 From: KeyffMS <124252104+KeyffMS@users.noreply.github.com> Date: Sun, 6 Sep 2026 21:08:06 +0200 Subject: [PATCH 6/8] Use distro bubblewrap and verify user/network namespaces before real CLI conformance --- .github/workflows/plananvil-tests.yml | 6 ++++++ tools/install_ci_codex.py | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/.github/workflows/plananvil-tests.yml b/.github/workflows/plananvil-tests.yml index 88d75f7..ae229c3 100644 --- a/.github/workflows/plananvil-tests.yml +++ b/.github/workflows/plananvil-tests.yml @@ -94,6 +94,12 @@ jobs: - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: "3.11" + - name: Install and verify the operating-system sandbox prerequisite + run: | + sudo apt-get update -qq + sudo apt-get install -y bubblewrap + /usr/bin/bwrap --version + /usr/bin/bwrap --unshare-user --unshare-net --ro-bind / / /bin/true - name: Install pinned official CLI into runner temp run: | python tools/install_ci_codex.py "$RUNNER_TEMP/codex-conformance-bin" diff --git a/tools/install_ci_codex.py b/tools/install_ci_codex.py index d4a1b37..6d3916a 100644 --- a/tools/install_ci_codex.py +++ b/tools/install_ci_codex.py @@ -14,7 +14,7 @@ from urllib.request import Request, urlopen TAG = "rust-v0.153.4" -ASSETS = ("codex-x86_64-unknown-linux-musl.tar.gz", "bwrap-x86_64-unknown-linux-musl.tar.gz") +ASSETS = ("codex-x86_64-unknown-linux-musl.tar.gz",) def fetch(url): From 0f9bf00423f2895efde517b8c860832b8c4dfc42 Mon Sep 17 00:00:00 2001 From: KeyffMS <124252104+KeyffMS@users.noreply.github.com> Date: Sun, 6 Sep 2026 21:12:35 +0200 Subject: [PATCH 7/8] Match upstream Codex hosted-CI namespace prerequisites and restore host policy after conformance --- .github/workflows/plananvil-tests.yml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/.github/workflows/plananvil-tests.yml b/.github/workflows/plananvil-tests.yml index ae229c3..97a56b6 100644 --- a/.github/workflows/plananvil-tests.yml +++ b/.github/workflows/plananvil-tests.yml @@ -94,6 +94,17 @@ jobs: - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: "3.11" + - name: Enable sandbox namespaces in this disposable hosted VM + # Matches openai/codex rust-v0.153.4 .github/actions/setup-ci/action.yml. + # This is a host prerequisite, not a Codex sandbox override. The + # self-hosted live runner and its security policy are not modified. + run: | + sysctl kernel.unprivileged_userns_clone > "$RUNNER_TEMP/codex-userns-restore.conf" + sudo sysctl -w kernel.unprivileged_userns_clone=1 + if [ -f /proc/sys/kernel/apparmor_restrict_unprivileged_userns ]; then + sysctl kernel.apparmor_restrict_unprivileged_userns >> "$RUNNER_TEMP/codex-userns-restore.conf" + sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 + fi - name: Install and verify the operating-system sandbox prerequisite run: | sudo apt-get update -qq @@ -107,3 +118,9 @@ jobs: echo "PLANANVIL_TEST_CODEX_BIN=$RUNNER_TEMP/codex-conformance-bin/codex" >> "$GITHUB_ENV" - name: Verify real CLI lifecycle against deterministic loopback responses run: python -m unittest discover -s tests -p test_qualification_c09_cli.py -v + - name: Restore hosted VM namespace policy + if: always() + run: | + if [ -f "$RUNNER_TEMP/codex-userns-restore.conf" ]; then + sudo sysctl -p "$RUNNER_TEMP/codex-userns-restore.conf" + fi From 1629eb1d3200b4ac36d78caf16624c2b65a4e6f7 Mon Sep 17 00:00:00 2001 From: KeyffMS <124252104+KeyffMS@users.noreply.github.com> Date: Sun, 6 Sep 2026 21:17:39 +0200 Subject: [PATCH 8/8] Record successful real CLI conformance and the scoped hosted-CI prerequisites --- ...EX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md b/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md index 4cf95bc..2e71f7f 100644 --- a/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md +++ b/docs/CODEX_C09_FINITE_RECOVERY_AUDIT_2026-09-06.md @@ -41,4 +41,23 @@ Executable offline tests run actual installer/start/checkpoint/Git/hooks and the A separate hosted conformance job downloads the exact official CLI release and verifies asset hashes. It runs the real CLI, real sandbox, tools and product hooks against a loopback Responses simulator, with no credentials or external model requests. The simulator selects responses and usage; passing demonstrates CLI integration, not autonomous live-model behavior. The normal offline suite does not require downloads or Codex. -Existing cross-platform product/harness suites, packaging, C10/C13 regression tests, evidence validation and release checks remain mandatory. The full model-backed C01-C16 workflow is the final qualification, not an outcome inferred from CI. No sandbox, approvals, auth ownership, product hook output, C13 fallback or release requirement is weakened. +Existing cross-platform product/harness suites, packaging, C10/C13 regression tests, evidence validation and release checks remain mandatory. The full model-backed C01-C16 workflow is the final qualification, not an outcome inferred from CI. No Codex sandbox, approvals, auth ownership, product hook output, C13 fallback or release requirement is weakened. + +## Completed real CLI verification in PR #33 + +[CI #115](https://github.com/KeyffMS/PlanAnvil/actions/runs/34054161715), head `0f9bf00423f2895efde517b8c860832b8c4dfc42`, passed the complete eight-job hosted matrix, including the real CLI conformance job. That job ran four tests without skips and emitted: + +```text +CODEX_01534_OFFLINE_CONFORMANCE_OK: 3 real tools, 2 compactions, 2 compact recovery contexts, 1 completed turn +``` + +The downloaded CLI asset was verified against official release metadata: `codex-x86_64-unknown-linux-musl.tar.gz`, SHA-256 `f479424eca092484dc40d87ae28c44f4cc40234a60045d6131e493800d814a30`. + +The conformance work caught and corrected two independent test-environment problems before merge: + +1. Responses Lite can omit `tools` on ordinary requests. The simulator now distinguishes normal turns from compaction using canonical `client_metadata.x-codex-turn-metadata.request_kind`, consistent with [pinned responses_metadata.rs](https://github.com/openai/codex/blob/rust-v0.153.4/codex-rs/core/src/responses_metadata.rs). Missing/conflicting metadata is rejected by executable tests. Recovery checks require the actual product context phrase, not just a token already present in the initial prompt. +2. The hosted Ubuntu VM denied bubblewrap network-namespace setup (`RTM_NEWADDR: Operation not permitted`). The separate conformance job now follows [pinned upstream setup-ci](https://github.com/openai/codex/blob/rust-v0.153.4/.github/actions/setup-ci/action.yml): enable unprivileged user namespaces and temporarily remove the host AppArmor restriction on their creation. It saves and restores the original sysctl values with an always-run cleanup step. Distro bubblewrap is installed and both user/network namespace creation are tested before Codex starts. Codex runs as the ordinary runner user with its read-only sandbox, approval never and model-tool network disabled. The self-hosted live runner and its policy are not changed. + +The original publication bundle was replayed into the exact candidate tree before these conformance corrections. Temporary publication files and workflow are absent from the final PR tree. No local/offline result is promoted into committed live capability evidence. + +After final PR and post-merge CI pass, the new main commit is ready for the requested full C01-C16 model-backed qualification. That run, not this deterministic loopback peer, decides whether the live model completes the repaired scenario. The full gate and the 900-second C09 deadline remain unchanged.