Harden Trust Pipeline v0.2.1: crash-consistent promotion, verifier independence, watcher/materializer integration - #52
Open
kam6l wants to merge 15 commits into
Open
Harden Trust Pipeline v0.2.1: crash-consistent promotion, verifier independence, watcher/materializer integration#52kam6l wants to merge 15 commits into
kam6l wants to merge 15 commits into
Conversation
added 11 commits
August 17, 2026 15:27
…-closed recovery - WorkspaceLease: persistent promotion.lock that is never unlinked; the OS-level advisory lock (flock / msvcrt) is the only thing released, closing the unlink-then-relock inode race. - Journal: per-entry write-ahead state machine (PREPARED -> APPLY_INTENT -> APPLIED, plus RECOVER_INTENT/RECOVERED/AMBIGUOUS) replacing the single applied: bool; transaction states gain RECOVERY_FAILED. - Journal load: explicit NO_JOURNAL / VALID_JOURNAL / CORRUPT_JOURNAL outcomes; corrupt journals now fail closed instead of reading as absent. Legacy schema-2 journals remain recoverable under their old semantics. - Recovery: validates every journal path (normalized, under root and the approved backup dir, no symlink parents, no AgentDiff-internal targets), re-verifies backup digest/size and opened identity before restoring, restores content and mode through an fsynced temp, converges on crashes between replace and chmod, and fails closed on ambiguity. - Engine: per-entry APPLY_INTENT persists before mutation and APPLIED after post-state verification; recovery failures propagate instead of being ignored; staging and payload copies open with O_NOFOLLOW and verify opened device/inode identity. - Tests: 37 adversarial fault-injection cases (corrupt/malformed journals, traversal, symlink/hardlink backups, crash matrix, mode restoration, cross-process lease exclusion, legacy journal recovery).
…trength - New deterministic verifier-file classifier (tests, fixtures, runner configs, manifests, lockfiles, CI workflows) with a mutation report. - Proof now runs two distinguishable verifications: patched tests on the complete patched workspace, and a baseline run against the sealed pre-run verifier files overlaid onto patched product code. Patch-added verifier files are removed from the baseline workspace. - PROVEN fails closed when verifier-related files were modified but the baseline verifier cannot confirm the patched run. - ProofStrength L0-L4 + WEAK/REVIEW/STRONG label and verifier-independence metadata, computed deterministically; verdict stays PROVEN/NOT_PROVEN. - Clean-room base materialization now preserves file modes and verifies opened identity; proof plan discovery uses the shared classifier. - Tests: classifier matrix, mutation reports, strength matrix, baseline overlay end-to-end, tamper-blocking, added-verifier-file fail-closed.
…on, safe materializer - RuntimeBackend now exposes static RuntimeCapabilities (backend, control levels per boundary, private_workspace, live-safety and source-snapshot support) before execution; runner no longer uses hasattr/getattr string hacks, and configure_source/configure_safety/close are protocol methods. - HybridSafetyWatcher is wired into transaction execution: watchdog event source (optional) feeds dirty-path/directory queues, targeted policy checks accelerate protected-path detection, full reconciliation runs on a configurable cadence/overflow/force, and backend failures degrade to polling with recorded watcher status. Final after-state stays an authoritative full capture. - WorkspaceMaterializer: strategies renamed accurately (CLONE=reflink, FAST_COPY=copy_file_range, STREAM_COPY=fallback; copy_file_range is not a guaranteed reflink), modes preserved on every strategy, symlink/hardlink/ special-file targets rejected instead of silently dropped, O_NOFOLLOW identity checks, target-symlink check before resolve, actual strategy reported; DockerRuntime now materializes through it. - LocalRuntime: reparented descendants are attributed via POSIX session id captured at process start (fixes flaky execution-domain test). - SafetyController split into budget checks and authoritative full reconciliation for targeted dirty-path scanning.
…bject store - verify_integrity routes by capsule version: spec-v2 verified with the structured manifest, legacy v1 verified under its original guarantees; a schema-2 mirror without integrity/manifest.json is an incomplete seal and fails closed. IntegrityReport carries the verified version. - CapsuleReader.compute_merkle_root renamed to compute_root_digest (flat aggregate, not a Merkle tree); deprecated alias retained. - Real content-addressed immutable ObjectStore under .agentdiff/objects: stream + digest + fsync + atomic rename, write-once immutability with re-verification, digest-validated paths, O_NOFOLLOW identity checks, fail-closed reads on mismatch. Capsule layout (spec v2) is unchanged; CAS is the foundation for incremental spec-v3 artifact references and export/import hydration. - Tests: v1/v2 verification, tamper detection, incomplete-seal fail-closed, root-digest determinism, CAS dedupe/immutability/corruption/symlink.
Fixes the CI-quality failures that shipped on main: E501/I001/TC001/TC003/ SIM1xx lint fixes, blind-exception narrowing, dead-code removal, accurate type annotations, msvcrt/yaml/watchdog typing, and updated watcher tests for the accelerated targeted-check behavior.
…to 0.2.1 README now leads with the Docker/prove/promote flow, the Trust Report example, accurate feature status, and an explicit tamper-evident-not- authenticated note. PROJECT_PLAN, SECURITY.md, quickstart, and version references in docs_src are synchronized; CHANGELOG adds the 0.2.1 entry and corrects the inaccurate Merkle/reflink terminology in the 0.2.0 notes. pyproject/__init__/uv.lock bumped to 0.2.1.
The engines existed but were unreachable from the CLI. Adds: - agentdiff prove RUN_ID: clean-room reproduction with the AgentDiff Trust Report summary (runtime, policy, immediate/future blast, trusted plan, baseline/patched test counts, verifier changes, hidden state, proof strength, promotion, verdict) plus machine-readable JSON. - agentdiff promote RUN_ID [--dry-run] [--safe-only] [--path ...]: crash-consistent promotion gated on PROVEN proof; fail-closed exit 9 on recovery ambiguity/corruption and on missing proof. - CLI tests for help surface, missing-run handling, and proof-gated fail-closed behavior.
The materialized workspace is an ephemeral copy derived from durable sealed capsule evidence, so per-file fsync added no durability and cost ~100x throughput (8s -> 0.07s for 1k files on this host). Destination writes are flushed; fsync remains in promotion staging and the sealed capsule where durability is a trust property. Adds benchmarks/materialize_bench.py with measured results (stream/fast/auto across 1k and 10k files) and sets explicit 0.4-or-later removal versions for the deprecated legacy modules. Moves watchdog typing to a pyproject mypy override so quality jobs pass with and without the optional docs group installed.
…ours-preferred promotion/proof/runtime resolution
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
added 2 commits
August 17, 2026 16:10
…across files Walk order is filesystem-dependent, so earlier regular files may already exist when a symlink is rejected; the caller discards the failed private workspace. The guarantee tested is that the symlink itself is never materialized or followed.
The prove/promote JSON output now passes through redact_data before serialization so evidence emitted for sharing/upload is defensively redacted; also addresses the CodeQL clear-text-logging finding on the report serialization path.
added 2 commits
August 17, 2026 16:24
…heuristic CodeQL's py/clear-text-logging-sensitive-data flags the attribute access as a sensitive-data source via its name heuristic, tainting the proof JSON report print path (false positive: the value is a boolean). Renamed the ProofResult field and JSON key to verification_confirmed; the human-facing report label remains 'Trusted Plan'. JSON output additionally passes through redact_data before serialization.
Windows st_mode is not a meaningful security property and restore cannot preserve it, so content is the ground truth there; this fixes the Windows fault-injection recovery failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardening pass on the AgentDiff Trust Pipeline targeting 0.2.1 — no new AI features. The primary story is unchanged: isolate → observe → control → analyze → prove → promote → tamper-evident evidence.
Baseline:
main@145dc8chad CI red at everyuv sync --lockedstep (staleuv.lockpinningagentdiff = 0.1.0), plus failing ruff/mypy/pytest. This branch fixes the causes (nothing disabled) and finishes the trust pipeline.Highlights
Promotion (P0)
promotion.lockis persistent and never unlinked; only the OS advisory lock is released (flock/msvcrt), with cross-process exclusion tests.PREPARED → APPLY_INTENT → APPLIED(+RECOVER_INTENT/RECOVERED/AMBIGUOUS);applied: boolremoved; legacy schema-2 journals remain recoverable under old semantics.NO_JOURNAL / VALID_JOURNAL / CORRUPT_JOURNALoutcomes; corrupt journals block promotion.Proof (P0)
Runtime / Safety (P1)
RuntimeCapabilitiesreplacehasattr/getattrstring sniffing;configure_source/configure_safety/closeare protocol methods.benchmarks/materialize_bench.py).Evidence
compute_merkle_rootrenamedcompute_root_digest(flat aggregate, not a Merkle tree)..agentdiff/objects): immutable, digest-validated, fail-closed reads; capsule spec-v2 layout unchanged (incremental spec-v3/export foundation).Product / Docs / CI
agentdiff proveandagentdiff promoteCLI commands added (previously only the engines existed; the GitHub Action already called them) with the AgentDiff Trust Report output.__version__, uv.lock).Known limitations (unchanged scope)
agentdiff gcand capsule export/import CLI remain Planned.