Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 32 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,46 @@

AgentDiff is pre-release software. APIs and artifact schemas may change before the first stable release.

## 0.2.1

### Security

- **Promotion crash window closed**: every filesystem mutation is now preceded by a persisted `APPLY_INTENT` journal entry and followed by post-state verification and a persisted `APPLIED` entry. Recovery distinguishes "mutation occurred", "mutation did not occur", and "ambiguous" by comparing current host state to the recorded base/result, and fails closed on ambiguity.
- **Workspace lease inode race fixed**: `promotion.lock` is a persistent lock file that is never unlinked; only the OS-level advisory lock (flock / msvcrt) is released, so concurrent AgentDiff promotions cannot both hold the lease.
- **Corrupt journal fails closed**: journal loading returns explicit `NO_JOURNAL` / `VALID_JOURNAL` / `CORRUPT_JOURNAL` outcomes; a corrupt journal blocks promotion instead of reading as "no journal". Legacy schema-2 journals remain recoverable under their old semantics.
- **Recovery path/digest hardening**: journal paths are validated (normalized, below root and the approved backup directory, no symlink parents, no AgentDiff-internal targets); backups are re-verified by SHA-256/size and opened identity before restore; restore writes through an fsynced temp and restores content plus mode. Legacy "applied: bool" journals map onto the new state machine.
- **Verifier independence (proof)**: proof now runs a baseline verification that restores the sealed pre-run verifier files over the patched product code, so agent-modified tests cannot silently weaken the verifier; PROVEN fails closed when verifier-related files were modified and the baseline cannot confirm.
- **Proof-strength metadata**: deterministic L0-L4 levels with WEAK/REVIEW/STRONG labels and verifier-independence ratings; the PROVEN/NOT_PROVEN verdict remains deterministic.
- **Terminology corrections**: `copy_file_range` is no longer described as a guaranteed reflink (CLONE = FICLONE only when supported, FAST_COPY = accelerated copy, STREAM_COPY = fallback); the capsule aggregate digest is renamed Capsule Root Digest, not "Merkle root".
- **Materializer mode preservation**: the executable bit and file modes are preserved on every copy strategy; symlinks, hardlinks, and special files are rejected instead of silently dropped; O_NOFOLLOW identity checks prevent source substitution.
- **Runtime capabilities**: explicit static `RuntimeCapabilities` (filesystem/network/process/privilege control levels, private workspace, live safety, source snapshot support) replace `hasattr`/`getattr` string sniffing before execution.
- **Hybrid safety watcher integrated**: OS event hints feed dirty-path targeted checks with periodic authoritative full reconciliation; backend failures degrade to polling with recorded status; final after-state always comes from an authoritative capture.
- **Capsule v1/v2 separation**: `verify_integrity` routes by capsule version; legacy v1 capsules verify under their original guarantees, and a schema-2 mirror without the structured manifest fails closed.
- **Content-addressed object store**: immutable `ObjectStore` (`.agentdiff/objects`) with write-once semantics, digest-validated paths, and fail-closed reads; capsule layout (spec v2) is unchanged, with the object store as the incremental spec-v3/export foundation.

### Changed

- `RuntimeBackend` protocol gains `capabilities`, `configure_source`, `configure_safety`, and `close`; all runtimes implement them.
- `WorkspaceMaterializer` strategies renamed (`CLONE`/`FAST_COPY`/`STREAM_COPY`; `REFLINK`/`COPY` retained as aliases) and reports the strategy actually used.
- Docker runtime materializes the private workspace through `WorkspaceMaterializer` and records materialization evidence in `runtime.json`.
- `agentdiff inspect` and proof JSON now include proof-strength, baseline-verifier, and watcher evidence.

### Tests

- Added adversarial promotion fault-injection coverage: corrupt/malformed journals, traversal, backup symlink/hardlink, crash at every write-ahead transition, mode restoration, ambiguous recovery, legacy journal recovery, cross-process lease exclusion.
- Added proof-strength matrix, verifier-mutation classifier, baseline overlay end-to-end, and tamper-blocking tests.
- Added materializer security tests (mode preservation per strategy, symlink/hardlink/special-file rejection, target symlink), capsule v1/v2 verification tests, CAS object-store tests, and hybrid watcher degradation/overflow tests.

## 0.2.0

### Added

- **Proof Trust Provenance (P0 Security)**: Base-snapshot verification plan auto-discovery with deterministic tamper rejection when patches modify build or test configuration files (`package.json`, `pyproject.toml`, `conftest.py`, `Makefile`, etc.) without an explicit policy override.
- **Crash-Consistent Promotion Gate**: Multi-file promotion with advisory workspace lease locking (`WorkspaceLease`), write-ahead transaction logging (`PromotionJournal`), two-phase staging with `fsync` validation (`PromotionStager`), and automatic crash recovery (`PromotionRecovery`).
- **Policy Schema v2**: Added first-class `proof:` section supporting container image, network mode, setup, build, and test command sequences.
- **Capsule Spec v2 & Merkle Validation**: Structured integrity manifests, content-addressed blob references, deterministic Merkle root hashing, and backward compatibility with v1 flat capsules.
- **Capsule Spec v2**: Structured integrity manifests, blob references, a deterministic Capsule Root Digest (flat aggregate — not a Merkle tree), and backward compatibility with v1 flat capsules.
- **Hybrid Safety Watcher**: Blends filesystem notification hints with deterministic snapshot validation and budget enforcement.
- **High-Speed Workspace Materializer**: Fast copy-on-write / reflink / copy directory materializer for isolated container workspaces.
- **High-Speed Workspace Materializer**: Fast clone (FICLONE where supported) / accelerated copy / streaming directory materializer for isolated container workspaces.

### Changed

Expand Down
32 changes: 19 additions & 13 deletions PROJECT_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,20 @@ The primary path is `agentdiff run --task "…" -- <command>`. Experimental memo
6. Evidence is redacted, bounded, and private by default.
7. Isolation, tracing, and agent protocols are integration seams, not features to rebuild.

## Current `0.1.0` surface
## Current `0.2.x` surface

### Beta
### Beta — trust pipeline

- No-follow filesystem manifests and private run capsules.
- Deterministic `allow`, `review`, and `deny` policy with provenance.
- Local shell-free command execution with timeout and best-effort process evidence.
- Explainable, capped blast-radius scoring.
- No-follow filesystem manifests and private run capsules (spec v2; legacy v1 capsules remain verifiable under their original guarantees).
- Deterministic `allow`, `review`, and `deny` policy with provenance and live budget enforcement.
- Local shell-free execution with timeout, process, and port evidence; Docker runtime with a private writable workspace (never a writable host repo), cap-drop, no-new-privileges, and explicit network modes.
- Explainable, capped immediate blast-radius scoring; separate future-execution-risk analysis.
- Hybrid safety watcher: OS event hints feed dirty-path targeted checks, authoritative full reconciliation runs on schedule/overflow/force, and backend failures degrade to polling with recorded status.
- Clean-room proof: trusted verification plan from pre-run evidence, patched tests, and an independent baseline verifier (pre-run verifier files over patched product code) with deterministic proof-strength metadata (L0-L4).
- Crash-consistent promotion gate: write-ahead journal with per-entry state machine, persistent workspace lease (never-unlinked lock file), validated backup restore with digest/mode checks, and fail-closed recovery on corrupt or ambiguous state.
- Run listing, inspection, checksum verification, and exact-identity cleanup.
- Conflict-safe recovery for eligible regular files.
- Content-addressed immutable object store as the migration foundation for spec-v3 artifact references and future export/import.
- Linux, macOS, and native Windows CI on Python 3.12–3.14.

### Experimental
Expand All @@ -46,7 +50,7 @@ The primary path is `agentdiff run --task "…" -- <command>`. Experimental memo
- Artifact migration and compatibility tooling.
- Larger external-state benchmark coverage.

An HTTP API, hosted dashboard, Docker backend, bundled sandbox, universal network blocking, and arbitrary external-state rollback are not implemented.
An HTTP API, hosted dashboard, bundled sandbox, universal network blocking, and arbitrary external-state rollback are not implemented. The Docker backend implements the isolation boundary this plan targets; it is a capability-bearing container boundary, not a virtual machine.

## Release gates

Expand Down Expand Up @@ -79,12 +83,14 @@ An HTTP API, hosted dashboard, Docker backend, bundled sandbox, universal networ

### Differentiated safety core

1. Add clean-room proof by replaying a captured patch in a fresh worktree before promotion.
2. Detect future execution risk in package scripts and GitHub Actions changes, then extend to Dockerfiles, Makefiles, hooks, and editor tasks.
3. Add an experimental copy-on-write Docker runtime where the real repository is changed only by an explicit, policy-filtered promotion step.
1. [x] Clean-room proof replays the captured patch in a fresh environment before promotion.
2. [x] Future execution risk analysis covers package scripts, GitHub Actions, Dockerfiles, Makefiles, hooks, and editor tasks.
3. [x] Docker runtime materializes a private workspace and the real repository is changed only by an explicit, policy-filtered promotion step.
4. Harden verifier independence further: external signed CI verification (proof strength L4) and verifier-file policy controls.

### Evidence moat

1. Add signed, shareable capsule export and standardized telemetry.
2. Add run attribution for changed lines and evidence-based comparison of parallel agent attempts.
3. Keep adversarial race, path, hardlink, redaction, and rollback tests ahead of new claims.
1. Add signed (authenticated) capsule support; current checksums are tamper-evident, not authenticated.
2. Add shareable capsule export/import (the CAS object store is the hydration foundation).
3. Add run attribution for changed lines and evidence-based comparison of parallel agent attempts.
4. Keep adversarial race, path, hardlink, redaction, promotion-crash, and rollback tests ahead of new claims.
105 changes: 73 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
<h1 align="center">AgentDiff</h1>

<p align="center">
<strong>See what the agent changed. Undo only the collateral.</strong><br>
Independent state observation, deterministic intent policy, explainable blast radius, and conflict-safe selective recovery for AI-agent commands.
<strong>DON'T TRUST AN AI PATCH. PROVE IT.</strong><br>
Isolate the agent, observe real state, enforce deterministic policy, reproduce the patch in a clean room, and promote only proven work through a crash-consistent gate.
</p>

<p align="center">
Expand Down Expand Up @@ -42,70 +42,98 @@ A command can exit successfully while leaving one intended edit, one dependency

## Start in under a minute

AgentDiff `0.1.0` requires Python 3.12+ and is currently installed from source:
AgentDiff `0.2.x` requires Python 3.12+ and is currently installed from source:

```bash
git clone https://github.com/kam6l/agentdiff.git
cd agentdiff
uv tool install .
```

From the project you want to observe:
From the project you want to guard, run the agent inside an isolated private
workspace, prove the result in a clean room, then promote only proven work
through the crash-consistent gate:

```bash
agentdiff policy init
agentdiff run --task "Fix authentication" -- codex
```

The summary leads with the decision you need:
agentdiff run \
--runtime docker \
--task "Fix authentication" \
-- codex

```text
Task completed
agentdiff inspect <run-id>

Expected changes: 4
Unexpected changes: 3
Protected changes: 1
agentdiff prove <run-id>

Blast Radius: HIGH (72/100)
Recovery available: YES
Policy outcome: DENY
agentdiff promote <run-id> --dry-run --safe-only
agentdiff promote <run-id> --safe-only

agentdiff verify <run-id>
```

Then inspect the durable capsule or recover unchanged collateral:
`agentdiff prove` reproduces the base-plus-patch workspace in a clean Docker
container, runs the **patched** tests, then re-runs the trusted **baseline**
tests (the pre-run verifier files restored over the patched product code) so
an agent cannot hide behind weakened tests. `agentdiff promote` applies only
proven, policy-selected changes with a write-ahead journal and automatic
crash recovery; `--safe-only` selects only `ALLOW` changes.

```bash
agentdiff inspect <run-id>
agentdiff verify <run-id>
agentdiff rollback <run-id> --safe-only
The trust report shows immediate vs future blast radius, verifier changes,
proof strength, and a single deterministic verdict:

```text
Runtime Docker / private isolated workspace
Policy ALLOW
Immediate Blast 12 / LOW
Future Blast 6 / LOW
Trusted Plan YES
Baseline Tests 184 / 184
Patched Tests 191 / 191
Verifier Changes 2
Proof Strength L3 / STRONG
Promotion CRASH-CONSISTENT / SAFE
✓ PROVEN
```

[Run the reproducible five-minute example](https://kam6l.github.io/agentdiff/docs/quickstart/)
For lower isolation, `--runtime local` runs the agent directly on the host
with observation and recovery, not a sandbox.

[Run the reproducible example](https://kam6l.github.io/agentdiff/docs/quickstart/)

## How it works

| Stage | Result |
|---|---|
| **Capture** | No-follow before-state manifest and bounded recovery backups |
| **Execute** | Exact argv, exit status, owned-process evidence, and machine-wide port observations |
| **Evaluate** | `allow` / `review` / `deny` decisions, rule provenance, warnings, and a 0-100 score |
| **Recover** | Exact post-state conflict checks before eligible collateral is changed |
| **Isolate** | Docker private workspace (never a writable host repo) or observed local run |
| **Observe** | No-follow before/after manifests, live hybrid safety watcher, owned-process and port evidence |
| **Control** | Deterministic `allow` / `review` / `deny` policy plus budget enforcement |
| **Analyze** | Immediate and future blast radius stay separate |
| **Prove** | Clean-room reproduction with trusted baseline + patched verification |
| **Promote** | Write-ahead journal, crash-consistent recovery, workspace lease |
| **Evidence** | Tamper-evident sealed capsules (spec v2, v1 still verifiable) |

## Feature status

| Status | Surface |
|---|---|
| **Beta** | Local transactions, policy, capsules, verification, scoring, and regular-file recovery (tested on Python 3.12-3.14) |
| **Experimental** | Cortex evidence memory and provider routing, Anthropic `srt` adapter, transport-neutral MCP policy hook, and LangChain callback |
| **Planned** | PyPI/binary releases, authenticated evidence, telemetry export, and a maintained hosted sandbox integration |
| **Beta** | Transactions, policy, blast radius, capsules, clean-room proof, baseline verifier, promotion gate with crash recovery, and Docker/local runtimes (tested on Python 3.12-3.14, Linux/macOS/Windows) |
| **Experimental** | Cortex evidence memory and provider routing, Anthropic `srt` adapter, transport-neutral MCP policy hook, LangChain callback, content-addressed object store (spec-v3 migration foundation) |
| **Planned** | PyPI/binary releases, authenticated (signed) capsules, capsule export/import CLI, hosted dashboard, maintained hosted sandbox integration |

There is no HTTP server, hosted dashboard, Docker backend, bundled sandbox, or claimed PyPI release today.
Capsule checksums are **tamper-evident, not authenticated**: they detect
accidental or modest modification but an attacker who can rewrite the whole
capsule can produce a new self-consistent one. Signing remains future work.
There is no HTTP server, hosted dashboard, or claimed PyPI release today.

## CLI

| Command | Purpose |
|---|---|
| `agentdiff run -- <cmd>` | Wrap an explicit argv in a transaction |
| `agentdiff runs` / `inspect` / `verify` | Find and validate local evidence capsules |
| `agentdiff run -- <cmd>` | Wrap an explicit argv in a transaction (`--runtime docker` for isolation) |
| `agentdiff runs` / `inspect` / `verify` | Find, inspect, and validate evidence capsules |
| `agentdiff prove <id>` | Clean-room reproduction + trusted baseline/patched verification |
| `agentdiff promote <id> [--dry-run] [--safe-only]` | Crash-consistent, proof-gated promotion |
| `agentdiff rollback <id> --safe-only` | Recover eligible `review` and `deny` changes |
| `agentdiff cleanup <id>` | Signal exact PID/create-time identities recorded for a run |
| `agentdiff doctor` | Report implemented capabilities and limits |
Expand All @@ -129,7 +157,20 @@ agentdiff cortex advise <run-id>

## Trust boundary

AgentDiff records symlinks without traversing them, redacts common secret-bearing values, verifies backups and capsule checksums, and identifies processes by PID plus creation time. It does **not** authenticate a capsule against an attacker who can replace the whole directory, attribute machine-wide port changes to one process, or undo APIs, databases, network effects, hardlinks, symlinks, and unbacked files.
AgentDiff never trusts the agent's explanation, environment, verifier, or
generated state. It observes independent filesystem state, reproduces patches
with trusted pre-run verification commands, and promotes only when evidence
supports it. Promotion recovery fails closed when host state is ambiguous or
the journal is corrupt, and the workspace lease never deletes its lock file
(the OS lock itself is released instead), so concurrent promotions cannot
both hold the lock.

It does **not** authenticate a capsule against an attacker who can replace
the whole directory, attribute machine-wide port changes to one process, or
undo APIs, databases, network effects, hardlinks, symlinks, and unbacked
files. Cortex (the experimental LLM surface) can read verified evidence and
generate advice but never decides policy, proof, blast radius, or promotion
outcomes.

Read the [runtime model](https://kam6l.github.io/agentdiff/docs/concepts/runtime/), [recovery guarantees](https://kam6l.github.io/agentdiff/docs/concepts/recovery/), and [security limits](https://kam6l.github.io/agentdiff/docs/trust/).

Expand Down
Loading
Loading