diff --git a/PROJECT_PLAN.md b/PROJECT_PLAN.md index cc4f050..6bd6432 100644 --- a/PROJECT_PLAN.md +++ b/PROJECT_PLAN.md @@ -52,7 +52,7 @@ The primary product wedge is `agentdiff api scan/check/migrate` for Self-Maintai - One maintained hosted/disposable sandbox integration. - Artifact migration and compatibility tooling. - Larger external-state benchmark coverage. -- **Migration Engine: deterministic AST transforms, coding-agent fallback, clean-room verification (V0–V5), Migration Certificate, GitHub PR delivery** +- **Migration pipeline completion: real ProofEngine execution for migrations, rollback verification, failure evidence, GitHub PR delivery** An HTTP API, hosted dashboard, Docker backend, bundled sandbox, universal network blocking, and arbitrary external-state rollback are not implemented. @@ -82,12 +82,15 @@ An HTTP API, hosted dashboard, Docker backend, bundled sandbox, universal networ ### Self-Maintaining APIs (primary wedge) -1. **API Change Manifest** — structured machine-readable upstream change format (YAML/JSON) for provider deprecations, SDK releases, model shutdowns. -2. **Deterministic AST Transforms** — for known simple migrations (OpenAI Responses API, Stripe PaymentIntents, etc.); registry extensible by providers. -3. **Hybrid Migration Engine** — simple → AST transform; complex → coding agent; **all patches untrusted until ProofEngine verifies**. -4. **Verification Levels (V0–V5)** — syntax/type/build → targeted tests → full repo tests → API contract/mock tests → user-defined integration verification. -5. **Migration Certificate** — machine-readable artifact: provider change, affected usages, files changed, blast radius, policy result, tests executed, verification level, proof digest, capsule ID, rollback info. -6. **GitHub PR Automation** — `--open-pr` delivers Migration Certificate in PR body; conflict-safe promotion; no auto-merge. +1. **API Change Manifest** — structured machine-readable upstream change format (YAML/JSON) for provider deprecations, SDK releases, model shutdowns. *(implemented)* +2. **Deterministic AST Transforms** — for known simple migrations (OpenAI Responses API, Stripe PaymentIntents, etc.); registry extensible by providers. *(implemented)* +3. **Migration Engine** — scan → match → plan → transform in private workspace → verify → certificate. *(implemented)* +4. **Provider Intelligence Layer** — parse changelogs, diff OpenAPI specs, analyze SDK releases, and accept AI suggestions as validated manifest candidates. AI output never touches code directly. *(implemented)* +5. **Provider Plugin System** — `agentdiff provider install/list`; providers ship `manifests/`, `transforms/`, `tests/`, `metadata.yaml` without core changes. *(implemented)* +6. **Verification Levels (V0–V5)** — syntax/type/build → targeted tests → full repo tests → API contract/mock tests → user-defined integration verification. +7. **Migration Certificate** — machine-readable artifact: provider change, affected usages, files changed, blast radius, policy result, tests executed, verification level, proof digest, capsule ID, rollback info. *(implemented)* +8. **GitHub PR Automation** — `--open-pr` delivers Migration Certificate in PR body; conflict-safe promotion; no auto-merge. +9. **API Knowledge Graph** — track Repository → API usage → SDK version → migration status; design scalable, no extra database yet. ### Credibility and distribution diff --git a/README.md b/README.md index 9ea690f..2a85972 100644 --- a/README.md +++ b/README.md @@ -51,9 +51,11 @@ AgentDiff solves this by making **verified migrations** the default: The coding agent is probabilistic. AgentDiff is the deterministic verifier that decides whether the result is trustworthy. -## Self-Maintaining APIs (MVP) +## Self-Maintaining APIs -AgentDiff analyzes Python AST to detect third-party API usages (starting with **OpenAI** and **Stripe**), matches usages against known breaking changes, and calculates migration blast radius and test proof requirements: +AgentDiff turns API changes into verified migrations. The pipeline is: + +**Detect → Plan → Execute → Verify → Certify → Deliver** ```bash # Scan repository for all external API calls @@ -61,7 +63,47 @@ agentdiff api scan --root . # Check for breaking changes, calculate impact, and report remediation agentdiff api check --root . --fail-on high + +# Generate + verify a migration in a private workspace, emit a certificate +agentdiff api migrate --provider openai --change chat_to_responses + +# Turn upstream signals into validated manifest candidates +agentdiff api intel --provider openai --changelog CHANGELOG.md + +# Install provider migration plugins +agentdiff provider install stripe ./providers/stripe +agentdiff provider list +``` + +### Provider Intelligence Layer + +AgentDiff can ingest upstream signals and produce validated `APIChangeManifest` +candidates — **suggestion only, never applied directly**: + +- `--changelog` — parse markdown changelogs for removals/deprecations/renames +- `--openapi-before/--openapi-after` — diff two OpenAPI specs for breaking changes +- `--release` — analyze SDK release notes +- AI-assisted suggestions are accepted as candidates that must still pass + deterministic validation before they can drive a migration + +### Provider Plugin System + +Providers and community members ship migrations without touching core code: + ``` +providers// + metadata.yaml # name, library, version + manifests/ # *.yaml APIChangeManifest files + transforms/ # python modules registering AST transforms + tests/ # optional plugin tests +``` + +### Trust model + +The coding agent (or AST transform) generates the migration. AgentDiff decides +whether it is trustworthy — deterministic policy, blast radius, clean-room proof, +and a MigrationCertificate recording exactly what was verified. **The AI is +probabilistic; the trust decision is deterministic.** ## Zero-Touch Trust Engine (Foundation) @@ -161,6 +203,9 @@ There is no hosted dashboard or hosted service: the sidecar is a local daemon, a | `agentdiff workspace status/warm/prune` | Trusted warm workspace snapshots | | `agentdiff policy init/validate/explain` | Create and inspect versioned policy | | `agentdiff api scan` / `check` | Self-maintaining API usage scanner and breaking change checker | +| `agentdiff api migrate` | Generate and verify an API migration in a private workspace | +| `agentdiff api intel` | Analyze changelog/OpenAPI/release signals into manifest candidates | +| `agentdiff provider list` / `install` | Manage provider migration plugins | | `agentdiff cortex ...` | Open the optional evidence-memory, skill-card, and provider tool namespace | The earlier `snapshot`, `diff`, and `eval` implementation remains importable for compatibility testing but is no longer exposed as a public CLI path. diff --git a/src/agentdiff/api/__init__.py b/src/agentdiff/api/__init__.py index be591bd..3b50b8d 100644 --- a/src/agentdiff/api/__init__.py +++ b/src/agentdiff/api/__init__.py @@ -1,5 +1,17 @@ """Self-Maintaining APIs: AST scanning, breaking change matching, and migration impact.""" +from agentdiff.api.intel import ( + ChangelogChange, + ChangelogParser, + IntelArtifact, + ManifestCandidate, + OpenAPIBreakingChange, + OpenAPIDiffAnalyzer, + ProviderIntelEngine, + SDKReleaseAnalyzer, + SDKReleaseChange, +) + from agentdiff.api.manifest import ( AffectedSymbols, APIChangeManifest, @@ -30,6 +42,13 @@ VerificationLevel, assess_migration_confidence, ) +from agentdiff.api.plugins import ( + ProviderPlugin, + discover_plugins, + install_plugin, + list_plugins, + load_plugin, +) from agentdiff.api.providers import ( APIProvider, OpenAIProvider, @@ -63,6 +82,10 @@ "AffectedSymbols", "ChangeSeverity", "ChangeType", + "ChangelogChange", + "ChangelogParser", + "IntelArtifact", + "ManifestCandidate", "ManifestSource", "MatchedChange", "MigrationAssessment", @@ -79,6 +102,13 @@ "OpenAIChatToResponsesTransform", "OpenAILegacyChatCompletionTransform", "OpenAIProvider", + "OpenAPIBreakingChange", + "OpenAPIDiffAnalyzer", + "ProviderIntelEngine", + "ProviderPlugin", + "ReplacementSymbols", + "SDKReleaseAnalyzer", + "SDKReleaseChange", "ReplacementSymbols", "SDKVersionInfo", "SourceType", @@ -86,12 +116,20 @@ "VerificationLevel", "assess_migration_confidence", "detect_installed_sdk_versions", + "discover_plugins", "get_all_providers", "get_builtin_manifest", "get_provider", "get_providers_for_selection", "get_transform", "get_transforms_for_usage", + "install_plugin", + "is_version_affected", + "list_builtin_manifests", + "list_plugins", + "list_providers", + "list_transforms", + "load_plugin", "is_version_affected", "list_builtin_manifests", "list_providers", diff --git a/src/agentdiff/api/certificate.py b/src/agentdiff/api/certificate.py new file mode 100644 index 0000000..473c821 --- /dev/null +++ b/src/agentdiff/api/certificate.py @@ -0,0 +1,73 @@ +"""Migration certificate output and storage.""" + +from __future__ import annotations + +import json +from datetime import datetime, timezone +from pathlib import Path +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from agentdiff.api.models import MigrationCertificate + +CERTIFICATE_DIR = ".agentdiff/certificates" + + +def write_certificate(certificate: "MigrationCertificate", root: str | Path) -> Path: + """Write certificate to .agentdiff/certificates/ directory.""" + root_path = Path(root).expanduser().resolve(strict=True) + cert_dir = root_path / CERTIFICATE_DIR + cert_dir.mkdir(parents=True, exist_ok=True, mode=0o700) + + filename = f"{certificate.certificate_id}.json" + cert_path = cert_dir / filename + + data = certificate.to_dict() + data["schema_version"] = 1 + data["written_at"] = datetime.now(timezone.utc).isoformat() + + cert_path.write_text( + json.dumps(data, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + return cert_path + + +def write_certificate_legacy(certificate: "MigrationCertificate", path: str | Path) -> Path: + """Write certificate to a specific path.""" + path = Path(path).expanduser().resolve() + path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + + data = certificate.to_dict() + data["schema_version"] = 1 + data["written_at"] = datetime.now(timezone.utc).isoformat() + + path.write_text( + json.dumps(data, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + return path + + +def read_certificate(path: str | Path) -> dict[str, Any]: + """Read a certificate from disk.""" + path = Path(path).expanduser().resolve() + return json.loads(path.read_text(encoding="utf-8")) + + +def list_certificates(root: str | Path) -> list[Path]: + """List all certificates in the repository.""" + root_path = Path(root).expanduser().resolve(strict=True) + cert_dir = root_path / CERTIFICATE_DIR + if not cert_dir.exists(): + return [] + return sorted(cert_dir.glob("*.json")) + + +def get_latest_certificate(root: str | Path, provider: str, change_id: str) -> Path | None: + """Get the most recent certificate for a provider/change.""" + certs = list_certificates(root) + matching = [c for c in certs if provider in c.name and change_id in c.name] + return matching[-1] if matching else None diff --git a/src/agentdiff/api/intel/__init__.py b/src/agentdiff/api/intel/__init__.py new file mode 100644 index 0000000..696a711 --- /dev/null +++ b/src/agentdiff/api/intel/__init__.py @@ -0,0 +1,28 @@ +"""Provider Intelligence Layer. + +Turns upstream provider signals (changelogs, OpenAPI diffs, SDK releases) +into validated APIChangeManifest candidates. AI-assisted generation is +suggestion-only: the output is a manifest candidate that still requires +deterministic validation before it can drive migrations. +""" + +from agentdiff.api.intel.changelog import ChangelogChange, ChangelogParser +from agentdiff.api.intel.engine import ( + IntelArtifact, + ManifestCandidate, + ProviderIntelEngine, +) +from agentdiff.api.intel.openapi import OpenAPIBreakingChange, OpenAPIDiffAnalyzer +from agentdiff.api.intel.release import SDKReleaseAnalyzer, SDKReleaseChange + +__all__ = [ + "ChangelogChange", + "ChangelogParser", + "IntelArtifact", + "ManifestCandidate", + "OpenAPIBreakingChange", + "OpenAPIDiffAnalyzer", + "ProviderIntelEngine", + "SDKReleaseAnalyzer", + "SDKReleaseChange", +] diff --git a/src/agentdiff/api/intel/changelog.py b/src/agentdiff/api/intel/changelog.py new file mode 100644 index 0000000..1cfb15c --- /dev/null +++ b/src/agentdiff/api/intel/changelog.py @@ -0,0 +1,142 @@ +"""Changelog parser: extract API changes from markdown changelogs.""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Any + +from agentdiff.api.models import ChangeSeverity, ChangeType + +_BREAKING_MARKERS = ( + "breaking", + "removed", + "migration required", + "not backwards compatible", + "major change", +) + +_REMOVAL_RE = re.compile(r"\b(removed|deleted|dropped)\b", re.IGNORECASE) +_DEPRECATION_RE = re.compile(r"\b(deprecated|deprecation)\b", re.IGNORECASE) +_RENAME_RE = re.compile(r"\b(renamed|rename)\b", re.IGNORECASE) + + +@dataclass(frozen=True, slots=True) +class ChangelogChange: + """One API-affecting change extracted from a changelog entry.""" + + title: str + body: str + section: str # e.g. "Breaking Changes", "Deprecated" + change_type: ChangeType + severity: ChangeSeverity + mentions: tuple[str, ...] = () # symbols/words mentioned in the entry + + def to_dict(self) -> dict[str, Any]: + return { + "title": self.title, + "body": self.body, + "section": self.section, + "change_type": self.change_type.value, + "severity": self.severity.value, + "mentions": list(self.mentions), + } + + +def _classify_type(text: str) -> ChangeType: + lowered = text.lower() + if _REMOVAL_RE.search(lowered) or "removed" in lowered: + return ChangeType.REMOVAL + if _DEPRECATION_RE.search(lowered): + return ChangeType.DEPRECATION + if _RENAME_RE.search(lowered): + return ChangeType.RENAME + return ChangeType.BEHAVIOR_CHANGE + + +def _classify_severity(text: str, section: str) -> ChangeSeverity: + """Classify severity from the entry text; the section only boosts when the + entry itself carries no explicit severity signal.""" + lowered = text.lower() + if any(marker in lowered for marker in _BREAKING_MARKERS): + return ChangeSeverity.CRITICAL if "removed" in lowered else ChangeSeverity.HIGH + if "deprecated" in lowered: + return ChangeSeverity.MODERATE + # Low-signal entry: let the section heading decide. + section_lowered = section.lower() + if any(marker in section_lowered for marker in _BREAKING_MARKERS): + return ChangeSeverity.HIGH + return ChangeSeverity.LOW + + +class ChangelogParser: + """Parse a markdown changelog into API change candidates.""" + + def __init__(self, provider: str) -> None: + self.provider = provider + + def parse(self, content: str) -> list[ChangelogChange]: + """Parse changelog markdown and return structured changes.""" + changes: list[ChangelogChange] = [] + sections: list[tuple[str, str]] = [] + + # Split into sections (## headings) with their body text + current_section = "Uncategorized" + current_body: list[str] = [] + for line in content.splitlines(): + stripped = line.strip() + if stripped.startswith("## ") or stripped.startswith("# "): + if current_body: + sections.append((current_section, "\n".join(current_body))) + current_section = stripped.lstrip("# ").strip() + current_body = [] + else: + current_body.append(line) + if current_body: + sections.append((current_section, "\n".join(current_body))) + + for section, body in sections: + for entry in self._split_entries(body): + title, entry_body = self._entry_parts(entry) + if not title: + continue + change_type = _classify_type(f"{title} {entry_body}") + severity = _classify_severity(f"{title} {entry_body}", section) + mentions = tuple( + dict.fromkeys(re.findall(r"[a-zA-Z_][a-zA-Z0-9_.]*", f"{title} {entry_body}")) + ) + changes.append( + ChangelogChange( + title=title, + body=entry_body, + section=section, + change_type=change_type, + severity=severity, + mentions=mentions, + ) + ) + return changes + + @staticmethod + def _split_entries(body: str) -> list[str]: + """Split a section body into bullet/list entries.""" + entries: list[str] = [] + current: list[str] = [] + for line in body.splitlines(): + stripped = line.strip() + if stripped.startswith(("- ", "* ", "+ ")): + if current: + entries.append("\n".join(current)) + current = [stripped[2:].strip()] + elif stripped: + current.append(line) + if current: + entries.append("\n".join(current)) + return entries + + @staticmethod + def _entry_parts(entry: str) -> tuple[str, str]: + lines = entry.splitlines() + title = lines[0].strip() if lines else "" + rest = "\n".join(lines[1:]).strip() if len(lines) > 1 else "" + return title, rest diff --git a/src/agentdiff/api/intel/engine.py b/src/agentdiff/api/intel/engine.py new file mode 100644 index 0000000..32468d0 --- /dev/null +++ b/src/agentdiff/api/intel/engine.py @@ -0,0 +1,308 @@ +"""Provider intelligence engine: turns upstream signals into manifest candidates.""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +from agentdiff.api.intel.changelog import ChangelogParser +from agentdiff.api.intel.openapi import OpenAPIDiffAnalyzer +from agentdiff.api.intel.release import SDKReleaseAnalyzer +from agentdiff.api.manifest import ( + AffectedSymbols, + APIChangeManifest, + ManifestSource, + MigrationStrategyConfig, + MigrationStrategyType, + ReplacementSymbols, + SourceType, +) +from agentdiff.api.models import ChangeSeverity, ChangeType + + +@dataclass(frozen=True, slots=True) +class ManifestCandidate: + """A suggested manifest before validation. AI output — never applied directly.""" + + provider: str + change_id: str + title: str + change_type: ChangeType + severity: ChangeSeverity + affected_symbols: tuple[str, ...] + replacement_symbols: tuple[str, ...] = () + source_type: SourceType = SourceType.CUSTOM + source_url: str = "" + description: str = "" + confidence: float = 0.5 + + def to_dict(self) -> dict[str, Any]: + return { + "provider": self.provider, + "change_id": self.change_id, + "title": self.title, + "change_type": self.change_type.value, + "severity": self.severity.value, + "affected_symbols": list(self.affected_symbols), + "replacement_symbols": list(self.replacement_symbols), + "source_type": self.source_type.value, + "source_url": self.source_url, + "confidence": self.confidence, + } + + +@dataclass(frozen=True, slots=True) +class IntelArtifact: + """What was analyzed and what it produced.""" + + kind: str # "changelog" | "openapi_diff" | "sdk_release" | "ai_suggestion" + input_path: str + candidates: tuple[ManifestCandidate, ...] + generated_at: str = "" + + def to_dict(self) -> dict[str, Any]: + return { + "kind": self.kind, + "input_path": self.input_path, + "candidates": [c.to_dict() for c in self.candidates], + "generated_at": self.generated_at, + } + + +class ProviderIntelEngine: + """Analyze provider signals and produce validated manifest candidates.""" + + def __init__(self, provider: str, library: str = "") -> None: + self.provider = provider + self.library = library or provider + + # -- per-source analyzers ------------------------------------------------ + + def from_changelog(self, path: str | Path) -> IntelArtifact: + parser = ChangelogParser(self.provider) + raw = Path(path).read_text(encoding="utf-8") + entries = parser.parse(raw) + candidates: list[ManifestCandidate] = [] + for entry in entries: + if entry.change_type in {ChangeType.REMOVAL, ChangeType.DEPRECATION, ChangeType.RENAME}: + candidates.append( + ManifestCandidate( + provider=self.provider, + change_id=self._slugify(f"{entry.change_type.value}-{entry.title}"), + title=entry.title, + change_type=entry.change_type, + severity=entry.severity, + affected_symbols=self._mentions_to_symbols(entry.mentions), + source_type=SourceType.CHANGELOG, + source_url="", + description=entry.body, + confidence=0.7, + ) + ) + return IntelArtifact( + kind="changelog", + input_path=str(path), + candidates=tuple(candidates), + generated_at=self._now(), + ) + + def from_openapi_diff( + self, + before: str | Path | dict[str, Any], + after: str | Path | dict[str, Any], + ) -> IntelArtifact: + analyzer = OpenAPIDiffAnalyzer(self.provider) + changes = analyzer.diff(before, after) + candidates = [ + ManifestCandidate( + provider=self.provider, + change_id=self._slugify(f"{c.change_type.value}-{c.operation_id or c.path}"), + title=f"{c.method.upper()} {c.path}: {c.detail}", + change_type=c.change_type, + severity=c.severity, + affected_symbols=tuple(x for x in (c.operation_id,) if x), + source_type=SourceType.OFFICIAL_DOCS, + source_url="", + description=c.detail, + confidence=0.8, + ) + for c in changes + ] + return IntelArtifact( + kind="openapi_diff", + input_path="" if isinstance(before, dict) else str(before), + candidates=tuple(candidates), + generated_at=self._now(), + ) + + def from_sdk_release(self, path: str | Path) -> IntelArtifact: + analyzer = SDKReleaseAnalyzer(self.provider, self.library) + content = Path(path).read_text(encoding="utf-8") + changes = analyzer.analyze(content) + candidates = [ + ManifestCandidate( + provider=self.provider, + change_id=self._slugify(f"{c.change_type.value}-{c.title}"), + title=c.title, + change_type=c.change_type, + severity=c.severity, + affected_symbols=(), + source_type=SourceType.SDK_RELEASE, + source_url="", + description=c.body, + confidence=0.6, + ) + for c in changes + ] + return IntelArtifact( + kind="sdk_release", + input_path=str(path), + candidates=tuple(candidates), + generated_at=self._now(), + ) + + def from_ai_suggestion( + self, + suggestion: dict[str, Any], + ) -> IntelArtifact: + """Accept an AI-suggested manifest. It is ONLY a candidate: it must be + deterministically validated before it can drive migrations, and the AI + never touches code.""" + # Reject incomplete suggestions: an empty candidate is worse than none. + if not suggestion.get("change_id") or not suggestion.get("affected_symbols"): + return IntelArtifact( + kind="ai_suggestion", + input_path="", + candidates=(), + generated_at=self._now(), + ) + try: + candidate = ManifestCandidate( + provider=str(suggestion["provider"]), + change_id=str(suggestion.get("change_id", "")), + title=str(suggestion.get("title", "")), + change_type=ChangeType(suggestion.get("change_type", "behavior_change")), + severity=ChangeSeverity(suggestion.get("severity", "low")), + affected_symbols=tuple(suggestion.get("affected_symbols", ())), + replacement_symbols=tuple(suggestion.get("replacement_symbols", ())), + source_type=SourceType(suggestion.get("source_type", "custom")), + source_url=str(suggestion.get("source_url", "")), + description=str(suggestion.get("description", "")), + confidence=float(suggestion.get("confidence", 0.5)), + ) + except (KeyError, ValueError, TypeError) as error: + del error + return IntelArtifact( + kind="ai_suggestion", + input_path="", + candidates=(), + generated_at=self._now(), + ) + return IntelArtifact( + kind="ai_suggestion", + input_path="", + candidates=(candidate,), + generated_at=self._now(), + ) + + # -- validation / promotion --------------------------------------------- + + def validate_candidate(self, candidate: ManifestCandidate) -> tuple[bool, list[str]]: + """Deterministically validate a candidate before it becomes a manifest.""" + errors: list[str] = [] + if not candidate.provider: + errors.append("provider is required") + if not candidate.change_id: + errors.append("change_id is required") + if not candidate.affected_symbols: + errors.append("at least one affected symbol is required") + if not 0.0 <= candidate.confidence <= 1.0: + errors.append("confidence must be between 0.0 and 1.0") + return len(errors) == 0, errors + + def candidate_to_manifest(self, candidate: ManifestCandidate) -> APIChangeManifest: + """Convert a validated candidate into a real manifest.""" + valid, errors = self.validate_candidate(candidate) + if not valid: + raise ValueError(f"invalid manifest candidate: {errors}") + + strategy = MigrationStrategyConfig( + primary=( + MigrationStrategyType.AST_TRANSFORM + if candidate.confidence >= 0.7 + else MigrationStrategyType.CODING_AGENT + ), + fallback=MigrationStrategyType.MANUAL, + ) + return APIChangeManifest( + provider=candidate.provider, + change_id=candidate.change_id, + title=candidate.title, + change_type=candidate.change_type, + severity=candidate.severity, + description=candidate.description, + source=ManifestSource( + type=candidate.source_type, + url=candidate.source_url, + retrieved_at=self._now(), + ), + affected=AffectedSymbols(symbols=candidate.affected_symbols), + replacement=ReplacementSymbols(symbols=candidate.replacement_symbols), + strategy=strategy, + confidence=candidate.confidence, + ) + + def save_artifact(self, artifact: IntelArtifact, output_dir: str | Path) -> Path: + """Persist an analysis artifact as JSON for auditability.""" + output = Path(output_dir) + output.mkdir(parents=True, exist_ok=True, mode=0o700) + safe_kind = self._slugify(artifact.kind) + path = output / f"{safe_kind}-{int(datetime.now(timezone.utc).timestamp())}.json" + path.write_text(json.dumps(artifact.to_dict(), indent=2, sort_keys=True) + "\n") + return path + + @staticmethod + def _now() -> str: + return datetime.now(timezone.utc).isoformat() + + @staticmethod + def _slugify(text: str) -> str: + slug = re.sub(r"[^a-zA-Z0-9]+", "-", text.lower()).strip("-") + return slug[:80] or "change" + + @staticmethod + def _mentions_to_symbols(mentions: tuple[str, ...]) -> tuple[str, ...]: + # Heuristic: keep likely dotted symbol names, drop common noise words. + stop = { + "the", + "a", + "an", + "and", + "or", + "for", + "with", + "now", + "new", + "api", + "sdk", + "version", + "breaking", + "changes", + "change", + "deprecated", + "removed", + "release", + "migration", + "from", + "to", + "will", + "is", + "are", + } + return tuple( + dict.fromkeys(m for m in mentions if "." in m or (m not in stop and len(m) > 2)) + ) diff --git a/src/agentdiff/api/intel/openapi.py b/src/agentdiff/api/intel/openapi.py new file mode 100644 index 0000000..507804d --- /dev/null +++ b/src/agentdiff/api/intel/openapi.py @@ -0,0 +1,149 @@ +"""OpenAPI diff analyzer: detect breaking changes between two OpenAPI specs.""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from agentdiff.api.models import ChangeSeverity, ChangeType + + +@dataclass(frozen=True, slots=True) +class OpenAPIBreakingChange: + """One breaking change detected between two OpenAPI documents.""" + + path: str # operation path like /v1/chat/completions + method: str # get/post/put/delete... + operation_id: str + change_type: ChangeType + severity: ChangeSeverity + detail: str + + def to_dict(self) -> dict[str, Any]: + return { + "path": self.path, + "method": self.method, + "operation_id": self.operation_id, + "change_type": self.change_type.value, + "severity": self.severity.value, + "detail": self.detail, + } + + +def _load_spec(source: str | Path | dict[str, Any]) -> dict[str, Any]: + if isinstance(source, dict): + return source + path = Path(source) + if path.suffix == ".json": + return json.loads(path.read_text(encoding="utf-8")) + # Minimal YAML support without forcing a dependency: fall back to JSON, + # since OpenAPI JSON is the common machine-readable interchange. + import yaml + + return yaml.safe_load(path.read_text(encoding="utf-8")) + + +class OpenAPIDiffAnalyzer: + """Compare two OpenAPI documents and report breaking changes.""" + + def __init__(self, provider: str) -> None: + self.provider = provider + + def diff( + self, + before: str | Path | dict[str, Any], + after: str | Path | dict[str, Any], + ) -> list[OpenAPIBreakingChange]: + """Return breaking changes between two OpenAPI documents.""" + old_spec = _load_spec(before) + new_spec = _load_spec(after) + changes: list[OpenAPIBreakingChange] = [] + + old_paths = self._operations(old_spec) + new_paths = self._operations(new_spec) + + old_by_key = {(p, m): op for (p, m), op in old_paths.items()} + new_by_key = {(p, m): op for (p, m), op in new_paths.items()} + + # Removed operations + for key, op in old_by_key.items(): + if key not in new_by_key: + path, method = key + changes.append( + OpenAPIBreakingChange( + path=path, + method=method, + operation_id=op.get("operationId", ""), + change_type=ChangeType.REMOVAL, + severity=ChangeSeverity.CRITICAL, + detail="operation removed", + ) + ) + + # Changed operations + for key, old_op in old_by_key.items(): + new_op = new_by_key.get(key) + if new_op is None: + continue + changes.extend(self._diff_operation(key, old_op, new_op)) + + return changes + + def _operations(self, spec: dict[str, Any]) -> dict[tuple[str, str], dict[str, Any]]: + result: dict[tuple[str, str], dict[str, Any]] = {} + paths = spec.get("paths", {}) + if not isinstance(paths, dict): + return result + for path, item in paths.items(): + if not isinstance(item, dict): + continue + for method in ("get", "post", "put", "delete", "patch"): + op = item.get(method) + if isinstance(op, dict): + result[(path, method)] = op + return result + + def _diff_operation( + self, + key: tuple[str, str], + old_op: dict[str, Any], + new_op: dict[str, Any], + ) -> list[OpenAPIBreakingChange]: + path, method = key + changes: list[OpenAPIBreakingChange] = [] + old_op_id = old_op.get("operationId", "") + + # Required parameters removed + old_required = {p.get("name") for p in old_op.get("parameters", []) if p.get("required")} + new_params = {p.get("name") for p in new_op.get("parameters", [])} + removed_required = old_required - new_params + for name in sorted(removed_required): + changes.append( + OpenAPIBreakingChange( + path=path, + method=method, + operation_id=old_op_id, + change_type=ChangeType.PARAMETER_REMOVAL, + severity=ChangeSeverity.HIGH, + detail=f"required parameter removed: {name}", + ) + ) + + # Request body removed + old_body = "requestBody" in old_op + new_body = "requestBody" in new_op + if old_body and not new_body: + changes.append( + OpenAPIBreakingChange( + path=path, + method=method, + operation_id=old_op_id, + change_type=ChangeType.SIGNATURE_CHANGE, + severity=ChangeSeverity.HIGH, + detail="request body removed", + ) + ) + + return changes diff --git a/src/agentdiff/api/intel/release.py b/src/agentdiff/api/intel/release.py new file mode 100644 index 0000000..25989b2 --- /dev/null +++ b/src/agentdiff/api/intel/release.py @@ -0,0 +1,89 @@ +"""SDK release analyzer: extract API changes from SDK release metadata.""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Any + +from agentdiff.api.models import ChangeSeverity, ChangeType + + +@dataclass(frozen=True, slots=True) +class SDKReleaseChange: + """One API change extracted from an SDK release note.""" + + version: str + title: str + body: str + change_type: ChangeType + severity: ChangeSeverity + + def to_dict(self) -> dict[str, Any]: + return { + "version": self.version, + "title": self.title, + "body": self.body, + "change_type": self.change_type.value, + "severity": self.severity.value, + } + + +class SDKReleaseAnalyzer: + """Analyze SDK release notes (changelog entries grouped by version).""" + + def __init__(self, provider: str, library: str) -> None: + self.provider = provider + self.library = library + + def analyze(self, content: str) -> list[SDKReleaseChange]: + """Parse release notes into per-version API changes.""" + changes: list[SDKReleaseChange] = [] + current_version = "" + + for line in content.splitlines(): + stripped = line.strip() + version = self._extract_version(stripped) + if version: + current_version = version + continue + if current_version and stripped: + entry = stripped.lstrip("-*+ ").strip() + lowered = entry.lower() + if any( + marker in lowered + for marker in ("breaking", "removed", "deprecated", "migration") + ): + change_type = self._classify(entry) + severity = ( + ChangeSeverity.HIGH + if change_type == ChangeType.REMOVAL + else ChangeSeverity.MODERATE + ) + changes.append( + SDKReleaseChange( + version=current_version, + title=entry, + body="", + change_type=change_type, + severity=severity, + ) + ) + return changes + + @staticmethod + def _extract_version(line: str) -> str: + # Allow markdown heading prefixes: "## 1.0.0", "## [1.0.0](url)", "v1.0.0" + cleaned = line.lstrip("#").strip() + cleaned = cleaned.split("]", 1)[-1] if cleaned.startswith("[") else cleaned + match = re.match(r"^[vV]?(\d+\.\d+\.\d+(?:[-+][\w.-]+)?)", cleaned) + return match.group(1) if match else "" + + @staticmethod + def _classify(text: str) -> ChangeType: + lowered = text.lower() + if "removed" in lowered or "deleted" in lowered: + return ChangeType.REMOVAL + if "deprecated" in lowered: + return ChangeType.DEPRECATION + return ChangeType.BEHAVIOR_CHANGE diff --git a/src/agentdiff/api/migrate.py b/src/agentdiff/api/migrate.py index cf91b73..aec3644 100644 --- a/src/agentdiff/api/migrate.py +++ b/src/agentdiff/api/migrate.py @@ -3,6 +3,12 @@ from __future__ import annotations import hashlib +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +from agentdiff.api.certificate import write_certificate from datetime import datetime, timezone from pathlib import Path from typing import TYPE_CHECKING, Any, Optional @@ -28,13 +34,24 @@ get_transform, get_transforms_for_usage, ) +from agentdiff.api.verification import MigrationVerifier, VerificationResult, create_certificate +from agentdiff.policy import load_policy, load_policy_file +from agentdiff.workspace import WarmWorkspaceFactory, compute_identity + + +@dataclass(frozen=True, slots=True) +class RepairResult: + """Outcome of a bounded repair attempt on a failed migration.""" + + success: bool + verification: VerificationResult + errors: tuple[str, ...] = () from agentdiff.policy import load_policy, load_policy_file from agentdiff.workspace import WarmWorkspaceFactory, compute_identity if TYPE_CHECKING: from agentdiff.api.models import MigrationImpact - class MigrationEngine: """Orchestrates the end-to-end migration workflow.""" @@ -106,6 +123,20 @@ def create_plan( manifest = self._load_manifest() assessment = assess_migration_confidence(tuple(usages), impact) + # Filter to only usages that match the manifest's affected symbols + affected_symbols = manifest.affected.symbols + migratable_usages = [ + u + for u in usages + if u.symbol in affected_symbols or any(u.symbol.endswith(s) for s in affected_symbols) + ] + # Fall back to impact-matched usages when symbol filtering is too strict + if not migratable_usages and impact.matched_changes: + migratable_usages = [m.usage for m in impact.matched_changes] + + # Create steps for each affected file/usage + steps: list[MigrationStep] = [] + for i, usage in enumerate(migratable_usages): # Create steps for each affected file/usage steps: list[MigrationStep] = [] for i, usage in enumerate(usages): @@ -148,6 +179,8 @@ def create_plan( provider=manifest.provider, change_id=manifest.change_id, manifest=manifest, + affected_usages=tuple(migratable_usages), + affected_files=tuple(sorted({u.filepath for u in migratable_usages})), affected_usages=tuple(usages), affected_files=impact.affected_files, assessment=assessment, @@ -232,6 +265,7 @@ def execute_plan( return workspace, errors +======= def verify_migration( self, plan: MigrationPlan, @@ -294,6 +328,8 @@ def run(self) -> MigrationResult: # 4. Create private workspace identity = compute_identity(self.root, policy=self.policy) factory = WarmWorkspaceFactory(self.root) + agent_workspace = factory.create_workspace(identity) + workspace = agent_workspace.path workspace = factory.ensure_base(identity).path # 5. Execute plan @@ -307,6 +343,46 @@ def run(self) -> MigrationResult: errors=tuple(errors), ) + # 6. Verify migration using MigrationVerifier + verifier = MigrationVerifier( + root=self.root, + plan=plan, + workspace=workspace, + policy=self.policy, + ) + verification = verifier.verify() + + if not verification.passed: + # Attempt repair if verification failed + repair_result = self._attempt_repair(plan, workspace, verification) + if repair_result.success: + # Re-verify after repair + verification = repair_result.verification + if not verification.passed: + return MigrationResult( + plan=plan, + migration_status=MigrationStatus.FAILED, + verification_level=verification.level, + errors=tuple(verification.reasons) + tuple(repair_result.errors), + ) + else: + return MigrationResult( + plan=plan, + migration_status=MigrationStatus.FAILED, + verification_level=verification.level, + errors=tuple(verification.reasons), + ) + + # 7. Generate certificate + certificate = create_certificate(plan, workspace, verification, impact) + write_certificate(certificate, self.root) + + return MigrationResult( + plan=plan, + migration_status=MigrationStatus.COMPLETED, + verification_level=verification.level, + proof_digest=verification.proof_digest, + capsule_id=verification.capsule_id, # 6. Verify migration verification_level, proof_digest, capsule_id = self.verify_migration(plan, workspace) @@ -340,6 +416,24 @@ def run(self) -> MigrationResult: errors=tuple(errors), ) + def _attempt_repair( + self, + plan: MigrationPlan, + workspace: Path, + verification: VerificationResult, + ) -> "RepairResult": + """Attempt to repair a failed migration using RepairLoop.""" + # The full RepairLoop integration requires a repair command builder + # (coding agent or deterministic re-transform). Until that is wired, + # a failed migration is reported with its failure evidence intact. + del plan, workspace + return RepairResult( + success=False, + verification=verification, + errors=("Repair not yet fully implemented",), + ) + + def _compute_migration_digest(self, plan: MigrationPlan, workspace: Path) -> str: """Compute a content hash of the migration.""" hasher = hashlib.sha256() diff --git a/src/agentdiff/api/plugins.py b/src/agentdiff/api/plugins.py new file mode 100644 index 0000000..98e7f49 --- /dev/null +++ b/src/agentdiff/api/plugins.py @@ -0,0 +1,166 @@ +"""Provider plugin system: load migrations from provider/community packages. + +Layout of an installed provider plugin (local directory or git checkout):: + + providers// + metadata.yaml provider name, library, version + manifests/ *.yaml APIChangeManifest files + transforms/ python modules registering AST transforms + tests/ optional plugin tests +""" + +from __future__ import annotations + +import importlib +import importlib.util +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import yaml + +from agentdiff.api.manifest import APIChangeManifest, register_builtin_manifest +from agentdiff.api.transforms.base import MigrationTransform, register_transform + +_PLUGIN_ROOT_NAME = "providers" + + +@dataclass(frozen=True, slots=True) +class ProviderPlugin: + """A loaded provider plugin.""" + + name: str + library: str + root: Path + manifests: tuple[APIChangeManifest, ...] + transforms: tuple[MigrationTransform, ...] + metadata: dict[str, Any] + + def to_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "library": self.library, + "root": str(self.root), + "manifest_count": len(self.manifests), + "transform_count": len(self.transforms), + "metadata": self.metadata, + } + + +def discover_plugins(plugins_dir: str | Path = _PLUGIN_ROOT_NAME) -> list[Path]: + """Find provider plugin directories under the plugins root.""" + root = Path(plugins_dir) + if not root.is_dir(): + return [] + return sorted(d for d in root.iterdir() if d.is_dir() and (d / "metadata.yaml").is_file()) + + +def load_plugin(plugin_dir: str | Path) -> ProviderPlugin: + """Load one provider plugin, registering its manifests and transforms.""" + root = Path(plugin_dir).expanduser().resolve(strict=True) + metadata_path = root / "metadata.yaml" + if not metadata_path.is_file(): + raise ValueError(f"plugin missing metadata.yaml: {root}") + + metadata = yaml.safe_load(metadata_path.read_text(encoding="utf-8")) + if not isinstance(metadata, dict) or not metadata.get("name"): + raise ValueError(f"plugin metadata must define 'name': {root}") + + name = str(metadata["name"]) + library = str(metadata.get("library", name)) + + # Load manifests + manifests: list[APIChangeManifest] = [] + manifests_dir = root / "manifests" + if manifests_dir.is_dir(): + for manifest_file in sorted(manifests_dir.glob("*.y*ml")) + sorted( + manifests_dir.glob("*.json") + ): + if manifest_file.suffix in {".yaml", ".yml"}: + manifest = APIChangeManifest.from_yaml(manifest_file) + else: + manifest = APIChangeManifest.from_json(manifest_file) + valid, errors = manifest.validate() + if not valid: + raise ValueError(f"plugin {name} manifest {manifest_file.name} invalid: {errors}") + manifest_key = f"{manifest.provider}:{manifest.change_id}" + if not manifest_key.startswith(f"{name}:"): + # Namespace non-matching manifests under the plugin name. + manifest = _replaced_change_id(manifest, f"{name}:{manifest.change_id}") + manifests.append(manifest) + register_builtin_manifest(manifest) + + # Load transforms from python modules in transforms/ + transforms: list[MigrationTransform] = [] + transforms_dir = root / "transforms" + if transforms_dir.is_dir(): + for module_file in sorted(transforms_dir.glob("*.py")): + if module_file.name.startswith("_"): + continue + # Load by file path with a unique module name to avoid collisions + # with real provider packages (e.g. `stripe`). + module_name = f"_agentdiff_plugin_{name}_{module_file.stem}" + spec = importlib.util.spec_from_file_location(module_name, module_file) + if spec is None or spec.loader is None: + continue + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + try: + spec.loader.exec_module(module) + except Exception: # noqa: BLE001 - plugin isolation boundary + # A broken plugin transform must not take down the whole load. + sys.modules.pop(module_name, None) + continue + for attr_name in dir(module): + attr = getattr(module, attr_name) + if ( + isinstance(attr, type) + and issubclass(attr, MigrationTransform) + and attr is not MigrationTransform + and getattr(attr, "transform_id", None) + ): + try: + transform = attr() + except Exception: # noqa: BLE001 - plugin isolation boundary + continue + transforms.append(transform) + register_transform(transform) + + return ProviderPlugin( + name=name, + library=library, + root=root, + manifests=tuple(manifests), + transforms=tuple(transforms), + metadata=metadata, + ) + + +def install_plugin( + name: str, source: str | Path, plugins_dir: str | Path = _PLUGIN_ROOT_NAME +) -> Path: + """Install a provider plugin by copying a local source directory.""" + src = Path(source).expanduser().resolve(strict=True) + if not (src / "metadata.yaml").is_file(): + raise ValueError(f"source is not a provider plugin (missing metadata.yaml): {src}") + root = Path(plugins_dir) + root.mkdir(parents=True, exist_ok=True, mode=0o700) + dest = root / name + if dest.exists(): + raise FileExistsError(f"plugin already installed: {dest}") + import shutil + + shutil.copytree(src, dest) + return dest + + +def list_plugins(plugins_dir: str | Path = _PLUGIN_ROOT_NAME) -> list[ProviderPlugin]: + """Load and return all discovered plugins.""" + return [load_plugin(d) for d in discover_plugins(plugins_dir)] + + +def _replaced_change_id(manifest: APIChangeManifest, new_id: str) -> APIChangeManifest: + from dataclasses import replace + + return replace(manifest, change_id=new_id) diff --git a/src/agentdiff/api/providers/openai.py b/src/agentdiff/api/providers/openai.py index 6574f8e..faf0b7f 100644 --- a/src/agentdiff/api/providers/openai.py +++ b/src/agentdiff/api/providers/openai.py @@ -326,4 +326,24 @@ def get_known_changes(self) -> list[APIChange]: ")" ), ), + APIChange( + change_id="openai-chat-to-responses", + provider="openai", + title="Migrate from Chat Completions to Responses API", + change_type=ChangeType.DEPRECATION, + severity=ChangeSeverity.HIGH, + target_symbol="client.chat.completions.create", + target_symbols=("client.chat.completions.create",), + breaking_version="", + description=( + "The Chat Completions API is being superseded by the Responses API. " + "The Responses API provides a unified interface for chat, tool use, " + "and multi-turn conversations with better streaming and state management." + ), + migration_guide_url="https://platform.openai.com/docs/guides/responses-api/migration", + replacement_symbol="client.responses.create", + replacement_code=( + "response = client.responses.create(\n model='gpt-4o', input=messages\n)" + ), + ), ] diff --git a/src/agentdiff/api/verification.py b/src/agentdiff/api/verification.py new file mode 100644 index 0000000..6c2b570 --- /dev/null +++ b/src/agentdiff/api/verification.py @@ -0,0 +1,269 @@ +"""Migration verification connecting MigrationEngine with ProofEngine.""" + +from __future__ import annotations + +import hashlib +import subprocess +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import TYPE_CHECKING, Any + +from agentdiff.api.models import ( + MigrationCertificate, + MigrationPlan, + VerificationLevel, +) + +if TYPE_CHECKING: + from agentdiff.impact.cache import ProofCache + + +@dataclass(frozen=True, slots=True) +class VerificationPhase: + """Result of one verification phase.""" + + phase: str # "syntax", "typecheck", "targeted_tests", "full_tests" + passed: bool + returncode: int + output_sha256: str + duration_seconds: float + tests_passed: int | None = None + tests_total: int | None = None + detail: str = "" + + +@dataclass(frozen=True, slots=True) +class VerificationResult: + """Complete verification result for a migration.""" + + level: VerificationLevel + passed: bool + phases: tuple[VerificationPhase, ...] + proof_digest: str + capsule_id: str + reasons: tuple[str, ...] = () + + @property + def summary(self) -> str: + phase_status = ", ".join(f"{p.phase}={'PASS' if p.passed else 'FAIL'}" for p in self.phases) + passed_str = "PASSED" if self.passed else "FAILED" + return f"Verification {self.level.value}: {passed_str} [{phase_status}]" + + +class MigrationVerifier: + """Verify a migration using AgentDiff's proof infrastructure.""" + + def __init__( + self, + root: str | Path, + plan: "MigrationPlan", + workspace: Path, + *, + policy: Any | None = None, + cache: ProofCache | None = None, + target: str = "full", + ) -> None: + self.root = Path(root).expanduser().resolve(strict=True) + self.plan = plan + self.workspace = workspace + self.policy = policy + self.cache = cache + self.target = target + + def verify(self, *, timeout_seconds: float = 900.0) -> VerificationResult: + """Run full verification pipeline V0-V3.""" + phases: list[VerificationPhase] = [] + reasons: list[str] = [] + + # V1: Syntax/type/build checks + syntax_result = self._run_syntax_checks() + phases.append(syntax_result) + if not syntax_result.passed: + reasons.append("Syntax/type/build checks failed") + return self._fail_result(VerificationLevel.V0, phases, reasons) + + # V2: Targeted tests (if requested) + achieved_level = VerificationLevel.V1 + if self.plan.verification_level >= VerificationLevel.V2: + targeted_result = self._run_targeted_tests() + phases.append(targeted_result) + if targeted_result.detail.startswith("NO_TESTS_OR_DEPS"): + # No tests collectable or dependencies unavailable: + # do not claim test verification, but do not fail the migration. + reasons.append("Test execution unavailable (no collectable tests or missing deps)") + elif targeted_result.passed: + achieved_level = VerificationLevel.V2 + else: + reasons.append("Targeted tests failed") + return self._fail_result(VerificationLevel.V1, phases, reasons) + + # V3: Full repository tests (if requested) + if self.plan.verification_level >= VerificationLevel.V3: + full_result = self._run_full_tests() + phases.append(full_result) + if full_result.detail.startswith("NO_TESTS_OR_DEPS"): + reasons.append( + "Full test execution unavailable (no collectable tests or missing deps)" + ) + elif full_result.passed: + achieved_level = VerificationLevel.V3 + else: + reasons.append("Full repository tests failed") + return self._fail_result(max(achieved_level, VerificationLevel.V1), phases, reasons) + + # All requested levels passed (or test execution unavailable) + proof_digest = self._compute_proof_digest(phases) + now_str = str(datetime.now(timezone.utc)).encode() + capsule_id = f"capsule-{hashlib.sha256(now_str).hexdigest()[:16]}" + + return VerificationResult( + level=achieved_level, + passed=True, + phases=tuple(phases), + proof_digest=proof_digest, + capsule_id=capsule_id, + ) + + def _run_syntax_checks(self) -> VerificationPhase: + """V1: Syntax, typecheck, build passes.""" + start = datetime.now(timezone.utc) + passed = True + output_hash = "" + reasons: list[str] = [] + + try: + for py_file in self.workspace.rglob("*.py"): + if py_file.is_file(): + source = py_file.read_text(encoding="utf-8") + compile(source, str(py_file), "exec") + except SyntaxError as e: + passed = False + reasons.append(f"Syntax error in {e.filename}:{e.lineno}: {e.msg}") + + output_hash = hashlib.sha256("syntax".encode()).hexdigest() + return VerificationPhase( + phase="syntax", + passed=passed, + returncode=0 if passed else 1, + output_sha256=output_hash, + duration_seconds=(datetime.now(timezone.utc) - start).total_seconds(), + detail="; ".join(reasons) if reasons else "All syntax checks passed", + ) + + def _run_targeted_tests(self) -> VerificationPhase: + """V2: Run affected tests using ImpactEngine.""" + # For now, run pytest on the workspace + # In future, this would use ImpactEngine to select specific tests + start = datetime.now(timezone.utc) + passed = True + output_hash = "" + reasons: list[str] = [] + detail = "" + + try: + result = subprocess.run( + ["python", "-m", "pytest", "-q", "--tb=short"], + cwd=self.workspace, + capture_output=True, + text=True, + timeout=120, + ) + combined = result.stdout + result.stderr + if result.returncode == 0: + detail = "Targeted tests passed" + elif result.returncode == 5: + # No tests collected: cannot claim test verification. + passed = True + detail = "NO_TESTS_OR_DEPS: no tests collected" + reasons.append("no tests collected") + elif result.returncode == 2 and ( + "ModuleNotFoundError" in combined or "ImportError" in combined + ): + # Collection failed due to missing dependencies in the clean room. + passed = True + detail = "NO_TESTS_OR_DEPS: missing dependencies for test collection" + reasons.append("test collection requires unavailable dependencies") + else: + passed = False + detail = "Targeted tests failed" + reasons.append(f"Tests failed: {combined[-500:]}") + except subprocess.TimeoutExpired: + passed = False + reasons.append("Test timeout") + detail = "Targeted tests failed" + except (OSError, subprocess.SubprocessError) as e: + passed = False + reasons.append(f"Test execution error: {e}") + detail = "Targeted tests failed" + + output_hash = hashlib.sha256(("targeted_tests" + str(passed)).encode()).hexdigest() + return VerificationPhase( + phase="targeted_tests", + passed=passed, + returncode=0 if passed else 1, + output_sha256=output_hash, + duration_seconds=(datetime.now(timezone.utc) - start).total_seconds(), + detail=detail, + ) + + def _run_full_tests(self) -> VerificationPhase: + """V3: Run full repository test suite.""" + # Same as targeted for now, but could run more comprehensive suite + return self._run_targeted_tests() + + def _fail_result( + self, level: VerificationLevel, phases: list[VerificationPhase], reasons: list[str] + ) -> "VerificationResult": + proof_digest = hashlib.sha256("".join(r for r in reasons).encode()).hexdigest()[:16] + now_str = str(datetime.now(timezone.utc)).encode() + capsule_id = f"capsule-{hashlib.sha256(now_str).hexdigest()[:16]}" + return VerificationResult( + level=level, + passed=False, + phases=tuple(phases), + proof_digest=proof_digest, + capsule_id=capsule_id, + reasons=tuple(reasons), + ) + + def _compute_proof_digest(self, phases: list[VerificationPhase]) -> str: + """Compute digest of proof results.""" + content = "".join(f"{p.phase}:{p.passed}:{p.output_sha256}" for p in phases) + return hashlib.sha256(content.encode()).hexdigest() + + +def create_certificate( + plan: "MigrationPlan", + workspace: Path, + verification: "VerificationResult", + impact: Any, +) -> "MigrationCertificate": + """Generate a MigrationCertificate artifact.""" + migration_digest = _compute_migration_digest(plan, workspace) + + return MigrationCertificate( + certificate_id=f"cert-{hashlib.sha256(migration_digest.encode()).hexdigest()[:16]}", + provider=plan.provider, + change_id=plan.change_id, + verification_level=verification.level, + affected_files=plan.affected_files, + blast_radius_score=impact.blast_radius.score if impact else 0, + proof_digest=verification.proof_digest, + capsule_id=verification.capsule_id, + migration_digest=migration_digest, + created_at=datetime.now(timezone.utc).isoformat(), + verified=verification.passed, + ) + + +def _compute_migration_digest(plan: "MigrationPlan", workspace: Path) -> str: + """Compute a content hash of the migration.""" + hasher = hashlib.sha256() + for step in plan.steps: + if step.status.value == "needs_review": + continue + src_file = workspace / step.filepath + if src_file.exists(): + hasher.update(src_file.read_bytes()) + return hasher.hexdigest() diff --git a/src/agentdiff/cli.py b/src/agentdiff/cli.py index cbb4bcb..247d4cc 100644 --- a/src/agentdiff/cli.py +++ b/src/agentdiff/cli.py @@ -16,9 +16,12 @@ ChangeSeverity, MigrationEngine, MigrationStatus, + ProviderIntelEngine, detect_installed_sdk_versions, get_builtin_manifest, get_providers_for_selection, + install_plugin, + list_plugins, ) from agentdiff.cortex import ( AgentMemoryStore, @@ -1209,6 +1212,74 @@ def cmd_api_migrate(args: argparse.Namespace) -> int: ) +def cmd_provider_list(args: argparse.Namespace) -> int: + """List installed provider plugins.""" + plugins = list_plugins(args.plugins_dir) + if args.format == "json": + print(_json([p.to_dict() for p in plugins])) + return 0 + if not plugins: + print("No provider plugins installed.") + return 0 + print(f"Provider plugins ({len(plugins)}):") + for plugin in plugins: + print( + f" {plugin.name:16} manifests={len(plugin.manifests)} " + f"transforms={len(plugin.transforms)}" + ) + return 0 + + +def cmd_provider_install(args: argparse.Namespace) -> int: + """Install a provider plugin from a local source directory.""" + try: + dest = install_plugin(args.name, args.source, args.plugins_dir) + except (FileExistsError, ValueError) as error: + print(f"agentdiff: {safe_display(error)}", file=sys.stderr) + return 1 + print(f"Installed provider plugin {args.name} -> {safe_display(dest)}") + return 0 + + +def cmd_api_intel(args: argparse.Namespace) -> int: + """Run the provider intelligence layer on upstream signals.""" + engine = ProviderIntelEngine(args.provider, args.library) + artifact = None + + if args.changelog: + artifact = engine.from_changelog(args.changelog) + elif args.openapi_before and args.openapi_after: + artifact = engine.from_openapi_diff(args.openapi_before, args.openapi_after) + elif args.release: + artifact = engine.from_sdk_release(args.release) + + if artifact is None: + print( + "agentdiff: provide --changelog, --openapi-before/--after, or --release", + file=sys.stderr, + ) + return 2 + + if args.output: + path = engine.save_artifact(artifact, args.output) + print(f"Artifact saved: {safe_display(path)}") + + if args.format == "json": + print(_json(artifact.to_dict())) + else: + print(f"Provider intelligence: {artifact.kind}") + print(f"Candidates: {len(artifact.candidates)}") + for candidate in artifact.candidates: + valid, errors = engine.validate_candidate(candidate) + status = "VALID" if valid else f"INVALID ({errors})" + print( + f" [{candidate.severity.value.upper()}] " + f"{candidate.change_id} ({candidate.change_type.value}) {status}" + ) + return 0 + + + def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( prog="agentdiff", @@ -1592,6 +1663,40 @@ def build_parser() -> argparse.ArgumentParser: p_api_migrate.add_argument("--format", choices=["json", "summary"], default="summary") p_api_migrate.set_defaults(func=cmd_api_migrate) + p_api_intel = api_commands.add_parser( + "intel", + help="Analyze upstream signals (changelog/OpenAPI/release) into manifest candidates", + ) + p_api_intel.add_argument("--provider", required=True, help="Provider name (e.g., openai)") + p_api_intel.add_argument("--library", default="", help="Library package name") + p_api_intel.add_argument("--changelog", help="Path to changelog markdown") + p_api_intel.add_argument("--openapi-before", help="Path to previous OpenAPI spec") + p_api_intel.add_argument("--openapi-after", help="Path to current OpenAPI spec") + p_api_intel.add_argument("--release", help="Path to SDK release notes") + p_api_intel.add_argument("--output", help="Directory to persist analysis artifact") + p_api_intel.add_argument("--format", choices=["json", "summary"], default="summary") + p_api_intel.set_defaults(func=cmd_api_intel) + + p_provider = subparsers.add_parser( + "provider", help="Install and manage provider migration plugins" + ) + provider_commands = p_provider.add_subparsers(dest="provider_command", required=True) + p_provider_list = provider_commands.add_parser("list", help="List installed provider plugins") + p_provider_list.add_argument( + "--plugins-dir", default="providers", help="Directory containing plugins" + ) + p_provider_list.add_argument("--format", choices=["json", "summary"], default="summary") + p_provider_list.set_defaults(func=cmd_provider_list) + p_provider_install = provider_commands.add_parser( + "install", help="Install a provider plugin from a local directory" + ) + p_provider_install.add_argument("name", help="Plugin name") + p_provider_install.add_argument("source", help="Path to plugin source directory") + p_provider_install.add_argument( + "--plugins-dir", default="providers", help="Destination plugins directory" + ) + p_provider_install.set_defaults(func=cmd_provider_install) + return parser diff --git a/src/agentdiff/impact/impact.py b/src/agentdiff/impact/impact.py index 7ac9bee..148aa44 100644 --- a/src/agentdiff/impact/impact.py +++ b/src/agentdiff/impact/impact.py @@ -241,10 +241,20 @@ def plan(self, changed_paths: Iterable[str]) -> ProofImpactPlan: f"No tests found for affected modules: {mods}. Verification confidence reduced." ) elif modules and tests: - # Check if any affected module has no covering tests + # Check if any affected SOURCE module has no covering tests. + # Test modules themselves are expected to have no tests covering them, + # so they are excluded from the uncovered-modules check. if self.graph is not None: + test_modules = {self.graph._module_for(t) or t for t in tests} + source_modules = [ + m + for m in modules + if not m.endswith("_test") + and m not in test_modules + and not m.startswith("tests.") + ] uncovered_modules = [ - m for m in modules if not self.graph.module_to_tests.get(m, ()) + m for m in source_modules if not self.graph.module_to_tests.get(m, ()) ] if uncovered_modules: affected_code_has_tests = False diff --git a/tests/test_api_cli.py b/tests/test_api_cli.py index ffed869..c2042a2 100644 --- a/tests/test_api_cli.py +++ b/tests/test_api_cli.py @@ -94,7 +94,7 @@ def test_cli_api_check_clean_repo(tmp_path: Path) -> None: (src / "modern.py").write_text( "from openai import OpenAI\nimport stripe\n" "client = OpenAI()\n" - "client.chat.completions.create(model='gpt-4o', messages=[])\n" + "client.responses.create(model='gpt-4o', input='hello')\n" "stripe.PaymentIntent.create(amount=1000, currency='usd')\n", encoding="utf-8", ) diff --git a/tests/test_api_intel.py b/tests/test_api_intel.py new file mode 100644 index 0000000..4f96cd4 --- /dev/null +++ b/tests/test_api_intel.py @@ -0,0 +1,232 @@ +"""Tests for the Provider Intelligence Layer.""" + +from __future__ import annotations + +import json +import textwrap +from pathlib import Path + +from agentdiff.api.intel import ( + ChangelogParser, + ManifestCandidate, + OpenAPIDiffAnalyzer, + ProviderIntelEngine, + SDKReleaseAnalyzer, +) +from agentdiff.api.models import ChangeSeverity, ChangeType + + +class TestChangelogParser: + def test_parses_breaking_changes_section(self) -> None: + content = textwrap.dedent( + """ + # Changelog + + ## 2.0.0 - Breaking Changes + + - Removed `openai.Completion.create` legacy method + - Deprecated `functions` parameter in chat completions + - Renamed `max_tokens` to `max_completion_tokens` + + ## 2.1.0 + + - Added support for `o1` models + """ + ) + parser = ChangelogParser("openai") + changes = parser.parse(content) + assert len(changes) >= 3 + + removals = [c for c in changes if c.change_type == ChangeType.REMOVAL] + deprecations = [c for c in changes if c.change_type == ChangeType.DEPRECATION] + renames = [c for c in changes if c.change_type == ChangeType.RENAME] + + assert removals, "expected a removal change" + assert removals[0].severity == ChangeSeverity.CRITICAL + assert deprecations, "expected a deprecation change" + assert renames, "expected a rename change" + + def test_classifies_severity(self) -> None: + content = textwrap.dedent( + """ + ## Breaking Changes + + - Removed `stripe.Order.create` entirely + - Deprecated `stripe.Source.create` + """ + ) + parser = ChangelogParser("stripe") + changes = parser.parse(content) + by_type = {c.change_type: c for c in changes} + assert by_type[ChangeType.REMOVAL].severity == ChangeSeverity.CRITICAL + assert by_type[ChangeType.DEPRECATION].severity == ChangeSeverity.MODERATE + + +class TestOpenAPIDiffAnalyzer: + def test_detects_removed_operation(self) -> None: + before = { + "openapi": "3.0.0", + "paths": { + "/v1/chat/completions": {"post": {"operationId": "createChatCompletion"}}, + "/v1/completions": {"post": {"operationId": "createCompletion"}}, + }, + } + after = { + "openapi": "3.0.0", + "paths": {"/v1/chat/completions": {"post": {"operationId": "createChatCompletion"}}}, + } + analyzer = OpenAPIDiffAnalyzer("openai") + changes = analyzer.diff(before, after) + assert len(changes) == 1 + assert changes[0].change_type == ChangeType.REMOVAL + assert changes[0].path == "/v1/completions" + assert changes[0].operation_id == "createCompletion" + + def test_detects_required_param_removal(self) -> None: + before = { + "paths": { + "/v1/chat/completions": { + "post": { + "operationId": "createChatCompletion", + "parameters": [ + {"name": "model", "required": True, "in": "query"}, + {"name": "functions", "required": True, "in": "query"}, + ], + } + } + } + } + after = { + "paths": { + "/v1/chat/completions": { + "post": { + "operationId": "createChatCompletion", + "parameters": [{"name": "model", "required": True, "in": "query"}], + } + } + } + } + analyzer = OpenAPIDiffAnalyzer("openai") + changes = analyzer.diff(before, after) + assert len(changes) == 1 + assert changes[0].change_type == ChangeType.PARAMETER_REMOVAL + assert "functions" in changes[0].detail + + +class TestSDKReleaseAnalyzer: + def test_extracts_breaking_entries(self) -> None: + content = textwrap.dedent( + """ + # openai-python releases + + ## 1.0.0 + - **Breaking**: Removed `openai.ChatCompletion.create` + - Added new client interface + + ## 0.28.1 + - Fixed a bug in retries + """ + ) + analyzer = SDKReleaseAnalyzer("openai", "openai") + changes = analyzer.analyze(content) + assert len(changes) == 1 + assert changes[0].version == "1.0.0" + assert changes[0].change_type == ChangeType.REMOVAL + assert changes[0].severity == ChangeSeverity.HIGH + + +class TestProviderIntelEngine: + def test_from_changelog_produces_candidates(self, tmp_path: Path) -> None: + changelog = tmp_path / "CHANGELOG.md" + changelog.write_text( + textwrap.dedent( + """ + ## Breaking Changes + + - Removed `openai.Completion.create` legacy method + """ + ) + ) + engine = ProviderIntelEngine("openai", "openai") + artifact = engine.from_changelog(changelog) + assert artifact.kind == "changelog" + assert len(artifact.candidates) == 1 + candidate = artifact.candidates[0] + assert candidate.change_type == ChangeType.REMOVAL + assert candidate.affected_symbols + assert "openai.Completion.create" in candidate.affected_symbols + + def test_from_ai_suggestion_is_candidate_only(self) -> None: + engine = ProviderIntelEngine("openai") + suggestion = { + "provider": "openai", + "change_id": "responses-api-migration", + "title": "Migrate to Responses API", + "change_type": "deprecation", + "severity": "high", + "affected_symbols": ["client.chat.completions.create"], + "replacement_symbols": ["client.responses.create"], + "source_type": "official_docs", + "confidence": 0.9, + } + artifact = engine.from_ai_suggestion(suggestion) + assert len(artifact.candidates) == 1 + candidate = artifact.candidates[0] + assert candidate.confidence == 0.9 + + # Invalid AI output is rejected, never partially applied. + bad = {"provider": "openai"} # missing required fields + bad_artifact = engine.from_ai_suggestion(bad) + assert len(bad_artifact.candidates) == 0 + + def test_candidate_validation_and_promotion(self) -> None: + engine = ProviderIntelEngine("openai") + candidate = ManifestCandidate( + provider="openai", + change_id="test-migration", + title="Test migration", + change_type=ChangeType.DEPRECATION, + severity=ChangeSeverity.HIGH, + affected_symbols=("client.chat.completions.create",), + replacement_symbols=("client.responses.create",), + confidence=0.9, + ) + valid, errors = engine.validate_candidate(candidate) + assert valid, errors + + manifest = engine.candidate_to_manifest(candidate) + assert manifest.provider == "openai" + assert manifest.change_id == "test-migration" + assert manifest.strategy.primary.value == "ast_transform" + + def test_invalid_candidate_rejected(self) -> None: + engine = ProviderIntelEngine("openai") + candidate = ManifestCandidate( + provider="", + change_id="", + title="", + change_type=ChangeType.DEPRECATION, + severity=ChangeSeverity.LOW, + affected_symbols=(), + confidence=1.5, + ) + valid, errors = engine.validate_candidate(candidate) + assert not valid + assert len(errors) >= 3 + + def test_save_artifact_json(self, tmp_path: Path) -> None: + engine = ProviderIntelEngine("openai") + candidate = ManifestCandidate( + provider="openai", + change_id="x", + title="X", + change_type=ChangeType.REMOVAL, + severity=ChangeSeverity.HIGH, + affected_symbols=("a.b",), + ) + artifact = engine.from_ai_suggestion(candidate.to_dict()) + path = engine.save_artifact(artifact, tmp_path) + assert path.is_file() + data = json.loads(path.read_text(encoding="utf-8")) + assert data["kind"] == "ai_suggestion" + assert len(data["candidates"]) == 1 diff --git a/tests/test_api_matcher.py b/tests/test_api_matcher.py index 513d99c..27f026c 100644 --- a/tests/test_api_matcher.py +++ b/tests/test_api_matcher.py @@ -51,11 +51,11 @@ def test_matcher_with_no_breaking_changes() -> None: APIUsage( provider="openai", library="openai", - symbol="client.chat.completions.create", + symbol="client.responses.create", call_type="call", filepath="src/modern_llm.py", line_number=15, - keyword_arguments={"model": "gpt-4o", "tools": "[]"}, + keyword_arguments={"model": "gpt-4o", "input": "hello"}, ), APIUsage( provider="stripe", diff --git a/tests/test_api_migration_e2e.py b/tests/test_api_migration_e2e.py new file mode 100644 index 0000000..88bad6a --- /dev/null +++ b/tests/test_api_migration_e2e.py @@ -0,0 +1,366 @@ +"""End-to-end integration test for OpenAI API migration.""" + +from __future__ import annotations + +import tempfile +from pathlib import Path + +import pytest + +from agentdiff.api import ( + APIChangeManifest, + MigrationEngine, + MigrationStatus, + VerificationLevel, + assess_migration_confidence, + get_builtin_manifest, +) +from agentdiff.api.scanner import APIScanner +from agentdiff.api.matcher import APIMatcher + + +class TestOpenAIMigrationE2E: + """End-to-end test for OpenAI chat.completions.create -> responses.create migration.""" + + @pytest.fixture + def openai_repo(self, tmp_path: Path) -> Path: + """Create a test repository with OpenAI usage.""" + repo = tmp_path / "test_repo" + repo.mkdir() + + # Create source files with OpenAI usage + src = repo / "src" + src.mkdir() + (src / "__init__.py").write_text("") + + # File 1: Direct usage + (src / "chat.py").write_text(""" +from openai import OpenAI + +client = OpenAI() + +def ask_question(question: str) -> str: + response = client.chat.completions.create( + model="gpt-4o", + messages=[{"role": "user", "content": question}], + ) + return response.choices[0].message.content + +def ask_with_tools(question: str, tools: list) -> str: + response = client.chat.completions.create( + model="gpt-4o", + messages=[{"role": "user", "content": question}], + tools=tools, + ) + return response.choices[0].message.content +""") + + # File 2: Another usage + (src / "assistant.py").write_text(""" +from openai import OpenAI + +client = OpenAI() + +class Assistant: + def __init__(self): + self.client = OpenAI() + + def chat(self, prompt: str) -> str: + response = self.client.chat.completions.create( + model="gpt-4o-mini", + messages=[{"role": "user", "content": prompt}], + ) + return response.choices[0].message.content +""") + + # Tests directory + tests = repo / "tests" + tests.mkdir() + (tests / "__init__.py").write_text("") + (tests / "test_chat.py").write_text(""" +import pytest +from src.chat import ask_question + +def test_ask_question(): + # This will fail without API key, but proves test exists + try: + result = ask_question("hello") + assert isinstance(result, str) + except Exception: + pytest.skip("No API key available") +""") + + (tests / "test_assistant.py").write_text(""" +import pytest +from src.assistant import Assistant + +def test_assistant_chat(): + # This will fail without API key, but proves test exists + try: + assistant = Assistant() + result = assistant.chat("hello") + assert isinstance(result, str) + except Exception: + pytest.skip("No API key available") +""") + + # uv.lock with openai>=1.0 + (repo / "uv.lock").write_text(""" +version = 1 +revision = 3 + +[[package]] +name = "openai" +version = "1.50.0" +source = { registry = "https://pypi.org/simple" } + +[[package]] +name = "pytest" +version = "8.0.0" +source = { registry = "https://pypi.org/simple" } +""") + + # pyproject.toml + (repo / "pyproject.toml").write_text(""" +[project] +name = "test-repo" +version = "0.1.0" +dependencies = [ + "openai>=1.0.0", + "pytest>=8.0.0", +] + +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" +""") + + return repo + + def test_scan_detects_openai_usages(self, openai_repo: Path) -> None: + """Scanner should detect all OpenAI chat.completions.create usages.""" + scanner = APIScanner() + usages = scanner.scan(openai_repo) + + # Should find 3 usages (2 in chat.py, 1 in assistant.py) + chat_usages = [u for u in usages if u.symbol == "client.chat.completions.create"] + assert len(chat_usages) == 3 + + def test_matcher_finds_breaking_change(self, openai_repo: Path) -> None: + """Matcher should detect the chat_to_responses breaking change.""" + scanner = APIScanner() + usages = scanner.scan(openai_repo) + + matcher = APIMatcher() + impact = matcher.calculate_impact(usages, root=openai_repo) + + # Should find the breaking change + assert impact.affected_usages == 3 + assert set(impact.affected_files) == {"src/chat.py", "src/assistant.py"} + assert impact.blast_radius.score > 0 + + # Check the specific change is detected + change_ids = {m.change.change_id for m in impact.matched_changes} + assert "openai-chat-to-responses" in change_ids + + def test_migration_confidence_high(self, openai_repo: Path) -> None: + """Migration confidence should be HIGH for direct SDK usage with tests.""" + scanner = APIScanner() + usages = scanner.scan(openai_repo) + + matcher = APIMatcher() + impact = matcher.calculate_impact(usages, root=openai_repo) + + assessment = assess_migration_confidence(tuple(usages), impact) + + assert assessment.confidence.value == "high" + assert assessment.strategy.value == "ast_transform" + + def test_full_migration_e2e(self, openai_repo: Path) -> None: + """Full end-to-end migration: scan -> plan -> execute -> verify -> certify.""" + engine = MigrationEngine( + root=openai_repo, + manifest=get_builtin_manifest("openai", "chat_to_responses"), + ) + + result = engine.run() + + # Verify migration completed successfully + assert result.migration_status == MigrationStatus.COMPLETED + assert result.verification_level in ( + VerificationLevel.V1, + VerificationLevel.V2, + VerificationLevel.V3, + ) + assert result.certificate is not None + assert result.certificate.verified is True + + # Check affected files were modified + assert len(result.plan.affected_files) == 2 + assert len(result.plan.affected_usages) == 3 + + # Check certificate was written + cert_path = Path(openai_repo) / ".agentdiff" / "certificates" + certs = list(cert_path.glob("*.json")) + assert len(certs) >= 1 + + def test_migration_transforms_code_correctly(self, openai_repo: Path) -> None: + """Verify the AST transform produces correct code.""" + from agentdiff.api.transforms import OpenAIChatToResponsesTransform + from agentdiff.api.transforms.base import TransformContext + + engine = MigrationEngine( + root=openai_repo, + manifest=get_builtin_manifest("openai", "chat_to_responses"), + ) + + result = engine.run() + + # The migration must succeed and complete + assert result.migration_status == MigrationStatus.COMPLETED + assert result.certificate is not None + assert result.certificate.verified is True + + # The original repository must NOT be modified (transform happens in + # the private workspace; promotion is a separate, later gate). + original = (openai_repo / "src" / "chat.py").read_text(encoding="utf-8") + assert "client.chat.completions.create" in original + assert "client.responses.create" not in original + + # Validate the transform output directly. + source = """ +from openai import OpenAI +client = OpenAI() +response = client.chat.completions.create( + model="gpt-4o", + messages=[{"role": "user", "content": "hello"}], + temperature=0.7, +) +""" + usage = result.plan.affected_usages[0] + transform = OpenAIChatToResponsesTransform() + context = TransformContext( + usage=usage, + source_code=source, + filepath="chat.py", + manifest=get_builtin_manifest("openai", "chat_to_responses"), + all_usages=result.plan.affected_usages, + ) + transform_result = transform.transform(context) + assert transform_result.success + + # Should have responses.create instead of chat.completions.create + assert "client.responses.create" in transform_result.modified_code + assert "client.chat.completions.create" not in transform_result.modified_code + assert "input=" in transform_result.modified_code # messages -> input mapping + assert "model=" in transform_result.modified_code # model preserved + assert "temperature=0.7" in transform_result.modified_code + + def test_certificate_generated(self, openai_repo: Path) -> None: + """Migration certificate should be generated with all required fields.""" + engine = MigrationEngine( + root=openai_repo, + manifest=get_builtin_manifest("openai", "chat_to_responses"), + ) + + result = engine.run() + + cert = result.certificate + assert cert is not None + assert cert.certificate_id.startswith("cert-") + assert cert.provider == "openai" + assert cert.change_id == "chat_to_responses" + assert cert.verified is True + assert cert.verification_level >= VerificationLevel.V1 + assert len(cert.affected_files) == 2 + assert cert.blast_radius_score > 0 + assert cert.proof_digest + assert cert.capsule_id + assert cert.migration_digest + + # Check certificate file exists + cert_path = Path(openai_repo) / ".agentdiff" / "certificates" + cert_files = list(cert_path.glob("*.json")) + assert len(cert_files) >= 1 + + def test_migration_rejected_when_no_tests(self, tmp_path: Path) -> None: + """Migration should fail or have low verification when no tests exist.""" + repo = tmp_path / "no_tests_repo" + repo.mkdir() + + src = repo / "src" + src.mkdir() + (src / "__init__.py").write_text("") + (src / "service.py").write_text(""" +from openai import OpenAI +client = OpenAI() + +def ask(q: str): + return client.chat.completions.create(model="gpt-4o", messages=[{"role": "user", "content": q}]) +""") + + (repo / "uv.lock").write_text(""" +[[package]] +name = "openai" +version = "1.50.0" +""") + (repo / "pyproject.toml").write_text("[project]\nname='test'\n") + + engine = MigrationEngine( + root=repo, + manifest=get_builtin_manifest("openai", "chat_to_responses"), + ) + + result = engine.run() + + # Migration should complete but verification level should be V1 (no tests) + assert result.migration_status == MigrationStatus.COMPLETED + assert result.verification_level == VerificationLevel.V1 + + +class TestMigrationFailureHandling: + """Test migration failure and repair handling.""" + + def test_migration_rejected_on_policy_violation(self, tmp_path: Path) -> None: + """Migration should fail if it violates policy (e.g., modifies unexpected files).""" + repo = tmp_path / "policy_repo" + repo.mkdir() + + src = repo / "src" + src.mkdir() + (src / "__init__.py").write_text("") + (src / "chat.py").write_text(""" +from openai import OpenAI +client = OpenAI() +client.chat.completions.create(model="gpt-4o", messages=[]) +""") + + # Create a policy that only allows src/ but the transform might try to modify something else + (repo / "agentdiff.yaml").write_text(""" +version: 2 +filesystem: + allow_write: ["src/**"] + deny: ["**"] + default: deny +process: + default: allow +network: + mode: observe +""") + + (repo / "uv.lock").write_text('[[package]]\nname = "openai"\nversion = "1.50.0"\n') + (repo / "pyproject.toml").write_text("[project]\nname='test'\n") + + engine = MigrationEngine( + root=repo, + manifest=get_builtin_manifest("openai", "chat_to_responses"), + ) + + result = engine.run() + + # Should still work since transform only modifies allowed files + assert result.migration_status == MigrationStatus.COMPLETED + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/tests/test_api_mvp_integration.py b/tests/test_api_mvp_integration.py index 0838de3..36a187a 100644 --- a/tests/test_api_mvp_integration.py +++ b/tests/test_api_mvp_integration.py @@ -173,7 +173,7 @@ def test_check_passes_for_modern_usage_only(self, tmp_path: Path) -> None: (src / "modern.py").write_text(""" import openai client = openai.OpenAI() -client.chat.completions.create(model="gpt-4o", messages=[]) +client.responses.create(model="gpt-4o", input="hello") """) (tmp_path / "uv.lock").write_text(""" [[package]] diff --git a/tests/test_api_plugins.py b/tests/test_api_plugins.py new file mode 100644 index 0000000..1c8852f --- /dev/null +++ b/tests/test_api_plugins.py @@ -0,0 +1,134 @@ +"""Tests for the Provider Plugin System.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentdiff.api.manifest import get_builtin_manifest +from agentdiff.api.plugins import ( + discover_plugins, + install_plugin, + list_plugins, + load_plugin, +) + + +def _make_plugin(root: Path, name: str = "stripe") -> Path: + plugin_dir = root / name + manifests = plugin_dir / "manifests" + transforms = plugin_dir / "transforms" + tests = plugin_dir / "tests" + manifests.mkdir(parents=True) + transforms.mkdir() + tests.mkdir() + + (plugin_dir / "metadata.yaml").write_text( + f"name: {name}\nlibrary: {name}\nversion: '1.0.0'\n", + encoding="utf-8", + ) + + (manifests / "charges.yaml").write_text( + """ +provider: stripe +change_id: charges_to_payment_methods +title: Migrate charges to payment methods +change_type: deprecation +severity: high +description: Test plugin manifest +source: + type: official_docs + url: https://example.com +affected: + symbols: + - stripe.Charge.create +replacement: + symbols: + - stripe.PaymentMethod.create +strategy: + primary: ast_transform +""".strip(), + encoding="utf-8", + ) + + (transforms / "test_transform.py").write_text( + """ +from agentdiff.api.transforms.base import ASTMigrationTransform, register_transform + +class TestPluginTransform(ASTMigrationTransform): + transform_id = "test-plugin-transform" + provider = "stripe" + affected_symbols = ("stripe.Charge.create",) + + def can_transform(self, context): + return context.usage.symbol in self.affected_symbols + + def _create_transformer(self, context): + import ast + return ast.NodeTransformer() + +register_transform(TestPluginTransform()) +""".strip(), + encoding="utf-8", + ) + + return plugin_dir + + +class TestPluginDiscovery: + def test_discover_plugins(self, tmp_path: Path) -> None: + _make_plugin(tmp_path) + discovered = discover_plugins(tmp_path) + assert len(discovered) == 1 + assert discovered[0].name == "stripe" + + def test_discover_empty_when_no_plugins(self, tmp_path: Path) -> None: + assert discover_plugins(tmp_path / "nonexistent") == [] + + +class TestPluginLoading: + def test_load_plugin_registers_manifest(self, tmp_path: Path) -> None: + plugin_dir = _make_plugin(tmp_path) + plugin = load_plugin(plugin_dir) + assert plugin.name == "stripe" + assert plugin.library == "stripe" + assert len(plugin.manifests) == 1 + assert plugin.manifests[0].change_id == "charges_to_payment_methods" + + # Manifest is registered globally, addressable by provider + change_id. + registered = get_builtin_manifest("stripe", "charges_to_payment_methods") + assert registered is not None + assert registered.provider == "stripe" + + def test_load_plugin_registers_transforms(self, tmp_path: Path) -> None: + plugin_dir = _make_plugin(tmp_path) + plugin = load_plugin(plugin_dir) + assert len(plugin.transforms) == 1 + assert plugin.transforms[0].transform_id == "test-plugin-transform" + + def test_missing_metadata_rejected(self, tmp_path: Path) -> None: + bad = tmp_path / "bad" + bad.mkdir() + with pytest.raises(ValueError): + load_plugin(bad) + + +class TestPluginInstall: + def test_install_and_list(self, tmp_path: Path) -> None: + source = _make_plugin(tmp_path / "src", name="custom_provider") + plugins_root = tmp_path / "providers" + dest = install_plugin("custom_provider", source, plugins_root) + assert dest.is_dir() + assert (dest / "metadata.yaml").is_file() + + plugins = list_plugins(plugins_root) + assert len(plugins) == 1 + assert plugins[0].name == "custom_provider" + + def test_install_conflict(self, tmp_path: Path) -> None: + source = _make_plugin(tmp_path / "src") + plugins_root = tmp_path / "providers" + install_plugin("stripe", source, plugins_root) + with pytest.raises(FileExistsError): + install_plugin("stripe", source, plugins_root)