⚡ PromptDiff

Fast, CI-Native Regression Testing for LLM Prompts ("Git Diff for Prompts")

Catch silent quality regressions, schema breakages, latency spikes, and cost inflation before merging prompt changes.

🚀 Quickstart • 🔍 Core Workflow • 🩺 Environment Doctor • 🐍 Python SDK • 📚 Recipe Catalog • 🧪 Pytest Integration • 📦 Installation

---

💡 Why PromptDiff?

Modifying system prompts or switching models often leads to unexpected side effects: broken JSON formatting, subtle hallucinations, increased latency, or ballooning token costs.

promptdiff brings standard software regression testing to prompt engineering:

• CLI & CI/CD First: Run lightweight local evaluations in seconds or gate pull requests in GitHub Actions.
• Deterministic Caching: SHA-256 keyed SQLite disk cache ensures identical runs cost $0 and execute in milliseconds.
• Accurate Token & Cost Gating: Model pricing registry with local tokenizers calculates exact financial and latency deltas.
• Hardened Subprocess Sandbox: Isolated code execution runner with resource limits and exploit-tested AST/memory sandboxing.
• Rich Reports: Standalone, zero-dependency interactive HTML reports and automated sticky PR comments.

---

⚖️ Honest Comparison: PromptDiff vs Alternatives

Selecting the right evaluation tool depends heavily on your team's workflow, runtime stack, and data sovereignty requirements:

Feature / Dimension	PromptDiff	promptfoo	LangSmith	Braintrust
Primary Focus	Local-first regression CI/CD & prompt version diffing	LLM red-teaming, security & multi-provider CLI evals	Production tracing, debug sessions & SaaS observability	Enterprise eval platform, proxy logging & collaboration
Runtime & Language	Pure Python 3.10+ (zero heavy dependencies)	Node.js / TypeScript	Hosted SaaS (Python / TS SDKs)	Hosted SaaS / Enterprise on-prem
Data Privacy	100% Local / On-prem (SQLite on local disk; zero telemetry exfiltration)	Local / Self-hosted	Cloud SaaS (prompts & traces sent to vendor servers)	Cloud SaaS / Enterprise Private Cloud
CI/CD Quality Gate	Native promptdiff test & Pytest plugin (exit code 1 on regression)	Native CLI runner & GitHub Actions	Webhook / CI SDK assertions	CI integration via CLI / SDK
Cost & Latency Diffing	Deterministic offline token & pricing delta engine	Basic cost approximations	Cloud dashboard cost tracking	Cloud dashboard cost analytics
Sandboxed Code Execution	Isolated OS subprocess (-I -s -B, memory & CPU limits)	Node VM sandbox	Cloud worker execution	Cloud execution sandbox
Automated Prompt Optimization	Reflexive meta-prompting & MCTS compiler	Optional external scripts	Playground prompt engineering	Automated AI prompt tuner
Full Distributed Tracing	⚠️ Telemetry logs only (OpenTelemetry / MLflow exportable)	⚠️ Eval-focused only	✅ Full distributed waterfall traces	✅ Distributed trace logging & proxy
Pricing Model	100% Free & Open Source (MIT)	Open Source (MIT) with Enterprise tier	Proprietary SaaS (Usage-based subscription)	Commercial SaaS / Enterprise license

When to choose which:

• Choose PromptDiff if you are a Python/MLOps team that treats prompts as code in Git, wants pytest-native integration, requires 100% data sovereignty without external cloud dependencies, and needs fast PR regression gates.
• Choose promptfoo if you have a Node.js/TypeScript stack, want a rich browser-based red-teaming workspace, or need pre-packaged adversarial jailbreak test suites.
• Choose LangSmith if your primary requirement is distributed production trace visualization across multi-agent LangChain graphs.
• Choose Braintrust if you want an enterprise-managed centralized cloud evaluation platform with web-based team playground collaboration.

---

🚀 Quickstart in 30 Seconds

# 1. Install promptdiff core (lightweight, zero heavy ML dependencies)
pip install promptdiff

# 2. Scaffold a starter evaluation project
promptdiff init my-evals
cd my-evals

# 3. Run regression tests offline (Zero API keys required)
promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs testcases.jsonl \
  --mock \
  --eval "json_validity,latency,cost,similarity" \
  --assert "cost_delta <= 15%, latency_delta <= 20%" \
  --export-html report.html

---

🩺 Environment Doctor

Diagnose local environment readiness, LLM API keys, optional packages (tiktoken, sentence-transformers, mlflow, wandb), and disk cache engine with a single command:

promptdiff doctor

---

🔍 How It Works: Pull Request Quality Gate

Integrate promptdiff directly into your CI/CD pipeline to block regressions before merging to main:

promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs datasets/testcases.jsonl \
  --model gpt-4o \
  --eval "json_validity,latency,cost,similarity,llm_judge,faithfulness,security" \
  --assert "cost_delta <= 10%, latency_delta <= 15%, similarity >= 0.75, faithfulness >= 0.85" \
  --fail-on-regression \
  --export-markdown report.md

Exit Code	CI Status	Action
0	PASSED	Quality assertions satisfied; safe to merge.
1	REGRESSION	Regression threshold violated (e.g. cost jump, latency spike, schema break). CI pipeline fails.

🤖 Standalone Pull Request Commenter (scripts/pr_commenter.py)

For non-composite CI environments (Jenkins, GitLab CI, Buildkite, or custom GitHub Actions steps), use the standalone PR commenting script:

# Run regression evaluation exporting report JSON
promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs datasets/testcases.jsonl \
  --mock \
  --export-json report.json

# Post or update sticky Markdown evaluation comment on PR
python scripts/pr_commenter.py \
  --report report.json \
  --repo "$GITHUB_REPOSITORY" \
  --pr "$PR_NUMBER" \
  --token "$GITHUB_TOKEN"

---

📚 Curated Recipe Catalog

Pull ready-to-use prompt templates, test suites, and tailored evaluators for your specific use case:

# List all domain recipes
promptdiff recipe list

# Pull a specific starter kit
promptdiff recipe pull rag-qa           # RAG Grounding & Faithfulness
promptdiff recipe pull json-extractor   # Strict Structured Output & Schema AST
promptdiff recipe pull sql-gen          # Natural Language to SQL
promptdiff recipe pull security-guard   # Prompt Injection & Extraction Defense

---

🧪 Pytest Plugin Integration

Use promptdiff fixtures directly in your standard Python unit test suites:

# tests/test_prompts.py
import pytest
from promptdiff.core.models import TestCase


@pytest.mark.asyncio
async def test_support_prompt_regression(prompt_diff):
    report = await prompt_diff.compare(
        v1="prompts/support_v1.txt",
        v2="prompts/support_v2.txt",
        test_cases=[
            TestCase(id="tc1", vars={"query": "How do I reset my password?"}),
            TestCase(id="tc2", vars={"query": "Request billing refund"}),
        ],
        model="gpt-4o",
        mock=True,
    )
    assert report.verdict.passed, f"Regression detected: {report.verdict.failed_assertions}"

Run with standard pytest:

pytest tests/test_prompts.py

---

🐍 Python SDK

Use promptdiff programmatically inside Python applications or evaluation scripts:

import promptdiff
from promptdiff.core.models import TestCase

# Run regression evaluation
report = promptdiff.compare(
    v1="prompts/support_v1.txt",
    v2="prompts/support_v2.txt",
    dataset=[
        TestCase(id="tc1", vars={"query": "Reset password"}),
        TestCase(id="tc2", vars={"query": "Billing question"}),
    ],
    model="gpt-4o",
    mock=True,
    assertions=["cost_delta <= 15%", "latency_delta <= 20%"],
)

print(f"Passed: {report.verdict.passed}")
print(f"Cost Delta: {report.verdict.cost_delta_pct:.1f}%")

# Compress prompt tokens while maintaining quality
shrunk = promptdiff.shrink(
    prompt="Please kindly act as an AI and answer: {{query}}",
    dataset=[TestCase(id="1", vars={"query": "Help"})],
    mock=True,
)
print(f"Compressed Prompt: {shrunk.compressed_prompt}")

---

📦 Installation & Modular Extras

PromptDiff is built with a slim, featherweight core and modular extras so you only install what you need:

# Core CLI & CI runner (typer, rich, pydantic, httpx, jinja2, pyyaml, tenacity, numpy)
pip install promptdiff

# Semantic dense embedding similarity (sentence-transformers)
pip install "promptdiff[semantic]"

# Interactive split-screen Terminal UI (Textual)
pip install "promptdiff[tui]"

# Streamlit telemetry web dashboard
pip install "promptdiff[ui]"

# All optional components
pip install "promptdiff[all]"

---

🧩 Advanced & Extended Modules

Command / Tool	Extra Required	Description
promptdiff cache-impact	Core	KV-cache prefix breakpoint analyzer & monthly financial cash loss forecaster.
promptdiff replay-traces	Core	Production OpenTelemetry & Langfuse shadow replayer with automated PII masking.
promptdiff arena	Core	Evaluate $N$ prompt versions with Bayesian Bradley-Terry & ELO skill ratings.
promptdiff studio	Core	Launch zero-dependency local-first visual diff web studio & playground.
promptdiff mcts	Core	Active Monte Carlo Tree Search prompt optimizer with Pareto frontier.
promptdiff redteam	Core	Multi-turn TAP adversarial red-teaming (steganography & CVSS risk matrix).
promptdiff cascade	Core	Confidence-aware model cascade router & enterprise ROI forecaster.
promptdiff check	Core	Static linting & token cost analysis for prompt templates.
promptdiff serve	Core	Launch FastAPI REST API server & playground (pip install fastapi uvicorn).
promptdiff diff	Core	Instant side-by-side terminal syntax diff without calling model APIs.
promptdiff pricing	Core	Query token pricing and cost calculations for 30+ providers.
promptdiff fuzz	Core	Red-teaming security fuzzer scanning 20 distinct adversarial injection vectors.
promptdiff tui	[tui]	Launch interactive split-screen terminal workspace (pip install promptdiff[tui]).
promptdiff ui	[ui]	Launch Streamlit web dashboard for interactive telemetry (pip install promptdiff[ui]).
promptdiff optimize	Core	Reflective auto-prompt optimizer (DSPy style) using meta-model feedback.
promptdiff shrink	Core	Token compressor pruning boilerplate fluff while preserving 100% output quality.
promptdiff cache-sim	Core	Prefix caching hit rate analyzer and ROI forecaster.
promptdiff history	Core	Benchmark prompt quality and cost evolution across Git revisions.

---

🛡️ Production Engineering Standards & Quality Assurance

PromptDiff is built to enterprise MLOps standards with zero tolerance for unverified code or silent regressions:

Dimension	Quality Standard	Verification
Comprehensive Test Suite	336 unit, integration, and security tests	pytest passing on Linux, macOS, and Windows
Test Coverage	89%+ branch & statement coverage	Automated threshold enforcement in CI (--cov-fail-under=85)
Isolated Code Sandbox	Subprocess execution with resource limits (RLIMIT_AS, RLIMIT_CPU)	Exploit-tested AST/memory barriers & strict timeout handling
Strict Type Safety	100% type-annotated codebase (PEP 561 compliant py.typed)	mypy --strict promptdiff (0 errors across 119 source files)
Code Formatting & Linting	Automated style checking & import order	ruff check . & ruff format --check . in pre-commit
Cryptographic Provenance	HMAC-SHA256 zero-width prompt steganography	Constant-time tamper detection (hmac.compare_digest)
Schema Drift Protection	Automated drift protection against JSON schema divergence	DiffReport.model_json_schema() verified in CI pipeline

---

🏛️ Architecture & Resume Showcase

Detailed mathematical formulations, system diagrams, and resume-ready STAR bullet points for senior AI Engineer and MLOps roles are available in:

👉 Technical Architecture & Portfolio Showcase (PORTFOLIO.md)

🔒 Data Privacy & Local Storage Disclosure

PromptDiff operates under an absolute local-first, zero-telemetry exfiltration guarantee:

• Local Persistence Only: Evaluation runs and token metrics are written to local SQLite storage (.promptdiff/telemetry.db). No prompt contents, outputs, or traces are ever sent to external cloud servers.
• Automated Retention Management: Automatically delete historical records older than $N$ days with --db-retention-days <N> or run promptdiff db prune --days 14.
• Ephemeral Storage: Run with --db-path ":memory:" for zero disk persistence.
• Complete security documentation and disclosure SLAs are available in SECURITY.md.

---

🤝 Community & Contributing

• Contributing Guide
• Code of Conduct
• Security Policy
• Changelog

---

📄 License

Distributed under the MIT License. See LICENSE for more information.