Change safety for agents.
Record, replay and fork AI agent runs: reproduce a failure exactly,
find the call that lost the run, and prove your fix holds before it ships.
Quickstart · Docs · CLI · CI · Compare
Your agent refunded $240 it should have escalated. Run it again, and it behaves.
You can't make it fail on demand, so you can't prove a fix works.
shadowlap records your agent's model calls and the tools you wrap, then re‑runs your real code against that recording. The refund happens again, call for call, with nothing sent upstream and nothing done twice. Change one thing, and the run forks at the first request that differs.
Record, replay, blame and fix, in the real CLI (source).
npm install -g shadowlap
shadowlap demoThe CLI needs Node 22.17 or later, not 23.
shadowlap demo runs the whole loop in about half a minute. It records a refund agent breaking its policy, replays it with no upstream requests, finds the call that lost the run, then proposes, proves and applies a fix. Its model is a scripted stand‑in, so the demo needs no key and comes out the same every time. The pictures in this README are made the same way, with the real CLI, dashboard and terminal UI.
One bug, from failure to fix. A support agent refunds three $240 orders, labelled refund-bug, though its policy sends refunds over $100 to a manager. A fourth refund, $60, is labelled good.
shadowlap blame asks model calls again, each from a fork that replays everything before it as recorded. It names the last call where a different answer can still save the run.
$ shadowlap blame 01M64J32D4 --expect @refund-check.json
…
⑂ The run was lost at model call #5 (3 of 5)
…
#5 3 of 5 5 of 10 passed · rate >= 0.10 (99.1%) rescued ◂ blamed
#7 4 of 5 0 of 6 passed · rate <= 0.60 (99.1%) lost
…Asked again from call #5, 5 of 10 attempts pass. From call #7 on, none do.
shadowlap repair asks a model, or your own Claude Code, to rewrite the system prompt or a tool description. It keeps the smallest rewrite that verify accepts twice.
$ shadowlap repair --fixes label:refund-bug --keeps label:good \
> --expect @refund-check.json --repairer anthropic
…
✓ Fixed: Escalate refunds above the policy limit
agent.mjs · system prompt
- You are a support agent. Follow the refund policy.
+ You are a support agent. Follow the refund policy: escalate every refund above the limit.
Verified: fixes 2 of 2 target runs, 1 of 1 held-out, keeps 1 of 1 good,
…Verify accepts a fix only when it fixes the failing runs, held‑out runs included, and leaves no good run 10 points or more worse, at 5% risk a run. shadowlap verify proves a fix you wrote yourself the same way.
--repairer anthropic needs the Anthropic SDK beside the CLI:
npm install -g @anthropic-ai/sdk
The same fix in the dashboard: blame, the prompt change and the verdict (source).
fix apply puts the patch in your working tree and keeps the fixed runs as goldens, recordings run as tests.
$ shadowlap fix apply fx-01M64J3BR1
⑂ Applying fx-01M64J3BR1 · issue_refund called with amount 240
…
✓ fx-01M64J3BR1 applied: the patch is in the working tree,
3 of 3 goldens copied to goldens/, shadowlap test passedCommit the goldens with the patch. In CI, shadowlap test replays them offline with no real key. It fails on a changed request, a live call, an error exit or a failed check.
On GitHub, add one step after checkout, Node setup and your build (full workflow):
- uses: devjoinedthechat/shadowlap@v0.1.0
with:
version: 0.1.0
recordings: goldens/*With fix-on-failure, the Action proposes a fix for a golden that fails, verifies it on your goldens and opens a pull request with it. On pull requests, it keeps one comment with each golden's verdict.
cd your-agent
shadowlap initinit finds your model SDKs, framework, agent command and tools, and prints your next steps. Model calls need no code changes. Tools need the SDK for your language:
# Node 18.18+
npm install @shadowlap/sdk # wrap tools
npm install @shadowlap/adapters # wrap them in your framework
# Node 22.17+, not 23
npm install -D @shadowlap/test # record tests in Vitest, Jest or node:test
# Python 3.9+
pip install shadowlap # all three in one packageWrap each tool where you define it, and declare its effect: read, write or pure. Outside shadowlap the wrapper only calls your function, so wrapped tools ship unchanged.
import { tool } from '@shadowlap/sdk';
export const issueRefund =
tool('issue_refund', { effect: 'write' }, refund);from shadowlap import tool
@tool(effect="read")
def lookup_order(order_id): ...Then record and replay:
shadowlap run -- node agent.mjs # record, with your usual key
shadowlap replay <run> # replay it; no real key if clean
shadowlap dashboard # step through it in your browserrun names every tool you haven't wrapped: those run for real on every replay. The quickstart walks through each step and what it prints.
shadowlap dashboard opens your runs on 127.0.0.1. A failed run opens where it went wrong: at the call blame found it lost, or else where it first failed. Fork from any call, edit a tool's result and fork from there, or compare two runs side by side. A fork's page ranks what it did differently.
shadowlap ui brings the same views to your terminal. Each call has conversation, request, response and meta tabs. Press : for the what‑ifs at the selected call, or to blame, repair or verify the run.
The terminal UI on the same project: the failed run, a fork and the fork menu (source).
- What‑ifs at any call. Override a result, switch the model, plant an instruction.
- Model and prompt changes. See which decisions change before you ship one.
- And more: injection tests, error paths, bisect, coverage, production traffic, a team store, training data.
| Area | Works with |
|---|---|
| Providers | Anthropic, OpenAI, Gemini, Azure OpenAI, Amazon Bedrock, Gemini Enterprise Agent Platform (Vertex AI) and Claude on Microsoft Foundry |
| Gateways | Vercel's AI Gateway, OpenAI‑compatible servers (Ollama, vLLM, OpenRouter, Groq) and any host that speaks one of these formats |
| SDKs | TypeScript and Python; any other language over the wire protocol |
| Frameworks (TypeScript) | Mastra, Vercel AI SDK, OpenAI Agents SDK, LangChain, LangGraph, Claude Agent SDK, the tool runner in Anthropic's TypeScript SDK |
| Frameworks (Python) | CrewAI, OpenAI Agents SDK, LangChain, LangGraph, Pydantic AI, Google ADK, LlamaIndex, Microsoft Agent Framework, Strands, smolagents, Agno, Claude Agent SDK |
| Test suites | pytest, Vitest, Jest and node:test: a test records once, then replays offline |
| MCP | Servers over stdio or Streamable HTTP, through the shadowlap mcp proxy |
| CI | The GitHub Action; JUnit, JSON and Markdown reports for any other CI |
| Coding agents | Claude Code, Codex, Cursor and Copilot, through JSON output and skills |
| Imports | HAR, nock, VCR, LiteLLM, OpenTelemetry, Langfuse, LangSmith, Phoenix, Logfire |
| Exports | Packs, one‑file HTML reports, HAR, OpenTelemetry, eval datasets and preference pairs |
shadowlap run starts a sidecar on 127.0.0.1 and points your agent at it:
shadowlap run -- node agent.mjs
│
├─ sidecar 127.0.0.1 on a free port, a token in every path
│ ├─ /anthropic/* /openai/* /gemini/* model APIs
│ ├─ /bedrock/* /vertex/* /azure/* cloud clients
│ ├─ /ai-gateway/* Vercel's AI Gateway
│ ├─ /http/* HTTP services
│ └─ /v1/calls/* tools, clock, randomness
├─ store .shadowlap/shadowlap.db (SQLite), out of git
└─ agent ANTHROPIC_BASE_URL, OPENAI_BASE_URL,
GOOGLE_GEMINI_BASE_URL for the model SDKs;
SHADOWLAP_URL, SHADOWLAP_RUN_ID for shadowlap's SDKs
- Matched by request, not order. A call is matched by its request, after your normalizers mask values such as ids, and by how many identical requests came before it. Parallel calls match in whatever order they arrive.
- Forked at the first change. After it, model calls and
readtools run live.writetools are blocked or stubbed unless you allow them, so a fork never repeats a refund.
How it works covers fingerprints, matching, effects and the network guard.
- Local only. The sidecar and dashboard listen on 127.0.0.1, behind a token. Runs stay in your project.
- Keys never stored. API keys pass through to the provider and are never written.
- Masked on the way out. Every copy of a run masks the credentials shadowlap recognises. Packs, exports, share links and what repair shows its model also apply your redaction rules.
- No telemetry by default. Nothing is sent unless
SHADOWLAP_TELEMETRY_URLnames a collector you run, and what is sent never includes prompts or tool data.DO_NOT_TRACK=1opts out.
SECURITY.md lists every safeguard and how to report a vulnerability.
CONTRIBUTING.md covers setup, the test suites, lint, commit messages and how a pull request is reviewed.
git clone https://github.com/devjoinedthechat/shadowlap
cd shadowlap && npm ci && npm run build
npx vitest run && npm run typecheck:tests
# regenerate the README's terminal, examples and pictures
npm run demo && node scripts/readme-shots.mjsApache-2.0. shadowlap has no accounts to sign up for.