Skip to content
devjoinedthechatPublic

About

Change safety for agents. Record, replay and fork AI agent runs: reproduce a failure exactly, find the call that lost the run, and prove your fix holds before it ships.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4,807 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

shadowlap

Change safety for agents.
Record, replay and fork AI agent runs: reproduce a failure exactly,
find the call that lost the run, and prove your fix holds before it ships.

CI npm version PyPI version License: Apache-2.0

Quickstart · Docs · CLI · CI · Compare

Your agent refunded $240 it should have escalated. Run it again, and it behaves.
You can't make it fail on demand, so you can't prove a fix works.

shadowlap records your agent's model calls and the tools you wrap, then re‑runs your real code against that recording. The refund happens again, call for call, with nothing sent upstream and nothing done twice. Change one thing, and the run forks at the first request that differs.

A terminal session: shadowlap run records a support agent that refunds a $240 order; shadowlap replay reproduces it with 0 upstream requests; shadowlap blame finds the run was lost at model call #5, with high confidence; after a prompt fix, the replay forks with the diff of the system prompt and the check passes.
Record, replay, blame and fix, in the real CLI (source).

Quickstart

npm install -g shadowlap
shadowlap demo

The CLI needs Node 22.17 or later, not 23.

shadowlap demo runs the whole loop in about half a minute. It records a refund agent breaking its policy, replays it with no upstream requests, finds the call that lost the run, then proposes, proves and applies a fix. Its model is a scripted stand‑in, so the demo needs no key and comes out the same every time. The pictures in this README are made the same way, with the real CLI, dashboard and terminal UI.

From failure to proof

One bug, from failure to fix. A support agent refunds three $240 orders, labelled refund-bug, though its policy sends refunds over $100 to a manager. A fourth refund, $60, is labelled good.

1. Find the call that lost the run

shadowlap blame asks model calls again, each from a fork that replays everything before it as recorded. It names the last call where a different answer can still save the run.

$ shadowlap blame 01M64J32D4 --expect @refund-check.json
…
⑂ The run was lost at model call #5 (3 of 5)
…
    #5  3 of 5  5 of 10 passed · rate >= 0.10 (99.1%)  rescued  ◂ blamed
    #7  4 of 5  0 of 6 passed · rate <= 0.60 (99.1%)   lost
…

Asked again from call #5, 5 of 10 attempts pass. From call #7 on, none do.

2. Get a fix, and prove it

shadowlap repair asks a model, or your own Claude Code, to rewrite the system prompt or a tool description. It keeps the smallest rewrite that verify accepts twice.

$ shadowlap repair --fixes label:refund-bug --keeps label:good \
>     --expect @refund-check.json --repairer anthropic
…
✓ Fixed: Escalate refunds above the policy limit
  agent.mjs · system prompt
  - You are a support agent. Follow the refund policy.
  + You are a support agent. Follow the refund policy: escalate every refund above the limit.

  Verified: fixes 2 of 2 target runs, 1 of 1 held-out, keeps 1 of 1 good,
…

Verify accepts a fix only when it fixes the failing runs, held‑out runs included, and leaves no good run 10 points or more worse, at 5% risk a run. shadowlap verify proves a fix you wrote yourself the same way.

--repairer anthropic needs the Anthropic SDK beside the CLI:

npm install -g @anthropic-ai/sdk

The shadowlap dashboard, from a failed run to its fix: Runs › Failed lists the issue's three runs, the newest opens at model call #5 where blame found it lost, and Open fix shows blame, repair's prompt change and verify's verdict, then the patch.
The same fix in the dashboard: blame, the prompt change and the verdict (source).

3. Keep it fixed in CI

fix apply puts the patch in your working tree and keeps the fixed runs as goldens, recordings run as tests.

$ shadowlap fix apply fx-01M64J3BR1
⑂ Applying fx-01M64J3BR1 · issue_refund called with amount 240
…
✓ fx-01M64J3BR1 applied: the patch is in the working tree,
  3 of 3 goldens copied to goldens/, shadowlap test passed

Commit the goldens with the patch. In CI, shadowlap test replays them offline with no real key. It fails on a changed request, a live call, an error exit or a failed check.

On GitHub, add one step after checkout, Node setup and your build (full workflow):

- uses: devjoinedthechat/shadowlap@v0.1.0
  with:
    version: 0.1.0
    recordings: goldens/*

With fix-on-failure, the Action proposes a fix for a golden that fails, verifies it on your goldens and opens a pull request with it. On pull requests, it keeps one comment with each golden's verdict.

Use it on your agent

cd your-agent
shadowlap init

init finds your model SDKs, framework, agent command and tools, and prints your next steps. Model calls need no code changes. Tools need the SDK for your language:

# Node 18.18+
npm install @shadowlap/sdk         # wrap tools
npm install @shadowlap/adapters    # wrap them in your framework
# Node 22.17+, not 23
npm install -D @shadowlap/test     # record tests in Vitest, Jest or node:test
# Python 3.9+
pip install shadowlap              # all three in one package

Wrap each tool where you define it, and declare its effect: read, write or pure. Outside shadowlap the wrapper only calls your function, so wrapped tools ship unchanged.

import { tool } from '@shadowlap/sdk';

export const issueRefund =
  tool('issue_refund', { effect: 'write' }, refund);
from shadowlap import tool

@tool(effect="read")
def lookup_order(order_id): ...

Then record and replay:

shadowlap run -- node agent.mjs   # record, with your usual key
shadowlap replay <run>            # replay it; no real key if clean
shadowlap dashboard               # step through it in your browser

run names every tool you haven't wrapped: those run for real on every replay. The quickstart walks through each step and what it prints.

Dashboard and terminal UI

shadowlap dashboard opens your runs on 127.0.0.1. A failed run opens where it went wrong: at the call blame found it lost, or else where it first failed. Fork from any call, edit a tool's result and fork from there, or compare two runs side by side. A fork's page ranks what it did differently.

shadowlap ui brings the same views to your terminal. Each call has conversation, request, response and meta tabs. Press : for the what‑ifs at the selected call, or to blame, repair or verify the run.

The shadowlap terminal UI: the newest refund that failed, open at model call #5, where blame found it lost, with its fix applied; the run tree, two refunds marked lost at #5; the fork, open at call 17, where it left its parent, with what it did differently; and the fork menu at lookup_policy's call, Override result first.
The terminal UI on the same project: the failed run, a fork and the fork menu (source).

Also

Integrations

Area Works with
Providers Anthropic, OpenAI, Gemini, Azure OpenAI, Amazon Bedrock, Gemini Enterprise Agent Platform (Vertex AI) and Claude on Microsoft Foundry
Gateways Vercel's AI Gateway, OpenAI‑compatible servers (Ollama, vLLM, OpenRouter, Groq) and any host that speaks one of these formats
SDKs TypeScript and Python; any other language over the wire protocol
Frameworks (TypeScript) Mastra, Vercel AI SDK, OpenAI Agents SDK, LangChain, LangGraph, Claude Agent SDK, the tool runner in Anthropic's TypeScript SDK
Frameworks (Python) CrewAI, OpenAI Agents SDK, LangChain, LangGraph, Pydantic AI, Google ADK, LlamaIndex, Microsoft Agent Framework, Strands, smolagents, Agno, Claude Agent SDK
Test suites pytest, Vitest, Jest and node:test: a test records once, then replays offline
MCP Servers over stdio or Streamable HTTP, through the shadowlap mcp proxy
CI The GitHub Action; JUnit, JSON and Markdown reports for any other CI
Coding agents Claude Code, Codex, Cursor and Copilot, through JSON output and skills
Imports HAR, nock, VCR, LiteLLM, OpenTelemetry, Langfuse, LangSmith, Phoenix, Logfire
Exports Packs, one‑file HTML reports, HAR, OpenTelemetry, eval datasets and preference pairs

How it works

shadowlap run starts a sidecar on 127.0.0.1 and points your agent at it:

shadowlap run -- node agent.mjs
│
├─ sidecar  127.0.0.1 on a free port, a token in every path
│  ├─ /anthropic/* /openai/* /gemini/*  model APIs
│  ├─ /bedrock/* /vertex/* /azure/*     cloud clients
│  ├─ /ai-gateway/*                     Vercel's AI Gateway
│  ├─ /http/*                           HTTP services
│  └─ /v1/calls/*                       tools, clock, randomness
├─ store    .shadowlap/shadowlap.db (SQLite), out of git
└─ agent    ANTHROPIC_BASE_URL, OPENAI_BASE_URL,
            GOOGLE_GEMINI_BASE_URL for the model SDKs;
            SHADOWLAP_URL, SHADOWLAP_RUN_ID for shadowlap's SDKs
  • Matched by request, not order. A call is matched by its request, after your normalizers mask values such as ids, and by how many identical requests came before it. Parallel calls match in whatever order they arrive.
  • Forked at the first change. After it, model calls and read tools run live. write tools are blocked or stubbed unless you allow them, so a fork never repeats a refund.

How it works covers fingerprints, matching, effects and the network guard.

Security

  • Local only. The sidecar and dashboard listen on 127.0.0.1, behind a token. Runs stay in your project.
  • Keys never stored. API keys pass through to the provider and are never written.
  • Masked on the way out. Every copy of a run masks the credentials shadowlap recognises. Packs, exports, share links and what repair shows its model also apply your redaction rules.
  • No telemetry by default. Nothing is sent unless SHADOWLAP_TELEMETRY_URL names a collector you run, and what is sent never includes prompts or tool data. DO_NOT_TRACK=1 opts out.

SECURITY.md lists every safeguard and how to report a vulnerability.

Contributing

CONTRIBUTING.md covers setup, the test suites, lint, commit messages and how a pull request is reviewed.

git clone https://github.com/devjoinedthechat/shadowlap
cd shadowlap && npm ci && npm run build
npx vitest run && npm run typecheck:tests
# regenerate the README's terminal, examples and pictures
npm run demo && node scripts/readme-shots.mjs

License

Apache-2.0. shadowlap has no accounts to sign up for.

About

Change safety for agents. Record, replay and fork AI agent runs: reproduce a failure exactly, find the call that lost the run, and prove your fix holds before it ships.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages