Skip to content

docs(examples): add a Playwright pack with a failed assertion - #24

Merged
siddhant573 merged 3 commits into
LambdaTest:mainfrom
siddhant573:claude/playwright-failed-assertion-example-x7fn0h
Nov 11, 2026
Merged

siddhant573 merged 3 commits into
LambdaTest:mainfrom
siddhant573:claude/playwright-failed-assertion-example-x7fn0h

Conversation

@siddhant573

@siddhant573 siddhant573 commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Check it in the viewer

Open the pack from this branch in evidence.lambdatest.com ↗

The link points ?pack= at the sealed zip on this branch in the fork siddhant573/evidence-cli-fork through raw.githubusercontent.com, which serves it with HTTP Range (206) and Access-Control-Allow-Origin: *. Once merged, the link in examples/playwright/overview.md (which uses main) works the same way. You can also download examples/playwright/sealed/playwright.evidence and drop it onto the viewer.

What to look for:

  • Run header: 2 tests, 1 passed, 1 failed.
  • Every step is labelled with its test.step title, shows https://shop.example.test/cart in the address bar, and has its own screenshot.
  • SAVE10 takes 10% off the cart total: step 3 is failed, with expected $44.99, actual $47.49, the failure screenshot (it shows $47.49), and the failure record (matcher message, locator getByTestId('cart-total'), source promo-code.spec.ts:13:50, snippet).
  • cart shows the subtotal of every line: passed, 3 steps, each with its screenshot.

What's in it

There was no Playwright example yet; examples/ only had the kane-cli pack (a clean run). This adds examples/playwright/:

  • playwright.evidence/: the pack from a real Playwright 1.63 run, sealed by evidence finalize and committed unpacked, like the kane-cli pack.
  • sealed/playwright.evidence: the sealed zip itself (112 KB), for the viewer link.
  • project/: the Playwright project that produced it (demo cart page served by page.route, two specs, a step() helper that records each step's URL and screenshot) and to-evidence.mjs, a small converter from Playwright's JSON report to a live pack.
  • overview.md: walkthrough of how a failed assertion is recorded: the step verdict in result.yaml, the step-level failure.yaml, and the finalize-generated root failure index, plus a Playwright-to-pack mapping table.

The demo page has a planted bug: the SAVE10 promo discounts only the first line instead of the subtotal.

Decisions to review

  • The converter marks a failed expect() as failed (defect: Product) and any other error (timeout, crash) as broken (defect: Test), following the format's verdict definitions and the e2e reporter's rule.
  • project/package.json depends on @testmuai/evidence-cli@^0.2.0, which is published on npm.
  • Each step folder carries a step.json (summary, url, kind, status, duration_ms). The format treats it as opaque; it is the per-step record evidence.lambdatest.com reads for a step's label and address bar (it does not read result.yaml title), the same file kane-cli packs carry.
  • README and site/ are unchanged; they still name only kane-cli as a working example.

Testing

  • evidence validate on the directory: ok valid at L1 and L0, no warnings. The sealed zip also validates at L1, and unpacks to exactly the committed directory.
  • Ran the committed project/ from a fresh copy: npx playwright test gives 1 passed, 1 failed; to-evidence.mjs + finalize give a valid pack.
  • npm test: 510 tests pass.
  • Not tested: rendering in the viewer itself. evidence.lambdatest.com was not reachable from the session that made this change, so please confirm with the link above.

🤖 Generated with Claude Code

A real Playwright 1.63 run of two specs against a small demo cart page:
one passes, one fails on expect(...).toHaveText() because the SAVE10 promo
discounts the first line instead of the subtotal (expected $44.99,
received $47.49).

The pack shows how a failed assertion is recorded at each layer: the step
verdict with kind/defect/expected/actual/check in result.yaml, the
step-level failure.yaml with the matcher message, locator, source snippet,
screenshot and page snapshot, and the finalize-generated root failure index.
It validates at L0 and L1 with no warnings.

project/ holds the Playwright project and to-evidence.mjs, a small
converter from the JSON reporter output to a live pack, so the pack can be
reproduced with npx playwright test && npm run evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013cUMVh2toFvpRV4GVDsgoB
sealed/playwright.evidence is the zip finalize wrote; the committed
playwright.evidence/ directory is that zip unpacked. GitHub's raw host
serves it with Range support and open CORS, so overview.md links straight
to it in evidence.lambdatest.com via ?pack=.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013cUMVh2toFvpRV4GVDsgoB
@siddhant573

Copy link
Copy Markdown
Collaborator Author

Same pack, served from this PR's ref in LambdaTest/evidence-cli (keeps working if the fork goes away): Open in evidence.lambdatest.com ↗

…t in the viewer

evidence.lambdatest.com labels a step from steps/<n>-<id>/step.json
(summary) and fills its address bar from step.json url; result.yaml's
step title is not read. The pack had no step.json and only the one
end-of-test screenshot, so steps showed as "Step 1", "Step 2",
"assertion" on about:blank.

A step() helper in tests/fixtures.ts wraps test.step and attaches the
page URL and a screenshot when each step ends. to-evidence.mjs now
writes a folder per step with step.json and that screenshot. Pack and
sealed zip regenerated from a fresh run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@siddhant573
siddhant573 merged commit 73f69a8 into LambdaTest:main Nov 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants