Repository navigation
docs(examples): add a Playwright pack with a failed assertion - #24
Merged
siddhant573 merged 3 commits intoNov 11, 2026
Conversation
A real Playwright 1.63 run of two specs against a small demo cart page: one passes, one fails on expect(...).toHaveText() because the SAVE10 promo discounts the first line instead of the subtotal (expected $44.99, received $47.49). The pack shows how a failed assertion is recorded at each layer: the step verdict with kind/defect/expected/actual/check in result.yaml, the step-level failure.yaml with the matcher message, locator, source snippet, screenshot and page snapshot, and the finalize-generated root failure index. It validates at L0 and L1 with no warnings. project/ holds the Playwright project and to-evidence.mjs, a small converter from the JSON reporter output to a live pack, so the pack can be reproduced with npx playwright test && npm run evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cUMVh2toFvpRV4GVDsgoB
sealed/playwright.evidence is the zip finalize wrote; the committed playwright.evidence/ directory is that zip unpacked. GitHub's raw host serves it with Range support and open CORS, so overview.md links straight to it in evidence.lambdatest.com via ?pack=. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cUMVh2toFvpRV4GVDsgoB
Collaborator
Author
|
Same pack, served from this PR's ref in LambdaTest/evidence-cli (keeps working if the fork goes away): Open in evidence.lambdatest.com ↗ |
…t in the viewer evidence.lambdatest.com labels a step from steps/<n>-<id>/step.json (summary) and fills its address bar from step.json url; result.yaml's step title is not read. The pack had no step.json and only the one end-of-test screenshot, so steps showed as "Step 1", "Step 2", "assertion" on about:blank. A step() helper in tests/fixtures.ts wraps test.step and attaches the page URL and a screenshot when each step ends. to-evidence.mjs now writes a folder per step with step.json and that screenshot. Pack and sealed zip regenerated from a fresh run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lt-zeeshan
approved these changes
Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Check it in the viewer
Open the pack from this branch in evidence.lambdatest.com ↗
The link points
?pack=at the sealed zip on this branch in the forksiddhant573/evidence-cli-forkthroughraw.githubusercontent.com, which serves it with HTTP Range (206) andAccess-Control-Allow-Origin: *. Once merged, the link inexamples/playwright/overview.md(which usesmain) works the same way. You can also downloadexamples/playwright/sealed/playwright.evidenceand drop it onto the viewer.What to look for:
test.steptitle, showshttps://shop.example.test/cartin the address bar, and has its own screenshot.failed, with expected$44.99, actual$47.49, the failure screenshot (it shows$47.49), and the failure record (matcher message, locatorgetByTestId('cart-total'), sourcepromo-code.spec.ts:13:50, snippet).What's in it
There was no Playwright example yet;
examples/only had the kane-cli pack (a clean run). This addsexamples/playwright/:playwright.evidence/: the pack from a real Playwright 1.63 run, sealed byevidence finalizeand committed unpacked, like the kane-cli pack.sealed/playwright.evidence: the sealed zip itself (112 KB), for the viewer link.project/: the Playwright project that produced it (demo cart page served bypage.route, two specs, astep()helper that records each step's URL and screenshot) andto-evidence.mjs, a small converter from Playwright's JSON report to a live pack.overview.md: walkthrough of how a failed assertion is recorded: the step verdict inresult.yaml, the step-levelfailure.yaml, and the finalize-generated root failure index, plus a Playwright-to-pack mapping table.The demo page has a planted bug: the
SAVE10promo discounts only the first line instead of the subtotal.Decisions to review
expect()asfailed(defect: Product) and any other error (timeout, crash) asbroken(defect: Test), following the format's verdict definitions and the e2e reporter's rule.project/package.jsondepends on@testmuai/evidence-cli@^0.2.0, which is published on npm.step.json(summary,url,kind,status,duration_ms). The format treats it as opaque; it is the per-step record evidence.lambdatest.com reads for a step's label and address bar (it does not readresult.yamltitle), the same file kane-cli packs carry.site/are unchanged; they still name only kane-cli as a working example.Testing
evidence validateon the directory:ok validat L1 and L0, no warnings. The sealed zip also validates at L1, and unpacks to exactly the committed directory.project/from a fresh copy:npx playwright testgives 1 passed, 1 failed;to-evidence.mjs+finalizegive a valid pack.npm test: 510 tests pass.evidence.lambdatest.comwas not reachable from the session that made this change, so please confirm with the link above.🤖 Generated with Claude Code