Repository navigation
feat(integrations): @testmuai/e2e-evidence, an e2e reporter that writes each run as a pack - #18
Open
siddhant573 wants to merge 1 commit into
Open
siddhant573 wants to merge 1 commit into
siddhant573 wants to merge 1 commit into
Conversation
…es each run as a pack A reporter for the e2e framework through its public reporters API: reporters: ['list', evidence()] writes one sealed pack per run in .e2e/evidence, validated at L1, and names it in the terminal summary. The pack is built from e2e's report-1 document and the artifacts it links: readable test names and step summaries, durations, failed vs broken verdicts, the failure record with expected and actual, locator, URL, frame and screen text, agent steps with their turns, and shareable traces. Options are outDir, profile, and enabled; E2E_EVIDENCE=0 turns it off for a run. It never changes the run's outcome. Its own package.json, ESM on Node 22.12+, with e2e >=0.17.0 <1 as a peer, tested in its own CI job; the root build and tests do not touch it.
1 of 2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
@testmuai/e2e-evidenceunderintegrations/e2e: a reporter for the e2e framework that writes each run as a sealed.evidencepack, validated at L1, using e2e's publicreportersAPI.The e2e maintainers prefer this as a standalone package over a built-in reporter (e2e-inc/e2e#754), so it lives here, next to the format library it seals with.
.e2e/evidence/, named in the terminal summary. Options:outDir,profile(L0/L1),enabled.E2E_EVIDENCE=0turns it off for a run. An unknown option throws.failed(onlyASSERTION_FAILED) vsbroken, the failure record with expected/actual, locator, URL, frame and screen text, agent steps with their turns, and shareable traces.completeornot-required, never the test source (the definition is the test's identity from the report).package.json(ESM, Node 22.12+, peere2e >=0.17.0 <1) and its own CI job. The root package's build, typecheck, and tests do not include it.Not yet, because e2e's report does not carry them: the value a step typed or picked, where a step acted (the viewer's cursor and element box), and a frame on every passed step (the validator reports those as warnings). The README says so; an e2e PR adding step detail to the report is in progress.
Publishing is not wired:
release.ymlpublishes the root package only. This needs its own release step or a manualnpm publishfromintegrations/e2e, to decide in review.Verified
Locally, with the packed tarball installed in a fresh project next to the published
e2e@0.17.0and@e2e-dev/web@0.12.0, against a one-page login app, one passing test and one failing on purpose:The failing step's
failure.yamlhasexpected: text "Welcome, grace",actual: text "Invalid credentials", the locator, the URL,screenshot.png, andscreen.txt. WithE2E_EVIDENCE=0no pack was written.npm ci && npm run typecheck && npm run build && npm testinintegrations/e2e: 33 tests pass. Rootnpx tsc --noEmitis unchanged.