Skip to content

feat(integrations): @testmuai/e2e-evidence, an e2e reporter that writes each run as a pack - #18

Open
siddhant573 wants to merge 1 commit into
mainfrom
feat/e2e-evidence
Open

siddhant573 wants to merge 1 commit into
mainfrom
feat/e2e-evidence

Conversation

@siddhant573

Copy link
Copy Markdown
Collaborator

Summary

Adds @testmuai/e2e-evidence under integrations/e2e: a reporter for the e2e framework that writes each run as a sealed .evidence pack, validated at L1, using e2e's public reporters API.

import { evidence } from '@testmuai/e2e-evidence';

export default {
  targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
  reporters: ['list', evidence()],
} satisfies E2EConfig;

The e2e maintainers prefer this as a standalone package over a built-in reporter (e2e-inc/e2e#754), so it lives here, next to the format library it seals with.

  • One pack per run in .e2e/evidence/, named in the terminal summary. Options: outDir, profile (L0/L1), enabled. E2E_EVIDENCE=0 turns it off for a run. An unknown option throws.
  • Built from e2e's report-1 document and the artifacts it links: readable test names (title path), step summaries and durations, failed (only ASSERTION_FAILED) vs broken, the failure record with expected/actual, locator, URL, frame and screen text, agent steps with their turns, and shareable traces.
  • Copies only artifacts whose redaction is complete or not-required, never the test source (the definition is the test's identity from the report).
  • It never changes a run's outcome: a pack it cannot write, or one that does not validate, is a summary line.
  • Its own package.json (ESM, Node 22.12+, peer e2e >=0.17.0 <1) and its own CI job. The root package's build, typecheck, and tests do not include it.

Not yet, because e2e's report does not carry them: the value a step typed or picked, where a step acted (the viewer's cursor and element box), and a frame on every passed step (the validator reports those as warnings). The README says so; an e2e PR adding step detail to the report is in progress.

Publishing is not wired: release.yml publishes the root package only. This needs its own release step or a manual npm publish from integrations/e2e, to decide in review.

Verified

Locally, with the packed tarball installed in a fresh project next to the published e2e@0.17.0 and @e2e-dev/web@0.12.0, against a one-page login app, one passing test and one failing on purpose:

   ✓ signs in with a known user 831ms
   × welcomes an unknown user (fails on purpose) 5.25s
      Tests  1 failed | 1 passed (2)
   Evidence  .e2e/evidence/01a10bfd-8b2d-7735-b776-83a7f5adb0ae.evidence

$ npx evidence validate .e2e/evidence/*.evidence --profile L1
! tests/t-bb3729ed188c3ecf/steps/1-0-0: step folder "1-0-0" has no screenshot.<ext> [l1.steps.screenshot_missing]
…
ok valid (profile L1, evidence 0.1, status finalized)

The failing step's failure.yaml has expected: text "Welcome, grace", actual: text "Invalid credentials", the locator, the URL, screenshot.png, and screen.txt. With E2E_EVIDENCE=0 no pack was written.

npm ci && npm run typecheck && npm run build && npm test in integrations/e2e: 33 tests pass. Root npx tsc --noEmit is unchanged.

…es each run as a pack

A reporter for the e2e framework through its public reporters API:
reporters: ['list', evidence()] writes one sealed pack per run in
.e2e/evidence, validated at L1, and names it in the terminal summary.

The pack is built from e2e's report-1 document and the artifacts it links:
readable test names and step summaries, durations, failed vs broken
verdicts, the failure record with expected and actual, locator, URL, frame
and screen text, agent steps with their turns, and shareable traces.
Options are outDir, profile, and enabled; E2E_EVIDENCE=0 turns it off for
a run. It never changes the run's outcome.

Its own package.json, ESM on Node 22.12+, with e2e >=0.17.0 <1 as a peer,
tested in its own CI job; the root build and tests do not touch it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant