Skip to content

test(evals): exercise the workspace V4 CLI - #2838

Draft
shrey150 wants to merge 1 commit into
agent/browse-v4-5-remove-xpathfrom
agent/browse-v4-6-release
Draft

test(evals): exercise the workspace V4 CLI#2838
shrey150 wants to merge 1 commit into
agent/browse-v4-5-remove-xpathfrom
agent/browse-v4-6-release

Conversation

@shrey150

@shrey150 shrey150 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Make workspace evals and packaging consume the V4 Browse CLI proven in #2835.

  • Point Stagehand evals at the workspace Browse package instead of published Browse 0.9.5.
  • Update native and external-agent adapters to the current Browse command grammar.
  • Keep eval session names within Unix socket limits, restrict the wrapper to driver commands, and clean daemons up on abort/signals.
  • Add command-mapping and real wrapper-process coverage.

The Browse migration changeset now belongs to #2835, the first complete supported-parity head; this PR has no changeset in its own diff. The unrelated eval CLI --verbose changes from the old draft are intentionally excluded.

Stack (#2872)

  1. chore(cli): import Browse V3 baseline #2833 — exact Browse V3 baseline import
  2. refactor(cli): migrate Browse runtime and commands to Stagehand V4 #2834 — Stagehand V4 runtime and standard command parity
  3. feat(cli): restore cursor overlay through page.evaluate #2869 — CLI-owned cursor overlay
  4. fix(cli): restore V3 network capture through a CDP sidecar #2849 — CLI-private CDP sidecar; V3 network parity
  5. refactor(cli): remove coordinate XPath return flag #2835 — remove --return-xpath; supported V3 parity/release checkpoint
  6. test(evals): exercise the workspace V4 CLI #2838 — eval and packaging integration
  7. fix(cli): persist context names in Browserbase #2839 — managed Context names (fast-follow)
  8. refactor(cli): consume shared Functions core #2701 — shared Functions core consumer (fast-follow)

Review boundary

This is a consumer/integration layer, not another runtime-parity layer. Its diff is limited to eval adapters, their tests, workspace dependency wiring, and the lockfile.

E2E Test Matrix

The current remote PR head is f13b62d4c17afc55673b82a34ea03622e6eec570. The complete matrix below ran at f8e2579571b5c059753881f14b80abb718066a94; the only inherited change since then is #2849's test-helper timeout diagnostic, whose focused network tests and Browse lint/typecheck passed at sidecar head 9887732b6. The product diff is unchanged from proof commit 1dd7d0c8ef7ac65697f91780620a417a8ceb6bdd, where the packed-consumer and browser-backed wrapper flows below ran. Browser-backed runs used an isolated daemon directory; the pre-existing default daemon was not touched.

Command / flow Observed output Confidence / sufficiency
Fully verified head f8e257957: pnpm install --frozen-lockfile; build browse; build @browserbasehq/stagehand-evals Frozen install passed; Browse generated its real Oclif manifest; eval ESM and CLI bundles both built. Proves the regenerated lockfile and exact current workspace dependency graph build.
Fully verified head f8e257957: full @browserbasehq/stagehand-evals test suite 81 test files / 633 tests passed in 102.97 seconds. Proves the propagated parent/lock changes did not regress the complete eval suite, including Browse integration coverage.
Product proof head 1dd7d0c8: pnpm --filter browse pack; install the tarball in an empty npm consumer; run node_modules/.bin/browse --version and browse open --help Tarball installed; publish rewrite changed workspace:* to Stagehand 4.1.0; installed binary reported browse/0.9.6; V4 --remote and --cdp flags were present. Proves the publish artifact is consumable outside the monorepo and does not leak an invalid workspace range. Product code is unchanged at the current head.
Product proof head 1dd7d0c8: built ESM prepareBrowseCliHarnessAdapter twice Metadata resolved the workspace packages/cli/bin/run.js at version 0.9.6; generated sessions and workdirs were distinct. Proves evals consume the workspace CLI and isolate concurrent/per-row state.
Product proof head 1dd7d0c8: execute the generated pinned wrapper as a real subprocess: browse open https://example.com, then browse get title Wrapper launched a real Browserbase V4 session, returned selected-target/session identity fields, and read Example Domain. Proves the wrapper executes the pinned entrypoint and appends its environment/session flags correctly.
Product proof head 1dd7d0c8: invoke browse cloud contexts list through that wrapper Rejected locally with exit 64. Proves non-driver commands cannot escape the eval harness boundary.
Product proof head 1dd7d0c8: register both live adapter cleanups; cooperative abort; aggressive abort Cooperative abort preserved both workdirs; aggressive abort stopped cleanup paths and removed both workdirs. Proves the two-stage abort semantics and real adapter cleanup are wired together.

@browserbasehq/stagehand-evals is private and is intentionally tested as a built workspace consumer, while the public browse package is additionally tested as an isolated packed consumer.

@changeset-bot

changeset-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: f13b62d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Pin eval agents to the V4 workspace CLI, isolate per-run sessions, restrict the wrapper to driver commands, and guarantee cleanup on abort and process shutdown.
@shrey150
shrey150 force-pushed the agent/browse-v4-6-release branch from f8e2579 to f13b62d Compare September 11, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant