Skip to content

Add critical user flow test gate - #125

Open
iatopilskii wants to merge 2 commits into
databricks:mainfrom
iatopilskii:critical-user-flow-tests
Open

Add critical user flow test gate#125
iatopilskii wants to merge 2 commits into
databricks:mainfrom
iatopilskii:critical-user-flow-tests

Conversation

@iatopilskii

Copy link
Copy Markdown

Summary

Adds a dedicated Playwright release gate for DevHub’s most critical developer and coding-agent user flows.

The suite runs against a local production build. It covers desktop Chromium and WebKit plus explicitly named Playwright-emulated Pixel and iPhone profiles. No production monitoring, scheduled checks, or alerts are added.

Why

DevHub already had broad Vitest and Chromium regression coverage, but there was no isolated cross-browser gate for journeys whose failure prevents a developer or coding agent from getting useful content.

Critical flows covered

  • Home page → fetch and copy a complete bootstrap prompt.
  • Desktop Cmd/Ctrl+K or mobile-menu search → open Databricks CLI documentation → copy useful Markdown.
  • Templates search → open SaaS Subscription Tracker → copy the agent prompt → validate the Replit handoff.
  • Simulated clipboard failure → display the retry state → recover without reloading.
  • Validate coding-agent entry points:
    • /api/bootstrap-prompt
    • /llms.txt
    • Raw documentation
    • MCP tools/list
    • MCP calls to list_docs_resources and get_doc_resource

Implementation

  • Adds playwright.critical.config.ts and a documented P0/P1 test plan.
  • Adds pnpm test:critical and an independent critical-flow CI step.
  • Uses the real browser Clipboard API with explicit permissions in desktop Chromium; deterministic clipboard behavior remains for cross-engine tests and controlled failure injection.
  • Verifies required MCP tools as an extensible subset and calls the tools to validate useful responses.
  • Names device-profile projects emulated-pixel-chromium and emulated-iphone-webkit so CI output cannot be mistaken for physical-device coverage.
  • Refuses to reuse an unrelated local server for the critical production-build gate.
  • Preserves screenshots, traces, and videos for failed critical tests as short-lived CI artifacts.

Validation

  • Critical suite: 17 passed, with 3 browser-independent API duplicates intentionally skipped.
  • MCP tests: 13 passed.
  • pnpm build: passed.
  • pnpm typecheck: passed.
  • Prettier check: passed.
  • Content validation: passed.
  • Image verification: passed.
  • fallow dead-code: no issues.
  • Manual browser and SEO smoke checks: passed with no runtime errors; title, H1, and canonical remain correct.

Existing baseline failures

The broader regression suite still reports unrelated failures in files not changed by this PR:

  • Vitest: 311 passed; 2 AppKit documentation broken-link tests failed.
  • Existing Chromium Playwright suite: 182 passed; 1 layout-height assertion failed by 6 px after the latest AppKit docs sync.
  • fallow dupes reports existing duplication groups in untouched production files.

Test plan

See tests/critical/README.md for priorities, expected outcomes, browser coverage, and local commands.

@iatopilskii
iatopilskii marked this pull request as ready for review August 1, 2026 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant