Prose-QA

Beta — Prose-QA is under active development. Expect rough edges and breaking changes between releases.

Write what you want to test in plain text, and let Prose-QA do the rest. This autonomous, LLM-driven testing engine executes complex web workflows and validation checkpoints without the overhead of heavy browser wrappers, bringing frictionless QA to modern development.

Requires Node.js 24+, PQA_LLM_API_KEY, and llm.provider / llm.model in config.

Quick start

npm install prose-qa

pqa install-browser chrome   # or lightpanda (headless, lighter for CI)

pqa config llm.provider [openai-compatible|fireworks|openai|anthropic|google]
pqa config llm.model [model-string]

export PQA_LLM_API_KEY=...
pqa run scenarios/**/*.md

New project checklist

1. Install the package in your app repo (or globally with npm install -g prose-qa).
2. Install a browser: pqa install-browser chrome or pqa install-browser lightpanda (headless only but lighter — good for CI).
3. Create pqa.config.json — use pqa config <key> <value> or copy the minimal example.
4. Add scenarios under scenarios/ (see 0_hello-world.md; advanced patterns in create-pqa-scenario skill).
5. Run pqa run or pqa debug.

What you get

• Natural language scenarios — # Goal, # Steps, and # Then checkpoints (format guide)
• CI + local debug modes with HTML/JSON reports
• MCP Server to help your usual agent create scenario tailored to your codebase
• Cache, healing, recording, and analysis — see HOWTO
• Extensions / hooks — lifecycle callbacks for auth, browser health, and custom logic (extensions guide)

Documentation

Doc	Purpose
docs/HOWTO.md	Step-by-step guide: scenarios → run → CI → MCP → record → cache → healing → analyze
docs/CONFIG.md	Full configuration reference
docs/extensions.md	Hooks system, defaultExtensionHooks, parallel workers
CONTEXT.md	Glossary (run unit, provisioning, profiles)
CONTRIBUTING.md	Pull request guidelines
SECURITY.md	Vulnerability reporting, secrets, and run artifacts
recorder-extension/README.md	Chrome extension recorder (WIP)

Run pqa help or pqa help <command> for the full CLI reference (src/cli/help.ts).

CLI

Command	Description
pqa config <key> <value>	Set a value in pqa.config.json
pqa run [globs]	Run scenarios (headless by default)
pqa debug [globs]	Verbose debug run (headed by default)
pqa clear-cache [scenario]	Clear scenario replay cache
pqa analyze [run...]	Post-run analysis and flaky detection (--last N)
pqa auth list / clear [profile]	List or clear cached auth profiles
pqa install-browser chrome / lightpanda	Install browser binaries for agent-browser
pqa record start / note / checkpoint / stop	Record browser actions → scenario markdown
pqa record generate <dir>	Regenerate scenario markdown from a saved recording
pqa skills list / show / sync	Discover and inspect agent skills
pqa mcp	Start MCP server (Cursor, Claude Desktop, …)
pqa help [command...]	Show CLI help for one or all commands

Tag filters, retries, and cache flags: see HOWTO §3–§4 and HOWTO §10.

Exit codes: 0 pass · 1 failure · 2 config/harness error

Configuration

Supported filenames (first match wins): pqa.config.json, pqa.config.mjs, pqa.config.js, pqa.config.ts.

{
  "envVars": ["PQA_TEST_EMAIL", "PQA_TEST_PASSWORD"],
  "llm": {
    "provider": "anthropic",
    "model": "claude-sonnet-4-20250514"
  }
}

Variable	Required when
PQA_LLM_API_KEY	Any cloud llm.provider (not openai-compatible)
PQA_LLM_BASE_URL	Required when llm.provider is openai-compatible (unless set in config)
PQA_LLM_PROVIDER	Optional env shortcut for llm.provider
PQA_LLM_MODEL	Optional env shortcut for llm.model

All options, env vars, and a full example: docs/CONFIG.md.

MCP

Add the following to your mcp.json:

{
  "mcpServers": {
    "prose-qa": {
      "command": "npx",
      "args": ["-y", "prose-qa", "mcp"],
      "cwd": "/path/to/your-app-with-pqa.config"
    }
  }
}

Tools: validate_scenario, run_scenario, get_create_pqa_scenario_skill. Details: HOWTO §7.

Development (this repo)

git clone https://github.com/FreakDev/Prose-QA.git
cd Prose-QA
npm ci && npm run install-chrome

npm run build   # required — bundled pqa.config.ts imports dist/hooks/defaults.js

export PQA_LLM_API_KEY=...

npm run demo:server   # terminal 1 — http://127.0.0.1:8080/
npm run dev -- debug scenarios/0_hello-world.md
npm run dev -- run scenarios/**/*.md --tags example   # CI smoke subset
npm run dev -- run scenarios/**/*.md --tags demo       # full demo suite

Bundled demo scenarios cover smoke tests, calculator widgets, auth profiles, multi-field forms, validation errors, navigation, and partials — see scenarios/ and demo-site/. Add scenarios/auth/login-admin.md locally to exercise example-authenticated (see create-pqa-scenario skill).

See CONTRIBUTING.md and docs/HOWTO.md for the full walkthrough.

License

MIT — see LICENSE.