Skip to content

Evaluation option matrix, API/responsive coverage, org-model API tests - #20

Merged
saqibmanan merged 40 commits into
mainfrom
fix/api-tests-use-org-models
Nov 6, 2026
Merged

Evaluation option matrix, API/responsive coverage, org-model API tests#20
saqibmanan merged 40 commits into
mainfrom
fix/api-tests-use-org-models

Conversation

@saqibmanan

@saqibmanan saqibmanan commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Opt-in evaluation matrix e2e (EVAL_MATRIX=pairwise|all): 14 pairwise / 135 total combinations, create -> run -> COMPLETED -> results -> report download. Non-Opus models only.
  • API round-trip for all 9 evaluator x scope options, plus matrix definition checks (smoke).
  • Responsive check for the Start-an-Evaluation modal (phone/tablet).
  • API tests now use existing org-1 models (never Opus); skips resolved.
  • Failure screenshots on fixture setup errors and in CI artifacts; session refresh verifies login and retries.
  • Stale xfails removed; ledger row 31 added.

Prod results (2026-11-05)

12 of 14 pairwise runs passed end to end. Two General-scope library runs stalled mid-run (audits 910/911) -> ParakhAI-Backend#134.

Notes

  • Matrix runs create real audits; completed ones are kept as evidence. Needs SANDBOX_ORG_SLUG.
  • The 135-case run was not executed.

@saqibmanan
Add first_org_model_id helper that skips Opus models
f33044f
@saqibmanan
Use org models in auditor model-shape test instead of myModels
0aad9b0
@saqibmanan
Use org models in playground tests instead of myModels
0b5035e
@saqibmanan
Accept DRAFT in runAudit status check when the run is rejected
3e10245
@saqibmanan
Take failure screenshot on fixture setup errors, not only test-body f…
468de44 
…ailures
@saqibmanan
Upload FAIL_*.png screenshots with the smoke report artifact
cf11a8d
@saqibmanan
Remove stale UX-005 overview-h1 xfail: h1 is present on dev and prod
6f756db
@saqibmanan
Remove stale CDS-001 xfails: editor no longer throws the JS syntax error
d74aaed
@saqibmanan
Mark ledger bug #14 fixed (overview h1 present on dev and prod)
d86f453
@saqibmanan
Add evaluation option matrix: 135 combos and a 14-run pairwise subset
cdbc7f2
@saqibmanan
Add locators for evaluation domain, custom prompts and playground steps
35f8517
@saqibmanan
Page object: modal domain select, per-module sub-modules, custom prom…
43842c3 
…pts, playground steps
@saqibmanan
Add opt-in evaluation matrix test: create, run to COMPLETED, verify r…
379dc05 
…esults and report
@saqibmanan
Playground helper: wait for generation, then Passed and Save and Test…
77228ac 
… New Input
@saqibmanan
Matrix test: playground finish is async too, use the full wait budget
59e0f10
@saqibmanan
Playground helper: fill the Evaluation Recommendation dialog when fin…
f10e13f 
…ishing
@saqibmanan
Modal random model pick: never choose an Opus model
036f637
@saqibmanan
Matrix test: generate the report before downloading it
ccbaef4
@saqibmanan
Per-module sub-module helper: reuse the working combobox trigger locator
4a85bcb
@saqibmanan
Matrix test: progress-aware wait (fail on stall or hard cap, not a fi…
078ee74 
…xed 10 min)
@saqibmanan
Page object: real uncheck_module, set_modules, select_smallest_prompt…
a1e3ccd 
…_library
@saqibmanan
Matrix test: exact module selection and smallest prompt library
a1c55c7
@saqibmanan
test: verify and retry the session refresh login so long runs do not …
dbfca0a 
…start logged out
@saqibmanan
test: pick bulk sub-modules with Enter on the highlighted option (cli…
3337b77 
…ck does not select)
@saqibmanan
test: API round-trip of all 9 evaluator x scope options plus matrix d…
a255bd9 
…efinition checks
@saqibmanan
test: responsive check for the Start-an-Evaluation modal
585dfe9
@saqibmanan
docs: ledger row 31 for stalled bulk library audits (ParakhAI-Backend…
97d4cf0 
…#134)
@saqibmanan
test: wait for the AI Maker overview h1 instead of a fixed 2s (flaky …
6041b59 
…on slow CI)
@saqibmanan
test: prompt-library tests follow the renamed KCC libraries; drop rem…
7519ef7 
…oved Public Interest Hindi
@saqibmanan
test: wait for the list URL after Back to List instead of reading it …
965ba3a 
…immediately
@saqibmanan saqibmanan closed this Nov 5, 2026
@saqibmanan saqibmanan reopened this Nov 5, 2026
@saqibmanan
saqibmanan merged commit 8f37e3e into main Nov 6, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@saqibmanan