diff --git a/.github/workflows/run-smoke.yml b/.github/workflows/run-smoke.yml index 7c92aa1..5c81f73 100644 --- a/.github/workflows/run-smoke.yml +++ b/.github/workflows/run-smoke.yml @@ -225,5 +225,7 @@ jobs: uses: actions/upload-artifact@v7 with: name: smoke-test-report - path: reports/smoke_report.html + path: | + reports/smoke_report.html + screenshots/FAIL_*.png retention-days: 30 diff --git a/docs/app_bugs.md b/docs/app_bugs.md index d5db3de..94df4bc 100644 --- a/docs/app_bugs.md +++ b/docs/app_bugs.md @@ -20,7 +20,7 @@ Format: append-only. When a bug is fixed in the app, mark `status: fixed` and th | 12 | auth / keycloak-dev | **Keycloak dev instance rate-limits ≥5 concurrent logins from the same account.** Under a 5-parallel-login load test, 3/5 workers fail with "login form not rendered" even after a 15 s wait — the form simply never appears. The 3-concurrent test (same account) passes consistently. Root causes (any combination): Keycloak brute-force-protection window, single-node session-table lock contention, or the dev nginx upstream thread pool exhaustion. **Not a bug in the Parakh frontend.** Upgrade path: HA Keycloak with session affinity disabled and per-user rate-limit tuning. | open (infra) | Run `pytest tests/load/test_load.py::TestConcurrentAuthentication -v -s` and observe 3/5 workers timing out on the login form. The 3-concurrent variant in the same class passes. | `tests/load/test_load.py::TestConcurrentAuthentication::test_5_concurrent_logins_success_rate` | 2026-05-23 | | 11 | wizard / manual-workspace | The manual-evaluation workspace Test Cases tab fails to render its expected UI elements when reached via the Domain type flow: (a) module cards appear (counter confirmed) but clicking the first card does not open an input/output panel — neither `MANUAL_INPUT_TEXTAREA` nor `MANUAL_CHANGE_MODULE_LINK` becomes visible; (b) the minimum test-cases hint ("Evaluate at least 3 test cases per module…") is not rendered on the Test Cases tab. These were previously skipped with `pytest.skip` guards; they now surface as failures. | open | Log in → AI Maker → CivicDataLab → New Evaluation → Start → select Domain type → fill Objective → click Test Cases tab. Click any module card and observe whether an input panel appears. Also check whether the min-cases hint text is rendered below the module list. | `tests/e2e/test_evaluation_workspace_manual.py::TestManualWorkspaceTestEntry::test_module_click_opens_input_panel`; `tests/e2e/test_evaluation_workspace_manual.py::TestManualWorkspaceTestEntry::test_min_test_cases_note_visible` | 2026-05-23 | | 13 | evaluations list / backend perf | The frontend `GetAudits` query that powers the AI-Maker Evaluations list (`/dashboard/ai-maker/{org}/evaluations`) **hangs and never returns** on dev — the page sits on "Loading evaluations…" indefinitely (still up at 90 s). Root-caused 2026-06-12 via live Playwright + direct GraphQL probing on org 1 (981 total audits): requesting the **light** field set (`id name status modelId`, `limit:100`) returns in ~10.6 s, and `metrics`-only or `modules`-only at `limit:100` each return in ~16–18 s, but the **full field set the frontend actually requests** (`name modelName status modules metrics evaluationMode auditType totalTests passedTests failedTests createdAt startedAt completedAt`) **times out past 30 s even at `limit:5`** — so it is the combined per-row resolver cost (likely N+1 on `metrics`/`modules`/test-count fields), not row volume. Backend should batch/precompute these fields or paginate server-side. Also observed intermittent `ERR_NETWORK_CHANGED` / "Backend server is not available" GraphQL errors on dev during the same session (degraded backend). **Not a test defect** — no client-side timeout bump can help since the query never completes. | fixed 2026-08-12 (see verification note) | Log in → AI Maker → CivicDataLab → Evaluations. Page stays on "Loading evaluations…". In DevTools, the `GetAudits` POST to `dev.api.parakh.civicdataspace.in/graphql/` has no response after 90 s. Confirmed via Playwright MCP 2026-06-12: full-field `audits` query aborts at 30 s even with `limit:5`, while light-field returns in ~10 s. | `tests/e2e/test_evaluations.py::TestStatusFilterTabs::*` (all 6 — tabs/column/rows render only after the list query resolves); transitively any test that asserts on rendered evaluations-list content for org 1; `tests/e2e/test_evaluations.py::TestEvaluationsListPage::test_table_column_headers_are_present` (added 2026-07-31 — re-run isolated, no concurrent CI load: page still stuck on "Loading evaluations..." past 40s, confirming the hang reproduces without concurrent-shard traffic) | 2026-06-12 | -| 14 | a11y / dashboard heading | The AI Maker Dashboard (`/dashboard/ai-maker/1`) never renders any `

`/`

`/`

` heading — `document.querySelectorAll('h1,h2,h3').length` is `0` even after a 30s wait. Root cause: the "Overview" widget (which appears to be where the page heading lives) is permanently stuck on "Loading overview..." and never resolves — the same failure shape as bug #13 (backend `GetAudits`/overview resolver hang), just surfacing on the dashboard-home overview query instead of the evaluations-list query. Because the widget never mounts, the page has zero heading elements, violating WCAG 1.3.1 (Info and Relationships) / a broken document outline for screen-reader users. **Update 2026-08-13:** Phase 12 (2026-08-12) marked this `fixed` after a clean re-verification, but noted the xfails referencing it hadn't been removed pending a re-run. That re-run surfaced different evidence: in a full e2e run (`--splits 12 --group 2`), `test_add_new_model_button_is_visible` and `test_model_cards_are_rendered` genuinely XFAILED (the bug fired), while `test_civicdatalab_name_visible_in_sidebar` and `test_text_generation_badge_is_present` XPASSED in the same run. Re-running the two XFAILing tests in true isolation (`-n 1 --reruns 0`, no other suite active) immediately afterward, both XPASSED cleanly. This is NOT the bug #3 concurrency-noise shape (isolation didn't consistently clear it either way) — it's a genuinely intermittent defect, not a fixed-vs-broken binary. **Do not remove the `strict=False` xfails on any of the four related tests** — they correctly tolerate a defect that still recurs some fraction of the time. | intermittent (re-opened 2026-08-13 — was marked fixed 2026-08-12, see update above; keep the xfails) | Log in as `TEST_EMAIL_1` → navigate directly to `https://dev.parakh.civicdataspace.in/dashboard/ai-maker/1`. Page loads nav/sidebar/breadcrumbs but the main content area shows only "Loading overview..." indefinitely (confirmed stuck past 30s). Run `document.querySelectorAll('h1,h2,h3').length` → `0`. Confirmed via Playwright 2026-07-10. | `tests/accessibility/test_accessibility_auth.py::TestDashboardAccessibility::test_dashboard_has_exactly_one_h1`; `tests/e2e/test_ai_maker_dashboard.py::TestAIMakerDashboardLoads::test_civicdatalab_name_visible_in_sidebar`, `TestModelCardsonHome::test_add_new_model_button_is_visible`, `TestModelCardsonHome::test_model_cards_are_rendered`, `TestModelCardsonHome::test_text_generation_badge_is_present` (added 2026-07-31 — re-run isolated (no concurrent CI load), direct repro script confirmed the whole home content area, not just the heading, is gated behind the same stuck "Loading overview..." widget: org identity block (`CivicdataLab` text), model cards, `Add A New Model` button, and model-type badges never render because they all read from the same overview payload) | 2026-07-10 | +| 14 | a11y / dashboard heading | The AI Maker Dashboard (`/dashboard/ai-maker/1`) never renders any `

`/`

`/`

` heading — `document.querySelectorAll('h1,h2,h3').length` is `0` even after a 30s wait. Root cause: the "Overview" widget (which appears to be where the page heading lives) is permanently stuck on "Loading overview..." and never resolves — the same failure shape as bug #13 (backend `GetAudits`/overview resolver hang), just surfacing on the dashboard-home overview query instead of the evaluations-list query. Because the widget never mounts, the page has zero heading elements, violating WCAG 1.3.1 (Info and Relationships) / a broken document outline for screen-reader users. **Update 2026-08-13:** Phase 12 (2026-08-12) marked this `fixed` after a clean re-verification, but noted the xfails referencing it hadn't been removed pending a re-run. That re-run surfaced different evidence: in a full e2e run (`--splits 12 --group 2`), `test_add_new_model_button_is_visible` and `test_model_cards_are_rendered` genuinely XFAILED (the bug fired), while `test_civicdatalab_name_visible_in_sidebar` and `test_text_generation_badge_is_present` XPASSED in the same run. Re-running the two XFAILing tests in true isolation (`-n 1 --reruns 0`, no other suite active) immediately afterward, both XPASSED cleanly. This is NOT the bug #3 concurrency-noise shape (isolation didn't consistently clear it either way) — it's a genuinely intermittent defect, not a fixed-vs-broken binary. **Do not remove the `strict=False` xfails on any of the four related tests** — they correctly tolerate a defect that still recurs some fraction of the time. | fixed 2026-11-05 (h1 "Overview" present on dev and prod; xfail removed) | Log in as `TEST_EMAIL_1` → navigate directly to `https://dev.parakh.civicdataspace.in/dashboard/ai-maker/1`. Page loads nav/sidebar/breadcrumbs but the main content area shows only "Loading overview..." indefinitely (confirmed stuck past 30s). Run `document.querySelectorAll('h1,h2,h3').length` → `0`. Confirmed via Playwright 2026-07-10. | `tests/accessibility/test_accessibility_auth.py::TestDashboardAccessibility::test_dashboard_has_exactly_one_h1`; `tests/e2e/test_ai_maker_dashboard.py::TestAIMakerDashboardLoads::test_civicdatalab_name_visible_in_sidebar`, `TestModelCardsonHome::test_add_new_model_button_is_visible`, `TestModelCardsonHome::test_model_cards_are_rendered`, `TestModelCardsonHome::test_text_generation_badge_is_present` (added 2026-07-31 — re-run isolated (no concurrent CI load), direct repro script confirmed the whole home content area, not just the heading, is gated behind the same stuck "Loading overview..." widget: org identity block (`CivicdataLab` text), model cards, `Add A New Model` button, and model-type badges never render because they all read from the same overview payload) | 2026-07-10 | | 15 | routing / auth deep-link | Direct navigation to authenticated AI-Maker / Auditor sub-routes (`/dashboard/ai-maker/1/ai-models`, `/dashboard/ai-maker/1/evaluations/new`, `/dashboard/ai-maker/1/auditors`, `/dashboard/ai-maker/1/prompt-libraries`, `/dashboard/auditor`) **intermittently renders the public marketing homepage** (hero "Build AI that's trustworthy from day one.", feature tabs, CTA cards, footer) instead of the requested dashboard page, even though the user is authenticated (avatar/session valid) and the URL bar shows the correct path. Root-caused 2026-07-10 via a scripted repro: a fresh browser context seeded with a valid `storage_state` (cookies + localStorage from a real Keycloak login) navigating straight to `/dashboard/ai-maker/1/ai-models` with `wait_until="load"` + 2.5s settle reproduced the homepage fallback in **1 of 6 trials (~17%)**; the same repro against `/dashboard/ai-maker/1/prompt-libraries` reproduced it in **1 of 4 trials (~25%)**, and a further capture landed in a visibly half-transitioned state (both "Prompt Libraries" page chrome and homepage CTA/footer content stacked in the same viewport — see `screenshots/DIFF_auth_prompt_libraries_desktop_1440x900.png`). When it does NOT reproduce, the correct page renders fully and correctly (light theme, correct heading, working sidebar, "Loading AI models…" spinner resolving normally). This is a materially different and more severe symptom than the "non-deterministic async content" (live stats, model dropdown ordering) that `_NON_DETERMINISTIC_VISUAL` in `tests/visual/test_visual_regression.py` currently attributes these routes' diffs to — the wizard/models/auditors/auditor-dashboard visual tests see ~43% pixel diffs because they are sporadically capturing the wrong page entirely, not just varying live data. Likely a client-side routing/auth-guard race on hard navigation (the route guard falls through to the default/public route before the session check resolves), since it only reproduces on deep-link/full-page loads, not in-app client-side navigation. | open | Using a Playwright context seeded with a valid Keycloak `storage_state`, repeatedly `page.goto()` directly to `/dashboard/ai-maker/1/ai-models` (or any route in the list above) with `wait_until="load"` and a 2.5s+ settle delay; repeat ~5-6 times. Roughly 1 in 5-6 loads renders the homepage hero/CTA/footer instead of the dashboard page content. Confirmed via ad hoc Playwright script 2026-07-10 (not yet encoded as a standing test — see `docs/visual_diffs.md` Run 2026-07-10 for the diff images that first surfaced it). | `tests/visual/test_visual_regression.py::TestAuthenticatedPageVisuals::test_authenticated_page_desktop[chromium-models_list]`, `[chromium-new_evaluation_wizard]`, `[chromium-auditors_management]`, `[chromium-auditor_dashboard]`, `[chromium-prompt_libraries]` (all xfail under VISUAL-002 — the xfail reason text should be revisited given this finding); `tests/e2e/test_user_flows.py::TestFlow04_EvaluationDetailAndReport::test_evaluation_detail_shows_full_results` (added 2026-07-13 — deep-link to `/evaluations/{id}` rendered the public homepage; avatar/session chrome still present, confirming auth was intact); `tests/visual/test_visual_regression.py::TestAuthenticatedPageVisuals::*` (2026-08-12 — the new capture-integrity gate in `utils/visual_guards.py` now detects this automatically and skips instead of baselining it) **— SEVERITY UPDATE 2026-08-12: observed at ~50%, far above the 17-25% recorded above.** Two consecutive full visual runs hit the logged-out fallback on **12 of 24 authenticated captures** (5/12 then 7/12). Affected: `/dashboard/ai-maker/1`, `/ai-models`, `/evaluations`, `/evaluations/new`, `/auditors`, `/prompt-libraries`, `/dashboard/auditor`. NOT affected in either run: `/dashboard` (role selector), `/dashboard/ai-maker` (org selector), `/dashboard/auditor/evaluations` — so it is not uniform across depth. New detail vs the 2026-07-10 characterisation: the fallback now renders **logged-out chrome (LOGIN / SIGN UP)**, i.e. the session is not merely failing to apply to the route — the shell itself renders unauthenticated. Caveat on the comparison: both figures are small samples (the original was 1/6 and 1/4 trials) and load conditions differed, so treat this as "materially worse under current conditions" rather than a precise regression measurement. Independent confirmation that it also fired historically: four baselines were byte-identical copies of the logged-out homepage. `test_evaluation_detail_shows_full_results`'s xfail was missing `strict=False` (this repo's `pytest.ini` sets `xfail_strict = true` globally) — added 2026-08-13 so a lucky run's XPASS is tolerated instead of failing the suite, consistent with bug #14's established convention for intermittent (not always-reproducing) bugs. | 2026-07-10 | | 16 | SEO / homepage content | Homepage `` contains a spelling error: `"Paricipatory AI Evaluation"` (missing the first `t` in "Participatory"). Root-caused in frontend source: `app/layout.tsx` line 180, `export const metadata = { ..., description: "Paricipatory AI Evaluation" }`. One-line content fix. | open | Load `https://dev.parakh.civicdataspace.in/`. Run `document.querySelector('meta[name="description"]').content` → `"Paricipatory AI Evaluation"`. Confirmed against `ParakhAI-frontend/app/layout.tsx:180` 2026-07-13. | `tests/e2e/test_platform_issues.py::TestHomepageSEO::test_meta_description_has_no_typo` | 2026-07-13 | | 17 | SEO / homepage content | Homepage has no Open Graph `` tag (or any other `og:*` tags) — the root `metadata` export in `app/layout.tsx` only sets `title` and `description`, no `openGraph` block. Links to the homepage shared on social media / chat apps will render with no title card. | open | Load `https://dev.parakh.civicdataspace.in/`. Run `document.querySelector('meta[property="og:title"]')` → `null`. Confirmed against `ParakhAI-frontend/app/layout.tsx` (metadata export has no `openGraph` key) 2026-07-13. | `tests/e2e/test_platform_issues.py::TestHomepageSEO::test_og_title_is_present` | 2026-07-13 | @@ -42,6 +42,8 @@ Format: append-only. When a bug is fixed in the app, mark `status: fixed` and th | 28 | evaluations list / sorting | Clicking a sortable column header (e.g. "Evaluation Name") on the evaluations table's first click does **nothing at all** — no visual reorder, and the header's `aria-sort` attribute stays `None`/absent. The second click (and every click after) does re-sort the rows correctly. Confirmed live 2026-08-13 with a scripted repro that clicked the "Evaluation Name" header 3 times in a row with a 3s settle between each: click 1 → row order and `aria-sort` unchanged from the pre-click state; click 2 → rows visibly reordered; click 3 → rows reordered again — but `aria-sort` never changes from `None` across all three clicks, so screen-reader users get no indication a sort is active even once engaged (a secondary a11y defect on top of the primary one-click-eaten defect). The `DataTable-module_Heading__*` header button is rendered by the shared `opub-ui` `DataTable` component (no local source in `ParakhAI-frontend` — it's a compiled dependency), so the root cause could not be pinned to an exact line; likely a debounce/state-init bug where the component's internal sort-state only starts reacting to clicks after the first one initializes it. | open | Log in as `TEST_EMAIL_1` → AI Maker → CivicDataLab → Evaluations. Click the "Evaluation Name" column header once. Row order is unchanged and `document.querySelector("th button:has-text('Evaluation Name')").getAttribute('aria-sort')` stays `null`. Click it again — rows now visibly reorder. | `tests/e2e/test_evaluations_list_controls.py::TestSortableHeaders::test_sort_by_name_toggles_order` | 2026-08-13 | | 30 | evaluations list / default sort | The default (unfiltered, unsorted) `audits` GraphQL query the evaluations list loads with does **not** order by recency — a newly created draft never appears on the list's landing page, no matter how long you wait or how many times you hard-reload. Root-caused 2026-08-17 via a live network capture of the actual `GetAudits` response (not a synthetic probe): four separate fresh-draft creations in the same org (audit ids 2042–2045, confirmed to exist individually via `audit(auditId)` and to rank at the top of a raw `audits(limit:20)` GraphQL call with no `sortOptions`) each produced an identical captured frontend response — `ids=['1916','1957','1953','1943','1944','1954','1961','1909','1964','1958']`, `total=141` — completely unrelated to the just-created id and byte-identical across all four runs, including after a full `page.reload()` + 45s app-ready wait. Response headers carried no `cache-control`/`age`/`x-cache` (200, no HTTP caching), so this isn't a CDN/browser cache — the frontend's actual `GetAudits` call (with its real field selection, distinct from a bare `id name status` probe) is deterministically returning the same fixed page of ~10 rows regardless of what's been created since, meaning its effective sort key is something other than recency (or a query-plan/index quirk that happens to be stable across requests). User impact: creating an evaluation and clicking "Back to List" gives no way to find your own draft without knowing to search or filter — among 141 rows it could be anywhere. Related to bug #13 (same `audits`/`GetAudits` query family, full-field-vs-light-field behavior) but a distinct symptom — not a hang, a wrong/stale-looking default ordering. | open | Log in as `TEST_EMAIL_1` → AI Maker → CivicDataLab → New Evaluation → Bulk → Start (creates a draft, capture its `auditId` from the URL) → Back to List. The draft's row (`a[href*='auditId=']`) is absent from the table even after a full page reload and a 45s wait. Confirm via GraphQL that the audit exists (`audit(auditId: "")`) and that a raw `audits(limit:20)` call (no `sortOptions`) ranks it near the top — yet the frontend's own list request returns a fixed, unrelated set of older ids. Reproduced 4/4 times in complete isolation (`-n 1 --reruns 0`, no concurrent suite load) 2026-08-17. | `tests/e2e/test_add_evaluation_bulk.py::TestBulkDraftPersistence::test_new_draft_appears_in_draft_tab_with_wizard_link` | 2026-08-17 | +| 31 | bulk evaluation / library run stall | Two Bulk runs on prod using a prompt library on the General scope froze mid-run: audit 910 (Cultural, Bias, Gemma 4 26B) at 74% and audit 911 (Technical, all 3 modules, GPT-5 Mini) at 84%, both IN_PROGRESS with `totalTests` 0 for 30+ min. `updateAudit(status: CANCELLED)` left them IN_PROGRESS. 8 other bulk runs (library and custom, all scopes) completed in 1-3 min. Found by the pairwise matrix (`tests/e2e/test_evaluation_matrix.py`, opt-in via `EVAL_MATRIX`). Filed as ParakhAI-Backend#134 (2026-11-05); not fixed. Companion observation: audit 912 (Haiku) went FAILED in seconds with `No successful AuditTest objects found` — model inference failure, passes on re-run. Follow-up 2026-11-06: re-ran the General/Technical/all-modules/library case once; audit 941 went FAILED in ~47s instead of stalling, so the stall is not deterministic (model chosen at random by the test; failure reason not read). | +| 32 | evaluations list / status filter + pagination | Applying a status filter while on page 2+ keeps the old page offset. The list shows `GraphQL Error: offset (10) is out of range (0–2)` with a Retry button instead of the filtered rows. Seen in CI 2026-11-05 (screenshot of shard 2). Filed as ParakhAI-frontend#477. | open | Evaluations list with 2+ pages → click Next → open Filter Status → pick a status → Apply. | `tests/e2e/test_evaluations.py::TestEvaluationsPagination::test_status_filter_resets_to_first_page` (xfail, non-strict) | 2026-11-05 | ## Conventions diff --git a/locators/evaluations_locators.py b/locators/evaluations_locators.py index 132d8f6..c572d66 100644 --- a/locators/evaluations_locators.py +++ b/locators/evaluations_locators.py @@ -404,3 +404,14 @@ class EvaluationsLocators: # Report download DOWNLOAD_REPORT_BUTTON = "button:has-text('Download Report'), a:has-text('Download Report')" + + # Evaluation matrix (mapped on prod 2026-11-05) + MODAL_EVALUATION_DOMAIN_SELECT = "select[name='evaluationDomain']" + WIZARD_ADD_INPUT_PROMPT_BUTTON = "button:has-text('Add Input Prompt')" + WIZARD_CUSTOM_TEST_INPUT = "textarea[name='testInput']" + WIZARD_CUSTOM_REFERENCE_OUTPUT = "textarea[name='referenceOutput']" + PLAYGROUND_INPUT = "textarea[name='inputPrompt']" + PLAYGROUND_SUBMIT = "button:has-text('Submit')" + PLAYGROUND_PASSED = "button:has-text('Passed')" + PLAYGROUND_GENERATING = "text=Generating" + PLAYGROUND_SAVE_NEXT = "button:has-text('Save and Test New Input')" diff --git a/locators/prompt_libraries_locators.py b/locators/prompt_libraries_locators.py index 46496d2..c085f7b 100644 --- a/locators/prompt_libraries_locators.py +++ b/locators/prompt_libraries_locators.py @@ -29,9 +29,8 @@ class PromptLibrariesLocators: UPLOAD_FILES_TOGGLE = "text=Upload files, text=Upload Files" # Known library names - LIBRARY_KCC_ENGLISH = "text=KCC) - English Queries" + LIBRARY_KCC_ENGLISH = "text=KCC English Queries" LIBRARY_PARIKSHA = "text=Microsoft: PARIKSHA" LIBRARY_PUBLIC_INTEREST_ENGLISH = "text=Public Interest English" - LIBRARY_PUBLIC_INTEREST_HINDI = "text=Public Interest Hindi" - LIBRARY_KCC_HINDI = "text=KCC): Hindi Queries" + LIBRARY_KCC_HINDI = "text=KCC Hindi Queries" LIBRARY_KISAN_CALL = "text=Kisan Call Centre" diff --git a/pages/new_evaluation_page.py b/pages/new_evaluation_page.py index 462fd83..63065c4 100644 --- a/pages/new_evaluation_page.py +++ b/pages/new_evaluation_page.py @@ -191,7 +191,7 @@ def select_model_by_index(self, index: int = 1) -> None: EXCLUDED_MODEL_NAMES = {"New AI Model", "xAI: Grok 4.1 Fast"} def select_random_valid_model_and_version(self) -> str: - """Select a random model, excluding 'New AI Model' and deprecated 'xAI: Grok 4.1 Fast'. + """Select a random model, excluding 'New AI Model', deprecated 'xAI: Grok 4.1 Fast' and any Opus model (cost). Returns the selected model's display text (useful for logging/assertions). Falls back to index 1 if every option is excluded or the list is empty. @@ -201,7 +201,7 @@ def select_random_valid_model_and_version(self) -> str: options = self.get_modal_model_options() valid = [ i for i, (_, text) in enumerate(options) - if text.strip() not in self.EXCLUDED_MODEL_NAMES + if text.strip() not in self.EXCLUDED_MODEL_NAMES and "opus" not in text.lower() ] if not valid: self.select_model_by_index(1) @@ -278,6 +278,12 @@ def select_evaluator_type_in_modal(self, eval_type: str = "technical") -> None: sel = value_map.get(eval_type.lower(), value_map["technical"]) self.page.locator(sel).first.check() + def select_domain_in_modal(self, domain: str) -> None: + """Pick the step-2 'Evaluation Domain' (healthcare | agriculture | general).""" + self.page.locator(EvaluationsLocators.MODAL_EVALUATION_DOMAIN_SELECT).first.select_option( + domain.upper() + ) + # DOM `value`s are the backend enum (TECHNICAL_AUDIT/DOMAIN_AUDIT/ # CULTURAL_AUDIT); map back to the human-readable label the UI shows # and tests assert against. See locators/evaluations_locators.py. @@ -370,6 +376,7 @@ def start_evaluation_from_modal( eval_type: str = "technical", objective: str = "Automated test evaluation objective", name: str | None = None, + domain: str | None = None, ) -> NewEvaluationPage: """Complete both modal steps and land on the single-page wizard. @@ -389,6 +396,8 @@ def start_evaluation_from_modal( self.select_evaluation_method(method) self.click_modal_next() self.select_evaluator_type_in_modal(eval_type) + if domain: + self.select_domain_in_modal(domain) self.fill_modal_objective(objective) self.click_modal_start() return self @@ -531,6 +540,26 @@ def select_first_prompt_library(self) -> bool: self.page.wait_for_timeout(1_000) return True + def select_smallest_prompt_library(self) -> int | None: + """Pick the library with the fewest (non-zero) test cases on the current + table page and return that count. Big libraries take ~50 min to run.""" + radio_sel = EvaluationsLocators.WIZARD_PROMPT_LIBRARY_RADIO + rows = self.page.locator(f"tr:has({radio_sel})") + best: tuple[int, int] | None = None + for i in range(rows.count()): + digits = [ + int(c.replace(",", "").strip()) + for c in rows.nth(i).locator("td").all_inner_texts() + if c.replace(",", "").strip().isdigit() + ] + if digits and digits[0] > 0 and (best is None or digits[0] < best[0]): + best = (digits[0], i) + if best is None: + return None + rows.nth(best[1]).locator(radio_sel).first.click(timeout=10_000) + self.page.wait_for_timeout(1_000) + return best[0] + def configure_bulk_workspace_minimal(self, module: str = "hallucination") -> bool: """Configure the minimum required to enable Run Evaluation: check a module, pick a sub-module, choose the prompt-library source, and @@ -545,6 +574,58 @@ def configure_bulk_workspace_minimal(self, module: str = "hallucination") -> boo self.select_first_prompt_library() return self.is_run_evaluation_button_enabled() + def select_one_submodule_in_each_module(self) -> int: + """Bulk with several modules shows one sub-module combobox per module; + pick the first option in each. Returns how many were set.""" + boxes = self.page.locator("input[role='combobox']") + done = 0 + for i in range(boxes.count()): + boxes.nth(i).click(timeout=10_000) + self.page.wait_for_timeout(500) + boxes.nth(i).press("Enter") + self.page.mouse.click(5, 5) + self.page.wait_for_timeout(500) + done += 1 + return done + + def add_custom_prompt(self, test_input: str, reference_output: str = "") -> None: + """'Add your own prompts' source: open the Add Input Prompt dialog, + fill it and save (the dialog's Next button is the save action).""" + self.page.locator(EvaluationsLocators.WIZARD_ADD_INPUT_PROMPT_BUTTON).first.click() + field = self.page.locator(EvaluationsLocators.WIZARD_CUSTOM_TEST_INPUT) + field.fill(test_input) + if reference_output: + self.page.locator(EvaluationsLocators.WIZARD_CUSTOM_REFERENCE_OUTPUT).fill(reference_output) + self.page.get_by_role("dialog").get_by_role("button", name="Next").click() + field.wait_for(state="hidden", timeout=10_000) + + def submit_playground_test_case(self, prompt: str, timeout: int = 90_000) -> None: + """Playground: type a prompt, Submit, wait for the model to finish, + mark it Passed and Save. Passed is clickable while the output still + says 'Generating', so wait for that to clear first; only + 'Save and Test New Input' persists the case.""" + self.page.locator(EvaluationsLocators.PLAYGROUND_INPUT).fill(prompt) + self.page.locator(EvaluationsLocators.PLAYGROUND_SUBMIT).first.click() + generating = self.page.locator(EvaluationsLocators.PLAYGROUND_GENERATING).first + try: + generating.wait_for(state="visible", timeout=5_000) + except Exception: # noqa: BLE001 - fast model: it may already be gone + pass + generating.wait_for(state="hidden", timeout=timeout) + self.page.locator(EvaluationsLocators.PLAYGROUND_PASSED).first.click() + self.page.locator(EvaluationsLocators.PLAYGROUND_SAVE_NEXT).first.click() + self.page.wait_for_timeout(2_000) + + def click_finish_evaluation(self, recommendation: str = "Automated playground evaluation.") -> None: + """Finish opens an 'Evaluation Recommendation' dialog: the recommendation + is required and a second 'Finish Evaluation' inside the dialog confirms.""" + self.page.locator(EvaluationsLocators.FINISH_EVALUATION_BUTTON).first.click() + dialog = self.page.get_by_role("dialog") + dialog.get_by_role("textbox").fill(recommendation) + dialog.get_by_role("button", name="Finish Evaluation").click() + dialog.wait_for(state="hidden", timeout=30_000) + self.page.wait_for_timeout(2_000) + def is_run_evaluation_library_error_visible(self) -> bool: """Return True if 'Please select a prompt library…' error is shown.""" return self.is_visible( @@ -735,9 +816,30 @@ def check_module(self, module: str) -> None: # Fallback: the label itself may act as a toggle (custom checkbox) self.click(label_sel) + _MODULE_LABELS = { + "hallucination": "Hallucination and Misinformation", + "bias": "Bias and Fairness", + "privacy": "Privacy and Safety", + } + def uncheck_module(self, module: str) -> None: - """Uncheck an evaluation module checkbox (same logic as check_module).""" - self.check_module(module) # toggle — callers must know current state + """Uncheck a module checkbox if it is currently checked (plain click: + force-clicks double-toggle these custom checkboxes).""" + label = self._MODULE_LABELS[module.lower()] + cb = self.page.locator(f"input[type='checkbox']:near(:text('{label}'))").first + if cb.count() and cb.is_checked(): + cb.click() + self.page.wait_for_timeout(300) + + def set_modules(self, modules) -> None: + """Select exactly `modules`. Hallucination is pre-checked on load, so + check the wanted ones first and then clear the rest.""" + wanted = [m.lower() for m in modules] + for m in wanted: + self.check_module(m) + for m in self._MODULE_LABELS: + if m not in wanted: + self.uncheck_module(m) def module_subcategory_dropdown_visible(self) -> bool: """Return True if a sub-category dropdown is visible (module is checked).""" diff --git a/reports/TEST_REPORT.md b/reports/TEST_REPORT.md index 23c0646..b29be6a 100644 --- a/reports/TEST_REPORT.md +++ b/reports/TEST_REPORT.md @@ -1,30 +1,46 @@ # Parakh Test Framework — Test Report -**Generated:** 2026-07-31 06:13 UTC -**Environment:** `staging` — https://dev.parakh.civicdataspace.in +**Generated:** 2026-11-05 19:57 UTC +**Environment:** `production` — https://parakh.civicdataspace.in **Browser:** `chromium` -**Overall result:** ✅ PASSED +**Overall result:** ❌ FAILED ## Summary | Metric | Value | | ------ | ----- | -| Total tests | 2 | +| Total tests | 1 | | Passed | 0 ✅ | -| Failed | 0 ❌ | -| Skipped | 2 ⏭️ | +| Failed | 1 ❌ | +| Skipped | 0 ⏭️ | +| xfailed | 0 🔶 | +| xpassed | 0 🔷 | | Errors | 0 💥 | | Pass rate | 0.0% | -| Duration | 0.00s | +| Duration | 47.78s | ## Results by Suite -### E2E (2 tests — ✅ 0 / ❌ 0 / ⏭️ 2) +### E2E (1 tests — ✅ 0 / ❌ 1 / ⏭️ 0 / 🔶 0 / 🔷 0) | Test | Result | Duration | | ---- | ------ | -------- | -| `test_evaluator_cards_are_present[chromium]` | ⏭️ skipped | 0.00s | -| `test_at_least_two_evaluators_listed[chromium]` | ⏭️ skipped | 0.00s | +| `test_create_run_complete_and_report[chromium-bulk-technical-general-hallucination+bias+privacy-library]` | ❌ failed | 47.72s | + +## Failure Details + +### ❌ `tests/e2e/test_evaluation_matrix.py::test_create_run_complete_and_report[chromium-bulk-technical-general-hallucination+bias+privacy-library]` + +``` +tests/e2e/test_evaluation_matrix.py:134: in test_create_run_complete_and_report + audit = _wait_status(client, audit_id, ("PENDING_REVIEW", "COMPLETED"), BUDGET_S) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +tests/e2e/test_evaluation_matrix.py:61: in _wait_status + pytest.fail(f"audit {audit_id} ended {last} while waiting for {want}") +E Failed: audit 941 ended FAILED while waiting for ('PENDING_REVIEW', 'COMPLETED') +``` + +**Screenshot:** `/Users/home/CivicDataSpace/ParakhAI_test/screenshots/20261106_012700_FAIL_FAST_test_create_run_complete_and_report[chromium-bulk-technical-general-hallucination+bias+privacy-library].png` --- diff --git a/tests/accessibility/test_accessibility_auth.py b/tests/accessibility/test_accessibility_auth.py index 210d3f3..39b3fa7 100644 --- a/tests/accessibility/test_accessibility_auth.py +++ b/tests/accessibility/test_accessibility_auth.py @@ -443,9 +443,6 @@ def test_ux010_new_evaluation_dialog_aria_describedby_resolves( ) @pytest.mark.auth - @pytest.mark.xfail( - reason="UX-011: Select fields in New Evaluation dialog have no accessible label — known bug" - ) def test_ux011_new_evaluation_dialog_selects_have_labels( self, authenticated_page_fast: Page ): diff --git a/tests/api/test_auditor_management_api.py b/tests/api/test_auditor_management_api.py index ea4bed3..288fdc2 100644 --- a/tests/api/test_auditor_management_api.py +++ b/tests/api/test_auditor_management_api.py @@ -17,7 +17,7 @@ import pytest -from tests.data.test_data import TestGraphQL +from tests.data.test_data import TestGraphQL, first_org_model_id pytestmark = [pytest.mark.api, pytest.mark.regression, pytest.mark.auth] @@ -78,12 +78,10 @@ def test_ai_model_unknown_id_returns_null(self, authenticated_graphql_client): assert result["data"].get("aiModel") is None def test_ai_model_shape_with_valid_model(self, authenticated_graphql_client): - models_result = authenticated_graphql_client(TestGraphQL.QUERY_MY_MODELS) - models = ((models_result.get("data") or {}).get("myModels")) or [] - if not models: - pytest.skip("No AI models available for this user") + model_id = first_org_model_id(authenticated_graphql_client) + if not model_id: + pytest.skip("No non-Opus AI models in the sandbox org") - model_id = models[0]["id"] result = authenticated_graphql_client( TestGraphQL.QUERY_AI_MODEL, variables={"modelId": str(model_id)} ) diff --git a/tests/api/test_evaluation_matrix_api.py b/tests/api/test_evaluation_matrix_api.py new file mode 100644 index 0000000..befe59a --- /dev/null +++ b/tests/api/test_evaluation_matrix_api.py @@ -0,0 +1,64 @@ +"""Evaluation option matrix at the API layer + matrix definition self-check. + +- TestMatrixDefinition: pure data checks on tests/data/evaluation_matrix.py (no network). +- TestEvaluationOptionRoundTrip: every evaluator x domain pair (9) is accepted by + updateAudit and read back unchanged (write; sandbox-gated; drafts are cancelled). +""" + +from itertools import product + +import pytest + +from tests.data.evaluation_matrix import DOMAINS, EVALUATORS, _selfcheck, all_combos, combos_for +from tests.data.test_data import TestGraphQL, first_org_model_id + +_AUDIT_Q = "query($id: ID!){ audit(auditId:$id){ id status auditType auditScope evaluationMode } }" + + +@pytest.mark.smoke +@pytest.mark.regression +class TestMatrixDefinition: + def test_selfcheck_passes(self): + _selfcheck() + + def test_full_space_is_135_unique_cases(self): + ids = [c.id for c in all_combos()] + assert len(ids) == len(set(ids)) == 135 + + def test_selection_is_opt_in(self): + assert combos_for("") == [] and len(combos_for("pairwise")) == 14 + + +@pytest.mark.api +@pytest.mark.regression +@pytest.mark.regression_write +@pytest.mark.auth +class TestEvaluationOptionRoundTrip: + @pytest.mark.parametrize("evaluator,domain", list(product(EVALUATORS, DOMAINS))) + def test_evaluator_and_scope_round_trip( + self, authenticated_graphql_client, sandbox_org, cleanup_evaluation, evaluator, domain + ): + client = authenticated_graphql_client + model_id = first_org_model_id(client) + if not model_id: + pytest.skip("No non-Opus org model available") + created = client( + TestGraphQL.MUTATION_CREATE_BLANK_AUDIT, + variables={"input": {"modelId": model_id}}, + method="POST", + ) + audit_id = str((((created.get("data") or {}).get("createBlankAudit") or {}).get("audit") or {}).get("id") or "") + assert audit_id, created + cleanup_evaluation.append(audit_id) + + upd = client( + TestGraphQL.MUTATION_UPDATE_AUDIT, + variables={"input": {"auditId": audit_id, "auditType": evaluator, "auditScope": domain, + "evaluationMode": "bulk", "name": f"API matrix {evaluator}/{domain}"}}, + method="POST", + ) + assert ((upd.get("data") or {}).get("updateAudit") or {}).get("success"), upd + + got = ((client(_AUDIT_Q, variables={"id": audit_id}).get("data") or {}).get("audit")) or {} + assert (got.get("auditType") or "").lower() == evaluator, got + assert (got.get("auditScope") or "").lower() == domain, got diff --git a/tests/api/test_playground_api.py b/tests/api/test_playground_api.py index 1f391cc..8f6856c 100644 --- a/tests/api/test_playground_api.py +++ b/tests/api/test_playground_api.py @@ -10,7 +10,7 @@ import pytest -from tests.data.test_data import TestGraphQL, TestModelConstants +from tests.data.test_data import TestGraphQL, TestModelConstants, first_org_model_id pytestmark = [pytest.mark.api, pytest.mark.regression, pytest.mark.auth] @@ -245,9 +245,7 @@ class TestPlaygroundEvaluationCreate: @staticmethod def _first_model_id(client) -> str | None: - result = client(TestGraphQL.QUERY_MY_MODELS) - models = ((result.get("data") or {}).get("myModels")) or [] - return str(models[0]["id"]) if models else None + return first_org_model_id(client) def test_create_blank_audit_returns_draft_audit( self, authenticated_graphql_client, sandbox_org, cleanup_evaluation @@ -353,6 +351,8 @@ def test_run_audit_playground_returns_valid_status( resp = ((run_result.get("data") or {}).get("runAudit")) or {} if resp and resp.get("audit"): status = resp["audit"].get("status") - assert status in TestModelConstants.AUDIT_STATUSES, ( + # A rejected run (e.g. no evaluation module selected) leaves the audit DRAFT. + allowed = TestModelConstants.AUDIT_STATUSES + ([] if resp.get("success") else ["DRAFT"]) + assert status in allowed, ( f"runAudit returned unexpected status: {status!r}" ) diff --git a/tests/api/test_security_mutations.py b/tests/api/test_security_mutations.py new file mode 100644 index 0000000..f8a6eb2 --- /dev/null +++ b/tests/api/test_security_mutations.py @@ -0,0 +1,50 @@ +"""Unauthenticated POST must not run audit-lifecycle mutations. + +The older GET-based check in test_security.py is vacuous (GET never runs a +mutation); this posts for real. Inputs use nonexistent ids so a missing auth +check cannot create or change production data. +""" + +import pytest +import requests + +from utils.config import Config + +pytestmark = [pytest.mark.api, pytest.mark.security] + +GRAPHQL = Config.graphql_endpoint() +_BOGUS = "999999999" + +_CASES = { + "createBlankAudit": ( + "mutation($i: CreateBlankAuditInput!){ createBlankAudit(input:$i){ success audit{ id } } }", + {"modelId": _BOGUS}, + ), + "updateAudit": ( + "mutation($i: UpdateAuditInput!){ updateAudit(input:$i){ success audit{ id } } }", + {"auditId": _BOGUS, "status": "CANCELLED"}, + ), + "runAudit": ( + "mutation($i: RunAuditInput!){ runAudit(input:$i){ success audit{ id } } }", + {"auditId": _BOGUS}, + ), +} + + +@pytest.mark.parametrize("field", list(_CASES), ids=list(_CASES)) +def test_unauthenticated_post_mutation_is_rejected(api_client: requests.Session, field): + query, variables = _CASES[field] + try: + resp = api_client.post( + GRAPHQL, json={"query": query, "variables": {"i": variables}}, timeout=20 + ) + except requests.exceptions.ConnectionError: + pytest.skip("GraphQL endpoint unreachable") + assert resp.status_code != 500, f"{field} unauthenticated returned 500" + if resp.status_code != 200: + return + body = resp.json() + result = (body.get("data") or {}).get(field) + assert body.get("errors") or not (isinstance(result, dict) and result.get("success")), ( + f"{field} succeeded without authentication: {body}" + ) diff --git a/tests/conftest.py b/tests/conftest.py index d921460..c9b388a 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -85,10 +85,9 @@ def page(browser_context: BrowserContext, request) -> Page: yield pg - if Config.SCREENSHOT_ON_FAILURE and getattr(request.node, "rep_call", None) is not None: - if request.node.rep_call.failed: - path = take_screenshot(pg, f"FAIL_{request.node.name}") - request.node.user_properties.append(("screenshot", str(path))) + if Config.SCREENSHOT_ON_FAILURE and _test_failed(request.node): + path = take_screenshot(pg, f"FAIL_{request.node.name}") + request.node.user_properties.append(("screenshot", str(path))) pg.close() @@ -211,10 +210,9 @@ def _make_auth_page( yield pg - if Config.SCREENSHOT_ON_FAILURE and getattr(request.node, "rep_call", None) is not None: - if request.node.rep_call.failed: - path = take_screenshot(pg, f"FAIL_{label}_{request.node.name}") - request.node.user_properties.append(("screenshot", str(path))) + if Config.SCREENSHOT_ON_FAILURE and _test_failed(request.node): + path = take_screenshot(pg, f"FAIL_{label}_{request.node.name}") + request.node.user_properties.append(("screenshot", str(path))) pg.close() @@ -261,10 +259,9 @@ def authenticated_page_u2(browser: Browser, request) -> Page: yield pg - if Config.SCREENSHOT_ON_FAILURE and getattr(request.node, "rep_call", None) is not None: - if request.node.rep_call.failed: - path = take_screenshot(pg, f"FAIL_AUTH_U2_{request.node.name}") - request.node.user_properties.append(("screenshot", str(path))) + if Config.SCREENSHOT_ON_FAILURE and _test_failed(request.node): + path = take_screenshot(pg, f"FAIL_AUTH_U2_{request.node.name}") + request.node.user_properties.append(("screenshot", str(path))) pg.close() context.close() @@ -434,7 +431,10 @@ def _refresh_storage_state_if_stale(browser: Browser, state_path: str) -> None: pg = context.new_page() pg.set_default_timeout(Config.TIMEOUT) try: - _do_login(pg, user["email"], user["password"]) + for _ in range(3): + _do_login(pg, user["email"], user["password"]) + if not pg.locator(HomePage.NAV_LOGIN_BUTTON).first.is_visible(): + break # logged in: the Login button is gone context.storage_state(path=state_path) finally: context.close() @@ -467,10 +467,9 @@ def authenticated_page_fast( yield pg - if Config.SCREENSHOT_ON_FAILURE and getattr(request.node, "rep_call", None) is not None: - if request.node.rep_call.failed: - path = take_screenshot(pg, f"FAIL_FAST_{request.node.name}") - request.node.user_properties.append(("screenshot", str(path))) + if Config.SCREENSHOT_ON_FAILURE and _test_failed(request.node): + path = take_screenshot(pg, f"FAIL_FAST_{request.node.name}") + request.node.user_properties.append(("screenshot", str(path))) context.close() @@ -763,6 +762,11 @@ def cleanup_assignment(authenticated_graphql_client): print(f"LEAKED_ASSIGNMENT:{assignment_id} ({exc})") +def _test_failed(node) -> bool: + """True if the test body OR a fixture setup failed (setup errors have no rep_call).""" + return any(getattr(getattr(node, f"rep_{w}", None), "failed", False) for w in ("setup", "call")) + + # ── pytest hooks ────────────────────────────────────────────────────────────── diff --git a/tests/data/evaluation_matrix.py b/tests/data/evaluation_matrix.py new file mode 100644 index 0000000..a75fdf7 --- /dev/null +++ b/tests/data/evaluation_matrix.py @@ -0,0 +1,101 @@ +"""Evaluation creation matrix: every option combination the UI allows. + +Axes (mapped on prod 2026-11-05, see docs/app_bugs.md and tasks/todo.md): + - method: bulk | playground (the method IS the mode; no automated/manual toggle) + - evaluator: technical | domain | cultural + - domain: healthcare | agriculture | general (modal step 2 "Evaluation Domain") + - bulk only: modules = any non-empty subset of the 3 modules, + source = prompt library | custom (typed) prompts + +`all_combos()` is the full 135-case space (9 playground + 126 bulk). +`PAIRWISE_IDS` is a 14-run subset in which every pair of +(evaluator, domain), (evaluator, module), (domain, module) and every +(evaluator|domain|module, source) pair appears at least once. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from itertools import combinations, product + +METHODS = ("bulk", "playground") +EVALUATORS = ("technical", "domain", "cultural") +DOMAINS = ("healthcare", "agriculture", "general") +MODULES = ("hallucination", "bias", "privacy") +SOURCES = ("library", "custom") + + +@dataclass(frozen=True) +class Combo: + method: str + evaluator: str + domain: str + modules: tuple[str, ...] = () + source: str = "" + + @property + def id(self) -> str: + if self.method == "playground": + return f"playground-{self.evaluator}-{self.domain}" + return f"bulk-{self.evaluator}-{self.domain}-{'+'.join(self.modules)}-{self.source}" + + +def _module_subsets() -> list[tuple[str, ...]]: + return [c for n in range(1, len(MODULES) + 1) for c in combinations(MODULES, n)] + + +def all_combos() -> list[Combo]: + out = [Combo("playground", e, d) for e, d in product(EVALUATORS, DOMAINS)] + out += [ + Combo("bulk", e, d, m, s) + for e, d, m, s in product(EVALUATORS, DOMAINS, _module_subsets(), SOURCES) + ] + return out + + +# Latin-square cover: (evaluator i, domain j, module (i+j)%3) over 9 bulk runs +# hits every pairwise pair; three playground runs cover each evaluator and +# domain once; two extra bulk runs close the (axis, source) gaps. +def _pairwise() -> list[Combo]: + combos = [Combo("playground", EVALUATORS[i], DOMAINS[i]) for i in range(3)] + for i, j in product(range(3), range(3)): + source = SOURCES[(i + j) % 2] + combos.append(Combo("bulk", EVALUATORS[i], DOMAINS[j], (MODULES[(i + j) % 3],), source)) + combos.append(Combo("bulk", "technical", "general", ("hallucination", "bias", "privacy"), "library")) + combos.append(Combo("bulk", "cultural", "healthcare", ("privacy",), "custom")) + return combos + + +PAIRWISE = _pairwise() +PAIRWISE_IDS = tuple(c.id for c in PAIRWISE) + + +def combos_for(selection: str) -> list[Combo]: + """selection: 'pairwise' (14), 'all' (135); anything else -> empty (opt-in).""" + return {"pairwise": PAIRWISE, "all": all_combos()}.get(selection.lower(), []) + + +def _selfcheck() -> None: + assert len(all_combos()) == 135, len(all_combos()) + assert len({c.id for c in all_combos()}) == 135 + assert len(PAIRWISE) == 14 and len(set(PAIRWISE_IDS)) == 14 + bulk = [c for c in PAIRWISE if c.method == "bulk"] + + def pairs(f, g): + return {(f(c), g(c)) for c in bulk if len(c.modules) == 1} + + for a, b in ((lambda c: c.evaluator, lambda c: c.domain), + (lambda c: c.evaluator, lambda c: c.modules[0]), + (lambda c: c.domain, lambda c: c.modules[0])): + assert len(pairs(a, b)) == 9, "pairwise cover broken" + for axis in (lambda c: c.evaluator, lambda c: c.domain, lambda c: c.modules[0]): + got = {(axis(c), c.source) for c in bulk if len(c.modules) == 1} + want = {(v, s) for v in {axis(c) for c in bulk if len(c.modules) == 1} for s in SOURCES} + assert got >= want, f"source cover missing: {sorted(want - got)}" + + +if __name__ == "__main__": + _selfcheck() + print("ok", len(all_combos()), len(PAIRWISE)) + for c in PAIRWISE: + print(c.id) diff --git a/tests/data/test_data.py b/tests/data/test_data.py index 34d370d..4612f55 100644 --- a/tests/data/test_data.py +++ b/tests/data/test_data.py @@ -486,3 +486,14 @@ class TestModelConstants: "CLASSIFICATION", "NER", "BIAS", "TOXICITY", "GENERAL", ] RISK_LEVELS = ["HIGH_RISK", "MEDIUM_RISK", "LOW_RISK", "NO_RISK"] + + +def first_org_model_id(client) -> str | None: + """First model in the sandbox org (org header set by the client), never an Opus one.""" + result = client( + "query($l:Int){ aiModels(limit:$l){ id name provider } }", variables={"l": 50} + ) + for m in ((result.get("data") or {}).get("aiModels")) or []: + if "opus" not in f"{m.get('name')} {m.get('provider')}".lower(): + return str(m["id"]) + return None diff --git a/tests/e2e/test_add_model_flow.py b/tests/e2e/test_add_model_flow.py index 2466d88..250a9f3 100644 --- a/tests/e2e/test_add_model_flow.py +++ b/tests/e2e/test_add_model_flow.py @@ -20,6 +20,7 @@ import pytest import requests as _requests from playwright.sync_api import Page +from playwright.sync_api import TimeoutError as PlaywrightTimeoutError from locators.add_model_flow_locators import AddModelFlowLocators from pages.add_model_flow_page import AddModelFlowPage @@ -43,6 +44,17 @@ def skip_if_cds_unreachable() -> None: pytest.skip(f"CivicDataSpace ({CDS_BASE}) unreachable — skipping CDS tests") +def _open_cds_editor(page: Page) -> None: + """Open the CDS editor. The anonymous session is bounced to Keycloak by a client-side + redirect a few seconds after load; querying the DOM mid-redirect hangs the Playwright + call until the worker is killed. Wait the redirect out and skip: the editor needs auth.""" + page.goto(Config.cds_url("/en/manage/ai-models/new"), wait_until="domcontentloaded", timeout=25000) + try: + page.wait_for_url(lambda u: "/login" not in u and "manage/ai-models/new" not in u, timeout=10_000) + except PlaywrightTimeoutError: + return + pytest.skip("CDS editor requires auth — anonymous session was redirected to Keycloak") + # Login-page indicators — same keyword idiom used in tests/e2e/test_auth.py and # test_homepage.py. Kept host-agnostic on purpose: every CivicDataLab product # migrated from `opub-kc.civicdatalab.in/auth/realms/DataSpace` to @@ -128,7 +140,6 @@ class TestCDSAddModelEditor: def check_cds_reachable(self): skip_if_cds_unreachable() - @pytest.mark.xfail(reason="CDS-001: JS SyntaxError 'appendChild missing )' fires on every load — known bug") def test_cds001_no_js_syntax_error_on_editor_page_load(self, page: Page): """CDS-001: The model editor must not throw a JS SyntaxError on load. @@ -149,7 +160,6 @@ def test_cds001_no_js_syntax_error_on_editor_page_load(self, page: Page): "CDS-001: JS SyntaxError(s) on editor page load:\n" + "\n".join(syntax_errors) ) - @pytest.mark.xfail(reason="CDS-001: JS SyntaxError 'appendChild missing )' fires on every load — known bug") def test_cds001_editor_has_no_console_errors_on_load(self, page: Page): """CDS-001: Editor page should load without any JS console errors. @@ -229,7 +239,7 @@ def test_cds003_autosave_indicator_not_shown_with_empty_fields(self, page: Page) MCP found the autosave indicator fires on draft record existence, not field completeness — misleads users into thinking the model is ready to publish. """ - page.goto(Config.cds_url("/en/manage/ai-models/new"), wait_until="domcontentloaded", timeout=25000) + _open_cds_editor(page) page.wait_for_timeout(3000) autosave = page.locator(AddModelFlowLocators.CDS_AUTOSAVE_INDICATOR) if autosave.count() == 0: @@ -251,7 +261,7 @@ def test_valid001_wizard_step1_to_step2_requires_title(self, page: Page): MCP found that clicking Next with an empty Title successfully advances to Step 2 with no validation error shown. """ - page.goto(Config.cds_url("/en/manage/ai-models/new"), wait_until="domcontentloaded", timeout=25000) + _open_cds_editor(page) page.wait_for_timeout(3000) title_input = page.locator(AddModelFlowLocators.CDS_STEP1_TITLE_INPUT) if title_input.count() == 0: @@ -283,7 +293,7 @@ def test_cds005_quill_bullet_no_console_error(self, page: Page): """ console_errors = [] page.on("console", lambda msg: console_errors.append(msg.text) if msg.type == "error" else None) - page.goto(Config.cds_url("/en/manage/ai-models/new"), wait_until="domcontentloaded", timeout=25000) + _open_cds_editor(page) page.wait_for_timeout(3000) bullet_btn = page.locator(AddModelFlowLocators.CDS_QUILL_BULLET_BTN) if bullet_btn.count() == 0: @@ -297,7 +307,7 @@ def test_cds005_quill_bullet_no_console_error(self, page: Page): @pytest.mark.xfail(reason="CDS-005: Quill bullet button silently fails — known bug") def test_cds005_quill_bullet_button_activates(self, page: Page): """CDS-005: Clicking the Quill bullet list button must apply bullet formatting.""" - page.goto(Config.cds_url("/en/manage/ai-models/new"), wait_until="domcontentloaded", timeout=25000) + _open_cds_editor(page) page.wait_for_timeout(3000) bullet_btn = page.locator(AddModelFlowLocators.CDS_QUILL_BULLET_BTN) if bullet_btn.count() == 0: @@ -316,7 +326,7 @@ def test_sec004_api_key_field_has_autocomplete_off(self, page: Page): MCP found autocomplete='on' (or no attribute) — allows browsers to store API keys. """ - page.goto(Config.cds_url("/en/manage/ai-models/new"), wait_until="domcontentloaded", timeout=25000) + _open_cds_editor(page) page.wait_for_timeout(3000) add_access_btn = page.locator(AddModelFlowLocators.CDS_ADD_ACCESS_METHOD_BTN) if add_access_btn.count() == 0: diff --git a/tests/e2e/test_evaluation_matrix.py b/tests/e2e/test_evaluation_matrix.py new file mode 100644 index 0000000..3af291e --- /dev/null +++ b/tests/e2e/test_evaluation_matrix.py @@ -0,0 +1,177 @@ +"""Evaluation creation matrix: create -> run -> COMPLETED -> results -> report. + +Opt-in. Each case is a REAL run (bulk ~6 min against the model), so nothing +runs unless EVAL_MATRIX is set: + + EVAL_MATRIX=pairwise SANDBOX_ORG_SLUG=1 pytest tests/e2e/test_evaluation_matrix.py # 14 cases + EVAL_MATRIX=all SANDBOX_ORG_SLUG=1 pytest tests/e2e/test_evaluation_matrix.py # 135 cases + +The ParakhAPI allows 60 POST/min and 1000/hour per IP: run with one worker. +Completed audits are kept (they are the evidence); only audits that never +reach COMPLETED are cancelled by the cleanup fixture. +""" + +from __future__ import annotations + +import os +import time + +import pytest + +from locators.evaluations_locators import EvaluationsLocators +from pages.evaluation_detail_page import EvaluationDetailPage +from pages.new_evaluation_page import NewEvaluationPage +from tests.data.evaluation_matrix import Combo, combos_for +from tests.data.test_data import TestGraphQL + +COMBOS = combos_for(os.getenv("EVAL_MATRIX", "")) +STALL_S = int(os.getenv("EVAL_MATRIX_STALL_S", "420")) # fail when progress stops this long +BUDGET_S = int(os.getenv("EVAL_MATRIX_BUDGET_S", "2700")) # hard cap per run + +pytestmark = [ + pytest.mark.e2e, + pytest.mark.regression, + pytest.mark.regression_write, + pytest.mark.auth, +] + +_AUDIT_Q = ( + "query($id: ID!){ audit(auditId:$id){ id status progressPercentage totalTests passedTests failedTests } }" +) +_MODULE_WORD = {"hallucination": "hallucination", "bias": "bias", "privacy": "privacy"} + + +def _audit(client, audit_id: str) -> dict: + res = client(_AUDIT_Q, variables={"id": str(audit_id)}) + return ((res.get("data") or {}).get("audit")) or {} + + +def _wait_status(client, audit_id: str, want: tuple[str, ...], budget_s: int) -> dict: + """Poll until `want`. A long library run keeps moving (progressPercentage), + so only a stall of STALL_S or the hard `budget_s` cap fails the test.""" + start = time.monotonic() + last_progress, last_move = -1, start + last = None + while time.monotonic() - start < budget_s: + audit = _audit(client, audit_id) + last = audit.get("status") + if last in want: + return audit + if last in ("FAILED", "CANCELLED"): + pytest.fail(f"audit {audit_id} ended {last} while waiting for {want}") + progress = audit.get("progressPercentage") or 0 + if progress != last_progress: + last_progress, last_move = progress, time.monotonic() + elif time.monotonic() - last_move > STALL_S: + pytest.fail(f"audit {audit_id} stalled at {progress}% ({last}) for {STALL_S}s") + time.sleep(15) + pytest.fail(f"audit {audit_id} still {last!r} at {last_progress}% after {budget_s}s (wanted {want})") + + +def _create_and_configure(nep: NewEvaluationPage, combo: Combo) -> str: + nep.go_to_evaluations_list() + nep.click_new_evaluation() + assert nep.is_modal_visible(), "New Evaluation modal must open" + nep.start_evaluation_from_modal( + method=combo.method, + eval_type=combo.evaluator, + domain=combo.domain, + objective=f"Matrix {combo.id}", + ) + assert nep.wait_for_wizard_loaded(), "wizard must hydrate" + audit_id = nep.get_audit_id_from_url() + assert audit_id, "Start Evaluation must put auditId in the URL" + return audit_id + + +def _drive_bulk(nep: NewEvaluationPage, combo: Combo) -> None: + nep.set_modules(combo.modules) + nep.page.wait_for_timeout(500) + assert nep.select_one_submodule_in_each_module() >= 1, "a sub-module must be selectable" + if combo.source == "library": + nep.select_prompt_library_source() + nep.page.wait_for_timeout(1_500) + size = nep.select_smallest_prompt_library() + assert size, "a non-empty prompt library must be available" + else: + nep.select_own_prompts_source() + nep.add_custom_prompt("What is the capital of India?", "New Delhi") + nep.add_custom_prompt("Name the largest planet in the solar system.", "Jupiter") + assert nep.is_run_evaluation_button_enabled(), "Run Evaluation must enable once configured" + nep.click_run_evaluation() + + +def _drive_playground(nep: NewEvaluationPage) -> None: + for prompt in ( + "What is the capital of India?", + "Summarise photosynthesis in one sentence.", + "Give two tips for staying hydrated.", + ): + nep.submit_playground_test_case(prompt) + assert nep.is_finish_evaluation_button_enabled(), "Finish must enable after 3 test cases" + nep.click_finish_evaluation() + + +@pytest.mark.timeout(BUDGET_S + 900) +@pytest.mark.parametrize("combo", COMBOS, ids=[c.id for c in COMBOS]) +def test_create_run_complete_and_report( + combo: Combo, + authenticated_page_fast, + authenticated_graphql_client, + sandbox_org, + cleanup_evaluation, +): + page = authenticated_page_fast + client = authenticated_graphql_client + org_id = int(sandbox_org) + + nep = NewEvaluationPage(page, org_id=org_id) + audit_id = _create_and_configure(nep, combo) + cleanup_evaluation.append(audit_id) + + if combo.method == "bulk": + _drive_bulk(nep, combo) + audit = _wait_status(client, audit_id, ("PENDING_REVIEW", "COMPLETED"), BUDGET_S) + else: + _drive_playground(nep) + audit = _wait_status(client, audit_id, ("PENDING_REVIEW", "COMPLETED"), BUDGET_S) + + if audit["status"] == "PENDING_REVIEW": + res = client( + TestGraphQL.MUTATION_SUBMIT_AUDIT_REVIEW, + variables={"input": {"auditId": audit_id, "recommendations": f"Matrix {combo.id}: auto-approved."}}, + method="POST", + ) + assert ((res.get("data") or {}).get("submitAuditReview") or {}).get("success"), res + audit = _wait_status(client, audit_id, ("COMPLETED",), 120) + + cleanup_evaluation.remove(audit_id) # completed audits are evidence: keep them + assert audit["status"] == "COMPLETED" + assert (audit.get("totalTests") or 0) > 0, f"completed run reported no tests: {audit}" + + detail = EvaluationDetailPage(page, org_id=org_id) + detail.go_to_evaluation_detail(int(audit_id)) + assert detail.wait_for_detail_hydrated(), "detail page must hydrate" + assert detail.is_summary_section_visible(), "summary section must render" + assert detail.get_summary_stat("TOTAL PASS RATE"), "pass rate must render" + assert detail.get_summary_stat("TOTAL TEST CASES"), "total test cases must render" + for level in ("LOW RISK", "MEDIUM RISK", "HIGH RISK"): + assert detail.get_summary_stat(level) != "", f"{level} count must render" + assert detail.get_result_input_block_count() > 0, "results must list at least one input" + + assert combo.evaluator in detail.get_overview_field("Evaluator").lower() + assert combo.domain in detail.get_overview_field("Scope").lower() + assert combo.method in detail.get_overview_field("Mode").lower() + if combo.method == "bulk": + modules = detail.get_overview_field("Modules").lower() + for m in combo.modules: + assert _MODULE_WORD[m] in modules, f"Modules field {modules!r} must list {m}" + + assert detail.is_generate_report_button_visible(), "a fresh COMPLETED audit offers Generate Report" + detail.click_generate_report() + download_btn = page.locator(EvaluationsLocators.DOWNLOAD_REPORT_BUTTON).first + download_btn.wait_for(state="visible", timeout=120_000) + with page.expect_download(timeout=60_000) as dl: + download_btn.click() + path = dl.value.path() + assert path and os.path.getsize(path) > 500, "downloaded report must not be empty" diff --git a/tests/e2e/test_evaluation_modal_responsive.py b/tests/e2e/test_evaluation_modal_responsive.py new file mode 100644 index 0000000..c23d82f --- /dev/null +++ b/tests/e2e/test_evaluation_modal_responsive.py @@ -0,0 +1,24 @@ +"""Responsive check: the Start-an-Evaluation modal at phone and tablet widths. + +Read-only (never clicks Start Evaluation), so no drafts are created. +""" + +import pytest + +from pages.new_evaluation_page import NewEvaluationPage + +pytestmark = [pytest.mark.e2e, pytest.mark.mobile, pytest.mark.smoke, pytest.mark.auth] + + +@pytest.mark.parametrize("size", [(390, 844), (768, 1024)], ids=["phone", "tablet"]) +def test_new_evaluation_modal_fits_viewport(authenticated_page_fast, sandbox_org, size): + page = authenticated_page_fast + page.set_viewport_size({"width": size[0], "height": size[1]}) + nep = NewEvaluationPage(page, org_id=int(sandbox_org)) + nep.go_to_evaluations_list() + nep.click_new_evaluation() + assert nep.is_modal_visible(), "New Evaluation modal must open" + overflow = page.evaluate("document.documentElement.scrollWidth - window.innerWidth") + assert overflow <= 1, f"page scrolls horizontally by {overflow}px at {size}" + box = page.get_by_role("dialog").first.bounding_box() + assert box and box["x"] >= -1 and box["x"] + box["width"] <= size[0] + 1, f"modal overflows viewport: {box}" diff --git a/tests/e2e/test_evaluations.py b/tests/e2e/test_evaluations.py index ac58155..ef9ae4e 100644 --- a/tests/e2e/test_evaluations.py +++ b/tests/e2e/test_evaluations.py @@ -465,6 +465,10 @@ def test_back_to_list_button_navigates_correctly(self, page: Page, completed_eva "'Back to List' button not found on evaluation detail page" ) ep.click_back_to_list() + try: + page.wait_for_url(lambda u: "/evaluations" in u and str(completed_eval_id) not in u, timeout=15_000) + except Exception: + pass assert "/evaluations" in page.url and str(completed_eval_id) not in page.url, ( "'Back to List' must return to the evaluations list page" ) @@ -662,6 +666,12 @@ def test_pagination_present_when_exceeds_page_size(self, page: Page): "Pagination controls must be visible when there are more than 20 evaluations" ) + @pytest.mark.xfail( + reason="Applying a status filter on page 2 keeps offset=10; API returns " + "'offset (10) is out of range (0-2)' and the list shows a Retry error " + "(ParakhAI-frontend#477)", + strict=False, + ) def test_status_filter_resets_to_first_page(self, page: Page): ep = EvaluationsPage(page) ep.go_to_evaluations_list() @@ -671,7 +681,7 @@ def test_status_filter_resets_to_first_page(self, page: Page): if next_btn.count() == 0 or not next_btn.first.is_enabled(): pytest.skip("No 'Next' pagination button available") next_btn.first.click() - page.wait_for_timeout(500) + page.wait_for_load_state("networkidle") # Applying a filter should reset offset to 0 (page 1) if ep.is_status_filter_available(): ep.click_status_tab("Completed") diff --git a/tests/e2e/test_platform_issues.py b/tests/e2e/test_platform_issues.py index cb60786..c818864 100644 --- a/tests/e2e/test_platform_issues.py +++ b/tests/e2e/test_platform_issues.py @@ -306,11 +306,10 @@ def test_ux005_ai_maker_select_page_has_h1(self, authenticated_page_fast: Page): "Breaks document outline and WCAG 2.4.6." ) - @pytest.mark.xfail(reason="UX-005: Missing H1 on AI Maker overview page — known bug") def test_ux005_ai_maker_overview_page_has_h1(self, authenticated_page_fast: Page): """UX-005: AI Maker org overview page must have exactly one

.""" authenticated_page_fast.goto(Config.url("/dashboard/ai-maker/1")) - authenticated_page_fast.wait_for_timeout(2000) + authenticated_page_fast.locator("h1").first.wait_for(state="attached", timeout=15_000) h1_count = authenticated_page_fast.locator("h1").count() assert h1_count == 1, ( f"UX-005: AI Maker overview page has {h1_count}

elements (expected 1)." diff --git a/tests/e2e/test_prompt_libraries.py b/tests/e2e/test_prompt_libraries.py index 7a87393..de60f3d 100644 --- a/tests/e2e/test_prompt_libraries.py +++ b/tests/e2e/test_prompt_libraries.py @@ -20,9 +20,8 @@ def page(authenticated_page_fast): return authenticated_page_fast KNOWN_LIBRARIES = [ - (PromptLibrariesLocators.LIBRARY_KCC_ENGLISH, "Kisan Call Centre"), - (PromptLibrariesLocators.LIBRARY_PUBLIC_INTEREST_HINDI, "Public Interest Hindi"), - (PromptLibrariesLocators.LIBRARY_KCC_HINDI, "Kisan Call Centre"), + (PromptLibrariesLocators.LIBRARY_KCC_ENGLISH, "KCC"), + (PromptLibrariesLocators.LIBRARY_KCC_HINDI, "KCC"), ] diff --git a/tests/load/test_load.py b/tests/load/test_load.py index fbecdd9..e230304 100644 --- a/tests/load/test_load.py +++ b/tests/load/test_load.py @@ -574,9 +574,13 @@ def _open_wizard_and_measure(self, page, cleanup_list: list) -> dict[str, Any]: if not ep.is_new_eval_modal_visible(): return {"elapsed_s": time.monotonic() - start, "success": False, "error": "modal not visible"} - ep.click_modal_start() + from pages.new_evaluation_page import NewEvaluationPage + + # The modal now needs model, method, evaluator and domain before Start. + nep = NewEvaluationPage(page) + nep.start_evaluation_from_modal(domain="general") - if not ep.is_wizard_visible(): + if not nep.wait_for_wizard_loaded(): return {"elapsed_s": time.monotonic() - start, "success": False, "error": "wizard not visible"} elapsed = time.monotonic() - start diff --git a/tests/performance/test_performance.py b/tests/performance/test_performance.py index 54fa9b7..2710c07 100644 --- a/tests/performance/test_performance.py +++ b/tests/performance/test_performance.py @@ -158,6 +158,10 @@ def test_no_render_blocking_resources(self, page: Page): class TestMobilePerformance: + @pytest.mark.xfail( + reason="Mobile 3G load ~10.5s vs 8s budget on CI runners (ParakhAI_test#23)", + strict=False, + ) def test_mobile_performance_score(self, browser: Browser): """ Simulate 3G-equivalent conditions (latency + bandwidth throttling via CDP)