diff --git a/docs/a11y_findings.md b/docs/a11y_findings.md index af94fc3..6a6a535 100644 --- a/docs/a11y_findings.md +++ b/docs/a11y_findings.md @@ -10,6 +10,14 @@ Each row links to the WCAG criterion and the axe rule id (helpful when filing ti | a2 | color-contrast | 1.4.3 (AA) | serious | Keycloak login (`opub-kc.civicdatalab.in/auth/realms/DataSpace/...`) | form labels / placeholder text | Foreground/background contrast below 4.5:1 for body text. Third-party (Keycloak) — see bug #5. | open (third-party) | 2026-05-08 | | a3 | link-name | 4.1.2 (A) / 2.4.4 (A) | serious | Keycloak login (footer) | `` containing only `` (4 social media links) | Icon-only links with no `aria-label` or visible text. Affects GitHub, LinkedIn, Twitter, Facebook links. Third-party — see bug #5. | open (third-party) | 2026-05-08 | | a4 | heading-order / page-has-heading-one | 1.3.1 (A) | serious | `/dashboard/ai-maker/1` (auth) | `
` (no h1/h2/h3 anywhere) | No heading elements at all — the Overview widget that would host the page heading never mounts because its data fetch hangs indefinitely. Linked to bug #14 (and bug #13's backend root cause). | open | 2026-07-10 | +| a5 | color-contrast | 1.4.3 (AA) | serious | `/` (anonymous, hydrated) | `button.bg-[#E8E4FF]` | Active feature tab text #6849EE on #E8E4FF = 4.49:1. Linked to bug #31. | open | 2026-10-31 | +| a6 | link-name | 4.1.2 (A) / 2.4.4 (A) | serious | `/` (anonymous, hydrated) | `footer a.w-10.h-10` x4 | Footer social icons unnamed; only visible after hydration. Same as bug #9. | open | 2026-10-31 | +| a7 | (manual) name-role-value | 4.1.2 (A) / 1.4.1 (A) | serious | `/` | feature tab `
` after the menu closes. Bug #40. | open | 2026-10-31 | ## How to interpret @@ -23,3 +31,4 @@ Each row links to the WCAG criterion and the axe rule id (helpful when filing ti - **Phase 2 run 2026-05-08:** 11 a11y tests → 3 fail (1 flake fixed in test code, 2 real bugs filed → bugs #4, #5). 4 pass / 5 skip (env / no-creds) / 2 xfail / 0 fail after fixes. - **Phase 7 run 2026-07-10:** 34 a11y tests (public + authenticated) → 1 fail, reproduced consistently (not flaky). Root-caused to dashboard overview widget hang; filed bug #14 / finding a4 and xfailed. 14 pass / 9 skip / 10 xfail / 0 fail after triage. +- **Phase 14 run 2026-10-31:** anonymous review of the hydrated homepage, sign-in page and mobile menu. a5-a12 added (bugs #31-#34, #38-#40). The pre-existing homepage axe test scans the pre-hydration skeleton, so a5/a6 were invisible to it. diff --git a/docs/app_bugs.md b/docs/app_bugs.md index d5db3de..3031345 100644 --- a/docs/app_bugs.md +++ b/docs/app_bugs.md @@ -22,7 +22,7 @@ Format: append-only. When a bug is fixed in the app, mark `status: fixed` and th | 13 | evaluations list / backend perf | The frontend `GetAudits` query that powers the AI-Maker Evaluations list (`/dashboard/ai-maker/{org}/evaluations`) **hangs and never returns** on dev — the page sits on "Loading evaluations…" indefinitely (still up at 90 s). Root-caused 2026-06-12 via live Playwright + direct GraphQL probing on org 1 (981 total audits): requesting the **light** field set (`id name status modelId`, `limit:100`) returns in ~10.6 s, and `metrics`-only or `modules`-only at `limit:100` each return in ~16–18 s, but the **full field set the frontend actually requests** (`name modelName status modules metrics evaluationMode auditType totalTests passedTests failedTests createdAt startedAt completedAt`) **times out past 30 s even at `limit:5`** — so it is the combined per-row resolver cost (likely N+1 on `metrics`/`modules`/test-count fields), not row volume. Backend should batch/precompute these fields or paginate server-side. Also observed intermittent `ERR_NETWORK_CHANGED` / "Backend server is not available" GraphQL errors on dev during the same session (degraded backend). **Not a test defect** — no client-side timeout bump can help since the query never completes. | fixed 2026-08-12 (see verification note) | Log in → AI Maker → CivicDataLab → Evaluations. Page stays on "Loading evaluations…". In DevTools, the `GetAudits` POST to `dev.api.parakh.civicdataspace.in/graphql/` has no response after 90 s. Confirmed via Playwright MCP 2026-06-12: full-field `audits` query aborts at 30 s even with `limit:5`, while light-field returns in ~10 s. | `tests/e2e/test_evaluations.py::TestStatusFilterTabs::*` (all 6 — tabs/column/rows render only after the list query resolves); transitively any test that asserts on rendered evaluations-list content for org 1; `tests/e2e/test_evaluations.py::TestEvaluationsListPage::test_table_column_headers_are_present` (added 2026-07-31 — re-run isolated, no concurrent CI load: page still stuck on "Loading evaluations..." past 40s, confirming the hang reproduces without concurrent-shard traffic) | 2026-06-12 | | 14 | a11y / dashboard heading | The AI Maker Dashboard (`/dashboard/ai-maker/1`) never renders any `

`/`

`/`

` heading — `document.querySelectorAll('h1,h2,h3').length` is `0` even after a 30s wait. Root cause: the "Overview" widget (which appears to be where the page heading lives) is permanently stuck on "Loading overview..." and never resolves — the same failure shape as bug #13 (backend `GetAudits`/overview resolver hang), just surfacing on the dashboard-home overview query instead of the evaluations-list query. Because the widget never mounts, the page has zero heading elements, violating WCAG 1.3.1 (Info and Relationships) / a broken document outline for screen-reader users. **Update 2026-08-13:** Phase 12 (2026-08-12) marked this `fixed` after a clean re-verification, but noted the xfails referencing it hadn't been removed pending a re-run. That re-run surfaced different evidence: in a full e2e run (`--splits 12 --group 2`), `test_add_new_model_button_is_visible` and `test_model_cards_are_rendered` genuinely XFAILED (the bug fired), while `test_civicdatalab_name_visible_in_sidebar` and `test_text_generation_badge_is_present` XPASSED in the same run. Re-running the two XFAILing tests in true isolation (`-n 1 --reruns 0`, no other suite active) immediately afterward, both XPASSED cleanly. This is NOT the bug #3 concurrency-noise shape (isolation didn't consistently clear it either way) — it's a genuinely intermittent defect, not a fixed-vs-broken binary. **Do not remove the `strict=False` xfails on any of the four related tests** — they correctly tolerate a defect that still recurs some fraction of the time. | intermittent (re-opened 2026-08-13 — was marked fixed 2026-08-12, see update above; keep the xfails) | Log in as `TEST_EMAIL_1` → navigate directly to `https://dev.parakh.civicdataspace.in/dashboard/ai-maker/1`. Page loads nav/sidebar/breadcrumbs but the main content area shows only "Loading overview..." indefinitely (confirmed stuck past 30s). Run `document.querySelectorAll('h1,h2,h3').length` → `0`. Confirmed via Playwright 2026-07-10. | `tests/accessibility/test_accessibility_auth.py::TestDashboardAccessibility::test_dashboard_has_exactly_one_h1`; `tests/e2e/test_ai_maker_dashboard.py::TestAIMakerDashboardLoads::test_civicdatalab_name_visible_in_sidebar`, `TestModelCardsonHome::test_add_new_model_button_is_visible`, `TestModelCardsonHome::test_model_cards_are_rendered`, `TestModelCardsonHome::test_text_generation_badge_is_present` (added 2026-07-31 — re-run isolated (no concurrent CI load), direct repro script confirmed the whole home content area, not just the heading, is gated behind the same stuck "Loading overview..." widget: org identity block (`CivicdataLab` text), model cards, `Add A New Model` button, and model-type badges never render because they all read from the same overview payload) | 2026-07-10 | | 15 | routing / auth deep-link | Direct navigation to authenticated AI-Maker / Auditor sub-routes (`/dashboard/ai-maker/1/ai-models`, `/dashboard/ai-maker/1/evaluations/new`, `/dashboard/ai-maker/1/auditors`, `/dashboard/ai-maker/1/prompt-libraries`, `/dashboard/auditor`) **intermittently renders the public marketing homepage** (hero "Build AI that's trustworthy from day one.", feature tabs, CTA cards, footer) instead of the requested dashboard page, even though the user is authenticated (avatar/session valid) and the URL bar shows the correct path. Root-caused 2026-07-10 via a scripted repro: a fresh browser context seeded with a valid `storage_state` (cookies + localStorage from a real Keycloak login) navigating straight to `/dashboard/ai-maker/1/ai-models` with `wait_until="load"` + 2.5s settle reproduced the homepage fallback in **1 of 6 trials (~17%)**; the same repro against `/dashboard/ai-maker/1/prompt-libraries` reproduced it in **1 of 4 trials (~25%)**, and a further capture landed in a visibly half-transitioned state (both "Prompt Libraries" page chrome and homepage CTA/footer content stacked in the same viewport — see `screenshots/DIFF_auth_prompt_libraries_desktop_1440x900.png`). When it does NOT reproduce, the correct page renders fully and correctly (light theme, correct heading, working sidebar, "Loading AI models…" spinner resolving normally). This is a materially different and more severe symptom than the "non-deterministic async content" (live stats, model dropdown ordering) that `_NON_DETERMINISTIC_VISUAL` in `tests/visual/test_visual_regression.py` currently attributes these routes' diffs to — the wizard/models/auditors/auditor-dashboard visual tests see ~43% pixel diffs because they are sporadically capturing the wrong page entirely, not just varying live data. Likely a client-side routing/auth-guard race on hard navigation (the route guard falls through to the default/public route before the session check resolves), since it only reproduces on deep-link/full-page loads, not in-app client-side navigation. | open | Using a Playwright context seeded with a valid Keycloak `storage_state`, repeatedly `page.goto()` directly to `/dashboard/ai-maker/1/ai-models` (or any route in the list above) with `wait_until="load"` and a 2.5s+ settle delay; repeat ~5-6 times. Roughly 1 in 5-6 loads renders the homepage hero/CTA/footer instead of the dashboard page content. Confirmed via ad hoc Playwright script 2026-07-10 (not yet encoded as a standing test — see `docs/visual_diffs.md` Run 2026-07-10 for the diff images that first surfaced it). | `tests/visual/test_visual_regression.py::TestAuthenticatedPageVisuals::test_authenticated_page_desktop[chromium-models_list]`, `[chromium-new_evaluation_wizard]`, `[chromium-auditors_management]`, `[chromium-auditor_dashboard]`, `[chromium-prompt_libraries]` (all xfail under VISUAL-002 — the xfail reason text should be revisited given this finding); `tests/e2e/test_user_flows.py::TestFlow04_EvaluationDetailAndReport::test_evaluation_detail_shows_full_results` (added 2026-07-13 — deep-link to `/evaluations/{id}` rendered the public homepage; avatar/session chrome still present, confirming auth was intact); `tests/visual/test_visual_regression.py::TestAuthenticatedPageVisuals::*` (2026-08-12 — the new capture-integrity gate in `utils/visual_guards.py` now detects this automatically and skips instead of baselining it) **— SEVERITY UPDATE 2026-08-12: observed at ~50%, far above the 17-25% recorded above.** Two consecutive full visual runs hit the logged-out fallback on **12 of 24 authenticated captures** (5/12 then 7/12). Affected: `/dashboard/ai-maker/1`, `/ai-models`, `/evaluations`, `/evaluations/new`, `/auditors`, `/prompt-libraries`, `/dashboard/auditor`. NOT affected in either run: `/dashboard` (role selector), `/dashboard/ai-maker` (org selector), `/dashboard/auditor/evaluations` — so it is not uniform across depth. New detail vs the 2026-07-10 characterisation: the fallback now renders **logged-out chrome (LOGIN / SIGN UP)**, i.e. the session is not merely failing to apply to the route — the shell itself renders unauthenticated. Caveat on the comparison: both figures are small samples (the original was 1/6 and 1/4 trials) and load conditions differed, so treat this as "materially worse under current conditions" rather than a precise regression measurement. Independent confirmation that it also fired historically: four baselines were byte-identical copies of the logged-out homepage. `test_evaluation_detail_shows_full_results`'s xfail was missing `strict=False` (this repo's `pytest.ini` sets `xfail_strict = true` globally) — added 2026-08-13 so a lucky run's XPASS is tolerated instead of failing the suite, consistent with bug #14's established convention for intermittent (not always-reproducing) bugs. | 2026-07-10 | -| 16 | SEO / homepage content | Homepage `` contains a spelling error: `"Paricipatory AI Evaluation"` (missing the first `t` in "Participatory"). Root-caused in frontend source: `app/layout.tsx` line 180, `export const metadata = { ..., description: "Paricipatory AI Evaluation" }`. One-line content fix. | open | Load `https://dev.parakh.civicdataspace.in/`. Run `document.querySelector('meta[name="description"]').content` → `"Paricipatory AI Evaluation"`. Confirmed against `ParakhAI-frontend/app/layout.tsx:180` 2026-07-13. | `tests/e2e/test_platform_issues.py::TestHomepageSEO::test_meta_description_has_no_typo` | 2026-07-13 | +| 16 | SEO / homepage content | Homepage `` contains a spelling error: `"Paricipatory AI Evaluation"` (missing the first `t` in "Participatory"). Root-caused in frontend source: `app/layout.tsx` line 180, `export const metadata = { ..., description: "Paricipatory AI Evaluation" }`. One-line content fix. | open on dev; fixed on `main` and prod by ParakhAI-frontend#437 (merged 2026-07-15) — the `dev` branch does not contain commit 9919bdb, so dev still serves the typo and `https://parakh.civicdataspace.in/` serves the corrected text (checked 2026-10-31; against prod this test XPASSes strict, which is the expected signal to retire the xfail once dev is merged). | Load `https://dev.parakh.civicdataspace.in/`. Run `document.querySelector('meta[name="description"]').content` → `"Paricipatory AI Evaluation"`. Confirmed against `ParakhAI-frontend/app/layout.tsx:180` 2026-07-13. | `tests/e2e/test_platform_issues.py::TestHomepageSEO::test_meta_description_has_no_typo` | 2026-07-13 | | 17 | SEO / homepage content | Homepage has no Open Graph `` tag (or any other `og:*` tags) — the root `metadata` export in `app/layout.tsx` only sets `title` and `description`, no `openGraph` block. Links to the homepage shared on social media / chat apps will render with no title card. | open | Load `https://dev.parakh.civicdataspace.in/`. Run `document.querySelector('meta[property="og:title"]')` → `null`. Confirmed against `ParakhAI-frontend/app/layout.tsx` (metadata export has no `openGraph` key) 2026-07-13. | `tests/e2e/test_platform_issues.py::TestHomepageSEO::test_og_title_is_present` | 2026-07-13 | | 18 | routing / 404 | Visiting any non-existent URL as an anonymous user does not show a 404/not-found message — it silently redirects to the generic NextAuth "sign in with Keycloak" page instead, with no indication the URL was invalid. Root cause confirmed in `middleware.ts`: `publicPages = ['/']` is the *only* unauthenticated route; the matcher (`'/((?!_next|api|.*\\..*).*)'`) sends every other path — including typos and genuinely nonexistent routes — through `withAuth`, which redirects unauthenticated requests to `/api/auth/signin` before Next.js ever gets a chance to render a not-found page. This is a distinct, more severe variant of the two existing sibling xfails in this file (`test_404_page_has_home_navigation_link` / `test_404_page_has_logo_or_branding`), which assumed a bare-but-present Next.js 404 page — there isn't one for anonymous users at all. | open | As an anonymous user, navigate to `https://dev.parakh.civicdataspace.in/this-page-does-not-exist-xyz-123`. Observe the generic Keycloak/NextAuth sign-in screen, not a 404 page. Confirmed against `ParakhAI-frontend/middleware.ts` (`publicPages = ['/']`) 2026-07-13. | `tests/e2e/test_platform_issues.py::TestNotFoundPage::test_404_page_shows_not_found_message` | 2026-07-13 | | 19 | ai-maker / org-selection race | Clicking the "AI Maker" role card intermittently shows a **"Redirect to CivicDataSpace — you are being redirected to add an organisation"** modal instead of the organization-selection list, even for accounts with real org memberships. Confirmed via direct GraphQL: `TEST_EMAIL_1`'s `myOrganizations` query returns **11 real organizations** (including `CivicDataLab`, id `1`) — this is not a data/membership issue. Root cause confirmed in `app/[locale]/dashboard/ai-maker/page.tsx:98-102`: `useEffect(() => { if (!isLoading && organizations.length === 0) setShowRedirectPrompt(true); }, [isLoading, organizations.length])` fires whenever `useMyOrganizations()`'s React Query `isLoading` flips to `false` with a still-empty `organizations` array — a data-arrival race, same family as bug #15's auth-guard race (the query is `enabled: isAuthenticated`, so an auth-hydration timing gap can produce exactly this transient empty-then-populated sequence). Worse: once triggered, the modal **never auto-dismisses** — `setShowRedirectPrompt(false)` only exists inside the two dialog button handlers (`page.tsx:160`, `173`), not as a reactive effect keyed off `organizations.length` becoming non-zero, so a real user who hits the race is stuck looking at a false "you have no orgs" prompt with no path back to their actual org list short of a page reload. | open | Log in as `TEST_EMAIL_1` (has 11 real orgs) → navigate to `/dashboard` → click "AI Maker" role card. Intermittently (reproduced during a live E2E run 2026-07-28) the "Redirect to CivicDataSpace" modal appears over an org-selection screen with zero cards, instead of the real 11-org grid. Verify org membership independently via `python -c "from scripts._api_client import get_access_token, graphql; t=get_access_token(); print(graphql(t, org_id='1', query='query{myOrganizations{id name slug}}'))"` — returns 11 orgs, confirming the frontend, not the account, is at fault. | `tests/e2e/test_evaluation_workspace.py::TestAIMakerRoleNavigation::test_clicking_ai_maker_goes_to_org_selection`, `test_org_selection_shows_civicdatalab`, `test_org_selection_has_multiple_orgs`, `test_selecting_civicdatalab_navigates_to_dashboard`; `tests/e2e/test_evaluation_workspace.py::TestEvaluatorRoleNavigation::test_switch_roles_returns_to_role_selection` (reclassified 2026-07-31 from bug #3 — a scripted repro isolated from concurrent CI load showed the `/dashboard/ai-maker` org-selection screen stuck on a bare "Loading" state for 40s+ after clicking the AI Maker card, with no redirect modal and no session loss; this is a more severe sibling of this row's race — instead of `isLoading` flipping to `false` with an empty array, the `myOrganizations` query itself never resolves within the observation window, so `isLoading` never flips at all. Same query, same family, worse symptom.) | 2026-07-28 | @@ -42,6 +42,16 @@ Format: append-only. When a bug is fixed in the app, mark `status: fixed` and th | 28 | evaluations list / sorting | Clicking a sortable column header (e.g. "Evaluation Name") on the evaluations table's first click does **nothing at all** — no visual reorder, and the header's `aria-sort` attribute stays `None`/absent. The second click (and every click after) does re-sort the rows correctly. Confirmed live 2026-08-13 with a scripted repro that clicked the "Evaluation Name" header 3 times in a row with a 3s settle between each: click 1 → row order and `aria-sort` unchanged from the pre-click state; click 2 → rows visibly reordered; click 3 → rows reordered again — but `aria-sort` never changes from `None` across all three clicks, so screen-reader users get no indication a sort is active even once engaged (a secondary a11y defect on top of the primary one-click-eaten defect). The `DataTable-module_Heading__*` header button is rendered by the shared `opub-ui` `DataTable` component (no local source in `ParakhAI-frontend` — it's a compiled dependency), so the root cause could not be pinned to an exact line; likely a debounce/state-init bug where the component's internal sort-state only starts reacting to clicks after the first one initializes it. | open | Log in as `TEST_EMAIL_1` → AI Maker → CivicDataLab → Evaluations. Click the "Evaluation Name" column header once. Row order is unchanged and `document.querySelector("th button:has-text('Evaluation Name')").getAttribute('aria-sort')` stays `null`. Click it again — rows now visibly reorder. | `tests/e2e/test_evaluations_list_controls.py::TestSortableHeaders::test_sort_by_name_toggles_order` | 2026-08-13 | | 30 | evaluations list / default sort | The default (unfiltered, unsorted) `audits` GraphQL query the evaluations list loads with does **not** order by recency — a newly created draft never appears on the list's landing page, no matter how long you wait or how many times you hard-reload. Root-caused 2026-08-17 via a live network capture of the actual `GetAudits` response (not a synthetic probe): four separate fresh-draft creations in the same org (audit ids 2042–2045, confirmed to exist individually via `audit(auditId)` and to rank at the top of a raw `audits(limit:20)` GraphQL call with no `sortOptions`) each produced an identical captured frontend response — `ids=['1916','1957','1953','1943','1944','1954','1961','1909','1964','1958']`, `total=141` — completely unrelated to the just-created id and byte-identical across all four runs, including after a full `page.reload()` + 45s app-ready wait. Response headers carried no `cache-control`/`age`/`x-cache` (200, no HTTP caching), so this isn't a CDN/browser cache — the frontend's actual `GetAudits` call (with its real field selection, distinct from a bare `id name status` probe) is deterministically returning the same fixed page of ~10 rows regardless of what's been created since, meaning its effective sort key is something other than recency (or a query-plan/index quirk that happens to be stable across requests). User impact: creating an evaluation and clicking "Back to List" gives no way to find your own draft without knowing to search or filter — among 141 rows it could be anywhere. Related to bug #13 (same `audits`/`GetAudits` query family, full-field-vs-light-field behavior) but a distinct symptom — not a hang, a wrong/stale-looking default ordering. | open | Log in as `TEST_EMAIL_1` → AI Maker → CivicDataLab → New Evaluation → Bulk → Start (creates a draft, capture its `auditId` from the URL) → Back to List. The draft's row (`a[href*='auditId=']`) is absent from the table even after a full page reload and a 45s wait. Confirm via GraphQL that the audit exists (`audit(auditId: "")`) and that a raw `audits(limit:20)` call (no `sortOptions`) ranks it near the top — yet the frontend's own list request returns a fixed, unrelated set of older ids. Reproduced 4/4 times in complete isolation (`-n 1 --reruns 0`, no concurrent suite load) 2026-08-17. | `tests/e2e/test_add_evaluation_bulk.py::TestBulkDraftPersistence::test_new_draft_appears_in_draft_tab_with_wizard_link` | 2026-08-17 | +| 31 | a11y / homepage contrast | The active feature tab on the homepage ("Automation-assisted Evaluation Environment", classes `bg-[#E8E4FF] text-[#6849EE]`) has a contrast ratio of 4.49:1 against the 4.5:1 AA minimum for 16px normal-weight text (WCAG 1.4.3). axe `color-contrast`, serious. Only visible once the client-rendered tabs have mounted — the pre-existing homepage axe test scans the pre-hydration skeleton and never sees it. Darkening the text (e.g. `#6242E6`) fixes it. | open | Load `https://dev.parakh.civicdataspace.in/`, wait for the tab row, run axe with tags `wcag2aa`: one `color-contrast` node `.bg-\[\#E8E4FF\]`, "contrast of 4.49 (foreground #6849ee, background #e8e4ff)". | `tests/accessibility/test_accessibility_hydrated.py::TestHydratedHomepageAxe::test_color_contrast_meets_aa` | 2026-10-31 | +| 32 | a11y / homepage feature tabs | The four feature "tabs" are plain `