diff --git a/README.md b/README.md index 3eb84d4..4040f52 100644 --- a/README.md +++ b/README.md @@ -77,6 +77,7 @@ its purpose, trigger phrases, and full instructions. | Skill | Description | |------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------| +| **[accessibility-infra-setup](./skills/accessibility-infra-setup/)** | Set up automated accessibility (a11y) check infrastructure in an existing Angular app with Playwright + axe-core (WCAG 2.2 AA) — detects existing test setup, scaffolds config, a shared axe fixture, one dummy example scan, npm scripts, and docs, then validates the sample passes. | | **[bdd-maintain](./skills/bdd-maintain/)** | REMOVE and DEAD CODE AUDIT modes — delete feature files, step definitions, and PageObjects linked to | **[create-repository](./skills/create-repository/)** | Creates and configures a GitHub repository under AbsaOSS (open source) or absa-group (internal). Runs a short interview, then shows an annotated `gh` command plan and executes it on confirmation — visibility, naming, admins, CODEOWNERS, labels, merge settings, and project links. | | **[data-cy-instrument](./skills/data-cy-instrument/)** | Resolve missing `data-cy` attributes in Angular component templates and sync PageObjects to use `getByTestId()` — run after a crawl when `coverage_gaps` are non-empty. | diff --git a/docs/README.md b/docs/README.md index 366284b..d0aff01 100644 --- a/docs/README.md +++ b/docs/README.md @@ -72,6 +72,7 @@ Task-specific guidance for using individual skills. Start by picking your use ca | Guide | Description | |---|---| +| [Accessibility Infra Setup](./guides/accessibility-infra-setup.md) | Bootstrap Playwright + axe-core a11y infrastructure in an Angular app (WCAG 2.2 AA) — config, shared fixture, one example scan, npm scripts, docs | | [TDD Workflow](./guides/tdd-workflow.md) | Test-driven development with: specification, confirmation gates, and vertical-sliced implementation | | [Unit Test Standards](./guides/test-unit-standards.md) | Reference for unit test standards across isolation, scope, naming, assertions, coverage, fixtures | | [Unit Test Writer](./guides/test-unit-write.md) | Generate complete unit tests from scratch following language-specific standards | diff --git a/docs/guides/accessibility-infra-setup.md b/docs/guides/accessibility-infra-setup.md new file mode 100644 index 0000000..5e1fd67 --- /dev/null +++ b/docs/guides/accessibility-infra-setup.md @@ -0,0 +1,94 @@ +# Accessibility Infra Setup Skill + +The `accessibility-infra-setup` skill bootstraps automated accessibility (a11y) check infrastructure in an existing Angular application using [Playwright](https://playwright.dev/) + [axe-core](https://github.com/dequelabs/axe-core) (`@axe-core/playwright`). It scaffolds the plumbing to catch automatically detectable **WCAG 2.2 AA** violations and proves it works with one passing example scan. + +It activates when you ask to set up accessibility checks, add a11y testing infrastructure, or wire axe-core into an Angular app. + +--- + +## What it does + +The skill extends your existing test setup rather than replacing it, then validates the result end to end: + +| Step | Action | +|------|--------| +| Detect | Inspects `package.json`, the dev-server script, package manager, and any existing Playwright / Cypress setup | +| Install | Adds `@playwright/test` + `@axe-core/playwright` and installs the Chromium browser | +| Scaffold | Creates `playwright.config.ts`, a shared axe fixture, and one dummy example scan | +| Wire scripts | Adds `test:a11y*` npm scripts to `package.json` | +| Validate | Runs the dummy scan until it passes green — the definition of done | +| Document | Adds `docs/accessibility.md` (or `playwright/README.md`) with how to run and where reports land | + +**Scope:** infrastructure plus exactly **one** dummy example scan. Authoring real accessibility tests, fixing violations, and CI wiring are all out of scope. + +--- + +## Expected outcome + +After the skill runs, the repository contains: + +``` +playwright/ +├── fixtures/ +│ └── axe-helpers.ts # Shared axe fixture (WCAG 2.2 AA tags) + assertion/incomplete-warning helpers +├── print-a11y-warnings.js # Prints tests carrying incomplete-result warnings +└── a11y/ + └── example.accessibility.spec.ts # ONE dummy scan of static, known-compliant HTML — passes +playwright.config.ts # `accessibility` project (Desktop Chrome, testMatch /\.accessibility\.spec\.ts$/) +docs/accessibility.md # How to run, where reports land +``` + +Running `npm run test:a11y` starts the dev server and runs the dummy scan on Desktop Chrome against +static, known-compliant HTML (via `page.setContent`, not a real app route), so it passes green +regardless of whether the app itself is WCAG-compliant yet — fixing app violations is a separate, +out-of-scope concern. axe `violations` fail the test; `incomplete` results (findings axe couldn't +confirm without human judgement) are non-blocking — they're surfaced as a warning annotation +instead. Run `npm run test:a11y:incomplete` to print any without opening the HTML report. + +--- + +## How to trigger it + +Ask naturally — the skill fires on intent, not exact wording: + +``` +set up accessibility checks +add a11y testing infrastructure +add axe-core to this Angular app +set up WCAG testing +add accessibility scans with Playwright +bootstrap a11y infra +``` + +--- + +## Requirements + +- An existing Angular application with a runnable dev server (`npm run start` on port 4200) +- Node.js 22+ +- Network access on first run (Playwright downloads the Chromium browser) + +--- + +## Coexistence rules + +- **Never touches Cypress.** If the repo uses Cypress, its config, specs, and dependencies are left untouched — Playwright is added alongside it. +- **Extends, doesn't recreate Playwright.** When a `playwright.config.*` already exists, the skill merges an `accessibility` project into it and reuses any existing axe fixture instead of adding a duplicate. + +--- + +## Installation + +The skill is installed along with the rest of the toolkit: + +```bash +npx skills add https://github.com/AbsaOSS/agentic-toolkit -g +``` + +To install only this skill: + +```bash +npx skills add https://github.com/AbsaOSS/agentic-toolkit -g --skill accessibility-infra-setup +``` + +See [Getting Started](../getting-started.md) for the full install guide. diff --git a/skills/accessibility-infra-setup/SKILL.md b/skills/accessibility-infra-setup/SKILL.md new file mode 100644 index 0000000..681f926 --- /dev/null +++ b/skills/accessibility-infra-setup/SKILL.md @@ -0,0 +1,272 @@ +--- +name: accessibility-infra-setup +description: > + Sets up automated accessibility (a11y) check infrastructure in an existing Angular application + using Playwright + axe-core (@axe-core/playwright) to catch automatically detectable WCAG 2.2 AA violations. + Detects the existing test setup (including package manager), installs and wires Playwright with a + dedicated `accessibility` project, scaffolds a shared axe fixture, adds one dummy example scan, + wires package.json scripts, documents how to run the checks, and validates that the sample passes. + Activates on requests like: "set up + accessibility checks", "add a11y testing infrastructure", "add axe-core to this Angular app", + "set up WCAG testing", "add accessibility scans with Playwright", "bootstrap a11y infra". + Scope is infrastructure only plus ONE dummy example test — authoring real accessibility tests is + a separate concern and out of scope. +license: Apache-2.0 +compatibility: > + Requires an existing Angular application with a runnable dev server (`start` script on port 4200, + run via npm, Yarn, or pnpm) and Node.js 22+. Installs Playwright browsers, which needs network + access on first run. +--- + +# accessibility-infra-setup + +Bootstraps Playwright + axe-core accessibility infrastructure in an existing Angular app. Follows the +[AbsaOSS/cps-shared-ui](https://github.com/AbsaOSS/cps-shared-ui) pattern, generalized for a plain +application (not a component library). + +**Scope:** infrastructure + exactly one dummy example scan. Do **not** author real accessibility +tests here — that is a separate follow-up task. + +## Expected outcome + +After this skill runs, the repository contains (fresh setup; see Step 3 for where these land instead +when merging into an existing `testDir`): + +``` +playwright/ +├── fixtures/ +│ └── axe-helpers.ts # Shared axe fixture (WCAG 2.2 AA tags) + assertion + incomplete-warning helpers +├── print-a11y-warnings.cjs # Prints tests carrying incomplete-result warnings (reusable module + CLI) +├── run-a11y-incomplete.cjs # Runs the accessibility project, then prints incomplete-result warnings +└── a11y/ + └── example.accessibility.spec.ts # ONE dummy scan of static, known-compliant HTML — passes +playwright.config.ts # `accessibility` project (Desktop Chrome, testMatch /\.accessibility\.spec\.ts$/) +docs/accessibility.md # How to run, where reports land (or playwright/README.md) +``` + +- `package.json` gains `@playwright/test` + `@axe-core/playwright` (devDependencies) and + `test:a11y*` scripts, including `test:a11y:incomplete`. +- Running `test:a11y` through the repo's detected package manager (see Step 1) starts the dev server + and runs the dummy scan on Desktop Chrome against static, known-compliant HTML (via + `page.setContent`, not a real app route) — it **passes green** regardless of whether the app + itself is WCAG-compliant yet. That green run is the definition of done; fixing app violations is a + separate, out-of-scope concern. +- axe `violations` fail the test; axe `incomplete` results (checks axe couldn't confirm without + human judgement) are non-blocking — reported as a warning annotation instead, never as a failure. + +## Workflow + +Copy this checklist and track progress: + +``` +- [ ] Step 1: Detect existing test infrastructure (incl. package manager) +- [ ] Step 2: Install dependencies +- [ ] Step 3: Scaffold config + fixture + example spec +- [ ] Step 4: Wire package.json scripts +- [ ] Step 5: Validate (run the dummy scan, confirm the pipeline itself works) +- [ ] Step 6: Document +``` + +### Step 1 · Detect existing test infrastructure + +Inspect the repo before writing anything — the goal is to **extend, not clobber**. + +1. Read `package.json`: note the dev-server script (usually `start` → `ng serve`, port 4200), the + package manager (from `package-lock.json` / `yarn.lock` / `pnpm-lock.yaml`, or a `packageManager` + field — distinguish Yarn Classic from Yarn Berry/PnP via `.yarnrc.yml`'s `nodeLinker`), and any + existing `@playwright/test`, `@axe-core/playwright`, Cypress, or Karma entries. +2. Check for an existing `playwright.config.*` and for Cypress (`cypress.config.*`, a `cypress/` + folder, or a `cypress` devDependency). + +**Carry the detected package manager through every later step.** All install and run commands below +are shown for npm, Yarn (Classic and Berry/PnP), and pnpm — use the one matching this repo, not npm +by default. This matters most under Yarn PnP: it doesn't hoist packages into `node_modules/.bin`, so +a plain `npm run ...` or bare `npx playwright ...` there fails to resolve the Playwright binary even +though setup is otherwise correct. + +Pick exactly one path based on what Playwright infra exists — Cypress never changes the decision: + +- **Playwright already configured** (a `playwright.config.*` exists) → **adjust it in place, do not + recreate it.** Merge in an `accessibility` project + `{ name: 'accessibility', testMatch: /\.accessibility\.spec\.ts$/, use: { ...devices['Desktop Chrome'] } }`, + add `testIgnore: /\.accessibility\.spec\.ts$/` to existing functional projects, and ensure `webServer` + starts the Angular dev server. The pattern is anchored to the filename suffix, not a bare + `/accessibility/` substring match, so a directory segment (e.g. a checkout under + `accessibility-app/`) can't cause unrelated specs to be picked up or excluded. Keep the user's + existing `testDir` and place a11y specs (and the axe fixture, if new) underneath it, with import + paths adjusted to resolve from that location — do not introduce a separate top-level `playwright/` + directory alongside it. Reuse any existing axe fixture instead of adding a second one. Merge the + `reporter` array so the existing reporter(s) (e.g. `'html'` or `['html']`) are kept as-is and a + `['json', { outputFile: 'playwright-report/summary.json' }]` entry is added alongside — never + replace an existing reporter outright. +- **No Playwright yet** → fresh setup. Copy `assets/playwright.config.ts` to the repo root and + scaffold the full structure in Step 3. + +**Cypress is off-limits either way.** If Cypress is present, treat it as read-only: never edit, +migrate, or delete Cypress config, specs, or dependencies, and never fold a11y scans into Cypress. +The two runners coexist — Playwright owns accessibility, Cypress keeps whatever it already covers. +Cypress existing does **not** make this a "fresh" or "adjust" decision; only the presence/absence of +a `playwright.config.*` does. + +Confirm the dev-server command and port with the user only if they differ from `start` / `4200`; +otherwise proceed. + +### Step 2 · Install dependencies + +Use the package manager detected in Step 1 — not npm by default: + +```bash +# npm +npm install -D @playwright/test @axe-core/playwright +npx playwright install chromium + +# Yarn (Classic or Berry/PnP) +yarn add -D @playwright/test @axe-core/playwright +yarn playwright install chromium + +# pnpm +pnpm add -D @playwright/test @axe-core/playwright +pnpm exec playwright install chromium +``` + +Under Yarn PnP there is no `node_modules/.bin`, so a bare `npx playwright ...` cannot resolve the +binary — always go through `yarn playwright ...` (or `yarn dlx playwright ...`) instead. The browser +download step needs network access regardless of package manager. If Playwright is already +installed, install `@axe-core/playwright` when it is missing, skip reinstalling `@playwright/test`, +and still ensure the Chromium browser is present. + +### Step 3 · Scaffold config, fixture, and example spec + +Create these files (templates live in this skill's `assets/`). The root is `playwright/` on a fresh +setup; when merging into an existing config, use the existing `testDir` as the root instead and keep +the same relative layout below it — otherwise Playwright's `testDir` never discovers the new files. + +1. `playwright.config.ts` (root) — from `assets/playwright.config.ts` (fresh setup only; otherwise + merge as in Step 1). Adjust `baseURL`, `webServer.command`, and port if the app differs, and set + `webServer.command` to launch the dev server through the package manager detected in Step 1 + (`npm run start`, `yarn start`, or `pnpm start`) rather than defaulting to npm. +2. `/fixtures/axe-helpers.ts` — copy verbatim from `assets/axe-helpers.ts`. This is the + single source of the WCAG 2.2 AA tag set; every scan must build from `makeAxeBuilder`. It also + exports `annotateIncomplete`, which every scan must call on `results.incomplete` so those + findings are reported as a warning annotation instead of silently dropped or failing the build. +3. `/a11y/example.accessibility.spec.ts` — from `assets/example.accessibility.spec.ts`. + It scans static, known-compliant HTML via `page.setContent(...)` rather than a real app route, so + the dummy proof stays green independent of whether the app itself is WCAG-compliant yet. Do not + point it at a real route (e.g. `page.goto('/')`) — that reintroduces exactly the failure mode this + avoids. Copy the template's HTML as-is; it only needs a landmark, a heading, and text. +4. `/print-a11y-warnings.cjs` — copy verbatim from `assets/print-a11y-warnings.cjs`. Exports a + `printWarnings(report)` helper (plus a `` CLI mode) that prints every test carrying an + incomplete-result warning, so they can be reviewed without opening the HTML report. +5. `/run-a11y-incomplete.cjs` — copy verbatim from `assets/run-a11y-incomplete.cjs`. The + cross-platform entry point behind `test:a11y:incomplete`: it spawns the accessibility project + itself with `--reporter=json` piped to a temp file (never a fixed path, and never stdout — test + attachments like screenshots/videos are base64-inlined and can blow past `spawnSync`'s stdout + buffer), calls `printWarnings` on it, then exits with the underlying test run's own exit code. + + Both scripts use the `.cjs` extension (not `.js`) so they load as CommonJS even in an Angular app + with `"type": "module"` in `package.json` — plain `.js` there is parsed as ESM and `require` throws + `ReferenceError: require is not defined`. Keep the `.cjs` extension regardless of the host + package's `type` field. + +`` is `playwright/` on a fresh setup, or the existing `testDir` when merging — e.g. with +`testDir: './e2e'` the fixture lands at `e2e/fixtures/axe-helpers.ts` and the dummy spec at +`e2e/a11y/example.accessibility.spec.ts`. Adjust import paths (e.g. `../fixtures/axe-helpers`) to +match wherever `` ends up; the relative layout between the files stays the same either way. + +The `.accessibility.spec.ts` filename suffix is what routes a spec to the accessibility project — +keep it. + +### Step 4 · Wire package.json scripts + +Add to `package.json` `scripts` (do not clobber existing entries). These entries are invoked via +whichever package manager Step 1 detected (`npm run test:a11y`, `yarn test:a11y`, or +`pnpm test:a11y`), so the script bodies themselves don't need to vary: + +```jsonc +"test:a11y": "playwright test --project=accessibility", +"test:a11y:headed": "playwright test --project=accessibility --headed", +"test:a11y:report": "playwright show-report", +"test:a11y:incomplete": "node playwright/run-a11y-incomplete.cjs" +``` + +Substitute `playwright/` in `test:a11y:incomplete` with the actual `` from Step 3 — e.g. for an +existing `testDir: './e2e'` the script must read `"node e2e/run-a11y-incomplete.cjs"`. The hardcoded +`playwright/` only applies on a fresh setup; using it verbatim when merging into an existing `testDir` +points at a path that was never created, and the script fails with `MODULE_NOT_FOUND`. + +`test:a11y:incomplete` is self-contained — it runs its own accessibility-project pass (JSON reporter +only, written to a temp file) and prints any incomplete-result warnings, then exits with that run's +status. Run it standalone; it does not depend on `test:a11y` having run first. + +### Step 5 · Validate — run the dummy scan + +Run the feedback loop until green, invoking the script through the package manager detected in +Step 1 (never assume npm): + +```bash +npm run test:a11y # npm +yarn test:a11y # Yarn (Classic or Berry/PnP) +pnpm test:a11y # pnpm +``` + +1. If it **passes**, the infrastructure is proven. Done. +2. If it fails, it is a **setup** problem (missing browser, wrong port, dev server timeout, import + error, or — under Yarn PnP — a package-manager mismatch where the wrong command was used to + invoke the script) — the dummy HTML is fixed and known-compliant, so a real WCAG violation should + never be the cause. Fix the config/paths and re-run. Common causes: dev server not on 4200, + `webServer.command` wrong, Chromium not installed, fixture import path incorrect, script run via + the wrong package manager. + +Do not finish until `test:a11y` exits green through the detected package manager. + +### Step 6 · Document + +Add `assets/accessibility-README.md` to the repo as `docs/accessibility.md` (or +`playwright/README.md`). Adjust file paths/scripts to match what you created, and rewrite the +`npm run ...` commands in it to match the package manager detected in Step 1 (`yarn ...` / `pnpm ...`) +so the docs don't point users at a command that can't resolve the binary. Add a short +"Accessibility" note with the run command to the main `README.md` if one exists. + +## Gotchas + +- **Separate project, Chrome only.** axe evaluates rendered DOM/ARIA, not browser rendering quirks — + scan once on Desktop Chrome. Do not fan a11y scans across webkit/firefox. +- **Wait for animations.** Scanning mid-transition produces false-positive color-contrast + violations. `waitForAnimationsToFinish` (in the fixture) prevents this — call it before every scan. +- **Never touch Cypress.** If the repo uses Cypress, leave its config, specs, and dependencies + untouched — add Playwright alongside it rather than migrating or editing anything Cypress owns. +- **Never default to npm.** Step 1 detects the repo's package manager (npm, Yarn Classic, Yarn + Berry/PnP, or pnpm) — use it consistently for installs, `webServer.command`, the validation run, + and the docs. Yarn PnP in particular has no `node_modules/.bin`, so a bare `npm run ...` or + `npx playwright ...` there fails to resolve the binary even when the rest of the setup is correct. +- **Extend, don't recreate Playwright.** When a `playwright.config.*` already exists, merge the + `accessibility` project into it and reuse any existing axe fixture — do not generate a second + config or a duplicate fixture. Scaffold the fixture and dummy spec under the existing `testDir`, + not a hardcoded `playwright/` — otherwise Playwright's own `testDir` setting never discovers them + and validation fails with "no tests found". +- **Filename routing.** A spec only lands in the accessibility project if its filename ends in + `.accessibility.spec.ts`. This is `testMatch: /\.accessibility\.spec\.ts$/`, anchored to the + filename suffix rather than a bare `/accessibility/` substring — a directory named e.g. + `accessibility-app/` must never cause unrelated specs to match. +- **git-ignore artifacts.** Ensure `test-results/` and `playwright-report/` are git-ignored. +- **Helper scripts must stay `.cjs`.** `print-a11y-warnings.cjs` and `run-a11y-incomplete.cjs` use + CommonJS `require`/`module.exports`. If the host `package.json` has `"type": "module"`, a plain + `.js` copy is loaded as ESM and fails immediately with `ReferenceError: require is not defined`. + Copy them with the `.cjs` extension regardless of the app's `type` field, and keep the `require` + between them pointed at `./print-a11y-warnings.cjs`. +- **WCAG tag set lives in one place.** Never inline `withTags(...)` in a spec — always go through + `makeAxeBuilder`, so the standard stays consistent as scans are added later. +- **Incomplete ≠ violation.** Never assert `expectNoViolations(results.incomplete)` in a fresh setup + — `incomplete` results need human judgement and must only be surfaced via `annotateIncomplete` + (a non-blocking warning annotation), never used to fail the build. +- **Dummy scan uses static HTML, not a real route.** The example spec calls `page.setContent(...)` + with known-compliant markup instead of `page.goto('/')`, so it proves the plumbing without + depending on whether the app itself is WCAG-compliant. Do not repoint it at a real route — real + per-page coverage belongs in dedicated specs added later, once the app's actual violations (if any) + are a separate, tracked concern. + +## Out of scope + +- Writing real accessibility tests / per-page coverage (separate skill/task). +- Fixing accessibility violations found in the app. +- pa11y-ci, Lighthouse, or CI-pipeline wiring — Playwright + axe-core only, run locally. diff --git a/skills/accessibility-infra-setup/assets/accessibility-README.md b/skills/accessibility-infra-setup/assets/accessibility-README.md new file mode 100644 index 0000000..5995fce --- /dev/null +++ b/skills/accessibility-infra-setup/assets/accessibility-README.md @@ -0,0 +1,74 @@ +# Accessibility Tests + +Automated accessibility checks for this app, powered by +[Playwright](https://playwright.dev/) + [axe-core](https://github.com/dequelabs/axe-core) +(via [`@axe-core/playwright`](https://github.com/dequelabs/axe-core-npm/tree/develop/packages/playwright)). +Scans assert compliance with **WCAG 2.2 AA** (plus axe best-practice rules); manual testing is still required for full compliance. + +axe-core's `violations` fail the test; `incomplete` results (checks axe couldn't confirm without +human judgement, e.g. combobox `aria-controls` patterns) are non-blocking and instead surfaced as a +`warning` annotation on the test, visible in the HTML report and the attached scan JSON. Run +`npm run test:a11y:incomplete` (or `yarn test:a11y:incomplete` / `pnpm test:a11y:incomplete`, +matching this repo's package manager) after a scan to print every test carrying such a warning +without opening the HTML report. + +## Layout + +``` +playwright/ +├── fixtures/ +│ └── axe-helpers.ts # Shared axe fixture (WCAG 2.2 AA tag set) + assertion helpers +├── print-a11y-warnings.cjs # Prints tests with incomplete-result warnings (reusable module + CLI) +├── run-a11y-incomplete.cjs # Runs the accessibility project, then prints incomplete-result warnings +└── a11y/ + └── example.accessibility.spec.ts # Dummy example scan of static, known-compliant HTML +playwright.config.ts # `accessibility` project routes any *accessibility* spec here +``` + +Any spec file whose name ends in `.accessibility.spec.ts` runs under the dedicated +`accessibility` Playwright project (Desktop Chrome only) — e.g. `accessibility-home.spec.ts` does +**not** match and is silently excluded. + +## Running + +Commands below use npm; substitute `yarn` or `pnpm` if that's this repo's package manager. + +```bash +npm run test:a11y # run all accessibility scans (auto-starts the dev server) +npm run test:a11y:headed # same, with the browser visible +npm run test:a11y:report # open the last HTML report +npm run test:a11y:incomplete # re-run the accessibility project and print incomplete-result warnings +``` + +Playwright auto-starts the Angular dev server and waits for `http://localhost:4200`. If a dev server +is already running locally, it is reused. + +## Reports and artifacts + +- HTML report → `playwright-report/` (open with `npm run test:a11y:report`) +- Full axe results JSON attached to each test → visible in the HTML report +- Screenshots / videos / traces on failure → `test-results/` + +Both directories are git-ignored. + +## Adding a scan + +1. Create a spec whose filename ends in `.accessibility.spec.ts`, e.g. + `playwright/a11y/checkout.accessibility.spec.ts`. +2. Import the shared fixture: + + ```ts + import { + test, + expectNoViolations, + annotateIncomplete, + waitForAnimationsToFinish + } from '../fixtures/axe-helpers'; + ``` + +3. Navigate, wait for animations, scan with `makeAxeBuilder()`, call + `annotateIncomplete(results.incomplete, testInfo)` to report (not fail on) incomplete results, + then assert with `expectNoViolations(results.violations)`. Use the example spec as a template. + +Writing real accessibility tests is intentionally out of scope of the setup — +this directory only ships the infrastructure and one dummy example. diff --git a/skills/accessibility-infra-setup/assets/axe-helpers.ts b/skills/accessibility-infra-setup/assets/axe-helpers.ts new file mode 100644 index 0000000..3b4ac1f --- /dev/null +++ b/skills/accessibility-infra-setup/assets/axe-helpers.ts @@ -0,0 +1,77 @@ +import { test as base, expect, type Page, type TestInfo } from '@playwright/test'; +import AxeBuilder from '@axe-core/playwright'; + +type Violations = Awaited>['violations']; +type Incomplete = Awaited>['incomplete']; + +type AxeFixture = { + makeAxeBuilder: () => AxeBuilder; +}; + +/** + * Shared axe-core fixture. Every accessibility scan builds its AxeBuilder from + * here so the WCAG 2.2 AA tag set stays identical across the whole suite. + */ +export const test = base.extend({ + makeAxeBuilder: async ({ page }, use) => { + const makeAxeBuilder = () => + new AxeBuilder({ page }).withTags([ + 'wcag2a', + 'wcag2aa', + 'wcag21a', + 'wcag21aa', + 'wcag22a', + 'wcag22aa', + 'best-practice' + ]); + await use(makeAxeBuilder); + } +}); + +export function formatViolations(violations: Violations): string { + if (violations.length === 0) return ''; + return violations + .map((v) => { + const nodes = v.nodes + .map((n) => ` - ${n.html}\n ${n.failureSummary}`) + .join('\n'); + return `\n[${v.impact}] ${v.id}: ${v.description}\n Help: ${v.helpUrl}\n${nodes}`; + }) + .join('\n'); +} + +export function expectNoViolations(violations: Violations) { + expect(violations, formatViolations(violations)).toHaveLength(0); +} + +/** + * Surface axe "incomplete" results (checks axe couldn't confirm without human + * judgement, e.g. combobox aria-controls patterns) as a non-blocking warning + * annotation instead of failing the test. Visible in the HTML report and + * printable via `print-a11y-warnings.cjs` without opening it. + */ +export function annotateIncomplete(incomplete: Incomplete, testInfo: TestInfo) { + if (incomplete.length === 0) return; + testInfo.annotations.push({ + type: 'warning', + description: incomplete + .map((r) => `[${r.impact ?? 'unknown'}] ${r.id} (${r.nodes.length} node(s))`) + .join('; ') + }); +} + +/** + * Wait for all animations/transitions to complete before scanning. Without this, + * axe may capture intermediate states (e.g. mid-transition background color) and + * report false-positive color-contrast violations. + */ +export async function waitForAnimationsToFinish(page: Page) { + await page.evaluate(() => + Promise.all( + document + .getAnimations() + .filter((a) => a.effect?.getTiming().iterations !== Infinity) + .map((a) => a.finished.catch(() => {})) + ) + ); +} diff --git a/skills/accessibility-infra-setup/assets/example.accessibility.spec.ts b/skills/accessibility-infra-setup/assets/example.accessibility.spec.ts new file mode 100644 index 0000000..213d30f --- /dev/null +++ b/skills/accessibility-infra-setup/assets/example.accessibility.spec.ts @@ -0,0 +1,39 @@ +import { + test, + expectNoViolations, + annotateIncomplete, + waitForAnimationsToFinish +} from '../fixtures/axe-helpers'; + +// Dummy example scan proving the infrastructure works end to end. Uses static, +// known-compliant HTML rather than a real app route, so it stays green regardless +// of whether the app itself is WCAG-compliant yet - fixing app violations is out +// of scope here. Add real per-page scans in dedicated *.accessibility.spec.ts files. +const COMPLIANT_HTML = ` + + Accessibility smoke test + +

Accessibility smoke test

+
+

This page exists only to prove the axe-core + Playwright plumbing works.

+
+ +`; + +test.describe('Accessibility - example scan', () => { + test('dummy page has no WCAG 2.2 AA violations', async ({ page, makeAxeBuilder }, testInfo) => { + await page.setContent(COMPLIANT_HTML); + await waitForAnimationsToFinish(page); + + const results = await makeAxeBuilder().analyze(); + + await testInfo.attach('dummy-accessibility-scan', { + body: JSON.stringify(results, null, 2), + contentType: 'application/json' + }); + + // "incomplete" results need human judgement - report, don't fail the build. + annotateIncomplete(results.incomplete, testInfo); + expectNoViolations(results.violations); + }); +}); diff --git a/skills/accessibility-infra-setup/assets/playwright.config.ts b/skills/accessibility-infra-setup/assets/playwright.config.ts new file mode 100644 index 0000000..444a80a --- /dev/null +++ b/skills/accessibility-infra-setup/assets/playwright.config.ts @@ -0,0 +1,54 @@ +import { defineConfig, devices } from '@playwright/test'; + +/** + * Playwright configuration with a dedicated `accessibility` project. + * + * Any spec whose filename ends in `.accessibility.spec.ts` runs axe-core scans on + * Desktop Chrome only — axe evaluates rendered DOM/ARIA state, so cross-browser + * variation adds no value. The pattern is anchored to the filename (not just + * `/accessibility/`) so a path segment like `accessibility-app/` can't false-match + * unrelated specs. Functional specs (if any) run under `chromium`. + * + * See https://playwright.dev/docs/test-configuration + */ +export default defineConfig({ + testDir: './playwright', + outputDir: 'test-results', + fullyParallel: true, + forbidOnly: !!process.env.CI, + retries: process.env.CI ? 2 : 0, + workers: process.env.CI ? 1 : undefined, + reporter: [['html'], ['json', { outputFile: 'playwright-report/summary.json' }]], + timeout: 30_000, + expect: { timeout: 5_000 }, + use: { + baseURL: 'http://localhost:4200', + actionTimeout: 5_000, + navigationTimeout: 5_000, + trace: 'on-first-retry', + screenshot: 'only-on-failure', + video: 'retain-on-failure' + }, + projects: [ + { + name: 'chromium', + testIgnore: /\.accessibility\.spec\.ts$/, + use: { ...devices['Desktop Chrome'] } + }, + { + name: 'accessibility', + testMatch: /\.accessibility\.spec\.ts$/, + use: { ...devices['Desktop Chrome'] } + } + ], + /* Start the Angular dev server before running tests. */ + webServer: { + // Replace with the detected package manager's run command, e.g. `yarn start` / `pnpm start`. + command: 'npm run start', + url: 'http://localhost:4200', + /* Reuse a running dev server locally; CI always starts fresh. */ + reuseExistingServer: !process.env.CI, + /* Angular builds can be slow — allow up to 2 minutes. */ + timeout: 120_000 + } +}); diff --git a/skills/accessibility-infra-setup/assets/print-a11y-warnings.cjs b/skills/accessibility-infra-setup/assets/print-a11y-warnings.cjs new file mode 100644 index 0000000..941c95b --- /dev/null +++ b/skills/accessibility-infra-setup/assets/print-a11y-warnings.cjs @@ -0,0 +1,47 @@ +#!/usr/bin/env node +// Reads a Playwright JSON-reporter report and prints every test that carries +// a 'warning' annotation (pushed by annotateIncomplete() for axe "incomplete" +// results) - lets you find those without opening the HTML report. +// .cjs (not .js): keeps this CommonJS regardless of the host package's "type": "module". +const fs = require('fs'); + +function* walkSpecs(suites) { + for (const suite of suites) { + yield* (suite.specs ?? []).map((spec) => spec); + if (suite.suites) yield* walkSpecs(suite.suites); + } +} + +// Exported so run-a11y-incomplete.cjs can reuse it on an in-memory report. +function printWarnings(report) { + let count = 0; + for (const spec of walkSpecs(report.suites ?? [])) { + for (const test of spec.tests ?? []) { + // TestResult.annotations only exists from Playwright 1.52+; older versions + // (e.g. 1.47, still in use by some projects) only populate test.annotations. + const resultAnnotations = (test.results ?? []).flatMap((r) => r.annotations ?? []); + const annotations = resultAnnotations.length > 0 ? resultAnnotations : (test.annotations ?? []); + const warnings = annotations.filter((a) => a.type === 'warning'); + if (warnings.length === 0) continue; + count++; + console.log(`\n${spec.file}:${spec.line} - ${spec.title} [${test.projectName}]`); + for (const w of warnings) console.log(` ${w.description}`); + } + } + + console.log( + count === 0 ? '\nNo tests with accessibility warnings found.' : `\n${count} test(s) with accessibility warnings.` + ); + return count; +} + +if (require.main === module) { + const reportPath = process.argv[2]; + if (!reportPath) { + console.error('Usage: node playwright/print-a11y-warnings.cjs '); + process.exit(1); + } + printWarnings(JSON.parse(fs.readFileSync(reportPath, 'utf8'))); +} + +module.exports = { printWarnings }; diff --git a/skills/accessibility-infra-setup/assets/run-a11y-incomplete.cjs b/skills/accessibility-infra-setup/assets/run-a11y-incomplete.cjs new file mode 100644 index 0000000..233e49a --- /dev/null +++ b/skills/accessibility-infra-setup/assets/run-a11y-incomplete.cjs @@ -0,0 +1,38 @@ +#!/usr/bin/env node +// Cross-platform replacement for a bash-only pipeline +// (`--reporter=json > tmp; TEST_EXIT=$?; ...; if [ ]; exit`), which breaks on +// Windows. Runs the accessibility project, writes the JSON report straight to +// a temp file (avoids piping potentially huge stdout - test attachments like +// screenshots/videos are base64-inlined and can exceed spawnSync's stdout +// buffer), prints axe "incomplete" warnings, then exits with the test run's +// own exit code. +// Usage: node playwright/run-a11y-incomplete.cjs [specFile] +// .cjs (not .js): keeps this CommonJS regardless of the host package's "type": "module". +const { spawnSync } = require('child_process'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const { printWarnings } = require('./print-a11y-warnings.cjs'); + +const specArgs = process.argv.slice(2); +const playwrightCli = require.resolve('@playwright/test/cli'); +const reportPath = path.join(os.tmpdir(), `pw-a11y-${process.pid}.json`); + +const result = spawnSync( + process.execPath, + [playwrightCli, 'test', ...specArgs, '--project=accessibility', '--reporter=json'], + { + stdio: 'inherit', + env: { ...process.env, PLAYWRIGHT_JSON_OUTPUT_NAME: reportPath } + } +); + +try { + printWarnings(JSON.parse(fs.readFileSync(reportPath, 'utf8'))); +} catch { + console.error('Could not read Playwright JSON report; skipping accessibility warning summary.'); +} finally { + fs.rmSync(reportPath, { force: true }); +} + +process.exit(result.status ?? 1); diff --git a/skills/accessibility-infra-setup/evals/evals.json b/skills/accessibility-infra-setup/evals/evals.json new file mode 100644 index 0000000..cd2cccb --- /dev/null +++ b/skills/accessibility-infra-setup/evals/evals.json @@ -0,0 +1,133 @@ +{ + "skill_name": "accessibility-infra-setup", + "evals": [ + { + "id": "happy-path-fresh-angular-app", + "category": "happy-path", + "prompt": "Set up accessibility check infrastructure in this Angular app. The project is in evals/files/fresh-angular.", + "expected_output": "Skill detects there is no existing Playwright setup, installs @playwright/test and @axe-core/playwright, scaffolds playwright.config.ts with a dedicated accessibility project, a shared axe-helpers fixture using the WCAG 2.2 AA tag set, one dummy example.accessibility.spec.ts scanning static known-compliant HTML via page.setContent, adds test:a11y npm scripts, runs the dummy scan to green, and documents how to run it.", + "files": [ + "evals/files/fresh-angular/package.json", + "evals/files/fresh-angular/angular.json", + "evals/files/fresh-angular/tsconfig.json", + "evals/files/fresh-angular/src/index.html", + "evals/files/fresh-angular/src/main.ts", + "evals/files/fresh-angular/src/app/app.component.ts" + ], + "expectations": [ + "Detects existing test infrastructure before writing files (reads package.json)", + "Installs @playwright/test and @axe-core/playwright as devDependencies", + "Installs the Chromium browser via playwright install", + "Creates playwright.config.ts with an 'accessibility' project using a filename-anchored testMatch (e.g. /\\.accessibility\\.spec\\.ts$/, not a bare /accessibility/ substring match) on Desktop Chrome", + "Creates a shared axe fixture (axe-helpers) whose makeAxeBuilder uses wcag2a/wcag2aa/wcag21a/wcag21aa/wcag22a/wcag22aa/best-practice tags", + "axe-helpers exports an annotateIncomplete helper that reports axe 'incomplete' results as a non-blocking warning annotation instead of failing the test", + "Creates exactly one dummy accessibility spec (filename contains 'accessibility') that scans static, known-compliant HTML via page.setContent (not a real app route), calls annotateIncomplete on results.incomplete and only asserts expectNoViolations on results.violations", + "Adds test:a11y npm scripts (including test:a11y:incomplete) to package.json without clobbering existing scripts", + "Runs npm run test:a11y and confirms it passes green", + "Adds documentation describing how to run the checks and where reports are generated", + "Does NOT author real per-page accessibility tests beyond the single dummy example" + ] + }, + { + "id": "happy-path-existing-playwright", + "category": "happy-path", + "prompt": "We already use Playwright for e2e. Add axe-core accessibility scanning on top. The project is in evals/files/existing-playwright.", + "expected_output": "Skill detects the existing playwright.config, does NOT overwrite it, merges in an accessibility project and adds a filename-anchored testIgnore to functional projects, adds the axe fixture and one dummy scan, and validates.", + "files": [ + "evals/files/existing-playwright/package.json", + "evals/files/existing-playwright/playwright.config.ts", + "evals/files/existing-playwright/e2e/example.spec.ts", + "evals/files/existing-playwright/angular.json", + "evals/files/existing-playwright/tsconfig.json", + "evals/files/existing-playwright/src/index.html", + "evals/files/existing-playwright/src/main.ts", + "evals/files/existing-playwright/src/app/app.component.ts" + ], + "expectations": [ + "Detects the existing playwright.config and does not overwrite it", + "Merges an accessibility project (testMatch anchored to the filename suffix, e.g. /\\.accessibility\\.spec\\.ts$/ rather than a bare /accessibility/ substring, on Desktop Chrome) into the existing config", + "Adds a matching filename-anchored testIgnore to existing functional projects so a11y scans run once, without false-matching unrelated specs under a path segment like 'accessibility-app/'", + "Installs @axe-core/playwright but does not reinstall Playwright if already present", + "Creates the shared axe fixture (with annotateIncomplete) and one dummy accessibility spec that reports incomplete results as warnings rather than failures", + "Validates the dummy scan passes", + "Places the axe fixture and dummy accessibility spec under the existing testDir ('./e2e'), not a new default 'playwright/' directory, with import paths that resolve correctly from that location", + "Merges the reporter config to keep the existing 'html' reporter while adding the 'json' reporter with outputFile 'playwright-report/summary.json', rather than replacing 'html' outright" + ] + }, + { + "id": "regression-cypress-untouched", + "category": "regression", + "prompt": "We use Cypress for e2e. Set up accessibility checks for this Angular app. The project is in evals/files/cypress-angular.", + "expected_output": "Skill introduces Playwright + axe-core accessibility infrastructure alongside Cypress and leaves all Cypress config, specs, and dependencies completely untouched.", + "files": [ + "evals/files/cypress-angular/package.json", + "evals/files/cypress-angular/cypress.config.ts", + "evals/files/cypress-angular/cypress/e2e/home.cy.ts", + "evals/files/cypress-angular/angular.json", + "evals/files/cypress-angular/tsconfig.json", + "evals/files/cypress-angular/src/index.html", + "evals/files/cypress-angular/src/main.ts", + "evals/files/cypress-angular/src/app/app.component.ts" + ], + "expectations": [ + "Detects Cypress and does NOT edit, migrate, or delete any Cypress config, specs, or dependencies", + "Does NOT fold accessibility scans into Cypress", + "Sets up a fresh Playwright config (no existing playwright.config present) with the accessibility project", + "Cypress and Playwright coexist after the change" + ] + }, + { + "id": "regression-infra-only-scope", + "category": "regression", + "prompt": "Set up accessibility infrastructure and write full a11y coverage for every page.", + "expected_output": "Skill sets up the infrastructure plus exactly one dummy example scan, and explicitly declines to author per-page coverage, noting that is a separate follow-up task.", + "files": [], + "expectations": [ + "Only one dummy example accessibility spec is created", + "Skill explicitly states per-page/real accessibility test authoring is out of scope", + "Infrastructure (config, fixture, scripts, docs) is still fully set up", + "Skill does not attempt to fix accessibility violations in application code" + ] + }, + { + "id": "regression-noncompliant-app-still-uses-static-dummy", + "category": "regression", + "prompt": "Set up accessibility check infrastructure in this Angular app. The project is in evals/files/fresh-angular.", + "expected_output": "The dummy example scan targets static, known-compliant HTML via page.setContent rather than a real app route (e.g. '/'), so it passes green regardless of whether the app itself has pre-existing WCAG violations. The skill does not point the dummy scan at a real route or route-shop between pages to force a pass.", + "files": [ + "evals/files/fresh-angular/package.json", + "evals/files/fresh-angular/angular.json", + "evals/files/fresh-angular/tsconfig.json", + "evals/files/fresh-angular/src/index.html", + "evals/files/fresh-angular/src/main.ts", + "evals/files/fresh-angular/src/app/app.component.ts" + ], + "expectations": [ + "Dummy spec uses page.setContent with static, known-compliant HTML instead of page.goto('/') or any other real app route", + "Does NOT attempt to discover or hunt for a 'clean' real route in the app", + "npm run test:a11y passes green independent of the app's actual WCAG compliance state", + "Skill states real per-page accessibility coverage against actual app routes is a separate, out-of-scope follow-up" + ] + }, + { + "id": "regression-do-not-clobber", + "category": "regression", + "prompt": "Add accessibility checks. Our package.json already has a bunch of test scripts and a playwright config. The project is in evals/files/existing-scripts-and-playwright.", + "expected_output": "Skill preserves existing scripts and config, merging additively rather than overwriting.", + "files": [ + "evals/files/existing-scripts-and-playwright/package.json", + "evals/files/existing-scripts-and-playwright/playwright.config.ts", + "evals/files/existing-scripts-and-playwright/angular.json", + "evals/files/existing-scripts-and-playwright/tsconfig.json", + "evals/files/existing-scripts-and-playwright/src/index.html", + "evals/files/existing-scripts-and-playwright/src/main.ts", + "evals/files/existing-scripts-and-playwright/src/app/app.component.ts" + ], + "expectations": [ + "Existing npm scripts are preserved (additive edit only)", + "Existing playwright.config is merged, not replaced", + "New test:a11y scripts are added alongside existing ones" + ] + } + ] +} diff --git a/skills/accessibility-infra-setup/evals/files/cypress-angular/angular.json b/skills/accessibility-infra-setup/evals/files/cypress-angular/angular.json new file mode 100644 index 0000000..0c2913b --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/cypress-angular/angular.json @@ -0,0 +1,37 @@ +{ + "$schema": "./node_modules/@angular/cli/lib/config/schema.json", + "version": 1, + "newProjectRoot": "projects", + "projects": { + "sample-angular-app": { + "projectType": "application", + "root": "", + "sourceRoot": "src", + "prefix": "app", + "architect": { + "build": { + "builder": "@angular-devkit/build-angular:application", + "options": { + "outputPath": "dist/sample-angular-app", + "index": "src/index.html", + "browser": "src/main.ts", + "tsConfig": "tsconfig.json" + }, + "configurations": { + "production": { "optimization": true, "outputHashing": "all" }, + "development": { "optimization": false, "sourceMap": true } + }, + "defaultConfiguration": "development" + }, + "serve": { + "builder": "@angular-devkit/build-angular:dev-server", + "configurations": { + "production": { "buildTarget": "sample-angular-app:build:production" }, + "development": { "buildTarget": "sample-angular-app:build:development" } + }, + "defaultConfiguration": "development" + } + } + } + } +} diff --git a/skills/accessibility-infra-setup/evals/files/cypress-angular/cypress.config.ts b/skills/accessibility-infra-setup/evals/files/cypress-angular/cypress.config.ts new file mode 100644 index 0000000..6aab236 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/cypress-angular/cypress.config.ts @@ -0,0 +1,8 @@ +import { defineConfig } from 'cypress'; + +export default defineConfig({ + e2e: { + baseUrl: 'http://localhost:4200', + specPattern: 'cypress/e2e/**/*.cy.ts', + }, +}); diff --git a/skills/accessibility-infra-setup/evals/files/cypress-angular/cypress/e2e/home.cy.ts b/skills/accessibility-infra-setup/evals/files/cypress-angular/cypress/e2e/home.cy.ts new file mode 100644 index 0000000..c6df447 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/cypress-angular/cypress/e2e/home.cy.ts @@ -0,0 +1,6 @@ +describe('home page', () => { + it('shows the expected heading', () => { + cy.visit('/'); + cy.get('h1').should('be.visible'); + }); +}); diff --git a/skills/accessibility-infra-setup/evals/files/cypress-angular/package.json b/skills/accessibility-infra-setup/evals/files/cypress-angular/package.json new file mode 100644 index 0000000..85191e4 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/cypress-angular/package.json @@ -0,0 +1,29 @@ +{ + "name": "sample-angular-app", + "version": "0.0.0", + "scripts": { + "ng": "ng", + "start": "ng serve", + "build": "ng build", + "test": "ng test", + "cypress:open": "cypress open", + "e2e": "cypress run" + }, + "private": true, + "dependencies": { + "@angular/common": "^22.0.0", + "@angular/core": "^22.0.0", + "@angular/platform-browser": "^22.0.0", + "@angular/router": "^22.0.0", + "rxjs": "~7.8.0", + "tslib": "^2.3.0", + "zone.js": "~0.15.0" + }, + "devDependencies": { + "@angular-devkit/build-angular": "^22.0.0", + "@angular/cli": "^22.0.0", + "@angular/compiler-cli": "^22.0.0", + "cypress": "^13.14.0", + "typescript": "~6.0.0" + } +} diff --git a/skills/accessibility-infra-setup/evals/files/cypress-angular/src/app/app.component.ts b/skills/accessibility-infra-setup/evals/files/cypress-angular/src/app/app.component.ts new file mode 100644 index 0000000..1564d33 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/cypress-angular/src/app/app.component.ts @@ -0,0 +1,15 @@ +import { Component } from '@angular/core'; + +@Component({ + selector: 'app-root', + standalone: true, + template: ` +
+

Welcome to the sample Angular app

+

Accessibility infrastructure smoke test.

+
+ `, +}) +export class AppComponent { + title = 'sample-angular-app'; +} diff --git a/skills/accessibility-infra-setup/evals/files/cypress-angular/src/index.html b/skills/accessibility-infra-setup/evals/files/cypress-angular/src/index.html new file mode 100644 index 0000000..084ae92 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/cypress-angular/src/index.html @@ -0,0 +1,12 @@ + + + + + Sample Angular App + + + + + + + diff --git a/skills/accessibility-infra-setup/evals/files/cypress-angular/src/main.ts b/skills/accessibility-infra-setup/evals/files/cypress-angular/src/main.ts new file mode 100644 index 0000000..31c5da4 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/cypress-angular/src/main.ts @@ -0,0 +1,4 @@ +import { bootstrapApplication } from '@angular/platform-browser'; +import { AppComponent } from './app/app.component'; + +bootstrapApplication(AppComponent).catch((err) => console.error(err)); diff --git a/skills/accessibility-infra-setup/evals/files/cypress-angular/tsconfig.json b/skills/accessibility-infra-setup/evals/files/cypress-angular/tsconfig.json new file mode 100644 index 0000000..a83858a --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/cypress-angular/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compileOnSave": false, + "compilerOptions": { + "outDir": "./dist/out-tsc", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "isolatedModules": true, + "experimentalDecorators": true, + "moduleResolution": "bundler", + "importHelpers": true, + "target": "ES2022", + "module": "ES2022" + }, + "angularCompilerOptions": { + "strictTemplates": true + }, + "files": ["src/main.ts"] +} diff --git a/skills/accessibility-infra-setup/evals/files/existing-playwright/angular.json b/skills/accessibility-infra-setup/evals/files/existing-playwright/angular.json new file mode 100644 index 0000000..0c2913b --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-playwright/angular.json @@ -0,0 +1,37 @@ +{ + "$schema": "./node_modules/@angular/cli/lib/config/schema.json", + "version": 1, + "newProjectRoot": "projects", + "projects": { + "sample-angular-app": { + "projectType": "application", + "root": "", + "sourceRoot": "src", + "prefix": "app", + "architect": { + "build": { + "builder": "@angular-devkit/build-angular:application", + "options": { + "outputPath": "dist/sample-angular-app", + "index": "src/index.html", + "browser": "src/main.ts", + "tsConfig": "tsconfig.json" + }, + "configurations": { + "production": { "optimization": true, "outputHashing": "all" }, + "development": { "optimization": false, "sourceMap": true } + }, + "defaultConfiguration": "development" + }, + "serve": { + "builder": "@angular-devkit/build-angular:dev-server", + "configurations": { + "production": { "buildTarget": "sample-angular-app:build:production" }, + "development": { "buildTarget": "sample-angular-app:build:development" } + }, + "defaultConfiguration": "development" + } + } + } + } +} diff --git a/skills/accessibility-infra-setup/evals/files/existing-playwright/e2e/example.spec.ts b/skills/accessibility-infra-setup/evals/files/existing-playwright/e2e/example.spec.ts new file mode 100644 index 0000000..672c915 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-playwright/e2e/example.spec.ts @@ -0,0 +1,6 @@ +import { test, expect } from '@playwright/test'; + +test('home page has expected heading', async ({ page }) => { + await page.goto('/'); + await expect(page.getByRole('heading', { level: 1 })).toBeVisible(); +}); diff --git a/skills/accessibility-infra-setup/evals/files/existing-playwright/package.json b/skills/accessibility-infra-setup/evals/files/existing-playwright/package.json new file mode 100644 index 0000000..5505ca0 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-playwright/package.json @@ -0,0 +1,29 @@ +{ + "name": "sample-angular-app", + "version": "0.0.0", + "scripts": { + "ng": "ng", + "start": "ng serve", + "build": "ng build", + "test": "ng test", + "e2e": "playwright test" + }, + "private": true, + "dependencies": { + "@angular/common": "^22.0.0", + "@angular/core": "^22.0.0", + "@angular/platform-browser": "^22.0.0", + "@angular/router": "^22.0.0", + "rxjs": "~7.8.0", + "tslib": "^2.3.0", + "zone.js": "~0.15.0" + }, + "devDependencies": { + "@angular-devkit/build-angular": "^22.0.0", + "@angular/cli": "^22.0.0", + "@angular/compiler-cli": "^22.0.0", + "@playwright/test": "^1.47.0", + "@types/node": "^22.10.0", + "typescript": "~6.0.0" + } +} diff --git a/skills/accessibility-infra-setup/evals/files/existing-playwright/playwright.config.ts b/skills/accessibility-infra-setup/evals/files/existing-playwright/playwright.config.ts new file mode 100644 index 0000000..272a7dd --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-playwright/playwright.config.ts @@ -0,0 +1,22 @@ +import { defineConfig, devices } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + fullyParallel: true, + reporter: 'html', + use: { + baseURL: 'http://localhost:4200', + trace: 'on-first-retry', + }, + projects: [ + { + name: 'chromium', + use: { ...devices['Desktop Chrome'] }, + }, + ], + webServer: { + command: 'npm run start', + url: 'http://localhost:4200', + reuseExistingServer: !process.env.CI, + }, +}); diff --git a/skills/accessibility-infra-setup/evals/files/existing-playwright/src/app/app.component.ts b/skills/accessibility-infra-setup/evals/files/existing-playwright/src/app/app.component.ts new file mode 100644 index 0000000..1564d33 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-playwright/src/app/app.component.ts @@ -0,0 +1,15 @@ +import { Component } from '@angular/core'; + +@Component({ + selector: 'app-root', + standalone: true, + template: ` +
+

Welcome to the sample Angular app

+

Accessibility infrastructure smoke test.

+
+ `, +}) +export class AppComponent { + title = 'sample-angular-app'; +} diff --git a/skills/accessibility-infra-setup/evals/files/existing-playwright/src/index.html b/skills/accessibility-infra-setup/evals/files/existing-playwright/src/index.html new file mode 100644 index 0000000..084ae92 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-playwright/src/index.html @@ -0,0 +1,12 @@ + + + + + Sample Angular App + + + + + + + diff --git a/skills/accessibility-infra-setup/evals/files/existing-playwright/src/main.ts b/skills/accessibility-infra-setup/evals/files/existing-playwright/src/main.ts new file mode 100644 index 0000000..31c5da4 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-playwright/src/main.ts @@ -0,0 +1,4 @@ +import { bootstrapApplication } from '@angular/platform-browser'; +import { AppComponent } from './app/app.component'; + +bootstrapApplication(AppComponent).catch((err) => console.error(err)); diff --git a/skills/accessibility-infra-setup/evals/files/existing-playwright/tsconfig.json b/skills/accessibility-infra-setup/evals/files/existing-playwright/tsconfig.json new file mode 100644 index 0000000..a83858a --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-playwright/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compileOnSave": false, + "compilerOptions": { + "outDir": "./dist/out-tsc", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "isolatedModules": true, + "experimentalDecorators": true, + "moduleResolution": "bundler", + "importHelpers": true, + "target": "ES2022", + "module": "ES2022" + }, + "angularCompilerOptions": { + "strictTemplates": true + }, + "files": ["src/main.ts"] +} diff --git a/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/angular.json b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/angular.json new file mode 100644 index 0000000..0c2913b --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/angular.json @@ -0,0 +1,37 @@ +{ + "$schema": "./node_modules/@angular/cli/lib/config/schema.json", + "version": 1, + "newProjectRoot": "projects", + "projects": { + "sample-angular-app": { + "projectType": "application", + "root": "", + "sourceRoot": "src", + "prefix": "app", + "architect": { + "build": { + "builder": "@angular-devkit/build-angular:application", + "options": { + "outputPath": "dist/sample-angular-app", + "index": "src/index.html", + "browser": "src/main.ts", + "tsConfig": "tsconfig.json" + }, + "configurations": { + "production": { "optimization": true, "outputHashing": "all" }, + "development": { "optimization": false, "sourceMap": true } + }, + "defaultConfiguration": "development" + }, + "serve": { + "builder": "@angular-devkit/build-angular:dev-server", + "configurations": { + "production": { "buildTarget": "sample-angular-app:build:production" }, + "development": { "buildTarget": "sample-angular-app:build:development" } + }, + "defaultConfiguration": "development" + } + } + } + } +} diff --git a/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/package.json b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/package.json new file mode 100644 index 0000000..c50c3a1 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/package.json @@ -0,0 +1,33 @@ +{ + "name": "sample-angular-app", + "version": "0.0.0", + "scripts": { + "ng": "ng", + "start": "ng serve", + "build": "ng build", + "test": "ng test", + "test:unit": "ng test --watch=false", + "test:ci": "ng test --watch=false --browsers=ChromeHeadless", + "e2e": "playwright test", + "e2e:headed": "playwright test --headed", + "lint": "ng lint" + }, + "private": true, + "dependencies": { + "@angular/common": "^22.0.0", + "@angular/core": "^22.0.0", + "@angular/platform-browser": "^22.0.0", + "@angular/router": "^22.0.0", + "rxjs": "~7.8.0", + "tslib": "^2.3.0", + "zone.js": "~0.15.0" + }, + "devDependencies": { + "@angular-devkit/build-angular": "^22.0.0", + "@angular/cli": "^22.0.0", + "@angular/compiler-cli": "^22.0.0", + "@playwright/test": "^1.47.0", + "@types/node": "^22.10.0", + "typescript": "~6.0.0" + } +} diff --git a/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/playwright.config.ts b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/playwright.config.ts new file mode 100644 index 0000000..bc1881a --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/playwright.config.ts @@ -0,0 +1,26 @@ +import { defineConfig, devices } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + fullyParallel: true, + reporter: 'html', + use: { + baseURL: 'http://localhost:4200', + trace: 'on-first-retry', + }, + projects: [ + { + name: 'chromium', + use: { ...devices['Desktop Chrome'] }, + }, + { + name: 'firefox', + use: { ...devices['Desktop Firefox'] }, + }, + ], + webServer: { + command: 'npm run start', + url: 'http://localhost:4200', + reuseExistingServer: !process.env.CI, + }, +}); diff --git a/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/app/app.component.ts b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/app/app.component.ts new file mode 100644 index 0000000..1564d33 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/app/app.component.ts @@ -0,0 +1,15 @@ +import { Component } from '@angular/core'; + +@Component({ + selector: 'app-root', + standalone: true, + template: ` +
+

Welcome to the sample Angular app

+

Accessibility infrastructure smoke test.

+
+ `, +}) +export class AppComponent { + title = 'sample-angular-app'; +} diff --git a/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/index.html b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/index.html new file mode 100644 index 0000000..084ae92 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/index.html @@ -0,0 +1,12 @@ + + + + + Sample Angular App + + + + + + + diff --git a/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/main.ts b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/main.ts new file mode 100644 index 0000000..31c5da4 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/src/main.ts @@ -0,0 +1,4 @@ +import { bootstrapApplication } from '@angular/platform-browser'; +import { AppComponent } from './app/app.component'; + +bootstrapApplication(AppComponent).catch((err) => console.error(err)); diff --git a/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/tsconfig.json b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/tsconfig.json new file mode 100644 index 0000000..a83858a --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/existing-scripts-and-playwright/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compileOnSave": false, + "compilerOptions": { + "outDir": "./dist/out-tsc", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "isolatedModules": true, + "experimentalDecorators": true, + "moduleResolution": "bundler", + "importHelpers": true, + "target": "ES2022", + "module": "ES2022" + }, + "angularCompilerOptions": { + "strictTemplates": true + }, + "files": ["src/main.ts"] +} diff --git a/skills/accessibility-infra-setup/evals/files/fresh-angular/angular.json b/skills/accessibility-infra-setup/evals/files/fresh-angular/angular.json new file mode 100644 index 0000000..0c2913b --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/fresh-angular/angular.json @@ -0,0 +1,37 @@ +{ + "$schema": "./node_modules/@angular/cli/lib/config/schema.json", + "version": 1, + "newProjectRoot": "projects", + "projects": { + "sample-angular-app": { + "projectType": "application", + "root": "", + "sourceRoot": "src", + "prefix": "app", + "architect": { + "build": { + "builder": "@angular-devkit/build-angular:application", + "options": { + "outputPath": "dist/sample-angular-app", + "index": "src/index.html", + "browser": "src/main.ts", + "tsConfig": "tsconfig.json" + }, + "configurations": { + "production": { "optimization": true, "outputHashing": "all" }, + "development": { "optimization": false, "sourceMap": true } + }, + "defaultConfiguration": "development" + }, + "serve": { + "builder": "@angular-devkit/build-angular:dev-server", + "configurations": { + "production": { "buildTarget": "sample-angular-app:build:production" }, + "development": { "buildTarget": "sample-angular-app:build:development" } + }, + "defaultConfiguration": "development" + } + } + } + } +} diff --git a/skills/accessibility-infra-setup/evals/files/fresh-angular/package.json b/skills/accessibility-infra-setup/evals/files/fresh-angular/package.json new file mode 100644 index 0000000..73d7a26 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/fresh-angular/package.json @@ -0,0 +1,38 @@ +{ + "name": "sample-angular-app", + "version": "0.0.0", + "scripts": { + "ng": "ng", + "start": "ng serve", + "build": "ng build", + "watch": "ng build --watch --configuration development", + "test": "ng test" + }, + "private": true, + "dependencies": { + "@angular/animations": "^22.0.0", + "@angular/common": "^22.0.0", + "@angular/compiler": "^22.0.0", + "@angular/core": "^22.0.0", + "@angular/forms": "^22.0.0", + "@angular/platform-browser": "^22.0.0", + "@angular/platform-browser-dynamic": "^22.0.0", + "@angular/router": "^22.0.0", + "rxjs": "~7.8.0", + "tslib": "^2.3.0", + "zone.js": "~0.15.0" + }, + "devDependencies": { + "@angular-devkit/build-angular": "^22.0.0", + "@angular/cli": "^22.0.0", + "@angular/compiler-cli": "^22.0.0", + "@types/jasmine": "~5.1.0", + "jasmine-core": "~5.4.0", + "karma": "~6.4.0", + "karma-chrome-launcher": "~3.2.0", + "karma-coverage": "~2.2.0", + "karma-jasmine": "~5.1.0", + "karma-jasmine-html-reporter": "~2.1.0", + "typescript": "~6.0.0" + } +} diff --git a/skills/accessibility-infra-setup/evals/files/fresh-angular/src/app/app.component.ts b/skills/accessibility-infra-setup/evals/files/fresh-angular/src/app/app.component.ts new file mode 100644 index 0000000..1564d33 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/fresh-angular/src/app/app.component.ts @@ -0,0 +1,15 @@ +import { Component } from '@angular/core'; + +@Component({ + selector: 'app-root', + standalone: true, + template: ` +
+

Welcome to the sample Angular app

+

Accessibility infrastructure smoke test.

+
+ `, +}) +export class AppComponent { + title = 'sample-angular-app'; +} diff --git a/skills/accessibility-infra-setup/evals/files/fresh-angular/src/index.html b/skills/accessibility-infra-setup/evals/files/fresh-angular/src/index.html new file mode 100644 index 0000000..084ae92 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/fresh-angular/src/index.html @@ -0,0 +1,12 @@ + + + + + Sample Angular App + + + + + + + diff --git a/skills/accessibility-infra-setup/evals/files/fresh-angular/src/main.ts b/skills/accessibility-infra-setup/evals/files/fresh-angular/src/main.ts new file mode 100644 index 0000000..31c5da4 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/fresh-angular/src/main.ts @@ -0,0 +1,4 @@ +import { bootstrapApplication } from '@angular/platform-browser'; +import { AppComponent } from './app/app.component'; + +bootstrapApplication(AppComponent).catch((err) => console.error(err)); diff --git a/skills/accessibility-infra-setup/evals/files/fresh-angular/tsconfig.json b/skills/accessibility-infra-setup/evals/files/fresh-angular/tsconfig.json new file mode 100644 index 0000000..a83858a --- /dev/null +++ b/skills/accessibility-infra-setup/evals/files/fresh-angular/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compileOnSave": false, + "compilerOptions": { + "outDir": "./dist/out-tsc", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "isolatedModules": true, + "experimentalDecorators": true, + "moduleResolution": "bundler", + "importHelpers": true, + "target": "ES2022", + "module": "ES2022" + }, + "angularCompilerOptions": { + "strictTemplates": true + }, + "files": ["src/main.ts"] +} diff --git a/skills/accessibility-infra-setup/evals/trigger-eval.json b/skills/accessibility-infra-setup/evals/trigger-eval.json new file mode 100644 index 0000000..690dae2 --- /dev/null +++ b/skills/accessibility-infra-setup/evals/trigger-eval.json @@ -0,0 +1,50 @@ +[ + { + "query": "Set up accessibility check infrastructure in this Angular app.", + "should_trigger": true + }, + { + "query": "Add a11y testing infrastructure with Playwright and axe-core.", + "should_trigger": true + }, + { + "query": "I want to add WCAG 2.2 AA accessibility scans to our frontend.", + "should_trigger": true + }, + { + "query": "Bootstrap axe-core accessibility testing for this Angular application.", + "should_trigger": true + }, + { + "query": "We already use Playwright — add accessibility scanning on top.", + "should_trigger": true + }, + { + "query": "Set up automated accessibility checks and a sample test.", + "should_trigger": true + }, + { + "query": "Add an accessibility infrastructure setup so we can catch WCAG issues.", + "should_trigger": true + }, + { + "query": "Write a full accessibility test for the checkout page.", + "should_trigger": false + }, + { + "query": "Fix the color contrast violation on the login button.", + "should_trigger": false + }, + { + "query": "Add unit tests for the discount service.", + "should_trigger": false + }, + { + "query": "Set up a CI pipeline for our Angular app.", + "should_trigger": false + }, + { + "query": "Migrate our Cypress e2e tests to Playwright.", + "should_trigger": false + } +]