Skip to content

docs(help): add accessibility review help topic - #4651

Merged
thymikee merged 6 commits into
mainfrom
docs/help-accessibility-review
Nov 18, 2026
Merged

thymikee merged 6 commits into
mainfrom
docs/help-accessibility-review

Conversation

@thymikee

@thymikee thymikee commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Agents had no single workflow for a whole-app accessibility review. This PR adds agent-device help accessibility. The topic gives a per-screen audit loop, whole-app coverage rules, and checks that audit cannot do (large text, dark mode, screen reader order and labels, gesture-only actions). It states the scope: audit findings are advisory, and no compliance verdict is given.

  • Routing: skills/agent-device/SKILL.md, the agent-setup rule, the root help topic list, and help dogfood Platform checks point to the topic.
  • Refs: audit's pinned refs stay valid only until the next snapshot. The topic and help dogfood tell the agent to act through the following snapshot -i refs, found by role and position (rects from audit --json and snapshot -i --json). Element-name matching waits until audit prints names (feat(audit): name findings and remove platform chrome noise #4652).
  • Bench: new accessibility-review help-conformance case (audit run, two or more audits, settings text-size, relaunch after the size change). The task states the starting settings, so a static plan can name the size to restore.
  • Touched files: 7. No change to audit output or the audit command description.

Validation

  • Tested SHA: 6a4ec0ee4f3f58a7afc6eb0b73265ce186f631bc.
  • pnpm check:affected --run: all runnable checks passed (exactCommit=true).
  • vitest run src/commands/schema: 116 passed.
  • pnpm bench:help-conformance --case accessibility-review --runner claude:sonnet --repeat 3: 3/3 trials, 8/8 checks each.
  • Docs and help text only. No runtime behavior changes, so no device run applies.
Rendered agent-device help accessibility
agent-device <version> — accessibility

Use this when asked for an accessibility review, a11y audit, or a screen-reader or touch-target check of an app.

Scope:
  Audit findings are advisory heuristics. Audit checks touch-target size, missing accessible names, and covered controls.
  This tool does not measure color contrast or focus order, and does not give a compliance verdict. Do not claim compliance.

Loop for each screen:
  1. Open the app: agent-device open <app>. Run mkdir -p ./a11y-output once.
  2. Run agent-device audit, then agent-device snapshot -i. Audit captures the screen itself, so it goes first.
  3. Record each finding with its kind, role, and size. Audit's pinned refs (@eN~sG) stay valid only until the next snapshot.
     To act on a finding, use the ref from that snapshot -i. The ref numbers can differ, so do not match by ref number.
     Find the element in the snapshot by role and position. To compare positions, use the rect from agent-device audit --json and from agent-device snapshot -i --json.
  4. Capture evidence: agent-device screenshot ./a11y-output/<screen>.png --overlay-refs

Coverage:
  Visit every tab, pushed screen, sheet, dialog, and empty, error, and loading state.
  Audit covers the visible screen only. On a long list, run agent-device scroll down 0.8, then run audit again.

Checks audit cannot do:
  Large text: first run agent-device settings text-size to record the current size. Then run agent-device settings text-size accessibility-extra-extra-extra-large, then agent-device open <app> --relaunch. The app adopts the size only after a relaunch.
    Then run audit and screenshot each key screen. Look for clipped, truncated, or overlapping text.
    At the end, set the recorded size again. For example, if the recorded size was large, run agent-device settings text-size large.
  Dark mode: first note the current mode from a screenshot. Then run agent-device settings appearance dark, then screenshot each key screen. Look for low-contrast text and icons by eye.
    At the end, set the noted mode again. For example, if the noted mode was light, run agent-device settings appearance light.
  If settings refuses on this device, such as a physical iPhone, list large text and dark mode as not done.
  Screen reader order and labels: read agent-device snapshot (without -i) from top to bottom. The order must match the visual order.
    A label must be specific and unique: "Save Lemon Herb Rice", not "Save". A control must expose its state (selected, checked, disabled).
  Gesture-only actions: every swipe or long-press action needs another way to reach it, such as a visible control or an accessibility action.
    On the iOS simulator, agent-device snapshot -i --actions names the custom actions that a merged element hides.

Fix loop (only when the user asks to fix):
  Change the app source, then reload (Fast Refresh, or agent-device open <app> --relaunch).
  Run audit again on the same screen and confirm the finding is gone. Report the before and after audit output.

Report:
  Group findings by screen. For each finding, give the kind, element name, ref, screenshot path, and suggested fix.
  List the checks you could not do. Include audit's limitation lines in that list.

Related:
  agent-device help <command>   command-specific flags
  agent-device help manual-qa   routine QA loop
  agent-device help workflow    full automation reference

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff Turn on auto-fix

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.38 MB 6.39 MB +3.5 kB
Package (unpacked) 6.38 MB 6.39 MB +3.5 kB
Package (download) 1.94 MB 1.94 MB +1.1 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 30.7 ms 24.3 ms -6.4 ms
CLI --help 47.9 ms 51.7 ms +3.7 ms

@apex-sentinel-staging

This comment has been minimized.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 3 new findings on b089cbc. The verdict and the full list are in the summary comment.

Comment thread scripts/help-conformance-cases.mjs Resolved
Comment thread skills/agent-device/SKILL.md Resolved
@apex-sentinel-staging

This comment has been minimized.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 922e495. The verdict and the full list are in the summary comment.

Comment thread scripts/help-conformance-cases.mjs Resolved
@apex-sentinel-staging

This comment has been minimized.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the topic. At 3046e9d I found one problem that should be fixed before merge. All 21 checks pass, and there are no conflicts.

Step 2 of the loop runs agent-device audit, then agent-device snapshot -i. Step 3 then says to act on the pinned refs (@eN~sG) that audit prints, or on refs from that snapshot (cli-help-workflows.ts#L215). Audit issues a partial ref frame at generation G. The snapshot -i that follows bumps the generation and activates a new frame, so a press or longpress on an audit pin is refused with ref_generation_mismatch. Only refs from the snapshot work. An agent that follows the loop in order and presses an audit pin gets a typed refusal. Also, audit prints no element name, so the name must come from the snapshot, matched by ref number. Please tell step 3 to take element names and action refs from the snapshot -i that follows the audit, matching each finding by ref number, and to treat audit's pinned refs as valid only until the next snapshot. The same sentence is in help dogfood, Platform checks, on main. Please fix both in this change and grep once so the two spellings change together.

Not blocking: you can take or leave these. On a physical iPhone settings text-size and settings appearance are refused, so the topic could add "If settings refuses on this device, such as a physical iPhone, list large text and dark mode as not done." The gesture-only line could name agent-device snapshot -i --actions, and Report could say to include audit's Limitation lines under the checks not done. The large-text and dark-mode sentences carry several actions each, which matches neighbouring topics, so splitting them is optional. The PR body still names tested SHA b089cbc with a 3/3 bench run, says 8 touched files (the diff has 7), and shows the first-draft rendered help. Later commits changed the help text and matchers, so please update the body, and re-run pnpm bench:help-conformance --case accessibility-review --runner claude:sonnet --repeat 3 at head if you want the bench claim to hold.

The four earlier inline threads are fixed at this head and can be resolved: the relaunch-order regex (thread), the SKILL.md routing to help accessibility (thread), the recorded and restored size and mode (thread), and the matcher comment now agreeing with the two-audit test (thread).

Before merge, the step 3 wording needs to change, together with the identical sentence in help dogfood. Then please refresh the validation section of the PR body.

@apex-sentinel-staging

This comment has been minimized.

@thymikee

Copy link
Copy Markdown
Member Author

Fixed at df32cc7.

  • Step 3 and help dogfood Platform checks now say: audit's pinned refs stay valid only until the next snapshot. To act on a finding, use the ref from the following snapshot -i. I grepped once, and both places changed together.
  • I changed one part of the request. The finding is matched by element name, not by ref number. Audit captures the full tree, and snapshot -i numbers refs after the interactive filter, so the same element can get different numbers. feat(audit): name findings and remove platform chrome noise #4652 adds the element name to each audit finding.
  • Taken from the optional list: a refusal line for physical iPhones, agent-device snapshot -i --actions for gesture-only actions, and audit's limitation lines in the report.
  • The restore steps now give concrete values (settings text-size large, settings appearance light) as examples. Before, Sonnet copied the <recorded-size> placeholder into its plan in 3 of 3 trials. The bench task now states the starting settings, so a static plan can name the size to restore.
  • The bench at head scores 3/3 on claude:sonnet, with 8/8 checks in each trial. check:affected passed with exactCommit=true. I updated the PR body.

@apex-sentinel-staging

This comment has been minimized.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the update. Both earlier inline threads are fixed at df32cc7, but one problem from the earlier review remains: the help text promises element names that audit does not return yet.

Step 3 in cli-help-workflows.ts:215 tells the agent to record each finding with its element name and to match the finding to the snapshot by element name. The dogfood platform checks (line 263) say the same. On this head, audit prints only <ref> <kind> <size or 'Button has no accessible name'> (audit-output.ts:44-70), and the finding contract has no name field (contracts/audit.ts:15-35). A missing-name finding therefore has no name anywhere. The name only arrives with #4652. An agent that follows the loop has nothing to record, and the ref numbers can differ after the interactive filter, so it has to guess which element each finding means. The evidence screenshot and the ref in the Report step can then point at the wrong element. Please either land #4652 first and rebase, or reword both sites together to what audit returns today: record kind, role and size, then find the element in the next snapshot -i by role and position (audit --json gives the rect, and snapshot nodes carry a rect). Please do not describe element-name matching until audit returns the name.

Not blocking: the new line at cli-help-workflows.ts:233 says snapshot -i --actions lists the accessibility actions of each element, but the flag only names custom actions an element merged away, works on the iOS simulator only, and costs one accessibility round trip per merged element (flag-definitions-workflow.ts:265-274), so you could scope it to the iOS simulator or drop it, and you can take or leave this.

The two inline threads at #4651 (comment) (text-size ordering before open --relaunch) and #4651 (comment) (SKILL.md routing to help accessibility) are fixed at df32cc7, so please resolve them.

CI is still running on this head and nothing has failed so far. The diff is help strings, tests, a bench script, SKILL.md and a website doc, with no runtime code. The help-conformance bench and check:affected results come from the PR description. Once the wording above is fixed and CI is green, this is ready to merge.

@thymikee

Copy link
Copy Markdown
Member Author

Fixed at 6a4ec0e.

  • Step 3 and the dogfood Platform checks now use only what audit returns today. The agent records the kind, role, and size of each finding. The agent finds the element in the next snapshot -i by role and position, with the rects from audit --json and snapshot -i --json. The help no longer describes matching by element name. I will change that wording after feat(audit): name findings and remove platform chrome noise #4652 lands.
  • The --actions line is now scoped: "On the iOS simulator, agent-device snapshot -i --actions names the custom actions that a merged element hides."
  • The two inline threads were already resolved.
  • Bench at head: claude:sonnet 3/3, 8/8 checks each. check:affected passed with exactCommit=true. The PR body shows the current rendered help.

@apex-sentinel-staging

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve · review of 6a4ec0e

Since the review of df32cc7: 0 new · 0 still open · 0 no longer raised

No open findings.

Not in the reviewers' prompt (7 files)
  • scripts/help-conformance-cases.mjs: no reviewer was selected for this change
  • skills/agent-device/SKILL.md: no reviewer was selected for this change
  • src/commands/schema/cli-help-first-screen.test.ts: no reviewer was selected for this change
  • src/commands/schema/cli-help-overview.ts: no reviewer was selected for this change
  • src/commands/schema/cli-help-workflows.test.ts: no reviewer was selected for this change
  • src/commands/schema/cli-help-workflows.ts: no reviewer was selected for this change
  • website/docs/docs/agent-setup.md: no reviewer was selected for this change

Reviewed with this repository's own Sentinel configuration (aa600e98051c).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at 6a4ec0e. The earlier findings from #4651 (comment) are fixed: step 3 and the dogfood Platform checks no longer match by name, and --actions is now scoped to the iOS simulator, which matches runner-interaction-facts.ts:95.

Not blocking, and you can take or leave it: step 2 runs plain agent-device audit, but the text rows omit role for small-target rows and rect for all rows, while step 3 asks the agent to record role and compare rects "from agent-device audit --json" (cli-help-workflows.ts:215). If you edit this text again, write agent-device audit --json in step 2 and say the rows carry role and rect. #4652 will rework this wording anyway.

The four inline threads from other reviewers are resolved and none are still open. No device run is needed for help text. The claim that audit rects and snapshot -i rects are comparable rests on contracts/audit.ts:20 and the snapshot help.

The Smoke Tests job failed in the Android smoke checks on assertAutomationSystem at test/integration/android-emulator-e2e/live-automation-scenario.ts:50. This PR changes only help text, the topic registry, a conformance case file and docs, and that scenario does not read any of them, so the failure looks like an emulator flake or an existing main issue. No other job fails. Please re-run Smoke Tests before merge. There are no conflicts.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 18, 2026
@thymikee
thymikee merged commit af752bc into main Nov 18, 2026
17 of 18 checks passed
@thymikee
thymikee deleted the docs/help-accessibility-review branch November 18, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant