Skip to content

perf(annotations): apply batched notes from source without re-checks - #125

Open
lauridskern wants to merge 2 commits into
mainfrom
perf/annotation-edits
Open

lauridskern wants to merge 2 commits into
mainfrom
perf/annotation-edits

Conversation

@lauridskern

Copy link
Copy Markdown
Collaborator

Users reported that sending several annotations at once (for example three small color changes) took much longer than one. This PR makes the panel's message and the Mobile Dev skill lead the agent straight to the edit, and adds a benchmark to measure it.

What was slow

With the old guidance, Codex:

  • reconciled several notes on one source one note at a time. "Change only this instance" for each of three notes that together cover every instance led to per-instance code half the time.
  • added a new prop and changed the call site even for a unique element.
  • sometimes inspected the device before editing and verified each change afterwards.
  • waited on slow checks after editing.

Changes

  • Grouped notes (formatAnnotationGroups): notes from one capture on one source say whether they cover every matching element, in which case the agent makes the change once at the source, or only some of them. Sent as model context and in the standard-host draft.
  • Per-note hints: "the only matching element in the inspected screen", and a search order when there is no source location: Target text, then identifier, then owner names.
  • Guidance: go straight to Source, don't re-inspect the device to confirm bounds, make every edit, then check once.
  • Skill: a Screen annotations section. Fast Refresh applies JS and style edits, so don't rebuild. Check once with device_open → device_screenshot → device_close.
  • Telemetry: ui.annotations.send_batch counts sends with a notes bucket (1, 2-3, 4+), documented in docs/telemetry.md.
  • tooling/annotation-eval: a Codex benchmark that compares the formatter and skill at a ref with the working tree, on a fixture app or on a live simulator. See its README.

Results

Codex gpt-6.1-sol, medium effort. Every one of the 108 runs made a correct edit.

Live simulator (Expo app, plugin MCP server, 12 runs per version)

Before After
Median 35.5 s 30.5 s (−14%)
Mean 55.1 s 33.4 s (−39%)
Runs over 60 s 4 / 12 0 / 12
Slowest run 104 s 49 s

Code only (21 runs per version, 7 scenarios): median 21.7 s → 19.0 s, mean 33.4 s → 20.9 s, runs over 60 s 3 → 0. With the old prompt, three notes without a source location (scenario G) had a median of 97 s; with the new one, 29 s.

Caveats

  • Most of the drop in mean time comes from the agent no longer waiting on a slow check after editing. In these runs that check was a user-level React Doctor skill, which took about 70 s each time it ran. Leaving out React Doctor and lint waits, after is 11–13% faster at the median.
  • The new prompt also produces simpler edits: a plain style change instead of a new prop for a unique element, and one shared edit for notes that cover every instance.
  • The fixture app is small, only Codex was tested, and nothing reproduced the multi-minute runs users reported.
  • Putting the source line in each note was also tested. It brought no gain and is not included.

Testing

  • node --test packages/ui/test/*.test.ts packages/core/test/mobile-dev-skill.test.ts: 460 passing
  • node --test packages/shared/test/*.test.ts tooling/test/*.test.ts: 190 passing
  • npm run typecheck:telemetry, plus tsc --strict on the changed sources and the harness
  • New tests cover grouping (all vs some instances, different edits, no grouping across captures, device names when notes span devices), the unique-target and search-order hints, and the send_batch attribute.

Annotation edits slowed down when the agent reconciled several notes on
one source, rewrote components for unique targets, and inspected or
verified the device around every note.

- Group notes from one capture and source: say when they cover every
  matching element (edit the source once) or only some of them.
- Mark a target as the only match on screen, and give a search order
  when a note has no source location.
- Guidance: go straight to Source, do not re-inspect the device before
  editing, make every edit, then check once.
- Skill: a Screen annotations section covering Fast Refresh (no
  rebuilds) and a single screenshot check.
- Telemetry: ui.annotations.send_batch counts sends by note count
  bucket (1, 2-3, 4+).
tooling/annotation-eval runs Codex on a fixture app with the annotation
formatter and skill at a git ref against the working tree, grades each
diff, and summarizes time, tool calls and correctness. Device mode drives
an Expo app on a simulator with the plugin's MCP server.
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
mobile-dev Ready Ready Preview Nov 18, 2026 4:11pm UTC

Request Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant