fix(webAgent): keep the aria snapshot fresh on client-rendered pages (SPA readiness + stale-ref recovery)

[lmorchard]

Two related fixes for the same underlying problem: on client-rendered (SPA) pages, the agent can end up acting on a stale or empty aria snapshot. Every action tool is ref-only — a [ref=…] minted by the snapshot, with no coordinate/vision fallback — so if the snapshot the model sees doesn't match the live DOM, the agent is stranded.

Fix 1 — retry an empty initial snapshot (SPA readiness guard)

Problem: On a client-rendered page (or one mid client-side redirect) the captured aria snapshot can land on the pre-hydration shell: a tree with no actionable elements (e.g. generic > img, img). With ref-only actions, an empty interactive tree strands the agent — it sees the form in the screenshot but has no ref to act on, and aborts. Observed on a Mattermost login page ("the accessibility tree is incomplete… I cannot proceed without valid element references").

Fix: In addPageSnapshot(), if the captured tree has no interactive refs, wait briefly and re-capture — bounded to 3 tries, 1s apart — before handing the snapshot to the model. Hypothesis-agnostic (pre-hydration, pre-redirect, briefly-hidden) and a no-op on any page that already has actionable elements. hasInteractiveRefs keys off the standard ARIA interactive roles that carry a ref.

Fix 2 — refresh the snapshot after a stale-ref tool error (recovery deadlock)

Problem: On a recoverable Invalid element reference error, the loop returned needsPageSnapshot: false, so addPageSnapshot was never called — the model stayed on a known-bad snapshot. Meanwhile the recovery guidance in prompts.ts tells the model "wait for the next page snapshot (it will arrive automatically)." It never arrived, so the model confabulated refs off a frozen view (asserting "the tree now shows E44, which differs from previous snapshots" — when no new snapshot was ever sent) until the consecutive-error budget killed the run. A loop-vs-prompt contract mismatch, general to any recoverable ref error on a client-rendered page.

Where Fix 1 handles an empty initial snapshot, Fix 2 handles mid-flow ref invalidation after the DOM mutates.

Fix: Force a snapshot refresh on the retry only for stale/invalid-ref errors (scoped via shouldRefreshSnapshotAfterError, keyed on the same phrase prompts.ts uses); other recoverable errors keep prior behavior. The existing consecutive-error budget bounds any refresh loop.

Validation (Zoo Magento guest-checkout, 11 reps each): baseline had 2/11 confabulation-deadlock failures; with the fix, 0/11, plus a follow-up 11/11 confirmation run. Invalid-ref errors are occasional — ~3% of actions, roughly one per checkout — but pre-fix a single one was deterministically fatal: no refresh → the model confabulated refs off the frozen snapshot → 5 consecutive errors → dead. The fix stops that cascade at the first error.

Tests

packages/core/test/webAgent.test.ts — 82 passing. hasInteractiveRefs covered against the exact sparse shell that stranded the agent vs. a hydrated form (+ link, ref-less role, name-vs-role false-positive, yaml-quoted key). shouldRefreshSnapshotAfterError covered for InvalidRefException, the ToolExecutionError carrying the invalid-ref message, the alternate phrasing, an unrelated recoverable error (no refresh), and non-Error inputs.

Provenance & confidence (honest)

Both were surfaced by an experimental eval-forensics pipeline (Zoo-benchmark failure → source-code join). Fix 1's specific trigger is a rare tail (the login task usually passes within the existing 1000ms settle) — treat it as a low-risk robustness guard. Fix 2's mechanism is verified in event logs (post-fix, the snapshot refreshes on every stale-ref error instead of freezing), and the confabulation-deadlock failure mode was eliminated in a small same-task comparison (2/11 → 0/11, N=11) plus an 11/11 confirmation on a different task. That is evidence the fix works; it is not a demonstrated benchmark-score gain (see the eval-scope note in the comments) — and on a Q&A-heavy set like WebVoyagerX we wouldn't expect one. It's the higher-impact of the two changes.

🤖 Generated with Claude Code

[lmorchard]

Addressed the Copilot review in 7446077:

1. Name-vs-role false positive — INTERACTIVE_REF_RE is now anchored to the node's role at line-start (^[ \t]*- '?<role>…), so an interactive word appearing inside a non-interactive node's accessible name (e.g. - generic "button" [ref=E5]) no longer matches. The optional leading ' keeps it matching yaml-quoted keys.
2. Retry count wording — kept 3 retries (the constant is …MAX_RETRIES), and clarified the comment to state "initial capture + up to MAX_RETRIES recaptures" so code and description agree.
3. Regression test — added coverage for the name-vs-role false positive (- generic "button" …, - text "click the link …") and for a yaml-quoted interactive key. Also fixed the test mock snapshots to use the real aria-YAML shape so the readiness guard doesn't spuriously retry against fake HTML fixtures.

Full webAgent.test.ts suite: 77 passing.

[lmorchard]

Folded in a second, related fix (257ea70): refresh the page snapshot after a stale-ref tool error.

Where the readiness guard (this PR's first fix) handles an empty initial snapshot, this handles mid-flow ref invalidation. On a recoverable Invalid element reference error the loop returned needsPageSnapshot: false, so addPageSnapshot was never called — the model stayed on a known-bad snapshot. Meanwhile the recovery guidance in prompts.ts tells the model "wait for the next page snapshot (it will arrive automatically)" — it never did, so the model confabulated refs off a frozen view until the consecutive-error budget ended the run.

The fix forces a refresh on the retry only for stale/invalid-ref errors (scoped via shouldRefreshSnapshotAfterError, keyed on the same phrase prompts.ts uses); other recoverable errors keep prior behavior, and the existing consecutive-error budget bounds any refresh loop.

Validation (Zoo Magento guest-checkout, 11 reps each): baseline had 2/11 confabulation-deadlock failures; with the fix, 0/11 — and reps that threw 7–15 invalid-ref errors now recover and pass instead of dying at 5 consecutive. Full webAgent.test.ts: 82 passing (added 5 tests for the new helper).

[lmorchard]

Going to give this a run through evals and see if things improve. 🤞 If so, this might fix a long-standing hard-to-fix bug.

[lmorchard]

Correction + eval context (post-validation)

Number correction: an earlier comment (and the original description) said reps threw "7–15 invalid-ref errors." That was a measurement artifact — I'd counted occurrences of the phrase "Invalid element reference" across all events, but that phrase lives in the recovery prompt (prompts.ts), so it appears in every generation message. Counting only genuine failed actions (browser:action_completed, success:false), the real rate is ~3% of actions — roughly one stale ref per checkout (9 across 11 confirmation reps).

This makes the fix's value clearer, not weaker: the trigger is occasional, but pre-fix a single stale ref was deterministically fatal — no refresh → the model confabulated refs off the frozen snapshot → 5 consecutive errors → dead. The before/after stands: 2/11 ref-deadlock failures → 0/11, plus an 11/11 confirmation run (same task, an in-stock product). The description has been updated.

On the partial WebVoyagerX eval (run pilo-batch-github-eval-xj9r6, 20/30): please don't read this as a verdict on these fixes. All 10 failures are unrelated — 3 browser disconnects (infra), 3 webSearch "insufficient credit" (a search-API billing issue, since fixed), 2 Gemini tool-protocol errors ("exactly one tool"), and 2 stale past-date Booking tasks (dataset). Zero implicate this change. WebVoyagerX is also Q&A/lookup-heavy, so it barely exercises the multi-step form-interaction path this fix targets — the real validation is the Zoo checkout above. A retry with the webSearch credit restored is running.

Follow-up: #636 tracks the more general element-ref robustness work (browser-layer self-heal on stale refs, nearest-valid-ref hints, etc.) — this PR is the targeted recovery fix; #636 is the broader direction.

[lmorchard]

Retry eval landed — fix confirmed firing, not implicated

The webSearch-credit-restored retry (pilo-batch-github-eval-667rj, same build 257ea70) came in at 23/30 (77%), up from 20/30. The +10pp is the webSearch fix + transient browser-disconnects clearing — none of it attributable to these code changes, and importantly no failure implicates this fix.

The one stale_element_refs failure (Booking--0) I dug into specifically since it wears the label this PR targets: its event log shows the refresh is working (fresh snapshots appended every iteration, 18→23 — no frozen-snapshot confabulation), but it couldn't recover on the ~1,984-ref Mexico results page where refs churn faster than the model can act, on an also-unsatisfiable past-date task. That's the fix's boundary (heavy high-churn pages), tracked as the residual case in #636 — not a regression here. The satisfiable-page case is fully covered (11/11 confirmation run above).

Remaining failures are dataset fragility (stale Booking dates + a stale Apple reference — pilo-evals-judge#99) and transient infra, not agent-quality regressions.

[lmorchard]

Eval-scope note (calibrating the "validated" claim honestly)

To be precise about what the evals do and don't support:

We cannot claim this fix improved aggregate eval performance. Both WebVoyagerX partial runs (67% → 77%) are on the same post-fix build (257ea70) — there is no pre-fix WebVoyagerX baseline, and the delta is webSearch-credit + transient-infra, not this change. WebVoyagerX is also Q&A/lookup-heavy, so it barely exercises the multi-step form path this fix targets; a proper before/after there would likely show ~nothing either way.

What the evidence does support (narrower, and solid):

• Mechanism, verified in event logs — pre-fix, a stale ref froze the snapshot and the model confabulated refs until the error budget killed it; post-fix, the snapshot refreshes on every stale-ref error (18→23 in the Booking-0 trace; recovery from real stale refs in the confirmation run). The deadlock signature is gone.
• Failure-mode count, small same-task comparison — the confabulation-deadlock occurred 2/11 in a pre-fix baseline (yoga-mat task) and 0/11 after.

Caveats on that count (why it's supporting evidence, not proof):

• N=11 is underpowered — at the ~18% base rate, you'd see 0 failures ~13% of the time by chance alone.
• The clean same-task A/B used the unconditional flip; the shipped scoped fix's 11/11 was on a different task (product swapped to remove availability noise). So there's no single adequately-powered, same-task, same-(shipped)-build A/B.
• The target failure is rare (~1/checkout) and stochastic, so its effect on any aggregate score is small and easily swamped by run-to-run variance.

So: mechanism verified + a specific failure mode eliminated, not "raised the benchmark by X." A real score-delta measurement would need an interaction-heavy set (the Zoo mutation suite, not WebVoyagerX), larger N, and a clean same-task pre/post of the shipped build — and would still expect a small delta, because the fix's value is converting a silent confabulation-inducing failure into clean recovery, which a pass/fail score barely captures.