BiModal Design

A design framework for building interfaces that work across the full AI agent capability spectrum — from HTTP retrievers to protocol-native agents.

---

Table of Contents

1. Overview
2. The Problem
3. The Agent Capability Spectrum
4. Defense in Depth
5. Quick Start
6. Key Concepts
7. Documentation
8. Tools & Examples
9. Maturity Levels
10. Contributing
11. Development Setup
12. Research & Citations
13. License
14. Author

---

Overview

BiModal Design is a design framework for building interfaces that remain functional and discoverable across the full spectrum of AI agent capabilities — from simple HTTP crawlers to vision agents to protocol-native AI systems.

The framework centers on two concepts:

• Agent Capability Spectrum: A six-level taxonomy of agent types, replacing the outdated binary "human vs. agent" model
• Defense in Depth: Five architectural layers ensuring graceful degradation across every agent type

As AI agents become primary users of web interfaces — for search, automation, commerce, and discovery — designing for agent accessibility is no longer optional. BiModal Design provides the principles, patterns, and tools to get there.

---

The Problem

Interfaces today face a spectrum of AI consumers, not a single one:

Agent Level	Example	What They See
Level 0 — HTTP Retrievers	curl, web scrapers	Raw HTML only
Level 1 — LLM & Agentic Browsers	Perplexity Comet	Parsed HTML, no JS
Level 2 — Browser Automation	Playwright, Agentic Chrome	Full rendered DOM
Level 3 — Vision & Computer-Use Agents	Claude Computer Use	Screenshots & AOM
Level 4 — Tool-Use Agents	OpenAI function calling	API responses
Level 5 — Protocol-Native	MCP-connected agents	Protocol data
Hybrid Agents	MCP-Universe eval targets	UI → API Handoff

A CSR-only app with <div id="root"></div> is invisible to Levels 0-1, fragile for Levels 2-3, and unreachable for Levels 4-5 without an API. Most interfaces fail at multiple levels simultaneously.

---

The Agent Capability Spectrum

BiModal Design v3.0 replaces the binary "human vs. agent" model with a graduated spectrum:

Level 0: HTTP Retrievers      → See only raw HTML (FR-1 critical)
Level 1: LLM & Agentic Browsers → Parse HTML, navigate on user's behalf
Level 2: Browser Automation   → Execute JS, agentic commerce workflows
Level 3: Vision & Computer-Use Agents → See rendered pages, query OS AOM, click UI elements
Level 4: Tool-Use Agents      → Call APIs directly via function calling
Level 5: Protocol-Native      → MCP, A2A, NLWeb — rich agent protocols

A single product page might be crawled by an HTTP retriever (Level 0), read by Perplexity (Level 1), automated by Playwright (Level 2), navigated by Claude Computer Use (Level 3), queried via API (Level 4), and accessed through MCP (Level 5) — all simultaneously.

---

Defense in Depth

Five architectural layers ensure every agent type is served:

Layer 5: Agent Protocols      (MCP, A2A, NLWeb)           → Level 5
Layer 4: API Surface           (REST, GraphQL, OpenAPI)    → Level 4-5
Layer 3: Structured Data       (schema.org, JSON-LD)      → Level 1-3
Layer 2: Semantic Structure    (HTML5, ARIA, headings)     → Level 1-3
Layer 1: Content Accessibility (FR-1: SSR/SSG)             → Level 0-1

Each layer serves a different segment of the spectrum. Together, they ensure graceful degradation — if an agent can't use Layer 5, it falls back to Layer 4, then Layer 3, and so on.

Hybrid Agents and Production Failure Modes

Modern AI workflows often rely on Hybrid Agents that traverse the UI but prefer to execute transactions via API or protocols (MCP) to avoid DOM selector drift (where brittle CSS classes break automations). A robust Layer 2 (Semantic Structure) allows agents to discover and handoff tasks to Layer 5 (Agent Protocols) seamlessly.

"Seamlessly" is the design goal, not the observed default. Under a hybrid GUI-MCP harness on OSWorld-MCP, a reasoning agent invoked a tool on only 23.9% of the tasks where a tool was reachable (arXiv:2608.03327). The handoff has to be made cheaper and more discoverable than continuing through the UI — not merely possible.

---

Key Research Findings

How to read these numbers. The ranges below are projections synthesized from public agent benchmarks (WebArena, VisualWebArena, ST-WebAgentBench, OSWorld, and others) — not results of a controlled A/B test of BiModal Design. Those benchmarks evaluate agents on the current web; they predate this framework and did not compare BiModal-compliant vs. conventional interfaces. We map their observed low and high ends to the design patterns BiModal Design codifies. Treat them as directional, not measured.

• ~12-20% baseline success for HTTP Retrievers on conventional CSR sites, in benchmarks associated with ~42-65% where Layer 1 patterns are present and ~60-75% with full Layer 1-3 patterns.
• ~35-50% baseline success for Browser Automation agents on conventional UI, associated with ~55-72% where semantic structure is present and up to ~75-88% with structured data.
• BrowseComp, VisualWebArena & MCP-Universe insights indicate that pure visual reasoning is brittle; well-structured Layer 2 and Layer 3 correlate with higher agent reliability and support reliable tool-use handoffs. MAG (2026) adds a fresh data point: evaluated on rendered screenshots rather than DOM or accessibility trees, the strongest model completes fewer than 40% of tasks.
• Semantic structure is a large lever on some agents, not a uniform one. ComponentBench (2026) instruments the middle layer between long-horizon workflows and atomic grounding tests: 97 canonical UI components instantiated as 2,910 verified tasks, each with a human reference trajectory. Within a single harness, GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only pixel control — the sharpest published measurement of what Layer 2 is worth. But the same paper states that "the benefit of structured aids is model-dependent", and for two of six models (GPT-5.4, GPT-5.4 mini) the ordering reverses, with pixel control beating Set-of-Mark. The defensible form of the Layer 2 claim is that structure removes a failure mode for the agents that consume it — not that it lifts every agent's score. Difficulty also inverts relative to humans: component families run from Command & Navigation at 91.6% down to Drag/Drop at 47.7%, and the fastest configuration still takes 3.7x as long as the matched human reference (arXiv:2608.18307).
• Where screenshot grounding actually breaks is finding the element, not understanding the instruction. GUI-Primitives (2026, EMNLP Main) holds the screenshot and anchor fixed across 994 contrastive instruction pairs and varies only the spatial relation. Nineteen vision-language models reach at most 32% strict point-in-box accuracy, and predictions land outside both candidate regions on 60-92% of items; conditional on landing inside one, target selection is 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal. Marking the two candidates raises selection accuracy by 35-57 percentage points — which the authors are careful to call "an upper bound on candidate discrimination rather than a deployable method" (arXiv:2608.21832). The paper makes no claim about accessibility trees or the DOM. What it establishes is narrower and still useful to Layer 2: the dominant failure is candidate localization, so the value of handing an agent an explicit, addressable element set is bounded well above what pixels alone deliver.
• Long-horizon orchestration remains unsolved, and reported scores may be optimistic. On WeaveBench (2026), which requires agents to combine GUI, CLI, and code operations within a single trajectory, the best frontier model-runtime pairing reaches a 41.2% PassRate. Its trajectory-aware judge also found that outcome-only grading substantially overestimates agent performance — a caution that applies to the benchmark figures the ranges above are derived from.
• Exposing a protocol surface is necessary but not sufficient — the agent has to choose it. A 2026 study of hybrid GUI-MCP agents ran one identical harness on the OSWorld-MCP benchmark (309 tasks) and found that merely making MCP tools available does not settle which way the effect goes: the same tools improved a reasoning model by +4.0pp and degraded a non-reasoning model by -5.9pp (5 runs each, both beyond 2 SE). Even the reasoning model called a tool on only 55 of 309 tasks — 23.9% of the tool-reachable ones, a shortfall the authors name the adoption gap (arXiv:2608.03327). For BiModal Design this sharpens what Layer 5 has to deliver: publishing tools is the precondition, but discoverability and an obviously cheaper tool route are what convert them into use.
• Production systems are now engineering that discoverability directly. SCOUT (2026), running in production at PayPal over a catalog of 2,000+ indexed tools, stops returning the full catalog on tools/list and instead surfaces two meta-tools — tool_search and execute_tool — so an agent retrieves only the schemas relevant to the current step. Reported effect: MCP tool-token consumption falls from 140.2k tokens (70.1% of context) to 1.3k (0.8%), a 99% reduction, with retrieval quality of Hit@1 84.8%, Hit@5 95.6%, and MRR 0.821 on the authors' own 49-query benchmark (45 evaluable) (arXiv:2608.23992). Read this as a cost-and-discoverability result, not a task-success one: the paper reports no baseline comparison against full-catalog exposure and no end-to-end agent success rate. It matters here because it is the same move the framework argues for at Layer 5 — make the tool route cheap and findable — implemented at enterprise scale rather than proposed.
• A populated accessibility tree is the floor, not the finish line. A three-week deployment study put a screen-reader-accessible computer-use agent in front of 8 blind users, collecting 1,258 real commands across 12 desktop applications; every model received both screenshots and UI trees carrying control roles and hierarchy. The strongest, GPT-5, succeeded on 52.5% (95% CI 49.8-55.3), ahead of Claude Sonnet 48.5%, Gemini 2.5 CU 43.9%, UI-TARS 39.8% and Qwen3-VL 37.9%. The failure classes the authors name are grounding, planning, constraint-tracking and termination (arXiv:2609.00524). The study does not compare agents with and without accessibility metadata, so it is not a measurement of Layer 2's benefit. What it does establish is the boundary of this framework's own claim: supplying a well-formed AOM removes a class of failure, and about half of real user commands still fail for reasons no structural layer reaches.
• The adoption gap is partly trainable, which bounds what interface design alone can fix. CUA-Universe (2026) turned 16 real desktop applications into hybrid GUI+CLI environments and post-trained a 9B model on the harvested trajectories. Against its own GUI-only operating point that model rises from 23.4% to 40.2% success on OSWorld (+16.8 pts); against the untuned base model it lifts OSWorld-MCP from 20.90% to 28.69% (+7.79 pts) while cutting steps 27% and tokens 30% (arXiv:2609.05374). The paper makes no claim about interface or website design — it is an agent-training result, recorded here because it cuts both ways for Layer 5. An agent can be taught to prefer the cheaper non-GUI route, so adoption is not solely the interface's problem; and 28.69% absolute on OSWorld-MCP says the remaining ceiling sits somewhere other than tool availability.
• The source benchmarks have themselves been shown to be gameable. BenchJack (2026) audited 10 popular agent benchmarks spanning software engineering, web navigation, desktop computing, and terminal operations, and synthesized reward-hacking exploits that achieve near-perfect scores on most of them without solving a single task — surfacing 219 distinct flaws across eight recurring classes. This is a reason to read the ranges above as directional rather than precise. It is not a reason to discard them: the same work patched the flaws it found, reducing the hackable-task ratio from near 100% to under 10% on four benchmarks and fully patching WebArena and OSWorld within three iterations (arXiv:2605.12673).

---

Quick Start

1. Test FR-1 Compliance (Layer 1)

# Check if your site exposes content in the initial HTML response
curl -s https://your-site.com | grep -E '<(main|nav|h1|article)'

If this returns semantic HTML with content — you pass Layer 1. If it returns <div id="root"></div> — content is invisible to Level 0-1 agents.

2. Run BiModal Design Validation

# Quick pass/fail FR-1 check
node tools/validators/fr1-validator.js https://your-site.com

# Comprehensive audit (structure, semantics, navigation, forms, agent features)
node tools/validators/fr1-checker.js https://your-site.com --verbose

3. Implement Core Patterns

<!-- Layer 1: Content in initial HTML (SSR/SSG) -->
<!-- Layer 2: Semantic structure with ARIA -->
<main role="main" aria-label="Product catalog">
  <h1>Wireless Headphones</h1>
  <nav role="navigation" aria-label="Main navigation">
    <a href="/products" aria-label="Browse all products">Products</a>
  </nav>
</main>

<!-- Layer 3: Structured data with schema.org -->
<script type="application/ld+json">
  {
    "@context": "https://schema.org",
    "@type": "Product",
    "name": "Wireless Headphones",
    "offers": {
      "@type": "Offer",
      "price": "99.99",
      "priceCurrency": "USD",
      "availability": "https://schema.org/InStock"
    }
  }
</script>

<!-- Layer 4: API documented via OpenAPI -->
<!-- Layer 5: MCP server discovery for protocol-native agents -->
<link rel="alternate" type="application/mcp+json" href="/mcp-server" />

---

Key Concepts

FR-1: Initial Payload Accessibility

The foundational requirement: critical content must exist in the initial HTTP response. This is Layer 1 of defense in depth — the floor, not the ceiling.

Two validation tools are included:

• fr1-validator.js — lightweight pass/fail check for FR-1 compliance (text content, semantic structure, SPA shell detection). Ideal for CI gates and quick checks.
• fr1-checker.js — comprehensive audit covering semantic content, navigation accessibility, form labels, heading hierarchy, ARIA landmarks, image alt text, and agent-specific features. Use --verbose for detailed scoring across five component categories (structure, semantics, navigation, forms, content), plus agent-feature checks.

Standards-First with Agent Attributes

Strategic Update (AOM Integration): For Web Components, BiModal Design heavily advocates using the ElementInternals API to natively define roles and ARIA states directly in the Accessibility Object Model (AOM). This removes the need for custom attributes on Custom Elements, dramatically improving discoverability for Level 2 (Browser Automation) and Level 3 (Vision & Computer-Use) agents as benchmarked by VisualWebArena. See our AOM Integration Proposal.

v3.0 uses established standards as the primary semantic layer, with data-agent-* attributes as a supplementary layer for intent and action metadata that standards don't cover:

Layer	Purpose	Example
Schema.org	Content identity and structure	itemscope itemtype="schema.org/Product"
WAI-ARIA	Accessibility and interaction	aria-label="Add to cart"
data-agent-*	Agent intent, actions, and hints	data-agent-action="add-to-cart"

Standards (schema.org, ARIA) describe what content is. Agent attributes describe what agents can do with it — actions, intents, component roles, and navigation priorities. See the API Reference for the full data-agent-* attribute specification.

Agent Protocols

BiModal Design v3.0 integrates emerging agent protocols:

• MCP: Expose tools, resources, and prompts for AI agents
• A2A: Enable agent-to-agent interoperability
• NLWeb: Support natural language queries against your data

GEO (Generative Engine Optimization)

As users discover content through AI assistants rather than search engines, BiModal Design compliance drives GEO performance. Layers 1-3 are essential for AI-assisted discoverability.

---

Documentation

Document	Description
White Paper	Framework specification v3.0
Implementation Guide	Development & deployment practices
Compliance Checklist	Layer-by-layer compliance criteria
API Reference	Tool and validator API documentation
Troubleshooting	Common errors and corrections

---

Tools & Examples

Validation Tools

• FR-1 Validator (fr1-validator.js) — quick pass/fail FR-1 compliance check (Layer 1)
• FR-1 Checker (fr1-checker.js) — comprehensive Layer 1-2 audit with detailed scoring across structure, semantics, navigation, forms, content meaning, and agent features
• Compliance Auditor — full BiModal Design compliance suite (Layers 1-3)

Implementation Examples

• Astro SSG Example — static rendering pattern (Layer 1)
• Next.js SSR Example — server rendering pattern (Layer 1)
• CSR Mitigation — client-rendered fallback strategies
• Hybrid Agent MCP Handoff — Layer 2 to Layer 5 transaction handoff
• AOM Web Component Example — Using ElementInternals to expose native semantics to agents

---

Maturity Levels

Maturity levels describe site compliance, not agent types. A Maturity Level 4 site implements all five defense-in-depth layers and serves all six agent capability levels (Levels 0-5).

Level	Name	Layers	Agent Coverage	Success Rate
0	Infrastructure Ready	Layer 1	Level 0-1	40-65%
1	Semantically Accessible	Layers 1-2	Level 0-2	55-75%
2	Data-Rich	Layers 1-3	Level 0-3	65-85%
3	API-Enabled	Layers 1-4	Level 0-4	80-92%
4	Agent-Native	Layers 1-5	All levels	90-98%

---

Contributing

Contributions are welcome!

1. Fork the repository
2. Create a feature branch
3. Commit with a Conventional Commit message
4. Submit a Pull Request

Refer to the Contributing Guidelines for review standards and code style.

---

Development Setup

# Clone the repository
git clone https://github.com/jgoldfoot/BiModalDesign.git
cd BiModalDesign

# Install dependencies (package.json is at the repo root)
npm install

# Run tests
npm test

---

Research & Citations

• WebAgents Survey 2025 — "A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation" (arXiv:2503.23350v4)

• MCP-Universe — "A comprehensive framework and benchmark for evaluating AI agents and LLMs through direct interaction with real-world Model Context Protocol servers"

• WebArena-Verified — Rigorous re-evaluation of autonomous web agents

• WorkArena++ — "Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks" (NeurIPS 2024)

• OSWorld — "Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments" (arXiv:2404.07972). Superseded by OSWorld 2.0 as of 2026-06-26.

• OSWorld 2.0 — "Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks" — Yuan, Zhou, Xiong et al., XLANG Lab (arXiv:2606.29537, June 2026). 108 long-horizon workflows; best frontier agent completes 20.6% at a 500-step budget.

• ST-WebAgentBench — "A Benchmark for Evaluating Safety and Trustworthiness in Web Agents" — Levy et al., IBM Research (arXiv:2410.06703, 2024; ICLR 2026)

• MCP — Model Context Protocol (modelcontextprotocol.io). Revision 2026-07-28, released 28 July 2026: stateless protocol core, mandatory server/discover RPC, Multi Round-Trip Requests, cacheable list results, Extensions framework (Tasks, MCP Apps), authorization hardening, formal deprecation policy.

• A2A — Agent-to-Agent Protocol. Launched by Google April 2025; contributed to the Linux Foundation, which announced it as the project's new home on 23 June 2025. Specification v1.0.0 released 12 March 2026 under Linux Foundation governance (latest patch v1.0.1, 28 May 2026)

• NLWeb — Natural Language Web Protocol (Microsoft, 2025)

• τ-bench — "A Benchmark for Tool-Agent-User Interaction in Real-World Domains" (arXiv:2406.12045)

• WebVoyager — Benchmarking end-to-end browser agents on live real-world websites

• Odysseys — "Benchmarking Web Agents on Realistic Long Horizon Tasks" — Jang, Koh, Fried, Salakhutdinov (arXiv:2604.24964, April 2026). 200 long-horizon live-web tasks; strongest evaluated model 44.5%.

• VisualWebArena — Evaluating Multimodal Agents on Realistic Visual Web Tasks

• WeaveBench — "A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces" — 114 tasks across 8 work domains requiring GUI, CLI, and code operations in one trajectory; best PassRate 41.2% (arXiv:2606.09426, 2026)

• MAG — "A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation" — screenshot-grounded evaluation via Set-of-Mark and pixel coordinates; strongest model completes fewer than 40% of tasks (arXiv:2607.10079, 2026)

• GUI-Primitives — "Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding" — Jahin, Parvez (arXiv:2608.21832, v1 22 Aug 2026, v2 27 Aug 2026; accepted to EMNLP 2026 Main Conference). 994 contrastive instruction pairs over seven spatial relations; 19 vision-language models reach at most 32% strict point-in-box accuracy, predictions fall outside both candidates on 60-92% of items, and marking the candidates raises selection accuracy by 35-57pp as an upper bound rather than a deployable method.

• Screenshots or Tools? — "Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents" — Fan, Li, Ma, Tan, Huang, Wu, Zhang, Shang, Chen (arXiv:2608.03327, v1 4 Aug 2026, v2 6 Aug 2026). One GUI-MCP harness on OSWorld-MCP (309 tasks): identical MCP tools move a reasoning model +4.0pp and a non-reasoning model -5.9pp; the reasoning model calls a tool on 55/309 tasks (23.9% of tool-reachable ones) — the adoption gap.

• SCOUT — "Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation" — Saha, Wang, Manoharan (arXiv:2608.23992, 25 Aug 2026). Selective Context Optimization for Universal Tooling: two MCP meta-tools (tool_search, execute_tool) replace full-catalog tools/list exposure, using BM25 + dense retrieval fused by Reciprocal Rank Fusion. In production at PayPal over 2,000+ indexed tools, MCP tool-token consumption drops from 140.2k (70.1% of context) to 1.3k (0.8%); Hit@1 84.8%, Hit@5 95.6%, MRR 0.821 on a 49-query internal benchmark. No baseline comparison and no end-to-end task-success measurement is reported.

• BenchJack — "Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack" — Wang et al., UC Berkeley (arXiv:2605.12673, May 2026). An automated red-teaming system for benchmark designers; audited 10 agent benchmarks, surfaced 219 distinct flaws across eight classes, and fully patched WebArena and OSWorld within three iterations.

• BrowseComp — Benchmark for Agentic Browser Navigation & Task Execution

• OpAgent — "Operator Agent for Web Navigation" achieving 71.6% on WebArena (arXiv:2602.13559)

• Operator — OpenAI's Computer-Using Agent (87% on WebVoyager, 58.1% on WebArena, 38.1% on OSWorld)

• Project Mariner — Google's agent featuring "Teach & Repeat" capabilities, achieving 84.0% on ScreenSpot and 83.5% on WebVoyager (Google, 2025)

• ScreenSpot — Benchmark for spatial and visual understanding in GUIs

• UFO² — "The Desktop AgentOS" featuring hybrid control detection that fuses Windows UI Automation with vision (Microsoft, 2025)

• WebMCP — Browser API letting a page register tools for in-browser agents via document.modelContext. W3C Web Machine Learning Community Group, Draft Community Group Report — a continuously updated editor's draft, so it is cited by retrieval date rather than by the date in its header (retrieved 7 September 2026; header then read 4 September 2026). Not a W3C Standard; not on the Standards Track. The ModelContext interface gained a specified executeTool() method on 14 August 2026, revised on 17 August 2026 to take a structured object rather than a JSON string; RegisteredTool.inputSchema was retyped from DOMString to object in the same window. On 19 August 2026 the draft specified AbortSignal integration for tool execution and preserved in-flight executions after unregistration (PRs #247, #248) — the cancellation and teardown semantics a page needs before exposing a long-running tool to an agent. On 3 September 2026 ToolAnnotations gained consequentialHint — "If true, indicates that executing the tool will result in consequential actions that are significant, real-world, or non-reversible, ex: booking a flight, transferring money" — joining readOnlyHint and untrustedContentHint. Implementation status as re-checked on 7 September 2026 (per the spec repository's own implementation-status.md, which is the source for these): origin trials live in Chrome 149 and Edge 150; experimental support in Brave's Leo AI chat; ChatGPT Desktop listed as supporting WebMCP — the first non-browser client on the page, added 26 August 2026 in a pull request from an OpenAI-affiliated author and merged by a spec editor. Firefox and Safari have standards-positions entries only.

• ComponentBench — "Diagnosing Component-Level Failures in Computer-Use Agents" — Guan, Lin, Cheng-Yue, Wang, Zhou (arXiv:2608.18307, 18 August 2026). 97 canonical UI components as 2,910 programmatically verified tasks with human reference trajectories. GPT-5 mini scores 83.1% on accessibility-tree observations vs 48.9% on coordinate-only pixel control; the paper reports the benefit of structured aids as model-dependent, reversing for GPT-5.4 and GPT-5.4 mini.

• MCP Roadmap — Model Context Protocol blog, 22 August 2026. Names a progressive discovery effort so "a server can offer a small entry point and reveal more of its catalog as the conversation narrows", plus a single standardized contract for tools/call result forms. Both target the discoverability half of the adoption gap. Tasks moved into an official extension (SEP-2663). https://blog.modelcontextprotocol.io/posts/mcp-roadmap/

• Computer-Use Agents for Blind Users — "Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications" — Kodandaram, Padma Reddy, Bi, Zhou, Ramakrishnan, Ashok (arXiv:2609.00524, 1 September 2026). Three-week diary study, 8 blind screen-reader users, 1,258 commands across 12 desktop applications, all models given screenshots and UI trees. GPT-5 52.5% (95% CI 49.8-55.3), Claude Sonnet 48.5%, Gemini 2.5 CU 43.9%, UI-TARS 39.8%, Qwen3-VL 37.9%. No with/without accessibility-metadata comparison is made.

• CUA-Universe — "A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents" — Shi, Wang, Fang, Liang, Jin, Zhao, Liu, Chen, Wang (arXiv:2609.05374, 4 September 2026). Environment-to-data pipeline over 16 desktop applications; a post-trained 9B model reaches 40.2% on OSWorld against its own GUI-only 23.4%, and 28.69% on OSWorld-MCP against the untuned base model's 20.90%. An agent-training result; makes no interface-design claim.

• Code execution with MCP — Building more efficient agents (Anthropic, Nov 2025)

---

License

Licensed under the Apache License 2.0. See LICENSE for full details.

---

Author

Joel Goldfoot Design Leader | AI + Human-Agent Interaction Researcher

joel@goldfoot.com linkedin.com/in/joelgoldfoot bimodal.design

---

BiModal Design — Designing for the full agent capability spectrum.