Feat: introduce lux models for agent

[gwynbleidd003]

why

• this PR introduces openagi models (lux-actor-1 and lux-thinker-1) for agent

what changed

• new agent client OpenAGICUAClient.ts
• new models for agent provider
• update cua-example.ts to use mode: cua to avoid ERROR: [DEPRECATED] The "cua: true" option is deprecated. Use "mode: 'cua'" instead. This option will be removed in a future version.

test plan

• update cua-example.ts: change the model name to openagi/lux-actor-1, use an api key from openagi
• pnpm run example cua-example

---

Summary by cubic

Adds OpenAGI computer-use agent support with lux-actor-1 and lux-thinker-1, enabling tasks to run via OpenAGI. Updates agent provider mapping, public types, and the example to use mode: "cua".

• New Features

  • Added OpenAGICUAClient to run steps with @oagi/oagi and translate actions (click, drag, type, scroll, hotkey, wait).
  • AgentProvider now supports provider "openagi" for models "lux-actor-1" and "lux-thinker-1".
  • Updated public types: AgentType includes "openagi"; AVAILABLE_CUA_MODELS and model provider list extended.
  • Example and tests updated to include OpenAGI models.

• Migration

  • Replace deprecated cua: true with mode: "cua" in agent config.
  • To use OpenAGI models, set an API key (OPENAGI_API_KEY or clientOptions.apiKey) and use modelName "openagi/lux-actor-1" or "openagi/lux-thinker-1". Provide a screenshot via setScreenshotProvider if your runtime doesn’t supply one.

^{Written for commit efdc88e. Summary will update automatically on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: efdc88e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

No issues found across 8 files

Architecture diagram

sequenceDiagram
    participant User as "Client Code"
    participant Factory as "AgentProvider"
    participant NewClient as "OpenAGICUAClient"
    participant OAGI as "@oagi/oagi SDK"
    participant API as "OpenAGI API"
    participant Browser as "Browser/Page"

    Note over User,Factory: Initialization Phase
    
    User->>Factory: stagehand.agent({ mode: 'cua', model: 'openagi/lux-actor-1' })
    
    Note right of Factory: CHANGED: Switch on 'openagi' provider
    alt Provider == 'openagi'
        Factory->>NewClient: NEW: instantiate(modelName, options)
        NewClient-->>Factory: client instance
    else Provider == 'anthropic' / 'openai' / etc
        Factory->>Factory: Instantiate existing clients
    end
    
    Factory-->>User: agent instance

    Note over User,Browser: Execution Phase (Computer Use Loop)

    User->>NewClient: act('Book a flight...')
    
    loop Action Cycle
        NewClient->>Browser: Capture State (Screenshot/DOM)
        Browser-->>NewClient: state data
        
        NewClient->>OAGI: NEW: client.step(state, instructions)
        OAGI->>API: Network Request (lux-actor-1)
        API-->>OAGI: Agent Actions (Click, Type, Scroll, etc.)
        OAGI-->>NewClient: Response Object
        
        Note over NewClient: NEW: Action Translation Layer
        
        alt Has Pending Actions
            NewClient->>NewClient: Map OpenAGI action to Stagehand/Playwright
            NewClient->>Browser: Execute Action (e.g., click(x,y))
        else Task Complete
            NewClient-->>User: Return Result
        end
    end

Loading

[greptile-apps]

Greptile Summary

• Adds OpenAGI as a new Computer Use Agent provider supporting lux-actor-1 and lux-thinker-1 models for browser automation
• Implements OpenAGICUAClient.ts with coordinate denormalization, action conversion logic, and integration with the @oagi/oagi package
• Updates deprecated cua: true configuration to mode: "cua" in the example to follow current API standards

Important Files Changed

Filename	Overview
packages/core/lib/v3/agent/OpenAGICUAClient.ts	New agent client implementing OpenAGI CUA integration with coordinate conversion and action mapping logic
packages/core/lib/v3/agent/AgentProvider.ts	Added OpenAGI provider mapping and client instantiation logic for lux models

Confidence score: 4/5

• This PR adds new functionality with well-structured type definitions and follows existing architectural patterns
• Score reduced due to potential edge cases in coordinate normalization logic, hardcoded scroll distances, and regex-based action parsing in the OpenAGI client
• Pay close attention to OpenAGICUAClient.ts for coordinate conversion accuracy and action parsing reliability

Sequence Diagram

sequenceDiagram
    participant User
    participant Stagehand
    participant Agent
    participant OpenAGICUAClient
    participant OpenAGIAPI
    participant Browser

    User->>Stagehand: "new Stagehand({ mode: 'cua', model: 'openagi/lux-actor-1' })"
    Stagehand->>Agent: "stagehand.agent(config)"
    Agent->>OpenAGICUAClient: "new OpenAGICUAClient(modelName, clientOptions)"
    
    User->>Agent: "agent.execute(instruction)"
    Agent->>OpenAGICUAClient: "execute(executionOptions)"
    
    loop "For each step (up to maxSteps)"
        OpenAGICUAClient->>Browser: "captureScreenshot()"
        Browser-->>OpenAGICUAClient: "base64 screenshot"
        OpenAGICUAClient->>OpenAGIAPI: "actor.step(screenshot)"
        OpenAGIAPI-->>OpenAGICUAClient: "step result with actions"
        OpenAGICUAClient->>OpenAGICUAClient: "convertAction(step, action)"
        OpenAGICUAClient->>Browser: "execute converted action"
    end
    
    OpenAGICUAClient-->>Agent: "AgentResult"
    Agent-->>User: "execution result"

Loading

[greptile-apps]

Additional Comments (1)

1. packages/core/lib/v3/agent/AgentProvider.ts, line 97-103 (link)

   logic: The OpenAGI client constructor call is missing the tools parameter that other CUA clients receive (see lines 72, 80, and 88)

_{7 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}