Skip to content

feat: enable UI actions from AI chat - #9881

Open
djbarnwal wants to merge 2 commits into
mainfrom
feat/ui-action-ai
Open

feat: enable UI actions from AI chat#9881
djbarnwal wants to merge 2 commits into
mainfrom
feat/ui-action-ai

Conversation

@djbarnwal

Copy link
Copy Markdown
Member

Adds an opt-in mechanism that allows Rill’s AI agents to trigger approved UI actions in the browser.

Adding capability to support ejection of Rill templates to vega lite spec. (#9850)

UI elements can expose themselves to the agent using a stable data-ai-action identifier. Available actions are sent as transient context with each chat request, and the agent can invoke the new click_ui tool to request an action.

Implementation

  • Added the click_ui backend tool for developer and analyst agents.
  • Added UIContext and UIAction to the completion API.
  • Discover visible, enabled, uniquely identified [data-ai-action] elements when sending a chat message.
  • Revalidate actions immediately before clicking them in the browser.
  • Added initial actions for:
    • Editing a canvas
    • Starting a dashboard pivot
  • Added frontend and backend tests for action discovery, validation, and execution.

Safety

UI actions are explicitly opt-in. The agent cannot use arbitrary CSS selectors or click arbitrary page elements.

An action is available only when it:

  • Has a non-empty data-ai-action identifier.
  • Is visible and enabled.
  • Has a unique identifier on the current page.
  • Is outside a data-ai-ignore subtree.
  • Is included in the current request’s UI context.

The action is checked again in the browser before the click is performed.

image

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

@djbarnwal
djbarnwal requested review from AdityaHegde and begelundmuller and removed request for begelundmuller September 11, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant