Skip to content

Add more androidDevice actions to the mobile-2 sample #2964

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
jebrans wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add more androidDevice actions to the mobile-2 sample #2964

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions android/samples/mobile-2/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,11 @@ interaction traffic.
| `composeSms` | `Intent.ACTION_SENDTO` with an `smsto:` URI and `sms_body` | Opens a pre-filled draft — the user still presses send, so no `SEND_SMS` permission is needed. With no recipient the draft opens with an empty To field; an *unusable* recipient is rejected rather than silently dropped. |
| `webSearch` | `Intent.ACTION_WEB_SEARCH` with `SearchManager.QUERY` | The query travels as an extra rather than being spliced into a URL, so it needs no encoding. |
| `openWebPage` | `Intent.ACTION_VIEW` with an `http`/`https` URI | The scheme allowlist is the load-bearing check: `ACTION_VIEW` would otherwise follow `market:`, `file:` or any app's own deep-link scheme, turning "open this page" into an arbitrary-app launcher driven by text the model read. URLs containing whitespace are refused rather than repaired into a different host. |
| `composeEmail` | `Intent.ACTION_SENDTO` with a bare `mailto:` URI plus `EXTRA_EMAIL`/`EXTRA_CC`/`EXTRA_BCC` | Opens a draft — the user still presses send, so nothing leaves the device unattended. `ACTION_SENDTO` is used rather than `ACTION_SEND` so only mail apps resolve, not every share target. Recipients ride as extras rather than being spliced into the URI, which keeps encoding out of the picture. One unusable address fails the whole action: a draft addressed to fewer people than asked for looks like success. |
| `shareText` | `Intent.ACTION_SEND` (`text/plain`) wrapped in `Intent.createChooser` | The user picks the destination app, so the model never chooses where the text goes. `createChooser` always resolves, so the inner `ACTION_SEND` intent is resolved first — otherwise a device with no text handler would report a false success. Newlines survive here (unlike the URI-bound actions) because shared text is a document, not a query. |
| `openSettings` | `Settings.ACTION_*` for a fixed screen | The model picks from a closed `AndroidSettingsScreen` enum, never a raw action string, so it cannot be steered into an arbitrary system activity. Screens only *display* settings; nothing is toggled. `appInfo` is pinned to this app's own package. |
| `createCalendarEvent` | `Intent.ACTION_INSERT` on `CalendarContract.Events.CONTENT_URI` | Opens the calendar app's pre-filled *new event* editor — the user still saves it, so no `WRITE_CALENDAR` permission is needed. Times are ISO-8601 and are resolved without `java.time` (minSdk 24). All-day events are anchored at UTC midnight as `CalendarContract` requires; an all-day event that also carries a time of day is rejected rather than guessed at. Spans longer than 366 days are refused. |
| `playMusicFromSearch` | `MediaStore.INTENT_ACTION_MEDIA_PLAY_FROM_SEARCH` | Asks whichever music app claims the intent to play the best match. What actually plays is up to that app, so the action reports what it dispatched, not what started. An unrecognised `focus` fails rather than falling back to `any`, which would quietly search for something broader than asked. |

All actions require the app to be in the foreground: Android 10+ silently refuses
background activity starts (no exception is thrown), so the app checks its own
Expand Down
71 changes: 71 additions & 0 deletions android/samples/mobile-2/app/src/main/AndroidManifest.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,77 @@
<action android:name="android.intent.action.VIEW" />
<data android:scheme="https" />
</intent>
<intent>
<action android:name="android.intent.action.SENDTO" />
<data android:scheme="mailto" />
</intent>
<!--
Only needed so shareText can tell an empty share sheet apart from a
working one: Intent.createChooser always resolves, because the chooser
itself is a system activity, so the inner SEND intent is resolved
first and this entry is what makes that check meaningful.
-->
<intent>
<action android:name="android.intent.action.SEND" />
<data android:mimeType="text/plain" />
</intent>
<!--
One entry per member of AndroidSettingsScreen. The enum is the
allowlist; these entries only make resolveActivity work for it.
-->
<intent>
<action android:name="android.settings.SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.WIFI_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.BLUETOOTH_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.DISPLAY_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.SOUND_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.LOCATION_SOURCE_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.BATTERY_SAVER_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.AIRPLANE_MODE_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.DATE_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.INTERNAL_STORAGE_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.ACCESSIBILITY_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.SECURITY_SETTINGS" />
</intent>
<intent>
<action android:name="android.settings.APPLICATION_DETAILS_SETTINGS" />
<data android:scheme="package" />
</intent>
<!--
The calendar insert intent is matched on the MIME type behind
CalendarContract.Events.CONTENT_URI, so the provider has to be visible
as well or resolveActivity cannot read that type back.
-->
<intent>
<action android:name="android.intent.action.INSERT" />
<data android:mimeType="vnd.android.cursor.dir/event" />
</intent>
<provider android:authorities="com.android.calendar" />
<intent>
<action android:name="android.media.action.MEDIA_PLAY_FROM_SEARCH" />
</intent>
</queries>

<uses-permission android:name="android.permission.INTERNET" />
Expand Down
125 changes: 124 additions & 1 deletion android/samples/mobile-2/app/src/main/assets/typeagent/androidDeviceSchema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,12 @@ export type AndroidDeviceAction =
| DialPhoneNumberAction
| ComposeSmsAction
| WebSearchAction
| OpenWebPageAction;
| OpenWebPageAction
| ComposeEmailAction
| ShareTextAction
| OpenSettingsAction
| CreateCalendarEventAction
| PlayMusicFromSearchAction;

// A day of the week an alarm can repeat on.
export type AlarmRepeatDay =
Expand Down Expand Up @@ -140,3 +145,121 @@ export type OpenWebPageAction = {
url: string;
};
};

// Opens the Android device's email app on a pre-filled draft message.
// The user still has to press send, so this never sends mail by itself.
// Use when the user asks to email someone.
export type ComposeEmailAction = {
actionName: "composeEmail";
parameters: {
// The original user request.
originalRequest: string;
// Recipient email addresses. Omit when the user did not name anyone;
// the email app then opens with an empty To field.
to?: string[];
// Carbon-copy recipients.
cc?: string[];
// Blind carbon-copy recipients.
bcc?: string[];
// The subject line.
subject?: string;
// The message body.
body?: string;
};
};

// Offers a piece of text to the Android share sheet so the user can pass it to
// another app. Use only when the user explicitly asks to share, send or post
// some text they have named; the user picks the destination app and confirms.
export type ShareTextAction = {
actionName: "shareText";
parameters: {
// The original user request.
originalRequest: string;
// The exact text to share.
text: string;
// An optional title or subject offered to apps that can use one.
subject?: string;
};
};

// A settings screen this device agent is allowed to open.
export type AndroidSettingsScreen =
| "settings"
| "wifi"
| "bluetooth"
| "display"
| "sound"
| "location"
| "battery"
| "airplaneMode"
| "dateAndTime"
| "storage"
| "accessibility"
| "security"
| "appInfo";

// Opens one of the Android device's settings screens.
// Use when the user asks to change a device setting - the app cannot toggle
// settings directly, so it takes the user to the right screen instead.
export type OpenSettingsAction = {
actionName: "openSettings";
parameters: {
// The original user request.
originalRequest: string;
// Which settings screen to open. Only these screens are supported;
// "settings" is the top-level settings app and "appInfo" is this chat
// app's own details page.
screen: AndroidSettingsScreen;
};
};

// Opens the Android device's calendar app on a pre-filled new event.
// The user still has to save it, so this never writes to a calendar by itself.
// Use when the user asks to create, schedule or add a calendar event.
export type CreateCalendarEventAction = {
actionName: "createCalendarEvent";
parameters: {
// The original user request.
originalRequest: string;
// The event title.
title: string;
// Local start time as ISO-8601 without a time zone, for example
// "2026-08-24T15:00". For an all-day event use a date only,
// "2026-08-24". Times are interpreted in the device's own time zone,
// so never convert to UTC and never append "Z" or an offset.
start: string;
// Local end time in the same format as start. Omit for a one hour
// event, or a single day when allDay is true.
end?: string;
// True when the event covers whole days rather than a time of day.
allDay?: boolean;
// Where the event takes place.
location?: string;
// Longer notes about the event.
description?: string;
};
};

// What a music search query names.
export type MusicSearchFocus =
| "any"
| "artist"
| "album"
| "song"
| "playlist";

// Asks the Android device's music app to play something matching a search.
// Use when the user asks to play music, an artist, an album or a song.
export type PlayMusicFromSearchAction = {
actionName: "playMusicFromSearch";
parameters: {
// The original user request.
originalRequest: string;
// What to search for, for example "Kind of Blue" or "Miles Davis".
query: string;
// What the query names. Omit or use "any" when it is unclear, which
// lets the music app decide.
focus?: MusicSearchFocus;
};
};
62 changes: 60 additions & 2 deletions android/samples/mobile-2/app/src/main/java/com/example/typeagentchat/AndroidDeviceAgent.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ internal object AndroidDeviceAgent {
private const val AGENT_DESCRIPTION =
"Acts on this Android device: sets alarms and countdown timers, shows the " +
"alarm and timer lists, searches for nearby places, shows a place on the " +
"map, opens the dialer or a text message draft, runs a web search and " +
"opens web pages."
"map, opens the dialer or a text message draft, runs a web search, " +
"opens web pages, drafts email, shares text with another app, opens " +
"device settings screens, drafts calendar events and plays music."

fun createRegistrationParams(
conversationId: String,
Expand Down Expand Up @@ -143,6 +144,56 @@ internal object AndroidDeviceAgent {
AndroidDeviceActionParseResult.Success(AndroidDeviceAction.OpenWebPage(parsed))
}

"composeEmail" -> {
val parsed = parseComposeEmailActionPayload(parameters)
?: return AndroidDeviceActionParseResult.ActionError(
"Invalid composeEmail parameters: every recipient must be a " +
"valid email address and the draft cannot be empty."
)
AndroidDeviceActionParseResult.Success(AndroidDeviceAction.ComposeEmail(parsed))
}

"shareText" -> {
val parsed = parseShareTextActionPayload(parameters)
?: return AndroidDeviceActionParseResult.ActionError(
"Invalid shareText parameters: text is required."
)
AndroidDeviceActionParseResult.Success(AndroidDeviceAction.ShareText(parsed))
}

"openSettings" -> {
val parsed = parseOpenSettingsActionPayload(parameters)
?: return AndroidDeviceActionParseResult.ActionError(
"Invalid openSettings parameters: screen must be one of " +
AndroidSettingsScreen.entries.joinToString { it.schemaName } + "."
)
AndroidDeviceActionParseResult.Success(AndroidDeviceAction.OpenSettings(parsed))
}

"createCalendarEvent" -> {
val parsed = parseCreateCalendarEventActionPayload(parameters)
?: return AndroidDeviceActionParseResult.ActionError(
"Invalid createCalendarEvent parameters: title is required and " +
"start/end must be local ISO-8601 values such as " +
"2026-08-24T15:00, with end after start."
)
AndroidDeviceActionParseResult.Success(
AndroidDeviceAction.CreateCalendarEvent(parsed)
)
}

"playMusicFromSearch" -> {
val parsed = parsePlayMusicFromSearchActionPayload(parameters)
?: return AndroidDeviceActionParseResult.ActionError(
"Invalid playMusicFromSearch parameters: query is required and " +
"focus must be one of " +
MusicSearchFocus.entries.joinToString { it.schemaName } + "."
)
AndroidDeviceActionParseResult.Success(
AndroidDeviceAction.PlayMusicFromSearch(parsed)
)
}

else -> AndroidDeviceActionParseResult.ActionError(
"Unsupported Android agent action: $actionName"
)
Expand Down Expand Up @@ -175,6 +226,13 @@ internal sealed interface AndroidDeviceAction {
data class ComposeSms(val action: ComposeSmsAction) : AndroidDeviceAction
data class WebSearch(val action: WebSearchAction) : AndroidDeviceAction
data class OpenWebPage(val action: OpenWebPageAction) : AndroidDeviceAction
data class ComposeEmail(val action: ComposeEmailAction) : AndroidDeviceAction
data class ShareText(val action: ShareTextAction) : AndroidDeviceAction
data class OpenSettings(val action: OpenSettingsAction) : AndroidDeviceAction
data class CreateCalendarEvent(val action: CreateCalendarEventAction) : AndroidDeviceAction
data class PlayMusicFromSearch(
val action: PlayMusicFromSearchAction
) : AndroidDeviceAction
}

internal sealed interface AndroidDeviceActionParseResult {
Expand Down
66 changes: 66 additions & 0 deletions android/samples/mobile-2/app/src/main/java/com/example/typeagentchat/ChatViewModel.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,31 @@ internal sealed interface ClientAction {
val action: OpenWebPageAction,
val completion: (AndroidDeviceExecutionResult) -> Unit
) : ClientAction

data class ComposeEmail(
val action: ComposeEmailAction,
val completion: (AndroidDeviceExecutionResult) -> Unit
) : ClientAction

data class ShareText(
val action: ShareTextAction,
val completion: (AndroidDeviceExecutionResult) -> Unit
) : ClientAction

data class OpenSettings(
val action: OpenSettingsAction,
val completion: (AndroidDeviceExecutionResult) -> Unit
) : ClientAction

data class CreateCalendarEvent(
val action: CreateCalendarEventAction,
val completion: (AndroidDeviceExecutionResult) -> Unit
) : ClientAction

data class PlayMusicFromSearch(
val action: PlayMusicFromSearchAction,
val completion: (AndroidDeviceExecutionResult) -> Unit
) : ClientAction
}

/**
Expand Down Expand Up @@ -207,6 +232,47 @@ class ChatViewModel(application: Application) : AndroidViewModel(application) {
) {
dispatchClientAction(ClientAction.OpenWebPage(action, completion), completion)
}

override fun onComposeEmail(
action: ComposeEmailAction,
completion: (AndroidDeviceExecutionResult) -> Unit
) {
dispatchClientAction(ClientAction.ComposeEmail(action, completion), completion)
}

override fun onShareText(
action: ShareTextAction,
completion: (AndroidDeviceExecutionResult) -> Unit
) {
dispatchClientAction(ClientAction.ShareText(action, completion), completion)
}

override fun onOpenSettings(
action: OpenSettingsAction,
completion: (AndroidDeviceExecutionResult) -> Unit
) {
dispatchClientAction(ClientAction.OpenSettings(action, completion), completion)
}

override fun onCreateCalendarEvent(
action: CreateCalendarEventAction,
completion: (AndroidDeviceExecutionResult) -> Unit
) {
dispatchClientAction(
ClientAction.CreateCalendarEvent(action, completion),
completion
)
}

override fun onPlayMusicFromSearch(
action: PlayMusicFromSearchAction,
completion: (AndroidDeviceExecutionResult) -> Unit
) {
dispatchClientAction(
ClientAction.PlayMusicFromSearch(action, completion),
completion
)
}
})

webSocketManager.setStaleConversationHandler {
Expand Down
Loading