Repository navigation
Add hybrid function calling sample - #2855
VinayGuthal wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new Hybrid Function Calling feature, adding the necessary ViewModel, UI screen, state representation, and navigation routing to support testing function calling tools across on-device, hybrid, and cloud inference modes. The feedback highlights opportunities to improve robustness and consistency, specifically by using safe casting to prevent potential exceptions during JSON parsing, ensuring the streaming generation path handles automatic function calling consistently with the unary path, and avoiding redundant model downloads in the UI when the selected option has not changed.
| if (toolOption == FunctionToolOption.MANUAL && detectedCalls.isNotEmpty()) { | ||
| val firstCall = detectedCalls.first() | ||
| val city = (firstCall.args["city"] as? JsonPrimitive)?.content ?: "Boston" | ||
| val state = (firstCall.args["state"] as? JsonPrimitive)?.content ?: "MA" | ||
| val date = (firstCall.args["date"] as? JsonPrimitive)?.content ?: "2024-10-17" | ||
| val weatherJson = WeatherRepository.fetchWeather(city, state, date) | ||
| executionResult = weatherJson.toString() | ||
|
|
||
| val followUp = model.generateContent( | ||
| content(role = "user") { text(promptText) }, | ||
| content(role = "model") { part(firstCall) }, | ||
| content(role = "user") { | ||
| part(FunctionResponsePart(firstCall.name, weatherJson, firstCall.id)) | ||
| } | ||
| ) | ||
| fullText = followUp.text.orEmpty() | ||
| source = followUp.inferenceSource | ||
| } |
There was a problem hiding this comment.
In the STREAMING generation method, there is no handling for FunctionToolOption.AUTO to capture and set the executionResult when a function call is detected. This causes an inconsistency with the UNARY method where the execution result is captured and displayed in the UI. Adding an else if block for AUTO will resolve this.
if (toolOption == FunctionToolOption.MANUAL && detectedCalls.isNotEmpty()) {
val firstCall = detectedCalls.first()
val city = (firstCall.args["city"] as? JsonPrimitive)?.content ?: "Boston"
val state = (firstCall.args["state"] as? JsonPrimitive)?.content ?: "MA"
val date = (firstCall.args["date"] as? JsonPrimitive)?.content ?: "2024-10-17"
val weatherJson = WeatherRepository.fetchWeather(city, state, date)
executionResult = weatherJson.toString()
val followUp = model.generateContent(
content(role = "user") { text(promptText) },
content(role = "model") { part(firstCall) },
content(role = "user") {
part(FunctionResponsePart(firstCall.name, weatherJson, firstCall.id))
}
)
fullText = followUp.text.orEmpty()
source = followUp.inferenceSource
} else if (toolOption == FunctionToolOption.AUTO && detectedCalls.isNotEmpty()) {
val firstCall = detectedCalls.first()
val city = (firstCall.args["city"] as? JsonPrimitive)?.content ?: "Boston"
val state = (firstCall.args["state"] as? JsonPrimitive)?.content ?: "MA"
val date = (firstCall.args["date"] as? JsonPrimitive)?.content ?: "2024-10-17"
executionResult = WeatherRepository.fetchWeather(city, state, date).toString()
}| onClick = { | ||
| selectedModelOption = modelOpt | ||
| viewModel.checkAndDownloadModel(modelOpt.option) | ||
| }, |
There was a problem hiding this comment.
To avoid redundant network requests and job cancellations when the user taps an already selected model option, add a check to ensure checkAndDownloadModel is only called when the selection actually changes.
onClick = {
if (selectedModelOption != modelOpt) {
selectedModelOption = modelOpt
viewModel.checkAndDownloadModel(modelOpt.option)
}
},Tested at firebase/quickstart-android#2855 ### Description Adds support for function calling when generating content using on-device and hybrid (`PREFER_ON_DEVICE`) inference modes, following the ML Kit GenAI Prompt APIs. When function calling tools are supplied, `OnDeviceGenerativeModelProvider` verifies that the underlying on-device model supports function calling via `isFunctionCallingAvailable()`. If function calling is not supported by the on-device model or if unsupported non-function tools (such as Google Search or Code Execution) are configured, a `FirebaseAIOnDeviceNotAvailableException` is raised, triggering automatic fallback to cloud inference in `PREFER_ON_DEVICE` mode. ### Summary of Changes - **`gradle/libs.versions.toml`**: - Updated `genaiPrompt` to `1.0.0-beta5` and `genaiSchema` to `1.0.0-alpha2`. - **`ai-logic/firebase-ai-ondevice-interop`**: - Added `FunctionDeclaration` and `FunctionSchema` interop data structures. - Added `FunctionCallPart` and `FunctionResponsePart` to `Part.kt`. - Added `functionCalls: List<FunctionCallPart>` to `Candidate`. - Added `tools: List<FunctionDeclaration>?` to `GenerateContentRequest`. - Added `suspend fun isFunctionCallingAvailable(): Boolean` to `GenerativeModel`. - Updated `api.txt` and `CHANGELOG.md`. - **`ai-logic/firebase-ai-ondevice`**: - Implemented `GenerativeModelImpl.isFunctionCallingAvailable()` delegating to ML Kit's `getAvailableFeatures().contains(Feature.FUNCTION_CALLING)`. - Added converters in `Converters.kt` for `FunctionCallPart`, `FunctionResponsePart`, `FunctionDeclaration`, and `FunctionSchema` to/from ML Kit equivalents. - Mapped candidate function calls in `GenerateContentResponse.toInterop()`. - Updated `api.txt` and `CHANGELOG.md`. - Added unit test coverage in `GenerativeModelImplTest` and `ConvertersTest`. - **`ai-logic/firebase-ai`**: - Added `tools: List<Tool>? = null` to `OnDeviceConfig` to allow configuring tools for on-device inference, defaulting to model-level tools when omitted. - Forwarded tools configuration in `GenerativeModel` to `OnDeviceGenerativeModelProvider`. - Updated `Candidate.fromInterop` to convert interop function calls into `FunctionCallPart` instances. - In `OnDeviceGenerativeModelProvider`: - Checks `onDeviceModel.isFunctionCallingAvailable()` when function declarations are provided. - Throws `FirebaseAIOnDeviceNotAvailableException` if non-function tools are passed to on-device inference or if function calling is unsupported. - Converts `FunctionCallPart` and `FunctionResponsePart` in conversation history. - Maps function declarations and auto-function declarations into interop schemas. - Updated `api.txt` and `CHANGELOG.md`. - Added unit tests in `OnDeviceGenerativeModelProviderTests` covering tool execution, candidate mapping, multi-turn history with function responses, streaming support, and cloud fallback behavior. ### Testing - `./gradlew :ai-logic:firebase-ai-ondevice-interop:check` - `./gradlew :ai-logic:firebase-ai-ondevice:check` - `./gradlew :ai-logic:firebase-ai:check` - `./gradlew spotlessCheck` --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…i/feature/hybrid/FunctionCallingViewModel.kt Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Description
Adds a new sample screen to test function calling tools across on-device, hybrid (
PREFER_ON_DEVICE), and cloud inference modes. This sample complements the function calling support added in firebase-android-sdk #8758.Features
FunctionDeclaration): DefinesfetchWeather(city, state, date), inspects returnedFunctionCallPartentries, executes the local tool, and sendsFunctionResponsePartto resolve the multi-turn exchange.AutoFunctionDeclaration): Validates automatic tool invocation handled directly by the SDK.Tool.googleSearch()): Demonstrates validation behavior and automatic cloud fallback when non-function tools are used with hybrid inference.ONLY_ON_DEVICE,PREFER_ON_DEVICE,ONLY_IN_CLOUD).generateContent) and streaming (generateContentStream) generation methods.ON_DEVICEvsIN_CLOUD).Summary of Changes
gradle/libs.versions.toml:firebase-aito18.2.0andfirebase-ai-ondeviceto16.0.0-beta08.firebase-ai/app:FunctionCallingViewModel.kt,FunctionCallingUiState.kt, andFunctionCallingScreen.ktundercom.google.firebase.quickstart.ai.FUNCTION_CALLINGinScreenType(Sample.kt).HYBRIDandTOOLS_FCcategories inFirebaseAISamples.kt.MainActivity.kt.Testing
./gradlew :app:assembleDebug./gradlew ktlintCheck