-
Notifications
You must be signed in to change notification settings - Fork 34
docs: document Vertex AI dependencies for the agent-server image #595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
9181693
793b108
cf39b7c
93966ca
55286b0
2da15fa
c25971b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,7 +3,7 @@ title: Google Gemini/Vertex | |
| description: OpenHands uses LiteLLM to make calls to Google's chat models. You can find their documentation on using Google as a provider -> [Gemini - Google AI Studio](https://docs.litellm.ai/docs/providers/gemini), [VertexAI - Google Cloud Platform](https://docs.litellm.ai/docs/providers/vertex) | ||
| --- | ||
|
|
||
| ## Gemini - Google AI Studio Configs | ||
| ## Gemini - Google AI Studio Configuration | ||
|
|
||
| When running OpenHands, you'll need to set the following in the OpenHands UI through the Settings under the `LLM` tab: | ||
| - `LLM Provider` to `Gemini` | ||
|
|
@@ -12,7 +12,7 @@ If the model is not in the list, enable `Advanced` options, and enter it in `Cus | |
| (e.g. gemini/<model-name> like `gemini/gemini-2.0-flash`). | ||
| - `API Key` to your Gemini API key | ||
|
|
||
| ## VertexAI - Google Cloud Platform Configs | ||
| ## VertexAI - Google Cloud Platform Configuration | ||
|
|
||
| To use Vertex AI through Google Cloud Platform when running OpenHands, you'll need to set the following environment | ||
| variables using `-e` in the docker run command: | ||
|
|
@@ -28,3 +28,82 @@ Then set the following in the OpenHands UI through the Settings under the `LLM` | |
| - `LLM Model` to the model you will be using. | ||
| If the model is not in the list, enable `Advanced` options, and enter it in `Custom Model` | ||
| (e.g. vertex_ai/<model-name>). | ||
|
|
||
| ### Vertex AI Dependencies | ||
|
|
||
| The `vertex_ai/*` models (including Gemini and Claude via Vertex AI) require the | ||
| `google-cloud-aiplatform` package, which is **not included by default** in the published | ||
| agent-server image. How you enable it depends on your deployment: | ||
|
|
||
| <Note> | ||
| Unlike AWS Bedrock (whose `boto3` dependency is bundled by default), Vertex AI support is | ||
| opt-in. If you skip this step, you will see a <code>ModuleNotFoundError: No module named | ||
| 'vertexai'</code> error when the agent tries to call a Vertex AI model. | ||
| </Note> | ||
|
|
||
| #### Local / Non-Docker Install | ||
|
|
||
| Install the `vertex` extra in your Python environment: | ||
|
|
||
| ```bash | ||
| pip install "openhands-sdk[vertex]" | ||
|
rajshah4 marked this conversation as resolved.
|
||
| # or, with uv (works in any Python environment): | ||
| uv pip install "openhands-sdk[vertex]" | ||
| ``` | ||
|
|
||
| #### Custom Agent-Server Image | ||
|
|
||
| Build the image with the `ENABLE_VERTEX` build flag (the container build file is in the | ||
| [`software-agent-sdk` repo](https://github.com/OpenHands/software-agent-sdk/blob/main/openhands-agent-server/openhands/agent_server/docker/Dockerfile); | ||
| run from the repo root): | ||
|
|
||
| ```bash | ||
| docker build \ | ||
| --build-arg ENABLE_VERTEX=1 \ | ||
|
rajshah4 marked this conversation as resolved.
|
||
| -t my-agent-server:vertex \ | ||
| -f openhands-agent-server/openhands/agent_server/docker/Dockerfile \ | ||
| . | ||
| ``` | ||
|
|
||
| Then point OpenHands at your custom image via the `AGENT_SERVER_IMAGE_REPOSITORY` and | ||
| `AGENT_SERVER_IMAGE_TAG` environment variables (see the | ||
| [Custom Sandbox Guide](/openhands/usage/advanced/custom-sandbox-guide) for details). | ||
|
|
||
| #### OpenHands Enterprise (Replicated / Kubernetes) | ||
|
|
||
| The default OHE installer Vertex path routes LLM calls through a LiteLLM proxy — the | ||
| agent-server uses a `litellm_proxy/...` model, and the proxy makes the actual Vertex call. | ||
| So the agent-server image does **not** need Vertex enabled for the default path; `ENABLE_VERTEX=1` | ||
| is only relevant if you customize OHE to bypass the proxy and call `vertex_ai/*` directly from | ||
| the agent-server. | ||
|
rajshah4 marked this conversation as resolved.
|
||
|
|
||
| ### Claude via Vertex AI | ||
|
|
||
| If you route Anthropic Claude through Google Vertex AI / Model Garden (rather than direct | ||
| Anthropic endpoints), use the `vertex_ai/` prefix with the Vertex-published model name, | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 Critical: The Anthropic Python SDK does not expose a
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Pushing back on this one —
|
||
| which is date-stamped: | ||
|
|
||
| - `Custom Model`: `vertex_ai/claude-sonnet-4-5@20250929` | ||
|
|
||
| Use the exact model name shown in your Vertex AI Model Garden console. | ||
|
|
||
| <Note> | ||
| For `vertex_ai/*` models, OpenHands still needs <code>google-cloud-aiplatform</code> | ||
| from the `vertex` extra above. In custom Claude via Vertex AI setups, if you encounter | ||
| <code>ModuleNotFoundError: No module named 'anthropic'</code>, install the Anthropic SDK too: | ||
| <code>pip install "anthropic[vertex]"</code>. | ||
| </Note> | ||
|
|
||
| ### Troubleshooting | ||
|
|
||
| #### Vertex AI SDK Import Error | ||
|
|
||
| If you encounter this error: | ||
| ``` | ||
| litellm.BadRequestError: Vertex_aiException BadRequestError - vertexai import failed | ||
| please run `pip install -U "google-cloud-aiplatform>=1.38"`. | ||
| Got error: No module named 'vertexai' | ||
| ``` | ||
|
|
||
| This means the agent-server image does not include the Vertex AI SDK. Enable the `vertex` | ||
| extra as described in [Vertex AI Dependencies](#vertex-ai-dependencies) above. | ||
Uh oh!
There was an error while loading. Please reload this page.