feat(eval): store eval files outside a read-only agents_dir - #7183
Open
Anusha0501 wants to merge 1 commit into
Open
Anusha0501 wants to merge 1 commit into
Anusha0501 wants to merge 1 commit into
Conversation
K8s and container images often mount agent code read-only, so writing eval sets next to the agent fails. Honor ADK_EVAL_STORAGE_DIR and ADK_EVAL_STORAGE_URI, and accept file:// in --eval_storage_uri. Fixes google#3887
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
2. Or, if no issue exists, describe the change:
Problem:
Local eval sets and results are written under
agents_dir. In Kubernetes and other container deployments that directory is often read-only, soadk web/adk evalfail withPermissionError.--eval_storage_urionly acceptedgs://, which is more infrastructure than many environments need.Solution:
Resolve eval storage with this precedence: explicit
--eval_storage_uri/eval_storage_uri, thenADK_EVAL_STORAGE_URI, thenADK_EVAL_STORAGE_DIR, thenagents_dir.file://URIs (andADK_EVAL_STORAGE_DIR) use the existing local eval managers and create the directory if needed. GCS behavior is unchanged.Testing Plan
Unit Tests:
The deselected case needs
google-cloud-storage(existing GCS manager test). New coverage includes URI/env precedence,file://path decoding, directory creation,~expansion, andget_eval_sets_manager/get_fast_api_appwiring.Manual End-to-End (E2E) Tests:
agents_dirat a read-only tree (orchmod a-w).export ADK_EVAL_STORAGE_DIR=/tmp/adk_evalsadk eval_set createor the web eval UI..evalset.jsonis written under/tmp/adk_evals/<app_name>/and not under the agent directory.Checklist
Additional context
Does not add a new
eval_storage_dir=kwarg onget_fast_api_app;eval_storage_uri="file://..."and the env vars cover the same use case with one resolution path.