Skip to content

feat(eval): store eval files outside a read-only agents_dir - #7183

Open
Anusha0501 wants to merge 1 commit into
google:mainfrom
Anusha0501:feat/eval-custom-storage-dir
Open

Anusha0501 wants to merge 1 commit into
google:mainfrom
Anusha0501:feat/eval-custom-storage-dir

Conversation

@Anusha0501

@Anusha0501 Anusha0501 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

2. Or, if no issue exists, describe the change:

Problem:
Local eval sets and results are written under agents_dir. In Kubernetes and other container deployments that directory is often read-only, so adk web / adk eval fail with PermissionError. --eval_storage_uri only accepted gs://, which is more infrastructure than many environments need.

Solution:
Resolve eval storage with this precedence: explicit --eval_storage_uri / eval_storage_uri, then ADK_EVAL_STORAGE_URI, then ADK_EVAL_STORAGE_DIR, then agents_dir. file:// URIs (and ADK_EVAL_STORAGE_DIR) use the existing local eval managers and create the directory if needed. GCS behavior is unchanged.

export ADK_EVAL_STORAGE_DIR=/tmp/adk_evals
# or
export ADK_EVAL_STORAGE_URI=file:///tmp/adk_evals
adk web /app/agents
app = get_fast_api_app(
    agents_dir="/app/agents",
    eval_storage_uri="file:///tmp/adk_evals",
    web=True,
)

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.
PYTHONPATH=src python -m pytest \
  tests/unittests/cli/utils/test_evals.py \
  tests/unittests/cli/utils/test_cli_eval.py \
  -q -k "not test_create_gcs_eval_managers_from_uri_success"

41 passed, 1 deselected

The deselected case needs google-cloud-storage (existing GCS manager test). New coverage includes URI/env precedence, file:// path decoding, directory creation, ~ expansion, and get_eval_sets_manager / get_fast_api_app wiring.

Manual End-to-End (E2E) Tests:

  1. Point agents_dir at a read-only tree (or chmod a-w).
  2. export ADK_EVAL_STORAGE_DIR=/tmp/adk_evals
  3. Create an eval set via adk eval_set create or the web eval UI.
  4. Confirm the .evalset.json is written under /tmp/adk_evals/<app_name>/ and not under the agent directory.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

Does not add a new eval_storage_dir= kwarg on get_fast_api_app; eval_storage_uri="file://..." and the env vars cover the same use case with one resolution path.

K8s and container images often mount agent code read-only, so writing
eval sets next to the agent fails. Honor ADK_EVAL_STORAGE_DIR and
ADK_EVAL_STORAGE_URI, and accept file:// in --eval_storage_uri.

Fixes google#3887
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Support Custom Evalset Storage Directory via Environment Variable

2 participants