Skip to content

AgentDetails.tool_declarations come back as dicts after a JSON round trip, breaking judge metrics on saved runs #7876

Closed

Description

@adityasingh2400

🔴 Required Information

Describe the Bug:
AgentDetails.tool_declarations is typed list[Any], so the genai_types.Tool objects in it don't survive JSON. Two ordinary flows round-trip through JSON:

  • Loading saved eval results (EvalSetResult.model_validate_json).
  • The eval service guide's "Score a run again without rerunning it" flow (InferenceResult.model_validate_json).

After either one, every tool comes back as a plain dict. Consumers of the field expect Tool objects:

  • rubric_based_multi_turn_trajectory_quality_v1 reads tool.function_declarations and raises AttributeError: 'dict' object has no attribute 'function_declarations'. LocalEvalService catches that, so the metric silently becomes NOT_EVALUATED.
  • The other judges serialize the tools through get_tool_declarations_as_json_str. Their prompt now carries the dict's null fields: 785 characters instead of 224 for one tool, including 17 nulls. The same run is therefore judged on different input depending on whether it was saved first.

Steps to Reproduce:

  1. Build an AppDetails whose agent has a genai_types.Tool in tool_declarations. Inference records this on every invocation.
  2. Round-trip it with AppDetails.model_validate_json(app_details.model_dump_json()).
  3. Inspect the type of tool_declarations[0], or score the round-tripped invocations with rubric_based_multi_turn_trajectory_quality_v1.

Expected Behavior:
The round-tripped tool_declarations equal the original Tool objects, and the judges see the same tools text as before saving.

Observed Behavior:

entry type after round trip: dict
judge tools text: original 224 chars, restored 785 chars, nulls in restored: 17

  File ".../rubric_based_multi_turn_trajectory_evaluator.py", line 270, in _assemble_dialogue_history
    for func in tool.function_declarations or []:
AttributeError: 'dict' object has no attribute 'function_declarations'
restored run -> rubric_based_multi_turn_trajectory_quality_v1 NOT_EVALUATED score None

Environment Details:

  • ADK Library Version (pip show google-adk): 2.13.0 (also on main at e9e8f61)
  • Desktop OS: macOS
  • Python Version (python -V): 3.11.16

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: any (the round trip doesn't involve a model)

🟡 Optional Information

Minimal Reproduction Code:

from google.genai import types
from google.adk.evaluation.app_details import AgentDetails, AppDetails

tool = types.Tool(function_declarations=[
    types.FunctionDeclaration(name="set_light", description="Turns a light on or off.")])
app = AppDetails(agent_details={"home": AgentDetails(name="home", tool_declarations=[tool])})

restored = AppDetails.model_validate_json(app.model_dump_json())
print(type(restored.agent_details["home"].tool_declarations[0]))  # <class 'dict'>

How often has this issue occurred?:

  • Always (100%), for any invocation whose agent has tools, once it has been through JSON

Activity

  1. added a commit that references this issue on Nov 3, 2026
    4255760
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions