feat(agent evals): promote agents/environments to be configurable, support multi-step agent evals - #3774
Open
hvent90 wants to merge 26 commits into
Open
feat(agent evals): promote agents/environments to be configurable, support multi-step agent evals#3774hvent90 wants to merge 26 commits into
hvent90 wants to merge 26 commits into
Conversation
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #3774 +/- ##
==========================================
+ Coverage 77.83% 77.95% +0.12%
==========================================
Files 1329 1341 +12
Lines 126868 127658 +790
Branches 11035 11189 +154
==========================================
+ Hits 98746 99521 +775
- Misses 24971 24986 +15
Partials 3151 3151
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
hvent90
commented
Aug 31, 2026
hvent90
commented
Aug 31, 2026
1 task
hvent90
force-pushed
the
feat/agent-evals-split
branch
from
August 31, 2026 23:19
77ba61e to
f22d7d8
Compare
tracing_http_client wraps the OpenAI wire so every provider call lands whole under <run log dir>/llm as <seq>-request/-response.json pairs; McpClient wires it into init_chat_model and exposes trace_dir() over RPC so harnesses can link agent steps to the exact payloads.
Tracing now happens only when McpClientConfig.trace_dir is set explicitly; a plain dimos run no longer taps the wire or writes under <run log dir>/llm. Harnesses that want capture pass trace_dir.
isinstance(model, ChatOpenAI) on the init_chat_model result replaces _attempt_infer_model_provider; a langchain bump can no longer break McpClient at import time. Traced set unchanged.
write_normalized and latest_pair have no callers here; they land with the eval framework that consumes them. This module is now just the tap.
One EvalCase: environment + instruction + grader, nothing else. The world is on the case (Dataset / ImageFile / Sim(blueprint=...)); agents are one dataclass module each (question_answer, blind, mcp_client, pi) whose constructor args are everything they decide, recorded whole in summary.json; every provider request/response is saved per call. Pi runs the pi CLI headless behind a RecordingProxy and reaches the robot through `dimos mcp call` from its bash tool. PassiveEval / InteractiveEval, generate.py and local.py are gone. See docs/development/agent_evals_design.md and agent_evals_restructure.md for the decisions.
One apartment case: map the rooms with deliberate move_to only, graded 0.5 exact room count + 0.5 room coverage from recorded odometry, with ground truth measured from the scene geometry (structure.glb walls + object manifest). Pi gains a `skills` pass-through to its native repeatable --skill flag (ambient discovery stays off), so the same agent runs the agent_encode, numpy, and spatial-skill conditions differing only in system_prompt / skills — same case, stack, and grader.
write_normalized and latest_pair were stripped from the extracted wire- trace PR (no callers there); they return here with their consumers, the eval recorder and pi agent.
The runner ended a case at agent idle and graded the last recorded pose, truncating navigation still in flight — navigate_with_text returns once the goal is set. Environments now settle on the budget the agent didn't use: Sim waits until the robot is at rest, frozen environments return immediately. The trace dir also rides environment.start() into the launched dimos instead of a shared env var, so concurrent runs can't collide.
hvent90
force-pushed
the
feat/agent-evals-split
branch
from
September 1, 2026 04:55
3139033 to
e738efe
Compare
/agent_idle is its own topic, so the idle flip can arrive before the last /agent message and the turn ended with an empty final_answer (3.14t CI). End the turn only once the received AIMessages match the raw trace, which is complete before the idle flip is published. Event-driven, no polling; a truly lost message now becomes an honest runner timeout.
hvent90
marked this pull request as ready for review
September 1, 2026 06:06
hvent90
requested review from
leshy,
mustafab0,
paul-nechifor and
spomichter
as code owners
September 1, 2026 06:06
spomichter
reviewed
Sep 1, 2026
|
|
||
|
|
||
| @dataclass | ||
| class McpClientAgent: |
Contributor
There was a problem hiding this comment.
We already have an MCP client
Contributor
Author
There was a problem hiding this comment.
Yes, this is the eval agent that invokes that MCP client.
spomichter
reviewed
Sep 1, 2026
spomichter
requested changes
Sep 1, 2026
The default system prompts forced bare-number/short-phrase replies on every suite, which fits parser-graded QA but forbids explanations for everything else. Agent prompts keep only the grounding contract; each case whose grader parses the reply now asks for its format in the question, next to the scorer that needs it.
spomichter
reviewed
Sep 1, 2026
Contributor
Greptile SummaryThis change expands eval execution with configurable agents and environments, multi-step trajectories, provider tracing, and persisted run metadata. Runtime verification found that an agent which exceeds its timeout can continue running after the environment has been stopped and while grading is in progress, allowing late tool activity or artifact writes to interfere with the result. Confidence Score: 4/5Not safe to merge until timed-out agent execution is prevented from overlapping environment cleanup and grading. A deterministic runtime reproduction exercised the timeout path and directly observed the agent resume after environment cleanup and grading began. Files Needing Attention: dimos/evals/runner.py needs timeout cancellation or process isolation before grading starts.
 T-Rex LogsWhat T-Rex did
Reviews (1): Last reviewed commit: "fix(evals): don't drop the final answer ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
spomichter
reviewed
Sep 1, 2026
Comment on lines
+192
to
+196
| assets = tmp_path / "assets" | ||
| assets.mkdir() | ||
| (tmp_path / "cases.jsonl").write_text('{"id":"q","image":"assets/q.png"}\n') | ||
| (tmp_path / "labels.jsonl").write_text('{"id":"q","answer":"yes"}\n') | ||
| (assets / "q.png").write_bytes(b"image") |
Contributor
There was a problem hiding this comment.
should pytest be saving to filesystem / creating dirs i forget best practice
Contributor
Author
There was a problem hiding this comment.
Using tmp_path is Pytest's suggested way of doing so.
They are saved under /tmp (docs) and will be cleared on system reboot.
Here is an example of us doing so elsewhere in the repo.
Drops the Pi coding agent, its recording proxy, and the point-cloud mapping suite (plus their tests and docs mentions) so they can land separately on top of the eval split.
github-actions
Bot
added
ready-to-merge
The at-rest distance, dwell, and poll interval move from module globals to Sim dataclass fields, so a case can tune them and tests configure them by construction instead of monkeypatching.
github-actions
Bot
added
the
ready-to-merge
Trajectory is now the ATIF document itself (schema ATIF-v1.7), so the runner writes it with asdict and Harbor's validator accepts it. Agents build it through TrajectoryBuilder; LangChain agents go through LangChainRecorder, which owns the builder. Tool results attach as observations on the step that made the call, the instruction is step 1, and per-step token metrics keep cache reads. results.jsonl columns take the same names (prompt_tokens, completion_tokens, cost_usd).
…ia RPC McpClient gains set_trace_dir(path): stores config.trace_dir and rebuilds the model + graph via _rebuild_agent (tools kept on _agent_tools). The whole rebuild runs under _lock so it cannot interleave with on_system_modules or swap the graph mid-turn. The eval McpClientAgent calls it through Dimos.connect() at turn start, pointing capture at run_dir/raw. This replaces the MCPCLIENT__TRACE_DIR env injection in Sim.start and the trace_dir parameter on Environment.start. Dataset now launches the agent's modules as their own stack (dimos run <modules>, no simulator), waits for MCP readiness, and tears it down in stop(). mcp_url= attach mode is kept; mcp_url plus modules is rejected in preflight. default_mcp_url/blueprint_modules move from sim.py to environments/lib/launch.py.
github-actions
Bot
added
ready-to-merge
…thread The runner ran the agent on a daemon thread and, at the case's timeout, moved on to settle and grade while the thread kept going: calling the model, writing into the case's raw dir, and (for McpClientAgent) staying subscribed to the topics the next case would use. Agent.run now takes timeout_s and runs on the caller's thread. Each agent honours the budget at its one blocking wait and returns what it has, marked timeout; nothing of the agent's outlives the call. A timed-out McpClient turn keeps the steps it made instead of being blanked.
github-actions
Bot
removed
the
ready-to-merge
RunningEnvironment.recording forced a Store, so Dataset materialized its selection into a MemoryStore via copy_streams: every blob decoded up front and held for the case, and transforming selects re-appended under the source's declared type. Main's PassiveEval never copied; it applied the selects to the open store and encoded the lazy Streams directly. Do that again. RunningEnvironment.streams is the selection; Dataset keeps the opened store (now _store, matching how the memory layer names what open_dataset returns) until stop(). copy_streams is gone. Sim exposes no streams: its agents act through MCP and graders reopen the artifact.
github-actions
Bot
added
ready-to-merge
hvent90
force-pushed
the
feat/agent-evals-split
branch
from
September 1, 2026 23:43
be304a5 to
493ac68
Compare
github-actions
Bot
added
the
ready-to-merge
github-actions
Bot
added
ready-to-merge
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What precipitated this
generate.pyto its consumers before it grows exponentially in the root of thedimos.evalsnamespaceWhat this introduces
Two new first-class citizens in
dimos.evals:agent_encodeif a dataset with appropriate streams are presenthumancliuses, and it can interrop with cases acting on datasets and simsDefining and running evals becomes more declarative: a single
EvalCasetype that takes in an environment. You determine the agent when invoking from the cli:dimos eval run <suite> --agent <agent> --set <agent_config, like max_steps>This also features our first multi-step agent eval for discovering every room in the dimsim apartment scene using pointcloud data and saying how many rooms are in there. More of these cases will be in a following PR along with many question/answer pointcloud evals from this pr, as well as an updated
agent_encode.Trajectory format adheres to Agent Trajectory Interchange Format.
Examples
I want to hillclimb on pointcloud's
agent_encodemethod where I am just testing interpretability of the data (meaning I'm handing the data to it instead of testing access/query patterns):I'm noticing the LLM is answering accurately! For example, it can detect a corridor versus open space as well as correctly identify discrete rooms about 50% of the time. But, I should test it against a baseline to check my assumptions - it should at least exceed the performance of a standard coding agent with bash:
Or maybe I write an opionated agent harness skill that gives 1,000 tokens of guidance on how to query memory and use numpy on pointcloud data:
Stepping forward in time, lets say this process was repeated across rgb streams, occupancy grid, and other data we've collected. The next rung on the ladder would be writing evals for the kind of task the pointcloud data was meant to solve - such as identifying rooms - but then testing it against each of our other optimized data paths.
Possible next steps
Besides the immediate goal of making more impactful multi-step evals, these can be added to the backlog:
AI assistance disclaimer
My associates
Fable 5,Opus 5, andGPT 5.6 Solwrote the vast majority of code found in this PR. I have spent extensive time combing through this but apologize for any oversights! Thanks for reading.