From 482169aefa5e91e1cfb42cd4967adaff885d1b0d Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Thu, 17 Sep 2026 19:44:04 -0700 Subject: [PATCH] feat(evals): correlate evaluation traces with the result that caused them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Runner` already accepted an `around_evaluation:` wrapper and already owned a `run_id` in its `metadata:`, and a telemetry backend already offers a per-block agent scope that can report the trace it opened. Nothing joined them, so every host that wanted a report row to link to the conversation behind it wrote the same correlation plumbing itself. `ActiveAgent::Evals::Correlation` is that join. A run mints a `run_id`, each evaluation a `result_id`, and both ride every trace opened inside them as `eval.`-prefixed attributes. The trace ids travel the other way onto `result.replay.metadata`: `trace_id` for the replay, `judge_trace_ids` for the judge calls that graded it. The verdict, being a judge call outside any evaluation, lands on the run metadata a Report carries rather than on whichever result was evaluated last. The tracer is injected, so the module takes on no telemetry dependency and `require "active_agent/evals"` still loads on its own — covered by a test that drives a correlated run out of process. Hand the object to `Runner.new(around_evaluation:)` directly; a plain lambda there is unchanged. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 13 ++ docs/framework/evaluations.md | 46 ++++ lib/active_agent/evals.rb | 4 +- lib/active_agent/evals/correlation.rb | 178 +++++++++++++++ test/evals/correlation_test.rb | 216 ++++++++++++++++++ test/evals/standalone_load_test.rb | 16 ++ .../support/standalone_correlation_script.rb | 32 +++ 7 files changed, 504 insertions(+), 1 deletion(-) create mode 100644 lib/active_agent/evals/correlation.rb create mode 100644 test/evals/correlation_test.rb create mode 100644 test/evals/standalone_load_test.rb create mode 100644 test/evals/support/standalone_correlation_script.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index dd90dc4a..e9903ac3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `Evaluation#replace_scenarios!` takes `on_removed:` — `:destroy` (the default, unchanged) or `:disable`, which keeps a scenario the suite no longer names as `enabled: false` so earlier runs' results still resolve. +- **An evaluation's traces link back to the result that caused them.** + `ActiveAgent::Evals::Correlation` joins two APIs the module already had but + never connected: `Runner`'s `around_evaluation:` hook and its `metadata:` + run identity, and a telemetry backend's per-block agent scope. A run mints a + `run_id`, each evaluation a `result_id`, and both ride every trace opened + inside them as `eval.`-prefixed attributes; the trace ids travel the other + way onto `result.replay.metadata` — `trace_id` for the replay, + `judge_trace_ids` for the judge calls that graded it, with a run-level + verdict landing on the run metadata the Report carries rather than on + whichever result was evaluated last. The tracer is injected, so the module + takes on no telemetry dependency and `require "active_agent/evals"` still + loads on its own. Hand the object to `Runner.new(around_evaluation:)` + directly; a plain lambda there keeps working unchanged. ## [1.6.2] - 2026-09-16 diff --git a/docs/framework/evaluations.md b/docs/framework/evaluations.md index 24d1f37c..ed894840 100644 --- a/docs/framework/evaluations.md +++ b/docs/framework/evaluations.md @@ -121,6 +121,52 @@ after the wrapper finishes; wrapper errors propagate. Direct `Runner#evaluate` calls bypass the wrapper so a host doing its own scheduling can establish context itself. +### Correlating traces with results + +`Correlation` is that wrapper, written once against a telemetry backend +instead of per host. It mints a `run_id` per run and a `result_id` per +evaluation, puts both on every trace opened inside them as `eval.`-prefixed +attributes, and records the trace ids back onto the result: +`replay.metadata["trace_id"]` for the replay and `["judge_trace_ids"]` for the +judge calls that graded it. A judge call made outside an evaluation — the +run-level verdict — lands on the run metadata instead, which is the hash the +Report carries, so the verdict is reachable from the report rather than +attributed to the last result evaluated. + +```ruby +correlation = ActiveAgent::Evals::Correlation.new( + agent_name: "SupportAgent", + judge_name: "SupportAgentJudge", + tracer: ->(name, action:, attributes:, on_trace:, &block) { + MyTelemetry.with_agent(name, action: action, attributes: attributes, + on_trace: on_trace, synchronous: true, &block) + } +) + +report = correlation.with_run("suite" => "support") do |metadata| + Runner.new( + scenarios: scenarios, models: models, metadata: metadata, + around_evaluation: correlation, + replay: ->(scenario, spec) { correlation.replay { agent.run(scenario.prompt) } }, + judge: Judge.new(label: "judge-model") { |instructions:, prompt:| + correlation.judge("score") { chat.with_instructions(instructions).ask(prompt).content } + } + ).call +end +``` + +Pass the yielded metadata to `Runner.new(metadata:)`: it is the caller's own +hash, mutated in place, so a run reopened around a later verdict accumulates +onto the metadata the Report already carries. + +The tracer is injected, so the evaluation module takes on no telemetry +dependency and still loads on its own. Without one, the correlation mints ids +and merges metadata while the blocks run untraced, which keeps a suite +runnable where no telemetry is configured. `trace_keys:` chooses which +correlation keys become attributes — the default is `run_id`, `result_id`, +`suite`, `scenario_key`, `model_label`, `model` and `provider`, and anything +else in the run metadata stays on the report but off the traces. + ## Faults | Fault | Meaning | diff --git a/lib/active_agent/evals.rb b/lib/active_agent/evals.rb index 0e299b9f..8f3c1318 100644 --- a/lib/active_agent/evals.rb +++ b/lib/active_agent/evals.rb @@ -25,6 +25,7 @@ require_relative "evals/report_html" require_relative "evals/report" require_relative "evals/runner" +require_relative "evals/correlation" require_relative "evals/publisher" # Scenario evaluations for agents that answer with tools. @@ -40,7 +41,8 @@ # fell short and what would fix it (Diagnosis, refined by an optional Judge), # and rolling everything up per model (Report). Runner ties them together # around one callable you supply: given a scenario and a model, run the -# agent and return a Replay. +# agent and return a Replay. Correlation is optional plumbing on top: it +# links the traces a run emits back to the result that caused them. # # scenarios = ActiveAgent::Evals::ScenarioParser.scenarios(pasted_text) # models = ActiveAgent::Evals::ModelSpec.parse_all(%w[gpt-5-mini qwen3:8b], default_provider: "openai") diff --git a/lib/active_agent/evals/correlation.rb b/lib/active_agent/evals/correlation.rb new file mode 100644 index 00000000..29538699 --- /dev/null +++ b/lib/active_agent/evals/correlation.rb @@ -0,0 +1,178 @@ +# frozen_string_literal: true + +require "securerandom" +require "active_support/isolated_execution_state" + +module ActiveAgent + module Evals + # Used to tie the traces an evaluation produces back to the run and the + # result that caused them, so a report row links to the exact conversation + # behind it. + # + # A run mints a `run_id`, each evaluation mints a `result_id`, and both ride + # every trace opened inside them as `eval.`-prefixed attributes. The trace + # ids travel the other way: a replay's trace id lands on + # `result.replay.metadata["trace_id"]`, and every judge call made while + # scoring that result appends to its `"judge_trace_ids"`. The verdict — a + # judge call made outside any evaluation — appends to the run metadata + # instead, which is the same Hash a Report carries as its `metadata`. + # + # A tracer is `(name, action:, attributes:, on_trace:) { ... }`: it opens a + # trace named for the agent, and calls `on_trace` with something answering + # to `#trace_id` once the trace is known. + # + # correlation = ActiveAgent::Evals::Correlation.new( + # agent_name: "SupportAgent", + # judge_name: "SupportAgentJudge", + # tracer: ->(name, action:, attributes:, on_trace:, &block) { + # MyTelemetry.with_agent(name, action: action, attributes: attributes, + # on_trace: on_trace, synchronous: true, &block) + # } + # ) + # + # correlation.with_run("suite" => "support") do |metadata| + # Runner.new( + # scenarios: scenarios, models: models, metadata: metadata, + # replay: ->(scenario, spec) { correlation.replay { agent.run(scenario.prompt) } }, + # judge: Judge.new(label: "judge-model") { |instructions:, prompt:| + # correlation.judge("score") { chat.with_instructions(instructions).ask(prompt).content } + # }, + # around_evaluation: correlation + # ).call + # end + # + # Without a tracer the correlation still mints ids and merges metadata. The + # blocks run untraced. + class Correlation + STATE_KEY = :active_agent_evals_correlation + + # DEFAULT_TRACE_KEYS names the correlation metadata that rides a trace as + # `eval.`-prefixed attributes. Anything else a caller puts in the run + # metadata (a tenant, a role) stays on the report but off the traces. + DEFAULT_TRACE_KEYS = %w[run_id result_id suite scenario_key model_label model provider].freeze + + attr_reader :agent_name, :judge_name, :trace_keys + + # @param agent_name [String] the trace name for the agent under evaluation + # @param judge_name [String] the trace name for judge traffic, kept distinct + # so grading calls do not read as the agent's own traffic + # @param tracer [#call, nil] `(name, action:, attributes:, on_trace:) { ... }`; + # nil runs every block untraced + # @param replay_action [String] the action name recorded for a replay trace + # @param trace_keys [Array] which correlation keys become attributes + def initialize(agent_name:, judge_name: "#{agent_name}Judge", tracer: nil, replay_action: "eval", + trace_keys: DEFAULT_TRACE_KEYS) + @agent_name = agent_name + @judge_name = judge_name + @tracer = tracer + @replay_action = replay_action + @trace_keys = trace_keys.map(&:to_s) + end + + # Opens a run. Mints `run_id` unless `metadata` carries one, and yields the + # metadata hash the traces will be correlated against — pass that same hash + # to `Runner.new(metadata:)` so the Report carries the run's identity and + # collects the verdict's trace id. + # + # The hash yielded is the caller's own, mutated in place, so a run + # reopened around a later verdict accumulates onto the metadata a Report + # already carries. + # + # @param metadata [Hash] opaque run metadata. A non-Hash is coerced with + # `#to_h`, and non-String keys are stringified. + # @yieldparam metadata [Hash] + def with_run(metadata = {}) + run = metadata.is_a?(Hash) ? metadata : metadata.to_h + run.transform_keys!(&:to_s) unless run.keys.all?(String) + run["run_id"] ||= SecureRandom.uuid + with_context({ run: run, result: nil }) { yield run } + end + + # Wraps one evaluation, in the shape `Runner.new(around_evaluation:)` calls: + # `(scenario, spec) { ... } → Result`. Mints a `result_id`, merges the + # correlation onto the Result's replay metadata, and returns the Result. + def around_evaluation(scenario, spec) + result_metadata = { + "run_id" => run_metadata["run_id"], + "result_id" => SecureRandom.uuid, + "scenario_key" => scenario.key, + "model_label" => spec.label, + "model" => spec.model, + "provider" => spec.provider + }.compact + + with_context(run: run_metadata, result: result_metadata) do + yield.tap { |result| result.replay.metadata.merge!(result_metadata) } + end + end + + # Delegates to `around_evaluation`, so the object satisfies + # `Runner.new(around_evaluation:)` directly. + def call(scenario, spec, &) + around_evaluation(scenario, spec, &) + end + + # Traces one replay of the agent under evaluation. The trace id lands on + # the current result's metadata, so a report row links to the conversation. + def replay(action = @replay_action, &) + trace(@agent_name, action, judge: false, &) + end + + # Traces one judge call. Appends to the current result's `judge_trace_ids`, + # or the run's when no evaluation is open (the verdict). + def judge(action = "score", &) + trace(@judge_name, action, judge: true, &) + end + + # The correlation metadata in scope, or nil outside a run. A result's + # values win over the run's. + def current + context = ActiveSupport::IsolatedExecutionState[STATE_KEY] + return nil unless context + + context.fetch(:run, {}).merge(context[:result] || {}) + end + + private + + def run_metadata + context = ActiveSupport::IsolatedExecutionState[STATE_KEY] + context&.fetch(:run, nil) || {} + end + + def with_context(context) + previous = ActiveSupport::IsolatedExecutionState[STATE_KEY] + ActiveSupport::IsolatedExecutionState[STATE_KEY] = context + yield + ensure + ActiveSupport::IsolatedExecutionState[STATE_KEY] = previous + end + + def trace(name, action, judge:, &block) + return block.call unless @tracer + + context = ActiveSupport::IsolatedExecutionState[STATE_KEY] || {} + correlation = context.fetch(:run, {}).merge(context[:result] || {}) + attributes = correlation.slice(*@trace_keys).transform_keys { |key| "eval.#{key}" } + # A replay belongs to the evaluation that opened it and nowhere else, so + # it records no trace id when called outside one. A judge call outside an + # evaluation is the verdict, which belongs to the run. + target = judge ? (context[:result] || context[:run]) : context[:result] + + @tracer.call(name, action: action, attributes: attributes, on_trace: recorder(target, judge: judge), &block) + end + + def recorder(target, judge:) + lambda do |trace| + next unless target + + if judge + (target["judge_trace_ids"] ||= []) << trace.trace_id + else + target["trace_id"] = trace.trace_id + end + end + end + end + end +end diff --git a/test/evals/correlation_test.rb b/test/evals/correlation_test.rb new file mode 100644 index 00000000..69a5c9f6 --- /dev/null +++ b/test/evals/correlation_test.rb @@ -0,0 +1,216 @@ +# frozen_string_literal: true + +require "test_helper" +require_relative "evals_test_support" + +class EvalsCorrelationTest < ActiveSupport::TestCase + include EvalsTestSupport + + # Stands in for a telemetry adapter, recording every trace it was asked to open. + class RecordingTracer + Trace = Struct.new(:trace_id) + + attr_reader :traces + + def initialize + @traces = [] + @sequence = 0 + end + + def to_proc + method(:call).to_proc + end + + def call(name, action:, attributes:, on_trace:, &block) + @sequence += 1 + trace_id = "trace-#{@sequence}" + @traces << { name: name, action: action, attributes: attributes, trace_id: trace_id } + on_trace&.call(Trace.new(trace_id)) + block.call + end + end + + def setup + @tracer = RecordingTracer.new + @correlation = ActiveAgent::Evals::Correlation.new(agent_name: "SupportAgent", judge_name: "SupportJudge", + tracer: @tracer.to_proc) + end + + def test_a_run_mints_identities_that_reach_every_result_and_its_replay_trace + report = run_evaluation(scenarios: [ scenario("lookup_1"), scenario("lookup_2") ]) + + result_ids = report.results.map { |result| result.replay.metadata.fetch("result_id") } + assert_equal 2, result_ids.uniq.size, "each evaluation should mint its own result identity" + report.results.each do |result| + metadata = result.replay.metadata + assert_equal "run-1", metadata["run_id"], "the run identity should reach every result" + assert_equal result.scenario.key, metadata["scenario_key"] + assert_equal result.spec.label, metadata["model_label"] + replay_trace = @tracer.traces.find { |trace| trace[:trace_id] == metadata.fetch("trace_id") } + assert replay_trace, "the replay's trace id should land on the result's replay metadata" + assert_equal "SupportAgent", replay_trace[:name] + end + end + + def test_correlation_rides_a_trace_as_eval_prefixed_attributes + run_evaluation(scenarios: [ scenario("lookup_1") ]) + + replay_trace = @tracer.traces.find { |trace| trace[:name] == "SupportAgent" } + assert_equal({ + "eval.run_id" => "run-1", + "eval.suite" => "support", + "eval.result_id" => replay_trace[:attributes].fetch("eval.result_id"), + "eval.scenario_key" => "lookup_1", + "eval.model_label" => "test-model", + "eval.model" => "test-model", + "eval.provider" => "openai" + }, replay_trace[:attributes]) + end + + def test_opaque_run_metadata_stays_off_the_traces + @correlation.with_run("suite" => "support", "tenant" => "acme") do |metadata| + @correlation.replay { nil } + assert_equal "acme", metadata["tenant"], "the caller's own metadata should stay on the run" + end + + refute_includes @tracer.traces.first[:attributes].keys, "eval.tenant" + end + + def test_judge_traffic_routes_to_the_result_and_the_verdict_to_the_run + metadata = nil + report = run_evaluation(scenarios: [ scenario("lookup_1") ], judge: judge_calling("score")) { |run| metadata = run } + @correlation.with_run(metadata) { @correlation.judge("verdict") { nil } } + + result_metadata = report.results.first.replay.metadata + judge_ids = result_metadata.fetch("judge_trace_ids") + assert_equal 1, judge_ids.size, "the result should collect exactly the one judge call made inside it" + judge_trace = @tracer.traces.find { |trace| trace[:trace_id] == judge_ids.first } + assert_equal "SupportJudge", judge_trace[:name], "a judge call should be traced under the judge name" + assert_equal result_metadata.fetch("result_id"), judge_trace[:attributes]["eval.result_id"] + + verdict_id = metadata.fetch("judge_trace_ids").last + verdict_trace = @tracer.traces.find { |trace| trace[:trace_id] == verdict_id } + assert_equal "verdict", verdict_trace[:action] + refute_includes verdict_trace[:attributes].keys, "eval.result_id", + "a verdict should not be attributed to the last result evaluated" + refute_includes judge_ids, verdict_id + end + + def test_extra_trace_keys_carry_a_callers_own_correlation_onto_its_traces + keys = ActiveAgent::Evals::Correlation::DEFAULT_TRACE_KEYS + %w[tenant] + correlation = ActiveAgent::Evals::Correlation.new(agent_name: "SupportAgent", tracer: @tracer.to_proc, + trace_keys: keys) + correlation.with_run("run_id" => "run-1", "tenant" => "acme") { correlation.replay { nil } } + + assert_equal "acme", @tracer.traces.first[:attributes].fetch("eval.tenant") + end + + def test_a_reopened_run_accumulates_onto_the_metadata_a_report_already_carries + report = nil + metadata = { "run_id" => "run-1" } + @correlation.with_run(metadata) do |run| + report = ActiveAgent::Evals::Runner.new( + scenarios: [ scenario("lookup_1") ], models: [ spec("test-model") ], metadata: run, + around_evaluation: @correlation, replay: ->(*) { replay(answer: "Order ABC-123 shipped on Monday.") } + ).call + end + @correlation.with_run(metadata) { @correlation.judge("verdict") { nil } } + + assert_same metadata, report.metadata, "the run metadata should be the caller's own hash" + assert_equal [ "trace-1" ], report.metadata.fetch("judge_trace_ids"), + "a verdict traced after the run should reach the report" + end + + def test_a_replay_trace_id_never_lands_on_the_run + @correlation.with_run("run_id" => "run-1") do |metadata| + @correlation.replay { nil } + assert_nil metadata["trace_id"], "a replay outside an evaluation should record no trace id on the run" + end + end + + def test_a_raising_replay_restores_the_enclosing_context + outer = nil + @correlation.with_run("run_id" => "outer") do |run| + outer = @correlation.current + assert_raises(IOError) do + @correlation.around_evaluation(scenario("lookup_1"), spec("test-model")) do + @correlation.replay { raise IOError, "synthetic failure" } + end + end + assert_equal outer, @correlation.current, "the evaluation context should be restored after a failure" + assert_equal "outer", run.fetch("run_id") + end + + assert_nil @correlation.current, "the run context should be restored after the run" + end + + def test_a_raising_run_restores_an_enclosing_run + @correlation.with_run("run_id" => "outer") do + assert_raises(IOError) do + @correlation.with_run("run_id" => "inner") { raise IOError, "synthetic failure" } + end + + assert_equal "outer", @correlation.current.fetch("run_id") + end + + assert_nil @correlation.current + end + + def test_a_run_mints_its_own_identity_when_the_caller_supplies_none + first = @correlation.with_run { |metadata| metadata.fetch("run_id") } + second = @correlation.with_run { |metadata| metadata.fetch("run_id") } + + refute_equal first, second + assert_match(/\A[0-9a-f-]{36}\z/, first) + end + + def test_without_a_tracer_identities_still_reach_the_results + correlation = ActiveAgent::Evals::Correlation.new(agent_name: "SupportAgent") + report = correlation.with_run("run_id" => "run-1") do |metadata| + ActiveAgent::Evals::Runner.new( + scenarios: [ scenario("lookup_1") ], models: [ spec("test-model") ], metadata: metadata, + around_evaluation: correlation, + replay: ->(*) { correlation.replay { replay(answer: "Order ABC-123 shipped on Monday.") } } + ).call + end + + metadata = report.results.first.replay.metadata + assert_equal "run-1", metadata.fetch("run_id") + refute_includes metadata.keys, "trace_id", "an untraced run records no trace id" + end + + def test_a_plain_around_evaluation_lambda_adds_no_correlation + seen = [] + wrapper = lambda do |scenario, spec, &evaluate| + seen << [ scenario.key, spec.label ] + evaluate.call + end + report = ActiveAgent::Evals::Runner.new( + scenarios: [ scenario("lookup_1") ], models: [ spec("test-model") ], around_evaluation: wrapper, + replay: ->(*) { replay(answer: "Order ABC-123 shipped on Monday.") } + ).call + + assert_equal [ [ "lookup_1", "test-model" ] ], seen + assert_equal 1, report.results.size + assert_empty report.results.first.replay.metadata, "a plain wrapper should add no correlation of its own" + end + + private + + def judge_calling(action) + fake_judge do |_instructions, _prompt| + @correlation.judge(action) { '{"score": 0.9}' } + end + end + + def run_evaluation(scenarios:, judge: nil) + @correlation.with_run("run_id" => "run-1", "suite" => "support") do |metadata| + yield metadata if block_given? + ActiveAgent::Evals::Runner.new( + scenarios: scenarios, models: [ spec("test-model") ], judge: judge, metadata: metadata, + around_evaluation: @correlation, + replay: ->(*) { @correlation.replay { replay(answer: "Order ABC-123 shipped on Monday.") } } + ).call + end + end +end diff --git a/test/evals/standalone_load_test.rb b/test/evals/standalone_load_test.rb new file mode 100644 index 00000000..8457e0c5 --- /dev/null +++ b/test/evals/standalone_load_test.rb @@ -0,0 +1,16 @@ +# frozen_string_literal: true + +require "test_helper" + +# The dummy app's test process has the whole framework loaded, so this check +# runs out of process. +class EvalsStandaloneLoadTest < ActiveSupport::TestCase + SCRIPT = File.expand_path("support/standalone_correlation_script.rb", __dir__) + LIB = File.expand_path("../../lib", __dir__) + + def test_the_evaluation_module_and_a_correlated_run_load_without_the_framework + output = IO.popen([ RbConfig.ruby, "-I#{LIB}", SCRIPT ], err: %i[child out], &:read) + + assert_equal "ok", output.strip, output + end +end diff --git a/test/evals/support/standalone_correlation_script.rb b/test/evals/support/standalone_correlation_script.rb new file mode 100644 index 00000000..1b4549aa --- /dev/null +++ b/test/evals/support/standalone_correlation_script.rb @@ -0,0 +1,32 @@ +# frozen_string_literal: true + +# Drives a correlated run end to end with only the evaluation module loaded. +require "active_agent/evals" + +raise "the framework should not be loaded" if defined?(ActiveAgent::Base) + +traces = [] +tracer = lambda do |name, action:, attributes:, on_trace:, &block| + trace_id = "trace-#{traces.size + 1}" + traces << { name: name, action: action, attributes: attributes } + on_trace&.call(Struct.new(:trace_id).new(trace_id)) + block.call +end + +correlation = ActiveAgent::Evals::Correlation.new(agent_name: "SupportAgent", tracer: tracer) +scenario = ActiveAgent::Evals::Scenario.from_hash({ "key" => "lookup_1", "prompt" => "Where is order ABC-123?" }) +model = ActiveAgent::Evals::ModelSpec.parse("test-model", default_provider: "openai") + +report = correlation.with_run("suite" => "support") do |metadata| + ActiveAgent::Evals::Runner.new( + scenarios: [ scenario ], models: [ model ], metadata: metadata, around_evaluation: correlation, + replay: ->(*) { correlation.replay { ActiveAgent::Evals::Replay.new(answer: "Order ABC-123 shipped on Monday.") } } + ).call +end + +metadata = report.results.first.replay.metadata +raise "expected a result id, got #{metadata.inspect}" unless metadata["result_id"] +raise "expected the replay trace id, got #{metadata.inspect}" unless metadata["trace_id"] == "trace-1" +raise "expected the eval attributes, got #{traces.inspect}" unless traces.first[:attributes]["eval.suite"] == "support" + +puts "ok"