Conversation
…tracer
A with_agent scope opened inside another one replaced it outright, and a turn captures whichever scope is active on its first round. So when an evaluation replay wrapped the code under test in an outer scope carrying `eval.*` attributes, an `on_trace` callback and `synchronous: true`, and that code opened its own scope around the actual `chat.ask`, the trace lost the evaluation attributes, the callback never received the trace id and delivery went back to the background thread, with no spec noticing.
Nested scopes now compose: attributes merge (enclosing first, nested keys win), `on_trace` and `synchronous` are inherited unless the nested call passes its own, and `synchronous:` defaults to nil ("inherit, else false") so an explicit false still overrides an inherited true. Without `pin` a nested scope still replaces name and action. `pin: true` makes a scope own the turn's identity: nested scopes may only add attributes while its name, action, on_trace and synchronous stay in force. The previous scope is restored exactly on exit, including when the block raises.
`correlation_tracer(synchronous: true, pin: true)` returns the `tracer:` lambda `ActiveAgent::Evals::Correlation` (activeagent >= 1.6.3) expects, `->(name, action:, attributes:, on_trace:, &block)`, opening a pinned, synchronous scope. activeagent is not a dependency; the contract is described in the YARD comment and README.
Core and adapter suites pass (72 tests, 223 assertions) and every Ruby file passes the Omakase lint configuration. Both gems are bumped to 0.3.2 (the core gem is republished unchanged so the two stay on one version). After merging, tag `v0.3.2` to publish both through the release workflow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011wAK6yEsed7nUqfvnDRfr5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
RubyLLM.with_agentsetThread.current[AGENT_KEY]to a fresh hash and restored the previous one on exit, and a turn captures whichever scope is active on its first round. So when a host wrapped an evaluation replay in an outer scope (viaActiveAgent::Evals::Correlation'stracer:— the evaluation's name,eval.*attributes, anon_tracethat records the trace id,synchronous: true) and the code under test opened its own inner scope around the actualchat.ask(its name and action,sparkle.role-style attributes), the inner scope silently replaced the outer one. The trace lost theeval.*attributes, theon_tracecallback never received the trace id, delivery went back to the background thread, and no spec noticed.The new rules
Nested
with_agentscopes compose:pin: truescopeattributeson_tracesynchronousname/actionsynchronous:now defaults tonil, meaning "inherit, else false", so an explicitsynchronous: falsestill overrides an inheritedtrue. A top-level scope still storesfalse, soagent[:synchronous]reads as before.pin: falseis a new keyword. A pinned scope owns the turn's identity: while it is active, a nestedwith_agentmay only add attributes, and it stays pinned however deep the nesting goes.ensurestill restores the previous scope exactly, including when the block raises.AGENT_KEYstays public.RubyLLM.correlation_tracer(synchronous: true, pin: true)returns->(name, action:, attributes:, on_trace:, &block), which callswith_agentwith those arguments plus the tracer'ssynchronousandpin. This is thetracer:thatActiveAgent::Evals::Correlation.new(agent_name:, tracer:)(activeagent >= 1.6.3) expects;on_tracereceives theTrace, which answers#trace_id. activeagent is not added as a dependency; the contract is described in the YARD comment and README.Changes
adapters/ruby_llm/lib/activeagents/telemetry/ruby_llm.rb:with_agentcomposes through a privatecompose_scope; newcorrelation_tracer.adapters/ruby_llm/test/test_nested_scopes.rb: merge + inheritance, name/action replacement when unpinned, pinned outer keeps identity/callback/delivery and merges attributes, pin holds two levels down, explicitsynchronous: falsebeats an inheritedtrue(delivery observed on the background thread), scope restored after the block and after a raise (asserted on the thread-local and on the reported traces), top-level defaults, the tracer opening a pinned synchronous scope and forwardingon_trace, and the tracer forwarding relaxed settings.adapters/ruby_llm/README.md: "Nested scopes" section with the table above and a "Correlating with ActiveAgent evaluations" section for the tracer.CHANGELOG.md: 0.3.2 entry. Both gems bumped to 0.3.2, following the 0.3.1 convention (the core gem has no code change and is republished so the two stay on one version).docs/{branches,issues,milestones,pull-requests}/nested-agent-scopes.md: the repo's per-feature notes.Validation
bundle exec rake test_all: core 39 runs / 100 assertions, adapter 33 runs / 123 assertions, 0 failures.rubocop-rails-omakaseconfig against every Ruby file in both gems: no offenses.bundle exec rake build_allbuilds both 0.3.2 gems.After merging
Tag
v0.3.2to publish both gems through the release workflow. Not done here.🤖 Generated with Claude Code
https://claude.ai/code/session_011wAK6yEsed7nUqfvnDRfr5
Generated by Claude Code