Skip to content

feat(ruby_llm): compose nested with_agent scopes and add correlation_tracer - #7

Draft
TonsOfFun wants to merge 1 commit into
mainfrom
claude/nested-agent-scopes
Draft

TonsOfFun wants to merge 1 commit into
mainfrom
claude/nested-agent-scopes

Conversation

@TonsOfFun

Copy link
Copy Markdown
Contributor

The problem

RubyLLM.with_agent set Thread.current[AGENT_KEY] to a fresh hash and restored the previous one on exit, and a turn captures whichever scope is active on its first round. So when a host wrapped an evaluation replay in an outer scope (via ActiveAgent::Evals::Correlation's tracer: — the evaluation's name, eval.* attributes, an on_trace that records the trace id, synchronous: true) and the code under test opened its own inner scope around the actual chat.ask (its name and action, sparkle.role-style attributes), the inner scope silently replaced the outer one. The trace lost the eval.* attributes, the on_trace callback never received the trace id, delivery went back to the background thread, and no spec noticed.

The new rules

Nested with_agent scopes compose:

Setting Nested scope inside a plain scope Nested scope inside a pin: true scope
attributes Merged: enclosing first, nested keys win on collision Merged the same way
on_trace Inherited unless the nested call passes its own The pinned scope's callback stays in force
synchronous Inherited unless the nested call passes its own The pinned scope's setting stays in force
name / action Replaced by the nested scope (as today) The pinned scope's identity stays in force
  • synchronous: now defaults to nil, meaning "inherit, else false", so an explicit synchronous: false still overrides an inherited true. A top-level scope still stores false, so agent[:synchronous] reads as before.
  • pin: false is a new keyword. A pinned scope owns the turn's identity: while it is active, a nested with_agent may only add attributes, and it stays pinned however deep the nesting goes.
  • ensure still restores the previous scope exactly, including when the block raises. AGENT_KEY stays public.
  • RubyLLM.correlation_tracer(synchronous: true, pin: true) returns ->(name, action:, attributes:, on_trace:, &block), which calls with_agent with those arguments plus the tracer's synchronous and pin. This is the tracer: that ActiveAgent::Evals::Correlation.new(agent_name:, tracer:) (activeagent >= 1.6.3) expects; on_trace receives the Trace, which answers #trace_id. activeagent is not added as a dependency; the contract is described in the YARD comment and README.

Changes

  • adapters/ruby_llm/lib/activeagents/telemetry/ruby_llm.rb: with_agent composes through a private compose_scope; new correlation_tracer.
  • adapters/ruby_llm/test/test_nested_scopes.rb: merge + inheritance, name/action replacement when unpinned, pinned outer keeps identity/callback/delivery and merges attributes, pin holds two levels down, explicit synchronous: false beats an inherited true (delivery observed on the background thread), scope restored after the block and after a raise (asserted on the thread-local and on the reported traces), top-level defaults, the tracer opening a pinned synchronous scope and forwarding on_trace, and the tracer forwarding relaxed settings.
  • adapters/ruby_llm/README.md: "Nested scopes" section with the table above and a "Correlating with ActiveAgent evaluations" section for the tracer.
  • CHANGELOG.md: 0.3.2 entry. Both gems bumped to 0.3.2, following the 0.3.1 convention (the core gem has no code change and is republished so the two stay on one version).
  • docs/{branches,issues,milestones,pull-requests}/nested-agent-scopes.md: the repo's per-feature notes.

Validation

  • bundle exec rake test_all: core 39 runs / 100 assertions, adapter 33 runs / 123 assertions, 0 failures.
  • rubocop with the rubocop-rails-omakase config against every Ruby file in both gems: no offenses.
  • bundle exec rake build_all builds both 0.3.2 gems.

After merging

Tag v0.3.2 to publish both gems through the release workflow. Not done here.

🤖 Generated with Claude Code

https://claude.ai/code/session_011wAK6yEsed7nUqfvnDRfr5


Generated by Claude Code

…tracer

A with_agent scope opened inside another one replaced it outright, and a turn captures whichever scope is active on its first round. So when an evaluation replay wrapped the code under test in an outer scope carrying `eval.*` attributes, an `on_trace` callback and `synchronous: true`, and that code opened its own scope around the actual `chat.ask`, the trace lost the evaluation attributes, the callback never received the trace id and delivery went back to the background thread, with no spec noticing.

Nested scopes now compose: attributes merge (enclosing first, nested keys win), `on_trace` and `synchronous` are inherited unless the nested call passes its own, and `synchronous:` defaults to nil ("inherit, else false") so an explicit false still overrides an inherited true. Without `pin` a nested scope still replaces name and action. `pin: true` makes a scope own the turn's identity: nested scopes may only add attributes while its name, action, on_trace and synchronous stay in force. The previous scope is restored exactly on exit, including when the block raises.

`correlation_tracer(synchronous: true, pin: true)` returns the `tracer:` lambda `ActiveAgent::Evals::Correlation` (activeagent >= 1.6.3) expects, `->(name, action:, attributes:, on_trace:, &block)`, opening a pinned, synchronous scope. activeagent is not a dependency; the contract is described in the YARD comment and README.

Core and adapter suites pass (72 tests, 223 assertions) and every Ruby file passes the Omakase lint configuration. Both gems are bumped to 0.3.2 (the core gem is republished unchanged so the two stay on one version). After merging, tag `v0.3.2` to publish both through the release workflow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011wAK6yEsed7nUqfvnDRfr5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants