chore(release): activeagent and actionagent 1.6.3

CHANGELOG.md

@@ -7,8 +7,89 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.6.3] - 2026-09-18
+
+Releases `activeagent` and `actionagent` 1.6.3 from one tag.
+
+A release about telling the truth on the screens that report what happened.
+The context meter now divides the provider's own `prompt_tokens` among its
+segments instead of subtracting estimates from it, and sizes each piece —
+tool schemas, MCP schemas, instructions and the transcript — before the span
+clips it for storage; previously a trace with dense tool schemas showed a
+large message history that was never sent. An adapted replay is metered as
+one execution like any other, so a host that supplies its own runtime is no
+longer silently uncounted, and a spec naming a provider the agent cannot
+serve now fails before the replay rather than reaching it.
+
+Three seams hosts were reaching around become API. `ActiveAgent::Evals::Correlation`
+joins `Runner`'s `around_evaluation:` hook to a telemetry backend's trace
+scope, so a report row links back to the conversation behind it. A `Judge`
+block that accepts `kind:` is told whether it is scoring, recommending or
+writing the verdict, instead of matching on the gem's own instruction prose.
+`Agent#generations` replaces the polymorphic join hosts were copying out of a
+private service method.
+
+Upgrading: no migration, and nothing that already worked changes. The judge
+keyword reaches only a block that asks for it, so existing judges are
+untouched; `Evaluation#replace_scenarios!` keeps `:destroy` as its default.
+Adapters should drop any `ActionAgent.record_usage` call of their own, which
+now double-counts, and any provider allow-list check of their own, which is
+now dead code.
+
+### Added
+
+- The prompt span records how large the tool schemas actually are, as
+  `prompt.input.tools.tokens`, `prompt.input.mcp_tools.tokens`,
+  `prompt.input.instructions.tokens` and `prompt.input.messages.tokens`. The
+  transcript's size is measured before the span trims the history to the turns
+  that fit, the others before their content is clipped. The content attributes
+  beside them are previews clipped for storage — and on the SDK path the tool
+  attribute is a roster of names and parameter keys, several times smaller than
+  the schema the model is sent — so a reader that sized the context from one
+  understated tool pressure badly.
+- MCP tool schemas are attributed apart from the toolbox's, so the context meter
+  can name which half fills the window.
+- `Evaluation#replace_scenarios!` takes `on_removed:` — `:destroy` (the
+  default, unchanged) or `:disable`, which keeps a scenario the suite no
+  longer names as `enabled: false` so earlier runs' results still resolve.
+- **An evaluation's traces link back to the result that caused them.**
+  `ActiveAgent::Evals::Correlation` joins two APIs the module already had but
+  never connected: `Runner`'s `around_evaluation:` hook and its `metadata:`
+  run identity, and a telemetry backend's per-block agent scope. A run mints a
+  `run_id`, each evaluation a `result_id`, and both ride every trace opened
+  inside them as `eval.`-prefixed attributes; the trace ids travel the other
+  way onto `result.replay.metadata` — `trace_id` for the replay,
+  `judge_trace_ids` for the judge calls that graded it, with a run-level
+  verdict landing on the run metadata the Report carries rather than on
+  whichever result was evaluated last. The tracer is injected, so the module
+  takes on no telemetry dependency and `require "active_agent/evals"` still
+  loads on its own. Hand the object to `Runner.new(around_evaluation:)`
+  directly; a plain lambda there keeps working unchanged.
+- A `Judge` block that accepts `kind:` is told which of the judge's three calls
+  it is serving — `:score`, `:recommend` or `:verdict` — so a host can trace,
+  budget or model them separately. Previously the only signal was the
+  `instructions` string, so hosts matched against the gem's own
+  `RECOMMEND_INSTRUCTIONS` / `VERDICT_INSTRUCTIONS` constants; rewording one
+  then sent every such host quietly down its `else` branch, mislabelling traces
+  rather than failing. The keyword reaches only a block that names it or
+  collects `**`, so judges taking `instructions:` and `prompt:` are unaffected.
+  (#462)
+- `Agent#generations` reads the generations recorded against an agent, with
+  `Agent#agent_contexts` beside it. Generations hang off `AgentContext`
+  polymorphically, so reaching them meant hand-writing that join — the engine
+  did it itself in a private service method a host could not reuse, which now
+  uses the association instead. Destroying an agent still leaves its contexts
+  alone, as it always has. (#464)
+
 ### Fixed
 
+- The dashboard's context meter divides the provider's own `prompt_tokens`
+  among its segments instead of subtracting its estimates from it. Charging the
+  difference to one segment made "Messages" absorb the whole approximation
+  error, so a trace with dense JSON tool schemas read as a large message history
+  that was never sent. The transcript is one of the divided segments: dividing
+  only the rest would hand its share to the segments that remained, so a long
+  conversation reported an enormous system prompt and no history at all.
 - A host that supplies a `scenario_evaluation_adapter_resolver` now has its
   replays metered as executions, one per scenario x model, the same unit the
   default path records. Previously an adapted replay was counted only if the
@@ -17,13 +98,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   serve fails with `ArgumentError` before any replay runs. A spec handed back
   as a Hash naming both `provider` and `model` bypassed the `providers:`
   allow-list, so the run reached the replay with a provider nothing serves.
-
-### Added
-
-- `Evaluation#replace_scenarios!` takes `on_removed:` — `:destroy` (the
-  default, unchanged) or `:disable`, which keeps a scenario the suite no
-  longer names as `enabled: false` so earlier runs' results still resolve.
-
 ## [1.6.2] - 2026-09-16
 
 Releases `activeagent` and `actionagent` 1.6.2 from one tag.

actionagent/app/models/action_agent/agent.rb

@@ -11,6 +11,14 @@ class ObservedAgentError < StandardError; end
     has_many :agent_runs, dependent: :destroy
     has_many :evaluations, dependent: :destroy
     has_many :agent_memories, as: :memorable, dependent: :destroy
+    # Generations hang off AgentContext polymorphically, which is an
+    # implementation detail of how contexts are modelled — so without these a
+    # host that wants an agent's recorded history writes that join itself and is
+    # coupled to the shape. Deliberately no `dependent:` on the contexts: the
+    # association is added to read them, and destroying an agent has never taken
+    # its conversations with it. Making it do so is a separate call.
+    has_many :agent_contexts, as: :contextable
+    has_many :generations, through: :agent_contexts
 
     # Polymorphic rows (agent_memories, agent_contexts) store this string.
     # A host app that grew these tables under its own Agent constant keeps

actionagent/app/services/action_agent/agent_execution_service.rb

@@ -33,9 +33,13 @@ class ProviderNotConfiguredError < StandardError; end
     # so a multi-gigabyte log named .csv costs a fixed slice of memory rather
     # than its whole size. Four bytes per character is UTF-8's worst case.
     ATTACHMENT_TEXT_BYTE_LIMIT = ATTACHMENT_TEXT_LIMIT * 4
+    # What one prompt-span attribute stores. The value is a preview for reading,
+    # so it is clipped; a size that has to stay exact travels as its own
+    # `*.tokens` attribute instead.
+    PROMPT_SPAN_ATTRIBUTE_LIMIT = 6000
     # The prompt span records the transcript, not the data URIs; keep the
     # whole serialized list within the same budget as the other attributes.
-    PROMPT_SPAN_MESSAGE_LIMIT = 6000
+    PROMPT_SPAN_MESSAGE_LIMIT = PROMPT_SPAN_ATTRIBUTE_LIMIT
     # Prior turns sent with a pinned conversation: the most recent ones,
     # trimmed oldest-first to a character budget.
     HISTORY_TURN_LIMIT = 40
@@ -164,11 +168,19 @@ def call
     def record_prompt_span(root_span)
       span = root_span.add_span("agent.prompt", span_type: :prompt)
       if composed_instructions.present?
-        span.set_attribute("prompt.input.instructions", composed_instructions.to_s.byteslice(0, 6000).to_s.scrub)
-      end
-      if tool_schemas.present?
-        span.set_attribute("prompt.input.tools", tool_schemas.to_json.byteslice(0, 6000).to_s.scrub)
+        instructions = composed_instructions.to_s
+        span.set_attribute("prompt.input.instructions", instructions.byteslice(0, PROMPT_SPAN_ATTRIBUTE_LIMIT).to_s.scrub)
+        span.set_attribute("prompt.input.instructions.tokens", estimated_tokens(instructions))
       end
+      # MCP and toolbox schemas are attributed separately so the meter can name
+      # which half fills the window, and each carries its size. The content
+      # attributes are truncated previews for reading: sizing the context from
+      # one understates it by whatever the clip dropped, which for a twelve-tool
+      # agent is most of the schema.
+      record_tool_schema_attributes(span)
+      full_transcript_json = prompt_turn[:transcript].map do |message|
+        { role: message[:role], content: message[:content].to_s }
+      end.to_json
       transcript = prompt_turn[:transcript].map do |message|
         { role: message[:role], content: message[:content].to_s.byteslice(0, 4000).to_s.scrub }
       end
@@ -181,6 +193,13 @@ def record_prompt_span(root_span)
       end
       span.set_attribute("prompt.input.messages", serialized)
       span.set_attribute("messages.count", transcript.size)
+      # The stored attribute is the tail of the history that fit, so its size is
+      # not the transcript's. The meter apportions the provider's prompt_tokens
+      # across the segments it can size, and a transcript missing from that set
+      # is not merely imprecise: the segments that remain are scaled up to cover
+      # it, so a long conversation reads as an enormous system prompt. Measured
+      # over the full turn, before either the per-message clip or the trim.
+      span.set_attribute("prompt.input.messages.tokens", estimated_tokens(full_transcript_json))
       span.finish
     rescue StandardError => e
       Rails.logger.warn("[AgentExecutionService] prompt span failed: #{e.message}")
@@ -385,6 +404,30 @@ def mcp_dispatcher
       @mcp_dispatcher ||= MCPToolDispatcher.new(@agent_record)
     end
 
+    # Splits the offered schemas the way `tool_schemas` assembles them, so the
+    # span reports what the model was actually sent: nothing for a mock run, and
+    # one MCP round trip rather than a second one for telemetry.
+    def record_tool_schema_attributes(span)
+      mcp_definitions, toolbox_definitions = tool_schema_halves
+      {
+        "prompt.input.tools" => toolbox_definitions,
+        "prompt.input.mcp_tools" => mcp_definitions
+      }.each do |key, definitions|
+        next if definitions.blank?
+
+        json = definitions.to_json
+        span.set_attribute(key, json.byteslice(0, PROMPT_SPAN_ATTRIBUTE_LIMIT).to_s.scrub)
+        span.set_attribute("#{key}.tokens", estimated_tokens(json))
+      end
+    end
+
+    # ~4 chars/token, the same approximation the context meter applies to content
+    # it sizes itself. Taken before truncation, so the meter reads the whole
+    # schema rather than the preview the attribute stores.
+    def estimated_tokens(text)
+      (text.length / 4.0).round
+    end
+
     def call_agent(slug:, message:)
       depth = Thread.current[:agent_call_depth].to_i
       return { error: "call_agent depth limit (#{MAX_CALL_DEPTH}) reached" } if depth >= MAX_CALL_DEPTH
@@ -562,13 +605,21 @@ def generate!
     # server-side implementations (none for mock runs — the mock provider
     # doesn't do tool calling).
     def tool_schemas
-      return [] if provider == :mock
-
-      # The agent's own MCP servers describe their tools; the toolbox describes
-      # the rest. Without the first half a tool the agent declares is never
-      # offered to the model, which then answers from memory instead of calling
-      # it.
-      mcp_dispatcher.tool_definitions + AgentToolbox.definitions_for(@agent_record.tools)
+      mcp_definitions, toolbox_definitions = tool_schema_halves
+      mcp_definitions + toolbox_definitions
+    end
+
+    # The agent's own MCP servers describe their tools; the toolbox describes the
+    # rest. Without the first half a tool the agent declares is never offered to
+    # the model, which then answers from memory instead of calling it. Memoized
+    # because listing a server's tools is a request to that server.
+    def tool_schema_halves
+      @tool_schema_halves ||=
+        if provider == :mock
+          [ [], [] ]
+        else
+          [ mcp_dispatcher.tool_definitions, AgentToolbox.definitions_for(@agent_record.tools) ]
+        end
     end
 
     # Persists the tool interaction stream to the solid_agent conversation

actionagent/app/services/action_agent/evaluation_runner_service.rb

@@ -149,9 +149,7 @@ def passed_count(per_sample_scores)
     end
 
     def sample_generations(model: nil)
-      scope = AgentGeneration
-        .joins(:agent_context)
-        .where(AgentContext.table_name => { contextable: @evaluation.agent })
+      scope = @evaluation.agent.generations
       scope = scope.where(model: model) if model
       scope.order(created_at: :desc).limit(@evaluation.sample_size).to_a
     end

actionagent/frontend/components/dashboard/ContextMeter.jsx

@@ -1,5 +1,8 @@
 import React, { useState } from 'react';
 
+// Re-exported for the views that size their own segments.
+export { contextWindowFor, estimateTokens } from '../../utils/traceContext.mjs';
+
 // Context-window pressure meter (from the Active Agent design system).
 // A segmented bar of what occupies the model's context — messages, tool
 // results, instructions, tool/MCP schemas, generated output — against the
@@ -17,24 +20,7 @@ const SEGMENT_COLORS = {
   output: '#7c3aed',
 };
 
-// Context-window sizes by model family. The provider reports real token
-// counts; the window is the constraint we hold them against.
-export const contextWindowFor = (model) => {
-  const name = (model || '').toLowerCase();
-  if (name.includes('claude')) return 200000;
-  if (name.includes('gemini')) return 1000000;
-  if (name.includes('llama')) return 131072;
-  if (name.includes('gpt-4o') || name.includes('gpt-4-turbo') || name.includes('gpt-4.1')) return 128000;
-  if (name.includes('gpt-5')) return 400000;
-  return 128000;
-};
 
-// ~4 chars/token, for estimating segment sizes from recorded content.
-export const estimateTokens = (value) => {
-  if (value == null) return 0;
-  const text = typeof value === 'string' ? value : JSON.stringify(value);
-  return Math.round(text.length / 4);
-};
 
 export const formatTokenCount = (value) => {
   if (value == null) return '—';

actionagent/frontend/components/dashboard/TraceDetail.jsx

@@ -1,6 +1,7 @@
 import React, { useCallback, useEffect, useState } from 'react';
 import { TYPOGRAPHY } from '../../utils/designTokens';
-import ContextMeter, { contextWindowFor, estimateTokens } from './ContextMeter';
+import ContextMeter from './ContextMeter';
+import { traceContext } from '../../utils/traceContext.mjs';
 import InteractionStream from './InteractionStream';
 import SpanWaterfall, {
   SPAN_SORTS,
@@ -9,6 +10,9 @@ import SpanWaterfall, {
 } from './SpanWaterfall';
 import { SegmentedControl, telemetryColors } from './TelemetryObject';
 
+// TraceCard reads a trace's context pressure through this module.
+export { traceContext };
+
 // Everything one trace has to say, in one panel: its spans as a waterfall or
 // its run as a conversation, what it held in context, where the wall clock
 // went, and what it spent. Rendered under a trace row in the Traces view and
@@ -20,67 +24,6 @@ const TRACE_VIEWS = [
   { id: 'conversation', label: 'Conversation', hint: 'The run as a message stream — prompt, tool calls, response' },
 ];
 
-// Context pressure: what the biggest generation in this trace held against the
-// model's window. Segment sizes are estimated from recorded content (~4
-// chars/token); the input/output totals are the provider's real counts.
-export const traceContext = (trace) => {
-  const spans = trace?.spans || [];
-  let peak = null;
-  for (const span of spans) {
-    const tokens = span.tokens || {};
-    const total = (tokens.input || 0) + (tokens.output || 0);
-    if (total > 0 && (!peak || total > peak.total)) {
-      peak = {
-        input: tokens.input || 0,
-        output: tokens.output || 0,
-        thinking: tokens.thinking || 0,
-        cached: tokens.cached || 0,
-        total,
-      };
-    }
-  }
-  if (!peak) return null;
-
-  const attr = (key) => {
-    for (const span of spans) {
-      const value = (span.attributes || {})[key];
-      if (value) return value;
-    }
-    return null;
-  };
-  // Both telemetry shapes: ActiveAgent SDK (prompt.input.*, tool.input/
-  // output.*) and the RubyLLM adapter (llm.instructions/tools,
-  // tool.arguments/result).
-  const instructions = estimateTokens(attr('prompt.input.instructions') || attr('llm.instructions'));
-  const toolSchemas = estimateTokens(attr('prompt.input.tools') || attr('llm.tools'));
-  const mcpSchemas = estimateTokens(attr('prompt.input.mcp_tools'));
-  let toolResults = 0;
-  for (const span of spans) {
-    const attrs = span.attributes || {};
-    const result = attrs['tool.output.result'] || attrs['tool.result'];
-    const args = attrs['tool.input.args'] || attrs['tool.arguments'];
-    if (result) toolResults += estimateTokens(result);
-    if (args) toolResults += estimateTokens(args);
-  }
-  toolResults = Math.min(toolResults, peak.input);
-  const messages = Math.max(peak.input - instructions - toolSchemas - mcpSchemas - toolResults, 0);
-
-  return {
-    used: peak.total,
-    limit: contextWindowFor(trace.model),
-    cached: peak.cached,
-    thinking: peak.thinking,
-    segments: [
-      { key: 'messages', label: 'Messages', tokens: messages },
-      { key: 'tool_results', label: 'Tool results', tokens: toolResults },
-      { key: 'instructions', label: 'Instructions', tokens: instructions },
-      { key: 'tool_schemas', label: 'Tool schemas', tokens: toolSchemas },
-      { key: 'mcp_schemas', label: 'MCP tool schemas', tokens: mcpSchemas },
-      { key: 'output', label: 'Generated output', tokens: peak.output },
-    ],
-  };
-};
-
 // Generation vs tool time for one trace: how much of the wall clock went to
 // the LLM, to each tool, and to unattributed overhead. Tool calls run inside
 // the provider's generate loop, so their time is subtracted from the llm span
@@ -245,7 +188,7 @@ export default function TraceDetail({ trace, darkMode, compact = false, showConv
 
       {context && (
         <div className="mt-3">
-          <ContextMeter {...context} label="Context pressure" estimated darkMode={darkMode} />
+          <ContextMeter {...context} label="Context pressure" darkMode={darkMode} />
         </div>
       )}