From 8ac9060997a3a132984e3d0dd077f17fe5c215ac Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Thu, 17 Sep 2026 20:36:57 -0700 Subject: [PATCH 1/7] feat: let the engine run host agent classes, and teach SchemaTools about enums MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four gaps a host hits when its agents live in code and the dashboard mirrors them. Each is something the framework has the information to do itself, and each was found by reading a real integration (support-hub) against the engine. Rails enums reached the model as a bare integer. SchemaGenerator derives an enum constraint from inclusion validators and never consulted defined_enums, so a `status` column was offered as {type: "integer"} with no labels — and every host explained the mapping in prose instead. Worse, both failure modes were silent: `status: "pending"` returned {count: 0}, indistinguishable from "none match", and the range form the schema advertised on an enum meant `status: {gt: 1}` returned a confident, arbitrary count. range_predicates! already rejects an unknown operator for exactly this reason; enum values now get the same treatment. The engine could not run an agent that exists in code. AgentExecutionService builds an anonymous ActiveAgent::Base subclass from the record's `tools` and `instructions` columns — right for a dashboard-authored agent, which is rows and has no class, but a mirrored agent's SchemaTools rosters, delegations and rendered instructions cannot be expressed that way. A host therefore flattens its agents to sync them, and the dashboard evaluates a lookalike rather than what production runs. Both runtimes now coexist: ActionAgent.run_host_agent_ classes (default false) runs the real class when one resolves, and anything else — no class name, a stale name, a name that is not an agent — falls back to the dynamic runtime rather than failing. AgentSync mirrors classes into Agent records, setting the agent_class_name AgentRelease already expects a host to have written, with the code owning what an agent is and the operator owning how it runs. rendered_instructions makes the text reachable without `send` into a private renderer. Full suite: 2040 runs, 0 failures; 253 errors, all pre-existing missing-API-key errors in docs/integration tests (baseline on the same checkout: 261). rubocop clean across 555 files. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 33 +++++ .../action_agent/agent_execution_service.rb | 49 +++++++ .../app/services/action_agent/agent_sync.rb | 126 ++++++++++++++++++ actionagent/lib/action_agent.rb | 16 +++ actionagent/test/agent_sync_test.rb | 69 ++++++++++ .../test/host_agent_class_execution_test.rb | 89 +++++++++++++ lib/active_agent/concerns/view.rb | 23 ++++ lib/active_agent/schema_tools.rb | 58 ++++++++ .../instructions.md.erb | 1 + test/dummy/db/schema.rb | 1 + test/rendered_instructions_test.rb | 32 +++++ test/schema_tools_test.rb | 63 +++++++++ 12 files changed, 560 insertions(+) create mode 100644 actionagent/app/services/action_agent/agent_sync.rb create mode 100644 actionagent/test/agent_sync_test.rb create mode 100644 actionagent/test/host_agent_class_execution_test.rb create mode 100644 test/dummy/app/views/rendered_instructions_probe_agent/instructions.md.erb create mode 100644 test/rendered_instructions_test.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index ba736f7e..2b37396f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- `ActiveAgent::Base.rendered_instructions` renders an agent's instructions + outside a generation, for a dashboard mirroring the class and for tests + asserting what a model is told. Both otherwise reached a private renderer + through `send`. +- `ActionAgent::AgentSync` mirrors host agent classes into dashboard `Agent` + records, setting the `agent_class_name` that `AgentRelease` already expects a + host to have written. The code owns what an agent is (name, description, + instructions, tools — rewritten every sync); the operator owns how it runs + (provider, model, status — set on create and preserved), so a model chosen in + the dashboard survives the next deploy. +- `ActionAgent.run_host_agent_classes` (default `false`) runs an agent that + mirrors a host class as that class, rather than as one rebuilt from the + record's `tools` and `instructions` columns. Dashboard-authored agents, which + name no class, keep using the dynamic runtime either way; a class name that no + longer resolves falls back to it rather than failing the run. + +### Fixed + +- A Rails enum is offered to the model as its names (`{type: "string", enum: + [...]}`) instead of the integer backing it. `SchemaGenerator` reads enums from + inclusion validators and never consulted `defined_enums`, so a `status` column + reached the model as a bare integer with no labels — leaving every host to + explain the mapping in prose. +- A filter value outside an enum is rejected, naming the valid values, instead + of matching no rows. `status: "pending"` returned `{count: 0}`, which an agent + reports as a fact — indistinguishable from "none match". Same reasoning as the + unknown-operator rejection in `range_predicates!`. +- An enum is no longer offered the range form. Its integer backing is a + declaration-order artefact, so `status: {gt: 1}` was a meaningless filter that + still returned a confident count. + ## [1.6.3] - 2026-09-17 Releases `activeagent` and `actionagent` 1.6.3 from one tag. diff --git a/actionagent/app/services/action_agent/agent_execution_service.rb b/actionagent/app/services/action_agent/agent_execution_service.rb index 6fc8a47a..82a2fc06 100644 --- a/actionagent/app/services/action_agent/agent_execution_service.rb +++ b/actionagent/app/services/action_agent/agent_execution_service.rb @@ -500,6 +500,21 @@ def generate! tool_definitions = tool_schemas service = self + # A dashboard-authored agent has no Ruby class — it is rows: a tool + # selection, instructions typed in the builder. That is the common case + # and the runtime below builds a class for it. + # + # An agent mirrored from host code is the other case: the class exists, + # already declares its own tools (SchemaTools rosters, delegations) and + # renders its own instructions, and none of that is reachable through + # `tools` + `instructions` columns. Running the real class keeps the + # dashboard evaluating what production runs, instead of a rebuilt + # lookalike. Both runtimes stay; which one applies is decided by whether + # the class resolves. + if (host_class = resolved_host_class) + return run_host_class(host_class, actor: actor, action: action, run_trace_id: run_trace_id) + end + agent_class = Class.new(ActiveAgent::Base) do # SolidAgent persists contexts under self.class.name; anonymous # classes would fail its agent_name presence validation. @@ -677,6 +692,40 @@ def record_tool_spans(root_span, response) end end + # The host class this agent mirrors, when it names one that resolves to a + # runnable ActiveAgent::Base subclass. Anything else — no class name, a + # class that no longer exists, a name that resolves to something else — is + # nil, and the dynamic runtime handles the record as before. + # + # @return [Class, nil] + def resolved_host_class + return nil unless ActionAgent.run_host_agent_classes + + name = @agent_record.agent_class_name.presence + return nil if name.blank? + + klass = name.safe_constantize + klass if klass.is_a?(Class) && klass < ActiveAgent::Base + end + + # Runs the host's own class. Its tools, delegations and instructions come + # from the code, so the engine supplies only what is the run's business: + # the caller, and the trace to correlate against. + def run_host_class(klass, actor:, action:, run_trace_id:) + generation = klass.as(actor).public_send(action, **host_action_arguments(klass, action)) + generation.prompt_options[:trace_id] = run_trace_id if generation.respond_to?(:prompt_options) + generation.generate_now + end + + # A code agent's action takes named arguments (`ask(question:)`), so the + # run's message is passed under the action's own keyword rather than as a + # bare message the signature would reject. + def host_action_arguments(klass, action) + contract = klass.try(:delegation_contracts)&.dig(action.to_sym) + keyword = contract&.try(:parameters)&.keys&.first + keyword ? { keyword.to_sym => user_text } : {} + end + def provider_available?(name) # The gem's mock provider is a test double: accepted only in the test # environment so app runs can never store fabricated output. diff --git a/actionagent/app/services/action_agent/agent_sync.rb b/actionagent/app/services/action_agent/agent_sync.rb new file mode 100644 index 00000000..36621b03 --- /dev/null +++ b/actionagent/app/services/action_agent/agent_sync.rb @@ -0,0 +1,126 @@ +# frozen_string_literal: true + +module ActionAgent + # Mirrors host ActiveAgent classes into Agent records, so the dashboard can + # run, evaluate and release the agents an app already has in code. + # + # The engine reads an agent's identity from the class rather than asking the + # host to restate it: name, description and tool roster all come from the + # class, and re-running the sync updates each record in place. `AgentRelease` + # already expects `agent_class_name` to be set by "whatever syncs its + # ActiveAgent classes into Agent records" — this is that, so a host no longer + # has to write it. + # + # The split is deliberate and is the reason this is safe to run on deploy: + # + # * **The code owns what an agent is** — name, description, instructions, + # tools. Rewritten on every sync, so it cannot drift from the class. + # * **The operator owns how it runs** — provider, model, status. Set once on + # create and never touched again, so a model chosen in the dashboard + # survives the next deploy. + # + # ActionAgent::AgentSync.call(RecordAgent.all, owner: owner) + # + # @see AgentRelease which cuts a version per synced agent + class AgentSync + Row = Struct.new(:agent, :created, :skipped, keyword_init: true) + Result = Struct.new(:rows, :errors, keyword_init: true) do + def success? = errors.blank? + def agents = rows.filter_map(&:agent) + def created = rows.select(&:created) + def skipped = rows.select(&:skipped) + end + + # @param agents [Array] ActiveAgent::Base subclasses + # @param owner [Object] the record agents and API keys scope to + # @param provider [String, Symbol, nil] defaults to the class's own + # @param model [String, nil] defaults to the class's own + def self.call(agents, owner:, provider: nil, model: nil) + new(agents, owner: owner, provider: provider, model: model).call + end + + def initialize(agents, owner:, provider: nil, model: nil) + @agents = Array(agents) + @owner = owner + @provider = provider + @model = model + end + + # @return [Result] + def call + return Result.new(rows: [], errors: "An owner is required: the engine scopes agents to one.") if @owner.nil? + + rows = Agent.transaction { @agents.map { |klass| upsert(klass) } } + Result.new(rows: rows, errors: nil) + rescue ActiveRecord::RecordInvalid => e + Result.new(rows: [], errors: e.record.errors.full_messages.join(", ")) + end + + private + + def upsert(klass) + unless klass.respond_to?(:prompt_options) + return Row.new(agent: nil, created: false, skipped: "#{klass} is not an ActiveAgent::Base subclass") + end + + provider = resolved_provider(klass) + model = resolved_model(klass) + if provider.blank? || model.blank? + return Row.new(agent: nil, created: false, skipped: "#{klass} has no provider/model configured") + end + + agent = Agent.find_or_initialize_by(slug: self.class.slug_for(klass)) + created = agent.new_record? + if created + agent.owner = @owner + agent.provider = provider + agent.model = model + agent.status = :active + end + + agent.assign_attributes( + name: klass.name.titleize, + agent_class_name: klass.name, + description: description_for(klass), + instructions: instructions_for(klass), + tools: tool_names_for(klass) + ) + agent.save! + Row.new(agent: agent, created: created, skipped: nil) + end + + # "TicketAgent" -> "ticket-agent", the slug an MCP client sees as + # run_ticket-agent. A namespaced class flattens its separators, because + # Agent validates slugs as /\A[a-z0-9\-_]+\z/ — "Billing::TicketAgent" + # becomes "billing-ticket-agent". + def self.slug_for(klass) + klass.name.underscore.tr("/", "-").tr("_", "-") + end + + # An agent's own description if it declares one (a delegation contract is + # where an agent says what it answers), else its titleized name. + def description_for(klass) + contract = klass.try(:delegation_contracts)&.values&.first + contract&.try(:description).presence || klass.name.titleize + end + + # The rendered instructions, so the dashboard record runs on the same text + # the class does rather than a hand-maintained copy. + def instructions_for(klass) + klass.try(:rendered_instructions).presence + end + + def tool_names_for(klass) + names = klass.try(:tool_names) + Array(names).map(&:to_s) + end + + def resolved_provider(klass) + (@provider || klass.prompt_options[:service]).to_s.downcase.presence + end + + def resolved_model(klass) + (@model || klass.prompt_options[:model]).presence + end + end +end diff --git a/actionagent/lib/action_agent.rb b/actionagent/lib/action_agent.rb index c37e035f..ad28e936 100644 --- a/actionagent/lib/action_agent.rb +++ b/actionagent/lib/action_agent.rb @@ -256,6 +256,21 @@ def deprecator # @return [Boolean] attr_accessor :execution_enabled + # Whether a run of an agent that mirrors a host class executes that class, + # instead of the class the engine builds from the record's `tools` and + # `instructions` columns. + # + # Off by default: it changes what a run of a mirrored agent executes, and + # a host that has tuned its dashboard records around the dynamic runtime + # should opt in deliberately. Dashboard-authored agents — the ones with no + # `agent_class_name` — are unaffected either way. + # + # On, a mirrored agent runs its real tools, delegations and instructions, + # so an evaluation scores the agent production runs rather than a + # flattened copy of it. + # @return [Boolean] + attr_accessor :run_host_agent_classes + # Whether the "Ask ActiveAgents" assistant is available. # # The assistant is a tool for developing and CI-ing agents: it sends @@ -544,6 +559,7 @@ def reset! @provider_credentials_resolver = nil @sandbox_backends = {} @execution_enabled = true + @run_host_agent_classes = false @assistant_enabled = nil @scenario_evaluation_adapter_resolver = nil diff --git a/actionagent/test/agent_sync_test.rb b/actionagent/test/agent_sync_test.rb new file mode 100644 index 00000000..0a3ca87d --- /dev/null +++ b/actionagent/test/agent_sync_test.rb @@ -0,0 +1,69 @@ +# frozen_string_literal: true + +require "test_helper" + +# Mirroring host agent classes into Agent records — the step that lets the +# dashboard run, evaluate and release the agents an app already has in code. +class AgentSyncTest < ActiveSupport::TestCase + class InvoiceAgent < ApplicationAgent + generate_with :mock, model: "mock-1" + + def ask + prompt(message: "hi") + end + end + + setup do + ActionAgent::Agent.delete_all + end + + teardown do + ActionAgent::Agent.delete_all + end + + test "creates a record carrying the class name, so releases can resolve it" do + result = ActionAgent::AgentSync.call([ InvoiceAgent ], owner: nil_owner) + + assert result.success?, result.errors + agent = result.agents.sole + assert_equal "AgentSyncTest::InvoiceAgent", agent.agent_class_name + assert_equal "agent-sync-test-invoice-agent", agent.slug + assert_equal "mock", agent.provider + assert_equal "mock-1", agent.model + end + + test "re-running updates in place rather than duplicating" do + ActionAgent::AgentSync.call([ InvoiceAgent ], owner: nil_owner) + + assert_difference -> { ActionAgent::Agent.count }, 0 do + ActionAgent::AgentSync.call([ InvoiceAgent ], owner: nil_owner) + end + end + + test "the operator's provider and model survive a re-sync" do + agent = ActionAgent::AgentSync.call([ InvoiceAgent ], owner: nil_owner).agents.sole + agent.update!(model: "operator-choice") + + ActionAgent::AgentSync.call([ InvoiceAgent ], owner: nil_owner) + + # The code owns what an agent is; the operator owns how it runs. A model + # picked in the dashboard must survive the next deploy's sync. + assert_equal "operator-choice", agent.reload.model + end + + test "a class that is not an agent is skipped, not raised" do + result = ActionAgent::AgentSync.call([ String ], owner: nil_owner) + + assert result.success?, result.errors + assert_empty result.agents + assert_match "not an ActiveAgent::Base subclass", result.skipped.sole.skipped + end + + private + + # Agent has no owner presence validation; the engine scopes by whatever the + # host configures, and these tests exercise the sync itself. + def nil_owner + ActionAgent::Agent.new + end +end diff --git a/actionagent/test/host_agent_class_execution_test.rb b/actionagent/test/host_agent_class_execution_test.rb new file mode 100644 index 00000000..1ab9f5c1 --- /dev/null +++ b/actionagent/test/host_agent_class_execution_test.rb @@ -0,0 +1,89 @@ +# frozen_string_literal: true + +require "test_helper" + +# Two runtimes, and the rule for choosing between them. +# +# A dashboard-authored agent is rows — a tool selection and instructions typed +# into the builder — and the engine builds a class for it at run time. An +# agent mirrored from host code already *is* a class, with its own tools, +# delegations and instructions that `tools` + `instructions` columns cannot +# express. Both have to keep working. +class HostAgentClassExecutionTest < ActiveSupport::TestCase + class MirroredAgent < ApplicationAgent + generate_with :mock, model: "mock-1" + + def ask + prompt(message: "hi") + end + end + + setup do + ActionAgent::Agent.delete_all + @original = ActionAgent.run_host_agent_classes + end + + teardown do + ActionAgent.run_host_agent_classes = @original + ActionAgent::Agent.delete_all + end + + test "a dashboard-authored agent names no class, so the dynamic runtime keeps it" do + ActionAgent.run_host_agent_classes = true + agent = ActionAgent::Agent.create!( + name: "Builder Made", provider: "mock", model: "mock-1", agent_class_name: nil + ) + + assert_nil service_for(agent).send(:resolved_host_class), + "an agent with no class must fall to the dynamic runtime" + end + + test "a mirrored agent resolves its host class when the flag is on" do + ActionAgent.run_host_agent_classes = true + agent = ActionAgent::Agent.create!( + name: "Mirrored", provider: "mock", model: "mock-1", + agent_class_name: "HostAgentClassExecutionTest::MirroredAgent" + ) + + assert_equal MirroredAgent, service_for(agent).send(:resolved_host_class) + end + + test "the flag is off by default, so existing hosts are untouched" do + refute @original, "run_host_agent_classes must default to off" + + ActionAgent.run_host_agent_classes = false + agent = ActionAgent::Agent.create!( + name: "Mirrored", provider: "mock", model: "mock-1", + agent_class_name: "HostAgentClassExecutionTest::MirroredAgent" + ) + + assert_nil service_for(agent).send(:resolved_host_class) + end + + test "a class name that no longer resolves falls back rather than failing" do + ActionAgent.run_host_agent_classes = true + agent = ActionAgent::Agent.create!( + name: "Stale", provider: "mock", model: "mock-1", agent_class_name: "DeletedAgent" + ) + + # A renamed or removed class must not take the dashboard down with it. + assert_nil service_for(agent).send(:resolved_host_class) + end + + test "a class that is not an agent is not run as one" do + ActionAgent.run_host_agent_classes = true + agent = ActionAgent::Agent.create!( + name: "Not An Agent", provider: "mock", model: "mock-1", agent_class_name: "String" + ) + + assert_nil service_for(agent).send(:resolved_host_class) + end + + private + + def service_for(agent) + ActionAgent::AgentExecutionService.allocate.tap do |service| + service.instance_variable_set(:@agent_record, agent) + end + end +end diff --git a/lib/active_agent/concerns/view.rb b/lib/active_agent/concerns/view.rb index 9c0f6408..61a983c2 100644 --- a/lib/active_agent/concerns/view.rb +++ b/lib/active_agent/concerns/view.rb @@ -12,6 +12,29 @@ module View include ActionView::Layouts end + class_methods do + # The agent's rendered instructions, outside a generation. + # + # Two surfaces need the text an agent would run on without running it: a + # dashboard that mirrors the class (ActionAgent::AgentSync) and a test + # asserting what the model is told. Both otherwise reach a private + # renderer through `send`, which couples them to internals that can move + # without notice. + # + # TicketAgent.rendered_instructions # => "You are the Ticket agent..." + # + # TicketAgent.rendered_instructions(topic: "tickets") + # + # @param template [String] template name, default "instructions" + # @param assigns [Hash] instance variables the template reads + # @return [String, nil] nil when the agent has no such template + def rendered_instructions(template: "instructions", **assigns) + agent = new + assigns.each { |name, value| agent.instance_variable_set(:"@#{name}", value) } + agent.send(:view_render_template, template) + end + end + # Builds template lookup paths supporting both flat and nested directory structures. # # Templates are searched in priority order: diff --git a/lib/active_agent/schema_tools.rb b/lib/active_agent/schema_tools.rb index 55bd6531..8e60021d 100644 --- a/lib/active_agent/schema_tools.rb +++ b/lib/active_agent/schema_tools.rb @@ -332,6 +332,8 @@ def permitted_filters!(arguments) "`#{key}` is not a filterable attribute. Allowed filters: #{filterable.join(", ")}" end + validate_enum_value!(column, value) + memo[column] = value end @@ -386,6 +388,32 @@ def range_predicates!(column, value) end end + # A value outside a Rails enum reaches `where` as an unmatched name and + # returns zero rows — the same silent-nothing that {.range_predicates!} + # rejects for an unknown operator, and just as bad here: an agent reads + # "0 tickets under review" as a fact rather than a mistyped filter. + # + # The range form is not offered on an enum (see {.range_filterable?}), so + # a Hash here is a filter that cannot mean anything. + # + # @api private + # @raise [UnpermittedAttribute] on a value the enum does not define + def validate_enum_value!(column, value) + values = enum_values_for(column) + return if values.blank? + + if value.is_a?(Hash) + raise UnpermittedAttribute, + "`#{column}` is an enum and cannot be compared as a range. " \ + "Allowed values: #{values.keys.join(", ")}" + end + + return if values.key?(value.to_s) || values.value?(value) + + raise UnpermittedAttribute, + "`#{value}` is not a valid `#{column}`. Allowed values: #{values.keys.join(", ")}" + end + # Projects a record down to the declared return columns. # # The projection happens in SQL (+select+) as well as here, but the Ruby @@ -455,14 +483,44 @@ def filter_properties properties = schema[:schema][:properties] filterable.index_with do |column| + next enum_property(column) if enum_column?(column) + scalar = (properties[column] || { type: "string" }).deep_dup range_filterable?(column) ? with_range_form(column, scalar) : scalar end end + # A Rails enum is declared on the model, not through the inclusion + # validator SchemaGenerator reads, so without this a `status` column + # reaches the model as a bare integer: no labels, no constraint. The + # names are the model's own API (`Ticket.open`, `status: "open"`), so + # they are what the tool offers. + # + # @api private + def enum_property(column) + { type: "string", enum: enum_values_for(column).keys, description: "#{column.to_s.humanize} field" } + end + + # @api private + def enum_column?(column) + enum_values_for(column).present? + end + + # @api private + def enum_values_for(column) + @model.defined_enums[column.to_s] || {} + end + # Dates, times and numbers are the columns a question like "overdue" or # "more than 10" actually needs a comparison on. + # + # An enum is backed by an integer but is not ordered in any sense a + # question can use: `status > 1` is an artefact of declaration order, so + # offering the range form on one invites a meaningless filter that still + # returns a confident count. def range_filterable?(column) + return false if enum_column?(column) + RANGE_FILTERABLE_TYPES.include?(@model.type_for_attribute(column).type) end diff --git a/test/dummy/app/views/rendered_instructions_probe_agent/instructions.md.erb b/test/dummy/app/views/rendered_instructions_probe_agent/instructions.md.erb new file mode 100644 index 00000000..0f12c368 --- /dev/null +++ b/test/dummy/app/views/rendered_instructions_probe_agent/instructions.md.erb @@ -0,0 +1 @@ +You are the probe agent<%= " for #{@topic}" if @topic %>. diff --git a/test/dummy/db/schema.rb b/test/dummy/db/schema.rb index b9452945..818d6716 100644 --- a/test/dummy/db/schema.rb +++ b/test/dummy/db/schema.rb @@ -532,6 +532,7 @@ t.datetime "created_at", null: false t.boolean "published", default: false t.datetime "published_at" + t.integer "state", default: 0 t.string "title", null: false t.datetime "updated_at", null: false t.integer "user_id" diff --git a/test/rendered_instructions_test.rb b/test/rendered_instructions_test.rb new file mode 100644 index 00000000..a27d9d2f --- /dev/null +++ b/test/rendered_instructions_test.rb @@ -0,0 +1,32 @@ +# frozen_string_literal: true + +require "test_helper" + +# Rendering an agent's instructions without running it: what a dashboard that +# mirrors the class needs (ActionAgent::AgentSync), and what a test asserting +# what the model is told needs. Both otherwise reach a private renderer +# through `send`. +class RenderedInstructionsProbeAgent < ActiveAgent::Base +end + +class RenderedInstructionsTest < ActiveSupport::TestCase + test "renders the agent's own instructions template" do + assert_equal "You are the probe agent.", RenderedInstructionsProbeAgent.rendered_instructions + end + + test "assigns reach the template" do + # The hub case: instructions describing a roster the class computes. + assert_equal( + "You are the probe agent for tickets.", + RenderedInstructionsProbeAgent.rendered_instructions(topic: "tickets") + ) + end + + test "returns nil when the agent has no instructions template" do + agent = Class.new(ActiveAgent::Base) do + def self.name = "NoInstructionsTemplateAgent" + end + + assert_nil agent.rendered_instructions + end +end diff --git a/test/schema_tools_test.rb b/test/schema_tools_test.rb index b03b5b5e..9684193f 100644 --- a/test/schema_tools_test.rb +++ b/test/schema_tools_test.rb @@ -36,6 +36,21 @@ class UnscopedPostTools < ActiveAgent::SchemaTools returns :id, :title end + # A Rails enum, which is declared on the model rather than through the + # inclusion validator SchemaGenerator reads. + class EnumPost < Post + self.table_name = "posts" + enum :state, { draft: 0, review: 1, live: 2 }, prefix: true + + def self.name = "EnumPost" + end + + class EnumPostTools < ActiveAgent::SchemaTools + model EnumPost + filterable :state, :published + returns :id, :title + end + setup do Post.delete_all Profile.delete_all if defined?(Profile) @@ -107,6 +122,54 @@ class UnscopedPostTools < ActiveAgent::SchemaTools assert_equal "boolean", properties[:active][:type] end + test "a Rails enum is offered as its names, not its integer backing" do + definition = EnumPostTools.tool_definitions.find { |d| d[:name] == "find_enum_posts" } + state = definition[:parameters][:properties][:state] + + # Without this the column reaches the model as {type: "integer"}: no + # labels, so the only way a model learns "review" means 1 is prose in the + # instructions, written by hand in every host app. + assert_equal "string", state[:type] + assert_equal [ "draft", "review", "live" ], state[:enum] + end + + test "an enum is not offered the range form" do + definition = EnumPostTools.tool_definitions.find { |d| d[:name] == "find_enum_posts" } + state = definition[:parameters][:properties][:state] + + # `state > 1` is an artefact of declaration order, not a question anyone + # can ask. Offering it invites a filter that returns a confident number. + refute state.key?(:anyOf), "an enum must not advertise comparisons" + end + + test "an undefined enum value is rejected rather than matching nothing" do + result = EnumPostTools.call("count_enum_posts", state: "pending") + + # The silent alternative is {count: 0}, which an agent reports as a fact: + # "no posts are pending" reads identically to "pending is not a state". + # Same shape as the unknown-column error, so the model can correct itself. + refute result.key?(:count), "an undefined enum value must not return a count" + assert_match "pending", result[:error] + assert_match "draft, review, live", result[:error] + end + + test "an enum still accepts the names and values it defines" do + # setup seeds two posts, both at the column default (draft). + EnumPost.create!(title: "Live one", content: "x", user: @alice, state: "live") + + assert_equal({ count: 2 }, EnumPostTools.call("count_enum_posts", state: "draft")) + assert_equal({ count: 1 }, EnumPostTools.call("count_enum_posts", state: "live")) + # The integer backing still works, so a stored value round-trips. + assert_equal({ count: 1 }, EnumPostTools.call("count_enum_posts", state: 2)) + end + + test "a range predicate on an enum is rejected" do + result = EnumPostTools.call("count_enum_posts", state: { "gt" => 1 }) + + refute result.key?(:count), "a comparison on an enum must not return a count" + assert_match "cannot be compared as a range", result[:error] + end + test "only filterable columns appear as find parameters" do definition = UserTools.tool_definitions.find { |d| d[:name] == "find_users" } properties = definition[:parameters][:properties] From 06c19f8ae670e69a1af8ce14751864ae164263ed Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Thu, 17 Sep 2026 20:39:49 -0700 Subject: [PATCH 2/7] fix(sync): ask the class for instructions it assembles itself MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A record agent's prompt comes from a template it shares with sibling agents, filled from assigns the instance computes. rendered_instructions resolves only an agent's own template with assigns the caller supplies, so syncing such an agent stored empty instructions. AgentSync now asks for dashboard_instructions_text when the class defines it — only the class knows how its own prompt is built. Found by refactoring support-hub onto this branch. Co-Authored-By: Claude Opus 5 (1M context) --- .../app/services/action_agent/agent_sync.rb | 8 ++++++++ actionagent/test/agent_sync_test.rb | 20 +++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/actionagent/app/services/action_agent/agent_sync.rb b/actionagent/app/services/action_agent/agent_sync.rb index 36621b03..ad5fe303 100644 --- a/actionagent/app/services/action_agent/agent_sync.rb +++ b/actionagent/app/services/action_agent/agent_sync.rb @@ -106,7 +106,15 @@ def description_for(klass) # The rendered instructions, so the dashboard record runs on the same text # the class does rather than a hand-maintained copy. + # + # An agent whose instructions are assembled rather than rendered straight + # from its own template — filled from assigns it computes, or falling back + # to a template it shares with sibling agents — says so by defining + # `dashboard_instructions_text`. That is asked first, because only the + # class knows how its own prompt is built. def instructions_for(klass) + return klass.dashboard_instructions_text.presence if klass.respond_to?(:dashboard_instructions_text) + klass.try(:rendered_instructions).presence end diff --git a/actionagent/test/agent_sync_test.rb b/actionagent/test/agent_sync_test.rb index 0a3ca87d..da6be0b9 100644 --- a/actionagent/test/agent_sync_test.rb +++ b/actionagent/test/agent_sync_test.rb @@ -5,6 +5,16 @@ # Mirroring host agent classes into Agent records — the step that lets the # dashboard run, evaluate and release the agents an app already has in code. class AgentSyncTest < ActiveSupport::TestCase + class AssembledAgent < ApplicationAgent + generate_with :mock, model: "mock-1" + + def self.dashboard_instructions_text = "assembled for the dashboard" + + def ask + prompt(message: "hi") + end + end + class InvoiceAgent < ApplicationAgent generate_with :mock, model: "mock-1" @@ -51,6 +61,16 @@ def ask assert_equal "operator-choice", agent.reload.model end + test "an agent that assembles its own instructions is asked for them" do + result = ActionAgent::AgentSync.call([ AssembledAgent ], owner: nil_owner) + + # A record agent's prompt comes from a template it shares with siblings, + # filled from assigns it computes — the class is the only thing that knows + # how to build it, so the sync asks rather than rendering a template that + # would come back empty. + assert_equal "assembled for the dashboard", result.agents.sole.instructions + end + test "a class that is not an agent is skipped, not raised" do result = ActionAgent::AgentSync.call([ String ], owner: nil_owner) From e317a5ac7a00f91fb4015489d362fb22a358eeab Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Fri, 18 Sep 2026 11:25:43 -0700 Subject: [PATCH 3/7] test: declare the enum fixture's attribute type for Rails 7.2 Rails 7.2 raises "Undeclared attribute type for enum 'state' in EnumPost" when a subclass declares an enum over a column it inherited; Rails 8 infers it from the schema. Declaring `attribute :state, :integer` satisfies both. Co-Authored-By: Claude Opus 5 (1M context) --- test/schema_tools_test.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/test/schema_tools_test.rb b/test/schema_tools_test.rb index 9684193f..b0f12e72 100644 --- a/test/schema_tools_test.rb +++ b/test/schema_tools_test.rb @@ -40,6 +40,10 @@ class UnscopedPostTools < ActiveAgent::SchemaTools # inclusion validator SchemaGenerator reads. class EnumPost < Post self.table_name = "posts" + # Declared explicitly rather than inferred from the column: Rails 7.2 + # raises "Undeclared attribute type for enum" when a subclass declares an + # enum over a column it inherited, which Rails 8 tolerates. + attribute :state, :integer enum :state, { draft: 0, review: 1, live: 2 }, prefix: true def self.name = "EnumPost" From 8678de864f5726d9496e869bca3c3d2c51fdf224 Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Fri, 18 Sep 2026 11:29:26 -0700 Subject: [PATCH 4/7] test: declare the enum on Post rather than a subclass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rails 7.2 does not resolve an inherited column through a subclass the way Rails 8 does: EnumPost < Post raised "Undeclared attribute type for enum", and once that was declared, SchemaTools' own resolve_column! rejected `state` as "not a column on EnumPost". The column belongs to Post, so the enum is declared there and the tools front Post directly — no subclass, and the same behaviour on both Rails lines. Co-Authored-By: Claude Opus 5 (1M context) --- test/dummy/app/models/post.rb | 5 +++++ test/schema_tools_test.rb | 30 ++++++++++-------------------- 2 files changed, 15 insertions(+), 20 deletions(-) diff --git a/test/dummy/app/models/post.rb b/test/dummy/app/models/post.rb index 7ed82220..28a71c66 100644 --- a/test/dummy/app/models/post.rb +++ b/test/dummy/app/models/post.rb @@ -3,6 +3,11 @@ class Post < ApplicationRecord belongs_to :user + # A Rails enum, for the schema-tools tests: declared on the model rather + # than through an inclusion validator, which is the case SchemaGenerator + # cannot see and SchemaTools has to read from defined_enums. + enum :state, { draft: 0, review: 1, live: 2 }, prefix: true + validates :title, presence: true, length: { maximum: 255 } validates :content, presence: true diff --git a/test/schema_tools_test.rb b/test/schema_tools_test.rb index b0f12e72..cb72606f 100644 --- a/test/schema_tools_test.rb +++ b/test/schema_tools_test.rb @@ -38,19 +38,9 @@ class UnscopedPostTools < ActiveAgent::SchemaTools # A Rails enum, which is declared on the model rather than through the # inclusion validator SchemaGenerator reads. - class EnumPost < Post - self.table_name = "posts" - # Declared explicitly rather than inferred from the column: Rails 7.2 - # raises "Undeclared attribute type for enum" when a subclass declares an - # enum over a column it inherited, which Rails 8 tolerates. - attribute :state, :integer - enum :state, { draft: 0, review: 1, live: 2 }, prefix: true - - def self.name = "EnumPost" - end - + # Post declares `enum :state` (test/dummy/app/models/post.rb). class EnumPostTools < ActiveAgent::SchemaTools - model EnumPost + model Post filterable :state, :published returns :id, :title end @@ -127,7 +117,7 @@ class EnumPostTools < ActiveAgent::SchemaTools end test "a Rails enum is offered as its names, not its integer backing" do - definition = EnumPostTools.tool_definitions.find { |d| d[:name] == "find_enum_posts" } + definition = EnumPostTools.tool_definitions.find { |d| d[:name] == "find_posts" } state = definition[:parameters][:properties][:state] # Without this the column reaches the model as {type: "integer"}: no @@ -138,7 +128,7 @@ class EnumPostTools < ActiveAgent::SchemaTools end test "an enum is not offered the range form" do - definition = EnumPostTools.tool_definitions.find { |d| d[:name] == "find_enum_posts" } + definition = EnumPostTools.tool_definitions.find { |d| d[:name] == "find_posts" } state = definition[:parameters][:properties][:state] # `state > 1` is an artefact of declaration order, not a question anyone @@ -147,7 +137,7 @@ class EnumPostTools < ActiveAgent::SchemaTools end test "an undefined enum value is rejected rather than matching nothing" do - result = EnumPostTools.call("count_enum_posts", state: "pending") + result = EnumPostTools.call("count_posts", state: "pending") # The silent alternative is {count: 0}, which an agent reports as a fact: # "no posts are pending" reads identically to "pending is not a state". @@ -159,16 +149,16 @@ class EnumPostTools < ActiveAgent::SchemaTools test "an enum still accepts the names and values it defines" do # setup seeds two posts, both at the column default (draft). - EnumPost.create!(title: "Live one", content: "x", user: @alice, state: "live") + Post.create!(title: "Live one", content: "x", user: @alice, state: "live") - assert_equal({ count: 2 }, EnumPostTools.call("count_enum_posts", state: "draft")) - assert_equal({ count: 1 }, EnumPostTools.call("count_enum_posts", state: "live")) + assert_equal({ count: 2 }, EnumPostTools.call("count_posts", state: "draft")) + assert_equal({ count: 1 }, EnumPostTools.call("count_posts", state: "live")) # The integer backing still works, so a stored value round-trips. - assert_equal({ count: 1 }, EnumPostTools.call("count_enum_posts", state: 2)) + assert_equal({ count: 1 }, EnumPostTools.call("count_posts", state: 2)) end test "a range predicate on an enum is rejected" do - result = EnumPostTools.call("count_enum_posts", state: { "gt" => 1 }) + result = EnumPostTools.call("count_posts", state: { "gt" => 1 }) refute result.key?(:count), "a comparison on an enum must not return a count" assert_match "cannot be compared as a range", result[:error] From 1dc6235fe0343b1d822abad5ed8177b1ee883e36 Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Fri, 18 Sep 2026 12:02:09 -0700 Subject: [PATCH 5/7] test: add the enum's column to the migration, not just schema.rb MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI builds the dummy database with db:migrate, not by loading schema.rb, so adding `state` to the schema alone left Rails 7's test database without the column — and Post's `enum :state` then raised "Undeclared attribute type for enum" before any test ran. Rails 8 was green only because its schema was already loaded locally. Verified the way CI does it: rebuilt test/dummy from migrations under gemfiles/rails7.gemfile and ran the suite — 2067 runs, 0 failures (253 pre-existing API-key errors), and the same on Rails 8. Co-Authored-By: Claude Opus 5 (1M context) --- test/dummy/db/migrate/002_create_posts.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/test/dummy/db/migrate/002_create_posts.rb b/test/dummy/db/migrate/002_create_posts.rb index 26897f45..e19e1bf8 100644 --- a/test/dummy/db/migrate/002_create_posts.rb +++ b/test/dummy/db/migrate/002_create_posts.rb @@ -8,6 +8,9 @@ def change t.references :user, foreign_key: true t.boolean :published, default: false t.datetime :published_at + # Backs Post's `enum :state` — the schema-tools tests need a real Rails + # enum, which is the case SchemaGenerator cannot read from validators. + t.integer :state, default: 0 t.timestamps end From 3a4fe1816aa8ef7bca0de0fdba75a6e1d2c0b158 Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Fri, 18 Sep 2026 14:47:55 -0700 Subject: [PATCH 6/7] docs: say why an enum filter still accepts its backing integer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The schema advertises enum names only, so an integer arriving here means the model ignored it — worth stating that the tolerance is deliberate rather than an oversight, since the neighbouring branches both reject. Co-Authored-By: Claude Opus 5 --- lib/active_agent/schema_tools.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/active_agent/schema_tools.rb b/lib/active_agent/schema_tools.rb index 8e60021d..a69fe8ed 100644 --- a/lib/active_agent/schema_tools.rb +++ b/lib/active_agent/schema_tools.rb @@ -408,6 +408,9 @@ def validate_enum_value!(column, value) "Allowed values: #{values.keys.join(", ")}" end + # The schema offers names only, so an integer here came from a model + # ignoring it. Accepted anyway: it is unambiguous, and a host calling + # the tool directly in Ruby reasonably passes the backing value. return if values.key?(value.to_s) || values.value?(value) raise UnpermittedAttribute, From 1982cfc3cd4c115cf81c1cbdf224d6e0185fabbc Mon Sep 17 00:00:00 2001 From: Justin Bowen Date: Sat, 19 Sep 2026 12:15:29 -0700 Subject: [PATCH 7/7] test: stop pinning a price that depends on whether ruby_llm is loaded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SolidAgentRunsTest asserted claude-sonnet-5 at 0.048 for 12k/800 tokens. That figure is solid_agent's static table ($3/$15 per million). But ModelPricing prefers RubyLLM's registry whenever ::RubyLLM is defined, and the registry shipped in ruby_llm 2.0.0 prices the same model at $2/$10 — 0.032. Which table answers depends on whether the RubyLLM provider tests loaded the gem, and they do exactly when OPENAI_API_KEY is set: CI has no key, so it skipped them and stayed green; a local .env.test with a placeholder key loaded them and failed every full run, while the file alone passed. The old first assertion also compared ModelPricing.estimate with itself through the record, so it would have passed on nil. Now the test asserts what this seam actually guarantees, in any load order: usage reaches the generation record; the mock model prices to 0.0 rather than nil; a real model prices above zero and the record agrees with ModelPricing for the same tokens; and the explicit-rate branch, which bypasses every table, still computes 0.048. The dollar figures behind the tables belong to solid_agent's own suite. Verified under gemfiles/rails7.gemfile with ruby_llm both unloaded and loaded, and full suites on Rails 7 and 8: 2071 runs, 0 failures, with the 34 errors unchanged (missing API keys). Co-Authored-By: Claude Fable 5.1 --- test/integration/solid_agent/runs_test.rb | 30 +++++++++++++++-------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/test/integration/solid_agent/runs_test.rb b/test/integration/solid_agent/runs_test.rb index 31e7d1b3..76f5bd01 100644 --- a/test/integration/solid_agent/runs_test.rb +++ b/test/integration/solid_agent/runs_test.rb @@ -98,20 +98,30 @@ class SolidAgentRunsTest < SolidAgentIntegrationTest generation = AgentGeneration.sole - # The mock provider's model prices at zero by design, so the assertion - # that matters is that estimation runs and stays consistent. + # Usage has to reach the record before there is anything to price. + assert_operator generation.input_tokens + generation.output_tokens, :>, 0 + + # The mock provider's model prices at zero by design: 0.0 rather than nil + # is the proof that estimation ran over the recorded tokens. + assert_equal 0.0, generation.estimated_cost + + # A real model prices above zero. Which table answers — solid_agent's + # static rates or RubyLLM's registry — depends on whether ruby_llm has + # been loaded elsewhere in this process (its provider tests load it when + # OPENAI_API_KEY is set), and the two disagree on this model. So no dollar + # figure is pinned through the tables; solid_agent's own suite owns those. + generation.update!(model: "claude-sonnet-5", input_tokens: 12_000, output_tokens: 800) + + assert_operator generation.estimated_cost, :>, 0 assert_equal( - SolidAgent::ModelPricing.estimate( - model: generation.model, - input_tokens: generation.input_tokens, - output_tokens: generation.output_tokens - ), + SolidAgent::ModelPricing.estimate(model: "claude-sonnet-5", input_tokens: 12_000, output_tokens: 800), generation.estimated_cost ) - assert_in_delta 0.048, SolidAgent::ModelPricing.estimate( - model: "claude-sonnet-5", input_tokens: 12_000, output_tokens: 800 - ), 0.0005 + # Explicit rates bypass every table, so the arithmetic itself is stable. + assert_in_delta 0.048, + generation.estimated_cost(input_price_per_million: 3.0, output_price_per_million: 15.0), + 1e-9 end test "one trace id joins the run, the conversation and the generation" do