Evalon Core

Actor-based agent evaluation framework with simulation and LLM-as-judge scoring. Built with Scala 3 and Apache Pekko.

Evalon simulates multi-party conversations between participants, then evaluates an agent's performance against behavioral criteria. Each participant and event source runs as an independent actor, with the ScenarioRunner orchestrating message routing and transcript recording.

Setup

1. Install dependencies:

sbt compile

2. Configure API credentials:

cp .env.example .env
# Edit .env with your credentials

Environment variables:

• ANTHROPIC_BASE_URL — API endpoint (auto-strips /bedrock suffix)
• ANTHROPIC_API_KEY — auth token
• EVALON_CA_CERTS — path to CA cert bundle (for corporate proxies / custom CA bundles)

3. Load environment and run:

set -a && source .env && set +a && sbt run

Running scenarios

# Flight rescheduling (direct topology: user <-> agent)
sbt "run scenarios/reschedule_flight.yaml"

# Hotel cancellation (assist topology: user <-> rep, agent observes + assists)
sbt "run scenarios/hotel_cancellation_assist.yaml"

# Payment activation (direct, 3 tools)
sbt "run scenarios/payment_activation.yaml"

# Billing dispute (assist topology: user <-> rep, agent observes + assists)
sbt "run scenarios/billing_dispute_assist.yaml"

# Complex payment issue (direct, 14 tools, 30 turns)
sbt "run scenarios/comprehensive_payment_issue.yaml"

# Order refund workflow (direct, 8 tools)
sbt "run scenarios/order_refund.yaml"

Publishing

Releases are published to Maven Central by GitHub Actions (.github/workflows/release.yml) via sbt-ci-release. The version is derived from git tags by sbt-dynver — pushing a tag vX.Y.Z publishes that release.

For local testing, publish to your local repository with an explicit version (no tag needed):

sbt 'set ThisBuild / version := "0.1.0-LOCAL-SNAPSHOT"' publishLocal

Architecture

Actor system

Actor	Role
ScenarioRunner	Orchestrator — spawns participants, routes messages to direct participants, records transcript
Participant	Shared protocol (ReceiveMessage, ReceiveEvents) for all participant actors
SimulatedParticipant	Self-driven LLM actor with per-conversation history and cross-conversation context
EvaluatedAgent	Wraps the Agent trait, bridges actor messages to Future-based calls
EventSourceActor	Observes simulation traffic, uses LLM reasoning to emit derived events

Core modules

• model/ — Domain types: Message, Action, Event, HistoryEntry, Transcript, Scenario, EvalResult
• actor/ — Pekko typed actors for simulation participants and orchestration
• agent/ — Agent trait, SimpleAgent (Java-friendly SAM), ClaudeAgent (in-process Claude with tool-use loop), and RemoteAgent (HTTP proxy for non-JVM agents)
• tool/ — Tool trait and mock implementations (flights, hotels, payments, refunds)
• llm/ — sttp-based Anthropic Messages API client with circe codecs
• scenario/ — YAML scenario loader (circe-yaml)
• eval/ — LLM-as-judge evaluator (one call per criterion)
• output/ — Colored transcript printer and dataset JSON writer

Concepts

• Participants — simulated (LLM-driven), evaluated (agent under test), or custom (user code). Simulated participants may set template to replace the default role-play prompt (the [END] instruction is always appended).
• Conversations — named channels between participants, with an optional initiator
• Observations — allow a participant to observe messages in conversations it's not directly part of (currently only EvaluatedAgent reacts to observed events; SimulatedParticipant ignores them — support will be added when a scenario needs it)
• Event sources — actors that observe all simulation traffic and emit derived events (⚠️ experimental: custom source type is not wired up, dedup of repeated emissions is not handled, and no shipped scenario exercises this path yet)
• Transcript — immutable ordered log of all messages, tool calls, tool results, and events. Each message may include an optional JSON trace (how that reply was generated)

Evaluating remote agents

Evalon can evaluate agent implementations written in any language by proxying the Agent trait over HTTP. To use a remote agent, set an endpoint: on the evaluated participant in the scenario YAML — Evalon will construct a RemoteAgent instead of ClaudeAgent.

The agent server implements one endpoint, POST /v1/step. Each request carries the full conversation history (turns and prior tool uses), accumulated system events, and the conversation the response should target. Evalon owns history; the server is stateless. The server runs its own tools and reports the trace in the response.

Wire format (request):

{
  "protocol_version": "1",
  "history": [
    {"type": "turn", "conversation": "support_chat", "sender": "end_user", "content": "..."},
    {"type": "turn", "conversation": "support_chat", "sender": "agent", "content": "...", "trace": {"intent": "lookup"}},
    {"type": "tool_use", "conversation": "agent_assist", "interaction": {"call": {...}, "result": {...}}}
  ],
  "events": [{"name": "case_created", "data": {...}}],
  "respond_in": "support_chat"
}

Wire format (response):

{
  "action": {
    "type": "send",
    "message": {"sender": "agent", "content": "...", "trace": {"intent": "lookup"}},
    "tool_interactions": []
  }
}

trace on a message is optional JSON describing how that reply was generated. tool_interactions is the list of tool call/result pairs this step actually ran.

Or {"action": {"type": "end"}} to end the conversation.

A Java SimpleAgent can attach the same generation trace as an optional JSON string:

AgentReply.send(content, objectMapper.writeValueAsString(traceMap));

A minimal Python server (FastAPI, ~70 lines) lives at examples/remote_agent_python/server.py. With uv installed:

uv run examples/remote_agent_python/server.py
# then in another terminal:
sbt "run scenarios/reschedule_flight.yaml"  # with endpoint: uncommented in the YAML

Evalon retries 429/5xx responses with exponential backoff (3 retries, 1s base). Other errors fail the run.

Simulation flow

1. ScenarioRunner spawns all participant and event source actors
2. Conversations with an initiated_by participant kick off; that participant receives a start message
3. Participants are self-driven: when a message arrives and the actor is idle, it generates a response asynchronously. Messages arriving during generation are accumulated and processed afterward
4. Messages are recorded in the transcript and delivered to direct participants (those in the conversation's between list)
5. Observer notifications deliver new messages as events (ReceiveEvents) to observing participants
6. Event sources receive all traffic and may emit additional events
7. Simulation ends when any participant signals [END] or max_turns is reached
8. The evaluator scores the transcript with one LLM call per criterion, then combines weighted scores

Scenario YAML format

name: reschedule_flight
description: >
  A customer contacts support because their flight has been delayed.

participants:
  end_user:
    type: simulated
    persona: You are Jane Doe, a frustrated customer...
    goal: Get rebooked on the next available flight.
    # Optional: replace the default simulated-participant prompt.
    # template: |
    #   You are role-playing as Jane Doe.
  agent:
    type: evaluated
    # Optional: point at an HTTP server speaking the remote-agent protocol (see
    # examples/remote_agent_python/server.py). Without this, evaluation uses ClaudeAgent.
    # endpoint: http://localhost:8080

conversations:
  - name: support_chat
    between: [end_user, agent]
    initiated_by: end_user

# Shared ground truth for tools (not per-participant).
context:
  flights:
    AA123:
      flight_number: AA123
      status: delayed
      passenger: Jane Doe
      booking_ref: BK-5678

observations:
  - participant: agent
    observes: [support_chat]

event_sources:
  case_events:
    type: simulated
    description: A case management system...
    emits:
      - event: case_created
        schema: { case_id: string, priority: string }

max_turns: 10

# Optional: replace the default judge prompt. The transcript, criterion, and
# response-format instructions are still appended.
# eval_prompt_template: |
#   You are judging only whether the agent used the right tools.

eval_criteria:
  - name: lookup_flight_status
    description: "Agent looked up the customer's current flight status"
    criterion_type: binary          # binary | scored | rubric
    require_tool_call: true
  - name: rebook_flight
    description: "Agent successfully rebooked the customer on a new flight"
    criterion_type: binary
    require_tool_call: true
    weight: 2.0
  - name: professional_tone
    description: "Agent was empathetic and professional in tone"
    criterion_type: scored
    require_tool_call: false
    pass_threshold: 0.7             # scored/rubric: passed if score >= this

A criterion may also be a plain string; that is treated as a binary check with require_tool_call: false.

Judge JSON is per criterion: binary returns {passed, reasoning}; scored and rubric return {score, reasoning}. Binary passed comes from the judge; scored/rubric passed is score >= pass_threshold when that field is set.