Skip to content

feat(phoenix): add AgentV eval adapter#1279

Merged
christso merged 6 commits into
mainfrom
feat/phoenix-adapter
Jun 2, 2026
Merged

feat(phoenix): add AgentV eval adapter#1279
christso merged 6 commits into
mainfrom
feat/phoenix-adapter

Conversation

@christso
Copy link
Copy Markdown
Collaborator

@christso christso commented Jun 2, 2026

Summary

  • Adds @agentv/phoenix-adapter as a repo-local package for converting AgentV-authored eval suites into Phoenix dataset/experiment payloads.
  • Reuses @agentv/core's eval loader so AgentV YAML remains the source of truth for case expansion, external datasets, Agent Skills evals.json, assertion parsing, interpolation, and metadata handling.
  • Ports deterministic Phoenix CODE evaluator support for contains, regex, equals, and is-json; reports LLM/code/trace/composite/metric/custom evaluator families as unsupported instead of silently adapting them.
  • Adds root scripts and CI smoke coverage via bun run phoenix:assert-smoke.

Verification

  • bun --filter @agentv/phoenix-adapter test — 18 tests passed.
  • bun --filter @agentv/phoenix-adapter typecheck — passed.
  • bun run phoenix:assert-smoke — dry-run smoke passed: 1 suite, 4 tests, baseline count 4, 0 failed suites.
  • bun run test — passed (core/eval/phoenix-adapter/cli tests).
  • Pre-push hook on git push passed: build, typecheck, lint, test, validate eval YAML files.

Smoke / before-after evidence

Before this branch, AgentV had no in-repo Phoenix adapter package or phoenix:assert-smoke script. On this branch:

bun run phoenix:assert-smoke
Suites: 1
Tests: 4
Passed suites: 1
Failed suites: 0
| passed | `examples/features/assert/evals/dataset.eval.yaml` | 4 | 4 | |

Full dry-run is intentionally not used as a CI gate yet. Current informational full dry-run result is 97 suites / 405 tests / 93 passed suites / 4 failed suites. The failures are documented in packages/phoenix-adapter/docs/e2e-verification.md and are source/baseline or source-reference parity issues, not Phoenix conversion crashes.

Follow-up work

  • Implement AgentV-compatible Phoenix adapters for llm-grader and code-grader semantics.
  • Decide whether to expand deterministic support to AgentV's extended string-match families (icontains-*, contains-any, starts-with, etc.).
  • Resolve the known full dry-run source/baseline drift before making full dry-run parity a blocking CI check.

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages Bot commented Jun 2, 2026

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 9be0b5f
Status: ✅  Deploy successful!
Preview URL: https://e2d3bd23.agentv.pages.dev
Branch Preview URL: https://feat-phoenix-adapter.agentv.pages.dev

View logs

@christso christso marked this pull request as ready for review June 2, 2026 11:33
@christso christso merged commit fe48070 into main Jun 2, 2026
4 checks passed
@christso christso deleted the feat/phoenix-adapter branch June 2, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant