From ae0013133815a1025f46b92875cb4b38d6771259 Mon Sep 17 00:00:00 2001 From: Johannes Vogt Date: Mon, 14 Sep 2026 15:01:16 +0200 Subject: [PATCH 1/2] short section about agent evals --- guides/ai/cap-agents.md | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/guides/ai/cap-agents.md b/guides/ai/cap-agents.md index ebed0856c..2a4637382 100644 --- a/guides/ai/cap-agents.md +++ b/guides/ai/cap-agents.md @@ -487,14 +487,46 @@ On all errors the plugin will summarize the progress till that point. The summar On execution timeouts the graph does not fail but instead interrupts and sends a HITL message asking the user whether to continue, including the summary about the progress. +### Evals-based Agent Testing +Evaluate your agent behavior using `cds.test`. Start your application and send a message to the agent. +Then check the response with deterministic assertions or utilize an LLM-as-a-Judge, which scores the result against specified criteria. + +```js +import cds from '@sap/cds' +import { Judge } from '@cap-js/agents/eval' + +cds.test() + +describe('catalog', () => { + test('should list drama books', async () => { + const catalog = await cds.connect.to('CatalogService') + const result = await catalog.chat('what drama books do you have?') + + const response = result.messages.at(-1) + expect(response.content).toContain('Wuthering Heights') + expect(response.content).toContain('Jane Eyre') + + const judge = new Judge() + const { score } = await judge + .criteria('Should list **only** drama books, and suggest ordering one') + .evaluate(result) + + expect(score).toBeGreaterThan(0.8) + }) +}) +``` + +When [MLflow](#mlflow) is enabled, your test runs also appear as evaluations grouped by their top-level `describe` block. + +For detailed guidance on writing evals, refer to the documentation at: +https://github.com/cap-js/agents/blob/main/.docs/testing/evals.md ### More to come... Following are features and areas we are currently working on, and plan to release in the near future: -- **Evals-based Agent Testing** – using best practice evaluation frameworks. - **RAG & Knowledge Graphs** – for agents to leverage structured knowledge. - **Alternative Agent Harnesses** – to provide different execution environments for agents. - **Custom-coded Agents** – to inject custom logic and behavior into agent harness. From acd976c304008fdf0ecd48c83252403a113a8e99 Mon Sep 17 00:00:00 2001 From: Johannes Vogt Date: Mon, 14 Sep 2026 15:04:18 +0200 Subject: [PATCH 2/2] wording --- guides/ai/cap-agents.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/guides/ai/cap-agents.md b/guides/ai/cap-agents.md index 2a4637382..aebc7f46c 100644 --- a/guides/ai/cap-agents.md +++ b/guides/ai/cap-agents.md @@ -489,8 +489,8 @@ On execution timeouts the graph does not fail but instead interrupts and sends a ### Evals-based Agent Testing -Evaluate your agent behavior using `cds.test`. Start your application and send a message to the agent. -Then check the response with deterministic assertions or utilize an LLM-as-a-Judge, which scores the result against specified criteria. +Evaluate your agent behavior using `cds.test`. First, start your application and send a message to the agent. +Then check the response with deterministic assertions or use an LLM-as-a-Judge to score it against specified criteria. ```js import cds from '@sap/cds'