diff --git a/agents/test/agent-tests.mdx b/agents/test/agent-tests.mdx index 990680f..ff51ac1 100644 --- a/agents/test/agent-tests.mdx +++ b/agents/test/agent-tests.mdx @@ -310,7 +310,7 @@ Tests always exercise the agent's latest **draft** configuration, including unpu Everything on this page is also available over the REST API with an [API key](/developer-guide/getting-started/api-key), so you can keep tests next to your agent configuration and gate releases on them. See the [Tests API reference](/api-reference/endpoint/agent/list-tests) for every endpoint. -- **Export and import.** [Get Test](/api-reference/endpoint/agent/get-test) returns a test's full definition. Post that body to [Create Test](/api-reference/endpoint/agent/create-test) as is to recreate it, for example from JSON files kept in your repository. Read-only fields such as `test_id` and `created_at` are ignored. Tool references use your workspace's tool ids, so an export imports as is within the same team. [List Test Tools](/api-reference/endpoint/agent/list-test-tools) returns the ids a test can reference for an agent, including integration tools. +- **Export and import.** [Get Test](/api-reference/endpoint/agent/get-test) returns a test's full definition. Post that body to [Create Test](/api-reference/endpoint/agent/create-test) as is to recreate it, for example from JSON files kept in your repository. Read-only fields such as `test_id` and `created_at` are ignored. Tool references use your workspace's tool ids, so an export imports as is within the same team. [List Test Tools](/api-reference/endpoint/agent/list-test-tools) returns the ids a test can reference for an agent, including integration tools. [Import and export tests](/agents/test/import-export-tests) has scripts for saving tests to files and for moving them to another team. - **Attach.** Pass `agent_ids` when you create a test, or attach it later with [Attach Test](/api-reference/endpoint/agent/attach-test). - **Run and wait.** [Run Tests](/api-reference/endpoint/agent/run-tests) starts a batch with every attached test, or only the `test_ids` you pass, and `repeat_count` overrides each test's repeat count for that batch. Poll [Get Test Batch](/api-reference/endpoint/agent/get-test-batch) until `completed` is `true`, then check `passed`, `failed`, `errors`, and `pass_rate`. `pass_rate` leaves out runs that ended in **Error**, so decide in your pipeline whether an error should fail the build or be retried. - **History.** [List Test Batches](/api-reference/endpoint/agent/list-test-batches) and [List Test Runs](/api-reference/endpoint/agent/list-test-runs) return past results, filterable by test, batch, and status. diff --git a/agents/test/import-export-tests.mdx b/agents/test/import-export-tests.mdx new file mode 100644 index 0000000..69f7186 --- /dev/null +++ b/agents/test/import-export-tests.mdx @@ -0,0 +1,142 @@ +--- +title: "Import and Export Tests" +description: "Save tests as JSON files, recreate them in the same team, or move them to another team over the API" +icon: "file-export" +--- + +A test has one JSON format for reading and writing. [Get Test](/api-reference/endpoint/agent/get-test) returns it, and [Create Test](/api-reference/endpoint/agent/create-test) accepts the same body as is. That makes it easy to keep tests in version control, copy them to another agent, or move them to another team. + +**Prerequisites** + +- A Fish Audio [API key](/developer-guide/getting-started/api-key) for each team involved. +- `bash`, `curl`, and `jq` 1.6 or later for the scripts below. + +## The export format + +An exported test contains every field of its [test type](/agents/test/agent-tests#test-types), plus: + +| Field | On import | +| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| `test_id`, `workspace_id`, `created_at`, `updated_at` | Ignored. The import gets its own values. | +| `agent_ids` | The agents the test was attached to. The import attaches to them again, so change it to target others. | + +Tools appear as `{"id", "name", "type"}` wherever a test references them: `referenced_tool`, `simulation.assertions`, `simulation.tool_mocks.tools`, and `simulation.tool_mocks.real_tools`. Webhook and client tool ids belong to your team. Integration tool ids, such as `google_calendar:create_event`, are the same in every team. + +Every call to Create Test creates a new test, so importing the same file twice gives you two copies. + +## Export tests to files + +This script saves every test attached to an agent as `tests/.json`. Drop the `agent_id` parameter to export your whole test library instead. + +```bash export-tests.sh +set -euo pipefail +api=https://api.fish.audio/v1/agent +auth="Authorization: Bearer $FISH_API_KEY" +mkdir -p tests + +cursor="" +while true; do + page=$(curl -sfG "$api/tests" -H "$auth" \ + --data-urlencode "agent_id=$AGENT_ID" \ + --data-urlencode "page_size=100" \ + ${cursor:+--data-urlencode "cursor=$cursor"}) + for id in $(jq -r '.tests[].test_id' <<<"$page"); do + curl -sf "$api/tests/$id" -H "$auth" > "tests/$id.json" + done + [ "$(jq -r .has_more <<<"$page")" = "true" ] || break + cursor=$(jq -r .next_cursor <<<"$page") +done +``` + +## Import into the same team + +Within one team, tool ids stay valid, so the files import as they are: + +```bash import-tests.sh +set -euo pipefail +api=https://api.fish.audio/v1/agent +auth="Authorization: Bearer $FISH_API_KEY" + +for file in tests/*.json; do + curl -sf -X POST "$api/tests" -H "$auth" -H "Content-Type: application/json" \ + --data @"$file" | jq -r '"\(.test_id) \(.name)"' +done +``` + +Each test is attached to the agents in its `agent_ids`. To attach the copies to a different agent instead, rewrite that field on the way in: + +```bash +jq --arg agent "$AGENT_ID" '.agent_ids = [$agent]' "$file" | + curl -sf -X POST "$api/tests" -H "$auth" -H "Content-Type: application/json" --data @- +``` + +All agents in `agent_ids` must be in the same workspace, and the test is created there. + +## Import into another team + +Webhook and client tool ids from the source team mean nothing in the target team, so each reference has to point at the target team's tool of the same name first. + + + Create Test does not reject webhook and client tool ids it doesn't know. A + test imported without remapping saves fine but misbehaves when it runs: its + mocks never apply, required tool calls fail with _This tool is not on the + agent_, and forbidden tool checks pass without checking anything. Always remap + before importing into another team. + + + + + In the target team, create the tools the tests reference, with the same + names, for example with [Create Tool](/api-reference/endpoint/agent/create-tool), + and attach them to the target agent. Connect the same integrations on that + agent if the tests reference integration tools. + + + Run this script with the target team's API key and agent. It reads the tools + the target agent offers from [List Test Tools](/api-reference/endpoint/agent/list-test-tools), + replaces every webhook and client tool id by name, attaches each test to + the target agent, and stops with the tool's name if the agent has no tool + with that name. + + ```bash import-into-team.sh + set -euo pipefail + api=https://api.fish.audio/v1/agent + auth="Authorization: Bearer $FISH_API_KEY" + + tools=$(curl -sf "$api/agents/$AGENT_ID/test-tools" -H "$auth" | + jq '[.tools[] | select(.type != "integration") | {(.name): .id}] | add // {}') + + for file in tests/*.json; do + jq --argjson tools "$tools" --arg agent "$AGENT_ID" ' + walk( + if type == "object" and (.type == "webhook" or .type == "client") + and has("id") and has("name") + then .id = ($tools[.name] // error("No tool named \(.name) on the target agent")) + else . end + ) + | .agent_ids = [$agent] + ' "$file" | + curl -sf -X POST "$api/tests" -H "$auth" -H "Content-Type: application/json" \ + --data @- | jq -r '"\(.test_id) \(.name)"' + done + ``` + + + + +Integration tool ids are left as they are, since they are the same in every team. + +## Going further + + + + Every test type and how its fields map to the API. + + + Run an agent's tests from your pipeline and gate on the result. + + diff --git a/docs.json b/docs.json index e73f471..17f06c1 100644 --- a/docs.json +++ b/docs.json @@ -107,6 +107,7 @@ "pages": [ "agents/test/preview-calls", "agents/test/agent-tests", + "agents/test/import-export-tests", "agents/test/simulation-tutorial" ] },