[Agents Extension] Add test scenarios using interactive test tool - #9342
Draft
trangevi wants to merge 39 commits into
Draft
[Agents Extension] Add test scenarios using interactive test tool#9342trangevi wants to merge 39 commits into
trangevi wants to merge 39 commits into
Conversation
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
…ll add back later Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Adds a workflow skill under .github/skills/agent-scenario-tests/ that resolves the current branch's PR, maps changed files to impacted cli-interactive-tester scenario tags, drives the matching scenarios through the tester MCP server, and posts a results comment on the PR. Cost-aware: Tier 2 runs only after explicit user confirmation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…coverage Add 9 cli-interactive-tester scenarios closing the coverage gaps found in the PR #8524 review: - eval (cmd:eval): 00-eval-context-required (offline endpoint-required) and 28-eval-lifecycle (Tier 2 init/run/list/show against the shared agent). - optimize (cmd:optimize): 00-optimize-apply-requires-candidate (offline required-flag) and 29-optimize-submit-and-cancel (Tier 2, capped iteration). - invoke: 00-invoke-validate-protocol (offline unsupported-protocol) and 23-invoke-protocol-invocations (Tier 2 invocations memory semantics). - init: 00-init-validate-deploy-mode (offline value/required-flag validation) and 10-init-deploy-mode-container (Tier 1 container scaffold). - doctor: 00-doctor-partial-failure (mixed PASS+FAIL, exit 1). Add cmd:eval and cmd:optimize to the tag taxonomy, update the scenarios README tier tables, and update the agent-scenario-tests skill impact-mapping (eval and optimize are now covered Tier 2 commands, no longer listed as gaps). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A local scenario run revealed that 00-init-validate-deploy-mode never exercised
the --deploy-mode validation: in an empty directory with --no-prompt, init fails
earlier with 'template selection requires interactive mode' because
validateCodeDeployInput is only reached after an init method resolves.
Reclassify Tier 0 -> Tier 1, rename to 10-init-validate-deploy-mode.yaml, and
seed the from-code fixture so the from-code method resolves and the bogus
--deploy-mode value is actually rejected ('--deploy-mode must be container or
code'). Reaching the check scaffolds a starter template (network), hence Tier 1.
Note the late-validation UX (template scaffolded before the flag is validated)
as a report_finding. Update the README tier tables accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dings A full Tier 0+1+2 cli-interactive-tester run against a freshly deployed shared agent surfaced two scenario-accuracy issues (the CLI itself behaved correctly): - 28-eval-lifecycle: 'eval init --no-wait' is ASYNC — it submits dataset (datagen-*) and evaluator (evaluatorgen-*) generation jobs and writes eval.yaml, but does NOT create an eval 'run'. So 'eval list' legitimately shows 0 rows right after init and 'eval show' (no id) errors cleanly. Refined the header + goals to describe the async semantics and treat an empty list / eval-id-required message as expected rather than a failure. - 29-optimize-submit-and-cancel: the optimize command group is preview-gated per subscription. On a non-enrolled subscription both 'optimize' and 'optimize list' return a clean 400 SubscriptionNotRegistered (signup: aka.ms/ao/quickstart), so the submit->status->cancel lifecycle can't run. Documented the Agent Optimizer enrollment prerequisite and added a gating check that accepts the clean SubscriptionNotRegistered error as a valid outcome when enrollment is absent. Add cspell words: datagen, evaluatorgen, signup. All Tier 0 (13) and Tier 1 (2) scenarios and the Tier 2 setup/invoke/teardown passed; resources were fully torn down with azd down --force --purge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 6 new cli-interactive-tester scenarios covering three recently merged commands: Tier 0 (offline help validation): - 00-delete-help.yaml: validates azd ai agent delete --help output - 00-endpoint-show-help.yaml: validates azd ai agent endpoint show --help output - 00-code-download-help.yaml: validates azd ai agent code download --help output Tier 2 (cloud E2E, run between 2A-doctor and 2Z-teardown): - 2B-endpoint-show.yaml: shows endpoint config (table + JSON output) - 2C-code-download.yaml: negative-path test (container agent returns AgentNotCodeBased) - 2D-delete.yaml: deletes agent with --force, confirms removal via show All scenarios tested locally: 6/6 PASS. Co-authored-by: Jian Wu <wujia@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The 4 Tier 1 scenarios that copy fixtures used a hardcoded /mnt/c/Repos/... fallback path (Travis's machine). Replace with bash :? expansion so that missing AZD_AGENTS_FIXTURES fails immediately with a clear message instead of a cryptic 'No such file or directory'. Co-authored-by: Jian Wu <wujia@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a setup script that cross-compiles azd core and the azure.ai.agents extension as native Linux/amd64 binaries and installs them in WSL. This eliminates git safe.directory, TTY detection, and file-locking errors caused by running Windows azd.exe via WSL interop. Also updates: - Scenarios README: documents the WSL setup workflow - Skill prerequisites: adds native binary check for Windows hosts - Skill running-scenarios: adds timestamp postfix to session_ids for collision avoidance across parallel agent sessions Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Failed scenarios must not be retried unless the scenario's goals explicitly instruct it. Retrying masks flaky behavior and undermines the test suite's value as a regression signal. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Use subshell cd into module directory before go build (fixes 'outside main module' error when script is run from the scenarios directory) - Use full 40-char commit hash (git rev-parse HEAD) instead of --short, matching the required version format '<semver> (commit <full-hash>)' Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
# Conflicts: # cli/azd/extensions/azure.ai.agents/cspell.yaml
The skill workflow now includes Step 1b which executes setup-wsl.sh via the tester at the start of every run. This guarantees WSL always has the latest local code compiled as native Linux binaries, eliminating stale binary issues when code changes between runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The scenario goals are the contract — a scenario PASSES only when the product's actual behavior matches what the goals describe. The driving agent must verify goals were met, not rationalize why they weren't. Added 'never adapt around broken goals' rule: if goals reference a command/flag that doesn't exist or expect output that doesn't appear, fail the scenario instead of working around it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The shared_agent_name derivation now includes a compact timestamp
(MMDDHHmm) so concurrent runs on the same machine get unique Azure
resource names and working directories, preventing deployment conflicts.
Format: {prefix}-{shared_agent_suffix}-{ts}
Example: alice-basic-responses-07141038
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Picker navigation: update goals to test wrapping behavior (intentional)
instead of expecting bounded clamping
- fixtures_dir: replace AZD_AGENTS_FIXTURES env var with {fixtures_dir}
session_var auto-derived by the orchestrator (4 scenarios updated)
- Container deploy: clarify that startup command prompt is expected;
only entry-point/runtime prompts are code-deploy-only
- Make fixtures_dir derivation platform-neutral (not WSL-specific)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- eval init -> eval generate, --max-iterations -> --max-candidates (stale flags) - Remove agent.yaml references; agent definition now inline in azure.yaml - Rename 10-init-from-manifest-url -> 10-init-from-azure-yaml-url with correct URL - Update deploy-mode scenarios: code deploy auto-selected for Python templates - Flip 2C-code-download from negative to positive path (code deploy is default) - Rewrite 26-endpoint-update pre-hook to inject agentCard into azure.yaml - Fix 2B-endpoint-show goals (command shows protocols/version/auth, not URL) - Update 10-init-flags-agent-name-model with correct azure.yaml URL Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- 10-init-template-python: fix scaffold verification goal (no 'instructions' or literal 'model' in schema; model is via env var). Remove stale Container deploy goal (auto-selects code for templates). - 10-init-template-dotnet: same Container deploy fix (.NET also auto-selects code) - 26-endpoint-update: rewrite pre-hook with sed (embedded Python broke YAML parser) - 00-sample-list-text: relax goal for terminal buffer overflow (JSON scenario covers full data verification) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Location select: use 'search for and select' wording (8 scenarios) to guide text-based matching instead of fragile positional indexing - endpoint-update: replace sed with awk for reliable multi-line agentCard injection into azure.yaml - optimize: add --evaluator coherence flag (required by validate) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- sample-list-json-filters: assert --type agent returns non-empty results - optimize-submit-and-cancel: broaden gating check to any clean HTTP 400 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Other HTTP 400 errors (e.g. train_dataset.version required) are unexpected API errors and should be reported as bugs, not observations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- profile.yaml: fix stale gpt-4.1-mini comment (value is gpt-5.4-mini) - README: fix model default in profile table, soften tier 0 'offline' claim (sample-list needs network), extend tier 2 range to 2D, add 2B/2C/2D rows, add cmd:code and cmd:delete to tag taxonomy, fix eval description - impact-mapping: add code/delete command mappings, add agent_endpoint.go - workflow.md: remove stale eval/optimize coverage gap refs, extend range - running-scenarios.md: extend serial range to 2D - reporting.md: update sample report (eval has scenarios now) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… registration Previously the script manually copied the extension binary without registering it in azd's config, which meant azd could silently fall back to the published registry version instead of the dev build. Now the script uses the official azd extension tooling: 1. Builds azd core with sudo install to /usr/local/bin 2. Ensures microsoft.azd.extensions is available (auto-installs if needed) 3. azd x build → azd x pack --bundle → azd extension install <bundle.zip> 4. Strict version verification that fails the script if dev build isn't active Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Remove continue_on_error from agentCard injection hook (if injection fails, the scenario cannot succeed — let it fail fast) - Add pre-update version capture goal so the driving agent records the version BEFORE running endpoint update, enabling a concrete comparison Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Update mutually-exclusive validation scenario to use azure.yaml - Remove legacy agent.manifest.yaml mention from init-from-url comment Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.