[Agents Extension] Add test scenarios using the cli-interactive-tester tool#8524
Open
trangevi wants to merge 13 commits into
Open
[Agents Extension] Add test scenarios using the cli-interactive-tester tool#8524trangevi wants to merge 13 commits into
trangevi wants to merge 13 commits into
Conversation
Signed-off-by: trangevi <trangevi@microsoft.com>
github-actions
Bot
added
the
ext-agents
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
…ll add back later Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
Signed-off-by: trangevi <trangevi@microsoft.com>
trangevi
requested review from
JeffreyCA,
glharper,
therealjohn and
trrwilson
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a comprehensive, goal-based suite of manual interactive test scenarios for the azure.ai.agents azd extension, designed to be driven via the cli-interactive-tester MCP server. This codifies repeatable end-to-end command flows (from offline help/version checks through Tier 2 cloud provision/deploy/invoke) along with a profile/override mechanism and supporting fixtures.
Changes:
- Introduces a tiered scenario catalog (
00-,10-,2x-) with tagging conventions for selective runs and fleet orchestration. - Adds shared profile defaults (
profile.yaml), a local override template (profile.local.yaml.example), and gitignore rules for local profiles and run artifacts. - Adds a minimal “from-code” Python fixture used by scaffold-only init scenarios, and documents scenario usage in both the scenarios README and the extension
AGENTS.md.
Reviewed changes
Copilot reviewed 37 out of 37 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| cli/azd/extensions/azure.ai.agents/AGENTS.md | Documents the existence/intent of the manual cli-interactive-tester scenario suite and how contributors should use it. |
| cli/azd/extensions/azure.ai.agents/cspell.yaml | Adds a new word to prevent false-positive spellcheck failures from scenario docs. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/.gitignore | Ignores local profiles and tester output artifacts. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/README.md | Provides full orchestration guidance (tiers, tags, WSL path rules, auth prerequisites, hooks, and fleet mode). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/profile.yaml | Defines repo-shared default profile values (region/model/shared suffix). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/profile.local.yaml.example | Provides a template for per-user/per-CI identifying values (prefix/subscription/optional tenant). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/fixtures/from-code/app.py | Minimal Python source fixture for “init from existing code” scenarios. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/fixtures/from-code/requirements.txt | Minimal requirements file to ensure Python project detection during init-from-code flows. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-version.yaml | Tier 0 scenario for azd ai agent version. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-help-root.yaml | Tier 0 scenario validating root help output/command discovery. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-sample-list-text.yaml | Tier 0 scenario for sample list text rendering. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-sample-list-json-filters.yaml | Tier 0 scenario for sample list JSON output and filtering flags. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-doctor-empty-dir.yaml | Tier 0 scenario for doctor behavior in an empty directory. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-doctor-local-only.yaml | Tier 0 scenario for doctor --local-only. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-init-validate-mutually-exclusive.yaml | Tier 0 negative-path scenario validating init argument conflicts. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-init-validate-no-prompt-missing.yaml | Tier 0 negative-path scenario validating --no-prompt missing inputs behavior. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/00-init-picker-navigation.yaml | Tier 0 scenario focusing on init picker UX (filtering, navigation, abort behavior). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-template-python.yaml | Tier 1 scenario scaffolding from a Python template (auth required; stops before provision). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-template-dotnet.yaml | Tier 1 scenario scaffolding from a .NET template (auth required; stops before provision). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-from-manifest-url.yaml | Tier 1 scenario scaffolding from a GitHub manifest URL (auth + gh auth prerequisite). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-from-code.yaml | Tier 1 scenario for “use code in current directory” flow using seeded fixture. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-flags-agent-name-model.yaml | Tier 1 scenario validating --agent-name/--model overrides when initializing from a manifest URL. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/10-init-deploy-mode-code.yaml | Tier 1 scenario validating interactive code-deploy mode prompts (entry point/runtime). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/20-setup-deploy-shared-agent.yaml | Tier 2 setup scenario that provisions and deploys a shared agent used by subsequent Tier 2 scenarios. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/21-show.yaml | Tier 2 scenario validating show table output. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/21-show-json.yaml | Tier 2 scenario validating show --output json. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/22-invoke-remote.yaml | Tier 2 scenario validating remote invoke. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/22-invoke-new-session.yaml | Tier 2 scenario validating session vs conversation memory semantics for invoke. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/22-invoke-input-file.yaml | Tier 2 scenario validating invoke -f request-body-from-file behavior. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/23-sessions-lifecycle.yaml | Tier 2 scenario validating the sessions lifecycle command group. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/24-files-lifecycle.yaml | Tier 2 scenario validating the files lifecycle command group. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/25-monitor-console.yaml | Tier 2 scenario validating monitor console logs. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/25-monitor-system.yaml | Tier 2 scenario validating monitor system/container events. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/26-endpoint-update.yaml | Tier 2 scenario validating endpoint update behavior (patching without new version). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/27-run-local-and-invoke-local.yaml | Tier 2 scenario validating run + invoke --local with allocated ports and two sessions. |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/2A-doctor-provisioned-all-pass.yaml | Tier 2 scenario validating doctor against a provisioned project (with a known-acceptable warning). |
| cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/2Z-teardown-down.yaml | Tier 2 teardown scenario to destroy resources and clean the shared working directory. |
Signed-off-by: trangevi <trangevi@microsoft.com>
📋 Prioritization NoteThanks for the contribution! The linked issue isn't in the current milestone yet. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.