Azure · trangevi · Jun 2, 2026 · Jun 2, 2026 · Jun 3, 2026 · Jun 3, 2026
@@ -38,6 +38,46 @@ replace github.com/azure/azure-dev/cli/azd => ../../
 
 That `replace` points this extension at your local `cli/azd` checkout instead of the version in `go.mod`. Do not merge the extension with that `replace` still present.
 
+## Interactive CLI test scenarios
+
+This extension ships a suite of goal-based scenarios for the
+[cli-interactive-tester](https://github.com/coreai-microsoft/cli-interactive-tester)
+MCP server under `tests/cli-interactive-tester-scenarios/`. They drive real
+`azd ai agent` flows end-to-end (init, provision, deploy, invoke, run, sessions,
+files, monitor, endpoint, doctor, down) and are organized by tier:
+
+- **Tier 0** — offline, no Azure auth, no cost (help, version, validation, picker UX)
+- **Tier 1** — local-only with Azure auth (init flows)
+- **Tier 2** — full cloud E2E against a deployed shared agent (incurs Azure cost)
+
+Each scenario carries a set of tags based on what is being tested and how.
+See `tests/cli-interactive-tester-scenarios/README.md` for the tag taxonomy,
+profile setup, and orchestration rules.
+
+### Guidance for coding agents
+
+These scenarios are **never run automatically** — they require the
+cli-interactive-tester MCP server, a populated `profile.local.yaml`, and
+(for Tier 2) real Azure resources. Do not invoke them on your own. Instead:
+
+1. **Surface them to the user** when you make a change that touches a
+   user-facing command path covered by an existing scenario (anything under
+   `internal/cmd/` that maps to a `cmd:*` tag, or shared helpers used by those
+   commands). In your summary, point the user at the relevant scenario(s)
+   and suggest they run the tester against the matching tag set to validate
+   the change.
+
+2. **Add or update a scenario** when your change introduces a new command,
+   flag, prompt, or user-visible flow — or meaningfully alters an existing
+   one. Place the new YAML alongside the others, follow the tagging taxonomy
+   documented in the scenarios README, and mention the new/changed scenario
+   in the PR description so reviewers know to exercise it.
+
+3. **Do not modify scenarios to match buggy behavior.** Scenarios are
+   user-facing specifications of how the command should behave; if a scenario
+   fails because of your change, prefer fixing the code unless the behavior
+   change is intentional and documented.
+
 ## Error handling
 
 This extension uses `internal/exterrors` so the azd host can show a useful message, attach an optional suggestion, and emit stable telemetry.

@@ -41,6 +41,7 @@ words:
   - CLIENTSECRET
   - curr
   - dataagent
+  - defaultyourvalue
   - envkey
   - exterrors
   - goyaml

@@ -0,0 +1,6 @@
+# cli-interactive-tester run artifacts (screenshots, HTML reports, scrollback)
+.reports/
+
+# Per-developer / per-CI scenario profile (identifying values). Bootstrap
+# from profile.local.yaml.example. See README "Profile / overrides".
+profile.local.yaml
@@ -0,0 +1,19 @@
+# Tier 0 (offline) — `doctor` in an empty directory degrades gracefully.
+name: "doctor-empty-dir"
+command: "azd ai agent doctor"
+cwd: "~/working/azd-agents-doctor-empty-{instance}"
+tags: ["tier:0", "cmd:doctor", "parallel-safe"]
+
+# Guarantee an empty working dir so the "no azd project" path is exercised.
+# start_session recreates the dir, so removing it is enough.
+pre:
+  - run: "rm -rf ~/working/azd-agents-doctor-empty-{instance}"
+    cwd: "~/working"
+    name: "reset to an empty working dir"
+
+goals:
+  - "Run doctor in a directory that has no azd project. Wait for the check report to render."
+  - "Confirm the command reports checks as skipped (or failed) with readable, non-crashing output — it should NOT panic or print a Go stack trace."
+  - "Note the exit code behavior described in help: 2 when all checks are skipped (preconditions unmet), 1 on failure, 0 when at least one passes."
+  - "Take a screenshot of the doctor report."
+  - "Report a finding if doctor crashes, prints a stack trace, or gives an unhelpful/confusing message for a missing project."
@@ -0,0 +1,18 @@
+# Tier 0 (offline) — `doctor --local-only` skips remote checks.
+name: "doctor-local-only"
+command: "azd ai agent doctor --local-only"
+cwd: "~/working/azd-agents-doctor-local-{instance}"
+tags: ["tier:0", "cmd:doctor", "parallel-safe"]
+
+# Guarantee an empty working dir for a deterministic local-only run.
+pre:
+  - run: "rm -rf ~/working/azd-agents-doctor-local-{instance}"
+    cwd: "~/working"
+    name: "reset to an empty working dir"
+
+goals:
+  - "Run doctor with --local-only. Wait for the report to render."
+  - "Confirm only local checks are attempted and remote/network-dependent checks are skipped (the report should indicate skipped remote checks)."
+  - "Confirm the run completes without attempting any network calls and without crashing."
+  - "Take a screenshot of the report."
+  - "Report a finding if remote checks still run, or if the output is confusing about what was skipped."
@@ -0,0 +1,12 @@
+# Tier 0 (offline) — verify root help lists every expected subcommand.
+name: "help-root"
+command: "azd ai agent --help"
+cwd: "/tmp"
+tags: ["tier:0", "cmd:help", "parallel-safe"]
+
+goals:
+  - "Wait for the help output to render (it includes an ASCII-art banner and a 'Usage:' section)."
+  - "Confirm the 'Available Commands' list includes: doctor, endpoint, eval, files, init, invoke, monitor, optimize, run, sample, sessions, show, version."
+  - "Confirm the global flags are listed: --cwd/-C, --debug, --environment/-e, --no-prompt, --output/-o."
+  - "Take a screenshot of the help output."
+  - "Report a finding if any listed command is missing, if the usage text is malformed, or if the command exits non-zero."
@@ -0,0 +1,37 @@
+# Tier 0 (offline) — exercise the `init` interactive picker UX, then abort.
+#
+# This scenario probes the early interactive prompts only (init mode, language,
+# template list). It intentionally ABORTS with Ctrl-C before reaching any Azure /
+# Foundry project prompts, so it needs no Azure auth and creates no resources.
+name: "init-picker-navigation"
+command: "azd ai agent init"
+cwd: "~/working/azd-agents-picker-{instance}"
+tags: ["tier:0", "cmd:init", "picker", "parallel-safe"]
+
+env:
+  AZD_DISABLE_AGENT_DETECT: "1"
+
+# Clean dir, then seed a file so the dir is non-empty. The init-method picker
+# ("Use the code in the current directory" / "Start new from a template") only
+# appears when the working directory is NOT empty; on an empty dir the command
+# auto-selects the template flow and skips the picker.
+pre:
+  - run: "rm -rf ~/working/azd-agents-picker-{instance}"
+    cwd: "~/working"
+    name: "reset working dir"
+  - run: "mkdir -p ~/working/azd-agents-picker-{instance} && touch ~/working/azd-agents-picker-{instance}/placeholder.txt"
+    cwd: "~/working"
+    name: "seed a placeholder file so the init-method picker appears"
+
+goals:
+  - "Wait for the first prompt asking how to initialize (it should appear because the directory is non-empty): 'Use the code in the current directory' / 'Start new from a template'."
+  - "Select 'Start new from a template', then wait for the language prompt."
+  - "Select a language (Python), then wait for the template list prompt."
+  - "On the template list, type a partial search string to filter the list; confirm the list narrows to matching entries."
+  - "Type a string that matches nothing; confirm the list shows an empty/no-match state and does not crash."
+  - "Clear the filter (backspace) and confirm the full list returns."
+  - "Press Down many times past the end of the list; confirm selection stays bounded and does not error."
+  - "If a prompt mentions a '?' hint, press '?' and confirm a helpful hint is shown."
+  - "Take a screenshot at each interesting state."
+  - "Press Ctrl-C (or Escape at the top-level prompt) to abort. Confirm it exits cleanly without a stack trace and without leaving a half-written azure.yaml."
+  - "Report a finding for any picker glitch: filter not recovering, crash on no-match, unbounded scrolling, unhelpful hints, or a messy/abrupt abort."
@@ -0,0 +1,25 @@
+# Tier 0 (offline) — `init` rejects a positional argument combined with --manifest.
+#
+# There is no `--from-code` flag. A real, offline-detectable conflict is passing
+# BOTH a positional manifest argument AND -m/--manifest, which the command
+# rejects up front (CodeConflictingArguments) before any wizard or network call.
+name: "init-validate-mutually-exclusive"
+command: "azd ai agent init agent.manifest.yaml -m https://example.com/agent.manifest.yaml"
+cwd: "~/working/azd-agents-validate-{instance}"
+tags: ["tier:0", "cmd:init", "negative-path", "parallel-safe"]
+
+env:
+  AZD_DISABLE_AGENT_DETECT: "1"
+
+# Start from a clean dir so leftover files can't affect the validation result.
+pre:
+  - run: "rm -rf ~/working/azd-agents-validate-{instance}"
+    cwd: "~/working"
+    name: "reset working dir"
+
+goals:
+  - "Wait for the command to fail fast (it should not start the interactive wizard)."
+  - "Confirm the error message clearly states that you cannot pass both a positional argument and --manifest."
+  - "Confirm the process exits non-zero and does NOT create or modify any project files in the directory."
+  - "Take a screenshot of the error output."
+  - "Report a finding if the conflicting arguments are silently accepted, if the wizard starts anyway, or if the error message is unclear about the conflict."
@@ -0,0 +1,21 @@
+# Tier 0 (offline) — `init --no-prompt` with no resolvable inputs fails helpfully.
+name: "init-validate-no-prompt-missing"
+command: "azd ai agent init --no-prompt"
+cwd: "~/working/azd-agents-validate-noprompt-{instance}"
+tags: ["tier:0", "cmd:init", "negative-path", "parallel-safe"]
+
+env:
+  AZD_DISABLE_AGENT_DETECT: "1"
+
+# Empty dir is a precondition: no existing code/manifest to resolve from.
+pre:
+  - run: "rm -rf ~/working/azd-agents-validate-noprompt-{instance}"
+    cwd: "~/working"
+    name: "reset to an empty working dir"
+
+goals:
+  - "Run init in --no-prompt mode in an empty directory with no flags and no existing code/manifest."
+  - "Confirm the command does NOT hang waiting for input (no-prompt must never block on a prompt)."
+  - "Confirm it exits non-zero with a helpful message explaining what required value or decision could not be resolved automatically (e.g. needing --src, --manifest, or --project-id)."
+  - "Take a screenshot of the error output."
+  - "Report a finding if it hangs, prompts interactively despite --no-prompt, or gives an unhelpful error."
@@ -0,0 +1,13 @@
+# Tier 0 (offline) — `sample list` JSON output and filter flags.
+name: "sample-list-json-filters"
+command: "bash"
+cwd: "/tmp"
+tags: ["tier:0", "cmd:sample", "parallel-safe"]
+
+goals:
+  - "Run: azd ai agent sample list --output json. Confirm the output is valid JSON (an array/object of samples)."
+  - "Run: azd ai agent sample list --language python --output json. Confirm results are filtered to python samples only."
+  - "Run: azd ai agent sample list --type agent --output json. Confirm results only include agent-type templates."
+  - "Run: azd ai agent sample list --featured-only --output json. Confirm only featured samples are returned (a subset of the full list)."
+  - "Take a screenshot after each command."
+  - "Report a finding if any command produces invalid JSON, ignores its filter, or errors."
@@ -0,0 +1,11 @@
+# Tier 0 (offline) — `sample list` renders the human-readable catalog.
+name: "sample-list-text"
+command: "azd ai agent sample list"
+cwd: "/tmp"
+tags: ["tier:0", "cmd:sample", "parallel-safe"]
+
+goals:
+  - "Wait for the curated sample catalog to render as human-readable text."
+  - "Confirm at least one sample entry is shown with a name/title and a manifest or repo reference usable with 'azd ai agent init -m' or 'azd init -t'."
+  - "Take a screenshot of the catalog output."
+  - "Report a finding if the list is empty, truncated, or the command errors."
@@ -0,0 +1,11 @@
+# Tier 0 (offline) — verify `azd ai agent version` prints a version string.
+name: "version"
+command: "azd ai agent version"
+cwd: "/tmp"
+tags: ["tier:0", "cmd:version", "parallel-safe"]
+
+goals:
+  - "Wait for the command to print a version string (e.g. a value like '0.1.x-preview' or a commit-based 'vdev' build identifier)."
+  - "Confirm the process exits cleanly without an error or stack trace."
+  - "Take a screenshot of the final output."
+  - "Report a finding if no version is printed, if the output is empty, or if the command errors."
@@ -0,0 +1,40 @@
+# Tier 1 (auth, scaffold only) — interactive code-deploy mode (entry point + runtime).
+#
+# Requires Azure login (see README "Authentication"). Does NOT run `azd provision`;
+# no cost incurred.
+# Targets the --deploy-mode code path which prompts for entry-point and runtime
+# (instead of building a container image). There is no --from-code flag; the
+# from-code flow is selected interactively at the init-method prompt.
+name: "init-deploy-mode-code"
+command: "azd ai agent init --deploy-mode code"
+cwd: "~/working/azd-agents-t1-code-deploy-{instance}"
+tags: ["tier:1", "cmd:init", "parallel-safe"]
+
+env:
+  AZD_DISABLE_AGENT_DETECT: "1"
+
+# Seed a committed Python fixture so code-deploy has real source to package.
+# Override the fixture location with AZD_AGENTS_FIXTURES if needed.
+pre:
+  - run: "rm -rf ~/working/azd-agents-t1-code-deploy-{instance}"
+    cwd: "~/working"
+    name: "reset working dir"
+  - run: "mkdir -p ~/working/azd-agents-t1-code-deploy-{instance} && cp -r \"${AZD_AGENTS_FIXTURES:-/mnt/c/Repos/azure-dev/cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/fixtures}/from-code/.\" ~/working/azd-agents-t1-code-deploy-{instance}/"
+    cwd: "~/working"
+    name: "seed from-code agent fixture (app.py + requirements.txt)"
+
+goals:
+  - "RESOURCE NAMING: whenever you are prompted to NAME a new Azure resource you are creating (Foundry project/account, azd environment, agent, model deployment, or resource group), give it a name prefixed with '{prefix}-' and suffixed with this run's instance id '-{instance}' (e.g. '{prefix}-basic-responses-{instance}') so concurrent instances don't collide on resource names. Some fields lowercase the value and replace invalid characters with hyphens;   that normalization is expected. If a name prompt comes pre-filled with a default value, CLEAR it first (select-all then delete, or backspaces) before typing so your name replaces the default instead of appending to it."
+  - "At the first 'How do you want to initialize your agent?' prompt, select 'Use the code in the current directory'."
+  - "Wait for the tool to inspect the current directory's code with code-deploy (ZIP upload) mode selected."
+  - "If an existing agent manifest is detected, confirm reuse."
+  - "When prompted for an entry point, provide the main entry file (e.g. 'app.py')."
+  - "When prompted for a runtime, select an appropriate runtime (e.g. 'python_3_13')."
+  - "If asked to select an Azure AI Foundry project, choose to create a new one, naming it with the '{prefix}-' prefix, and follow the prompts."
+  - "If asked to select a subscription, search for and select the '{subscription}' subscription."
+  - "If asked for a location/region, select '{region}'."
+  - "If asked to select a model, choose '{model}' and accept the remaining model defaults."
+  - "Wait for initialization to complete — look for 'Next:' or a success message."
+  - "Verify the scaffold: confirm azure.yaml/agent.yaml reflect code-deploy mode with the chosen entry point and runtime, and that a .agentignore file controls ZIP packaging."
+  - "Take a screenshot of the completed init output."
+  - "STOP here — do NOT run 'azd provision'. Report a finding if the entry-point or runtime prompts are missing, confusing, or not persisted."
@@ -0,0 +1,39 @@
+# Tier 1 (auth, scaffold only) — init from a manifest with explicit --agent-name and --model.
+#
+# Requires Azure login (see README "Authentication"). Does NOT run `azd provision`;
+# no cost incurred.
+# Also requires GitHub login: `gh auth login` (manifest download can fall back to
+# the gh CLI when the anonymous GitHub API is rate-limited). The pre hook fails
+# fast if gh is not authenticated.
+# Verifies that the override flags are honored in the generated files.
+name: "init-flags-agent-name-model"
+command: "azd ai agent init -m https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/agent-framework/responses/01-basic/agent.manifest.yaml --agent-name {prefix}-qa-named-agent-{instance} --model {model}"
+cwd: "~/working/azd-agents-t1-flags-{instance}"
+tags: ["tier:1", "cmd:init", "parallel-safe"]
+
+env:
+  AZD_DISABLE_AGENT_DETECT: "1"
+
+# Require GitHub auth up front (like az login), then start from a clean dir so a
+# prior run's scaffold can't trigger overwrite prompts.
+pre:
+  - run: "gh auth status || { echo 'ERROR: GitHub CLI not authenticated. Run: gh auth login'; exit 1; }"
+    cwd: "~/working"
+    name: "require gh auth login (manifest download)"
+  - run: "rm -rf ~/working/azd-agents-t1-flags-{instance}"
+    cwd: "~/working"
+    name: "reset working dir"
+
+goals:
+  - "RESOURCE NAMING: whenever you are prompted to NAME a new Azure resource you are creating (Foundry project/account, azd environment, agent, model deployment, or resource group), give it a name prefixed with '{prefix}-' and suffixed with this run's instance id '-{instance}' (e.g. '{prefix}-basic-responses-{instance}') so concurrent instances don't collide on resource names. Some fields lowercase the value and replace invalid characters with hyphens;   that normalization is expected. If a name prompt comes pre-filled with a default value, CLEAR it first (select-all then delete, or backspaces) before typing so your name replaces the default instead of appending to it."
+  - "Wait for the manifest to download and parse."
+  - "When asked how to deploy, select 'Container' (hosted agent)."
+  - "If asked to select an Azure AI Foundry project, choose to create a new one, naming it with the '{prefix}-' prefix, and follow the prompts."
+  - "If asked to select a subscription, search for and select the '{subscription}' subscription."
+  - "If asked for a location/region, select '{region}'."
+  - "Accept any remaining model defaults (version, SKU, capacity). If prompted for a model deployment name, use a '{prefix}-'-prefixed name."
+  - "If asked for container/resource size, select 'Small'."
+  - "Wait for initialization to complete — look for 'Next:' or a success message."
+  - "Verify the overrides: confirm agent.yaml records the Foundry agent name as '{prefix}-qa-named-agent-{instance}' (the flag value passed via --agent-name) and the model '{model}' — the values passed via flags, not the manifest defaults."
+  - "Take a screenshot of the completed init output."
+  - "STOP here — do NOT run 'azd provision'. Report a finding if --agent-name or --model is ignored, or if the wizard still prompts for these values despite the flags."