[Agents Extension] Add test scenarios using interactive test tool

.github/skills/agent-scenario-tests/SKILL.md

@@ -0,0 +1,76 @@
+---
+name: agent-scenario-tests
+license: MIT
+metadata:
+  version: "1.0"
+  # Bump major on breaking prompt/trigger changes; bump minor on new references or mapping rules.
+description: >-
+  **WORKFLOW SKILL** — Runs the azure.ai.agents extension's cli-interactive-tester
+  scenarios locally as a PR regression check. Resolves the current branch's PR,
+  maps changed files to impacted scenario tags, drives the matching scenarios
+  through the cli-interactive-tester MCP server, and posts a results comment on the PR.
+
+  INVOKES: git CLI, gh CLI, cli-interactive-tester MCP tools (list_scenarios,
+  load_scenario, run_pre_hooks, start_session, send_action, finish_session,
+  run_post_hooks), ask_user.
+
+  USE FOR: run agent scenarios, scenario regression check, cli-interactive-tester,
+  test agents extension PR, run impacted scenarios, check agents extension for regressions,
+  agent scenario tests, validate azure.ai.agents change.
+
+  DO NOT USE FOR: azd core preflight (use azd-preflight), changelog (use changelog-generation),
+  creating PRs (use pull-request), authoring brand-new scenarios from scratch without a code
+  change, running scenarios for any extension other than azure.ai.agents.
+---
+
+# agent-scenario-tests
+
+Runs the `azure.ai.agents` extension's interactive CLI scenarios as a **local**
+PR regression gate and reports the results back on the pull request.
+
+## Overview
+
+The `azure.ai.agents` extension ships goal-based scenarios for the
+[cli-interactive-tester](https://github.com/coreai-microsoft/cli-interactive-tester)
+MCP server under `cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/`.
+These scenarios are **never run in CI** — they need the tester MCP server, a populated
+`profile.local.yaml`, manual `az`/`gh` login, and (for Tier 2) real Azure resources.
+
+This skill is the **opt-in, run-locally** flow a PR submitter uses to check their change
+for regressions. It:
+
+1. Resolves the current branch's PR link.
+2. Maps the PR's changed files to the impacted scenario **tag set** (`cmd:*` / `tier:*`).
+3. Enumerates and drives only the impacted scenarios through the tester.
+4. Posts a per-scenario results comment back on the PR.
+
+It is cost- and side-effect-aware: Tier 0 is free/offline, Tier 1 needs Azure auth but
+provisions nothing, **Tier 1b** (`verify-deploy`) provisions per-scenario Azure resources to
+verify Tier 1 scaffolds actually deploy, and **Tier 2** incurs Azure cost for cloud-feature
+testing — both Tier 1b and Tier 2 are only run after explicit user confirmation.
+
+> This skill drives scenarios **deliberately, with user consent**. That is different from
+> the extension's `AGENTS.md` rule that coding agents must not invoke scenarios on their
+> own during ordinary work — here the user has explicitly asked for a scenario run.
+
+{{ references/prerequisites.md }}
+
+{{ references/workflow.md }}
+
+{{ references/impact-mapping.md }}
+
+{{ references/running-scenarios.md }}
+
+{{ references/reporting.md }}
+
+## Exit Criteria
+
+- The current branch's PR was resolved (or the user supplied one / chose to skip the comment).
+- The impacted scenario set was derived from the PR diff and **confirmed by the user**
+  (including an explicit cost acknowledgement before any Tier 1b or Tier 2 run).
+- Every selected scenario was driven to completion with a recorded PASS/FAIL/SKIPPED, duration,
+  and any findings, and a `FINAL-REPORT.md` was written under `.reports/<run-timestamp>/`.
+- Scenarios with a `requires:` field whose prerequisite did not PASS are marked ⏭️ SKIPPED
+  (not FAIL) with a clear reason.
+- A results comment was posted on the PR (unless the user opted out), and any Tier 1b/Tier 2
+  run was followed by appropriate teardown so no Azure resources are left running.

.github/skills/agent-scenario-tests/references/impact-mapping.md

@@ -0,0 +1,85 @@
+<!-- cspell:ignore nextstep exterrors -->
+# Impact mapping — changed files → scenario tags
+
+Goal: from a PR's changed-file list, derive the **smallest** scenario tag set that still
+covers the change, plus the tier ceiling for cost gating.
+
+All paths below are relative to `cli/azd/extensions/azure.ai.agents/`.
+
+## 1. Command source → `cmd:*` tag
+
+Files under `internal/cmd/` map to the command they implement:
+
+| Changed file (glob) | Tag(s) | Notes |
+| --- | --- | --- |
+| `internal/cmd/init*.go` | `cmd:init` | Includes `init_from_code*`, `init_from_templates*`, `init_models`, `init_locations`, `init_validate`, `init_copy`, `init_foundry_resources_helpers`. |
+| `internal/cmd/show.go` | `cmd:show` | |
+| `internal/cmd/invoke*.go` | `cmd:invoke` | `invoke.go`, `invoke_raw.go`. |
+| `internal/cmd/run.go` | `cmd:run` | |
+| `internal/cmd/session.go` | `cmd:sessions` | |
+| `internal/cmd/files.go` | `cmd:files` | |
+| `internal/cmd/monitor*.go` | `cmd:monitor` | `monitor.go`, `monitor_format.go`. |
+| `internal/cmd/update.go` | `cmd:endpoint` | `update.go` defines `endpoint update`. |
+| `internal/cmd/doctor*.go` | `cmd:doctor` | `doctor.go`, `doctor_format.go`. |
+| `internal/cmd/eval*.go` | `cmd:eval` | `eval.go`, `eval_init.go`, `eval_run.go`, `eval_list.go`, `eval_show.go`, etc. Tier 2 (needs a deployed agent + Foundry endpoint). |
+| `internal/cmd/optimize*.go` | `cmd:optimize` | `optimize.go`, `optimize_apply.go`, `optimize_status.go`, etc. Tier 2 (submits a cloud optimization job). |
+| `internal/cmd/sample*.go` | `cmd:sample` | `sample.go`, `sample_list.go`. |
+| `internal/cmd/code*.go` | `cmd:code` | `code.go` (code download). |
+| `internal/cmd/delete*.go` | `cmd:delete` | `delete.go` (agent deletion). |
+| `internal/cmd/version.go` | `cmd:version` | |
+| `internal/cmd/root.go` | `cmd:help` + broad | Touches the whole command tree — treat as broad (see §3). |
+| `internal/cmd/listen.go` | — | gRPC host entrypoint; not scenario-testable. |
+
+## 2. Changed command with NO scenario coverage (gaps)
+
+These commands have **no** scenario in the suite yet. If the PR touches them, you cannot
+run a regression check — **report the gap** and recommend the author add a scenario
+(per the extension `AGENTS.md`), rather than silently passing:
+
+| Changed file (glob) | Uncovered command |
+| --- | --- |
+| `internal/cmd/mcp.go` | `mcp start` (hidden/preview) |
+
+## 3. Shared / cross-cutting code → broaden
+
+Changes outside a single command file affect many flows. When the diff touches any of
+these, broaden the impacted set (and ask the user how wide to go):
+
+| Changed file (glob) | Broaden to |
+| --- | --- |
+| `internal/cmd/helpers.go`, `internal/cmd/agent_context.go`, `internal/cmd/agent_endpoint.go`, `internal/cmd/*_context.go` | All `cmd:*` for commands that resolve project/agent context — at minimum `cmd:init`, `cmd:invoke`, `cmd:show`, `cmd:doctor`. |
+| `internal/cmd/root.go`, `internal/cmd/banner.go`, `internal/cmd/nextstep_output.go` | Run a Tier 0 smoke set (`tier:0`) across all commands. |
+| `internal/pkg/**`, `internal/project/**`, `internal/exterrors/**` | Map by what the package feeds: parsers/manifests → `cmd:init`; deployment/project target → `cmd:provision` + `cmd:deploy` (Tier 2). When unclear, propose a Tier 0/1 sweep and ask before any Tier 2. |
+| `go.mod` / `go.sum` / dependency bumps | Tier 0 smoke + ask whether a fuller sweep is warranted. |
+| files **outside** `cli/azd/extensions/azure.ai.agents/` (e.g. `cli/azd/` core) | This skill is scoped to the agents extension; note that core changes may need core azd testing instead, and proceed only with the agents-relevant subset. |
+
+## 4. Tier ceiling (cost gate)
+
+From the impacted `cmd:*` set, decide the **highest tier to offer**:
+
+- Default to **Tier 0 + Tier 1** for any change to a covered command (free + auth-only).
+- Offer **Tier 1b** (`verify-deploy`) when the change touches `cmd:init` code — these
+  scenarios verify that init scaffolds actually deploy. Since they reuse Tier 1 scaffolds,
+  any init change warrants running them. Tier 1b requires cost acknowledgement (provisions
+  real Azure resources) but is `parallel-safe`.
+- Offer **Tier 2** only when the change can plausibly affect cloud behavior — i.e. it
+  touches `cmd:invoke`, `cmd:sessions`, `cmd:files`, `cmd:monitor`, `cmd:endpoint`,
+  `cmd:show`, `cmd:run`, `cmd:eval`, `cmd:optimize`, `cmd:doctor` provisioned paths,
+  deployment/project code, or the provision/deploy flow. `cmd:eval` and `cmd:optimize`
+  are Tier 2-only (no offline happy path beyond their negative-path validation
+  scenarios). Tier 2 always requires the explicit cost confirmation in
+  `workflow.md` Step 4.
+- A pure Tier 0 change (e.g. `version.go`, help text, `sample list` formatting) should run
+  Tier 0 only.
+
+## 5. Translate tags → run list
+
+Combine the derived `cmd:*` tags with the chosen tier tags and call
+`list_scenarios(tags=[...])`. Example: an `invoke.go` change approved for Tier 2 →
+`list_scenarios(tags=["cmd:invoke"])`, then keep the Tier 0/1 results plus the Tier 2
+`2.*-invoke-*` scenarios, prefixed by `2.00-setup` and suffixed by `2.99-teardown`.
+
+For **Tier 1b** (`verify-deploy`): when `cmd:init` is impacted and the user approves
+cost-incurring tiers, include `list_scenarios(tags=["verify-deploy"])`. These scenarios
+have `requires:` fields — the orchestrator will check prerequisites at runtime and SKIP
+any whose Tier 1 prerequisite did not PASS.

.github/skills/agent-scenario-tests/references/prerequisites.md

@@ -0,0 +1,57 @@
+# Prerequisites
+
+Verify these before doing anything else. If a hard prerequisite is missing, stop and tell
+the user exactly what to fix — do **not** try to work around it.
+
+### Repo location
+
+1. Locate the scenarios directory:
+   `cli/azd/extensions/azure.ai.agents/tests/cli-interactive-tester-scenarios/`.
+   Resolve its absolute path; everything else is relative to it.
+2. Note the **WSL path** of that directory for MCP tool arguments. On Windows hosts the
+   tester runs inside WSL, so a Windows path like
+   `C:\Repos\azure-dev\...\scenarios\tier0\0.01-version.yaml` must be passed as
+   `/mnt/c/Repos/azure-dev/.../scenarios/tier0/0.01-version.yaml`. On macOS/Linux use the native
+   absolute path. See `running-scenarios.md` § Path style.
+
+### Tooling
+
+| Requirement | Check | If missing |
+| --- | --- | --- |
+| `git` + `gh` CLIs | `gh auth status` | Ask the user to run `gh auth login`. |
+| cli-interactive-tester MCP server | The `list_scenarios` / `start_session` MCP tools are available to you | Stop. Tell the user to register the cli-interactive-tester MCP server (see its README) and re-run. |
+| `profile.local.yaml` | File exists in the scenarios dir | Stop. Tell the user to `cp profile.local.yaml.example profile.local.yaml` and set `prefix` + `subscription`. |
+| Native Linux `azd` in WSL (Windows only) | Inside WSL, run `which azd` — must return `/usr/local/bin/azd` (not `/mnt/c/…` or a path ending in `azd.exe`). Then run `azd version` — must contain the expected dev version string. | **Hard stop.** If `which azd` returns a Windows interop path, the file-locking on UNC paths will fail all init/provision scenarios. Run Step 1b (`setup-wsl.sh`) to build and install native Linux azd. Do not proceed until both checks pass. |
+| Native Linux `azd` (native Linux/macOS) | `azd version` returns the expected dev build | Ask the user to build and install `azd` from source. No special path check is needed — any valid `azd` path works on native Linux. |
+
+### Auth (tier-dependent — only enforce for tiers actually selected)
+
+- **Tier 0** needs no auth.
+- **Tier 1 / Tier 2** read from / write to Azure. A human must `az login` inside WSL
+  **before** the run (the agent cannot complete the browser sign-in). If the selected set
+  includes Tier 1/2, remind the user to `az login` first.
+- **Manifest scenarios** (`1.03-init-from-azure-yaml-url`, `1.05-init-flags-agent-name-model`)
+  download from GitHub and can fall back to the `gh` CLI; they need `gh auth login` inside
+  WSL. Their `pre` hook fails fast if it isn't set up.
+
+### Profiles
+
+The scenarios reference `{prefix}`, `{subscription}`, `{region}`, `{model}`, `{tenant}`
+(optional) and `{shared_agent_name}` via placeholders. You must:
+
+1. Read both `profile.yaml` (checked-in defaults) and `profile.local.yaml` (developer
+   overrides) and **merge them, local overriding shared**.
+2. Derive `shared_agent_name = "{prefix}-{shared_agent_suffix}-{ts}"` where `{ts}` is
+   a compact timestamp of the form `MMDDHHmm` (e.g. `07141038`). This isolates
+   concurrent runs so two agents on the same machine don't collide on Azure resource
+   names or working directories.
+3. Derive `fixtures_dir` = the tester-side absolute path of the `fixtures/` subdirectory
+   inside the scenarios directory. On Windows (where the tester runs inside WSL) this is
+   the WSL-translated path (e.g. `/mnt/c/Repos/azure-dev/.../fixtures`); on native
+   Linux/macOS it is the regular absolute path. Apply the same path-style logic used for
+   scenario paths (see `running-scenarios.md` § Path style).
+   Scenario pre-hooks use `{fixtures_dir}` to locate test fixture files.
+4. Pass the merged map (including `shared_agent_name` and `fixtures_dir`) as `session_vars`
+   on **every** `load_scenario`, `run_pre_hooks`, `start_session`, and `run_post_hooks`
+   call. Omitting it leaves placeholders unresolved and the run executes against literal
+   `{prefix}` strings.

.github/skills/agent-scenario-tests/references/reporting.md

@@ -0,0 +1,59 @@
+# Reporting
+
+Two outputs: a local `FINAL-REPORT.md` artifact and a PR comment.
+
+## FINAL-REPORT.md
+
+Write to `<scenarios-dir>/.reports/<run-timestamp>/FINAL-REPORT.md` (the `.reports/` tree is
+git-ignored). Include:
+
+- Run header: timestamp, PR number/URL, branch, base ref, the derived tag set, and the
+  tiers actually run.
+- A per-tier table of scenarios with columns: `Scenario | Tier | Result | Duration | Findings`.
+- A short "Coverage gaps" section listing any changed command(s) with no scenario (from
+  `impact-mapping.md` §2), so the author knows to add one.
+- Links to the per-scenario `tester-reports/<run_name>/` folders for screenshots/HTML.
+
+## PR comment
+
+Post with `gh pr comment <number> --body-file <path>` (use a temp file to preserve
+formatting). Keep it scannable — full detail lives in the artifact. Suggested shape:
+
+```markdown
+## 🧪 Agent scenario regression check
+
+**Branch:** `<headRef>` → `<baseRef>` · **Run:** `<run-timestamp>`
+**Impacted tags:** `cmd:init`, `cmd:invoke` · **Tiers run:** 0, 1, 1b, 2
+
+| Scenario | Tier | Result | Duration |
+| --- | --- | --- | --- |
+| 0.01-version | 0 | ✅ PASS | 4s |
+| 1.01-init-template-python | 1 | ✅ PASS | 22s |
+| 1b.01-deploy-template-python | 1b | ✅ PASS | 3m 45s |
+| 1b.02-deploy-template-dotnet | 1b | ⏭️ SKIPPED | — |
+| 2.03-invoke-remote | 2 | ✅ PASS | 1m 12s |
+| 2.04-invoke-new-session | 2 | ❌ FAIL | 1m 40s |
+
+**Findings**
+- `1b.02-deploy-template-dotnet`: ⏭️ SKIPPED — prerequisite `tier1/1.02-init-template-dotnet.yaml` failed.
+- `2.04-invoke-new-session`: `--new-conversation` still recalled the prior name — memory
+  was not reset. (screenshot: …)
+
+**Coverage gaps:** this PR also touches `mcp.go`, which has no scenario — consider adding one.
+
+<sub>Run locally via the `agent-scenario-tests` skill. Not run in CI.</sub>
+```
+
+Rules:
+
+- Use ✅ PASS / ❌ FAIL / ⏭️ SKIPPED (and ⚠️ for a scenario that completed but raised a
+  non-fatal finding).
+- **SKIPPED** means the scenario's `requires:` prerequisite did not PASS — include the reason
+  in the Findings section (e.g. "prerequisite `tier1/1.01-…` failed"). SKIPPED scenarios
+  have no duration (`—`).
+- **Never** soften a real regression to make the table green. A scenario that failed because
+  of the PR's change is a FAIL — report it and recommend fixing the code, not the scenario.
+- If the user opted out of posting (or there's no PR), write only the artifact and print the
+  summary to the user instead.
+- Mention any Tier 2 teardown status explicitly (e.g. "`2.99-teardown-down` ran — no resources
+  left provisioned") so the reader knows nothing is still costing money.