Skip to content

Comments

feat: add AI-powered issue triage workflow#178

Merged
ayeshurun merged 215 commits intomicrosoft:mainfrom
ayeshurun:main
Feb 23, 2026
Merged

feat: add AI-powered issue triage workflow#178
ayeshurun merged 215 commits intomicrosoft:mainfrom
ayeshurun:main

Conversation

@ayeshurun
Copy link
Collaborator

@ayeshurun ayeshurun commented Feb 23, 2026

📥 Pull Request

Adds an AI triage workflow using github/ai-assessment-comment-labeler that automatically triages issues labeled needs triage.

What it does: Selects a prompt based on issue labels (bug/question/enhancement), runs an AI assessment via GitHub Models (gpt-4.1), posts a comment, applies labels, and escalates to @microsoft/fabric-cli-dev when needed. Supports re-triage when authors respond.

✨ Description of new changes

Prompts (.github/prompts/): Three self-contained prompts with full CLI command reference, standards (PEPs, RFCs), and concise response guidelines. Each maps assessment categories to specific label actions (escalate, help wanted, auto-close, request feedback).

Phase control: SUPPRESS_LABELS and SUPPRESS_COMMENTS env flags allow gradual rollout — set both to false for production.


Examples

### AI Assessment: Valuable Enhancement

This proposal directly addresses a usability gap in the managed private endpoint (MPE) creation flow. Handling a 401 Unauthorized during MPE status polling as a non-fatal, actionable state ("Pending") improves clarity and matches the experience of other Azure tools. The suggested message removes ambiguity and guides users when their principal lacks access to private endpoint connections. This approach maintains backward compatibility and avoids introducing breaking changes, as it only impacts post-creation status retrieval.

The design aligns with CLI conventions: error swallowing is justified when the underlying create operation is successful, and non-error status messaging is preferable to hard failures for expected authorization gaps. Mapping unverified status to "Pending" fits existing status semantics. The optional --no-status-check flag is a reasonable extension, but the proposed default behavior change already solves the core problem.

This enhancement is scoped, feasible, and will reduce support friction for common service principal scenarios.

**Next Steps**  
**📋 Backlog candidate** — This enhancement has been triaged and will be considered for the team's backlog.

---
> 💡 If this issue requires the team's attention and was not escalated, you can tag @microsoft/fabric-cli-dev to notify the team.
### AI Assessment: Potential Bug

The described behavior indicates that `fab ls` is inconsistently handling workspace and folder paths. Listing the workspace root returns items as if they are recursively enumerated through subfolders, but listing a specific folder results in a path error. Per documented hierarchy and command reference ([Hierarchy docs](https://microsoft.github.io/fabric-cli/essentials/hierarchy/), [ls command doc](https://microsoft.github.io/fabric-cli/commands/fs/ls/)), the CLI should support listing valid folders without invalid path errors and not automatically recurse unless specifically commanded.

The invalid path error in CASE 2 suggests either incorrect parsing of folder paths or broken entity resolution logic, especially since the workspace path is accepted but folder paths are not. The recursive listing in CASE 1 deviates from standard `ls` utility conventions; items should not be flattened into compound names unless requested, and folders should be navigable directly.

Severity is elevated due to potential regression in folder path resolution and entity hierarchy traversal, affecting access to nested resources and breaking expected filesystem semantics.

**Next Steps**
**🔔 Escalated to team** — This issue requires team review and has been flagged for attention.

---
> 💡 If this issue requires the team's attention and was not escalated, you can tag @microsoft/fabric-cli-dev to notify the team.
### AI Assessment: Potential Bug

The report describes a discrepancy between expected and actual output format for the `fab exists` command when `--output_format json` is specified. Boolean results are rendered as strings ("true" or "false") rather than native JSON boolean values (true or false). This behaviour diverges from conventional JSON serialization and creates downstream usability and interoperability issues, especially in automation and scripting environments.

The CLI documentation specifies that output format can be `json` or `text` (https://microsoft.github.io/fabric-cli/essentials/output_format/), implying strict compliance with JSON standards (RFC 7159/8259). Returning a string where a boolean is expected is inconsistent with standard JSON behavior and can cause logic errors in consumer scripts (e.g., treating "false" as truthy). The issue is plausibly reproducible given the provided steps and output samples.

This does not appear to be a misconfiguration or a documented CLI limitation. Correction may require changes in the result serialization logic to ensure type fidelity.

**Next Steps**

**🔔 Escalated to team** — This issue requires team review and has been flagged for attention.

---
> 💡 If this issue requires the team's attention and was not escalated, you can tag @microsoft/fabric-cli-dev to notify the team.

ayeshurun and others added 30 commits September 17, 2025 17:11
chore: Introduce release workflowc
Alon Yeshurun and others added 26 commits February 19, 2026 13:58
gpt-4o is being retired. gpt-5.2 is the current default in GitHub
Models with improved reasoning and instruction-following.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
gpt-5.2 does not exist in the catalog. gpt-4.1 is the best available
general-purpose model with 1M token context and high rate limit tier.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Switch model from openai/gpt-4.1 to openai/gpt-5 for better reasoning
- Add correct --output_format flag info (json/text, not --output)
- Add instruction to never invent CLI flags or features

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
gpt-5 returns 'unexpected error' on custom rate tier.
o3 is OpenAI's dedicated reasoning model for better accuracy.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
o3 and gpt-5 both fail on custom rate tier. gpt-4.1 is the best
available model on the high rate tier. Accuracy improvements from
prompt fixes (correct flags, anti-hallucination) remain in place.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Detect 'needs author feedback' from response text (awaiting author)
  in addition to assessment label (missing details)
- Remove redundant ai:* assessment labels when ai:needs team attention
  is added, keeping only the consolidated label

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When AI asks for author info, skip team notification. Team is only
notified after author responds and re-triage still can't resolve.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move ai:* label cleanup before the if/else branch so it runs
unconditionally. Prevents stale labels like ai:question-triage:needs
maintainer input from remaining on issues.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add 'Needs Author Feedback' assessment category to all 3 prompts
- Rename 'Missing Details' to 'Needs Author Feedback' in bug-triage
- Add new category to question-triage and feature-triage
- Remove redundant response text fallback and needsHumanReview override
- Update workflow label detection to match new category name

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Merge Needs Maintainer Input and Needs Team Review into single
  Needs Team Review category in question-triage prompt
- Remove duplicate needs author feedback label from post-processing
  (action's ai:* label is sufficient)
- Remove ai:* label cleanup (keep action's assessment labels)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract CLI commands, flags, item types, auth methods, and error
codes from actual source files. Inject into system prompt (not issue
body) so AI backs up answers with real codebase data. Comments and
issue body stay clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All AI triage comments now end with a footer instructing users
to tag @microsoft/fabric-cli-dev if the issue needs team attention
but wasn't escalated by the AI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Approach A (Action): github/ai-assessment-comment-labeler with codebase context injection
- Approach B (Code Agent): Custom scripts/ai_triage.py that greps source code and injects into prompt
- Both trigger on 'needs triage' label and post tagged comments (🅰️/🅱️)
- Both include collapsible label decision tables in comments
- Labels suppressed on both for fair testing
- Added CLI doc page URLs to all 3 prompt files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- A: Use yaml.safe_load/dump to avoid YAML alias errors from markdown **bold**
- A: Remove bold markdown from injected context text
- B: Reduce code context limits (30K total, 4K/file, 8 files max)
- B: Add API error details to stderr for debugging

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Bug: Potential Bug and Needs Team Review → Escalated to team
- Question: Tied each category explicitly to its footer
- Feature: Tied each category explicitly to its footer
- Removed stale Codebase Reference instruction from feature prompt

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove numbered dimension checklists from all 3 prompts
- Instruct model to skip adequate aspects, highlight only missing/wrong/notable
- Reduce target length from 3-5 to 2-4 paragraphs
- Trim verbose preamble in ai_triage.py code context header

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Valuable Enhancement → 📋 Backlog candidate (no escalation)
- Help Wanted → 🤝 Community contribution welcome
- Out of Scope → ✅ No action needed
- Add 'needs discussion' to needsHumanReview in both workflows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add tailored PEP/RFC/convention references to each prompt:
- Bug: packaging, CLI conventions, HTTP/REST, auth, file I/O, runtime
- Question: packaging, CLI conventions, HTTP/REST, auth, data formats
- Feature: CLI conventions, packaging, HTTP/REST, backward compat, auth

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mpts

- Remove Approach B (agent workflow + ai_triage.py), archived on branch
- Move codebase reference from pipeline build step into prompt files
- Prompts now self-contained: commands, flags, item types, auth, hidden entities
- Remove /tmp/prompts indirection, use .github/prompts directly
- Clean up A/B testing artifacts (tagged comments, approach labels)
- Restore phase control flags (SUPPRESS_LABELS + SUPPRESS_COMMENTS)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…inks

Reference https://microsoft.github.io/fabric-cli/essentials/resource_types/
instead of static lists, keeping prompts always up to date.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ayeshurun ayeshurun requested a review from a team as a code owner February 23, 2026 07:58
@ayeshurun ayeshurun merged commit c2eadeb into microsoft:main Feb 23, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants