feat: add AI-powered issue triage workflow#178
Merged
ayeshurun merged 215 commits intomicrosoft:mainfrom Feb 23, 2026
Merged
Conversation
fix: semantic pr
chore: Introduce release workflowc
gpt-4o is being retired. gpt-5.2 is the current default in GitHub Models with improved reasoning and instruction-following. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
gpt-5.2 does not exist in the catalog. gpt-4.1 is the best available general-purpose model with 1M token context and high rate limit tier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Switch model from openai/gpt-4.1 to openai/gpt-5 for better reasoning - Add correct --output_format flag info (json/text, not --output) - Add instruction to never invent CLI flags or features Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
gpt-5 returns 'unexpected error' on custom rate tier. o3 is OpenAI's dedicated reasoning model for better accuracy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
o3 and gpt-5 both fail on custom rate tier. gpt-4.1 is the best available model on the high rate tier. Accuracy improvements from prompt fixes (correct flags, anti-hallucination) remain in place. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Detect 'needs author feedback' from response text (awaiting author) in addition to assessment label (missing details) - Remove redundant ai:* assessment labels when ai:needs team attention is added, keeping only the consolidated label Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When AI asks for author info, skip team notification. Team is only notified after author responds and re-triage still can't resolve. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move ai:* label cleanup before the if/else branch so it runs unconditionally. Prevents stale labels like ai:question-triage:needs maintainer input from remaining on issues. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add 'Needs Author Feedback' assessment category to all 3 prompts - Rename 'Missing Details' to 'Needs Author Feedback' in bug-triage - Add new category to question-triage and feature-triage - Remove redundant response text fallback and needsHumanReview override - Update workflow label detection to match new category name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Merge Needs Maintainer Input and Needs Team Review into single Needs Team Review category in question-triage prompt - Remove duplicate needs author feedback label from post-processing (action's ai:* label is sufficient) - Remove ai:* label cleanup (keep action's assessment labels) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract CLI commands, flags, item types, auth methods, and error codes from actual source files. Inject into system prompt (not issue body) so AI backs up answers with real codebase data. Comments and issue body stay clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
All AI triage comments now end with a footer instructing users to tag @microsoft/fabric-cli-dev if the issue needs team attention but wasn't escalated by the AI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Approach A (Action): github/ai-assessment-comment-labeler with codebase context injection - Approach B (Code Agent): Custom scripts/ai_triage.py that greps source code and injects into prompt - Both trigger on 'needs triage' label and post tagged comments (🅰️ /🅱️ ) - Both include collapsible label decision tables in comments - Labels suppressed on both for fair testing - Added CLI doc page URLs to all 3 prompt files Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- A: Use yaml.safe_load/dump to avoid YAML alias errors from markdown **bold** - A: Remove bold markdown from injected context text - B: Reduce code context limits (30K total, 4K/file, 8 files max) - B: Add API error details to stderr for debugging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Bug: Potential Bug and Needs Team Review → Escalated to team - Question: Tied each category explicitly to its footer - Feature: Tied each category explicitly to its footer - Removed stale Codebase Reference instruction from feature prompt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove numbered dimension checklists from all 3 prompts - Instruct model to skip adequate aspects, highlight only missing/wrong/notable - Reduce target length from 3-5 to 2-4 paragraphs - Trim verbose preamble in ai_triage.py code context header Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Valuable Enhancement → 📋 Backlog candidate (no escalation) - Help Wanted → 🤝 Community contribution welcome - Out of Scope → ✅ No action needed - Add 'needs discussion' to needsHumanReview in both workflows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add tailored PEP/RFC/convention references to each prompt: - Bug: packaging, CLI conventions, HTTP/REST, auth, file I/O, runtime - Question: packaging, CLI conventions, HTTP/REST, auth, data formats - Feature: CLI conventions, packaging, HTTP/REST, backward compat, auth Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mpts - Remove Approach B (agent workflow + ai_triage.py), archived on branch - Move codebase reference from pipeline build step into prompt files - Prompts now self-contained: commands, flags, item types, auth, hidden entities - Remove /tmp/prompts indirection, use .github/prompts directly - Clean up A/B testing artifacts (tagged comments, approach labels) - Restore phase control flags (SUPPRESS_LABELS + SUPPRESS_COMMENTS) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…inks Reference https://microsoft.github.io/fabric-cli/essentials/resource_types/ instead of static lists, keeping prompts always up to date. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tarostok
approved these changes
Feb 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📥 Pull Request
Adds an AI triage workflow using
github/ai-assessment-comment-labelerthat automatically triages issues labeledneeds triage.What it does: Selects a prompt based on issue labels (bug/question/enhancement), runs an AI assessment via GitHub Models (
gpt-4.1), posts a comment, applies labels, and escalates to@microsoft/fabric-cli-devwhen needed. Supports re-triage when authors respond.✨ Description of new changes
Prompts (
.github/prompts/): Three self-contained prompts with full CLI command reference, standards (PEPs, RFCs), and concise response guidelines. Each maps assessment categories to specific label actions (escalate, help wanted, auto-close, request feedback).Phase control:
SUPPRESS_LABELSandSUPPRESS_COMMENTSenv flags allow gradual rollout — set both tofalsefor production.Examples
fab existsreturns strings instead of booleans in--output_format json#107: