📚 Documentation Noob Test Report - 2026-03-10

[github-actions[bot]]

Summary

Date of Test: 2026-03-10
Testing Role: Complete beginner with no prior experience with GitHub Agentic Workflows
Pages Visited:

• Home page (/gh-aw/)
• Quick Start Guide (/gh-aw/setup/quick-start/)
• Creating Workflows (/gh-aw/setup/creating-workflows/)
• CLI Commands (/gh-aw/setup/cli/)
• Examples: Issue & PR Events (/gh-aw/examples/issue-pr-events/)
• Introduction/About Workflows (/gh-aw/introduction/overview/)
• Reference pages (Glossary, Frontmatter, Engines)

Overall Impression: The documentation is comprehensive and well-organized with excellent reference material. However, there are several barriers to getting started for complete beginners. The learning path is unclear, terminology is not defined upfront, and some critical commands are underdocumented.

---

🔴 Critical Issues (Blocking)

1. Missing add-wizard Command Documentation

• Page: Quick Start Guide, Step 2
• Issue: The guide instructs users to run gh aw add-wizard githubnext/agentics/daily-repo-status, but this command is not documented on the CLI Commands page
• Impact: Users cannot find help, understand parameters, or troubleshoot this critical first-run command
• Recommendation: Add add-wizard command section to CLI documentation with examples and parameter descriptions

2. "Frontmatter" Terminology Not Defined for Beginners

• Page: Quick Start, Step 4 & Creating Workflows
• Issue: The term "frontmatter" is used extensively without definition. New users don't understand this is the YAML section at the top of the workflow file
• Impact: Users confused about mandatory workflow structure; may not understand what to edit
• Recommendation: Introduce "frontmatter" with a clear definition and visual example in Quick Start and Creating Workflows pages, BEFORE asking users to edit it

3. Empty or Minimal "Get Started" Section

• Page: Site navigation /gh-aw/get-started/
• Issue: The URL exists but appears to have minimal content, yet docs reference it as a navigation destination
• Impact: Users searching for "Getting Started" material in the sidebar cannot find structured tutorials
• Recommendation: Create a "Getting Started" section that bridges Quick Start and Advanced Topics, or consolidate into Setup section

4. Overwhelming Frontmatter Reference for Beginners

• Page: Reference → Frontmatter Full (/gh-aw/reference/frontmatter-full/)
• Issue: The complete frontmatter documentation lists 100+ configuration options with minimal explanation
• Impact: Beginners immediately confronted with too many options; unclear which are essential vs optional
• Recommendation: Create a "Frontmatter Quick Reference" showing only the 10-15 most common fields for beginners, with clear "essential", "common", and "advanced" sections

---

🟡 Confusing Areas (Slowing Down Learning)

1. Three Paths for Creating Workflows Not Ranked

• Page: Creating Workflows
• Issue: The page presents Web Interface, VSCode/CLI agents, and Manual Editing as equal options without guidance on which is best for beginners
• Current Flow: "Here are 3 options, pick one"
• Impact: Beginners unsure which method to choose
• Recommendation: Add a decision table comparing:
  • Method | Difficulty | Setup Time | Interactivity | Best For
  • Web Interface | Easy | 2 min | Low | Quick one-off workflows
  • CLI Agent | Medium | 5 min | High | Interactive workflow design
  • Manual | Hard | 10 min | None | Advanced users

2. Jargon Without Definitions on Home Page

• Page: Home page hero section
• Issue: Terms like "Continuous AI", "safe outputs", "sandboxed execution" used without explanation
• Impact: Non-technical users may be intimidated before diving into docs
• Recommendation: Add inline definitions or "Learn More" links for technical jargon on home page

3. Example Workflows Not Embedded in Documentation

• Page: Examples section
• Issue: Example pages link to external repositories but don't show actual workflow markdown/YAML inline
• Impact: Beginners must jump between docs and GitHub repo to see actual workflow structure
• Recommendation: Include 1-2 real workflow examples inline on each examples page showing the full markdown and key frontmatter sections

4. AI Engine Selection Unclear

• Page: Quick Start Step 2 & Reference → Engines
• Issue: Users must choose between Copilot, Claude, Codex with different setup requirements, but no comparison table exists
• Current: Each engine documented separately in reference; users don't know how to choose
• Impact: Beginners might pick wrong engine and waste time on setup
• Recommendation: Add a "Choosing Your AI Engine" section with decision tree or comparison table showing:
  • Cost | Ease of Setup | Features | Best For | Auth Method

5. Sidebar Order Doesn't Match Learning Path

• Page: Site navigation
• Issue: Sidebar shows Quick Start (order: 1) → Creating Workflows (order: 1) → CLI Commands (order: 200), but there's no foundational introduction or "Why Agentic Workflows?" section visible in sidebar
• Impact: Users might skip important context about what workflows are and why they matter
• Recommendation: Add an "Introduction" section to sidebar above "Quick Start" with pages like:
  • What are Agentic Workflows?
  • How They Work (visual overview)
  • Common Use Cases

6. Custom Workflow Prompts Too Technical for Beginners

• Page: Creating Workflows → GitHub Web Interface
• Issue: Example prompts assume users know how to articulate workflow purpose: "The purpose of the workflow is to triage new issues: label them by type and priority, identify duplicates..."
• Impact: Non-technical users unsure how to write their own prompt
• Recommendation: Provide a "Workflow Prompt Worksheet" template:
  • What do you want to automate?
  • When should it run?
  • What information does it need?
  • What actions should it take?

7. Missing Visual Learning Path

• Page: All setup pages
• Issue: Quick Start has a video, but Creating Workflows and CLI pages are text-only
• Impact: Visual learners less engaged with advanced topics
• Recommendation: Add diagrams showing:
  • Workflow lifecycle (markdown → compile → run → action)
  • Trigger types and when to use each
  • Permission model and safe outputs

---

🟢 What Worked Well

Strengths of Current Documentation

1. Clear Value Proposition on Home Page

   • The tagline "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" immediately clarifies purpose
   • Action buttons ("Quick Start with CLI" / "Creating Workflows") are prominent and helpful

2. Prerequisite Checklist in Quick Start

   • Lists all 5 required items (AI Account, GitHub Repo, GitHub Actions, GitHub CLI, OS)
   • Links to official docs for setup when needed
   • Prevents user frustration from missing prerequisites

3. Video Tutorial in Quick Start

   • Shows the full workflow installation and first run in action
   • Demonstrates actual CLI usage, not just screenshots
   • Great for visual learners

4. Excellent Command Reference

   • CLI Commands page has clear table with descriptions
   • Links to detailed documentation for each command
   • Shows most common commands first

5. Comprehensive Reference Section

   • 40+ detailed reference pages covering every feature
   • Good resource for users ready to dive into details
   • Well-indexed and searchable

6. Good Error Recovery Information

   • Quick Start includes [!TIP] boxes for common auth issues
   • Links to FAQ and Common Issues pages
   • Troubleshooting guide exists

7. Diverse Example Coverage

   • Examples page shows multiple trigger types (events, scheduled, manual, multi-repo)
   • Demonstrates wide range of use cases
   • Shows workflows are flexible

8. "About Workflows" Explanation

   • Introduction page clearly contrasts agentic workflows vs traditional automation
   • Includes concrete example showing markdown input and expected behavior
   • Helps users understand the value proposition

---

📊 Summary of Findings

Category	Count	Severity
Critical (Blocking)	4	🔴
Confusing/Yellow	7	🟡
Positive/Working	8	🟢
Total	19

---

🎯 Recommended Improvements (Prioritized)

Quick Wins (1-2 hours each)

1. Add add-wizard command to CLI reference - Users need documentation for this critical first-run command
2. Define "frontmatter" in Quick Start - Add 1-2 sentences with visual example before asking users to edit it
3. Create "Choosing Your AI Engine" guide - Help users select Copilot, Claude, or Codex
4. Add comparison table for creating workflow methods - Show Web Interface vs CLI vs Manual with pros/cons

Medium Effort (3-4 hours each)

5. Create "Frontmatter Quick Reference" for beginners - 10-15 essential fields with explanations and examples
6. Add inline workflow examples to Examples pages - Show real markdown/YAML, not just links to external repos
7. Add decision tree/worksheet for custom workflow prompts - Help users articulate what they want to automate
8. Restructure sidebar to include Introduction section - Put foundational content before Quick Start

Longer-Term (1-2 days each)

9. Create visual workflow lifecycle diagram - Show markdown → compile → run → action with illustrations
10. Add videos for Creating Workflows and CLI pages - Match the quality of Quick Start video tutorial
11. Consolidate or clarify "Get Started" section - Either populate it properly or consolidate with Setup
12. Create "Frontmatter by Use Case" guide - Show different frontmatter configs for common workflow types

---

Conclusion

The documentation has a strong foundation with excellent reference material and a clear quick start guide. However, new users face friction around terminology, command documentation, and decision-making for choosing workflow methods and AI engines.

Key takeaway for new user experience: Beginners can follow the Quick Start video and get their first workflow running, but customizing that workflow or creating a new one from scratch requires navigating confusing documentation and undefined terminology. Better definition of terms upfront, documented commands, and clear decision paths would dramatically improve the getting started experience.

Test Rating: ⭐⭐⭐ (3/5) - Functional but needs beginner-focused improvements

• ✅ Quick Start works well
• ✅ Reference material is comprehensive
• ⚠️ Terminology confusing for beginners
• ❌ Command documentation incomplete
• ❌ Decision paths unclear

AI generated by Documentation Noob Tester · history

• expires on Mar 11, 2026, 7:57 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-11T07:57:55.564Z.

Closed by Workflow