[workshop-sim] Workshop Simulation Report — 2026-07-25 (Run #11000, 1000×Monte Carlo)

[github-actions]

Overview

• Date: 2026-07-25
• Students simulated: 46 × 1,000 Monte Carlo runs (11,000 accumulated runs per student)
• Workshop steps available: 31/31 main steps (plus side quests)
• Overall success rate: 17.0% (95% Monte Carlo interval: 16.6%–17.3%)
• Highest-dropout step: 07-first-workflow — 33.0% conditional dropout among 23,349 at-risk runs (95% CI: 32.4%–33.6%)
• Lowest curriculum quality step: 07a-your-first-workflow-terminal.md (overall score 7.54/10)
• Learning KPI index: 7.30/10 (active_learning 4.38 · checkpoint_quality 8.55 · scaffolding 9.52)
• Model: unknown / 2026-07-assumption-model-v2 (parameter hash unknown)
• Limitation: synthetic results reflect explicit model assumptions; intervals exclude model and population-assumption uncertainty

Part Summary

Part	Files	Mean Score	Std Dev
Part 1 — core path (lessons 00–14)	19	8.99 / 10.0	±0.77
Part 2 — advanced (lessons 15+)	12	8.77 / 10.0	±0.40
Overall corpus	31	8.91 / 10.0	±0.64

No steps are classified as other.

Critical Findings

1. Step 07 is the single largest dropout gate (33% conditional, Part 1): workflow-authoring friction and Copilot billing configuration together account for the majority of failures at the most pivotal hands-on step. The terminal file 07a-your-first-workflow-terminal.md scores 7.54/10 overall with the lowest checkpoint_quality (5.0) in the corpus — learners advance without sufficient feedback that their compile passed correctly.
2. Conceptual overload through Steps 04–05c blocks 37% of at-risk learners before they reach any hands-on step: agentic-concept-gap and concept-overload are the two most common failure categories across all 46,000 runs. active_learning scores for 05-agentic-intro (2.9) and 05b-agentic-security (3.2) are well below the corpus mean.
3. Learning quality health is mixed: for learners who stay through the workshop, the learning KPI index (7.30/10) is acceptable — scaffolding (9.52) and checkpoint_quality (8.55) are strong — but active_learning (4.38) is consistently the weakest dimension across both Part 1 and Part 2. Most exercises are passive or observational rather than requiring learners to produce artifacts.
4. All five highest-dropout steps are in Part 1 (lessons 00–14): the core path has the heaviest completion burden. Repairs to Part 1 will have the largest impact on overall workshop completion.

Top Repairs to Prioritize

Note: some student dropout is expected and acceptable. Repairs must maintain or improve the learning KPI index — do not lower the cognitive bar or remove practice to chase headline completion numbers.

1. Strengthen Step 07a checkpoint and compile feedback — add an intermediate checkpoint after the first compile succeeds and before billing selection, so learners confirm the lock file was generated correctly (completion impact: ↑ · learning KPI impact: ↑)
2. Add active exercises to Steps 05 and 05b — replace or supplement the passive reveal-answer format with one short produce-an-output task per concept page (e.g., draft one line of frontmatter, classify a scenario) to increase the active_learning dimension from 2.9/3.2 toward the corpus mean (completion impact: ↔ · learning KPI impact: ↑)
3. Add a test-path section to Step 14b — explicitly instruct learners to open a draft PR to trigger the new PR reviewer workflow and verify a comment appears, with a short troubleshooting checklist for when the trigger does not fire (completion impact: ↑ · learning KPI impact: ↑)

Dropout by step

Step	At-risk runs	Conditional dropout	95% CI	Failure mode	Top reason
07-first-workflow	23,349	33.0%	32.4%–33.6%	Access + Learning barrier	Workflow authoring friction in Copilot Chat and billing path ambiguity
14b-pr-reviewer-workflow	13,794	25.3%	24.6%–26.0%	Learning barrier	Copilot skill guidance missing for event-driven workflow creation
05-agentic-intro	40,019	19.9%	19.5%–20.3%	Learning barrier	Conceptual gap between agentic workflow and standard Actions concepts
05c-agentic-practice	32,057	14.2%	13.8%–14.6%	Learning barrier	Cannot correctly classify agentic workflow components in exercises
05b-agentic-security	27,504	10.6%	10.2%–11.0%	Learning barrier	Security model concepts (sandboxing, safe-outputs) not fully absorbed
04-actions-intro	44,714	10.5%	10.2%–10.8%	Learning barrier	Concept overload for beginners with no prior Actions experience
12-test-and-iterate	14,561	5.3%	4.9%–5.6%	Learning barrier	Test and iterate cycle friction — unclear how to observe and modify
06-install-gh-aw	24,588	5.0%	4.8%–5.3%	Access barrier	Extension install friction in Codespace environment
17-add-mcp-tools	9,858	4.4%	4.0%–4.8%	Access + Learning barrier	MCP tooling configuration complexity
09-agentic-editing	15,081	3.4%	3.2%–3.8%	Learning barrier	Workflow editing friction when revising task briefs

Learning quality KPIs

Step file	Overall	active_learning	checkpoint_quality	scaffolding	Learning KPI	Repair priority
07a-your-first-workflow-terminal.md	7.54	4.6	5.0	10.0	6.22	🔴 High
05-agentic-workflows-intro.md	7.88	2.9	10.0	10.0	7.42	🔴 High
02a-setup-codespace.md	7.90	5.3	10.0	5.0	6.93	🟡 Medium
05b-agentic-workflows-security.md	8.64	3.2	10.0	10.0	7.53	🟡 Medium
14b-pr-reviewer-workflow.md	8.64	4.3	10.0	10.0	7.93	🟡 Medium
08-run-your-workflow.md	8.44	3.0	10.0	10.0	7.45	🟡 Medium
17-add-mcp-tools.md	8.36	3.1	10.0	10.0	7.49	🟡 Medium
15-conditional-logic.md	8.20	3.8	10.0	10.0	7.75	🟡 Medium
20-persistent-memory.md	8.25	3.7	10.0	10.0	7.71	🟡 Medium
07-your-first-workflow.md	9.37	0.0	0.0	10.0	2.73	⬜ Nav page
06-install-gh-aw.md	10.0	0.0	0.0	10.0	2.73	⬜ Nav page
00-welcome.md	10.0	0.0	0.0	5.0	1.36	⬜ Nav page
01-prerequisites.md	10.0	0.0	0.0	5.0	1.36	⬜ Nav page
Cohort mean	8.91	4.38	8.55	9.52	7.30	—

Curriculum quality metrics

Step file	Overall	Lowest rubric dimension	Recommended repair focus
07a-your-first-workflow-terminal.md	7.54	checkpoint_quality (5.0)	Add intermediate compile-success checkpoint before billing section
05-agentic-workflows-intro.md	7.88	active_learning (2.9)	Replace reveal-answer exercises with one produce-an-output task
02a-setup-codespace.md	7.90	scaffolding (5.0)	Add step-by-step scaffold for Codespace first-launch verification
15-conditional-logic.md	8.20	active_learning (3.8)	Add one branching scenario exercise learner must solve themselves
20-persistent-memory.md	8.25	active_learning (3.7)	Add a short write-the-memory-entry task before the reference example
05b-agentic-workflows-security.md	8.64	active_learning (3.2)	Add a classify-the-safe-output exercise at end of security concepts
14b-pr-reviewer-workflow.md	8.64	active_learning (4.3)	Add trigger-verification test step and troubleshooting checklist
08-run-your-workflow.md	8.44	active_learning (3.0)	Add an observe-and-interpret task for first workflow run output

Segment breakdowns

By technical level

Level	Students	Mean success rate
beginner	11	0.2%
github-basic	19	8.8%
actions-user	11	39.0%
advanced	5	36.3%

By personality

Personality	Students	Mean success rate
confused	6	19.3%
curious	15	14.6%
impatient	6	18.4%
methodical	12	17.4%
skeptical	7	18.1%

By UI preference

UI preferred	Students	Mean success rate
Yes	22	5.4%
No	24	27.5%

Notable student journeys (3)

Surprising success — Learner 026 (advanced/confused/devops): Despite a confused personality that often correlates with lower completion, this DevOps engineer with prior Actions experience achieved a 56.8% success rate. Their prior actions-user-adjacent knowledge of YAML triggers and permissions meant the conceptual sections were near-instant skips; their confusion manifested only at billing configuration, which they resolved through trial and error. Advanced technical background more than offset the personality penalty.

Unexpected dropout — Learner 006 (actions-user/impatient/backend-dev, SR=0.0%): Although Actions-familiar, this learner's impatient personality combined with 14b-pr-reviewer-workflow as the most common failure step suggests they consistently skipped the prerequisite model-access step, then hit an event-trigger failure they did not know how to debug. The lack of a guided test path in Step 14b meant they could not recover without re-reading earlier steps — an action impatient learners rarely take.

Content-gap case — Learner 008 (github-basic/enterprise-dev, SR=8.4%): This enterprise developer repeatedly dropped out at 05-agentic-intro despite having GitHub experience. The agentic-concept-gap failure category dominated their run history, pointing to a genuine conceptual gap: the enterprise Actions mental model (deterministic YAML, fixed runners) does not map well to the agentic reasoning loop described in Step 05. Adding a concrete enterprise-scenario analogy (e.g., "replace your hardcoded JIRA sync script with a task brief") to Step 05 would directly address this profile's barrier.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔬 Workshop Student Simulator · 64.5 AIC · ⌖ 5.35 AIC · ⊞ 10.5K · ◷

• expires on Jul 26, 2026, 4:07 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #2039

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.