[workshop-sim] Workshop Simulation Report — 2026-09-10 (Run #3000, 1000×Monte Carlo)

[github-actions]

Overview

• Date: 2026-09-10
• Students simulated: 46 × 1000 Monte Carlo runs
• Workshop steps available: 30/30
• Overall success rate: 23% (95% Monte Carlo interval: 22.8%–23.6%)
• Highest-dropout step: 05-agentic-intro (20.7% conditional dropout among 36,906 at-risk runs; 95% Monte Carlo interval: 20.3%–21.2%)
• Lowest curriculum quality step: 04-github-actions-intro.md (overall score 5.4/10)
• Learning KPI index: 2.9/10 (active_learning 4.2 · checkpoint_quality 0.0 · scaffolding 5.0)
• Model: 2026-07-survival-model-v2 / 2026-07-assumption-model-v2 (parameter hash 2024391902)
• Limitation: synthetic results reflect explicit model assumptions; intervals exclude model and population-assumption uncertainty

Part Summary

Part	Files	Mean Score	Std Dev
Part 1 — core path (lessons 00–14)	15	6.69 / 10.0	±1.74
Part 2 — advanced (lessons 15+)	15	6.07 / 10.0	±0.36
Overall corpus	30	6.38 / 10.0	—

No pages are classified as other.

Critical Findings

1. Three Part 1 concept pages (05-agentic-workflows-intro.md, 04-github-actions-intro.md, 07-your-first-workflow.md) drive the majority of dropout before learners reach the hands-on Step 8 run — concept density and task-stacking, not tooling access, are the primary blockers for beginners and no-coding-background learners.
2. checkpoint_quality scores 0.0/10 on every step in the corpus, including pages with clear, complete checklists (e.g. 05-agentic-workflows-intro.md, 07-your-first-workflow.md). Root cause: the shared rubric's CHECKPOINT_RE only matches a literal ✅ character, but the corpus consistently uses the :white_check_mark: emoji shortcode — a scoring-tool gap, not a missing-checkpoint gap. This artificially depresses the Learning KPI index (2.9/10) and should be fixed in the rubric detector before trusting checkpoint-quality-driven repairs.
3. The learning KPI index (2.9/10) indicates learners who persist through the workshop are not being well served by active-learning and checkpoint design — active_learning (4.2/10) and the (likely mis-scored) checkpoint_quality (0.0/10) are the weakest dimensions cohort-wide, while scaffolding (5.0/10) is moderate.
4. The most important repair belongs to Part 1 (00–14): 05-agentic-workflows-intro.md has both the highest conditional dropout (20.7%) and the highest cognitive_load (8.5/10) in the entire corpus, making it the single biggest lever for both completion and learning-quality gains.
5. A confirmed scoring-tool bug in the Monte Carlo simulator (word-boundary regex \b(commit|push)\b fails to match past-tense "committed and pushed") had inflated Step 8 dropout to 100% before this run's content-aware correction (see 07-first-workflow insight); the corrected model now shows Step 8 dropout at 1.4%, consistent with its well-scaffolded pre-flight checklist and browser-first design.

Top Repairs to Prioritize

Note: some student dropout is expected and acceptable. Repairs must maintain or improve the learning KPI index — do not lower the cognitive bar or remove practice to chase headline completion numbers.

1. Add an intermediate active-recall checkpoint between the "Three key terms" table and the "Two-file structure" section in 05-agentic-workflows-intro.md, and fix the emoji-shortcode/rubric mismatch (completion impact: ↑ · learning KPI impact: ↑)
2. Split 04-github-actions-intro.md's single long page into a shorter core refresher plus an optional deep-dive, adding one formative check-in after the anatomy diagram (completion impact: ↑ · learning KPI impact: ↑)
3. Reduce concept/task stacking in 07-your-first-workflow.md by moving the prompt-engineering explanation into a linked callout and keeping the page focused on the single authoring task (completion impact: ↑ · learning KPI impact: ↔)

Dropout by step

Step	At-risk runs	Dropouts	Conditional dropout rate	95% MC interval	Failure mode	Top reason
05-agentic-intro	36,906	7,652	20.7%	20.3%–21.2%	learning barrier	Learners cannot connect the agentic-workflow concept to what they'd actually deploy (agentic-concept-gap / deployment-capability-gap)
07-first-workflow	21,923	4,388	20.0%	19.5%–20.6%	access barrier	Learners struggle to translate the tutorial into a valid first workflow file (workflow-authoring-friction) or hit missing Copilot access (copilot-access-missing)
04-actions-intro	44,200	7,294	16.5%	16.2%–16.9%	learning barrier	Concept overload in the Actions refresher (concept-overload)
05c-agentic-practice	29,254	3,472	11.9%	11.5%–12.2%	learning barrier	Learners misclassify agentic vs. standard workflow examples (agentic-classification-gap)
05b-agentic-security	25,782	2,430	9.4%	9.1%–9.8%	learning barrier	Learners can't articulate the security guardrails (agentic-security-gap)
06-install-gh-aw	23,352	1,429	6.1%	5.8%–6.4%	access barrier	CLI install/auth friction (extension-install-friction)
17-add-mcp-tools	14,939	747	5.0%	4.7%–5.4%	learning barrier	General content-readiness friction at this advanced step
19-research-driven-training-node	13,663	597	4.1%	3.8%–4.5%	learning barrier	General content-readiness friction at this advanced step
02-setup	46,000	1,800	3.9%	3.7%–4.1%	access barrier	Codespace setup friction (setup-friction)
15-conditional-logic	15,806	597	3.8%	3.5%–4.1%	learning barrier	General content-readiness friction at this advanced step

Curriculum quality and learning KPIs

Step file	Overall score	active_learning	checkpoint_quality	scaffolding	Learning KPI index	Lowest rubric dimension	Repair priority
04-github-actions-intro.md	5.39	3.9	0.0	5.0	2.78	checkpoint_quality (rubric artifact)	High
05-agentic-workflows-intro.md	5.43	2.4	0.0	5.0	2.24	checkpoint_quality (rubric artifact)	High
05b-agentic-workflows-security.md	5.75	—	0.0	5.0	2.27	checkpoint_quality (rubric artifact)	Medium
05c-agentic-workflows-practice.md	6.22	6.2	0.0	5.0	3.62	checkpoint_quality (rubric artifact)	Low
07-your-first-workflow.md	6.25	6.3	0.0	5.0	3.65	checkpoint_quality (rubric artifact)	Medium
08-run-your-workflow.md	5.67	3.0	0.0	5.0	2.45	checkpoint_quality (rubric artifact)	Low (upstream-driven dropout, not this page)
Cohort mean (30 steps)	6.38	4.2	0.0	5.0	2.88	checkpoint_quality (rubric artifact)	—

Segment breakdowns

By technical level:

Level	Mean success rate	N
beginner	0.5%	11
github-basic	15.4%	19
actions-user	47.0%	11
advanced	50.5%	5

By personality:

Personality	Mean success rate	N
confused	21.9%	6
skeptical	22.1%	7
methodical	25.7%	12
curious	20.6%	15
impatient	27.4%	6

By UI preference:

Preference	Mean success rate	N
Prefers browser UI (ui_preferred: true)	12.0%	22
Prefers CLI (ui_preferred: false)	33.5%	24

Notable student journeys (3)

Surprising success: Learner 026 (advanced, confused personality, devops background, CLI-preferred) still reached a 61% success rate despite a "confused" personality tag — strong technical-level baseline and CLI comfort compensated for personality-driven hesitation, showing that background/level dominate personality effects once past Step 6.

Unexpected dropout: Several beginner/no-coding learners on the CCA tool path (Learners 015, 017, 018) hit 0% success, all failing at 04-actions-intro — the Actions refresher's concept density combined with no prior YAML/CI exposure is a harder wall for true beginners than the later agentic-specific content.

Content-gap case: 05-agentic-workflows-intro.md is the single highest-dropout step in the entire curriculum (20.7%) despite being well-written and having a complete checklist — the rubric under-scores its checkpoint entirely due to the :white_check_mark: vs. ✅ emoji-shortcode mismatch, masking that the real problem is concept stacking (three abstract terms + two multi-part activities with no intermediate recall check), not a missing checkpoint.

Generated by 🔬 Workshop Student Simulator · copilot · auto · 271 AIC · ⌖ 27.2 AIC · ⊞ 15.2K · ◷

• expires on Sep 11, 2026, 3:18 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #3157

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #3158

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.