test-perf: diagnose and cut the wall clock of the local test and gate corpus, especially on Windows Git Bash

Summary

Running the local test suites and the pre-PR gate scripts takes long enough on a Windows Git Bash host that a single phase close-out spends most of its wall clock waiting on them, and several individual scripts exceed the Claude Code Bash tool's two-minute default timeout on their own. The repository already documents the cause in scripts/affected-tests.sh: "The full corpus is tens of minutes of wall clock on a Windows box (Git Bash pays ~140ms per process spawn, and these suites are spawn-bound), which is long enough that nobody runs it locally and regressions reach CI." This issue asks for a measured program to troubleshoot, diagnose, and optimize test and gate performance so the corpus is as fast as it can be, with Windows as the primary target.

Observed (2026-09-04, Windows 11, Git Bash, worktree on the `feat/handoff-prompt-qol` branch)

Corpus size: 354 *.test.sh suites plus 42 Python test files under plugins/.
scripts/check-purged-em-dashes.sh exceeded the 120-second tool default on its first run (killed, exit 143) while scanning 105 files; it passed on a rerun under a 10-minute timeout.
scripts/check-changed-skills.sh origin/main exceeds 120 seconds for six touched skills and has to be backgrounded.
scripts/affected-tests.sh --run on a 56-file diff takes several minutes per run; two runs were needed this session.
plugins/session-flow/scripts/save_point.test.sh: 52 pytest cases take 10 to 14 seconds; plugins/session-flow/skills/retro/scripts/parse-transcript.test.sh: 38 cases take 2.6 to 5.4 seconds. Both are dominated by interpreter startup and per-case subprocess spawns, not by the assertions.
scripts/run-plugin-tests.sh keeps --jobs 1 as the default on Windows because a parallel run measured sublinear (the box saturates on process creation), and tool-honesty and jira conformance suites fail under run-plugin-tests --jobs 4, pass serially #3694 shows two suites fail under --jobs 4.
Side observation while diagnosing: with a merge in progress in the worktree, affected-tests.sh resolves its diff base to the fork point rather than the merge target, so files changed only on origin/main are reported as unmapped and the run exits 1 before executing any suite. Worth a guard or a clearer message.

Hook performance program: measure and cut every runtime-cost surface on the Windows host #3623 (hook performance program on the Windows host) and perf(hooks): Windows process-creation tax makes the hook set unusable - 1,778 timeouts / 20.6 hours blocked over 9 days #3508 (Windows process-creation tax on hooks): the same root cost, on the hook surface rather than the test surface.
perf(host): melo-desk-001 process creation costs 1.4 to 4 s system-wide; 3.81M leaked kernel Token objects (~10 GB paged pool) since boot #3715 (host-level process creation of 1.4 to 4 seconds system-wide): measurements for this issue should be taken on a host that does not carry that condition, or at least record it, so the baseline reflects the scripts rather than the machine.
ci: push-to-main wall time rose from 6 to 21 minutes after the six-job consolidation; parallelise test-linux on the push path #3705 (CI push-to-main wall time) and tool-honesty and jira conformance suites fail under run-plugin-tests --jobs 4, pass serially #3694 (suites that fail under --jobs 4): the CI-side and parallelism-side halves of the same problem.
Five suites fail only on a Windows Git Bash host; give them the host-skip pattern or fix them #3683 (five suites fail only on Windows Git Bash).

Proposed work

Instrument first. Add a timing mode to scripts/run-plugin-tests.sh and scripts/affected-tests.sh that records per-suite wall clock and a process-spawn count (a DEBUG trap or bash -x line count is enough on Git Bash where strace is unavailable), and commit a ranked baseline table so every later change is measured against it.
Diagnose the top offenders by mechanism: command substitutions inside per-item loops, grep/sed/jq/git invoked once per file or per line, find walks repeated per suite, mktemp -d per case, cygpath and uname per call, npx cold starts, and one pytest interpreter start per wrapper.
Optimize by the same pattern lib/hook-utils.sh already used (perf(hook-utils): build the telemetry envelope and read file_path with builtins #3678): builtins over spawns, one batched git call over many, one jq parse reused, one find walk hoisted, and one pytest invocation for co-located Python suites (with -p no:cacheprovider so no .pytest_cache is left for markdownlint to trip over).
Make parallelism safe on Windows: fix the suites in tool-honesty and jira conformance suites fail under run-plugin-tests --jobs 4, pass serially #3694 that fail under --jobs, then measure whether --jobs 2 or --jobs 4 pays on a healthy host and set the default accordingly.
Guard the result: a per-suite wall-clock budget lane (warn, then fail above a documented ceiling) so a regression is caught locally, and a documented fast local loop (the smallest command that covers a typical diff).

Acceptance

A committed baseline table of per-suite wall clock on Windows and on the Linux CI runner.
No single gate script in scripts/ exceeds 60 seconds on a healthy Windows host.
scripts/affected-tests.sh --run for a typical 50-file diff completes in under 2 minutes on Windows.
The full scripts/run-plugin-tests.sh corpus completes in under 10 minutes on Windows, with the number recorded.
Each optimization is behavior-preserving and shown so by the existing suites passing unchanged.

chore(topic-docs): prune the handoff-prompt-qol contract slice before…

feat(session-flow): shape-2 handoff save-points with the resume promp…

ci: shard test-linux four ways and gate the lint self-tests (#3773)

perf(guardrails): drop dirname and sed execs from the always-on path (#…

This was generated by AI during triage.

Merged from #3717 (duplicate)

#3717 was filed the same day, by the same author, on the same Windows Git Bash host, against the same root cost: per-process spawn overhead making the local gate and test corpus slow enough that a per-edit check becomes a per-phase check. Closing it as a duplicate of this issue, which carries the acceptance criteria. Carrying over its measurements, which are the per-script numbers this issue asks for in step 1 and did not yet have:

Script	Invocation	Measured
`plugins/skill-quality/scripts/check-skill.sh`	`--require-evals visualize` (one skill, markdownlint skipped)	> 120 s foreground timeout; finished under a 540 s background ceiling (two independent runs)
`scripts/check-changelog-parity.sh`	`--check-bump origin/main`	1248 s (one run); `--check` > 180 s; `--check-preserved origin/main` 16 s
`scripts/affected-tests.sh`	`--run` after an `evals/evals.json` edit	selection alone > 120 s; the basename reference rule selects ~148 suites for one evals file
`scripts/check-purged-em-dashes.sh`	bare	did not finish within a 600 s background window on one run (timing otherwise unverified)
`plugins/planning/scripts/check-open-questions.sh`	`--ledger … --brief …`	> 120 s foreground timeout; the `--ledger`-only form returns instantly

Conditions for those numbers: branch feat/visualize-absorb-show-me, base origin/main 8eae8e8, Claude Code 2.1.260, Windows 11, Git Bash, no node_modules until npm ci.

Suspected mechanisms #3717 named, for the step 2 diagnosis:

check-skill.sh is 1630 lines and counts codepoints through iconv, one spawn per file.
check-changelog-parity.sh is 728 lines and appears to diff every plugin's manifest and changelog against the base ref, so --check-bump scales with the fleet rather than the diff.
affected-tests.sh's basename reference rule maps any evals.json edit to every suite mentioning evals.json, hence ~148 suites for one plugin's file. This is a selection defect as much as a performance one.
check-open-questions.sh --brief reads the same ledger the fast path reads, so the cost is in the Brief extraction (extract_section), not the register parse.

Two scope items this issue's acceptance should absorb:

The acceptance criterion "no single gate script in scripts/ exceeds 60 seconds" does not reach two of the worst offenders above, which live in plugin trees: plugins/skill-quality/scripts/check-skill.sh and plugins/planning/scripts/check-open-questions.sh. Worth widening the criterion to plugin-scoped gates, or they get optimized last.
perf(scripts): local gate scripts take minutes on Windows Git Bash (check-skill, changelog-parity --check-bump, affected-tests, em-dash purge, check-open-questions --brief) #3717 asked for a wall-clock budget per gate published in each script's --help or the CI docs, so a contributor knows when to background a run rather than waiting on it. That complements this issue's step 5 budget lane: one is the enforcement, the other is the contributor-facing statement.

Also from #3717, worth stating in the framing: AGENTS.md tells every contributor to validate with scripts/affected-tests.sh --run. At these timings the honest path is "defer to CI", which makes this a documented-contract problem, not only a comfort problem.

Generated by Claude Code

fix(session-flow): treat a relocated handoff save-point as a WARN, no…

Additional measurements (2026-09-07, Windows 11, Git Bash, `perf/hook-performance-program`)

Session: comment-cleanup pass touching 10 files (ruff.toml, plugins/code-tidying/**, one claude-ops script).

scripts/affected-tests.sh --run --explain over that diff ran 40+ minutes with no completion and was killed.
Selection alone, no --run, takes 54 seconds wall clock (real 0m54.474s, user 0m0.962s, sys 0m3.860s): almost all of it is process-spawn overhead, not CPU.
plugins/claude-ops/skills/plugins/scripts/fleet-state.test.sh (2369 lines) exceeds the 600-second Bash tool maximum on its own, so no tool call can ever run it to completion here. It reached 15 PASS before the cap.
plugins/disk-hygiene/skills/clean/scripts/test_hygiene.py: a -k subset of 206 cases takes 65 seconds.

Over-selection compounds the cost

The reverse lookup matches on basename, so a change to plugins/code-tidying/skills/dissolve-comments/reference/safety.md selected three unrelated suites:

select: plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_merge.py  (references safety.md)
select: plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_merge_review_settle.py  (references safety.md)
select: plugins/source-control/skills/babysit-prs/scripts/tests/test_skill_contract.py  (references safety.md)

Those tests reference the babysit-prs safety.md, not the code-tidying one. A 10-file diff selected 23 suites. Matching on the repository-relative path, or at least on <skill>/<file>, would cut selection sharply for the common case of same-named reference files across skills.

Asks, in addition to the program above

Path-qualified reverse lookup instead of basename matching.
A per-suite wall-clock budget in CI that fails a suite exceeding it, so a suite cannot grow past what a developer can run locally. fleet-state.test.sh is already past 600 seconds.
A documented local escape hatch for the case where the affected set is known to be unrunnable, so a developer can record "NOT RUN locally, CI covers" without the change being reported as validated.

Same session, one more: scripts/check-changelog-parity.sh --check-order alone exceeded the 600-second Bash tool maximum on this host (2026-09-07, Windows 11, Git Bash) and was killed. A combined --check plus --check-bump main run earlier in the session finished, but only when backgrounded past the 180-second mark. Both are pure read-only gates over CHANGELOG.md files; they belong in the same spawn-count audit as the test suites.

feat(code-tidying): lift HARD path exclusions through three channels,…

fix(code-tidying): self-parse before triage, grant workflow scripts, …

perf(docs-hygiene): cut apply-rename suite spawns on Windows (#4171)

Re-triage. The owner restated on 2026-10-27 when an item stops for them:

Signal me (a needs-human / needs-decision label, or a direct question) only for:

A change to the original intent or vision.

A finding during implementation (things don't always go to plan) that changes the intent, vision or primary design.

A real HIGH/CRITICAL security or cost risk, as defined in section 3.

Everything else moves through the factory floor on its own. [...] Any other stop is a defect to fix, not a reason to ask me.

The umbrella is parked by the owner's 2026-09-29 decision (#3716 (comment)):

Owner decision (2026-09-29): the umbrella stays parked. It is not unparked as one program and not closed. If you want progress, the bounded piece to file is the path-qualified reverse lookup in affected-tests.sh (acceptance criterion 5).

The park was kept on 2026-10-13 (#3716 (comment)). Criterion 5, #4607 and #4608 have shipped.

Decided: the park stands as the owner set it. The umbrella is not worked as one program and is not admitted. The park says progress comes as a bounded piece filed as its own issue. It does not need the owner to sign each one, so needs-human is a stop outside the three reasons and comes off. Agents file each remaining piece as its own issue under this umbrella. Criterion 1 (the Windows-host baseline table) is filed as a host-run slice, since it is a measurement on the Windows host. This replaces the r29 classification that kept needs-human, which predates the 2026-10-27 rule.

Target labels: remove needs-human, add status: confirmed. The umbrella stays unadmitted.

New measurement for this issue (2026-10-29, Linux/WSL host, local run during PR #10610 work):

Suites run together	Local wall clock	Recorded CI seconds (`scripts/suite-seconds.txt`)
`plugins/planning/surface/test_round.py` + `test_server.py` + `test_exporters.py`	406 s, 411 s, 413 s (3 runs)	45 + 70 + 74 = 189 s

Local runs take about twice the recorded CI time for the same three suites. Per the timeout rule, the fix is speeding the suites up (or moving them by scripts/suite-seconds.txt balance), never a longer limit. The next step is a profile of test_server.py and test_exporters.py locally, to see whether a shared fixture (for example, per-test server start-up) dominates.

Status: criteria 5 (#5656) and 6 (#10230) are done. Criteria 1-4 are tracked in #10200 (timing option and the Linux column) and #10241 (the Windows column, the 60 s gate ceiling, the 2-minute affected-tests run and the 10-minute corpus). #10241's closing comment will say which of criteria 2-4 are met. Any per-offender slice it names is filed under this umbrella. This issue closes when those children close and every criterion has a child.

This was generated by AI.

Windows measurements from #10241, in #13139 (docs/test-wall-clock-baseline.md). Host: Windows 11 build 10.0.26300.9711, Git for Windows 2.56.0.windows.4, GNU bash 5.3.15, origin/main at 284b5303ae65, 2026-11-04. Fifty bash -c true launches averaged 37.2 ms each, so the host does not carry the #3715 condition.

Criterion 2, no gate script over 60 s: met. Medians of three runs over a one-commit skill diff (52a529d9cf~1..52a529d9cf): check-changed-skills.sh 17.414 s, check-purged-em-dashes.sh 17.376 s, check-skill.sh 15.631 s, check-changelog-parity.sh 0.525 s, check-open-questions.sh 0.214 s. Slowest single run: 31.174 s (cold cache).
Criterion 3, affected-tests.sh --run on a ~50-file diff under 2 min: not met. 130.8 s on 62ee899e8f~1..62ee899e8f (feat(evals): shared tested grader matchers with a dual-engine harness #12555, 47 files): 5 shell suites ran, all passing; 6 Python and Node suites were selected but not run.
Criterion 4, full corpus under 10 min: not met. A --jobs 1 run of the fd862a49b848 tree under a 60-minute deadline stopped after 3716 s with 18 of 829 suites done (3647 suite-seconds) and 811 never started. Largest Windows times against Linux CI: audit-engine.test.sh 1660 s against 135 s, cant-fail-scan.test.sh 891 s against 128 s, audit-fleet.test.sh 511 s against 34 s. 9 of the 18 exited nonzero on Windows and pass on Linux CI.

Criteria 3 and 4 need per-offender slices: start with the three suites above and the 9 Windows-only failures.

Slices filed from #10241's Windows result (#10241 closed; numbers in docs/test-wall-clock-baseline.md "Windows host").

test(windows): fix or host-skip the 9 suites that fail only on Windows Git Bash in the #10241 run #13228: the 9 suites that fail only on Windows Git Bash (criterion 4 cannot be read while they fail)
perf(test): cut audit-engine.test.sh's Windows wall clock (1660 s against 135 s on Linux) #13229: audit-engine.test.sh, 1660 s on Windows against 135 s on Linux
perf(test): cut cant-fail-scan.test.sh's Windows wall clock (891 s against 128 s on Linux) #13231: cant-fail-scan.test.sh, 891 s against 128 s
perf(test): cut audit-fleet.test.sh's Windows wall clock (511 s against 34 s on Linux) #13232: audit-fleet.test.sh, 511 s against 34 s

Kept here until those merge, then re-measured on a Windows host:

Criterion 3: affected-tests.sh --run on a ~50-file diff under 2 min (130.8 s now, on 62ee899e8f~1..62ee899e8f)
Criterion 4: full corpus under 10 min (not finished in 60 min: 18 of 829 suites). The re-measure names the next offenders, and they get their own slices.

Criterion 2 (no gate script over 60 s) is met.

This was generated by AI.

No description provided.

Navigation Menu

Repository navigation

test-perf: diagnose and cut the wall clock of the local test and gate corpus, especially on Windows Git Bash #3716

Summary

Observed (2026-09-04, Windows 11, Git Bash, worktree on the `feat/handoff-prompt-qol` branch)

Related

Proposed work

Acceptance

Merged from #3717 (duplicate)

Additional measurements (2026-09-07, Windows 11, Git Bash, `perf/hook-performance-program`)

Over-selection compounds the cost

Asks, in addition to the program above

37 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

test-perf: diagnose and cut the wall clock of the local test and gate corpus, especially on Windows Git Bash #3716

Description

Summary

Observed (2026-09-04, Windows 11, Git Bash, worktree on the feat/handoff-prompt-qol branch)

Related

Proposed work

Acceptance

Activity

kyle-sexton commented on Sep 6, 2026

Merged from #3717 (duplicate)

kyle-sexton commented on Sep 7, 2026

Additional measurements (2026-09-07, Windows 11, Git Bash, perf/hook-performance-program)

Over-selection compounds the cost

Asks, in addition to the program above

kyle-sexton commented on Sep 8, 2026

37 remaining items

kyle-sexton commented on Oct 27, 2026

kyle-sexton commented on Oct 29, 2026

kyle-sexton commented on Nov 2, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 4, 2026

kyle-sexton commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Observed (2026-09-04, Windows 11, Git Bash, worktree on the `feat/handoff-prompt-qol` branch)

Additional measurements (2026-09-07, Windows 11, Git Bash, `perf/hook-performance-program`)