perf: reduce test-suite execution time (tracking)

[Chemaclass]

Tracking: reduce test-suite execution time

Status 2026-07-14 (round 2 complete)

All round-2 child issues have landed (#772–#775); per-change before/after numbers are recorded in each PR. Round 1 (#767–#771) remains as history below.

Round 1 recap (commit 1ae3e82, macOS bash 3.2.57, Apple Silicon):

• 100 trivial tests: ~0.68s (was ~5.0s) — per-test framework overhead ~48ms → ~6.8ms (~7x).
• Full suite --parallel: ~52s for 1250 tests. The dominant remaining cost is the 258 nested ./bashunit cold starts in the acceptance suite, not per-test overhead.

The remaining work is the cold-start frontier plus two small fork leftovers (below); the per-test hot path is now largely fork-free.

Child issues — round 1 (done)

• perf(state): avoid base64 forks for empty title and hook message in export_subshell_context #762 perf(state): avoid base64 forks for empty title and hook message (PR perf(state): skip base64 forks for empty subshell-context fields #768)
• perf(runner): cache data-provider lookup per file instead of grep+sed per test #763 perf(runner): cache data-provider lookup per file (PR perf(runner): cache data-provider lookup per file #767)
• perf(runner): remove avoidable subshell forks from the run_test hot path #764 perf(runner): remove avoidable subshell forks from run_test (PR perf(runner): remove avoidable subshell forks from the run_test hot path #770)
• perf(clock): skip per-test clock reads when the clock impl forks an interpreter #765 perf(clock): skip per-test clock reads when the clock forks an interpreter (PR perf(runner): auto-skip per-test timing when the clock forks an interpreter #769)
• perf(parallel): detect CPU count and benchmark a default --jobs cap #766 perf(parallel): --jobs auto CPU-count cap (PR feat(parallel): add --jobs auto to cap concurrency at CPU cores #771)

Child issues — round 2 (done)

• perf(assert): remove per-assertion subshell forks (strip_ansi, label, alias probe) #772 perf(assert): per-assertion subshell forks — strip_ansi capture, failure-path label forks, alias probe
• perf(tags): build per-file function-to-tags map in one pass instead of per-function grep/sed walk #773 perf(tags): per-function grep/sed walk when --tag is used
• perf(runner): remaining run_test hot-path forks (labels, clock capture, parallel::is_enabled) #774 perf(runner): remaining run_test forks — normalize label captures, clock capture for pure-bash impls, parallel::is_enabled recomputed per test
• perf(snapshot): remove success-path forks from assert_match_snapshot #775 perf(snapshot): ~6-8 forks per assert_match_snapshot even on the success path

Smaller leftovers

Not worth their own issue; folded into a round-2 cleanup PR:

• per-arg base64 -d in data-provider parsing — resolved by perf(runner): cache data-provider lookup per file instead of grep+sed per test #763 (parsing is now pure-bash via build_provider_map)
• jobs -r | wc -l fork per wait_for_job_slot iteration (src/runner.sh:305, :317)
• check_duplicate_functions grep+awk per file (src/helpers.sh:208)

Next frontier (needs measurement before filing)

Spun out into #798 — the acceptance suite's 258 nested ./bashunit cold-start runs now dominate wall-clock. See #798 for candidate directions and the measure-first plan.

Constraints for all work

• Bash 3.0+ compatibility is non-negotiable (no declare -A, no [[ ]], no ${var,,}, no negative array indexing, no &>>). The perf target machine IS bash 3.2 on macOS.
• TDD, quality gate (./bashunit tests/, ./bashunit --parallel tests/, make sa, make lint, shfmt -w .) on every PR.
• One issue per PR; no batching unrelated changes.

Shared benchmark protocol

Run before and after each change, report numbers in the PR:

mkdir -p /tmp/bashunit-bench
for i in $(seq 1 100); do echo "function test_noop_$i() { assert_true true; }"; done > /tmp/bashunit-bench/noop100_test.sh
time ./bashunit --no-parallel /tmp/bashunit-bench/noop100_test.sh   # current: ~0.68s

# fork census (external binaries per run)
PS4='+X ${BASH_SOURCE##*/}:$LINENO ' bash -x ./bashunit --no-parallel /tmp/bashunit-bench/noop100_test.sh \
  2>/tmp/bashunit-bench/trace.txt >/dev/null
grep -oE ' (perl|base64|grep|sed|mktemp|rm|date) ' /tmp/bashunit-bench/trace.txt | sort | uniq -c

For assertion-heavy paths (#772) use 100 tests x 10 assert_equals; for tags (#773) use 100 tagged tests with --tag foo.

Also considered and NOT filed: lazy-sourcing large optional modules (coverage.sh, learn.sh) at startup. Sourcing all of src/ measures only ~28ms, so the win across 258 nested cold starts is ~5-8s and it complicates the single-file release build. Revisit as part of the "next frontier" item above.

Original baseline (commit 4d80e7c, for history)

• 100 trivial tests took ~5.0s (~48ms/test): 2x perl clock forks, 3x base64 pipelines, 1x grep+sed provider probe, 1x base64 -d decode, ~12-14 $(...) subshells per test.
• Sequential full suite >4 min; --parallel ~48s with no default job cap.

Line references valid at commit 49064ed and will drift.