Speed up the test-ae HTTP-server sweep (batching / parallelism strategies)

[paul-hammant]

Problem

make test-ae's HTTP portion spins up a large number of real server
instances and runs them strictly serially, which dominates the sweep's
wall-clock. This is a time concern, not a flakiness one (though the
fixed-port design that forces the serialization is the same root cause behind
the port-squatting flake class).

Measured on a local Linux dev box

• ~93 shell tests start a real server (ae run server.ae / proxy / origin),
  out of 366 test_*.sh total.
• Shell tests run at SH_NPROC=1 (serial) — see Makefile (sh_nproc=${SH_NPROC:-1}).
• Back-to-back, each server test costs ~3–4s (a warm singleton looks like
  ~0.5s, but in sequence it's ~4s), so the HTTP portion alone is ~5–6 min.
• Per test the time is server compile-check + startup, a sleep 0.1 poll loop
  plus a fixed sleep 0.3, several curl --max-time 5 spawns, and
  teardown — very little of it is actual assertion work.

Strategies, roughly by payoff

1. Parallelize the shell tests — biggest single win, lowest effort.
   SH_NPROC=1 is the throttle. The reason it's 1 is the fixed-port collision
   problem from the flake memory — parallel servers fight over 18120/19100/etc.
   Two ways to unlock it:

   • Ephemeral ports: have server.ae bind port 0, print the actual port in
     the READY line, and have the shell test read it. Removes the collision
     entirely → safe SH_NPROC=$(nproc). This is the right fix and also kills
     the port-squatting flake class. Est: 5 min → ~1 min on an 8-core box.
   • Cheaper interim: assign each test a unique port derived from its
     name/index so parallel runs don't collide, then bump SH_NPROC. Less
     robust than port-0 but a small diff.

2. Kill the sleep padding — easy, universal.
   Replace grep READY + sleep 0.3 with a real readiness probe (curl-until-
   response loop), which the Windows-flake memory already prescribes. Removes
   ~0.4s guaranteed per test + tightens the poll. ~93 × 0.4s ≈ 35s off, and
   it's strictly more correct.

3. Batch multiple assertions per server instance.
   Many tests start a whole server to fire 1–3 curls. Where several sibling
   tests exercise the same server build, fold them into one server lifecycle
   with multiple curl assertions (already done in keepalive's multi-URL curl).
   Fewer compile+startup+teardown cycles. Higher effort, test-by-test.

4. Share one long-lived server across a group.
   A session-scoped fixture: start one server per behavior family, run all its
   tests against it, tear down once. Biggest structural win but the most
   invasive — needs a harness concept the sweep doesn't have today.

5. Avoid recompiling the server per test.
   ae run re-checks/links each time even on cache hit. ae build once to a
   binary, reuse across tests. Marginal here (~0.3–0.5s) since the module cache
   already helps.

Suggested first step

#1 (ephemeral ports) + #2 (real readiness probe) compound: correct readiness
enables safe parallelism, they address the time problem AND retire the
port-squatting flake family in one move, and both are mechanical rather than a
per-test rewrite. Prototype on a handful of tests and measure before rolling
the pattern across all 93.

[nicolas-maman]

Measurement worth recording here, because it changes what this issue should aim at.

I converted the HTTP server fixtures to ephemeral ports (#1936 and #1937) and
then tried the thing this issue is ultimately after, raising SH_NPROC:

	failures	wall
serial (SH_NPROC=1)	0	312s
SH_NPROC=8	10 or more	630s

Parallel was both broken and slower than serial.

The failures are not port collisions. They cluster on cache_build_flags,
cache_libdir_invalidation and cache_subdir_entry_root_module, and each one
passes alone and fails in the parallel sweep. They assert cache HIT and
MISS against the single shared build-cache directory, so any other test
compiling at that moment changes what they observe. Several proxy tests fell
over in the same run.

That matters for the plan: strategies 1 to 5 above all treat the fixed ports as
the throttle. Ports were a real problem, and worth fixing on their own merits
(two fixtures both hardcoded 18107, which only serial execution hid), but they
are not the binding constraint on parallelism. Per-test build-cache isolation
is the prerequisite, and it is not on the list.

Strategy 2 (a real readiness probe instead of the fixed sleep 0.3) is done and
merged, and it is worth keeping regardless: it is strictly more correct, and the
measured probe returns in 0.036s where the sleep took 0.300s.

SH_NPROC stays at 1 for now, with the measurement recorded next to the default
in the Makefile so the next person to try does not have to rediscover it.