feat: Split pool sample forecasts across all endpoints and drop the single-request utilization stage

[shackmann]

Summary

With JOINTFM_DEPLOYMENT_IDS, forecast_samples(...) now splits every sample request evenly across all reachable endpoints, whatever its size. Before, it split only requests larger than the smallest endpoint's max_sample_count. Each endpoint still serves at most one of the client's batches at a time, which leaves its remaining request slots free for other callers.

The maximum-utilization check loses its single_request stage. A pool forecast is now always split, so that stage could no longer send one maximal request. The check now runs only concurrent_saturation, which still fills every advertised slot on every endpoint.

Why

The response download, not the GPU, takes most of a hosted sample request's time. Measured on 2026-10-06 against a 4-endpoint pool (max_sample_count 5000 and max_concurrent_requests 2 per endpoint, 732-step history, 92-step horizon), as median wall time:

Columns	Samples	1 endpoint	4 endpoints, 1 request each
10	100	1.22 s	0.97 s
10	1,000	3.22 s	2.13 s
10	5,000	12.71 s	6.85 s
100	1,000	21.35 s	12.25 s
100	5,000	58.62 s	50.12 s

One HTTP stream downloaded at about 4 MB/s, and concurrent streams from separate endpoints overlap up to about 10 MB/s in total. Before this change, a 5,000-sample request went to a single endpoint.

Changes

• Batch sizing (JointFMClient._sample_batch_sizes): the batch count is max(ceil(n / cap), min(n, routable endpoints)), and samples are split evenly across the batches. No batch exceeds the cap, every routable endpoint gets a batch, and none is empty (n_samples=1 stays one request). A single endpoint splits only above its cap, as before, but now into even batches instead of cap-sized ones.
• One request per endpoint: the existing per-peer POST lock already enforced this. A batch beyond the endpoint count, or one failing over onto a busy endpoint, waits instead of overlapping. A new test now covers it.
• Max-utilization check: removes the single_request stage and the pool-failover recording. Failover cannot happen any more, because the saturation burst sends through per-endpoint clients. concurrent_saturation is always present and also runs for a single-slot deployment.
• Docs: README, docs/api-reference.md and notebooks/max_utilization.ipynb are updated.

Breaking changes

• Removed UtilizationStage.failovers, MaxUtilizationReport.single_request and MaxUtilizationReport.stages. MaxUtilizationReport.concurrent_saturation is no longer Optional.
• run_max_utilization_check no longer requires the jointfm_client.pool logger to emit WARNING records.
• A seeded pool sample forecast returns different samples than before, because batch i uses seed + i and the batch layout changed. The result now also depends on how many endpoints are reachable.

Testing

• task pre-commit passes: ruff, ty, typos, nbstripout, and the test suite with coverage.
• New tests: a sample request below the cap is split across every peer (n=3 becomes [2, 1] with seeds 7 and 8; n=1 stays one request), and 6 concurrent batches on 2 peers never overlap on one peer.
• Live: a 1,000-sample, 10-column forecast went to the 4-endpoint pool as 4 × 250, one batch per endpoint, in 1.97 s. The single-endpoint baseline was 3.2 s.
• The max-utilization notebook cells were run against the test fake, not against live deployments.

---

Note

Medium Risk
Changes default multi-endpoint sample-forecast routing and reproducibility, with breaking changes to the max-utilization report API; behavior is covered by new tests but affects production forecast outputs for pool users.

Overview
With JOINTFM_DEPLOYMENT_IDS, forecast_samples(...) now splits every sample request evenly across all routable peers, not only when n_samples exceeds max_sample_count. Batch count is max(ceil(n / cap), min(n, peer_count)), with batch i using seed seed + i, so seeded pool results change and depend on how many endpoints are reachable. Each peer still handles at most one of this client's batches at a time (extra batches queue on the per-peer POST lock).

run_max_utilization_check drops the single_request stage and pool-failover tracking: it only runs concurrent_saturation, filling every advertised slot via endpoint_client, so the check is not distorted by automatic sample splitting. Breaking: removed UtilizationStage.failovers, MaxUtilizationReport.single_request, and stages; concurrent_saturation is always present. Docs, notebooks, and tests are updated accordingly.

^{Reviewed by Cursor Bugbot for commit 340abeb. Configure here.}