Pipeline LoRA publication and bound tau-bench requests

[bradhilton]

Summary

• split LoRA publication into an immutable distributed CPU capture followed by off-queue conversion and serialization
• synchronize runtime and planning failures through the trainer Gloo group before blocking tensor exchange
• make sampled-token lineage and text-equivalent suffix reconciliation linear instead of repeatedly rescanning long histories
• restore ordinary nonstreaming tau-bench policy calls with a bounded retry budget

Why

LoRA conversion and serialization were still running on Caladan's strict trainer queue. For the measured 116 MiB Qwen adapter, that delayed the next gradient by about 227 seconds even after upload itself had been reduced to seconds.

ART #804 changed URL-backed tau-bench policy calls to token-delta streaming. Live 512-concurrency qualification showed that this saturated one Python core, relayed more than 1 GB of SSE data, and still produced connection retries. This PR reverts the tau-bench portion of that change rather than adding more stream-specific pooling, sharding, parsing, or transport behavior.

Long sampled histories could additionally trigger repeated suffix and lineage scans. Those paths now use linear failure-table matching while preserving the existing conservative reconciliation rules.

Tau-bench request bound

URL-backed policy clients now use:

• a 10-minute read timeout per attempt
• one SDK retry, for an approximately 20-minute worst-case request budget
• 30-second connect, write, and pool timeouts
• the existing 2,048-connection limit needed to admit the experiment's 512-way concurrency

The longest legitimate Qwen3.5-4B cold request observed was 351 seconds, while Caladan's benchmark default is 15 minutes. The approximately 20-minute retry-inclusive budget provides headroom without inheriting Caladan's unrelated 45-minute cross-model control-plane safety limit.

Tau-bench now calls chat.completions.create(..., stream=False) directly. It no longer sends stream options, progress headers, or a private final-response header, and it has no stream-body retry or reconstruction path. ART's generic public stream consumer remains available to callers that explicitly use it.

Scope

The tau-bench production diff against main is 11 additions and 100 deletions. The removed code includes automatic streaming selection, usage trailers, custom stream headers, body-level retry classification, and completion reconstruction.

Validation

• 94 focused tau-bench, generic OpenAI stream, and trajectory-history tests pass
• ruff, formatting, ty, and lock consistency checks pass
• the Megatron GPU workflow remains the authority for the retained LoRA publication tests and will rerun on this rewritten head

Delivery note

This remains a draft for review. Caladan must pin ART's eventual merge SHA rather than this branch commit.

[bradhilton]

Live qualification at Caladan batch 40 exposed a policy APITimeoutError whose cause was httpx.ConnectTimeout during chat.completions.create(). That call was outside ART’s streamed-policy retry guard, so the rollout failed after the SDK exhausted its internal retries.

Commit e1f04eb fixes the root cause without changing any timeout: streamed calls now use ART as the sole retry owner, disable SDK-internal retries on a shared-transport client clone, and retain exactly max_retries + 1 total attempts. Retryable pre-header HTTP status codes are covered too; non-stream calls remain SDK-owned.

Verification: 18 focused stream tests pass; Ruff, formatting, ty, and lock checks pass. A fresh Codex review and a persistent Claude Fable 5 re-review both approved the revised exact-attempt-budget implementation with no blockers. The full unconstrained local test collection is not feasible in this lightweight environment because GPU/Megatron torch extras and an example API credential are absent; GitHub CI is running the repository-defined environments.

[bradhilton]

Follow-up: Claude Fable 5 correctly found that three existing tau-bench test doubles lacked with_options(). Commit 7da0985 updates only those fakes. The affected suites now pass together (32 tests), all prek hooks pass, and the production retry design is unchanged.

[bradhilton]

Final post-fix status: Claude Fable 5 re-reviewed commit 7da0985 after the fake-client compatibility repair and approved with no remaining blockers. GitHub quality checks, install smoke test, and two-H200 TrainerRank validation are all green. The PR remains draft and unmerged pending live experiment qualification and downstream pinning.

[bradhilton]

Post-remediation review record:

• Fresh Codex review initially found unconditional event-loop JSON parsing, a post-fork executor hang, SDK parsing divergence, usage-trailer loss, unbounded per-stream buffering, scheduler unfairness, and incomplete capture after partial stream consumption. Each issue now has a focused regression; the final review approves with no blockers.
• Persistent Claude Fable 5 independently found the unguarded capture snapshot and partial-stream re-read/capture cases. Both are fixed; its settled-diff review ran all 30 stream tests and approves with no blockers.
• The pool intentionally uses two workers, not one or four. In the 512-stream stress reproduction, one worker produced 1.78 s maximum loop lag, four intermittently exceeded 0.5 s, and two stayed below the bound; six consecutive two-worker runs passed.
• Live audits passed against both providers: OpenAI over HTTP/2 returned expected content/usage with exact auto-capture equality, and Caladan/Qwen over HTTP/2 preserved reasoning, content, 141 token IDs, usage, and exact auto-capture equality.

This remains draft pending patched live qualification and downstream pinning.

[FurtherAI]

I believe lora publishing is well optimized in the new multinode pr #808. A snapshot is taken and moved to the cpu, then the forward-backward work proceeds. There's a bunch of other overlap around this, but point is the time between consecutive fwd-bwd work is roughly 200ms.

[bradhilton]

Superseded by #809, which contains the complete #807 change set plus the final TrainerRank isolation, trajectory-group compatibility, and compile telemetry fixes. Caladan will pin #809's merged SHA.