Skip to content

Harden pipeline publication, trajectories, and trainer telemetry - #809

Merged
bradhilton merged 8 commits into
mainfrom
agent/compile-phase-telemetry
Aug 19, 2026
Merged

Harden pipeline publication, trajectories, and trainer telemetry#809
bradhilton merged 8 commits into
mainfrom
agent/compile-phase-telemetry

Conversation

@bradhilton

@bradhilton bradhilton commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

This combines and supersedes #807 so the dependent Caladan change has one ART merge and one final SHA to pin.

  • restore bounded nonstreaming tau-bench policy calls after high-concurrency streaming saturated the driver
  • keep token-lineage and text-equivalent suffix reconciliation linear
  • move TrainerRank LoRA conversion/serialization off the strict rank queue after immutable distributed CPU capture
  • keep src/art/megatron byte-identical to main; ART PipelineTrainer continues using its original Megatron LoRA publication path
  • let art.trajectory_group() accept resolved trajectories, exceptions, and arbitrary awaitables while resolving repeated awaitable identities once and preserving metadata/metrics/logs
  • emit structured trainer phase and torch.compile telemetry, deduplicating plan identities on low-cardinality topology/request-mix fields while retaining exact token counts in event payloads

Tau-bench request bound

URL-backed policy clients use a 10-minute read timeout per attempt and one SDK retry, for an approximately 20-minute worst-case request budget. They no longer send private streaming/progress headers or reconstruct streamed policy completions.

PipelineTrainer isolation

The final net diff under src/art/megatron is zero. New staged export ownership lives in art.trainer_rank._lora_export and calls existing low-level Megatron helpers without changing the PipelineTrainer entry points or implementation.

Validation

  • fresh blocker-only Codex review: two blockers found, fixed, and re-reviewed with no remaining blockers
  • fresh Claude Fable 5 blocker review: no merge blockers
  • 162 focused trajectory/history/tau-bench/telemetry tests passed
  • 47 ART PipelineTrainer tests passed
  • 11 legacy and staged LoRA export tests passed
  • Ruff, format, ty, lock consistency, and diff checks passed
  • GitHub CPU and 2xH200 checks will run on this consolidated head

Downstream

Caladan #113 must pin the actual merged SHA from this PR; it must not retain the prior #807 draft SHA.

@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 19, 2026 16:00 — with GitHub Actions Error
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 19, 2026 16:42 — with GitHub Actions Inactive
@bradhilton bradhilton changed the title Attribute trainer compile stalls Harden pipeline publication, trajectories, and trainer telemetry Aug 19, 2026
@bradhilton
bradhilton changed the base branch from agent/stream-terminal-drain to main August 19, 2026 21:01
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 19, 2026 21:01 — with GitHub Actions Error
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 19, 2026 21:08 — with GitHub Actions Error
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 19, 2026 21:23 — with GitHub Actions Inactive
@bradhilton
bradhilton marked this pull request as ready for review August 19, 2026 21:39
@bradhilton
bradhilton merged commit 98f2a79 into main Aug 19, 2026
9 of 11 checks passed
@bradhilton
bradhilton deployed to trainer-rank-gpu-validation August 19, 2026 21:39 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant