Skip to content

Integrate topology-portable TrainerRank checkpoints - #791

Open
bradhilton wants to merge 10 commits into
austin/monarch_multinode_trainingfrom
agent/topology-portable-austin-compat
Open

Integrate topology-portable TrainerRank checkpoints#791
bradhilton wants to merge 10 commits into
austin/monarch_multinode_trainingfrom
agent/topology-portable-austin-compat

Conversation

@bradhilton

@bradhilton bradhilton commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • apply the merged ART TrainerRank checkpoint API and final hardening to Austin’s latest multinode branch
  • preserve the branch-specific PP/local-chunk guard and model-support-aware CP configuration
  • keep PipelineTrainer, ambient Megatron training, runtime, and publication code unchanged

Scope guard

  • production changes are restricted to art.trainer_rank
  • no changes to pipeline_trainer, Megatron runtime/training, model handlers, or weights/lora_publish
  • branch-specific compatibility changes are limited to dev/CI utilities and tests
  • production Python delta against Austin’s branch: +1,526 lines

Validation

  • based on Austin target a7973734b1ac72636b8bff4ed39b70b175a91631, rechecked immediately before readiness
  • final-head quality, package-install, and 2×H200 checkpoint validation all pass (H200: 19m26s)
  • local prek passes after applying merged ART bf5c6052d36c7ab365964582285aff82f5fca98b
  • 121 dependency-available focused tests passed locally; Megatron-extra paths pass in CI/H200
  • independent PipelineTrainer-regression and finalizer reviews: clean
  • Claude Fable 5 final focused review: clean
  • no unresolved GitHub review threads

@bradhilton
Add topology-portable TrainerRank checkpoints (#790)
d7aaba8 
* Add topology-portable TrainerRank checkpoints

* Run GPU validation for TrainerRank changes

* Harden distributed checkpoint finalization

* Authenticate checkpoint artifact manifests
@bradhilton
Resolve Austin branch TrainerRank test imports
3762956
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 12, 2026 18:13 — with GitHub Actions Error
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 12, 2026 18:16 — with GitHub Actions Error
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 12, 2026 18:19 — with GitHub Actions Error
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 12, 2026 18:22 — with GitHub Actions Error
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 12, 2026 18:25 — with GitHub Actions Inactive
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 12, 2026 19:22 — with GitHub Actions Inactive
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 12, 2026 20:43 — with GitHub Actions Inactive
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 12, 2026 21:09 — with GitHub Actions Inactive
@bradhilton
Harden distributed checkpoint finalization (#793)
8489047 
* Harden distributed checkpoint finalization

* Recover checkpoint cleanup across rank skew

* Keep checkpoint recovery collectives aligned
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 12, 2026 22:08 — with GitHub Actions Inactive
@bradhilton
bradhilton marked this pull request as ready for review August 12, 2026 22:28
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 12, 2026 22:28 — with GitHub Actions Failure
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 12, 2026 22:41 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bradhilton