Skip to content

feat(rl): add pinned Miles LoRA federation - #39

Open
AlexEisie wants to merge 6 commits into
mainfrom
feat/miles_rl_support_v0_impl
Open

feat(rl): add pinned Miles LoRA federation#39
AlexEisie wants to merge 6 commits into
mainfrom
feat/miles_rl_support_v0_impl

Conversation

@AlexEisie

@AlexEisie AlexEisie commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Pin agentenv/miles@yeto-sync at a91bd34e50416aeb1da111f74d52b296e8216b96.
  • Add the INIT thin boundary in Miles: replicated canonical LoRA export/apply, LoRA optimizer reset with scheduler progress preservation, and a native post-train/pre-SGLang synchronization hook.
  • Keep Miles responsible for rollout, reward, GRPO, offload, saving, and SGLang transport; remove the previous Yeto actor/private-broadcast/train-loop/provider/logging/IPC runtime injection.
  • Preserve pre-train policy-version checks, exact-base equal-weight fixed-roster averaging, completed-group recovery, long-task/session support, and standard PEFT export.

Remaining intentional differences from INIT

INIT plan Current implementation Assessment
Recover the authoritative checkpoint across syncer VM or disk replacement. Automatic recovery requires the original syncer disk to remain available. The RL algorithm is unaffected, but cross-VM disaster recovery remains an operational gap.
Connect RL and synchronization metrics to a dashboard. Emit complete round/sync metrics to JSONL. Sufficient for validation and offline diagnosis, without centralized live monitoring.

Validation

  • Miles thin-branch tests: 9 passed; Ruff passed.
  • Yeto focused RL tests: 67 passed.
  • Full Yeto Python suite: 760 passed, 4 skipped.
  • Rust syncer: 58 passed; cargo fmt --check passed.
  • The earlier 8xA100 real-model matrix validates the RL algorithm/model path but predates the thin-hook migration; the new boundary has not yet repeated that GPU matrix.

AlexEisie added 6 commits July 30, 2026 09:45
@AlexEisie
feat(rl): add pinned Miles LoRA federation
0f0f2c7 
Implement the Miles RL v0 path around radixark/miles at dfc66ff38752bfa2c5d325e0037ebc4b537c06de: provision multi-GPU islands, run Miles rollout and GRPO, export complete canonical fp32 LoRA updates, and commit exact-base equal-weight fixed-roster averages through the existing syncer protocol.

Add optimizer-state reset with scheduler progress preservation, completed-group recovery and oversampling, custom generation/session-server/TITO forwarding, strict failure handling, standard PEFT export, metrics, documentation, and focused launcher/runtime/Rust coverage. The implementation was validated with dense DP=8, two DP=4 islands, MoE EP=8, direct Miles parity, tool-use sessions, fault injection, syncer restart, and 20 consecutive real-model merges on eight A100 GPUs.

Differences from INIT:

1. Use process-local actor/provider adapters and the pinned non-FT private _broadcast path against a clean upstream Miles checkout, instead of maintaining the proposed Miles thin branch and train-loop hook.

2. Keep the authoritative syncer checkpoint on retained local disk; the INIT cross-VM/disk durable checkpoint mount is not implemented.

3. Emit the planned RL and sync metrics to JSONL, but do not enable the INIT dashboard integration.
@AlexEisie
docs(rl): clarify Miles RL implementation
3caa18a
@AlexEisie
feat(rl): adopt maintained Miles sync hooks
9ebd420 
Pin agentenv/miles yeto-sync and move replicated LoRA export/apply, optimizer reset, native post-train synchronization, and round metrics behind its public boundary. Keep Miles in control of rollout, GRPO training, offload, and SGLang publication while removing the old actor, train-loop, provider, logging, and IPC runtime injection.

Validate rollout policy versions before optimizer work, preserve exact-base fixed-roster averaging and completed-group recovery, and document the maintained integration. The remaining INIT differences are syncer checkpoint loss with its VM/disk and JSONL-only monitoring without a dashboard.
@AlexEisie
feat(rl): add reproducible Miles LM benchmark
4f2b963 
Compare native Miles, one strict Yeto island, and fixed-roster federated Yeto on identical GPU, prompt-group, trajectory, optimizer, and expert-parallel budgets.

Run real Miles rollout and GRPO jobs, retain paired rollout captures, validate prompt identity and completed work, evaluate standard PEFT artifacts on a held-out set, and write atomic resumable reward, pass@k, throughput, synchronization, and cost reports.

Reuse the public Miles runner without the Yeto hook for the native reference, isolate concurrent island ports, derive canonical adapter targets through Megatron-Bridge mappings, preserve declared Hugging Face model architectures and provider rotary settings, and pin the maintained gated-attention Miles revision.

Reject incomplete native adapter layouts and missing PEFT keys, preserve adapter tensor values during PEFT namespace normalization, and allow a replacement learner joining at terminal state to apply and acknowledge the final committed policy.

Add focused coverage for workload fairness, provenance, artifact validation, report aggregation, generic model mappings, native-hook isolation, and terminal replacement recovery.
@AlexEisie
docs(rl): record current Miles validation
8e1eef9 
Describe the maintained external policy hook, the gated-attention Megatron-Bridge compatibility in the pinned c951c667 Miles revision, and the equal-hardware native, single-island, and federated benchmark workflow.

Record the three-seed Qwen3.6-27B run on eight H200 GPUs: nine eight-round jobs, 2,304 real training trajectories, 576 held-out generations, aggregate reward and pass@k results, and the shared 768-token response-cap limitation.

Keep the broader four-A100 MoE, recovery, parity, export, session/tool, and 20-merge campaign distinct from the current-pin benchmark, update automated-test evidence, and state the remaining multi-node, Spot, durable-checkpoint, dashboard, and soak boundaries precisely.
@AlexEisie
docs(benchmarks): publish Miles RL results
bb3ecda 
Archive the three-seed equal-hardware Qwen3.6-27B comparison across native Miles, one Yeto island, and two Yeto islands. Record configuration provenance, aggregate and per-seed quality, execution and synchronization metrics, and the verified adapter contract.

Document the response-cap limitation and keep the result scoped to strict v0 averaging rather than treating the small observed reward differences as convergence or an algorithmic win.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AlexEisie