feat(rl): add decoupled Miles RL adapter lifecycle - #43
Open
AlexEisie wants to merge 58 commits into
Open
Conversation
Implement the Miles RL v0 path around radixark/miles at dfc66ff38752bfa2c5d325e0037ebc4b537c06de: provision multi-GPU islands, run Miles rollout and GRPO, export complete canonical fp32 LoRA updates, and commit exact-base equal-weight fixed-roster averages through the existing syncer protocol. Add optimizer-state reset with scheduler progress preservation, completed-group recovery and oversampling, custom generation/session-server/TITO forwarding, strict failure handling, standard PEFT export, metrics, documentation, and focused launcher/runtime/Rust coverage. The implementation was validated with dense DP=8, two DP=4 islands, MoE EP=8, direct Miles parity, tool-use sessions, fault injection, syncer restart, and 20 consecutive real-model merges on eight A100 GPUs. Differences from INIT: 1. Use process-local actor/provider adapters and the pinned non-FT private _broadcast path against a clean upstream Miles checkout, instead of maintaining the proposed Miles thin branch and train-loop hook. 2. Keep the authoritative syncer checkpoint on retained local disk; the INIT cross-VM/disk durable checkpoint mount is not implemented. 3. Emit the planned RL and sync metrics to JSONL, but do not enable the INIT dashboard integration.
Pin agentenv/miles yeto-sync and move replicated LoRA export/apply, optimizer reset, native post-train synchronization, and round metrics behind its public boundary. Keep Miles in control of rollout, GRPO training, offload, and SGLang publication while removing the old actor, train-loop, provider, logging, and IPC runtime injection. Validate rollout policy versions before optimizer work, preserve exact-base fixed-roster averaging and completed-group recovery, and document the maintained integration. The remaining INIT differences are syncer checkpoint loss with its VM/disk and JSONL-only monitoring without a dashboard.
Compare native Miles, one strict Yeto island, and fixed-roster federated Yeto on identical GPU, prompt-group, trajectory, optimizer, and expert-parallel budgets. Run real Miles rollout and GRPO jobs, retain paired rollout captures, validate prompt identity and completed work, evaluate standard PEFT artifacts on a held-out set, and write atomic resumable reward, pass@k, throughput, synchronization, and cost reports. Reuse the public Miles runner without the Yeto hook for the native reference, isolate concurrent island ports, derive canonical adapter targets through Megatron-Bridge mappings, preserve declared Hugging Face model architectures and provider rotary settings, and pin the maintained gated-attention Miles revision. Reject incomplete native adapter layouts and missing PEFT keys, preserve adapter tensor values during PEFT namespace normalization, and allow a replacement learner joining at terminal state to apply and acknowledge the final committed policy. Add focused coverage for workload fairness, provenance, artifact validation, report aggregation, generic model mappings, native-hook isolation, and terminal replacement recovery.
Describe the maintained external policy hook, the gated-attention Megatron-Bridge compatibility in the pinned c951c667 Miles revision, and the equal-hardware native, single-island, and federated benchmark workflow. Record the three-seed Qwen3.6-27B run on eight H200 GPUs: nine eight-round jobs, 2,304 real training trajectories, 576 held-out generations, aggregate reward and pass@k results, and the shared 768-token response-cap limitation. Keep the broader four-A100 MoE, recovery, parity, export, session/tool, and 20-merge campaign distinct from the current-pin benchmark, update automated-test evidence, and state the remaining multi-node, Spot, durable-checkpoint, dashboard, and soak boundaries precisely.
Archive the three-seed equal-hardware Qwen3.6-27B comparison across native Miles, one Yeto island, and two Yeto islands. Record configuration provenance, aggregate and per-seed quality, execution and synchronization metrics, and the verified adapter contract. Document the response-cap limitation and keep the result scoped to strict v0 averaging rather than treating the small observed reward differences as convergence or an algorithmic win.
Integrate the latest upstream SFT, accelerator, diffusion, data-format, and CI changes while preserving the Miles RL launcher, exporter, documentation, and benchmark paths.
- support attested strict resume and intentional adapter branching - preserve training recipe and parent lineage in artifact provenance - add safe PEFT-to-base merge with sharded SafeTensors output - validate and stage parent adapters before learner startup
- compare lifecycle metadata against the learner's effective defaults - document that resume, branch, and merge currently apply to causal LoRA SFT - cover launcher-only default normalization in lifecycle tests
- add the experimental single-process PPO command and environment boundary - provide CyberGym prompt and reward adapters with content checks - forward worker endpoint settings and API credentials without serializing secrets - reject SFT adapter lifecycle flags on Miles RL launches - preserve nested prompt metadata for environment-backed rewards
- deploy fixed-roster Miles RL directly to caller-owned SSH hosts - content-address local prompt inputs and mount them read-only on islands - rehash deployed Rust syncer sources before the remote release build - retain round inputs for an independent f32 AVG oracle - collect manifests, event tapes, checkpoints, logs, and failure-injection evidence
- allow native, single-island, and federated arms to be selected explicitly - skip syncer construction for native-only reference runs - forward identical chat-template kwargs through training and held-out evaluation - cover arm filtering and Qwen thinking-mode parity
- rely on the learner's existing matrix-merge default - keep parent adapter forwarding compact - remove explanatory comments that duplicate the lifecycle contract
Describe the supported Miles causal-LM LoRA boundary and the strict-avg and decoupled synchronization contracts.\n\nDocument policy snapshots, staged BCAST apply and commit ordering, exact-snapshot completed-group recovery, finalization and export, and the equal-work RL benchmark with fragment payload and queue-time metrics.
Add the explicit decoupled RL preset with deterministic all-AVG LoRA fragments, exact-base pipelined outer rounds, complete policy snapshot tokens, and post-train safe-boundary application.\n\nStage broadcasts until Miles applies and verifies the complete trainer state, preserve optimizer moments for in-process fragment updates, and retain exact-snapshot completed groups across recovery. Add authoritative multi-fragment finalization and PEFT export, and pin the stop-capable Miles commit.\n\nKeep strict-avg as the default and leave Rust, SFT, Diffusion, local PPO, and CyberGym behavior unchanged.
Cover deterministic fragment layout, horizon and exact-base gating, staged BCAST commit ordering, optimizer preservation, policy snapshot identity, checkpoint recovery, budget consolidation, and terminal finalization.\n\nAdd direct syncer integration and two-island f32 outer-update oracle coverage, plus multi-fragment export, launcher contract, standard PEFT artifact, and final publication checks.
Add a decoupled benchmark arm alongside native Miles, single-island strict, and federated strict controls. Hold local optimizer work constant with learner budgets and a deterministic terminal consolidation sweep.\n\nRecord artifact-ready time, time-weighted GPU activity, realized horizon, PULL-to-PUSH and BCAST queue latency, fragment payload traffic, source identity, and resume-safe run metadata. Extend benchmark tests for allocation, commands, aggregation, reporting, and failure handling.
Define the causal-LM LoRA-only decoupled Miles RL contract, deterministic fragment layout, safe-boundary state machine, outer optimizer, snapshot identity, recovery, finalization, benchmark fairness, validation matrix, and explicit non-goals.\n\nUpdate the implementation status after pinning the stop-capable Miles commit and keep the local Chinese guide and original INIT plan excluded from commits.
During learner-budget consolidation, a duplicate HELLO from the same connection generation could replay an already queued in-flight PULL and make the frozen learner answer twice. Treat only a genuinely newer generation as a reconnect before replaying outstanding pulls.\n\nAdd a regression test covering unchanged, reconnected, and unrelated generations. This keeps the existing fixed-roster and consolidation behavior otherwise unchanged.
CyberGym can race its non-atomic lookup and insert when one island submits the same task and PoC concurrently. Serialize only identical server, agent, task, and payload submissions while preserving concurrency for distinct candidates and retaining fatal HTTP error handling.
Allow a fresh Decoupled Miles RL phase to warm-start from a completed Yeto PEFT adapter without reopening the prior run or carrying optimizer and progress state across phases. Hash and stage the parent read-only, validate its causal-LM LoRA contract, exact canonical tensors, finite values, and export provenance, then seed and attest the version-zero syncer cut before rollout zero. Persist the parent digest in in-phase checkpoints and document the deliberately narrow continuation boundary.
Add a Decoupled-specific acceptance oracle that checks the ordered fragment schedule, fixed two-island roster, progress ledger, complete terminal cut, and identical applied final policy hash. Route SSH verification and PEFT export through the configured RL sync preset while retaining the existing Strict-AVG oracle unchanged. Document the two verification contracts and cover dispatch, export arguments, and invariant rejection.
Persist the effective Miles runtime values for dynamic-sampling replacement limits, train offload, and distributed timeout so island checkpoints attest what actually ran. Emit dynamic-filter drop reasons as numeric per-reason metrics instead of a JSON-valued metric. Balance the temporary actor wake used for initial adapter export by restoring offload before Miles enters its training loop. This prevents the first train step from resuming an already-awake torch memory-saver allocation, as exposed by the real two-island H200 acceptance run.
Pin Miles f8c0726dc33e8e516477129c964b2e39eaccbc02, where the external policy boundary keeps an offloaded trainer resident through initial adapter export, policy application, and the first SGLang weight publication before restoring offload. Remove Yeto's duplicate onload/offload ownership so Miles has one lifecycle authority, retain strict initialization failure reporting, and update the regression test and user documentation for the corrected ordering.
Pin the reviewed Miles commit that stages LoRA weights before trainer offload and publishes them through independent CUDA IPC storage. When a decoupled learner submits the configured final fragment step, wait for and acknowledge the authoritative final cut instead of racing into an unnecessary next rollout. Cover that boundary with a focused regression test.
# Conflicts: # README.md # docs/ADAPTER_LIFECYCLE.md # docs/PROVENANCE.md # tests/test_adapter_lifecycle.py # tests/test_launch_auto.py # yeto/adapter_lifecycle.py # yeto/cli.py # yeto/launcher.py # yeto/learner.py
A run is a syncer plus one learner island per --gpu entry, scattered across clouds. Its observability was a log line every ten steps and the syncer's event tape, both of which need `sky logs` plus a script to turn into an answer about which island is falling behind. `--wandb` maps a fleet onto one W&B group: one run per island (rank 0 only), one for the syncer. Island runs use a deterministic id with resume="allow", so a preempted spot island reattaches to its own curve instead of starting a second one. Per-island steps stay on their own run because async DiLoCo advances them independently and a shared run would log a non-monotonic step series. Sync metrics come from an optional observer on sync_diloco_boundary, the one module all four backends route their sync through, so staleness, push size and delta norm land once and apply everywhere. The Rust syncer is unchanged. Its event tape already records quorum and grace timings, per-island staleness, and per-island RDA contribution, so in head-controller mode a daemon thread tails the tape into the syncer's run. tests/test_wandb_tape.py parses the Rust format literal and fails if a field is added there without a decision on the Python side. Telemetry never costs a run: a missing package, a failed init, or a failed log all degrade to a no-op sink, and the learner's pip step is non-fatal. Without a credential the run goes offline rather than blocking a headless spot VM. WANDB_API_KEY rides the same path as HF_TOKEN and is kept out of wandb.config. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Local controller mode runs the syncer on its own cluster, so the event tape was unreachable from the controller on the submitting machine and only island metrics streamed. Rather than shipping the tape back over sky, the reader goes to the tape: `yeto.wandb_tape --follow` now runs as a sidecar beside the syncer on that VM. It is backgrounded deliberately. The syncer must stay the job's foreground process, because FleetController reads that job's exit code as the syncer's health; and as a separate process the forwarder cannot take the syncer down with it. Telemetry pulls the repo workdir onto the syncer cluster, which the cross-build path already did; wandb_tape and wandb_logger are stdlib-only, so that VM still never needs the training stack, and a test pins it. A forwarder now records the byte offset of the last record it forwarded next to the tape, so a restarted one resumes instead of re-logging every past merge. Head mode gets the same treatment for a reused head. The offset store found a real bug in the shipped --follow path: sky ends a job with SIGTERM, which the loop did not handle, so throttled offsets never reached disk and a restart replayed them. A subprocess test now signals a real process and checks the flush; it fails against the old loop. Without --wandb the syncer task is byte-identical to before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A real 2-island SmolLM2 run across two H200 boxes, in both controller modes, surfaced two things the unit tests could not. The syncer's W&B run had almost no config in local controller mode. Head mode hands its forwarder the launch namespace directly, but the sidecar is a separate process and only had its own argv, so the syncer run could not be filtered by model, region, or recipe the way the island runs can. The launcher now serializes build_config(args) into --config-json, the same way cmd_launch_head already ships the launch args to the head; the credential filter runs before serialization. An island that ran fewer steps than one logging window reported sync curves and no training curve at all. That is not hypothetical: the late joiner in the run took 4 local steps against the other island's 80, because it was still downloading weights, and finished with no loss series. Every backend now emits a final training point when its loop ends on a partial window. The gate is args.wandb rather than the run's enabled flag, because rank > 0 holds a no-op sink and the collective inside _global_loss_sum has to stay balanced. TELEMETRY_EVERY moves to wandb_logger, which all four backends already import and none of which import each other. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merges feature/wandb-telemetry. Three conflicts were additive (README doc list, the head job's env block, an import). The fourth was real: this branch moves the syncer's event tape under ~/yeto-output for RL runs, so the forwarder would have tailed a file nothing writes. syncer_event_tape(args) now resolves that path from the training mode, and syncer_command, LocalSyncer's strict-failure reader and the W&B sidecar all go through it. A test pins that all three agree, for both modes. For RL itself there is no inner loop here to instrument: MILES owns it. But the island already writes a structured event tape carrying reward, KL, ESS, clip fraction, action tokens, rollout percentiles, delta norms and payload sizes. yeto/rl/wandb_rl.py tees each event into W&B as it is appended — a second reader of the tape rather than a second instrumentation pass, with the file written first so the tape stays authoritative when the network is not. The projection keeps scalars, turns lists into counts, and drops identifiers: a policy hash names a state rather than measuring one, and charting it is meaningless. RL runs join the fleet group with job_type="rl-learner" on an rl/rollout_id x-axis. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
yeto-rl-ssh builds its own learner command from the plan rather than going through make_rl_learner_task, so the launcher plumbing did not reach it and an on-host RL run would have started with telemetry off. The plan now carries the project/entity/mode, _learner_argv emits the flags, and the container is told its fleet group via YETO_RUN_GROUP from the run id. WANDB_API_KEY stays out of the plan and rides in the remote env file next to HF_TOKEN, which is the promise the harness already makes about secrets — a test pins that the plan never serializes a credential. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The RL step metric was rl/rollout_id, which appears in no event the RL path actually emits. define_metric therefore pointed at a series that was never logged and every RL curve silently fell back to W&B's internal step counter, so island runs could not be compared against each other or against the syncer. Checked against a real DeepSeek-V4 island tape: a rollout round carries base_policy_version and local_round_id, a policy apply carries policy_version, and an apply-progress tick carries only policy_version — no single key spans them. event_metrics now synthesizes rl/step from the first of those present, so every event lands on one axis. The synthetic fixtures were what hid this, so the tests now carry one real event of each kind, captured from that tape, and assert every kind gets an x-axis. Verified live: reward_mean, reward_std, mean_kl, ess_ratio, clip_fraction, action_tokens, delta_l2_norm, the group percentiles and sync/fragment_payload_bytes_* all arrive on the W&B server against a monotonic rl/step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces Yeto's Miles-backed causal-LM RL path and the supporting
adapter lifecycle, synchronization, reproducibility, and acceptance tooling.
It keeps Miles responsible for rollout, reward evaluation, GRPO, and local
training, while Yeto owns the cross-island canonical-LoRA boundary,
authoritative checkpointing, recovery identity, and final export.
Motivation
Yeto previously synchronized SFT adapters but did not provide a production
boundary for independent Miles RL islands. RL needs more than a generic tensor
exchange: each rollout must be bound to one complete policy snapshot, incoming
global updates must be applied only at a safe post-train boundary, and resumed
runs must reject a mismatched adapter layout instead of silently mixing model
state.
What Changed
Miles RL integration
yeto.rlpackage with canonical LoRA export/apply, policy snapshotidentity, a Miles bridge, initial-adapter handling, and a Miles learner
launch path.
--training-mode rl --rl-runtime mileslauncher options plusyeto rl,yeto-rl-export, andyeto-rl-sshentry points.strict-avg: a complete-LoRA, fixed-roster, equal-weight correctnessbaseline.
decoupled: deterministic multi-fragment LoRA synchronization that letslocal RL continue while distinct fragment rounds are in flight.
vector. Stale, malformed, or mixed-policy trajectories are rejected before
training.
re-exported policy hash, then publish the coherent policy to SGLang.
final publication so external policy synchronization does not use an
offloaded actor.
Syncer correctness and recovery
equal learner weighting, reconnect-safe PULL replay, and per-fragment
in-flight ownership.
strict RL runs fail closed for a missing or mismatched fingerprint.
markers, and add structured event-tape metrics for layout identity,
responders, merge duration, and global delta norm.
CyberGym, local reference path, and operational tooling
deterministic reward submission, text Level 1 prompts, shaped rewards,
checksum coverage, curriculum selection, and variance-aware sampling.
SSH acceptance harness for deploying an attested Yeto source plus detached
Miles and SGLang revisions to existing hosts.
generated benchmark reports from source deployment.
Reproducibility and compatibility
remote Miles and SGLang dependencies.
LoRA transport revision, including lazy P2P import and offload-safe runtime
behavior needed by the integration.
local-rldependencies (gymnasium,requests, andnumpy) without changing the base install.Documentation and coverage
benchmark workflow, CyberGym RL, SSH acceptance runbook, and adapter
lifecycle.
strict and decoupled synchronization, initial adapters, exports, launcher
construction, benchmark/resume accounting, CyberGym scoring and curriculum,
and SSH artifact attestation.
Documentation alignment and task organization
docs/MILES_RL.mdwith the current implementation: exact Miles,SGLang, PEFT, and image pins; exact-base syncer behavior; checkpoint layout
verification; offload-safe publication; template forwarding; event-tape
fields; and benchmark-only reconnect behavior.
yeto/tasks/cybergym, usethe
yeto.tasks.cybergymimport path everywhere, and leave the genericdistributed-runtime contract in
docs/MILES_RL.md.Design Constraints
Miles/Megatron/SGLang stack. Full-parameter RL, TP/PP LoRA gather-scatter,
dynamic membership, and optimizer-state federation remain out of scope.
decoupledis opt-in and uses deterministic fragments, exact raw anchors,complete-roster equal-weight averaging, and a post-train safe boundary. It
does not make generic asynchronous RL a supported mode.
verifier intentionally rejects a dirty checkout or an unpinned dependency.
Validation
python -m pytest -q tests/1080 passed, 5 skippedin 149.14s.cargo testinsyncer/59 passed.credentials; it is not run from this development environment.
Size
58 files changed: 21,233 additions and 82 deletions. The changes are grouped
into the RL implementation, syncer safety contract, operational tooling, test
coverage, and documentation so the behavior can be reviewed by ownership area.