Make ART a native multi-node RL runtime with Monarch

[FurtherAI]

Summary

This PR turns ART into a native distributed RL runtime rather than a single-node trainer wrapped in remote launch glue. Monarch owns process lifecycle and typed RPC, ART owns RL semantics and policy state, Megatron/NCCL owns distributed training, and vLLM owns inference execution.

Distribution remains an execution topology, not a second programming model. Existing single-node training code continues to use MegatronBackend() and PipelineTrainer; internally, it now compiles to the same one-host runtime used by multi-node jobs. Advanced users can provide explicit placement and service topology without changing the training loop.

The PR also adds production GLM-5.2 support optimized on H200 and B300, composable CP/EP/DP/PP/VPP training, multi-node inference, CUDA 13/Blackwell support, and the asynchronous data, packing, publication, and durability paths needed to keep those systems efficient.

Design Decisions

• The ART controller may run on any host. It is not tied to trainer rank 0.
• CPU rollout functions execute in distributed host-local worker processes. User code is referenced by verified import path rather than shipped as opaque pickled closures.
  • This does come with a utility to add your script to the PYTHONPATH so your rollout fn can be imported.
• Trainer ranks are ordinary ranks across hosts; TP, CP, EP, ETP, DP, PP, and VPP compose without node-specific trainer logic.
• One model service is one coordinated native vLLM deployment, which may span hosts. Independent inference replicas remain isolated on the follow-up branch rather than adding complexity to the core runtime.
• Shared storage is used only for durable checkpoints, optimizer generations, adapters, and retained artifacts. Live jobs, events, trajectories, packed batches, and health state use typed direct communication.
• Rollout owners retain trajectory records. The controller selects compact references, and packing workers fetch leased records directly from owners.
• Packing has a lookahead window of 1, moving it off the critical path.
• Also added cross-batch CP planning lookahead, moving it generally off the critical path.
• Warm Megatron ranks retain model and optimizer state. No recurring checkpoint copy, content hash, model reload, or optimizer reload remains.
• LoRA activation and durable persistence are independent asynchronous operations over immutable snapshots. Local multi-node transfer uses CPU-to-CPU NIXL through a transport-neutral manifest.
  • This is a nice optimization, especially for glm 5.2/large models, making publishing a lora fast.
• Failures are surfaced and cleanup is fail-closed; the runtime does not silently fall back to filesystem polling, NCCL Socket, naive cross-host EP, or stale policy state.
• Generalized-RL data models and TrainingProgramSpec are intentionally not prerequisites. Existing trajectories, trajectory groups, backend.train(), and pipeline training remain valid.

Public API

API	Purpose
art.init_megatron_runtime_config(...)	Configures topology, packed sequence length, snapshot capacity, compilation, and streaming offload while preserving the existing single-node call shape.
HostSpec, ClusterSpec, GpuPlacement	Describe physical hosts, CPU capacity, GPU identity, controller placement, and transport configuration.
TrainerMeshSpec, ModelServiceSpec, VllmParallelSpec	Describe trainer ranks and user-named model services without assigning algorithm-specific roles such as policy, judge, or teacher.
compile_topology(...)	Validates placement, rank ordering, parallel-world divisibility, endpoints, ports, GPU ownership, NCCL, and NIXL before model allocation.
ArtRuntime.start(...)	Attaches ART to an explicit Monarch host mesh for multi-node execution.
ArtRuntime.start_local(...)	Collapses the same runtime onto one host; this is what the default MegatronBackend() path uses.
InstalledAsyncCallable and runtime.rollout_executor(...)	Register a source-verified top-level async rollout function and distribute the actual autotuner-selected worker count across host CPU slots.
PackingRequest and PackedBatchRef	Move versioned trajectory references into immutable trainer-ready batches with explicit provenance and lease ownership.
runtime.start_trainer(...)	Starts a warm typed Megatron run using TrainerRuntimeSpec, TrainingRunSpec, job contracts, and progress/completion events.
runtime.start_model_service(...)	Starts and supervises one native single- or multi-host vLLM deployment as one health, version, update, and recovery domain.
art-monarch	Runs the same package-owned bootstrap locally, through SkyPilot, or on explicitly supplied hosts.

Normal single-node usage remains:

art.init_megatron_runtime_config(
    topology=art.MegatronTopologyConfig(),
    packed_sequence_length=128 * 1024,
)

async with MegatronBackend() as backend:
    ...

Explicit multi-node usage supplies a compiled runtime and rollout executor, then uses the same backend and PipelineTrainer APIs:

runtime = await ArtRuntime.start(host_mesh, compile_topology(...))
rollouts = runtime.rollout_executor(
    InstalledAsyncCallable.from_callable(rollout),
    target_workers=num_rollout_workers,
)

async with MegatronBackend(runtime=runtime) as backend:
    trainer = PipelineTrainer(..., rollout_executor=rollouts, backend=backend)

These APIs allow multiprocessing rollout workers, which is useful when rollouts perform CPU-heavy environment execution.

Runtime Flow

rollout CPU workers
    -> owner-local trajectory records
    -> descriptor/reference queue
    -> selection + immutable leases
    -> direct owner fetch
    -> prefix-tree packing + route replay finalization
    -> SHM or authenticated cross-host batch fanout
    -> warm Megatron ranks
    -> forward/backward + optimizer step
    -> bounded immutable snapshot pool
       -> NIXL transfer + vLLM activation
       -> asynchronous adapter/optimizer durability

Policy version, adapter generation, logprobs, rewards, timing, MoE routes, and mid-prefill policy changes remain attributable through this flow.

Implementation Map

The total branch diff is 280 files, +64,465/-17,589. Excluding tests and lockfiles, production, setup, examples, and documentation contribute +40,016/-9,631; tests contribute +17,163/-5,841.

Area	Change
src/art/distributed/	19 files, +10,221: typed topology, Monarch lifecycle, rollout execution, trajectory ownership, leased queues, packing, batch transport, NIXL adapter transfer, admission, and model-service supervision.
src/art/megatron/runtime/	16 files, +4,426/-211: typed runtime/job/event contracts, local and Monarch executors, warm trainer supervision, managed package runtime, compilation identity, publication, and recovery.
Remaining src/art/megatron/	64 files, +12,851/-5,189: backend cutover, distributed service coordination, CP/EP/HybridEP, optimizer state, asynchronous snapshots, BF16 LoRA serialization, and trainer instrumentation. The old filesystem service.py is deleted.
GLM-5.2 core and handler	12 files, +3,535: sparse MLA, indexer, CP stages, LoRA projections, model spec/state, TileLang kernel, and ART model-support integration.
training/pipeline_schedule.py	+995: PP/VPP scheduling with variable sequence lengths, executed batch size one, recomputation, CP, and route-replay integration.
ART/vLLM serving runtime	20 files, +3,955/-1,699 excluding lockfiles: vLLM 0.25.1 integration, distributed deployment lifecycle, binary MoE routes, policy spans, pooled fast metrics, and model-specific patches still required upstream.
Pipeline, autotuner, preprocessing, trajectories	14 files, +1,959/-781: bounded queue control, packing lookahead, logical/executed token accounting, and async trainer dispatch.
Release packaging and examples/multinode/	One-command CUDA 12/13 profiles, a locked content-addressed Megatron runtime, bundled HybridEP and NIXL/UCX build assets, managed etcd, and SkyPilot/local bootstrap examples.
tests/	80 files, +17,163/-5,841: runtime lifecycle, topology, data-plane, failure/recovery, publication, model correctness, numerical parity, packing, trainability, and E2E throughput coverage.

Workflow Tests

The workflow is a set of tests which run for each handler, proving things like parity with HF transformers, invariance to prefix tree packing, correct parallelism implementations, minimal train-inf mismatch, trainability and now e2e throughput. The throughput test uses a set of layers which fits a 128k packed seq on 2 gpus, cp2 ep2. vLLM is deployed with 2 gpus as well, and a synthetic workload is trained on. We assert things like a gap under 230ms p50 between consecutive fwd_bwd work, vLLM and trainer load, time to activate an adapter, trainer throughput matching expected isolated throughput (also catches recompilation issues), and overall tok/s. These ensure that the system is properly async and components are performing at peak speed.

In addition to the new stage, we redesigned how the workflow schedules itself, combined stages, and minimized imports, process startup, and repeated work. This turns a 70-90 minute run for one handler into approximately 60 minutes for all ten, with further scaling from additional GPUs.

Performance And Validation

• H200 CP8/EP8 retained 95.0% throughput when moved from one host to a 4+4 cross-host layout.
• Completion-heavy CP8/EP8 at 64K to CP16/EP16 at 128K retained 94.6% raw weak-scaling throughput.
• On two B300 training nodes, full 78-layer GLM-5.2 CP8/EP8/DP2 reached 15,713 logical tok/s and approximately 9.79% useful MFU, 12.0% faster than CP16/EP16 and 43.3% faster than the measured PP2/VPP3 topology at identical useful work.
  • With plenty of exploration, we have determined that cpN/epN is the most efficient topology in general for a model's minimum number of gpus. Single-node or multi-node.
  • We did make improvements to multi-node CP
• The selected three-node E2E layout uses 16 trainer GPUs and 8 inference GPUs. It reached score 195 and 308.19 accepted tok/s while retaining 96.4% of the 8:8 control's score per provisioned GPU.
  • Pretty impressive to scale the system throughput with gpus cleanly, requires balancing load carefully as well as good scaling from the gpu-heavy systems (vLLM and Megatron).
• Recurring trainer non-forward/backward wall time fell from 10.403s to 1.494s.
• The former 4.357s synchronous save-and-publish region became a 1.65ms enqueue plus 176.46ms immutable snapshot preparation; transfer, activation, and durability proceed asynchronously with bounded backpressure.
• Warm representative packing improved from 1.839s to 0.949s, replay finalization from 85.07ms to 0.030ms, and SHM finalization from 74.14ms to 18.41ms.
• The definitive post-main-merge workflow passed all ten required stages for all ten handlers on 24 B300 GPUs in approximately 50m18s. Sensitivity variants were intentionally excluded.
• Covered handlers are Llama 3 dense, Qwen 3 dense/MoE, Qwen 3.5 dense/MoE, Gemma 4 dense/MoE, DeepSeek V4, GLM-5.2, and GPT-OSS MoE.
• Release wheels were qualified from a fresh package install on CUDA 12/H200 with Apex fused extensions and on two CUDA 13/B300 hosts with EP2, HybridEP, NIXL, managed etcd, and vLLM. The final wheel, sdist, package-content checks, Ruff, formatting, type checking, hooks, and lock validation pass.
• Physical multi-node gates exited cleanly without residual ART, Monarch, Megatron, vLLM, NCCL, or GPU processes.

Intentional Scope

This PR does not add a second training API, Ray, generalized-RL program definitions, multiple independent inference replicas, merged-weight serving, the old NCCL weight-transfer engine, file-backed job dispatch, JSONL polling, or nested multi-node torchrun. Those omissions are deliberate: the delivered core is the smallest coherent runtime compatible with the efficiency target that provides correct multi-node rollout, inference, training, data movement, policy publication, durability, and single-node collapse.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
openpipe-art	🟢 Ready	View Preview	Aug 19, 2026, 1:32 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.