Make ART a native multi-node RL runtime with Monarch - #808

Draft

FurtherAI wants to merge 1411 commits into

mainfrom

austin/monarch_multinode_training

This pull request is big! We're only showing the most recent 250 commits

Commits on Aug 11, 2026

Support training-only distributed runtimes
FurtherAI
committed
b28a170
Defer trajectory queue shrink during collection
FurtherAI
committed
1077c74
Bound full-model trainer shutdown
FurtherAI
committed
f91c332
Preserve live KV across policy updates
FurtherAI
committed
7375005
Exercise policy transition hash patch
FurtherAI
committed
2f29744
Allow full-model service teardown to finish
FurtherAI
committed
b845dd7
Rebase policy cache hashes after preemption
FurtherAI
committed
ed6f555
Fix trajectory queue lease backpressure
FurtherAI
committed
4d3fcaf
Record final cutover acceptance blockers
FurtherAI
committed
1ae05bd
Stabilize throughput load evidence
FurtherAI
committed
1752bc4
Make throughput evidence statistically symmetric
FurtherAI
committed
8465a3f
Stabilize throughput measurement windows
FurtherAI
committed
a51ef7b
Aggregate stable throughput evidence
FurtherAI
committed
fb0df2f
Update stable-window rejection assertion
FurtherAI
committed
82730d3
Derive stable settings from execution history
FurtherAI
committed
fad26e0
Handle multi-window throughput evidence
FurtherAI
committed
1dde72e
Stabilize matched throughput parity sampling
FurtherAI
committed
bb26624
Align matched throughput test steps
FurtherAI
committed
2dc328a
Bound throughput evidence to measured steps
FurtherAI
committed
b1172de
Bound slow workflow acceptance budgets
FurtherAI
committed
7d18a0e
Align DSV4 trainability resource contract
FurtherAI
committed
1860748
Install final B300 throughput calibrations
FurtherAI
committed
59d6e62
Stabilize real-path parity inputs
FurtherAI
committed
135e97c
Fix parity rollout inputs deterministically
FurtherAI
committed
1a5b494
Fix deterministic parity decoding
FurtherAI
committed
0a3ea45

Commits on Aug 12, 2026

Restore representative train inference retries
FurtherAI
committed
917c290
Trim redundant correctness topologies
FurtherAI
committed
a797373
test: schedule reusable model workflow sessions
FurtherAI
committed
0779671
test: stabilize GPT OSS mismatch KL gate
FurtherAI
committed
c708fe2
test: shorten steady state throughput validation
FurtherAI
committed
887cda6
test: schedule workflows across cluster hosts
FurtherAI
committed
a0f4d44
test: pin remote workflow runtime environment
FurtherAI
committed
1c1901f
test: capture matched inputs from packed generations
FurtherAI
committed
15e1f51
test: schedule default workflow resources
FurtherAI
committed
d4f7cd9
test: overlap matched capture under source leases
FurtherAI
committed
22a0e82
test: share production width functional fixtures
FurtherAI
committed
5be2fc9
test: account for functional MTP layers
FurtherAI
committed
91ab236
fix: pin vllm runtime build tools
FurtherAI
committed
3c94660
Optimize sliding context masks
FurtherAI
committed
edba6e7
test: measure throughput after shape warmup
FurtherAI
committed
3ed139a
test: cover two post-warmup windows
FurtherAI
committed
7bcb7ee
test: update throughput lag percentile fixture
FurtherAI
committed
cc3c18b
test: retain mean activation lag failure case
FurtherAI
committed
c04f4c3
Measure trainer gaps on the CUDA timeline
FurtherAI
committed
4183558
Optimize DSV4 prefix compression layouts
FurtherAI
committed
28c03ab
Reuse functional LoRA vLLM sessions safely
FurtherAI
committed
84bc4a1
Reuse pinned publication buffers by snapshot slot
FurtherAI
committed
c8f7f4e
Optimize oracle comparison lifecycle
FurtherAI
committed
561d9a8
Reuse base Megatron runtime for parity checks
FurtherAI
committed
2d6f1de
Avoid repacking fused DSV4 LoRA tensors
FurtherAI
committed
aeafe31
Reuse functional validation runtimes across stages
FurtherAI
committed
a6d5729
Retry only transient train inference failures
FurtherAI
committed
564245c
Handle empty vLLM unload responses
FurtherAI
committed
8a2d630
Allocate disjoint shared workflow roles
FurtherAI
committed
7f74941
Fix concurrent workflow fixture execution
FurtherAI
committed
bd80bb4
Use native DSV4 serving kernels
FurtherAI
committed
567c3e1
Preserve reused inference runtime contracts
FurtherAI
committed
cbad7fb
Fit wide-head FlexAttention backward on Blackwell
FurtherAI
committed
498cfcd
Handle local DSV4 dummy LoRA experts
FurtherAI
committed
2add349
Keep replay-built MoE runtimes stage-local
FurtherAI
committed
4acfa92
Preserve exact sliding and local expert layouts
FurtherAI
committed
5d44ffc
Reduce throughput workflow measurement horizon
FurtherAI
committed
9dfab04
Align throughput test with shorter horizon
FurtherAI
committed
a2cc8f2
Keep trainability on coherent pretrained models
FurtherAI
committed
8910104
Preserve exact workflow session topology
FurtherAI
committed
3cc563c
Keep throughput calibrations source-stable
FurtherAI
committed
b0ca2c5
Align workflow tests with final protocol
FurtherAI
committed
e7d86b7
Keep throughput fixture on measured suffix
FurtherAI
committed
0bfc300
Fence throughput calibrations by runtime contract
FurtherAI
committed
f916bb7
Refresh B300 throughput contract identities
FurtherAI
committed
d890981
Amortize workflow worker imports per host
FurtherAI
committed
9c828e0
Stabilize finite-horizon throughput gap sampling
FurtherAI
committed
c0d2cbf
Harden workflow forkserver cleanup
FurtherAI
committed
3e78245
Schedule correctness references independently
FurtherAI
committed
b103f71
Record exact correctness phase topology
FurtherAI
committed
0a3da83
Overlap Megatron trainer preparation
FurtherAI
committed
469c6be
Overlap workflow preload with preparation
FurtherAI
committed
fff22b9
Allow variable throughput role counts
FurtherAI
committed
6aea4ef
Use a shorter DSV4 throughput sequence
FurtherAI
committed
85be028
Allow trainer startup during vLLM launch
FurtherAI
committed
b02c59d
Remove merged rollout serving
FurtherAI
committed
bc1343e
Restore numerical mismatch retries
FurtherAI
committed
2e649c8
Align mismatch retry invariant
FurtherAI
committed
4ab90c3
Scale DSV4 throughput work to 32K
FurtherAI
committed
560aded
Reuse resident trainer for functional validation
FurtherAI
committed
1236714
Create trainability artifact directory
FurtherAI
committed
85a4e28
Prefetch trainer with external inference
FurtherAI
committed
b071fee
Preserve trainer run identity in actors
FurtherAI
committed
acd2e06
Handle EP-local dummy MoE adapters generically
FurtherAI
committed
effec27
Capture MoE routes in resident functional sessions
FurtherAI
committed
987dce1
Overlap functional trainer and vLLM startup
FurtherAI
committed
77dd8a2
Fence functional rollouts on serving readiness
FurtherAI
committed
6748204
Schedule workflow resources without fragmentation
FurtherAI
committed
62b5091
Cache trainer Dynamo graphs per rank
FurtherAI
committed
0cb3e0e
Enable Dynamo caching after trainer imports
FurtherAI
committed
d5c7a05
Support aliased methods in trainer compile cache
FurtherAI
committed
69b485c
Preserve non-strict Dynamo cache bypass
FurtherAI
committed
9ccf3d4
Cache late Dynamo identity guards
FurtherAI
committed
7261458

Commits on Aug 13, 2026

Skip transient CUDA events in trainer cache
FurtherAI
committed
e834923
Give lazy Qwen bridges stable identities
FurtherAI
committed
b819e95
Key trainer caches by resolved compile plan
FurtherAI
committed
29cf697
Serialize guarded module context variables
FurtherAI
committed
cece4be
Reconstruct guarded autograd nodes
FurtherAI
committed
5cf75ab
Prune unguarded HybridEP compile state
FurtherAI
committed
8c61dad
Register reused Dynamo resume code
FurtherAI
committed
56684e5
Bundle no-op autograd graphs for precompile
FurtherAI
committed
f656f92
Serialize trainer compile cache sources
FurtherAI
committed
f2bd018
Serialize bundled trainer code objects
FurtherAI
committed
37d4e76
Allow isolated workflow compiler caches
FurtherAI
committed
fba84fb
Integrate qualified DSV4 throughput geometry
FurtherAI
committed
75c3766
Scope trainer precompile serialization
FurtherAI
committed
c191d05
Match resident vLLM startup deadline
FurtherAI
committed
5653399
Serialize dynamic Dynamo resume functions
FurtherAI
committed
d71573c
Require resident trainability continuation
FurtherAI
committed
b9d40f5
Preserve dtype identities in compile guards
FurtherAI
committed
0b1be17
Fail closed on workflow operation failures
FurtherAI
committed
450b52a
Complete resident compiler cache artifacts
FurtherAI
committed
ec36a5b
Preserve resident phase evidence after failures
FurtherAI
committed
e72388e
Namespace regional compile cache entries
FurtherAI
committed
8994fb1
Make HybridEP handles cache serializable
FurtherAI
committed
3f1541a
Scope HybridEP handle cache serialization
FurtherAI
committed
fd46450
Correct DSV4 throughput update geometry
FurtherAI
committed
2cf319a
Serialize HybridEP handles in guard caches
FurtherAI
committed
1de9e47
Preserve regional precompile entries
FurtherAI
committed
a9e6b3d
Resolve serialized nested compile frames
FurtherAI
committed
3d6ff6b
Classify generated resume frames from origin metadata
FurtherAI
committed
f6d00b9
Reuse native compiler cache across trainer runs
FurtherAI
committed
5a83c39
Cache import-time Megatron compile wrappers
FurtherAI
committed
ab57f7a
Support compiled callable classes in trainer cache
FurtherAI
committed
167ffac
Hydrate import-time compiler guards safely
FurtherAI
committed
8c8356e
Canonicalize hydrated nested compile code
FurtherAI
committed
0f2a916
Canonicalize guarded compile bytecode
FurtherAI
committed
5db8f3d
Handle transformed compile constants safely
FurtherAI
committed
269322b
Resolve nested compile code across packages
FurtherAI
committed
44a436b
Canonicalize generated compile frames
FurtherAI
committed
654e0a0
Intern hydrated resume code identities
FurtherAI
committed
a8b6ae6
Keep compile identity interning fast
FurtherAI
committed
7fcda63
Drop trainer precompile compatibility
FurtherAI
committed
f5b56ca
Make throughput groups per step global
FurtherAI
committed
7b146d4
Retry inconclusive throughput load once
FurtherAI
committed
87ee200
Balance DSV4 throughput decode load
FurtherAI
committed
9b4b801
Bound DSV4 inference concurrency
FurtherAI
committed
e31b8c9
Balance DSV4 inference capacity
FurtherAI
committed
d304ff1
Tune DSV4 inference pressure
FurtherAI
committed
151def4
Calibrate DSV4 two-group throughput
FurtherAI
committed
f8326c0
Give DSV4 pressure margin
FurtherAI
committed
d82490e
Use representative policy for resident parity
FurtherAI
committed
a2ed9f3
Calibrate DSV4 balanced throughput load
FurtherAI
committed
16c7aba
Update mismatch policy fence fixture
FurtherAI
committed
eadfe41
Calibrate Qwen resident parity variance
FurtherAI
committed
6febaea
Align DSV4 HF norm output precision
FurtherAI
committed
03b0fa1
Use DP-aware resident score scheduling
FurtherAI
committed
3e94cef
Bound repeated HF route score noise
FurtherAI
committed
488d0d5
Require utilized throughput workloads
FurtherAI
committed
146a025
Fix throughput utilization test fixture
FurtherAI
committed
a6f44a1
Cover utilization gate combinations
FurtherAI
committed
88d0ee7
Attribute trainer schedule gap phases
FurtherAI
committed
81b811a
Reuse repeated prefixes in throughput workloads
FurtherAI
committed
89b9c12
Trace pipeline trainer critical path phases
FurtherAI
committed
92cf734
Decompose trainer schedule gaps
FurtherAI
committed
d9f901e
Remove nonverbose trainer progress overhead
FurtherAI
committed
1ffaf35
Allow homogeneous throughput depth expansion
FurtherAI
committed
8167f90
Stabilize throughput workflow acceptance
FurtherAI
committed
83b80a1
Retry transient throughput failures
FurtherAI
committed
946802f
Isolate throughput runtime provenance
FurtherAI
committed
51d6ce8

Commits on Aug 14, 2026

Balance DSV4 throughput qualification
FurtherAI
committed
3b451e9
Stabilize DSV4 throughput workload
FurtherAI
committed
8d76cad
Load DSV4 inference reliably
FurtherAI
committed
2796992
Handle appended functional fixture layers
FurtherAI
committed
e9a5e4d
Restore valid Blackwell wide-head Flex tiles
FurtherAI
committed
3fd9c08
Fence functional serving baseline before registration
FurtherAI
committed
42cf811
Preserve workflow runtime placement invariants
FurtherAI
committed
d958557
Serialize incompatible base model workflow stages
FurtherAI
committed
4c19948
Apply workflow fixture environment before imports
FurtherAI
committed
ae37ba3
Size Gemma resident functional context
FurtherAI
committed
7a001c2
Align resident length sequence limits
FurtherAI
committed
5af20f2
Align resident scoring sequence length
FurtherAI
committed
6ac0a7b
Scope wide-head Flex workaround to Blackwell
FurtherAI
committed
a0bda28
Select settled throughput execution windows
FurtherAI
committed
2dce744
Exercise measured throughput row validation
FurtherAI
committed
054627b
Stabilize measured workflow gates
FurtherAI
committed
17c4501
Keep adapter publication off the trainer path
FurtherAI
committed
68a62ae
Calibrate learned-policy parity gates
FurtherAI
committed
b652f5f
Record learned-policy calibration evidence
FurtherAI
committed
cce36cb
Stabilize functional workflow gates
FurtherAI
committed
b61bab2
Own packing rendezvous per worker
FurtherAI
committed
ac51473
Stabilize GLM length movement
FurtherAI
committed
7eb93ab
Allow router-free virtual pipeline chunks
FurtherAI
committed
cc8c6a3
Preplan CP batches from packing lookahead
FurtherAI
committed
b67893f
Overlap controller finalization with training
FurtherAI
committed
17d62f6
Preserve source times for async pipeline metrics
FurtherAI
committed
58bdfaa
Fence controller finalization behind trainer dispatch
FurtherAI
committed
2bd401e
Stabilize Python GC after trainer compilation
FurtherAI
committed
d31ad2a
Allow 230ms workflow trainer gap p50
FurtherAI
committed
7e3549c
Preserve resident trainability scenario order
FurtherAI
committed
fd2a1b5
Give GPT OSS trainability a reliable horizon
FurtherAI
committed
451143d
Balance persistent workflow affinities
FurtherAI
committed
872e71f
Stabilize workflow host admission
FurtherAI
committed
98d2bcf

Commits on Aug 15, 2026

Calibrate Gemma MoE learned-policy parity
FurtherAI
committed
83ef9ec
Align DSV4 inference wave with training batch
FurtherAI
committed
265cc8d
Sustain DSV4 throughput wave pressure
FurtherAI
committed
ea7f51b
Calibrate balanced DSV4 throughput workload
FurtherAI
committed
e9fbc31
Disambiguate parity choices by full token path
FurtherAI
committed
e0dbc26
Isolate direct parity vLLM environments
FurtherAI
committed
9c4e3dd
Match duplicate parity leaves by source scores
FurtherAI
committed
349e25b
Calibrate Gemma parity to execution variance
FurtherAI
committed
474c232
Drain stale packing before throughput evidence
FurtherAI
committed
adcdded
Remove stale DSV4 host exclusivity
FurtherAI
committed
c1de2e0
Sustain DSV4 inference pressure
FurtherAI
committed
3a807db
Fully balance DSV4 throughput load
FurtherAI
committed
34896d3
Remove single-rank TCP rendezvous races
FurtherAI
committed
4a4bbab
Remove obsolete diagnostics and compatibility helpers
FurtherAI
committed
252d7f5
Promote distributed contracts and drop debug artifacts
FurtherAI
committed
219f7f6
Consolidate distributed lifecycle helpers
FurtherAI
committed
e96b531
Remove residual debug output
FurtherAI
committed
cef32e1
Root-cause and tighten train inference parity gates
FurtherAI
committed
f478ab2
Test architecture-scoped runtime caches
FurtherAI
committed
7cf4392
Clarify Gemma parity calibration evidence
FurtherAI
committed
480124b

Commits on Aug 17, 2026

Merge remote-tracking branch 'origin/main' into austin/monarch_multinode_training
FurtherAI
committed
67567e3
Retry inconclusive throughput evidence once
FurtherAI
committed
590627f
Refresh B300 throughput calibration identities
FurtherAI
committed
756b360
Skip numeric trajectory interning walks
FurtherAI
committed
fa7aa21
Avoid transient packing string interning
FurtherAI
committed
63861de
Skip interning transient rollout results
FurtherAI
committed
138b06d
Calibrate Llama train-inference parity envelope
FurtherAI
committed
1fdfd14
Calibrate Qwen 3.5 packing-order parity variance
FurtherAI
committed
efc9e7e
Record complete post-merge workflow pass
FurtherAI
committed
494d119
Make trajectory memory compaction explicit
FurtherAI
committed
a155ac6

Commits on Aug 18, 2026

Simplify model workflow orchestration
FurtherAI
committed
1904bf7
Remove stale workflow resource helpers
FurtherAI
committed
a02ac70
Trim workflow scheduler contract tests
FurtherAI
committed
2d9ad72
Focus oracle harness invariant coverage
FurtherAI
committed
87efc6d
Focus train inference invariant coverage
FurtherAI
committed
ca64556
Reduce workflow assertion scaffolding
FurtherAI
committed
e3b547c
Remove optional yes-no workflow coverage
FurtherAI
committed
4837b0a
Consolidate fast metrics contract tests
FurtherAI
committed
14c534d
Share runtime policy request fixtures
FurtherAI
committed
3016691
Accept source scripts in Monarch launcher
FurtherAI
committed
ce487ed
Fix workstation multi-node launch portability
FurtherAI
committed
fe978c1
Pin uv in cluster setup
FurtherAI
committed
cfa4b10
Add typed multi-node launch context
FurtherAI
committed
2e49ec6
Harden multi-node cluster setup
FurtherAI
committed
e9be341

Commits on Aug 19, 2026

Package managed Megatron runtimes for releases
FurtherAI
committed
bfb2370
Build package artifacts only for release PRs
FurtherAI
committed
a1f5158
Merge remote-tracking branch 'origin/main' into austin/monarch_multinode_training
FurtherAI
committed
364b477
Fix managed Megatron CI setup
FurtherAI
committed
3cfe6a2
Fingerprint managed runtime cache inputs
FurtherAI
committed
74ea2bd
Exercise managed Megatron runtime in CI
FurtherAI
committed
72e353d
Pin TileLang-compatible TVM FFI runtime
FurtherAI
committed
5d026ef
Pin the validated FLA kernel contract
FurtherAI
committed
e932bff