feat: add on-demand models with shared LoRA and exclusive training

.dev_scripts/ci_container_test.sh

@@ -26,7 +26,7 @@ if [ "$MODELSCOPE_SDK_DEBUG" == "True" ]; then
     pip uninstall autoawq -y
     pip uninstall lmdeploy -y
     pip uninstall tensorflow -y
-    pip install ray==2.48
+    pip install ray==2.55.1
     pip install optimum
 
     # test with install

cookbook/client/server/megatron/server_config.yaml

@@ -1,4 +1,8 @@
-# Twinkle Server Configuration - Megatron Backend
+# Twinkle Server Configuration - Megatron LoRA Backend
+# Candidate for 8 x L20D (~269 GiB each): 2-4 single-GPU sampler instances
+# plus up to 2 two-GPU LoRA training instances (3 handles each, 6 total).
+# Validate 27B/32K peak memory, throughput and optimizer resume on L20D
+# before production rollout; local smaller-model tests do not prove those limits.
 
 # proxy_location: determines where the HTTP proxy runs.
 # "EveryNode" means each Ray node runs its own proxy (good for multi-node).
@@ -26,6 +30,13 @@ persistence:
   mode: redis
   redis_url: redis://localhost:6380
 
+# Independent allocation, startup and confirmed-shutdown budgets.
+training:
+  allocation_timeout_s: 3
+  startup_timeout_s: 1800
+  shutdown_timeout_s: 180
+  sweep_interval_s: 5
+
 # Applications: each entry defines a service component deployed on the server
 applications:
 
@@ -36,7 +47,7 @@ applications:
     import_path: server            # Python module to import
     args:
       server_config:
-        per_token_model_limit: 3      # Maximum number of models (adapters) per token (server-globally enforced)
+        per_token_model_limit: 3      # Global per-token limit; shared-token clients share these 3 slots
       supported_models:
         - Qwen/Qwen3.8-27B
 
@@ -56,30 +67,33 @@ applications:
 
   # 3. Sampler Service - Runs inference / sampling using vLLM engine
   #    Used for generating text from the model (e.g., evaluating LoRA results).
-  #    Config: TP=2 x DP=2 on 4 GPUs, ~27GB weights/GPU, ~37GB for KV cache + LoRA
+  #    Each instance has DP=1 and one GPU. Two are preloaded; scale to four.
+  #    Memory usage includes each engine's preallocated KV cache, not just weights.
   - name: sampler-Qwen3.8-27B
     route_prefix: /api/v1/sampler/Qwen/Qwen3.8-27B
     import_path: sampler
     args:
       model_id: "ms://Qwen/Qwen3.8-27B"  # ModelScope model identifier
-      nproc_per_node: 4               # Number of GPU processes per node
+      nproc_per_node: 1               # One GPU per independently allocated instance
+      min_replicas: 2
+      max_replicas: 4
+      target_ongoing_requests: 16
+      idle_timeout_s: 600
       sampler_type: vllm              # Inference engine: vllm
       engine_args:                    # vLLM engine-specific settings
         max_model_len: 32768           # Maximum sequence length the engine supports
-        gpu_memory_utilization: 0.75   # 80% utilization, ~64GB/GPU, leaves buffer for safety
+        gpu_memory_utilization: 0.75   # 75% engine memory budget; does not free Ray GPU reservations
         enable_lora: true             # Allow loading LoRA adapters during inference
         max_loras: 5                  # Max allowed loras working on vLLM at the same time
-        max_lora_rank: 32             # Support up to rank 64 LoRA adapters
+        max_lora_rank: 32             # Maximum supported LoRA rank
         enable_tower_connector_lora: true
       device_group:                   # Logical device group for the sampler
         name: sampler
-        gpus_per_worker: 2
-        ranks: 4                   # GPU rank indices to use
+        ranks: 1                   # Per-instance device count, not fixed physical GPU IDs
         device_type: cuda
       device_mesh:
         device_type: cuda
-        dp_size: 2
-        tp_size: 2                  # 2 TP replicas for multi-tenant throughput
+        dp_size: 1                 # Pool instance count is controlled by args.min/max_replicas
       queue_config:
         rps_limit: 20                               # Max requests per second
         tps_limit: 131072                            # Max tokens per second
@@ -97,32 +111,37 @@ applications:
               TWINKLE_TRUST_REMOTE_CODE: "0"
               TWINKLE_LONG_POLL_TIMEOUT: "120"
 
-  # 2. Model Service - Hosts the base model for training.
-  #    Config: PP=2 x DP=2 on 4 GPUs, ~27GB weights/GPU, comfortable for LoRA training
+  # 2. Model Service - LoRA-only, independently queued two-GPU instances.
+  #    Megatron instances require at least two devices; PP=2 shards each model.
+  #    min=0 enables cold startup; max=2 limits total training devices to four.
   - name: models-Qwen3.8-27B
     route_prefix: /api/v1/model/Qwen/Qwen3.8-27B
     import_path: model
     args:
       backend: megatron                          # Use Megatron-LM backend
       model_id: "ms://Qwen/Qwen3.8-27B"          # ModelScope model identifier
       max_length: 32768                           # model max length
-      max_loras: 3                                # model max loras
-      nproc_per_node: 4                           # Number of GPU processes per node
+      train_mode: lora
+      max_loras: 3                                # Handles per instance, not simultaneous backward calls
+      min_replicas: 0                             # Do not reserve training GPUs at startup
+      max_replicas: 2                             # Up to 2 independent two-GPU instances
+      idle_timeout_s: 600                         # Business idle; heartbeats/polling do not renew it
+      nproc_per_node: 2                           # Two workers per training instance
       device_group:
         name: model
-        ranks: 4
+        ranks: 2
         device_type: cuda
       device_mesh:
         device_type: cuda
-        dp_size: 2
+        dp_size: 1
         pp_size: 2
 
       queue_config:
         rps_limit: 20                               # Max requests per second
         tps_limit: 131072                            # Max tokens per second
         max_input_tokens: 32768
       adapter_config:
-        adapter_timeout: 120                       # Seconds before idle adapter unload
+        adapter_timeout: 120                       # Session heartbeat timeout, not business idle
         adapter_max_lifetime: 36000               # Maximum lifetime of an adapter in seconds (e.g., 10 hours)
     deployments:
       - name: ModelManagement

cookbook/client/server/on_demand_training.yaml

@@ -0,0 +1,41 @@
+# On-demand training: applications stays the public configuration entry.
+training:
+  allocation_timeout_s: 3
+  startup_timeout_s: 1800
+  # PPU actor/PlacementGroup death confirmation can exceed the 60s framework default.
+  shutdown_timeout_s: 180
+  sweep_interval_s: 5
+applications:
+  - name: model-qwen
+    route_prefix: /api/v1/model/Qwen/Qwen3-1.7B
+    import_path: model
+    args:
+      model_id: ms://Qwen/Qwen3-1.7B
+      backend: transformers
+      device_type: cuda
+      modes:
+        lora:
+          num_devices: 2
+          device_mesh: {device_type: cuda, dp_size: 2}
+          min_replicas: 1
+          max_loras: 5
+          max_r: 32
+          idle_timeout_s: 1800
+          target_modules: all-linear
+        full:
+          num_devices: 4
+          device_mesh: {device_type: cuda, dp_size: 4}
+          min_replicas: 0
+          idle_timeout_s: 1800
+  - name: model-small
+    route_prefix: /api/v1/model/Qwen/Qwen3-0.6B
+    import_path: model
+    args:
+      model_id: ms://Qwen/Qwen3-0.6B
+      backend: megatron
+      device_type: cuda
+      modes:
+        lora:
+          num_devices: 2
+          device_mesh: {device_type: cuda, dp_size: 2}
+          max_loras: 5

docs/source_en/Usage Guide/Server and Client/Server.md

@@ -479,3 +479,59 @@ raises. During that window newly arriving tasks fail fast with a `server`/503 er
 Each persisted future stores its immutable `absolute_deadline` when it is created.
 Cleanup therefore reaches the same decision regardless of which deployment process
 holds the cleanup lease.
+
+## On-demand training and device reclamation
+
+Existing `applications` entries with `import_path: model` remain valid. Set `args.min_replicas: 0` for lazy startup and explicitly set `args.idle_timeout_s` for business inactivity. Legacy `adapter_config.adapter_timeout` remains a heartbeat timeout and `adapter_max_lifetime` retains its absolute lifetime. New `modes` default to 1800 seconds of business inactivity; heartbeats and polling do not renew that deadline; queued and running operations protect training state. Empty instances above the resident minimum are destroyed so other models can reuse their devices. Existing deployment min/max replicas become training instance minimums/limits.
+
+`modes` is optional: single-mode training does not require rewriting existing YAML. Model `deployments` options configure the CPU router; autoscaling request targets apply to router replicas. The Controller owns training-instance minimums/limits, and each compute instance has exactly one Serve replica. Non-autoscaling request limits, health/shutdown settings, and `ray_actor_options.runtime_env` also reach compute instances. `args.env_vars` overrides matching deployment environment variables. Unsupported `adapter_config` fields fail validation instead of being silently ignored.
+
+To offer both LoRA and full training for one model, configure `num_devices`, `device_mesh`, `min_replicas`, and `idle_timeout_s` separately under `args.modes.lora/full`. LoRA users share compatible instances; each full-training handle owns an instance. Sampling configuration stays the same. See `cookbook/client/server/on_demand_training.yaml` in the repository.
+
+### Single-node, eight-GPU LoRA capacity
+
+`cookbook/client/server/megatron/server_config.yaml` is a candidate for large-memory L20D GPUs. It retains single-mode YAML: two preloaded single-GPU sampling instances (DP=1, no TP setting), scaling up to four, plus up to two two-GPU training instances (PP=2, DP=1). Megatron configurations require at least two devices per instance before allocation; single-device Megatron is not supported. Only `train_mode: lora` is configured; full training remains available in other framework configurations. Key model args:
+
+```yaml
+train_mode: lora
+nproc_per_node: 2
+device_group: {name: model, ranks: 2, device_type: cuda}
+device_mesh: {device_type: cuda, dp_size: 1, pp_size: 2}
+max_loras: 3
+min_replicas: 0
+max_replicas: 2
+idle_timeout_s: 600
+```
+
+- `max_replicas × max_loras = 6` bounds training handles, not throughput. Creation fills existing compatible LoRA slots before starting another instance; active training is not migrated for load balancing. Stateful operations are serial within an instance, with at most two independent training queues. `per_token_model_limit: 3` remains a server-wide per-token quota, not a total-user cap. Clients sharing one token share those three slots; independent users need independent tokens, or a deliberately increased shared-token quota. Set it to 1 separately if the service requires one handle per independent token.
+- PP=2 shards each model over two devices; removing DP replication from the original PP=2/DP=2 instance leaves room for two independent instances. Account for 32K activations, LoRA optimizers, communication and temporary workspaces. Smaller-model or different-hardware E2E tests do not certify L20D/27B/32K peak memory or throughput.
+- Save checkpoints during training and explicitly call `remove_adapter()` or unload the Tinker model when finished. Heartbeats and polling do not renew business idle. Expired handles require a new create and explicit checkpoint resume; old training is not resumed automatically.
+- `min_replicas: 0` permits reclaiming every empty training instance at the cost of cold startup. Both pools use the same device-allocation and instance-lifecycle implementation with independent state. Reclaiming training devices does not directly scale sampling; sampling grows according to its own in-flight load. Lowering vLLM `gpu_memory_utilization` does not release Ray's whole-device reservation. No fractional-GPU or training/sampling colocation is enabled.
+
+Before rollout, validate the actual 27B model on L20D: a two-GPU PP=2/DP=1 32K workload at the maximum business batch, three resident rank-32 LoRAs, two instances/six handles alongside sampling, optimizer save/resume, tenant isolation and confirmed device reclamation. Measure peak memory, training tokens/s, sampling output tokens/s, queue wait and P95 latency against the original four-GPU PP=2/DP=2 layout using the same work. If the two-GPU layout does not meet business requirements, restore four GPUs per instance (PP=2, DP=2, at most one instance); do not infer throughput from an idle memory snapshot.
+
+### On-demand sampling instances and dispatch
+
+A stable CPU Router selects the ready instance with the fewest actual in-flight operations for ordinary sampling, DataRefs and streaming. Clients do not supply an instance ID, and users/LoRAs are not pinned to a device. Per-instance DP/TP describes an engine's parallel shape, not pool replicas. The production candidate uses one GPU and DP=1 per instance, without TP configuration.
+
+```yaml
+nproc_per_node: 1
+device_group: {name: sampler, ranks: 1, device_type: cuda}
+device_mesh: {device_type: cuda, dp_size: 1}
+min_replicas: 2
+max_replicas: 4
+target_ongoing_requests: 16
+idle_timeout_s: 600
+```
+
+Both minimum instances must finish loading before readiness. Demand is `ceil((in-flight requests + 1) / 16)`, capped at four. Starting and quarantined instances count against that cap. Existing ready instances serve during expansion; only idle extras with no actual generation are reclaimed after 600 seconds, down to two. Neither pool preempts or colocates on a GPU; four sampling plus four training GPUs fit the eight-device maximum.
+
+New instances load only the base model and default template. LoRAs load from each request's `adapter_uri`, `model_path` or persistent sampling-session checkpoint path, into independent disposable caches. No cache synchronization or new cache database is introduced. `engine_args.max_loras: 5` bounds active engine LoRAs, not training handles. Custom templates, named adapter settings and patches are broadcast under pool-wide exclusive protection and replayed before new instances become ready. Partial broadcast failure fences all participating replicas. Controller restart clears the old pool and dynamic control settings; clients must reapply them, while existing sampling-session/checkpoint paths remain unchanged.
+
+HTTP disconnect and future timeout do not certify generation exit. Completion handles and weight guards remain held until actual exit; unconfirmed death remains `QUARANTINED` without early capacity release. Full-checkpoint sampling retains pool-wide exclusive weight changes. L20D/27B single-GPU 32K memory, four-engine concurrency, output tokens/s and P95 latency require controlled same-hardware validation; local PPU/4B tests cannot certify them.
+
+### Rollout compatibility and safe states
+
+Keeping YAML syntax does not keep the old SDK wire unchanged. The Twinkle SDK performs the `training_handles` handshake, generates `client_create_id` and supplies `training_id`; upgrade it together with the server. Tinker keeps its own model-handle wire.
+
+Ray 2.55 is the minimum dependency: Serve ingress must await asynchronous user constructors; do not use the old CI Ray 2.48 pin. Startup, business idle, task execution and confirmed shutdown have separate budgets. Atomic create start and cancel are mutually exclusive; an unknown state-backend acknowledgement must not authorize execution. `QUARANTINED` means device death is unconfirmed and still counts toward the instance limit; release must not report unconfirmed reclamation as success. Only an actually executed operation renews business idle, not a queue failure or unknown start acknowledgement.