Skip to content

[Docs] Service groups and Replica IP interpolation #4185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Bihan wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[Docs] Service groups and Replica IP interpolation #4185

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 33 additions & 18 deletions mkdocs/docs/concepts/services.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,8 @@ Setting the minimum number of replicas to `0` allows the service to scale down t
<span id="replica-groups"></span>

??? info "Replica groups"

@peterschmidt85 peterschmidt85 Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why its collapsible section? Shouldn't we do it similar to Tasks - as a separate section?

@peterschmidt85 peterschmidt85 Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, shouldn't we cross-link Replica groups and PD disaggregation sections?

A service can include multiple replica groups. Each group can define its own `commands`, `resources` requirements, and `scaling` rules.
A service can define multiple replica groups. Each group has its own `replicas` count (or range),
`resources`, `commands`, and `scaling` rules.

<div editor-title="service.dstack.yml">

Expand All @@ -303,8 +304,8 @@ Setting the minimum number of replicas to `0` allows the service to scale down t
env:
- MODEL_ID=deepseek-ai/DeepSeek-R1-Distill-Llama-8B

replicas:
- count: 1..2
groups:
- replicas: 1..2
scaling:
metric: rps
target: 10
Expand All @@ -317,7 +318,7 @@ Setting the minimum number of replicas to `0` allows the service to scale down t
resources:
gpu: 48GB

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 5
Expand All @@ -336,7 +337,21 @@ Setting the minimum number of replicas to `0` allows the service to scale down t

</div>

> Properties such as `regions`, `port`, `image`, `env` and some other cannot be configured per replica group. This support is coming soon.
[`groups`](../reference/dstack.yml/service.md#groups) and top-level [`replicas`](../reference/dstack.yml/service.md#replicas) are mutually exclusive.

> Properties such as `regions`, `port`, `env` and some other cannot be configured per replica group. This support is coming soon.

### Accessing replica IPs

@peterschmidt85 peterschmidt85 Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be under "Replica groups" section?
Also shall we make it collapsible ("???") section?


Commands in any group can reference the internal IP address of any replica in the run via
`${{ groups[i].replicas[j].IP_ADDRESS }}`, where `i` is the index of the group in `groups` and `j` is
the index of the replica within that group.

> Only replicas guaranteed at start can be referenced:
>
> - `replicas: 2` → replica indexes `0` and `1` can be referenced
> - `replicas: 1..4` → replica index `0` can be referenced
> - `replicas: 0..4` → no replica indexes can be referenced

### PD disaggregation

Expand All @@ -363,9 +378,9 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
- HF_TOKEN
- MODEL_ID=zai-org/GLM-4.5-Air-FP8

replicas:
- count: 1
# For now replica group with router must have count: 1
groups:
- replicas: 1
# For now the router group must have replicas: 1
commands:
- pip install smg
- |
Expand All @@ -379,7 +394,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
router:
type: sglang

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 3
Expand All @@ -394,7 +409,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
resources:
gpu: H200

- count: 1..8
- replicas: 1..8
scaling:
metric: rps
target: 2
Expand Down Expand Up @@ -437,8 +452,8 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
- HF_TOKEN
- MODEL_ID=zai-org/GLM-4.5-Air-FP8

replicas:
- count: 1
groups:
- replicas: 1
docker: true
commands:
- apt-get update
Expand All @@ -460,7 +475,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
router:
type: dynamo

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 3
Expand Down Expand Up @@ -489,7 +504,7 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`:
resources:
gpu: H200

- count: 1..8
- replicas: 1..8
scaling:
metric: rps
target: 2
Expand Down Expand Up @@ -551,8 +566,8 @@ env:
- RDMA_DEVICES=bnxt_re0,bnxt_re1,bnxt_re2,bnxt_re3,bnxt_re4,bnxt_re5,bnxt_re6,bnxt_re7
- NCCL_IB_DISABLE=1

replicas:
- count: 1
groups:
- replicas: 1
commands:
- pip install smg
- |
Expand All @@ -565,7 +580,7 @@ replicas:
router:
type: sglang

- count: 1..2
- replicas: 1..2
scaling:
metric: rps
target: 300
Expand All @@ -591,7 +606,7 @@ replicas:
cpu: 96..
memory: 512GB..

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 300
Expand Down
8 changes: 4 additions & 4 deletions mkdocs/docs/examples/inference/dynamo.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ env:
- HF_TOKEN
- MODEL_ID=zai-org/GLM-4.5-Air-FP8

replicas:
- count: 1
groups:
- replicas: 1
docker: true
commands:
- apt-get update
Expand All @@ -47,7 +47,7 @@ replicas:
router:
type: dynamo

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 3
Expand Down Expand Up @@ -76,7 +76,7 @@ replicas:
resources:
gpu: H200

- count: 1..8
- replicas: 1..8
scaling:
metric: rps
target: 2
Expand Down
28 changes: 14 additions & 14 deletions mkdocs/docs/examples/inference/sglang.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,9 +152,9 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
- HF_TOKEN
- MODEL_ID=zai-org/GLM-4.5-Air-FP8

replicas:
- count: 1
# For now replica group with router must have count: 1
groups:
- replicas: 1
# For now replica group with router must have replicas: 1
commands:
- pip install smg
- |
Expand All @@ -168,7 +168,7 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
router:
type: sglang

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 3
Expand All @@ -184,7 +184,7 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
resources:
gpu: H200

- count: 1..8
- replicas: 1..8
scaling:
metric: rps
target: 2
Expand Down Expand Up @@ -225,9 +225,9 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
- HF_TOKEN
- MODEL_ID=zai-org/GLM-4.5-Air-FP8

replicas:
- count: 1
# For now replica group with router must have count: 1
groups:
- replicas: 1
# For now replica group with router must have replicas: 1
python: "3.12"
commands:
- pip install smg
Expand All @@ -244,7 +244,7 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
resources:
cpu: 4

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 3
Expand All @@ -262,7 +262,7 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
resources:
gpu: H200

- count: 1..8
- replicas: 1..8
scaling:
metric: rps
target: 2
Expand Down Expand Up @@ -309,8 +309,8 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
- RDMA_DEVICES=bnxt_re0,bnxt_re1,bnxt_re2,bnxt_re3,bnxt_re4,bnxt_re5,bnxt_re6,bnxt_re7
- NCCL_IB_DISABLE=1

replicas:
- count: 1
groups:
- replicas: 1
commands:
- pip install smg
- |
Expand All @@ -323,7 +323,7 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
router:
type: sglang

- count: 1..2
- replicas: 1..2
scaling:
metric: rps
target: 300
Expand All @@ -349,7 +349,7 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/
cpu: 96..
memory: 512GB..

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 300
Expand Down
8 changes: 4 additions & 4 deletions mkdocs/docs/examples/inference/vllm.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,8 @@ env:
- HF_TOKEN
- MODEL_ID=zai-org/GLM-4.5-Air-FP8

replicas:
- count: 1
groups:
- replicas: 1
python: "3.12"
commands:
- pip install smg
Expand All @@ -158,7 +158,7 @@ replicas:
resources:
cpu: 4

- count: 1..4
- replicas: 1..4
scaling:
metric: rps
target: 3
Expand All @@ -173,7 +173,7 @@ replicas:
resources:
gpu: H200

- count: 1..8
- replicas: 1..8
scaling:
metric: rps
target: 2
Expand Down
4 changes: 2 additions & 2 deletions mkdocs/docs/reference/dstack.yml/service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,9 @@ The `service` configuration type allows running [services](../../concepts/servic
required: true


### `replicas`
### `groups`

#### `replicas[n]`
#### `groups[n]`

#SCHEMA# dstack._internal.core.models.configurations.ReplicaGroup
overrides:
Expand Down
Loading