From 1b3ae1d1fc2a006255acf878159f698d54495c72 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 9 Jun 2026 11:01:38 -0700 Subject: [PATCH 01/10] Add a prefill/decode disaggregation design doc Issue #34's original sketch predates the KServe drop and describes disaggregation through KServe's LLMInferenceService. This documents it on the current architecture: a self-contained prefill block on the deployment, KV transfer via the engine's kv-transfer-config and NIXL, routing through the same swappable EPP on a GAIE InferencePool that unified serving uses, the correctness constraints to enforce as matching matures, and when disaggregation is worth it. It parallels design/modelcache.md and the decisions discussed on the issue. Towards #34. Signed-off-by: Dennis Ramdass --- design/disaggregation.md | 182 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 design/disaggregation.md diff --git a/design/disaggregation.md b/design/disaggregation.md new file mode 100644 index 000000000..cc09566cd --- /dev/null +++ b/design/disaggregation.md @@ -0,0 +1,182 @@ +# Prefill/Decode Disaggregation + +**Status:** Draft +**Date:** June 2026 +**Author:** Dennis Ramdass + +This document proposes prefill/decode disaggregation for Modelplane. It builds +on the base design in [design.md](./design.md) and the routing it relies on. + +## Summary + +LLM inference has two phases with opposite hardware profiles. Prefill processes +the whole prompt at once and is compute-bound; it sets time-to-first-token. +Decode generates one token at a time and is memory-bandwidth-bound; it sets +inter-token latency. Run on the same pods, a prefill burst stalls in-flight +decodes and neither phase can be tuned independently. + +Disaggregation runs the two phases as separate pod sets. A prefill instance +processes the prompt and transfers its KV cache to a decode instance, which +generates the output. Modelplane expresses this with a `prefill` block on the +deployment: the top-level `workers` is the decode (or unified) role, and adding +a `prefill` block makes the deployment disaggregated. + +```yaml +apiVersion: modelplane.ai/v1alpha1 +kind: ModelDeployment +metadata: + name: llama-405b + namespace: ml-team +spec: + replicas: 1 + modelCacheRef: + name: llama-405b + # Top-level workers: the decode role. + workers: + count: 3 + topology: + tensor: 8 + template: + spec: + containers: + - name: engine + image: vllm/vllm-openai:v0.9.1 + args: + - "--model=/mnt/models" + - '--kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' + # Prefill role. Self-contained. + prefill: + workers: + count: 5 + topology: + tensor: 1 + template: + spec: + containers: + - name: engine + image: vllm/vllm-openai:v0.9.1 + args: + - "--model=/mnt/models" + - '--kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_producer"}' +``` + +## The prefill block + +The `prefill` block is self-contained: its own `workers.count`, `topology`, +`template`, and `nodeSelector`. It repeats settings rather than inheriting from +the root, because explicit repetition is easier to reason about than an implicit +merge. This matches the shape design.md already sketches for disaggregation. + +The prefill:decode ratio is the two `workers.count` values. It is a topology +parameter fixed per deployment, not a scaling knob, and both counts are +explicit. There is no default ratio, consistent with design.md avoiding +cross-resource defaulting. + +Because the block carries its own `nodeSelector` and `topology`, an operator can +place prefill and decode on different GPU classes through the normal +capability-matching mechanism. Prefill is compute-bound and suits high-FLOPS +GPUs; decode is memory-bandwidth-bound and suits high-bandwidth GPUs. Modelplane +does not choose that hardware. It exposes the knob, and the in-cluster scheduler +places the pods, the same as the unified path. Prefill and decode of a replica +stay on one InferenceCluster, since KV transfer needs co-location, so distinct +hardware means different pools within that cluster rather than different +clusters. + +A deployment without a `prefill` block is unified serving and is unaffected. + +## KV cache transfer + +The prefill engine produces the KV cache and the decode engine consumes it, +configured through the engine's `--kv-transfer-config` (`NixlConnector`, with +`kv_role` `kv_producer` on prefill and `kv_consumer` on decode). NIXL moves the +cache between the two over the fastest available interconnect. + +KV cache size grows roughly linearly with input length, on the order of 0.1 GB +per 1K input tokens for an 8B model, so under load the transfer can reach tens +of GB/s. That is comfortable over NVLink within a node and over RDMA/InfiniBand +across nodes, but it saturates PCIe or plain ethernet. Where the engine supports +it, the transfer is hidden behind compute (layer by layer, asynchronous, +chunked), keeping the disaggregation overhead small. + +## Routing + +Disaggregation needs a router that sends a request to a prefill instance, then +to a decode instance holding the transferred KV cache. Modelplane uses the same +routing layer as unified serving: a Gateway API Inference Extension +`InferencePool` fronted by a swappable endpoint-picker (EPP), defaulting to the +llm-d inference-scheduler. The EPP's prefill/decode scorer sequences the two +phases, and its prefix-cache scorer still applies. Disaggregation runs on the +multi-pod (llm-d) path, which already goes through this routing layer, so it +adds no separate proxy. + +A deployment with a `prefill` block selects the multi-pod backend even at +`pipeline: 1`, because disaggregation needs cross-pod coordination regardless of +the per-role topology. + +## Constraints + +These are documented now and enforced as the matching and validation surfaces +mature. + +- **Co-location.** A replica's prefill and decode must be schedulable on one + InferenceCluster. The fleet scheduler rejects the deployment if no matched + cluster can host both roles. +- **Interconnect.** KV transfer needs NVLink within a node or RDMA/InfiniBand + across nodes; over PCIe or ethernet it bottlenecks. It is required as a + cluster or pool capability (e.g. `networkInterNode`) and matched the same way + as other hardware requirements. +- **Connector and model compatibility.** Both roles run a compatible KV + connector (`NixlConnector`, paired `kv_role`) on the same model and dtype, + with compatible parallelism so the KV layout matches. +- **Both roles explicit.** A disaggregated deployment sets both `workers.count` + and `prefill.workers.count`. + +## When to use + +Disaggregation pays off for large models under load with strict TTFT and ITL +targets, long context, and a fast interconnect, where prefill and decode load +are large enough and skewed enough to tune separately. For small models, short +context, or low traffic, the KV-transfer overhead outweighs the benefit; +aggregated serving, optionally with chunked prefill, is simpler and usually +faster. The decision is the operator's. Modelplane serves unified by default and +disaggregates only when a `prefill` block is set. + +## Alternatives considered + +### KServe prefill section + +The original sketch (issue #34) expressed disaggregation through KServe's +`LLMInferenceService.prefill` section. Modelplane dropped KServe for a backend +dispatcher (native and llm-d), so disaggregation now lives in the `prefill` +block on the deployment and is emitted by the llm-d backend. The concept carries +over; the resource does not. + +### A bespoke prefill/decode proxy + +vLLM and Ray ship a small proxy that sequences prefill and decode. Running our +own proxy would work, but the GAIE `InferencePool` plus a swappable EPP is the +standard seam and already gives prefix- and KV-aware routing for unified +serving. Reusing it means one routing component for both unified and +disaggregated serving rather than a disaggregation-only proxy. + +### A routing discriminator instead of a template + +The EPP could be selected by a `picker` enum. Instead it is a curated PodSpec +subset (`routing.template`), the same shape and owner as the engine, defaulting +to the llm-d EPP and overridable by image and args. This avoids a discriminator +for a component that is really just a container, matching the engine convention +and design.md's preference against gratuitous discriminators. + +### Modelplane choosing per-role hardware + +Modelplane could read the compute-bound and bandwidth-bound profiles and place +each role on a chosen GPU class. It does not. Placement stays a user-declared +`nodeSelector` resolved by the in-cluster scheduler, the same as every other +workload. Modelplane exposes the knob and guards correctness; it does not make +in-cluster scheduling decisions. + +### An implicit prefill:decode ratio + +A default ratio would let a deployment request disaggregation without prefill +counts. Both counts are required instead, so the topology is explicit and +nothing depends on cross-resource defaulting. From 815861e3176c55163819df5ed234cf4ae2b77084 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 9 Jun 2026 11:07:35 -0700 Subject: [PATCH 02/10] Document disaggregated serving in the concepts guide The concepts guide covers unified and multi-node serving but not prefill/decode disaggregation, which the design doc now specifies. This adds a Disaggregated Serving section describing the prefill block, the compute-bound vs bandwidth-bound split, KV transfer over NIXL, the ModelCache and co-location requirements, and when disaggregation is worth it. It ties into the existing Multi-node Inference and ModelCache sections. Towards #34. Signed-off-by: Dennis Ramdass --- docs/content/concepts.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/content/concepts.md b/docs/content/concepts.md index 4d482618c..b913c56f6 100644 --- a/docs/content/concepts.md +++ b/docs/content/concepts.md @@ -136,6 +136,34 @@ across GPUs within a node. Multi-node deployments require a [ModelCache](#modelcache) referenced via `spec.modelCacheRef.name`. +## Disaggregated Serving + +Prefill (processing the whole prompt) and decode (generating tokens one at a +time) have opposite hardware profiles. Prefill is compute-bound and sets +time-to-first-token; decode is memory-bandwidth-bound and sets inter-token +latency. On shared pods a prefill burst stalls in-flight decodes, and neither +phase can be tuned on its own. + +Set a `prefill` block on the ModelDeployment to split them. The top-level +`workers` becomes the decode role and `prefill` is its own self-contained role, +with its own `workers.count`, `topology`, `template`, and `nodeSelector`, so +each phase can land on a different GPU class through the usual capability +matching. The prefill engine transfers its KV cache to a decode engine over +NIXL, configured through the engine's `--kv-transfer-config`. Like multi-node +serving, a disaggregated deployment requires a [ModelCache](#modelcache); both +roles mount the same PVC and stay co-located on one cluster, since KV transfer +needs a fast interconnect (NVLink within a node, RDMA across nodes). + +Disaggregation runs on the multi-node (llm-d) path. A request is routed to a +prefill instance and then to the decode instance holding its KV cache by the +same endpoint picker that fronts multi-node serving. A deployment without a +`prefill` block is unified serving and is unaffected. + +Disaggregation pays off for large models under load with strict latency targets +and long context. For small models or low traffic the KV-transfer overhead +outweighs the benefit, so aggregated serving (optionally with chunked prefill) +is the default. + ## ModelCache A ModelCache stages a model artifact on workload-cluster storage as a From f16b60e35bb0a6757e01cfbca4623d0eab9d9166 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 9 Jun 2026 13:45:19 -0700 Subject: [PATCH 03/10] Match the concepts guide to the ModelCache source discriminator The ModelCache section still described spec.source as a discriminated union keyed on which field is set, with lowercase future types under "their own discriminator". The shipped API is a required source enum naming the kind (source: HuggingFace) with the matching object set alongside it, validated by CEL. This updates the bullet to that shape so the guide matches the resource users actually write. Towards #34. Signed-off-by: Dennis Ramdass --- docs/content/concepts.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/content/concepts.md b/docs/content/concepts.md index b913c56f6..83103cb0f 100644 --- a/docs/content/concepts.md +++ b/docs/content/concepts.md @@ -181,11 +181,11 @@ engine container's `env`). Each cache has: -- A **source**: a discriminated union of where to fetch the artifact, with - source-specific fields (e.g. `huggingFace.repo` and `huggingFace.sizeGiB` - today). Future types (`dragonfly` for P2P distribution, `oci` for NIM-style - bundled artifacts) will declare different fields under their own - discriminator. +- A **source**: a required `source` enum naming the kind, with the matching + source object set alongside it (e.g. `source: HuggingFace` selects + `spec.huggingFace`, which carries `repo` and `sizeGiB`). `HuggingFace` is the + only value today; future sources add an enum value and a sibling object + (`Dragonfly` for P2P distribution, `OCI` for NIM-style bundled artifacts). - An optional **clusterSelector** to scope replication. Omitting `spec.clusterSelector` stages the cache on every matched cluster; setting `matchLabels` restricts it to clusters carrying those labels. From 2e87f15a80afef63afc3b655b0a609ff5091aa3c Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Tue, 9 Jun 2026 14:06:50 -0700 Subject: [PATCH 04/10] Skip the checklist requirement on draft PRs The checklist-completed job runs require-checklist-action on every pull_request event, including while a PR is still a draft. A WIP draft that hasn't filled out the template checklist yet fails the check, which is noise: the checklist is something you complete before asking for review, not while iterating. This gates the job on the PR not being a draft and adds the ready_for_review trigger, so the check is skipped while the PR is a draft and runs when it's marked ready for review. Signed-off-by: Dennis Ramdass --- .github/workflows/pr.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index cddda4638..c05221918 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -2,13 +2,17 @@ name: PR on: pull_request: - types: [opened, edited, synchronize] + types: [opened, edited, synchronize, ready_for_review] permissions: contents: read jobs: checklist-completed: + # Draft PRs are still in progress, so don't require a completed checklist + # until they're marked ready for review. The ready_for_review trigger above + # re-runs this job when that happens. + if: github.event.pull_request.draft == false permissions: # The action reads the PR body through the issues API, so it needs issues # access on top of pull-requests. The repo's default workflow token is From 909c2ede1cfea53034f379c4342df186cf6209c6 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 10 Jun 2026 15:55:35 -0700 Subject: [PATCH 05/10] Address design review feedback on disaggregation The example predated #101 merging, where nodeSelector became required: show it on both the decode and prefill roles, each with its own device selectors (illustrating distinct per-role hardware), and model the InfiniBand fabric as a Synthetic device. Fold the operator "when to use" guidance into the summary as background. Give the routing-discriminator alternative an API sketch so the discriminator-vs-template tradeoff is concrete, and add the "two ModelDeployments" alternative with why a single MD is better (co-location and that a prefill-only MD isn't conceptually a model deployment). Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Dennis Ramdass --- design/disaggregation.md | 98 ++++++++++++++++++++++++++++++++-------- 1 file changed, 79 insertions(+), 19 deletions(-) diff --git a/design/disaggregation.md b/design/disaggregation.md index cc09566cd..734ee5e77 100644 --- a/design/disaggregation.md +++ b/design/disaggregation.md @@ -21,6 +21,14 @@ generates the output. Modelplane expresses this with a `prefill` block on the deployment: the top-level `workers` is the decode (or unified) role, and adding a `prefill` block makes the deployment disaggregated. +It pays off for large models under load with strict TTFT/ITL targets, long +context, and a fast interconnect, where the two phases' loads are large and +skewed enough to tune separately; for small models, short context, or low +traffic the KV-transfer overhead outweighs the benefit and aggregated serving +(optionally with chunked prefill) is simpler. The choice is the operator's: +Modelplane serves unified by default and disaggregates only when a `prefill` +block is set. + ```yaml apiVersion: modelplane.ai/v1alpha1 kind: ModelDeployment @@ -31,7 +39,7 @@ spec: replicas: 1 modelCacheRef: name: llama-405b - # Top-level workers: the decode role. + # Top-level workers: the decode role (memory-bandwidth-bound). workers: count: 3 topology: @@ -44,7 +52,19 @@ spec: args: - "--model=/mnt/models" - '--kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' - # Prefill role. Self-contained. + # Decode's hardware. nodeSelector is required (a list of DRA device requests); + # here a high-VRAM GPU plus the InfiniBand fabric the KV transfer needs. + nodeSelector: + devices: + - name: gpu + count: 8 + selectors: + - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("141Gi")) >= 0 + - name: nic + count: 8 + selectors: + - cel: device.attributes["nic.nvidia.com"].linkType == "infiniband" + # Prefill role. Self-contained, with its own (compute-bound, single-GPU) hardware. prefill: workers: count: 5 @@ -58,6 +78,16 @@ spec: args: - "--model=/mnt/models" - '--kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_producer"}' + nodeSelector: + devices: + - name: gpu + count: 1 + selectors: + - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("80Gi")) >= 0 + - name: nic + count: 1 + selectors: + - cel: device.attributes["nic.nvidia.com"].linkType == "infiniband" ``` ## The prefill block @@ -122,26 +152,34 @@ mature. InferenceCluster. The fleet scheduler rejects the deployment if no matched cluster can host both roles. - **Interconnect.** KV transfer needs NVLink within a node or RDMA/InfiniBand - across nodes; over PCIe or ethernet it bottlenecks. It is required as a - cluster or pool capability (e.g. `networkInterNode`) and matched the same way - as other hardware requirements. + across nodes; over PCIe or ethernet it bottlenecks. The fabric is modeled as a + `Synthetic` device on the InferenceClass (e.g. a `nic` device with + `claim: Synthetic`, since no DRA driver claims it) and matched by a + `nodeSelector` device request, the same way as a claimable GPU — see the + example above. - **Connector and model compatibility.** Both roles run a compatible KV connector (`NixlConnector`, paired `kv_role`) on the same model and dtype, with compatible parallelism so the KV layout matches. - **Both roles explicit.** A disaggregated deployment sets both `workers.count` and `prefill.workers.count`. -## When to use +## Alternatives considered -Disaggregation pays off for large models under load with strict TTFT and ITL -targets, long context, and a fast interconnect, where prefill and decode load -are large enough and skewed enough to tune separately. For small models, short -context, or low traffic, the KV-transfer overhead outweighs the benefit; -aggregated serving, optionally with chunked prefill, is simpler and usually -faster. The decision is the operator's. Modelplane serves unified by default and -disaggregates only when a `prefill` block is set. +### Two ModelDeployments (one prefill, one decode) -## Alternatives considered +Disaggregation could be expressed as two separate ModelDeployments — one for +prefill, one for decode — reusing existing primitives with no new `prefill` +block. It is close to what's proposed and appealingly minimal, but the single-MD +form is better on two counts: + +- **Co-location.** With one MD the scheduler has everything it needs to place + prefill and decode workers on the same cluster sensibly. With two MDs the + author must either get crafty with `nodeSelector`s to force co-location, or we + add cross-MD co-scheduling hints (and the scheduler would then have to reason + about all MDs together). +- **It's still a model deployment.** A disaggregated MD is conceptually one + thing — "a model deployment." A prefill-only MD isn't one; it's an + implementation half that only makes sense paired with a decode MD. ### KServe prefill section @@ -161,11 +199,33 @@ disaggregated serving rather than a disaggregation-only proxy. ### A routing discriminator instead of a template -The EPP could be selected by a `picker` enum. Instead it is a curated PodSpec -subset (`routing.template`), the same shape and owner as the engine, defaulting -to the llm-d EPP and overridable by image and args. This avoids a discriminator -for a component that is really just a container, matching the engine convention -and design.md's preference against gratuitous discriminators. +The EPP could be selected by a `picker` enum: + +```yaml +spec: + routing: + picker: llm-d # enum: llm-d | ...; each value hard-codes an EPP +``` + +Instead it is a curated PodSpec subset, the same shape and owner as the engine — +defaulting to the llm-d EPP, overridable by image and args: + +```yaml +spec: + routing: + template: + spec: + containers: + - name: epp + image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.8.0 # default + args: ["--config-file=/config/epp.yaml"] # override to tune scorers +``` + +A discriminator would force Modelplane to enumerate and version every supported +picker; the template treats the EPP as what it is — a container — and lets a +user swap or tune it (different image, extra scorer args) without an API change, +matching the engine convention and design.md's preference against gratuitous +discriminators. ### Modelplane choosing per-role hardware From 6bc1e2a6405527db0274768239431851c03cae5a Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 10 Jun 2026 15:57:47 -0700 Subject: [PATCH 06/10] Fold the confirmed disaggregation routing design into the doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A feasibility spike pinned down how the routing actually works, so the Routing section now describes it concretely rather than as an aspiration: one InferencePool fronts both roles and the EPP partitions them by an llm-d.ai/role label, picking a decode pod then a prefill pod and handing the decode pod the prefill address (x-prefiller-host-port) for a routing sidecar to forward the prompt over NIXL — correcting the earlier decode-only-pool sketch. It also records the gateway decision: InferencePool needs a GAIE-conformant gateway, which core Envoy Gateway is not, so serving clusters run Envoy AI Gateway (layered on the same data plane, leaving plain routes untouched); unified serving stays on its plain Service route for now. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Dennis Ramdass --- design/disaggregation.md | 51 ++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 15 deletions(-) diff --git a/design/disaggregation.md b/design/disaggregation.md index 734ee5e77..fc59f7317 100644 --- a/design/disaggregation.md +++ b/design/disaggregation.md @@ -130,18 +130,39 @@ chunked), keeping the disaggregation overhead small. ## Routing -Disaggregation needs a router that sends a request to a prefill instance, then -to a decode instance holding the transferred KV cache. Modelplane uses the same -routing layer as unified serving: a Gateway API Inference Extension -`InferencePool` fronted by a swappable endpoint-picker (EPP), defaulting to the -llm-d inference-scheduler. The EPP's prefill/decode scorer sequences the two -phases, and its prefix-cache scorer still applies. Disaggregation runs on the -multi-pod (llm-d) path, which already goes through this routing layer, so it -adds no separate proxy. - -A deployment with a `prefill` block selects the multi-pod backend even at -`pipeline: 1`, because disaggregation needs cross-pod coordination regardless of -the per-role topology. +Disaggregation needs to route a request to a prefill instance, transfer the KV +cache, then have a decode instance generate from it. Modelplane does this with a +Gateway API Inference Extension (GAIE) `InferencePool` fronted by a swappable +endpoint-picker (EPP), defaulting to the llm-d inference-scheduler — no bespoke +proxy. + +**One `InferencePool` fronts both roles.** Its selector matches a deployment's +prefill and decode pods alike; the EPP partitions them internally by a role +label (`llm-d.ai/role: prefill|decode`). Per request the EPP picks a decode pod, +then a prefill pod, and passes the decode pod the chosen prefill's address (an +`x-prefiller-host-port` header). A small routing sidecar on the decode pod +forwards the prompt to that prefill, which runs prefill and transfers its KV +cache over NIXL; the decode engine then generates. The EPP's prefix-cache scorer +still applies, so cache-aware placement carries over. (An earlier sketch assumed +a decode-only pool with the EPP pair-picking across two pools; the llm-d +mechanism is the single-pool, role-partitioned form above.) + +The EPP itself is configured through `routing.template` — a curated PodSpec +subset, defaulting to the llm-d EPP and overridable by image and args, the same +shape and owner as the engine. + +A deployment with a `prefill` block selects the multi-pod (llm-d) backend even +at `pipeline: 1`, because disaggregation needs cross-pod coordination regardless +of the per-role topology. + +**Gateway.** `InferencePool` as an `HTTPRoute` backend needs a GAIE-conformant +gateway; core Envoy Gateway — which ServingStack installs today for plain +`HTTPRoute → Service` routing — does not serve it. Modelplane therefore runs +**Envoy AI Gateway** on the serving clusters: it layers on the same Envoy +Gateway data plane, so existing plain routes are unaffected. Unified serving +keeps its plain `Service` route for now; the `InferencePool`/EPP path is used by +disaggregated serving (and can later carry KV-/load-aware routing for unified +serving too). ## Constraints @@ -193,9 +214,9 @@ over; the resource does not. vLLM and Ray ship a small proxy that sequences prefill and decode. Running our own proxy would work, but the GAIE `InferencePool` plus a swappable EPP is the -standard seam and already gives prefix- and KV-aware routing for unified -serving. Reusing it means one routing component for both unified and -disaggregated serving rather than a disaggregation-only proxy. +standard seam for prefix- and KV-aware routing. Reusing it means one routing +component we can extend to unified serving later, rather than a +disaggregation-only proxy. ### A routing discriminator instead of a template From 4fea4982b7158711544b002b194e06424ab94944 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 10 Jun 2026 16:00:03 -0700 Subject: [PATCH 07/10] Explain InferencePool, the gateway choice, and the role labels The routing section named InferencePool without saying what it is: add a one-line definition and link to the GAIE docs. Frame Envoy AI Gateway as a deliberate choice among the conformant options (Envoy AI Gateway, Istio, kgateway), picked because it layers on the Envoy Gateway data plane ServingStack already runs rather than replacing the gateway. Note that Modelplane stamps the llm-d.ai/role label the EPP filters on, alongside its own internal modelplane.ai/pd-role label. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Dennis Ramdass --- design/disaggregation.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/design/disaggregation.md b/design/disaggregation.md index fc59f7317..ebcfbcb7b 100644 --- a/design/disaggregation.md +++ b/design/disaggregation.md @@ -136,9 +136,20 @@ Gateway API Inference Extension (GAIE) `InferencePool` fronted by a swappable endpoint-picker (EPP), defaulting to the llm-d inference-scheduler — no bespoke proxy. +An [`InferencePool`][gaie] is a Gateway API backend (used in an `HTTPRoute` like +a `Service`) that groups model-serving pods and delegates the per-request +endpoint choice to an EPP over the Endpoint Picker Protocol. It is the standard +seam where inference-aware routing — prefix-cache locality, load, prefill/decode +sequencing — plugs in, instead of the round-robin a plain `Service` does. + +[gaie]: https://gateway-api-inference-extension.sigs.k8s.io/ + **One `InferencePool` fronts both roles.** Its selector matches a deployment's prefill and decode pods alike; the EPP partitions them internally by a role -label (`llm-d.ai/role: prefill|decode`). Per request the EPP picks a decode pod, +label (`llm-d.ai/role: prefill|decode`) its prefill/decode filters select on. +Modelplane stamps that `llm-d.ai/role` label on each role's pods — alongside the +`modelplane.ai/pd-role` label its own compositions use internally, since the EPP +does not read Modelplane's label. Per request the EPP picks a decode pod, then a prefill pod, and passes the decode pod the chosen prefill's address (an `x-prefiller-host-port` header). A small routing sidecar on the decode pod forwards the prompt to that prefill, which runs prefill and transfers its KV @@ -157,12 +168,14 @@ of the per-role topology. **Gateway.** `InferencePool` as an `HTTPRoute` backend needs a GAIE-conformant gateway; core Envoy Gateway — which ServingStack installs today for plain -`HTTPRoute → Service` routing — does not serve it. Modelplane therefore runs -**Envoy AI Gateway** on the serving clusters: it layers on the same Envoy -Gateway data plane, so existing plain routes are unaffected. Unified serving -keeps its plain `Service` route for now; the `InferencePool`/EPP path is used by -disaggregated serving (and can later carry KV-/load-aware routing for unified -serving too). +`HTTPRoute → Service` routing — does not serve it. Of the conformant options +(Envoy AI Gateway, Istio, kgateway), **Modelplane chooses Envoy AI Gateway**: it +layers on the Envoy Gateway data plane ServingStack already runs (an additive +controller + CRDs, not a gateway swap), so it's the lowest-friction change and +leaves existing plain routes untouched — where Istio or kgateway would replace +the gateway. Unified serving keeps its plain `Service` route for now; the +`InferencePool`/EPP path is used by disaggregated serving (and can later carry +KV-/load-aware routing for unified serving too). ## Constraints From 7ca8ffd5b8355765964113f3228678ec85939fc7 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 10 Jun 2026 16:03:23 -0700 Subject: [PATCH 08/10] Reflect the 1:1 default ratio and fix the NIC count in the example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit workers.count defaults to 1, so an omitted prefill/decode count runs 1:1 rather than being required — say so instead of claiming the ratio is always explicit, and reframe the rejected alternative as "requiring both counts explicit" (which we don't, since it would make count mandatory for unified too). Drop the example's decode nic count from 8 to 1: it selects the InfiniBand fabric, not a per-GPU device. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Dennis Ramdass --- design/disaggregation.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/design/disaggregation.md b/design/disaggregation.md index ebcfbcb7b..d62c15acb 100644 --- a/design/disaggregation.md +++ b/design/disaggregation.md @@ -61,7 +61,7 @@ spec: selectors: - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("141Gi")) >= 0 - name: nic - count: 8 + count: 1 selectors: - cel: device.attributes["nic.nvidia.com"].linkType == "infiniband" # Prefill role. Self-contained, with its own (compute-bound, single-GPU) hardware. @@ -97,10 +97,10 @@ The `prefill` block is self-contained: its own `workers.count`, `topology`, the root, because explicit repetition is easier to reason about than an implicit merge. This matches the shape design.md already sketches for disaggregation. -The prefill:decode ratio is the two `workers.count` values. It is a topology -parameter fixed per deployment, not a scaling knob, and both counts are -explicit. There is no default ratio, consistent with design.md avoiding -cross-resource defaulting. +The prefill:decode ratio is the two `workers.count` values — a topology +parameter fixed per deployment, not a scaling knob. Each `count` defaults to 1, +so a disaggregated deployment that omits them runs 1:1; set them explicitly to +tune the ratio. Because the block carries its own `nodeSelector` and `topology`, an operator can place prefill and decode on different GPU classes through the normal @@ -194,8 +194,9 @@ mature. - **Connector and model compatibility.** Both roles run a compatible KV connector (`NixlConnector`, paired `kv_role`) on the same model and dtype, with compatible parallelism so the KV layout matches. -- **Both roles explicit.** A disaggregated deployment sets both `workers.count` - and `prefill.workers.count`. +- **Ratio.** The prefill:decode ratio is `workers.count` to + `prefill.workers.count`; each defaults to 1, so omitting them runs 1:1. Set + them explicitly to tune the ratio. ## Alternatives considered @@ -269,8 +270,10 @@ each role on a chosen GPU class. It does not. Placement stays a user-declared workload. Modelplane exposes the knob and guards correctness; it does not make in-cluster scheduling decisions. -### An implicit prefill:decode ratio +### Requiring both counts to be explicit -A default ratio would let a deployment request disaggregation without prefill -counts. Both counts are required instead, so the topology is explicit and -nothing depends on cross-resource defaulting. +We considered rejecting a disaggregated deployment that doesn't set both +`workers.count` and `prefill.workers.count`, to force the ratio to be stated. We +don't: `count` already defaults to 1, and enforcing explicit-only would mean +dropping that default — making `count` mandatory for unified deployments too. A +1:1 default is a reasonable starting point; tuning the ratio is an explicit edit. From 0e34d15f5d1c61ade39dbb4675f219434d46e631 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 10 Jun 2026 16:08:20 -0700 Subject: [PATCH 09/10] Frame the gateway as a pluggable default, not a least-change choice MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The earlier justification — Envoy AI Gateway is lowest-friction because it layers on what ServingStack already runs — is a non-argument for a greenfield product: there's no installed base to preserve, and it wrongly implied the gateway is fixed. Reframe it as the default Modelplane installs (chosen on merits: Envoy-based, LLM-purpose-built, InferencePool-conformant) while making the gateway a swappable seam like the EPP, so customers can plug in their own GAIE-conformant gateway — notably on BYO clusters that already run one. Signed-off-by: Dennis Ramdass --- design/disaggregation.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/design/disaggregation.md b/design/disaggregation.md index d62c15acb..f4826110a 100644 --- a/design/disaggregation.md +++ b/design/disaggregation.md @@ -167,13 +167,15 @@ at `pipeline: 1`, because disaggregation needs cross-pod coordination regardless of the per-role topology. **Gateway.** `InferencePool` as an `HTTPRoute` backend needs a GAIE-conformant -gateway; core Envoy Gateway — which ServingStack installs today for plain -`HTTPRoute → Service` routing — does not serve it. Of the conformant options -(Envoy AI Gateway, Istio, kgateway), **Modelplane chooses Envoy AI Gateway**: it -layers on the Envoy Gateway data plane ServingStack already runs (an additive -controller + CRDs, not a gateway swap), so it's the lowest-friction change and -leaves existing plain routes untouched — where Istio or kgateway would replace -the gateway. Unified serving keeps its plain `Service` route for now; the +gateway. ServingStack installs **Envoy AI Gateway** as the default — it's +Envoy-based, purpose-built for LLM traffic, and serves `InferencePool` v1 — so +disaggregation works out of the box on a Modelplane-provisioned cluster. But the +gateway is a swappable seam, like the EPP: any GAIE-conformant gateway works +(Istio, kgateway, GKE Gateway, …), so a customer can plug in their own — for a +BYO cluster that already runs one, or simply a different preference. (Plain +Envoy Gateway is not conformant on its own; `InferencePool` support comes from +the AI Gateway layer, which is why we install that rather than bare Envoy +Gateway.) Unified serving keeps its plain `Service` route for now; the `InferencePool`/EPP path is used by disaggregated serving (and can later carry KV-/load-aware routing for unified serving too). From b2e42103ff815ad6194752985c575d2d2f515ec4 Mon Sep 17 00:00:00 2001 From: Dennis Ramdass Date: Wed, 10 Jun 2026 16:35:51 -0700 Subject: [PATCH 10/10] Make routing a first-class explicit block; require explicit counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move the routing/EPP API out of the alternatives and into the main body: the example now shows a full deployment with top-level workers, prefill, and routing, and the Routing section describes routing.template directly. Per review, nothing in a disaggregated deployment is guessed: routing is required (no default EPP) and both workers.count and prefill.workers.count are explicit (no default ratio), symmetrical with how the engine is specified. Note where the EPP runs — a lightweight CPU Deployment on ordinary nodes, no GPU nodeSelector, one per InferencePool. Co-Authored-By: Claude Opus 4.8 (1M context) Signed-off-by: Dennis Ramdass --- design/disaggregation.md | 71 ++++++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 32 deletions(-) diff --git a/design/disaggregation.md b/design/disaggregation.md index f4826110a..07aeced01 100644 --- a/design/disaggregation.md +++ b/design/disaggregation.md @@ -88,6 +88,17 @@ spec: count: 1 selectors: - cel: device.attributes["nic.nvidia.com"].linkType == "infiniband" + # Routing: required for a disaggregated deployment. The endpoint picker (EPP) + # that sequences prefill -> decode, given explicitly like the engine — no + # guessed default. routing.template is a curated PodSpec subset (the EPP + # container); the llm-d inference-scheduler is the usual choice. + routing: + template: + spec: + containers: + - name: epp + image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.8.0 + args: ["--config-file=/config/epp.yaml"] ``` ## The prefill block @@ -98,9 +109,10 @@ the root, because explicit repetition is easier to reason about than an implicit merge. This matches the shape design.md already sketches for disaggregation. The prefill:decode ratio is the two `workers.count` values — a topology -parameter fixed per deployment, not a scaling knob. Each `count` defaults to 1, -so a disaggregated deployment that omits them runs 1:1; set them explicitly to -tune the ratio. +parameter fixed per deployment, not a scaling knob. Both are stated explicitly; +there is no default ratio. Defaulting one block of a disaggregated deployment +would be asymmetrical with the rest — we don't guess an engine config or the +`routing` EPP if you omit them — and the ratio is a deliberate topology choice. Because the block carries its own `nodeSelector` and `topology`, an operator can place prefill and decode on different GPU classes through the normal @@ -133,7 +145,7 @@ chunked), keeping the disaggregation overhead small. Disaggregation needs to route a request to a prefill instance, transfer the KV cache, then have a decode instance generate from it. Modelplane does this with a Gateway API Inference Extension (GAIE) `InferencePool` fronted by a swappable -endpoint-picker (EPP), defaulting to the llm-d inference-scheduler — no bespoke +endpoint-picker (EPP), typically the llm-d inference-scheduler — no bespoke proxy. An [`InferencePool`][gaie] is a Gateway API backend (used in an `HTTPRoute` like @@ -158,9 +170,17 @@ still applies, so cache-aware placement carries over. (An earlier sketch assumed a decode-only pool with the EPP pair-picking across two pools; the llm-d mechanism is the single-pool, role-partitioned form above.) -The EPP itself is configured through `routing.template` — a curated PodSpec -subset, defaulting to the llm-d EPP and overridable by image and args, the same -shape and owner as the engine. +The EPP is configured through `routing.template` — a curated PodSpec subset, the +same shape and owner as the engine — given **explicitly** on a disaggregated +deployment (the example above). There is no default EPP: we don't guess the +routing component any more than we guess the engine. `routing` is required when a +`prefill` block is set. + +The EPP is a lightweight CPU controller — it watches the deployment's pods and +scores requests; it serves no model and holds no GPU. So it runs as a small +Deployment on ordinary nodes in the serving namespace (no GPU, no special +`nodeSelector`), one per deployment's `InferencePool`. Its pod shape comes from +`routing.template`, the same way the engine's comes from `workers.template`. A deployment with a `prefill` block selects the multi-pod (llm-d) backend even at `pipeline: 1`, because disaggregation needs cross-pod coordination regardless @@ -196,9 +216,10 @@ mature. - **Connector and model compatibility.** Both roles run a compatible KV connector (`NixlConnector`, paired `kv_role`) on the same model and dtype, with compatible parallelism so the KV layout matches. -- **Ratio.** The prefill:decode ratio is `workers.count` to - `prefill.workers.count`; each defaults to 1, so omitting them runs 1:1. Set - them explicitly to tune the ratio. +- **Both counts explicit.** A disaggregated deployment sets both `workers.count` + and `prefill.workers.count`; there is no default ratio. +- **Routing explicit.** A disaggregated deployment sets `routing`; the EPP is + not defaulted. ## Alternatives considered @@ -236,7 +257,8 @@ disaggregation-only proxy. ### A routing discriminator instead of a template -The EPP could be selected by a `picker` enum: +The EPP could be selected by a `picker` enum instead of the `routing.template` +shown in the example above: ```yaml spec: @@ -244,22 +266,8 @@ spec: picker: llm-d # enum: llm-d | ...; each value hard-codes an EPP ``` -Instead it is a curated PodSpec subset, the same shape and owner as the engine — -defaulting to the llm-d EPP, overridable by image and args: - -```yaml -spec: - routing: - template: - spec: - containers: - - name: epp - image: ghcr.io/llm-d/llm-d-inference-scheduler:v0.8.0 # default - args: ["--config-file=/config/epp.yaml"] # override to tune scorers -``` - A discriminator would force Modelplane to enumerate and version every supported -picker; the template treats the EPP as what it is — a container — and lets a +picker. The template treats the EPP as what it is — a container — and lets a user swap or tune it (different image, extra scorer args) without an API change, matching the engine convention and design.md's preference against gratuitous discriminators. @@ -272,10 +280,9 @@ each role on a chosen GPU class. It does not. Placement stays a user-declared workload. Modelplane exposes the knob and guards correctness; it does not make in-cluster scheduling decisions. -### Requiring both counts to be explicit +### An implicit prefill:decode ratio -We considered rejecting a disaggregated deployment that doesn't set both -`workers.count` and `prefill.workers.count`, to force the ratio to be stated. We -don't: `count` already defaults to 1, and enforcing explicit-only would mean -dropping that default — making `count` mandatory for unified deployments too. A -1:1 default is a reasonable starting point; tuning the ratio is an explicit edit. +A default ratio — say 1:1 when the counts are omitted — would be convenient. We +require both counts instead: defaulting one block of a disaggregated deployment +is asymmetrical with the rest (we don't guess an engine config or the `routing` +EPP), and the ratio is a deliberate topology choice, not a sensible default.