docs(training-guides): add GPU slicing with Dynamic Resource Allocation (DRA)#284
Open
typhoonzero wants to merge 1 commit into
Open
docs(training-guides): add GPU slicing with Dynamic Resource Allocation (DRA)#284typhoonzero wants to merge 1 commit into
typhoonzero wants to merge 1 commit into
Conversation
…on (DRA) Add a solution guide showing how to request a *slice* of a GPU via Kubernetes DRA (resource.k8s.io/v1, GA in 1.34) with the NVIDIA DRA driver, and run a LoRA supervised fine-tune inside it with Kubeflow Trainer v2. - New guide docs/en/training_guides/gpu-slicing-with-dra.mdx: device plugin vs. DRA, the DeviceClass -> ResourceClaimTemplate -> Pod claim model, smoke-testing a slice, a Trainer v2 fine-tune confined to a MIG slice, profile sizing, the non-MIG time-slicing fallback, Kueue quota, wiring into other orchestrators, admin enablement, and troubleshooting. - assets/dra/: MIG + shared-GPU ResourceClaimTemplates, a DRA smoke Pod, a self-contained (no-download) LoRA-SFT TrainingRuntime, and a TrainJob. - e2e/cases/c15_dra_gpu_slice.sh: end-to-end case that applies the template/runtime/TrainJob and asserts the fine-tune finishes inside the slice; self-skips when no gpu.nvidia.com ResourceSlices are advertised. - index.mdx: add the guide to the "Pick a path" table. The built-in DRA objects (both ResourceClaimTemplates + smoke Pod) pass server-side dry-run against a live K8s 1.34 cluster running the NVIDIA DRA driver v25.8.1; doom lint is clean. The live GPU e2e is deferred: the target cluster's only GPU is a HAMi-served A30 and Kubeflow Trainer v2 is not installed there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploying alauda-ai with
|
| Latest commit: |
deb0db5
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7e67fe79.alauda-ai.pages.dev |
| Branch Preview URL: | https://docs-gpu-slicing-with-dra.alauda-ai.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new training-guides solution document showing how to request a slice of a GPU with Kubernetes Dynamic Resource Allocation (DRA) —
resource.k8s.io/v1, GA in Kubernetes 1.34 — using the NVIDIA DRA driver, and run a LoRA supervised fine-tune inside that slice with Kubeflow Trainer v2.The guide teaches the DRA model (
DeviceClass→ResourceClaimTemplate→ PodresourceClaims+ containerresources.claims) and contrasts it with the whole-GPU device-plugin approach, so a small fine-tune no longer has to occupy an entire card.Contents
docs/en/training_guides/gpu-slicing-with-dra.mdx— device plugin vs. DRA, the three-object slice model, confirming the driver advertises slices, a DRA smoke test, a Trainer v2 fine-tune confined to a MIG slice, MIG profile sizing, the non-MIG time-slicing / MPS fallback, combining slices with Kueue quota, wiring the same claim into KFP / Volcano / plain Jobs, admin enablement, and troubleshooting.docs/en/training_guides/assets/dra/— MIG and shared-GPUResourceClaimTemplates, a DRA smokePod, a self-contained (no model/dataset download) LoRA-SFTTrainingRuntime, and aTrainJob.e2e/cases/c15_dra_gpu_slice.sh— end-to-end case that applies the template + runtime + TrainJob and asserts the fine-tune finishes inside the slice. It self-skips when nogpu.nvidia.comResourceSlices are advertised, so it stays green on device-plugin-only clusters.index.mdx— one new row in the "Pick a path" table.Validation
doom lintclean (0 errors, 0 warnings).ResourceClaimTemplates + the smokePod) pass server-side dry-run against a live Kubernetes 1.34 cluster running the NVIDIA DRA driver v25.8.1 — theresource.k8s.io/v1exactlyrequest shape, CEL selectors, and the opaqueGpuConfigare all accepted by the real API.TrainingRuntime/TrainJobare modeled on the repo's published LlamaFactory runtime and parse cleanly.Note on the live e2e
The full GPU e2e is deferred, not yet run live: the available dev cluster's only GPU is a HAMi-served A30 (actively serving an inference workload) and Kubeflow Trainer v2 is not installed there — activating the DRA kubelet-plugin on that card would conflict with HAMi and disrupt the running service. The MIG-profile attribute string in the selector follows the documented NVIDIA-driver convention; the guide explicitly directs readers to confirm it against their own
kubectl get resourceslices -o yaml, andc15will pin it down once a HAMi-free DRA GPU node is available.🤖 Generated with Claude Code