Skip to content

docs(training-guides): add GPU slicing with Dynamic Resource Allocation (DRA)#284

Open
typhoonzero wants to merge 1 commit into
masterfrom
docs/gpu-slicing-with-dra
Open

docs(training-guides): add GPU slicing with Dynamic Resource Allocation (DRA)#284
typhoonzero wants to merge 1 commit into
masterfrom
docs/gpu-slicing-with-dra

Conversation

@typhoonzero

Copy link
Copy Markdown
Contributor

What

Adds a new training-guides solution document showing how to request a slice of a GPU with Kubernetes Dynamic Resource Allocation (DRA)resource.k8s.io/v1, GA in Kubernetes 1.34 — using the NVIDIA DRA driver, and run a LoRA supervised fine-tune inside that slice with Kubeflow Trainer v2.

The guide teaches the DRA model (DeviceClassResourceClaimTemplate → Pod resourceClaims + container resources.claims) and contrasts it with the whole-GPU device-plugin approach, so a small fine-tune no longer has to occupy an entire card.

Contents

  • docs/en/training_guides/gpu-slicing-with-dra.mdx — device plugin vs. DRA, the three-object slice model, confirming the driver advertises slices, a DRA smoke test, a Trainer v2 fine-tune confined to a MIG slice, MIG profile sizing, the non-MIG time-slicing / MPS fallback, combining slices with Kueue quota, wiring the same claim into KFP / Volcano / plain Jobs, admin enablement, and troubleshooting.
  • docs/en/training_guides/assets/dra/ — MIG and shared-GPU ResourceClaimTemplates, a DRA smoke Pod, a self-contained (no model/dataset download) LoRA-SFT TrainingRuntime, and a TrainJob.
  • e2e/cases/c15_dra_gpu_slice.sh — end-to-end case that applies the template + runtime + TrainJob and asserts the fine-tune finishes inside the slice. It self-skips when no gpu.nvidia.com ResourceSlices are advertised, so it stays green on device-plugin-only clusters.
  • index.mdx — one new row in the "Pick a path" table.

Validation

  • doom lint clean (0 errors, 0 warnings).
  • All three built-in DRA objects (both ResourceClaimTemplates + the smoke Pod) pass server-side dry-run against a live Kubernetes 1.34 cluster running the NVIDIA DRA driver v25.8.1 — the resource.k8s.io/v1 exactly request shape, CEL selectors, and the opaque GpuConfig are all accepted by the real API.
  • Trainer v2 TrainingRuntime / TrainJob are modeled on the repo's published LlamaFactory runtime and parse cleanly.

Note on the live e2e

The full GPU e2e is deferred, not yet run live: the available dev cluster's only GPU is a HAMi-served A30 (actively serving an inference workload) and Kubeflow Trainer v2 is not installed there — activating the DRA kubelet-plugin on that card would conflict with HAMi and disrupt the running service. The MIG-profile attribute string in the selector follows the documented NVIDIA-driver convention; the guide explicitly directs readers to confirm it against their own kubectl get resourceslices -o yaml, and c15 will pin it down once a HAMi-free DRA GPU node is available.

🤖 Generated with Claude Code

…on (DRA)

Add a solution guide showing how to request a *slice* of a GPU via
Kubernetes DRA (resource.k8s.io/v1, GA in 1.34) with the NVIDIA DRA
driver, and run a LoRA supervised fine-tune inside it with Kubeflow
Trainer v2.

- New guide docs/en/training_guides/gpu-slicing-with-dra.mdx: device
  plugin vs. DRA, the DeviceClass -> ResourceClaimTemplate -> Pod claim
  model, smoke-testing a slice, a Trainer v2 fine-tune confined to a MIG
  slice, profile sizing, the non-MIG time-slicing fallback, Kueue quota,
  wiring into other orchestrators, admin enablement, and troubleshooting.
- assets/dra/: MIG + shared-GPU ResourceClaimTemplates, a DRA smoke Pod,
  a self-contained (no-download) LoRA-SFT TrainingRuntime, and a TrainJob.
- e2e/cases/c15_dra_gpu_slice.sh: end-to-end case that applies the
  template/runtime/TrainJob and asserts the fine-tune finishes inside the
  slice; self-skips when no gpu.nvidia.com ResourceSlices are advertised.
- index.mdx: add the guide to the "Pick a path" table.

The built-in DRA objects (both ResourceClaimTemplates + smoke Pod) pass
server-side dry-run against a live K8s 1.34 cluster running the NVIDIA
DRA driver v25.8.1; doom lint is clean. The live GPU e2e is deferred: the
target cluster's only GPU is a HAMi-served A30 and Kubeflow Trainer v2
is not installed there.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying alauda-ai with  Cloudflare Pages  Cloudflare Pages

Latest commit: deb0db5
Status: ✅  Deploy successful!
Preview URL: https://7e67fe79.alauda-ai.pages.dev
Branch Preview URL: https://docs-gpu-slicing-with-dra.alauda-ai.pages.dev

View logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant