Example for using RAPIDS with Dagster-Slurm#718
Open
HPicatto wants to merge 5 commits into
Open
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Contributor
|
We're gonna trigger CI to run additional tests, and test the whole thing on a SLURM cluster. Like we discussed before, It would be ideal to replace |
|
… example Adds a workflow example showing RAPIDS cuML (UMAP + HDBSCAN) running on a Slurm HPC cluster orchestrated with dagster-slurm: a partitioned CPU fan-out of gensim LDA jobs followed by GPU reduction/clustering stages in a separate packed pixi environment, with a CPU fallback path for GPU-less development. Includes UI screenshots from a real cluster run and a refined variant with metaxy sample-level incremental tracking. Refs rapidsai#715
- add a quickstart and an explicit 'by hand vs. here' contrast up front - call out the local iteration loop (identical code, laptop to cluster) - state which assets run in local mode plus expected size/runtime - mention run-scoped Slurm allocations for the queue-wait case - mention multi-cluster targeting via SLURM_EDGE_NODE_* configuration - add an 'HPC in the middle, Dagster end to end' section - soften the topic_map claim (plot not among the screenshots) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs: address tutorial review feedback (quickstart, personas, E2E story)
- Quickstart block right after the intro; 'what you would normally do' contrast paragraph for first-time HPC users - Explicit note that local mode materializes 3 of 6 assets - Download-size / wall-clock expectations for the docker chain - Queue-wait answer (experimental session/het-job modes) and multi-cluster deployment note for experienced Slurm users - Note that all-sbatch is a choice; ingest/publish can run off-Slurm in the same graph (and must, on sites without compute-node internet) - Iteration-loop paragraph in the conclusion - Add the topic_map artifact from the cluster run as the closing image - Stable cell ids (nbformat 5)
From a full end-to-end backfill on the real cluster (19m28s): the topic_map run view with cluster-count/path metadata, the topic_map preview rendering the scatter inline in the Dagster UI (closing the review gap about the promised 'streams a labeled meta-topic map' claim), and the all-green backfill overview with observed wall-clock numbers in the surrounding text.
HPicatto
force-pushed
the
feat/dagster-slurm-topic-modeling-example
branch
from
July 19, 2026 16:24
6661597 to
7ee9795
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… example
Adds a workflow example showing RAPIDS cuML (UMAP + HDBSCAN) running on a Slurm HPC cluster orchestrated with dagster-slurm: a partitioned CPU fan-out of gensim LDA jobs followed by GPU reduction/clustering stages in a separate packed pixi environment, with a CPU fallback path for GPU-less development. Includes UI screenshots from a real cluster run and a refined variant with metaxy sample-level incremental tracking.
Refs #715