Address CSCS review feedback and add web deployment by robobryce · Pull Request #220 · NVIDIA/accelerated-computing-hub

robobryce · 2026-07-22T15:22:53Z

Summary

address the remaining review feedback on the PyHPC tutorial, including links, reproducible Colab installs, the notebook 10 typo, CUDA math, and safe nanobind buffers
add a supported 10-hour Daint launcher that pulls the GitHub CI image, bind-mounts the checkout, and serves JupyterLab, Nsight Systems, and Nsight Compute over HTTPS
document the five-port SSH tunnel required for Streamer WebRTC over TURN/TCP, plus reconnect, cancellation, persistence, and cleanup behavior
keep the generic Docker/Brev path unchanged while adapting rootless Podman for Daint host networking

This PR deliberately targets event/2026-07-cscs-summer-school; merging it updates the event branch reviewed by #215.

Live Daint validation

Tested from this node against Daint job 4258261 on nid006001 using the GitHub-CI PyHPC image and pinned NVIDIA Streamer images:

JupyterLab returned HTTPS 200, and a file written through its HTTPS API appeared in the bind-mounted $SCRATCH checkout
both Streamer HTTPS interfaces loaded through the SSH tunnel
both Streamers selected connected, nominated TURN/TCP relay candidate pairs
both input data channels opened and both 1280x800 video streams advanced with zero dropped frames in the samples
graceful TERM cleanup removed containers, listeners, private TLS/TURN state, and job-local image stores while preserving student work

Validation

Bash/Python syntax and ShellCheck for the new launcher
all tutorial Docker Compose files render
default and CSCS rootless-Podman transforms, including generic nsight and isolated NCU ports/display
all 26 notebook JSON/rename/install consistency checks
nanobind compile/runtime buffer guards
CUDA 13.3 compile validation
Jupyter HTTPS/proxy/plugin checks
link check: 87 valid, 0 errors
live browser HTTPS/WebRTC/persistence/cleanup test on Daint

copy-pr-bot · 2026-07-22T15:22:57Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-07-22T15:23:02Z

👋 Thank you for your contribution!

This pull request is from a forked repository so GitHub Actions will not be able to run CI. A maintainer will review your changes shortly and manually trigger the CI.

@maintainers Please review this PR when you have a chance and follow the instructions in the CONTRIBUTING.md file to trigger the CI.

brycelelbach · 2026-07-23T11:45:37Z

+```bash
+export CSCS_ACCOUNT="YOUR_ACCOUNT"
+export ACH_STATE="${SCRATCH}/ach-pyhpc-web"
+mkdir -p -m 700 "${ACH_STATE}"
+umask 077
+
+cd "${ACH_REPO}"
+JOB_ID=$(sbatch --parsable \
+  --account="${CSCS_ACCOUNT}" --partition=normal --time=10:00:00 \
+  --nodes=1 --ntasks=1 --gpus=1 --signal=B:TERM@60 \
+  --job-name=ach-pyhpc-web \
+  --chdir="${ACH_REPO}" --output="${ACH_STATE}/slurm-%j.log" \
+  --export=ALL,ACH_REPO="${ACH_REPO}",ACH_STATE="${ACH_STATE}",ACH_BRANCH="${ACH_BRANCH}" \
+  tutorials/pyhpc/brev/start-cscs-web.bash)
+echo "JOB_ID=${JOB_ID}"
+tail -F "${ACH_STATE}/slurm-${JOB_ID}.log"
+```


It sounds like we should probably just have a script for doing the job launch, and the script should also check out the git repo.

If there is an existing repo, it should only update to top of trunk if the existing repo is on main and not dirty.

Addressed in launch-cscs-web.bash: the launcher now owns checkout preparation. It clones the event branch when the checkout is absent and only changes an existing checkout when it is on main and completely clean, including untracked files. Dirty or non-main checkouts are left untouched; a separate managed release checkout supplies the deployment code.

Final naming update: this behavior is now in cscs-launch-tutorial. It still clones when absent, only moves an existing clean main checkout, preserves dirty/non-main student checkouts, and uses a separate clean runtime checkout.

brycelelbach · 2026-07-23T11:46:37Z

+```
+
+Wait for `READY node=nidXXXXXX`, then copy that node name. `Ctrl-C` only stops
+`tail`; it does not stop the job. The first launch takes several minutes while


Why did you put this note about ctrl-c here?

Are users required to ctrl-c (e.g. tail runs forever)?

I would prefer a script that completes after the node is ready.

Addressed. The launcher waits for the Slurm job and all three services, prints CSCS_WEB_JOB_ID and CSCS_WEB_NODE, and then exits. There is no tail or required Ctrl-C; the workstation connector owns the long-lived SSH shell.

Confirmed unchanged after the latest update: cscs-launch-tutorial returns once the job and all three services are ready; no tail or Ctrl-C step is involved.

brycelelbach · 2026-07-23T11:53:22Z

+```bash
+export CSCS_USER="YOUR_CSCS_USERNAME"
+export NODE="nidXXXXXX"
+
+ssh -N \
+  -o ExitOnForwardFailure=yes \
+  -o ServerAliveInterval=30 -o ServerAliveCountMax=3 \
+  -J "${CSCS_USER}@ela.cscs.ch,${CSCS_USER}@daint.alps.cscs.ch" \
+  -L 127.0.0.1:8888:127.0.0.1:8888 \
+  -L 127.0.0.1:8080:127.0.0.1:8080 \
+  -L 127.0.0.1:3478:127.0.0.1:3478 \
+  -L 127.0.0.1:8081:127.0.0.1:8081 \
+  -L 127.0.0.1:3479:127.0.0.1:3479 \
+  "${CSCS_USER}@${NODE}"


We should have three scripts total instead of embedded instructions here.

One script that is run on the Daint headnode that launches the job, waits for it start, and reports the node ID.

One script that is run from the workstation/browser with the browser that connects to the node with port forwarding and leaves the user in a shell on the node.

One end-to-end script that you run from workstation/browser that SSHs to Daint (should be resilient to authentication/2FA issues), invokes script 1, and then invokes script 2 locally.

Addressed with the three requested scripts: launch-cscs-web.bash launches/waits/reports on Daint, connect-cscs-web.bash opens the five forwards and an interactive compute-node shell, and run-cscs-web.bash performs the end-to-end workstation flow. The last opens one Daint ControlMaster before submission, retries once via cscs-key sign after public-key rejection, reuses that connection, and then invokes the connector locally.

Final names in 3085f6b are cscs-launch-tutorial, cscs-connect-tutorial, and cscs-run-tutorial. The end-to-end helper uploads/invokes the launcher, then runs the connector through the same authenticated Daint control connection.

brycelelbach · 2026-07-23T11:55:03Z

 The generated EDF sets `PMIX_MCA_gds=hash` and `PMIX_MCA_psec=native`, which
 are the PMIx settings used for clean Slurm MPI runs on Daint.

+## Run JupyterLab and Nsight Streamer for 10 hours


The instructions should include instructions on how to setup SSH access to Daint from scratch on the user's workstation/laptop with the browser

Added from-scratch workstation setup to docs/cscs.md: account/MFA prerequisites, cscs-key installation, key creation and signing, ssh-agent, ela/daint/nid* SSH configuration, file permissions, and an ssh daint hostname end-to-end test, with links to the official CSCS instructions.

The final setup is now shorter and alias-free: docs/cscs.md has an exact pinned cscs-key install command plus key creation/signing. The helpers accept --user and route through ela.cscs.ch to daint.alps.cscs.ch directly, so no user SSH config is required.

robobryce · 2026-07-23T13:29:42Z

Live Daint validation completed for 4f0cdda.

Ran the end-to-end workstation helper against job 4265767 on nid006544 (GH200), using the published ARM64 images only; no image was built on Daint.
JupyterLab, Nsight Systems Streamer, and Nsight Compute Streamer all returned HTTPS 200 through the five SSH forwards (8889 was used locally because this workstation already owns 8888).
Both Streamers were forced to TURN relay mode and connected over TURN/TCP through SSH. Video frames advanced and both input data channels opened and sent messages.
A file written through the Jupyter contents API appeared in the existing dirty student checkout's tutorials/pyhpc/notebooks tree, not the managed runtime checkout. The test marker was removed afterward.
Exiting and re-running the standalone connection helper restored all three services while the Slurm job continued.
The documented full-job TERM removed job-local TLS material and the allocation disappeared from squeue.

The repeated early Podman termination was traced with strace to Daint's short-lived per-user systemd manager. The final fix keeps rootless Podman in the Slurm job cgroup by unsetting the inherited user-bus address before any Podman operation. Local Bash syntax, ShellCheck, Compose transformation, port validation, and mocked three-script/authentication flows also pass.

brycelelbach · 2026-07-24T08:53:03Z

+CSCS does not support username/password SSH. If the certificate expires, run
+`cscs-key sign` again; the private key does not need to be regenerated. See the
+official [CSCS SSH instructions](https://docs.cscs.ch/access/ssh/) for account,
+MFA, key-signing, and platform-specific installation details.


This is overengineered and too wordy. Make it simpler and more concise.

Simplified in c236d3a. The setup is now just the exact cscs-key install command, key creation/signing, and the one launch command; the ssh-agent and user SSH-config instructions are gone.

brycelelbach · 2026-07-24T08:54:30Z

+
+1. Ensure the CSCS account belongs to a project with Daint access and has
+   [multi-factor authentication](https://docs.cscs.ch/access/mfa/) configured.
+2. Install [`cscs-key`](https://docs.cscs.ch/access/ssh/#command-line-access).


Provide an exact curl command for this.

Added in c236d3a: the docs now install the pinned cscs-key v1.1.0 Linux binary with an exact curl-to-tar command into ${HOME}/.local/bin (plus the Homebrew command for macOS).

brycelelbach · 2026-07-24T08:55:24Z

+   Host ela
+       HostName ela.cscs.ch
+       User YOUR_CSCS_USERNAME
+       IdentityFile ~/.ssh/cscs-key
+       IdentitiesOnly yes


Why do I need this one?

Removed in c236d3a. Users no longer need an Ela alias (or any SSH alias); the helpers construct the Ela-to-Daint route themselves using the real hostnames.

brycelelbach · 2026-07-24T08:55:31Z

+   Host nid*
+       User YOUR_CSCS_USERNAME
+       IdentityFile ~/.ssh/cscs-key
+       IdentitiesOnly yes


What's this for?

Removed in c236d3a. The workstation SSH config no longer needs a nid* block; cscs-connect-tutorial supplies the compute-node user and identity directly.

brycelelbach · 2026-07-24T09:18:56Z

+used for student work; this lets the launcher preserve an older or modified
+student checkout without running stale infrastructure from it.
+
+Download the three web helpers from the event branch onto the workstation:


This should be ONE curl | bash that you run on the workstation.

Changed in c236d3a to one curl | bash command. The streamed cscs-run-tutorial clones or fast-forwards the event branch under ${HOME}/accelerated-computing-hub, then launches and connects.

Follow-up in 8445075: the final command is a no-argument curl | bash using cscs-run-tutorial.bash. It reads the username from the signed SSH certificate and lets Slurm select the default account. The NVIDIA event-branch URL will resolve once this PR is merged; the pushed fork/head raw URL currently returns 200.

brycelelbach · 2026-07-24T09:20:26Z

+mkdir -p "${HOME}/ach-cscs-web"
+cd "${HOME}/ach-cscs-web"


This is a bad directory name. It should just be ${HOME}/accelerated-computing-hub

Changed in c236d3a. The workstation checkout is now exactly ${HOME}/accelerated-computing-hub.

brycelelbach · 2026-07-24T09:23:33Z

+This is the recommended path from the workstation:
+
+```bash
+./run-cscs-web.bash --account YOUR_CSCS_ACCOUNT


This script should not assume that the user has an SSH alias called daint, but it can assume that the user has already setup keys.

Fixed in c236d3a. The scripts do not read or require a daint alias. They accept --user, use the configured key (default ${HOME}/.ssh/cscs-key), and build a private Ela-to-Daint route with ela.cscs.ch and daint.alps.cscs.ch.

brycelelbach · 2026-07-24T09:24:36Z

+cd "${HOME}/ach-cscs-web"
+BASE_URL="https://raw.githubusercontent.com/NVIDIA/accelerated-computing-hub/event/2026-07-cscs-summer-school/tutorials/pyhpc/brev"
+for SCRIPT in launch-cscs-web.bash connect-cscs-web.bash run-cscs-web.bash; do
+  curl --fail --location --remote-name "${BASE_URL}/${SCRIPT}"


I don't like these script names. It should be cscs-launch-tutorial, cscs-connect-tutorial, cscs-run-tutorial.

Renamed exactly as requested in c236d3a: cscs-launch-tutorial, cscs-connect-tutorial, and cscs-run-tutorial. All documentation and inter-script references use the new names.

Follow-up in 8445075: per the latest request, all three filenames now include the suffix: cscs-launch-tutorial.bash, cscs-connect-tutorial.bash, and cscs-run-tutorial.bash. Bootstrap, upload, sibling lookup, usage, and docs all match.

Address CSCS review feedback and add web deployment

90c688d

brycelelbach reviewed Jul 23, 2026

View reviewed changes

brycelelbach added 2 commits July 23, 2026 12:20

Add scripted CSCS web deployment workflow

c2a19ed

Harden CSCS web deployment on Daint

4f0cdda

brycelelbach reviewed Jul 24, 2026

View reviewed changes

brycelelbach added 3 commits July 24, 2026 09:35

Simplify CSCS tutorial bootstrap

c236d3a

Clarify CSCS event branch default

3085f6b

Default CSCS identity and suffix helpers

8445075

brycelelbach merged commit 5c02432 into NVIDIA:event/2026-07-cscs-summer-school Jul 24, 2026

Uh oh!

Conversation

robobryce commented Jul 22, 2026

Summary

Live Daint validation

Validation

Uh oh!

copy-pr-bot Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robobryce commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!