Add Dask Gateway support to lc run (JupyterHub deployments)#159
Draft
EiffL wants to merge 4 commits into
Draft
Conversation
cluster_for_run gains a fourth branch, selected when a Gateway
environment is ambient (DASK_GATEWAY__ADDRESS, as on a lightcone
JupyterHub pod) or a cluster is named via LIGHTCONE_GATEWAY_CLUSTER.
Gateway scheduler addresses use a gateway:// comm scheme that a bare
distributed.Client cannot dial, so the parent/child rendezvous changes
shape: cluster_for_run now yields the env overlay the child snakemake
needs ({DASK_SCHEDULER_ADDRESS: ...} on existing branches,
{LIGHTCONE_GATEWAY_CLUSTER: name} on the new one), and the executor
rejoins the cluster by name through the authenticated Gateway API —
the same rendezvous-by-ambient-context pattern as the SLURM branch.
Ownership mirrors the address branch: clusters we create are adaptive
(minimum=1, bounded by --jobs) and shut down on exit; clusters named
by the user (e.g. created from the JupyterLab Dask sidebar) are left
running with their scaling untouched. Startup fails fast when gateway
workers don't advertise the cpus/memory/gpus resource contract, which
otherwise hangs every task silently.
Also: a lightcone-hub site entry detected via env markers (pods have
generated hostnames, so detect_current_site now checks declared env
markers before hostname patterns), and a [gateway] optional extra for
the dask-gateway client.
Validated against the lightcone-hub GKE staging deployment; design
notes and the deployment procedure live in
docs/design/jupyterhub-dask-gateway-gcp.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
lightcone-cli | 4b39e63 | Commit Preview URL Branch Preview URL |
Jul 14 2026, 05:17 AM |
✅ Eval Results
Graders✅ spec_valid (1.00) Full output |
Reworks the Dask Gateway integration around one idea: on a hub deployment, the worker pod IS the container. The Containerfile keeps defining the recipe environment on every path; only its realization differs (OCI wrap locally/SLURM, pod image on the hub). - container.py: new "kubernetes" runtime — wrap_recipe no-ops, a site-declared non-OCI runtime resolves as explicit in load_runtime (kills the false "no container runtime found" warning on the hub; Perlmutter's OCI hint semantics unchanged). LIGHTCONE_REGISTRY env names the deployment registry; registry_image_ref maps the local lc-<project>-<hash> tag to <registry>/lc-<project>:<hash> (same content hash — same artifact). resolve_image_for_run stays site-agnostic so code_version is stable across laptop<->hub. Best-effort registry probe via the GKE metadata token. - dask_cluster.py: the Gateway branch is attach-only. lc run never creates clusters; it discovers the user's running cluster through the user-scoped Gateway API (one -> attach; zero -> error with a copy-pasteable new_cluster snippet, image pre-filled and shutdown_on_close=False; several -> LIGHTCONE_GATEWAY_CLUSTER disambiguates). Scaling untouched, cluster left running — the Gateway flavor of the DASK_SCHEDULER_ADDRESS convention. Verifies the resource contract on live workers (now cpus+memory together; cpus-only would still hang mem_mb rules) and the worker image via the deployment-injected LIGHTCONE_WORKER_IMAGE, warning on mismatch. Gateway()/connect failures translate to guided errors. - executor.py: _connect_client now mirrors the parent's branch priority (address first), and lc run strips the losing variable from the child env — a stale shell-exported gateway name can no longer redirect the child to an unverified cluster. - commands.py: lc build on kubernetes checks the registry and prints publish instructions that reproduce the hash-attested build context (lc build + docker tag/push, not a raw docker build). lc run cross-checks runtime vs cluster branch and warns on both mismatch directions. Manifests record the pod's actual image (worker_image). - tests: attach-only gateway suite (discovery, lifecycle, contract, image verification, stale-name guidance), registry helpers, runtime precedence; conftest neutralizes ambient hub env markers so the suite passes inside a hub pod. Validated end-to-end under a simulated hub environment; reviewed by a 25-finding multi-agent workflow with adversarial verification, all confirmed findings fixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRVPnWVUtpumnFsCzEEA8U
Validated live on the lightcone-hub GKE deployment; a two-rule pipeline now runs end to end on a Gateway cluster (attach, contract+image verification, DAG on spot workers, manifests with worker_image ground truth, lc verify clean, idempotent re-run). Three real-world fixes, all the same root cause — Gateway worker pods share the project volume with the driver but not its HOME, install prefix, or cwd, unlike local/SLURM workers which inherit the driver's environment by process ancestry: - Executor overrides get_python_executable(): the driver's sys.executable (conda notebook path) doesn't exist in a slim worker image — exit 127 before snakemake starts. The worker's own PATH resolves the interpreter; the worker image is the software deployment. - Executor pins --directory on the child snakemake: worker pods start in their own HOME, so relative outputs "succeeded" into the pod's ephemeral filesystem while the driver saw nothing. - lc run passes --shared-fs-usage input-output/persistence/sources/ storage-local-copies (not source-cache: the child mkdir'd the driver's ~/.cache path and died on PermissionError) and --latency-wait 60 (NFS attribute caching hid freshly written worker outputs from the driver past snakemake's 5s default) — gateway branch only; local/SLURM keep snakemake defaults. Also: _run_shell forwards a bounded stderr tail when the child fails (exit 127 with zero diagnostics is a debugging dead end), and gateway_branch_active() exposes the branch decision so lc run can shape the snakemake invocation before entering the cluster context. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRVPnWVUtpumnFsCzEEA8U
Found live on the hub: an ASTRA analysis named "Union2.1 Flat ΛCDM MAP Fit" produced the image name lc-union2.1-flat-λcdm-map-fit — not a valid OCI reference (registries and docker/podman require lowercase ASCII), and the λ reached http.client's ASCII-only request line, so `lc build` crashed with UnicodeEncodeError instead of reporting. - compute_image_tag routes through image_name_slug(): NFKD transliteration for accented latin, other non-ASCII dropped, invalid chars collapsed to "-". Already-valid names (gwtest, union2.1) pass through unchanged, so existing tags keep their names. - registry_image_exists percent-encodes the URL path and treats ValueError/UnicodeError as "cannot tell" (None) — the probe's contract is tri-state, never a traceback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HRVPnWVUtpumnFsCzEEA8U
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lc runnow works on Dask Gateway deployments (our JupyterHub-on-GKE playground) with zero user configuration: on a hub pod,cd /shared/<project> && lc runrequests an adaptive Gateway cluster, runs the pipeline on the spot pool, and shuts the cluster down afterwards. SettingLIGHTCONE_GATEWAY_CLUSTER=<name>instead attaches to an existing cluster (e.g. one created from the JupyterLab Dask sidebar, dashboard panels docked) and leaves it running on exit.Why the rendezvous changes shape
Gateway scheduler addresses are not dialable: they use a custom
gateway://comm scheme (traefik multiplexing + per-cluster TLS + SNI) that only thedask-gatewayclient library understands — verified on the staging cluster (unknown address scheme 'gateway'from a bareClient). Sincelc runand the executor plugin live in different processes, the fix is the same rendezvous-by-ambient-context pattern the SLURM branch already uses: the parent passes the cluster name through the env, and the executor rejoins by name through the authenticated Gateway API.Concretely,
cluster_for_runnow yields the env overlay the child snakemake needs ({"DASK_SCHEDULER_ADDRESS": ...}on existing branches,{"LIGHTCONE_GATEWAY_CLUSTER": name}on the new one) instead of a bare address string — making the actual parent→child contract explicit.Changes
engine/dask_cluster.py— fourth branch (priority: explicit address → gateway → SLURM → local): creates an adaptive cluster (minimum=1, bounded by--jobs) or attaches by name; owned clusters are shut down on exit, attached ones untouched — mirroring the address-branch convention. Fails fast when workers don't advertise thecpus/memory/gpusresource contract (otherwise every task hangs with no error).snakemake_executor_plugin_dask/executor.py— client resolution extracted to_connect_client(): Gateway name → rejoin via Gateway API; elseClient(DASK_SCHEDULER_ADDRESS)as before.engine/site_registry.py—lightcone-hubsite entry detected via env markers (JUPYTERHUB_USER+DASK_GATEWAY__ADDRESS; pods have generated hostnames), checked ahead of hostname patterns indetect_current_site.pyproject.toml— optional extralightcone-cli[gateway]; mypy override fordask_gateway.docs/design/jupyterhub-dask-gateway-gcp.md.Testing
/sharedRWX + flock, hub-proxied gateway) was validated end to end on thelightcone-hubGKE staging cluster. End-to-endlc runon the hub additionally needs the worker image (lc+snakemakebaked in) — tracked in lightcone-hub.🤖 Generated with Claude Code