Snakemake orchestration for the real-data pipeline (design for discussion)

[cailmdaley]

Snakemake orchestration for the real-data pipeline

Summary

Replace the shell-script orchestration of real-data ShapePipe runs (curl_canfar_local.sh → run_job_sp_canfar_v2.0.bash → job_sp_canfar_v2.0.bash, plus per-site sbatch reimplementations like the nibi P3 scripts) with a Snakemake workflow living in this repo. First pass touches no module code: rules call shapepipe_run -c <config> on the existing config chains, at the same granularities the bash layer uses today. What changes is who tracks state: instead of hand-written file-count thresholds, done-log files, and flock-guarded dedup, Snakemake's file-based DAG does the bookkeeping — with science products as targets, a completeness count-floor as the failure policy, and one execution profile per machine.

Round history: round 1 posted as the original body of this issue (see edit history) → four-lens adversarial review (36 findings) → Cail's simplification feedback plus a Snakemake-v9 / nibi-networking research pass → a second four-lens adversarial review of the revised design (30 raw findings → 22 verified, 4 blockers) → this revision, which folds all 22. The through-line of Cail's feedback is simplify, and most of that simplification dissolves review findings rather than trading them away: dropping the failure-classifier removes the "misclassification swallows real bugs" hole; using the full default rerun-triggers removes the "code/config fixes don't propagate" contradiction; collapsing to one execution mode removes the multi-instance race. The second review then exposed a harder class of bug at the seam between Snakemake's state model and ShapePipe's — those fixes are folded below as design truth. This is a living PRD, kept updated as implementation proceeds; the eventual PR just says "implements #848." Please keep poking holes.

Motivation, concretely:

• The v2.0 bash layer works but re-implements a build system: input discovery, completeness checks (ls | wc -l vs thresholds), done-markers (logs/log_job_<ID>_<bit>.txt), cross-tile dedup (flock on .sp_lock), retry/force semantics. Every one maps to a Snakemake primitive — and the bash versions live in three layers that each site then re-wraps (job_p3_batch1.sh on nibi is a fourth, parallel reimplementation).
• The im_sims Snakefile (Image simulations workflow #766 lineage) proved the approach but kept the bash layer underneath and used log files as targets — completion ≠ product existence. This design inverts that.
• Once the DAG is explicit, the questions we actually care about — how many jobs per node given memory, why did this tile fail, what is safe to rerun — become answerable instead of archaeological.

What the bash layer does today (what Snakemake must absorb)

Level 3 (curl_canfar_local.sh): fan out one job per tile, throttled batches, CANFAR/skaha REST.
Level 2 (run_job_sp_canfar_v2.0.bash): per-tile driver — work-dir forest (tiles/<IDra>/<ID>, exp/<2-digit-prefix>/<expbase>), loops exposures from exp_numbers-*.txt, per-exposure flock dedup, per-runner file-count completeness checks (with :warn/:subpath classes), done-logs, --force/--retry.
Level 1 (job_sp_canfar_v2.0.bash): job-bit → config → shapepipe_run, per-unit work dirs, init_exp_work_dir unit isolation. Unit isolation is per-unit work-dir content, not NUMBER_LIST — set_config_number_list exists (job_sp_canfar_v2.0.bash:255) but is never exercised in the forest flow: both call sites omit -e/--exclusive. A unit is isolated by its own work dir, its cfis symlink, and its own list files (a tile's tile_numbers.txt; an exposure's fabricated single-entry exp_numbers-000-000.txt). This is the mechanism the wrapper must reproduce (finding 9).

The pipeline being orchestrated:

flowchart TD
    subgraph TP["tile-level, pre"]
        Git["get_images (download)"] --> Uz[uncompress weight] --> Fe[find_exposures]
    end
    subgraph EX["exposure-level (modules parallelize over 40 CCDs internally)"]
        Gie[get_images exp] --> Sp[split into CCDs] --> Ma[mask] --> Psf["SExtractor → setools → PSFEx → PSF interp"]
    end
    subgraph TQ["tile-level, post"]
        Mh[merge_headers] --> Mat[mask tile] --> Sxt[SExtractor tile] --> Vi["PSF interp + vignets"] --> Ng["ngmix × N chunks"] --> Ms[merge_sep_cats] --> Mc["make_cat → final_cat"]
    end
    Fe -->|"exposure list (tile FITS HISTORY)"| Gie
    Psf -->|"PSF models, stamps"| Vi
    Sp -->|headers| Mh

Loading

Key structural facts:

• The tile→exposure edge is data-derived: which exposures a tile needs is known only after the tile image is downloaded and its HISTORY header read (find_exposures.py reads hdu[0].header["HISTORY"], emits exp_numbers-<IDra>-<IDdec>.txt). Exposures overlap tiles (~7–10 per tile) and must be processed exactly once. In a contiguous survey the exposure-overlap graph is one giant connected component — exposure-disjoint tile batching is geometrically impossible (finding 6), which is why the shared-store and cleanup story below matters.
• ShapePipe modules already parallelize within a shapepipe_run call (SMP over CCDs/objects), so orchestration jobs stay at tile/exposure granularity — never per-CCD.
• Between modules, ShapePipe resolves inputs through its own run-log (log_run_sp.txt, the last:/run_sp_<prefix>: INPUT_DIR mechanism). This resolver wiring is load-bearing and stays. What we disable per-unit is the timestamp: the wrapper writes RUN_DATETIME=False into every per-unit config copy, so each stage lands at a deterministic output/run_sp_<prefix>/ and the resolver never sees ambiguity (finding 1/12/21).

Design

D1. Staged execution: prepare, then run

A run is declared by a tile list. Execution is honestly three static steps (findings 7/17: the exposure job set is data-derived from the tiles' find_exposures output, so it cannot be scheduled in the same static DAG that produces it), chained by the sp driver so the UX stays one command:

snakemake prepare_tiles          # static per-tile DAG: Git → Uz → Fe, --keep-going
python workflow/scripts/build_index.py   # PLAIN SCRIPT, not a DAG node
snakemake prepare_exposures      # static per-exposure DAG: Gie + per-unit star cats
snakemake                        # phase B — the main compute DAG, parsed from the index

The index step exists for exactly one reason: run_index.sqlite must exist on disk before the later invocations parse it — the Snakefile loads the index once at parse time into plain dicts, and every input function is a dict lookup (no per-job DB access, no checkpoint, no directory globbing at scale). build_index.py iterates the declared tile list, checks each tile's Fe output at its deterministic path (O(tile-list) existence checks — no ls at scale), and writes tiles/exposures/tile_exposures plus index/missing.json.

The index build is threshold-gated, not all-or-nothing: a tile whose download persistently failed (retries + the FITS-integrity check below make that rare) simply lands in missing.json and its downstream is never scheduled; build_index.py exits nonzero only if the missing fraction exceeds a configurable floor. A bad tile costs that tile, not the run — one dead CADC file must never block a 7000-tile campaign. (There is no --allow-missing flag — that Snakemake option does not exist; tile list + --keep-going + the threshold + missing.json is the whole mechanism.)

This structure is implemented and green on nibi: a probe job ran shapepipe_run in-container through Snakemake and the prepare phase came up clean on a 4-tile subset — deterministic run_sp_tile_{Git,Uz,Fe} dirs on disk, run_index.sqlite built, 4 tiles / 30 exposures indexed.

The index is parse-time data, never a rule input: in phase B. Extending scope (append tiles, rebuild index) changes which jobs exist but does not touch the mtime chain of completed work — so 50 finished tiles stay finished when tile 51 arrives. This invariant is only true because the shared exposure store is persistent, not temp() (D2, finding 4): were the shared exposure intermediates temp(), tile 51 would force regeneration of border exposures, marking its finished spatial neighbours stale and wiping their RNG-bearing products. The invariant and temp()-on-shared-exposures are mutually exclusive; we keep the invariant.

Index schema:

tiles(tile_id, ra_dir, status)
exposures(exp_id, n_hdu)               -- deduplicated; live-CCD count from the actual file
tile_exposures(tile_id, exp_id)

Exposure dedup is structural: exposure-level rules are keyed by exp_id alone, so every tile wanting exposure E requests the same output paths and Snakemake schedules E once within an invocation. Cross-invocation sharing over a common store is a contract, not a free ride (D6, finding 16).

Downloads run in-DAG on compute nodes. Round 1 assumed nibi compute nodes have no outbound internet; that was wrong — a compute node (c4.nibi.sharcnet) empirically reached Google (HTTP/2 200) and resolved CADC DNS, and Alliance docs confirm "all nodes on Nibi have internet access." So downloads sit inside the DAG: tile get_images in phase A (prepare, feeding find_exposures), exposure get_images and star-cat retrieval in phase B (the main run, keyed by the index). Each download rule carries retries: (endemic vcp flakiness) and a FITS-integrity verification step (open with astropy, check expected HDU count / NAXIS) so an exit-0-but-truncated transfer is a retryable failure, never a silent input — vcp's exit code alone is not trusted.

D2. Rules = existing config files; work-dir forest preserved

Rule granularity follows the existing job-bit boundaries — each rule wraps one shapepipe_run -c <config> on one unit (tile or exposure), in the same per-unit work-dir forest as v2.0. Because the wrapper writes RUN_DATETIME=False into each unit's config copy, every stage's output dir is a deterministic literal output/run_sp_<prefix>/ (prefixes distinct per stage, each stage runs once per unit tree → no collisions), declared as the rule's directory() output. Snakemake's delete-before-rerun then removes the real products, log entries stay stable, and last:/run_sp_<prefix>: resolution is unambiguous (get_run's >1-match RuntimeError can never fire). No rule declares a glob — there are no glob outputs anywhere in the DAG. The rows marked (phase A) build during snakemake prepare; everything else runs in phase B (the main invocation), exposure downloads and star cats included.

rule	unit	config (example/cfis)	declared output path
tile_get_images+tile_find_exposures (prepare_tiles)	tile	config_tile_Git_*, Uz, Fe	.../run_sp_tile_Git/, run_sp_tile_Uz/, run_sp_tile_Fe/find_exposures_runner/output/exp_numbers-<IDra>-<IDdec>.txt
(index)	run	build_index.py — plain script between the two prepare targets, threshold-gated	index/run_index.sqlite, index/missing.json
exp_get_images (prepare_exposures)	exposure	config_exp_Gie_*	.../run_sp_exp_Gie/
star_cat_exp (prepare_exposures)	exposure	per-unit link from the pre-generated store	star_cat-<exp>.fits (+ per-CCD symlinks)
star_cat_tile (prepare_exposures)	tile	per-unit link from the pre-generated store	star_cat-<IDra>-<IDdec>.fits
exp_split	exposure	config_exp_Sp.ini	.../run_sp_exp_Sp/ (headers-<exp>.npy + CCD files, ensure(non_empty=True))
exp_mask	exposure	config_exp_Ma_onthefly.ini	.../run_sp_exp_Ma/ (per-CCD masks)
exp_psf	exposure	config_exp_psfex.ini	.../run_sp_exp_SxSePsfPi/ (per-CCD PSF models)
tile_exp_forest	tile	(symlink curation)	per-tile $SP_EXP view — runtime convenience, not a DAG edge
tile_merge_headers	tile	config_tile_Mh_exp.ini	.../run_sp_tile_Mh/merge_headers_runner/output/log_exp_headers-<IDra>-<IDdec>.sqlite
tile_detect	tile	uc / sx_nomask / sx+onthefly-mask (below)	.../run_sp_tile_<Uc|Sx>/ (tile sexcat)
tile_vignets	tile	config_tile_PiViVi_*.ini	.../run_sp_tile_PiViVi/ (vignet files)
tile_ngmix	tile × chunk	generated config_tile_Ng<k>.ini	.../run_sp_tile_ngmix_Ng<k>u/ (chunk catalog)
tile_merge_cats	tile	config_merge_sep_cats.ini	.../run_sp_tile_msc/ (merged ngmix cat)
tile_make_cat	tile	config_tile_Mc_*.ini	final_cat-<IDra>-<IDdec>.fits

tile_merge_headers writes log_exp_headers-<IDra>-<IDdec>.sqlite — the tile suffix comes from the exp_numbers filename (merge_headers.py:47), and downstream Sx/PiViVi/Ng consume it via NUMBERING_SCHEME -000-000, so the declared output must carry the tile suffix (finding 12).

tile_detect (job bit 256) has three selectable variants, all representable: uc (external UNIONS catalogue → read_ext_sexcat_runner, config_tile_Git_cat_vos.ini+config_tile_Uc.ini — v2.0's data default), sx_nomask (config_tile_Sx_nomask.ini alone), and sx+onthefly-mask (config_tile_Ma_onthefly.ini+config_tile_Sx.ini). DR6 will standardise on one; the config selector carries the choice.

The data/image_sims axis (example/cfis vs example/cfis_image_sims, retrieve=vos vs symlink) is kept, but sims parity is a named follow-up, not a free ride — the sims chain diverges structurally (fake-Uz symlinking, fake_psf folded into vignets, different thresholds, forced tile_det=sx). First pass targets data.

Star cats are re-keyed per-unit. create_star_cat.py already emits star_cat-<exp>.fits (plus 40 per-CCD symlinks) and star_cat-<IDra>-<IDdec>.fits, so we declare per-exposure and per-tile star-cat outputs. Tile 51's cats become new DAG nodes rather than a mutation of an input to 50 finished units — preserving the growth invariant — and exp_mask/tile_detect declare their own unit's cat as a proper input: (finding 11a). A single run-level star_cats rule would dirty every finished mask job on append; that is out. First pass, the per-unit rules link cats from a pre-generated store (gen_star_cats.sh on a networked login node, the existing mechanism); folding create_star_cat.py itself into the DAG is a named follow-up.

Why the directory is the contract for per-CCD stages, and why that's the right primitive. Snakemake's unit of success is the job: one 40-CCD exp_psf invocation is a single exit code with all-or-nothing declared outputs, and Snakemake has no optional-output primitive (feature request #4016 open). Declaring per-CCD files is doubly impossible: at DR6 (~20k exposures) it is ~2.4M declared paths for split alone (~8M across mask+psf) — the DAG never builds against that many Lustre stats — and any missing CCD becomes a hard failure poisoning the whole downstream cone. Exploding to ~800k per-CCD jobs hits documented DAG-scale walls (#292/#3003). So the per-CCD stages (exp_mask, exp_psf, exp_split) declare their directory() output only; the count-floor (D3) counts the products in that directory at job time against the committed threshold table (workflow/scripts/completeness.py, stdlib-only so wrapper and report import the same table in-container) and sets the exit code — no per-unit manifest is written (delete-before-rerun on the declared directory is the whole state contract). It is exactly the primitive the flagship pipelines converged on — DES's per-CCD blacklist flagged on "too few stars" (our ~2% case), LSST/Rubin's quantum registry with pipetask report (which defaults to not fail-fast).

The wrapper (workflow/scripts/sp_rule.py). Each rule runs through a thin wrapper that:

1. materialises the unit (init_exp_work_dir parity, the real v2.0 isolation): the cfis config symlink; the per-unit list file (a tile's tile_numbers.txt in dot format; an exposure's fabricated single-entry exp_numbers-000-000.txt under output/run_sp_tile_Fe/find_exposures_runner/output/, so last:find_exposures_runner resolves to this exposure); and star_cat_exp/star_cat_tiles symlinks (dir-level links to the shared cats — the file handler intersects on NUMBERING_SCHEME, so extra files are harmless) in every unit work dir, because configs resolve $SP_RUN/star_cat_exp / $SP_RUN/star_cat_tiles inside each unit dir and mask stages fail with "No files found matching star_cat" without them (finding 11b). NUMBER_LIST is not the isolation path and is injected only for stages whose NUMBERING_SCHEME is the unit ID itself (tile-scheme stages Uz/Fe/Mh/Ma/Sx/PiViVi/Ng/merge/make_cat, exp_Sp); it is never applied to tile_get_images, exp_get_images, exp_mask, or exp_psf — for those, a -<ID> entry fails the [HELP] #746 startup validation against the per-CCD numbering scheme, and a 40-entry expansion would convert tolerated per-CCD attrition into whole-exposure hard failure (findings 9/10). The per-unit work dir already restricts scanning to this unit's files, reproducing the monolith's graceful per-CCD degradation exactly.
2. writes the per-unit config copy with RUN_DATETIME=False (deterministic output dir) and SMP_BATCH_SIZE={threads} (D4 fork-width invariant).
3. runs update_runs_log_file.py at job START — after Snakemake's pre-delete — so log_run_sp.txt (regenerated from the actual directory tree) never references a just-deleted run dir, then runs shapepipe_run. Concurrent tile_ngmix chunk jobs are exempt from this step (finding 18) — see D4.
4. does the completeness count-floor check (D3) → sets the exit code;
5. rewrites declared outputs only when content changed, so mtimes move only when reality moves (no per-unit manifest — the count-floor's verdict is the exit code), plus a cheap wrong-unit guard: output filenames must contain the unit's ID (finding 10);
6. tees module stdout/stderr to a per-unit log declared via log:.

State ownership (finding 21). Two state trackers meet here, and the design defines which wins: the directory tree is the source of truth; log_run_sp.txt is a derived cache; Snakemake owns existence. RUN_DATETIME=False makes each stage's dir a literal path Snakemake can delete-before-rerun; update_runs_log_file.py at job start rebuilds the log from disk so it can never point at a stale run. There is no pre-clean step and no rename-to-stable-name — both were failure modes (the rename left the timestamped dir behind for the resolver to resurrect). The wrapper is also the hand-run escape hatch (sp_rule.py tile_detect 123.456 reproduces the rule exactly); when hand-run it first deletes the target run dir if present (FileHandler.mkdir raises on an existing dir), reproducing Snakemake's delete-before-rerun so manual surgery stays visible to Snakemake afterwards.

tile_exp_forest is convenience, never the DAG edge (findings 5/19). Today the tile stages find exposure products by globbing $SP_EXP for the newest run dir (get_exp_output_files takes matches[-1]). Instead a per-tile rule builds a curated symlink view of exactly the index-listed exposures' verified products and points $SP_EXP at it — symlinking by exact run_sp_exp_<prefix> name (RUN_DATETIME=False guarantees exactly one). But the forest is not a sufficient input for any rule that reads through it: Snakemake has no notion of reachability through a symlink, so if the forest were the only declared consumer, temp() refcounting would delete the exposure dirs the instant the forest is built. Therefore tile_merge_headers, tile_vignets, and PSF-interp declare the real directory() outputs of their index-listed exposures (run_sp_exp_Sp, run_sp_exp_Ma, run_sp_exp_SxSePsfPi) as input: via an index-driven dict lookup. The last real reader — not tile_exp_forest — is the last DAG consumer.

Cleanup — persistent store, explicit verb (findings 4/5/6/19). Isn't there a temp() flag for exactly this? Yes — and temp() is the right tool for strictly-private intermediates. But it is per-invocation, and exposures are shared across tiles and batches: marking a shared exposure temp() combines with append-driven index growth, mtime propagation, and delete-before-rerun to destroy the "finished tiles stay finished" invariant and silently recompute RNG-bearing products under a later append or batch. So shared exposure intermediates are persistent, not temp(). Space is reclaimed by an explicit index-driven cleanup verb (sp clean-exposures) that deletes an exposure's intermediates only when run_index.sqlite's tile_exposures says every consuming tile has final_cat present across the whole output root — the index already carries exactly those edges, and cleanup can run periodically so peak storage stays bounded (accepting that later batches sharing a cleaned exposure re-trigger its regeneration). temp() may remain only on genuinely tile-private intermediates that cannot cross-couple units.

D3. Failure handling: the count-floor is the policy

Round 1 had a three-class taxonomy plus a failure_classes.yaml whitelist plus a verdict-classifier. That was over-engineering beyond what the bash layer does, and it manufactured exactly the opacity Martin fears (a misclassifier writing green over a real bug). All of it is dropped. The policy is the count table the bash complete_check already encodes: the wrapper does a completeness count-floor check — the ported per-runner thresholds, with :warn and :subpath semantics — and exits 0 above the floor, nonzero below (zero output is always fatal). The thresholds are committed as the table in workflow/scripts/completeness.py (stdlib-only, imported by both the wrapper and the report); the table below is its human rendering.

stage	check
tile get_images	get_images_runner:4 (2 for sims)
tile uncompress	uncompress_fits_runner:1
tile find_exposures	find_exposures_runner:1
exp get_images	get_images_runner:6 (3 sims)
exp split	split_exp_runner:121 (40 CCD × 3 + 1 header)
exp mask	mask_runner:40
exp psf	sextractor_runner:80 psfex_runner:80 psfex_interp_runner:40::warn setools_runner:80:rand_split
tile merge_headers	merge_headers_runner:1
tile detect (sx)	sextractor_runner:2
tile detect (uc)	get_images_runner:2 read_ext_sexcat_runner:1
tile vignets (data)	psfex_interp_runner:1 vignetmaker_runner_run_1:1 vignetmaker_runner_run_2:4 (:3 sims)
tile ngmix	ngmix_runner:1
tile make_cat	make_cat_runner:1

:warn means a short count does not fail the unit (expected sparse-CCD PSF loss — psfex_interp_runner:40::warn); :subpath (setools_runner:80:rand_split) counts a nested dir. So exp_psf tolerates <40 PSF-interp CCDs but still fails hard on 0-of-40, a short SExtractor/PSFEx count, or a nonzero shapepipe_run exit — the count table is the fail-closed contract, anything below the floor is a real failure with no reclassification. Tolerated-but-below-40 cases simply pass (exit 0); the standalone report (below) re-derives them by disk-scanning the run dirs against expected_counts.yaml.

Three mechanisms carry the three real situations, and only these:

• Transient (vcp hiccups, node flakiness, OOM): per-rule retries: with attempt-scaled resources (mem_mb = lambda wc, attempt: base * attempt). Downloads additionally get the integrity check (D1). A deterministic failure exhausts retries and fails.
• Hard (real bug, unrecoverable short count): the job exits nonzero; with keep-going: true everything independent continues and only the failed downstream cone waits.
• Below-floor-but-expected (too few stars on some CCDs): absorbed by :warn, exit 0 — surfaced by the standalone report's disk-scan, no separate class, no whitelist.

Rerun triggers — the full default. Use the full default triggers {mtime, params, input, software-env, code} (the v9 default since 7.8) and remove any override from the profile. A params change should re-run — params make a big scientific difference — and a fixed module/config propagates natively. The one honest cost: the first run after adopting this may rerun more than mtime-only would; the mitigation is in the wrapper (declared outputs rewritten only when content changed), so timestamped-run-dir churn is gone anyway (RUN_DATETIME=False) and mtime cascades stay real, never cosmetic.

One deliberate scoping of the code trigger: rules invoke the wrapper via shell: (python sp_rule.py <stage> <unit>), not Snakemake's script: directive. So code tracks each rule's shell string (config path + params — exactly what should re-trigger) while an edit to the shared sp_rule.py does not invalidate the entire DAG (as script: would, every rule routing through the one wrapper). Wrapper-logic changes are an explicit --forcerun; science-affecting changes still propagate — configs/params via the shell string, module code via the software-env trigger on the container image.

Debugging path. Every rule declares its per-unit log via log:; the profile sets show-failed-logs: true, so on failure the head log prints rule, wildcards, unit log path, and its tail. In one-allocation mode (D4) the OOM/walltime trace also lands in the allocation's single stdout log; the standalone report (below) links every failed unit to its log path.

The report is out of the DAG (finding 8). A run_report rule is a descendant of every tile, so under --keep-going any single hard failure (the very thing a report must enumerate) poisons its cone and the rule never runs — it would execute only on all-green runs, where it has nothing to say. So the report is a standalone verb (sp report): it reads run_index.sqlite and expected_counts.yaml, disk-scans the tiles/ and exp/ trees (the persistent exposure store makes re-counting cheap and always available), and enumerates failed cones — runnable at any time, including mid-run. onsuccess/onerror handlers in the Snakefile invoke the same script so a report lands automatically at the end of every invocation regardless of failures. The report distinguishes whole-exposure absence (all 40 CCDs missing) from tolerated per-CCD attrition, so no forest/deletion bug can be absorbed by the expected-by-data floor (finding 5).

D4. One execution mode: the one-big-allocation

Round 1 offered two modes. We keep only the one proven on nibi (job_p3_batch1.sh) and defer the slurm-executor:

# inside the sbatch allocation:
module load apptainer            # via the committed launcher (D5), not shell state
snakemake --profile profiles/nibi <targets>   # local scheduler bin-packs the whole allocation
# snakemake wraps each shapepipe_run in `apptainer exec project.sif` (--sdm apptainer)

sbatch an allocation; run snakemake with its local scheduler inside it. snakemake itself runs bare on the node — never inside the container. The orchestrator only needs snakemake + the stdlib (its parse-time job is loading run_index.sqlite; all heavy science lives in the container, reached per-job via --sdm apptainer, D5). snakemake's own environment lives on the shared FS (/project) because the local executor re-invokes its python inside jobs — a login-node-only venv would not be visible to the compute node. It is installed once as a small uv venv (or an Alliance module) and pinned in the workflow. The local executor bin-packs jobs against both declared cores/threads and the mem_mb cap — that cap is required: in one-allocation mode SLURM runs --mem=0 --exclusive and enforces nothing per-job, so without a total mem_mb resource the scheduler packs by cores alone, attempt-scaled OOM retries don't thin concurrency, and the Linux OOM killer arbitrates. The cap makes higher mem_mb actually reduce concurrency on retry. --latency-wait 60 handles Lustre metadata propagation (outputs of a just-finished job appearing to the next).

threads == SMP fork-width invariant (finding 13). A rule's threads/cpus_per_task must equal ShapePipe's internal SMP fork width, or the scheduler's packing density decouples from the true per-job process count and the node-wide nproc budget blows (the im_sims crash beyond ~4 jobs/node at ulimit -u 1200). Enforced mechanically: the wrapper writes SMP_BATCH_SIZE={threads} into the per-unit config, so cpus_per_task and fork width are one number by construction (the exclusive-node-tuned 96/48/32 in the stock configs must never reach a per-unit job). mem_mb = threads × measured per-worker footprint (nibi P3: ngmix ≈ 3–4 GB/worker after #843, PSF light, SExtractor mid). P0 measures once: ulimit -u, pids.max and cpuset/memory cgroup confinement on a compute node, plus one deliberate max-packing test before P2 scale.

Orchestrator hosting (finding 14). The login-node user slice is measured at 6 cores / 16 GiB / 512 tasks, shared with editors and sessions — too small to host a long-lived orchestrator at batch scale, and an OOM-killed or rebooted login-node snakemake orphans in-flight jobs that then double-write directory() outputs on --rerun-incomplete. So: run snakemake inside a long-walltime sbatch job (compute nodes have internet and can sbatch) or, minimally, under tmux with the caveat that reboots still orphan jobs. Pin local-cores: 4 in the profile (it otherwise defaults to the node's ~192 visible CPUs; under the local executor it caps localrule/aggregation concurrency). Add an sp cancel verb: in one-allocation mode there are no per-job SLURM jobs, so it cancels the allocation itself; the deferred slurm-executor variant is a scancel-by-run-UUID sweep (squeue --me --format=%j | grep <uuid> | xargs scancel). Either way the rule is the same and mandatory: never --unlock/--rerun-incomplete while any process of the dead run may still be writing.

Stated plainly: this mode is single-node. Multi-node scaling = multiple tile-disjoint snakemake invocations, each its own allocation — explicitly deferred. First pass is single-node subsets and P0–P2 scale. (The slurm-executor path — login-node snakemake submitting one sbatch per rule via the apptainer SDM — is not hypothetical: a probe job ran it end-to-end on nibi and completed. It stays available from the same Snakefile as a later scaling knob; deferred, not designed out.)

ngmix straggler fix (the 13-hour chunk on the 50-tile run), retained and sharpened:

• N chunks per tile is static (config), so the DAG shape is fixed and N_SPLIT_MAX == chunk count in tile_merge_cats (empty/short chunks tolerated by the merge).
• The tile sexcat is a declared input: of each chunk job, and the wrapper computes that chunk's ID range at execution time from the tile's own object count — equal shares, closed ranges, no open-ended final chunk (the real root cause: ID_OBJ_MAX=-1 on the last chunk). The wrapper writes the computed range into the chunk's generated config copy (ID_OBJ_MIN/ID_OBJ_MAX). (A params: function can't do this — params evaluate at DAG-construction, before the sexcat exists.)
• Chunk jobs are exempt from the run-log rewrite (finding 18): N chunks share one tiles/<ID>/output/ tree, and concurrent update_runs_log_file.py truncate-rewrites are a lost-update race that drops a sibling's just-appended entry. Concurrent appends (what shapepipe_run does natively) and reads are proven safe; only the rewrite races. Chunk configs use literal INPUT_DIRs (./output/run_sp_tile_Sx|PiViVi|Mh etc., deterministic under RUN_DATETIME=False), bypassing the log entirely. merge_sep_cats — DAG-serialized after all chunks — is the single per-tile log-sync point before make_cat reads the log. Design rule: log_run_sp.txt is shared mutable state within a unit tree; only DAG-serialized stages may rewrite it; concurrent stages may only append.
• Thread/arena hygiene the p3 runs use: MALLOC_ARENA_MAX=2, MALLOC_TRIM_THRESHOLD_=0, OMP/OPENBLAS/MKL_NUM_THREADS=1.

D5. One profile; Snakemake owns the containers

profiles/nibi/config.yaml carries everything site-specific; rules carry only intrinsic needs (threads, mem_mb, runtime):

# profiles/nibi/config.yaml
cores: 192
local-cores: 4
resources: {mem_mb: 700000}          # node-RAM cap for the local scheduler
default-resources: {mem_mb: 2000, runtime: 2h}
software-deployment-method: apptainer
apptainer-args: "--cleanenv --env OMP_NUM_THREADS=1 --env MALLOC_ARENA_MAX=2 --env PYTHONPATH=<prod worktree>/src --home /home/cdaley --bind /project --bind /scratch"
keep-going: true
show-failed-logs: true
rerun-incomplete: true
latency-wait: 60
# no rerun-triggers override — full default triggers (D3)

Containers: Snakemake wraps every job in apptainer exec automatically — the workflow never calls apptainer itself. Rules carry a container: directive pointing at the local /project .sif (/project/def-mjhudson/cdaley/containers/shapepipe-develop-runtime.sif), never docker:// — no apptainer-prefix cache, no registry pull mid-campaign (findings 15/2). The profile sets software-deployment-method: apptainer and the args above, and Snakemake wraps each command. This kills the hand-rolled in_container/apptainer_noslurm machinery. Per-rule containers are allowed → modularity (default to the project SIF; a different SIF per stage is permitted).

Container environment is explicit, because the SDM drops the proven recipe (findings 2/3). Snakemake's apptainer SDM generates apptainer exec with no --cleanenv, so only what the submitting shell exported would pass through, and it hardcodes --home <cwd> (singularity.py:143), which would set $HOME to the workflow workdir and hide ~/.ssl/cadcproxy.pem from vos/vcp — every get_images download would fail auth. The profile's args fix all three:

• --home /home/cdaley is appended after snakemake's hardcoded --home <cwd> and wins (verified on apptainer 1.4.5), restoring $HOME so ~/.ssl/cadcproxy.pem is visible (and astropy/vos caches don't collide in the shared workdir).
• --env OMP_NUM_THREADS=1 caps the sif's scipy-openblas pool (verified 32→1); without it 40 forked CCD workers each size a pool to the 192-core node.
• --cleanenv is stricter than the proven recipe but safe: APPTAINERENV_* survive it (Snakemake's envvars: keeps working) and the wrapper's SP_RUN/SP_CONFIG/SP_EXP exports happen inside the container.
• --env PYTHONPATH=<pinned checkout>/src is set for P0 (finding 2c): P0 validates science content against bash runs that use a worktree src override, so P0 must run known code. The pin must be a frozen checkout at a named commit, never a moving branch checkout — the live p3-batch1 run demonstrated the failure (its shapepipe-prod worktree switched branches mid-run, so late stages run different code than early ones). Production later rebuilds the sif at the exact validation commit and drops the override — which-code-runs is a decision, never a build-time accident.

A P0 smoke rule, run inside a submitted SLURM job, prints shapepipe.__file__, sys.path, and threadpoolctl.threadpool_info() to prove code precedence and thread caps before any science validation, and one containerized tile_get_images runs end-to-end before anything else in phase A (proves cert visibility).

The launcher owns module load apptainer (finding 20). apptainer is a cvmfs module, not on the default PATH; snakemake resolves the binary at job runtime via shutil.which, so a snakemake started from any shell without the module loaded fails every job after burning queue time. A committed launcher (workflow/bin/sp, or a Makefile target) does module load apptainer/1.4.5 before exec-ing snakemake --profile profiles/nibi — the documented entry point, so PATH never depends on shell state. It is also the path P0 exercises.

Group-compatibility is a design constraint (finding 22). Per-unit-per-stage granularity at DR6 is ~120–170k SLURM jobs/campaign — not yet priced against the proven single-node monolith (14 shapepipe_run calls for 50 tiles). Snakemake's designed answer is group: fusion, but only if rules stay fusable: no mid-chain localrules, no pipe/service outputs in the tile/exposure chains (currently true — state it so it stays true), making group: a pure profile-level knob. P0 measures the granularity cost: DAG build time/memory for a synthetic index at 2–4k-tile scale (snakemake -n on fake index rows), per-job overhead (apptainer cold-start + shapepipe_run startup vs stage runtime) on the shortest rules (exp_split, tile_merge_headers, tile_merge_cats), and a one-line check of nibi's MaxSubmitJobs/short-job policy. If short-rule overhead exceeds ~20–30% or scheduler load is unfriendly, set group: labels (fuse merge_headers→mask+detect→vignets per tile, or the exposure Sp→Ma→psf chain) in the profile — no rule or contract change, directory() outputs and failure classes unaffected.

CANFAR is a future profile — a one-line note, not implemented here. When it comes it is a second profiles/canfar/ (cluster-generic wrapping the skaha REST calls curl_canfar_local.sh already makes); the Snakefile is unchanged.

D6. Repo layout, storage, sharing, migration, and the complexity budget

workflow/
  Snakefile              # parse-time index load; onsuccess/onerror → sp report; includes rule files
  rules/{prepare,exposure,tile}.smk
  scripts/{sp_rule.py, build_index.py, build_forest.py, run_report.py, clean_exposures.py, completeness.py}
  bin/sp                 # launcher: module load apptainer → snakemake --profile profiles/nibi
profiles/nibi/config.yaml

Pin snakemake>=9,<10 (latest 9.23.1). v8→v9 breaking changes are minor for us: --use-singularity became --sdm, executors are now plugins (the local executor we use stays built-in), the full rerun-triggers set is the default.

Storage placement (finding 15). /scratch purges by age; a P3 campaign spans months. So durable, low-volume products — final_cat-<ID>.fits, index/run_index.sqlite, index/missing.json, reports — land on /project; bulk intermediates (per-CCD split/mask/PSF run dirs) live on /scratch, each batch sized to complete well inside the ~60-day purge. The Snakemake workdir (.snakemake/ — DAG metadata + incomplete-markers) is explicitly on /project so purge never breaks the mtime chain or resurrects finished-tile reruns.

Sharing model is a contract, not a convenience (finding 16). Dedup over a shared output root is per-invocation DAG semantics only. The contract: exactly one Snakemake workdir (one .snakemake) per output root, so all invocations share provenance metadata and the workdir lock serializes them; invocations are strictly sequential and the launcher refuses --nolock (two concurrent invocations would each delete-before-rerun a directory the other is reading/writing). Even sequentially, under full rerun-triggers, code/params drift between invocations would mark run A's finished exposure dirs stale and delete-before-rerun their RNG-bearing products. So final_cat and completed exposure-product directories are protected() — check_protected_output fires before deletion, converting a silent destructive recompute into a loud failure exactly when the full-rerun-trigger × delete-before-rerun combination would otherwise strike. Intentional recomputation routes only through the sp rerun (--forcerun) verb. (In Snakemake 9 a cross-workdir invocation is not itself a rerun hazard — missing metadata is informational, excluded from rerun reasons — but it silently drops provenance tracking, which is its own reason to forbid it.)

Is this a fourth orchestration generation? The honest accounting: sp_rule.py + build_index.py + the index + the report replace ~2000 lines across three bash levels plus each site's sbatch wrapper. The wrapper is smaller than round 1 — no classifier, no whitelist, just unit setup, config copy, run-log, count-floor, tee — and everything it owns the bash layer already does, scattered. The claim to hold us to: net less code, in one place, testable, with the DAG delegated to a tool whose whole job is DAGs. If an implementation PR doesn't deliver that, this design has failed and we should say so.

Phased migration, each phase independently useful, bash layers untouched until Martin retires them (P0/P1 run on a separate output tree — no shared exp/ with a bash-driven run during validation):

1. P0 — prepare + exposure stages on a 4-tile P3 subset. Gates: the container smoke rule (shapepipe.__file__, sys.path, threadpool_info), one end-to-end containerized tile_get_images (cert visibility), the nproc/cgroup + max-packing measurement, and the granularity measurement (DAG build cost, short-rule overhead, MaxSubmitJobs). Validate against a bash-driven run by scientific equivalence, not byte-identity (FITS DATE headers, setools rand_split RNG, and ngmix stochasticity all break bytes) — product counts per stage, catalogue row counts, shape/PSF parameters to numerical tolerance. Add the 3-rule synthetic append test (run 2 tiles, append 1, dry-run must schedule nothing for finished tiles) to CI as the growth-invariant regression — it runs in seconds.
2. P1 — full tile chain through final_cat; retire job_p3_batch1.sh for new runs.
3. P2 — scale via batched one-allocation invocations (50-tile → P3, ~7300 tiles, tile-disjoint sky partitions); exercise the count-floor, retries, rerun propagation, and sp clean-exposures for real.
4. P3 — CANFAR profile (cluster-generic wrapping skaha); decide whether the deferred slurm-executor mode is worth a custom plugin.¹

Open questions

1. Batch sizing for multi-invocation scaling: how many tile-disjoint sky partitions, how large (bounded above by the 60-day purge, below by exposure-overlap waste at batch borders)? Named now, decided at P2.
2. make_cat over partial tiles: if k of 40 CCDs failed PSF on some exposure, the tile completes with fewer epochs. Keep that (report records the gaps), or add a quality gate (quarantine a tile above X% missing)?
3. Where do generated per-chunk ngmix configs live — wrapper writes them into the unit work dir (proposed), or keep the repo-level sed-template approach?
4. Repo layout: workflow/ at top level per Snakemake convention, or under scripts/ next to the bash it replaces?

---

Round-2 design (revised), grounded in run_job_sp_canfar_v2.0.bash / job_sp_canfar_v2.0.bash on develop, the im_sims Snakefile lineage (#766), and the nibi P3 runs (#808). Simplified after Cail's feedback on round 1 and a second research pass (Snakemake v9, nibi compute-node networking, the field's partial-failure idiom), then hardened against a second adversarial review. Living PRD — kept current as implementation proceeds.

— Claude (Fable) on behalf of Cail

Footnotes

1. The deferred slurm-executor mode (per-rule sbatch instead of one big allocation) is not hypothetical — a probe completed end-to-end on nibi (login-node snakemake, one sbatch per rule via the apptainer SDM; slurm_account=def-mjhudson_cpu, per-rule cpus_per_task/mem/runtime from the same measured footprints). It runs from the same Snakefile; deferred to a later scaling decision, not designed out. ↩

[cailmdaley]

Round 2 posted — the body above has been rewritten in place (round 1 is preserved in the edit history). What happened between rounds:

1. Cail's feedback on round 1, whose through-line is simplify: one execution mode (the one-big-allocation), one profile, Snakemake owns the containers (--sdm apptainer — nothing in the workflow ever calls apptainer itself), full default rerun-triggers, a single run-level report, snakemake>=9,<10, CANFAR deferred to a future profile.
2. A second research pass that overturned two round-1 assumptions: nibi compute nodes do have outbound internet (verified empirically from a compute node), so downloads run in-DAG; and the failure-handling machinery collapsed to the count-floor that the bash complete_check already encodes — the three-class taxonomy, whitelist, and verdict-classifier are gone. Per-CCD stages now declare deterministic directory() outputs (RUN_DATETIME=False per-unit), with an in-job per-runner count-floor from a committed expected_counts.yaml.
3. A second four-lens adversarial review of the revised design (30 raw findings → 22 verified, 4 blockers), all folded in. The blockers were seam bugs between Snakemake's state model and ShapePipe's: timestamped run dirs vs. declared outputs, the apptainer SDM silently dropping the proven container environment (--home, thread caps), and temp() on shared exposure products destroying the "finished tiles stay finished" invariant on the workflow's most routine operation (appending tiles). That last one is why cleanup is now an explicit index-driven verb instead of temp() on the exposure store.
4. Empirical grounding on nibi: the prepare phase already runs green on a 4-tile P3 subset (deterministic run dirs, run_index.sqlite with 4 tiles / 30 exposures), and both execution paths were probed live (the one-allocation container mechanics, and a slurm-executor probe job that completed end-to-end — kept as a deferred scaling option).

A draft PR with the implementation skeleton follows shortly; this issue stays the living PRD and will be updated as implementation teaches us things. The open questions at the foot are still the discussion agenda.

— Claude (Fable) on behalf of Cail

🤖 Generated with Claude Code

[cailmdaley]

What the first real runs taught (P0 night 1)

The design has now met real data on nibi and delivered. Every compute attempt failed on a genuine defect, each one at the seam between Snakemake's state model and ShapePipe's directory-as-database bookkeeping — never a design error, always a concrete v2.0 mechanism the abstraction hadn't yet been told about. The recovery run then drove the full tile chain through all eight ngmix chunks on every tile: job 17808445 COMPLETED in 3h47, four final_cats on disk, and all four tiles PASS scientific-equivalence validation against the bash baseline. This is what the runs taught, organized by the design section each learning touches.

P0 delivered — four final_cats, validation PASS

• Job 17808445 COMPLETED (3h47): 4/4 final_cats on disk (46/42/47/44 MB, 173 columns). Tile 210.296 = 34,321 rows — exactly its SExtractor object count: the closed-range ngmix chunking covered every object once, no straggler drop, no double-count.
• run_report.json: 4 tiles / 19 exposures, no missing tiles, zero failures at exposure granularity. (Attrition granularity caveat in D3.)
• Append-invariant test PASSED. Appending the 196 quad to the tile list and dry-running scheduled exactly 65 jobs, all for the new tiles (every rule count = 4, ngmix 32 = 4×8) and zero for the finished quad; no exposure jobs until the index is rebuilt (parse-time index doing its job). Run under --rerun-triggers mtime — the first default-trigger invocation would storm the DAG on the script-hash param change (metadata reconciliation, D3).
• Smoke gates green in-job: shapepipe.__file__ = pinned worktree src in both driver logs; OMP pool 32→1; ulimit -u = 3.09 M (no nproc ceiling).
• Validation: all four tiles PASS scientific equivalence to the p3-batch1 bash baseline — table and the PSFEx caveat in the Validation section below.

D1 — Staged execution / the index

The index must carry the exposure's original name, not just the dedup key. The exposure store keeps CFHT's p suffix (2086319p), while the DAG keys exposures by bare base id. The base id is the right dedup key, but get_images needs the original name: the index carries both, and the wrapper writes the original name verbatim into the fabricated per-unit exp_numbers list. This confirms the posted design's own note ("carries both bare exposure ids and original names, 2605805p") — it's now load-bearing in code, not just prose, and the write must be unconditional (an exists-guard had pinned a stale pre-fix copy of the list file across a rerun).

Structural dedup is real and observed. prepare came up green on the 210/211 overlap quad: 34 tile→exposure edges collapsed to 19 unique exposure chains, each scheduled exactly once. The dedup-within-an-invocation contract holds against actual overlap geometry.

D2 — Wrapper fabrication, forest, star-cats, count flavors

Four D2 mechanisms needed correction against the running modules:

1. The exposure forest must be sharded, not flat. exp_utils.get_exp_output_files hardwires the v2.0 sharded layout (forest/<2-digit-prefix>/<base>/output) into its $SP_EXP glob, so a flat forest fails every tile gather stage. The design's flat-store decision remains correct — nothing globs the store, the index drives every path — but the module-facing view the forest presents must carry the shard. This is a refinement of the posted text, not a contradiction: the store is flat, the forest view is sharded.

2. Star cats are consumed as directories, not per-unit files. They're per-CCD directories (~40/exposure) reached via $SP_RUN/star_cat_{exp,tiles} dir symlinks — the v2.0 mechanism the wrapper already promises to reproduce. The per-unit star-cat file rules I had sketched linked names that don't exist and nothing read them; they're deleted. The wrapper's dir-symlink step (finding 11b) is the whole mechanism.

3. get_images completeness counts are config-flavor-dependent. The nibi symlink flavor produces tile 2 / exp 3 files, versus the canfar vos table's 4 / 6. The committed count table now follows the actual configs in use, verified against the p3-batch1 baseline (100 files / 50 tiles). This directly touches the D3 count table (see below).

4. Shell-string placeholders don't reach into params:. {threads}/{output} survive literally into the process if they sit inside a params: value — Snakemake formats the shell: string once. Placeholders must live in shell:. This is a wrapper-plumbing fix, but it validates the D4 insistence that SMP_BATCH_SIZE={threads} be written by the wrapper into the config copy, not passed as a shell placeholder.

D3 — Failure policy, observed live + the quota postmortem

The count-floor works, watched in real job exits. attempt 3's log shows both OK lines (above floor → exit 0) and FATAL breaches (below floor → nonzero) — the P0 failure-policy bullet now has live evidence, not just a ported table.

The report counter needs to be per-CCD, not per-exposure. Validation exposed a granularity gap in run_report: its "zero attrition" is exposure-granularity only. The real signal lives per-CCD — psfex_interp (validation_psf) attrition is 46/760 CCDs = 6.05% (smk) vs 41/760 = 5.39% (bash), ~30× the 0.2% lore figure, on both sides. The run is healthy; the counter is just aggregated one level too coarse to show where the loss happens. This is the DES-blacklist granularity exactly (per-CCD, not per-exposure), so the report should count at that grain. FOLLOW-UP: make the run_report attrition counter per-CCD.

The DES/Rubin precedent is citable. The count-floor + standalone report is the field's converged pattern, and I've verified the citations: DES's per-CCD PSF blacklist (Y3: <25 stars → blacklisted, ~2% of data removed, survey proceeds — Jarvis et al. 2021 §4.2; also 2016, Y6 2025) is our sparse-CCD setools case almost literally; Rubin's Butler quantum registry + pipetask report (Jenness 2022, Lust 2023) is the same "record per-unit, gate on a floor, report and continue" shape at survey scale. Both avoid a workflow engine's per-file targets; Snakemake forces the same choice (#4016, no optional-output primitive). Worth adding these citations to the D3 precedent sentence.

Quota-kill postmortem — two independent failure layers. attempt 3 (job 17798771) computed the full chain of tile 210.296 — all 8 ngmix chunks passed with in-job closed ID ranges (n_obj=34321 → chunk 8 = [30031, 34321], no open-ended last chunk: the straggler fix working) — then died writing Snakemake job metadata, because /project def-mjhudson had gone hard-full (27/27 TiB). Two layers:

• The forest-staleness layer, which is a D3 rerun-trigger finding: three of four tiles FATAL'd at tile_merge_headers on stale flat forests from attempt 2. The forest rule never re-ran, because the code trigger tracks the shell: string but not the external sp_rule.py it invokes — exactly the tradeoff the design accepts to keep wrapper edits from invalidating the whole DAG. The fix folds a wrapper-script content hash into every rule's params:, closing the gap. The one-time cost: a params mismatch on every already-recorded job → a full-DAG storm at the next default-trigger invocation. So the recovery (attempt 4) runs --rerun-triggers mtime deliberately, with the 3 stale forests deleted by hand, and defers metadata reconciliation until after green.
• The quota layer is a real-world instance of D6's storage story, below.

D5 — Profile env, state placement, execution mode

PYTHONPATH pin and container env held up — the smoke gate (attempt 1, job 17794661) confirmed the pinned src resolves and threadpools cap at 1, exactly as D5 specifies.

State placement had to move off /project under duress. The posted design places the Snakemake workdir and index_db on /project (so scratch purge can't break the mtime chain). /project going hard-full forced the opposite tonight: the working repo moved to /home/cdaley/shapepipe-smk (clone @ a273e763, pushed), bin/sp runs Snakemake with --directory <run_dir>-state on /scratch, the Snakefile resolves its configfile relative to itself, and index_db moved to scratch. This is a forced deviation, not a design change — but it means the D6 "durable products + .snakemake/ on /project" placement is currently blocked and should be flagged as such until group-level /project reclamation happens. The design intent stands; the environment doesn't currently permit it.

One-allocation mode delivered — attempt 4 = job 17808445, one deliberate invocation, COMPLETED in 3h47. Confirms the D4 single-allocation model in practice, start to four catalogues.

D6 — Bench numbers, storage reality, precedent

DAG-scale gate cleared, and it reframes P2. Synthetic index (realistic ~2.7 exp/tile overlap), snakemake -n, nibi login, Snakemake 9.23.1, parse-time dict-lookup input functions:

tiles	jobs	DAG build	peak RSS
250	6,689	14.6 s	238 MB
1,000	26,689	77 s	616 MB
2,000	53,357	163 s	1.1 GB
4,000	106,689	351 s	2.1 GB
DR6 (~7,300)	~195k	~11–12 min	~4 GB

Near-linear (~3.3 ms, ~20 KB per job). All-of-DR6 builds in ~11–12 min at ~4 GB — viable on a login node or allocation head. Consequence: the D5/D6 batching-for-scale rationale should say batching is an operational choice (walltime, storage), not a DAG-scale necessity. The group:-fusion escape hatch stays in reserve, unforced.

Storage: ~34 GB/tile of un-cleaned intermediates (smk-p0: 136 G/4 tiles; p3-batch1: 1.6 T/50 tiles). This is the concrete number behind the sp clean-exposures verb — at DR6 the intermediate footprint alone exceeds any single allocation's scratch, so index-driven cleanup isn't optional at scale. And it's exactly what bit us: /project hard-full (27/27 TiB) killed attempt 3's metadata write; scratch is over its soft quota (1693/1024 GiB) even after a 357 G reclaim tonight. The storage story in D6 is now measured, not projected.

---

Validation — scientific equivalence to the bash baseline

All four tiles PASS scientific equivalence to the p3-batch1 bash baseline (verdict: /home/cdaley/scratch-validation/validation_verdict-p0night1.json). Rows are exactly equal on every tile (100% NUMBER crossmatch), column sets are identical, and every deterministic (non-NGMIX) column is bit-identical: SExtractor 760/760 CCD cells exact, setools star counts exact (max delta 0), WCS headers max-diff 0.0 across all 19 exposures. Shape statistics agree — per-object g1/g2 correlation 0.964–0.989, g1/g2/T/SNR/FLUX medians within 1–2%, NGMIX_FLAGS_NOSHEAR==0 at a rate of exactly 1.00000 both sides, all tiles.

tile	rows	g-corr	T rel-Δ	SNR rel-Δ
210.296	34,321	0.988–0.989	0.4%	1.1%
210.297	30,637	0.964	0.2%	0.2%
211.296	35,037	0.983–0.984	0.2%	0.04%
211.297	32,712	0.987–0.988	0.9%	1.7%

Finding — the residual shape differences are PSFEx, not orchestration. The g-correlations sit at 0.96–0.99 rather than 1.00 because PSFEx diverges structurally under the baseline's known mid-run code drift, not because of the Snakemake orchestration. Concretely: bash drops CCD 0 on 6 exposures where smk keeps it; that propagates through NGMIX_N_EPOCH on 0.02–13.1% of objects (tile-dependent) and surfaces as a few 2–2.6σ paired g2/SNR offsets at the ~1e-3 level. Two consequences. (1) FOLLOW-UP: confirm PSFEx config parity between the two code pins before treating the two catalogs as the same data product — parity confirmation is what lets us attribute the whole residual to the drift. (2) Provenance: the smk run's pinned code makes it the better-controlled product of the two — worth one sentence in #848, because it inverts the usual "is the new orchestration trustworthy" framing.

Naming divergence to reconcile

The posted D3 text names the count table two ways: the wrapper's committed table is workflow/scripts/completeness.py (a stdlib-only Python table), while the standalone report is described as reading expected_counts.yaml (D3 final paragraph and the finding-8 report paragraph). Implementation settled on the Python table (completeness.py) as the single source both wrapper and report import in-container. The PRD should pick one name — completeness.py — and drop the expected_counts.yaml references so there's one table, imported, not a YAML the report re-reads.

PRD edits needed (checklist for tightening #848 tomorrow)

• D2, forest bullet — state the store is flat but the module-facing forest view is sharded forest/<2-digit-prefix>/<base> (because get_exp_output_files globs the v2.0 layout). Current text implies the forest is just "convenience"; make the shard explicit.
• D2, exp_numbers fabrication — note the fabricated list must carry the exposure's original p-suffixed name and be written unconditionally (no exists-guard).
• D2, star-cat bullet — the per-exposure/per-tile star-cat outputs are consumed as directories via $SP_RUN symlinks, not per-unit declared files; drop any per-unit star-cat file rule language.
• D2 count table + D3 — the tile get_images:4 (2 sims) / exp get_images:6 (3 sims) rows are the canfar vos flavor; the nibi symlink flavor is 2 / 3. Either parameterize the table by config flavor or state it follows the active configs.
• D3, rerun-triggers section — add the caveat that the code trigger does not cover the external sp_rule.py, so a wrapper-script content hash is carried in each rule's params: to make wrapper edits propagate. (Currently the text implies code on the shell string is sufficient; it isn't for external-script logic.)
• D3 / finding 8 — rename all expected_counts.yaml references to completeness.py; one table, imported by both wrapper and report.
• D3, precedent paragraph — add the DES (arXiv:2011.03409 §4.2) and Rubin (arXiv:2206.14941, 2303.03313) citations.
• D3, attrition counter — make the run_report attrition counter per-CCD, not per-exposure: validation showed exposure-granularity "zero attrition" hides the real per-CCD psfex_interp loss (46/760 smk, 41/760 bash ≈ 6%). Same grain as the DES per-CCD blacklist.
• D3 / metadata reconciliation — record the decision the script-hash param forces: the params mismatch on every already-recorded job means the next default-trigger invocation storms the whole DAG. Pick the reconciliation path (one-time controlled rebuild vs. accepting the storm once) and state it; P0 ran --rerun-triggers mtime to sidestep it, which is not a standing answer.
• Validation, PSFEx parity — before the smk and bash catalogs are called the same data product, confirm PSFEx config parity across the two code pins; the residual shape differences (g-corr 0.96–0.99) trace to the baseline's mid-run PSFEx drift, not orchestration. Note that the smk run's pinned code makes it the better-controlled product.
• D5/D6, batching rationale — reframe batched invocations as an operational choice, citing the bench (DR6 ~195k jobs ≈ 11–12 min / ~4 GB), not a DAG-scale necessity.
• D6, storage — replace projected storage language with the measured ~34 GB/tile.
• D6, state placement — note that /project placement of .snakemake/ and durable products is the intent but is currently blocked by /project being hard-full (27/27 TiB, group-level); state runs off /scratch via --directory until group-level reclamation. This is the storage-placement reality, not the design intent.
• D6 P0 gate list — the DAG-scale/granularity, compute-DAG-green, failure-policy, final_cat delivery, and append-invariant bullets are all now done (job 17808445, 4/4 on disk, validation PASS); the P0 gate is cleared.

Status note for the issue: P0 is delivered — job 17808445 COMPLETED (3h47), four final_cats on disk, all four tiles PASS scientific-equivalence validation against the bash baseline, and the append-invariant regression passed. The open items are the four PRD edits' follow-ups (per-CCD attrition counter, PSFEx config parity, metadata reconciliation, /project reclamation), none of which block the Gate-2 exit. This comment records what the runs proved.

— Claude (Fable) on behalf of Cail

🤖 Generated with Claude Code

[martinkilbinger]

Looks nice!

Comments:

• Level 3 (curl_canfar_local.sh) this is obsolete and already no longer used (was used up to the skaha python canfar update in 2025).
• Setting RUN_DATETIME=False is fine as long as Snakemake handles re-runs of failed modules (shapepipe would exit with an error that output dir already exists)
• exposure index file: Is it flexible that exposures can be added if we run more tiles? E.g. do a test run of a few tiles, then add more for a larger run?
• tiles/exposures/tile_exposures: I am still a bit worried that having all tiles (exposures) in one root dir will overload the file system. Even with index sql files we still want to type ls, find, wc, etc. Are or order 50k subdirectories ok? And what is tiles_exposures?
• star_cat_exp, star catalogue for masking: This will be no longer used once healsparse masks replace ShapePipe-internal masking
• config_merge_sep_cats.ini: running ngmix as parallel jobs on object batches and the merging in the next module has been removed for early v2.0 runs; this was fairly unstable and difficult to manage in v1.4. Make sure this is done properly in the new Snakemake setup.
• I'd like to implement a global statistics of tiles and exposures, how many succeeded and how many failed, and for what reason. See previous script shapepipe/scripts/python/summary_tiles.py.
• When running tiles in parallel, the code has to make sure that not more than one job accesses the same exposure; this should be already implemented in v2.0, but check.