Set up ShapePipe on Nibi and run end-to-end on a first batch of P3 tiles

[cailmdaley]

Goal

Get ShapePipe running end-to-end on Nibi over a handful of P3 tiles — the first real run on data. This is deliberately small: the point is to stand up the container, wire the pipeline to our pre-staged data, get a valid shape catalogue out the far end, and measure how long a tile actually takes and how it parallelizes — so we can size the full P3 run realistically. Second issue on the First run on data (Nibi) milestone, following #801.

What's already in place (from #801)

P3 data is staged locally in /project/def-mjhudson/unions-wl/ (see its README.md):

• tiles/ — 1268 P3 coadd images + weights (detection layer)
• exposures/ — 1180 unique single exposures, image+weight+flag (measurement layer; downloading)

So the run can consume everything off local disk — no CADC calls needed during processing.

Setup tasks

• Container. apptainer pull the runtime image (build from develop, not the stale v1.0.1 tag), caching off $HOME:

  export APPTAINER_CACHEDIR=/project/def-mjhudson/unions-wl/.apptainer
  apptainer pull /project/def-mjhudson/unions-wl/shapepipe-runtime.sif \
      docker://ghcr.io/cosmostat/shapepipe:develop-runtime

• Clone shapepipe (develop) for the example/cfis/ configs + auxdir/.
• Point ShapePipe at local data. In config_tile_Git_vos.ini and config_exp_Gie_vos.ini, set RETRIEVE = symlink and INPUT_PATH → /project/def-mjhudson/unions-wl/{tiles,exposures}. Verify INPUT_FILE_PATTERN/INPUT_FILE_EXT match our staged filenames (CFIS.XXX.YYY.r.fits / .r.weight.fits.fz; exposures NNNNNNNp.fits.fz + .weight/.flag).
• Tile list. Write tile_numbers.txt with the 2–4 chosen tiles.
• SLURM. Adapt example/pbs/candide_smp.sh for Nibi (partition, --account=def-mjhudson, --cpus-per-task, real --time), with [EXECUTION] MODE = smp and [JOB] SMP_BATCH_SIZE = cpus. Bind the clone at an identical path (--bind "$SPDIR:$SPDIR") so config paths resolve in/out of the container.
• Run the module chain (PSFEx path) in order — get_images → find_exposures → get_images(exp) → split_exp → merge_headers → mask → sextractor → setools → psfex/psfex_interp → vignetmaker → spread_model → vignetmaker → ngmix → merge_sep → make_cat — and confirm a final_cat FITS appears.
• Record wall-time per tile and per stage; note where ngmix (shape measurement) dominates.

Scope decisions for run 1

• PSFEx, not MCCD — simpler, better-trodden for a first validation. MCCD (the UNIONS-WL production model) is a later run.
• ~10 tiles — a big enough chunk to see per-tile timing spread, throughput scaling, and a first failure rate, while still finishing comfortably on one node.
• SMP / joblib on one node — tiles are independent, so ShapePipe fans them across workers within the node. That's all the parallelism this run needs; MPI / multi-node is a throughput lever for the full run, with nothing to add here.

What we're measuring

• Wall-time per tile, and where it goes across stages (ngmix shape measurement is expected to dominate).
• Throughput scaling with SMP_BATCH_SIZE (cores) — tiles are independent, so total run time should scale ~linearly with cores; this run calibrates the unit cost.
• Peak memory per tile.
• Failure rate across the ~10 tiles — how many complete cleanly vs. error out, and why.
• A valid final_cat shape catalogue as the "it works end-to-end" artifact.

Open questions

• Do our staged P3 filenames match the CFIS DR6 INPUT_FILE_PATTERNs, or do the patterns need editing?
• Audit the full set of input data products a run needs beyond tiles + exposures — star catalogues / star masks for mask, auxdir/ contents, any PSF/validation inputs — and confirm we have them (and whether any reintroduce a CADC dependency). To be nailed down in a follow-up comment.
• Nibi has no ShapePipe reference config (candide is the documented cluster) — the sbatch header needs porting, not copy-paste.

— Claude (Fable) on behalf of Cail

[cailmdaley]

First full test run on Nibi: CFIS P3, 10 tiles / 1960 single-CCD exposures, through step 7 (SExtractor + setools star selection + PSFEx + interp) so far. Along the way we poked at the data and the failures, and found five things worth fixing — all now up as PRs (#823–#827). The story is best told in plots.

4 of 1960 CCDs crashed star selection with ValueError: Too many bins for data range. Here's a healthy CCD for reference — image with detections (red = selected stars), weight, pipeline flags, and the size–magnitude plane with the setools cuts:

And here's a "failed" one. Spot the difference? Neither could we. Clean image, ~1400 detections, a textbook stellar locus with ~55 stars sitting in the selection box:

Nothing is wrong with these CCDs. The crash is a numerical edge case in StrInterpreter._mode: its densest-bin zoom can collapse onto near-identical float32 FWHM values before converging, at which point np.histogram gives up — np.histogram(np.array([4.002815, 4.002817], dtype=np.float32), 11) is the whole bug in one line. Which CCDs fail is a coin flip on float32 duplicates. Fixed in #823 (all 4 CCDs recover; healthy CCDs bit-for-bit unchanged).

A second failed CCD had something more interesting in it. Look at the upper part of the image — a huge, bright, unmasked glow, full of accepted detections:

The bright-object halo mask should have caught this. It didn't, because mask.py averages the four GSC magnitudes with is not None guards that let NaN through: this object has only Fmag (9.5), its mean goes NaN, NaN < HALO_MAG_LIM is False, and it is silently skipped. The brightest (most saturated) sources are exactly the ones most likely to be missing GSC bands — the bias hits the objects that most need masking. This is very plausibly a big part of #662. Fixed in #824: on this exposure's star catalogue, halo stars go from 57 to 118, and the recovered ones are the brightest in the field (mag 5.7–9.5):

But look closer at the glow itself: no diffraction spikes, and perfectly elliptical. It isn't a star at all — it's the nearby galaxy NGC 5949, shredded by GSC into catalogue entries, one of them (likely the nucleus) misclassified as a Class-0 "star". Masking it via a star halo is the right answer for the wrong reason… which raised the question: doesn't ShapePipe mask bright galaxies deliberately? It does — the NGC/Messier components are enabled and NGC 5949 is in the shipped ngc_cat.fits — but the overlap test only fires when an image corner lies within the object's radius, so a mid-CCD galaxy is silently skipped (Found 0 NGC objects overlapping with image, on this very CCD). Fixed in #827 with a footprint-based test — the NGC mask now lands exactly on the galaxy:

Two more, text-only. With astropy 7 (the recent major-6 pin bump), the tile-level epoch-matching post-process crashed on 7/10 tiles: it inverse-projects every tile object through every contributing CCD's WCS, and wcslib 8.4 now raises for objects far outside a CCD's distortion domain — routine at tile scale, tolerated by astropy 6. Fixed in #825 (forward-project the footprint first; previously-succeeding tiles bit-for-bit unchanged). And a recurring theme throughout was failures that are hard to read where they happen — a bare numpy error, a silent skip, an unset $SP_EXP becoming a literal relative path, merge_headers silently merging only the first tile, example configs referencing run names their siblings don't produce. #826 collects those: loud failures and consistent examples.

Five bugs, five PRs, ten tiles. Not bad for a first look at the data.

— Claude (Fable) on behalf of Cail.