image-sims workflow: campaign hardening + snakemake-native containers#287
Open
cailmdaley wants to merge 38 commits into
Open
image-sims workflow: campaign hardening + snakemake-native containers#287cailmdaley wants to merge 38 commits into
cailmdaley wants to merge 38 commits into
Conversation
Bring the image-simulation shear-bias core over from the divergent
image_sims_val branch as a clean, self-contained unit:
- src/sp_validation/image_sims.py: ImageSimMBias reads the five
calibrated grid catalogues (reference + g1/g2 +-shear pairs), matches
each to the reference by RA/Dec, and returns m1/m2/c1/c2 with
bootstrap errors. Its only sp_validation dependency is
match_catalogs_radec.
- src/sp_validation/catalog.py: add match_catalogs_radec (nearest-
neighbour RA/Dec match within a threshold) alongside match_stars2 --
develop had no such helper, and image_sims_val's cat.py is not on
develop.
- scripts/compute_m_bias_image_sims.py: CLI driver; now casts numpy
scalars to plain floats before dumping so m_bias_results.yaml is
human-readable rather than !!python/object binary.
- config/calibration/mask_v1.X.{4,9}_im_sim.yaml: the image-sim
calibration/cut configs the calibrate step consumes.
- tests/test_image_sims.py: analytic synthetic-recovery test (injects a
known m/c, asserts exact recovery) plus a match_catalogs_radec check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QcYFjEWXxyCtUGzQJ7AhLH
Add workflow/rules/image_sims.smk and a standalone entry point that drive the image-simulation validation chain end to end across two containers, the sp_validation-side half of UNIONS-WL/MultiBand_ImSim#1: ShapePipe container: im_pipeline (raw sim tiles -> per-tile cats), im_merge (create_final_cat -> final_cat_{sim}) sp_validation container: im_extract -> comprehensive, im_calibrate -> cut, im_mbias -> m_bias_results.yaml Design: - Native container pattern: every rule sets `container: None` and calls `apptainer exec` explicitly, since the two halves live in different images. Extract/calibrate use the sp_validation image's own self-consistent calibration environment; m-bias injects this branch's extracted core on PYTHONPATH. - Fully parameterised under config["image_sims"] -- container paths, repo roots, data roots, PSF dict, tile list, sim/calibration knobs -- so a fresh user drives it from config alone, no hard-coded clone layout (the "runnable by someone other than Fabian" gap). - Shell blocks avoid f-strings, so snakemake's {input}/{wildcards} placeholders interpolate correctly -- fixing the {input.tiles} proxy- repr bug from the original Snakefile. - params_im_sim.py derives the field name from the run-dir basename, so a single shared template serves every sim. - Included in the main workflow under `if "image_sims" in config`; standalone via workflow/image_sims/Snakefile. Verified on candide: DAG resolves (36 jobs, correct chaining); im_init and im_calibrate run in-container and produce a real cut catalogue; the extracted m-bias core produces a real m2/c2 from grid_2's calibrated ShapePipe catalogues. Full end-to-end from Martin's on-disk intermediates is blocked by data-vintage inconsistencies (see fiber), not the workflow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QcYFjEWXxyCtUGzQJ7AhLH
… from scratch
im_init only staged params.py / mask / cfis, but ShapePipe's get_images_runner
resolves the raw tile+exposure images via $SP_DIR/input_tiles and
$SP_DIR/input_exp -- top-level symlinks that neither run_job nor any init
script creates. Without them im_pipeline can only run off pre-staged
intermediates, not from raw images, which defeats the "runnable from raw
SKiLLS" purpose of the workflow.
Stage input_tiles -> {input_sims_base}/{sim}/images/SP_tiles and input_exp ->
.../SP_exp in im_init, matching the ground-truth run-dir layout of Martin's
working image-sims runs. Verified: init produces resolving symlinks to the
real CFIS_simu_image-*.fits for all five grid_1 sims.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013uTmBAtdP5Wvav2M76MgCA
… grammar
The image-sim merge->extract->calibrate chain had never been run end-to-end
on the post-#761 ngmix column grammar; doing so surfaced a cascade of
old-grammar and interface defects, all fixed here (validated on 1z2z_grid_1:
2345 -> 1684 post-cut objects, <e1>,<e2> ~ 0 for the no-shear reference):
- params_im_sim.py: migrate add_cols / add_cols_pre_cal / centers to the v2
grammar (NGMIX_ELL_* -> NGMIX_G{1,2}_*, NGMIX_Tpsf_* -> NGMIX_T_PSF_RECONV_*,
drop NGMIX_MOM_FAIL, keep NGMIX_MCAL_TYPES_FAIL; IMAFLAGS_ISO omitted -- the
sims run no imaging-flag masking).
- extract_info.py: guard every star-catalogue-dependent block under
`if star_cat_path`. The sims have no star catalogue (star_cat_path=None), so
the star match/metacal/PSF-leakage diagnostics -- all downstream of the
comprehensive-cat write -- must be skipped rather than crash on undefined
`ind_star`. Galaxy diagnostics preserved; mixed gal+star R histograms fall
back to galaxy-only.
- catalog_builders.py: make CalibrateCat.read_cat FITS-aware. The sims produce
a single per-run comprehensive catalogue (FITS, from write_shape_catalog),
not the joined multi-patch HDF5 the data path builds; read it directly with
dat_ext=None instead of forcing h5py.
- mask_v1.X.9_im_sim.yaml: v2 grammar (NGMIX_MOM_FAIL -> NGMIX_MCAL_TYPES_FAIL,
NGMIX_ELL_PSFo_NOSHEAR_{0,1} -> NGMIX_G{1,2}_PSF_ORIG_NOSHEAR) and .fits input.
- image_sims.smk: comprehensive cat is .fits (write_shape_catalog always writes
FITS); default extract/calibrate to the repo checkout and inject
PYTHONPATH={sp_validation_repo}/src on every SPV stage, so the branch code
(these fixes) wins over the lagging baked container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D9atA9VPrg7VMC7oBArny9
im_init staged params.py by `cp {params.template}` where the template was a
snakemake `params:` value, not a tracked `input:`. Editing the template (e.g.
the ngmix-v2 grammar migration) therefore did NOT retrigger im_init, so run
dirs kept stale old-grammar params.py and extract died on
`no field of name NGMIX_Tpsf_NOSHEAR` -- only for sims whose run dir predated
the edit. Move template + mask_src to `input:` so a template change re-stages
into every run dir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D9atA9VPrg7VMC7oBArny9
…tion)
The extracted ImageSimMBias._m_c_pair was UNPAIRED: it matched the +g and
-g sims each to the reference separately, meaned them independently, and
subtracted. Intrinsic shape (sigma_e ~ 0.3) never cancelled, so a single
tile gave sigma(m) ~ 1.5 -- unconstrained.
Restore the paired estimator the SKiLLS sims are built for: match the +g and
-g sims directly to each other (same galaxies, opposite input shear; verified
98.8% co-located to <0.05" with e-correlation 0.81), difference per object,
m = <(e_+ - e_-)/(2 g_in) - 1> , c = <(e_+ + e_-)/2> ,
and bootstrap the paired objects (one resample applied to both sims) so the
per-object cancellation carries into the error. Intrinsic shape cancels in m;
c is a sum so it stays shape-noise limited (as it must). This also restores
Martin's direct +g<->-g matching, which the extraction (10baa52) regressed,
and goes beyond it (his bootstrap still resampled the two means independently).
New test test_mbias_pool_cancels_shape_noise builds catalogues with realistic
intrinsic scatter and asserts sigma(m) sits far below the unpaired shape-noise
floor -- the property the pooling exists to deliver. Existing exact-recovery
test still passes to machine precision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HmxPiKJEechqueT3YU1yrx
Fresh-eyes review notes on 4e38326: document that 1z2z is the unsheared reference (loaded for null-test diagnostics, not consumed by the +g/-g pool estimator) and that the RA/Dec match is nearest-neighbour, not a strict bijection -- effectively 1:1 on well-separated grid sims (~99% co-located to <0.05" on SKiLLS grid_1) but could dilute cancellation on denser fields. No behavioural change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HmxPiKJEechqueT3YU1yrx
…mulations with constant shear
) Fabian's #227 verdict excludes shape weights from sim m-bias calibration (w_des is pathological on sims, N_eff ~1-5%; w_iv behaves but is also excluded). w_col: null in the workflow config now flows through the .smk untouched and lands here as unit weights. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6
… the branch layout Martin implemented the #226 fix (skip additive-bias subtraction for constant-shear sims) on his image_sims_val branch (e38ace9) against the old notebooks/ layout; #225 moved calibration to scripts/calibration/. This ports the identical semantics: get_calibrated_m_c gains additive_correction (default True, data path unchanged); the calibrate script reads it from the mask config's metacal: section; both im_sim mask configs set False. c/c_err still computed as diagnostics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6
) The w_des-weighted global R was measured on N_eff ~ 20-100 objects and differs from the unweighted mean by up to 17% (0.780 vs 0.669 on tile 233.293) — no weights anywhere in sim m-bias, per Fabian's #227 verdict. calibrate_comprehensive_cat reads the key fail-fast, so both im_sim mask configs now carry it explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6
…#225 branch His side trusted, ours vetted, new grammar wins: - image_sims.py: our paired pool estimator + paired bootstrap kept; his pair_match knob adopted (False = his independent-means estimator with independent bootstraps — the toggle switches estimator flavor, documented); his print_mean_ellipticities + catalog_name default adopted; grids_dir key. - compute_m_bias_image_sims.py: ours (workflow-integrated, convergence tracking); his diagnostics capability preserved via diagnostics_image_sims.py, taken wholesale. - catalog.py/io.py: his hdf5 write branch ported into our layout (+ the import os it needed and a duplicated match_catalogs_radec removed — both caught by the reproduction gate / pre-commit). - calibration.py: identical additive_correction semantics both sides; our signature (no shape_method). - mask configs: ours (new grammar, #226/#227 flags); his old-grammar versions preserved in his branch history. Verified: calibrate+mbias rerun on tile 233.293 reproduces the pre-merge m_bias_results.yaml BIT-EXACTLY (m1=+0.274362, m2=+0.231224, all 8 values). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6
…| of FHP's grid sims The sims inject g_cosmic = +/-0.025 (basic_info.txt of all four sheared branches; applied once as galaxy.shear() in MultiBand_ImSim). Every m-bias run divided by 0.02, producing a flat, isotropic, c-clean spurious m = 0.025/0.02 - 1 = +0.25 — the entire 'S7' anomaly, including the apparent Pujol-vs-metacal response deficit (0.80 vs 0.64 = the same x1.25). Confirmed by: binned m (flat +0.22-0.25 in every S/N and rel_size bin, where measurement pathologies cannot be flat), FHP's own estimator dividing by 0.025 (shapepipe#796 comment), and ablation — at 0.025 the reference tile gives m1=+0.019+-0.022, m2=-0.015+-0.017, zero at percent level. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HPZYMKjec2dXavyURPFeR4
The psf_dict config key was read into PSF_DICT but never used -- a dead
key. Meanwhile the ShapePipe fake_psf config now expects its dictionary
path from the environment (PSF_DICT_PATH = $PSF_DICT). Add
--env PSF_DICT={PSF_DICT} to the SP_EXEC apptainer prefix so the
workflow's psf_dict value flows into the container and getexpanded
resolves it. This closes the seam: the path is supplied by the workflow,
not hardcoded in the ShapePipe repo. The path carries no shell/brace
hazards, so it interpolates cleanly through Snakemake's {SP_EXEC}.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
…ed list ImageSimMBias derived its 5-branch sim list and +/- pairing from module-level constants (the hardcoded sim_names in load_catalogs and _PAIRS). Move both into config-driven fields (branches, pairs), so the campaign manifest is the single source of truth for which sims exist and how they pair. The conventional 5-branch layout survives as _DEFAULT_BRANCHES / _DEFAULT_PAIRS, used only when the config carries no branch map (the synthetic-recovery tests). Branch load order and per-pair processing order are preserved exactly, so the seeded-bootstrap m/c result is bit-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
The injected-shear facts (amplitude, per-branch (g1,g2), pairing) had multiple
homes: a shear_amplitude literal in config.yaml, its fallback in the im_mbias
rule, and the hardcoded branch list in the estimator. Give them exactly one:
manifest.yaml, built at the head of the DAG from the sims' own basic_info.txt.
im_build_manifest.py parses g_cosmic from each requested branch's basic_info.txt
(stdlib string ops on the "g_cosmic = g1 g2" line -- runs inside the container,
no YAML/regex dep for the parse), then fails loud, naming the file and field, if
any branch's sign/axis disagrees with its 1{X}2{Y} name, the reference is not
(0,0), or the derived |g| differs across the sheared branches. Validation is
scoped to the branches the run requests.
im_mbias now depends on manifest.yaml and injects its shear_amplitude, branches,
and pairs into the generated m_bias_config.yaml; the estimator's sim list and
pairing follow from there. The shear_amplitude literal is deleted from
config.yaml and from the im_mbias fallback.
Bit-exact: the seed-42 bootstrap m/c on arm-A catalogs (m1=+0.0195, m2=-0.0150,
c1=-0.0026, c2=-0.0033, with errors) reproduces the reference through the
manifest-derived config.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
Split the image_sims config into science keys and operational keys with one home apiece, and validate it at DAG parse so typos and omissions die before any compute — replacing scattered .get() fallbacks whose silent defaults could mask a misconfigured run. - Science keys (w_col, pair_match, match_radius_deg, n_bootstrap, bootstrap_seed, mask_config) are required from the run config: no default in the .smk, none in the estimator. The workflow config.yaml carries them only as commented template lines. - Operational keys (binds, sims_type, branches, shape, config_dir, psf_model, n_smp, and the now-surfaced extract_script/calibrate_script) default visibly in config.yaml and nowhere else; the .smk reads every key as bare IMSIM[key], so config.yaml is each default's single home. - Schema check at load: an unknown key under image_sims: is a hard error (typo protection); a missing science key errors naming the key. - tile_ids is the one tile-input mechanism; tile_ids_file handling removed. - im_mbias now emits every key the estimator requires, pair_match and bootstrap_seed included — requiring a key without emitting it would be a KeyError at run time. - The estimator reads w_col/pair_match/match_radius_deg/n_bootstrap as required, and takes the bootstrap seed from config (bootstrap_seed) at the identical call site (np.random.default_rng(seed=...) per pair), so the seed-42 reference errors reproduce bit-exactly. Verified: dry-run parses green with a complete config, and errors informatively on an unknown key and on a missing science key. Estimator tail-check against the arm-A grids reproduces m1/m2/c1/c2 and all errors bit-exactly against the reference m_bias_results.yaml. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
Collapse the two-container wiring (shapepipe_sif + sp_validation_sif,
SP_EXEC + SPV_EXEC + SPV_EXEC_MBIAS) into a single image driven by one
shared exec prefix. The sp_validation image is built FROM the ShapePipe
image, so it already carries both stacks -- Source Extractor, PSFEx, the
shapepipe_run console entry, plus sp_validation -- and can host every
stage of the chain.
Config: a single `sif` key replaces shapepipe_sif/sp_validation_sif in
both the schema (_STRUCTURAL_KEYS) and the workflow config.yaml template.
The unified EXEC prefix injects branch code by PYTHONPATH rather than the
old bind-mount shadow: PYTHONPATH prepends BOTH repos' src
({shapepipe_repo}/src:{spv_repo}/src) so the worktree builds win over the
baked /app and /sp_validation copies -- the local-testing counterpart of
the git-ref deps. It carries PSF_DICT (folding in the concurrent
imsims-ab addition to the old SP_EXEC -- same semantics, from the
psf_dict config key) and keeps the SLURM-var strip (env -u SLURM_JOBID
-u SLURM_JOB_ID -u SLURM_PROCID) so the ShapePipe pipeline stage's
OpenMPI does not attach to the host launcher; the strip is harmless for
the pure-Python sp_validation stages, so one prefix serves all. Bash
entry points (run_job) and the ShapePipe/sp_validation scripts stay
invoked at the repo paths from config -- PYTHONPATH shadows the Python
packages only, stated in a comment.
Verified: snakemake -n builds the DAG (single sif/EXEC resolve, schema
green); the new prefix shadows both packages to the worktree src
(shapepipe -> wt-803-imsims-ab/src, sp_validation -> wt-225-restructure/
src, not the baked copies); shapepipe_run is on PATH in the one image;
and the estimator tail reproduces the seed-42 reference m/c bit-exactly
through the new prefix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
Replace the uncommitted user-global `SNAKEMAKE_PROFILE=slurm` with a repo
profile at workflow/profiles/candide/config.yaml, so scheduling is repo state
rather than an operator's shell. It carries only cluster policy: the slurm
executor, account cusers, partition comp,pscomp, node excludes
--exclude=n17,n09,n36 (via slurm_extra), a per-job runtime floor, retries and
rerun-triggers. Per-rule mem_mb/runtime stay on the rules; the profile is the
floor.
Deliberately kept OUT of the profile:
* Container settings. The image-sims rules set `container: None` and own
their `apptainer exec` call, so `software-deployment-method: apptainer` /
`apptainer-args` would wrap a redundant second container.
* OMP_NUM_THREADS. Pinned to 1 on the `apptainer exec` line in the shared
EXEC prefix (image_sims.smk), not here. The slurm executor submits with
`--export=ALL`, which propagates the *driver's* ambient env — but a profile
only sets CLI flags, never the driver's own env, so an OMP_NUM_THREADS there
would silently depend on the operator having exported it by hand. Injecting
it at the container boundary puts it where the compute runs, committed and
independent of the launching shell. Verified: apptainer on candide inherits
host env into the container by default, and this is the same lever the
prefix already uses for PYTHONPATH/PSF_DICT.
Document the one drive command in the image_sims Snakefile docstring and the
workflow README:
snakemake --profile workflow/profiles/candide \
-s workflow/image_sims/Snakefile <target> --configfile <run config>
The target goes before --configfile: --configfile takes one-or-more paths, so a
target after it is read as a config file.
Validated with `snakemake -n --profile workflow/profiles/candide ...`: the full
im_mbias DAG parses (32 jobs), the profile is applied, and every job carries the
account/partition/excludes defaults. Estimator tail-check under the new EXEC env
(OMP_NUM_THREADS=1) reproduces the reference m_bias_results.yaml bit-exactly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
…ntract The image_sims_val merge flipped the params template default to .hdf5, but the im_extract/im_calibrate rules and the im_sim mask config input_path all declare .fits. Extract then writes an .hdf5 the DAG never consumes and snakemake fails on missing declared output. Never caught before: the post-merge reproduction check used the recompute-only-downstream trick and never re-ran extract. Caught by the first fresh-root full-chain run (#766 reproduction gate). The hdf5 write path remains available by setting output_format explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
…iagnostics alignment Round out the m-bias estimator tail so a result file stands on its own and every reader of the code sees what the numbers actually mean. - Column semantics: the estimator consumes the *calibrated* e1/e2 columns (g = R^-1 g_uncal - c), not e*_uncal, so the headline m/c is the residual bias after the chain's own metacal calibration. Docstring the class and _load_cat honestly to that effect. - Provenance: im_mbias assembles a provenance block (sha256 of manifest.yaml, both repos' branch+commit read via git, container sif + its GHCR revision scanned plain-text from the SIF's OCI labels) into the generated m_bias_config.yaml; the compute script copies it verbatim into the output results yaml as a separate top-level key, so a result file records which manifest, commits, and container built it. Numeric m/c fields serialise byte-for-byte as before -- provenance is a pure addition. - diagnostics_image_sims.py: adopt the estimator's config schema (grids_dir, not base; branches from the config; w_col honouring null -> unit weights, so no KeyError when the weight column is absent), and colour branches from a palette instead of a hard-coded five-branch map. - Cumulative writer: overwrite an existing n_tiles entry instead of silently skipping -- the old skip left a stale (possibly wrong) number in place on a re-run. Plots regenerate on every update. The bootstrap-count in the text output reads from config, not a 500 literal. Tail-check (.tailcheck/S5): m1/m2/c1/c2 + errors reproduce the seed-42 reference bit-exactly; the added provenance block is the only difference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
…ta config The image-sim calibration reuses the data mask config and changes only what the sims genuinely differ on. Keeping a second full copy (mask_v1.X.9_im_sim.yaml) lets it silently drift from the data config it was branched from. Make the delta legible and drift-proof instead: - config/calibration/mask_v1.X.9_im_sim.overlay.yaml declares, as a list of block ops on the base, exactly what the sims drop or change and one line of why each: the sim input path, the dropped IMAFLAGS_ISO cut (no ShapePipe coverage flags on sims), the dropped dat_ext group (no coverage masks on sims), global_R_weight null (unweighted, #227) and additive_correction False (constant-shear sims). - workflow/scripts/im_compose_mask.py applies the overlay to the base as structure-anchored text edits, so the resolved file preserves the base's exact formatting and comments -- which is what makes byte-identity with a hand-maintained runtime file achievable. Each anchor must match the base exactly once, so an overlay that has fallen out of sync fails loudly. - test_mask_overlay.py locks compose(base, overlay) == the committed mask_v1.X.9_im_sim.yaml byte-for-byte, and locks the fail-fast on a stale anchor. The runtime file is left byte-identical (it is a tracked input to im_init; changing its bytes would cascade a full-chain rerun), so the estimator tail reproduces the reference m/c bit-exactly -- verified against m_bias_results.yaml. test_config_paths_exist's path walker skips the overlay's declaration keys (why/replace/with/drop are prose and verbatim base-config text, not paths); its one real path, base:, is still validated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
Fail-fast configuration made both keys required (no code defaults); the synthetic-recovery and pool-cancellation fixtures predated that and relied on the removed defaults. All three image-sims tests pass again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
Bring the Phase A restructure onto the #225 PR branch: manifest rule at the head of the DAG, fail-fast config schema (six required science keys — w_col, pair_match, match_radius_deg, n_bootstrap, bootstrap_seed, mask_config — validated at parse; unknown keys error), the single-container exec model, committed candide SLURM profile, self-describing results with a provenance block, and the mask overlay + compose stage. Conflict was in workflow/rules/image_sims.smk, at the container exec prefix. Resolved to the restructure branch's unified EXEC prefix: one apptainer prefix over a single `sif`, injecting PYTHONPATH=<shapepipe-src>:<spv-src>, PSF_DICT, and OMP_NUM_THREADS=1. This retires the old SP_EXEC/SPV_EXEC split and, with it, 4a89da2's --env PSF_DICT bolt-on on the old prefix — PSF_DICT injection now lives in the unified prefix. The 6378746 fix survives: params_im_sim.py keeps output_format = ".fits" (the restructure branch forked before that fix, so its stale ".hdf5" is superseded). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
gate225a (2026-07-11) ran the whole chain in the published sp_validation image and failed at ngmix: 'Numba needs NumPy 2.4 or less. Got NumPy 2.5'. Root cause is env drift in the derived image, not architecture: sp_validation has no lockfile and does not declare cosmo_numba (its numba-bearing dep), so an unpinned install layer bumped NumPy past numba's window; PYTHONPATH shadowing covers pure-Python code, never binary deps. Split the exec prefix by image — sif_pipeline (ShapePipe image) for the pipeline and merge stages, sif for the sp_validation stages — one shared prefix shape (SLURM-strip, PYTHONPATH, PSF_DICT, OMP_NUM_THREADS=1). This is the split the gate766 baseline ran, so the reproduction gate compares restructure mechanics against identical images. Collapse back to one image once sp_validation is uv-locked with its deps declared (spun off). Provenance now records both images (containers: sif + sif_pipeline with their GHCR revisions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H2wXqsx5RprRXVh4MQsn6h
…esults Fold Martin's per-run multiple-weight-scheme capability (origin 9a80223) into this branch's restructured m-bias surfaces, keeping both: the multi-scheme estimator AND the fail-fast config + self-describing results the Phase A work established. ImageSimMBias now computes m/c for every scheme in `w_cols` in one run. "none" (or a null entry) means unit weights; the first entry is the primary/headline result. Our fiducial leads with the unweighted paired estimator (`w_cols: [none]`), per #227. `run()` returns a document with the per-scheme block under `weights` plus the primary scheme mirrored at the top level, so a reader wanting the headline m/c never needs the scheme name and the provenance block still rides through unchanged. Bit-exactness preserved. The bootstrap resample indices are drawn once per pair (a single `(n_boot, n)` block) and shared across schemes, so a scheme comparison is a clean weighting comparison. Pre-drawing the block in one call is bit-identical to the previous per-replicate `rng.integers(0, n, n)` loop (numpy fills row-major), verified against the committed pre-merge estimator on identical catalogues: all eight m/c and bootstrap-error values match to the last bit. Config surfaces track the new shape: `w_cols` is the required science key in the workflow schema (the deprecated scalar `w_col` stays accepted in code and in an allowed-but-not-required set, so legacy run configs still parse), the config.yaml template documents it, and the m_bias rule emits `w_cols` into the generated estimator config. The driver reports every scheme in console/yaml/txt, casts the whole results tree to plain Python floats (recursive `to_python`), and self-heals legacy cumulative files carrying numpy object tags. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uild
On a fresh grids_base the DAG build aborted before any job ran:
MissingInputException in rule im_pipeline
Missing input files for rule im_pipeline:
.../{sim}/cfis
im_pipeline declared the per-sim `cfis` symlink as an input, but no rule
produces it as an output: im_init creates cfis (and input_tiles/input_exp)
as an *untracked* side effect of its shell block — snakemake will not
accept a symlink/directory as a tracked output, so those links are staged
imperatively, not declared. Asking the DAG for cfis therefore demanded a
file nothing generates; it happened to resolve only when a prior run had
already left the symlink on disk, and failed on any clean root.
The fix drops cfis from im_pipeline's inputs. The im_init -> im_pipeline
edge is already carried by `params.py`, which *is* a tracked im_init
output, and im_init stages cfis in the same shell block — so cfis exists
whenever params does. The other cwd-relative readers of these untracked
links (im_merge reads cfis/final_cat.param, im_calibrate reads
config_mask.yaml) never declared them as inputs and so were already
correct, depending on im_init transitively through params/tiles/cat.
Verified: `snakemake im_mbias -n` against a fresh grids_base now builds
the full 27-job chain (im_init -> im_pipeline -> merge -> extract ->
calibrate -> m_bias) with no MissingInputException.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lore
Port the three hard-won candide SLURM lessons from shapepipe's retired
image_sims_pipeline/Snakefile docstring (569ffe98) into our committed
profile, adapted to what the profile actually sets:
* runtime must carry a unit. Snakemake's CLI/default-resources parser
reads a bare number as SECONDS, so `runtime: 60` would give every job
a 60-second wall clock. Our value is already `"60m"`; the comment now
guards it and notes the seconds trap is CLI-only (a rule's numeric
`resources: runtime=720` is read as minutes).
* cpus_per_task=12 is now pinned in default-resources. This is a real
fix, not just documentation: the profile set no cpus_per_task, so
SLURM defaulted to 1 and would pack ~48 jobs onto a 48-core node --
past candide's per-user `ulimit -u 1200`, which crashes apptainer
beyond ~4 concurrent jobs/node ("can't start new thread"). The chain
is MPI-free and pins OMP_NUM_THREADS=1, so 12 CPUs is not for
threading; it exists solely to cap SLURM to ~4 jobs per node. The
compute-heavy im_pipeline stage inherits it (it overrides mem/runtime
but not cpus), which is exactly where the ceiling would otherwise bite.
* verify after launch with `squeue -u $USER -o "%C %l"` (must show 12
and the intended wall clock) — a misparsed runtime/cpus surfaces there
before it wastes a queue slot.
Verified: a profile-driven dry-run resolves cpus_per_task=12 onto every
job including im_pipeline (runtime=720, mem=16GB).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ShapePipe pipeline stage's ngmix ini (which owns METACAL_PSF) is resolved from the shapepipe worktree's example/cfis_image_sims via run_job's self-computed config_dir, so nothing in the run config could select a different ini -- an A/B on the metacal reconvolution PSF needed a separate worktree per arm. Add an optional operational key `pipeline_config_dir`. When set it is passed to run_job as `-c DIR`, redirecting the pipeline stage's whole ShapePipe config tree (ngmix ini included) at a per-arm copy; when empty (the default) no `-c` is passed and run_job self-computes exactly as before, so runs predating this key reproduce bit-exact. Merge/final_cat still read the top-level cfis symlink (config_dir), untouched. Candidate upstream PR for the image-sims orchestration branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XSWPiisH6y7NFFbjHqthuC
…ctive Replace the hand-rolled ``apptainer exec`` prefix (``_EXEC_PREFIX`` / ``EXEC`` / ``EXEC_PIPELINE``) with Snakemake's own per-rule ``container:`` directive: ``sif_pipeline`` for the ShapePipe stages (im_pipeline, im_merge), ``sif`` for the sp_validation stages (im_manifest, im_extract, im_calibrate). The Snakefile drops ``container: None``; ``--software-deployment-method apptainer`` (set in the candide profile) turns the directive into an ``apptainer exec <image>`` wrap around each job. Bind mounts move to the profile's ``apptainer-args``; the three per-run env injections (PYTHONPATH shadowing the branch build, PSF_DICT, OMP_NUM_THREADS=1) are set once in ``os.environ`` and reach the container natively -- the slurm executor submits ``sbatch --export=ALL`` and apptainer inherits the host environment by default. Setting them in committed workflow code, not by hand in the launching shell, preserves the "one run command, no implicit uncommitted state" property. The old ``env -u SLURM_*`` scrub is dropped: the pipeline image's OpenMPI is built ``--with-pmix=internal`` with no SLURM RAS plugin (``ompi_info --param ras slurm`` reports the framework absent), so it sees a singleton regardless of host SLURM vars -- the scrub was defensive, not load-bearing. im_mbias is a ``run:`` block, which Snakemake runs in the driver process where ``container:`` does not reach, so its one ``shell()`` keeps an explicit ``apptainer exec`` (M_BIAS_EXEC), built from the same config for lockstep. No behaviour change intended: same two images, same env, same commands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ofile Add ``software-deployment-method: apptainer`` and ``apptainer-args`` (the static candide bind mounts) so Snakemake wraps every job carrying a ``container:`` directive in ``apptainer exec``. Rewrite the "deliberately NOT here" note: container settings now belong here (they drive the per-rule directives), while the per-run env injections stay in committed .smk code because they derive from the run config's repo/PSF paths, not static policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # workflow/image_sims/Snakefile # workflow/image_sims/config.yaml # workflow/profiles/candide/config.yaml # workflow/rules/image_sims.smk
ImageSimMBias.load_catalogs hardcoded the '_grid_' infix between branch
name and run number, so blended arms ({branch}_{num} dirs) hit
FileNotFoundError on a '{branch}_grid_{num}' path that never existed.
Derive the suffix from sims_type (defaulting to grid for backward
compat and the synthetic-recovery tests), mirroring the workflow's own
_SUFFIX contract. Emit sims_type into the generated m_bias_config, and
bump im_mbias mem to 8G since blended catalogues are ~5x grid size and
OOM the snakemake 1000M default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HubFhoU6bH8ij6hcTWxw7s
Collaborator
Author
|
Added 36dd99b: the m-bias step hardcoded a — Claude (Fable) on behalf of Cail |
…lization
Add an optional `exp_num` config key so the exposure input can be pulled from a
different SKiLLS realization than the tiles. Tiles stay on `{branch}_{num}`;
exposures resolve `{branch}_{exp_num}` via a new SIM_EXP_BASE mapping, consumed
by rule im_init's `sim_exp` symlink source. Unset `exp_num` defaults to `num`,
reproducing prior behaviour bit-for-bit. Guarded to non-grid sims (grid names
embed sims_type; the pathology this addresses is a blended-family concern).
Motivation: the blended-family realizations _1.._4 carry stale cross-linked
SP_exp (grid-sims exposures symlinked in, signal-free); _5 is the clean all-real
exposure set. This reproduces Fabian's production split (exposures from _5, tiles
from _1), unblocking the blended smoke arms whose ngmix flux collapsed ~10x on
the contaminated z-branch exposures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HubFhoU6bH8ij6hcTWxw7s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstreams the image-sims workflow evolution from the calibration campaign, plus a refactor to snakemake-native container handling. Net diff vs develop is small (4 files, +102/−60) — most campaign work was already in #225; this brings the remainder:
Campaign hardening
pipeline_config_dirconfig key: point the ShapePipe stage at an alternative config tree (per-run ngmix ini swaps for A/B arms) without touching the shared worktree; unset reproduces prior behaviour byte-for-byte.Snakemake-native containers
apptainer execshell prefixes replaced by per-rulecontainer:directives (ShapePipe stages → pipeline image; sp_validation stages → validation image) withsoftware-deployment-method: apptainer+apptainer-args(binds) in the candide profile.env -u SLURM_*scrub is dropped: verified non-load-bearing (the pipeline image's OpenMPI has no SLURM RAS plugin;MPI.COMM_WORLD.Get_size()= 1 regardless of the env).im_mbiasis arun:block (executes in the driver, wherecontainer:cannot reach) and keeps one explicitapptainer exec, noted in code.Verification: full single-tile smoke (233.293, 5 shear branches, 27 jobs) through the new mechanism on candide — m1/c1 bitwise-identical to the reference run; m2/c2 within cross-node realization noise.
This branch supersedes the deleted
feature/image-sims-orchestrationlineage (history preserved via merge).— Claude (Fable) on behalf of Cail
🤖 Generated with Claude Code
https://claude.ai/code/session_01HubFhoU6bH8ij6hcTWxw7s