Calibration fail-fast, pipeline_config_dir, jobs cap, response docs (imsims A/B campaign)#285
Draft
cailmdaley wants to merge 4 commits into
Draft
Calibration fail-fast, pipeline_config_dir, jobs cap, response docs (imsims A/B campaign)#285cailmdaley wants to merge 4 commits into
cailmdaley wants to merge 4 commits into
Conversation
The ShapePipe pipeline stage's ngmix ini (which owns METACAL_PSF) is resolved from the shapepipe worktree's example/cfis_image_sims via run_job's self-computed config_dir, so nothing in the run config could select a different ini -- an A/B on the metacal reconvolution PSF needed a separate worktree per arm. Add an optional operational key `pipeline_config_dir`. When set it is passed to run_job as `-c DIR`, redirecting the pipeline stage's whole ShapePipe config tree (ngmix ini included) at a per-arm copy; when empty (the default) no `-c` is passed and run_job self-computes exactly as before, so runs predating this key reproduce bit-exact. Merge/final_cat still read the top-level cfis symlink (config_dir), untouched. Candidate upstream PR for the image-sims orchestration branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XSWPiisH6y7NFFbjHqthuC
An empty no-shear selection (every object failing the flag/SNR/size cuts, typically because upstream shape measurement failed wholesale) previously propagated a 0-row catalogue all the way to get_w_des, where pandas died with the opaque "cannot set a frame with no defined index". Raise at the point emptiness is first knowable -- the end of metacal._masking_gal -- naming the actual condition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dA9sas4vvNBcNKRmi973L
The Snakefile docstring claimed no -j is needed because "the slurm executor sets the job cap"; snakemake 9.16.3 in fact refuses to run without an explicit --jobs. Pair the doc correction with jobs: 100 in the candide profile so the promise (no -j on the command line) becomes true again rather than merely corrected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dA9sas4vvNBcNKRmi973L
The design choice was implicit, spread across metacal.R construction, get_calibrate_e_from_cat's np.mean loop and get_quantities_binned's per-bin means; state it once on the function every calibration consumer flows through. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dA9sas4vvNBcNKRmi973L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #284
Four small fixes from the A/B characterization campaign (shapepipe#813), one commit each:
pipeline_config_dir(cherry-pick from the campaign branch): optional run-config key pointing the ShapePipe pipeline stage at a per-run config tree; unset ⇒ byte-exact prior behaviour. This is what let every config-only A/B arm run without forking the shared worktree.metacal._masking_galnow raises a named error ("0 objects survive the metacal selection … upstream shape measurement likely failed wholesale") instead of letting a 0-row frame die three layers down in pandas.jobs: 100, so the documented profile-only invocation works again (9.16.3 hard-requires a job cap).get_calibrated_quantitiesstating that a single run-averaged R is applied to every object (per-object responses exist but are only consumed as means).Reviewer call intentionally left open: none here — the debatable piece (RuntimeError on wholesale failure) lives in the shapepipe sibling PR.
— Claude (Fable), on behalf of Cail.
🤖 Generated with Claude Code
https://claude.ai/code/session_019dA9sas4vvNBcNKRmi973L