Smokescreen-fork blinding: per-part-at-birth blind, hash-commitment custody, CAMB↔CCL cross-check (PR 6)

[cailmdaley]

Closes #252.

Blinds the tomographic data vector per part, at birth, through the UNIONS-WL/Smokescreen fork: each blindable intermediate SACC product — reporting ξ±, integration ξ±, pseudo-Cℓ — is shifted by t(hidden) − t(fiducial) (Muir et al. 2019) the moment the pipeline computes it, with the hidden cosmology drawn inside a fixed amplitude envelope by the fork's own CCL-native, per-key-independent draw. One seed per catalogue version drives every part's draw, so all parts share one hidden cosmology by construction. Only blinded parts persist on disk; derived statistics (COSEBIs, pure-E/B) are computed downstream from the already-blinded integration ξ± by the pipeline's own estimators, so they are born blinded — there is no re-derivation step inside the blinding code. No one can read S8 off the data until the collaboration unblinds; both the blind and the unblind are verifiable without a trusted keyholder.

Stacks on #245 (sacc_io); depends on the fork protocol (ConcealDataVector(fiducial_params, shifts_dict, sacc_data, *, seed, theory_fn), pinned fork SHA 588a6b9). No firecrown anywhere in this PR's diff — the fork is the concealment engine, and every theory vector is a direct CCL computation.

Reconciliation with #245's canonical sacc_io

This branch was rebuilt to stack on feat/sacc-2-sacc-io (#245) rather than carry its own copy of sacc_io. src/sp_validation/sacc_io.py is now #245's canonical module verbatim plus one appended gather() (the terminal part assembly); everything else in the module is byte-identical to #245. The reconciled surface:

• Grid vocabulary is sacc_io: SACC read/write layer for the standard data-product layout #245's: grid='reporting' (analysis ξ±) / grid='integration' (COSEBIs/pure-EB input), swept from this branch's former coarse/fine everywhere (tags, filenames, docstrings, tests).
• save(s, path, *, type) stamps type='data'|'mock' in metadata (with a re-stamp guard); the blinding tooling passes each part's own provenance through.
• The fail-closed gate lives in sacc_io.load(), per the PRD's letter ("sacc_io fails closed on load"): load(path, *, allow_unblinded=False) raises on a type='data' file lacking concealed=True. The blinding/unblinding tooling — the two legitimate readers of an unblinded real vector — passes allow_unblinded=True explicitly (blind_part); loads of blinded files carry concealed=True and need no escape hatch. This branch's custody logic (commitment verification, escrow, seed math) stays in blinding.py; only the type/concealed load gate is sacc_io's.
• gather() builds on sacc_io: SACC read/write layer for the standard data-product layout #245's merge(): the tracer-union / ordered-point / block-diagonal-covariance assembly is merge, so gather delegates to it and adds only the one blind-aware thing merge cannot know — the assert_consistent_blind custody call and the shared-stamp write.

The three-verb surface

blind-init (once per catalogue version) — draws an OS-entropy seed, writes a repo-committable commitment.json (sha256(seed) + a canonical config digest binding the envelope and fiducial), and encrypts the seed into a Fernet bundle (smokescreen.encryption). The plaintext seed is never written. These outputs are the blind's fixed state; every other call reads them.

blind-part (per intermediate, at birth) — reads the encrypted seed (verifying both hashes against the commitment first), loads the plaintext part via sacc_io.load(..., allow_unblinded=True), conceals it through its matching theory_fn backend under the shared seed, writes the blinded part (type= inherited from the part) with the part's true vector escrowed into a per-part encrypted bundle beside it, and deletes the plaintext part. Each escrow is self-contained: restoring a part needs only that part's bundle plus the seed bundle — corruption of one bundle loses one part, not all. Blinded metadata: concealed=True, blind label, blind_commitment, blind_config_digest; seed_smokescreen stripped.

unblind (per part, or the assembled file via grid-tag selection) — verifies both sha256(seed) and the config digest against commitment.json before subtracting anything, recomputes the part's shift from the seed through the same backend, and subtracts — the seed-subtracted vector is the authority. When the part's escrow bundle is beside it and its stored seed_sha256 matches the commitment, the escrow serves two subordinate roles: a tighter equality check (fail closed if it disagrees) and clearing the add-then-subtract ulp residue so the restore is bit-for-bit. An escrow bound to a different seed, or one that disagrees materially, never determines the output.

Terminal assembly — sacc_io.gather assembles the standalone part SACCs into the one-file {version}.sacc (via #245's merge: tracer merge, ordered points with all tags including bandpower windows, block-diagonal covariance). Its single blind-aware touch is one call to blinding.assert_consistent_blind, which fails the gather closed unless every blindable part carries the identical blind_commitment (mixed blinded/plaintext assemblies also refuse), and stamps the shared commitment on the assembled file. ρ/τ diagnostic parts are exempt and pass through untouched.

What's in the box

sp_validation/blinding_theory.py — the blinding backend's theory surface. TheoryConfig (frozen dataclass; fail-fast overrides), the home of the blinding/inference fiducial (CosmoSIS v1.4.6.3 IA-fiducial values: S8=0.80, Ωm=0.30, h=0.70, n_s=0.96, Σm_ν=0.06 eV, HMCode2020+feedback with logT_AGN=7.5). Carries two halofit tokens (ccl_halofit_version, camb_halofit_version) naming the one recipe per stack; Neff=3.046/T_CMB=2.7255 are emitted explicitly to both stacks. Two independent ξ± paths live here: the CCL-native route (Boltzmann-CAMB P(k) → CCL Limber + FFTLog — the blinding recipe) and an independent pycamb P(k) → Pk2D → CCL projection with a closed-form σ8-matched A_s rescale. numpy-only at module scope.

No sp_validation/cosmology.py. develop deleted the local cosmology module (#223), moving cosmology to cs_util.cosmo, and this PR does not resurrect it. Per the UNIONS layering — cs_util = the cosmology library (generic machinery), sp_validation = configuration + survey-specific implementations — TheoryConfig (config) and the three part backends in blinding.py (reporting/integration ξ±, pseudo-Cℓ; survey-specific) are correctly homed in the blinding namespace. The generic pieces here (CCL ξ± path, CAMB P(k)→Pk2D path, σ8/A_s rescale) live in blinding_theory.py for now but carry an explicit module-docstring note that they are cosmology-library code destined for cs_util.cosmo (tracked in cs_util#80).

sp_validation/blinding.py —

• Envelope calibration: the intended (|ΔS8| ≤ 0.075, |ΔΩm| ≤ 0.1) box maps to {sigma8, Omega_c} half-widths at the fiducial (σ8 = S8/√(Ωm/0.3); Ω_c one-to-one). Half-widths are config, not code.
• Three theory_fn backends, one per part: reporting ξ±, integration ξ± (grid tags), and pseudo-Cℓ (W @ Cℓ_EE on the stored BandpowerWindows; ΔBB = ΔEB ≡ 0 — the shift is pure E-mode). Two-tracer cross-pairs; rows aligned to each part's own stored order, never an assumed pairing.
• Per-part conceal: extract the part's blindable block into a sub-SACC (exactly the rows the theory_fn spans — the fork's length guard enforces the agreement), drive ConcealDataVector, write the shifted values back at their recorded indices (row order preserved). This block carve/write-back is deliberately index-based (_extract_block/_set_values), not sacc_io: SACC read/write layer for the standard data-product layout #245's tag-matching extract()/update_statistic(): ConcealDataVector aligns its theory_fn output to the sub-SACC mean by row position, so the rows must be addressed by contiguous integer index. Covariance (including the integration part's dense per-pair block) and BB/EB rows are byte-identical to the input.
• Custody: blind_init / blind_part / unblind_part as above, plus assert_consistent_blind for the gather. A part with no blindable block (ρ/τ diagnostics) refuses a blind call loudly.

sp_validation/sacc_io.py — #245's canonical module verbatim + gather (terminal part assembly, one blind-aware call site as above).

sp_validation/b_modes.py — gains the values-only estimator seams cosebis_from_xi and pure_eb_from_xi, exposing the same cosmo_numba kernels calculate_cosebis / calculate_pure_eb_correlation drive to callers holding ξ± arrays (e.g. a blinded integration-ξ± SACC part) rather than TreeCorr correlations. This is where born-blinded derived statistics come from — the estimators stay in the pipeline's namespace, not the blinding module's.

scripts/blind_data_vector.py — blind-init / blind-part / unblind / verify CLI; blind-init refuses to overwrite an existing blind's state before drawing anything.

Tests (all observed in the shared-venv run on candide)

test_blinding.py covers the per-part acceptance criteria AC1–AC9; test_camb_ccl_crosscheck.py covers AC10–14. Heavy precision tests are @pytest.mark.slow; the derived-statistics tests importorskip("cosmo_numba"). The blinding suite + test_sacc_io.py on candide (this reconciled branch): 90 passed, 0 failed, 0 skipped (slow theory tests included; test_sacc_io.py + test_blinding.py + test_camb_ccl_crosscheck.py).

Observed numbers from the runs:

Quantity	Observed	Bound
AC2 on-file shift vs theory difference (all 3 parts, 2-bin; one hidden cosmology across parts asserted)	6.4e-14 rel	≤ 1e-10
AC3 realized shift vs independently written CCL reference	1.0e-20 abs (reporting ξ±)	< 1e-8
AC4 ΔBₙ (born-blinded COSEBIs) amplitude-independence	1.5e-10 of |ΔBₙ|	≤ 1e-9
AC4 leakage magnitude	ΔBₙ/Bₙ = 4.9e-6 (B=2e-5), 1.4e-5 (B=2e-6); Δξ⁺_B = 1.5e-9 = 0.08% of a 2e-6 B-mode	reported, not asserted
AC4 Δξ⁺_B amplitude-independence	8.7e-14 (5.8e-05 of the leakage)	≤ 5% of leakage
AC9 pure-E/B NaN pattern (born from blinded vs true parts)	identical (finite values shifted)	exact parity
AC10 nominal-A_s σ8 offset	3.4%	> 2% (the offset is real)
AC10 σ8-matched residual	1.1e-16	< 1e-4
AC11 ξ± floor at fiducial	ξ+ 0.24%, ξ− 0.10%	< 0.5% / 1.0%
AC12 ξ± floor off-fiducial (S8+0.075, Ωm−0.05)	ξ+ 0.28%, ξ− 0.12%	< 0.5% / 1.0%

The AC4/AC9 comparands run the pipeline's own downstream convention — measured reporting ξ± + integration grid + edge-based bounds, through the b_modes seams — on blinded vs true parts, so they check the born-blinded derivation path against the pipeline convention rather than any blinding-internal seam. AC2 is fork-draw recovery + placement (both sides share the backend; backend correctness is AC3's independently written reference). AC1 (zero-shift identity, per part + downstream derivation), AC5 (covariance/row-order/BB-EB/ρτ untouched), AC6 (custody: init writes only commitment + encrypted bundle; plaintext parts deleted; gather fails closed on mismatched or mixed blinds; unblind fails closed on either tampered hash), AC7 (same seed ⇒ identical shifts), and AC8 (end-to-end init → three blind-parts → gather → per-part bit-for-bit unblind → true derived statistics reproduced) are asserted exactly.

Not in this PR

Snakemake wiring — which pipeline rule invokes blind-part, where blind-init fires per catalogue version — belongs to the Snakemake-parts row; the per-part surface here is designed for that wiring, but the DAG edges are authored there. Covariance blinding, unblinding-criteria policy, and the fiducial-cosmology group decision are likewise out of scope (config values, not code).

Dependencies

The theory stack (sacc, the UNIONS-WL Smokescreen fork pinned by commit, cosmo_numba, cryptography, pyccl) is already declared as a core dependency on the feat/sacc-2-sacc-io base (inherited from develop), so this branch adds no dependency changes and no [blinding] extra — the earlier fork-pin duplication is gone now that the base carries the fork as core. The CI blinding-stack smoke test still imports the fork-only draw_param_shifts and the kernel modules (cosmo_numba.B_modes.cosebis.COSEBIS, schneider2022.get_pure_EB_modes), so an upstream Smokescreen package or a missing kernel cannot ship green.

Review-convergence additions (PRD #241 review, #280)

Boltzmann backend is one config knob (#280). TheoryConfig.transfer_function (default boltzmann_camb) threads through ccl_cosmology into all three theory backends, so blinding and the CosmoSIS+CAMB inference share one power-spectrum path. Halofit follows the choice — CAMB HMCode tokens only under boltzmann_camb, CCL's own halofit otherwise (documented as a cross-check tool, not a production setting) — and the backend is bound into the blinding config digest (a different P(k) path is a different blind). AC13 pins the shipped default against the inference stack; AC2 is parametrized over a non-default backend (eisenstein_hu) to prove the knob threads consistently.

Mock-only plaintext assembly (PRD §4, "Mocks vs data"). Terminal assembly is additionally gated on declared mocks: every unconcealed blindable part must carry metadata type: mock (the tag sacc_io.save stamps on #245) or the gather fails closed — an unconcealed data part, or one missing the tag, can never silently assemble, so skipping the blind cannot expose real data. Mixed concealed/plaintext assemblies fail closed regardless of type.

Closes #280.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WzUt7VbtXwr2SCHUdiQTyt