Skip to content

Migrate cosmo_val + Snakemake to the SACC writers #247

Description

@cailmdaley

Make SACC the sole native output format of the cosmology-validation pipeline. Every two-point product (ξ± coarse and fine, pseudo-Cℓ, COSEBIs, pure-E/B, ρ/τ, n(z)) is written through the sacc_io builders into single-statistic SACC part files, and the Snakemake workflow assembles those parts into one terminal SACC file per catalogue version, {version}.sacc. No bespoke writer remains on the data-vector path.

Desired end state

  • Each cosmo_val statistic's write step is replaced by a call to the matching sacc_io builder, producing a part file that holds exactly one statistic plus the self-describing n(z) tracers it references. A producer writes a part with a single save(s, path) call; it never assembles a terminal file and never emits a bespoke format alongside.
  • A dedicated Snakemake assembly rule reads the parts, concatenates them in canonical order, attaches the BlockDiagonal covariance (analysis blocks plus the dense per-pair fine block), and writes the single terminal {version}.sacc. The terminal file is a pure gather; anything that wants only one statistic targets the corresponding part rule, not a second terminal file.
  • Part filenames are DAG internals: consumers bind parts through the workflow's path helpers, never by hardcoding names. {version}.sacc is the only stable name.
  • Blinding rides the DAG here. Blinding is per part, at birth (PRD: SACC data format + Smokescreen blinding #241 §4; CLIs and the assembly-time assertion are delivered by Smokescreen blinding wiring (fork protocol, three theory backends, custody) #252): a blind-init rule fires once per catalogue version, each blindable part rule (coarse ξ±, fine ξ±, pseudo-Cℓ) is followed by blind-part so only blinded parts persist on disk, COSEBIs/pure-E/B are derived from the blinded fine ξ± (born blinded), and the terminal assembly asserts blind_commitment is identical across parts. This issue authors the rules; Smokescreen blinding wiring (fork protocol, three theory backends, custody) #252 supplies the machinery.
  • The bespoke .txt/custom-FITS/.npz writers on the data-vector path are deleted; nothing downstream reads them.

sacc_io is a fixed dependency and the single source of truth for every builder signature. Metadata keys required on every file: catalogue_version, sp_validation_version, created, npatch.

Acceptance

The workflow produces {version}.sacc per version from parts, with only blinded blindable parts persisted and the commitment assertion exercised; the deleted bespoke writers have no remaining readers; the assembled file matches the sacc_io layout contract.

— Fable on behalf of Cail.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions