feat(release): reproducible source archives, reproducibility checks, and ASF automated signing

[potiuk]

Summary

• The source artefact is now a reproducible export of the tag. A new stdlib-only tool, tools/reproducible-archive (repro-archive build | check | compare | recipe | epoch), wraps git archive (tracked files only, .gitattributes export-ignore honoured) and applies every rule from reproducible-builds.org § Archive metadata: one SOURCE_DATE_EPOCH mtime, sorted members, uid/gid 0, a=rX,u+w, no PAX atime/ctime, gzip -n, zip -X. Two builders of the same tag get byte-identical bytes regardless of their git/tar/zip version. release-rc-cut emits it as the source build; the single-file module can be copied verbatim into a CI workflow.
• Optional reproducibility checks, for source and binaries. release-build.md § Reproducibility checks turns on a rebuild-and-compare in release-rc-cut Step 2b (RM self-check before signing) and release-verify-rc Step 9 (every voter), with identical / content-identical / differs verdicts; binaries by byte-identical or documented-divergence. Motivation: Magpie's own 0.1.0-rc1 got a -1 for a non-reproducible zip -r artefact (PRINCIPLES.md § 11).
• First-release .gitattributes education step. release-prepare prep Step 2e walks the operator through every top-level path, classifies it, checks references before proposing an export-ignore, confirms each entry, lands .gitattributes in the prep PR and records export_ignore_reviewed; release-rc-cut blocks while the review is outstanding.
• 🪶 ASF-specific automated release signing (opt-in, gated on organization: ASF). release-prepare automated-signing drafts (never files/sends) the Infra key-request ticket, the Security Team notification and a reproducible-build workflow PR from projects/_template/workflows/release-candidate.yml (no key material, ATR OIDC upload), per Infra § Automated release signing. When enabled, verify-rc's reproducibility check becomes the mandatory bit-by-bit validation on trusted hardware and release-promote blocks without the attestation. Non-ASF adopters never see it.
• The record carries a SWHID next to the commit and the repository URL. repro-archive build prints swh:1:rev:<commit> and swh:1:dir:<tree> of the archive's expanded content (with origin= / anchor= qualifiers), computed in memory the way git computes a tree id and tested against git write-tree over the extracted tree, so it agrees with asfswhid and with what ATR computes at compose time; a note says whether it equals the repository tree (i.e. whether .gitattributes altered the export). check --swhid and swhid let a voter recompute and compare from the staged bytes; release-verify-rc reports swhid_matches; the [VOTE] record lists repository, commit, SWHID, epoch and sha512. Two more machine-dependent inputs are pinned as well: the builder's core.autocrlf / core.eol (which git archive would apply), and the commit id travels in the archive as git archive does it (global PAX comment / zip comment). Shape follows the discussion on apache/tooling-actions#37.
• Design doc. docs/designs/2026-09-20-reproducible-releases.md records the decisions, the rejected alternatives and the known limits (compression bytes across zlib builds, export-subst, the ATR SWHID exposure).
• Convenience artefacts are project-specific and config-declared; reproducibility decides whether one is good. The framework assumes nothing beyond the source. release-build.md § Convenience artefacts lists each artefact a project ships besides the source (binary tarball, wheels, jars, container image, chart, …) with its own build_command, staging / stage_command, reproducibility mode, vote_included flag and publish_channel / publish_command. release-rc-cut builds and stages them under the tag's SOURCE_DATE_EPOCH, release-verify-rc rebuilds and compares each one (a binary cannot be reviewed, only rebuilt — that check is what establishes it is what the voted source produces), release-vote-draft lists them with their vote scope, release-promote publishes only the ones that reproduced and emits a HOLD for any that did not (the source promotion is never held back), and release-announce-draft names their channels. release-verify-rc Step 7 now runs the project's own source_tree_validators instead of hard-coded framework tools.
• [VOTE] template tells PMC members how to verify. Every release-vote-draft body carries a How to verify this candidate section: the reproducibility record (commit, SOURCE_DATE_EPOCH, sha512), the agentic one-liner (/magpie-release-management:verify-rc <rc>), the human-readable verification page and reproducibility page at the RC tag, the ATR candidate link, and the voter-obligation sentence. New config keys vote_verification_doc_url, reproducibility_doc_url, vote_verification_skill; ATR runbook and the manual voter guide (with a reply template) updated to match.

Type of change

• Skill change (.claude/skills/<name>/) — eval fixtures updated below
• Tool / bridge contract (tools/<system>/*.md)
• Python package (tools/*/ with pyproject.toml)
• Groovy reference impl
• Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
• Documentation (docs/, README.md, CONTRIBUTING.md)
• Project template (projects/_template/)
• CI / dev loop (prek, workflows, validators)
• Other:

Test plan

• prek run --all-files passes
• For Python packages touched: uv run pytest / ruff check / mypy passes — reproducible-archive has 21 tests: every reproducible-builds.org rule has a positive case (the builder applies it) and a negative case (check catches a violation), compare on all three verdicts, and two builds at different wall-clock times under a different umask are byte-identical
• For Groovy bridges touched: command-line invocation tested end-to-end
• For skill changes: eval suite passes for the affected skill — run with claude -p: release-prepare 15/15, release-rc-cut 14/14, release-promote 8/8, release-vote-draft step-2 3/3, release-verify-rc 16/17 (the one failure is the untouched step-2-verify-signatures/case-2, where the semantic grader objected to the model picking downloads.apache.org over dist.apache.org for the KEYS URL — pre-existing grader sensitivity, not touched by this PR)
• For skill behaviour changes: a new or updated eval fixture is included in this PR — new suites release-rc-cut/step-2b-reproducibility (3) and release-verify-rc/step-9-reproducibility (5, incl. a convenience artefact whose rebuild differs); new cases for the unreviewed-archive block, the repro-archive build source command, the first-release archive review, the ASF-only automated-signing gate (blocked for non-ASF, proceeds for ASF), and release-promote publishing a reproduced wheel while holding a container image that did not reproduce; release-vote-draft step-2 now asserts the How to verify section via assertions.json, including the case that must omit the reproducibility record rather than invent it
• Other: tools/spec-loop/specs/release-management-lifecycle.md updated by hand for the new behaviour; .last-sync not bumped

RFC-AI-0004 compliance

• HITL — every new state change is a proposal: .gitattributes entries confirmed one by one, the Infra ticket and Security Team mail are drafts, the workflow PR opens as a draft, release-verify-rc never asserts trusted hardware on the committer's behalf (--trusted-hardware is theirs)
• Sandbox — repro-archive runs offline on the local clone; no new network reach
• Vendor neutrality — placeholders throughout; the ASF-specific option is resolved from organizations/ASF/organization.md → release_process.automated_signing (null in independent) and marked 🪶 wherever it appears
• Conversational + correctable — --allow-unreviewed-archive, --skip-repro-check, --skip-repro, --review-archive are the explicit, logged overrides; config keys default to today's behaviour for source_archive_method: custom
• Write-access discipline — no autonomous outbound messages; the agent holds neither the RM's key nor the CI key (spec § Boundary 1 extended to say so)
• Privacy LLM — n/a

Linked issues

Refs the 0.1.0-rc1 -1 recorded in docs/release-management/manual-release-process.md § Caveats.

Notes for reviewers (optional)

• A new tool-capability value, substrate:release, was added to docs/labels-and-capabilities.md and the validator's TOOL_CAPABILITIES, since none of the existing substrate:* values fit an adopter-facing release helper. Happy to fold it into an existing value if you prefer.
• The ATR upload action's exact inputs in projects/_template/workflows/release-candidate.yml are placeholders (<COMMIT_SHA>, # files:); the template says to confirm them against apache/tooling-actions when rendering. The workflow deliberately contains no signing step: the mechanism is whatever Infra agrees on the ticket.
• release-verify-rc renumbered its hand-back step from 9 to 10 to make room for the reproducibility step; no eval referenced the old number.
• Genericity audit: the only project-specific text found in the release skills was verify-rc Step 7's hard-coded Magpie validators and rc1 story, now replaced by the config-driven source_tree_validators (Magpie's own list moved to projects/magpie/release-build.md). Remaining "Magpie" mentions in the skills are references to the framework (skill names, the shared pre-flight block).

Gen-AI disclosure

Authored with Claude Code (Fable 5.1) driving the edits, tests and eval runs; reviewed and directed by the PR author.

🤖 Generated with Claude Code

[potiuk]

This one should make our ATR release process smooth :)