Skip to content

feat(release): add ASF source candidate workflow - #3278

Merged
Astro-Han merged 10 commits into
apache:mainfrom
M4n5ter:feat/asf-source-release
Aug 20, 2026
Merged

feat(release): add ASF source candidate workflow#3278
Astro-Han merged 10 commits into
apache:mainfrom
M4n5ter:feat/asf-source-release

Conversation

@M4n5ter

@M4n5ter M4n5ter commented Aug 20, 2026

Copy link
Copy Markdown
Member
English

Summary

  • Build Apache Maka Incubator source candidates from an immutable Git commit while isolating every Git subprocess from inherited repository, configuration, template, and attribute controls; archive permissions remain independent of ambient tar.umask.
  • Verify archive boundaries, required release documents, SHA-512, and detached signatures against a reviewed KEYS file without allowing ambient tar or gzip options to alter tool behavior.
  • Before signing, require Release Managers to reproduce the canonical uncompressed Git tar payload on trusted hardware. The downloaded workflow archive remains the exact artifact that is signed, so platform-specific gzip encodings cannot change source identity or block cross-platform signing.
  • Bind signing to a complete PGP fingerprint; require the actual signing key or subkey to be RSA with at least 2048 bits; create SHA-512 signatures; and accept only SHA-256, SHA-384, or SHA-512 signatures while failing closed on expired, revoked, bad, weak-digest, or missing signatures.
  • Give ASF source mechanics a dedicated required CI lane, independent from npm and Desktop convenience-artifact gates, and route it only from inputs that it actually verifies.
  • Document candidate tags, local signing, Apache dist/dev staging, every binding voter's verification duty, complete podling/IPMC vote thresholds, and immutable RC handling.

Refs #3269
Part of #2974

Verification

  • npm run check:asf-source — 9/9 passed, including hostile GIT_DIR/GIT_TEMPLATE_DIR/Git config/GZIP isolation, GNU TAR_OPTIONS bypass rejection, equivalent tar payloads with different gzip bytes, strong-digest enforcement, RSA signing-subkey acceptance, Ed25519/RSA-1024 rejection, and isolated-KEYS verification.

  • npm run check:release — 32/32 passed after separating the ASF source gate.

  • node --test --test-concurrency=1 scripts/ci-test-plan.test.mjs — 18/18 passed.

  • npm run typecheck

  • npm run lint

  • npm run format:check

  • actionlint v1.7.7 on the changed CI and candidate workflows.

  • An earlier npm test run completed the full build and most suites, but remained non-green on two known macOS environment cases: Eval expected /var/... instead of resolved /private/var/..., and one Runtime node-pty lifecycle test timed out after 10 seconds. Neither failing area is changed by this PR.

Remaining validation

  • Obtain independent human review of the release policy and Release Manager runbook.
  • Run Prepare ASF source candidate successfully from a reviewed commit on main.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Codex prepared and reviewed the source-candidate automation, trust-boundary tests, workflows, and Release Manager runbook under human direction.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, build, and affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No
简体中文

概要

  • 从不可变 Git commit 构建 Apache Maka Incubator 源码候选包,同时隔离每个 Git 子进程继承的仓库、配置、模板与 attributes 控制;归档权限仍不受本机 tar.umask 影响。
  • 验证归档边界、必需发版文件、SHA-512,以及基于已审查 KEYS 文件的 detached signature,并阻止本机 tar 或 gzip 选项改变工具行为。
  • 签名前要求 Release Manager 在受信硬件上重建并逐字节匹配规范的未压缩 Git tar payload;最终签署的仍是 workflow 下载到的原始归档,因此不同平台的 gzip 编码不会改变源码身份或阻断跨平台签名。
  • 将签名绑定到完整 PGP 指纹,要求实际签名 key 或 subkey 至少为 2048-bit RSA,生成 SHA-512 签名,并只接受 SHA-256、SHA-384 或 SHA-512;对过期、撤销、错误、弱 digest 或缺失签名 fail closed。
  • 为 ASF 源码发版机制建立独立 required CI lane,与 npm 和 Desktop convenience artifact gate 解耦,并只由它实际验证的输入触发。
  • 记录 candidate tag、本地签名、Apache dist/dev 暂存、每位 binding voter 的验包义务、完整 podling/IPMC 投票门槛及不可变 RC 处理流程。

关联 #3269
#2974 的一部分

验证

  • npm run check:asf-source——9/9 通过,覆盖恶意 GIT_DIR/GIT_TEMPLATE_DIR/Git config/GZIP 隔离、GNU TAR_OPTIONS 绕过拒绝、不同 gzip 字节下的等价 tar payload、强 digest 强制、RSA signing subkey 正向路径、Ed25519/RSA-1024 拒绝及隔离 KEYS 验签。

  • npm run check:release——ASF source gate 拆分后 32/32 通过。

  • node --test --test-concurrency=1 scripts/ci-test-plan.test.mjs——18/18 通过。

  • npm run typecheck

  • npm run lint

  • npm run format:check

  • 使用 actionlint v1.7.7 检查变更的 CI 与候选包 workflow。

  • 较早的一次 npm test 完成全量构建及绝大多数 suite,但仍有两个已知 macOS 环境问题:Eval 预期 /var/...,实际得到 /private/var/...;另有一个 Runtime node-pty 生命周期测试在 10 秒后超时。这两个失败区域都未被本 PR 修改。

剩余验证

  • 对发版政策和 Release Manager runbook 完成独立人工审查。
  • main 上经过审查的 commit 成功运行 Prepare ASF source candidate

AI 使用

二选一:

  • 没有生成式工具做出实质性贡献
  • 生成式工具做出了实质性贡献

工具及范围:Codex 在人工指导下准备并审查了源码候选包自动化、信任边界测试、workflow 和 Release Manager runbook。

检查清单

  • 测试覆盖此变更,并且缺少此变更时会失败
  • lint、format、build 和受影响的测试套件均在本地通过

此 PR 是否导致行为变化?

  • 是——已在上述概要中说明

Build source candidates from immutable commits while keeping Release Manager signing credentials outside CI. Add archive verification, reproducibility coverage, and the staging and voting runbook needed for an Incubator release.

Generated-by: Codex
Bind local signing to a byte-for-byte rebuild from the reviewed revision, require exact signing-key fingerprints, and interpret GPG status records so expired or revoked signatures fail closed. Isolate source mechanics in their own required CI lane, make tar access portable across Windows paths, and keep package and lockfile release identities aligned. The runbook now reflects the trusted-hardware handoff and every binding voter's verification duty.
Make source archives independent of ambient Git permission settings and enforce the ASF RSA key policy against the actual signing key or subkey on both signing and verification paths. Complete the two-stage Incubator vote contract and remove unreachable CI routing entries so the release authority stays explicit.
Compare the canonical Git tar payload instead of platform-specific gzip bytes so Release Managers can reproduce workflow candidates across trusted machines. Isolate archive attributes from local Git state and enforce one SHA-2/RSA signing policy for both local and KEYS verification.
Source candidate creation and verification must derive identity only from explicit repository and artifact inputs. Strip inherited Git, tar, and gzip control variables so trusted-checkout reproduction and archive policy cannot be redirected by the caller environment.

Keep Release Manager guidance authoritative by linking the exact-revision runbook from the workflow handoff instead of duplicating policy prose.
Publish verified source artifacts atomically from destination-local staging so interrupted or concurrent writers cannot expose partial release files. Authenticate signed candidates before parsing, stream payload digests, rebind the checksum during signing, and isolate GNU tar from ambient gzip options.

Remove the workflow's redundant verification pass and retain only release-identity assertions and regression coverage that protect observable policy boundaries.
@M4n5ter
M4n5ter force-pushed the feat/asf-source-release branch from 2ecca1a to 8d14f5c Compare August 20, 2026 07:29
Verify candidate bytes from a private snapshot so signature, checksum, and archive checks cannot drift across path replacement. Keep unsigned workflow artifacts commit-addressed and defer RC identity until immutable tagging and staging.

Remove the duplicate workflow version grammar and unreachable CI planner entries while preserving their fail-closed behavior.
The documented release path operates on completed local artifacts without a concurrent writer. Remove snapshot copies and mutation-only rebinding tests while retaining reproducibility, signature policy, and atomic publication guarantees.\n\nAlso narrow unused CLI and result contracts and remove duplicated workflow and archive-validation state.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at ac36d4fc. No P0 or P1. The core mechanics here are unusually solid, and I verified the load-bearing ones by building the candidate rather than by reading: in a throwaway worktree I ran create, verify, reproduce, and check:asf-source (9/9), then diffed the extracted tarball against a clean checkout of the same commit.

What that established, so it does not have to be re-litigated during the vote:

  • The archive is deterministic. uid/gid 0, uname/gname root, a single mtime from the commit, modes forced by -c tar.umask=0002 regardless of ambient umask, and gzip -n -9 so nothing leaks into the gzip header. The longest tracked path splits cleanly into ustar prefix+name, so no per-entry pax headers appear; git's pax_global_header is in the payload but is not listed by tar -t and so does not trip validateArchiveEntries.
  • The contents are exactly the committed tree minus the three export-ignore pathsdiff -rq differs only by .claude/, .maka-shots/, and maka-proposal-zh-review.txt. No binaries beyond five PNGs (README hero images and desktop icons), no symlinks, no gitlinks, no node_modules, no .git.
  • The forbiddenSegments backstop is live, not decorative. Building from the pre-PR base commit, whose .gitattributes lacks the export-ignore lines, fails with Forbidden archive entry: …/.claude/.
  • The signature covers the published bytes. signSourceCandidate signs archivePath, verifySha512File hashes the same file, and publishSourceCandidate hardlinks it — same inode throughout.
  • The secret surface is genuinely clean. Zero secrets.* references; workflow_dispatch only, so no fork-reachable path; permissions: contents: read; persist-credentials: false; all three third-party actions pinned to full commit SHAs; and every ${{ }} reaching a run: goes through a job-level env: and is dereferenced as a quoted shell variable. No private key ever enters CI — signing is local by design.
  • The GPG status parsing is correct: VALIDSIG field indices, rejection of EXPKEYSIG/REVKEYSIG/BADSIG/NO_PUBKEY/NODATA before the GOODSIG count, a hash allowlist of exactly SHA-256/384/512, and subkey fingerprint resolution. Verification runs in a mkdtemp homedir seeded only from the reviewed KEYS file, so an unlisted signer yields NO_PUBKEY.
  • The ci-test-plan.mjs edit is not a coverage regression. Dropping five entries from CLI_PACKAGE_FILES looked like one; running planTests on each shows all five still resolve to full: true through the unknown-root-file fallback.

On the seam question: this extends rather than parallels. No existing script produces a checksum file or touches GPG, the ASF source tarball is a different artifact with a different authority from the npm convenience tarball and the DMG, and asf-source-workflow-policy.test.mjs mirrors the existing release-cli-workflow-policy.test.mjs convention. Splitting check:asf-source out of check:release is right — the authoritative artifact should not be gated on Desktop and npm convenience checks. The one genuine overlap is the npm audit step, inline.

Two of the findings below are process decisions rather than defects and need a named human owner rather than a code change: the candidate always builds main's tip at dispatch (deliberate — a test asserts the absence of a SHA input), and NOTICE is not yet in ASF form (deliberate — DISCLAIMER-WIP says so). Both are legitimate WIP postures; the risk is that the PPMC discovers them mid-vote rather than deciding them beforehand.

Reviewed with Claude Opus as an analysis assistant. Findings marked "reproduced by execution" were built and run in an isolated worktree that has been removed; the root checkout was not modified and no repository-wide tests were run. Evidence grade is stated per finding.

Comment thread .github/workflows/asf-source-candidate.yml
Comment thread .github/ASF_SOURCE_RELEASE.md Outdated
Comment thread .github/workflows/asf-source-candidate.yml
Comment thread .github/workflows/asf-source-candidate.yml
Comment thread scripts/asf-source-release.mjs
Comment thread .github/workflows/asf-source-candidate.yml
Comment thread scripts/asf-source-release.mjs
Comment thread scripts/asf-source-release.mjs
Comment thread .github/ASF_SOURCE_RELEASE.md
Comment thread .github/ASF_SOURCE_RELEASE.md
Validate generated attribution inventories from the extracted source candidate so the vote-time gate covers the exact archived payload. Clarify that release keys are appended to the reviewed KEYS history rather than replacing it.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 251728b. Two of my P2s from ac36d4f are resolved and I verified both:

  • Attribution inventories — the lane now runs check:third-party-notices and check:cli-third-party-notices against the extracted source, after a full npm ci, with asf-source-workflow-policy.test.mjs asserting the workflow still contains them. Both scripts exist in the archived package.json and need only devDependencies, which the preceding step installs. Correct fix, and pinning it with a policy assertion is the right instinct.
  • KEYS truncation> KEYS is now >> KEYS and the runbook starts from a reviewed copy of the published file. (gpg --list-sigs <fpr> && gpg --armor --export <fpr>) >> KEYS produces the conventional ASF KEYS block and cannot destroy existing keys.

One follow-up on the inventory fix below, and my remaining threads from the previous head carry over unchanged — the dispatch-time $GITHUB_SHA resolution, npm audit against the live advisory feed on the release path, NOTICE being checked for existence rather than content, the workflow_dispatch first-run shakeout, and the four P3s. None of them are new, so I have not re-posted them.

Still COMMENT rather than approve: the open items are all P2, none of them blocks a release by itself, but this is the lane that decides what an IPMC vote sees, and I would rather the RM path be settled before it becomes the default.

AI disclosure: this review was produced with Claude Code (Opus 5). For this pass I read the delta commit, the workflow and runbook at 251728b, asf-source-release.mjs's archive boundary set, .gitattributes, and the notice-generation scripts and their call sites; I did not run the workflow or the tests. Per AGENTS.md this is not independent human review.

Comment thread .github/workflows/asf-source-candidate.yml
Check the Windows sandbox Cargo attribution inventory from the extracted source payload so its committed notices cannot drift from the archived lockfile. Keep the workflow-policy test aligned with all shipped attribution inventories.
@M4n5ter
M4n5ter requested a review from Astro-Han August 20, 2026 10:36

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cargo-inventory P2 is fixed at 5d52e8f, including the part I care about most — extending the asf-source-workflow-policy.test.mjs assertion so the lane cannot lose an inventory again without a test noticing.

I checked the one thing that could have made my own suggestion wrong: this job is runs-on: ubuntu-24.04 with no Rust setup step, so check:windows-cargo-notices depends on the runner image's preinstalled toolchain. That is already this repository's established assumption — windows-sandbox-w0.yml runs cargo build --locked and cargo test --locked on windows-2025 with no setup step either — and experiments/windows-sandbox/launcher/{Cargo.toml,Cargo.lock} are neither in forbiddenSegments nor export-ignored, so they are present in the extracted candidate. It should hold. Confirmed by reading the workflows and the archive boundary set; not executed, so the first real run is still the proof.

Worth distinguishing from my open npm audit thread, since they look similar: cargo metadata --locked resolves from the committed lockfile and is deterministic, so this does not add the kind of time-varying external dependency that one is about.

My remaining threads from the previous heads carry over unchanged — the dispatch-time $GITHUB_SHA resolution, npm audit against the live advisory feed on the release path, NOTICE being checked for existence rather than content, the workflow_dispatch first-run shakeout, and the four P3s. Nothing new from me on this head, so I have not re-posted them.

Still COMMENT rather than approve: everything open is P2 or below and none of it blocks a release on its own, but this lane decides what an IPMC vote sees, and I would rather the Release Manager path be settled before it becomes the default.

AI disclosure: this review was produced with Claude Code (Opus 5). I diffed 251728b..5d52e8f and read the workflow, the sandbox workflow and the archive boundary code at this head; I did not run the workflow or the tests. Per AGENTS.md this is not independent human review.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 5d52e8f, and correcting my own bar while I do it.

I withheld approval on the previous two heads on the grounds that this lane decides what an IPMC vote sees, so I wanted the Release Manager path settled first. That was a standard I invented for this PR rather than the one I apply everywhere else: P0 and P1 block, P2 should be fixed but does not. Everything still open here is P2 or P3, and holding a release-mechanics PR to a stricter rule than the product PRs I approved today is not defensible. My mistake, and it cost you two extra rounds.

What actually landed across those rounds: the attribution inventories are verified inside the extracted candidate for all three sources now — npm, CLI and cargo — each pinned by an assertion in asf-source-workflow-policy.test.mjs so the lane cannot quietly lose one; and the KEYS runbook no longer truncates the published file. All of my review threads are resolved.

The four P2s I am approving over, so they are on the record rather than forgotten: the candidate commit resolves at dispatch time rather than being named or asserted; npm audit puts a time-varying external feed on the release path; NOTICE is checked for existence rather than content; and workflow_dispatch only becomes available once this is on the default branch, so the first real run is the first syntax check. None of them blocks a release on its own, and the last one cannot be resolved before merging by construction.

Note on CI: package is failing at "Verify automatic update end to end", which is the Windows autoupdate lane. This PR's only workflow change adds an asf_source gate and one step to the test job and does not touch package; that lane is the one #3241 and #3265 are currently fixing. test and audit are green.

AI disclosure: this approval was produced with Claude Code (Opus 5). Across the three heads I read the workflow, the runbook, asf-source-release.mjs's archive boundary, .gitattributes, the notice-generation scripts and their call sites, and confirmed the failing job's step. I have not run the workflow. Per AGENTS.md this does not count as the independent human review this PR still needs.

@Astro-Han
Astro-Han merged commit 84fbe05 into apache:main Aug 20, 2026
2 of 3 checks passed
@M4n5ter
M4n5ter deleted the feat/asf-source-release branch August 20, 2026 11:46
Astro-Han added a commit that referenced this pull request Aug 20, 2026
…pendencies

A second round of adversarial review found the previous revision had
replaced one overreach with another, and that both rounds had missed
shipped dependencies a name-based scan does not surface.

The previous revision corrected "the EAR requires notification" but then
asserted as established fact that Maka is entirely standard cryptography.
That does not follow. 772.1 reaches proprietary or unpublished
cryptographic protocols, not only algorithms, so an enumeration of
algorithm names cannot settle it. Retract the claim and name the three
constructions that need a protocol-level determination.

Two shipped dependencies were absent. dugite is a direct root dependency
that bundles a complete 141 MB Git distribution into the desktop app,
carrying the .NET cryptography stack, OpenSSL native libraries, and TLS
transport; it is the largest third-party crypto payload in the product.
@jackwener/opencli is a direct desktop dependency carrying vendor-private
API signing with hardcoded key material against undocumented endpoints,
which is the clearest input to the 772.1 question. Neither is JavaScript
crypto a package-name scan would find, which is why the coverage limit
now says so instead of claiming completeness.

Move the snapshot to current main. The prior baseline predated #3278 by
28 minutes, which had already defined the source artifact: git archive
with only three export-ignore entries, so experiments/ and
packages/eval/harbor/ are inside it, and the CA-generation path with
them. That resolves one open question and adds GPG detached signing as a
new surface.

Also record the third lockfile, qualify the SSH finding for the Windows
MinGit payload, correct a PowerShell call mislabelled as DPAPI-backed,
and split manufacturer into which item an artifact contains versus who
built it.

Refs #3273

Generated-by: Claude Code (claude-opus-5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants