Skip to content

chore: add ASF source headers and a release header audit gate - #3397

Open
Astro-Han wants to merge 2 commits into
mainfrom
chore/asf-headers-rat
Open

chore: add ASF source headers and a release header audit gate#3397
Astro-Han wants to merge 2 commits into
mainfrom
chore/asf-headers-rat

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Applies the ASF source-header policy and enforces it with an automated audit of the source release candidate.

Nothing in the repository decided which files the header policy covers, and nothing checked compliance. scripts/asf-license-headers.mjs is now that decision. It classifies every file exactly once:

  • covered — the file type carries the ASF header; missing it fails the audit;
  • excluded — the file matches one reviewed exclusion rule that records why the header does not belong there;
  • unclassified — neither, which fails the audit.

Unclassified is the design. A new file extension or an unexpected path cannot reach a release candidate until someone writes down which of the two it is, so the exclusion list stays a list of reasons rather than a list of paths that happened to be noisy. The audit also fails when an exclusion rule matches nothing in a checkout, so the reviewed list cannot accumulate dead rules. .github/ASF_SOURCE_HEADERS.md documents the policy; npm run check:asf-headers -- --report prints the resolved file list behind every rule.

The same module applies the policy via npm run write:asf-headers, so the gate and the sweep cannot disagree about the header text or the covered set.

Why not Apache RAT

The issue allows "Apache RAT or an equivalent mentor-approved audit". This PR implements the equivalent, for three reasons:

  1. The audit must be reproducible from the archive. The release gate runs inside the extracted apache-maka-<version>-incubating-src.tar.gz. RAT would put a JVM and a downloaded jar into the path a voter is expected to reproduce; this audit needs nothing beyond the Node the archive already requires.
  2. RAT's exclusions fail open. They are globs: a new, unexpected file inside an excluded directory is approved silently. This audit fails closed, which is what "fail on an unexpected unapproved file" asks for.
  3. One seam, not a parallel path. The repository already gates releases with scripts/*.mjs regenerate-and-diff checks wired into check: npm scripts and run from the extracted candidate. This extends that seam.

Mentors who prefer literal RAT output should say so on the issue — the policy data would carry over to a rat-excludes file, but the fail-closed property would not.

Reviewed exclusions

Rule Files Why
not-in-source-release 10 export-ignore keeps them out of the archive
asf-release-documents 3 LICENSE, NOTICE, DISCLAIMER-WIP are the license and notice themselves
third-party-license-texts 12 verbatim upstream texts, digest-verified by the executor preparation scripts
third-party-source 30 third-party work under its own license, and diffs against third-party sources
generated-files 9 byte-compared against a fresh generator run; the generators carry the header
verbatim-runtime-payloads 2 bundled skill payloads are digest-pinned and delivered to the model as instructions
verbatim-github-templates 1 GitHub copies the PR template into every PR description
byte-significant-fixtures 6 recorded inputs and captured historical state
no-comment-syntax 38 JSON and CSV cannot carry a comment
binary-files 5 binary content
no-creative-content 8 VCS metadata and platform manifests

Third-party license classification stays with #3270 / G3; this gate only decides where an ASF header may be asserted.

Two commits, on purpose

  • 6c7445d — policy, audit, CI wiring, and release gate. On its own the audit reports every existing source file as missing a header, rather than passing silently.
  • c6f3418 — the sweep: 2,591 files, purely mechanical. Each changed file is exactly its previous content with the header inserted at the top, below a shebang, an HTML doctype, or Markdown front matter where one has to open the file. Verified programmatically: for every changed file, applyHeader(git show HEAD:<path>) === <working copy>.

.git-blame-ignore-revs records the sweep. Following the convention that file documents, this PR adds the placeholder entry and the landed squash hash goes in a follow-up immediately after merge — a hash that never reaches main makes git blame fail outright.

Merging this will require open pull requests to rebase. With ~60 open PRs that is a real cost, but it is paid once; splitting the sweep by directory would multiply the rebases rather than avoid them.

DISCLAIMER-WIP no longer discloses that source files lack Apache license headers. The NOTICE and software-grant/ICLA disclosures are unchanged and still open.

Refs #3271

Exit conditions satisfied

  • Define the file types, generated files, third-party files, fixtures, and other justified exclusions covered by the header policy — scripts/asf-license-headers.mjs and .github/ASF_SOURCE_HEADERS.md.
  • Add ASF license headers to applicable tracked source files — 2,591 files. .asf.yaml already carried the header and is untouched.
  • Add Apache RAT or an equivalent mentor-approved audit with a reviewed exclusion list — pending mentor confirmation of the equivalence argument above.
  • Run the audit against the extracted source archive, not only the Git checkout — Prepare ASF source candidate runs it inside candidate-source before npm ci.
  • Make the release workflow fail on an unexpected unapproved file.
  • Update DISCLAIMER-WIP when its missing-header disclosure is no longer true.
  • Record the mechanical header-only change in .git-blame-ignore-revs — placeholder added; the hash follows the squash-merge per the file's own convention.

Verification

Run in a clean worktree at c6f3418, rebased onto e955575:

  • npm run check:asf-headers — 2,592 covered, 124 excluded, no unclassified file, no entangled header.
  • Against the extracted archive, which is what the gate actually does: built apache-maka-0.1.11-incubating-src.tar.gz with npm run release:asf:source, extracted it, and ran the audit inside it at c6f3418 — 2,592 covered, 114 excluded (the 10 export-ignore files are correctly absent), exit 0.
  • Negative tests in that extracted archive: an unexpected .kt file → reported unclassified, exit 1; a header removed from packages/core/src/settings.ts → reported missing, exit 1.
  • npm run check:asf-source — 28 pass (includes the new asf-license-headers.test.mjs and the workflow-policy assertion that the audit precedes npm ci).
  • npm run lint, npm run format:check, npm run build, npm run typecheck — pass.
  • npm run check:stale, npm run astryx:theme -- --check, npm run astryx:surface-inventory, npm run windows:inventory, npm run check:third-party-notices, npm run check:cli-third-party-notices, npm run check:windows-cargo-notices, npx knip (desktop and ui) — pass. astryx:theme caught a generated file the first pass had missed (maka.js); it is now in the generated-files rule. check:stale needed one clean && build of @maka/ui first — a stale incremental tsbuildinfo, not a code problem.
  • npm --workspace @maka/desktop run test:dist — 1017 pass, 0 fail. This is the surface whose source-contract tests read .ts/.tsx bytes, so it is the one the sweep could plausibly have broken.
  • npm --workspace @maka/core run test:dist — 579 pass, 0 fail.

Not run locally: the remaining workspace suites, Electron e2e, Storybook, and the Windows lanes. CI covers them.

Review focus

The exclusion list is the part that needs human judgement, not the header insertion. npm run check:asf-headers -- --report prints every rule with its justification and the exact files it claims — one screen, and the part of this PR where being wrong has release consequences.

The sweep does not need to be read file by file. Its claim is that every changed file equals its parent with the canonical header applied, and that claim is checkable:

// against c6f3418
applyHeader(execFileSync('git', ['show', `HEAD~1:${path}`], ), style) === readFileSync(path, 'utf8')

That holds for all 2,591 files, and no file in the tree contains the license text more than once.

Addressed from review

@jackwener's automated review at d3e0eef found a real blocker: .asf.yaml already carried an ASF header whose license-URL indentation differed from the rendering, hasHeader matched byte-exactly, and the sweep prepended a second license block that the audit then accepted. Fixed at the class level rather than for the one file:

  • classifyExistingHeader recognizes an existing header by its normalized prose, independent of comment syntax, indentation, and line wrapping. A formatting variant is replaced with the canonical rendering, never doubled.
  • A leading comment that mixes the ASF text with other content — someone else's copyright line, say — is reported as entangled; write refuses and the audit fails. Separating those is a decision, not a rewrite.
  • Six regression tests cover the variant, the block-comment variant, the entangled case, an unrelated leading comment, and idempotence after replacement.
  • The canonical rendering now uses the license-URL indentation that .asf.yaml already used, which is also the common ASF form, so that file is untouched by this PR.
  • Verified across the whole tree: no file contains the license text more than once.

A repository-wide scan found .asf.yaml was the only pre-existing header.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code (Opus) drafted the policy module, its tests, the CI and release-workflow wiring, the policy document, and ran the sweep and the verification above. Both commits carry Generated-by: Claude Code. The exclusion rules and their justifications are AI-drafted and need a human contributor of record to confirm them; licensing conclusions remain with the PPMC, mentors, and IPMC.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@jackwener jackwener left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review of exact head d3e0eefd1f8934d1e1ac94a1e7b3bbafcc634b69.

Blocking correctness finding: the mechanical sweep prepends a second ASF header to .asf.yaml. The parent already begins with the standard ASF header, but its license-URL indentation is four spaces while renderHeader("hash") emits two. Because hasHeader accepts only byte-exact rendered text, applyHeader treats the existing valid header as absent and adds another; the resulting file has two consecutive license blocks. The new audit then passes because it sees its exact block at offset zero, so neither the gate nor the tests catch the duplication.

Please make application recognize/replace an existing standard ASF header (or otherwise prevent duplicate semantic headers), clean the duplicated .asf.yaml header, and add a regression covering this pre-existing whitespace variant. This matters beyond one file: a future valid formatting variant would otherwise be silently doubled again.

Merge readiness: not ready. The PR is draft, required CI is still running, independent licensing/mentor review is required, and the duplicate-header blocker must be resolved on a new exact head.

@Astro-Han
Astro-Han force-pushed the chore/asf-headers-rat branch 2 times, most recently from c6f3418 to f1d7d83 Compare August 21, 2026 09:45
@Astro-Han

Copy link
Copy Markdown
Contributor Author

Thanks — good catch, and the failure mode was worse than the one file: my own mechanical-purity check proved applyHeader(parent) === child, which is true and completely blind to the parent already carrying an equivalent header.

Fixed at f1d7d830 at the class level rather than for .asf.yaml alone:

  • Header detection now matches on normalized prose, with comment syntax, indentation, and line wrapping removed, so a formatting variant is recognized and replaced with the canonical rendering instead of doubled.
  • A leading comment that carries the ASF text together with other content is reported as entangled: write refuses and the audit fails, since separating that is a decision rather than a rewrite.
  • Six regression tests cover the indentation variant, a block-comment variant, the entangled case, an unrelated leading comment, and idempotence after replacement.
  • The canonical rendering now uses the license-URL indentation .asf.yaml already had, so that file is untouched by this PR.

Two checks added because the original one was not enough: a scan of the pre-sweep tree confirming .asf.yaml was the only pre-existing header, and a whole-tree assertion that no file contains the license text more than once.

Agreed on merge readiness. The package job in Release Windows check fails on main itself with the same signature and is being fixed in #3382; the required test check is green here.

@Astro-Han
Astro-Han marked this pull request as ready for review August 21, 2026 10:05
The first Apache Maka (Incubating) source release has to apply the ASF header
policy and enforce it, and nothing in the repository decided which files the
policy covers or checked that they comply.

`scripts/asf-license-headers.mjs` is that decision. It classifies every file
exactly once: covered by a comment syntax and required to carry the ASF header,
or matched by one reviewed exclusion rule that records why the header does not
belong there. A file that is neither fails the audit, so a new file type or an
unexpected path cannot reach a release candidate until someone writes down
which of the two it is. The exclusions cover the ASF release documents
themselves, third-party work kept under its own license, generated files that a
`check:` script byte-compares against a fresh run, payloads republished
verbatim as prompt text or as a pull request body, byte-significant fixtures,
formats with no comment syntax, binaries, and version-control metadata.

The same module applies the policy through `npm run write:asf-headers`, so the
gate and the sweep cannot disagree about the header text or the covered set.

`Prepare ASF source candidate` runs the audit inside the extracted archive
before anything installs or builds into that tree, so the release gate reads
the exact bytes a voter downloads instead of a checkout that resembles them. CI
runs it unconditionally beside the other install-free checks, because a header
gate that only sees the affected surfaces is not a gate.

Apache RAT was the alternative. It would put a JVM and a downloaded jar into
the audit path that reviewers are expected to reproduce, and its glob
exclusions fail open: a new unexpected file inside an excluded directory is
approved silently. This audit needs no toolchain beyond the Node the archive
already requires, and an unexpected file fails it.

The headers themselves are not in this commit. Until the sweep lands, the audit
reports every existing source file as missing one.

Refs #3271

Generated-by: Claude Code
`npm run write:asf-headers` produced this commit, and `npm run
check:asf-headers` now passes on the checkout and on the extracted source
archive. Nothing here was hand-edited: every changed file is exactly its
previous content with the standard ASF header inserted at the top, below an
interpreter shebang, an HTML doctype, or Markdown front matter where one of
those has to open the file. `.asf.yaml`, which already carried the header, is
untouched.

`DISCLAIMER-WIP` no longer discloses that source files lack Apache license
headers, because they no longer do. The remaining disclosures about `NOTICE`
and about the software grant and ICLAs are unchanged and still open.

`.git-blame-ignore-revs` gets an entry for this sweep. Following the
convention the file documents, the placeholder records the landed hash in a
follow-up right after the squash-merge, since a hash that never reaches main
makes `git blame` fail outright.

Merging this will require open pull requests to rebase. That is the cost of
doing the sweep once; splitting it per directory would multiply the rebases
instead of avoiding them.

Refs #3271

Generated-by: Claude Code
@Astro-Han
Astro-Han force-pushed the chore/asf-headers-rat branch from f1d7d83 to 0f97abe Compare August 21, 2026 10:33

@M4n5ter M4n5ter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

English

Reviewed exact head 0f97abe69fca7b14fab6e0e057e59ae1ded73f12.

The mechanical sweep is internally consistent, the focused 35 tests pass, and all four CI checks are green. However, I found four policy-level gaps that should be addressed before this becomes the repository's release gate.

1. [Blocking] Keep known mixed-origin files out of the mechanical ASF-header sweep

The extension fallback adds whole-file ASF headers to:

  • packages/runtime/src/tool-output.ts
  • packages/runtime/src/edit-replace.ts
  • packages/runtime/src/model-protocol.ts

The first two explicitly state that they adapt opencode code; #3325 records the pinned MIT sources and describes them as mixed-origin. #2907 similarly records the Vercel AI SDK material adapted in model-protocol.ts.

The ASF source-header policy defines third-party work to include parts of a file not submitted by their copyright owner, says not to mechanically add the standard ASF header to third-party source, and leaves major modifications to a case-by-case PMC decision.

Please classify these paths as third-party/mixed-source exclusions for now and preserve their upstream attribution. If the PPMC later chooses a combined header for a heavily modified file, that decision can be applied explicitly. Landing #3325 or #2907 later would restore attribution, but would not remove the whole-file header added here.

2. [Blocking] Audit every source-candidate entry instead of skipping build-looking directory names

scripts/asf-license-headers.mjs:254-267,410-427 silently skips every archive directory named dist, release, test-results, and several others. asf-source-release.mjs does not reject most of those names, so an extracted source candidate can contain unaudited source files.

A concrete extracted-tree repro with release/unexpected.kt reports:

Audited 0 covered and 0 excluded file(s)
Every source file carries the ASF header or a reviewed exclusion.

The comment claiming that verifySourceCandidate already rejects these directories is therefore incorrect.

Please remove the build-output skip from archive mode and classify every regular file. Unsupported entry types such as symlinks should be explicitly rejected or classified rather than silently omitted. Since the workflow performs this audit before npm ci, legitimate checkout build output does not need an archive-mode exception. An extracted-candidate fixture containing dist/unexpected.kt should fail as unclassified.

3. [Important] Replace directory-wide exclusions with reviewed inventories

Several exclusions use an entire directory as the policy boundary:

  • everything under apps/desktop/resources/licenses
  • everything under apps/desktop/src/renderer/assets/provider-brands
  • everything under packages/runtime/resources/bundled-skills

This means a future Maka-authored .ts or .md file placed in any of those directories silently inherits a third-party/verbatim justification. It also already misdescribes git/NOTICE.txt and git/SOURCE_OFFER.txt: those are Maka-authored legal prose, not byte-identical upstream texts verified by the preparation scripts.

This contradicts the gate's fail-closed objective. Please use exact reviewed inventories or structurally provable shapes—for example, exact bundled payload paths and only *.svg provider-brand assets—and add negative tests showing that an unexpected .ts/README.md inside each directory remains unclassified.

4. [Important] Fail closed on existing ASF headers the writer cannot safely canonicalize

The formatting-variant fix still leaves several forms of the original duplicate-header failure open:

  • A TypeScript file using a valid leading // ASF header is parsed only as block, classified as absent, and receives a second header.
  • A common /** ... */ rendering behaves the same way.
  • The ASF policy explicitly accepts https in the license URL, but that variant is also classified as absent and duplicated.
  • Conversely, the exact-header fast path returns canonical before inspecting the complete leading hash/slash comment, so a directly adjacent third-party copyright line bypasses the entangled safeguard.

In each duplication case, the next audit passes because the newly inserted canonical header is at offset zero.

The simpler and safer boundary is: keep the audit byte-exact, but if any leading ASF signature is present and is not the isolated canonical rendering, refuse to mutate it and require manual resolution. After this one-time sweep, consider removing the variant-rewriting state machine—or the write mode entirely—instead of continually expanding a semantic license-comment parser.

Simplify audit

The same fixes can materially reduce the policy surface:

  1. Delete the second archive inclusion authority (buildOutputDirectories).
  2. Replace broad prefix exclusions with explicit reviewed inventories.
  3. Retire the one-shot variant-rewriting machinery after the initial sweep, leaving one small exact audit gate.

Verdict: request changes. The current tree appears mechanically sound, but the gate does not yet enforce the fail-closed licensing boundary described by the PR.

简体中文

已审查精确 head 0f97abe69fca7b14fab6e0e057e59ae1ded73f12

机械 header sweep 内部一致,35 项定向测试通过,四项 CI 也全部通过。不过,我确认了四个 policy 层面的缺口;在它成为仓库 release gate 前应当修复。

1. [Blocking] 不要对已知 mixed-origin 文件机械添加 whole-file ASF header

扩展名 fallback 给以下文件添加了整个文件的 ASF header:

  • packages/runtime/src/tool-output.ts
  • packages/runtime/src/edit-replace.ts
  • packages/runtime/src/model-protocol.ts

前两个文件明确说明其改编自 opencode;#3325 固定了 MIT 来源,并将其描述为 mixed-origin。#2907 同样记录了 model-protocol.ts 改编的 Vercel AI SDK 材料。

ASF source-header policy 明确将文件内并非由版权所有者提交的部分也视为 third-party work,要求不要机械地在第三方源码顶部添加标准 ASF header,并要求 PMC 对重大修改逐案决定。

请暂时将这些路径分类为 third-party/mixed-source exclusion,并保留上游 attribution。如果 PPMC 后续决定某个重度修改文件使用组合 header,可以再显式实施。之后合入 #3325#2907 只能补回 attribution,无法移除本 PR 已经添加的 whole-file header。

2. [Blocking] 审计 source candidate 中的每个 entry,不要按看似 build output 的目录名跳过

scripts/asf-license-headers.mjs:254-267,410-427 会静默跳过 archive 中所有名为 distreleasetest-results 等的目录。asf-source-release.mjs 并不会拒绝其中大多数目录,因此解压后的 source candidate 可以包含完全未被审计的源码。

在解压目录中放入 release/unexpected.kt 后,实际输出为:

Audited 0 covered and 0 excluded file(s)
Every source file carries the ASF header or a reviewed exclusion.

所以,代码中声称 verifySourceCandidate 已经拒绝这些目录的注释并不成立。

请移除 archive mode 的 build-output skip,并分类每个普通文件。symlink 等不支持的 entry type 也应明确拒绝或分类,而不是静默忽略。workflow 在 npm ci 前运行该审计,因此 archive mode 不需要为 checkout build output 提供例外。建议增加包含 dist/unexpected.kt 的 extracted-candidate fixture,并断言其因 unclassified 而失败。

3. [Important] 用经过审查的文件清单替代整个目录级别的 exclusion

当前多个 exclusion 把整个目录作为 policy 边界:

  • apps/desktop/resources/licenses 下的所有内容
  • apps/desktop/src/renderer/assets/provider-brands 下的所有内容
  • packages/runtime/resources/bundled-skills 下的所有内容

这会导致未来放入这些目录的 Maka 自有 .ts.md 文件静默继承 third-party/verbatim 理由。当前理由也已经错误描述了 git/NOTICE.txtgit/SOURCE_OFFER.txt:它们是 Maka 编写的法律说明,不是由 preparation scripts 校验 digest 的上游逐字节文本。

这与 gate 的 fail-closed 目标相矛盾。请使用精确、经过审查的文件清单,或者能由结构严格证明的规则——例如精确列出 bundled payload,只允许 provider-brand 的 *.svg——并增加反向测试,确保这些目录中新出现的 .ts/README.md 仍然会被判为 unclassified。

4. [Important] 对 writer 无法安全 canonicalize 的已有 ASF header 应当 fail closed

当前 formatting-variant 修复仍留下了几种原始重复 header 问题:

  • TypeScript 文件若使用有效的 // ASF header,只会按 block style 解析,因此被判为 absent 并叠加第二份 header。
  • 常见的 /** ... */ 格式有同样的问题。
  • ASF policy 明确接受 license URL 使用 https,但该变体仍会被判为 absent 并重复添加。
  • 反过来,exact-header 快速路径会在检查完整的 hash/slash leading comment 前直接返回 canonical,导致紧邻 header 的第三方 copyright 行绕过 entangled 保护。

在所有重复场景中,下一次 audit 都会通过,因为新添加的 canonical header 位于 offset zero。

更简单且安全的边界是:audit 保持 byte-exact;但只要发现 leading ASF 标志,而它不是独立的 canonical rendering,就拒绝自动修改并要求人工处理。完成这次一次性 sweep 后,建议删除 variant rewrite 状态机——甚至整个 write mode——而不是继续扩展一个语义化的 license-comment parser。

简化审计

上述修复也能明显缩小 policy surface:

  1. 删除第二套 archive inclusion authority,即 buildOutputDirectories
  2. 用明确的 reviewed inventory 替代宽泛的 prefix exclusion。
  3. 初次 sweep 完成后收回一次性的 variant rewrite 机制,只保留一个小型、精确的 audit gate。

结论:request changes。当前 tree 的机械改动本身没有发现问题,但这个 gate 尚未真正实现 PR 所描述的 fail-closed licensing boundary。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants