diff --git a/.context/decisions/archive-member-executables.md b/.context/decisions/archive-member-executables.md new file mode 100644 index 0000000..cffadd3 --- /dev/null +++ b/.context/decisions/archive-member-executables.md @@ -0,0 +1,28 @@ +# Executable archive members are declared beside the extraction + +**Decided 2026-11-04, for the next npm minor release after 1.4.0.** + +`assetArchives[].executables` lists paths relative to that entry's destination after stripping. +An absent or empty list declares none. Each must be a distinct regular member of that archive, +reached without a link and still present after pruning and compressed-source removal. Diagnostics +name the field and offending path. Paths follow the forward-slash payload convention and additionally +reject dot segments and colons so alias spellings and Windows stream names cannot hide duplicates. +Repeated resulting payload paths across entries are refused, including after base/target joining. + +The validated paths join the existing executable asset/local-file rule. Modes remain synthesized: +0755 on POSIX, 0644 for every Windows file entry, with identical declaration validation on Windows. +This also admits an archive member as a native execution binary. + +**Rejected:** a top-level executable-path list, which disconnects origin validation from the +extraction that must supply the file. **Rejected:** inheriting upstream permission bits, which +would let extraction and host modes decide a reproducibility promise. + +**Compatibility:** schema version 3 stays unchanged. Only optional build input grows; the signed +release, box manifest, digest, kind strings, signature and frozen golden documents do not change. +Older builders reject a scroll using the field through the archive item's closed properties. +Older consumers already understand the resulting ZIP modes. This is an npm minor, not a wire break. + +Authoring commands preserve whole nested objects. There is no archive-specific `add` command; +`edit scroll` intentionally excludes collections, `new scroll` creates no archive entries, and +`refresh` changes pins only. Edit the archive JSON directly. Base/target merging joins entries +whole, so declarations travel with their entry and need no separate merge rule. diff --git a/.context/decisions/environment-executable-paths.md b/.context/decisions/environment-executable-paths.md new file mode 100644 index 0000000..7cba92a --- /dev/null +++ b/.context/decisions/environment-executable-paths.md @@ -0,0 +1,32 @@ +# Executables the solved environment installs are declared in `executablePaths` + +**Decided 2026-11-04, for the same npm minor release as archive-member executables.** + +The runtime rule marks only the interpreter and the scripts directory. A package that keeps a +helper binary in its own directory inside the prefix — a compiler Triton launches, a server +PyTorch spawns — has always shipped as 0644, and once the self-test ran with archive modes it +failed at build time with nothing an author could declare. `executablePaths` closes that gap. + +It is a top-level list of literal payload paths under the runtime's environment root (`venv/` for +every runtime). The reader refuses a path outside that root, a path an asset or local file brings +in, and a path under any `assetArchives` destination, so each executable keeps one declaration on +the entry that supplies it. Spelling follows `assetArchives[].executables`: forward slashes, no dot +segments, colons or trailing slash, distinct within a whole scroll; base/target joining appends +base-first and drops repeats, like `prunePaths`. After unpacking and pruning, the build requires +each path to be a regular file reached without a link, then joins it to the same mode rule as every +other declaration. Windows validates identically and keeps extension-based executability. + +Packing materialises directory links in the prefix, so a path that went through one on the build +machine is a real path in the payload; a file link inside the prefix stays a link on POSIX targets +and is refused there, while Windows receives the file. + +**Rejected:** globs, which would make the declaration describe whatever the solve happened to +produce rather than what the author reviewed; a literal path that moves with a new `python3.x` +directory fails the build by name instead. **Rejected:** widening the runtime rule to a package +directory, which would mark data files executable. **Rejected:** accepting any payload path, which +would give archive members and assets a second, competing declaration — the reason +[`archive-member-executables.md`](archive-member-executables.md) rejected a top-level list. + +**Compatibility:** optional build input in schema version 3, like `assetArchives[].executables`. +Signed documents, the box manifest, the payload digest and ZIP interpretation do not change; +older builders reject the unknown top-level property; consumers need no upgrade. diff --git a/.context/decisions/self-test-archive-modes.md b/.context/decisions/self-test-archive-modes.md new file mode 100644 index 0000000..f57c4c4 --- /dev/null +++ b/.context/decisions/self-test-archive-modes.md @@ -0,0 +1,32 @@ +# Self-tests use the modes the consumer will receive + +**Decided 2026-11-04, alongside executable archive members.** + +Before optional Apple signing and the build self-test, `applyArchiveFileModes()` applies the +archive writer's own `archiveFileMode` rule to every collected regular payload file. It never +chmods a link. Runtime entry points and script directories, executable assets/local files and +declared archive members receive 0755; every other POSIX regular file receives 0644. Windows +skips chmod and retains its existing extension-based semantics. Parity sees the same tree. +The entry-point executable guard now runs before the self-test. Self-test failures retain the +original error and explain which declaration can give a helper its missing executable bit. + +Modes are reapplied after the probes and the final digest write, so generated output has the +same final mode in the build directory and the ZIP. The writer still synthesizes modes itself; +normalization is a check of what will ship, never permission to trust the build filesystem. +This makes two cheap metadata walks rather than reading or copying a second multi-GB payload. + +**Rejected:** archive-then-extract validation (candidate b). It reproduces more than modes but +costs another full write, read and temporary payload, and changes pipeline order. The defect is +the permission rule, which is already centralized. Consumer conformance and the packed-package +end-to-end check still exercise real extraction. **Rejected:** chmodding only declared files, +which would leave undeclared upstream executables runnable during self-testing. + +This deliberately changes build behavior: boxes that passed only because their build directory +inherited upstream executable bits now fail where installed consumers already failed. Rebuilding +with the appropriate declaration repairs them without changing the signed format. No dependency +versions or runtime rules are silently broadened to conceal a failure. + +Evidence: the new real-subprocess regression first failed because the undeclared build wrongly +succeeded; it passes after normalization by observing an OS permission failure, before archiving. +The declared counterpart runs. A separate test proves that a link under the runtime scripts +directory cannot chmod its data-file target back to executable. diff --git a/.context/history/archive-member-executables-plan.md b/.context/history/archive-member-executables-plan.md new file mode 100644 index 0000000..9568e70 --- /dev/null +++ b/.context/history/archive-member-executables-plan.md @@ -0,0 +1,352 @@ +# Executable members of asset archives + +Prepared and completed: 2026-11-04. Status: implemented and locally verified; unreleased. +Branch: `feat/archive-member-executables`, fresh clone of upstream `main` at `e512f00` +(`scrollcase` 1.4.0). The other working copies are untouched. + +## Baseline and boundaries + +- Read `AGENTS.md`, `CLAUDE.md`, `AGENT-POLICY.md`, `CONTRIBUTING.md`, the context index, + architecture, conventions, current state, relevant decisions and public design decisions. +- `npm ci`, then `npm test`: **38 files passed; 588 tests passed, 1 skipped** on Linux x86_64 + in WSL. The skip is the macOS code-signing case. Raw logs are outside the clone in + `/home/lorenzo/scrollcase-executables-evidence/`. +- No push, release tag, publication, hosted workflow or private-key output. Real local validation + is explicitly requested; use a small synthetic archive and an existing pinned toolchain, never + rebuild the CI-signed demos. If a real solve is needed, measure it and retain the exact inputs. +- Keep every implementation commit green. Record defects and guard failures here before moving + to the next expensive step. Restore mutations immediately after proving a guard red. + +## Decisions and exact rules + +1. Add optional `assetArchives[].executables`, an array of distinct paths relative to that entry's + `destination`, after `stripComponents`. Empty and absent both mean no declarations. Use the + existing safe-path convention, with no dot segments or alias spellings; no globbing, directories, + absolute/drive paths, traversal, NULs, links or link ancestors. Diagnostics name + `assetArchives[].executables`, the offending path and its archive. Reject duplicates before + extraction, including normalized duplicates and repeated resulting payload paths across entries. +2. Validate against regular extracted files before pruning and revalidate after pruning, before + runtime preparation could recreate a removed file. A declaration removed by `prunePaths` or + `removeAfterExtract` is a contradiction. Keep the default compressed-source removal and optional + retention working. Declarations do not borrow a pre-existing file from another source. +3. Join the validated paths to the existing asset/local-file declarations. Both the archive writer + and the execution guard receive this same list. ZIP, TAR and TAR.GZ use the same rules. Windows + validates declarations identically, writes 0644 as before and uses extension-based execution. +4. Choose mode normalization before the self-test (candidate a). A helper in `archive.mjs` uses + `archiveFileMode` itself on regular payload files, never follows links, and skips chmod on a + Windows target. Run it before optional Apple signing, self-test and parity. Keep the entry-point + guard ahead of execution. Wrap self-test failures with actionable declaration guidance while + preserving the original error. Reapply modes after probes so probe-created files also match + the archive. Do not duplicate the rule, infer executable bits from input modes or extract a + second multi-GB payload (candidate b). +5. Stay inside schema version 3: only the optional build-input schema changes. No signed document, + manifest, digest format, namespace, signature or frozen golden envelope changes. Older builders + reject the new field through `additionalProperties: false`; older consumers already understand + the unchanged ZIP modes. This calls for the next npm **minor** (1.5.0), left unreleased here. + +Record the declaration decision and the self-test decision separately in `.context/decisions/`, +including the rejected top-level executable list and archive-round-trip designs. + +## Files and implementation commits + +### 1. Declare and validate archive members + +- `src/contract/schema/scroll.schema.json`: optional described field; regenerate types with + `npm run types`, public v3 schemas with `npm run docs:schemas`, Python schemas with + `python/scripts/sync_schemas.py`, Rust assets with `rust/scripts/sync-assets.mjs`. +- `src/build/assets.mjs`, `src/build/archive.mjs`, `src/build/scroll.mjs`, `src/build/box.mjs`: + validate declarations, reject link traversal before copying, check post-prune survival, collect + executable paths and feed the existing archive/entry-point checks. Preserve checksum validation, + collisions, injection seams and the single substrate. +- Authoring audit: `scroll-edit.mjs` spreads complete objects, so add/remove/edit/refresh preserve + nested archive entries. `edit scroll` intentionally excludes collections; authors edit archive + JSON directly. `add` has no archive-member command to extend. `authoring.mjs` scaffolds no asset + archives. `scroll.mjs` joins whole archive entries base-first; nested declarations travel with + their entry, without an independent merge rule. Add preservation and fragment tests. +- Tests: synthetic ZIP/TAR/TAR.GZ; with/without stripping and source removal; regular-file and + missing/directory/link/ancestor/unsafe/duplicate cases; prune contradictions; native execution + guard; Windows mode rule; byte-identical rebuild. Observe focused new tests fail first. +- Docs in this commit: scroll reference, managing-assets guide, white paper, public decisions, + changelog, context architecture and decision record. Updating-boxes does not currently describe + archive declarations; add migration guidance there alongside the behavior change in step 2. +- Regenerate the committed GitHub Action bundle and schemas with `npm run action:build`. + Gate: focused tests, `npm test`, type/copy/action checks, docs build, Syngraphe. + +### 2. Self-test the modes that will ship + +- `src/build/archive.mjs`, `src/build/box.mjs`: shared mode normalization, earlier execution guard, + self-test diagnostics and parity ordering. Preserve safe prefix symlinks and Apple signing. +- Tests: run a real local subprocess against an executable archive member; prove an undeclared + 0755 input fails at self-test time and the declared counterpart succeeds; inspect normalized + modes in the runner and the final ZIP. Cover runtime-generated console scripts and data files. +- Measure every tracked scroll and every fixture/inline declaration for archive usage and + execution outside runtime rules. Do not claim large demos were rebuilt. Document measured + static impact and limits, including dependencies that might spawn helpers outside scripts paths. +- Update the self-test decision, relevant reference/guide/architecture/white-paper/changelog pages. + Regenerate types and Action. Gate: focused tests and all step-1 checks. + +### 3. Consumer parity and complete verification + +- Extend the existing restrictive-umask conformance case in + `src/contract/fixtures/consumer-conformance.json` with a declared archive-member path. + This behavioral case is not a frozen signature golden. Extend the JS, Python and Rust fixture + builders and regenerate Rust fixture copies. JS stages a real synthetic asset archive; the + mirrors express the resulting unchanged signed box. Under umask 077 all three must extract 0755. +- Full checks from `AGENTS.md` and `.github/workflows/ci.yml`: Node suite, types, Action drift, + npm pack dry-run, production dependency audit, public schemas, docs build; isolated Python unit + suite, mypy, schema check, wheel/sdist, distribution inspection and clean wheel import; Rust + tests, clippy, asset check and cargo package. No hosted CI. Report host/version limits honestly. +- Pack the finished repository with `npm pack` into scratch, install that tarball in a separate + scratch project with a committed scroll and synthetic executable TAR.GZ. Build with the real + pinned pixi/conda-pack if available, install through the JS consumer under umask 077 and run the + member. Verify release signature, archive hash, mode and output; keep no scratch state in git. +- Gate: all required checks green; document counts, guard failures, source inventory, artifact + hashes, commands and remaining limits here; commit evidence and move the finished plan to + `.context/history/`. Update `.context/index.md` and `.context/state/current.md` in the same change. + +## Risks and stop conditions + +- Existing ZIP asset copying can turn a link into a regular file: validate the extracted source + before copying, retaining the stricter asset-archive boundary without changing consumer links. +- A destination may already traverse a prefix link: check destination ancestors before extraction, + not only the final declared file. Do not broaden this work into a different extraction backend. +- chmod must visit regular files only, not mutate a symlink target under an alias. New self-test + output and the last-written digest file need their final modes applied too. +- Mode normalization deliberately reveals boxes that passed only on build-directory permissions. + Preserve the subprocess error and point to every applicable executable declaration. +- The Action includes build code and schemas, so source changes require regenerated committed + artifacts. Generated types and language copies are never hand-edited. +- Stop a local real build on unexpected downloads or a pin mismatch; inspect exact inputs before + any retry. Never weaken native-host, provenance, checksum, signer or licence guards. + +## Living evidence + +- [x] Fresh current source, repository instructions, baseline suite. +- [x] Plan committed before implementation (`765119c`). +- [x] Declaration, validation, authoring preservation, determinism and docs proved. +- [x] Consumer-mode self-test regression proved red, then green; source impact measured. +- [x] Three consumer implementations preserve the archive-member bit under umask 077. +- [x] Every required local check and packed-package end-to-end validation passed. +- [x] Implementation commits and owner-only release steps recorded; final clean-tree check follows + the evidence commit. + +Step 1 evidence in progress: 33 new asset tests failed against the baseline (4 existing passed). +The expanded Node suite exposed Ajv strict mode requiring an explicit string type beside a +referenced path pattern; fix the schema rather than weakening strict mode. A first native test +omitted required `defaultArgs`; its corrected declaration passes. System Python lacks ensurepip, +so tests use uv's isolated CPython 3.13 environment outside the clone instead. The existing pinned +toolchain is present under the WSL user's `.pixi/`; do not infer absence from PATH. + +Safety review found that checking only an extraction destination misses a link deeper beneath it. +Move the lstat walk into the existing filesystem module and apply it before every copied archive +entry as well as when validating declarations. Add a regression asserting no file is written +through that ancestor. No second extractor or mode rule is needed. + +Step 1 complete: **640 Node tests passed, 1 macOS-only skip, 38 files**. The source membership, +link traversal, stripping/removal combinations, prune contradictions, Windows modes, native +entry point and byte-identical ZIP/TAR/TAR.GZ builds pass. Authoring preservation and split-scroll +tests pass. Generated contract/runtime types, 8 public schemas and Action bundle checks pass; +Python and Rust synchronization made no copy changes because neither consumer ships the build-only +scroll schema. Python: **68 tests**, mypy (9 source files), schema check, wheel/sdist and distribution +inspection pass. Rust: **112 tests**, clippy clean. Docs build and postbuild verification pass: +82 pages, 8 schemas, 82 Markdown twins and 46-page canonical/LLM coverage. `syngraphe check` passes +with the pre-existing AGENT006 managed-block-version warning; no managed block was hand-edited. + +The docs build was first started concurrently with schema generation and correctly refused drift. +It passed after synchronization; dependent checks will run after their generators from now on. + +Step 2 focused evidence: the undeclared real-helper regression failed because the baseline build +wrongly succeeded; the runtime-mode assertion also failed (0644 instead of 0755). After applying +the writer's rule, all 10 focused self-test cases pass. The undeclared helper now fails with an +OS permission error before archive creation; declaring it succeeds. Probe outputs and the final +digest receive final modes too; the link-alias regression checks that chmod never follows links. + +Static source inventory (script and JSON in the external evidence directory): **26 tracked +scroll.json files, 20 effective example/demo scrolls**, all read through the real split-scroll +reader. Zero example archive entries; every execution command satisfies the existing executable +rule. Per family: codon-demo, dataset-demo, hello-box, llm-demo, sentiment-demo and transcode-demo +each have 3 targets; hello-box-node and hello-box-native have 1 each. Native commands are the +runtime-provided h5dump, zstd and ffmpeg under the scripts directory; interpreted commands are +the runtime entry point. Scanning tracked JSON found 10 complete scroll inputs: the only archive +declaration is the canonical scroll-pixi example, a source ZIP with no execution. The pre-existing +inline archive fixtures exercise reading or intentionally invalid `embed`; no undeclared helper +is run. The full suite measures these paths dynamically with the stubbed toolchain. This does +not establish that every third-party dependency avoids spawning a helper outside runtime script +paths; real builds may expose those, and CI-signed demos cannot be rebuilt with local keys. + +Step 2 gate: **644 Node tests passed, 1 skipped, 38 files**; contract/runtime types and Action +drift checks pass; docs build and all generated-route checks pass; Syngraphe passes with the same +pre-existing managed-block warning. No Python or Rust implementation/copy changed in this step. + +Step 3 in progress: extending the existing umask-077 case with `tools/bin/archive-tool` (0755) +and an undeclared upstream-executable README (0644) first failed in **all three** implementations, +because the extra paths were missing. The JS fixture now expands a real synthetic TAR.GZ through +the build declaration; Python and Rust describe the resulting unchanged ZIP. No consumer runtime +or signed schema changes are needed. Next gates include the full language checks and packaging. + +The lost-bit guard was then proven directly in all three fixture builders: set only the archive +member's synthesized mode to 0644, run the conformance case, observe 0644 versus required 0755, +and restore the exact original file in a finally block. JS exited 1, Python 1 and Rust 101. +Separately, the 1.4.0 schema accepted its original canonical scroll, then rejected that same scroll +with the new field as `$.assetArchives[0].executables is not allowed`. Compatibility is observed, +not inferred. Full suites after the fixture additions passed: Node 644 + 1 skip, Python 68, +Rust 112. Types, Action drift, npm pack dry-run, production audit (0 vulnerabilities), Python +schema/mypy/build/distribution checks and Rust clippy/asset checks pass. Docs build passes too. +The restored guards will be checked again before the conformance commit; clean cargo packaging, +clean wheel import and the real packed-package smoke remain. + +Guard cleanup finding: Python reused bytecode compiled during the deliberate same-size 0755 to +0644 mutation, because mutation and restoration fell within the same timestamp second. The +source was correctly restored to 0755, but the next suite still observed the cached 0644 fixture. +Remove only the identified generated conformance-support pyc, then rerun the full Python suite; +this is mutation-harness cleanup, not a production-code fallback or weakened expectation. + +The Python suite passes again after removing that one generated pyc. Clean wheel installation +and import pass from a second venv outside the checkout. `cargo package` passes from the clean +conformance commit. The real packed build also succeeded in 0.734 seconds using the installed +pixi 0.77.0 and conda-pack global environment pinned to 0.9.2, with a tiny locked libzlib prefix. +Its first consumer extraction verified signature/hash/digest and restored 0755 under umask 077; +the execution harness then passed a memory-only Writable where Node spawn needs a file descriptor. +Correct only the scratch verifier to capture stdout through an opened file descriptor, commit that +scratch correction, and rebuild cleanly. Do not change the consumer API as part of this task. + +Real smoke now passes completely, including a second clean build with the same ZIP hash. Final +review identified a schema consistency gap: `uniqueItems` is new to the lightweight validator, +so its own structural check would ignore it even though executable-path validation already +rejects duplicates. Add one direct schema regression, observe it fail, implement the keyword +using the validator's existing deep-equality primitive, update the module description, regenerate +the Action, and rerun checks. This does not relax or replace the field/path diagnostic guard. +The self-test regression was also strengthened to run the scroll's real Node probe source and +argv through a tiny fixture interpreter, observing the nested helper's EACCES rather than +substituting a helper launch for the probe. Both declared and undeclared cases pass. + +Final consistency gate: the new direct schema test failed before keyword support and passes +afterwards. **645 Node tests passed, 1 skipped, 38 files**. Types, regenerated Action drift and +docs/postbuild checks pass. The finished package was repacked and installed in a new scratch +project; its real build and full consumer preparation, payload verification and execution pass. + +## Delivered commits + +| Commit | Result | +| --- | --- | +| `765119c` | Plan committed before implementation, with baseline and compatibility analysis | +| `0e3a9e5` | Declaration, extraction safety, prune checks, authoring preservation and docs | +| `d2499b4` | Self-test and parity use the archive writer's synthesized modes | +| `f0d594f` | Existing umask conformance case extended across JS, Python and Rust | +| `22ea9d9` | Schema uniqueness support and real scroll-probe subprocess regression | + +The final documentation commit closes this record, updates current context and records the owner +release procedure. No version bump, release tag, push, hosted CI run or publication was performed. +The fresh clone is `/home/lorenzo/scrollcase-executables`; the unrelated working copies were not +used. Runtime dependencies, lockfiles, export map, signed schemas and frozen signature fixtures +are unchanged. Generated types, public schemas and the committed Action were regenerated. + +## Initial Linux verification + +Host: WSL Ubuntu Linux x86_64; Node 22.14.0, Python 3.13.15 in isolated venvs, Rust/Cargo 1.95.0. + +| Check | Observed result | +| --- | --- | +| `npm test` | 38 files, **645 passed, 1 skipped**; baseline was 588 passed + 1 skipped | +| `npm run types:check`, `npm run action:check` | Pass | +| Public schema check and both language-copy checks | Pass; 8 public schemas | +| `npm pack --dry-run`, actual `npm pack` and scratch installation | Pass | +| `npm audit --omit=dev` | **0 vulnerabilities** | +| Docs build and postbuild verification | Pass; 82 pages/twins, 8 schemas, 46-page canonical/LLM coverage | +| Python unittest suite | **68 passed**, including all 86 shared conformance cases | +| Python `mypy src`, schema check, wheel/sdist and distribution inspection | Pass; 9 source files typed | +| Clean wheel installation/import from another venv | Pass; import resolves from its site-packages | +| Rust `cargo test --all-targets` | **112 passed**, including all 86 shared conformance cases | +| Rust clippy, copied-assets check and clean `cargo package` | Pass | +| `syngraphe check` | Pass with the existing AGENT006 warning about the old managed block | + +The single skipped Node test requires native macOS code signing. At this initial handoff, native +macOS/Windows and other interpreter-version CI matrix legs had not run locally; no hosted +workflows were started. Native Windows was subsequently tested: see +[`windows-native-validation.md`](windows-native-validation.md) for actual results, the existing +symlink-privilege limitation and the successful Windows EXE smoke. Native macOS remains untested. +All three target adapters' mode rules were exercised, and Windows declarations share the same +target-independent validation. Large CI-signed demos were statically inventoried, not rebuilt. + +### Packed-package smoke + +The final npm tarball is the unreleased source still numbered 1.4.0, not the registry's published +1.4.0. It was installed by absolute local tarball path in a separate scratch checkout. + +- Tarball SHA-256: `0dbc1114f28abd1dbac540869913f10850bafbdf707b839cef64c782d72ea500`. +- Scratch root: `/home/lorenzo/scrollcase-executables-evidence/packed-smoke-aSn2R7`. +- Scratch source commit: `c21da0cd8eead53db6b16d3d3c72a961c3f9a51f`, clean at build time. +- A statically compiled C helper prints `archive-member-ok`. Its TAR.GZ has two roots, + `package/bin/tool` and `._metadata`, no stripping, and expands into `tools`. The scroll declares + `executables: ["package/bin/tool"]`, prunes only `tools/._metadata` and removes the compressed + source. The input is a hash-pinned local file, so this test needs no asset server. +- Input TAR.GZ SHA-256: `603b5f249bee9e4d77c87f7e03d64229c32a0ee0c15c9fb11fd723acba216d9b`. +- The manifest pins one conda-forge package, `libzlib = "=1.3.2"`, on `linux-64`. The real lock + SHA-256 is `9a16d25f318f7e6ed84c82aba81701fd3d9412f68e8a6d74d5833c6445cbc23e`. +- Installed pixi 0.77.0 and the existing conda-pack global environment pinned to 0.9.2 performed + the real solve/install/pack. No toolchain was installed. The final build took **0.738 seconds**. +- Consumer preparation under umask 077 verifies the signature, archive hash and manifest, then + restores **0755**. Explicit installed-payload verification succeeds for 10 entries. Running + through `runExtractedBox` exits **0**, prints **archive-member-ok** and raises no signal. +- A second clean real build yields the same **401,550-byte** archive, SHA-256 + `1e375528e5d2f7d02868be220f9969b386434d6743ce1530951a32ef54502f74`. +- Signed release file SHA-256: + `10b8e25ee11988e9ae31cb62b6b43539954795cadf13c9ad6270e3a05f034379`. + +Exact smoke commands, after creating the synthetic project and installing the local tarball: + +```sh +npm pack --pack-destination /home/lorenzo/scrollcase-executables-evidence +# From the scratch project, using its installed package, not source imports: +node node_modules/scrollcase/src/cli.mjs lock archive-member-smoke --pixi /home/lorenzo/.pixi/bin/pixi +# Commit the scroll, input, manifest, lock and package lock before the build; ignore .scrollcase/. +node node_modules/scrollcase/src/cli.mjs keygen +node node_modules/scrollcase/src/cli.mjs build archive-member-smoke \ + --pixi /home/lorenzo/.pixi/bin/pixi --conda-pack /home/lorenzo/.pixi/bin/conda-pack +node verify.mjs +``` + +The scratch verifier sets umask 077 around `verifyAndExtractBox`, asserts the installed mode and +pruned/source absence, calls `verifyExtractedPayload`, and runs the receipt with stdout captured +through a file descriptor. Raw logs and executable scripts are in the external evidence directory; +the decisions, results, exact inputs and hashes needed to continue are recorded here. + +## Owner-only release procedure + +The intended npm release is **1.5.0**: a new optional build-input field is a minor. Schema version +stays 3. Existing consumers need no release; their runtime implementations and versions did not +change. Older builders clearly reject scrolls using the field. Boxes whose undeclared helpers only +worked under build-directory modes now fail early; the 20 effective repository examples need no +declaration changes based on the measured static inventory above. + +After reviewing this branch, integrate it into current main. In this fresh clone main starts at +the cloned base, so a fast-forward is available unless the owner has advanced it independently: + +```sh +git switch main +git merge --ff-only feat/archive-member-executables +``` + +Then, following `CONTRIBUTING.md`, **close the changelog before bumping**: rename the current +Unreleased section to `[1.5.0]` with the actual release date, and open a fresh empty Unreleased +section above it. The following commands are for the maintainer to execute, not an authorization +for an agent to publish: + +```sh +git add CHANGELOG.md +git commit -m "Close the changelog for 1.5.0" +npm version 1.5.0 +node scripts/check-release-version.mjs v1.5.0 +npm test +npm run types:check +npm run action:check +npm pack --dry-run +(cd docs && npm run build) +git push --dry-run --follow-tags origin main +# Inspect exactly which branch and tags would travel. Push only the intended release. +git push origin main --follow-tags +``` + +Pushing `v1.5.0` starts the npm Trusted Publishing workflow. The maintainer checks its result and +the registry version/provenance afterwards. Do not publish manually, backfill old release tags, +or push Python/Rust release tags for this change. No GitHub release or paid workflow is needed. diff --git a/.context/history/windows-native-validation.md b/.context/history/windows-native-validation.md new file mode 100644 index 0000000..251321c --- /dev/null +++ b/.context/history/windows-native-validation.md @@ -0,0 +1,125 @@ +# Native Windows verification follow-up + +Completed: 2026-11-04. Source: `feat/archive-member-executables` at `5158b43`, plus the +code-signing fixture correction described below. Native link-dependent suites remain blocked by +the local account's Windows symlink privilege; they are not reported green. + +The prior executable-member verification used WSL even though native Windows Node, Python and +Rust are available. Correct that omission without changing the user's original working copies +or pushing the branch. The existing CI workflow already covers Node 20/22/24 and Rust on Linux, +macOS and Windows; Python 3.10/3.14 runs on Ubuntu. It triggers for pull requests and pushes to main, +so the unpublished local branch has no CI result. No hosted workflow is authorized here. + +- [x] Export the exact committed branch as a Git bundle; clone it into the new native Windows + directory `C:/Users/loren/Documents/GitHub/scrollcase-executables-windows`. Keep its dependencies + separate from WSL. Verify the source SHA and clean checkout before testing. +- [x] Run native `npm ci`, `npm test`, types and Action drift checks. Capture actual failures and + applicable platform skips, including link-creation privileges, without weakening tests. +- [x] Run the Python consumer in an isolated Windows venv and the Rust consumer with its native + toolchain; run relevant copy and lint checks. Local compiler prerequisites must be checked, + not assumed absent. +- [x] Review failures in the main WSL branch, with failing coverage + observed first. Preserve unrelated pre-existing failures as such only after comparing the + affected path with the upstream baseline. Do not bypass any native-host or safety guard. +- [x] Record final native results in the tracked implementation report and current context; + rerun affected checks, commit changes locally and verify the branch is clean. + +Logs and the bundle belong in the new sibling directory +`C:/Users/loren/Documents/GitHub/scrollcase-executables-evidence`. Native tests are reversible local +work already authorized by the requested cross-platform validation. No push, tag, publication, +real box/toolchain installation or paid CI run is part of this follow-up. + +Observed first native run: Node **576 passed, 57 failed, 13 skipped**. The untouched upstream +baseline at `e512f00`, run in its own new Windows worktree, has **534 passed, 46 failed, 9 skipped**. +Both expose the same missing Windows symlink privilege (EPERM), plus an existing code-signing +refusal test that changes its target to Linux but retains Windows' `venv/python.exe` declaration. +The branch's 11 additional failures are archive/link and build cases stopped by the same privilege. +Do not disable those tests or change machine-wide Developer Mode. Correct the contradictory +code-signing fixture by letting the declared target derive its interpreter path, and rerun it +on Windows and Linux. Its Windows failure is already observed, on both baseline and branch. + +Python native: 68 run, **62 passed, 6 platform skips**; mypy (9 source files) and copied schemas +pass. Native Node generated types and Action drift pass. Rust builds successfully with the +installed MSVC Build Tools; the first archive test binary exposes missing symlink privilege +(OS error 1314). Run remaining Rust binaries with no-fail-fast, compare the unchanged failing +archive case with baseline, and record exact outcomes. + +Add a separate native Windows signed build/extract/run smoke using a tiny locally compiled EXE +inside a TAR.GZ and a link-free stub for only the environment installer, as Windows prefixes +normally are. Keep actual staging, declaration validation, self-test, ZIP/signing, verification, +extraction and process execution real. This exercises the new field without changing existing +link guards or requiring a machine-wide permission change. The earlier WSL smoke retains the +real pixi/conda-pack environment check. No new npm test dependency or runtime fallback is needed. + +## Final outcomes + +Native Windows (NT 10.0.26200), Node 22.14.0, CPython 3.13.3, Rust/Cargo 1.95.0 with the existing +Visual Studio 2022 Build Tools. No machine-wide setting or privilege was changed. + +| Check | Observed result | +| --- | --- | +| Untouched main `npm test` | 534 passed, 46 failed, 9 skipped | +| Feature branch before fixture correction | 576 passed, 57 failed, 13 skipped | +| Feature branch after correction | **577 passed, 56 failed, 13 skipped** | +| Corrected code-signing refusal case | Pass on native Windows and in the full Linux suite | +| Node types, Action drift and language-copy checks | Pass | +| Python unit suite | **62 passed, 6 platform skips** out of 68 | +| Python mypy, schema check, wheel/sdist and distribution inspection | Pass | +| Rust suite with `--no-fail-fast` | **98 passed, 1 failed**; all test binaries attempted | +| Rust clippy with warnings denied | Pass | +| Windows signed archive/consumer smoke | Pass, including two byte-identical builds | +| Linux suite after the fixture correction | **645 passed, 1 macOS-only skip**, 38 files | + +The 56 remaining Node failures all stop at Windows denying symlink creation (EPERM), either +directly or through an assertion expecting a later validation error. Main already has 45 failures +with this cause; the 11 additional feature cases encounter the same limitation before their +intended assertions. The separate existing fixture failure is now fixed: a Linux target must +derive its own interpreter, rather than inherit Windows' `venv/python.exe` from the host fixture. +No runtime implementation or executable-field guarantee was changed to make these results pass. + +The one Rust failure is `a_link_the_contract_permits_survives_listing_and_extraction`: Windows +reports OS error 1314, a required privilege is not held. Its test and the entire Rust runtime +are byte-unchanged from main (verified with `git diff --exit-code e512f00 -- rust/src +rust/tests/archive.rs`). All other Rust binaries, including applicable shared conformance cases, +pass. POSIX-only cases are excluded by the repository's existing platform rules. + +The full native Node/Rust suites therefore remain **not green on this Windows account**. Running +them with Windows symlink creation enabled, or on the configured CI runners, is still needed to +close that environment-specific gap. Do not hide it by skipping tests or declaring the feature +perfect. GitHub CI is already configured, but there is no result for this unpublished branch; +the original no-push/no-paid-workflow constraint was preserved. + +## Native Windows execution evidence + +The scratch script compiles a tiny Rust EXE locally, puts it in a real TAR.GZ, commits the scroll +and input, and calls the real builder. Only pixi installation/conda-pack are represented by a +link-free prefix stub; staging, member validation, native self-test, deterministic ZIP creation, +signing, consumer preparation, payload verification and process launch are the real code. + +- Scratch: `C:/Users/loren/Documents/GitHub/scrollcase-executables-evidence/native-smoke-o11GMn`. +- Declared member: `executables: ["bin/tool.exe"]`, destination `tools`, stripping one wrapper. +- Both build self-tests print `native-archive-member-ok`. +- The Windows ZIP mode is **0644**, as documented; the `.exe` runs after consumer extraction. +- Installed-payload verification succeeds and execution exits **0** with + `native-archive-member-ok`. +- Both builds have SHA-256 + `1604650631e936cee1b6af9d7e84cd8299fbe4788340f60e5b5eca40d31523b1`. + +Exact command in the sibling evidence directory: `node native-smoke.mjs`. Native suite logs, +baseline comparison, build artifacts and `windows-smoke-result.json` are kept there. This native +execution check supplements the earlier WSL smoke that used real pixi and conda-pack. + +## Existing GitHub Actions coverage + +`.github/workflows/ci.yml` already defines: + +- Node 20, 22 and 24 on Ubuntu, macOS and Windows: nine suite jobs, with Action drift checks; +- the Action bundle smoke on all three operating systems; +- Rust tests, clippy and copied-assets checks on all three operating systems; +- Python 3.10 and 3.14 on Ubuntu; +- package-surface and documentation checks. + +Its triggers are pull requests and pushes to main. It does not run merely because local commits +exist, and it defines no manual dispatch for uploading an unpublished local checkout. No workflow +was modified or run, and no branch or tag was pushed. The maintainer's original release authority +and the prohibition on paid CI remain intact. diff --git a/.context/index.md b/.context/index.md index 548c094..8753162 100644 --- a/.context/index.md +++ b/.context/index.md @@ -25,6 +25,12 @@ system is, where the work stands, what was decided and why, and what already wen no migration path. - `decisions/bundled-licence-declaration.md` — the one field added inside version 3 rather than deferred to a v4. + - `decisions/archive-member-executables.md` — declaring expanded members and validating their + origin, safe paths and survival, without inheriting input modes or changing the signed format. + - `decisions/environment-executable-paths.md` — the top-level list for helpers the solved + environment installs outside the scripts directory, and why it accepts nothing else. + - `decisions/self-test-archive-modes.md` — applying the archive mode rule before self-testing, + instead of copying a second multi-gigabyte payload. - `decisions/payload-digest-v1.md` — the signed path list, and why a receipt is never serialised. - `decisions/box-environment-report.md` — accepted, unimplemented: report, do not enforce. - `decisions/demo-boxes-are-signed-by-ci.md` — why a demo box cannot be rebuilt locally. @@ -36,6 +42,10 @@ system is, where the work stands, what was decided and why, and what already wen ## Historical - `history/` — completed, historical, or superseded operational context: + - `history/archive-member-executables-plan.md` — completed executable archive members, archive-mode + self-tests, cross-language conformance, packed-package evidence and owner release steps. + - `history/windows-native-validation.md` — native Windows results, the upstream symlink-privilege + limitation, corrected target fixture and actual Windows EXE archive/consumer smoke. - `history/extraction-and-origin.md` — where Scrollcase came from, and how the maintainer works. - `history/mistakes-and-what-they-taught.md` — every trap fallen into, several of them twice. - `history/v3-plan.md` and `history/v3-phase-a.md`, `history/v3-phase-b.md`, diff --git a/.context/state/current.md b/.context/state/current.md index afe9540..9b13ba7 100644 --- a/.context/state/current.md +++ b/.context/state/current.md @@ -1,9 +1,42 @@ # Current State -Last reviewed: 2026-09-28. +Last reviewed: 2026-11-04. ## In flight +**Executable members of expanded asset archives are complete and verified, unreleased** — branch +`feat/archive-member-executables`, based on main at `e512f00` (1.4.0). The declaration, mode-faithful +self-test, authoring preservation, documentation and three-consumer umask conformance are delivered. +Linux: Node 645 pass + 1 macOS-only skip; Python 68 pass; Rust 112 pass. A real packed-package build, +installation and execution under umask 077 pass, including a byte-identical rebuild. The plan, +decisions, artifact hashes, source inventory and owner-only 1.5.0 release steps are in +[`../history/archive-member-executables-plan.md`](../history/archive-member-executables-plan.md). +No version bump, release tag, push, publication or hosted workflow was performed. + +**Executables inside the solved environment — delivered on the same branch.** A helper a package +installs in the prefix outside the scripts directory had always shipped as `0644`, and the +mode-faithful self-test made that a build failure with nothing to declare. Top-level +`executablePaths` now names such files under `venv/`; see +[`../decisions/environment-executable-paths.md`](../decisions/environment-executable-paths.md). +Real smoke on 2026-11-05 (Linux/WSL, packed tarball of `9582e80`, pixi 0.77.0, conda-pack 0.9.2): +a native box whose execution binary is conda-forge `libtool 2.5.4`'s +`venv/share/libtool/build-aux/config.guess`, installed `0755` by conda. Undeclared, the build +refuses it at the entry-point guard; declared, it builds in 1.4 s (2,792,682-byte archive, SHA-256 +`248da16e67b1f798d77055b881c656415328f1f38ac0836f1e9b21d27c088950`) with the helper `0755` and the +undeclared `ltmain.sh` `0644` in the ZIP. The Node consumer under umask 077 verifies the signature, +archive and 94 payload entries, restores `0755`, and runs it: exit 0, `x86_64-pc-linux-gnu`. A +clean rebuild is byte-identical. Scripts and logs: `/home/lorenzo/scrollcase-executables-evidence/` +(`envexec-*`). + +**Native Windows follow-up:** Python 62 pass + 6 skips; Node 577 pass, 56 fail, 13 skips; +Rust 98 pass, 1 fail. All remaining native failures are blocked by this account's missing Windows +symlink privilege, also observed on untouched main. A pre-existing fixture carrying a Windows +interpreter path into a Linux scroll is corrected. Types, Action drift, Python packaging/mypy +and Rust clippy pass. A native EXE archive/build/consumer smoke and identical rebuild pass with +the environment installer stubbed. Full details and existing GitHub CI matrix coverage are in +[`../history/windows-native-validation.md`](../history/windows-native-validation.md). The full +Windows suites are not green on this account; no machine-wide setting or test guard was changed. + **A box may now contain PyPI dependencies** — branch `pypi-licence-declaration`, unreleased. Until now it could not, and nobody had noticed: pixi records an SPDX licence for every conda package @@ -96,9 +129,11 @@ Nothing is authorised and in progress. What is owed, in the order it was last di None. Two standing limits shape what can be proven locally rather than blocking work: -- **Linux and Windows exist only in CI.** The local suite runs on macOS with the toolchain stubbed, - and `MTLCreateSystemDefaultDevice()` returns nil on the maintainer's Mac, so anything - GPU-touching fails locally and passes in CI. +- **Native host coverage depends on the session.** The 2026-11-04 executable-member work ran on + Linux x86_64 in WSL and native Windows x86_64. Windows EXE execution passes, while link-dependent + native suites require a symlink-capable account. macOS native execution remains untested. + On the maintainer's Mac, `MTLCreateSystemDefaultDevice()` returns nil, + so GPU checks still require the appropriate CI host. - **The demo boxes cannot be rebuilt locally.** CI signs them with a private key held in a GitHub secret — deliberately. See [`../decisions/demo-boxes-are-signed-by-ci.md`](../decisions/demo-boxes-are-signed-by-ci.md). diff --git a/.context/truth/architecture.md b/.context/truth/architecture.md index 867a032..061f8f7 100644 --- a/.context/truth/architecture.md +++ b/.context/truth/architecture.md @@ -50,6 +50,22 @@ A consumer reverses it: verify the signature, check the payload shape, check the hash, check safe entries and manifest agreement — *then* extract, and only then run. Verification precedes execution, always, in all three consumers. +Expanded archive members may declare executable bits through `assetArchives[].executables`, +relative to the destination after stripping. The build checks regular-file membership, link-free +paths and survival after pruning before joining the existing asset/local-file mode rule. No mode +comes from the upstream archive or build filesystem. Windows validates the list identically and +keeps 0644 file entries. See [`../decisions/archive-member-executables.md`](../decisions/archive-member-executables.md). + +Files the solved environment installs outside the scripts directory are declared in top-level +`executablePaths`, literal paths under `venv/`. The reader refuses any path an asset, local file +or archive destination owns; the build requires a link-free regular file after pruning. See +[`../decisions/environment-executable-paths.md`](../decisions/environment-executable-paths.md). + +Before code signing, self-test and parity, the build applies the archive writer's own mode rule to +regular payload files, skipping links and Windows chmod. It reapplies modes after probes and the +digest write. Thus inherited input modes cannot hide a missing executable declaration during the +build. See [`../decisions/self-test-archive-modes.md`](../decisions/self-test-archive-modes.md). + ## External systems - **conda-forge**, through `pixi`. The only package source. diff --git a/.context/truth/conventions.md b/.context/truth/conventions.md index 9954f53..cbd64d4 100644 --- a/.context/truth/conventions.md +++ b/.context/truth/conventions.md @@ -66,8 +66,13 @@ habits that were learnt the expensive way. pushed tag, the dangerous command is `git push`, not `npm publish`. - **`MTLCreateSystemDefaultDevice()` returns nil on the maintainer's Mac.** Anything GPU-touching fails locally and passes in CI. Not a code defect; do not chase it. -- **Linux and Windows are covered by CI and nowhere else.** The local suite runs on macOS with the - toolchain stubbed. +- **Record the host actually tested.** CI covers Linux, macOS and Windows. The 2026-11-04 archive + executable work ran on WSL Linux x86_64 and native Windows. Do not infer that Windows is + unavailable merely because the working clone is in WSL: native Node, Python and Rust exist. + This local Windows account cannot create symbolic links, so the full native suite has recorded + EPERM/1314 failures even on main; do not report it green or weaken those guards. The dedicated + native Windows EXE smoke passes. See + [`../history/windows-native-validation.md`](../history/windows-native-validation.md). - **Durable project knowledge belongs in tracked files** — this directory, `AGENTS.md`, `docs/`, `CHANGELOG.md`. Machine-local agent memory is a convenience, never the only copy: the original project-memory file was lost with a deleted clone, and what survives is the reconstruction in diff --git a/CHANGELOG.md b/CHANGELOG.md index eee6a36..ea01ece 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,35 @@ All notable changes to Scrollcase are documented here. The format follows ## [Unreleased] +### Added + +- **`assetArchives[].executables` declares executable expanded members.** Paths are relative to + the archive's destination after stripping. The build rejects missing or non-regular members, + links and link traversal, unsafe or duplicate paths, and declarations removed by pruning. + Linux and macOS receive synthesized `0755` modes; Windows validates the same declarations and + continues to use extensions. This optional build-input field stays in schema version 3 and + requires a newer builder; existing consumers need no format upgrade. Intended for a minor release. +- **`executablePaths` declares helpers the solved environment installs.** Literal paths under + `venv/`, for an executable a package keeps outside the scripts directory, which until now always + shipped as `0644`. A path outside `venv/`, or one an asset, local file or archive destination + already owns, is refused when the scroll is read; after pruning, each path must be a regular file + reached without a link. Same modes, Windows behaviour and schema-version-3 compatibility as + `assetArchives[].executables`. + +### Fixed + +- ZIP asset archives reject links before copying, matching the existing TAR asset boundary. +- **Build self-tests and parity now see the file modes the consumer receives.** Regular POSIX + files receive the archive writer's synthesized modes before testing; links are skipped and + Windows keeps its extension-based behavior. Boxes that passed only because of inherited + build-directory executable bits now fail at build time, where their users already failed. + Declare the helper on its archive entry, asset or local file, or in `executablePaths` when the + environment installs it, and rebuild. A static inventory of all 20 effective repository + example/demo scrolls found no asset archives and no entry points missing declarations; the large + CI-signed demos were not rebuilt. Dependency helpers outside runtime script paths may still + expose missing declarations during a real build. On Windows a self-test failure keeps its + original error with no mode hint. + ## [1.4.0] — 2026-09-28 ### Added diff --git a/action/contract/schema/scroll.schema.json b/action/contract/schema/scroll.schema.json index 7ab3c28..eea2fd9 100644 --- a/action/contract/schema/scroll.schema.json +++ b/action/contract/schema/scroll.schema.json @@ -215,6 +215,16 @@ }, "removeAfterExtract": { "type": "boolean" + }, + "executables": { + "type": "array", + "uniqueItems": true, + "description": "Regular archive members that need the executable bit, relative to this entry's destination after stripComponents. Paths must be distinct, use forward slashes and contain no dot segments or colons. Every path must be a member of this archive, exist before pruning, traverse no link and survive prunePaths and removeAfterExtract. The bit is synthesised from this declaration, never inherited from the upstream archive or build machine. Windows validates the same paths but writes 0644 and decides executability by extension. Empty or absent declares none.", + "items": { + "$ref": "#/$defs/payloadPath", + "type": "string", + "pattern": "^(?!.*(?:^|/)\\.(?:/|$))(?!.*[:\\u0000])(?!.*?/$).+$" + } } } } @@ -263,6 +273,16 @@ "$ref": "#/$defs/payloadPath" } }, + "executablePaths": { + "type": "array", + "uniqueItems": true, + "description": "Files the solved environment installs under venv/ that need the executable bit beyond the runtime's own scripts directory, such as a helper binary a package keeps in its own directory. Literal payload paths, with no glob expansion: forward slashes, no dot segments or colons. Each must be a regular file after pruning, traverse no link, and not be a file an asset, local file or asset archive brings in, which carry their own declaration. The bit is synthesised from this declaration, never inherited from the build machine. Windows validates the same paths but writes 0644 and decides executability by extension. Empty or absent declares none.", + "items": { + "$ref": "#/$defs/payloadPath", + "type": "string", + "pattern": "^(?!.*(?:^|/)\\.(?:/|$))(?!.*[:\\u0000])(?!.*?/$).+$" + } + }, "selfTest": { "type": "object", "additionalProperties": false, diff --git a/action/dist/index.cjs b/action/dist/index.cjs index 0cd516b..be14fab 100644 --- a/action/dist/index.cjs +++ b/action/dist/index.cjs @@ -22529,6 +22529,30 @@ function safeRelativePath(value) { } return normalized; } +async function assertPathWithoutLinks(root, path, field, { + allowMissing = false, + kind = "file", + missingHint = "check prunePaths and removeAfterExtract" +} = {}) { + const parts = safeRelativePath(path).split("/"); + let current2 = root; + for (const [index, part] of parts.entries()) { + current2 = (0, import_node_path.join)(current2, part); + let entry; + try { + entry = await (0, import_promises.lstat)(current2); + } catch (error2) { + if (error2.code !== "ENOENT" && error2.code !== "ENOTDIR") throw error2; + if (allowMissing) return; + fail(`${field}: ${JSON.stringify(path)} is missing (${missingHint}).`); + } + if (entry.isSymbolicLink()) fail(`${field}: ${JSON.stringify(path)} is a link or traverses a link at ${parts.slice(0, index + 1).join("/")}.`); + const needsDirectory = index < parts.length - 1 || kind === "directory"; + if (needsDirectory ? !entry.isDirectory() : !entry.isFile()) { + fail(`${field}: ${JSON.stringify(path)} must be a regular file with directory ancestors.`); + } + } +} async function collectEntries(root, current2 = root) { const entries = await (0, import_promises.readdir)(current2, { withFileTypes: true }); const collected = []; @@ -25856,6 +25880,13 @@ function archiveMarksExecutable(adapter, runtimeId, declared, relativePath) { relativePath ); } +async function applyArchiveFileModes(payloadDir, adapter, { runtimeId, executablePaths: declared = [] }) { + if (adapter.host.platform === "win32") return; + const rule = declaredExecutablePaths(runtimeId, adapter, declared); + for (const path of await collectRegularFiles(payloadDir)) { + await (0, import_promises4.chmod)((0, import_node_path11.join)(payloadDir, ...path.split("/")), archiveFileMode(adapter, path, rule) & 511); + } +} function isDeclaredUncompressed(path, declared) { return declared.some((entry) => path === entry || path.startsWith(`${entry}/`)); } @@ -26055,6 +26086,7 @@ async function extractScrollArchive(archivePath, format, destination, stripCompo } else { fail(`Unsupported archive format: ${format}`); } + await validateExtractedTree(tempRoot); let source = tempRoot; for (let index = 0; index < stripComponents; index += 1) { const entries = await collectFiles(source); @@ -26068,6 +26100,7 @@ async function extractScrollArchive(archivePath, format, destination, stripCompo } const files = await collectFiles(source); for (const file of files) { + await assertPathWithoutLinks(destination, file, "Scroll archive entry", { allowMissing: true }); if (await fileExists((0, import_node_path11.join)(destination, ...file.split("/")))) { fail(`Scroll archive entry already exists in destination: ${file}`); } @@ -26078,6 +26111,7 @@ async function extractScrollArchive(archivePath, format, destination, stripCompo await (0, import_promises4.mkdir)((0, import_node_path11.dirname)(outputPath), { recursive: true }); await (0, import_promises4.copyFile)((0, import_node_path11.join)(source, ...file.split("/")), outputPath, import_node_fs8.constants.COPYFILE_EXCL); } + return files; } finally { await (0, import_promises4.rm)(tempRoot, { recursive: true, force: true }); } @@ -26146,11 +26180,50 @@ async function moveIntoPlace(source, destination) { await (0, import_promises6.rm)(source, { force: true }); } } +function assetArchiveExecutablePaths(archive) { + const paths = archive.executables === void 0 ? [] : archive.executables; + const field = `assetArchives[].executables in ${archive.relativePath}`; + if (!Array.isArray(paths)) fail(`${field}: expected an array, received ${JSON.stringify(paths)}.`); + const seen = /* @__PURE__ */ new Set(); + return paths.map((path) => { + try { + safeRelativePath(path); + if (typeof path !== "string" || path.includes("\\") || path.includes(":") || path.split("/").includes(".")) fail("Use a relative forward-slash path without dot segments or colons."); + } catch (error2) { + fail(`${field}: ${JSON.stringify(path)} is unsafe. ${error2.message}`); + } + if (seen.has(path)) fail(`${field}: ${JSON.stringify(path)} is a duplicate.`); + seen.add(path); + return import_node_path12.posix.join(safeRelativePath(archive.destination), path); + }); +} +async function assertAssetArchiveExecutables(payloadDir, archive) { + const field = `assetArchives[].executables ${JSON.stringify(archive.executables ?? [])} in ${archive.relativePath}`; + for (const path of assetArchiveExecutablePaths(archive)) { + await assertPathWithoutLinks(payloadDir, path, field); + } +} async function expandAssetArchive(payloadDir, archive) { + const executablePaths = assetArchiveExecutablePaths(archive); const archivePath = (0, import_node_path12.join)(payloadDir, safeRelativePath(archive.relativePath)); const destination = (0, import_node_path12.join)(payloadDir, safeRelativePath(archive.destination)); - await extractScrollArchive(archivePath, archive.format, destination, Number(archive.stripComponents ?? 0)); + const field = `assetArchives[].executables ${JSON.stringify(archive.executables ?? [])} in ${archive.relativePath}`; + await assertPathWithoutLinks(payloadDir, archive.destination, field, { allowMissing: true, kind: "directory" }); + let extracted; + try { + extracted = await extractScrollArchive(archivePath, archive.format, destination, Number(archive.stripComponents ?? 0)); + } catch (error2) { + if (executablePaths.length) fail(`${field}: ${error2.message}`); + throw error2; + } + await assertAssetArchiveExecutables(payloadDir, archive); + const members = new Set(extracted.map((path) => import_node_path12.posix.join(safeRelativePath(archive.destination), path))); + for (const path of executablePaths) { + if (!members.has(path)) fail(`${field}: ${JSON.stringify(path)} is not a member of this archive.`); + } if (archive.removeAfterExtract !== false) await (0, import_promises6.rm)(archivePath, { force: true }); + await assertAssetArchiveExecutables(payloadDir, archive); + return executablePaths; } // src/build/codesign.mjs @@ -26329,6 +26402,14 @@ function validate(value, schema, context, path, errors) { errors.push(`${path} must contain at least ${schema.minItems} item${schema.minItems === 1 ? "" : "s"}`); return; } + if (schema.uniqueItems) { + for (let index = 0; index < value.length; index += 1) { + if (value.slice(0, index).some((item) => (0, import_node_util.isDeepStrictEqual)(item, value[index]))) { + errors.push(`${path}[${index}] is a duplicate item`); + return; + } + } + } if (schema.items) { for (let index = 0; index < value.length && errors.length === 0; index += 1) { validate(value[index], schema.items, context, `${path}[${index}]`, errors); @@ -27278,7 +27359,7 @@ async function loadScrollSchemas() { } var SCROLL_BASE_REFERENCE = "../scroll.json"; var JOINED_ENTRY_LISTS = Object.freeze(["assets", "assetArchives", "localFiles"]); -var JOINED_STRING_LISTS = Object.freeze(["prunePaths", "uncompressedPaths"]); +var JOINED_STRING_LISTS = Object.freeze(["prunePaths", "uncompressedPaths", "executablePaths"]); var JOINED_MAPS = Object.freeze(["compatibility", "environment"]); function assertDistinctPayloadDestinations(scroll) { const claimed = /* @__PURE__ */ new Map(); @@ -27363,6 +27444,25 @@ function scrollDirectory(reference) { if (path === root || !path.startsWith(`${root}${import_node_path21.sep}`)) fail(`Invalid scroll: ${reference}`); return path; } +function assertEnvironmentExecutablePaths(scroll, environmentRoot) { + const entryPaths = new Set([...scroll.assets, ...scroll.localFiles ?? []].map((entry) => import_node_path21.posix.normalize(safeRelativePath(entry.relativePath)))); + for (const path of scroll.executablePaths ?? []) { + const field = `executablePaths: ${JSON.stringify(path)}`; + if (!path.startsWith(`${environmentRoot}/`)) { + fail(`${field} is outside ${environmentRoot}/. This field marks files the solved environment installs; declare a file you bring in on its asset, local file or archive entry.`); + } + if (entryPaths.has(path)) { + fail(`${field} is brought in by an asset or local file; mark that entry "executable": true instead.`); + } + const archive = (scroll.assetArchives ?? []).find((entry) => { + const destination = import_node_path21.posix.normalize(safeRelativePath(entry.destination)).replace(/\/$/, ""); + return path === destination || path.startsWith(`${destination}/`); + }); + if (archive) { + fail(`${field} lies under the destination of ${archive.relativePath}; declare it in that entry's assetArchives[].executables instead.`); + } + } +} function effectiveScroll(scroll, adapter, targetId) { const layout = runtimeAdapter(scroll.runtime.id).layout(adapter); const runtime = layout.entryPoint === null ? { ...scroll.runtime } : { ...scroll.runtime, entryPoint: scroll.runtime.entryPoint ?? layout.entryPoint }; @@ -27387,6 +27487,16 @@ async function readExactScroll(reference) { const fragment = JSON.parse(await (0, import_promises16.readFile)((0, import_node_path21.resolve)(dir, "scroll.json"), "utf8")); const extended = fragment.extends !== void 0; const declared = extended ? joinScrollFragment(await readScrollBase(fragment, dir, normalized), fragment) : fragment; + const executableMembers = /* @__PURE__ */ new Set(); + if (Array.isArray(declared.assetArchives)) { + for (const archive of declared.assetArchives) { + if (!archive || typeof archive !== "object" || typeof archive.destination !== "string" || typeof archive.relativePath !== "string") continue; + for (const path of assetArchiveExecutablePaths(archive)) { + if (executableMembers.has(path)) fail(`assetArchives[].executables: ${JSON.stringify(path)} is declared by more than one archive.`); + executableMembers.add(path); + } + } + } const [scrollSchema, targetSchema, executionSchema] = await loadScrollSchemas(); const validationError = schemaValidationError(declared, scrollSchema, [targetSchema, executionSchema]); if (validationError) { @@ -27420,6 +27530,7 @@ async function readExactScroll(reference) { ...(scroll.localFiles ?? []).flatMap((file) => [file.sourcePath, file.relativePath]), ...scroll.prunePaths ?? [], ...scroll.uncompressedPaths ?? [], + ...scroll.executablePaths ?? [], ...scroll.selfTest.files, ...scroll.selfTest.script ? [scroll.selfTest.script] : [], ...scroll.execution?.script ? [scroll.execution.script] : [], @@ -27431,6 +27542,7 @@ async function readExactScroll(reference) { ]; for (const path of payloadPaths) safeRelativePath(path); assertDistinctPayloadDestinations(scroll); + assertEnvironmentExecutablePaths(scroll, runtime.layout(adapter).root); const [boxDirectory, targetDirectory] = parts; if (boxDirectory !== scroll.boxId) { fail(`Nested scroll box directory ${boxDirectory} does not match scroll boxId ${scroll.boxId}.`); @@ -27544,11 +27656,16 @@ function runSelfTest({ adapter, scroll, payloadDir, run: run2, extraCode = null ); const resolve6 = (argument) => argument.kind === "payload-path" ? (0, import_node_path22.join)(payloadDir, ...safeRelativePath(argument.value).split("/")) : argument.value; for (const invocation of invocations) { - run2(resolve6(invocation.command), invocation.args.map(resolve6), { - cwd: payloadDir, - env, - expectExitCode: invocation.expectExitCode - }); + try { + run2(resolve6(invocation.command), invocation.args.map(resolve6), { + cwd: payloadDir, + env, + expectExitCode: invocation.expectExitCode + }); + } catch (error2) { + if (adapter.host.platform === "win32") throw error2; + fail(`Self-test failed with the file modes the archive will carry. ${error2 instanceof Error ? error2.message : String(error2)} If a helper lost its executable bit, declare it in assetArchives[].executables (relative to the archive's destination), assets[].executable or localFiles[].executable, or in executablePaths if the solved environment installs it.`); + } } } async function writeNotice(payloadDir, name, value) { @@ -27664,12 +27781,21 @@ async function buildBox(name, options2 = {}) { for (const file of scroll.localFiles ?? []) { await copyVerifiedLocalFile(file, payloadDir, workspace.root); } + const archiveExecutablePaths = []; for (const archive of scroll.assetArchives ?? []) { - await expandAssetArchive(payloadDir, archive); + archiveExecutablePaths.push(...await expandAssetArchive(payloadDir, archive)); } for (const prunePath of scroll.prunePaths ?? []) { await (0, import_promises17.rm)((0, import_node_path22.join)(payloadDir, safeRelativePath(prunePath)), { recursive: true, force: true }); } + for (const archive of scroll.assetArchives ?? []) { + await assertAssetArchiveExecutables(payloadDir, archive); + } + for (const path of scroll.executablePaths ?? []) { + await assertPathWithoutLinks(payloadDir, path, "executablePaths", { + missingHint: "check the path against the solved environment and prunePaths" + }); + } for (const written of await runtimeBuilder(scroll.runtime.id).preparePayload?.(payloadDir) ?? []) { log(`Writing ${written}`); } @@ -27694,6 +27820,25 @@ async function buildBox(name, options2 = {}) { runtimeVersion: scroll.runtime.version, files: payloadFiles }); + const declaredExecutablePaths2 = [ + ...archiveExecutablePaths, + ...scroll.executablePaths ?? [], + ...[ + ...scroll.assets, + ...scroll.localFiles ?? [] + ].filter((entry) => entry.executable && entry.embed !== false).map((entry) => safeRelativePath(entry.relativePath)) + ]; + const archiveModes = { runtimeId: scroll.runtime.id, executablePaths: declaredExecutablePaths2 }; + if (scroll.execution) { + const { command } = runtimeAdapter(scroll.runtime.id).buildArgv({ + execution: scroll.execution, + target: adapter + }); + const commandPath = safeRelativePath(command.value); + if (!archiveMarksExecutable(adapter, scroll.runtime.id, declaredExecutablePaths2, commandPath)) { + fail(`This box runs ${commandPath}, which the archive would not mark executable. Declare it with "executable": true on the asset or local file that brings it in, in assetArchives[].executables relative to that archive's destination, or in executablePaths if the solved environment installs it.`); + } + } const provenance = { scrollId: scroll.scrollId, scrollVersion: scroll.scrollVersion, @@ -27750,6 +27895,7 @@ async function buildBox(name, options2 = {}) { provenance }, null, 2)} `); + await applyArchiveFileModes(payloadDir, adapter, archiveModes); if (codesignIdentity !== null) { log("Code signing Mach-O files"); const signed = await codesignPayload({ @@ -27784,6 +27930,7 @@ async function buildBox(name, options2 = {}) { const digestStream = payloadDigestStream(await payloadDigestEntries(payloadDir)); await (0, import_promises17.writeFile)((0, import_node_path22.join)(payloadDir, PAYLOAD_DIGEST_FILE), digestStream); const payloadDigestValue = { format: PAYLOAD_DIGEST_FORMAT, sha256: sha256Hex2(digestStream) }; + await applyArchiveFileModes(payloadDir, adapter, archiveModes); await normalizeTree(payloadDir); const installedSizeBytes = await payloadSize(payloadDir); await (0, import_promises17.mkdir)(workspace.distDir, { recursive: true }); @@ -27791,25 +27938,10 @@ async function buildBox(name, options2 = {}) { ...scroll.assets.map((asset) => safeRelativePath(asset.relativePath)), ...(scroll.uncompressedPaths ?? []).map((path) => safeRelativePath(path)) ]; - const declaredExecutablePaths2 = [ - ...scroll.assets, - ...scroll.localFiles ?? [] - ].filter((entry) => entry.executable && entry.embed !== false).map((entry) => safeRelativePath(entry.relativePath)); - if (scroll.execution) { - const { command } = runtimeAdapter(scroll.runtime.id).buildArgv({ - execution: scroll.execution, - target: adapter - }); - const commandPath = safeRelativePath(command.value); - if (!archiveMarksExecutable(adapter, scroll.runtime.id, declaredExecutablePaths2, commandPath)) { - fail(`This box runs ${commandPath}, which the archive would not mark executable. Declare it with "executable": true on the asset or local file that brings it in.`); - } - } log("Creating deterministic archive"); await createDeterministicZip(payloadDir, archivePath, adapter, { uncompressedPaths, - runtimeId: scroll.runtime.id, - executablePaths: declaredExecutablePaths2 + ...archiveModes }); log("Hashing deterministic archive"); const archiveSha = await sha256File(archivePath); diff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.md index 80f6ecc..c3119ec 100644 --- a/docs/concepts/architecture.md +++ b/docs/concepts/architecture.md @@ -110,9 +110,11 @@ reviewed copy. A licence problem is a legal problem, and it is cheaper to hit it expensive checks. **Self-test with the box's own interpreter.** The builder runs post-prune file assertions, the -target assertion, imports, and optional scroll `code`. Schema version 2 signs the target -assertion and import subset for a consumer to repeat; the richer scroll-only checks are not -misrepresented as consumer checks. +runtime's probe and optional scroll source. First it applies the archive writer's own mode rule +to regular payload files, so an undeclared helper cannot pass only on inherited build-directory +permissions. Declare such a helper on its asset, local file or archive entry. Windows validates +those declarations too and uses extensions. Schema version 3 signs the imports and commands a +consumer can repeat; the richer scroll-only checks remain builder checks. **Parity after the self-test, on the same payload.** There is no point comparing accelerators in a box that cannot import its dependencies in the first place. diff --git a/docs/concepts/design-decisions.md b/docs/concepts/design-decisions.md index 08917eb..c12c3ed 100644 --- a/docs/concepts/design-decisions.md +++ b/docs/concepts/design-decisions.md @@ -665,6 +665,46 @@ does not grant it by default: an entitlement is a security decision about the ca signing in the consuming application after extraction, which is too late — notarization inspects the application before anything is extracted. +## Expanded executable members are declared beside their archive + +`assetArchives[].executables` names regular members relative to the destination after stripping. +The build validates their origin and survival through pruning, then adds them to the same mode +rule as executable assets and local files. Upstream permission bits never decide what ships. +Windows checks the declarations too, while continuing to decide executability by extension. + +**Rejected:** a top-level executable-path list, which separates a member from the archive that +must supply it; and inheriting upstream modes, which would make the declaration incomplete and +let extraction behavior determine the result. The optional build-input field stays in version 3: +signed documents and ZIP interpretation are unchanged, older builders refuse the new property, +and existing consumers already preserve its synthesized executable bit. + +## Environment executables have a list of their own + +A package can install a helper binary in its own directory inside the prefix, outside the scripts +directory the runtime rule covers. Such a file has no scroll entry to carry a flag, so top-level +`executablePaths` names it: literal paths under `venv/`, checked as link-free regular files after +pruning and added to the same mode rule. The reader refuses a path an asset, local file or archive +destination owns, so this is not the top-level list rejected above — every executable still has +exactly one declaration, on whatever supplies it. + +**Rejected:** globs, which would declare whatever the solve happened to produce rather than what +the author reviewed; a literal path that moves with a new `python3.x` directory fails the build by +name instead. **Rejected:** marking a whole package directory, which would make data files +executable. Like the archive field, it is optional build input inside version 3. + +## A build self-test sees archive modes + +Before code signing, self-test and parity, the builder applies the archive writer's own mode rule +to every regular payload file. Links are skipped, and Windows keeps its extension-based behavior. +Modes are reapplied after probes and the final digest write so generated files also match. A +helper whose executable bit was never declared therefore fails at build time, where an installed +consumer already failed. The error points to the declarations while retaining the original cause. + +**Rejected:** validating an extracted second copy of the archive. That costs another full write, +read and temporary multi-gigabyte tree to check a rule already centralized in `archiveFileMode`. +Normalizing only declared files was also rejected: it would leave undeclared upstream executables +available during self-testing and preserve the defect. + ## Deliberately out of scope Publishing to object storage, downloading boxes, selecting or promoting a channel, updating an diff --git a/docs/guides/managing-assets.md b/docs/guides/managing-assets.md index fc193c0..9cc5c6c 100644 --- a/docs/guides/managing-assets.md +++ b/docs/guides/managing-assets.md @@ -85,6 +85,12 @@ When the upstream artefact is a tarball or zip, declare it as an asset and then - Extraction never overwrites a file already present in the destination. - `removeAfterExtract` defaults to `true`: the compressed original is dead weight inside the payload once unpacked. +- Add `"executables": ["bin/tool", "launch.sh"]` to the archive entry when expanded members must + run. Paths are relative to its `destination` after stripping. The build checks that each is a + distinct regular member, reaches it without traversing a link and still carries it after pruning. + Removing a declared executable is an error. Input archive modes never determine output modes. + Linux and macOS receive `0755`; Windows validates the list but keeps `0644` archive entries and + uses filename extensions. See the [complete rules](/reference/scroll#assetarchives). ## Compression diff --git a/docs/guides/updating-boxes.md b/docs/guides/updating-boxes.md index 6a7babd..b739f26 100644 --- a/docs/guides/updating-boxes.md +++ b/docs/guides/updating-boxes.md @@ -23,6 +23,16 @@ native binary; the update model does not change. ## Start with what changed +Upgrading to a builder that supports `assetArchives[].executables` also makes its self-test use +the exact modes the archive will carry. A helper that only ran because extraction left it +executable on the build machine now fails during the build, where installed users already failed. +Add the member's path to its archive entry's `executables`, relative to the destination after +stripping, then increment the box version and rebuild. Use `executable: true` for a direct asset +or local file, and [`executablePaths`](/reference/scroll#executablepaths) for a helper the solved +environment installs under `venv/`. See [archive declarations](/reference/scroll#assetarchives). +Older builders reject this new optional field; installed consumers already understand the ZIP +permissions and need no format upgrade. + diff --git a/docs/public/schema/v3/scroll.schema.json b/docs/public/schema/v3/scroll.schema.json index 7ab3c28..eea2fd9 100644 --- a/docs/public/schema/v3/scroll.schema.json +++ b/docs/public/schema/v3/scroll.schema.json @@ -215,6 +215,16 @@ }, "removeAfterExtract": { "type": "boolean" + }, + "executables": { + "type": "array", + "uniqueItems": true, + "description": "Regular archive members that need the executable bit, relative to this entry's destination after stripComponents. Paths must be distinct, use forward slashes and contain no dot segments or colons. Every path must be a member of this archive, exist before pruning, traverse no link and survive prunePaths and removeAfterExtract. The bit is synthesised from this declaration, never inherited from the upstream archive or build machine. Windows validates the same paths but writes 0644 and decides executability by extension. Empty or absent declares none.", + "items": { + "$ref": "#/$defs/payloadPath", + "type": "string", + "pattern": "^(?!.*(?:^|/)\\.(?:/|$))(?!.*[:\\u0000])(?!.*?/$).+$" + } } } } @@ -263,6 +273,16 @@ "$ref": "#/$defs/payloadPath" } }, + "executablePaths": { + "type": "array", + "uniqueItems": true, + "description": "Files the solved environment installs under venv/ that need the executable bit beyond the runtime's own scripts directory, such as a helper binary a package keeps in its own directory. Literal payload paths, with no glob expansion: forward slashes, no dot segments or colons. Each must be a regular file after pruning, traverse no link, and not be a file an asset, local file or asset archive brings in, which carry their own declaration. The bit is synthesised from this declaration, never inherited from the build machine. Windows validates the same paths but writes 0644 and decides executability by extension. Empty or absent declares none.", + "items": { + "$ref": "#/$defs/payloadPath", + "type": "string", + "pattern": "^(?!.*(?:^|/)\\.(?:/|$))(?!.*[:\\u0000])(?!.*?/$).+$" + } + }, "selfTest": { "type": "object", "additionalProperties": false, diff --git a/docs/reference/scroll.md b/docs/reference/scroll.md index 2bd54f3..32bfa88 100644 --- a/docs/reference/scroll.md +++ b/docs/reference/scroll.md @@ -185,7 +185,7 @@ leave `execution` half from each half, producing a `python-script` that inherite | --- | --- | | Scalars, and the cohesive objects `target`, `execution`, `parity` | The fragment replaces the base | | `assets`, `assetArchives`, `localFiles` | Joined, base entries first. Two entries claiming one `relativePath` is an **error** | -| `prunePaths`, `uncompressedPaths`, `selfTest.imports`, `selfTest.files` | Joined, base first, repeats dropped | +| `prunePaths`, `uncompressedPaths`, `executablePaths`, `selfTest.imports`, `selfTest.files` | Joined, base first, repeats dropped | | `compatibility`, `environment` | Joined key by key; on a shared key the fragment wins | | `selfTest.code` / `selfTest.script` | One slot: a fragment naming either replaces both | | `extends` | Dropped — the joined scroll extends nothing | @@ -593,7 +593,8 @@ refuses to overwrite them. "format": "tar.gz", "destination": "cache/hello", "stripComponents": 1, - "removeAfterExtract": true + "removeAfterExtract": true, + "executables": ["bin/tool", "launch.sh"] } ] ``` @@ -605,6 +606,28 @@ validated and expanded exactly like `tar.gz`. An archive is expanded at build ti "leave it out and let the caller fetch it" names nothing that could happen. Version 2 refused that combination with a cross-field check; version 3 makes it unspeakable. +`executables` optionally lists regular archive members that need the executable bit. Each path is +relative to this entry's `destination`, **after** `stripComponents`: the example marks +`cache/hello/bin/tool` and `cache/hello/launch.sh`. It works with all three formats and with either +value of `removeAfterExtract`. Empty and absent both declare none. Upstream modes are never inherited. + +Paths are literal, with no glob expansion. They use forward slashes, with no absolute or drive +paths, colons, empty or dot segments, or NULs. Each must come from this archive and be a regular +file before pruning; missing files, +directories, links, paths through links and duplicates fail with the field and path named. +Two archive entries cannot declare the same resulting payload path. A declaration removed by +`prunePaths` or `removeAfterExtract` also fails, even if a later step would recreate the file. + +On Linux and macOS, the declared file ships as `0755`; other files outside the runtime's executable +paths ship as `0644`. Windows validates every declaration identically, but archive file entries +remain `0644`: Windows decides executability by extension. The declaration also satisfies the +guard on a native box's execution binary. + +Edit this nested list in `scroll.json`; `edit scroll` intentionally handles scalar fields only. +Other editing commands and `refresh` preserve it, and base-plus-target joining keeps it with its +archive entry. This optional build-input field stays within schema version 3. Older Scrollcase +builders reject it as an unknown property; consumers need no upgrade to read the resulting ZIP. + ### `localFiles` Files copied from your own repository into the payload. Added and removed with @@ -669,9 +692,32 @@ automatically and does not need repeating here. The decision is taken from the scroll alone — nothing opens the file or reads its extension — which is what keeps two builds of the same commit byte-identical. +### `executablePaths` + +Files the solved environment installs under `venv/` that need the executable bit, beyond the +runtime's own scripts directory — a helper binary a package keeps in its own directory, for +instance. Without a declaration such a file ships as `0644`, whatever mode conda gave it on the +build machine. + +```jsonc +"executablePaths": ["venv/lib/python3.11/site-packages/triton/backends/nvidia/bin/ptxas"] +``` + +Paths are **literal**, with no glob expansion, and use the same spelling rules as +`assetArchives[].executables`: forward slashes, no dot segments, colons or trailing slash. The +reader refuses a path outside `venv/`, a path an asset or local file brings in (mark that entry +`"executable": true` instead) and a path under an `assetArchives` destination (declare it in that +entry's `executables`), so every executable has exactly one declaration. + +The build checks each path once the environment is unpacked and pruned: it must be a regular file +and traverse no link. A path the solve did not produce fails with the field named — which is also +what happens when an upgrade moves the file, for example to a new `python3.x` directory, so the +scroll cannot silently go stale. Linux and macOS receive `0755`; Windows validates the same paths +and keeps deciding executability by extension. + ::: warning Payload paths Every path inside the payload (`relativePath`, `destination`, `prunePaths`, `uncompressedPaths`, -`selfTest.files`) is relative and may never escape the payload root. Absolute paths, `..` segments, +`executablePaths`, `selfTest.files`) is relative and may never escape the payload root. Absolute paths, `..` segments, and drive letters are rejected. ::: @@ -711,6 +757,15 @@ Supply it later with [`edit scroll`](/reference/cli#edit-scroll), or per build w ## Self-test +The build applies the exact regular-file modes the archive will carry **before** running the +self-test and parity. On Linux and macOS an undeclared helper outside the runtime's executable +paths is `0644`, even when its upstream archive or conda says `0755`. Declare it with +`assetArchives[].executables`, `assets[].executable` or `localFiles[].executable` as appropriate, +or with [`executablePaths`](#executablepaths) when the solved environment installs it. +Self-test failures retain their original error and point to these declarations. Windows validates +the declarations identically and continues to use extensions rather than POSIX modes, so its +self-test failures carry no such hint. + Builder checks run with the payload's **own runtime** before the box is archived. Schema version 3 signs the probe for a consumer to repeat; it does not carry the richer file or `code` assertions. diff --git a/docs/white-paper.md b/docs/white-paper.md index ca3b42f..34d4edb 100644 --- a/docs/white-paper.md +++ b/docs/white-paper.md @@ -2246,9 +2246,10 @@ The optional fields are where a scroll expresses intent: | `condaDependencyLicenseAudit` | Path to the reviewed licence inventory the build must still match | | `bundledLicenseDeclaration` | Path to the reviewed licences of what was linked *inside* a binary the box ships — the half `pixi.lock` cannot see | | `publishBaseUrl` | Where the built archive and its signed documents will be published, so each can point at the next. Optional: a box built to run locally is never published, and the build then omits both links rather than inventing an address | -| `assetArchives` | Downloaded archives to expand into the payload, with `stripComponents` and `removeAfterExtract` | +| `assetArchives` | Downloaded archives to expand into the payload, with `stripComponents`, `removeAfterExtract` and optional destination-relative `executables` | | `localFiles` | Files copied from the project's own repository, optionally pinned to a declared hash | | `prunePaths` | Payload paths deleted before packing | +| `executablePaths` | Literal paths under `venv/` that the solved environment installs and that need the executable bit beyond the scripts directory | | `execution` | The application entry point | | `parity` | The cross-accelerator numerical gate | @@ -2901,10 +2902,10 @@ declares a workspace. Scroll structure is validated at runtime, from the shipped schemas, before tool discovery or any -build-directory mutation. The validator is written from scratch, in 194 lines, and implements the +build-directory mutation. The dependency-free validator implements the subset of JSON Schema 2020-12 the shipped schemas use: `$ref` (local pointers and absolute registered `$id`s), `const`, `enum`, `type`, `minLength`, `pattern`, numeric bounds, `minItems`, -`items`, `minProperties`, `required`, `properties`, `additionalProperties: false`, +`items`, `uniqueItems`, `minProperties`, `required`, `properties`, `additionalProperties: false`, `dependentRequired`, `allOf`, `oneOf`, `if`/`then`/`else` and `not`. @@ -3150,6 +3151,25 @@ surprising layout fails loudly rather than producing a wrong tree. The compresse removed after expansion unless the scroll asks otherwise, since it is dead weight inside the payload once unpacked. +`assetArchiveExecutablePaths()` resolves an entry's optional `executables` relative to its +destination after stripping. The scroll reader rejects unsafe paths and duplicate resulting paths +before tool discovery. `expandAssetArchive()` checks regular-file membership before pruning and +returns the declared payload paths; `assertAssetArchiveExecutables()` repeats the filesystem check +after pruning. A missing member, directory, link or link ancestor fails with its declaration named. +The extracted source is checked before copying, including ZIP links that a copy would otherwise +conceal as regular files. A pre-existing destination file cannot satisfy an archive declaration. +The original's removal and pruning cannot silently cancel a declaration. Empty and absent lists +both declare none, and Windows applies the same validation. +`assertPathWithoutLinks()` in `filesystem.mjs` supplies the shared lstat walk, also applied to +each extraction destination before copying so a link deeper below the destination cannot be used. + +Top-level `executablePaths` covers what no entry supplies: a file the solved environment installs +outside the scripts directory. `assertEnvironmentExecutablePaths()` in `scroll.mjs` refuses, at +read time, a path outside the runtime's environment root and any path an asset, local file or +archive destination already owns, so each executable keeps one declaration. `buildBox()` runs the +same lstat walk on each path after pruning, then adds the paths to the declared set that drives the +entry-point guard, the self-test modes and the archive. +
@@ -3330,6 +3350,16 @@ tested with the MPS fallback disabled rather than quietly falling back to CPU. T environment declaration is applied here too, before the target controls, so a bad runtime path fails the build while accelerator validation remains authoritative. +Before optional code signing and these checks, `applyArchiveFileModes()` in `archive.mjs` uses +the writer's own `archiveFileMode` rule on every collected regular file. Undeclared helpers outside +runtime paths are 0644 even if extraction left them 0755, so the self-test now finds the failure +that users of the installed archive already encountered. Failures retain the subprocess cause +and point to executable asset, local-file and archive-member declarations. Links are skipped so +an alias under the scripts directory cannot change its target's mode. Windows skips chmod and +uses extensions. Parity sees the same modes; after probes and the final digest write, modes are +applied again to cover newly generated files. The ZIP writer still synthesizes modes independently +from the same rule. Two metadata walks avoid extracting a second multi-gigabyte payload. + The code that runs is the adapter's platform assertion, then the declared imports, then the scroll's optional extra Python. Only the import subset reaches the signed release, with `timeoutSeconds: 180` recorded as the bound a consumer should apply when repeating it. The scroll's @@ -3498,7 +3528,7 @@ zip.addFile(join(payloadDir, ...entry.path.split('/')), entry.path, { compress: compressionLevel !== 0, compressionLevel, mtime: FIXED_ARCHIVE_TIME, - mode: archiveFileMode(adapter, entry.path), + mode: archiveFileMode(adapter, entry.path, executablePaths), forceDosTimestamp: true, }); ``` @@ -3506,13 +3536,14 @@ zip.addFile(join(payloadDir, ...entry.path.split('/')), entry.path, { Every variable that could differ between two runs has been removed. Entry order comes from the sorted enumeration; the timestamp is fixed and forced into DOS form so no local timezone leaks in; the compression level is pinned per path, because a different level produces different bytes for -identical input; and the mode comes from the **target adapter** rather than from the filesystem: +identical input; and the mode comes from the **target, runtime and scroll** rather than from the filesystem: | Path | Mode | Reason | | --- | --- | --- | | Windows target, anything | `0644` | Windows has no executable bit to preserve | | The interpreter entry point | `0755` | It must be executable after extraction | | Anything under the scripts directory | `0755` | Console scripts are executables | +| A declared executable asset, local file, expanded archive member or environment path | `0755` | The scroll explicitly requires it to run | | Everything else | `0644` | | Reading the mode from disk instead would make the archive depend on the umask of whoever ran the @@ -6345,6 +6376,12 @@ The two helpers are shared rather than duplicated: `consumer-box-fixture.mjs` bu and the external signer, and `consumer-conformance.mjs` builds the mutated archives the shared fixture describes. +The restrictive-umask conformance case extracts both a declared executable asset and a declared +archive member under `077`, asserting 0755 for each and 0644 for ordinary data. The JS fixture +expands a real synthetic TAR.GZ through `assetArchives[].executables`; Python and Rust describe +the resulting signed ZIP entries. Thus all three prove the installed mode without introducing a +second interpretation of the builder's scroll field into a consumer. +
### 11.4 The Python and Rust suites diff --git a/python/tests/conformance_support.py b/python/tests/conformance_support.py index 72e6f2a..ec147f4 100644 --- a/python/tests/conformance_support.py +++ b/python/tests/conformance_support.py @@ -149,6 +149,13 @@ def _fixture_options(spec: dict[str, Any]) -> dict[str, Any]: # The mode is synthesised from the scroll's declaration, and extraction has to hand it back # whatever umask the process is running under. options["extra_files"] = {"bin/tool": (b"#!/bin/sh\nexit 0\n", 0o755)} + if spec.get("executableArchiveMember"): + # The builder expanded a declared member and synthesized its ZIP mode. The consumer sees + # the result, without a second definition of the build-only scroll field in this package. + options.setdefault("extra_files", {}).update({ + "tools/bin/archive-tool": (b"#!/bin/sh\nexit 0\n", 0o755), + "tools/README": (b"ordinary data despite upstream executable permissions\n", 0o644), + }) if "environment" in spec: options["environment"] = spec["environment"] if "labels" in spec: diff --git a/rust/fixtures/consumer-conformance.json b/rust/fixtures/consumer-conformance.json index 9d3d787..1f7e463 100644 --- a/rust/fixtures/consumer-conformance.json +++ b/rust/fixtures/consumer-conformance.json @@ -1404,7 +1404,8 @@ "action": "prepare", "requiresPosixModes": true, "fixture": { - "executableAsset": true + "executableAsset": true, + "executableArchiveMember": true }, "runtime": { "umask": "077" @@ -1421,6 +1422,8 @@ "targetId": "$NATIVE_TARGET", "executableModes": { "bin/tool": "755", + "tools/bin/archive-tool": "755", + "tools/README": "644", "box.json": "644" } } diff --git a/rust/src/contract/payload_digest.rs b/rust/src/contract/payload_digest.rs index 2f9647b..ff107c2 100644 --- a/rust/src/contract/payload_digest.rs +++ b/rust/src/contract/payload_digest.rs @@ -261,7 +261,7 @@ mod tests { fn an_empty_payload_still_commits_to_its_format() { let stream = payload_digest_stream(&[]).unwrap(); assert_eq!(stream, format!("{PAYLOAD_DIGEST_FORMAT}\n").into_bytes()); - assert!(parse_payload_digest_stream(&stream).unwrap().is_empty()); + assert_eq!(parse_payload_digest_stream(&stream).unwrap(), Vec::::new()); } #[test] diff --git a/rust/src/environment.rs b/rust/src/environment.rs index 799f231..83a25b3 100644 --- a/rust/src/environment.rs +++ b/rust/src/environment.rs @@ -494,7 +494,7 @@ mod tests { )] }; let summary = resolve_environment(&options("linux", layers(), &[], false, false)).unwrap(); - assert!(summary.report.variables.is_empty()); + assert_eq!(summary.report.variables, Vec::::new()); assert_eq!(summary.report.remaining_variable_count, 3); assert!(!summary.report.is_worth_reporting()); diff --git a/rust/tests/conformance.rs b/rust/tests/conformance.rs index 3a976fd..b4738c4 100644 --- a/rust/tests/conformance.rs +++ b/rust/tests/conformance.rs @@ -213,6 +213,14 @@ impl Fixture { if spec.get("executableAsset").is_some() { entries.push(Entry::file("bin/tool", b"#!/bin/sh\nexit 0\n", 0o755)); } + // The builder expands the archive and synthesizes modes. Consumers receive this result, + // without needing to interpret the build-only executable-member declaration themselves. + if spec.get("executableArchiveMember").is_some() { + entries.push(Entry::file("tools/bin/archive-tool", b"#!/bin/sh\nexit 0\n", 0o755)); + entries.push(Entry::file( + "tools/README", b"ordinary data despite upstream executable permissions\n", 0o644, + )); + } if execution["kind"] == "python-module" { entries.push(Entry::file("example/application.py", b"print('module')\n", 0o644)); } else { diff --git a/rust/tests/prepare.rs b/rust/tests/prepare.rs index cc0b56f..65b3b50 100644 --- a/rust/tests/prepare.rs +++ b/rust/tests/prepare.rs @@ -51,7 +51,7 @@ fn preparing_a_box_yields_a_receipt_describing_what_was_verified() { assert_eq!(prepared.box_id(), "fixture-box"); assert_eq!(prepared.labels().unwrap()["model"], "fixture-model"); assert_eq!(prepared.signing_key_ids(), [support::KEY_ID]); - assert!(prepared.required_assets().is_empty()); + assert_eq!(prepared.required_assets(), []); assert!(prepared.execution().is_some()); // The receipt reports what was measured, and the tree is really there. assert!(destination.join("box.json").is_file()); diff --git a/src/build/archive.d.mts b/src/build/archive.d.mts index 620aa85..e23fb1e 100644 --- a/src/build/archive.d.mts +++ b/src/build/archive.d.mts @@ -13,6 +13,23 @@ * @returns {boolean} */ export function archiveMarksExecutable(adapter: import("../contract/targets.mjs").BoxTargetAdapter, runtimeId: string, declared: readonly string[], relativePath: string): boolean; +/** + * Gives the build tree the exact regular-file modes the archive will carry. + * + * A self-test against extracted upstream modes can pass on a helper the consumer receives as + * 0644. Ask the archive's own rule before testing, rather than maintaining a second definition or + * extracting a second multi-gigabyte tree. Links are never chmodded through an alias. Windows + * decides executability by extension and has no POSIX permissions to normalize. + * + * @param {string} payloadDir + * @param {import('../contract/targets.mjs').BoxTargetAdapter} adapter + * @param {{ runtimeId: string, executablePaths?: readonly string[] }} options + * @returns {Promise} + */ +export function applyArchiveFileModes(payloadDir: string, adapter: import("../contract/targets.mjs").BoxTargetAdapter, { runtimeId, executablePaths: declared }: { + runtimeId: string; + executablePaths?: readonly string[]; +}): Promise; /** * Streams a deterministic, Zip64-capable box archive using the pinned Node backend. * @@ -77,6 +94,6 @@ export function extractZipArchive(archivePath: string, destination: string): Pro * @param {'zip' | 'tar' | 'tar.gz'} format * @param {string} destination * @param {number} [stripComponents] - * @returns {Promise} + * @returns {Promise} extracted regular-file paths, relative to the destination */ -export function extractScrollArchive(archivePath: string, format: "zip" | "tar" | "tar.gz", destination: string, stripComponents?: number): Promise; +export function extractScrollArchive(archivePath: string, format: "zip" | "tar" | "tar.gz", destination: string, stripComponents?: number): Promise; diff --git a/src/build/archive.mjs b/src/build/archive.mjs index 4d2eb8c..fdaade4 100644 --- a/src/build/archive.mjs +++ b/src/build/archive.mjs @@ -22,8 +22,10 @@ import { findEntryThroughLink, findUnresolvableLink } from '../contract/links.mj import { isExecutablePayloadPath, runtimeAdapter } from '../contract/runtimes.mjs'; import { FIXED_ARCHIVE_TIME, + assertPathWithoutLinks, collectEntries, collectFiles, + collectRegularFiles, fileExists, safeRelativePath, validateExtractedTree, @@ -88,6 +90,27 @@ export function archiveMarksExecutable(adapter, runtimeId, declared, relativePat ); } +/** + * Gives the build tree the exact regular-file modes the archive will carry. + * + * A self-test against extracted upstream modes can pass on a helper the consumer receives as + * 0644. Ask the archive's own rule before testing, rather than maintaining a second definition or + * extracting a second multi-gigabyte tree. Links are never chmodded through an alias. Windows + * decides executability by extension and has no POSIX permissions to normalize. + * + * @param {string} payloadDir + * @param {import('../contract/targets.mjs').BoxTargetAdapter} adapter + * @param {{ runtimeId: string, executablePaths?: readonly string[] }} options + * @returns {Promise} + */ +export async function applyArchiveFileModes(payloadDir, adapter, { runtimeId, executablePaths: declared = [] }) { + if (adapter.host.platform === 'win32') return; + const rule = declaredExecutablePaths(runtimeId, adapter, declared); + for (const path of await collectRegularFiles(payloadDir)) { + await chmod(join(payloadDir, ...path.split('/')), archiveFileMode(adapter, path, rule) & 0o777); + } +} + /** * Whether a payload path was declared as one whose bytes are already compressed. * @@ -387,7 +410,7 @@ async function validateTarArchive(archivePath) { * @param {'zip' | 'tar' | 'tar.gz'} format * @param {string} destination * @param {number} [stripComponents] - * @returns {Promise} + * @returns {Promise} extracted regular-file paths, relative to the destination */ export async function extractScrollArchive(archivePath, format, destination, stripComponents = 0) { const tempRoot = await mkdtemp(join(tmpdir(), 'scrollcase-extract-')); @@ -406,6 +429,10 @@ export async function extractScrollArchive(archivePath, format, destination, str } else { fail(`Unsupported archive format: ${format}`); } + // A box may carry checked prefix links, but an asset archive is copied into an existing tree. + // Copying a ZIP link would silently turn it into a regular file and conceal its origin from + // executable-member validation, so the same strict boundary applies to every asset format. + await validateExtractedTree(tempRoot); let source = tempRoot; for (let index = 0; index < stripComponents; index += 1) { @@ -423,6 +450,7 @@ export async function extractScrollArchive(archivePath, format, destination, str const files = await collectFiles(source); // Archives may add a subtree beside verified assets, but must never replace those assets. for (const file of files) { + await assertPathWithoutLinks(destination, file, 'Scroll archive entry', { allowMissing: true }); if (await fileExists(join(destination, ...file.split('/')))) { fail(`Scroll archive entry already exists in destination: ${file}`); } @@ -433,6 +461,7 @@ export async function extractScrollArchive(archivePath, format, destination, str await mkdir(dirname(outputPath), { recursive: true }); await copyFile(join(source, ...file.split('/')), outputPath, constants.COPYFILE_EXCL); } + return files; } finally { await rm(tempRoot, { recursive: true, force: true }); } diff --git a/src/build/assets.mjs b/src/build/assets.mjs index 5f6dcdb..4b71ca9 100644 --- a/src/build/assets.mjs +++ b/src/build/assets.mjs @@ -9,10 +9,10 @@ import { createWriteStream } from 'node:fs'; import { copyFile, mkdir, rename, rm, stat } from 'node:fs/promises'; -import { dirname, join, sep } from 'node:path'; +import { dirname, join, posix, sep } from 'node:path'; import { pipeline } from 'node:stream/promises'; import { extractScrollArchive as extractArchive } from './archive.mjs'; -import { fileExists, safeRelativePath, sha256File } from './filesystem.mjs'; +import { assertPathWithoutLinks, fileExists, safeRelativePath, sha256File } from './filesystem.mjs'; import { fail } from './process.mjs'; const MAX_DOWNLOAD_ATTEMPTS = 5; @@ -114,6 +114,41 @@ export async function moveIntoPlace(source, destination) { } } +/** + * Resolves executable archive members without inheriting an upstream file mode. + * + * An archive entry owns its declarations: paths are relative to its destination after stripping, + * rather than a second top-level list whose relationship to the extraction would be implicit. + * Refusing aliases keeps both duplicate detection and diagnostics independent of the host OS. + * This is also asked while reading a scroll, before any download or toolchain invocation. + */ +export function assetArchiveExecutablePaths(archive) { + const paths = archive.executables === undefined ? [] : archive.executables; + const field = `assetArchives[].executables in ${archive.relativePath}`; + if (!Array.isArray(paths)) fail(`${field}: expected an array, received ${JSON.stringify(paths)}.`); + const seen = new Set(); + return paths.map((path) => { + try { + safeRelativePath(path); + if (typeof path !== 'string' || path.includes('\\') || path.includes(':') + || path.split('/').includes('.')) fail('Use a relative forward-slash path without dot segments or colons.'); + } catch (error) { + fail(`${field}: ${JSON.stringify(path)} is unsafe. ${error.message}`); + } + if (seen.has(path)) fail(`${field}: ${JSON.stringify(path)} is a duplicate.`); + seen.add(path); + return posix.join(safeRelativePath(archive.destination), path); + }); +} + +/** Validates the files again after staging or pruning; modes are irrelevant even on Windows. */ +export async function assertAssetArchiveExecutables(payloadDir, archive) { + const field = `assetArchives[].executables ${JSON.stringify(archive.executables ?? [])} in ${archive.relativePath}`; + for (const path of assetArchiveExecutablePaths(archive)) { + await assertPathWithoutLinks(payloadDir, path, field); + } +} + /** * Unpacks a downloaded archive into the payload tree. * @@ -123,9 +158,28 @@ export async function moveIntoPlace(source, destination) { * a wrong tree. */ export async function expandAssetArchive(payloadDir, archive) { + const executablePaths = assetArchiveExecutablePaths(archive); const archivePath = join(payloadDir, safeRelativePath(archive.relativePath)); const destination = join(payloadDir, safeRelativePath(archive.destination)); - await extractArchive(archivePath, archive.format, destination, Number(archive.stripComponents ?? 0)); + const field = `assetArchives[].executables ${JSON.stringify(archive.executables ?? [])} in ${archive.relativePath}`; + // A packed prefix may already hold links. Check before copying anything into it, not only after + // a declared member has been written through an ancestor whose name conceals its destination. + await assertPathWithoutLinks(payloadDir, archive.destination, field, { allowMissing: true, kind: 'directory' }); + let extracted; + try { + extracted = await extractArchive(archivePath, archive.format, destination, Number(archive.stripComponents ?? 0)); + } catch (error) { + if (executablePaths.length) fail(`${field}: ${error.message}`); + throw error; + } + await assertAssetArchiveExecutables(payloadDir, archive); + // Joined exactly as assetArchiveExecutablePaths joins, so membership never depends on spelling. + const members = new Set(extracted.map((path) => posix.join(safeRelativePath(archive.destination), path))); + for (const path of executablePaths) { + if (!members.has(path)) fail(`${field}: ${JSON.stringify(path)} is not a member of this archive.`); + } // The compressed original is dead weight inside the payload once unpacked. if (archive.removeAfterExtract !== false) await rm(archivePath, { force: true }); + await assertAssetArchiveExecutables(payloadDir, archive); + return executablePaths; } diff --git a/src/build/box.mjs b/src/build/box.mjs index dbb5e56..37b4cc9 100644 --- a/src/build/box.mjs +++ b/src/build/box.mjs @@ -2,9 +2,9 @@ * `build` — assemble the environment, prove it works, archive it, sign it. * * In order: solve and pack the conda-forge environment from the committed lock, fetch and unpack the - * declared assets, prune what is not needed at run time, audit dependency licences, self-test with - * the payload's *own* interpreter, normalise timestamps, zip deterministically, and emit a signed - * release plus a signed channel pointer. + * declared assets, prune what is not needed at run time, audit dependency licences, apply archive + * modes, self-test with the payload's *own* interpreter, normalise timestamps, zip deterministically, + * and emit a signed release plus a signed channel pointer. * * The self-test is the step that earns the box its name. The builder runs the platform assertion, * the probe the scroll declared, its file assertions and any extra source it named; the release @@ -29,9 +29,16 @@ import { payloadDigestStream, } from '../contract/payload-digest.mjs'; import { signDocument } from '../sign/index.mjs'; -import { copyVerifiedLocalFile, downloadVerified, expandAssetArchive, moveIntoPlace } from './assets.mjs'; -import { archiveMarksExecutable, createDeterministicZip } from './archive.mjs'; import { + assertAssetArchiveExecutables, + copyVerifiedLocalFile, + downloadVerified, + expandAssetArchive, + moveIntoPlace, +} from './assets.mjs'; +import { applyArchiveFileModes, archiveMarksExecutable, createDeterministicZip } from './archive.mjs'; +import { + assertPathWithoutLinks, collectFiles, fileExists, normalizeTree, @@ -98,11 +105,18 @@ function runSelfTest({ adapter, scroll, payloadDir, run, extraCode = null }) { ? join(payloadDir, ...safeRelativePath(argument.value).split('/')) : argument.value); for (const invocation of invocations) { - run(resolve(invocation.command), invocation.args.map(resolve), { - cwd: payloadDir, - env, - expectExitCode: invocation.expectExitCode, - }); + try { + run(resolve(invocation.command), invocation.args.map(resolve), { + cwd: payloadDir, + env, + expectExitCode: invocation.expectExitCode, + }); + } catch (error) { + // Windows has no POSIX modes to lose, so a hint about the executable bit would only mislead. + // Elsewhere the hint is conditional: most failures are an ordinary import error, not a mode. + if (adapter.host.platform === 'win32') throw error; + fail(`Self-test failed with the file modes the archive will carry. ${error instanceof Error ? error.message : String(error)} If a helper lost its executable bit, declare it in assetArchives[].executables (relative to the archive's destination), assets[].executable or localFiles[].executable, or in executablePaths if the solved environment installs it.`); + } } } @@ -280,14 +294,26 @@ export async function buildBox(name, options = {}) { } // An asset archive is expanded at build time and has no descriptor to defer, so the schema gives // it no `embed` field and there is nothing to skip here. + const archiveExecutablePaths = []; for (const archive of scroll.assetArchives ?? []) { - await expandAssetArchive(payloadDir, archive); + archiveExecutablePaths.push(...await expandAssetArchive(payloadDir, archive)); } // Drops what is only needed to build (tests, docs, bundled sample data). A box is a multi-gigabyte // download for an end user, so pruning is a user-facing concern rather than tidiness. for (const prunePath of scroll.prunePaths ?? []) { await rm(join(payloadDir, safeRelativePath(prunePath)), { recursive: true, force: true }); } + for (const archive of scroll.assetArchives ?? []) { + await assertAssetArchiveExecutables(payloadDir, archive); + } + // A declared environment file is checked once the prefix is unpacked and pruned: the path names + // what the solve produced, so a version bump that moved it fails here rather than shipping a box + // whose helper silently lost its bit. + for (const path of scroll.executablePaths ?? []) { + await assertPathWithoutLinks(payloadDir, path, 'executablePaths', { + missingHint: 'check the path against the solved environment and prunePaths', + }); + } // Whatever the runtime needs in the payload that nothing declares. It runs after the prunes, so // a project cannot prune a file the runtime is about to write, and before the payload is read, // so what it writes is archived and digested like everything else. @@ -321,6 +347,30 @@ export async function buildBox(name, options = {}) { runtimeVersion: scroll.runtime.version, files: payloadFiles, }); + // The same declaration set drives the entry-point guard, the self-test tree and the ZIP. An + // upstream mode is never a fourth source of executability, even temporarily during validation. + const declaredExecutablePaths = [ + ...archiveExecutablePaths, + ...(scroll.executablePaths ?? []), + ...[ + ...scroll.assets, + ...(scroll.localFiles ?? []), + ].filter((entry) => entry.executable && entry.embed !== false) + .map((entry) => safeRelativePath(entry.relativePath)), + ]; + const archiveModes = { runtimeId: scroll.runtime.id, executablePaths: declaredExecutablePaths }; + // Whatever the box starts must be runnable after extraction. Check before the self-test so a + // native entry point receives the declaration diagnostic rather than only an OS launch error. + if (scroll.execution) { + const { command } = runtimeAdapter(scroll.runtime.id).buildArgv({ + execution: scroll.execution, + target: adapter, + }); + const commandPath = safeRelativePath(command.value); + if (!archiveMarksExecutable(adapter, scroll.runtime.id, declaredExecutablePaths, commandPath)) { + fail(`This box runs ${commandPath}, which the archive would not mark executable. Declare it with "executable": true on the asset or local file that brings it in, in assetArchives[].executables relative to that archive's destination, or in executablePaths if the solved environment installs it.`); + } + } // Everything needed to answer "where did this box come from, and could I rebuild it?". const provenance = { scrollId: scroll.scrollId, @@ -393,6 +443,7 @@ export async function buildBox(name, options = {}) { provenance, }, null, 2)}\n`); + await applyArchiveFileModes(payloadDir, adapter, archiveModes); if (codesignIdentity !== null) { log('Code signing Mach-O files'); const signed = await codesignPayload({ @@ -437,6 +488,9 @@ export async function buildBox(name, options = {}) { const digestStream = payloadDigestStream(await payloadDigestEntries(payloadDir)); await writeFile(join(payloadDir, PAYLOAD_DIGEST_FILE), digestStream); const payloadDigestValue = { format: PAYLOAD_DIGEST_FORMAT, sha256: sha256Hex(digestStream) }; + // Probes may create files, and the digest itself was written afterwards. Finish their modes too + // so inspection of the build directory still sees exactly what extraction will produce. + await applyArchiveFileModes(payloadDir, adapter, archiveModes); await normalizeTree(payloadDir); const installedSizeBytes = await payloadSize(payloadDir); await mkdir(workspace.distDir, { recursive: true }); @@ -448,36 +502,10 @@ export async function buildBox(name, options = {}) { ...scroll.assets.map((asset) => safeRelativePath(asset.relativePath)), ...(scroll.uncompressedPaths ?? []).map((path) => safeRelativePath(path)), ]; - // The executable bit is synthesised, never read off the build machine, so what carries it has to - // be *declared*: the runtime's own rule covers the interpreter and the console scripts a conda - // prefix generates, and the scroll covers everything it brought in itself. A downloaded file - // arrives with no mode at all — HTTP carries content, not permissions — so without this a box - // could not ship an asset that runs. - const declaredExecutablePaths = [ - ...scroll.assets, - ...(scroll.localFiles ?? []), - ].filter((entry) => entry.executable && entry.embed !== false) - .map((entry) => safeRelativePath(entry.relativePath)); - // Whatever the box starts has to come out of the archive runnable. For an interpreted runtime - // that is the interpreter, and the runtime's own rule covers it; for a native one it is a file - // the scroll brought in, and only the scroll can say the bit belongs on it. Asked through the - // argv rule rather than by naming an execution kind here, so the guard holds for every runtime - // and keeps holding for the next one. - if (scroll.execution) { - const { command } = runtimeAdapter(scroll.runtime.id).buildArgv({ - execution: scroll.execution, - target: adapter, - }); - const commandPath = safeRelativePath(command.value); - if (!archiveMarksExecutable(adapter, scroll.runtime.id, declaredExecutablePaths, commandPath)) { - fail(`This box runs ${commandPath}, which the archive would not mark executable. Declare it with "executable": true on the asset or local file that brings it in.`); - } - } log('Creating deterministic archive'); await createDeterministicZip(payloadDir, archivePath, adapter, { uncompressedPaths, - runtimeId: scroll.runtime.id, - executablePaths: declaredExecutablePaths, + ...archiveModes, }); log('Hashing deterministic archive'); diff --git a/src/build/filesystem.d.mts b/src/build/filesystem.d.mts index 3e5bf04..149c0af 100644 --- a/src/build/filesystem.d.mts +++ b/src/build/filesystem.d.mts @@ -21,6 +21,26 @@ export function compareStableStrings(left: string, right: string): -1 | 0 | 1; * @throws {Error} when the path is absolute, empty, contains `..`, a drive letter or a NUL */ export function safeRelativePath(value: unknown): string; +/** + * Checks an archive destination or member without following any filesystem link. + * + * The lexical safety rule still runs first; lstat then catches the different case where a safe + * name reaches a prefix link already on disk. With `allowMissing`, the first absent component is + * where extraction will start creating directories. Without it, a declared member must already + * exist. Callers supply the field so one shared walk reports the declaration an author can fix, and + * the hint, because what usually removed a file depends on which declaration named it. + * + * @param {string} root + * @param {string} path + * @param {string} field + * @param {{ allowMissing?: boolean, kind?: 'file' | 'directory', missingHint?: string }} [options] + * @returns {Promise} + */ +export function assertPathWithoutLinks(root: string, path: string, field: string, { allowMissing, kind, missingHint, }?: { + allowMissing?: boolean; + kind?: "file" | "directory"; + missingHint?: string; +}): Promise; /** * Lists payload entries in the stable order used by hashing and archive creation. * diff --git a/src/build/filesystem.mjs b/src/build/filesystem.mjs index 9ac431d..6bc7620 100644 --- a/src/build/filesystem.mjs +++ b/src/build/filesystem.mjs @@ -69,6 +69,46 @@ export function safeRelativePath(value) { return normalized; } +/** + * Checks an archive destination or member without following any filesystem link. + * + * The lexical safety rule still runs first; lstat then catches the different case where a safe + * name reaches a prefix link already on disk. With `allowMissing`, the first absent component is + * where extraction will start creating directories. Without it, a declared member must already + * exist. Callers supply the field so one shared walk reports the declaration an author can fix, and + * the hint, because what usually removed a file depends on which declaration named it. + * + * @param {string} root + * @param {string} path + * @param {string} field + * @param {{ allowMissing?: boolean, kind?: 'file' | 'directory', missingHint?: string }} [options] + * @returns {Promise} + */ +export async function assertPathWithoutLinks(root, path, field, { + allowMissing = false, + kind = 'file', + missingHint = 'check prunePaths and removeAfterExtract', +} = {}) { + const parts = safeRelativePath(path).split('/'); + let current = root; + for (const [index, part] of parts.entries()) { + current = join(current, part); + let entry; + try { + entry = await lstat(current); + } catch (error) { + if (error.code !== 'ENOENT' && error.code !== 'ENOTDIR') throw error; + if (allowMissing) return; + fail(`${field}: ${JSON.stringify(path)} is missing (${missingHint}).`); + } + if (entry.isSymbolicLink()) fail(`${field}: ${JSON.stringify(path)} is a link or traverses a link at ${parts.slice(0, index + 1).join('/')}.`); + const needsDirectory = index < parts.length - 1 || kind === 'directory'; + if (needsDirectory ? !entry.isDirectory() : !entry.isFile()) { + fail(`${field}: ${JSON.stringify(path)} must be a regular file with directory ancestors.`); + } + } +} + /** * Lists payload entries in the stable order used by hashing and archive creation. * diff --git a/src/build/schema-validation.mjs b/src/build/schema-validation.mjs index f9fd966..e76a7d9 100644 --- a/src/build/schema-validation.mjs +++ b/src/build/schema-validation.mjs @@ -98,6 +98,14 @@ function validate(value, schema, context, path, errors) { errors.push(`${path} must contain at least ${schema.minItems} item${schema.minItems === 1 ? '' : 's'}`); return; } + if (schema.uniqueItems) { + for (let index = 0; index < value.length; index += 1) { + if (value.slice(0, index).some((item) => isDeepStrictEqual(item, value[index]))) { + errors.push(`${path}[${index}] is a duplicate item`); + return; + } + } + } if (schema.items) { for (let index = 0; index < value.length && errors.length === 0; index += 1) { validate(value[index], schema.items, context, `${path}[${index}]`, errors); diff --git a/src/build/scroll-edit.mjs b/src/build/scroll-edit.mjs index b3ad2f3..41e2774 100644 --- a/src/build/scroll-edit.mjs +++ b/src/build/scroll-edit.mjs @@ -531,7 +531,7 @@ export async function removeSelfTestCommand({ boxId, target, args }) { const UNEDITABLE_FIELDS = Object.freeze(new Set([ '$schema', 'schemaVersion', 'extends', 'boxId', 'target', 'runtime', 'labels', 'compatibility', 'environment', 'assets', 'assetArchives', 'localFiles', - 'prunePaths', 'uncompressedPaths', 'selfTest', 'execution', 'parity', + 'prunePaths', 'uncompressedPaths', 'executablePaths', 'selfTest', 'execution', 'parity', ])); /** diff --git a/src/build/scroll.mjs b/src/build/scroll.mjs index 7dcc2b0..63a62c7 100644 --- a/src/build/scroll.mjs +++ b/src/build/scroll.mjs @@ -13,7 +13,7 @@ */ import { readFile, readdir } from 'node:fs/promises'; -import { join, resolve, sep } from 'node:path'; +import { join, posix, resolve, sep } from 'node:path'; import { boxTargetAdapter, boxTargetId } from '../contract/targets.mjs'; import { assertRuntimeEntryPoint, @@ -26,6 +26,7 @@ import { compareStableStrings, fileExists, safeRelativePath } from './filesystem import { fail, runResult } from './process.mjs'; import { schemaValidationError } from './schema-validation.mjs'; import { getWorkspace } from './workspace.mjs'; +import { assetArchiveExecutablePaths } from './assets.mjs'; const scrollSchemaUrl = new URL('../contract/schema/scroll.schema.json', import.meta.url); const targetSchemaUrl = new URL('../contract/schema/target.schema.json', import.meta.url); @@ -60,7 +61,7 @@ const JOINED_ENTRY_LISTS = Object.freeze(['assets', 'assetArchives', 'localFiles * importing a module twice is idempotent, and refusing a base and a fragment that both name `json` * would be hostile for no gain. */ -const JOINED_STRING_LISTS = Object.freeze(['prunePaths', 'uncompressedPaths']); +const JOINED_STRING_LISTS = Object.freeze(['prunePaths', 'uncompressedPaths', 'executablePaths']); /** * Open maps: joined key by key, fragment winning a shared key. @@ -149,7 +150,7 @@ function joinSelfTests(base = {}, fragment = {}) { * | --- | --- | * | Scalars, and the cohesive objects `target`, `runtime`, `execution`, `parity` | The fragment replaces the base | * | `assets`, `assetArchives`, `localFiles` | Joined base-first; a repeated `relativePath` is an error | - * | `prunePaths`, `uncompressedPaths`, `selfTest.imports`, `selfTest.files` | Joined base-first, repeats dropped | + * | `prunePaths`, `uncompressedPaths`, `executablePaths`, `selfTest.imports`, `selfTest.files` | Joined base-first, repeats dropped | * | `selfTest.commands` | Joined base-first, repeats kept | * | `compatibility`, `environment` | Joined key by key, the fragment winning a shared key | * | `extends` | Dropped: the joined scroll extends nothing | @@ -224,6 +225,39 @@ export function scrollDirectory(reference) { return path; } +/** + * Checks what `executablePaths` may name before anything is downloaded or solved. + * + * The field exists for one gap: a file the solved environment installs outside the runtime's + * scripts directory has no entry of its own to carry a flag. Anything an asset, local file or asset + * archive brings in already has one, so it keeps that single owner rather than a second declaration + * whose relationship to the first would be implicit. Whether the file exists is a build question, + * answered once the environment is unpacked and pruned. + * + * @param {Record} scroll + * @param {string} environmentRoot + */ +function assertEnvironmentExecutablePaths(scroll, environmentRoot) { + const entryPaths = new Set([...scroll.assets, ...(scroll.localFiles ?? [])] + .map((entry) => posix.normalize(safeRelativePath(entry.relativePath)))); + for (const path of scroll.executablePaths ?? []) { + const field = `executablePaths: ${JSON.stringify(path)}`; + if (!path.startsWith(`${environmentRoot}/`)) { + fail(`${field} is outside ${environmentRoot}/. This field marks files the solved environment installs; declare a file you bring in on its asset, local file or archive entry.`); + } + if (entryPaths.has(path)) { + fail(`${field} is brought in by an asset or local file; mark that entry "executable": true instead.`); + } + const archive = (scroll.assetArchives ?? []).find((entry) => { + const destination = posix.normalize(safeRelativePath(entry.destination)).replace(/\/$/, ''); + return path === destination || path.startsWith(`${destination}/`); + }); + if (archive) { + fail(`${field} lies under the destination of ${archive.relativePath}; declare it in that entry's assetArchives[].executables instead.`); + } + } +} + /** * Fills in everything a scroll does not have to say twice. * @@ -268,6 +302,19 @@ async function readExactScroll(reference) { const declared = extended ? joinScrollFragment(await readScrollBase(fragment, dir, normalized), fragment) : fragment; + // Report the path as well as the field, including for duplicates the schema would otherwise + // describe only by array index. Malformed containers are left to the schema validator. + const executableMembers = new Set(); + if (Array.isArray(declared.assetArchives)) { + for (const archive of declared.assetArchives) { + if (!archive || typeof archive !== 'object' + || typeof archive.destination !== 'string' || typeof archive.relativePath !== 'string') continue; + for (const path of assetArchiveExecutablePaths(archive)) { + if (executableMembers.has(path)) fail(`assetArchives[].executables: ${JSON.stringify(path)} is declared by more than one archive.`); + executableMembers.add(path); + } + } + } const [scrollSchema, targetSchema, executionSchema] = await loadScrollSchemas(); const validationError = schemaValidationError(declared, scrollSchema, [targetSchema, executionSchema]); if (validationError) { @@ -312,6 +359,7 @@ async function readExactScroll(reference) { ...(scroll.localFiles ?? []).flatMap((file) => [file.sourcePath, file.relativePath]), ...(scroll.prunePaths ?? []), ...(scroll.uncompressedPaths ?? []), + ...(scroll.executablePaths ?? []), ...scroll.selfTest.files, ...(scroll.selfTest.script ? [scroll.selfTest.script] : []), ...(scroll.execution?.script ? [scroll.execution.script] : []), @@ -323,6 +371,7 @@ async function readExactScroll(reference) { ]; for (const path of payloadPaths) safeRelativePath(path); assertDistinctPayloadDestinations(scroll); + assertEnvironmentExecutablePaths(scroll, runtime.layout(adapter).root); const [boxDirectory, targetDirectory] = parts; if (boxDirectory !== scroll.boxId) { fail(`Nested scroll box directory ${boxDirectory} does not match scroll boxId ${scroll.boxId}.`); diff --git a/src/contract/fixtures/consumer-conformance.json b/src/contract/fixtures/consumer-conformance.json index 9d3d787..1f7e463 100644 --- a/src/contract/fixtures/consumer-conformance.json +++ b/src/contract/fixtures/consumer-conformance.json @@ -1404,7 +1404,8 @@ "action": "prepare", "requiresPosixModes": true, "fixture": { - "executableAsset": true + "executableAsset": true, + "executableArchiveMember": true }, "runtime": { "umask": "077" @@ -1421,6 +1422,8 @@ "targetId": "$NATIVE_TARGET", "executableModes": { "bin/tool": "755", + "tools/bin/archive-tool": "755", + "tools/README": "644", "box.json": "644" } } diff --git a/src/contract/schema/scroll.schema.json b/src/contract/schema/scroll.schema.json index 7ab3c28..eea2fd9 100644 --- a/src/contract/schema/scroll.schema.json +++ b/src/contract/schema/scroll.schema.json @@ -215,6 +215,16 @@ }, "removeAfterExtract": { "type": "boolean" + }, + "executables": { + "type": "array", + "uniqueItems": true, + "description": "Regular archive members that need the executable bit, relative to this entry's destination after stripComponents. Paths must be distinct, use forward slashes and contain no dot segments or colons. Every path must be a member of this archive, exist before pruning, traverse no link and survive prunePaths and removeAfterExtract. The bit is synthesised from this declaration, never inherited from the upstream archive or build machine. Windows validates the same paths but writes 0644 and decides executability by extension. Empty or absent declares none.", + "items": { + "$ref": "#/$defs/payloadPath", + "type": "string", + "pattern": "^(?!.*(?:^|/)\\.(?:/|$))(?!.*[:\\u0000])(?!.*?/$).+$" + } } } } @@ -263,6 +273,16 @@ "$ref": "#/$defs/payloadPath" } }, + "executablePaths": { + "type": "array", + "uniqueItems": true, + "description": "Files the solved environment installs under venv/ that need the executable bit beyond the runtime's own scripts directory, such as a helper binary a package keeps in its own directory. Literal payload paths, with no glob expansion: forward slashes, no dot segments or colons. Each must be a regular file after pruning, traverse no link, and not be a file an asset, local file or asset archive brings in, which carry their own declaration. The bit is synthesised from this declaration, never inherited from the build machine. Windows validates the same paths but writes 0644 and decides executability by extension. Empty or absent declares none.", + "items": { + "$ref": "#/$defs/payloadPath", + "type": "string", + "pattern": "^(?!.*(?:^|/)\\.(?:/|$))(?!.*[:\\u0000])(?!.*?/$).+$" + } + }, "selfTest": { "type": "object", "additionalProperties": false, diff --git a/src/contract/types/index.d.ts b/src/contract/types/index.d.ts index e3241e4..8927f98 100644 --- a/src/contract/types/index.d.ts +++ b/src/contract/types/index.d.ts @@ -214,6 +214,10 @@ export interface BoxScroll { destination: PayloadPath; stripComponents?: number; removeAfterExtract?: boolean; + /** + * Regular archive members that need the executable bit, relative to this entry's destination after stripComponents. Paths must be distinct, use forward slashes and contain no dot segments or colons. Every path must be a member of this archive, exist before pruning, traverse no link and survive prunePaths and removeAfterExtract. The bit is synthesised from this declaration, never inherited from the upstream archive or build machine. Windows validates the same paths but writes 0644 and decides executability by extension. Empty or absent declares none. + */ + executables?: string[]; }[]; /** * Files copied from the consumer's own repository into the payload. A file already under the project's own version control needs no second copy of its identity here, and what ships is hashed into the signed release either way; declaring sha256 pins one that must not change without review, which suits a licence notice and not a script still being written. @@ -238,6 +242,10 @@ export interface BoxScroll { * Payload paths stored in the archive instead of deflated, because their bytes are already compressed and re-compressing them costs build time while making the archive marginally larger. A path matches itself and everything beneath it, so one entry can name a single large file or the directory an expanded asset archive landed in. Declared assets are stored automatically; this is for anything else the project knows to be already compressed. */ uncompressedPaths?: PayloadPath[]; + /** + * Files the solved environment installs under venv/ that need the executable bit beyond the runtime's own scripts directory, such as a helper binary a package keeps in its own directory. Literal payload paths, with no glob expansion: forward slashes, no dot segments or colons. Each must be a regular file after pruning, traverse no link, and not be a file an asset, local file or asset archive brings in, which carry their own declaration. The bit is synthesised from this declaration, never inherited from the build machine. Windows validates the same paths but writes 0644 and decides executability by extension. Empty or absent declares none. + */ + executablePaths?: string[]; /** * Builder checks run against the payload before archiving. The signed release carries the probe — the imports and commands a consumer can repeat after extraction — while file assertions and the optional extra source stay builder-only. At least one of imports and commands is required: a box that proves nothing about itself is not a box worth signing. */ diff --git a/tests/helpers/asset-archive.mjs b/tests/helpers/asset-archive.mjs new file mode 100644 index 0000000..43e24ff --- /dev/null +++ b/tests/helpers/asset-archive.mjs @@ -0,0 +1,43 @@ +/** Small third-party archives: real extraction inputs, with no network or installed toolchain. */ +import { createWriteStream } from 'node:fs'; +import { chmod, mkdir, readFile, symlink, writeFile } from 'node:fs/promises'; +import { dirname, join } from 'node:path'; +import { pipeline } from 'node:stream/promises'; +import * as tar from 'tar'; +import yazl from 'yazl'; + +export async function writeAssetArchive(root, { + format = 'tar.gz', + files = { 'wrapper/bin/tool': '#!/bin/sh\nprintf "archive-member-ok\\n"\n' }, + links = {}, +} = {}) { + await mkdir(root, { recursive: true }); + const path = join(root, `asset.${format}`); + if (format === 'zip') { + const zip = new yazl.ZipFile(); + const output = pipeline(zip.outputStream, createWriteStream(path)); + for (const [name, contents] of Object.entries(files)) { + zip.addBuffer(Buffer.from(contents), name, { mode: 0o100755 }); + } + for (const [name, target] of Object.entries(links)) { + zip.addBuffer(Buffer.from(target), name, { mode: 0o120777 }); + } + zip.end(); + await output; + } else { + const source = join(root, 'source'); + for (const [name, contents] of Object.entries(files)) { + await mkdir(dirname(join(source, name)), { recursive: true }); + await writeFile(join(source, name), contents); + if (process.platform !== 'win32') await chmod(join(source, name), 0o755); + } + for (const [name, target] of Object.entries(links)) { + await mkdir(dirname(join(source, name)), { recursive: true }); + await symlink(target, join(source, name)); + } + await tar.c({ file: path, cwd: source, gzip: format === 'tar.gz' }, [ + ...Object.keys(files), ...Object.keys(links), + ]); + } + return { path, bytes: await readFile(path) }; +} diff --git a/tests/helpers/consumer-box-fixture.mjs b/tests/helpers/consumer-box-fixture.mjs index f81ca9c..bc94cc4 100644 --- a/tests/helpers/consumer-box-fixture.mjs +++ b/tests/helpers/consumer-box-fixture.mjs @@ -1,5 +1,6 @@ import { createHash } from 'node:crypto'; import { + copyFile, mkdir, mkdtemp, stat, @@ -18,6 +19,8 @@ import { BOX_SCHEMA_VERSION, documentKinds } from '../../src/contract/documents. import { boxTargetAdapter } from '../../src/contract/targets.mjs'; import { runtimeAdapter } from '../../src/contract/runtimes.mjs'; import { generateSigningKey, signDocument } from '../../src/sign/index.mjs'; +import { expandAssetArchive } from '../../src/build/assets.mjs'; +import { writeAssetArchive } from './asset-archive.mjs'; export function nativeTarget() { if (process.platform === 'darwin' && process.arch === 'arm64') { @@ -74,6 +77,7 @@ export async function createConsumerBoxFixture({ labels = undefined, runtimeId = 'python', executablePaths = [], + executableArchiveMember = false, extraFiles = {}, } = {}) { const root = await mkdtemp(join(tmpdir(), 'scrollcase-consumer-fixture-')); @@ -91,6 +95,23 @@ export async function createConsumerBoxFixture({ await writeFile(filePath, contents); } + const declaredPaths = [...executablePaths]; + if (executableArchiveMember) { + // Exercise the build-side declaration and extraction too: consumers see only the resulting + // signed ZIP, whose modes must survive the restrictive extraction umask in the shared case. + const { path } = await writeAssetArchive(join(root, 'upstream'), { + files: { + 'wrapper/bin/archive-tool': '#!/bin/sh\nexit 0\n', + 'wrapper/README': 'ordinary data despite upstream executable permissions\n', + }, + }); + await copyFile(path, join(payload, 'source.tar.gz')); + declaredPaths.push(...await expandAssetArchive(payload, { + relativePath: 'source.tar.gz', format: 'tar.gz', destination: 'tools', + stripComponents: 1, executables: ['bin/archive-tool'], + })); + } + if (execution?.kind === 'python-script') { const scriptPath = join(payload, ...execution.script.split('/')); await mkdir(dirname(scriptPath), { recursive: true }); @@ -142,7 +163,7 @@ export async function createConsumerBoxFixture({ } const installedSizeBytes = await payloadSize(payload); const archivePath = join(root, 'box.zip'); - await createDeterministicZip(payload, archivePath, adapter, { runtimeId, executablePaths }); + await createDeterministicZip(payload, archivePath, adapter, { runtimeId, executablePaths: declaredPaths }); const archiveMetadata = await stat(archivePath); const release = { ...shared, diff --git a/tests/helpers/consumer-conformance.mjs b/tests/helpers/consumer-conformance.mjs index 9bb4a98..4286761 100644 --- a/tests/helpers/consumer-conformance.mjs +++ b/tests/helpers/consumer-conformance.mjs @@ -393,6 +393,7 @@ function fixtureOptions(spec = {}) { payloadDigest: spec.payloadDigest !== false, environment: spec.environment, executablePaths, + executableArchiveMember: spec.executableArchiveMember ?? false, ...(spec.executableAsset ? { extraFiles: { 'bin/tool': '#!/bin/sh\nexit 0\n' } } : {}), ...(spec.labels ? { labels: spec.labels } : {}), }; diff --git a/tests/unit/archive-security.test.mjs b/tests/unit/archive-security.test.mjs index a272187..17caa4b 100644 --- a/tests/unit/archive-security.test.mjs +++ b/tests/unit/archive-security.test.mjs @@ -1,5 +1,7 @@ import { createWriteStream } from 'node:fs'; import { + chmod, + lstat, mkdir, mkdtemp, readFile, @@ -14,10 +16,13 @@ import { afterEach, describe, expect, it } from 'vitest'; import * as tar from 'tar'; import yazl from 'yazl'; import { + applyArchiveFileModes, + createDeterministicZip, extractScrollArchive, extractZipArchive, listZipEntries, } from '../../src/build/archive.mjs'; +import { boxTargetAdapters } from '../../src/contract/targets.mjs'; import { collectFiles, fileExists, safeRelativePath } from '../../src/build/filesystem.mjs'; const created = []; @@ -42,6 +47,31 @@ async function writeZip(path, entries) { } describe('archive boundaries', () => { + it.runIf(process.platform !== 'win32')('normalises regular modes without chmod through a script-directory link', async () => { + const root = await scratch(); + await mkdir(join(root, 'venv', 'bin'), { recursive: true }); + await writeFile(join(root, 'data'), 'data'); + await chmod(join(root, 'data'), 0o755); + await symlink('../../data', join(root, 'venv', 'bin', 'alias')); + const adapter = boxTargetAdapters().find(({ platform }) => platform === 'linux'); + await applyArchiveFileModes(root, adapter, { runtimeId: 'python' }); + expect((await lstat(join(root, 'data'))).mode & 0o777).toBe(0o644); + expect((await lstat(join(root, 'venv', 'bin', 'alias'))).isSymbolicLink()).toBe(true); + }); + for (const adapter of boxTargetAdapters()) { + it(`synthesises declared-member modes for ${adapter.id} independently of the host`, async () => { + const root = await scratch(); + const payload = join(root, 'payload'); + await mkdir(payload); + await writeFile(join(payload, 'tool'), 'executable'); + await writeFile(join(payload, 'data'), 'ordinary'); + const archive = join(root, 'box.zip'); + await createDeterministicZip(payload, archive, adapter, { runtimeId: 'native', executablePaths: ['tool'] }); + const entries = await listZipEntries(archive); + expect(entries.find(({ path }) => path === 'tool').mode).toBe(adapter.platform === 'windows' ? 0o644 : 0o755); + expect(entries.find(({ path }) => path === 'data').mode).toBe(0o644); + }); + } it('rejects traversal and absolute path spellings before joining them to a destination', () => { for (const path of ['', '../escape', 'safe/../../escape', '/absolute', 'C:/absolute', 'a//b']) { expect(() => safeRelativePath(path), path).toThrow(/Unsafe relative path/); diff --git a/tests/unit/assets.test.mjs b/tests/unit/assets.test.mjs index e51b88e..01b1dcf 100644 --- a/tests/unit/assets.test.mjs +++ b/tests/unit/assets.test.mjs @@ -1,10 +1,12 @@ import { createHash } from 'node:crypto'; -import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { copyFile, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { afterEach, describe, expect, it } from 'vitest'; -import { downloadVerified } from '../../src/build/assets.mjs'; +import { downloadVerified, expandAssetArchive } from '../../src/build/assets.mjs'; import { fileExists } from '../../src/build/filesystem.mjs'; +import { writeAssetArchive } from '../helpers/asset-archive.mjs'; +import { schemaValidationError } from '../../src/build/schema-validation.mjs'; const created = []; afterEach(async () => { @@ -19,6 +21,114 @@ async function scratch() { const sha256 = (bytes) => createHash('sha256').update(bytes).digest('hex'); +describe('executable members of asset archives', () => { + it('enforces the schema uniqueness keyword as well as the field diagnostic', async () => { + const scrollSchema = JSON.parse(await readFile(new URL('../../src/contract/schema/scroll.schema.json', import.meta.url), 'utf8')); + const schema = { + ...scrollSchema.properties.assetArchives.items.properties.executables, + $defs: scrollSchema.$defs, + }; + expect(schemaValidationError(['bin/tool', 'bin/tool'], schema)).toMatch(/duplicate/); + expect(schemaValidationError(['bin/tool', 'bin/other'], schema)).toBeNull(); + }); + async function staged(format = 'tar.gz', overrides = {}, contents = {}) { + const root = await scratch(); + const { path } = await writeAssetArchive(join(root, 'upstream'), { format, ...contents }); + const payload = join(root, 'payload'); + await mkdir(payload); + await copyFile(path, join(payload, `source.${format}`)); + return { + payload, + archive: { + relativePath: `source.${format}`, format, destination: 'tools', + stripComponents: 1, executables: ['bin/tool'], ...overrides, + }, + }; + } + + for (const format of ['zip', 'tar', 'tar.gz']) { + for (const stripComponents of [0, 1]) { + for (const removeAfterExtract of [true, false]) { + it(`validates ${format} members with strip=${stripComponents}, remove=${removeAfterExtract}`, async () => { + const member = stripComponents ? 'bin/tool' : 'wrapper/bin/tool'; + const { payload, archive } = await staged(format, { + stripComponents, removeAfterExtract, executables: [member], + }); + await expect(expandAssetArchive(payload, archive)).resolves.toEqual([`tools/${member}`]); + expect(await fileExists(join(payload, `source.${format}`))).toBe(!removeAfterExtract); + }); + } + } + } + + for (const path of ['missing', 'bin', '../escape', '/absolute', 'C:/drive', 'C:drive', + 'bin/../tool', './bin/tool', 'bin/./tool', 'bin//tool', 'bin/tool/', 'bin\\tool', 'bad\0path', '']) { + it(`refuses the invalid member ${JSON.stringify(path)} and names its field`, async () => { + const { payload, archive } = await staged('tar.gz', { executables: [path] }); + const error = await expandAssetArchive(payload, archive).catch((error) => error); + expect(error).toBeInstanceOf(Error); + expect(error.message).toContain('assetArchives[].executables'); + expect(error.message).toContain(JSON.stringify(path)); + expect(await fileExists(join(payload, archive.relativePath))).toBe(true); + }); + } + + it('refuses a duplicate declaration before extraction', async () => { + const { payload, archive } = await staged('zip', { executables: ['bin/tool', 'bin/tool'] }); + await expect(expandAssetArchive(payload, archive)).rejects.toThrow(/assetArchives\[\]\.executables.*bin\/tool.*duplicate/i); + expect(await fileExists(join(payload, 'tools'))).toBe(false); + }); + + it('does not borrow a regular file already in the destination', async () => { + const { payload, archive } = await staged('tar', { executables: ['other'] }); + await mkdir(join(payload, 'tools')); + await writeFile(join(payload, 'tools', 'other'), 'from elsewhere'); + await expect(expandAssetArchive(payload, archive)).rejects.toThrow(/assetArchives\[\]\.executables.*other.*member/); + }); + + it('accepts empty and absent declarations', async () => { + for (const executables of [undefined, []]) { + const { payload, archive } = await staged('zip', { executables }); + await expect(expandAssetArchive(payload, archive)).resolves.toEqual([]); + } + }); + + for (const format of ['zip', 'tar', 'tar.gz']) { + it(`refuses a declared link in ${format} before it can become a copied regular file`, async () => { + const { payload, archive } = await staged(format, { executables: ['bin/alias'] }, { + links: { 'wrapper/bin/alias': 'tool' }, + }); + await expect(expandAssetArchive(payload, archive)).rejects.toThrow(/assetArchives\[\]\.executables.*bin\/alias.*link/); + }); + } + + it('refuses a destination traversing a link', async () => { + const { payload, archive } = await staged(); + await mkdir(join(payload, 'real')); + await symlink('real', join(payload, 'tools'), 'dir'); + await expect(expandAssetArchive(payload, archive)).rejects.toThrow(/assetArchives\[\]\.executables.*bin\/tool.*link/); + expect(await fileExists(join(payload, 'real', 'bin', 'tool'))).toBe(false); + }); + + it('refuses a link below the destination before copying through it', async () => { + const { payload, archive } = await staged(); + const outside = join(payload, '..', 'outside'); + await mkdir(outside); + await mkdir(join(payload, 'tools')); + await symlink(outside, join(payload, 'tools', 'bin'), 'dir'); + await expect(expandAssetArchive(payload, archive)).rejects.toThrow(/assetArchives\[\]\.executables.*bin\/tool.*link/); + expect(await fileExists(join(outside, 'tool'))).toBe(false); + }); + + it('refuses declaring the compressed original instead of a member', async () => { + const { payload, archive } = await staged('tar.gz', { + destination: '.', executables: ['source.tar.gz'], + }); + await expect(expandAssetArchive(payload, archive)).rejects.toThrow(/assetArchives\[\]\.executables.*source.tar.gz.*not a member/); + expect(await fileExists(join(payload, 'source.tar.gz'))).toBe(true); + }); +}); + function response(bytes, status = 200) { return { ok: status >= 200 && status < 300, diff --git a/tests/unit/build-pipeline.test.mjs b/tests/unit/build-pipeline.test.mjs index 002a634..68bc9d7 100644 --- a/tests/unit/build-pipeline.test.mjs +++ b/tests/unit/build-pipeline.test.mjs @@ -1,6 +1,6 @@ import { execFileSync } from 'node:child_process'; import { createHash, randomBytes } from 'node:crypto'; -import { existsSync, mkdirSync, symlinkSync, writeFileSync } from 'node:fs'; +import { chmodSync, existsSync, mkdirSync, statSync, symlinkSync, writeFileSync } from 'node:fs'; import { lstat, mkdir, mkdtemp, readFile, realpath, rename, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { dirname, join, relative, sep } from 'node:path'; @@ -23,6 +23,8 @@ import { configureWorkspace, resetWorkspace } from '../../src/build/workspace.mj import { generateSigningKey, signDocument } from '../../src/sign/index.mjs'; import { boxTargetAdapters, boxTargetId, decodeDocumentPayload, documentKinds } from '../../src/contract/index.mjs'; import { runtimeAdapter } from '../../src/contract/runtimes.mjs'; +import { writeAssetArchive } from '../helpers/asset-archive.mjs'; +import { run as runProcess } from '../../src/build/process.mjs'; // The pipeline is the same on every platform, but the native-host gate (rightly) refuses to build // a box for any other one — so the test scroll targets whatever host the suite is running on. @@ -159,14 +161,22 @@ function fakeToolchain(payloadDir, { // What the box starts. A native box starts a file the scroll brought in, and its packed prefix // carries no interpreter at all — so both are parameters rather than the Python answer baked in. interpreter = true, + interpreterContents = null, entrySegments = ENTRY_SEGMENTS, selfTestCommand = join(payloadDir, ...ENTRY_SEGMENTS), + // Files a package installs in the prefix, relative to it, written with mode 0755 the way conda + // installs a helper binary — so a test can prove the box does not inherit that bit. + prefixFiles = {}, } = {}) { const run = function run(command, args = [], options = {}) { if (command === 'pixi' && args[0] === 'install') { const manifest = args[args.indexOf('--manifest-path') + 1]; const prefix = join(dirname(manifest), '.pixi', 'envs', 'default'); - if (interpreter) writeDeep(join(prefix, ...entrySegments.slice(1)), '#!/bin/sh\nexit 0\n'); + if (interpreter) { + const entry = join(prefix, ...entrySegments.slice(1)); + writeDeep(entry, interpreterContents ?? '#!/bin/sh\nexit 0\n'); + if (interpreterContents && process.platform !== 'win32') chmodSync(entry, 0o755); + } if (consoleScript) { // What conda actually generates: the *build machine's* interpreter, reached through the // shell trampoline it falls back to when an absolute shebang would be too long. @@ -190,6 +200,11 @@ function fakeToolchain(payloadDir, { writeDeep(join(prefix, ...sitePackages, ...modulePath.slice(0, -1), `${modulePath.at(-1)}.py`), 'print("module ready")\n'); } + for (const [path, contents] of Object.entries(prefixFiles)) { + const file = join(prefix, ...path.split('/')); + writeDeep(file, contents); + if (process.platform !== 'win32') chmodSync(file, 0o755); + } plantPrefixSymlinks(prefix); return ''; } @@ -272,6 +287,204 @@ describe('the build pipeline', () => { return { root, scrollDir, keys, payloadDir: join(root, '.scrollcase', 'build', scrollId, 'payload') }; } + async function archiveProject({ format = 'tar.gz', archive = {}, scroll = {} } = {}) { + const upstream = await mkdtemp(join(tmpdir(), 'scrollcase-upstream-')); + created.push(upstream); + const { bytes } = await writeAssetArchive(upstream, { format }); + const project = await makeProject({ + ...SCROLL, + assets: [{ + url: `https://assets.example.org/source.${format}`, + relativePath: `cache/source.${format}`, + sizeBytes: bytes.length, + sha256: createHash('sha256').update(bytes).digest('hex'), + }], + assetArchives: [{ + relativePath: `cache/source.${format}`, format, destination: 'tools', + stripComponents: 1, executables: ['bin/tool'], ...archive, + }], + ...scroll, + }); + return { + ...project, + options: { + ...project.keys, + ...fakeToolchain(project.payloadDir), + fetchImpl: async () => ({ ok: true, status: 200, body: Readable.from([bytes]) }), + log: () => {}, + }, + }; + } + + for (const format of ['zip', 'tar', 'tar.gz']) { + it(`builds and rebuilds a declared ${format} member byte-identically`, async () => { + const { options } = await archiveProject({ format }); + const first = await buildBox(SCROLL_REF, options); + const firstBytes = await readFile(first.archivePath); + const modes = await zipModes(first.archivePath); + expect(modes.get('tools/bin/tool')).toBe(process.platform === 'win32' ? 0o100644 : 0o100755); + expect(modes.has(`cache/source.${format}`)).toBe(false); + const second = await buildBox(SCROLL_REF, options); + expect(await readFile(second.archivePath)).toEqual(firstBytes); + expect(second.archiveSha256).toBe(first.archiveSha256); + await expect(verifyBox(second.releasePath, { publicPath: options.publicPath, log: () => {} })) + .resolves.toMatchObject({ status: 'passed' }); + }); + } + + for (const pruned of ['tools/bin/tool', 'tools/bin', 'tools']) { + it(`refuses prunePaths removing the executable through ${pruned}`, async () => { + const { options } = await archiveProject({ scroll: { prunePaths: [pruned] } }); + await expect(buildBox(SCROLL_REF, options)) + .rejects.toThrow(/assetArchives\[\]\.executables.*bin\/tool.*prunePaths/); + }); + } + + it('lets a native box start a declared archive member', async () => { + const { options, payloadDir } = await archiveProject({ + scroll: { + runtime: { id: 'native' }, + execution: { kind: 'native-binary', binary: 'tools/bin/tool', defaultArgs: [] }, + selfTest: { commands: [{ args: [] }] }, + }, + }); + await expect(buildBox(SCROLL_REF, { + ...options, + ...fakeToolchain(payloadDir, { interpreter: false, selfTestCommand: join(payloadDir, 'tools', 'bin', 'tool') }), + })).resolves.toMatchObject({ archiveSha256: expect.any(String) }); + }); + + for (const declared of [false, true]) { + it.runIf(process.platform !== 'win32')(`self-tests a real archive helper with declared=${declared}`, async () => { + const layout = runtimeAdapter('node').layout(HOST_ADAPTER); + const { options, payloadDir } = await archiveProject({ + archive: { executables: declared ? ['bin/tool'] : [] }, + scroll: { + runtime: { id: 'node', version: process.versions.node }, + selfTest: { + imports: ['node:child_process'], + code: 'require("node:child_process").execFileSync("./tools/bin/tool", { stdio: "inherit" });', + }, + }, + }); + const messages = []; + const build = () => buildBox(SCROLL_REF, { + ...options, + ...fakeToolchain(payloadDir, { + entrySegments: layout.entryPoint.split('/'), + selfTestCommand: join(payloadDir, ...layout.entryPoint.split('/')), + // A small fixture interpreter forwards the real invocation to Node without copying and + // compressing the host's entire executable in every unit test. The scroll's actual probe + // code, argument vector and helper launch run, rather than an assertion replacing them. + interpreterContents: `#!/bin/sh\nexec '${process.execPath.replaceAll("'", "'\\''")}' "$@"\n`, + onSelfTest: ({ command, args, options }) => { + expect(runProcess(command, args, { ...options, capture: true })) + .toBe('archive-member-ok'); + }, + }), + log: (message) => messages.push(message), + }); + if (declared) { + await expect(build()).resolves.toMatchObject({ archiveSha256: expect.any(String) }); + } else { + await expect(build()).rejects.toThrow(/Self-test failed.*tools\/bin\/tool.*EACCES.*If a helper lost.*assetArchives\[\]\.executables.*executablePaths/s); + expect(messages).not.toContain('Creating deterministic archive'); + expect(statSync(join(payloadDir, 'tools', 'bin', 'tool')).mode & 0o777).toBe(0o644); + } + }); + } + + const ENVIRONMENT_HELPER = 'lib/pkg/bin/helper'; + const environmentHelperFiles = { + [ENVIRONMENT_HELPER]: '#!/bin/sh\nprintf environment-helper-ok\n', + 'lib/pkg/README': 'not a program\n', + }; + + for (const declared of [false, true]) { + it.runIf(process.platform !== 'win32')(`self-tests a real environment helper with declared=${declared}`, async () => { + const layout = runtimeAdapter('node').layout(HOST_ADAPTER); + const project = await makeProject({ + ...SCROLL, + runtime: { id: 'node', version: process.versions.node }, + selfTest: { + imports: ['node:child_process'], + code: `require("node:child_process").execFileSync("./venv/${ENVIRONMENT_HELPER}", { stdio: "inherit" });`, + }, + executablePaths: declared ? [`venv/${ENVIRONMENT_HELPER}`] : [], + }); + const build = () => buildBox(SCROLL_REF, { + ...project.keys, + ...fakeToolchain(project.payloadDir, { + entrySegments: layout.entryPoint.split('/'), + selfTestCommand: join(project.payloadDir, ...layout.entryPoint.split('/')), + interpreterContents: `#!/bin/sh\nexec '${process.execPath.replaceAll("'", "'\\''")}' "$@"\n`, + prefixFiles: environmentHelperFiles, + onSelfTest: ({ command, args, options }) => { + expect(runProcess(command, args, { ...options, capture: true })) + .toBe('environment-helper-ok'); + }, + }), + log: () => {}, + }); + if (declared) { + const built = await build(); + const modes = await zipModes(built.archivePath); + expect(modes.get(`venv/${ENVIRONMENT_HELPER}`)).toBe(0o100755); + expect(modes.get('venv/lib/pkg/README')).toBe(0o100644); + } else { + // conda installed it 0755; the box would not, and the build now says so before signing. + await expect(build()).rejects.toThrow(/Self-test failed.*venv\/lib\/pkg\/bin\/helper.*EACCES.*executablePaths/s); + } + }); + } + + const environmentFailures = { + 'a path the solve did not produce': [{ executablePaths: ['venv/lib/pkg/bin/missing'] }, + /executablePaths: "venv\/lib\/pkg\/bin\/missing" is missing \(check the path against the solved environment and prunePaths\)/], + 'a pruned helper': [{ executablePaths: [`venv/${ENVIRONMENT_HELPER}`], prunePaths: ['venv/lib/pkg'] }, + /executablePaths: "venv\/lib\/pkg\/bin\/helper" is missing/], + 'a directory': [{ executablePaths: ['venv/lib/pkg/bin'] }, + /executablePaths: "venv\/lib\/pkg\/bin" must be a regular file/], + // Packing materialises directory links, so `lib/icu/current/` is a real directory by now; a file + // link inside the prefix survives as a link on POSIX targets only, and Windows gets the file. + ...(process.platform === 'win32' ? {} : { + 'a prefix link itself': [{ executablePaths: ['venv/lib/icu/pkgdata.inc'] }, + /executablePaths: "venv\/lib\/icu\/pkgdata\.inc" is a link or traverses a link at venv\/lib\/icu\/pkgdata\.inc/], + }), + }; + for (const [name, [declaration, message]] of Object.entries(environmentFailures)) { + it(`refuses an environment executable that is ${name}`, async () => { + const project = await makeProject({ ...SCROLL, ...declaration }); + const messages = []; + await expect(buildBox(SCROLL_REF, { + ...project.keys, + ...fakeToolchain(project.payloadDir, { prefixFiles: environmentHelperFiles }), + log: (line) => messages.push(line), + })).rejects.toThrow(message); + expect(messages).not.toContain('Running self-test'); + }); + } + + it.runIf(process.platform !== 'win32')('self-tests runtime and declared modes, then normalises files the probe created', async () => { + const { options, payloadDir } = await archiveProject(); + const built = await buildBox(SCROLL_REF, { + ...options, + ...fakeToolchain(payloadDir, { + consoleScript: 'example-cli', + onSelfTest: () => { + expect(statSync(join(payloadDir, ...ENTRY_SEGMENTS)).mode & 0o777).toBe(0o755); + expect(statSync(join(payloadDir, ...HOST_LAYOUT.scriptsDirectory.split('/'), 'example-cli')).mode & 0o777).toBe(0o755); + expect(statSync(join(payloadDir, 'tools', 'bin', 'tool')).mode & 0o777).toBe(0o755); + expect(statSync(join(payloadDir, 'box.json')).mode & 0o777).toBe(0o644); + writeFileSync(join(payloadDir, 'probe-output'), 'probe generated data', { mode: 0o777 }); + }, + }), + }); + expect(statSync(join(payloadDir, 'probe-output')).mode & 0o777).toBe(0o644); + expect(statSync(join(payloadDir, PAYLOAD_DIGEST_FILE)).mode & 0o777).toBe(0o644); + expect((await zipModes(built.archivePath)).get('probe-output')).toBe(0o100644); + }); + it('rejects the removed flat scroll layout', async () => { await makeProject(SCROLL, { commit: false, dirName: SCROLL.scrollId }); await expect(readScroll(SCROLL.scrollId)).rejects.toThrow(/contains no target scrolls/); @@ -884,7 +1097,13 @@ describe('the build pipeline', () => { it('refuses to code sign a box that is not for macOS, before tool discovery', async () => { // Refused ahead of the native-host gate, so the same assertion holds on every host. - const scroll = { ...SCROLL, target: { platform: 'linux', arch: 'x86_64', accelerator: 'cpu' } }; + // Let the target derive its entry point: keeping the host's Windows path would reject the + // contradictory runtime declaration before the code-signing option is ever inspected. + const scroll = { + ...SCROLL, + runtime: { id: RUNTIME_ID, version: PYTHON_VERSION }, + target: { platform: 'linux', arch: 'x86_64', accelerator: 'cpu' }, + }; const { keys } = await makeProject(scroll); const calls = []; await expect(buildBox(`${SCROLL.boxId}/linux-x86_64-cpu`, { diff --git a/tests/unit/scroll-editing.test.mjs b/tests/unit/scroll-editing.test.mjs index 0471f74..27261f7 100644 --- a/tests/unit/scroll-editing.test.mjs +++ b/tests/unit/scroll-editing.test.mjs @@ -112,6 +112,27 @@ describe('editing an existing scroll', () => { expect(other.scroll.assets).toHaveLength(1); }); + it('preserves archive and environment executable declarations through add, edit and refresh', async () => { + const archive = { relativePath: 'source.tar.gz', format: 'tar.gz', destination: 'tools', executables: ['bin/tool'] }; + const executablePaths = ['venv/lib/pkg/bin/helper']; + const { root } = await splitBox({ base: { ...SHARED, assetArchives: [archive], executablePaths } }); + await writeFile(join(root, 'notice.txt'), 'reviewed'); + await addFile({ boxId: SHARED.boxId, target: ALL_TARGETS, sourcePath: 'notice.txt', pin: true }); + await addAsset({ boxId: SHARED.boxId, target: ALL_TARGETS, url: 'https://example.org/data', fetchImpl: servingBytes('data') }); + await setScrollField({ boxId: SHARED.boxId, target: ALL_TARGETS, field: 'version', value: '1.1.0' }); + await writeFile(join(root, 'notice.txt'), 'changed after review'); + expect((await refreshScroll({ boxId: SHARED.boxId })).updated).toEqual(['notice.txt']); + await removeScrollEntry({ boxId: SHARED.boxId, target: ALL_TARGETS, field: 'localFiles', relativePath: 'notice.txt' }); + for (const reference of [REFERENCE, `example-model/${OTHER_TARGET_ID}`]) { + const { scroll } = await readScroll(reference); + expect(scroll.assetArchives).toEqual([archive]); + expect(scroll.executablePaths).toEqual(executablePaths); + } + const editable = (await editableScrollFields()).map(({ name }) => name); + expect(editable).not.toContain('assetArchives'); + expect(editable).not.toContain('executablePaths'); + }); + it('records an asset for one target only when asked', async () => { await splitBox(); await addAsset({ diff --git a/tests/unit/scroll-extends.test.mjs b/tests/unit/scroll-extends.test.mjs index b675688..2d82d04 100644 --- a/tests/unit/scroll-extends.test.mjs +++ b/tests/unit/scroll-extends.test.mjs @@ -73,6 +73,74 @@ describe('joining a base scroll with a target fragment', () => { expect(scroll.scrollId).toBe(`example-model-${TARGET_ID}`); }); + it('keeps executable members on both base and target archive entries', async () => { + const shared = { relativePath: 'shared.tar', format: 'tar', destination: 'shared', executables: ['bin/tool'] }; + const target = { relativePath: 'target.zip', format: 'zip', destination: 'target', executables: ['bin/tool'] }; + await family({ ...BASE, assetArchives: [shared] }, { + extends: '../scroll.json', target: TARGET, assetArchives: [target], + }); + expect((await readScroll(REFERENCE)).scroll.assetArchives).toEqual([shared, target]); + }); + + it('refuses duplicate resulting executable paths across base and target archives', async () => { + const archive = { relativePath: 'one.tar', format: 'tar', destination: 'tools', executables: ['bin/tool'] }; + await family({ ...BASE, assetArchives: [archive] }, { + extends: '../scroll.json', target: TARGET, + assetArchives: [{ ...archive, relativePath: 'two.tar', destination: './tools' }], + }); + await expect(readScroll(REFERENCE)).rejects.toThrow(/assetArchives\[\]\.executables.*tools\/bin\/tool.*more than one/); + }); + + for (const path of ['../escape', 'bin/./tool', 'bin\\tool', '']) { + it(`refuses the unsafe archive declaration ${JSON.stringify(path)} while reading`, async () => { + await family({ ...BASE, assetArchives: [{ + relativePath: 'source.zip', format: 'zip', destination: 'tools', executables: [path], + }] }); + const error = await readScroll(REFERENCE).catch((error) => error); + expect(error.message).toContain('assetArchives[].executables'); + expect(error.message).toContain(JSON.stringify(path)); + }); + } + + it('joins environment executable paths base-first and drops repeats', async () => { + await family({ ...BASE, executablePaths: ['venv/lib/pkg/bin/shared', 'venv/lib/pkg/bin/both'] }, { + extends: '../scroll.json', target: TARGET, + executablePaths: ['venv/lib/pkg/bin/both', 'venv/lib/pkg/bin/target'], + }); + expect((await readScroll(REFERENCE)).scroll.executablePaths) + .toEqual(['venv/lib/pkg/bin/shared', 'venv/lib/pkg/bin/both', 'venv/lib/pkg/bin/target']); + }); + + const owned = { + 'outside the environment': [{ executablePaths: ['tools/bin/helper'] }, /executablePaths: "tools\/bin\/helper" is outside venv\//], + 'an asset brings in': [ + { assets: [{ ...asset('helper', 'a'), relativePath: 'venv/extra/helper' }], executablePaths: ['venv/extra/helper'] }, + /executablePaths: "venv\/extra\/helper" is brought in by an asset or local file/, + ], + 'an archive expands': [{ + assetArchives: [{ relativePath: 'cache/tools.zip', format: 'zip', destination: './venv/extra' }], + executablePaths: ['venv/extra/bin/helper'], + }, /executablePaths: "venv\/extra\/bin\/helper" lies under the destination of cache\/tools\.zip/], + }; + for (const [name, [declaration, message]] of Object.entries(owned)) { + it(`refuses an executable path ${name}`, async () => { + await family({ ...BASE, ...declaration }); + await expect(readScroll(REFERENCE)).rejects.toThrow(message); + }); + } + + for (const path of ['venv/lib/./helper', 'venv/lib/../helper', 'venv\\lib\\helper', 'venv/lib/helper/', 'C:venv/helper']) { + it(`refuses the unsafe environment declaration ${JSON.stringify(path)} while reading`, async () => { + await family({ ...BASE, executablePaths: [path] }); + await expect(readScroll(REFERENCE)).rejects.toThrow(/executablePaths/); + }); + } + + it('refuses a repeated environment executable path in a whole scroll', async () => { + await family(null, { ...BASE, target: TARGET, executablePaths: ['venv/lib/helper', 'venv/lib/helper'] }); + await expect(readScroll(REFERENCE)).rejects.toThrow(/executablePaths.*duplicate/); + }); + it('lets a fragment replace a scalar and a cohesive object', async () => { await family( { ...BASE, version: '1.0.0', execution: { kind: 'python-module', module: 'shared.main', defaultArgs: [] } },