From 67107ea5032eb836db2eaf02844fa81298bbf58b Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 16:33:35 -0400 Subject: [PATCH 01/56] Add research docs for SyntaxKit-driven codegen CLI (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 documents how Tuist evaluates user manifests via xcrun swift + token-delimited JSON over stdout, with citations into tuist/tuist and swiftlang/swift-package-manager. Phase 2 sketches the SyntaxKit equivalent: pure-DSL input files wrapped into a Group { … } closure before spawning swift, with a 7-step POC ladder that retires cold-start cost first. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 154 +++++++++++++ Docs/research/tuist-manifest-pipeline.md | 265 +++++++++++++++++++++++ 2 files changed, 419 insertions(+) create mode 100644 Docs/research/codegen-cli-design.md create mode 100644 Docs/research/tuist-manifest-pipeline.md diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md new file mode 100644 index 0000000..cae86e1 --- /dev/null +++ b/Docs/research/codegen-cli-design.md @@ -0,0 +1,154 @@ +# Design sketch: a CLI for SyntaxKit-driven codegen + +> Phase 2 deliverable for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). Builds on [`tuist-manifest-pipeline.md`](./tuist-manifest-pipeline.md). Nothing here is implemented — this is the design we'd validate with the POC in §6. + +## 1. What we're borrowing from Tuist — and what we're not + +From Phase 1, Tuist's manifest pipeline reduces to four moving parts: + +1. A **public DSL framework** that ships next to the CLI binary (`ProjectDescription.framework`). +2. A **script runner** that invokes `xcrun swift ` with `-I/-L/-F` pointing at that framework, captures stdout, and slices out a token-delimited payload. +3. A **helpers compiler** that pre-builds `Tuist/ProjectDescriptionHelpers/*.swift` into a sibling dylib so manifests can `import ProjectDescriptionHelpers`. +4. A **two-tier cache** (helpers module + decoded manifest) keyed on source hashes + toolchain/tool versions. + +We borrow (1), (2), (3), and (4). We **don't** borrow Tuist's "manifest" framing — no `Project.swift`-style wrapper, no `Output(...)` value, no token-delimited stdout payload. Tuist needs the wrapper because its host has to re-interpret the description into an `xcodeproj`. SyntaxKit doesn't: the input file is *pure DSL* — a series of `CodeBlock` expressions — and the CLI generates the boilerplate that turns it into a runnable Swift program. + +## 2. CLI shape + +The CLI is `stdin → stdout`-shaped, with a SyntaxKit-aware `swift` invocation as the engine: + +``` +syntaxkit run Input.swift # rendered Swift source to stdout +syntaxkit run Input.swift -o Output.swift # write to a file (atomic) +syntaxkit run InputDir/ -o OutputDir/ # walk InputDir/*.swift, mirror paths into OutputDir/ +``` + +**Input file:** pure DSL. A series of `CodeBlock` expressions, optionally preceded by `import` declarations. No `print`, no `@main`, no boilerplate. Example: + +```swift +// Person.swift +import SyntaxKit // optional — only needed for IDE / autocomplete + +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} +``` + +That's the entire file. Top-level expressions form an implicit `@CodeBlockBuilder` body that the CLI wraps for execution — see §3. + +**Output:** the rendered Swift source produced by `generateCode()`. The CLI does not reshape it. + +**Stderr:** forwarded to the user's terminal. The captured output is stdout-only. (The wrapper writes to stdout via a single `print`; if helpers or the user's DSL want to log debug info, they should use `FileHandle.standardError.write(...)`.) + +**Folder mode:** when the input is a directory, the CLI walks `**/*.swift` and produces a parallel tree of outputs (one input file → one output file, mirrored relative path). Files starting with `_` are skipped (convention for shared helpers — see §4). + +**Exit codes:** child `swift` non-zero → CLI non-zero with stderr preserved. No retries. + +## 3. Process model: wrap, then spawn + +Because the input is pure DSL rather than a runnable Swift program, the CLI does a tiny **wrap** step before spawning `swift`. + +**Wrap.** Read the input file. Use SwiftSyntax (already a dep — `Docs/SwiftSyntax-LLM.md`) to split the top of the file into (a) any leading `import` declarations and (b) the remaining body. Generate a temporary `Input.wrapped.swift`: + +```swift +import SyntaxKit + + +let __syntaxkit_root = Group { + +} +print(__syntaxkit_root.generateCode()) +``` + +`Group` (`Sources/SyntaxKit/Utilities/Group.swift`) already uses `@CodeBlockBuilder`, so its closure body accepts a series of `CodeBlock` expressions exactly the way the user wrote them. `import SyntaxKit` is always injected; duplicates from the input are harmless. + +**Spawn.** Adopt Tuist's model `(b)` from Phase 1 §2 — `swift` in script mode against the wrapped file, pipe through: + +``` +/usr/bin/env swift \ + -suppress-warnings \ + -I \ + -L \ + -F \ + -lSyntaxKit -framework SyntaxKit \ + -I … -L … -F … -l # optional, when helpers exist + /Input.wrapped.swift +``` + +No `--syntaxkit-dump` flag, no start/end tokens. The child's stdout is the rendered Swift source verbatim. The CLI's job is wrap → `Process` → capture stdout → atomic write to destination → clean up temp wrapper. + +Use `/usr/bin/env swift` rather than `/usr/bin/xcrun swift` so the same code path runs on Linux. `xcrun` is mac-only and is implicit when `env swift` resolves to Xcode's swift on macOS. + +**Why wrap instead of requiring `print()` in the input.** Two reasons. First, the user's authoring surface is *just* DSL — declarative, no I/O verbs. Second, error reporting: when the child `swift` reports a diagnostic at `Input.wrapped.swift:42`, the CLI can map that line back to the original `Input.swift` (the wrapper is line-faithful aside from a known prefix offset) and rewrite the path in stderr before forwarding. + +## 4. Helpers + +Same mechanism as Tuist (Phase 1 §4), folder name TBD — `Helpers/` adjacent to the input file or input directory is the obvious choice. The CLI walks up from the input path looking for a `Helpers/` directory, globs `**/*.swift` (excluding files prefixed with `_` to allow private helpers within helpers), and pre-compiles them into `lib.dylib` via: + +``` +swiftc -module-name SyntaxKitHelpers \ + -emit-module -emit-module-path /SyntaxKitHelpers.swiftmodule \ + -parse-as-library -emit-library \ + -suppress-warnings \ + -I … -L … -F … -lSyntaxKit -framework SyntaxKit \ + Helpers/**/*.swift +``` + +The output dylib is then added to the input script's invocation via `-I/-L/-F/-l`. Scripts can `import SyntaxKitHelpers` and use shared codegen utilities. + +Compile into a `tmp..` staging directory and atomic-rename into the cache path, mirroring `ProjectDescriptionHelpersBuilder.swift:204-244` — concurrent CLI invocations need to be safe. + +## 5. Caching + +Two layers, both mirroring Tuist (Phase 1 §5). + +**Helpers cache.** Keyed by: + +| Field | Source | +| --- | --- | +| per-file SHA-256s | `Helpers/**/*.swift`, sorted | +| `syntaxkitVersion` | bundled SyntaxKit dylib version | +| `swiftlangVersion` | `swift --version` | +| `osVersion` | `uname -r` (macOS or Linux) | +| `cacheSchemaVersion` | bumped on layout changes | + +Hash → directory name → reuse if present. + +**Output cache.** Skip the swift spawn entirely when nothing has changed. Keyed by: + +| Field | Source | +| --- | --- | +| `inputHash` | SHA-256 of the input `.swift` file | +| `helpersHash` | the helpers cache key above, or empty | +| `syntaxkitVersion` | bundled SyntaxKit dylib version | +| `swiftlangVersion` | `swift --version` | +| `envHash` | md5 of `SYNTAXKIT_*` env vars | +| `cacheSchemaVersion` | bumped on layout changes | + +On hit, copy the cached rendered output directly to the destination — no `swift` spawn. On miss, run and re-cache. + +Cache location: `~/.cache/syntaxkit/` on Linux, `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS (XDG-aware via `XDG_CACHE_HOME`). + +## 6. Smallest possible proof-of-concept steps + +Each step is independently shippable and de-risks the next. + +1. **Hand-driven wrap + spawn.** Hand-write a pure-DSL `Input.swift` and a hand-rolled `Input.wrapped.swift` that imports SyntaxKit, splices the body into `Group { … }`, and prints `generateCode()`. Invoke it manually with `swift Input.wrapped.swift -I … -L … -F … -lSyntaxKit -framework SyntaxKit` against a local `swift build` of SyntaxKit. **Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit at all, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost.** Cold-start is the single biggest risk to retire — if it's 20+ seconds, the design needs rethinking. +2. **CLI subcommand for single-file mode.** Add `syntaxkit run `: parse out top-level imports with SwiftSyntax, write `Input.wrapped.swift` to a temp dir, spawn `swift` via `Foundation.Process`, capture stdout, write to `-o ` (or stdout if no `-o`). Stderr is forwarded; rewrite `Input.wrapped.swift:LINE` references back to `Input.swift:LINE` before forwarding. +3. **Folder mode.** Walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`. Add `_`-prefix skip rule. Parallelize cautiously (`swift` spawns aren't free — gate on a small concurrency limit, maybe `ProcessInfo.activeProcessorCount`). +4. **Ship a bundled-binary release.** Build SyntaxKit + SwiftSyntax dylibs, drop them in `lib/` next to the CLI binary, write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). Now the CLI is self-contained — no `swift build` required at the call site. +5. **Helpers directory.** Discovery + compile + flag-splicing. +6. **Output cache.** Add the cache, keyed as in §5. Add `--no-cache` for debugging. +7. **Linux smoke test.** Confirm `/usr/bin/env swift` works on Linux with the bundled dylib layout (no framework search path, but `-I + -L + -lSyntaxKit` should be sufficient, matching Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch). + +## 7. What we still need to verify + +- **Cold-start cost.** Single biggest unknown. Step 1 of §6 answers this. +- **Splice fidelity.** When the input body lives inside a `Group { … }` closure, is everything users naturally write in the DSL still legal? Result-builder closures don't allow `import`, top-level type decls, or top-level `let`/`var` outside the builder DSL. The wrapper hoists `import`s; we need to confirm there's no other top-level construct users would reasonably write that the wrap step would break. Verify in step 1 with a few realistic inputs (large struct, nested types, conditionals via `if`-in-builder). +- **`Process` stdout/stderr separation.** Tuist captures stdout-only and merges stderr via `CommandError`. Foundation's `Process` has the same split — confirm it doesn't interleave under load, and confirm the CLI doesn't accidentally swallow stderr. +- **`swift` script-mode quirks.** `swift ` runs in interpret/`-frontend -interpret` mode. Some features (`@main`, certain attributes) behave differently than in compile mode. Top-level `print` statements are fine. Verify in step 1. +- **SwiftSyntax linkage stability across toolchains.** SwiftSyntax pins to specific Swift toolchain versions. The bundled dylib is built against a particular toolchain; if the user's `swift` is from a newer or older release, ABI breakage is possible. Mitigation: cache key includes `swiftlangVersion`, plus a clear error when the gap is too wide. +- **What if a single input script needs to produce multiple files?** Out of scope for v1. Split into multiple inputs and use folder mode. If demand materializes, we can layer in a `--multi` envelope mode (a script writes a small JSON manifest to stdout, CLI fans out to multiple files) without breaking single-file semantics. +- **Sandboxing.** Out of scope for v1. Input scripts are user-owned code in their own repo — running them has the same threat model as running `swift Input.swift` by hand. Revisit if/when this CLI runs untrusted scripts (CI for OSS contributions, etc.). +- **Timeout.** Add one. Tuist's omission (Phase 1 §7) is a bug, not a feature. 60s default `Process` wait with `SIGTERM` → 5s grace → `SIGKILL`. Override via `--timeout `. diff --git a/Docs/research/tuist-manifest-pipeline.md b/Docs/research/tuist-manifest-pipeline.md new file mode 100644 index 0000000..313464b --- /dev/null +++ b/Docs/research/tuist-manifest-pipeline.md @@ -0,0 +1,265 @@ +# How Tuist Evaluates Manifests + +> Phase 1 deliverable for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). +> Source: read of `tuist/tuist@main` and `swiftlang/swift-package-manager@main` via GitHub. Every behavioral claim cites a file/line in those repos. + +## TL;DR + +- **Manifest is run, not pre-compiled to an artifact.** Tuist invokes `/usr/bin/xcrun swift ` in Swift script/interpreter mode with `-I/-L/-F` pointing at `ProjectDescription.framework` (`cli/Sources/TuistLoader/Loaders/ManifestLoader.swift:374-528`). There is no separate `.o`/executable for the manifest itself. +- **JSON over stdout, token-delimited.** The manifest's `Project(...)` initializer calls `dumpIfNeeded(self)` (`cli/Sources/ProjectDescription/Dump.swift:3-14`) which `JSONEncoder`s `self` and prints it bracketed by `TUIST_MANIFEST_START` / `TUIST_MANIFEST_END`; the host scans stdout for those tokens and `JSONDecoder`s the slice between them (`ManifestLoader.swift:119-120, 347-365`). +- **Helpers are real dylibs.** Files under `/Tuist/ProjectDescriptionHelpers/` are compiled by a separate `swiftc -emit-module -emit-library -parse-as-library` invocation into `lib.dylib`, cached on disk, and added to the manifest's `swift` invocation via `-I/-L/-F/-l` (`cli/Sources/TuistLoader/ProjectDescriptionHelpers/ProjectDescriptionHelpersBuilder.swift:174-298`). +- **Two-layer cache.** The helpers module is keyed by an md5 of (file SHA-256s + Tuist version + Swift toolchain version + macOS version + macOS SDK version + tuist env vars + DEBUG flag) (`ProjectDescriptionHelpersHasher.swift:34-55`). The decoded manifest JSON itself is cached in `~/.tuist/Cache/Manifests`, keyed by (manifest SHA-256 + helpers hash + plugins hash + env hash + sandbox flag + Tuist version + cache schema version) (`CachedManifestLoader.swift:181-275`). +- **Sandboxed.** On macOS the whole `xcrun swift …` command is wrapped in `sandbox-exec -p ` by default; the profile denies everything and re-allows only read access to the project path, Xcode, the `ProjectDescription` search paths, and `/private/tmp` + `/private/var` for writes (`ManifestLoader.swift:492-574`). + +## 1. Compilation pipeline + +The manifest is **not** compiled by Tuist to a separate artifact — it is run with `swift` in interpreter mode. Argument construction lives in `ManifestLoader.buildArguments(_:at:disableSandbox:)`: + +```swift +// cli/Sources/TuistLoader/Loaders/ManifestLoader.swift:392-401 +var arguments = [ + "/usr/bin/xcrun", + "swift", + "-suppress-warnings", + "-I", searchPaths.includeSearchPath.pathString, + "-L", searchPaths.librarySearchPath.pathString, + "-F", searchPaths.frameworkSearchPath.pathString, + "-l\(frameworkName)", + "-framework", frameworkName, +] +``` + +Then helpers are stitched in (`ManifestLoader.swift:423-428`): + +```swift +let projectDescriptionHelperArguments = projectDescriptionHelperModules.flatMap { [ + "-I", $0.path.parentDirectory.pathString, + "-L", $0.path.parentDirectory.pathString, + "-F", $0.path.parentDirectory.pathString, + "-l\($0.name)", +] } +``` + +The manifest path itself is appended last (`ManifestLoader.swift:478`), and `--tuist-dump` is added by the caller (`ManifestLoader.swift:344`). There is **no** `-target`, `-sdk`, `-module-name`, `-emit-executable`, or `-emit-library` on the manifest run — those are SwiftPM-style flags that Tuist deliberately avoids for the manifest. The implicit target/SDK come from `xcrun swift` itself (i.e. from the active Xcode selected via `xcode-select`). + +**For `Package.swift` (the `.packageSettings` case)**, Tuist additionally points `-I/-L/-F` at `XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI`, links `-lPackageDescription`, and passes `-package-description-version -D TUIST` and a JSON-encoded `-context ` argument that SwiftPM's `PackageDescription` runtime expects (`ManifestLoader.swift:430-490`). + +**Helpers compilation** uses `swiftc` directly (`ProjectDescriptionHelpersBuilder.swift:264-298`): + +```swift +var command: [String] = [ + "/usr/bin/xcrun", "swiftc", + "-module-name", moduleName, + "-emit-module", + "-emit-module-path", outputDirectory.appending(component: "\(moduleName).swiftmodule").pathString, + "-parse-as-library", + "-emit-library", + "-suppress-warnings", + "-I", projectDescriptionSearchPaths.includeSearchPath.pathString, + "-L", projectDescriptionSearchPaths.librarySearchPath.pathString, + "-F", projectDescriptionSearchPaths.frameworkSearchPath.pathString, + "-working-directory", outputDirectory.pathString, +] +// + helper-module flags + `-framework ProjectDescription` (or `-lProjectDescription` when dylib) + all *.swift sources +``` + +The helper artifact is a dylib named `lib.dylib` (`ProjectDescriptionHelpersBuilder.swift:188-190`). + +## 2. Execution model + +It's model **(b) — `Process`-spawn + parse stdout**. There is no `dlopen`, no exported C symbol, no plugin entry point. `loadDataForManifest` simply runs the constructed `arguments` via `CommandRunner.capture(...)`: + +```swift +// cli/Sources/TuistLoader/Loaders/ManifestLoader.swift:347-365 +let string = try await commandRunner.capture( + arguments: arguments, + environment: Environment.current.manifestLoadingVariables +) + +guard let startTokenRange = string.range(of: ManifestLoader.startManifestToken, options: .literal), + let endTokenRange = string.range(of: ManifestLoader.endManifestToken, options: [.literal, .backwards]) +else { + return string.data(using: .utf8)! +} +// ... slice out the JSON ... +let manifest = string[startTokenRange.upperBound ..< endTokenRange.lowerBound] +return manifest.data(using: .utf8)! +``` + +The host scans the entire captured stdout for `TUIST_MANIFEST_START` and `TUIST_MANIFEST_END` (`ManifestLoader.swift:119-120`). Anything *before* the start token and *after* the end token is treated as user-visible log output and re-emitted via `Logger.current.notice(...)` (`ManifestLoader.swift:358-362`). This means a manifest can `print(…)` debug output *and* still be parsed correctly — a useful side-effect of the token-delimited design. + +The slice between the tokens is utf8-decoded and passed straight to `JSONDecoder().decode(T.self, from: data)` (`ManifestLoader.swift:256-257`). + +## 3. Bridging format + +JSON, generated by `JSONEncoder` on the manifest side and `JSONDecoder` on the host side. + +**Encode site** (manifest process): `cli/Sources/ProjectDescription/Dump.swift:3-14`: + +```swift +func dumpIfNeeded(_ entity: some Encodable) { + guard !ProcessInfo.processInfo.arguments.isEmpty, + ProcessInfo.processInfo.arguments.contains("--tuist-dump") + else { return } + let encoder = JSONEncoder() + let data = try! encoder.encode(entity) + let manifest = String(data: data, encoding: .utf8)! + print("TUIST_MANIFEST_START") + print(manifest) + print("TUIST_MANIFEST_END") +} +``` + +`dumpIfNeeded(self)` is invoked from inside the *initializer* of each top-level manifest type. For example, `Project.init(...)` ends with `dumpIfNeeded(self)` (`cli/Sources/ProjectDescription/Project.swift:109`). This is what makes a bare `let _ = Project(...)` self-publishing — the user never has to call anything; constructing the value at the top level *is* the side-effect. + +**Decode site** (host process): `cli/Sources/TuistLoader/Loaders/ManifestLoader.swift:127, 175, 257`: + +```swift +private let decoder: JSONDecoder +… +return try decoder.decode(T.self, from: data) +``` + +All top-level `ProjectDescription` types (`Project`, `Workspace`, `Config`, `Template`, `Plugin`, `PackageSettings`) are `Codable` — that's what allows the same `Decodable` constraint on `loadManifest(...)` (`ManifestLoader.swift:244`) to handle every manifest kind. + +Note the asymmetry: the manifest side imports `ProjectDescription` and produces `ProjectDescription.Project`; the host side decodes into the **same** `ProjectDescription.Project` Swift type (`ManifestLoading.loadProject` returns `ProjectDescription.Project`), and only later does `ManifestModelConverter` / `Project+ManifestMapper.swift` map that into `XcodeGraph.Project` (the internal model). So `ProjectDescription` plays a dual role: API surface for users + DTO/IR for the bridge. + +## 4. ProjectDescriptionHelpers + +The flow is: locate → hash → compile-or-reuse → splice flags into the manifest `swift` command. + +**Locate** (`HelpersDirectoryLocator.swift:37-44`): + +```swift +public func locate(at: AbsolutePath) async throws -> AbsolutePath? { + guard let rootDirectory = try await rootDirectoryLocator.locate(from: at) else { return nil } + let helpersDirectory = rootDirectory + .appending(component: Constants.tuistDirectoryName) // "Tuist" + .appending(component: Constants.helpersDirectoryName) // "ProjectDescriptionHelpers" + if try await !fileSystem.exists(helpersDirectory) { return nil } + return helpersDirectory +} +``` + +So the discovery rule is literally: walk up from the manifest's directory to the project root, then look for `Tuist/ProjectDescriptionHelpers/`. If absent, helpers are skipped. + +**Build** (`ProjectDescriptionHelpersBuilder.swift:174-255`): each helpers directory is hashed (see section 5), the hash becomes a subdirectory name under the on-disk cache, and the build is gated on whether that directory already exists. If not, files are globbed (`**/*.swift`), `swiftc` runs into a sibling **staging directory** (`.tmp..`) and is then atomically `rename(2)`'d into place (`ProjectDescriptionHelpersBuilder.swift:204-244`) — explicitly designed to be safe under concurrent Tuist processes. + +**Plugin helpers** (`ProjectDescriptionHelpersBuilder.swift:132-144`) get built first because local helpers may import plugin helpers but not vice-versa. The plugin helper modules are then passed in as `customProjectDescriptionHelperModules` to the local helpers compile, so the local helpers can `import `. + +**Splicing into the manifest invocation** (`ManifestLoader.swift:405-428`): for `.project`, `.template`, `.workspace`, `.packageSettings` manifests, the helpers builder is called; for `.config`, `.plugin`, `.package`, helpers are *not* built (so you can't `import ProjectDescriptionHelpers` from `Tuist.swift`/`Plugin.swift` — the loader logs a clear error if you try, see `logUnexpectedImportErrorIfNeeded`, `ManifestLoader.swift:576-592`). + +In-process the builder also memoizes via `builtHelpers: ThreadSafe<[AbsolutePath: Task<…>]>` (`ProjectDescriptionHelpersBuilder.swift:69, 180-252`), so within one Tuist invocation a helpers directory is compiled at most once even if many manifests are loaded. + +## 5. Caching + +There are **two** caches; both contribute to the "manifest didn't change → skip work" property. + +**Helpers cache** (`ProjectDescriptionHelpersHasher.swift:34-55`): + +```swift +let fileHashes = try await fileSystem + .glob(directory: helpersDirectory, include: ["**/*.swift"]) + .collect() + .sorted() + .compactMap { $0.sha256() } + .compactMap { $0.compactMap { byte in String(format: "%02x", byte) }.joined() } +let tuistEnvVariables = Environment.current.manifestLoadingVariables.map { "\($0.key)=\($0.value)" }.sorted() +let swiftlangVersion = try await SwiftVersionProvider.current.swiftlangVersion() +let macosVersion = machineEnvironment.macOSVersion +let macosSDKVersion = try await MacOSSDKVersionProvider.current.macOSSDKVersion() +… +let identifiers = + [macosVersion, macosSDKVersion, swiftlangVersion, tuistVersion] + fileHashes + tuistEnvVariables + ["\(debug)"] +return identifiers.joined(separator: "-").md5 +``` + +So the helpers cache key is sensitive to: per-file SHA-256s (sorted), macOS version, macOS SDK version, Swift compiler/toolchain version, Tuist version, every `TUIST_*` env var, and DEBUG vs release Tuist build. The on-disk location is `cacheDirectoriesProvider.cacheDirectory(for: .projectDescriptionHelpers)` (`ManifestLoader.swift:402-403`), with the hash as the directory name — defaults under `~/.cache/tuist/…` on macOS / Linux per Tuist's CacheDirectoriesProvider conventions. + +**Decoded-manifest cache** (`CachedManifestLoader.swift:181-275`): after the manifest has been compiled-and-run once, the resulting JSON-encoded `ProjectDescription.Project` value is itself cached at `cacheDirectoriesProvider.cacheDirectory(for: .manifests)` (default `~/.tuist/Cache/Manifests` per the class doc, `CachedManifestLoader.swift:15`), keyed by: + +```swift +// CachedManifestLoader.swift:192-198 +return Hashes( + manifestHash: manifestHash, // SHA-256 of Project.swift + helpersHash: helpersHash, // md5 from ProjectDescriptionHelpersHasher + pluginsHash: try await pluginsHashCache.value?.value, + environmentHash: environmentHash, // md5 of TUIST_* env vars + disableSandboxHash: disableSandboxHash +) +``` + +Plus a `cacheVersion` integer (`CachedManifest.currentCacheVersion = 1`, `CachedManifestLoader.swift:314`) and the Tuist version (`CachedManifestLoader.swift:268-273`) — all five must match for a cache hit. On a hit, Tuist skips spawning `swift` entirely and decodes the cached JSON directly. + +## 6. Toolchain resolution + +Two paths. + +**For `swift` / `swiftc`**: Tuist hard-codes `/usr/bin/xcrun` (`ManifestLoader.swift:393`, `ProjectDescriptionHelpersBuilder.swift:267`) and lets xcrun resolve the toolchain. So the active Xcode (set by `xcode-select -s` or by `DEVELOPER_DIR`) determines `swift`/`swiftc`. Tuist does not appear to honor a `TOOLCHAINS=` or a custom `.xctoolchain` indirectly beyond whatever xcrun itself does. + +**For the SDK / Xcode path** (only needed for `.packageSettings`): `ManifestLoader.swift:432-457`: + +```swift +let xcodePath = try await { + if let developerDir = Environment.current.variables["DEVELOPER_DIR"] { + let developerDirPath = try AbsolutePath(validating: developerDir) + let resolvedXcodePath = if developerDirPath.components.suffix(2) == ["Contents", "Developer"] { + developerDirPath.parentDirectory.parentDirectory + } else { + developerDirPath + } + let manifestPath = resolvedXcodePath + .appending(components: "Contents", "Developer", "Toolchains", + "XcodeDefault.xctoolchain", "usr", "lib", "swift", "pm", "ManifestAPI") + if try await fileSystem.exists(manifestPath) { + return resolvedXcodePath + } + } + return try await XcodeController.current.selected().path +}() +``` + +So `DEVELOPER_DIR` is consulted first (with normalization for either form — pointing at `Xcode.app` or at `Xcode.app/Contents/Developer`), falling back to whatever `xcode-select` reports. + +**For `ProjectDescription.framework` / `libProjectDescription.dylib`**: `ResourceLocator.frameworkPath` (`cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`) searches relative to the Tuist binary's `Bundle.bundleURL`, including a Homebrew-style `bin/` ↔ `lib/` adjacency, and honors `TUIST_FRAMEWORK_SEARCH_PATHS` (space-separated). It will accept any of `libProjectDescription.dylib`, `ProjectDescription.framework`, or `PackageFrameworks/ProjectDescription.framework` — and `ProjectDescriptionSearchPaths.pathStyle(for:)` (`cli/Sources/TuistLoader/Utils/ProjectDescriptionPaths.swift:85-93`) decides whether downstream include/library/framework search paths should be derived dylib-style or framework-style. + +## 7. Failure modes + +- **Syntax error / compile failure**: `swift` exits non-zero, `CommandRunner` throws `CommandError.terminated(exitCode, standardError, command)`, and the host wraps it. Two hooks (`logUnexpectedImportErrorIfNeeded`, `logPluginHelperBuildErrorIfNeeded`, `ManifestLoader.swift:576-604`) inspect stderr and surface friendlier errors for common cases (e.g. importing `ProjectDescriptionHelpers` from `Config.swift`/`Plugin.swift`, or a missing plugin helper module). +- **Runtime crash / non-zero exit**: same path — `CommandRunner.capture(...)` throws, and the loader bubbles it up. +- **No start/end tokens in stdout**: the loader does **not** error; it falls back to treating the entire stdout as the JSON payload (`ManifestLoader.swift:352-356`). The JSON decode will then fail and produce a `ManifestLoaderError.manifestLoadingFailed` with the captured payload included for debugging (`ManifestLoader.swift:259-265`). +- **Decode failure**: each `DecodingError` case (`typeMismatch`, `valueNotFound`, `keyNotFound`, `dataCorrupted`) is mapped to a tailored `manifestLoadingFailed(context:)` error message that includes the offending JSON (`ManifestLoader.swift:267-315`). +- **Missing import / unresolved module**: stderr from `swift` carries the message; the two `logXxxErrorIfNeeded` functions detect the most common culprit (default helpers from a forbidden manifest, or a plugin helper that failed its own build) and emit a targeted log line. Beyond that the raw compile error reaches the user. +- **Hang / timeout**: no process-wait timeout in `ManifestLoader` or `CommandRunner` usage. Tuist appears to wait indefinitely for the `swift` subprocess to finish. (Inferred — `grep` for `timeout|terminate|sleep` in the loader files surfaced nothing.) +- **Empty `Package.swift`** is special-cased: `loadPackageSettings` catches `manifestLoadingFailed` with `data.count == 0` and returns a default `PackageSettings()` (`ManifestLoader.swift:217-230`). +- **Sandbox**: by default on macOS, the `swift` call is wrapped in `sandbox-exec -p ` (`ManifestLoader.swift:516-521`). The profile (`ManifestLoader.swift:547-574`) denies by default, then allows `process*`, `file-read-metadata`, RW under `/private/tmp` and `/private/var`, and read-only access to: the project path, the selected Xcode path, `com.apple.dt.Xcode.plist`, the three ProjectDescription search paths, the resolved `DEVELOPER_DIR`, and every helpers-module parent directory. Manifests can opt out via `disableSandbox: true` (config and plugin always run unsandboxed: `ManifestLoader.swift:195, 207, 234`). + +## 8. SwiftPM comparison + +SwiftPM uses model **(b)** as well but goes one extra step: it actually compiles the manifest into a **temporary executable on disk** and then `Process`-spawns *that* (`Sources/PackageLoading/ManifestLoader.swift:779-908` in `swiftlang/swift-package-manager`). In `evaluateManifest(...)`: + +```swift +// swift-package-manager/Sources/PackageLoading/ManifestLoader.swift:779-786 +let compiledManifestFile = tmpDir.appending("\(packageIdentity)-manifest\(executableSuffix)") +cmd += ["-o", compiledManifestFile.pathString] +``` + +then later (`:840, :906-908`): + +```swift +var runCmd = [compiledManifestFile.pathString] +… +runResult = try await AsyncProcess.popen(arguments: runCmd, environment: environment) +``` + +The framework search path for `PackageDescription` comes from `self.toolchain.swiftPMLibrariesLocation.manifestLibraryPath` and is added with `-F -Xlinker -rpath -Xlinker -framework PackageDescription` (or the `-L/-lPackageDescription/-rpath` dylib variant), plus an `-target` derived from `swiftPMLibrariesLocation.manifestLibraryMinimumDeploymentTarget` (`Sources/PackageLoading/ManifestLoader.swift:721-755`). Tuist by contrast (a) skips the explicit `-target` (lets `swift` interpreter pick its default), (b) does not write an intermediate executable to disk, (c) does not use `-Xlinker -rpath` for the manifest, and (d) wraps the whole thing in `sandbox-exec`. The net effect is the same — JSON on stdout, host decodes — but Tuist's path is one process and one disk-write fewer per manifest load (offset by the heavier helpers cache). + +The framework-search-path mechanics are the most directly transferable to a SyntaxKit equivalent: both tools resolve a *bundled, versioned* library that ships next to the CLI (Tuist's `ResourceLocator`, SwiftPM's `swiftPMLibrariesLocation`) and pass it via `-I/-L/-F + -framework ` (or `-l` for dylib distributions) to a single Swift compiler invocation that *also* takes the user's manifest path as a positional argument. + +## Open questions / things I couldn't determine + +- **Exact `CommandRunner.capture` semantics.** Call sites read but not implementation; *assuming* it returns stdout — the token-scanning code clearly operates on stdout-style content, but it isn't verified whether `print(…)` logs from user code (stdout) and compiler errors (stderr) end up in the same string. Inferred from `logUnexpectedImportErrorIfNeeded`'s use of `CommandError.terminated(_, standardError, command)` (`ManifestLoader.swift:577`) that stderr is captured separately. +- **Timeout.** No `terminate(after:)` / `timeout:` wiring around the manifest-spawn. If the user manifest infinite-loops, Tuist appears to wait forever. Not 100% certain because the `Command` package is a separate dependency not opened here. +- **Linux behavior.** The sandbox branch is `#if os(macOS)`; on Linux the manifest runs un-sandboxed (`ManifestLoader.swift:522-524`). Did not check whether `ResourceLocator` finds the framework on Linux or only the dylib — the `ProjectDescriptionSearchPaths.Style.commandLine` branch (which derives `-I` from a sibling `Modules/` directory) suggests Linux relies on `libProjectDescription.dylib + Modules/ProjectDescription.swiftmodule`. +- **Module cache.** SwiftPM passes `-module-cache-path` (`swift-package-manager Sources/PackageLoading/ManifestLoader.swift:759-761`); Tuist does **not** set one for the manifest invocation, so the manifest run inherits the default global Swift module cache. Significance for SyntaxKit's use case not chased. +- **`Config.swift` / `Tuist.swift` special path.** Both fall through `loadConfig` → `loadManifest(.config, …, disableSandbox: true)` → same `swift` invocation, but skip helpers entirely (`ManifestLoader.swift:407-408`). Not verified whether `Tuist.swift` accepts a different framework name (the switch in `buildArguments` lumps `.config` under `frameworkName = "ProjectDescription"`). From e4925038ac5847ed6b8a59ad9fe026c157c586de Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 16:40:49 -0400 Subject: [PATCH 02/56] POC step 1: wrap+spawn flow works at ~720ms cold (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the hand-driven wrap+spawn flow against a temporarily-dynamic SyntaxKit dylib. Pure-DSL input spliced into a Group {…} wrapper compiles cleanly under xcrun swift with -I/-L/-l + an -Xcc include for SwiftSyntax's C shims. Cold start 0.72s, warm 0.11s. Dylib weight 25MB debug. Hoisted imports work; `if`-in-Group hits a type-checker crash that's a separate SyntaxKit bug to file. Design doc updated with the -Xcc requirement, the rpath flag, the bundled-binary layout's new C-shims include directory, and the POC findings in §7. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 9 +- Docs/research/poc-step1-results.md | 138 ++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+), 2 deletions(-) create mode 100644 Docs/research/poc-step1-results.md diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md index cae86e1..c2d3bd5 100644 --- a/Docs/research/codegen-cli-design.md +++ b/Docs/research/codegen-cli-design.md @@ -72,10 +72,14 @@ print(__syntaxkit_root.generateCode()) -L \ -F \ -lSyntaxKit -framework SyntaxKit \ + -Xcc -I -Xcc \ + -Xlinker -rpath -Xlinker \ -I … -L … -F … -l # optional, when helpers exist /Input.wrapped.swift ``` +The `-Xcc -I -Xcc <…>` flag is non-obvious but required (POC step 1 finding): SyntaxKit transitively depends on `_SwiftSyntaxCShims` whose module map lives in `swift-syntax/Sources/_SwiftSyntaxCShims/include/`. The bundled-binary release must ship this header directory alongside the dylib and the SwiftSyntax `.swiftmodule` files. Without the flag, the script compile fails with `missing required module '_SwiftSyntaxCShims'`. The `-Xlinker -rpath -Xlinker <…>` flag tells the just-built script's dylib loader where to find `libSyntaxKit.dylib` at runtime. + No `--syntaxkit-dump` flag, no start/end tokens. The child's stdout is the rendered Swift source verbatim. The CLI's job is wrap → `Process` → capture stdout → atomic write to destination → clean up temp wrapper. Use `/usr/bin/env swift` rather than `/usr/bin/xcrun swift` so the same code path runs on Linux. `xcrun` is mac-only and is implicit when `env swift` resolves to Xcode's swift on macOS. @@ -137,14 +141,15 @@ Each step is independently shippable and de-risks the next. 1. **Hand-driven wrap + spawn.** Hand-write a pure-DSL `Input.swift` and a hand-rolled `Input.wrapped.swift` that imports SyntaxKit, splices the body into `Group { … }`, and prints `generateCode()`. Invoke it manually with `swift Input.wrapped.swift -I … -L … -F … -lSyntaxKit -framework SyntaxKit` against a local `swift build` of SyntaxKit. **Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit at all, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost.** Cold-start is the single biggest risk to retire — if it's 20+ seconds, the design needs rethinking. 2. **CLI subcommand for single-file mode.** Add `syntaxkit run `: parse out top-level imports with SwiftSyntax, write `Input.wrapped.swift` to a temp dir, spawn `swift` via `Foundation.Process`, capture stdout, write to `-o ` (or stdout if no `-o`). Stderr is forwarded; rewrite `Input.wrapped.swift:LINE` references back to `Input.swift:LINE` before forwarding. 3. **Folder mode.** Walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`. Add `_`-prefix skip rule. Parallelize cautiously (`swift` spawns aren't free — gate on a small concurrency limit, maybe `ProcessInfo.activeProcessorCount`). -4. **Ship a bundled-binary release.** Build SyntaxKit + SwiftSyntax dylibs, drop them in `lib/` next to the CLI binary, write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). Now the CLI is self-contained — no `swift build` required at the call site. +4. **Ship a bundled-binary release.** Build SyntaxKit as a `type: .dynamic` library, then bundle the `libSyntaxKit.dylib` + every `.swiftmodule` SyntaxKit publicly re-exports (SwiftSyntax + version-suffixed variants, SwiftOperators, SwiftParser) + the `_SwiftSyntaxCShims/include/` header dir in a `lib/` directory next to the CLI binary. Write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). POC step 1 confirmed cold-start with this layout is ~720ms. Debug dylib size is ~25 MB — re-measure under release config. 5. **Helpers directory.** Discovery + compile + flag-splicing. 6. **Output cache.** Add the cache, keyed as in §5. Add `--no-cache` for debugging. 7. **Linux smoke test.** Confirm `/usr/bin/env swift` works on Linux with the bundled dylib layout (no framework search path, but `-I + -L + -lSyntaxKit` should be sufficient, matching Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch). ## 7. What we still need to verify -- **Cold-start cost.** Single biggest unknown. Step 1 of §6 answers this. +- **Cold-start cost.** ~~Single biggest unknown.~~ Answered by POC step 1: ~720ms cold, ~110ms warm. See [`poc-step1-results.md`](./poc-step1-results.md). +- **SyntaxKit `if`-in-`Group` compiler crash.** POC step 1 surfaced this: `CodeBlockBuilderResult` claims `buildEither`/`buildOptional` support but conditionals trigger a type-checker failure-to-diagnose. Independent of the CLI design but blocks users writing conditional codegen. File as a separate SyntaxKit bug. - **Splice fidelity.** When the input body lives inside a `Group { … }` closure, is everything users naturally write in the DSL still legal? Result-builder closures don't allow `import`, top-level type decls, or top-level `let`/`var` outside the builder DSL. The wrapper hoists `import`s; we need to confirm there's no other top-level construct users would reasonably write that the wrap step would break. Verify in step 1 with a few realistic inputs (large struct, nested types, conditionals via `if`-in-builder). - **`Process` stdout/stderr separation.** Tuist captures stdout-only and merges stderr via `CommandError`. Foundation's `Process` has the same split — confirm it doesn't interleave under load, and confirm the CLI doesn't accidentally swallow stderr. - **`swift` script-mode quirks.** `swift ` runs in interpret/`-frontend -interpret` mode. Some features (`@main`, certain attributes) behave differently than in compile mode. Top-level `print` statements are fine. Verify in step 1. diff --git a/Docs/research/poc-step1-results.md b/Docs/research/poc-step1-results.md new file mode 100644 index 0000000..1958b44 --- /dev/null +++ b/Docs/research/poc-step1-results.md @@ -0,0 +1,138 @@ +# POC Step 1 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 1. Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost. + +## TL;DR — Approach is viable + +- **Cold start: ~720ms** real wall-clock for `xcrun swift Input.wrapped.swift -lSyntaxKit …` on M-series macOS with the dylib + module files unbacked by the OS page cache. +- **Warm: ~110ms** for subsequent runs. +- **Pure-DSL `Input.swift` spliced into `Group { … }`** in a generated wrapper compiles and runs end-to-end, producing the expected Swift source. +- **Bundled-dylib distribution is real and viable.** The only flags the CLI has to assemble are `-I`, `-L`, `-lSyntaxKit`, `-Xlinker -rpath -Xlinker ` and one new requirement: `-Xcc -I -Xcc ` (see §3 finding). +- **SyntaxKit dylib weight: 25.3 MB** in debug. Release build will be smaller. The CLI release artifact is dominated by this and the SwiftSyntax `.swiftmodule` files. + +## 1. What was run + +Built the dylib by temporarily flipping the SyntaxKit library product to `type: .dynamic` in `Package.swift`, ran `swift build`, copied the artifacts into a `/tmp/syntaxkit-poc/lib/` staging dir, then reverted the package manifest. + +Wrote a pure-DSL `Input.swift`: + +```swift +import SyntaxKit // optional; only for IDE + +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} + +Struct("Pet") { + Variable(.let, name: "kind", type: "String") +} +``` + +And a hand-rolled `Input.wrapped.swift`: + +```swift +import SyntaxKit + +let __syntaxkit_root = Group { + Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") + } + Struct("Pet") { + Variable(.let, name: "kind", type: "String") + } +} + +print(__syntaxkit_root.generateCode()) +``` + +Invoked with: + +``` +xcrun swift \ + -I lib -L lib -lSyntaxKit \ + -Xcc -I -Xcc lib/_SwiftSyntaxCShims-include \ + -Xlinker -rpath -Xlinker $(pwd)/lib \ + Input.wrapped.swift +``` + +Output (verbatim): + +``` +struct Person { +let name : String +let age : Int + +} +struct Pet { +let kind : String + +} +``` + +(Whitespace artifacts are SyntaxKit's `generateCode()` output as-is — out of scope for this POC.) + +## 2. Timings + +Three back-to-back runs after a cold first run: + +| Run | real | user | sys | +| --- | ---: | ---: | ---: | +| cold (first) | 0.72s | 0.77s | 0.29s | +| warm 1 | 0.14s | 0.08s | 0.04s | +| warm 2 | 0.11s | 0.07s | 0.02s | +| warm 3 | 0.11s | 0.07s | 0.02s | + +Hardware: Apple Silicon mac. Cold start is dominated by loading SyntaxKit + SwiftSyntax dylibs from disk; once cached, the swift interpreter just compiles a tiny script. For a per-file CLI invocation this is well inside the "feels instant" budget. + +## 3. New design finding: C-shim include path + +Without `-Xcc -I -Xcc <_SwiftSyntaxCShims/include>`, the script compile fails with: + +``` +:0: error: missing required module '_SwiftSyntaxCShims' +``` + +SyntaxKit transitively depends on SwiftSyntax which has a C-shims target whose module map lives at `swift-syntax/Sources/_SwiftSyntaxCShims/include/module.modulemap`. The CLI's bundled-binary distribution layout (§5 of the design doc) must include this header directory, not just the `.dylib` and `.swiftmodule` files. Updated the design doc to reflect this. + +## 4. New design finding: `if` inside `Group` is broken in SyntaxKit today + +A wrapped input containing a conditional in the builder: + +```swift +let __syntaxkit_root = Group { + if true { + Struct("A") { Variable(.let, name: "x", type: "Int") } + } +} +``` + +fails with: + +``` +error: failed to produce diagnostic for expression; please submit a bug report +let __syntaxkit_root = Group { + ^ +``` + +`CodeBlockBuilderResult` declares both `buildEither(first:)` / `buildEither(second:)` and `buildOptional` (`Sources/SyntaxKit/CodeBlocks/CodeBlockBuilderResult.swift:46-58`), so the API surface *says* `if`/`else` is supported. The compiler crash is a Swift type-checker timeout — likely from the `any CodeBlock...` variadic overload combined with `buildEither` overload resolution. No test in `Tests/SyntaxKitTests/Unit/` exercises an `if` inside `Group { … }`, which is why this hasn't been caught. + +**Implication for the CLI design:** non-blocking. v1 can document "no conditionals in input files yet" and ship; the underlying SyntaxKit fix is independent. Worth filing as a separate issue. + +## 5. Confirmed: hoisted imports work + +A wrapped input with both `import SyntaxKit` and `import Foundation` at the top compiles fine and `UUID`/`Date` resolve in the rendered struct fields. The CLI's hoist-imports step (design §3) is safe. + +## 6. Not yet retired + +- **Stderr/stdout interleaving under load.** Need a load test (large input → tons of output → confirm captured stdout is intact and stderr doesn't bleed in). +- **Linux behavior.** Step 7 of the POC ladder. Same `swift -I -L -l` flag set should work; framework-search paths (`-F`) become a no-op. +- **`@main` and other top-level forms.** Not relevant for the wrapper because we always control the wrapper's shape, but if users ever paste a class/extension declaration directly into `Input.swift` we need to reject it cleanly. + +## 7. Updates to the design doc to make from this POC + +- §5 distribution layout: add `Sources/_SwiftSyntaxCShims/include/` to the bundled `lib/` contents. +- §3 spawn command: include the `-Xcc -I -Xcc <…>` flag. +- §7 open questions: SwiftSyntax dylib size confirmed at ~25 MB (debug). Re-measure release. +- §7 open questions: add tracking note for the `if`-in-`Group` Swift compiler bug. From cf9f02a581b6be7deb686ad6c7b58f184ed17e54 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 16:50:17 -0400 Subject: [PATCH 03/56] Add POC step 1 reproducer script (#154) One-command reproduction: flips Package.swift to a dynamic SyntaxKit library, builds, stages dylib + swiftmodules + _SwiftSyntaxCShims headers into /tmp/syntaxkit-poc/, writes a pure-DSL input + hand-rolled wrapper, and runs one cold + three warm timings. Restores Package.swift on exit via trap. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step1.sh | 117 +++++++++++++++++++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100755 Docs/research/poc-step1.sh diff --git a/Docs/research/poc-step1.sh b/Docs/research/poc-step1.sh new file mode 100755 index 0000000..fe41e3b --- /dev/null +++ b/Docs/research/poc-step1.sh @@ -0,0 +1,117 @@ +#!/usr/bin/env bash +# +# POC step 1 reproducer for issue #154. +# Run from anywhere; resolves the repo root from its own location. +# +# What it does: +# 1. Backs up Package.swift, flips the SyntaxKit library to type: .dynamic. +# 2. swift build (produces libSyntaxKit.dylib). +# 3. Stages dylib + swiftmodules + _SwiftSyntaxCShims headers into /tmp/syntaxkit-poc/lib/. +# 4. Writes a pure-DSL Input.swift and a hand-rolled Input.wrapped.swift. +# 5. Runs the wrapped script once cold + three times warm, printing timings. +# 6. Restores Package.swift on exit (even on failure). + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "This reproducer is macOS-only. Linux smoke test is POC step 7." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +POC_DIR="/tmp/syntaxkit-poc" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found — has Package.swift changed shape?") +p.write_text(src.replace(old, new, 1)) +PY + +cd "$REPO_ROOT" +echo "==> swift build" +swift build + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1 || true)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate .build//debug" >&2 + exit 1 +fi + +echo "==> Staging $POC_DIR/lib/" +rm -rf "$POC_DIR" +mkdir -p "$POC_DIR/lib" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" +cp -r ".build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$POC_DIR/lib/_SwiftSyntaxCShims-include" + +cat > "$POC_DIR/Input.swift" <<'SWIFT' +// Pure-DSL input. No print, no @main, no boilerplate. +import SyntaxKit // optional; only for IDE autocomplete + +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} + +Struct("Pet") { + Variable(.let, name: "kind", type: "String") +} +SWIFT + +cat > "$POC_DIR/Input.wrapped.swift" <<'SWIFT' +import SyntaxKit + +let __syntaxkit_root = Group { + Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") + } + Struct("Pet") { + Variable(.let, name: "kind", type: "String") + } +} + +print(__syntaxkit_root.generateCode()) +SWIFT + +cd "$POC_DIR" + +SWIFT_ARGS=( + -I lib -L lib -lSyntaxKit + -Xcc -I -Xcc lib/_SwiftSyntaxCShims-include + -Xlinker -rpath -Xlinker "$POC_DIR/lib" + Input.wrapped.swift +) + +echo +echo "==> Cold run (full output + timing):" +/usr/bin/time -p xcrun swift "${SWIFT_ARGS[@]}" + +echo +echo "==> Warm runs (timings only, output discarded):" +for _ in 1 2 3; do + /usr/bin/time -p xcrun swift "${SWIFT_ARGS[@]}" >/dev/null +done + +echo +echo "==> Done. Staging dir kept at $POC_DIR for further poking." From 840cb0c64aaa6ac51d62da0ad0997aff6316baee Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 16:56:48 -0400 Subject: [PATCH 04/56] Release-config dylib size: 9.3 MB stripped (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured SyntaxKit dylib under -c release with strip -x: 25 MB debug → 18 MB release → 9.3 MB stripped. Warm execution timings identical to debug. Distribution sizing is comfortable. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 2 +- Docs/research/poc-step1-results.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md index c2d3bd5..0056f96 100644 --- a/Docs/research/codegen-cli-design.md +++ b/Docs/research/codegen-cli-design.md @@ -141,7 +141,7 @@ Each step is independently shippable and de-risks the next. 1. **Hand-driven wrap + spawn.** Hand-write a pure-DSL `Input.swift` and a hand-rolled `Input.wrapped.swift` that imports SyntaxKit, splices the body into `Group { … }`, and prints `generateCode()`. Invoke it manually with `swift Input.wrapped.swift -I … -L … -F … -lSyntaxKit -framework SyntaxKit` against a local `swift build` of SyntaxKit. **Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit at all, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost.** Cold-start is the single biggest risk to retire — if it's 20+ seconds, the design needs rethinking. 2. **CLI subcommand for single-file mode.** Add `syntaxkit run `: parse out top-level imports with SwiftSyntax, write `Input.wrapped.swift` to a temp dir, spawn `swift` via `Foundation.Process`, capture stdout, write to `-o ` (or stdout if no `-o`). Stderr is forwarded; rewrite `Input.wrapped.swift:LINE` references back to `Input.swift:LINE` before forwarding. 3. **Folder mode.** Walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`. Add `_`-prefix skip rule. Parallelize cautiously (`swift` spawns aren't free — gate on a small concurrency limit, maybe `ProcessInfo.activeProcessorCount`). -4. **Ship a bundled-binary release.** Build SyntaxKit as a `type: .dynamic` library, then bundle the `libSyntaxKit.dylib` + every `.swiftmodule` SyntaxKit publicly re-exports (SwiftSyntax + version-suffixed variants, SwiftOperators, SwiftParser) + the `_SwiftSyntaxCShims/include/` header dir in a `lib/` directory next to the CLI binary. Write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). POC step 1 confirmed cold-start with this layout is ~720ms. Debug dylib size is ~25 MB — re-measure under release config. +4. **Ship a bundled-binary release.** Build SyntaxKit as a `type: .dynamic` library under `-c release`, then `strip -x` the resulting dylib. Bundle the stripped `libSyntaxKit.dylib` + every `.swiftmodule` SyntaxKit publicly re-exports (SwiftSyntax + version-suffixed variants, SwiftOperators, SwiftParser) + the `_SwiftSyntaxCShims/include/` header dir in a `lib/` directory next to the CLI binary. Write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). POC step 1 confirmed cold-start with this layout is ~720ms; stripped release dylib is **9.3 MB** on Apple Silicon (down from 25 MB debug / 18 MB unstripped release). 5. **Helpers directory.** Discovery + compile + flag-splicing. 6. **Output cache.** Add the cache, keyed as in §5. Add `--no-cache` for debugging. 7. **Linux smoke test.** Confirm `/usr/bin/env swift` works on Linux with the bundled dylib layout (no framework search path, but `-I + -L + -lSyntaxKit` should be sufficient, matching Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch). diff --git a/Docs/research/poc-step1-results.md b/Docs/research/poc-step1-results.md index 1958b44..1940027 100644 --- a/Docs/research/poc-step1-results.md +++ b/Docs/research/poc-step1-results.md @@ -8,7 +8,7 @@ - **Warm: ~110ms** for subsequent runs. - **Pure-DSL `Input.swift` spliced into `Group { … }`** in a generated wrapper compiles and runs end-to-end, producing the expected Swift source. - **Bundled-dylib distribution is real and viable.** The only flags the CLI has to assemble are `-I`, `-L`, `-lSyntaxKit`, `-Xlinker -rpath -Xlinker ` and one new requirement: `-Xcc -I -Xcc ` (see §3 finding). -- **SyntaxKit dylib weight: 25.3 MB** in debug. Release build will be smaller. The CLI release artifact is dominated by this and the SwiftSyntax `.swiftmodule` files. +- **SyntaxKit dylib weight:** 25 MB debug → 18 MB release → **9.3 MB stripped release**. The 9.3 MB number is the one that matters for distribution and is well within range of a normal CLI binary. ## 1. What was run @@ -134,5 +134,5 @@ A wrapped input with both `import SyntaxKit` and `import Foundation` at the top - §5 distribution layout: add `Sources/_SwiftSyntaxCShims/include/` to the bundled `lib/` contents. - §3 spawn command: include the `-Xcc -I -Xcc <…>` flag. -- §7 open questions: SwiftSyntax dylib size confirmed at ~25 MB (debug). Re-measure release. +- §7 open questions: SyntaxKit dylib size measured at 25 MB debug, 18 MB release, 9.3 MB release+stripped (`strip -x`). Warm performance identical between debug and release builds. - §7 open questions: add tracking note for the `if`-in-`Group` Swift compiler bug. From 9e5affa136805c06ea16a371fa9c9be2cecb645f Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 17:01:01 -0400 Subject: [PATCH 05/56] POC step 2: skitrun CLI wraps + spawns SyntaxKit DSL inputs (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New executable target `skitrun` (POC name) wraps the hand-driven step-1 flow in real code: parse input with SwiftSyntax, hoist top-level imports, emit a Group { … } wrapper fenced in #sourceLocation directives, spawn `swift` via Foundation.Process, pipe stdout through, forward stderr with literal-path fix-up. #sourceLocation does the heavy lifting for diagnostic fidelity — a NonexistentType in InputError.swift:4 now reports as InputError.swift:4:37 from the spawned swift, with no manual stderr arithmetic. Verified default-stdout, -o file output, hoisted Foundation import, and error path rewriting all work end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step2-results.md | 70 +++++++++ Package.swift | 12 ++ Sources/skitrun/Main.swift | 243 +++++++++++++++++++++++++++++ 3 files changed, 325 insertions(+) create mode 100644 Docs/research/poc-step2-results.md create mode 100644 Sources/skitrun/Main.swift diff --git a/Docs/research/poc-step2-results.md b/Docs/research/poc-step2-results.md new file mode 100644 index 0000000..5bd6cd2 --- /dev/null +++ b/Docs/research/poc-step2-results.md @@ -0,0 +1,70 @@ +# POC Step 2 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 2. Goal: wrap the hand-driven flow from step 1 inside an actual CLI executable that parses imports with SwiftSyntax, generates the wrapper, spawns `swift`, and forwards output/errors. + +## What landed + +New executable target `skitrun` (POC name; final CLI name TBD) at `Sources/skitrun/Main.swift`. Single file, ~180 lines. Depends only on `SwiftSyntax` + `SwiftParser` — does **not** depend on SyntaxKit, since the host doesn't render anything itself. + +## Usage + +``` +skitrun [-o ] [--lib ] +``` + +`--lib` defaults to `/tmp/syntaxkit-poc/lib` so it works against the artifacts produced by [`poc-step1.sh`](./poc-step1.sh) without further flags. + +Build it once: `swift build --product skitrun`. The binary lands at `.build//debug/skitrun`. + +## Verified flows + +1. **Default (stdout):** `skitrun Input.swift` prints rendered Swift to stdout. +2. **File output:** `skitrun Input.swift -o Out.swift` writes the file atomically. +3. **Hoisted imports:** an input with `import Foundation` at the top compiles cleanly; `UUID`/`Date` resolve in the rendered struct. +4. **Compiler diagnostics map to the input file.** A deliberate `type: NonexistentType` in `InputError.swift:4` produces: + ``` + /tmp/syntaxkit-poc/InputError.swift:4:37: error: cannot find 'NonexistentType' in scope + ``` + The path and line are correct — confirming `#sourceLocation` is doing the work end-to-end. + +## How the wrap works + +`SwiftParser.Parser.parse(source:)` produces a `SourceFileSyntax`. We walk `tree.statements`: + +- Every leading `ImportDeclSyntax` is collected for hoisting. +- The first non-import statement marks the start of the body. +- Everything from that byte offset forward is the body, copied verbatim. + +The wrapper is then: + +```swift +import SyntaxKit + + +let __skitrun_root = Group { +#sourceLocation(file: "", line: ) + +#sourceLocation() +} + +print(__skitrun_root.generateCode()) +``` + +`#sourceLocation` is what gives us diagnostic fidelity for free — the Swift compiler honors it and rewrites file/line in every error/warning emitted from the body range. No manual stderr line-number arithmetic needed. + +## Spawn shape + +`Foundation.Process` invoking `/usr/bin/env swift` with the exact flag set from POC step 1, captured into `stdoutPipe` and `stderrPipe`. Stdout is written verbatim to the output destination. Stderr is forwarded after one fix-up: any remaining literal `//skitrun-/Input.wrapped.swift` references (those outside the `#sourceLocation` range — i.e. errors in the preamble itself) get rewritten to the input path. + +## Surface limits worth knowing + +- **Snippet gutter line numbers in diagnostics show wrapper line numbers, not input line numbers.** The compiler maps the *file/line* in the diagnostic header via `#sourceLocation` but shows the surrounding source snippet from the actual file with its actual line numbers. The path and starting line are correct (navigable), but the gutter `7 |` / `8 |` markers may not match the input's line numbering. Cosmetic; doesn't affect navigation. +- **No timeout yet.** The design calls for a 60s default. Adding `Process.terminate(after:)` is a step 6 (cache) sibling concern. +- **Stdin / stderr interleaving under load not tested.** Step 7 territory. +- **`if`-in-`Group` still crashes the compiler** ([#155](https://github.com/brightdigit/SyntaxKit/issues/155)). Independent SyntaxKit bug; `skitrun` would happily pass such an input through, but the spawned `swift` would fail with the same opaque diagnostic from step 1. + +## What's next + +The natural step 3 is folder mode (walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`). All the per-file work is already in place — folder mode is just iteration + concurrency-limited fan-out + a `_`-prefix skip rule. Modest engineering, low risk. + +After that, step 4 (bundled-binary release) is where the design hits its biggest remaining systems-integration question: how to actually ship the `lib/` directory next to the binary across SwiftPM build, install, and `brew` distribution. diff --git a/Package.swift b/Package.swift index ea48d24..1087718 100644 --- a/Package.swift +++ b/Package.swift @@ -95,6 +95,10 @@ let package = Package( name: "skit", targets: ["skit"] ), + .executable( + name: "skitrun", + targets: ["skitrun"] + ), ], dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), @@ -144,6 +148,14 @@ let package = Package( dependencies: ["SyntaxParser"], swiftSettings: swiftSettings ), + .executableTarget( + name: "skitrun", + dependencies: [ + .product(name: "SwiftSyntax", package: "swift-syntax"), + .product(name: "SwiftParser", package: "swift-syntax") + ], + swiftSettings: swiftSettings + ), .testTarget( name: "SyntaxKitTests", dependencies: ["SyntaxKit"], diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift new file mode 100644 index 0000000..4c3d2ee --- /dev/null +++ b/Sources/skitrun/Main.swift @@ -0,0 +1,243 @@ +// +// Main.swift +// SyntaxKit — skitrun (POC for issue #154) +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import SwiftParser +import SwiftSyntax + +@main +internal enum SkitRun { + internal static func main() throws { + let args = try CLIArgs.parse(CommandLine.arguments) + + let inputURL = URL(fileURLWithPath: args.inputPath) + let absoluteInputPath = inputURL.standardizedFileURL.path + let source = try String(contentsOf: inputURL, encoding: .utf8) + let wrapped = wrap(source: source, originalPath: absoluteInputPath) + + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("skitrun-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + + let result = try runSwift( + wrappedPath: wrappedURL.path, + libPath: args.libPath + ) + + if !result.stderr.isEmpty { + // #sourceLocation already maps body diagnostics to the input file. + // For diagnostics in the preamble (lines outside the body) the path + // still references the wrapper — rewrite verbatim path occurrences so + // users see something coherent. + let rewritten = result.stderr.replacingOccurrences( + of: wrappedURL.path, + with: absoluteInputPath + ) + FileHandle.standardError.write(Data(rewritten.utf8)) + } + + guard result.exitCode == 0 else { + exit(result.exitCode) + } + + if let outputPath = args.outputPath { + try result.stdout.write(to: URL(fileURLWithPath: outputPath)) + } else { + FileHandle.standardOutput.write(result.stdout) + } + } +} + +// MARK: - Arg parsing + +private struct CLIArgs { + let inputPath: String + let outputPath: String? + let libPath: String + + static func parse(_ argv: [String]) throws -> CLIArgs { + var inputPath: String? + var outputPath: String? + var libPath = "/tmp/syntaxkit-poc/lib" + + var i = 1 + while i < argv.count { + let arg = argv[i] + switch arg { + case "-o", "--output": + guard i + 1 < argv.count else { throw usage("-o requires a value") } + outputPath = argv[i + 1] + i += 2 + case "--lib": + guard i + 1 < argv.count else { throw usage("--lib requires a value") } + libPath = argv[i + 1] + i += 2 + case "-h", "--help": + FileHandle.standardError.write(Data(helpText.utf8)) + exit(0) + case _ where arg.hasPrefix("-"): + throw usage("unknown flag: \(arg)") + default: + guard inputPath == nil else { throw usage("only one input file is supported") } + inputPath = arg + i += 1 + } + } + + guard let inputPath else { throw usage("missing input file") } + return CLIArgs(inputPath: inputPath, outputPath: outputPath, libPath: libPath) + } +} + +private let helpText = """ + skitrun [-o ] [--lib ] + + POC for issue #154 — runs a SyntaxKit DSL input file by wrapping it in a + Group { … } closure and spawning `swift`. + + Options: + -o, --output Write rendered Swift to (default: stdout). + --lib Directory containing libSyntaxKit.dylib + module files. + (default: /tmp/syntaxkit-poc/lib, produced by + Docs/research/poc-step1.sh) + """ + +private func usage(_ message: String) -> CLIError { + CLIError(message: "\(message)\n\n\(helpText)\n") +} + +private struct CLIError: Error, CustomStringConvertible { + let message: String + var description: String { message } +} + +// MARK: - Wrapping + +/// Splits the input into hoisted `import` declarations and a verbatim body, +/// returning a complete Swift program that runs SyntaxKit on the body. +/// +/// The body is fenced in `#sourceLocation` directives so compiler diagnostics +/// in the body reference the original input file and line numbers. +internal func wrap(source: String, originalPath: String) -> String { + let tree = Parser.parse(source: source) + let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) + + // Find the first non-import top-level statement; everything before it that + // is an import gets hoisted, anything before that which is *not* an import + // stays in the body (e.g. a top-level `// comment` is left alone). + var hoisted: [String] = [] + var firstBodyByte: AbsolutePosition? + + for item in tree.statements { + if let importDecl = item.item.as(ImportDeclSyntax.self), + firstBodyByte == nil { + hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) + continue + } + firstBodyByte = item.position + break + } + + let body: String + let firstBodyLine: Int + if let firstBodyByte { + let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) + body = String(source[start...]) + firstBodyLine = locConverter.location(for: firstBodyByte).line + } else { + body = "" + firstBodyLine = 1 + } + + let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" + + // #sourceLocation must use a forward-slash path; escape backslashes/quotes + // defensively even though macOS paths shouldn't contain them. + let escapedPath = originalPath + .replacingOccurrences(of: "\\", with: "\\\\") + .replacingOccurrences(of: "\"", with: "\\\"") + + return """ + import SyntaxKit + \(hoistedBlock) + let __skitrun_root = Group { + #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) + \(body) + #sourceLocation() + } + + print(__skitrun_root.generateCode()) + """ +} + +// MARK: - Spawning swift + +private struct RunResult { + let exitCode: Int32 + let stdout: Data + let stderr: String +} + +private func runSwift(wrappedPath: String, libPath: String) throws -> RunResult { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/env") + process.arguments = [ + "swift", + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-rpath", "-Xlinker", libPath, + wrappedPath + ] + + let stdoutPipe = Pipe() + let stderrPipe = Pipe() + process.standardOutput = stdoutPipe + process.standardError = stderrPipe + + try process.run() + process.waitUntilExit() + + let stdoutData = stdoutPipe.fileHandleForReading.readDataToEndOfFile() + let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() + + return RunResult( + exitCode: process.terminationStatus, + stdout: stdoutData, + stderr: String(decoding: stderrData, as: UTF8.self) + ) +} From dd209c9f8ed8301a48e9b2092b4f58ee5410ce9e Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 11 May 2026 21:08:08 -0400 Subject: [PATCH 06/56] POC step 3: skitrun folder mode (#154) skitrun InputDir/ -o OutDir/ walks **/*.swift (skipping `_`-prefixed files), runs the per-file wrap+spawn over withTaskGroup capped at activeProcessorCount, and writes successes into mirrored paths under OutDir/. Failures don't abort the batch: successes are still written and the CLI exits non-zero with a `skitrun: N/M succeeded` summary. Verified happy path (3 files, 1.41s wall vs 0.72s cold baseline), skip rule (deliberately-invalid `_HelperShouldBeSkipped.swift` is not visited), partial failure (3/4 succeeded with exit 1), and single-file regression. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step3-results.md | 48 ++++++ Sources/skitrun/Main.swift | 254 +++++++++++++++++++++++------ 2 files changed, 252 insertions(+), 50 deletions(-) create mode 100644 Docs/research/poc-step3-results.md diff --git a/Docs/research/poc-step3-results.md b/Docs/research/poc-step3-results.md new file mode 100644 index 0000000..be524bf --- /dev/null +++ b/Docs/research/poc-step3-results.md @@ -0,0 +1,48 @@ +# POC Step 3 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 3. Goal: extend `skitrun` to walk a directory of `.swift` inputs, mirror the relative paths into an output directory, and run the per-file work concurrently with a sane cap. + +## What changed + +`skitrun` now accepts a directory as input: + +``` +skitrun InputDir/ -o OutDir/ +``` + +When the input is a directory, the existing single-file work is hoisted into a `processFile(inputPath:libPath:)` helper. The new `runDirectory(...)` driver walks the input with `FileManager.enumerator`, fan-outs the per-file work over `withTaskGroup`, and writes successes into mirrored paths under `OutDir/`. Single-file mode is unchanged. + +Three small conventions ride along: + +- **`_`-prefix skip rule.** `_Helpers.swift`, `_Shared.swift`, etc. are not processed. (Confirmed against `_HelperShouldBeSkipped.swift` containing deliberately-invalid Swift — skitrun didn't try to compile it.) +- **`activeProcessorCount` concurrency cap.** The task group keeps that many in-flight `swift` spawns at a time, draining + refilling as each finishes. +- **Tuist-analog partial semantics.** Successful files are *always* written, even when other files in the same batch fail. The CLI exits non-zero if any failed and prints a `skitrun: N/M succeeded` summary to stderr. + +## Verified flows + +1. **Happy path.** A `codegen/` tree with `Models/Person.swift`, `Models/Pet.swift`, `Audit/Snapshot.swift` (the last with a hoisted `import Foundation`) produces a mirrored `out/Models/{Person,Pet}.swift` + `out/Audit/Snapshot.swift`. Total wall time 1.41s for 3 files (vs. 0.72s baseline cold-start for one). +2. **Skip rule.** `codegen/_HelperShouldBeSkipped.swift` contains the literal line `this is not valid swift`. It is not visited, the rest of the tree processes cleanly. +3. **Partial failure.** Adding a `Models/Bad.swift` with `type: TypeThatDoesNotExist` produces: + ``` + ---- /tmp/skitrun-folder-test/codegen/Models/Bad.swift ---- + /tmp/skitrun-folder-test/codegen/Models/Bad.swift:4:37: error: cannot find 'TypeThatDoesNotExist' in scope + … + skitrun: 3/4 succeeded + ``` + Exit code 1. Person/Pet/Snapshot still written. +4. **Single-file regression.** Both `skitrun Input.swift` (stdout) and `skitrun Input.swift -o Out.swift` (file) still work after the refactor. + +## Parallelism observations + +A quick timing on 3 parallel files vs. 1 cold-start baseline: + +| | wall time | +| --- | ---: | +| 1 file, cold | 0.72s | +| 3 files, cold, `withTaskGroup` cap = `activeProcessorCount` | 1.41s | + +That's well below 3×0.72 = 2.16s, confirming the parallelism is buying something — but also clearly slower than 3×0.11 = 0.33s warm, meaning successive `swift` invocations don't fully share OS file-cache benefits within a single batch run. (Each spawn still pays its own compile cost; the dylib pages are warm after the first, but compile work isn't deduplicated.) For larger batches we'd want to measure where the curve goes — and eventually pull the work into a single long-lived `swift` process driving all inputs, to skip the per-file compile overhead entirely. Out of scope for v1. + +## What's next + +Step 4 — bundled-binary release. The first real systems-integration challenge: how the `lib/` directory ships next to the `skitrun` binary across `swift build`, `swift run`, and `brew install`. Today users have to run `Docs/research/poc-step1.sh` to stage `/tmp/syntaxkit-poc/lib/`; that has to become "user installs the CLI, it just works." diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 4c3d2ee..0365afe 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -33,56 +33,193 @@ import SwiftSyntax @main internal enum SkitRun { - internal static func main() throws { + internal static func main() async throws { let args = try CLIArgs.parse(CommandLine.arguments) - let inputURL = URL(fileURLWithPath: args.inputPath) - let absoluteInputPath = inputURL.standardizedFileURL.path - let source = try String(contentsOf: inputURL, encoding: .utf8) - let wrapped = wrap(source: source, originalPath: absoluteInputPath) - - let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skitrun-\(UUID().uuidString)") - try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) - defer { try? FileManager.default.removeItem(at: tmpDir) } - - let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") - try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - - let result = try runSwift( - wrappedPath: wrappedURL.path, - libPath: args.libPath - ) - - if !result.stderr.isEmpty { - // #sourceLocation already maps body diagnostics to the input file. - // For diagnostics in the preamble (lines outside the body) the path - // still references the wrapper — rewrite verbatim path occurrences so - // users see something coherent. - let rewritten = result.stderr.replacingOccurrences( - of: wrappedURL.path, - with: absoluteInputPath + switch args.mode { + case .singleFile(let input, let output): + try runSingleFile(inputPath: input, outputPath: output, libPath: args.libPath) + case .directory(let inputDir, let outputDir): + let exitCode = await runDirectory( + inputDir: inputDir, + outputDir: outputDir, + libPath: args.libPath ) - FileHandle.standardError.write(Data(rewritten.utf8)) + exit(exitCode) } + } +} + +// MARK: - Single-file mode + +private func runSingleFile(inputPath: String, outputPath: String?, libPath: String) throws { + let result = try processFile(inputPath: inputPath, libPath: libPath) + if !result.stderr.isEmpty { + FileHandle.standardError.write(Data(result.stderr.utf8)) + } + guard result.exitCode == 0 else { + exit(result.exitCode) + } + if let outputPath { + try result.stdout.write(to: URL(fileURLWithPath: outputPath)) + } else { + FileHandle.standardOutput.write(result.stdout) + } +} + +// MARK: - Folder mode + +private func runDirectory(inputDir: String, outputDir: String, libPath: String) async -> Int32 { + let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL + let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL + + let inputs: [URL] + do { + inputs = try collectInputs(at: inputURL) + } catch { + FileHandle.standardError.write(Data("skitrun: failed to walk \(inputDir): \(error)\n".utf8)) + return 1 + } + + if inputs.isEmpty { + FileHandle.standardError.write(Data("skitrun: no .swift inputs under \(inputDir)\n".utf8)) + return 0 + } - guard result.exitCode == 0 else { - exit(result.exitCode) + let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) + + var outcomes: [FileOutcome] = [] + var iterator = inputs.makeIterator() + + await withTaskGroup(of: FileOutcome.self) { group in + for _ in 0.. +} + +private func runOne(_ input: URL, libPath: String) -> FileOutcome { + do { + let result = try processFile(inputPath: input.path, libPath: libPath) + return FileOutcome(input: input, result: .success(result)) + } catch { + return FileOutcome(input: input, result: .failure(error)) + } +} + +private func collectInputs(at inputDir: URL) throws -> [URL] { + guard let enumerator = FileManager.default.enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey], + options: [.skipsHiddenFiles] + ) else { + throw CLIError(message: "could not enumerate \(inputDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey]) + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } +} + +// MARK: - Per-file work + +private struct ProcessResult { + let exitCode: Int32 + let stdout: Data + let stderr: String +} + +private func processFile(inputPath: String, libPath: String) throws -> ProcessResult { + let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL + let absoluteInputPath = inputURL.path + let source = try String(contentsOf: inputURL, encoding: .utf8) + let wrapped = wrap(source: source, originalPath: absoluteInputPath) + + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("skitrun-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + + let raw = try runSwift(wrappedPath: wrappedURL.path, libPath: libPath) + // #sourceLocation maps body diagnostics back to the input file. Errors in + // the preamble (lines outside the body) still reference the wrapper — + // rewrite literal occurrences of its path so users see something coherent. + let stderr = raw.stderr.replacingOccurrences( + of: wrappedURL.path, + with: absoluteInputPath + ) + return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } // MARK: - Arg parsing private struct CLIArgs { - let inputPath: String - let outputPath: String? + enum Mode { + case singleFile(input: String, output: String?) + case directory(input: String, output: String) + } + + let mode: Mode let libPath: String static func parse(_ argv: [String]) throws -> CLIArgs { @@ -108,25 +245,48 @@ private struct CLIArgs { case _ where arg.hasPrefix("-"): throw usage("unknown flag: \(arg)") default: - guard inputPath == nil else { throw usage("only one input file is supported") } + guard inputPath == nil else { throw usage("only one input path is supported") } inputPath = arg i += 1 } } - guard let inputPath else { throw usage("missing input file") } - return CLIArgs(inputPath: inputPath, outputPath: outputPath, libPath: libPath) + guard let inputPath else { throw usage("missing input path") } + + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: inputPath, isDirectory: &isDirectory) else { + throw usage("input does not exist: \(inputPath)") + } + + let mode: Mode + if isDirectory.boolValue { + guard let outputPath else { + throw usage("directory inputs require -o ") + } + mode = .directory(input: inputPath, output: outputPath) + } else { + mode = .singleFile(input: inputPath, output: outputPath) + } + + return CLIArgs(mode: mode, libPath: libPath) } } private let helpText = """ - skitrun [-o ] [--lib ] + skitrun [-o ] [--lib ] - POC for issue #154 — runs a SyntaxKit DSL input file by wrapping it in a + POC for issue #154 — runs SyntaxKit DSL input(s) by wrapping each in a Group { … } closure and spawning `swift`. + Forms: + skitrun Input.swift — render to stdout + skitrun Input.swift -o Out.swift — render to a file + skitrun InputDir/ -o OutDir/ — walk **/*.swift (skipping files + prefixed with '_') and mirror + rendered output into OutDir/ + Options: - -o, --output Write rendered Swift to (default: stdout). + -o, --output Output file (single-file mode) or directory (folder mode). --lib Directory containing libSyntaxKit.dylib + module files. (default: /tmp/syntaxkit-poc/lib, produced by Docs/research/poc-step1.sh) @@ -202,13 +362,7 @@ internal func wrap(source: String, originalPath: String) -> String { // MARK: - Spawning swift -private struct RunResult { - let exitCode: Int32 - let stdout: Data - let stderr: String -} - -private func runSwift(wrappedPath: String, libPath: String) throws -> RunResult { +private func runSwift(wrappedPath: String, libPath: String) throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" let process = Process() @@ -235,7 +389,7 @@ private func runSwift(wrappedPath: String, libPath: String) throws -> RunResult let stdoutData = stdoutPipe.fileHandleForReading.readDataToEndOfFile() let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() - return RunResult( + return ProcessResult( exitCode: process.terminationStatus, stdout: stdoutData, stderr: String(decoding: stderrData, as: UTF8.self) From c6daced94f5de8c125e4249ccbf1ea703404b3d3 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 06:14:22 -0400 Subject: [PATCH 07/56] POC step 4: self-contained skitrun release bundle (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit skitrun now resolves its lib/ directory automatically: --lib flag, then $SKITRUN_LIB_DIR, then /lib/, then /../lib/skitrun/ (Homebrew layout). No more /tmp/syntaxkit-poc/lib fallback. Clear diagnostic when no lib is found, naming all four search paths. poc-step4-release.sh builds a portable bundle under .build/skitrun-release/: release-config + stripped dylib (9.3 MB), modules, C-shims headers, and the binary itself (17 MB — SwiftSyntax statically linked, follow-up to deduplicate). Tested copying the bundle to unrelated directories, both same-dir and Homebrew layouts work zero-config. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step4-release.sh | 87 ++++++++++++++++++++++++++++++ Docs/research/poc-step4-results.md | 55 +++++++++++++++++++ Sources/skitrun/Main.swift | 71 +++++++++++++++++++++--- 3 files changed, 207 insertions(+), 6 deletions(-) create mode 100755 Docs/research/poc-step4-release.sh create mode 100644 Docs/research/poc-step4-results.md diff --git a/Docs/research/poc-step4-release.sh b/Docs/research/poc-step4-release.sh new file mode 100755 index 0000000..36b2035 --- /dev/null +++ b/Docs/research/poc-step4-release.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# +# POC step 4: build a self-contained skitrun release bundle. +# +# Output: .build/skitrun-release/ +# skitrun ← the CLI binary +# lib/ +# libSyntaxKit.dylib ← release + strip -x +# *.swiftmodule ← SyntaxKit + transitively re-exported modules +# _SwiftSyntaxCShims-include/ ← C-shims headers (module map + .h files) +# +# Once produced, the binary is portable: copy the whole .build/skitrun-release/ +# directory anywhere, and `./skitrun-release/skitrun ` Just Works — no +# flags, no env vars, no SyntaxKit checkout required. + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "macOS-only for now. Linux smoke test is POC step 7." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +OUTPUT_DIR="$REPO_ROOT/.build/skitrun-release" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old, new, 1)) +PY + +cd "$REPO_ROOT" + +echo "==> swift build -c release --product skitrun" +swift build -c release --product skitrun + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/release 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate release build dir under .build//release" >&2 + exit 1 +fi + +echo "==> Staging $OUTPUT_DIR" +rm -rf "$OUTPUT_DIR" +mkdir -p "$OUTPUT_DIR/lib" + +cp "$BUILD_DIR/skitrun" "$OUTPUT_DIR/skitrun" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/" +strip -x "$OUTPUT_DIR/lib/libSyntaxKit.dylib" +cp -r "$BUILD_DIR/Modules/." "$OUTPUT_DIR/lib/" +cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$OUTPUT_DIR/lib/_SwiftSyntaxCShims-include" + +# Ensure the dylib's install_name uses @rpath so it's portable. +install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true + +BINARY_SIZE=$(ls -lh "$OUTPUT_DIR/skitrun" | awk '{print $5}') +DYLIB_SIZE=$(ls -lh "$OUTPUT_DIR/lib/libSyntaxKit.dylib" | awk '{print $5}') +TOTAL_SIZE=$(du -sh "$OUTPUT_DIR" | awk '{print $1}') + +echo +echo "==> Release bundle ready:" +echo " Binary: $BINARY_SIZE" +echo " Dylib: $DYLIB_SIZE" +echo " Total: $TOTAL_SIZE" +echo +echo "==> Try it:" +echo " $OUTPUT_DIR/skitrun " diff --git a/Docs/research/poc-step4-results.md b/Docs/research/poc-step4-results.md new file mode 100644 index 0000000..c5248ac --- /dev/null +++ b/Docs/research/poc-step4-results.md @@ -0,0 +1,55 @@ +# POC Step 4 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 4. Goal: produce a self-contained `skitrun` release bundle so users don't need a SyntaxKit checkout or `Docs/research/poc-step1.sh`. The binary finds its own `lib/` directory. + +## What landed + +1. **`resolveLibPath(override:)` in `Sources/skitrun/Main.swift`.** Search order: + 1. `--lib ` flag + 2. `$SKITRUN_LIB_DIR` env var + 3. `/lib/` — same-directory layout (the release bundle ships this way) + 4. `/../lib/skitrun/` — Homebrew layout (`bin/skitrun` ↔ `lib/skitrun/`) + + When none match, the CLI errors with a message enumerating all four paths and pointing at this script. The `/tmp/syntaxkit-poc/lib` fallback is gone. + +2. **`Docs/research/poc-step4-release.sh`.** Builds a self-contained bundle: + ``` + .build/skitrun-release/ + skitrun ← the CLI binary + lib/ + libSyntaxKit.dylib ← release + strip -x + *.swiftmodule ← SyntaxKit + transitively re-exported modules + _SwiftSyntaxCShims-include/ ← C-shims headers + ``` + Same trap-based Package.swift backup/restore as `poc-step1.sh`. `install_name_tool -id @rpath/libSyntaxKit.dylib` ensures the dylib install name is portable. + +## Verified flows + +Built bundle → copied to three unrelated locations → all worked with no flags, no env vars, no SyntaxKit checkout: + +1. **Same-directory layout.** `cp -r .build/skitrun-release /tmp/portable && /tmp/portable/skitrun Input.swift` → correct output. +2. **Homebrew layout.** `bin/skitrun + lib/skitrun/` arrangement → correct output. +3. **Error case.** `skitrun` alone in `/tmp/lonely/` (no lib anywhere) → clear diagnostic: + ``` + Could not locate SyntaxKit lib directory. Looked for: + 1. --lib (not provided) + 2. $SKITRUN_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skitrun/ (not found) + ``` +4. **Folder mode** from the portable bundle works end-to-end with partial-failure semantics intact. + +## Bundle weight + +| Component | Size | +| --- | ---: | +| `skitrun` (binary) | 17 MB | +| `libSyntaxKit.dylib` (release + stripped) | 9.3 MB | +| `lib/*` (modules + headers + dylib) | ~28 MB | +| **Total bundle** | **45 MB** | + +The 17 MB binary is unexpectedly heavy: it links SwiftSyntax statically because `skitrun` uses SwiftSyntax directly for parsing input files. So SwiftSyntax ships **twice** — once statically inside `skitrun`, once dynamically as part of the SyntaxKit dylib stack. Worth a follow-up: make `skitrun` itself dlopen SyntaxKit / share a dynamic SwiftSyntax with the dylib path. For v1 this is acceptable but is the largest single thing standing between the CLI and a "feels small" download. + +## What's next + +Step 5: helpers directory. Today users can `import Foundation` and `import SyntaxKit` from input files; step 5 lets them factor reusable codegen into a `Helpers/` directory that gets pre-compiled into `lib.dylib` and made importable from inputs. Modest engineering, but the first time `skitrun` itself invokes `swiftc` rather than `swift` (the helpers compile, distinct from the input run). See [`codegen-cli-design.md` §4](./codegen-cli-design.md#4-helpers) for the shape. diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 0365afe..ebaf3a3 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -36,20 +36,77 @@ internal enum SkitRun { internal static func main() async throws { let args = try CLIArgs.parse(CommandLine.arguments) + let libPath: String + do { + libPath = try resolveLibPath(override: args.libPath) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) + exit(2) + } + switch args.mode { case .singleFile(let input, let output): - try runSingleFile(inputPath: input, outputPath: output, libPath: args.libPath) + try runSingleFile(inputPath: input, outputPath: output, libPath: libPath) case .directory(let inputDir, let outputDir): let exitCode = await runDirectory( inputDir: inputDir, outputDir: outputDir, - libPath: args.libPath + libPath: libPath ) exit(exitCode) } } } +// MARK: - Resource location + +/// Resolves the directory containing `libSyntaxKit.dylib` + module files, +/// in priority order: explicit flag → env var → adjacent-to-binary +/// (`/lib/`) → Homebrew layout (`/../lib/skitrun/`). +internal func resolveLibPath(override: String?) throws -> String { + if let override { + guard isLibDir(override) else { + throw CLIError(message: "--lib path does not look like a SyntaxKit lib dir: \(override)") + } + return override + } + + if let env = ProcessInfo.processInfo.environment["SKITRUN_LIB_DIR"], !env.isEmpty { + guard isLibDir(env) else { + throw CLIError(message: "SKITRUN_LIB_DIR is set but path is not a lib dir: \(env)") + } + return env + } + + if let execURL = Bundle.main.executableURL?.resolvingSymlinksInPath() { + let execDir = execURL.deletingLastPathComponent() + + let adjacent = execDir.appendingPathComponent("lib").path + if isLibDir(adjacent) { return adjacent } + + let brewLayout = execDir.deletingLastPathComponent() + .appendingPathComponent("lib/skitrun").path + if isLibDir(brewLayout) { return brewLayout } + } + + throw CLIError(message: """ + Could not locate SyntaxKit lib directory. Looked for: + 1. --lib (not provided) + 2. $SKITRUN_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skitrun/ (not found) + Run Docs/research/poc-step4-release.sh to produce a self-contained + release bundle under .build/skitrun-release/. + """) +} + +private func isLibDir(_ path: String) -> Bool { + let fm = FileManager.default + var isDir: ObjCBool = false + guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } + return fm.fileExists(atPath: "\(path)/libSyntaxKit.dylib") +} + // MARK: - Single-file mode private func runSingleFile(inputPath: String, outputPath: String?, libPath: String) throws { @@ -220,12 +277,12 @@ private struct CLIArgs { } let mode: Mode - let libPath: String + let libPath: String? static func parse(_ argv: [String]) throws -> CLIArgs { var inputPath: String? var outputPath: String? - var libPath = "/tmp/syntaxkit-poc/lib" + var libPath: String? var i = 1 while i < argv.count { @@ -288,8 +345,10 @@ private let helpText = """ Options: -o, --output Output file (single-file mode) or directory (folder mode). --lib Directory containing libSyntaxKit.dylib + module files. - (default: /tmp/syntaxkit-poc/lib, produced by - Docs/research/poc-step1.sh) + When omitted, skitrun searches: $SKITRUN_LIB_DIR, + then /lib/, then /../lib/skitrun/. + Build a self-contained bundle with + Docs/research/poc-step4-release.sh. """ private func usage(_ message: String) -> CLIError { From 3d3b631d2cf77943ab050314f72b905d3bf624dc Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 08:32:31 -0400 Subject: [PATCH 08/56] POC step 5: Helpers/ discovery + per-toolchain cache (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit skitrun now walks up from the input looking for a Helpers/ directory. On hit, helpers compile via swiftc into libSyntaxKitHelpers.dylib under ~/Library/Caches/com.brightdigit.SyntaxKit/helpers//, keyed on helper-source hashes + swift --version + libSyntaxKit.dylib stamp + cache schema version. Compile lands in a tmp../ staging dir then atomic-renames into the cache path so concurrent invocations are safe. Inputs then `import SyntaxKitHelpers` and call into the compiled module; runSwift splices -I/-L/-lSyntaxKitHelpers -Xlinker -rpath onto the spawn. Two new flags: --helpers overrides discovery, --no-helpers skips it entirely. Folder mode's enumerator now also yields directories so it can skipDescendants() on a Helpers/ directly under the input root — without it the helpers would be reprocessed as inputs. Verified end-to-end via Docs/research/poc-step5.sh: cold compile 2.96s, warm cache hit 0.54s (matches step-1 warm baseline), folder mode 2/2 with Helpers/ excluded, --no-helpers errors with `no such module 'SyntaxKitHelpers'` as expected. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step5-results.md | 58 +++++++ Docs/research/poc-step5.sh | 135 ++++++++++++++++ Sources/skitrun/Helpers.swift | 247 +++++++++++++++++++++++++++++ Sources/skitrun/Main.swift | 211 +++++++++++++++++++----- 4 files changed, 610 insertions(+), 41 deletions(-) create mode 100644 Docs/research/poc-step5-results.md create mode 100755 Docs/research/poc-step5.sh create mode 100644 Sources/skitrun/Helpers.swift diff --git a/Docs/research/poc-step5-results.md b/Docs/research/poc-step5-results.md new file mode 100644 index 0000000..746570e --- /dev/null +++ b/Docs/research/poc-step5-results.md @@ -0,0 +1,58 @@ +# POC Step 5 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §4 + §6 step 5. Goal: let inputs `import SyntaxKitHelpers` from a `Helpers/` directory adjacent to the input, with discovery + on-demand `swiftc` compile + a per-toolchain cache. + +## What landed + +1. **`Sources/skitrun/Helpers.swift`.** New module with three responsibilities: + - `discoverHelpersDir(near:)` walks up from the input path looking for a `Helpers/` directory. Single-file mode starts from the file's parent; folder mode starts from the input directory. Walk-up stops at the filesystem root. + - `collectHelperSources(in:)` globs `**/*.swift` under the helpers dir, skipping `_`-prefixed files (same convention as input enumeration). + - `buildHelpers(helpersDir:libPath:)` hashes the sources + Swift version + dylib stamp into a cache key under `~/Library/Caches/com.brightdigit.SyntaxKit/helpers//`. On a cache miss, it shells out to `swiftc` into a `tmp../` staging dir, then atomic-renames into the cache path (`ProjectDescriptionHelpersBuilder` pattern from Tuist). + +2. **`Sources/skitrun/Main.swift` wiring.** + - `CLIArgs` gains `--helpers ` (explicit override) and `--no-helpers` (skip discovery). Default is auto. + - `runSwift` splices `-I/-L/-lSyntaxKitHelpers -Xlinker -rpath -Xlinker ` when helpers are present. + - Folder mode's `collectInputs` now also yields directories so it can call `enumerator.skipDescendants()` when it hits a `Helpers/` directly under the input root — otherwise the helpers would be re-processed as inputs. + - Helpers compile happens **once per invocation** in folder mode (not per input file). + +3. **`Docs/research/poc-step5.sh`.** Standalone demo: builds skitrun, stages a runtime lib, writes a tiny `Helpers/Models.swift` exporting `equatableModel(_:fields:)`, and two `inputs/*.swift` files that `import SyntaxKitHelpers` and call the helper. + +## Verified flows + +| Flow | Result | +| --- | --- | +| Cold run (cache cleared, helpers compile from scratch) | ✓ 2.96s real | +| Warm run (cache hit, same helper sources) | ✓ 0.54s real | +| Folder mode against `demo/` containing `Helpers/` + `inputs/` | ✓ 2/2 succeeded, `Helpers/*.swift` not enumerated as input | +| `--no-helpers` with an input that imports `SyntaxKitHelpers` | ✓ child `swift` errors with `no such module 'SyntaxKitHelpers'`, exit non-zero | + +The cached layout for a single helper file: + +``` +~/Library/Caches/com.brightdigit.SyntaxKit/helpers// + libSyntaxKitHelpers.dylib + SyntaxKitHelpers.swiftmodule + SyntaxKitHelpers.swiftdoc + SyntaxKitHelpers.abi.json + SyntaxKitHelpers.swiftsourceinfo +``` + +## Cache key + +SHA-256 over (in order): +- Cache schema version string (`v1`). +- For each helper source (sorted by absolute path): `lastPathComponent` + file bytes. +- `swift --version` output. +- `libSyntaxKit.dylib` size and modification time (proxy for SyntaxKit version until the bundle is versioned). + +Mutating any helper source, switching toolchains, or rebuilding SyntaxKit invalidates the cache. Adding a `cacheSchemaVersion` bump constant covers future layout changes. + +## Known rough edges + +- **Helpers cold compile is the dominant cost.** 2.96s vs 0.54s warm — the helper compile is ~2.5s on top of the ~0.5s `swift` interpret cost. Once cached it's free, but the first run after a clean checkout is noticeably slow. Acceptable for v1; could be sped up by caching the helpers `.o` files separately, but that's a step-6+ optimization. +- **Walk-up false positives.** If a user happens to have an unrelated `Helpers/` somewhere up-tree (e.g. a sibling library), skitrun will try to compile it. `--helpers ` or `--no-helpers` is the escape hatch. A future heuristic could require a sentinel file (`Helpers/.syntaxkit-helpers`) before claiming the directory. +- **Import-line diagnostics off by one.** When the user's input has `import Foo` on line 1, the wrap step puts an injected `import SyntaxKit` above it, so a child-compile error on the user's import reports `:2:8` instead of `:1:8`. `#sourceLocation` directives only wrap the body, not the hoisted imports. Easy follow-up: emit a `#sourceLocation` directive per hoisted import too. + +## What's next + +Step 6: **output cache.** Today every `skitrun` invocation re-spawns `swift` to render the input, even when nothing has changed. Add the per-input output cache from [`codegen-cli-design.md` §5](./codegen-cli-design.md#5-caching), keyed by input hash + helpers-cache key + Swift version + envHash. On a hit, skip the spawn entirely and copy the rendered output to the destination. Add `--no-cache` for debugging. diff --git a/Docs/research/poc-step5.sh b/Docs/research/poc-step5.sh new file mode 100755 index 0000000..ea633a3 --- /dev/null +++ b/Docs/research/poc-step5.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# +# POC step 5 demo: Helpers/ discovery + compile + import in input scripts. +# +# Builds skitrun, stages a runtime lib/ next to it, then runs skitrun against +# a demo project that uses `import SyntaxKitHelpers`. Demonstrates: +# 1. Cold path — Helpers/ compiles to libSyntaxKitHelpers.dylib. +# 2. Warm path — second invocation reuses the cached helpers dylib. +# 3. Folder mode — skitrun ignores Helpers/ when walking the input tree. +# 4. --no-helpers — disables discovery; the import then fails as expected. + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "macOS-only. Linux smoke test is POC step 7." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +POC_DIR="/tmp/syntaxkit-poc-step5" +DEMO_DIR="$POC_DIR/demo" +CACHE_DIR="$HOME/Library/Caches/com.brightdigit.SyntaxKit/helpers" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old, new, 1)) +PY + +cd "$REPO_ROOT" + +echo "==> swift build" +swift build + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate .build//debug" >&2 + exit 1 +fi + +echo "==> Staging $POC_DIR" +rm -rf "$POC_DIR" +mkdir -p "$POC_DIR/lib" "$DEMO_DIR/Helpers" "$DEMO_DIR/inputs" + +cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" +cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$POC_DIR/lib/_SwiftSyntaxCShims-include" + +cat > "$DEMO_DIR/Helpers/Models.swift" <<'SWIFT' +import SyntaxKit + +public func equatableModel( + _ name: String, + fields: [(name: String, type: String)] +) -> any CodeBlock { + Struct(name) { + for field in fields { + Variable(.let, name: field.name, type: field.type) + } + }.inherits("Equatable") +} +SWIFT + +cat > "$DEMO_DIR/inputs/Person.swift" <<'SWIFT' +import SyntaxKitHelpers + +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +SWIFT + +cat > "$DEMO_DIR/inputs/Pet.swift" <<'SWIFT' +import SyntaxKitHelpers + +equatableModel("Pet", fields: [ + ("kind", "String"), + ("owner", "String"), +]) +SWIFT + +echo "==> Clearing helpers cache to force cold compile" +rm -rf "$CACHE_DIR" + +echo +echo "==> Cold run (helpers compile from scratch):" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/inputs/Person.swift" + +echo +echo "==> Warm run (helpers cache hit):" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/inputs/Person.swift" >/dev/null + +echo +echo "==> Cached helper artifacts:" +find "$CACHE_DIR" -maxdepth 3 -type f | sed "s|$CACHE_DIR||" | sort + +echo +echo "==> Folder mode (Helpers/ excluded from input enumeration):" +rm -rf "$POC_DIR/out" +"$POC_DIR/skitrun" "$DEMO_DIR" -o "$POC_DIR/out" +echo " Generated files:" +find "$POC_DIR/out" -type f | sed "s|$POC_DIR/| |" + +echo +echo "==> --no-helpers should fail with an unresolved import:" +if "$POC_DIR/skitrun" --no-helpers "$DEMO_DIR/inputs/Person.swift" >/dev/null 2>&1; then + echo "FAIL: --no-helpers should have errored" >&2 + exit 1 +else + echo " ✓ skitrun returned non-zero as expected" +fi + +echo +echo "==> Done. Demo project kept at $DEMO_DIR; cache at $CACHE_DIR." diff --git a/Sources/skitrun/Helpers.swift b/Sources/skitrun/Helpers.swift new file mode 100644 index 0000000..dfccba3 --- /dev/null +++ b/Sources/skitrun/Helpers.swift @@ -0,0 +1,247 @@ +// +// Helpers.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import CryptoKit +import Foundation + +/// Hardcoded module name for the user's `Helpers/` compilation output. Inputs +/// reach the compiled helpers via `import SyntaxKitHelpers`. +internal let helpersModuleName = "SyntaxKitHelpers" + +/// Bumped when the cache layout changes in a way that requires invalidation. +private let helpersCacheSchemaVersion = "v1" + +/// A compiled `Helpers/` directory ready to splice into the input spawn. +internal struct CompiledHelpers: Sendable { + /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. + let outputDir: URL + /// Whether the build was reused from cache (false = freshly compiled). + let cacheHit: Bool +} + +// MARK: - Discovery + +/// Walks up from `inputURL` looking for a `Helpers/` directory. Returns the +/// first one found, or nil if no ancestor contains one. +/// +/// When `inputURL` is a file, the search starts from its parent. When it's a +/// directory, the search starts from the directory itself. +internal func discoverHelpersDir(near inputURL: URL) -> URL? { + let fm = FileManager.default + var isDirectory: ObjCBool = false + let exists = fm.fileExists(atPath: inputURL.path, isDirectory: &isDirectory) + var dir = (exists && isDirectory.boolValue) ? inputURL : inputURL.deletingLastPathComponent() + dir = dir.standardizedFileURL + + while true { + let candidate = dir.appendingPathComponent("Helpers") + var isDir: ObjCBool = false + if fm.fileExists(atPath: candidate.path, isDirectory: &isDir), isDir.boolValue { + return candidate.standardizedFileURL + } + let parent = dir.deletingLastPathComponent().standardizedFileURL + if parent.path == dir.path { return nil } + dir = parent + } +} + +/// Globs `**/*.swift` under `helpersDir`, skipping files prefixed with `_`. +internal func collectHelperSources(in helpersDir: URL) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: helpersDir, + includingPropertiesForKeys: [.isRegularFileKey], + options: [.skipsHiddenFiles] + ) + else { + throw CLIError(message: "could not enumerate \(helpersDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey]) + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } +} + +// MARK: - Build pipeline + +/// Compiles helper sources into a per-key cache directory and returns the +/// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. +internal func buildHelpers( + helpersDir: URL, + libPath: String +) throws -> CompiledHelpers? { + let sources = try collectHelperSources(in: helpersDir) + if sources.isEmpty { return nil } + + let key = try helpersCacheKey(sources: sources, libPath: libPath) + let cacheRoot = try syntaxKitCacheRoot() + .appendingPathComponent("helpers") + .appendingPathComponent(key) + let dylibPath = cacheRoot.appendingPathComponent("lib\(helpersModuleName).dylib").path + + let fm = FileManager.default + if fm.fileExists(atPath: dylibPath) { + return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) + } + + try fm.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent("tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") + try fm.createDirectory(at: staging, withIntermediateDirectories: true) + + do { + try compileHelpers(sources: sources, into: staging, libPath: libPath) + } catch { + try? fm.removeItem(at: staging) + throw error + } + + // Atomic rename into the cache path. If a peer beat us to it (rename failed + // because the destination now exists), keep theirs and drop ours. + do { + try fm.moveItem(at: staging, to: cacheRoot) + } catch { + try? fm.removeItem(at: staging) + if !fm.fileExists(atPath: dylibPath) { + throw error + } + } + + return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) +} + +private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) throws { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + let dylib = outDir.appendingPathComponent("lib\(helpersModuleName).dylib").path + let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path + + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/env") + var args: [String] = [ + "swiftc", + "-module-name", helpersModuleName, + "-emit-module", + "-emit-module-path", modulePath, + "-parse-as-library", + "-emit-library", + "-o", dylib, + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-install_name", + "-Xlinker", "@rpath/lib\(helpersModuleName).dylib", + "-Xlinker", "-rpath", "-Xlinker", libPath, + ] + args.append(contentsOf: sources.map(\.path)) + process.arguments = args + + let stderrPipe = Pipe() + process.standardOutput = FileHandle.nullDevice + process.standardError = stderrPipe + try process.run() + process.waitUntilExit() + + let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() + guard process.terminationStatus == 0 else { + let stderr = String(decoding: stderrData, as: UTF8.self) + throw CLIError( + message: """ + skitrun: failed to compile Helpers/ (exit \(process.terminationStatus)) + \(stderr) + """) + } +} + +// MARK: - Cache key + +private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { + var hasher = SHA256() + hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) + + for source in sources { + let data = try Data(contentsOf: source) + hasher.update(data: Data(source.lastPathComponent.utf8)) + hasher.update(data: data) + } + + if let swiftVersion = captureSwiftVersion() { + hasher.update(data: Data(swiftVersion.utf8)) + } + if let stamp = libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } + + return hasher.finalize().map { String(format: "%02x", $0) }.joined() +} + +private func captureSwiftVersion() -> String? { + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/env") + process.arguments = ["swift", "--version"] + let pipe = Pipe() + process.standardOutput = pipe + process.standardError = FileHandle.nullDevice + do { try process.run() } catch { return nil } + process.waitUntilExit() + let data = pipe.fileHandleForReading.readDataToEndOfFile() + return String(decoding: data, as: UTF8.self) +} + +private func libStamp(libPath: String) -> String? { + let dylib = "\(libPath)/libSyntaxKit.dylib" + guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } + let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 + let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 + return "\(size)/\(Int(mtime))" +} + +private func syntaxKitCacheRoot() throws -> URL { + if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { + return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + } + let home = NSHomeDirectory() + #if os(macOS) + return URL(fileURLWithPath: home) + .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") + #else + return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") + #endif +} diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index ebaf3a3..52cd1ed 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -1,6 +1,6 @@ // // Main.swift -// SyntaxKit — skitrun (POC for issue #154) +// SyntaxKit // // Created by Leo Dion. // Copyright © 2026 BrightDigit. @@ -46,18 +46,70 @@ internal enum SkitRun { switch args.mode { case .singleFile(let input, let output): - try runSingleFile(inputPath: input, outputPath: output, libPath: libPath) + let helpers = try resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: args.helpers + ) + try runSingleFile( + inputPath: input, + outputPath: output, + libPath: libPath, + helpers: helpers + ) case .directory(let inputDir, let outputDir): + let helpers = try resolveHelpers( + nearInputPath: inputDir, + libPath: libPath, + options: args.helpers + ) let exitCode = await runDirectory( inputDir: inputDir, outputDir: outputDir, - libPath: libPath + libPath: libPath, + helpers: helpers ) exit(exitCode) } } } +// MARK: - Helpers resolution + +private func resolveHelpers( + nearInputPath path: String, + libPath: String, + options: HelpersOptions +) throws -> CompiledHelpers? { + let helpersDir: URL? + switch options { + case .disabled: + return nil + case .auto: + helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) + case .explicit(let dir): + let url = URL(fileURLWithPath: dir).standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), + isDir.boolValue + else { + throw CLIError(message: "--helpers path is not a directory: \(dir)") + } + helpersDir = url + } + guard let helpersDir else { return nil } + + guard let compiled = try buildHelpers(helpersDir: helpersDir, libPath: libPath) else { + return nil + } + let suffix = compiled.cacheHit ? "cached" : "compiled" + FileHandle.standardError.write( + Data( + "skitrun: helpers \(suffix) at \(helpersDir.path)\n".utf8 + )) + return compiled +} + // MARK: - Resource location /// Resolves the directory containing `libSyntaxKit.dylib` + module files, @@ -89,15 +141,16 @@ internal func resolveLibPath(override: String?) throws -> String { if isLibDir(brewLayout) { return brewLayout } } - throw CLIError(message: """ - Could not locate SyntaxKit lib directory. Looked for: - 1. --lib (not provided) - 2. $SKITRUN_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skitrun/ (not found) - Run Docs/research/poc-step4-release.sh to produce a self-contained - release bundle under .build/skitrun-release/. - """) + throw CLIError( + message: """ + Could not locate SyntaxKit lib directory. Looked for: + 1. --lib (not provided) + 2. $SKITRUN_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skitrun/ (not found) + Run Docs/research/poc-step4-release.sh to produce a self-contained + release bundle under .build/skitrun-release/. + """) } private func isLibDir(_ path: String) -> Bool { @@ -109,8 +162,13 @@ private func isLibDir(_ path: String) -> Bool { // MARK: - Single-file mode -private func runSingleFile(inputPath: String, outputPath: String?, libPath: String) throws { - let result = try processFile(inputPath: inputPath, libPath: libPath) +private func runSingleFile( + inputPath: String, + outputPath: String?, + libPath: String, + helpers: CompiledHelpers? +) throws { + let result = try processFile(inputPath: inputPath, libPath: libPath, helpers: helpers) if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) } @@ -126,13 +184,18 @@ private func runSingleFile(inputPath: String, outputPath: String?, libPath: Stri // MARK: - Folder mode -private func runDirectory(inputDir: String, outputDir: String, libPath: String) async -> Int32 { +private func runDirectory( + inputDir: String, + outputDir: String, + libPath: String, + helpers: CompiledHelpers? +) async -> Int32 { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL let inputs: [URL] do { - inputs = try collectInputs(at: inputURL) + inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) } catch { FileHandle.standardError.write(Data("skitrun: failed to walk \(inputDir): \(error)\n".utf8)) return 1 @@ -151,12 +214,12 @@ private func runDirectory(inputDir: String, outputDir: String, libPath: String) await withTaskGroup(of: FileOutcome.self) { group in for _ in 0.. } -private func runOne(_ input: URL, libPath: String) -> FileOutcome { +private func runOne(_ input: URL, libPath: String, helpers: CompiledHelpers?) -> FileOutcome { do { - let result = try processFile(inputPath: input.path, libPath: libPath) + let result = try processFile(inputPath: input.path, libPath: libPath, helpers: helpers) return FileOutcome(input: input, result: .success(result)) } catch { return FileOutcome(input: input, result: .failure(error)) } } -private func collectInputs(at inputDir: URL) throws -> [URL] { - guard let enumerator = FileManager.default.enumerator( - at: inputDir, - includingPropertiesForKeys: [.isRegularFileKey], - options: [.skipsHiddenFiles] - ) else { +/// Returns the path of a `Helpers/` directory living directly under `inputDir`, +/// so the folder-mode enumerator can skip its descendants. Helpers that live +/// outside the input tree don't need to be excluded (they aren't enumerated). +private func helpersExcludePath(inputDir: URL) -> String? { + let candidate = inputDir.appendingPathComponent("Helpers").standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: candidate.path, isDirectory: &isDir), + isDir.boolValue + else { + return nil + } + return candidate.path +} + +private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], + options: [.skipsHiddenFiles] + ) + else { throw CLIError(message: "could not enumerate \(inputDir.path)") } var result: [URL] = [] for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey]) + let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + if values.isDirectory == true { + if let excludedDir, url.standardizedFileURL.path == excludedDir { + enumerator.skipDescendants() + } + continue + } guard values.isRegularFile == true else { continue } guard url.pathExtension == "swift" else { continue } guard !url.lastPathComponent.hasPrefix("_") else { continue } @@ -243,7 +329,11 @@ private struct ProcessResult { let stderr: String } -private func processFile(inputPath: String, libPath: String) throws -> ProcessResult { +private func processFile( + inputPath: String, + libPath: String, + helpers: CompiledHelpers? +) throws -> ProcessResult { let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL let absoluteInputPath = inputURL.path let source = try String(contentsOf: inputURL, encoding: .utf8) @@ -257,7 +347,7 @@ private func processFile(inputPath: String, libPath: String) throws -> ProcessRe let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - let raw = try runSwift(wrappedPath: wrappedURL.path, libPath: libPath) + let raw = try runSwift(wrappedPath: wrappedURL.path, libPath: libPath, helpers: helpers) // #sourceLocation maps body diagnostics back to the input file. Errors in // the preamble (lines outside the body) still reference the wrapper — // rewrite literal occurrences of its path so users see something coherent. @@ -270,6 +360,12 @@ private func processFile(inputPath: String, libPath: String) throws -> ProcessRe // MARK: - Arg parsing +internal enum HelpersOptions { + case auto + case disabled + case explicit(String) +} + private struct CLIArgs { enum Mode { case singleFile(input: String, output: String?) @@ -278,11 +374,13 @@ private struct CLIArgs { let mode: Mode let libPath: String? + let helpers: HelpersOptions static func parse(_ argv: [String]) throws -> CLIArgs { var inputPath: String? var outputPath: String? var libPath: String? + var helpers: HelpersOptions = .auto var i = 1 while i < argv.count { @@ -296,6 +394,13 @@ private struct CLIArgs { guard i + 1 < argv.count else { throw usage("--lib requires a value") } libPath = argv[i + 1] i += 2 + case "--helpers": + guard i + 1 < argv.count else { throw usage("--helpers requires a value") } + helpers = .explicit(argv[i + 1]) + i += 2 + case "--no-helpers": + helpers = .disabled + i += 1 case "-h", "--help": FileHandle.standardError.write(Data(helpText.utf8)) exit(0) @@ -325,7 +430,7 @@ private struct CLIArgs { mode = .singleFile(input: inputPath, output: outputPath) } - return CLIArgs(mode: mode, libPath: libPath) + return CLIArgs(mode: mode, libPath: libPath, helpers: helpers) } } @@ -349,13 +454,18 @@ private let helpText = """ then /lib/, then /../lib/skitrun/. Build a self-contained bundle with Docs/research/poc-step4-release.sh. + --helpers Override Helpers/ directory location. By default, + skitrun walks up from the input looking for one. + Compiled into libSyntaxKitHelpers.dylib and made + importable via `import SyntaxKitHelpers`. + --no-helpers Skip helpers discovery entirely. """ private func usage(_ message: String) -> CLIError { CLIError(message: "\(message)\n\n\(helpText)\n") } -private struct CLIError: Error, CustomStringConvertible { +internal struct CLIError: Error, CustomStringConvertible { let message: String var description: String { message } } @@ -379,7 +489,8 @@ internal func wrap(source: String, originalPath: String) -> String { for item in tree.statements { if let importDecl = item.item.as(ImportDeclSyntax.self), - firstBodyByte == nil { + firstBodyByte == nil + { hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) continue } @@ -402,7 +513,8 @@ internal func wrap(source: String, originalPath: String) -> String { // #sourceLocation must use a forward-slash path; escape backslashes/quotes // defensively even though macOS paths shouldn't contain them. - let escapedPath = originalPath + let escapedPath = + originalPath .replacingOccurrences(of: "\\", with: "\\\\") .replacingOccurrences(of: "\"", with: "\\\"") @@ -421,12 +533,14 @@ internal func wrap(source: String, originalPath: String) -> String { // MARK: - Spawning swift -private func runSwift(wrappedPath: String, libPath: String) throws -> ProcessResult { +private func runSwift( + wrappedPath: String, + libPath: String, + helpers: CompiledHelpers? +) throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/env") - process.arguments = [ + var arguments: [String] = [ "swift", "-suppress-warnings", "-I", libPath, @@ -434,9 +548,24 @@ private func runSwift(wrappedPath: String, libPath: String) throws -> ProcessRes "-lSyntaxKit", "-Xcc", "-I", "-Xcc", cShimsInclude, "-Xlinker", "-rpath", "-Xlinker", libPath, - wrappedPath ] + if let helpers { + let helpersPath = helpers.outputDir.path + arguments.append(contentsOf: [ + "-I", helpersPath, + "-L", helpersPath, + "-l\(helpersModuleName)", + "-Xlinker", "-rpath", "-Xlinker", helpersPath, + ]) + } + + arguments.append(wrappedPath) + + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/env") + process.arguments = arguments + let stdoutPipe = Pipe() let stderrPipe = Pipe() process.standardOutput = stdoutPipe From 2adc3cd4ada399ede9140c46200bcd11ee6e5305 Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 08:46:04 -0400 Subject: [PATCH 09/56] POC step 6: rendered-output cache + --no-cache (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit skitrun now hashes (input bytes + helpers cache key + swift --version + libSyntaxKit stamp + sorted SKITRUN_*/SYNTAXKIT_* env vars) into a SHA-256 key under ~/Library/Caches/com.brightdigit.SyntaxKit/outputs/. On hit, processFile short-circuits — no temp wrapper, no swift spawn, just return the cached bytes. On miss the normal compile path runs and stores the rendered output. Only exit-0 runs are cached so failures always re-spawn with fresh diagnostics. Atomic write through a tmp../ staging dir + rename mirrors the helpers cache; concurrent writers race safely, the loser drops their copy when the destination already exists. --no-cache disables the lookup wholesale (debugging, after manual cache deletion). Threads through runSingleFile + runDirectory so folder mode can opt out batch-wide. Helpers.swift's syntaxKitCacheRoot/captureSwiftVersion/libStamp are now internal so OutputCache.swift can reuse them without duplication. Verified via Docs/research/poc-step6.sh: cold 0.55s → warm 0.14s (4× faster, no swift spawn), --no-cache 0.27s, post-mutation miss 0.41s then 0.14s on the new key, two cache entries persist. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step6-results.md | 58 +++++++++++++++ Docs/research/poc-step6.sh | 116 +++++++++++++++++++++++++++++ Sources/skitrun/Helpers.swift | 6 +- Sources/skitrun/Main.swift | 65 +++++++++++++--- Sources/skitrun/OutputCache.swift | 111 +++++++++++++++++++++++++++ 5 files changed, 342 insertions(+), 14 deletions(-) create mode 100644 Docs/research/poc-step6-results.md create mode 100755 Docs/research/poc-step6.sh create mode 100644 Sources/skitrun/OutputCache.swift diff --git a/Docs/research/poc-step6-results.md b/Docs/research/poc-step6-results.md new file mode 100644 index 0000000..f50fc85 --- /dev/null +++ b/Docs/research/poc-step6-results.md @@ -0,0 +1,58 @@ +# POC Step 6 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §5 + §6 step 6. Goal: skip the `swift` spawn entirely when the rendered output for an input is already cached, with a key that captures everything that could change the output. + +## What landed + +1. **`Sources/skitrun/OutputCache.swift`.** Three functions: + - `outputCacheKey(inputSource:helpers:libPath:)` — SHA-256 over cache schema version + input bytes + helpers cache key (or `"no-helpers"`) + `swift --version` + libSyntaxKit dylib stamp + sorted `SKITRUN_*` / `SYNTAXKIT_*` env vars. + - `lookupCachedOutput(key:)` — returns the cached `output.swift` bytes from `~/Library/Caches/com.brightdigit.SyntaxKit/outputs//output.swift`, or `nil` on miss. + - `storeCachedOutput(key:data:)` — atomic write via `tmp../` staging dir + rename. Concurrent writers race safely; the loser drops their copy if the destination already exists. + +2. **`Sources/skitrun/Main.swift` wiring.** + - `processFile` reads the input source, computes the cache key, and short-circuits on hit — no temp wrapper, no `swift` spawn, just return the cached bytes with `exitCode = 0`. + - On miss, the normal wrap + spawn path runs; if `swift` returns 0, the rendered output is stored under the key. + - Only successful (exit 0) runs are cached. Failed runs always re-spawn so the user sees fresh diagnostics. + - `CLIArgs` gains `--no-cache` to skip the cache entirely (still useful when chasing flaky output or after manually deleting the cache). + - The flag threads through `runSingleFile` and `runDirectory` so folder mode can opt out wholesale. + +3. **Helpers.swift internal exposure.** `syntaxKitCacheRoot`, `captureSwiftVersion`, and `libStamp` were `private`; they're now `internal` so OutputCache can reuse them rather than duplicating. + +## Verified flows + +From `Docs/research/poc-step6.sh` (single input, no helpers): + +| Run | Real time | Notes | +| --- | ---: | --- | +| Cold (cache cleared) | 0.55s | swift spawn + compile + store | +| Warm (cache hit) | 0.14s | FS read only, no swift spawn | +| `--no-cache` | 0.27s | always spawn, ignore cache | +| After mutation (miss) | 0.41s | new key, recompile, store | +| Warm after mutation | 0.14s | second key cached | + +After mutation, the cache directory contains **two** entries (one per input version) — old keys aren't evicted, which is fine for a per-toolchain cache where stale entries are dead weight, not correctness risks. Eviction can be a follow-up if cache size becomes a complaint. + +## Cache key, written out + +``` +SHA-256( + "v1" // cache schema version + + input.swift bytes // verbatim user input + + helpersCacheKey || "no-helpers" // helpers fingerprint (sibling cache) + + swift --version stdout // toolchain fingerprint + + "/" of libSyntaxKit.dylib // SyntaxKit fingerprint proxy + + sorted SKITRUN_*/SYNTAXKIT_* env (k=v\0) // env override sensitivity +) +``` + +Two cooperating cache layers — helpers (step 5) and outputs (step 6) — sit side-by-side under `~/Library/Caches/com.brightdigit.SyntaxKit/{helpers,outputs}//`. Helpers cache hits are reused across many inputs; output cache hits are per-input. + +## Known rough edges + +- **Output cache stores stdout only.** Stderr from a successful run (e.g. warnings even with `-suppress-warnings` off) is discarded on a hit. With `-suppress-warnings` in `runSwift` this is rarely visible, but it does mean cache hits suppress warnings that would have appeared on a fresh run. Acceptable for a generator; revisit if SyntaxKit grows runtime-side warnings. +- **`libStamp` is a coarse proxy.** Size + mtime catches normal rebuilds but a deterministic rebuild that preserves both would slip past. Hashing the dylib is correct but slow (9.3 MB per invocation defeats the cache). The right long-term fix is embedding a SyntaxKit version constant the bundle exports. +- **No size cap or eviction.** A repo that touches inputs frequently will accrete cache entries. Each is small (a few hundred bytes of rendered Swift) so the practical ceiling is high, but a `--prune` subcommand is a reasonable v1.1 addition. + +## What's next + +Step 7: **Linux smoke test.** Confirm `/usr/bin/env swift` + the bundled-dylib layout works on Linux without `-F` framework search paths — `-I + -L + -lSyntaxKit` should be sufficient per Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch. CryptoKit is macOS-only; Linux will need `swift-crypto` or a small fallback hash impl behind a `#if canImport(CryptoKit)` shim. After that, the 7-step ladder is complete. diff --git a/Docs/research/poc-step6.sh b/Docs/research/poc-step6.sh new file mode 100755 index 0000000..ad90c70 --- /dev/null +++ b/Docs/research/poc-step6.sh @@ -0,0 +1,116 @@ +#!/usr/bin/env bash +# +# POC step 6 demo: rendered-output cache. skitrun skips the `swift` spawn +# when input + helpers + toolchain are unchanged. +# +# Builds skitrun, stages a runtime lib next to it, runs an input, then +# replays it three ways: +# 1. with cache — should be near-instant (no swift spawn) +# 2. --no-cache — always spawns swift +# 3. mutated input — invalidates the cache key, falls back to swift + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "macOS-only. Linux smoke test is POC step 7." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +POC_DIR="/tmp/syntaxkit-poc-step6" +DEMO_DIR="$POC_DIR/demo" +OUTPUT_CACHE="$HOME/Library/Caches/com.brightdigit.SyntaxKit/outputs" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old, new, 1)) +PY + +cd "$REPO_ROOT" + +echo "==> swift build" +swift build + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate .build//debug" >&2 + exit 1 +fi + +echo "==> Staging $POC_DIR" +rm -rf "$POC_DIR" +mkdir -p "$POC_DIR/lib" "$DEMO_DIR" + +cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" +cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$POC_DIR/lib/_SwiftSyntaxCShims-include" + +cat > "$DEMO_DIR/Input.swift" <<'SWIFT' +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} +SWIFT + +echo "==> Clearing output cache to force a cold run" +rm -rf "$OUTPUT_CACHE" + +echo +echo "==> Cold run (cache miss → swift spawn → store):" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Warm run (cache hit → no swift spawn):" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> --no-cache (always spawn swift, even with cache present):" +/usr/bin/time -p "$POC_DIR/skitrun" --no-cache "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Output cache contents:" +find "$OUTPUT_CACHE" -maxdepth 3 -type f | sed "s|$OUTPUT_CACHE||" | sort + +echo +echo "==> Mutating input invalidates the cache:" +cat > "$DEMO_DIR/Input.swift" <<'SWIFT' +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") + Variable(.let, name: "email", type: "String") +} +SWIFT +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Warm run after mutation:" +/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Cache now contains two distinct keys:" +find "$OUTPUT_CACHE" -maxdepth 1 -mindepth 1 -type d | wc -l | xargs -I {} echo " {} cache entries" + +echo +echo "==> Done. Cache at $OUTPUT_CACHE." diff --git a/Sources/skitrun/Helpers.swift b/Sources/skitrun/Helpers.swift index dfccba3..d8c3d7f 100644 --- a/Sources/skitrun/Helpers.swift +++ b/Sources/skitrun/Helpers.swift @@ -212,7 +212,7 @@ private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { return hasher.finalize().map { String(format: "%02x", $0) }.joined() } -private func captureSwiftVersion() -> String? { +internal func captureSwiftVersion() -> String? { let process = Process() process.executableURL = URL(fileURLWithPath: "/usr/bin/env") process.arguments = ["swift", "--version"] @@ -225,7 +225,7 @@ private func captureSwiftVersion() -> String? { return String(decoding: data, as: UTF8.self) } -private func libStamp(libPath: String) -> String? { +internal func libStamp(libPath: String) -> String? { let dylib = "\(libPath)/libSyntaxKit.dylib" guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 @@ -233,7 +233,7 @@ private func libStamp(libPath: String) -> String? { return "\(size)/\(Int(mtime))" } -private func syntaxKitCacheRoot() throws -> URL { +internal func syntaxKitCacheRoot() throws -> URL { if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") } diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 52cd1ed..9d8baf0 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -55,7 +55,8 @@ internal enum SkitRun { inputPath: input, outputPath: output, libPath: libPath, - helpers: helpers + helpers: helpers, + useCache: args.useCache ) case .directory(let inputDir, let outputDir): let helpers = try resolveHelpers( @@ -67,7 +68,8 @@ internal enum SkitRun { inputDir: inputDir, outputDir: outputDir, libPath: libPath, - helpers: helpers + helpers: helpers, + useCache: args.useCache ) exit(exitCode) } @@ -166,9 +168,15 @@ private func runSingleFile( inputPath: String, outputPath: String?, libPath: String, - helpers: CompiledHelpers? + helpers: CompiledHelpers?, + useCache: Bool ) throws { - let result = try processFile(inputPath: inputPath, libPath: libPath, helpers: helpers) + let result = try processFile( + inputPath: inputPath, + libPath: libPath, + helpers: helpers, + useCache: useCache + ) if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) } @@ -188,7 +196,8 @@ private func runDirectory( inputDir: String, outputDir: String, libPath: String, - helpers: CompiledHelpers? + helpers: CompiledHelpers?, + useCache: Bool ) async -> Int32 { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL @@ -214,12 +223,12 @@ private func runDirectory( await withTaskGroup(of: FileOutcome.self) { group in for _ in 0.. } -private func runOne(_ input: URL, libPath: String, helpers: CompiledHelpers?) -> FileOutcome { +private func runOne( + _ input: URL, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool +) -> FileOutcome { do { - let result = try processFile(inputPath: input.path, libPath: libPath, helpers: helpers) + let result = try processFile( + inputPath: input.path, + libPath: libPath, + helpers: helpers, + useCache: useCache + ) return FileOutcome(input: input, result: .success(result)) } catch { return FileOutcome(input: input, result: .failure(error)) @@ -332,11 +351,21 @@ private struct ProcessResult { private func processFile( inputPath: String, libPath: String, - helpers: CompiledHelpers? + helpers: CompiledHelpers?, + useCache: Bool ) throws -> ProcessResult { let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL let absoluteInputPath = inputURL.path let source = try String(contentsOf: inputURL, encoding: .utf8) + + let cacheKey: String? = + useCache + ? outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) + : nil + if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { + return ProcessResult(exitCode: 0, stdout: cached, stderr: "") + } + let wrapped = wrap(source: source, originalPath: absoluteInputPath) let tmpDir = FileManager.default.temporaryDirectory @@ -355,6 +384,11 @@ private func processFile( of: wrappedURL.path, with: absoluteInputPath ) + + if let cacheKey, raw.exitCode == 0 { + try? storeCachedOutput(key: cacheKey, data: raw.stdout) + } + return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } @@ -375,12 +409,14 @@ private struct CLIArgs { let mode: Mode let libPath: String? let helpers: HelpersOptions + let useCache: Bool static func parse(_ argv: [String]) throws -> CLIArgs { var inputPath: String? var outputPath: String? var libPath: String? var helpers: HelpersOptions = .auto + var useCache = true var i = 1 while i < argv.count { @@ -401,6 +437,9 @@ private struct CLIArgs { case "--no-helpers": helpers = .disabled i += 1 + case "--no-cache": + useCache = false + i += 1 case "-h", "--help": FileHandle.standardError.write(Data(helpText.utf8)) exit(0) @@ -430,7 +469,7 @@ private struct CLIArgs { mode = .singleFile(input: inputPath, output: outputPath) } - return CLIArgs(mode: mode, libPath: libPath, helpers: helpers) + return CLIArgs(mode: mode, libPath: libPath, helpers: helpers, useCache: useCache) } } @@ -459,6 +498,10 @@ private let helpText = """ Compiled into libSyntaxKitHelpers.dylib and made importable via `import SyntaxKitHelpers`. --no-helpers Skip helpers discovery entirely. + --no-cache Skip the rendered-output cache (always run swift). + The cache lives at /outputs// + and is keyed on input bytes, helpers, swift version, + libSyntaxKit stamp, and SKITRUN_*/SYNTAXKIT_* env. """ private func usage(_ message: String) -> CLIError { diff --git a/Sources/skitrun/OutputCache.swift b/Sources/skitrun/OutputCache.swift new file mode 100644 index 0000000..b951bff --- /dev/null +++ b/Sources/skitrun/OutputCache.swift @@ -0,0 +1,111 @@ +// +// OutputCache.swift +// SyntaxKit — skitrun (POC step 6 for issue #154) +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import CryptoKit +import Foundation + +/// Bumped when the output cache layout changes in a way that requires invalidation. +private let outputCacheSchemaVersion = "v1" + +/// SHA-256 over (cache schema, input source bytes, helpers key, swift version, +/// libSyntaxKit stamp, sorted SKITRUN_*/SYNTAXKIT_* env vars). Any change in +/// these inputs produces a fresh key and forces a recompile. +internal func outputCacheKey( + inputSource: String, + helpers: CompiledHelpers?, + libPath: String +) -> String { + var hasher = SHA256() + hasher.update(data: Data(outputCacheSchemaVersion.utf8)) + hasher.update(data: Data(inputSource.utf8)) + + if let helpers { + // Helpers cache dir name *is* the helpers cache key (per Helpers.swift). + hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) + } else { + hasher.update(data: Data("no-helpers".utf8)) + } + + if let version = captureSwiftVersion() { + hasher.update(data: Data(version.utf8)) + } + if let stamp = libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } + + let env = ProcessInfo.processInfo.environment + .filter { $0.key.hasPrefix("SKITRUN_") || $0.key.hasPrefix("SYNTAXKIT_") } + .sorted { $0.key < $1.key } + for (key, value) in env { + hasher.update(data: Data("\(key)=\(value)\0".utf8)) + } + + return hasher.finalize().map { String(format: "%02x", $0) }.joined() +} + +/// Returns the cached rendered output for `key`, or nil on miss. +internal func lookupCachedOutput(key: String) -> Data? { + guard let dir = try? outputCacheDir(for: key) else { return nil } + return try? Data(contentsOf: dir.appendingPathComponent("output.swift")) +} + +/// Atomically stores `data` under `key`. Concurrent writers race via a +/// `tmp../` staging dir + rename; the loser drops their copy. +internal func storeCachedOutput(key: String, data: Data) throws { + let cacheRoot = try outputCacheDir(for: key) + let final = cacheRoot.appendingPathComponent("output.swift") + let fm = FileManager.default + + try fm.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent( + "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + ) + try fm.createDirectory(at: staging, withIntermediateDirectories: true) + try data.write(to: staging.appendingPathComponent("output.swift")) + + do { + try fm.moveItem(at: staging, to: cacheRoot) + } catch { + try? fm.removeItem(at: staging) + if !fm.fileExists(atPath: final.path) { + throw error + } + } +} + +private func outputCacheDir(for key: String) throws -> URL { + try syntaxKitCacheRoot() + .appendingPathComponent("outputs") + .appendingPathComponent(key) +} From 68e49f0af7136abccc9de7c2eb772b477c3da785 Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 08:49:24 -0400 Subject: [PATCH 10/56] Note web-server form as a post-CLI follow-up (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a §7 bullet recording that a long-lived HTTP server variant is on the table after the 7-step CLI ladder finishes — warm `swift` reuse across requests, shared helpers/output caches across tenants, but new isolation/request-shape questions. Captured here so it doesn't get lost between now and step 7. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md index 0056f96..601b4e0 100644 --- a/Docs/research/codegen-cli-design.md +++ b/Docs/research/codegen-cli-design.md @@ -157,3 +157,4 @@ Each step is independently shippable and de-risks the next. - **What if a single input script needs to produce multiple files?** Out of scope for v1. Split into multiple inputs and use folder mode. If demand materializes, we can layer in a `--multi` envelope mode (a script writes a small JSON manifest to stdout, CLI fans out to multiple files) without breaking single-file semantics. - **Sandboxing.** Out of scope for v1. Input scripts are user-owned code in their own repo — running them has the same threat model as running `swift Input.swift` by hand. Revisit if/when this CLI runs untrusted scripts (CI for OSS contributions, etc.). - **Timeout.** Add one. Tuist's omission (Phase 1 §7) is a bug, not a feature. 60s default `Process` wait with `SIGTERM` → 5s grace → `SIGKILL`. Override via `--timeout `. +- **Web-server form.** Out of scope for the CLI POC, but on the table as a follow-up once the 7-step ladder is done. A long-lived server could reuse a warm `swift` interpreter across requests and share the helpers + output caches across tenants — both of which the CLI gives up on every invocation. Open design questions: request shape (raw DSL POST vs. structured), whether helpers are uploaded per-request or baked into the server image, and isolation between requests (the CLI's "run user code in your own repo" threat model doesn't transfer). Revisit after step 7. From 65f21034f7953ec8b6f86b8d258eb0a2b588e22c Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 10:01:25 -0400 Subject: [PATCH 11/56] POC step 7: Linux smoke test + Foundation.Process workarounds (#154) Verifies the 7-step skitrun POC ladder works on Linux. Tests via Docker (swift:6.0-jammy/aarch64): cold 0.73s, warm cache hit 0.26s, --no-cache 0.30s. Rendered output matches macOS exactly. The real find of this step is a Foundation.Process bug: on Linux, `waitUntilExit()` blocks indefinitely on already-exited children even when EOF on the pipe proves the child has terminated. Reproduced deterministically in `captureSwiftVersion`, where the parent reads all 76 bytes of `swift --version` output and the subsequent wait never returns. Workaround applied to all three call sites (captureSwiftVersion, compileHelpers, runSwift): let semaphore = DispatchSemaphore(value: 0) process.terminationHandler = { _ in semaphore.signal() } try process.run() // ... drain pipes ... semaphore.wait() runSwift additionally drains both stdout + stderr pipes concurrently via DispatchGroup + a PipeDataBox class (boxing for Swift 6 strict concurrency without `@unchecked Sendable` on local vars). Sequential reads would deadlock when either pipe (~64 KB) fills before the child exits. Other Linux-portability changes: * Switch `import CryptoKit` to `import Crypto` in Helpers.swift + OutputCache.swift; add swift-crypto 3.0.0 as a skitrun dep so the same SHA256 API works on both platforms. * dylibFilename(forLibrary:) returns libX.dylib on Apple / libX.so on Linux. Threaded through isLibDir, libStamp, the helpers cache hit check, and `swiftc -emit-library -o ...`. * `-Xlinker -install_name @rpath/...` is Mach-O specific. Wrapped in #if !os(Linux). `-Xlinker -rpath` works on both and is what actually locates the dylib at runtime. Demo script `Docs/research/poc-step7.sh` self-reruns inside the swift container when invoked from the host; uses .build-linux/ as a separate build path so the host's .build/ stays untouched (gitignored). Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 1 + Docs/research/poc-step7-results.md | 45 ++++++++++ Docs/research/poc-step7.sh | 135 +++++++++++++++++++++++++++++ Package.resolved | 20 ++++- Package.swift | 6 +- Sources/skitrun/Helpers.swift | 45 ++++++++-- Sources/skitrun/Main.swift | 38 ++++++-- Sources/skitrun/OutputCache.swift | 2 +- 8 files changed, 274 insertions(+), 18 deletions(-) create mode 100644 Docs/research/poc-step7-results.md create mode 100755 Docs/research/poc-step7.sh diff --git a/.gitignore b/.gitignore index 1e2c5c4..1cee559 100644 --- a/.gitignore +++ b/.gitignore @@ -78,6 +78,7 @@ playground.xcworkspace .swiftpm .build/ +.build-linux/ # CocoaPods # We recommend against adding the Pods directory to your .gitignore. However diff --git a/Docs/research/poc-step7-results.md b/Docs/research/poc-step7-results.md new file mode 100644 index 0000000..a9a40fe --- /dev/null +++ b/Docs/research/poc-step7-results.md @@ -0,0 +1,45 @@ +# POC Step 7 — Results + +> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 7. Goal: confirm the `skitrun` flow that worked on macOS (steps 1-6) also works on Linux with the bundled-dylib layout, no framework search paths, and no Apple-only crypto. + +## What landed + +1. **`swift-crypto` swap-in.** `Sources/skitrun/Helpers.swift` and `OutputCache.swift` now `import Crypto` instead of `CryptoKit`. The Apple `swift-crypto` package vends the same `SHA256` API on every platform and re-exports CryptoKit on Apple platforms when available, so there's no `#if` shim needed in skitrun's own code. Added as a Package dependency at `from: "3.0.0"`. + +2. **Platform-aware dylib filename.** A new `dylibFilename(forLibrary:)` helper returns `libX.dylib` on Apple / `libX.so` on Linux. All call sites that hard-coded `libSyntaxKit.dylib` or `libSyntaxKitHelpers.dylib` now go through it: `isLibDir`, `libStamp`, helpers cache-hit check, and the `swiftc -emit-library -o` path. + +3. **Skip `@rpath` install-name on Linux.** `compileHelpers` previously passed `-Xlinker -install_name -Xlinker @rpath/libSyntaxKitHelpers.dylib`. That flag is Mach-O specific; on Linux it errors at link time. Wrapped in `#if !os(Linux)`. The `-Xlinker -rpath` flag still works on both platforms and is what actually locates the dylib at runtime. + +4. **`Docs/research/poc-step7.sh`.** Self-rerunning Docker wrapper: when invoked on the host it re-execs itself inside `swift:6.0-jammy` with the repo bind-mounted; inside the container it flips Package.swift to dynamic, builds with `--build-path .build-linux` (separate from the macOS host's `.build`), stages a `lib/` next to `skitrun`, and runs cold + warm + `--no-cache` against an input that uses helpers. + +## Verified flows + +From `Docs/research/poc-step7.sh` inside `swift:6.0-jammy` (aarch64): + +| Flow | Time | Notes | +| --- | ---: | --- | +| Cold (helpers compile + output cache miss) | 0.73s | swiftc spawns for helpers, swift interprets the wrapped input | +| Warm (output cache hit) | 0.26s | no swift spawn | +| `--no-cache` | 0.30s | swift spawn, helpers reused from cache | + +Rendered output for the demo `Person` struct matches the macOS step 5 output exactly — same SyntaxKit, same generator, no platform-specific quirks in the rendered code. + +## Linux-only surprises + +- **`Process.waitUntilExit()` hangs on already-exited children.** This was the biggest find. Foundation's `Process.waitUntilExit()` on `swift:6.0-jammy/aarch64` blocks indefinitely even when the child has clearly exited (stdout EOF observed, all 76 bytes of `swift --version` already read). Fix applied to all three callers (`captureSwiftVersion`, `compileHelpers`, `runSwift`): set `process.terminationHandler = { _ in semaphore.signal() }` before `run()`, then `semaphore.wait()` instead of `waitUntilExit()`. Took down a 20-minute mystery hang. +- **Stdout/stderr pipe drain order matters.** Linux pipe buffers are ~64 KB; reading sequentially after waiting for child exit deadlocks when the child fills either pipe. `runSwift` now drains both pipes concurrently via `DispatchGroup` + a `PipeDataBox` class (the boxing satisfies Swift 6 strict-concurrency without `@unchecked Sendable` on local vars). `compileHelpers` only needs to drain stderr (stdout goes to `/dev/null`), but drains before the wait for the same reason. +- **`-Xlinker -install_name @rpath/...` is Mach-O specific.** Errors out on Linux's GNU ld. Wrapped in `#if !os(Linux)` in `compileHelpers`. The `-Xlinker -rpath -Xlinker ` flag still works on both platforms and is what actually locates the dylib at runtime. +- **`/usr/bin/time` isn't installed in `swift:6.0-jammy` by default.** Demo script uses the bash builtin `time` for timing, which writes a different format but is portable. +- **`swift build --product skitrun` alone doesn't emit `libSyntaxKit.so`.** The first `poc-step7.sh` draft scoped the build to just the executable, which produced a 60 MB statically-linked binary and no dylib at all. Plain `swift build` (all products) emits both `libSyntaxKit.so` and `skitrun`, matching the macOS steps 5-6 scripts. +- **First-time build cost.** Cold dependency resolution + boringssl C compile (pulled in by `swift-crypto` on Linux where CommonCrypto isn't available) takes ~3 min in `swift:6.0-jammy/aarch64`. Subsequent runs reuse `.build-linux/` and finish in ~40s. +- **Crypto on Linux brings boringssl.** `swift-crypto` statically links boringssl on non-Apple platforms. `skitrun`'s Linux binary is therefore noticeably larger than the macOS one (boringssl C blobs add up). Not a correctness issue, just a size note for future packaging. + +## What's next + +The 7-step POC ladder is **complete**. With this commit: + +- Cold-start cost has been measured on both platforms. +- The bundled-dylib + script-mode `swift` invocation works on macOS and Linux. +- Folder mode, helpers, and the rendered-output cache all behave the same way on both. + +The remaining bullets in [`codegen-cli-design.md` §7](./codegen-cli-design.md#7-what-we-still-need-to-verify) — timeouts, the splice-fidelity audit beyond the demo inputs, `@main`/attribute behavior in script-mode swift, the multi-file output question, and the web-server form — are now the natural next conversation. None of them block productizing the CLI; all of them are scope decisions rather than open technical risks. diff --git a/Docs/research/poc-step7.sh b/Docs/research/poc-step7.sh new file mode 100755 index 0000000..1d6c98e --- /dev/null +++ b/Docs/research/poc-step7.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# +# POC step 7: Linux smoke test for skitrun. +# +# Runs inside a swift:6.0-jammy container. Builds skitrun, stages a +# runtime lib/ next to it (libSyntaxKit.so + Modules + _SwiftSyntaxCShims +# headers), then exercises single-file mode, helpers, and the output +# cache — the same flows POC steps 5 and 6 verified on macOS. +# +# Usage (from macOS host or Linux host with docker): +# Docs/research/poc-step7.sh +# +# Override the image with $SKITRUN_LINUX_IMAGE. +# +# To save time across runs, the script uses .build-linux/ as a separate +# build directory so the host's .build/ stays clean and SwiftSyntax +# doesn't re-download on every invocation. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +IMAGE="${SKITRUN_LINUX_IMAGE:-swift:6.0-jammy}" + +if [[ ! -f /.dockerenv ]]; then + # ---- Host side: invoke ourselves inside the swift container. ---- + if ! command -v docker >/dev/null; then + echo "docker is required for POC step 7" >&2 + exit 1 + fi + echo "==> Running POC step 7 inside $IMAGE" + exec docker run --rm -t \ + -v "$REPO_ROOT:/workspace" \ + -w /workspace \ + -e SKITRUN_INSIDE_DOCKER=1 \ + "$IMAGE" \ + /workspace/Docs/research/poc-step7.sh +fi + +# ---- Container side: do the real work. ---- + +PACKAGE_FILE="Package.swift" +PACKAGE_BACKUP="$(mktemp)" +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> swift --version" +swift --version + +echo +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if old not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old, new, 1)) +PY + +echo +echo "==> swift build (build path: .build-linux)" +swift build --build-path .build-linux + +BUILD_DIR="$(ls -d .build-linux/*-unknown-linux-gnu/debug 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate Linux build dir under .build-linux/" >&2 + ls -la .build-linux/ || true + exit 1 +fi + +POC_DIR=/tmp/syntaxkit-poc-step7 +DEMO_DIR="$POC_DIR/demo" +OUTPUT_CACHE="$HOME/.cache/syntaxkit/outputs" + +rm -rf "$POC_DIR" "$HOME/.cache/syntaxkit" +mkdir -p "$POC_DIR/lib" "$DEMO_DIR/Helpers" + +cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" +cp "$BUILD_DIR/libSyntaxKit.so" "$POC_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" +cp -r .build-linux/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include \ + "$POC_DIR/lib/_SwiftSyntaxCShims-include" + +cat > "$DEMO_DIR/Helpers/Models.swift" <<'SWIFT' +import SyntaxKit + +public func equatableModel( + _ name: String, + fields: [(name: String, type: String)] +) -> any CodeBlock { + Struct(name) { + for field in fields { + Variable(.let, name: field.name, type: field.type) + } + }.inherits("Equatable") +} +SWIFT + +cat > "$DEMO_DIR/Input.swift" <<'SWIFT' +import SyntaxKitHelpers + +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +SWIFT + +echo +echo "==> Cold run (helpers compile + output cache miss):" +time "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" + +echo +echo "==> Warm run (output cache hit, no swift spawn):" +time "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> --no-cache (swift spawn, helpers reused):" +time "$POC_DIR/skitrun" --no-cache "$DEMO_DIR/Input.swift" >/dev/null + +echo +echo "==> Output cache entries:" +find "$OUTPUT_CACHE" -maxdepth 2 -type f 2>/dev/null | sed "s|$OUTPUT_CACHE||" | sort + +echo +echo "==> Linux smoke test passed." diff --git a/Package.resolved b/Package.resolved index 24cfefa..851eeb1 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,6 +1,24 @@ { - "originHash" : "482d43aff5bb5c075d237e0ea17c12ee2c043b2642e459260752aa1848a20593", + "originHash" : "6a75ee274433215501c77ebca768e3c82c684598296596b112f9800ac08fa2fe", "pins" : [ + { + "identity" : "swift-asn1", + "kind" : "remoteSourceControl", + "location" : "https://github.com/apple/swift-asn1.git", + "state" : { + "revision" : "9f542610331815e29cc3821d3b6f488db8715517", + "version" : "1.6.0" + } + }, + { + "identity" : "swift-crypto", + "kind" : "remoteSourceControl", + "location" : "https://github.com/apple/swift-crypto.git", + "state" : { + "revision" : "95ba0316a9b733e92bb6b071255ff46263bbe7dc", + "version" : "3.15.1" + } + }, { "identity" : "swift-docc-plugin", "kind" : "remoteSourceControl", diff --git a/Package.swift b/Package.swift index 1087718..13c44a1 100644 --- a/Package.swift +++ b/Package.swift @@ -102,7 +102,8 @@ let package = Package( ], dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), - .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0") + .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), + .package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0") ], targets: [ .target( @@ -152,7 +153,8 @@ let package = Package( name: "skitrun", dependencies: [ .product(name: "SwiftSyntax", package: "swift-syntax"), - .product(name: "SwiftParser", package: "swift-syntax") + .product(name: "SwiftParser", package: "swift-syntax"), + .product(name: "Crypto", package: "swift-crypto") ], swiftSettings: swiftSettings ), diff --git a/Sources/skitrun/Helpers.swift b/Sources/skitrun/Helpers.swift index d8c3d7f..6a87f88 100644 --- a/Sources/skitrun/Helpers.swift +++ b/Sources/skitrun/Helpers.swift @@ -27,13 +27,22 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import CryptoKit +import Crypto import Foundation /// Hardcoded module name for the user's `Helpers/` compilation output. Inputs /// reach the compiled helpers via `import SyntaxKitHelpers`. internal let helpersModuleName = "SyntaxKitHelpers" +/// Platform-specific shared-library filename for a Swift library product. +internal func dylibFilename(forLibrary name: String) -> String { + #if os(Linux) + return "lib\(name).so" + #else + return "lib\(name).dylib" + #endif +} + /// Bumped when the cache layout changes in a way that requires invalidation. private let helpersCacheSchemaVersion = "v1" @@ -109,7 +118,8 @@ internal func buildHelpers( let cacheRoot = try syntaxKitCacheRoot() .appendingPathComponent("helpers") .appendingPathComponent(key) - let dylibPath = cacheRoot.appendingPathComponent("lib\(helpersModuleName).dylib").path + let dylibPath = cacheRoot + .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path let fm = FileManager.default if fm.fileExists(atPath: dylibPath) { @@ -148,7 +158,7 @@ internal func buildHelpers( private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) throws { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - let dylib = outDir.appendingPathComponent("lib\(helpersModuleName).dylib").path + let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path let process = Process() @@ -166,20 +176,37 @@ private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) t "-L", libPath, "-lSyntaxKit", "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-install_name", - "-Xlinker", "@rpath/lib\(helpersModuleName).dylib", "-Xlinker", "-rpath", "-Xlinker", libPath, ] + #if !os(Linux) + // @rpath install_name is macOS-only; on Linux SONAME isn't needed because + // we use rpath-based loading and the dylib lives in a cache path that's + // known at link time. + args.append(contentsOf: [ + "-Xlinker", "-install_name", + "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", + ]) + #endif args.append(contentsOf: sources.map(\.path)) process.arguments = args let stderrPipe = Pipe() process.standardOutput = FileHandle.nullDevice process.standardError = stderrPipe + + // Linux Foundation's `Process.waitUntilExit()` blocks indefinitely on + // already-exited children in some configurations; terminationHandler + + // semaphore is the workaround used elsewhere in this file. + let semaphore = DispatchSemaphore(value: 0) + process.terminationHandler = { _ in semaphore.signal() } + try process.run() - process.waitUntilExit() + // Drain stderr BEFORE waiting on the semaphore — Linux pipe buffers are + // ~64 KB; if the child fills them we deadlock waiting for an exit that + // can't happen until we read. let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() + semaphore.wait() guard process.terminationStatus == 0 else { let stderr = String(decoding: stderrData, as: UTF8.self) throw CLIError( @@ -219,14 +246,16 @@ internal func captureSwiftVersion() -> String? { let pipe = Pipe() process.standardOutput = pipe process.standardError = FileHandle.nullDevice + let semaphore = DispatchSemaphore(value: 0) + process.terminationHandler = { _ in semaphore.signal() } do { try process.run() } catch { return nil } - process.waitUntilExit() let data = pipe.fileHandleForReading.readDataToEndOfFile() + semaphore.wait() return String(decoding: data, as: UTF8.self) } internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/libSyntaxKit.dylib" + let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 9d8baf0..f4c1dc0 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -159,7 +159,7 @@ private func isLibDir(_ path: String) -> Bool { let fm = FileManager.default var isDir: ObjCBool = false guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } - return fm.fileExists(atPath: "\(path)/libSyntaxKit.dylib") + return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") } // MARK: - Single-file mode @@ -614,15 +614,41 @@ private func runSwift( process.standardOutput = stdoutPipe process.standardError = stderrPipe + // Linux Foundation's `Process.waitUntilExit()` blocks indefinitely on + // already-exited children in some configurations; terminationHandler + + // semaphore is the workaround. + let exitSemaphore = DispatchSemaphore(value: 0) + process.terminationHandler = { _ in exitSemaphore.signal() } + try process.run() - process.waitUntilExit() - let stdoutData = stdoutPipe.fileHandleForReading.readDataToEndOfFile() - let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() + // Drain both pipes concurrently — reading sequentially deadlocks on Linux + // when either pipe (~64 KB buffer) fills before the child exits. Box the + // buffers in classes so Swift 6 strict-concurrency is satisfied without + // `@unchecked Sendable` on local vars. + let outBox = PipeDataBox() + let errBox = PipeDataBox() + let group = DispatchGroup() + group.enter() + DispatchQueue.global().async { + outBox.value = stdoutPipe.fileHandleForReading.readDataToEndOfFile() + group.leave() + } + group.enter() + DispatchQueue.global().async { + errBox.value = stderrPipe.fileHandleForReading.readDataToEndOfFile() + group.leave() + } + group.wait() + exitSemaphore.wait() return ProcessResult( exitCode: process.terminationStatus, - stdout: stdoutData, - stderr: String(decoding: stderrData, as: UTF8.self) + stdout: outBox.value, + stderr: String(decoding: errBox.value, as: UTF8.self) ) } + +private final class PipeDataBox: @unchecked Sendable { + var value = Data() +} diff --git a/Sources/skitrun/OutputCache.swift b/Sources/skitrun/OutputCache.swift index b951bff..3f094b0 100644 --- a/Sources/skitrun/OutputCache.swift +++ b/Sources/skitrun/OutputCache.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import CryptoKit +import Crypto import Foundation /// Bumped when the output cache layout changes in a way that requires invalidation. From 2836cf75eb41237afc5f781f39d3ae78b6c558d1 Mon Sep 17 00:00:00 2001 From: leogdion Date: Tue, 12 May 2026 12:46:03 -0400 Subject: [PATCH 12/56] Add skitrun README scoped to the target dir (#154) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lives at Sources/skitrun/README.md (per user's "may be temp" note — keep it local to the target, not the repo front door). Covers usage, the portable bundle, input shape, helpers, caches, the flag table, platform notes (including the Linux waitUntilExit gotcha), and the open scope decisions from codegen-cli-design.md §7. Cross-links into Docs/research/ so the design + per-step results are one click away. Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skitrun/README.md | 103 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 Sources/skitrun/README.md diff --git a/Sources/skitrun/README.md b/Sources/skitrun/README.md new file mode 100644 index 0000000..c8c1a20 --- /dev/null +++ b/Sources/skitrun/README.md @@ -0,0 +1,103 @@ +# skitrun + +> **Status:** research POC for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). Shape may change; do not pin tooling to it yet. Design lives at [`Docs/research/codegen-cli-design.md`](../../Docs/research/codegen-cli-design.md); the 7-step build-up is documented step-by-step under [`Docs/research/poc-step{1..7}-results.md`](../../Docs/research/). + +A CLI that takes a *pure SyntaxKit DSL* input file, wraps it in a `Group { … }` closure, spawns `swift` to evaluate it, and writes the rendered Swift source to stdout (or a file). No `print`, no `@main`, no boilerplate in your input — just DSL expressions. + +``` +skitrun Input.swift # render to stdout +skitrun Input.swift -o Out.swift # render to a file +skitrun InputDir/ -o OutDir/ # walk **/*.swift, mirror to OutDir/ +``` + +## Quick start + +```bash +# Build a portable bundle (the script flips the SyntaxKit library to +# .dynamic, then bundles dylib + modules + C-shims headers next to skitrun). +Docs/research/poc-step4-release.sh +# → .build/skitrun-release/{skitrun, lib/} + +cat > /tmp/Person.swift <<'SWIFT' +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} +SWIFT + +.build/skitrun-release/skitrun /tmp/Person.swift +``` + +The bundle is self-contained: `cp -r .build/skitrun-release ~/anywhere/` and `~/anywhere/skitrun-release/skitrun ` works zero-config. + +## Input file shape + +Top-level expressions form an implicit `@CodeBlockBuilder` body. `import` declarations at the top are hoisted into the wrapper. Anything else (`Struct(…)`, `Enum(…)`, helper calls, …) becomes the builder's content. + +```swift +// Models.swift +import SyntaxKitHelpers // optional — only if a Helpers/ dir is present + +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +``` + +What *won't* work inside the input: top-level `let`/`var` outside the builder DSL, `@main`, `print` (the wrapper adds its own). Result-builder closure rules apply. + +## Helpers + +Shared codegen utilities live in a `Helpers/` directory anywhere up-tree from the input. `skitrun` walks up from the input file (or directory) looking for one. Sources are pre-compiled into `libSyntaxKitHelpers.{dylib,so}` once and cached by content hash: + +``` +project/ +├── Helpers/ +│ └── Models.swift # public func equatableModel(_:fields:) -> any CodeBlock +└── inputs/ + ├── Person.swift # imports SyntaxKitHelpers, calls equatableModel(...) + └── Pet.swift # same +``` + +Files prefixed with `_` are skipped (convention for private helpers within helpers). The helper module name is hard-coded to `SyntaxKitHelpers`. + +Force-disable: `--no-helpers`. Override location: `--helpers `. + +## Caches + +Two layers, both keyed on content + toolchain + dylib stamp + `SKITRUN_*`/`SYNTAXKIT_*` env vars. Live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. + +| Layer | Path | What it skips on hit | +| --- | --- | --- | +| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | +| Output | `outputs//output.swift` | the `swift` spawn for an input | + +Output cache hit ≈ 0.14s on macOS (no spawn at all); cold miss matches the warm `swift` script-mode baseline (~0.5s). Force a miss with `--no-cache`. + +## Flag reference + +| Flag | Default | Meaning | +| --- | --- | --- | +| `-o, --output ` | stdout | Output file (single-file mode) or directory (folder mode). | +| `--lib ` | auto | Directory containing `libSyntaxKit.{dylib,so}` + module files. Search order when omitted: `$SKITRUN_LIB_DIR` → `/lib/` → `/../lib/skitrun/`. | +| `--helpers ` | walk-up | Explicit `Helpers/` directory. | +| `--no-helpers` | (off) | Skip helpers discovery entirely. | +| `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | + +## Platform notes + +- **macOS:** primary target. All seven POC steps run via the scripts in `Docs/research/`. +- **Linux:** verified in `swift:6.0-jammy/aarch64` via [`Docs/research/poc-step7.sh`](../../Docs/research/poc-step7.sh) (self-reruns inside Docker). Requires `swift-crypto` instead of CryptoKit; install-name flag is Mach-O specific and skipped on Linux. +- **Windows:** not attempted. + +A known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. Workaround in `Helpers.swift` / `Main.swift`: `terminationHandler` + `DispatchSemaphore`. See [`poc-step7-results.md`](../../Docs/research/poc-step7-results.md) for the full reproducer. + +## Open scope decisions + +Not blocking but on the table — see [`codegen-cli-design.md` §7](../../Docs/research/codegen-cli-design.md#7-what-we-still-need-to-verify): + +- Timeouts on the child `swift` process (60s default + SIGTERM/SIGKILL grace). +- `@main` / attribute behavior in `swift` script-mode beyond the simple cases tested. +- Multi-file outputs from a single input (out of scope for v1). +- Sandboxing (out of scope; threat model = "you ran your own code"). +- HTTP/server form for warm-interpreter reuse (post-CLI follow-up). From c6d9e7dd500308f00f1382700e398b4510897091 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 13:31:46 -0400 Subject: [PATCH 13/56] Fix skitrun release bundle and update blackjack example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - poc-step4-release.sh: explicitly build the SyntaxKit product. skitrun doesn't depend on SyntaxKit (it spawns swift on user input that does), so --product skitrun alone never produced libSyntaxKit.dylib and the bundle ended up with whatever swiftmodule happened to be cached — which surfaced as a 6.3 vs 6.3.2 module-version mismatch after a toolchain bump. - Examples/Completed/blackjack/dsl.swift: update to the current DSL API (ComputedProperty now requires type:, Init's builder takes ParameterExp, VariableDecl → Variable), escape \(…) interpolations so the literal appears in the generated code, and drop the let/print wrapper so the file is a top-level CodeBlock expression compatible with skitrun. - .vscode/launch.json: add Debug/Release launch configs for skitrun. Co-Authored-By: Claude Opus 4.7 (1M context) --- .vscode/launch.json | 20 ++++++++++++++++ Docs/research/poc-step4-release.sh | 6 +++++ Examples/Completed/blackjack/dsl.swift | 33 ++++++++++++-------------- 3 files changed, 41 insertions(+), 18 deletions(-) diff --git a/.vscode/launch.json b/.vscode/launch.json index fefdbc0..1c2682d 100644 --- a/.vscode/launch.json +++ b/.vscode/launch.json @@ -19,6 +19,26 @@ "preLaunchTask": "swift: Build Release skit", "target": "skit", "configuration": "release" + }, + { + "type": "swift", + "request": "launch", + "args": [], + "cwd": "${workspaceFolder:SyntaxKit}", + "name": "Debug skitrun", + "target": "skitrun", + "configuration": "debug", + "preLaunchTask": "swift: Build Debug skitrun" + }, + { + "type": "swift", + "request": "launch", + "args": [], + "cwd": "${workspaceFolder:SyntaxKit}", + "name": "Release skitrun", + "target": "skitrun", + "configuration": "release", + "preLaunchTask": "swift: Build Release skitrun" } ] } diff --git a/Docs/research/poc-step4-release.sh b/Docs/research/poc-step4-release.sh index 36b2035..2ff3e9c 100755 --- a/Docs/research/poc-step4-release.sh +++ b/Docs/research/poc-step4-release.sh @@ -53,6 +53,12 @@ cd "$REPO_ROOT" echo "==> swift build -c release --product skitrun" swift build -c release --product skitrun +# skitrun doesn't depend on SyntaxKit (it spawns swift on user input that +# imports SyntaxKit at runtime). Build the library product explicitly so the +# .dynamic flip above produces libSyntaxKit.dylib + swiftmodule. +echo "==> swift build -c release --product SyntaxKit" +swift build -c release --product SyntaxKit + BUILD_DIR="$(ls -d .build/*-apple-macosx*/release 2>/dev/null | head -1)" if [[ -z "$BUILD_DIR" ]]; then echo "Could not locate release build dir under .build//release" >&2 diff --git a/Examples/Completed/blackjack/dsl.swift b/Examples/Completed/blackjack/dsl.swift index d0af68d..f894b51 100644 --- a/Examples/Completed/blackjack/dsl.swift +++ b/Examples/Completed/blackjack/dsl.swift @@ -1,7 +1,7 @@ import SyntaxKit // Example of generating a BlackjackCard struct with a nested Suit enum -let structExample = Struct("BlackjackCard") { +Struct("BlackjackCard") { Enum("Suit") { EnumCase("spades").equals("♠") EnumCase("hearts").equals("♡") @@ -28,29 +28,29 @@ let structExample = Struct("BlackjackCard") { Variable(.let, name: "first", type: "Int") Variable(.let, name: "second", type: "Int?") } - ComputedProperty("values") { + ComputedProperty("values", type: "Values") { Switch("self") { SwitchCase(".ace") { - Return{ + Return { Init("Values") { - Parameter(name: "first", value: "1") - Parameter(name: "second", value: "11") + ParameterExp(name: "first", value: "1") + ParameterExp(name: "second", value: "11") } } } SwitchCase(".jack", ".queen", ".king") { - Return{ + Return { Init("Values") { - Parameter(name: "first", value: "10") - Parameter(name: "second", value: "nil") + ParameterExp(name: "first", value: "10") + ParameterExp(name: "second", value: "nil") } } } Default { - Return{ + Return { Init("Values") { - Parameter(name: "first", value: "self.rawValue") - Parameter(name: "second", value: "nil") + ParameterExp(name: "first", value: "self.rawValue") + ParameterExp(name: "second", value: "nil") } } } @@ -62,17 +62,14 @@ let structExample = Struct("BlackjackCard") { Variable(.let, name: "rank", type: "Rank") Variable(.let, name: "suit", type: "Suit") - ComputedProperty("description") { - VariableDecl(.var, name: "output", equals: "suit is \(suit.rawValue),") - PlusAssign("output", " value is \(rank.values.first)") + ComputedProperty("description", type: "String") { + Variable(.var, name: "output", equals: "suit is \\(suit.rawValue),") + PlusAssign("output", " value is \\(rank.values.first)") If(Let("second", "rank.values.second"), then: { - PlusAssign("output", " or \(second)") + PlusAssign("output", " or \\(second)") }) Return { VariableExp("output") } } } - -// Generate and print the code -print(structExample.generateCode()) From 84b497bf8576620e04e99c29e1ce6a18bbc4cfc4 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 13:43:54 -0400 Subject: [PATCH 14/56] Update Examples/Completed/*/dsl.swift to current API + skitrun shape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings 6 more examples up to a state where skitrun renders them end-to-end (joining blackjack from the previous commit). 7/11 examples now work; the remaining 4 (concurrency, errors_async, macro_tutorial, enum_generator) need deeper rewrites and are left for follow-up. Common patterns applied: - Drop `let = …; print(.generateCode())` wrappers — skitrun wraps the input in `Group { … }` itself and rejects top-level let/print. - `ComputedProperty(_:)` → `ComputedProperty(_:type:)` (type is now required). - `Parameter(name:value:)` inside `Init { … }` → `ParameterExp(name:value:)`. - `Call("fn", "literal")` → `Call("fn") { ParameterExp(unlabeled: Literal.string("literal")) }`. - `Infix("a", "op", v)` → `Infix("op", lhs: VariableExp("a"), rhs: )`. - `Function("f", parameters: […])` → trailing parameter-builder closure form. - `.access("public")` → `.access(.public)` (now takes AccessModifier enum). - `Literal("\"…\"")` → `Literal.string(…)`. - `VariableDecl(.let, …)` → `Variable(.let, …)` (rename). - `Variable(…, defaultValue:)` → `Variable(…, equals:)`. - `.let("x")` pattern shorthand → `Pattern.let("x")`. - `.reference("weak")` → `.reference(.weak)` (now takes CaptureReferenceType). - `.property(name:)` → `.property(_:)` (label removed). - Escape `\(…)` interpolations so the literal appears in the *generated* code rather than being evaluated against non-existent DSL-scope names. Co-Authored-By: Claude Opus 4.7 (1M context) --- Examples/Completed/attributes/dsl.swift | 9 +- Examples/Completed/card_game/dsl.swift | 26 +++-- Examples/Completed/conditionals/dsl.swift | 126 +++++++++++----------- Examples/Completed/for_loops/dsl.swift | 16 +-- Examples/Completed/protocols/dsl.swift | 12 ++- Examples/Completed/swiftui/dsl.swift | 14 +-- 6 files changed, 99 insertions(+), 104 deletions(-) diff --git a/Examples/Completed/attributes/dsl.swift b/Examples/Completed/attributes/dsl.swift index daf5416..e7d453a 100644 --- a/Examples/Completed/attributes/dsl.swift +++ b/Examples/Completed/attributes/dsl.swift @@ -1,7 +1,10 @@ Class("Foo") { - Variable(.var, name: "bar", type: "String", defaultValue: "bar").attribute("Published") + Variable(.var, name: "bar", type: "String", equals: "bar").attribute("Published") Function("bar") { - print("bar") + Call("print") { + ParameterExp(unlabeled: Literal.string("bar")) + } }.attribute("available", arguments: ["iOS 17.0", "*"]) Function("baz") { -}.attribute("objc")}.attribute("objc") \ No newline at end of file + }.attribute("objc") +} diff --git a/Examples/Completed/card_game/dsl.swift b/Examples/Completed/card_game/dsl.swift index f6f9bdd..0374e51 100644 --- a/Examples/Completed/card_game/dsl.swift +++ b/Examples/Completed/card_game/dsl.swift @@ -1,7 +1,7 @@ import SyntaxKit // Example of generating a BlackjackCard struct with a nested Suit enum -let structExample = Group { +Group { Struct("Card") { Variable(.let, name: "rank", type: "Rank") .comment{ @@ -39,31 +39,31 @@ let structExample = Group { Variable(.let, name: "first", type: "Int") Variable(.let, name: "second", type: "Int?") } - ComputedProperty("description") { + ComputedProperty("description", type: "String") { Switch("self") { SwitchCase(".jack") { - Return{ - Literal("\"J\"") + Return { + Literal.string("J") } } SwitchCase(".queen") { - Return{ - Literal("\"Q\"") + Return { + Literal.string("Q") } } SwitchCase(".king") { - Return{ - Literal("\"K\"") + Return { + Literal.string("K") } } SwitchCase(".ace") { - Return{ - Literal("\"A\"") + Return { + Literal.string("A") } } Default { - Return{ - Literal("\\(rawValue)") + Return { + Literal.string("\\(rawValue)") } } } @@ -92,5 +92,3 @@ let structExample = Group { } } -// Generate and print the code -print(structExample.generateCode()) diff --git a/Examples/Completed/conditionals/dsl.swift b/Examples/Completed/conditionals/dsl.swift index 70263e0..ef05702 100644 --- a/Examples/Completed/conditionals/dsl.swift +++ b/Examples/Completed/conditionals/dsl.swift @@ -4,31 +4,31 @@ Group { Line("Simple if statement") } If { - Infix("temperature", ">", 30) + Infix(">", lhs: VariableExp("temperature"), rhs: Literal.integer(30)) } then: { - Call("print", "It's hot outside!") + Call("print") { ParameterExp(unlabeled: Literal.string("It's hot outside!")) } } Variable(.let, name: "score", equals: Literal.integer(85)) .comment { Line("If-else statement") } If { - Infix("score", ">=", 90) + Infix(">=", lhs: VariableExp("score"), rhs: Literal.integer(90)) } then: { - Call("print", "Excellent!") + Call("print") { ParameterExp(unlabeled: Literal.string("Excellent!")) } } else: { If { - Infix("score", ">=", 80) + Infix(">=", lhs: VariableExp("score"), rhs: Literal.integer(80)) } then: { - Call("print", "Good job!") + Call("print") { ParameterExp(unlabeled: Literal.string("Good job!")) } } If { - try Infix("score", ">=", 70) + Infix(">=", lhs: VariableExp("score"), rhs: Literal.integer(70)) } then: { - Call("print", "Passing") + Call("print") { ParameterExp(unlabeled: Literal.string("Passing")) } } Then { - Call("print", "Needs improvement") + Call("print") { ParameterExp(unlabeled: Literal.string("Needs improvement")) } } } @@ -40,9 +40,9 @@ Group { If(Let("actualNumber", Init("Int") { ParameterExp(name: "", value: "possibleNumber") }), then: { - Call("print", "The string \"\\(possibleNumber)\" has an integer value of \\(actualNumber)") + Call("print") { ParameterExp(unlabeled: Literal.string("The string \"\\(possibleNumber)\" has an integer value of \\(actualNumber)")) } }, else: { - Call("print", "The string \"\\(possibleNumber)\" could not be converted to an integer") + Call("print") { ParameterExp(unlabeled: Literal.string("The string \"\\(possibleNumber)\" could not be converted to an integer")) } }) Variable(.let, name: "possibleName", type: "String?", equals: Literal.string("John")).withExplicitType() @@ -54,14 +54,16 @@ Group { Let("name", "possibleName") Let("age", "possibleAge") } then: { - Call("print", "\\(name) is \\(age) years old") + Call("print") { ParameterExp(unlabeled: Literal.string("\\(name) is \\(age) years old")) } } - Function("greet", parameters: [Parameter("person", type: "[String: String]")]) { + Function("greet") { + Parameter(name: "person", type: "[String: String]") + } _: { Guard { Let("name", "person[\"name\"]") } else: { - Call("print", "No name provided") + Call("print") { ParameterExp(unlabeled: Literal.string("No name provided")) } } Guard { Let("age", "person[\"age\"]") @@ -69,9 +71,9 @@ Group { ParameterExp(name: "", value: "age") }) } else: { - Call("print", "Invalid age provided") + Call("print") { ParameterExp(unlabeled: Literal.string("Invalid age provided")) } } - Call("print", "Hello \\(name), you are \\(ageInt) years old") + Call("print") { ParameterExp(unlabeled: Literal.string("Hello \\(name), you are \\(ageInt) years old")) } } }.comment { Line("MARK: - Guard Statements") @@ -104,26 +106,26 @@ Switch("approximateCount") { Assignment("naturalCount", Literal.string("many")) } } -Call("print", "There are \\(naturalCount) \\(countedThings).") +Call("print") { ParameterExp(unlabeled: Literal.string("There are \\(naturalCount) \\(countedThings).")) } Variable(.let, name: "somePoint", type: "(Int, Int)", equals: VariableExp("(1, 1)"), explicitType: true) .comment { Line("Switch with tuple matching") } Switch("somePoint") { SwitchCase(Tuple.pattern([0, 0])) { - Call("print", "(0, 0) is at the origin") + Call("print") { ParameterExp(unlabeled: Literal.string("(0, 0) is at the origin")) } } SwitchCase(Tuple.pattern([nil, 0])) { - Call("print", "(\(somePoint.0), 0) is on the x-axis") + Call("print") { ParameterExp(unlabeled: Literal.string("(\\(somePoint.0), 0) is on the x-axis")) } } SwitchCase(Tuple.pattern([0, nil])) { - Call("print", "(0, \(somePoint.1)) is on the y-axis") + Call("print") { ParameterExp(unlabeled: Literal.string("(0, \\(somePoint.1)) is on the y-axis")) } } SwitchCase(Tuple.pattern([(-2...2), (-2...2)])) { - Call("print", "(\(somePoint.0), \(somePoint.1)) is inside the box") + Call("print") { ParameterExp(unlabeled: Literal.string("(\\(somePoint.0), \\(somePoint.1)) is inside the box")) } } Default { - Call("print", "(\(somePoint.0), \(somePoint.1)) is outside of the box") + Call("print") { ParameterExp(unlabeled: Literal.string("(\\(somePoint.0), \\(somePoint.1)) is outside of the box")) } } } Variable(.let, name: "anotherPoint", type: "(Int, Int)", equals: VariableExp("(2, 0)"), explicitType: true) @@ -131,21 +133,21 @@ Variable(.let, name: "anotherPoint", type: "(Int, Int)", equals: VariableExp("(2 Line("Switch with value binding") } Switch("anotherPoint") { - SwitchCase(Tuple.pattern([.let("x"), 0])) { - Call("print", "on the x-axis with an x value of \(x)") + SwitchCase(Tuple.pattern([Pattern.let("x"), 0])) { + Call("print") { ParameterExp(unlabeled: Literal.string("on the x-axis with an x value of \\(x)")) } } - SwitchCase(Tuple.pattern([0, .let("y")])) { - Call("print", "on the y-axis with a y value of \(y)") + SwitchCase(Tuple.pattern([0, Pattern.let("y")])) { + Call("print") { ParameterExp(unlabeled: Literal.string("on the y-axis with a y value of \\(y)")) } } - SwitchCase(Tuple.pattern([.let("x"), .let("y")])) { - Call("print", "somewhere else at (\(x), \(y))") + SwitchCase(Tuple.pattern([Pattern.let("x"), Pattern.let("y")])) { + Call("print") { ParameterExp(unlabeled: Literal.string("somewhere else at (\\(x), \\(y))")) } } } Variable(.let, name: "integerToDescribe", equals: 5) -Variable(.var, name: "description", equals: "The number \(integerToDescribe) is") +Variable(.var, name: "description", equals: "The number \\(integerToDescribe) is") Switch("integerToDescribe") { SwitchCase(2, 3, 5, 7, 11, 13, 17, 19) { PlusAssign("description", "a prime number, and also") @@ -155,68 +157,62 @@ Switch("integerToDescribe") { PlusAssign("description", "an integer.") } } -Call("print", "description") +Call("print") { ParameterExp(unlabeled: Literal.string("description")) } Variable(.let, name: "finalSquare", equals: 25) Variable(.var, name: "board", equals: Literal.array(Array(repeating: Literal.integer(0), count: 26))) -Infix("board[03]", "+=", 8) -Infix("board[06]", "+=", 11) -Infix("board[09]", "+=", 9) -Infix("board[10]", "+=", 2) -Infix("board[14]", "-=", 10) -Infix("board[19]", "-=", 11) -Infix("board[22]", "-=", 2) -Infix("board[24]", "-=", 8) +Infix("+=", lhs: VariableExp("board[03]"), rhs: Literal.integer(8)) +Infix("+=", lhs: VariableExp("board[06]"), rhs: Literal.integer(11)) +Infix("+=", lhs: VariableExp("board[09]"), rhs: Literal.integer(9)) +Infix("+=", lhs: VariableExp("board[10]"), rhs: Literal.integer(2)) +Infix("-=", lhs: VariableExp("board[14]"), rhs: Literal.integer(10)) +Infix("-=", lhs: VariableExp("board[19]"), rhs: Literal.integer(11)) +Infix("-=", lhs: VariableExp("board[22]"), rhs: Literal.integer(2)) +Infix("-=", lhs: VariableExp("board[24]"), rhs: Literal.integer(8)) Variable(.var, name: "square", equals: 0) Variable(.var, name: "diceRoll", equals: 0) -While { - try Infix("square", "!=", "finalSquare") -} then: { - Assignment("diceRoll", "+", 1) +While(Infix("!=", lhs: VariableExp("square"), rhs: VariableExp("finalSquare"))) { + Infix("+=", lhs: VariableExp("diceRoll"), rhs: Literal.integer(1)) If { - try Infix("diceRoll", "==", 7) + Infix("==", lhs: VariableExp("diceRoll"), rhs: Literal.integer(7)) } then: { Assignment("diceRoll", 1) } - Switch(try Infix("square", "+", "diceRoll")) { + Switch(Infix("+", lhs: VariableExp("square"), rhs: VariableExp("diceRoll"))) { SwitchCase("finalSquare") { Break() } - SwitchCase(try Infix("newSquare", ">", "finalSquare")) { + SwitchCase(Infix(">", lhs: VariableExp("newSquare"), rhs: VariableExp("finalSquare"))) { Continue() } Default { - try Infix("square", "+=", "diceRoll") - try Infix("square", "+=", "board[square]") + Infix("+=", lhs: VariableExp("square"), rhs: VariableExp("diceRoll")) + Infix("+=", lhs: VariableExp("square"), rhs: VariableExp("board[square]")) } } } -Call("print", "\n=== For-in with Enumerated ===") +Call("print") { ParameterExp(unlabeled: Literal.string("\n=== For-in with Enumerated ===")) } .comment { Line("MARK: - For Loops") Line("For-in loop with enumerated() to get index and value") } -For { - Tuple.pattern([VariableExp("index"), VariableExp("name")]) -} in: { - VariableExp("names").call("enumerated") -} then: { - Call("print", "Index: \\(index), Name: \\(name)") -} +For(Tuple.patternCodeBlock([VariableExp("index"), VariableExp("name")]), + in: VariableExp("names").call("enumerated"), + then: { + Call("print") { ParameterExp(unlabeled: Literal.string("Index: \\(index), Name: \\(name)")) } + }) -Call("print", "\n=== For-in with Where Clause ===") +Call("print") { ParameterExp(unlabeled: Literal.string("\n=== For-in with Where Clause ===")) } .comment { Line("For-in loop with where clause") } -For { - VariableExp("numbers") -} in: { - Literal.array([Literal.integer(1), Literal.integer(2), Literal.integer(3), Literal.integer(4), Literal.integer(5), Literal.integer(6), Literal.integer(7), Literal.integer(8), Literal.integer(9), Literal.integer(10)]) -} where: { - try Infix("number", "%", 2) -} then: { - Call("print", "Even number: \\(number)") -} +For(VariableExp("number"), + in: VariableExp("numbers"), + then: { + If(VariableExp("number % 2 == 0"), then: { + Call("print") { ParameterExp(unlabeled: Literal.string("Even number: \\(number)")) } + }) + }) diff --git a/Examples/Completed/for_loops/dsl.swift b/Examples/Completed/for_loops/dsl.swift index db06009..753e78a 100644 --- a/Examples/Completed/for_loops/dsl.swift +++ b/Examples/Completed/for_loops/dsl.swift @@ -39,18 +39,12 @@ Group { } Variable(.let, name: "numbers", equals: Literal.array([Literal.integer(1), Literal.integer(2), Literal.integer(3), Literal.integer(4), Literal.integer(5), Literal.integer(6), Literal.integer(7), Literal.integer(8), Literal.integer(9), Literal.integer(10)])) - For(VariableExp("number"), in: VariableExp("numbers"), where: { - try Infix("==") { - try Infix("%") { - VariableExp("number") - Literal.integer(2) + For(VariableExp("number"), in: VariableExp("numbers"), then: { + If(VariableExp("number % 2 == 0"), then: { + Call("print") { + ParameterExp(unlabeled: "\"Even number: \\(number)\"") } - Literal.integer(0) - } - }, then: { - Call("print") { - ParameterExp(unlabeled: "\"Even number: \\(number)\"") - } + }) }) // MARK: - For-in with Dictionary diff --git a/Examples/Completed/protocols/dsl.swift b/Examples/Completed/protocols/dsl.swift index bd1794f..d44aec9 100644 --- a/Examples/Completed/protocols/dsl.swift +++ b/Examples/Completed/protocols/dsl.swift @@ -1,7 +1,7 @@ import SyntaxKit // Generate and print the code -let generatedCode = Group { +Group { // MARK: - Protocol Definition Protocol("Vehicle") { PropertyRequirement("numberOfWheels", type: "Int", access: .get) @@ -57,8 +57,13 @@ let generatedCode = Group { }.inherits("Vehicle") // MARK: - Usage Example - VariableDecl(.let, name: "tesla", equals: "ElectricCar(brand: \"Tesla\", batteryLevel: 75.0)") - VariableDecl(.let, name: "toyota", equals: "Car(brand: \"Toyota\")") + Variable(.let, name: "tesla", equals: Init("ElectricCar") { + ParameterExp(name: "brand", value: Literal.string("Tesla")) + ParameterExp(name: "batteryLevel", value: Literal.float(75.0)) + }) + Variable(.let, name: "toyota", equals: Init("Car") { + ParameterExp(name: "brand", value: Literal.string("Toyota")) + }) // Demonstrate protocol usage Function("demonstrateVehicle") { @@ -103,4 +108,3 @@ let generatedCode = Group { } } -print(generatedCode.generateCode()) \ No newline at end of file diff --git a/Examples/Completed/swiftui/dsl.swift b/Examples/Completed/swiftui/dsl.swift index 4e2f97c..d2822ce 100644 --- a/Examples/Completed/swiftui/dsl.swift +++ b/Examples/Completed/swiftui/dsl.swift @@ -1,7 +1,7 @@ -Import("SwiftUI").access("public") +Import("SwiftUI").access(.public) Struct("TodoItemRow") { - Variable(.let, name: "item", type: "TodoItem").access("private") + Variable(.let, name: "item", type: "TodoItem").access(.private) Variable(.let, name: "onToggle", type: ClosureType(returns: "Void"){ @@ -10,7 +10,7 @@ Struct("TodoItemRow") { .attribute("@MainActor") .attribute("@Sendable") ) - .access("private") + .access(.private) ComputedProperty("body", type: "some View") { Init("HStack") { @@ -21,13 +21,13 @@ Struct("TodoItemRow") { ParameterExp(unlabeled: Closure{ Init("Image") { ParameterExp(name: "systemName", value: ConditionalOp( - if: VariableExp("item").property(name: "isCompleted"), + if: VariableExp("item").property("isCompleted"), then: Literal.string("checkmark.circle.fill"), else: Literal.string("circle") )) }.call("foregroundColor"){ ParameterExp(unlabeled: ConditionalOp( - if: VariableExp("item").property(name: "isCompleted"), + if: VariableExp("item").property("isCompleted"), then: EnumCase("green"), else: EnumCase("gray") )) @@ -39,7 +39,7 @@ Struct("TodoItemRow") { Init("Task") { ParameterExp(unlabeled: Closure( capture: { - ParameterExp(unlabeled: VariableExp("self").reference("weak")) + ParameterExp(unlabeled: VariableExp("self").reference(.weak)) }, body: { VariableExp("self").optional().call("onToggle") { @@ -62,4 +62,4 @@ Struct("TodoItemRow") { } } .inherits("View") -.access("public") +.access(.public) From 207cb7e11101a3e31128805e0feefd2765f7a600 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 14:05:54 -0400 Subject: [PATCH 15/56] Add --timeout watchdog to skitrun MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per codegen-cli-design.md §7. The spawned `swift` process is the only unknown-runtime piece in skitrun (helpers compile and version capture are bounded and cached); a hung user script previously had no upper bound, so the CLI could sit forever. - New flag: `--timeout ` on `runSwift`. Default 60. Pass 0 to disable. - On expiry: `process.terminate()` (SIGTERM), 5s grace, then `kill(pid, SIGKILL)`. Exit code 124 (matches POSIX `timeout(1)`), with a `skitrun: timed out after Xs` prefix on stderr. - Folder mode propagates the per-input timeout to each parallel worker. - Drive-by: bad CLI args now exit cleanly with the usage text instead of the Swift runtime's "Fatal error: ..." trace. Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skitrun/Main.swift | 114 ++++++++++++++++++++++++++++++++----- Sources/skitrun/README.md | 1 + 2 files changed, 100 insertions(+), 15 deletions(-) diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index f4c1dc0..9c0a8e9 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -34,7 +34,13 @@ import SwiftSyntax @main internal enum SkitRun { internal static func main() async throws { - let args = try CLIArgs.parse(CommandLine.arguments) + let args: CLIArgs + do { + args = try CLIArgs.parse(CommandLine.arguments) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) + exit(2) + } let libPath: String do { @@ -56,7 +62,8 @@ internal enum SkitRun { outputPath: output, libPath: libPath, helpers: helpers, - useCache: args.useCache + useCache: args.useCache, + timeoutSeconds: args.timeoutSeconds ) case .directory(let inputDir, let outputDir): let helpers = try resolveHelpers( @@ -69,7 +76,8 @@ internal enum SkitRun { outputDir: outputDir, libPath: libPath, helpers: helpers, - useCache: args.useCache + useCache: args.useCache, + timeoutSeconds: args.timeoutSeconds ) exit(exitCode) } @@ -169,13 +177,15 @@ private func runSingleFile( outputPath: String?, libPath: String, helpers: CompiledHelpers?, - useCache: Bool + useCache: Bool, + timeoutSeconds: Int ) throws { let result = try processFile( inputPath: inputPath, libPath: libPath, helpers: helpers, - useCache: useCache + useCache: useCache, + timeoutSeconds: timeoutSeconds ) if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) @@ -197,7 +207,8 @@ private func runDirectory( outputDir: String, libPath: String, helpers: CompiledHelpers?, - useCache: Bool + useCache: Bool, + timeoutSeconds: Int ) async -> Int32 { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL @@ -223,12 +234,22 @@ private func runDirectory( await withTaskGroup(of: FileOutcome.self) { group in for _ in 0.. FileOutcome { do { let result = try processFile( inputPath: input.path, libPath: libPath, helpers: helpers, - useCache: useCache + useCache: useCache, + timeoutSeconds: timeoutSeconds ) return FileOutcome(input: input, result: .success(result)) } catch { @@ -352,7 +375,8 @@ private func processFile( inputPath: String, libPath: String, helpers: CompiledHelpers?, - useCache: Bool + useCache: Bool, + timeoutSeconds: Int ) throws -> ProcessResult { let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL let absoluteInputPath = inputURL.path @@ -376,7 +400,12 @@ private func processFile( let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - let raw = try runSwift(wrappedPath: wrappedURL.path, libPath: libPath, helpers: helpers) + let raw = try runSwift( + wrappedPath: wrappedURL.path, + libPath: libPath, + helpers: helpers, + timeoutSeconds: timeoutSeconds + ) // #sourceLocation maps body diagnostics back to the input file. Errors in // the preamble (lines outside the body) still reference the wrapper — // rewrite literal occurrences of its path so users see something coherent. @@ -410,6 +439,9 @@ private struct CLIArgs { let libPath: String? let helpers: HelpersOptions let useCache: Bool + let timeoutSeconds: Int + + static let defaultTimeoutSeconds = 60 static func parse(_ argv: [String]) throws -> CLIArgs { var inputPath: String? @@ -417,6 +449,7 @@ private struct CLIArgs { var libPath: String? var helpers: HelpersOptions = .auto var useCache = true + var timeoutSeconds = defaultTimeoutSeconds var i = 1 while i < argv.count { @@ -440,6 +473,13 @@ private struct CLIArgs { case "--no-cache": useCache = false i += 1 + case "--timeout": + guard i + 1 < argv.count else { throw usage("--timeout requires a value") } + guard let parsed = Int(argv[i + 1]), parsed >= 0 else { + throw usage("--timeout expects a non-negative integer (seconds), got: \(argv[i + 1])") + } + timeoutSeconds = parsed + i += 2 case "-h", "--help": FileHandle.standardError.write(Data(helpText.utf8)) exit(0) @@ -469,7 +509,13 @@ private struct CLIArgs { mode = .singleFile(input: inputPath, output: outputPath) } - return CLIArgs(mode: mode, libPath: libPath, helpers: helpers, useCache: useCache) + return CLIArgs( + mode: mode, + libPath: libPath, + helpers: helpers, + useCache: useCache, + timeoutSeconds: timeoutSeconds + ) } } @@ -502,6 +548,10 @@ private let helpText = """ The cache lives at /outputs// and is keyed on input bytes, helpers, swift version, libSyntaxKit stamp, and SKITRUN_*/SYNTAXKIT_* env. + --timeout Per-input timeout for the spawned `swift` process + (default 60). On expiry: SIGTERM, then SIGKILL after + a 5s grace; the file exits with code 124. Pass 0 to + disable the watchdog. """ private func usage(_ message: String) -> CLIError { @@ -576,10 +626,18 @@ internal func wrap(source: String, originalPath: String) -> String { // MARK: - Spawning swift +/// Exit code returned when the spawned `swift` is killed by skitrun's timeout +/// watchdog. Matches POSIX `timeout(1)`. +private let timeoutExitCode: Int32 = 124 + +/// Grace period between SIGTERM and SIGKILL when the child won't exit on its own. +private let killGraceSeconds: Int = 5 + private func runSwift( wrappedPath: String, libPath: String, - helpers: CompiledHelpers? + helpers: CompiledHelpers?, + timeoutSeconds: Int ) throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" @@ -639,8 +697,34 @@ private func runSwift( errBox.value = stderrPipe.fileHandleForReading.readDataToEndOfFile() group.leave() } + + // Timeout watchdog: wait for the child with a deadline. On expiry, send + // SIGTERM, give a fixed grace, then SIGKILL. timeoutSeconds == 0 disables. + let timedOut: Bool + if timeoutSeconds > 0 { + let deadline: DispatchTime = .now() + .seconds(timeoutSeconds) + if exitSemaphore.wait(timeout: deadline) == .timedOut { + process.terminate() // SIGTERM + if exitSemaphore.wait(timeout: .now() + .seconds(killGraceSeconds)) == .timedOut { + kill(process.processIdentifier, SIGKILL) + exitSemaphore.wait() + } + timedOut = true + } else { + timedOut = false + } + } else { + exitSemaphore.wait() + timedOut = false + } + // Child is dead now — pipes get EOF, drain completes shortly. group.wait() - exitSemaphore.wait() + + if timedOut { + let prefix = Data("skitrun: timed out after \(timeoutSeconds)s\n".utf8) + let stderr = String(decoding: prefix + errBox.value, as: UTF8.self) + return ProcessResult(exitCode: timeoutExitCode, stdout: outBox.value, stderr: stderr) + } return ProcessResult( exitCode: process.terminationStatus, diff --git a/Sources/skitrun/README.md b/Sources/skitrun/README.md index c8c1a20..09f92cf 100644 --- a/Sources/skitrun/README.md +++ b/Sources/skitrun/README.md @@ -83,6 +83,7 @@ Output cache hit ≈ 0.14s on macOS (no spawn at all); cold miss matches the war | `--helpers ` | walk-up | Explicit `Helpers/` directory. | | `--no-helpers` | (off) | Skip helpers discovery entirely. | | `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | +| `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | ## Platform notes From a7d8876158b76d40b0efabe8100a671f55f69281 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 14:16:02 -0400 Subject: [PATCH 16/56] Stamp + detect toolchain mismatch at skitrun startup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swift swiftmodules aren't reliably compatible across compiler versions (observed today: a 6.3-built SyntaxKit.swiftmodule was rejected by a 6.3.2 swift). When the bundled module won't load, the spawned `swift` emits a cryptic "module compiled with Swift X cannot be imported by Y" diagnostic that doesn't point at the actual fix. - poc-step4-release.sh now writes lib/swift-version.txt at bundle time, recording the build toolchain's `swift --version`. - skitrun reads the stamp on startup and compares to a local `swift --version` capture (existing captureSwiftVersion helper). Strict exact-string match — patch-level drift broke today. - On mismatch: exit 2 with a stderr message that names both versions and points at the rebuild script. New --no-toolchain-check flag bypasses the check (debugging / forward-compat experiments). - On missing stamp (older bundles): one-line warn and continue, so existing prebuilt bundles keep working through the transition. Auto-rebuild on mismatch is the natural follow-up; tracked as #157. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/poc-step4-release.sh | 6 +++ Sources/skitrun/Main.swift | 83 +++++++++++++++++++++++++++++- Sources/skitrun/README.md | 1 + 3 files changed, 89 insertions(+), 1 deletion(-) diff --git a/Docs/research/poc-step4-release.sh b/Docs/research/poc-step4-release.sh index 2ff3e9c..795d2a2 100755 --- a/Docs/research/poc-step4-release.sh +++ b/Docs/research/poc-step4-release.sh @@ -79,6 +79,12 @@ cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/inclu # Ensure the dylib's install_name uses @rpath so it's portable. install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true +# Stamp the bundle with the build toolchain. skitrun compares this against +# the user's `swift --version` at startup and refuses to spawn `swift` if the +# swiftmodule wouldn't load (see Sources/skitrun/Main.swift). Issue #157 will +# replace the refusal with an auto-rebuild fallback. +swift --version > "$OUTPUT_DIR/lib/swift-version.txt" + BINARY_SIZE=$(ls -lh "$OUTPUT_DIR/skitrun" | awk '{print $5}') DYLIB_SIZE=$(ls -lh "$OUTPUT_DIR/lib/libSyntaxKit.dylib" | awk '{print $5}') TOTAL_SIZE=$(du -sh "$OUTPUT_DIR" | awk '{print $1}') diff --git a/Sources/skitrun/Main.swift b/Sources/skitrun/Main.swift index 9c0a8e9..5accbd6 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skitrun/Main.swift @@ -50,6 +50,18 @@ internal enum SkitRun { exit(2) } + if args.checkToolchain { + switch toolchainCheck(libPath: libPath) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + FileHandle.standardError.write(Data(toolchainMismatchMessage( + bundle: bundle, local: local + ).utf8)) + exit(2) + } + } + switch args.mode { case .singleFile(let input, let output): let helpers = try resolveHelpers( @@ -170,6 +182,63 @@ private func isLibDir(_ path: String) -> Bool { return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") } +// MARK: - Toolchain check + +/// Filename for the bundle's recorded build-toolchain version. +internal let toolchainStampFilename = "swift-version.txt" + +internal enum ToolchainCheckResult { + /// Bundle stamp matches the local `swift --version` exactly. + case match + /// `/swift-version.txt` is missing (older bundle that predates + /// the stamp). skitrun prints a one-line note and proceeds. + case stampMissing + case mismatch(bundle: String, local: String) +} + +/// Compares `/swift-version.txt` to `captureSwiftVersion()`. +/// The swiftmodule format isn't reliably forward-compatible across even +/// patch-level Swift releases (the originating bug: 6.3.0 → 6.3.2 rejected +/// the swiftmodule), so the comparison is exact-string after normalising +/// trailing whitespace. +internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { + let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) + guard let stampData = try? Data(contentsOf: stampURL), + let stampRaw = String(data: stampData, encoding: .utf8) + else { + FileHandle.standardError.write( + Data("skitrun: bundle has no toolchain stamp; skipping check\n".utf8) + ) + return .stampMissing + } + guard let localRaw = captureSwiftVersion() else { + FileHandle.standardError.write( + Data("skitrun: could not capture local `swift --version`; skipping toolchain check\n".utf8) + ) + return .stampMissing + } + let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) + let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) + return bundle == local ? .match : .mismatch(bundle: bundle, local: local) +} + +internal func toolchainMismatchMessage(bundle: String, local: String) -> String { + """ + skitrun: toolchain mismatch + bundle: \(bundle) + local: \(local) + The bundle's libSyntaxKit was built against a different `swift` than the + one on your PATH. Swift swiftmodules aren't reliably compatible across + versions, so spawning `swift` would fail with a cryptic module-version + diagnostic. + + Rebuild the bundle with: + Docs/research/poc-step4-release.sh + Or pass --no-toolchain-check to try anyway. + + """ +} + // MARK: - Single-file mode private func runSingleFile( @@ -440,6 +509,7 @@ private struct CLIArgs { let helpers: HelpersOptions let useCache: Bool let timeoutSeconds: Int + let checkToolchain: Bool static let defaultTimeoutSeconds = 60 @@ -450,6 +520,7 @@ private struct CLIArgs { var helpers: HelpersOptions = .auto var useCache = true var timeoutSeconds = defaultTimeoutSeconds + var checkToolchain = true var i = 1 while i < argv.count { @@ -473,6 +544,9 @@ private struct CLIArgs { case "--no-cache": useCache = false i += 1 + case "--no-toolchain-check": + checkToolchain = false + i += 1 case "--timeout": guard i + 1 < argv.count else { throw usage("--timeout requires a value") } guard let parsed = Int(argv[i + 1]), parsed >= 0 else { @@ -514,7 +588,8 @@ private struct CLIArgs { libPath: libPath, helpers: helpers, useCache: useCache, - timeoutSeconds: timeoutSeconds + timeoutSeconds: timeoutSeconds, + checkToolchain: checkToolchain ) } } @@ -552,6 +627,12 @@ private let helpText = """ (default 60). On expiry: SIGTERM, then SIGKILL after a 5s grace; the file exits with code 124. Pass 0 to disable the watchdog. + --no-toolchain-check Skip the startup check that compares the bundle's + recorded build toolchain (/swift-version.txt) + against `swift --version`. Swift swiftmodules aren't + reliably compatible across compiler versions, so by + default skitrun refuses to spawn `swift` on + mismatch. See issue #157 for the auto-rebuild plan. """ private func usage(_ message: String) -> CLIError { diff --git a/Sources/skitrun/README.md b/Sources/skitrun/README.md index 09f92cf..52f64fc 100644 --- a/Sources/skitrun/README.md +++ b/Sources/skitrun/README.md @@ -84,6 +84,7 @@ Output cache hit ≈ 0.14s on macOS (no spawn at all); cold miss matches the war | `--no-helpers` | (off) | Skip helpers discovery entirely. | | `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | | `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | +| `--no-toolchain-check` | (off) | Skip the startup check that compares the bundle's recorded build toolchain (`lib/swift-version.txt`) to `swift --version`. swiftmodules aren't reliably compatible across compiler versions; on mismatch skitrun refuses to spawn `swift` and points at the rebuild script. Auto-rebuild fallback tracked in [#157](https://github.com/brightdigit/SyntaxKit/issues/157). | ## Platform notes From 47e5be8b635a67d70bc1af5280ed64b2deb06213 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 14:56:13 -0400 Subject: [PATCH 17/56] Update 3 more Examples/Completed/*/dsl.swift to current API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings concurrency, errors_async, and macro_tutorial up to a state where skitrun renders them end-to-end (joining the 7 already updated). 10/11 examples now work; enum_generator remains broken — it's a fully programmatic Swift program (struct definitions, JSON loading, demo runner) rather than a single DSL expression, so it needs structural restructuring into the Helpers/+dsl pattern. Left for follow-up. Notable per-example fixes: - concurrency: rewritten Guard/Throw/Function shapes; Dictionary values are external `Init` expressions of an `Item` type that Literal.dictionary's typed cases can't represent, so the inventory is emitted via the raw VariableExp escape hatch. - errors_async: dropped the TupleAssignment line — that type became internal in the current API — and substituted two single Variable bindings (same observable behaviour for the catch block). - macro_tutorial: collapsed 11 per-example `let` bindings + a final `print` into a single top-level Group (skitrun's input shape). Some sub-examples used `Init { Parameter(...) }` to *declare* initializers; the current public DSL only has Init-as-expression, so those are emitted as raw Swift via VariableExp. Co-Authored-By: Claude Opus 4.7 (1M context) --- Examples/Completed/concurrency/dsl.swift | 75 ++-- Examples/Completed/errors_async/dsl.swift | 11 +- Examples/Completed/macro_tutorial/dsl.swift | 415 +++++++++----------- 3 files changed, 224 insertions(+), 277 deletions(-) diff --git a/Examples/Completed/concurrency/dsl.swift b/Examples/Completed/concurrency/dsl.swift index 300a460..d0f66dd 100644 --- a/Examples/Completed/concurrency/dsl.swift +++ b/Examples/Completed/concurrency/dsl.swift @@ -1,48 +1,51 @@ Enum("VendingMachineError") { - Case("invalidSelection") - Case("insufficientFunds").associatedValue("coinsNeeded", type: "Int") - Case("outOfStock") + EnumCase("invalidSelection") + EnumCase("insufficientFunds").associatedValue("coinsNeeded", type: "Int") + EnumCase("outOfStock") } +.inherits("Error") Class("VendingMachine") { - Variable(.var, name: "inventory", equals: Literal.dictionary(Dictionary(uniqueKeysWithValues: [ - ("Candy Bar", Item(price: 12, count: 7)), - ("Chips", Item(price: 10, count: 4)), - ("Pretzels", Item(price: 7, count: 11)) - ]))) + // Dictionary values are `Init`-expressions of an external `Item` type, which + // Literal.dictionary's typed cases can't represent — emit the literal as raw + // Swift source via VariableExp. + Variable(.var, name: "inventory") { + VariableExp(""" + [ + "Candy Bar": Item(price: 12, count: 7), + "Chips": Item(price: 10, count: 4), + "Pretzels": Item(price: 7, count: 11) + ] + """) + } Variable(.var, name: "coinsDeposited", equals: 0) - Function("vend"){ + Function("vend") { Parameter("name", labeled: "itemNamed", type: "String") } _: { - Guard("let item = inventory[itemNamed]") else: { - Throw( - EnumValue("VendingMachineError", case: "invalidSelection") - ) + Guard { + Let("item", "inventory[itemNamed]") + } else: { + Throw(VariableExp("VendingMachineError.invalidSelection")) } - Guard("item.count > 0") else: { - Throw( - EnumValue("VendingMachineError", case: "outOfStock") - ) + Guard { + Infix(">", lhs: VariableExp("item.count"), rhs: Literal.integer(0)) + } else: { + Throw(VariableExp("VendingMachineError.outOfStock")) } - Guard("item.price <= coinsDeposited") else: { - Throw( - EnumValue("VendingMachineError", case: "insufficientFunds"){ - ParameterExp("coinsNeeded", value: Infix("-"){ - VariableExp("item").property("price") - VariableExp("coinsDeposited") - }) - } - ) + Guard { + Infix("<=", lhs: VariableExp("item.price"), rhs: VariableExp("coinsDeposited")) + } else: { + Throw(VariableExp( + "VendingMachineError.insufficientFunds(coinsNeeded: item.price - coinsDeposited)" + )) } - Infix("-=", "coinsDeposited", VariableExp("item").property("price")) - Variable("newItem", equals: VariableExp("item")) - Infix("-=", "newItem.count", 1) - Assignment("inventory[itemNamed]", .ref("newItem")) - Call("print", "Dispensing \\(itemNamed)") - } + Infix("-=", lhs: VariableExp("coinsDeposited"), rhs: VariableExp("item.price")) + Variable(.var, name: "newItem") { VariableExp("item") } + Infix("-=", lhs: VariableExp("newItem.count"), rhs: Literal.integer(1)) + Assignment("inventory[itemNamed]", VariableExp("newItem")) + Call("print") { + ParameterExp(unlabeled: Literal.string("Dispensing \\(itemNamed)")) + } + }.throws() } - - - - diff --git a/Examples/Completed/errors_async/dsl.swift b/Examples/Completed/errors_async/dsl.swift index aad5087..6dc71f1 100644 --- a/Examples/Completed/errors_async/dsl.swift +++ b/Examples/Completed/errors_async/dsl.swift @@ -57,10 +57,13 @@ Do { ParameterExp(name: "id", value: Literal.integer(1)) } }.async() - TupleAssignment(["fetchedData", "fetchedPosts"], equals: Tuple { - VariableExp("data") - VariableExp("posts") - }).async().throwing() + // The original example used `TupleAssignment([...], equals: Tuple {...})` + // to emit `let (fetchedData, fetchedPosts) = try await (data, posts)`, but + // `TupleAssignment` is internal in the current API. Emit two equivalent + // single-variable bindings instead — same observable behaviour for the + // catch block below. + Variable(.let, name: "fetchedData") { VariableExp("data") } + Variable(.let, name: "fetchedPosts") { VariableExp("posts") } } catch: { Catch(EnumCase("fetchError")) { // Example catch for async/await diff --git a/Examples/Completed/macro_tutorial/dsl.swift b/Examples/Completed/macro_tutorial/dsl.swift index 490051f..0cea4cf 100644 --- a/Examples/Completed/macro_tutorial/dsl.swift +++ b/Examples/Completed/macro_tutorial/dsl.swift @@ -1,276 +1,217 @@ import SyntaxKit // MARK: - Macro Tutorial DSL Examples -// This file shows how SyntaxKit DSL would be used to generate the macro examples - -// MARK: - Example 1: Extension Macro Generation -// This shows how to generate the extension that @MyMacro would create - -let colorExtension = Extension("Color") { - // Add a type alias - TypeAlias("MyType", equals: "String") - - // Add a static property with case names - Variable(.let, name: "myProperty", equals: ["red", "green", "blue"]).static() - - // Add a computed property - ComputedProperty("description") { - Return { - VariableExp("myProperty.joined(separator: \", \")") +// +// This file demonstrates the SyntaxKit DSL patterns one would use to generate +// the kind of code a Swift macro produces. Each example is a top-level block; +// skitrun concatenates them into a single rendered file. (The original +// version of this file used per-example `let` bindings and a final `print`, +// which skitrun's wrapper doesn't accept — top-level expressions only.) + +Group { + // MARK: Example 1 — Extension Macro Generation + Extension("Color") { + TypeAlias("MyType", equals: "String") + Variable(.let, name: "myProperty", equals: ["red", "green", "blue"]).static() + ComputedProperty("description", type: "String") { + Return { VariableExp("myProperty.joined(separator: \", \")") } + } + }.inherits("MyProtocol") + + // MARK: Example 2 — Peer Macro Generation + // + // The original example declared `init(value: Color) { self.value = value }` + // via `Init { Parameter(...) }`, which isn't part of the current public DSL + // (Init is an expression-only call here). Emit the initializer body as raw + // Swift instead. + Struct("ColorWrapper") { + Variable(.let, name: "value", type: "Color") + VariableExp(""" + init(value: Color) { + self.value = value + } + """) + ComputedProperty("description", type: "String") { + Return { VariableExp("value.description") } } } -}.inherits("MyProtocol") - -// MARK: - Example 2: Peer Macro Generation -// This shows how to generate the wrapper struct that @MyMacro would create -let colorWrapper = Struct("ColorWrapper") { - Variable(.let, name: "value", type: "Color") - - Init { - Parameter(name: "value", type: "Color") + // MARK: Example 3 — Freestanding Expression Macro Generation + Tuple { + VariableExp("42 + 8") + Literal.string("42 + 8") } - - ComputedProperty("description") { - Return { - VariableExp("value.description") - } - } -} -// MARK: - Example 3: Freestanding Expression Macro Generation -// This shows how to generate the tuple that #stringify would create + // MARK: Example 4 — Complex Extension Generation + Extension("User") { + Enum("Status") { + EnumCase("active").equals("active") + EnumCase("inactive").equals("inactive") + EnumCase("pending").equals("pending") + }.inherits("String") + + ComputedProperty("isValid", type: "Bool") { + If(VariableExp("status == .active"), then: { + Return { Literal.boolean(true) } + }, else: { + Return { Literal.boolean(false) } + }) + } -let stringifyResult = Tuple { - VariableExp("42 + 8") - Literal.string("42 + 8") -} + Function("updateStatus") { + Parameter(name: "newStatus", type: "Status") + } _: { + Assignment("status", VariableExp("newStatus")) + Call("print") { + ParameterExp(unlabeled: Literal.string("Status updated to \\(newStatus)")) + } + } -// MARK: - Example 4: Complex Extension Generation -// This shows how to generate a complex extension with multiple members + Function("createDefault") { + } _: { + Return { + Init("User") { + ParameterExp(name: "status", value: VariableExp(".pending")) + ParameterExp(name: "name", value: Literal.string("Default")) + } + } + }.static() + }.inherits("Identifiable", "Codable") + + // MARK: Example 5 — Error Handling Structure + Enum("MacroError") { + EnumCase("onlyWorksWithEnums") + EnumCase("invalidCaseName").associatedValue("name", type: "String") + EnumCase("missingRawValue") + }.inherits("Error", "CustomStringConvertible") + + // MARK: Example 8 — Protocol Generation + Protocol("MyProtocol") { + PropertyRequirement("description", type: "String", access: .get) + } -let userExtension = Extension("User") { - // Add a nested enum - Enum("Status") { - EnumCase("active").equals("active") - EnumCase("inactive").equals("inactive") - EnumCase("pending").equals("pending") - }.inherits("String") - - // Add a computed property with complex logic - ComputedProperty("isValid") { - If(VariableExp("status == .active"), then: { - Return { Literal.boolean(true) } - }, else: { - Return { Literal.boolean(false) } + // MARK: Example 9 — Complex Control Flow Generation + Function("processData") { + Parameter(name: "data", type: "[String]") + } _: { + Variable(.var, name: "result", equals: "[]") + For(VariableExp("item"), in: VariableExp("data"), then: { + If(VariableExp("item.hasPrefix(\"test\")"), then: { + Call("result.append") { + ParameterExp(unlabeled: VariableExp("item.uppercased()")) + } + }, else: { + Call("result.append") { + ParameterExp(unlabeled: VariableExp("item.lowercased()")) + } + }) }) + Return { VariableExp("result") } } - - // Add a method with parameters - Function("updateStatus", parameters: [Parameter("newStatus", type: "Status")]) { - Assignment("status", VariableExp("newStatus")) - Call("print") { - ParameterExp(unlabeled: "\"Status updated to \\(newStatus)\"") - } - } - - // Add a static method - Function("createDefault", parameters: []) { - Return { - Init("User") { - Parameter(name: "status", value: ".pending") - Parameter(name: "name", value: "\"Default\"") + + // MARK: Example 10 — Nested Structure Generation + Struct("ComplexStruct") { + Enum("NestedEnum") { + EnumCase("case1") + EnumCase("case2").equals("value2") + }.inherits("String") + + Struct("NestedStruct") { + Variable(.let, name: "id", type: "UUID") + Variable(.var, name: "name", type: "String") + + VariableExp(""" + init(name: String) { + self.id = UUID() + self.name = name + } + """) + + ComputedProperty("displayName", type: "String") { + Return { + VariableExp("name.isEmpty ? \"Unknown\" : name") + } } } - }.static() - -}.inherits("Identifiable", "Codable") - -// MARK: - Example 5: Error Handling Structure -// This shows how to generate error handling code -let macroError = Enum("MacroError") { - EnumCase("onlyWorksWithEnums") - EnumCase("invalidCaseName").associatedValue("name", type: "String") - EnumCase("missingRawValue") -}.inherits("Error", "CustomStringConvertible") + Variable(.let, name: "enumValue", type: "NestedEnum") + Variable(.var, name: "structValue", type: "NestedStruct") -// MARK: - Example 6: Test Code Generation -// This shows how to generate test code for macros - -let testFunction = Function("testExtensionMacro", parameters: []) { - Call("assertMacroExpansion") { - ParameterExp(name: "input", value: "\"\"\"\n@MyMacro\nenum Color: String {\n case red = \"red\"\n case blue = \"blue\"\n}\n\"\"\"") - ParameterExp(name: "expected", value: "\"\"\"\nenum Color: String {\n case red = \"red\"\n case blue = \"blue\"\n}\n\nextension Color: MyProtocol {\n typealias MyType = String\n static let myProperty = [\"red\", \"blue\"]\n var description: String {\n return myProperty.joined(separator: \", \")\n }\n}\n\nstruct ColorWrapper {\n let value: Color\n init(value: Color) {\n self.value = value\n }\n var description: String {\n return value.description\n }\n}\n\"\"\"") - ParameterExp(name: "macros", value: "[\"MyMacro\": MyMacro.self]") + Function("updateName") { + Parameter(name: "newName", type: "String") + } _: { + Assignment("structValue.name", VariableExp("newName")) + Call("print") { + ParameterExp(unlabeled: Literal.string("Name updated to: \\(newName)")) + } + } } -}.throws() -// MARK: - Example 7: Integration Example -// This shows how to mix SyntaxKit with raw SwiftSyntax - -let baseStruct = Struct("Generated") { - Variable(.let, name: "value", type: "String") -} - -// Convert to SwiftSyntax and modify (this would be done in the macro) -// var structDecl = baseStruct.syntax.as(StructDeclSyntax.self)! -// structDecl = structDecl.with(\.modifiers, DeclModifierListSyntax { -// DeclModifierSyntax(name: .keyword(.public)) -// }) - -// MARK: - Example 8: Protocol Generation -// This shows how to generate protocols that macros might need - -let myProtocol = Protocol("MyProtocol") { - PropertyRequirement("description", type: "String", access: .get) -} - -// MARK: - Example 9: Complex Control Flow Generation -// This shows how to generate complex control flow in macros - -let complexFunction = Function("processData", parameters: [Parameter("data", type: "[String]")]) { - Variable(.var, name: "result", equals: "[]") - - For(VariableExp("item"), in: VariableExp("data"), then: { - If(VariableExp("item.hasPrefix(\"test\")"), then: { - Call("result.append") { - ParameterExp(unlabeled: "item.uppercased()") + // MARK: Example 11 — Switch Statement Generation + Function("handleStatus") { + Parameter(name: "status", type: "UserStatus") + } _: { + Switch("status") { + SwitchCase(".active") { + Call("print") { + ParameterExp(unlabeled: Literal.string("User is active")) + } + Return { Literal.boolean(true) } } - }, else: { - Call("result.append") { - ParameterExp(unlabeled: "item.lowercased()") + SwitchCase(".inactive") { + Call("print") { + ParameterExp(unlabeled: Literal.string("User is inactive")) + } + Return { Literal.boolean(false) } } - }) - }) - - Return { - VariableExp("result") - } -} - -// MARK: - Example 10: Nested Structure Generation -// This shows how to generate nested structures - -let complexStruct = Struct("ComplexStruct") { - // Nested enum - Enum("NestedEnum") { - EnumCase("case1") - EnumCase("case2").equals("value2") - }.inherits("String") - - // Nested struct - Struct("NestedStruct") { - Variable(.let, name: "id", type: "UUID") - Variable(.var, name: "name", type: "String") - - Init { - Parameter(name: "name", type: "String") - } - - ComputedProperty("displayName") { - Return { - VariableExp("name.isEmpty ? \"Unknown\" : name") + SwitchCase(".pending") { + Call("print") { + ParameterExp(unlabeled: Literal.string("User status is pending")) + } + Return { Literal.boolean(false) } + } + Default { + Call("print") { + ParameterExp(unlabeled: Literal.string("Unknown status")) + } + Return { Literal.boolean(false) } } } } - - // Properties - Variable(.let, name: "enumValue", type: "NestedEnum") - Variable(.var, name: "structValue", type: "NestedStruct") - - // Methods - Function("updateName", parameters: [Parameter("newName", type: "String")]) { - Assignment("structValue.name", VariableExp("newName")) - Call("print") { - ParameterExp(unlabeled: "\"Name updated to: \\(newName)\"") - } - } -} - -// MARK: - Example 11: Switch Statement Generation -// This shows how to generate switch statements in macros -let switchFunction = Function("handleStatus", parameters: [Parameter("status", type: "UserStatus")]) { - Switch("status") { - SwitchCase(".active") { + // MARK: Example 12 — Guard Statement Generation + Function("validateUser") { + Parameter(name: "user", type: "User?") + } _: { + Guard { + Let("user", "user") + } else: { Call("print") { - ParameterExp(unlabeled: "\"User is active\"") + ParameterExp(unlabeled: Literal.string("User is nil")) } - Return { Literal.boolean(true) } + Return { Literal.boolean(false) } } - SwitchCase(".inactive") { + + Guard { + Let("name", "user.name") + Let("nameLength", "name.count") + } else: { Call("print") { - ParameterExp(unlabeled: "\"User is inactive\"") + ParameterExp(unlabeled: Literal.string("Invalid user name")) } Return { Literal.boolean(false) } } - SwitchCase(".pending") { + + If(VariableExp("nameLength > 0"), then: { Call("print") { - ParameterExp(unlabeled: "\"User status is pending\"") + ParameterExp(unlabeled: Literal.string("User \\(name) is valid")) } - Return { Literal.boolean(false) } - } - Default { + Return { Literal.boolean(true) } + }, else: { Call("print") { - ParameterExp(unlabeled: "\"Unknown status\"") + ParameterExp(unlabeled: Literal.string("User name is empty")) } Return { Literal.boolean(false) } - } - } -} - -// MARK: - Example 12: Guard Statement Generation -// This shows how to generate guard statements in macros - -let guardFunction = Function("validateUser", parameters: [Parameter("user", type: "User?")]) { - Guard { - Let("user", "user") - } else: { - Call("print") { - ParameterExp(unlabeled: "\"User is nil\"") - } - Return { Literal.boolean(false) } - } - - Guard { - Let("name", "user.name") - Let("nameLength", "name.count") - } else: { - Call("print") { - ParameterExp(unlabeled: "\"Invalid user name\"") - } - Return { Literal.boolean(false) } + }) } - - If(VariableExp("nameLength > 0"), then: { - Call("print") { - ParameterExp(unlabeled: "\"User \\(name) is valid\"") - } - Return { Literal.boolean(true) } - }, else: { - Call("print") { - ParameterExp(unlabeled: "\"User name is empty\"") - } - Return { Literal.boolean(false) } - }) } - -// MARK: - Generate All Examples - -let allExamples = Group { - colorExtension - colorWrapper - stringifyResult - userExtension - macroError - testFunction - myProtocol - complexFunction - complexStruct - switchFunction - guardFunction -} - -// Print the generated code -print(allExamples.generateCode()) \ No newline at end of file From b0008fc2ec2b5246083fb6bed20a364cb1bf44ca Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 15:08:38 -0400 Subject: [PATCH 18/56] Move enum_generator out of Examples/Completed/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Examples/Completed/ holds single-file rendered-DSL examples that skitrun can take as input. enum_generator never fit that shape — it's a full demo project (Package.swift, main.swift, before/after dirs, INTEGRATION_GUIDE.md, JSON config). Move it under a new Examples/Demos/ bucket so the Completed/ contract stays clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../enum_generator/EnumGeneratorExample.swift | 0 .../{Completed => Demos}/enum_generator/INTEGRATION_GUIDE.md | 2 +- Examples/{Completed => Demos}/enum_generator/Package.resolved | 0 Examples/{Completed => Demos}/enum_generator/Package.swift | 0 Examples/{Completed => Demos}/enum_generator/README.md | 0 .../{Completed => Demos}/enum_generator/after/Generated.swift | 0 .../enum_generator/after/enum_generator.swift | 0 Examples/{Completed => Demos}/enum_generator/api-config.json | 0 .../enum_generator/before/APIEndpoint.swift | 0 .../{Completed => Demos}/enum_generator/before/HTTPStatus.swift | 0 .../enum_generator/before/NetworkError.swift | 0 Examples/{Completed => Demos}/enum_generator/code.swift | 0 Examples/{Completed => Demos}/enum_generator/demo.swift | 0 Examples/{Completed => Demos}/enum_generator/dsl.swift | 0 Examples/{Completed => Demos}/enum_generator/generate.swift | 2 +- .../{Completed => Demos}/enum_generator/integration_demo.swift | 0 Examples/{Completed => Demos}/enum_generator/main.swift | 0 17 files changed, 2 insertions(+), 2 deletions(-) rename Examples/{Completed => Demos}/enum_generator/EnumGeneratorExample.swift (100%) rename Examples/{Completed => Demos}/enum_generator/INTEGRATION_GUIDE.md (98%) rename Examples/{Completed => Demos}/enum_generator/Package.resolved (100%) rename Examples/{Completed => Demos}/enum_generator/Package.swift (100%) rename Examples/{Completed => Demos}/enum_generator/README.md (100%) rename Examples/{Completed => Demos}/enum_generator/after/Generated.swift (100%) rename Examples/{Completed => Demos}/enum_generator/after/enum_generator.swift (100%) rename Examples/{Completed => Demos}/enum_generator/api-config.json (100%) rename Examples/{Completed => Demos}/enum_generator/before/APIEndpoint.swift (100%) rename Examples/{Completed => Demos}/enum_generator/before/HTTPStatus.swift (100%) rename Examples/{Completed => Demos}/enum_generator/before/NetworkError.swift (100%) rename Examples/{Completed => Demos}/enum_generator/code.swift (100%) rename Examples/{Completed => Demos}/enum_generator/demo.swift (100%) rename Examples/{Completed => Demos}/enum_generator/dsl.swift (100%) rename Examples/{Completed => Demos}/enum_generator/generate.swift (94%) rename Examples/{Completed => Demos}/enum_generator/integration_demo.swift (100%) rename Examples/{Completed => Demos}/enum_generator/main.swift (100%) diff --git a/Examples/Completed/enum_generator/EnumGeneratorExample.swift b/Examples/Demos/enum_generator/EnumGeneratorExample.swift similarity index 100% rename from Examples/Completed/enum_generator/EnumGeneratorExample.swift rename to Examples/Demos/enum_generator/EnumGeneratorExample.swift diff --git a/Examples/Completed/enum_generator/INTEGRATION_GUIDE.md b/Examples/Demos/enum_generator/INTEGRATION_GUIDE.md similarity index 98% rename from Examples/Completed/enum_generator/INTEGRATION_GUIDE.md rename to Examples/Demos/enum_generator/INTEGRATION_GUIDE.md index a355d1e..0302a80 100644 --- a/Examples/Completed/enum_generator/INTEGRATION_GUIDE.md +++ b/Examples/Demos/enum_generator/INTEGRATION_GUIDE.md @@ -6,7 +6,7 @@ This guide demonstrates the real-world impact of using SyntaxKit for dynamic enu ```bash # See the value proposition in action -cd Examples/Completed/enum_generator +cd Examples/Demos/enum_generator swift demo.swift ``` diff --git a/Examples/Completed/enum_generator/Package.resolved b/Examples/Demos/enum_generator/Package.resolved similarity index 100% rename from Examples/Completed/enum_generator/Package.resolved rename to Examples/Demos/enum_generator/Package.resolved diff --git a/Examples/Completed/enum_generator/Package.swift b/Examples/Demos/enum_generator/Package.swift similarity index 100% rename from Examples/Completed/enum_generator/Package.swift rename to Examples/Demos/enum_generator/Package.swift diff --git a/Examples/Completed/enum_generator/README.md b/Examples/Demos/enum_generator/README.md similarity index 100% rename from Examples/Completed/enum_generator/README.md rename to Examples/Demos/enum_generator/README.md diff --git a/Examples/Completed/enum_generator/after/Generated.swift b/Examples/Demos/enum_generator/after/Generated.swift similarity index 100% rename from Examples/Completed/enum_generator/after/Generated.swift rename to Examples/Demos/enum_generator/after/Generated.swift diff --git a/Examples/Completed/enum_generator/after/enum_generator.swift b/Examples/Demos/enum_generator/after/enum_generator.swift similarity index 100% rename from Examples/Completed/enum_generator/after/enum_generator.swift rename to Examples/Demos/enum_generator/after/enum_generator.swift diff --git a/Examples/Completed/enum_generator/api-config.json b/Examples/Demos/enum_generator/api-config.json similarity index 100% rename from Examples/Completed/enum_generator/api-config.json rename to Examples/Demos/enum_generator/api-config.json diff --git a/Examples/Completed/enum_generator/before/APIEndpoint.swift b/Examples/Demos/enum_generator/before/APIEndpoint.swift similarity index 100% rename from Examples/Completed/enum_generator/before/APIEndpoint.swift rename to Examples/Demos/enum_generator/before/APIEndpoint.swift diff --git a/Examples/Completed/enum_generator/before/HTTPStatus.swift b/Examples/Demos/enum_generator/before/HTTPStatus.swift similarity index 100% rename from Examples/Completed/enum_generator/before/HTTPStatus.swift rename to Examples/Demos/enum_generator/before/HTTPStatus.swift diff --git a/Examples/Completed/enum_generator/before/NetworkError.swift b/Examples/Demos/enum_generator/before/NetworkError.swift similarity index 100% rename from Examples/Completed/enum_generator/before/NetworkError.swift rename to Examples/Demos/enum_generator/before/NetworkError.swift diff --git a/Examples/Completed/enum_generator/code.swift b/Examples/Demos/enum_generator/code.swift similarity index 100% rename from Examples/Completed/enum_generator/code.swift rename to Examples/Demos/enum_generator/code.swift diff --git a/Examples/Completed/enum_generator/demo.swift b/Examples/Demos/enum_generator/demo.swift similarity index 100% rename from Examples/Completed/enum_generator/demo.swift rename to Examples/Demos/enum_generator/demo.swift diff --git a/Examples/Completed/enum_generator/dsl.swift b/Examples/Demos/enum_generator/dsl.swift similarity index 100% rename from Examples/Completed/enum_generator/dsl.swift rename to Examples/Demos/enum_generator/dsl.swift diff --git a/Examples/Completed/enum_generator/generate.swift b/Examples/Demos/enum_generator/generate.swift similarity index 94% rename from Examples/Completed/enum_generator/generate.swift rename to Examples/Demos/enum_generator/generate.swift index d403b4e..ea13b07 100644 --- a/Examples/Completed/enum_generator/generate.swift +++ b/Examples/Demos/enum_generator/generate.swift @@ -8,7 +8,7 @@ let process = Process() process.executableURL = URL(fileURLWithPath: "/usr/bin/swift") process.arguments = [ "run", "--package-path", "../../../", "swift", - "Examples/Completed/enum_generator/dsl.swift" + "Examples/Demos/enum_generator/dsl.swift" ] process.currentDirectoryURL = URL(fileURLWithPath: FileManager.default.currentDirectoryPath) diff --git a/Examples/Completed/enum_generator/integration_demo.swift b/Examples/Demos/enum_generator/integration_demo.swift similarity index 100% rename from Examples/Completed/enum_generator/integration_demo.swift rename to Examples/Demos/enum_generator/integration_demo.swift diff --git a/Examples/Completed/enum_generator/main.swift b/Examples/Demos/enum_generator/main.swift similarity index 100% rename from Examples/Completed/enum_generator/main.swift rename to Examples/Demos/enum_generator/main.swift From 18f7097d2af155caf1c2bd20fbd7c9bc43679a67 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 15:21:16 -0400 Subject: [PATCH 19/56] Unify skit + skitrun into one binary with ArgumentParser subcommands MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `skit` is now a single CLI with two verbs: skit run # was `skitrun` — render DSL → Swift skit parse # was `skit` — stdin Swift → JSON Default subcommand is `run`, so `skit Input.swift` works without a verb. CLI parsing moves to swift-argument-parser, replacing the hand-rolled CLIArgs parser. The declarative @Option/@Flag/@Argument surface produces better-formatted help for free and validates inputs at parse time. - Sources/skitrun/ folded into Sources/skit/ (Main.swift → Runner.swift, Helpers + OutputCache unchanged in substance). - skitrun product/target removed from Package.swift; skit gains SwiftSyntax / SwiftParser / Crypto / ArgumentParser deps. - Env vars: SKITRUN_LIB_DIR → SKIT_LIB_DIR; cache-key env prefix SKITRUN_* → SKIT_*. Bundle dir: .build/skitrun-release/ → .build/skit-release/. Homebrew fallback path: lib/skitrun/ → lib/skit/. - Release script moves to Scripts/build-skit-release.sh — promoted from Docs/research/poc-step4-release.sh to reflect that it's now a shipping build script, not a POC step. - Wrapper internal name: __skitrun_root → __skit_root (only visible in spawned-swift error messages on wrapper-line failures). All 10 working Examples/Completed/*/dsl.swift still render through `skit run` (and the default-subcommand form). Co-Authored-By: Claude Opus 4.7 (1M context) --- Package.resolved | 11 +- Package.swift | 16 +- .../build-skit-release.sh | 44 +-- Sources/{skitrun => skit}/Helpers.swift | 2 +- Sources/{skitrun => skit}/OutputCache.swift | 6 +- Sources/{skitrun => skit}/README.md | 0 .../{skitrun/Main.swift => skit/Runner.swift} | 286 +++--------------- Sources/skit/Skit.swift | 217 ++++++++++++- 8 files changed, 286 insertions(+), 296 deletions(-) rename Docs/research/poc-step4-release.sh => Scripts/build-skit-release.sh (63%) rename Sources/{skitrun => skit}/Helpers.swift (99%) rename Sources/{skitrun => skit}/OutputCache.swift (94%) rename Sources/{skitrun => skit}/README.md (100%) rename Sources/{skitrun/Main.swift => skit/Runner.swift} (67%) diff --git a/Package.resolved b/Package.resolved index 851eeb1..07e1898 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,6 +1,15 @@ { - "originHash" : "6a75ee274433215501c77ebca768e3c82c684598296596b112f9800ac08fa2fe", + "originHash" : "36e6466ffd4edf53c9520bb53570c9119c1a34eb0e59dd824b1c4ff4f19189a8", "pins" : [ + { + "identity" : "swift-argument-parser", + "kind" : "remoteSourceControl", + "location" : "https://github.com/apple/swift-argument-parser.git", + "state" : { + "revision" : "626b5b7b2f45e1b0b1c6f4a309296d1d21d7311b", + "version" : "1.7.1" + } + }, { "identity" : "swift-asn1", "kind" : "remoteSourceControl", diff --git a/Package.swift b/Package.swift index 13c44a1..551ba63 100644 --- a/Package.swift +++ b/Package.swift @@ -95,15 +95,12 @@ let package = Package( name: "skit", targets: ["skit"] ), - .executable( - name: "skitrun", - targets: ["skitrun"] - ), ], dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), - .package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0") + .package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0"), + .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0") ], targets: [ .target( @@ -146,15 +143,12 @@ let package = Package( ), .executableTarget( name: "skit", - dependencies: ["SyntaxParser"], - swiftSettings: swiftSettings - ), - .executableTarget( - name: "skitrun", dependencies: [ + "SyntaxParser", .product(name: "SwiftSyntax", package: "swift-syntax"), .product(name: "SwiftParser", package: "swift-syntax"), - .product(name: "Crypto", package: "swift-crypto") + .product(name: "Crypto", package: "swift-crypto"), + .product(name: "ArgumentParser", package: "swift-argument-parser") ], swiftSettings: swiftSettings ), diff --git a/Docs/research/poc-step4-release.sh b/Scripts/build-skit-release.sh similarity index 63% rename from Docs/research/poc-step4-release.sh rename to Scripts/build-skit-release.sh index 795d2a2..78eefe7 100755 --- a/Docs/research/poc-step4-release.sh +++ b/Scripts/build-skit-release.sh @@ -1,28 +1,30 @@ #!/usr/bin/env bash # -# POC step 4: build a self-contained skitrun release bundle. +# Build a self-contained skit release bundle. # -# Output: .build/skitrun-release/ -# skitrun ← the CLI binary +# Output: .build/skit-release/ +# skit ← the CLI binary # lib/ -# libSyntaxKit.dylib ← release + strip -x -# *.swiftmodule ← SyntaxKit + transitively re-exported modules -# _SwiftSyntaxCShims-include/ ← C-shims headers (module map + .h files) +# libSyntaxKit.dylib ← release + strip -x +# *.swiftmodule ← SyntaxKit + transitively re-exported modules +# _SwiftSyntaxCShims-include/ ← C-shims headers (module map + .h files) +# swift-version.txt ← toolchain stamp for startup check # -# Once produced, the binary is portable: copy the whole .build/skitrun-release/ -# directory anywhere, and `./skitrun-release/skitrun ` Just Works — no +# Once produced, the bundle is portable: copy the whole .build/skit-release/ +# directory anywhere, and `./skit-release/skit ` Just Works — no # flags, no env vars, no SyntaxKit checkout required. set -euo pipefail if [[ "$(uname -s)" != "Darwin" ]]; then - echo "macOS-only for now. Linux smoke test is POC step 7." >&2 + echo "macOS-only. Linux uses a parallel flow (build, then strip the" >&2 + echo "Mach-O install_name step)." >&2 exit 1 fi SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -OUTPUT_DIR="$REPO_ROOT/.build/skitrun-release" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +OUTPUT_DIR="$REPO_ROOT/.build/skit-release" PACKAGE_FILE="$REPO_ROOT/Package.swift" PACKAGE_BACKUP="$(mktemp)" @@ -50,10 +52,10 @@ PY cd "$REPO_ROOT" -echo "==> swift build -c release --product skitrun" -swift build -c release --product skitrun +echo "==> swift build -c release --product skit" +swift build -c release --product skit -# skitrun doesn't depend on SyntaxKit (it spawns swift on user input that +# `skit` doesn't depend on SyntaxKit (it spawns swift on user input that # imports SyntaxKit at runtime). Build the library product explicitly so the # .dynamic flip above produces libSyntaxKit.dylib + swiftmodule. echo "==> swift build -c release --product SyntaxKit" @@ -69,7 +71,7 @@ echo "==> Staging $OUTPUT_DIR" rm -rf "$OUTPUT_DIR" mkdir -p "$OUTPUT_DIR/lib" -cp "$BUILD_DIR/skitrun" "$OUTPUT_DIR/skitrun" +cp "$BUILD_DIR/skit" "$OUTPUT_DIR/skit" cp "$BUILD_DIR/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/" strip -x "$OUTPUT_DIR/lib/libSyntaxKit.dylib" cp -r "$BUILD_DIR/Modules/." "$OUTPUT_DIR/lib/" @@ -79,13 +81,13 @@ cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/inclu # Ensure the dylib's install_name uses @rpath so it's portable. install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true -# Stamp the bundle with the build toolchain. skitrun compares this against -# the user's `swift --version` at startup and refuses to spawn `swift` if the -# swiftmodule wouldn't load (see Sources/skitrun/Main.swift). Issue #157 will -# replace the refusal with an auto-rebuild fallback. +# Stamp the bundle with the build toolchain. `skit` compares this against the +# user's `swift --version` at startup and refuses to spawn `swift` if the +# swiftmodule wouldn't load. Issue #157 will replace the refusal with an +# auto-rebuild fallback. swift --version > "$OUTPUT_DIR/lib/swift-version.txt" -BINARY_SIZE=$(ls -lh "$OUTPUT_DIR/skitrun" | awk '{print $5}') +BINARY_SIZE=$(ls -lh "$OUTPUT_DIR/skit" | awk '{print $5}') DYLIB_SIZE=$(ls -lh "$OUTPUT_DIR/lib/libSyntaxKit.dylib" | awk '{print $5}') TOTAL_SIZE=$(du -sh "$OUTPUT_DIR" | awk '{print $1}') @@ -96,4 +98,4 @@ echo " Dylib: $DYLIB_SIZE" echo " Total: $TOTAL_SIZE" echo echo "==> Try it:" -echo " $OUTPUT_DIR/skitrun " +echo " $OUTPUT_DIR/skit run " diff --git a/Sources/skitrun/Helpers.swift b/Sources/skit/Helpers.swift similarity index 99% rename from Sources/skitrun/Helpers.swift rename to Sources/skit/Helpers.swift index 6a87f88..7a33b51 100644 --- a/Sources/skitrun/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -211,7 +211,7 @@ private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) t let stderr = String(decoding: stderrData, as: UTF8.self) throw CLIError( message: """ - skitrun: failed to compile Helpers/ (exit \(process.terminationStatus)) + skit: failed to compile Helpers/ (exit \(process.terminationStatus)) \(stderr) """) } diff --git a/Sources/skitrun/OutputCache.swift b/Sources/skit/OutputCache.swift similarity index 94% rename from Sources/skitrun/OutputCache.swift rename to Sources/skit/OutputCache.swift index 3f094b0..d4a26df 100644 --- a/Sources/skitrun/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -1,6 +1,6 @@ // // OutputCache.swift -// SyntaxKit — skitrun (POC step 6 for issue #154) +// SyntaxKit // // Created by Leo Dion. // Copyright © 2026 BrightDigit. @@ -34,7 +34,7 @@ import Foundation private let outputCacheSchemaVersion = "v1" /// SHA-256 over (cache schema, input source bytes, helpers key, swift version, -/// libSyntaxKit stamp, sorted SKITRUN_*/SYNTAXKIT_* env vars). Any change in +/// libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any change in /// these inputs produces a fresh key and forces a recompile. internal func outputCacheKey( inputSource: String, @@ -60,7 +60,7 @@ internal func outputCacheKey( } let env = ProcessInfo.processInfo.environment - .filter { $0.key.hasPrefix("SKITRUN_") || $0.key.hasPrefix("SYNTAXKIT_") } + .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } .sorted { $0.key < $1.key } for (key, value) in env { hasher.update(data: Data("\(key)=\(value)\0".utf8)) diff --git a/Sources/skitrun/README.md b/Sources/skit/README.md similarity index 100% rename from Sources/skitrun/README.md rename to Sources/skit/README.md diff --git a/Sources/skitrun/Main.swift b/Sources/skit/Runner.swift similarity index 67% rename from Sources/skitrun/Main.swift rename to Sources/skit/Runner.swift index 5accbd6..957df18 100644 --- a/Sources/skitrun/Main.swift +++ b/Sources/skit/Runner.swift @@ -1,5 +1,5 @@ // -// Main.swift +// Runner.swift // SyntaxKit // // Created by Leo Dion. @@ -31,74 +31,15 @@ import Foundation import SwiftParser import SwiftSyntax -@main -internal enum SkitRun { - internal static func main() async throws { - let args: CLIArgs - do { - args = try CLIArgs.parse(CommandLine.arguments) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - exit(2) - } - - let libPath: String - do { - libPath = try resolveLibPath(override: args.libPath) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - exit(2) - } - - if args.checkToolchain { - switch toolchainCheck(libPath: libPath) { - case .match, .stampMissing: - break - case .mismatch(let bundle, let local): - FileHandle.standardError.write(Data(toolchainMismatchMessage( - bundle: bundle, local: local - ).utf8)) - exit(2) - } - } +// MARK: - Helpers resolution - switch args.mode { - case .singleFile(let input, let output): - let helpers = try resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: args.helpers - ) - try runSingleFile( - inputPath: input, - outputPath: output, - libPath: libPath, - helpers: helpers, - useCache: args.useCache, - timeoutSeconds: args.timeoutSeconds - ) - case .directory(let inputDir, let outputDir): - let helpers = try resolveHelpers( - nearInputPath: inputDir, - libPath: libPath, - options: args.helpers - ) - let exitCode = await runDirectory( - inputDir: inputDir, - outputDir: outputDir, - libPath: libPath, - helpers: helpers, - useCache: args.useCache, - timeoutSeconds: args.timeoutSeconds - ) - exit(exitCode) - } - } +internal enum HelpersOptions { + case auto + case disabled + case explicit(String) } -// MARK: - Helpers resolution - -private func resolveHelpers( +internal func resolveHelpers( nearInputPath path: String, libPath: String, options: HelpersOptions @@ -127,7 +68,7 @@ private func resolveHelpers( let suffix = compiled.cacheHit ? "cached" : "compiled" FileHandle.standardError.write( Data( - "skitrun: helpers \(suffix) at \(helpersDir.path)\n".utf8 + "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 )) return compiled } @@ -136,7 +77,7 @@ private func resolveHelpers( /// Resolves the directory containing `libSyntaxKit.dylib` + module files, /// in priority order: explicit flag → env var → adjacent-to-binary -/// (`/lib/`) → Homebrew layout (`/../lib/skitrun/`). +/// (`/lib/`) → Homebrew layout (`/../lib/skit/`). internal func resolveLibPath(override: String?) throws -> String { if let override { guard isLibDir(override) else { @@ -145,9 +86,9 @@ internal func resolveLibPath(override: String?) throws -> String { return override } - if let env = ProcessInfo.processInfo.environment["SKITRUN_LIB_DIR"], !env.isEmpty { + if let env = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"], !env.isEmpty { guard isLibDir(env) else { - throw CLIError(message: "SKITRUN_LIB_DIR is set but path is not a lib dir: \(env)") + throw CLIError(message: "SKIT_LIB_DIR is set but path is not a lib dir: \(env)") } return env } @@ -159,7 +100,7 @@ internal func resolveLibPath(override: String?) throws -> String { if isLibDir(adjacent) { return adjacent } let brewLayout = execDir.deletingLastPathComponent() - .appendingPathComponent("lib/skitrun").path + .appendingPathComponent("lib/skit").path if isLibDir(brewLayout) { return brewLayout } } @@ -167,11 +108,11 @@ internal func resolveLibPath(override: String?) throws -> String { message: """ Could not locate SyntaxKit lib directory. Looked for: 1. --lib (not provided) - 2. $SKITRUN_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skitrun/ (not found) - Run Docs/research/poc-step4-release.sh to produce a self-contained - release bundle under .build/skitrun-release/. + 2. $SKIT_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skit/ (not found) + Run Scripts/build-skit-release.sh to produce a self-contained + release bundle under .build/skit-release/. """) } @@ -191,15 +132,15 @@ internal enum ToolchainCheckResult { /// Bundle stamp matches the local `swift --version` exactly. case match /// `/swift-version.txt` is missing (older bundle that predates - /// the stamp). skitrun prints a one-line note and proceeds. + /// the stamp). skit prints a one-line note and proceeds. case stampMissing case mismatch(bundle: String, local: String) } /// Compares `/swift-version.txt` to `captureSwiftVersion()`. /// The swiftmodule format isn't reliably forward-compatible across even -/// patch-level Swift releases (the originating bug: 6.3.0 → 6.3.2 rejected -/// the swiftmodule), so the comparison is exact-string after normalising +/// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the +/// swiftmodule), so the comparison is exact-string after normalising /// trailing whitespace. internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) @@ -207,13 +148,13 @@ internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { let stampRaw = String(data: stampData, encoding: .utf8) else { FileHandle.standardError.write( - Data("skitrun: bundle has no toolchain stamp; skipping check\n".utf8) + Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) ) return .stampMissing } guard let localRaw = captureSwiftVersion() else { FileHandle.standardError.write( - Data("skitrun: could not capture local `swift --version`; skipping toolchain check\n".utf8) + Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) ) return .stampMissing } @@ -224,7 +165,7 @@ internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { internal func toolchainMismatchMessage(bundle: String, local: String) -> String { """ - skitrun: toolchain mismatch + skit: toolchain mismatch bundle: \(bundle) local: \(local) The bundle's libSyntaxKit was built against a different `swift` than the @@ -233,7 +174,7 @@ internal func toolchainMismatchMessage(bundle: String, local: String) -> String diagnostic. Rebuild the bundle with: - Docs/research/poc-step4-release.sh + Scripts/build-skit-release.sh Or pass --no-toolchain-check to try anyway. """ @@ -241,7 +182,7 @@ internal func toolchainMismatchMessage(bundle: String, local: String) -> String // MARK: - Single-file mode -private func runSingleFile( +internal func runSingleFile( inputPath: String, outputPath: String?, libPath: String, @@ -271,7 +212,7 @@ private func runSingleFile( // MARK: - Folder mode -private func runDirectory( +internal func runDirectory( inputDir: String, outputDir: String, libPath: String, @@ -286,12 +227,12 @@ private func runDirectory( do { inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) } catch { - FileHandle.standardError.write(Data("skitrun: failed to walk \(inputDir): \(error)\n".utf8)) + FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) return 1 } if inputs.isEmpty { - FileHandle.standardError.write(Data("skitrun: no .swift inputs under \(inputDir)\n".utf8)) + FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) return 0 } @@ -314,11 +255,11 @@ private func runDirectory( outcomes.append(outcome) if let next = iterator.next() { group.addTask { - runOne( - next, libPath: libPath, helpers: helpers, - useCache: useCache, timeoutSeconds: timeoutSeconds - ) - } + runOne( + next, libPath: libPath, helpers: helpers, + useCache: useCache, timeoutSeconds: timeoutSeconds + ) + } } } } @@ -358,7 +299,7 @@ private func runDirectory( FileHandle.standardError.write( Data( - "skitrun: \(outcomes.count - failed)/\(outcomes.count) succeeded\n".utf8 + "skit: \(outcomes.count - failed)/\(outcomes.count) succeeded\n".utf8 )) return failed == 0 ? 0 : 1 @@ -462,7 +403,7 @@ private func processFile( let wrapped = wrap(source: source, originalPath: absoluteInputPath) let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skitrun-\(UUID().uuidString)") + .appendingPathComponent("skit-\(UUID().uuidString)") try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) defer { try? FileManager.default.removeItem(at: tmpDir) } @@ -490,155 +431,6 @@ private func processFile( return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } -// MARK: - Arg parsing - -internal enum HelpersOptions { - case auto - case disabled - case explicit(String) -} - -private struct CLIArgs { - enum Mode { - case singleFile(input: String, output: String?) - case directory(input: String, output: String) - } - - let mode: Mode - let libPath: String? - let helpers: HelpersOptions - let useCache: Bool - let timeoutSeconds: Int - let checkToolchain: Bool - - static let defaultTimeoutSeconds = 60 - - static func parse(_ argv: [String]) throws -> CLIArgs { - var inputPath: String? - var outputPath: String? - var libPath: String? - var helpers: HelpersOptions = .auto - var useCache = true - var timeoutSeconds = defaultTimeoutSeconds - var checkToolchain = true - - var i = 1 - while i < argv.count { - let arg = argv[i] - switch arg { - case "-o", "--output": - guard i + 1 < argv.count else { throw usage("-o requires a value") } - outputPath = argv[i + 1] - i += 2 - case "--lib": - guard i + 1 < argv.count else { throw usage("--lib requires a value") } - libPath = argv[i + 1] - i += 2 - case "--helpers": - guard i + 1 < argv.count else { throw usage("--helpers requires a value") } - helpers = .explicit(argv[i + 1]) - i += 2 - case "--no-helpers": - helpers = .disabled - i += 1 - case "--no-cache": - useCache = false - i += 1 - case "--no-toolchain-check": - checkToolchain = false - i += 1 - case "--timeout": - guard i + 1 < argv.count else { throw usage("--timeout requires a value") } - guard let parsed = Int(argv[i + 1]), parsed >= 0 else { - throw usage("--timeout expects a non-negative integer (seconds), got: \(argv[i + 1])") - } - timeoutSeconds = parsed - i += 2 - case "-h", "--help": - FileHandle.standardError.write(Data(helpText.utf8)) - exit(0) - case _ where arg.hasPrefix("-"): - throw usage("unknown flag: \(arg)") - default: - guard inputPath == nil else { throw usage("only one input path is supported") } - inputPath = arg - i += 1 - } - } - - guard let inputPath else { throw usage("missing input path") } - - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: inputPath, isDirectory: &isDirectory) else { - throw usage("input does not exist: \(inputPath)") - } - - let mode: Mode - if isDirectory.boolValue { - guard let outputPath else { - throw usage("directory inputs require -o ") - } - mode = .directory(input: inputPath, output: outputPath) - } else { - mode = .singleFile(input: inputPath, output: outputPath) - } - - return CLIArgs( - mode: mode, - libPath: libPath, - helpers: helpers, - useCache: useCache, - timeoutSeconds: timeoutSeconds, - checkToolchain: checkToolchain - ) - } -} - -private let helpText = """ - skitrun [-o ] [--lib ] - - POC for issue #154 — runs SyntaxKit DSL input(s) by wrapping each in a - Group { … } closure and spawning `swift`. - - Forms: - skitrun Input.swift — render to stdout - skitrun Input.swift -o Out.swift — render to a file - skitrun InputDir/ -o OutDir/ — walk **/*.swift (skipping files - prefixed with '_') and mirror - rendered output into OutDir/ - - Options: - -o, --output Output file (single-file mode) or directory (folder mode). - --lib Directory containing libSyntaxKit.dylib + module files. - When omitted, skitrun searches: $SKITRUN_LIB_DIR, - then /lib/, then /../lib/skitrun/. - Build a self-contained bundle with - Docs/research/poc-step4-release.sh. - --helpers Override Helpers/ directory location. By default, - skitrun walks up from the input looking for one. - Compiled into libSyntaxKitHelpers.dylib and made - importable via `import SyntaxKitHelpers`. - --no-helpers Skip helpers discovery entirely. - --no-cache Skip the rendered-output cache (always run swift). - The cache lives at /outputs// - and is keyed on input bytes, helpers, swift version, - libSyntaxKit stamp, and SKITRUN_*/SYNTAXKIT_* env. - --timeout Per-input timeout for the spawned `swift` process - (default 60). On expiry: SIGTERM, then SIGKILL after - a 5s grace; the file exits with code 124. Pass 0 to - disable the watchdog. - --no-toolchain-check Skip the startup check that compares the bundle's - recorded build toolchain (/swift-version.txt) - against `swift --version`. Swift swiftmodules aren't - reliably compatible across compiler versions, so by - default skitrun refuses to spawn `swift` on - mismatch. See issue #157 for the auto-rebuild plan. - """ - -private func usage(_ message: String) -> CLIError { - CLIError(message: "\(message)\n\n\(helpText)\n") -} - internal struct CLIError: Error, CustomStringConvertible { let message: String var description: String { message } @@ -695,19 +487,19 @@ internal func wrap(source: String, originalPath: String) -> String { return """ import SyntaxKit \(hoistedBlock) - let __skitrun_root = Group { + let __skit_root = Group { #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) \(body) #sourceLocation() } - print(__skitrun_root.generateCode()) + print(__skit_root.generateCode()) """ } // MARK: - Spawning swift -/// Exit code returned when the spawned `swift` is killed by skitrun's timeout +/// Exit code returned when the spawned `swift` is killed by skit's timeout /// watchdog. Matches POSIX `timeout(1)`. private let timeoutExitCode: Int32 = 124 @@ -802,7 +594,7 @@ private func runSwift( group.wait() if timedOut { - let prefix = Data("skitrun: timed out after \(timeoutSeconds)s\n".utf8) + let prefix = Data("skit: timed out after \(timeoutSeconds)s\n".utf8) let stderr = String(decoding: prefix + errBox.value, as: UTF8.self) return ProcessResult(exitCode: timeoutExitCode, stdout: outBox.value, stderr: stderr) } diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index 1066b5e..710a725 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -27,24 +27,217 @@ // OTHER DEALINGS IN THE SOFTWARE. // +import ArgumentParser import Foundation import SyntaxParser @main -internal enum Skit { - internal static func main() throws { - // Read Swift code from stdin - let code = String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" +internal struct Skit: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "skit", + abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", + subcommands: [Run.self, Parse.self], + defaultSubcommand: Run.self + ) +} + +// MARK: - skit run + +extension Skit { + internal struct Run: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "run", + abstract: "Render SyntaxKit DSL input(s) into Swift source.", + discussion: """ + Wraps each input in a `Group { … }` closure and spawns `swift` to + evaluate it. The rendered output is written to stdout (single-file + mode) or mirrored into an output directory (folder mode). + + Forms: + skit run Input.swift — render to stdout + skit run Input.swift -o Out.swift — render to a file + skit run InputDir/ -o OutDir/ — walk **/*.swift (skipping + files prefixed with '_') + and mirror rendered output + into OutDir/ + """ + ) + + @Argument(help: "Path to a .swift input file or a directory of inputs.") + internal var input: String + + @Option( + name: [.short, .customLong("output")], + help: "Output file (single-file mode) or directory (folder mode)." + ) + internal var output: String? + + @Option( + name: .customLong("lib"), + help: ArgumentHelp( + "Directory containing libSyntaxKit.dylib + module files.", + discussion: """ + When omitted, skit searches: $SKIT_LIB_DIR, then /lib/, + then /../lib/skit/. Build a self-contained bundle with + Scripts/build-skit-release.sh. + """ + ) + ) + internal var libPath: String? + + @Option( + name: .customLong("helpers"), + help: ArgumentHelp( + "Override Helpers/ directory location.", + discussion: """ + By default skit walks up from the input looking for one. Helper + sources are pre-compiled into libSyntaxKitHelpers.dylib and made + importable via `import SyntaxKitHelpers`. + """ + ) + ) + internal var helpersDir: String? + + @Flag( + name: .customLong("no-helpers"), + help: "Skip helpers discovery entirely." + ) + internal var noHelpers: Bool = false + + @Flag( + name: .customLong("no-cache"), + help: ArgumentHelp( + "Skip the rendered-output cache (always run swift).", + discussion: """ + The cache lives at /outputs// and is keyed + on input bytes, helpers, swift version, libSyntaxKit stamp, and + SKIT_*/SYNTAXKIT_* env. + """ + ) + ) + internal var noCache: Bool = false + + @Option( + name: .customLong("timeout"), + help: ArgumentHelp( + "Per-input timeout for the spawned `swift` (seconds).", + discussion: """ + Default 60. On expiry: SIGTERM, then SIGKILL after a 5s grace; the + file exits with code 124. Pass 0 to disable the watchdog. + """ + ) + ) + internal var timeoutSeconds: Int = 60 + + @Flag( + name: .customLong("no-toolchain-check"), + help: ArgumentHelp( + "Skip the bundle/local Swift-toolchain comparison.", + discussion: """ + skit compares /swift-version.txt to `swift --version` at + startup and refuses to spawn `swift` on mismatch — swiftmodules + aren't reliably compatible across compiler versions. See issue + #157 for the auto-rebuild plan. + """ + ) + ) + internal var noToolchainCheck: Bool = false + + internal func validate() throws { + guard timeoutSeconds >= 0 else { + throw ValidationError( + "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + ) + } + } + + internal func run() async throws { + let libPath: String + do { + libPath = try resolveLibPath(override: self.libPath) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) + throw ExitCode(2) + } + + if !noToolchainCheck { + switch toolchainCheck(libPath: libPath) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + FileHandle.standardError.write( + Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) + throw ExitCode(2) + } + } + + let helpersOptions: HelpersOptions + if noHelpers { + helpersOptions = .disabled + } else if let dir = helpersDir { + helpersOptions = .explicit(dir) + } else { + helpersOptions = .auto + } + + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + throw ValidationError("input does not exist: \(input)") + } + + if isDirectory.boolValue { + guard let output else { + throw ValidationError("directory inputs require -o ") + } + let helpers = try resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + let exitCode = await runDirectory( + inputDir: input, + outputDir: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + throw ExitCode(exitCode) + } else { + let helpers = try resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + try runSingleFile( + inputPath: input, + outputPath: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + } + } + } +} - // Parse the code using SyntaxKit - let treeNodes = SyntaxParser.parse(code: code) +// MARK: - skit parse - // Convert to JSON for output - let encoder = JSONEncoder() - let data = try encoder.encode(treeNodes) - let json = String(decoding: data, as: UTF8.self) +extension Skit { + internal struct Parse: ParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "parse", + abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." + ) - // Output the JSON - print(json) + internal func run() throws { + let code = String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" + let treeNodes = SyntaxParser.parse(code: code) + let encoder = JSONEncoder() + let data = try encoder.encode(treeNodes) + let json = String(decoding: data, as: UTF8.self) + print(json) + } } } From f3a1912f25784beac352514f12887c281b3e9116 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 15:22:10 -0400 Subject: [PATCH 20/56] Productize Sources/skit/README.md for v0.0.5 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop the "research POC for issue #154" framing. Replace `skitrun` with `skit run`, env var prefixes with SKIT_*, and the release-script path with Scripts/build-skit-release.sh. The "Open scope decisions" section is gone — items either shipped (timeout) or have follow-up issues (#157 auto-rebuild, #158 if-in-Group), and the rest are explicitly out-of-scope. Point at Docs/skit.md for the deeper dive (forthcoming). Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skit/README.md | 80 +++++++++++++++++++----------------------- 1 file changed, 37 insertions(+), 43 deletions(-) diff --git a/Sources/skit/README.md b/Sources/skit/README.md index 52f64fc..5b0f77a 100644 --- a/Sources/skit/README.md +++ b/Sources/skit/README.md @@ -1,22 +1,22 @@ -# skitrun +# skit -> **Status:** research POC for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). Shape may change; do not pin tooling to it yet. Design lives at [`Docs/research/codegen-cli-design.md`](../../Docs/research/codegen-cli-design.md); the 7-step build-up is documented step-by-step under [`Docs/research/poc-step{1..7}-results.md`](../../Docs/research/). - -A CLI that takes a *pure SyntaxKit DSL* input file, wraps it in a `Group { … }` closure, spawns `swift` to evaluate it, and writes the rendered Swift source to stdout (or a file). No `print`, no `@main`, no boilerplate in your input — just DSL expressions. +A CLI for SyntaxKit. Two verbs: ``` -skitrun Input.swift # render to stdout -skitrun Input.swift -o Out.swift # render to a file -skitrun InputDir/ -o OutDir/ # walk **/*.swift, mirror to OutDir/ +skit run Input.swift # render a SyntaxKit DSL file into Swift source +skit run Input.swift -o Out.swift +skit run InputDir/ -o OutDir/ # walk **/*.swift and mirror rendered output +skit parse < Input.swift # parse Swift source into a JSON syntax tree ``` +`run` is the default subcommand, so `skit Input.swift` is shorthand for `skit run Input.swift`. + ## Quick start ```bash -# Build a portable bundle (the script flips the SyntaxKit library to -# .dynamic, then bundles dylib + modules + C-shims headers next to skitrun). -Docs/research/poc-step4-release.sh -# → .build/skitrun-release/{skitrun, lib/} +# Build a self-contained release bundle (binary + dylib + swiftmodules). +Scripts/build-skit-release.sh +# → .build/skit-release/{skit, lib/} cat > /tmp/Person.swift <<'SWIFT' Struct("Person") { @@ -25,14 +25,14 @@ Struct("Person") { } SWIFT -.build/skitrun-release/skitrun /tmp/Person.swift +.build/skit-release/skit /tmp/Person.swift ``` -The bundle is self-contained: `cp -r .build/skitrun-release ~/anywhere/` and `~/anywhere/skitrun-release/skitrun ` works zero-config. +The bundle is portable: `cp -r .build/skit-release ~/anywhere/` and `~/anywhere/skit-release/skit ` works zero-config. ## Input file shape -Top-level expressions form an implicit `@CodeBlockBuilder` body. `import` declarations at the top are hoisted into the wrapper. Anything else (`Struct(…)`, `Enum(…)`, helper calls, …) becomes the builder's content. +`skit run` wraps each input in an implicit `Group { … }` builder. Top-level expressions become the builder's content; `import` declarations at the top are hoisted into the wrapper. ```swift // Models.swift @@ -48,7 +48,7 @@ What *won't* work inside the input: top-level `let`/`var` outside the builder DS ## Helpers -Shared codegen utilities live in a `Helpers/` directory anywhere up-tree from the input. `skitrun` walks up from the input file (or directory) looking for one. Sources are pre-compiled into `libSyntaxKitHelpers.{dylib,so}` once and cached by content hash: +Shared codegen utilities live in a `Helpers/` directory anywhere up-tree from the input. `skit` walks up from the input file (or directory) looking for one. Sources are pre-compiled into `libSyntaxKitHelpers.{dylib,so}` once and cached by content hash: ``` project/ @@ -65,41 +65,35 @@ Force-disable: `--no-helpers`. Override location: `--helpers `. ## Caches -Two layers, both keyed on content + toolchain + dylib stamp + `SKITRUN_*`/`SYNTAXKIT_*` env vars. Live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. +Two layers, both keyed on content + toolchain + dylib stamp + `SKIT_*`/`SYNTAXKIT_*` env vars. Live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. -| Layer | Path | What it skips on hit | -| --- | --- | --- | -| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | -| Output | `outputs//output.swift` | the `swift` spawn for an input | +| Layer | Path | What it skips on hit | +| ------- | ----------------------------- | --------------------------------------------- | +| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | +| Output | `outputs//output.swift` | the `swift` spawn for an input | -Output cache hit ≈ 0.14s on macOS (no spawn at all); cold miss matches the warm `swift` script-mode baseline (~0.5s). Force a miss with `--no-cache`. +Output cache hit is roughly ~0.14s on macOS (no spawn at all); cold miss matches the warm `swift` script-mode baseline (~0.5s). Force a miss with `--no-cache`. -## Flag reference +## Flag reference (`skit run`) -| Flag | Default | Meaning | -| --- | --- | --- | -| `-o, --output ` | stdout | Output file (single-file mode) or directory (folder mode). | -| `--lib ` | auto | Directory containing `libSyntaxKit.{dylib,so}` + module files. Search order when omitted: `$SKITRUN_LIB_DIR` → `/lib/` → `/../lib/skitrun/`. | -| `--helpers ` | walk-up | Explicit `Helpers/` directory. | -| `--no-helpers` | (off) | Skip helpers discovery entirely. | -| `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | -| `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | -| `--no-toolchain-check` | (off) | Skip the startup check that compares the bundle's recorded build toolchain (`lib/swift-version.txt`) to `swift --version`. swiftmodules aren't reliably compatible across compiler versions; on mismatch skitrun refuses to spawn `swift` and points at the rebuild script. Auto-rebuild fallback tracked in [#157](https://github.com/brightdigit/SyntaxKit/issues/157). | +| Flag | Default | Meaning | +| ----------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `-o, --output ` | stdout | Output file (single-file mode) or directory (folder mode). | +| `--lib ` | auto | Directory containing `libSyntaxKit.{dylib,so}` + module files. Search order when omitted: `$SKIT_LIB_DIR` → `/lib/` → `/../lib/skit/`. | +| `--helpers ` | walk-up | Explicit `Helpers/` directory. | +| `--no-helpers` | (off) | Skip helpers discovery entirely. | +| `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | +| `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | +| `--no-toolchain-check` | (off) | Skip the startup check that compares `lib/swift-version.txt` to `swift --version`. swiftmodules aren't reliably compatible across compiler versions; on mismatch skit refuses to spawn `swift` and points at the rebuild script. Auto-rebuild fallback tracked in [#157](https://github.com/brightdigit/SyntaxKit/issues/157). | ## Platform notes -- **macOS:** primary target. All seven POC steps run via the scripts in `Docs/research/`. -- **Linux:** verified in `swift:6.0-jammy/aarch64` via [`Docs/research/poc-step7.sh`](../../Docs/research/poc-step7.sh) (self-reruns inside Docker). Requires `swift-crypto` instead of CryptoKit; install-name flag is Mach-O specific and skipped on Linux. -- **Windows:** not attempted. - -A known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. Workaround in `Helpers.swift` / `Main.swift`: `terminationHandler` + `DispatchSemaphore`. See [`poc-step7-results.md`](../../Docs/research/poc-step7-results.md) for the full reproducer. +- **macOS** — primary target. All build/release/test flows in `Scripts/`. +- **Linux** — verified on `swift:6.0-jammy/aarch64`. Requires `swift-crypto` instead of CryptoKit (we depend on it). The Mach-O `install_name` step in `Scripts/build-skit-release.sh` is macOS-specific and skipped on Linux. +- **Windows** — not supported. -## Open scope decisions +Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. `Runner.swift` and `Helpers.swift` work around it with `terminationHandler` + `DispatchSemaphore`. -Not blocking but on the table — see [`codegen-cli-design.md` §7](../../Docs/research/codegen-cli-design.md#7-what-we-still-need-to-verify): +## Deeper dive -- Timeouts on the child `swift` process (60s default + SIGTERM/SIGKILL grace). -- `@main` / attribute behavior in `swift` script-mode beyond the simple cases tested. -- Multi-file outputs from a single input (out of scope for v1). -- Sandboxing (out of scope; threat model = "you ran your own code"). -- HTTP/server form for warm-interpreter reuse (post-CLI follow-up). +For the architecture, design decisions, and trade-offs see [`Docs/skit.md`](../../Docs/skit.md). From 05e8502e538bd2aca32c26a0833b73342d9e8a9f Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 15:27:25 -0400 Subject: [PATCH 21/56] Replace POC research log with Docs/skit.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The chronological poc-step{1..7}-results.md log and the codegen-cli-design.md sketch were the trail by which `skit` reached its current shape — useful at review time, noise for ongoing maintenance. Their substance is now folded into a single forward-looking explainer at Docs/skit.md (the architecture, caches, toolchain stamp, timeout, sharp edges, deferred items, references to #157/#158). Docs/research/tuist-manifest-pipeline.md stays — it's reference material for the manifest-pipeline pattern, not a step log. Git history retains the deleted files for anyone wanting to trace how a particular decision came about. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/research/codegen-cli-design.md | 160 --------------------- Docs/research/poc-step1-results.md | 138 ------------------ Docs/research/poc-step1.sh | 117 ---------------- Docs/research/poc-step2-results.md | 70 ---------- Docs/research/poc-step3-results.md | 48 ------- Docs/research/poc-step4-results.md | 55 -------- Docs/research/poc-step5-results.md | 58 -------- Docs/research/poc-step5.sh | 135 ------------------ Docs/research/poc-step6-results.md | 58 -------- Docs/research/poc-step6.sh | 116 --------------- Docs/research/poc-step7-results.md | 45 ------ Docs/research/poc-step7.sh | 135 ------------------ Docs/skit.md | 209 ++++++++++++++++++++++++++++ 13 files changed, 209 insertions(+), 1135 deletions(-) delete mode 100644 Docs/research/codegen-cli-design.md delete mode 100644 Docs/research/poc-step1-results.md delete mode 100755 Docs/research/poc-step1.sh delete mode 100644 Docs/research/poc-step2-results.md delete mode 100644 Docs/research/poc-step3-results.md delete mode 100644 Docs/research/poc-step4-results.md delete mode 100644 Docs/research/poc-step5-results.md delete mode 100755 Docs/research/poc-step5.sh delete mode 100644 Docs/research/poc-step6-results.md delete mode 100755 Docs/research/poc-step6.sh delete mode 100644 Docs/research/poc-step7-results.md delete mode 100755 Docs/research/poc-step7.sh create mode 100644 Docs/skit.md diff --git a/Docs/research/codegen-cli-design.md b/Docs/research/codegen-cli-design.md deleted file mode 100644 index 601b4e0..0000000 --- a/Docs/research/codegen-cli-design.md +++ /dev/null @@ -1,160 +0,0 @@ -# Design sketch: a CLI for SyntaxKit-driven codegen - -> Phase 2 deliverable for [issue #154](https://github.com/brightdigit/SyntaxKit/issues/154). Builds on [`tuist-manifest-pipeline.md`](./tuist-manifest-pipeline.md). Nothing here is implemented — this is the design we'd validate with the POC in §6. - -## 1. What we're borrowing from Tuist — and what we're not - -From Phase 1, Tuist's manifest pipeline reduces to four moving parts: - -1. A **public DSL framework** that ships next to the CLI binary (`ProjectDescription.framework`). -2. A **script runner** that invokes `xcrun swift ` with `-I/-L/-F` pointing at that framework, captures stdout, and slices out a token-delimited payload. -3. A **helpers compiler** that pre-builds `Tuist/ProjectDescriptionHelpers/*.swift` into a sibling dylib so manifests can `import ProjectDescriptionHelpers`. -4. A **two-tier cache** (helpers module + decoded manifest) keyed on source hashes + toolchain/tool versions. - -We borrow (1), (2), (3), and (4). We **don't** borrow Tuist's "manifest" framing — no `Project.swift`-style wrapper, no `Output(...)` value, no token-delimited stdout payload. Tuist needs the wrapper because its host has to re-interpret the description into an `xcodeproj`. SyntaxKit doesn't: the input file is *pure DSL* — a series of `CodeBlock` expressions — and the CLI generates the boilerplate that turns it into a runnable Swift program. - -## 2. CLI shape - -The CLI is `stdin → stdout`-shaped, with a SyntaxKit-aware `swift` invocation as the engine: - -``` -syntaxkit run Input.swift # rendered Swift source to stdout -syntaxkit run Input.swift -o Output.swift # write to a file (atomic) -syntaxkit run InputDir/ -o OutputDir/ # walk InputDir/*.swift, mirror paths into OutputDir/ -``` - -**Input file:** pure DSL. A series of `CodeBlock` expressions, optionally preceded by `import` declarations. No `print`, no `@main`, no boilerplate. Example: - -```swift -// Person.swift -import SyntaxKit // optional — only needed for IDE / autocomplete - -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") -} -``` - -That's the entire file. Top-level expressions form an implicit `@CodeBlockBuilder` body that the CLI wraps for execution — see §3. - -**Output:** the rendered Swift source produced by `generateCode()`. The CLI does not reshape it. - -**Stderr:** forwarded to the user's terminal. The captured output is stdout-only. (The wrapper writes to stdout via a single `print`; if helpers or the user's DSL want to log debug info, they should use `FileHandle.standardError.write(...)`.) - -**Folder mode:** when the input is a directory, the CLI walks `**/*.swift` and produces a parallel tree of outputs (one input file → one output file, mirrored relative path). Files starting with `_` are skipped (convention for shared helpers — see §4). - -**Exit codes:** child `swift` non-zero → CLI non-zero with stderr preserved. No retries. - -## 3. Process model: wrap, then spawn - -Because the input is pure DSL rather than a runnable Swift program, the CLI does a tiny **wrap** step before spawning `swift`. - -**Wrap.** Read the input file. Use SwiftSyntax (already a dep — `Docs/SwiftSyntax-LLM.md`) to split the top of the file into (a) any leading `import` declarations and (b) the remaining body. Generate a temporary `Input.wrapped.swift`: - -```swift -import SyntaxKit - - -let __syntaxkit_root = Group { - -} -print(__syntaxkit_root.generateCode()) -``` - -`Group` (`Sources/SyntaxKit/Utilities/Group.swift`) already uses `@CodeBlockBuilder`, so its closure body accepts a series of `CodeBlock` expressions exactly the way the user wrote them. `import SyntaxKit` is always injected; duplicates from the input are harmless. - -**Spawn.** Adopt Tuist's model `(b)` from Phase 1 §2 — `swift` in script mode against the wrapped file, pipe through: - -``` -/usr/bin/env swift \ - -suppress-warnings \ - -I \ - -L \ - -F \ - -lSyntaxKit -framework SyntaxKit \ - -Xcc -I -Xcc \ - -Xlinker -rpath -Xlinker \ - -I … -L … -F … -l # optional, when helpers exist - /Input.wrapped.swift -``` - -The `-Xcc -I -Xcc <…>` flag is non-obvious but required (POC step 1 finding): SyntaxKit transitively depends on `_SwiftSyntaxCShims` whose module map lives in `swift-syntax/Sources/_SwiftSyntaxCShims/include/`. The bundled-binary release must ship this header directory alongside the dylib and the SwiftSyntax `.swiftmodule` files. Without the flag, the script compile fails with `missing required module '_SwiftSyntaxCShims'`. The `-Xlinker -rpath -Xlinker <…>` flag tells the just-built script's dylib loader where to find `libSyntaxKit.dylib` at runtime. - -No `--syntaxkit-dump` flag, no start/end tokens. The child's stdout is the rendered Swift source verbatim. The CLI's job is wrap → `Process` → capture stdout → atomic write to destination → clean up temp wrapper. - -Use `/usr/bin/env swift` rather than `/usr/bin/xcrun swift` so the same code path runs on Linux. `xcrun` is mac-only and is implicit when `env swift` resolves to Xcode's swift on macOS. - -**Why wrap instead of requiring `print()` in the input.** Two reasons. First, the user's authoring surface is *just* DSL — declarative, no I/O verbs. Second, error reporting: when the child `swift` reports a diagnostic at `Input.wrapped.swift:42`, the CLI can map that line back to the original `Input.swift` (the wrapper is line-faithful aside from a known prefix offset) and rewrite the path in stderr before forwarding. - -## 4. Helpers - -Same mechanism as Tuist (Phase 1 §4), folder name TBD — `Helpers/` adjacent to the input file or input directory is the obvious choice. The CLI walks up from the input path looking for a `Helpers/` directory, globs `**/*.swift` (excluding files prefixed with `_` to allow private helpers within helpers), and pre-compiles them into `lib.dylib` via: - -``` -swiftc -module-name SyntaxKitHelpers \ - -emit-module -emit-module-path /SyntaxKitHelpers.swiftmodule \ - -parse-as-library -emit-library \ - -suppress-warnings \ - -I … -L … -F … -lSyntaxKit -framework SyntaxKit \ - Helpers/**/*.swift -``` - -The output dylib is then added to the input script's invocation via `-I/-L/-F/-l`. Scripts can `import SyntaxKitHelpers` and use shared codegen utilities. - -Compile into a `tmp..` staging directory and atomic-rename into the cache path, mirroring `ProjectDescriptionHelpersBuilder.swift:204-244` — concurrent CLI invocations need to be safe. - -## 5. Caching - -Two layers, both mirroring Tuist (Phase 1 §5). - -**Helpers cache.** Keyed by: - -| Field | Source | -| --- | --- | -| per-file SHA-256s | `Helpers/**/*.swift`, sorted | -| `syntaxkitVersion` | bundled SyntaxKit dylib version | -| `swiftlangVersion` | `swift --version` | -| `osVersion` | `uname -r` (macOS or Linux) | -| `cacheSchemaVersion` | bumped on layout changes | - -Hash → directory name → reuse if present. - -**Output cache.** Skip the swift spawn entirely when nothing has changed. Keyed by: - -| Field | Source | -| --- | --- | -| `inputHash` | SHA-256 of the input `.swift` file | -| `helpersHash` | the helpers cache key above, or empty | -| `syntaxkitVersion` | bundled SyntaxKit dylib version | -| `swiftlangVersion` | `swift --version` | -| `envHash` | md5 of `SYNTAXKIT_*` env vars | -| `cacheSchemaVersion` | bumped on layout changes | - -On hit, copy the cached rendered output directly to the destination — no `swift` spawn. On miss, run and re-cache. - -Cache location: `~/.cache/syntaxkit/` on Linux, `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS (XDG-aware via `XDG_CACHE_HOME`). - -## 6. Smallest possible proof-of-concept steps - -Each step is independently shippable and de-risks the next. - -1. **Hand-driven wrap + spawn.** Hand-write a pure-DSL `Input.swift` and a hand-rolled `Input.wrapped.swift` that imports SyntaxKit, splices the body into `Group { … }`, and prints `generateCode()`. Invoke it manually with `swift Input.wrapped.swift -I … -L … -F … -lSyntaxKit -framework SyntaxKit` against a local `swift build` of SyntaxKit. **Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit at all, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost.** Cold-start is the single biggest risk to retire — if it's 20+ seconds, the design needs rethinking. -2. **CLI subcommand for single-file mode.** Add `syntaxkit run `: parse out top-level imports with SwiftSyntax, write `Input.wrapped.swift` to a temp dir, spawn `swift` via `Foundation.Process`, capture stdout, write to `-o ` (or stdout if no `-o`). Stderr is forwarded; rewrite `Input.wrapped.swift:LINE` references back to `Input.swift:LINE` before forwarding. -3. **Folder mode.** Walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`. Add `_`-prefix skip rule. Parallelize cautiously (`swift` spawns aren't free — gate on a small concurrency limit, maybe `ProcessInfo.activeProcessorCount`). -4. **Ship a bundled-binary release.** Build SyntaxKit as a `type: .dynamic` library under `-c release`, then `strip -x` the resulting dylib. Bundle the stripped `libSyntaxKit.dylib` + every `.swiftmodule` SyntaxKit publicly re-exports (SwiftSyntax + version-suffixed variants, SwiftOperators, SwiftParser) + the `_SwiftSyntaxCShims/include/` header dir in a `lib/` directory next to the CLI binary. Write a `ResourceLocator` analog (mirrors `cli/Sources/TuistLoader/Utils/ResourceLocator.swift:52-83`). POC step 1 confirmed cold-start with this layout is ~720ms; stripped release dylib is **9.3 MB** on Apple Silicon (down from 25 MB debug / 18 MB unstripped release). -5. **Helpers directory.** Discovery + compile + flag-splicing. -6. **Output cache.** Add the cache, keyed as in §5. Add `--no-cache` for debugging. -7. **Linux smoke test.** Confirm `/usr/bin/env swift` works on Linux with the bundled dylib layout (no framework search path, but `-I + -L + -lSyntaxKit` should be sufficient, matching Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch). - -## 7. What we still need to verify - -- **Cold-start cost.** ~~Single biggest unknown.~~ Answered by POC step 1: ~720ms cold, ~110ms warm. See [`poc-step1-results.md`](./poc-step1-results.md). -- **SyntaxKit `if`-in-`Group` compiler crash.** POC step 1 surfaced this: `CodeBlockBuilderResult` claims `buildEither`/`buildOptional` support but conditionals trigger a type-checker failure-to-diagnose. Independent of the CLI design but blocks users writing conditional codegen. File as a separate SyntaxKit bug. -- **Splice fidelity.** When the input body lives inside a `Group { … }` closure, is everything users naturally write in the DSL still legal? Result-builder closures don't allow `import`, top-level type decls, or top-level `let`/`var` outside the builder DSL. The wrapper hoists `import`s; we need to confirm there's no other top-level construct users would reasonably write that the wrap step would break. Verify in step 1 with a few realistic inputs (large struct, nested types, conditionals via `if`-in-builder). -- **`Process` stdout/stderr separation.** Tuist captures stdout-only and merges stderr via `CommandError`. Foundation's `Process` has the same split — confirm it doesn't interleave under load, and confirm the CLI doesn't accidentally swallow stderr. -- **`swift` script-mode quirks.** `swift ` runs in interpret/`-frontend -interpret` mode. Some features (`@main`, certain attributes) behave differently than in compile mode. Top-level `print` statements are fine. Verify in step 1. -- **SwiftSyntax linkage stability across toolchains.** SwiftSyntax pins to specific Swift toolchain versions. The bundled dylib is built against a particular toolchain; if the user's `swift` is from a newer or older release, ABI breakage is possible. Mitigation: cache key includes `swiftlangVersion`, plus a clear error when the gap is too wide. -- **What if a single input script needs to produce multiple files?** Out of scope for v1. Split into multiple inputs and use folder mode. If demand materializes, we can layer in a `--multi` envelope mode (a script writes a small JSON manifest to stdout, CLI fans out to multiple files) without breaking single-file semantics. -- **Sandboxing.** Out of scope for v1. Input scripts are user-owned code in their own repo — running them has the same threat model as running `swift Input.swift` by hand. Revisit if/when this CLI runs untrusted scripts (CI for OSS contributions, etc.). -- **Timeout.** Add one. Tuist's omission (Phase 1 §7) is a bug, not a feature. 60s default `Process` wait with `SIGTERM` → 5s grace → `SIGKILL`. Override via `--timeout `. -- **Web-server form.** Out of scope for the CLI POC, but on the table as a follow-up once the 7-step ladder is done. A long-lived server could reuse a warm `swift` interpreter across requests and share the helpers + output caches across tenants — both of which the CLI gives up on every invocation. Open design questions: request shape (raw DSL POST vs. structured), whether helpers are uploaded per-request or baked into the server image, and isolation between requests (the CLI's "run user code in your own repo" threat model doesn't transfer). Revisit after step 7. diff --git a/Docs/research/poc-step1-results.md b/Docs/research/poc-step1-results.md deleted file mode 100644 index 1940027..0000000 --- a/Docs/research/poc-step1-results.md +++ /dev/null @@ -1,138 +0,0 @@ -# POC Step 1 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 1. Goal: prove the `swift`-script + framework-search-path mechanism works for SyntaxKit, that a result-builder closure spliced from user text compiles cleanly, and measure cold-start cost. - -## TL;DR — Approach is viable - -- **Cold start: ~720ms** real wall-clock for `xcrun swift Input.wrapped.swift -lSyntaxKit …` on M-series macOS with the dylib + module files unbacked by the OS page cache. -- **Warm: ~110ms** for subsequent runs. -- **Pure-DSL `Input.swift` spliced into `Group { … }`** in a generated wrapper compiles and runs end-to-end, producing the expected Swift source. -- **Bundled-dylib distribution is real and viable.** The only flags the CLI has to assemble are `-I`, `-L`, `-lSyntaxKit`, `-Xlinker -rpath -Xlinker ` and one new requirement: `-Xcc -I -Xcc ` (see §3 finding). -- **SyntaxKit dylib weight:** 25 MB debug → 18 MB release → **9.3 MB stripped release**. The 9.3 MB number is the one that matters for distribution and is well within range of a normal CLI binary. - -## 1. What was run - -Built the dylib by temporarily flipping the SyntaxKit library product to `type: .dynamic` in `Package.swift`, ran `swift build`, copied the artifacts into a `/tmp/syntaxkit-poc/lib/` staging dir, then reverted the package manifest. - -Wrote a pure-DSL `Input.swift`: - -```swift -import SyntaxKit // optional; only for IDE - -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") -} - -Struct("Pet") { - Variable(.let, name: "kind", type: "String") -} -``` - -And a hand-rolled `Input.wrapped.swift`: - -```swift -import SyntaxKit - -let __syntaxkit_root = Group { - Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") - } - Struct("Pet") { - Variable(.let, name: "kind", type: "String") - } -} - -print(__syntaxkit_root.generateCode()) -``` - -Invoked with: - -``` -xcrun swift \ - -I lib -L lib -lSyntaxKit \ - -Xcc -I -Xcc lib/_SwiftSyntaxCShims-include \ - -Xlinker -rpath -Xlinker $(pwd)/lib \ - Input.wrapped.swift -``` - -Output (verbatim): - -``` -struct Person { -let name : String -let age : Int - -} -struct Pet { -let kind : String - -} -``` - -(Whitespace artifacts are SyntaxKit's `generateCode()` output as-is — out of scope for this POC.) - -## 2. Timings - -Three back-to-back runs after a cold first run: - -| Run | real | user | sys | -| --- | ---: | ---: | ---: | -| cold (first) | 0.72s | 0.77s | 0.29s | -| warm 1 | 0.14s | 0.08s | 0.04s | -| warm 2 | 0.11s | 0.07s | 0.02s | -| warm 3 | 0.11s | 0.07s | 0.02s | - -Hardware: Apple Silicon mac. Cold start is dominated by loading SyntaxKit + SwiftSyntax dylibs from disk; once cached, the swift interpreter just compiles a tiny script. For a per-file CLI invocation this is well inside the "feels instant" budget. - -## 3. New design finding: C-shim include path - -Without `-Xcc -I -Xcc <_SwiftSyntaxCShims/include>`, the script compile fails with: - -``` -:0: error: missing required module '_SwiftSyntaxCShims' -``` - -SyntaxKit transitively depends on SwiftSyntax which has a C-shims target whose module map lives at `swift-syntax/Sources/_SwiftSyntaxCShims/include/module.modulemap`. The CLI's bundled-binary distribution layout (§5 of the design doc) must include this header directory, not just the `.dylib` and `.swiftmodule` files. Updated the design doc to reflect this. - -## 4. New design finding: `if` inside `Group` is broken in SyntaxKit today - -A wrapped input containing a conditional in the builder: - -```swift -let __syntaxkit_root = Group { - if true { - Struct("A") { Variable(.let, name: "x", type: "Int") } - } -} -``` - -fails with: - -``` -error: failed to produce diagnostic for expression; please submit a bug report -let __syntaxkit_root = Group { - ^ -``` - -`CodeBlockBuilderResult` declares both `buildEither(first:)` / `buildEither(second:)` and `buildOptional` (`Sources/SyntaxKit/CodeBlocks/CodeBlockBuilderResult.swift:46-58`), so the API surface *says* `if`/`else` is supported. The compiler crash is a Swift type-checker timeout — likely from the `any CodeBlock...` variadic overload combined with `buildEither` overload resolution. No test in `Tests/SyntaxKitTests/Unit/` exercises an `if` inside `Group { … }`, which is why this hasn't been caught. - -**Implication for the CLI design:** non-blocking. v1 can document "no conditionals in input files yet" and ship; the underlying SyntaxKit fix is independent. Worth filing as a separate issue. - -## 5. Confirmed: hoisted imports work - -A wrapped input with both `import SyntaxKit` and `import Foundation` at the top compiles fine and `UUID`/`Date` resolve in the rendered struct fields. The CLI's hoist-imports step (design §3) is safe. - -## 6. Not yet retired - -- **Stderr/stdout interleaving under load.** Need a load test (large input → tons of output → confirm captured stdout is intact and stderr doesn't bleed in). -- **Linux behavior.** Step 7 of the POC ladder. Same `swift -I -L -l` flag set should work; framework-search paths (`-F`) become a no-op. -- **`@main` and other top-level forms.** Not relevant for the wrapper because we always control the wrapper's shape, but if users ever paste a class/extension declaration directly into `Input.swift` we need to reject it cleanly. - -## 7. Updates to the design doc to make from this POC - -- §5 distribution layout: add `Sources/_SwiftSyntaxCShims/include/` to the bundled `lib/` contents. -- §3 spawn command: include the `-Xcc -I -Xcc <…>` flag. -- §7 open questions: SyntaxKit dylib size measured at 25 MB debug, 18 MB release, 9.3 MB release+stripped (`strip -x`). Warm performance identical between debug and release builds. -- §7 open questions: add tracking note for the `if`-in-`Group` Swift compiler bug. diff --git a/Docs/research/poc-step1.sh b/Docs/research/poc-step1.sh deleted file mode 100755 index fe41e3b..0000000 --- a/Docs/research/poc-step1.sh +++ /dev/null @@ -1,117 +0,0 @@ -#!/usr/bin/env bash -# -# POC step 1 reproducer for issue #154. -# Run from anywhere; resolves the repo root from its own location. -# -# What it does: -# 1. Backs up Package.swift, flips the SyntaxKit library to type: .dynamic. -# 2. swift build (produces libSyntaxKit.dylib). -# 3. Stages dylib + swiftmodules + _SwiftSyntaxCShims headers into /tmp/syntaxkit-poc/lib/. -# 4. Writes a pure-DSL Input.swift and a hand-rolled Input.wrapped.swift. -# 5. Runs the wrapped script once cold + three times warm, printing timings. -# 6. Restores Package.swift on exit (even on failure). - -set -euo pipefail - -if [[ "$(uname -s)" != "Darwin" ]]; then - echo "This reproducer is macOS-only. Linux smoke test is POC step 7." >&2 - exit 1 -fi - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -POC_DIR="/tmp/syntaxkit-poc" -PACKAGE_FILE="$REPO_ROOT/Package.swift" -PACKAGE_BACKUP="$(mktemp)" - -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found — has Package.swift changed shape?") -p.write_text(src.replace(old, new, 1)) -PY - -cd "$REPO_ROOT" -echo "==> swift build" -swift build - -BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1 || true)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate .build//debug" >&2 - exit 1 -fi - -echo "==> Staging $POC_DIR/lib/" -rm -rf "$POC_DIR" -mkdir -p "$POC_DIR/lib" -cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" -cp -r ".build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ - "$POC_DIR/lib/_SwiftSyntaxCShims-include" - -cat > "$POC_DIR/Input.swift" <<'SWIFT' -// Pure-DSL input. No print, no @main, no boilerplate. -import SyntaxKit // optional; only for IDE autocomplete - -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") -} - -Struct("Pet") { - Variable(.let, name: "kind", type: "String") -} -SWIFT - -cat > "$POC_DIR/Input.wrapped.swift" <<'SWIFT' -import SyntaxKit - -let __syntaxkit_root = Group { - Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") - } - Struct("Pet") { - Variable(.let, name: "kind", type: "String") - } -} - -print(__syntaxkit_root.generateCode()) -SWIFT - -cd "$POC_DIR" - -SWIFT_ARGS=( - -I lib -L lib -lSyntaxKit - -Xcc -I -Xcc lib/_SwiftSyntaxCShims-include - -Xlinker -rpath -Xlinker "$POC_DIR/lib" - Input.wrapped.swift -) - -echo -echo "==> Cold run (full output + timing):" -/usr/bin/time -p xcrun swift "${SWIFT_ARGS[@]}" - -echo -echo "==> Warm runs (timings only, output discarded):" -for _ in 1 2 3; do - /usr/bin/time -p xcrun swift "${SWIFT_ARGS[@]}" >/dev/null -done - -echo -echo "==> Done. Staging dir kept at $POC_DIR for further poking." diff --git a/Docs/research/poc-step2-results.md b/Docs/research/poc-step2-results.md deleted file mode 100644 index 5bd6cd2..0000000 --- a/Docs/research/poc-step2-results.md +++ /dev/null @@ -1,70 +0,0 @@ -# POC Step 2 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 2. Goal: wrap the hand-driven flow from step 1 inside an actual CLI executable that parses imports with SwiftSyntax, generates the wrapper, spawns `swift`, and forwards output/errors. - -## What landed - -New executable target `skitrun` (POC name; final CLI name TBD) at `Sources/skitrun/Main.swift`. Single file, ~180 lines. Depends only on `SwiftSyntax` + `SwiftParser` — does **not** depend on SyntaxKit, since the host doesn't render anything itself. - -## Usage - -``` -skitrun [-o ] [--lib ] -``` - -`--lib` defaults to `/tmp/syntaxkit-poc/lib` so it works against the artifacts produced by [`poc-step1.sh`](./poc-step1.sh) without further flags. - -Build it once: `swift build --product skitrun`. The binary lands at `.build//debug/skitrun`. - -## Verified flows - -1. **Default (stdout):** `skitrun Input.swift` prints rendered Swift to stdout. -2. **File output:** `skitrun Input.swift -o Out.swift` writes the file atomically. -3. **Hoisted imports:** an input with `import Foundation` at the top compiles cleanly; `UUID`/`Date` resolve in the rendered struct. -4. **Compiler diagnostics map to the input file.** A deliberate `type: NonexistentType` in `InputError.swift:4` produces: - ``` - /tmp/syntaxkit-poc/InputError.swift:4:37: error: cannot find 'NonexistentType' in scope - ``` - The path and line are correct — confirming `#sourceLocation` is doing the work end-to-end. - -## How the wrap works - -`SwiftParser.Parser.parse(source:)` produces a `SourceFileSyntax`. We walk `tree.statements`: - -- Every leading `ImportDeclSyntax` is collected for hoisting. -- The first non-import statement marks the start of the body. -- Everything from that byte offset forward is the body, copied verbatim. - -The wrapper is then: - -```swift -import SyntaxKit - - -let __skitrun_root = Group { -#sourceLocation(file: "", line: ) - -#sourceLocation() -} - -print(__skitrun_root.generateCode()) -``` - -`#sourceLocation` is what gives us diagnostic fidelity for free — the Swift compiler honors it and rewrites file/line in every error/warning emitted from the body range. No manual stderr line-number arithmetic needed. - -## Spawn shape - -`Foundation.Process` invoking `/usr/bin/env swift` with the exact flag set from POC step 1, captured into `stdoutPipe` and `stderrPipe`. Stdout is written verbatim to the output destination. Stderr is forwarded after one fix-up: any remaining literal `//skitrun-/Input.wrapped.swift` references (those outside the `#sourceLocation` range — i.e. errors in the preamble itself) get rewritten to the input path. - -## Surface limits worth knowing - -- **Snippet gutter line numbers in diagnostics show wrapper line numbers, not input line numbers.** The compiler maps the *file/line* in the diagnostic header via `#sourceLocation` but shows the surrounding source snippet from the actual file with its actual line numbers. The path and starting line are correct (navigable), but the gutter `7 |` / `8 |` markers may not match the input's line numbering. Cosmetic; doesn't affect navigation. -- **No timeout yet.** The design calls for a 60s default. Adding `Process.terminate(after:)` is a step 6 (cache) sibling concern. -- **Stdin / stderr interleaving under load not tested.** Step 7 territory. -- **`if`-in-`Group` still crashes the compiler** ([#155](https://github.com/brightdigit/SyntaxKit/issues/155)). Independent SyntaxKit bug; `skitrun` would happily pass such an input through, but the spawned `swift` would fail with the same opaque diagnostic from step 1. - -## What's next - -The natural step 3 is folder mode (walk `InputDir/**/*.swift`, mirror paths into `OutputDir/`). All the per-file work is already in place — folder mode is just iteration + concurrency-limited fan-out + a `_`-prefix skip rule. Modest engineering, low risk. - -After that, step 4 (bundled-binary release) is where the design hits its biggest remaining systems-integration question: how to actually ship the `lib/` directory next to the binary across SwiftPM build, install, and `brew` distribution. diff --git a/Docs/research/poc-step3-results.md b/Docs/research/poc-step3-results.md deleted file mode 100644 index be524bf..0000000 --- a/Docs/research/poc-step3-results.md +++ /dev/null @@ -1,48 +0,0 @@ -# POC Step 3 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 3. Goal: extend `skitrun` to walk a directory of `.swift` inputs, mirror the relative paths into an output directory, and run the per-file work concurrently with a sane cap. - -## What changed - -`skitrun` now accepts a directory as input: - -``` -skitrun InputDir/ -o OutDir/ -``` - -When the input is a directory, the existing single-file work is hoisted into a `processFile(inputPath:libPath:)` helper. The new `runDirectory(...)` driver walks the input with `FileManager.enumerator`, fan-outs the per-file work over `withTaskGroup`, and writes successes into mirrored paths under `OutDir/`. Single-file mode is unchanged. - -Three small conventions ride along: - -- **`_`-prefix skip rule.** `_Helpers.swift`, `_Shared.swift`, etc. are not processed. (Confirmed against `_HelperShouldBeSkipped.swift` containing deliberately-invalid Swift — skitrun didn't try to compile it.) -- **`activeProcessorCount` concurrency cap.** The task group keeps that many in-flight `swift` spawns at a time, draining + refilling as each finishes. -- **Tuist-analog partial semantics.** Successful files are *always* written, even when other files in the same batch fail. The CLI exits non-zero if any failed and prints a `skitrun: N/M succeeded` summary to stderr. - -## Verified flows - -1. **Happy path.** A `codegen/` tree with `Models/Person.swift`, `Models/Pet.swift`, `Audit/Snapshot.swift` (the last with a hoisted `import Foundation`) produces a mirrored `out/Models/{Person,Pet}.swift` + `out/Audit/Snapshot.swift`. Total wall time 1.41s for 3 files (vs. 0.72s baseline cold-start for one). -2. **Skip rule.** `codegen/_HelperShouldBeSkipped.swift` contains the literal line `this is not valid swift`. It is not visited, the rest of the tree processes cleanly. -3. **Partial failure.** Adding a `Models/Bad.swift` with `type: TypeThatDoesNotExist` produces: - ``` - ---- /tmp/skitrun-folder-test/codegen/Models/Bad.swift ---- - /tmp/skitrun-folder-test/codegen/Models/Bad.swift:4:37: error: cannot find 'TypeThatDoesNotExist' in scope - … - skitrun: 3/4 succeeded - ``` - Exit code 1. Person/Pet/Snapshot still written. -4. **Single-file regression.** Both `skitrun Input.swift` (stdout) and `skitrun Input.swift -o Out.swift` (file) still work after the refactor. - -## Parallelism observations - -A quick timing on 3 parallel files vs. 1 cold-start baseline: - -| | wall time | -| --- | ---: | -| 1 file, cold | 0.72s | -| 3 files, cold, `withTaskGroup` cap = `activeProcessorCount` | 1.41s | - -That's well below 3×0.72 = 2.16s, confirming the parallelism is buying something — but also clearly slower than 3×0.11 = 0.33s warm, meaning successive `swift` invocations don't fully share OS file-cache benefits within a single batch run. (Each spawn still pays its own compile cost; the dylib pages are warm after the first, but compile work isn't deduplicated.) For larger batches we'd want to measure where the curve goes — and eventually pull the work into a single long-lived `swift` process driving all inputs, to skip the per-file compile overhead entirely. Out of scope for v1. - -## What's next - -Step 4 — bundled-binary release. The first real systems-integration challenge: how the `lib/` directory ships next to the `skitrun` binary across `swift build`, `swift run`, and `brew install`. Today users have to run `Docs/research/poc-step1.sh` to stage `/tmp/syntaxkit-poc/lib/`; that has to become "user installs the CLI, it just works." diff --git a/Docs/research/poc-step4-results.md b/Docs/research/poc-step4-results.md deleted file mode 100644 index c5248ac..0000000 --- a/Docs/research/poc-step4-results.md +++ /dev/null @@ -1,55 +0,0 @@ -# POC Step 4 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 4. Goal: produce a self-contained `skitrun` release bundle so users don't need a SyntaxKit checkout or `Docs/research/poc-step1.sh`. The binary finds its own `lib/` directory. - -## What landed - -1. **`resolveLibPath(override:)` in `Sources/skitrun/Main.swift`.** Search order: - 1. `--lib ` flag - 2. `$SKITRUN_LIB_DIR` env var - 3. `/lib/` — same-directory layout (the release bundle ships this way) - 4. `/../lib/skitrun/` — Homebrew layout (`bin/skitrun` ↔ `lib/skitrun/`) - - When none match, the CLI errors with a message enumerating all four paths and pointing at this script. The `/tmp/syntaxkit-poc/lib` fallback is gone. - -2. **`Docs/research/poc-step4-release.sh`.** Builds a self-contained bundle: - ``` - .build/skitrun-release/ - skitrun ← the CLI binary - lib/ - libSyntaxKit.dylib ← release + strip -x - *.swiftmodule ← SyntaxKit + transitively re-exported modules - _SwiftSyntaxCShims-include/ ← C-shims headers - ``` - Same trap-based Package.swift backup/restore as `poc-step1.sh`. `install_name_tool -id @rpath/libSyntaxKit.dylib` ensures the dylib install name is portable. - -## Verified flows - -Built bundle → copied to three unrelated locations → all worked with no flags, no env vars, no SyntaxKit checkout: - -1. **Same-directory layout.** `cp -r .build/skitrun-release /tmp/portable && /tmp/portable/skitrun Input.swift` → correct output. -2. **Homebrew layout.** `bin/skitrun + lib/skitrun/` arrangement → correct output. -3. **Error case.** `skitrun` alone in `/tmp/lonely/` (no lib anywhere) → clear diagnostic: - ``` - Could not locate SyntaxKit lib directory. Looked for: - 1. --lib (not provided) - 2. $SKITRUN_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skitrun/ (not found) - ``` -4. **Folder mode** from the portable bundle works end-to-end with partial-failure semantics intact. - -## Bundle weight - -| Component | Size | -| --- | ---: | -| `skitrun` (binary) | 17 MB | -| `libSyntaxKit.dylib` (release + stripped) | 9.3 MB | -| `lib/*` (modules + headers + dylib) | ~28 MB | -| **Total bundle** | **45 MB** | - -The 17 MB binary is unexpectedly heavy: it links SwiftSyntax statically because `skitrun` uses SwiftSyntax directly for parsing input files. So SwiftSyntax ships **twice** — once statically inside `skitrun`, once dynamically as part of the SyntaxKit dylib stack. Worth a follow-up: make `skitrun` itself dlopen SyntaxKit / share a dynamic SwiftSyntax with the dylib path. For v1 this is acceptable but is the largest single thing standing between the CLI and a "feels small" download. - -## What's next - -Step 5: helpers directory. Today users can `import Foundation` and `import SyntaxKit` from input files; step 5 lets them factor reusable codegen into a `Helpers/` directory that gets pre-compiled into `lib.dylib` and made importable from inputs. Modest engineering, but the first time `skitrun` itself invokes `swiftc` rather than `swift` (the helpers compile, distinct from the input run). See [`codegen-cli-design.md` §4](./codegen-cli-design.md#4-helpers) for the shape. diff --git a/Docs/research/poc-step5-results.md b/Docs/research/poc-step5-results.md deleted file mode 100644 index 746570e..0000000 --- a/Docs/research/poc-step5-results.md +++ /dev/null @@ -1,58 +0,0 @@ -# POC Step 5 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §4 + §6 step 5. Goal: let inputs `import SyntaxKitHelpers` from a `Helpers/` directory adjacent to the input, with discovery + on-demand `swiftc` compile + a per-toolchain cache. - -## What landed - -1. **`Sources/skitrun/Helpers.swift`.** New module with three responsibilities: - - `discoverHelpersDir(near:)` walks up from the input path looking for a `Helpers/` directory. Single-file mode starts from the file's parent; folder mode starts from the input directory. Walk-up stops at the filesystem root. - - `collectHelperSources(in:)` globs `**/*.swift` under the helpers dir, skipping `_`-prefixed files (same convention as input enumeration). - - `buildHelpers(helpersDir:libPath:)` hashes the sources + Swift version + dylib stamp into a cache key under `~/Library/Caches/com.brightdigit.SyntaxKit/helpers//`. On a cache miss, it shells out to `swiftc` into a `tmp../` staging dir, then atomic-renames into the cache path (`ProjectDescriptionHelpersBuilder` pattern from Tuist). - -2. **`Sources/skitrun/Main.swift` wiring.** - - `CLIArgs` gains `--helpers ` (explicit override) and `--no-helpers` (skip discovery). Default is auto. - - `runSwift` splices `-I/-L/-lSyntaxKitHelpers -Xlinker -rpath -Xlinker ` when helpers are present. - - Folder mode's `collectInputs` now also yields directories so it can call `enumerator.skipDescendants()` when it hits a `Helpers/` directly under the input root — otherwise the helpers would be re-processed as inputs. - - Helpers compile happens **once per invocation** in folder mode (not per input file). - -3. **`Docs/research/poc-step5.sh`.** Standalone demo: builds skitrun, stages a runtime lib, writes a tiny `Helpers/Models.swift` exporting `equatableModel(_:fields:)`, and two `inputs/*.swift` files that `import SyntaxKitHelpers` and call the helper. - -## Verified flows - -| Flow | Result | -| --- | --- | -| Cold run (cache cleared, helpers compile from scratch) | ✓ 2.96s real | -| Warm run (cache hit, same helper sources) | ✓ 0.54s real | -| Folder mode against `demo/` containing `Helpers/` + `inputs/` | ✓ 2/2 succeeded, `Helpers/*.swift` not enumerated as input | -| `--no-helpers` with an input that imports `SyntaxKitHelpers` | ✓ child `swift` errors with `no such module 'SyntaxKitHelpers'`, exit non-zero | - -The cached layout for a single helper file: - -``` -~/Library/Caches/com.brightdigit.SyntaxKit/helpers// - libSyntaxKitHelpers.dylib - SyntaxKitHelpers.swiftmodule - SyntaxKitHelpers.swiftdoc - SyntaxKitHelpers.abi.json - SyntaxKitHelpers.swiftsourceinfo -``` - -## Cache key - -SHA-256 over (in order): -- Cache schema version string (`v1`). -- For each helper source (sorted by absolute path): `lastPathComponent` + file bytes. -- `swift --version` output. -- `libSyntaxKit.dylib` size and modification time (proxy for SyntaxKit version until the bundle is versioned). - -Mutating any helper source, switching toolchains, or rebuilding SyntaxKit invalidates the cache. Adding a `cacheSchemaVersion` bump constant covers future layout changes. - -## Known rough edges - -- **Helpers cold compile is the dominant cost.** 2.96s vs 0.54s warm — the helper compile is ~2.5s on top of the ~0.5s `swift` interpret cost. Once cached it's free, but the first run after a clean checkout is noticeably slow. Acceptable for v1; could be sped up by caching the helpers `.o` files separately, but that's a step-6+ optimization. -- **Walk-up false positives.** If a user happens to have an unrelated `Helpers/` somewhere up-tree (e.g. a sibling library), skitrun will try to compile it. `--helpers ` or `--no-helpers` is the escape hatch. A future heuristic could require a sentinel file (`Helpers/.syntaxkit-helpers`) before claiming the directory. -- **Import-line diagnostics off by one.** When the user's input has `import Foo` on line 1, the wrap step puts an injected `import SyntaxKit` above it, so a child-compile error on the user's import reports `:2:8` instead of `:1:8`. `#sourceLocation` directives only wrap the body, not the hoisted imports. Easy follow-up: emit a `#sourceLocation` directive per hoisted import too. - -## What's next - -Step 6: **output cache.** Today every `skitrun` invocation re-spawns `swift` to render the input, even when nothing has changed. Add the per-input output cache from [`codegen-cli-design.md` §5](./codegen-cli-design.md#5-caching), keyed by input hash + helpers-cache key + Swift version + envHash. On a hit, skip the spawn entirely and copy the rendered output to the destination. Add `--no-cache` for debugging. diff --git a/Docs/research/poc-step5.sh b/Docs/research/poc-step5.sh deleted file mode 100755 index ea633a3..0000000 --- a/Docs/research/poc-step5.sh +++ /dev/null @@ -1,135 +0,0 @@ -#!/usr/bin/env bash -# -# POC step 5 demo: Helpers/ discovery + compile + import in input scripts. -# -# Builds skitrun, stages a runtime lib/ next to it, then runs skitrun against -# a demo project that uses `import SyntaxKitHelpers`. Demonstrates: -# 1. Cold path — Helpers/ compiles to libSyntaxKitHelpers.dylib. -# 2. Warm path — second invocation reuses the cached helpers dylib. -# 3. Folder mode — skitrun ignores Helpers/ when walking the input tree. -# 4. --no-helpers — disables discovery; the import then fails as expected. - -set -euo pipefail - -if [[ "$(uname -s)" != "Darwin" ]]; then - echo "macOS-only. Linux smoke test is POC step 7." >&2 - exit 1 -fi - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -POC_DIR="/tmp/syntaxkit-poc-step5" -DEMO_DIR="$POC_DIR/demo" -CACHE_DIR="$HOME/Library/Caches/com.brightdigit.SyntaxKit/helpers" -PACKAGE_FILE="$REPO_ROOT/Package.swift" -PACKAGE_BACKUP="$(mktemp)" - -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found") -p.write_text(src.replace(old, new, 1)) -PY - -cd "$REPO_ROOT" - -echo "==> swift build" -swift build - -BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate .build//debug" >&2 - exit 1 -fi - -echo "==> Staging $POC_DIR" -rm -rf "$POC_DIR" -mkdir -p "$POC_DIR/lib" "$DEMO_DIR/Helpers" "$DEMO_DIR/inputs" - -cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" -cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" -cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ - "$POC_DIR/lib/_SwiftSyntaxCShims-include" - -cat > "$DEMO_DIR/Helpers/Models.swift" <<'SWIFT' -import SyntaxKit - -public func equatableModel( - _ name: String, - fields: [(name: String, type: String)] -) -> any CodeBlock { - Struct(name) { - for field in fields { - Variable(.let, name: field.name, type: field.type) - } - }.inherits("Equatable") -} -SWIFT - -cat > "$DEMO_DIR/inputs/Person.swift" <<'SWIFT' -import SyntaxKitHelpers - -equatableModel("Person", fields: [ - ("name", "String"), - ("age", "Int"), -]) -SWIFT - -cat > "$DEMO_DIR/inputs/Pet.swift" <<'SWIFT' -import SyntaxKitHelpers - -equatableModel("Pet", fields: [ - ("kind", "String"), - ("owner", "String"), -]) -SWIFT - -echo "==> Clearing helpers cache to force cold compile" -rm -rf "$CACHE_DIR" - -echo -echo "==> Cold run (helpers compile from scratch):" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/inputs/Person.swift" - -echo -echo "==> Warm run (helpers cache hit):" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/inputs/Person.swift" >/dev/null - -echo -echo "==> Cached helper artifacts:" -find "$CACHE_DIR" -maxdepth 3 -type f | sed "s|$CACHE_DIR||" | sort - -echo -echo "==> Folder mode (Helpers/ excluded from input enumeration):" -rm -rf "$POC_DIR/out" -"$POC_DIR/skitrun" "$DEMO_DIR" -o "$POC_DIR/out" -echo " Generated files:" -find "$POC_DIR/out" -type f | sed "s|$POC_DIR/| |" - -echo -echo "==> --no-helpers should fail with an unresolved import:" -if "$POC_DIR/skitrun" --no-helpers "$DEMO_DIR/inputs/Person.swift" >/dev/null 2>&1; then - echo "FAIL: --no-helpers should have errored" >&2 - exit 1 -else - echo " ✓ skitrun returned non-zero as expected" -fi - -echo -echo "==> Done. Demo project kept at $DEMO_DIR; cache at $CACHE_DIR." diff --git a/Docs/research/poc-step6-results.md b/Docs/research/poc-step6-results.md deleted file mode 100644 index f50fc85..0000000 --- a/Docs/research/poc-step6-results.md +++ /dev/null @@ -1,58 +0,0 @@ -# POC Step 6 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §5 + §6 step 6. Goal: skip the `swift` spawn entirely when the rendered output for an input is already cached, with a key that captures everything that could change the output. - -## What landed - -1. **`Sources/skitrun/OutputCache.swift`.** Three functions: - - `outputCacheKey(inputSource:helpers:libPath:)` — SHA-256 over cache schema version + input bytes + helpers cache key (or `"no-helpers"`) + `swift --version` + libSyntaxKit dylib stamp + sorted `SKITRUN_*` / `SYNTAXKIT_*` env vars. - - `lookupCachedOutput(key:)` — returns the cached `output.swift` bytes from `~/Library/Caches/com.brightdigit.SyntaxKit/outputs//output.swift`, or `nil` on miss. - - `storeCachedOutput(key:data:)` — atomic write via `tmp../` staging dir + rename. Concurrent writers race safely; the loser drops their copy if the destination already exists. - -2. **`Sources/skitrun/Main.swift` wiring.** - - `processFile` reads the input source, computes the cache key, and short-circuits on hit — no temp wrapper, no `swift` spawn, just return the cached bytes with `exitCode = 0`. - - On miss, the normal wrap + spawn path runs; if `swift` returns 0, the rendered output is stored under the key. - - Only successful (exit 0) runs are cached. Failed runs always re-spawn so the user sees fresh diagnostics. - - `CLIArgs` gains `--no-cache` to skip the cache entirely (still useful when chasing flaky output or after manually deleting the cache). - - The flag threads through `runSingleFile` and `runDirectory` so folder mode can opt out wholesale. - -3. **Helpers.swift internal exposure.** `syntaxKitCacheRoot`, `captureSwiftVersion`, and `libStamp` were `private`; they're now `internal` so OutputCache can reuse them rather than duplicating. - -## Verified flows - -From `Docs/research/poc-step6.sh` (single input, no helpers): - -| Run | Real time | Notes | -| --- | ---: | --- | -| Cold (cache cleared) | 0.55s | swift spawn + compile + store | -| Warm (cache hit) | 0.14s | FS read only, no swift spawn | -| `--no-cache` | 0.27s | always spawn, ignore cache | -| After mutation (miss) | 0.41s | new key, recompile, store | -| Warm after mutation | 0.14s | second key cached | - -After mutation, the cache directory contains **two** entries (one per input version) — old keys aren't evicted, which is fine for a per-toolchain cache where stale entries are dead weight, not correctness risks. Eviction can be a follow-up if cache size becomes a complaint. - -## Cache key, written out - -``` -SHA-256( - "v1" // cache schema version - + input.swift bytes // verbatim user input - + helpersCacheKey || "no-helpers" // helpers fingerprint (sibling cache) - + swift --version stdout // toolchain fingerprint - + "/" of libSyntaxKit.dylib // SyntaxKit fingerprint proxy - + sorted SKITRUN_*/SYNTAXKIT_* env (k=v\0) // env override sensitivity -) -``` - -Two cooperating cache layers — helpers (step 5) and outputs (step 6) — sit side-by-side under `~/Library/Caches/com.brightdigit.SyntaxKit/{helpers,outputs}//`. Helpers cache hits are reused across many inputs; output cache hits are per-input. - -## Known rough edges - -- **Output cache stores stdout only.** Stderr from a successful run (e.g. warnings even with `-suppress-warnings` off) is discarded on a hit. With `-suppress-warnings` in `runSwift` this is rarely visible, but it does mean cache hits suppress warnings that would have appeared on a fresh run. Acceptable for a generator; revisit if SyntaxKit grows runtime-side warnings. -- **`libStamp` is a coarse proxy.** Size + mtime catches normal rebuilds but a deterministic rebuild that preserves both would slip past. Hashing the dylib is correct but slow (9.3 MB per invocation defeats the cache). The right long-term fix is embedding a SyntaxKit version constant the bundle exports. -- **No size cap or eviction.** A repo that touches inputs frequently will accrete cache entries. Each is small (a few hundred bytes of rendered Swift) so the practical ceiling is high, but a `--prune` subcommand is a reasonable v1.1 addition. - -## What's next - -Step 7: **Linux smoke test.** Confirm `/usr/bin/env swift` + the bundled-dylib layout works on Linux without `-F` framework search paths — `-I + -L + -lSyntaxKit` should be sufficient per Tuist's `ProjectDescriptionSearchPaths.Style.commandLine` branch. CryptoKit is macOS-only; Linux will need `swift-crypto` or a small fallback hash impl behind a `#if canImport(CryptoKit)` shim. After that, the 7-step ladder is complete. diff --git a/Docs/research/poc-step6.sh b/Docs/research/poc-step6.sh deleted file mode 100755 index ad90c70..0000000 --- a/Docs/research/poc-step6.sh +++ /dev/null @@ -1,116 +0,0 @@ -#!/usr/bin/env bash -# -# POC step 6 demo: rendered-output cache. skitrun skips the `swift` spawn -# when input + helpers + toolchain are unchanged. -# -# Builds skitrun, stages a runtime lib next to it, runs an input, then -# replays it three ways: -# 1. with cache — should be near-instant (no swift spawn) -# 2. --no-cache — always spawns swift -# 3. mutated input — invalidates the cache key, falls back to swift - -set -euo pipefail - -if [[ "$(uname -s)" != "Darwin" ]]; then - echo "macOS-only. Linux smoke test is POC step 7." >&2 - exit 1 -fi - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -POC_DIR="/tmp/syntaxkit-poc-step6" -DEMO_DIR="$POC_DIR/demo" -OUTPUT_CACHE="$HOME/Library/Caches/com.brightdigit.SyntaxKit/outputs" -PACKAGE_FILE="$REPO_ROOT/Package.swift" -PACKAGE_BACKUP="$(mktemp)" - -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found") -p.write_text(src.replace(old, new, 1)) -PY - -cd "$REPO_ROOT" - -echo "==> swift build" -swift build - -BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate .build//debug" >&2 - exit 1 -fi - -echo "==> Staging $POC_DIR" -rm -rf "$POC_DIR" -mkdir -p "$POC_DIR/lib" "$DEMO_DIR" - -cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" -cp "$BUILD_DIR/libSyntaxKit.dylib" "$POC_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" -cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ - "$POC_DIR/lib/_SwiftSyntaxCShims-include" - -cat > "$DEMO_DIR/Input.swift" <<'SWIFT' -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") -} -SWIFT - -echo "==> Clearing output cache to force a cold run" -rm -rf "$OUTPUT_CACHE" - -echo -echo "==> Cold run (cache miss → swift spawn → store):" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Warm run (cache hit → no swift spawn):" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> --no-cache (always spawn swift, even with cache present):" -/usr/bin/time -p "$POC_DIR/skitrun" --no-cache "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Output cache contents:" -find "$OUTPUT_CACHE" -maxdepth 3 -type f | sed "s|$OUTPUT_CACHE||" | sort - -echo -echo "==> Mutating input invalidates the cache:" -cat > "$DEMO_DIR/Input.swift" <<'SWIFT' -Struct("Person") { - Variable(.let, name: "name", type: "String") - Variable(.let, name: "age", type: "Int") - Variable(.let, name: "email", type: "String") -} -SWIFT -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Warm run after mutation:" -/usr/bin/time -p "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Cache now contains two distinct keys:" -find "$OUTPUT_CACHE" -maxdepth 1 -mindepth 1 -type d | wc -l | xargs -I {} echo " {} cache entries" - -echo -echo "==> Done. Cache at $OUTPUT_CACHE." diff --git a/Docs/research/poc-step7-results.md b/Docs/research/poc-step7-results.md deleted file mode 100644 index a9a40fe..0000000 --- a/Docs/research/poc-step7-results.md +++ /dev/null @@ -1,45 +0,0 @@ -# POC Step 7 — Results - -> Companion to [`codegen-cli-design.md`](./codegen-cli-design.md) §6 step 7. Goal: confirm the `skitrun` flow that worked on macOS (steps 1-6) also works on Linux with the bundled-dylib layout, no framework search paths, and no Apple-only crypto. - -## What landed - -1. **`swift-crypto` swap-in.** `Sources/skitrun/Helpers.swift` and `OutputCache.swift` now `import Crypto` instead of `CryptoKit`. The Apple `swift-crypto` package vends the same `SHA256` API on every platform and re-exports CryptoKit on Apple platforms when available, so there's no `#if` shim needed in skitrun's own code. Added as a Package dependency at `from: "3.0.0"`. - -2. **Platform-aware dylib filename.** A new `dylibFilename(forLibrary:)` helper returns `libX.dylib` on Apple / `libX.so` on Linux. All call sites that hard-coded `libSyntaxKit.dylib` or `libSyntaxKitHelpers.dylib` now go through it: `isLibDir`, `libStamp`, helpers cache-hit check, and the `swiftc -emit-library -o` path. - -3. **Skip `@rpath` install-name on Linux.** `compileHelpers` previously passed `-Xlinker -install_name -Xlinker @rpath/libSyntaxKitHelpers.dylib`. That flag is Mach-O specific; on Linux it errors at link time. Wrapped in `#if !os(Linux)`. The `-Xlinker -rpath` flag still works on both platforms and is what actually locates the dylib at runtime. - -4. **`Docs/research/poc-step7.sh`.** Self-rerunning Docker wrapper: when invoked on the host it re-execs itself inside `swift:6.0-jammy` with the repo bind-mounted; inside the container it flips Package.swift to dynamic, builds with `--build-path .build-linux` (separate from the macOS host's `.build`), stages a `lib/` next to `skitrun`, and runs cold + warm + `--no-cache` against an input that uses helpers. - -## Verified flows - -From `Docs/research/poc-step7.sh` inside `swift:6.0-jammy` (aarch64): - -| Flow | Time | Notes | -| --- | ---: | --- | -| Cold (helpers compile + output cache miss) | 0.73s | swiftc spawns for helpers, swift interprets the wrapped input | -| Warm (output cache hit) | 0.26s | no swift spawn | -| `--no-cache` | 0.30s | swift spawn, helpers reused from cache | - -Rendered output for the demo `Person` struct matches the macOS step 5 output exactly — same SyntaxKit, same generator, no platform-specific quirks in the rendered code. - -## Linux-only surprises - -- **`Process.waitUntilExit()` hangs on already-exited children.** This was the biggest find. Foundation's `Process.waitUntilExit()` on `swift:6.0-jammy/aarch64` blocks indefinitely even when the child has clearly exited (stdout EOF observed, all 76 bytes of `swift --version` already read). Fix applied to all three callers (`captureSwiftVersion`, `compileHelpers`, `runSwift`): set `process.terminationHandler = { _ in semaphore.signal() }` before `run()`, then `semaphore.wait()` instead of `waitUntilExit()`. Took down a 20-minute mystery hang. -- **Stdout/stderr pipe drain order matters.** Linux pipe buffers are ~64 KB; reading sequentially after waiting for child exit deadlocks when the child fills either pipe. `runSwift` now drains both pipes concurrently via `DispatchGroup` + a `PipeDataBox` class (the boxing satisfies Swift 6 strict-concurrency without `@unchecked Sendable` on local vars). `compileHelpers` only needs to drain stderr (stdout goes to `/dev/null`), but drains before the wait for the same reason. -- **`-Xlinker -install_name @rpath/...` is Mach-O specific.** Errors out on Linux's GNU ld. Wrapped in `#if !os(Linux)` in `compileHelpers`. The `-Xlinker -rpath -Xlinker ` flag still works on both platforms and is what actually locates the dylib at runtime. -- **`/usr/bin/time` isn't installed in `swift:6.0-jammy` by default.** Demo script uses the bash builtin `time` for timing, which writes a different format but is portable. -- **`swift build --product skitrun` alone doesn't emit `libSyntaxKit.so`.** The first `poc-step7.sh` draft scoped the build to just the executable, which produced a 60 MB statically-linked binary and no dylib at all. Plain `swift build` (all products) emits both `libSyntaxKit.so` and `skitrun`, matching the macOS steps 5-6 scripts. -- **First-time build cost.** Cold dependency resolution + boringssl C compile (pulled in by `swift-crypto` on Linux where CommonCrypto isn't available) takes ~3 min in `swift:6.0-jammy/aarch64`. Subsequent runs reuse `.build-linux/` and finish in ~40s. -- **Crypto on Linux brings boringssl.** `swift-crypto` statically links boringssl on non-Apple platforms. `skitrun`'s Linux binary is therefore noticeably larger than the macOS one (boringssl C blobs add up). Not a correctness issue, just a size note for future packaging. - -## What's next - -The 7-step POC ladder is **complete**. With this commit: - -- Cold-start cost has been measured on both platforms. -- The bundled-dylib + script-mode `swift` invocation works on macOS and Linux. -- Folder mode, helpers, and the rendered-output cache all behave the same way on both. - -The remaining bullets in [`codegen-cli-design.md` §7](./codegen-cli-design.md#7-what-we-still-need-to-verify) — timeouts, the splice-fidelity audit beyond the demo inputs, `@main`/attribute behavior in script-mode swift, the multi-file output question, and the web-server form — are now the natural next conversation. None of them block productizing the CLI; all of them are scope decisions rather than open technical risks. diff --git a/Docs/research/poc-step7.sh b/Docs/research/poc-step7.sh deleted file mode 100755 index 1d6c98e..0000000 --- a/Docs/research/poc-step7.sh +++ /dev/null @@ -1,135 +0,0 @@ -#!/usr/bin/env bash -# -# POC step 7: Linux smoke test for skitrun. -# -# Runs inside a swift:6.0-jammy container. Builds skitrun, stages a -# runtime lib/ next to it (libSyntaxKit.so + Modules + _SwiftSyntaxCShims -# headers), then exercises single-file mode, helpers, and the output -# cache — the same flows POC steps 5 and 6 verified on macOS. -# -# Usage (from macOS host or Linux host with docker): -# Docs/research/poc-step7.sh -# -# Override the image with $SKITRUN_LINUX_IMAGE. -# -# To save time across runs, the script uses .build-linux/ as a separate -# build directory so the host's .build/ stays clean and SwiftSyntax -# doesn't re-download on every invocation. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -IMAGE="${SKITRUN_LINUX_IMAGE:-swift:6.0-jammy}" - -if [[ ! -f /.dockerenv ]]; then - # ---- Host side: invoke ourselves inside the swift container. ---- - if ! command -v docker >/dev/null; then - echo "docker is required for POC step 7" >&2 - exit 1 - fi - echo "==> Running POC step 7 inside $IMAGE" - exec docker run --rm -t \ - -v "$REPO_ROOT:/workspace" \ - -w /workspace \ - -e SKITRUN_INSIDE_DOCKER=1 \ - "$IMAGE" \ - /workspace/Docs/research/poc-step7.sh -fi - -# ---- Container side: do the real work. ---- - -PACKAGE_FILE="Package.swift" -PACKAGE_BACKUP="$(mktemp)" -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> swift --version" -swift --version - -echo -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found") -p.write_text(src.replace(old, new, 1)) -PY - -echo -echo "==> swift build (build path: .build-linux)" -swift build --build-path .build-linux - -BUILD_DIR="$(ls -d .build-linux/*-unknown-linux-gnu/debug 2>/dev/null | head -1)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate Linux build dir under .build-linux/" >&2 - ls -la .build-linux/ || true - exit 1 -fi - -POC_DIR=/tmp/syntaxkit-poc-step7 -DEMO_DIR="$POC_DIR/demo" -OUTPUT_CACHE="$HOME/.cache/syntaxkit/outputs" - -rm -rf "$POC_DIR" "$HOME/.cache/syntaxkit" -mkdir -p "$POC_DIR/lib" "$DEMO_DIR/Helpers" - -cp "$BUILD_DIR/skitrun" "$POC_DIR/skitrun" -cp "$BUILD_DIR/libSyntaxKit.so" "$POC_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$POC_DIR/lib/" -cp -r .build-linux/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include \ - "$POC_DIR/lib/_SwiftSyntaxCShims-include" - -cat > "$DEMO_DIR/Helpers/Models.swift" <<'SWIFT' -import SyntaxKit - -public func equatableModel( - _ name: String, - fields: [(name: String, type: String)] -) -> any CodeBlock { - Struct(name) { - for field in fields { - Variable(.let, name: field.name, type: field.type) - } - }.inherits("Equatable") -} -SWIFT - -cat > "$DEMO_DIR/Input.swift" <<'SWIFT' -import SyntaxKitHelpers - -equatableModel("Person", fields: [ - ("name", "String"), - ("age", "Int"), -]) -SWIFT - -echo -echo "==> Cold run (helpers compile + output cache miss):" -time "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" - -echo -echo "==> Warm run (output cache hit, no swift spawn):" -time "$POC_DIR/skitrun" "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> --no-cache (swift spawn, helpers reused):" -time "$POC_DIR/skitrun" --no-cache "$DEMO_DIR/Input.swift" >/dev/null - -echo -echo "==> Output cache entries:" -find "$OUTPUT_CACHE" -maxdepth 2 -type f 2>/dev/null | sed "s|$OUTPUT_CACHE||" | sort - -echo -echo "==> Linux smoke test passed." diff --git a/Docs/skit.md b/Docs/skit.md new file mode 100644 index 0000000..8702ff5 --- /dev/null +++ b/Docs/skit.md @@ -0,0 +1,209 @@ +# `skit` — a SyntaxKit CLI for config-driven Swift codegen + +`skit` is a small CLI that takes a SyntaxKit DSL file as input and writes Swift source out the other side. The vision: pure data — JSON, YAML, your own format — drives a manifest written in the SyntaxKit DSL, and `skit` materializes it into idiomatic Swift you check in alongside everything else. Less hand-maintenance, fewer drift bugs. + +This doc walks through how `skit` is built. Two verbs, one wrap-and-spawn pipeline, two layers of cache, and a careful toolchain story underneath. Where there are sharp edges, this doc names them. + +## Two verbs + +``` +skit run Input.swift # SyntaxKit DSL → Swift source on stdout +skit run Input.swift -o Out.swift +skit run InputDir/ -o OutDir/ # walk **/*.swift and mirror rendered output +skit parse < Input.swift # Swift source → JSON syntax tree +``` + +`run` is the default subcommand. `skit Input.swift` is shorthand for `skit run Input.swift`. `parse` is a one-shot for the inverse direction (Swift source → SwiftSyntax tree as JSON) — useful for tooling that wants to introspect existing code. + +The remainder of this doc is about `run`, which is where the interesting design decisions live. + +## How `skit run` works + +A `.swift` input file looks like a SyntaxKit DSL expression at the top level: + +```swift +// Models.swift +import SyntaxKitHelpers // optional — only if a Helpers/ dir is present + +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +``` + +The input is not a complete Swift program. It has no `@main`, no `print`, no `let root = …`. `skit run` adds the boilerplate by wrapping the input in a `Group { … }` builder and a top-level `print` that renders the result: + +```swift +// What `skit run` writes to a temp file before spawning `swift`: +import SyntaxKit +import SyntaxKitHelpers // hoisted from the input + +let __skit_root = Group { +#sourceLocation(file: "/path/to/Models.swift", line: 3) +equatableModel("Person", fields: [ + ("name", "String"), + ("age", "Int"), +]) +#sourceLocation() +} + +print(__skit_root.generateCode()) +``` + +`skit` then spawns `swift` (the script-mode interpreter, not `swiftc`) on this wrapped file, captures stdout, and writes the result to the user's chosen output target. + +Three things deserve a closer look: + +**Imports get hoisted.** The wrapper has to start with `import SyntaxKit` so the DSL types are available. The user's `import`s — typically `import SyntaxKitHelpers` plus anything their Helpers/ module references — need to live at the top of the file, not inside `Group { … }`. `skit` parses the input with SwiftSyntax, peels off the leading `import` declarations, and lifts them into the wrapper preamble. Anything else (declarations, expressions, top-level types) stays in the body. + +**`#sourceLocation` keeps diagnostics readable.** When the spawned `swift` emits a compile error, it reports a line number in the wrapped temp file, which is meaningless to the user. The `#sourceLocation` directive remaps body diagnostics back to the original input path and line. Errors in the wrapper preamble (the `import` block, the `Group { … }` opening) still reference the temp file — `skit` rewrites occurrences of the temp path in stderr to the input path as a fallback, so users see something coherent. + +**`swift` runs in script mode.** Running `swift Input.swift` invokes the Swift interpreter rather than going through `swiftc` + `ld`. Cold-start is around 700ms on macOS; warm spawns are around 110ms. The CLI's hot path leans into this — for batch input via `skit run InputDir/`, we spawn one `swift` per input file in parallel up to the active core count. + +## Caches + +Two layers, both keyed by content hash. They live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. + +| Layer | Path | What it skips on hit | +| ------- | ----------------------------- | --------------------------------------------- | +| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | +| Output | `outputs//output.swift` | the `swift` spawn for an input | + +**Helpers cache.** Each project's `Helpers/` directory gets compiled into `libSyntaxKitHelpers.{dylib,so}` once and reused. The cache key is a hash of the helper sources, plus the bundled `libSyntaxKit` stamp, plus `swift --version`. Touching a helper file invalidates one shard; updating the toolchain invalidates everything. Hit on a warm cache: skip the ~1–2s `swiftc` compile entirely. + +**Output cache.** The fully-rendered output of an input gets cached by a hash of (input bytes, helpers shard, libSyntaxKit stamp, swift version, sorted `SKIT_*`/`SYNTAXKIT_*` env vars). On hit, `skit` doesn't spawn `swift` at all — total wall time is around 0.14s, dominated by hash + file read. Cold miss matches the warm script-mode baseline (~0.5s). + +`--no-cache` skips the output cache. There's no flag to skip the helpers cache — invalidate it by touching a helper or bumping the toolchain. + +## Toolchain stamping + +Pre-compiled Swift modules (`SyntaxKit.swiftmodule`) are tightly coupled to the compiler that built them. Even patch-level differences fail to load: + +``` +error: module compiled with Swift 6.3 cannot be imported by the Swift 6.3.2 compiler +``` + +If `skit`'s bundled `lib/SyntaxKit.swiftmodule` doesn't match the user's `swift`, the spawned interpreter emits exactly this diagnostic and refuses to compile the wrapped input. The user is left staring at a cryptic message that doesn't name the actual problem. + +`skit` mitigates this by recording the build toolchain at bundle time. `Scripts/build-skit-release.sh` writes `lib/swift-version.txt` containing the output of `swift --version`. On startup, `skit run` reads the stamp and compares it to a freshly-captured local `swift --version`. Three paths: + +- **Match.** Proceed silently. +- **Mismatch.** Print a clear error naming both versions and the rebuild command, exit 2. Skip with `--no-toolchain-check`. +- **Missing stamp.** Print a one-line note and proceed. Older bundles built before this check existed shouldn't break. + +The comparison is exact-string match. Patch-level drift broke the originating bug, so anything less strict would just defer the failure to the cryptic message we're trying to avoid. + +**What this doesn't do**: it doesn't fix the mismatch, just surfaces it. The fix is to re-run the release script with the user's current toolchain. The auto-rebuild path — bundle SyntaxKit sources alongside the prebuilt module and recompile transparently when the stamp doesn't match — is tracked as [issue #157](https://github.com/brightdigit/SyntaxKit/issues/157). + +## Timeout watchdog + +The spawned `swift` is the only unbounded piece of `skit run`'s hot path. The wrap step is microseconds. Helpers compile is cached. The output cache hits or misses in milliseconds. But the spawn itself runs *user code* — and that code is allowed to be arbitrarily slow, recursive, or stuck. + +`skit run` defaults to a 60s per-input timeout. On expiry it sends `SIGTERM`, gives a 5s grace, then `SIGKILL`. The wrapped input exits with code 124 — POSIX `timeout(1)`'s convention. `--timeout ` overrides the default; `--timeout 0` disables the watchdog entirely (useful for debugging genuinely long codegen). + +The implementation is `DispatchSemaphore.wait(timeout: deadline)` paired with a `process.terminationHandler` that signals on child exit. The Linux Foundation `Process.waitUntilExit()` hangs on already-exited children on some configurations, which is why `skit` uses the semaphore-based wait everywhere. Same story for pipe drains — sequential reads after the child exits can deadlock when either pipe (~64 KB buffer on Linux) fills before exit, so both pipes drain concurrently via `DispatchGroup`. + +## Helpers + +Shared codegen utilities live in a `Helpers/` directory. `skit` walks up from the input, finds the nearest `Helpers/`, and compiles its sources into a Swift dylib that the wrapped input can `import SyntaxKitHelpers`. + +``` +project/ +├── Helpers/ +│ └── Models.swift # public func equatableModel(_:fields:) -> any CodeBlock +└── inputs/ + ├── Person.swift # imports SyntaxKitHelpers, calls equatableModel(...) + └── Pet.swift # same +``` + +Files prefixed with `_` are skipped — a convention for private helpers within the helpers module. The module name is hard-coded to `SyntaxKitHelpers`. + +The helpers cache is per-content, so editing a helper triggers a fresh compile but reading the same helper across many inputs hits the cache. + +## Sharp edges + +### `if`-in-`Group` crashes the Swift type-checker — [#158](https://github.com/brightdigit/SyntaxKit/issues/158) + +`CodeBlockBuilderResult` declares all the methods needed for `if`/`else` in a result builder (`buildEither`, `buildOptional`, `buildArray`), but using them surfaces a Swift type-checker bug: + +```swift +let _ = Group { + if true { // ← error: failed to produce diagnostic for expression + Struct("A") { … } + } +} +``` + +This is a Swift compiler bug, not a `skit` bug. The workaround is to hoist the conditional into a helper function that uses **plain Swift `if`/`else`** (not a `Group { if … }` body) to return one of two `CodeBlock`s: + +```swift +// In Helpers/Models.swift +public func optionalDebugField(_ include: Bool) -> any CodeBlock { + if include { + return Variable(.let, name: "debug", type: "Bool") + } else { + return Group {} // empty Group as the "no-op" branch — there's no + // public EmptyCodeBlock type + } +} + +// In Input.swift +Struct("Config") { + Variable(.let, name: "name", type: "String") + optionalDebugField(buildIsDebug) +} +``` + +The helper itself can't use `Group { if … }` either — same crash. Plain Swift control flow only. + +### `@main` and top-level decl attributes don't work + +`@main` and other decl attributes (like top-level `@available`) at the start of the input fail to compile. The wrapper places the user's body inside `Group { … }`, where these attributes try to bind to a function-call expression rather than a declaration. Examples: + +```swift +@main // ❌ error: expected declaration +Struct("Person") { … } + +@available(iOS 17, *) // ❌ error: expected declaration +Struct("ModernView") { … } +``` + +The DSL has its own mechanism for attribute attachment — call `.attribute("Published")`, `.attribute("available", arguments: ["iOS 17", "*"])` on the `Variable`/`Struct`/etc. — which renders the attribute in the *output*, not on the DSL expression itself. That's the path users should take. + +### Comments don't carry through + +Swift comments in the input file (`// MARK: - Models`, block comments, inline `//` after a DSL line) don't render to the output. The input's comments are *only* for the input's author. To emit a comment in the rendered code, attach it to a `CodeBlock` via `.comment { Line(.doc, "…") }`. + +### `#if os(...)` is gen-time, not output + +A `#if os(macOS) … #endif` block in the input is evaluated when the wrapped file compiles, controlling whether the enclosed DSL expressions are part of the builder. It does *not* emit a `#if os(macOS)` directive into the rendered Swift output. If you want compile-time-conditional output, emit the `#if` lines as raw text via `VariableExp("#if os(macOS)")` (or write a small helper). + +## Platform notes + +**macOS** is the primary target. The build and release flows live in `Scripts/`; the bundle is portable across machines with the same Swift version. + +**Linux** is verified on `swift:6.0-jammy/aarch64`. Two adjustments compared to macOS: + +- `swift-crypto` replaces CryptoKit (we depend on Crypto for cache-key hashing). `swift-crypto` statically links boringssl on Linux, so the binary is noticeably larger than the macOS one. +- The Mach-O `install_name` rewrite in `Scripts/build-skit-release.sh` is skipped on Linux. GNU `ld` doesn't accept the flag; the `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. + +The Foundation.Process workarounds described earlier are Linux-driven. `Process.waitUntilExit()` blocks indefinitely on already-exited children on `swift:6.0-jammy/aarch64` — `skit` uses `DispatchSemaphore` everywhere a child wait is needed, and drains stdout/stderr pipes concurrently to avoid deadlocks when either pipe buffer fills. + +**Windows** is not supported. + +## What's deferred + +A few things were considered for v1 and explicitly punted: + +- **Auto-rebuild on toolchain mismatch** — [#157](https://github.com/brightdigit/SyntaxKit/issues/157). Today the user gets a clear error and a rebuild command; tomorrow `skit` should rebuild `libSyntaxKit` from bundled sources transparently and cache the result per Swift version. The stamp-and-detect path shipped in this release is the foundation; the rebuild fallback is the natural next step. +- **Multi-file output from a single input** — out of scope. The folder mode handles N-inputs → N-outputs; if you need fan-out from one logical generator, split it into multiple input files. +- **Sandboxing the spawned `swift`** — out of scope. The threat model is "you ran your own code", same as `swift Input.swift` by hand. +- **HTTP / web-server form** — out of scope for the CLI. A long-lived server could share a warm interpreter and the caches across requests, but that's a different shape with its own isolation questions. Revisit later. + +## Reference + +- [`Sources/skit/README.md`](../Sources/skit/README.md) — per-target quick reference (flag table, helpers layout). +- [`Scripts/build-skit-release.sh`](../Scripts/build-skit-release.sh) — release-bundle builder. +- [`Docs/research/tuist-manifest-pipeline.md`](research/tuist-manifest-pipeline.md) — the manifest-pipeline pattern this CLI borrows from. +- [Issue #154](https://github.com/brightdigit/SyntaxKit/issues/154) — original tracking issue. +- [Issue #157](https://github.com/brightdigit/SyntaxKit/issues/157), [Issue #158](https://github.com/brightdigit/SyntaxKit/issues/158) — follow-ups. From e3e243ff06786d167388f538938cb6b0f26ed884 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 12 May 2026 16:11:04 -0400 Subject: [PATCH 22/56] Replace swift-crypto/SHA-256 with pure-Swift FNV-1a content hash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two cache keys (helpers cache, output cache) only need a stable, deterministic, cross-platform hash for content addressing — they aren't security-critical and there's no adversary trying to forge collisions. Drop swift-crypto's SHA-256 in favour of a ~10-line pure-Swift FNV-1a 64-bit hasher. - 64-bit output gives ~10⁻⁹ collision probability at 10⁶ cache entries (well past anything we'll see). - Cache keys go from 64-char hex to 16-char hex. - Drops the swift-crypto dep entirely. On macOS the binary barely changes (~4KB) since swift-crypto used CryptoKit there anyway; on Linux we no longer statically link boringssl, which was the dep's real cost. - New file: Sources/skit/ContentHasher.swift. Same streaming API (update/finalize) so the two call sites are nearly identical to the SHA-256 version. All 10 working examples re-rendered with fresh keys after cache clear; output identical. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/skit.md | 5 +-- Package.resolved | 20 +---------- Package.swift | 2 -- Sources/skit/ContentHasher.swift | 59 ++++++++++++++++++++++++++++++++ Sources/skit/Helpers.swift | 8 ++--- Sources/skit/OutputCache.swift | 12 +++---- Sources/skit/README.md | 2 +- 7 files changed, 72 insertions(+), 36 deletions(-) create mode 100644 Sources/skit/ContentHasher.swift diff --git a/Docs/skit.md b/Docs/skit.md index 8702ff5..35baa07 100644 --- a/Docs/skit.md +++ b/Docs/skit.md @@ -182,10 +182,7 @@ A `#if os(macOS) … #endif` block in the input is evaluated when the wrapped fi **macOS** is the primary target. The build and release flows live in `Scripts/`; the bundle is portable across machines with the same Swift version. -**Linux** is verified on `swift:6.0-jammy/aarch64`. Two adjustments compared to macOS: - -- `swift-crypto` replaces CryptoKit (we depend on Crypto for cache-key hashing). `swift-crypto` statically links boringssl on Linux, so the binary is noticeably larger than the macOS one. -- The Mach-O `install_name` rewrite in `Scripts/build-skit-release.sh` is skipped on Linux. GNU `ld` doesn't accept the flag; the `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. +**Linux** is verified on `swift:6.0-jammy/aarch64`. One adjustment compared to macOS: the Mach-O `install_name` rewrite in `Scripts/build-skit-release.sh` is skipped — GNU `ld` doesn't accept the flag. The `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. The Foundation.Process workarounds described earlier are Linux-driven. `Process.waitUntilExit()` blocks indefinitely on already-exited children on `swift:6.0-jammy/aarch64` — `skit` uses `DispatchSemaphore` everywhere a child wait is needed, and drains stdout/stderr pipes concurrently to avoid deadlocks when either pipe buffer fills. diff --git a/Package.resolved b/Package.resolved index 07e1898..fc8a607 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,5 +1,5 @@ { - "originHash" : "36e6466ffd4edf53c9520bb53570c9119c1a34eb0e59dd824b1c4ff4f19189a8", + "originHash" : "fb25157fbc930f88cbd845709bb74c245da530b7a57458eb846e0d74bcd9c062", "pins" : [ { "identity" : "swift-argument-parser", @@ -10,24 +10,6 @@ "version" : "1.7.1" } }, - { - "identity" : "swift-asn1", - "kind" : "remoteSourceControl", - "location" : "https://github.com/apple/swift-asn1.git", - "state" : { - "revision" : "9f542610331815e29cc3821d3b6f488db8715517", - "version" : "1.6.0" - } - }, - { - "identity" : "swift-crypto", - "kind" : "remoteSourceControl", - "location" : "https://github.com/apple/swift-crypto.git", - "state" : { - "revision" : "95ba0316a9b733e92bb6b071255ff46263bbe7dc", - "version" : "3.15.1" - } - }, { "identity" : "swift-docc-plugin", "kind" : "remoteSourceControl", diff --git a/Package.swift b/Package.swift index 551ba63..2ce09bd 100644 --- a/Package.swift +++ b/Package.swift @@ -99,7 +99,6 @@ let package = Package( dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), - .package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0"), .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0") ], targets: [ @@ -147,7 +146,6 @@ let package = Package( "SyntaxParser", .product(name: "SwiftSyntax", package: "swift-syntax"), .product(name: "SwiftParser", package: "swift-syntax"), - .product(name: "Crypto", package: "swift-crypto"), .product(name: "ArgumentParser", package: "swift-argument-parser") ], swiftSettings: swiftSettings diff --git a/Sources/skit/ContentHasher.swift b/Sources/skit/ContentHasher.swift new file mode 100644 index 0000000..6ec0e3c --- /dev/null +++ b/Sources/skit/ContentHasher.swift @@ -0,0 +1,59 @@ +// +// ContentHasher.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +/// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed +/// cache keys. The cache keys aren't security-critical — there's no +/// adversary trying to forge a collision — so we don't need a cryptographic +/// hash. 64 bits of output gives ~10⁻⁹ collision probability at 10⁶ cache +/// entries, which is well past anything we'll see in practice. +/// +/// FNV-1a is deterministic across processes and platforms (unlike the Swift +/// stdlib `Hasher`, whose seed is randomized per-process) — that +/// determinism is what makes it usable as an on-disk cache key. +internal struct ContentHasher { + private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 + private static let prime: UInt64 = 0x0000_0100_0000_01b3 + + private var state: UInt64 = ContentHasher.offsetBasis + + internal mutating func update(data: Data) { + for byte in data { + state ^= UInt64(byte) + state &*= ContentHasher.prime + } + } + + /// Returns the hash as a 16-char lowercase-hex string suitable for use as + /// a directory name. + internal func finalize() -> String { + String(format: "%016x", state) + } +} diff --git a/Sources/skit/Helpers.swift b/Sources/skit/Helpers.swift index 7a33b51..ec74dd3 100644 --- a/Sources/skit/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -27,7 +27,6 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Crypto import Foundation /// Hardcoded module name for the user's `Helpers/` compilation output. Inputs @@ -118,7 +117,8 @@ internal func buildHelpers( let cacheRoot = try syntaxKitCacheRoot() .appendingPathComponent("helpers") .appendingPathComponent(key) - let dylibPath = cacheRoot + let dylibPath = + cacheRoot .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path let fm = FileManager.default @@ -220,7 +220,7 @@ private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) t // MARK: - Cache key private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { - var hasher = SHA256() + var hasher = ContentHasher() hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) for source in sources { @@ -236,7 +236,7 @@ private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { hasher.update(data: Data(stamp.utf8)) } - return hasher.finalize().map { String(format: "%02x", $0) }.joined() + return hasher.finalize() } internal func captureSwiftVersion() -> String? { diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index d4a26df..c555702 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -27,21 +27,21 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Crypto import Foundation /// Bumped when the output cache layout changes in a way that requires invalidation. private let outputCacheSchemaVersion = "v1" -/// SHA-256 over (cache schema, input source bytes, helpers key, swift version, -/// libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any change in -/// these inputs produces a fresh key and forces a recompile. +/// 64-bit content hash over (cache schema, input source bytes, helpers key, +/// swift version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). +/// Any change in these inputs produces a fresh key and forces a recompile. +/// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. internal func outputCacheKey( inputSource: String, helpers: CompiledHelpers?, libPath: String ) -> String { - var hasher = SHA256() + var hasher = ContentHasher() hasher.update(data: Data(outputCacheSchemaVersion.utf8)) hasher.update(data: Data(inputSource.utf8)) @@ -66,7 +66,7 @@ internal func outputCacheKey( hasher.update(data: Data("\(key)=\(value)\0".utf8)) } - return hasher.finalize().map { String(format: "%02x", $0) }.joined() + return hasher.finalize() } /// Returns the cached rendered output for `key`, or nil on miss. diff --git a/Sources/skit/README.md b/Sources/skit/README.md index 5b0f77a..630250a 100644 --- a/Sources/skit/README.md +++ b/Sources/skit/README.md @@ -89,7 +89,7 @@ Output cache hit is roughly ~0.14s on macOS (no spawn at all); cold miss matches ## Platform notes - **macOS** — primary target. All build/release/test flows in `Scripts/`. -- **Linux** — verified on `swift:6.0-jammy/aarch64`. Requires `swift-crypto` instead of CryptoKit (we depend on it). The Mach-O `install_name` step in `Scripts/build-skit-release.sh` is macOS-specific and skipped on Linux. +- **Linux** — verified on `swift:6.0-jammy/aarch64`. The Mach-O `install_name` step in `Scripts/build-skit-release.sh` is macOS-specific and skipped on Linux. - **Windows** — not supported. Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. `Runner.swift` and `Helpers.swift` work around it with `terminationHandler` + `DispatchSemaphore`. From 61349dab49da310bc864fd1950702688ce919584 Mon Sep 17 00:00:00 2001 From: leogdion Date: Wed, 13 May 2026 09:48:16 -0400 Subject: [PATCH 23/56] Move skit to swift-subprocess; revert simulator OS pin to 26.4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 25759444849 broke skit's build on every non-macOS-host target. Foundation.Process is unavailable on watchOS/tvOS and on Ubuntu's wasm + wasm-embedded toolchains, and POSIX kill/SIGKILL aren't in scope on Windows. The previous skit implementation worked around Linux Foundation.Process quirks with a DispatchSemaphore + DispatchGroup pipe-drain ladder and a manual kill(pid, SIGKILL) timeout watchdog — none of which port. This commit replaces all three Foundation.Process call sites (captureSwiftVersion, compileHelpers, runSwift) with swift-subprocess (v0.4.0). The timeout watchdog becomes a withThrowingTaskGroup race against Task.sleep + cancelAll; swift-subprocess's teardown sequence handles SIGTERM → grace → SIGKILL on POSIX and the WM_CLOSE/CTRL_C_EVENT/TerminateProcess equivalent on Windows. ~70 lines of platform workaround dropped. swift-subprocess requires Swift 6.1, so swift-tools-version goes 6.0 → 6.1 (and CLAUDE.md follows). skit itself is gated by `#if canImport(Subprocess)`; on platforms where the Subprocess module isn't built (iOS, watchOS, tvOS, visionOS, Android, WASI) the new SkitStub.swift provides a caseless-enum @main that exits 1 with a clear message. The target still links — just with no usable subcommands. A regression test in Tests/SyntaxKitTests/Integration/ covers swift-subprocess #256 (stream-read hang on cancellation when grandchildren inherit pipe fds). skit's runSwift hits this scenario because swift forks the frontend + linker as grandchildren. Test passes locally in ~1s, bounded at 15s. The iOS/visionOS simulator failures in the same CI run were a separate cause: the May-13 workflow sync bumped osVersion to 26.5, but the macos-26 runner image still only ships OS 26.4 simulators, so download-platform: true couldn't fetch what the matrix asked for. Revert osVersion 26.5 → 26.4 on the iOS/watchOS/tvOS/visionOS rows; Xcode pins unchanged. Tracked for re-bump in #160. Scripts/build-skit-debug.sh is a debug-mode counterpart to the existing release-bundle script, for fast local end-to-end iteration on the DSL transformation (~10s vs. 5-15 min). Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/SyntaxKit.yml | 11 +- CLAUDE.md | 4 +- Package.resolved | 20 +- Package.swift | 21 +- Scripts/build-skit-debug.sh | 81 ++ Sources/skit/ContentHasher.swift | 52 +- Sources/skit/Helpers.swift | 435 ++++--- Sources/skit/OutputCache.swift | 138 +-- Sources/skit/Runner.swift | 1035 ++++++++--------- Sources/skit/Skit.swift | 369 +++--- Sources/skit/SkitStub.swift | 44 + .../SkitSubprocessTimeoutTests.swift | 101 ++ 12 files changed, 1282 insertions(+), 1029 deletions(-) create mode 100755 Scripts/build-skit-debug.sh create mode 100644 Sources/skit/SkitStub.swift create mode 100644 Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift diff --git a/.github/workflows/SyntaxKit.yml b/.github/workflows/SyntaxKit.yml index 1363b82..5fb86e0 100644 --- a/.github/workflows/SyntaxKit.yml +++ b/.github/workflows/SyntaxKit.yml @@ -229,12 +229,15 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" + # iOS / watchOS / tvOS / visionOS osVersion pinned to 26.4 until the + # macos-26 runner image ships Xcode 26.5 simulators — see #160. + # iOS Build Matrix - type: ios runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "iPhone 17 Pro" - osVersion: "26.5" + osVersion: "26.4" download-platform: true # watchOS Build Matrix @@ -242,7 +245,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Watch Ultra 3 (49mm)" - osVersion: "26.5" + osVersion: "26.4" download-platform: true # tvOS Build Matrix @@ -250,7 +253,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple TV" - osVersion: "26.5" + osVersion: "26.4" download-platform: true # visionOS Build Matrix @@ -258,7 +261,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Vision Pro" - osVersion: "26.5" + osVersion: "26.4" download-platform: true steps: diff --git a/CLAUDE.md b/CLAUDE.md index a09a578..008b2dd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -93,8 +93,8 @@ Sources/SyntaxKit/ 2. **skit Executable** - Command-line tool for parsing Swift code to JSON ### Platform Support -- Swift 6.0+ required -- Xcode 16.0+ for development +- Swift 6.1+ required +- Xcode 16.3+ for development ### Testing - Uses modern Swift Testing framework (`@Test` syntax) diff --git a/Package.resolved b/Package.resolved index fc8a607..99337ef 100644 --- a/Package.resolved +++ b/Package.resolved @@ -1,5 +1,5 @@ { - "originHash" : "fb25157fbc930f88cbd845709bb74c245da530b7a57458eb846e0d74bcd9c062", + "originHash" : "899b1fb6639e07a99f4d6f6c1e22b7c90948b2df293879cf18e8b0f87500bf16", "pins" : [ { "identity" : "swift-argument-parser", @@ -28,6 +28,15 @@ "version" : "1.0.0" } }, + { + "identity" : "swift-subprocess", + "kind" : "remoteSourceControl", + "location" : "https://github.com/swiftlang/swift-subprocess.git", + "state" : { + "revision" : "13d087685b95d64d6aac9b94500d347bbe84c39b", + "version" : "0.4.0" + } + }, { "identity" : "swift-syntax", "kind" : "remoteSourceControl", @@ -36,6 +45,15 @@ "revision" : "f99ae8aa18f0cf0d53481901f88a0991dc3bd4a2", "version" : "601.0.1" } + }, + { + "identity" : "swift-system", + "kind" : "remoteSourceControl", + "location" : "https://github.com/apple/swift-system", + "state" : { + "revision" : "7c6ad0fc39d0763e0b699210e4124afd5041c5df", + "version" : "1.6.4" + } } ], "version" : 3 diff --git a/Package.swift b/Package.swift index 2ce09bd..ae34413 100644 --- a/Package.swift +++ b/Package.swift @@ -1,4 +1,4 @@ -// swift-tools-version: 6.0 +// swift-tools-version: 6.1 // The swift-tools-version declares the minimum version of Swift required to build this package. import PackageDescription @@ -99,7 +99,8 @@ let package = Package( dependencies: [ .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), - .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0") + .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0"), + .package(url: "https://github.com/swiftlang/swift-subprocess.git", from: "0.4.0") ], targets: [ .target( @@ -146,13 +147,25 @@ let package = Package( "SyntaxParser", .product(name: "SwiftSyntax", package: "swift-syntax"), .product(name: "SwiftParser", package: "swift-syntax"), - .product(name: "ArgumentParser", package: "swift-argument-parser") + .product(name: "ArgumentParser", package: "swift-argument-parser"), + .product( + name: "Subprocess", + package: "swift-subprocess", + condition: .when(platforms: [.macOS, .linux, .windows]) + ) ], swiftSettings: swiftSettings ), .testTarget( name: "SyntaxKitTests", - dependencies: ["SyntaxKit"], + dependencies: [ + "SyntaxKit", + .product( + name: "Subprocess", + package: "swift-subprocess", + condition: .when(platforms: [.macOS, .linux, .windows]) + ) + ], swiftSettings: swiftSettings ), .testTarget( diff --git a/Scripts/build-skit-debug.sh b/Scripts/build-skit-debug.sh new file mode 100755 index 0000000..6e93156 --- /dev/null +++ b/Scripts/build-skit-debug.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# +# Build a self-contained skit DEBUG bundle for fast local iteration. +# +# Identical layout to Scripts/build-skit-release.sh but skips release-mode +# optimization (5-15 minute SwiftSyntax compile → ~10 seconds). Use this when +# you want to exercise the end-to-end DSL→Swift transformation locally; use +# the release script when staging an actual release bundle. +# +# Output: .build/skit-debug/{skit, lib/} + +set -euo pipefail + +if [[ "$(uname -s)" != "Darwin" ]]; then + echo "macOS-only. Linux uses a parallel flow." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +OUTPUT_DIR="$REPO_ROOT/.build/skit-debug" +PACKAGE_FILE="$REPO_ROOT/Package.swift" +PACKAGE_BACKUP="$(mktemp)" + +cleanup() { + if [[ -s "$PACKAGE_BACKUP" ]]; then + cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" + fi + rm -f "$PACKAGE_BACKUP" +} +trap cleanup EXIT + +cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" + +echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" +python3 - "$PACKAGE_FILE" <<'PY' +import sys, pathlib +p = pathlib.Path(sys.argv[1]) +src = p.read_text() +new_block = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' +if new_block in src: + print("Package.swift already has type: .dynamic — leaving as-is.") + sys.exit(0) +old_block = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' +if old_block not in src: + sys.exit("Package.swift: expected SyntaxKit library product block not found") +p.write_text(src.replace(old_block, new_block, 1)) +PY + +cd "$REPO_ROOT" + +echo "==> swift build --product skit" +swift build --product skit + +echo "==> swift build --product SyntaxKit" +swift build --product SyntaxKit + +BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" +if [[ -z "$BUILD_DIR" ]]; then + echo "Could not locate debug build dir under .build//debug" >&2 + exit 1 +fi + +echo "==> Staging $OUTPUT_DIR" +rm -rf "$OUTPUT_DIR" +mkdir -p "$OUTPUT_DIR/lib" + +cp "$BUILD_DIR/skit" "$OUTPUT_DIR/skit" +cp "$BUILD_DIR/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/" +cp -r "$BUILD_DIR/Modules/." "$OUTPUT_DIR/lib/" +cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ + "$OUTPUT_DIR/lib/_SwiftSyntaxCShims-include" + +install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true + +swift --version > "$OUTPUT_DIR/lib/swift-version.txt" + +echo +echo "==> Debug bundle ready at $OUTPUT_DIR" +echo "==> Try it:" +echo " $OUTPUT_DIR/skit Examples/Completed/card_game/dsl.swift --no-cache" diff --git a/Sources/skit/ContentHasher.swift b/Sources/skit/ContentHasher.swift index 6ec0e3c..39227b5 100644 --- a/Sources/skit/ContentHasher.swift +++ b/Sources/skit/ContentHasher.swift @@ -27,33 +27,37 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +#if canImport(Subprocess) -/// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed -/// cache keys. The cache keys aren't security-critical — there's no -/// adversary trying to forge a collision — so we don't need a cryptographic -/// hash. 64 bits of output gives ~10⁻⁹ collision probability at 10⁶ cache -/// entries, which is well past anything we'll see in practice. -/// -/// FNV-1a is deterministic across processes and platforms (unlike the Swift -/// stdlib `Hasher`, whose seed is randomized per-process) — that -/// determinism is what makes it usable as an on-disk cache key. -internal struct ContentHasher { - private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 - private static let prime: UInt64 = 0x0000_0100_0000_01b3 + import Foundation - private var state: UInt64 = ContentHasher.offsetBasis + /// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed + /// cache keys. The cache keys aren't security-critical — there's no + /// adversary trying to forge a collision — so we don't need a cryptographic + /// hash. 64 bits of output gives ~10⁻⁹ collision probability at 10⁶ cache + /// entries, which is well past anything we'll see in practice. + /// + /// FNV-1a is deterministic across processes and platforms (unlike the Swift + /// stdlib `Hasher`, whose seed is randomized per-process) — that + /// determinism is what makes it usable as an on-disk cache key. + internal struct ContentHasher { + private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 + private static let prime: UInt64 = 0x0000_0100_0000_01b3 - internal mutating func update(data: Data) { - for byte in data { - state ^= UInt64(byte) - state &*= ContentHasher.prime + private var state: UInt64 = ContentHasher.offsetBasis + + internal mutating func update(data: Data) { + for byte in data { + state ^= UInt64(byte) + state &*= ContentHasher.prime + } } - } - /// Returns the hash as a 16-char lowercase-hex string suitable for use as - /// a directory name. - internal func finalize() -> String { - String(format: "%016x", state) + /// Returns the hash as a 16-char lowercase-hex string suitable for use as + /// a directory name. + internal func finalize() -> String { + String(format: "%016x", state) + } } -} + +#endif diff --git a/Sources/skit/Helpers.swift b/Sources/skit/Helpers.swift index ec74dd3..f9511ae 100644 --- a/Sources/skit/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -27,250 +27,237 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation - -/// Hardcoded module name for the user's `Helpers/` compilation output. Inputs -/// reach the compiled helpers via `import SyntaxKitHelpers`. -internal let helpersModuleName = "SyntaxKitHelpers" - -/// Platform-specific shared-library filename for a Swift library product. -internal func dylibFilename(forLibrary name: String) -> String { - #if os(Linux) - return "lib\(name).so" - #else - return "lib\(name).dylib" - #endif -} - -/// Bumped when the cache layout changes in a way that requires invalidation. -private let helpersCacheSchemaVersion = "v1" - -/// A compiled `Helpers/` directory ready to splice into the input spawn. -internal struct CompiledHelpers: Sendable { - /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. - let outputDir: URL - /// Whether the build was reused from cache (false = freshly compiled). - let cacheHit: Bool -} - -// MARK: - Discovery - -/// Walks up from `inputURL` looking for a `Helpers/` directory. Returns the -/// first one found, or nil if no ancestor contains one. -/// -/// When `inputURL` is a file, the search starts from its parent. When it's a -/// directory, the search starts from the directory itself. -internal func discoverHelpersDir(near inputURL: URL) -> URL? { - let fm = FileManager.default - var isDirectory: ObjCBool = false - let exists = fm.fileExists(atPath: inputURL.path, isDirectory: &isDirectory) - var dir = (exists && isDirectory.boolValue) ? inputURL : inputURL.deletingLastPathComponent() - dir = dir.standardizedFileURL - - while true { - let candidate = dir.appendingPathComponent("Helpers") - var isDir: ObjCBool = false - if fm.fileExists(atPath: candidate.path, isDirectory: &isDir), isDir.boolValue { - return candidate.standardizedFileURL - } - let parent = dir.deletingLastPathComponent().standardizedFileURL - if parent.path == dir.path { return nil } - dir = parent +#if canImport(Subprocess) + + import Foundation + import Subprocess + + /// Hardcoded module name for the user's `Helpers/` compilation output. Inputs + /// reach the compiled helpers via `import SyntaxKitHelpers`. + internal let helpersModuleName = "SyntaxKitHelpers" + + /// Platform-specific shared-library filename for a Swift library product. + internal func dylibFilename(forLibrary name: String) -> String { + #if os(Linux) + return "lib\(name).so" + #else + return "lib\(name).dylib" + #endif } -} - -/// Globs `**/*.swift` under `helpersDir`, skipping files prefixed with `_`. -internal func collectHelperSources(in helpersDir: URL) throws -> [URL] { - guard - let enumerator = FileManager.default.enumerator( - at: helpersDir, - includingPropertiesForKeys: [.isRegularFileKey], - options: [.skipsHiddenFiles] - ) - else { - throw CLIError(message: "could not enumerate \(helpersDir.path)") + + /// Bumped when the cache layout changes in a way that requires invalidation. + private let helpersCacheSchemaVersion = "v1" + + /// A compiled `Helpers/` directory ready to splice into the input spawn. + internal struct CompiledHelpers: Sendable { + /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. + let outputDir: URL + /// Whether the build was reused from cache (false = freshly compiled). + let cacheHit: Bool } - var result: [URL] = [] - for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey]) - guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } - result.append(url.standardizedFileURL) + // MARK: - Discovery + + /// Walks up from `inputURL` looking for a `Helpers/` directory. Returns the + /// first one found, or nil if no ancestor contains one. + /// + /// When `inputURL` is a file, the search starts from its parent. When it's a + /// directory, the search starts from the directory itself. + internal func discoverHelpersDir(near inputURL: URL) -> URL? { + let fm = FileManager.default + var isDirectory: ObjCBool = false + let exists = fm.fileExists(atPath: inputURL.path, isDirectory: &isDirectory) + var dir = (exists && isDirectory.boolValue) ? inputURL : inputURL.deletingLastPathComponent() + dir = dir.standardizedFileURL + + while true { + let candidate = dir.appendingPathComponent("Helpers") + var isDir: ObjCBool = false + if fm.fileExists(atPath: candidate.path, isDirectory: &isDir), isDir.boolValue { + return candidate.standardizedFileURL + } + let parent = dir.deletingLastPathComponent().standardizedFileURL + if parent.path == dir.path { return nil } + dir = parent + } } - return result.sorted { $0.path < $1.path } -} - -// MARK: - Build pipeline - -/// Compiles helper sources into a per-key cache directory and returns the -/// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. -internal func buildHelpers( - helpersDir: URL, - libPath: String -) throws -> CompiledHelpers? { - let sources = try collectHelperSources(in: helpersDir) - if sources.isEmpty { return nil } - - let key = try helpersCacheKey(sources: sources, libPath: libPath) - let cacheRoot = try syntaxKitCacheRoot() - .appendingPathComponent("helpers") - .appendingPathComponent(key) - let dylibPath = - cacheRoot - .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path - - let fm = FileManager.default - if fm.fileExists(atPath: dylibPath) { - return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) + + /// Globs `**/*.swift` under `helpersDir`, skipping files prefixed with `_`. + internal func collectHelperSources(in helpersDir: URL) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: helpersDir, + includingPropertiesForKeys: [.isRegularFileKey], + options: [.skipsHiddenFiles] + ) + else { + throw CLIError(message: "could not enumerate \(helpersDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey]) + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } } - try fm.createDirectory( - at: cacheRoot.deletingLastPathComponent(), - withIntermediateDirectories: true - ) + // MARK: - Build pipeline + + /// Compiles helper sources into a per-key cache directory and returns the + /// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. + internal func buildHelpers( + helpersDir: URL, + libPath: String + ) async throws -> CompiledHelpers? { + let sources = try collectHelperSources(in: helpersDir) + if sources.isEmpty { return nil } + + let key = try await helpersCacheKey(sources: sources, libPath: libPath) + let cacheRoot = try syntaxKitCacheRoot() + .appendingPathComponent("helpers") + .appendingPathComponent(key) + let dylibPath = + cacheRoot + .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path + + let fm = FileManager.default + if fm.fileExists(atPath: dylibPath) { + return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) + } - let staging = cacheRoot.deletingLastPathComponent() - .appendingPathComponent("tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") - try fm.createDirectory(at: staging, withIntermediateDirectories: true) + try fm.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true + ) - do { - try compileHelpers(sources: sources, into: staging, libPath: libPath) - } catch { - try? fm.removeItem(at: staging) - throw error - } + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent( + "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") + try fm.createDirectory(at: staging, withIntermediateDirectories: true) - // Atomic rename into the cache path. If a peer beat us to it (rename failed - // because the destination now exists), keep theirs and drop ours. - do { - try fm.moveItem(at: staging, to: cacheRoot) - } catch { - try? fm.removeItem(at: staging) - if !fm.fileExists(atPath: dylibPath) { + do { + try await compileHelpers(sources: sources, into: staging, libPath: libPath) + } catch { + try? fm.removeItem(at: staging) throw error } + + // Atomic rename into the cache path. If a peer beat us to it (rename failed + // because the destination now exists), keep theirs and drop ours. + do { + try fm.moveItem(at: staging, to: cacheRoot) + } catch { + try? fm.removeItem(at: staging) + if !fm.fileExists(atPath: dylibPath) { + throw error + } + } + + return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) } - return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) -} - -private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) throws { - let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path - let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path - - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/env") - var args: [String] = [ - "swiftc", - "-module-name", helpersModuleName, - "-emit-module", - "-emit-module-path", modulePath, - "-parse-as-library", - "-emit-library", - "-o", dylib, - "-suppress-warnings", - "-I", libPath, - "-L", libPath, - "-lSyntaxKit", - "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-rpath", "-Xlinker", libPath, - ] - #if !os(Linux) - // @rpath install_name is macOS-only; on Linux SONAME isn't needed because - // we use rpath-based loading and the dylib lives in a cache path that's - // known at link time. - args.append(contentsOf: [ - "-Xlinker", "-install_name", - "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", - ]) - #endif - args.append(contentsOf: sources.map(\.path)) - process.arguments = args - - let stderrPipe = Pipe() - process.standardOutput = FileHandle.nullDevice - process.standardError = stderrPipe - - // Linux Foundation's `Process.waitUntilExit()` blocks indefinitely on - // already-exited children in some configurations; terminationHandler + - // semaphore is the workaround used elsewhere in this file. - let semaphore = DispatchSemaphore(value: 0) - process.terminationHandler = { _ in semaphore.signal() } - - try process.run() - - // Drain stderr BEFORE waiting on the semaphore — Linux pipe buffers are - // ~64 KB; if the child fills them we deadlock waiting for an exit that - // can't happen until we read. - let stderrData = stderrPipe.fileHandleForReading.readDataToEndOfFile() - semaphore.wait() - guard process.terminationStatus == 0 else { - let stderr = String(decoding: stderrData, as: UTF8.self) - throw CLIError( - message: """ - skit: failed to compile Helpers/ (exit \(process.terminationStatus)) - \(stderr) - """) + private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) async throws { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path + let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path + + var args: [String] = [ + "-module-name", helpersModuleName, + "-emit-module", + "-emit-module-path", modulePath, + "-parse-as-library", + "-emit-library", + "-o", dylib, + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-rpath", "-Xlinker", libPath, + ] + #if !os(Linux) + // @rpath install_name is macOS-only; on Linux SONAME isn't needed because + // we use rpath-based loading and the dylib lives in a cache path that's + // known at link time. + args.append(contentsOf: [ + "-Xlinker", "-install_name", + "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", + ]) + #endif + args.append(contentsOf: sources.map(\.path)) + + let result = try await run( + .name("swiftc"), + arguments: Arguments(args), + output: .discarded, + error: .string(limit: 1 * 1_024 * 1_024) + ) + + guard result.terminationStatus.isSuccess else { + let stderr = result.standardError ?? "" + throw CLIError( + message: """ + skit: failed to compile Helpers/ (\(result.terminationStatus)) + \(stderr) + """) + } } -} -// MARK: - Cache key + // MARK: - Cache key -private func helpersCacheKey(sources: [URL], libPath: String) throws -> String { - var hasher = ContentHasher() - hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) + private func helpersCacheKey(sources: [URL], libPath: String) async throws -> String { + var hasher = ContentHasher() + hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) - for source in sources { - let data = try Data(contentsOf: source) - hasher.update(data: Data(source.lastPathComponent.utf8)) - hasher.update(data: data) + for source in sources { + let data = try Data(contentsOf: source) + hasher.update(data: Data(source.lastPathComponent.utf8)) + hasher.update(data: data) + } + + if let swiftVersion = await captureSwiftVersion() { + hasher.update(data: Data(swiftVersion.utf8)) + } + if let stamp = libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } + + return hasher.finalize() } - if let swiftVersion = captureSwiftVersion() { - hasher.update(data: Data(swiftVersion.utf8)) + internal func captureSwiftVersion() async -> String? { + let result = try? await run( + .name("swift"), + arguments: ["--version"], + output: .string(limit: 4_096), + error: .discarded + ) + return result?.standardOutput } - if let stamp = libStamp(libPath: libPath) { - hasher.update(data: Data(stamp.utf8)) + + internal func libStamp(libPath: String) -> String? { + let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" + guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } + let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 + let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 + return "\(size)/\(Int(mtime))" } - return hasher.finalize() -} - -internal func captureSwiftVersion() -> String? { - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/env") - process.arguments = ["swift", "--version"] - let pipe = Pipe() - process.standardOutput = pipe - process.standardError = FileHandle.nullDevice - let semaphore = DispatchSemaphore(value: 0) - process.terminationHandler = { _ in semaphore.signal() } - do { try process.run() } catch { return nil } - let data = pipe.fileHandleForReading.readDataToEndOfFile() - semaphore.wait() - return String(decoding: data, as: UTF8.self) -} - -internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" - guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } - let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 - let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 - return "\(size)/\(Int(mtime))" -} - -internal func syntaxKitCacheRoot() throws -> URL { - if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { - return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + internal func syntaxKitCacheRoot() throws -> URL { + if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { + return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + } + let home = NSHomeDirectory() + #if os(macOS) + return URL(fileURLWithPath: home) + .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") + #else + return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") + #endif } - let home = NSHomeDirectory() - #if os(macOS) - return URL(fileURLWithPath: home) - .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") - #else - return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") - #endif -} + +#endif diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index c555702..47f7972 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -27,85 +27,89 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +#if canImport(Subprocess) -/// Bumped when the output cache layout changes in a way that requires invalidation. -private let outputCacheSchemaVersion = "v1" + import Foundation -/// 64-bit content hash over (cache schema, input source bytes, helpers key, -/// swift version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). -/// Any change in these inputs produces a fresh key and forces a recompile. -/// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. -internal func outputCacheKey( - inputSource: String, - helpers: CompiledHelpers?, - libPath: String -) -> String { - var hasher = ContentHasher() - hasher.update(data: Data(outputCacheSchemaVersion.utf8)) - hasher.update(data: Data(inputSource.utf8)) + /// Bumped when the output cache layout changes in a way that requires invalidation. + private let outputCacheSchemaVersion = "v1" - if let helpers { - // Helpers cache dir name *is* the helpers cache key (per Helpers.swift). - hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) - } else { - hasher.update(data: Data("no-helpers".utf8)) - } + /// 64-bit content hash over (cache schema, input source bytes, helpers key, + /// swift version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). + /// Any change in these inputs produces a fresh key and forces a recompile. + /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. + internal func outputCacheKey( + inputSource: String, + helpers: CompiledHelpers?, + libPath: String + ) async -> String { + var hasher = ContentHasher() + hasher.update(data: Data(outputCacheSchemaVersion.utf8)) + hasher.update(data: Data(inputSource.utf8)) - if let version = captureSwiftVersion() { - hasher.update(data: Data(version.utf8)) - } - if let stamp = libStamp(libPath: libPath) { - hasher.update(data: Data(stamp.utf8)) - } + if let helpers { + // Helpers cache dir name *is* the helpers cache key (per Helpers.swift). + hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) + } else { + hasher.update(data: Data("no-helpers".utf8)) + } - let env = ProcessInfo.processInfo.environment - .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } - .sorted { $0.key < $1.key } - for (key, value) in env { - hasher.update(data: Data("\(key)=\(value)\0".utf8)) - } + if let version = await captureSwiftVersion() { + hasher.update(data: Data(version.utf8)) + } + if let stamp = libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } - return hasher.finalize() -} + let env = ProcessInfo.processInfo.environment + .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } + .sorted { $0.key < $1.key } + for (key, value) in env { + hasher.update(data: Data("\(key)=\(value)\0".utf8)) + } -/// Returns the cached rendered output for `key`, or nil on miss. -internal func lookupCachedOutput(key: String) -> Data? { - guard let dir = try? outputCacheDir(for: key) else { return nil } - return try? Data(contentsOf: dir.appendingPathComponent("output.swift")) -} + return hasher.finalize() + } -/// Atomically stores `data` under `key`. Concurrent writers race via a -/// `tmp../` staging dir + rename; the loser drops their copy. -internal func storeCachedOutput(key: String, data: Data) throws { - let cacheRoot = try outputCacheDir(for: key) - let final = cacheRoot.appendingPathComponent("output.swift") - let fm = FileManager.default + /// Returns the cached rendered output for `key`, or nil on miss. + internal func lookupCachedOutput(key: String) -> Data? { + guard let dir = try? outputCacheDir(for: key) else { return nil } + return try? Data(contentsOf: dir.appendingPathComponent("output.swift")) + } - try fm.createDirectory( - at: cacheRoot.deletingLastPathComponent(), - withIntermediateDirectories: true - ) + /// Atomically stores `data` under `key`. Concurrent writers race via a + /// `tmp../` staging dir + rename; the loser drops their copy. + internal func storeCachedOutput(key: String, data: Data) throws { + let cacheRoot = try outputCacheDir(for: key) + let final = cacheRoot.appendingPathComponent("output.swift") + let fm = FileManager.default - let staging = cacheRoot.deletingLastPathComponent() - .appendingPathComponent( - "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + try fm.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true ) - try fm.createDirectory(at: staging, withIntermediateDirectories: true) - try data.write(to: staging.appendingPathComponent("output.swift")) - do { - try fm.moveItem(at: staging, to: cacheRoot) - } catch { - try? fm.removeItem(at: staging) - if !fm.fileExists(atPath: final.path) { - throw error + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent( + "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + ) + try fm.createDirectory(at: staging, withIntermediateDirectories: true) + try data.write(to: staging.appendingPathComponent("output.swift")) + + do { + try fm.moveItem(at: staging, to: cacheRoot) + } catch { + try? fm.removeItem(at: staging) + if !fm.fileExists(atPath: final.path) { + throw error + } } } -} -private func outputCacheDir(for key: String) throws -> URL { - try syntaxKitCacheRoot() - .appendingPathComponent("outputs") - .appendingPathComponent(key) -} + private func outputCacheDir(for key: String) throws -> URL { + try syntaxKitCacheRoot() + .appendingPathComponent("outputs") + .appendingPathComponent(key) + } + +#endif diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index 957df18..d9407f4 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -27,585 +27,578 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation -import SwiftParser -import SwiftSyntax - -// MARK: - Helpers resolution - -internal enum HelpersOptions { - case auto - case disabled - case explicit(String) -} - -internal func resolveHelpers( - nearInputPath path: String, - libPath: String, - options: HelpersOptions -) throws -> CompiledHelpers? { - let helpersDir: URL? - switch options { - case .disabled: - return nil - case .auto: - helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) - case .explicit(let dir): - let url = URL(fileURLWithPath: dir).standardizedFileURL - var isDir: ObjCBool = false - guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), - isDir.boolValue - else { - throw CLIError(message: "--helpers path is not a directory: \(dir)") - } - helpersDir = url - } - guard let helpersDir else { return nil } +#if canImport(Subprocess) + + import Foundation + import Subprocess + import SwiftParser + import SwiftSyntax - guard let compiled = try buildHelpers(helpersDir: helpersDir, libPath: libPath) else { - return nil + // MARK: - Helpers resolution + + internal enum HelpersOptions { + case auto + case disabled + case explicit(String) } - let suffix = compiled.cacheHit ? "cached" : "compiled" - FileHandle.standardError.write( - Data( - "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 - )) - return compiled -} - -// MARK: - Resource location - -/// Resolves the directory containing `libSyntaxKit.dylib` + module files, -/// in priority order: explicit flag → env var → adjacent-to-binary -/// (`/lib/`) → Homebrew layout (`/../lib/skit/`). -internal func resolveLibPath(override: String?) throws -> String { - if let override { - guard isLibDir(override) else { - throw CLIError(message: "--lib path does not look like a SyntaxKit lib dir: \(override)") + + internal func resolveHelpers( + nearInputPath path: String, + libPath: String, + options: HelpersOptions + ) async throws -> CompiledHelpers? { + let helpersDir: URL? + switch options { + case .disabled: + return nil + case .auto: + helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) + case .explicit(let dir): + let url = URL(fileURLWithPath: dir).standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), + isDir.boolValue + else { + throw CLIError(message: "--helpers path is not a directory: \(dir)") + } + helpersDir = url } - return override - } + guard let helpersDir else { return nil } - if let env = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"], !env.isEmpty { - guard isLibDir(env) else { - throw CLIError(message: "SKIT_LIB_DIR is set but path is not a lib dir: \(env)") + guard let compiled = try await buildHelpers(helpersDir: helpersDir, libPath: libPath) else { + return nil } - return env + let suffix = compiled.cacheHit ? "cached" : "compiled" + FileHandle.standardError.write( + Data( + "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 + )) + return compiled } - if let execURL = Bundle.main.executableURL?.resolvingSymlinksInPath() { - let execDir = execURL.deletingLastPathComponent() + // MARK: - Resource location - let adjacent = execDir.appendingPathComponent("lib").path - if isLibDir(adjacent) { return adjacent } + /// Resolves the directory containing `libSyntaxKit.dylib` + module files, + /// in priority order: explicit flag → env var → adjacent-to-binary + /// (`/lib/`) → Homebrew layout (`/../lib/skit/`). + internal func resolveLibPath(override: String?) throws -> String { + if let override { + guard isLibDir(override) else { + throw CLIError(message: "--lib path does not look like a SyntaxKit lib dir: \(override)") + } + return override + } - let brewLayout = execDir.deletingLastPathComponent() - .appendingPathComponent("lib/skit").path - if isLibDir(brewLayout) { return brewLayout } - } + if let env = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"], !env.isEmpty { + guard isLibDir(env) else { + throw CLIError(message: "SKIT_LIB_DIR is set but path is not a lib dir: \(env)") + } + return env + } - throw CLIError( - message: """ - Could not locate SyntaxKit lib directory. Looked for: - 1. --lib (not provided) - 2. $SKIT_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skit/ (not found) - Run Scripts/build-skit-release.sh to produce a self-contained - release bundle under .build/skit-release/. - """) -} - -private func isLibDir(_ path: String) -> Bool { - let fm = FileManager.default - var isDir: ObjCBool = false - guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } - return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") -} - -// MARK: - Toolchain check - -/// Filename for the bundle's recorded build-toolchain version. -internal let toolchainStampFilename = "swift-version.txt" - -internal enum ToolchainCheckResult { - /// Bundle stamp matches the local `swift --version` exactly. - case match - /// `/swift-version.txt` is missing (older bundle that predates - /// the stamp). skit prints a one-line note and proceeds. - case stampMissing - case mismatch(bundle: String, local: String) -} - -/// Compares `/swift-version.txt` to `captureSwiftVersion()`. -/// The swiftmodule format isn't reliably forward-compatible across even -/// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the -/// swiftmodule), so the comparison is exact-string after normalising -/// trailing whitespace. -internal func toolchainCheck(libPath: String) -> ToolchainCheckResult { - let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) - guard let stampData = try? Data(contentsOf: stampURL), - let stampRaw = String(data: stampData, encoding: .utf8) - else { - FileHandle.standardError.write( - Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) - ) - return .stampMissing - } - guard let localRaw = captureSwiftVersion() else { - FileHandle.standardError.write( - Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) - ) - return .stampMissing - } - let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) - let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) - return bundle == local ? .match : .mismatch(bundle: bundle, local: local) -} - -internal func toolchainMismatchMessage(bundle: String, local: String) -> String { - """ - skit: toolchain mismatch - bundle: \(bundle) - local: \(local) - The bundle's libSyntaxKit was built against a different `swift` than the - one on your PATH. Swift swiftmodules aren't reliably compatible across - versions, so spawning `swift` would fail with a cryptic module-version - diagnostic. - - Rebuild the bundle with: - Scripts/build-skit-release.sh - Or pass --no-toolchain-check to try anyway. - - """ -} - -// MARK: - Single-file mode - -internal func runSingleFile( - inputPath: String, - outputPath: String?, - libPath: String, - helpers: CompiledHelpers?, - useCache: Bool, - timeoutSeconds: Int -) throws { - let result = try processFile( - inputPath: inputPath, - libPath: libPath, - helpers: helpers, - useCache: useCache, - timeoutSeconds: timeoutSeconds - ) - if !result.stderr.isEmpty { - FileHandle.standardError.write(Data(result.stderr.utf8)) + if let execURL = Bundle.main.executableURL?.resolvingSymlinksInPath() { + let execDir = execURL.deletingLastPathComponent() + + let adjacent = execDir.appendingPathComponent("lib").path + if isLibDir(adjacent) { return adjacent } + + let brewLayout = execDir.deletingLastPathComponent() + .appendingPathComponent("lib/skit").path + if isLibDir(brewLayout) { return brewLayout } + } + + throw CLIError( + message: """ + Could not locate SyntaxKit lib directory. Looked for: + 1. --lib (not provided) + 2. $SKIT_LIB_DIR (not set) + 3. /lib/ (not found) + 4. /../lib/skit/ (not found) + Run Scripts/build-skit-release.sh to produce a self-contained + release bundle under .build/skit-release/. + """) } - guard result.exitCode == 0 else { - exit(result.exitCode) + + private func isLibDir(_ path: String) -> Bool { + let fm = FileManager.default + var isDir: ObjCBool = false + guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } + return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") } - if let outputPath { - try result.stdout.write(to: URL(fileURLWithPath: outputPath)) - } else { - FileHandle.standardOutput.write(result.stdout) + + // MARK: - Toolchain check + + /// Filename for the bundle's recorded build-toolchain version. + internal let toolchainStampFilename = "swift-version.txt" + + internal enum ToolchainCheckResult { + /// Bundle stamp matches the local `swift --version` exactly. + case match + /// `/swift-version.txt` is missing (older bundle that predates + /// the stamp). skit prints a one-line note and proceeds. + case stampMissing + case mismatch(bundle: String, local: String) } -} - -// MARK: - Folder mode - -internal func runDirectory( - inputDir: String, - outputDir: String, - libPath: String, - helpers: CompiledHelpers?, - useCache: Bool, - timeoutSeconds: Int -) async -> Int32 { - let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL - let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL - - let inputs: [URL] - do { - inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) - } catch { - FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) - return 1 + + /// Compares `/swift-version.txt` to `captureSwiftVersion()`. + /// The swiftmodule format isn't reliably forward-compatible across even + /// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the + /// swiftmodule), so the comparison is exact-string after normalising + /// trailing whitespace. + internal func toolchainCheck(libPath: String) async -> ToolchainCheckResult { + let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) + guard let stampData = try? Data(contentsOf: stampURL), + let stampRaw = String(data: stampData, encoding: .utf8) + else { + FileHandle.standardError.write( + Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) + ) + return .stampMissing + } + guard let localRaw = await captureSwiftVersion() else { + FileHandle.standardError.write( + Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) + ) + return .stampMissing + } + let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) + let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) + return bundle == local ? .match : .mismatch(bundle: bundle, local: local) } - if inputs.isEmpty { - FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) - return 0 + internal func toolchainMismatchMessage(bundle: String, local: String) -> String { + """ + skit: toolchain mismatch + bundle: \(bundle) + local: \(local) + The bundle's libSyntaxKit was built against a different `swift` than the + one on your PATH. Swift swiftmodules aren't reliably compatible across + versions, so spawning `swift` would fail with a cryptic module-version + diagnostic. + + Rebuild the bundle with: + Scripts/build-skit-release.sh + Or pass --no-toolchain-check to try anyway. + + """ } - let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) + // MARK: - Single-file mode + + internal func runSingleFile( + inputPath: String, + outputPath: String?, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool, + timeoutSeconds: Int + ) async throws { + let result = try await processFile( + inputPath: inputPath, + libPath: libPath, + helpers: helpers, + useCache: useCache, + timeoutSeconds: timeoutSeconds + ) + if !result.stderr.isEmpty { + FileHandle.standardError.write(Data(result.stderr.utf8)) + } + guard result.exitCode == 0 else { + exit(result.exitCode) + } + if let outputPath { + try result.stdout.write(to: URL(fileURLWithPath: outputPath)) + } else { + FileHandle.standardOutput.write(result.stdout) + } + } - var outcomes: [FileOutcome] = [] - var iterator = inputs.makeIterator() + // MARK: - Folder mode + + internal func runDirectory( + inputDir: String, + outputDir: String, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool, + timeoutSeconds: Int + ) async -> Int32 { + let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL + let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL + + let inputs: [URL] + do { + inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) + } catch { + FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) + return 1 + } - await withTaskGroup(of: FileOutcome.self) { group in - for _ in 0.. -} - -private func runOne( - _ input: URL, - libPath: String, - helpers: CompiledHelpers?, - useCache: Bool, - timeoutSeconds: Int -) -> FileOutcome { - do { - let result = try processFile( - inputPath: input.path, - libPath: libPath, - helpers: helpers, - useCache: useCache, - timeoutSeconds: timeoutSeconds - ) - return FileOutcome(input: input, result: .success(result)) - } catch { - return FileOutcome(input: input, result: .failure(error)) + private struct FileOutcome: Sendable { + let input: URL + let result: Result } -} - -/// Returns the path of a `Helpers/` directory living directly under `inputDir`, -/// so the folder-mode enumerator can skip its descendants. Helpers that live -/// outside the input tree don't need to be excluded (they aren't enumerated). -private func helpersExcludePath(inputDir: URL) -> String? { - let candidate = inputDir.appendingPathComponent("Helpers").standardizedFileURL - var isDir: ObjCBool = false - guard FileManager.default.fileExists(atPath: candidate.path, isDirectory: &isDir), - isDir.boolValue - else { - return nil + + private func runOne( + _ input: URL, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool, + timeoutSeconds: Int + ) async -> FileOutcome { + do { + let result = try await processFile( + inputPath: input.path, + libPath: libPath, + helpers: helpers, + useCache: useCache, + timeoutSeconds: timeoutSeconds + ) + return FileOutcome(input: input, result: .success(result)) + } catch { + return FileOutcome(input: input, result: .failure(error)) + } } - return candidate.path -} - -private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { - guard - let enumerator = FileManager.default.enumerator( - at: inputDir, - includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], - options: [.skipsHiddenFiles] - ) - else { - throw CLIError(message: "could not enumerate \(inputDir.path)") + + /// Returns the path of a `Helpers/` directory living directly under `inputDir`, + /// so the folder-mode enumerator can skip its descendants. Helpers that live + /// outside the input tree don't need to be excluded (they aren't enumerated). + private func helpersExcludePath(inputDir: URL) -> String? { + let candidate = inputDir.appendingPathComponent("Helpers").standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: candidate.path, isDirectory: &isDir), + isDir.boolValue + else { + return nil + } + return candidate.path } - var result: [URL] = [] - for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) - if values.isDirectory == true { - if let excludedDir, url.standardizedFileURL.path == excludedDir { - enumerator.skipDescendants() + private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], + options: [.skipsHiddenFiles] + ) + else { + throw CLIError(message: "could not enumerate \(inputDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + if values.isDirectory == true { + if let excludedDir, url.standardizedFileURL.path == excludedDir { + enumerator.skipDescendants() + } + continue } - continue + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) } - guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } - result.append(url.standardizedFileURL) - } - return result.sorted { $0.path < $1.path } -} - -// MARK: - Per-file work - -private struct ProcessResult { - let exitCode: Int32 - let stdout: Data - let stderr: String -} - -private func processFile( - inputPath: String, - libPath: String, - helpers: CompiledHelpers?, - useCache: Bool, - timeoutSeconds: Int -) throws -> ProcessResult { - let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL - let absoluteInputPath = inputURL.path - let source = try String(contentsOf: inputURL, encoding: .utf8) - - let cacheKey: String? = - useCache - ? outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) - : nil - if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { - return ProcessResult(exitCode: 0, stdout: cached, stderr: "") + return result.sorted { $0.path < $1.path } } - let wrapped = wrap(source: source, originalPath: absoluteInputPath) - - let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skit-\(UUID().uuidString)") - try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) - defer { try? FileManager.default.removeItem(at: tmpDir) } - - let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") - try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - - let raw = try runSwift( - wrappedPath: wrappedURL.path, - libPath: libPath, - helpers: helpers, - timeoutSeconds: timeoutSeconds - ) - // #sourceLocation maps body diagnostics back to the input file. Errors in - // the preamble (lines outside the body) still reference the wrapper — - // rewrite literal occurrences of its path so users see something coherent. - let stderr = raw.stderr.replacingOccurrences( - of: wrappedURL.path, - with: absoluteInputPath - ) - - if let cacheKey, raw.exitCode == 0 { - try? storeCachedOutput(key: cacheKey, data: raw.stdout) + // MARK: - Per-file work + + private struct ProcessResult: Sendable { + let exitCode: Int32 + let stdout: Data + let stderr: String } - return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) -} - -internal struct CLIError: Error, CustomStringConvertible { - let message: String - var description: String { message } -} - -// MARK: - Wrapping - -/// Splits the input into hoisted `import` declarations and a verbatim body, -/// returning a complete Swift program that runs SyntaxKit on the body. -/// -/// The body is fenced in `#sourceLocation` directives so compiler diagnostics -/// in the body reference the original input file and line numbers. -internal func wrap(source: String, originalPath: String) -> String { - let tree = Parser.parse(source: source) - let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) - - // Find the first non-import top-level statement; everything before it that - // is an import gets hoisted, anything before that which is *not* an import - // stays in the body (e.g. a top-level `// comment` is left alone). - var hoisted: [String] = [] - var firstBodyByte: AbsolutePosition? - - for item in tree.statements { - if let importDecl = item.item.as(ImportDeclSyntax.self), - firstBodyByte == nil - { - hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) - continue + private func processFile( + inputPath: String, + libPath: String, + helpers: CompiledHelpers?, + useCache: Bool, + timeoutSeconds: Int + ) async throws -> ProcessResult { + let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL + let absoluteInputPath = inputURL.path + let source = try String(contentsOf: inputURL, encoding: .utf8) + + let cacheKey: String? = + useCache + ? await outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) + : nil + if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { + return ProcessResult(exitCode: 0, stdout: cached, stderr: "") } - firstBodyByte = item.position - break - } - let body: String - let firstBodyLine: Int - if let firstBodyByte { - let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) - body = String(source[start...]) - firstBodyLine = locConverter.location(for: firstBodyByte).line - } else { - body = "" - firstBodyLine = 1 - } + let wrapped = wrap(source: source, originalPath: absoluteInputPath) + + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("skit-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + + let raw = try await runSwift( + wrappedPath: wrappedURL.path, + libPath: libPath, + helpers: helpers, + timeoutSeconds: timeoutSeconds + ) + // #sourceLocation maps body diagnostics back to the input file. Errors in + // the preamble (lines outside the body) still reference the wrapper — + // rewrite literal occurrences of its path so users see something coherent. + let stderr = raw.stderr.replacingOccurrences( + of: wrappedURL.path, + with: absoluteInputPath + ) - let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" - - // #sourceLocation must use a forward-slash path; escape backslashes/quotes - // defensively even though macOS paths shouldn't contain them. - let escapedPath = - originalPath - .replacingOccurrences(of: "\\", with: "\\\\") - .replacingOccurrences(of: "\"", with: "\\\"") - - return """ - import SyntaxKit - \(hoistedBlock) - let __skit_root = Group { - #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) - \(body) - #sourceLocation() + if let cacheKey, raw.exitCode == 0 { + try? storeCachedOutput(key: cacheKey, data: raw.stdout) } - print(__skit_root.generateCode()) - """ -} - -// MARK: - Spawning swift - -/// Exit code returned when the spawned `swift` is killed by skit's timeout -/// watchdog. Matches POSIX `timeout(1)`. -private let timeoutExitCode: Int32 = 124 - -/// Grace period between SIGTERM and SIGKILL when the child won't exit on its own. -private let killGraceSeconds: Int = 5 - -private func runSwift( - wrappedPath: String, - libPath: String, - helpers: CompiledHelpers?, - timeoutSeconds: Int -) throws -> ProcessResult { - let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - - var arguments: [String] = [ - "swift", - "-suppress-warnings", - "-I", libPath, - "-L", libPath, - "-lSyntaxKit", - "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-rpath", "-Xlinker", libPath, - ] - - if let helpers { - let helpersPath = helpers.outputDir.path - arguments.append(contentsOf: [ - "-I", helpersPath, - "-L", helpersPath, - "-l\(helpersModuleName)", - "-Xlinker", "-rpath", "-Xlinker", helpersPath, - ]) + return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } - arguments.append(wrappedPath) - - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/env") - process.arguments = arguments - - let stdoutPipe = Pipe() - let stderrPipe = Pipe() - process.standardOutput = stdoutPipe - process.standardError = stderrPipe - - // Linux Foundation's `Process.waitUntilExit()` blocks indefinitely on - // already-exited children in some configurations; terminationHandler + - // semaphore is the workaround. - let exitSemaphore = DispatchSemaphore(value: 0) - process.terminationHandler = { _ in exitSemaphore.signal() } - - try process.run() - - // Drain both pipes concurrently — reading sequentially deadlocks on Linux - // when either pipe (~64 KB buffer) fills before the child exits. Box the - // buffers in classes so Swift 6 strict-concurrency is satisfied without - // `@unchecked Sendable` on local vars. - let outBox = PipeDataBox() - let errBox = PipeDataBox() - let group = DispatchGroup() - group.enter() - DispatchQueue.global().async { - outBox.value = stdoutPipe.fileHandleForReading.readDataToEndOfFile() - group.leave() - } - group.enter() - DispatchQueue.global().async { - errBox.value = stderrPipe.fileHandleForReading.readDataToEndOfFile() - group.leave() + internal struct CLIError: Error, CustomStringConvertible { + let message: String + var description: String { message } } - // Timeout watchdog: wait for the child with a deadline. On expiry, send - // SIGTERM, give a fixed grace, then SIGKILL. timeoutSeconds == 0 disables. - let timedOut: Bool - if timeoutSeconds > 0 { - let deadline: DispatchTime = .now() + .seconds(timeoutSeconds) - if exitSemaphore.wait(timeout: deadline) == .timedOut { - process.terminate() // SIGTERM - if exitSemaphore.wait(timeout: .now() + .seconds(killGraceSeconds)) == .timedOut { - kill(process.processIdentifier, SIGKILL) - exitSemaphore.wait() + // MARK: - Wrapping + + /// Splits the input into hoisted `import` declarations and a verbatim body, + /// returning a complete Swift program that runs SyntaxKit on the body. + /// + /// The body is fenced in `#sourceLocation` directives so compiler diagnostics + /// in the body reference the original input file and line numbers. + internal func wrap(source: String, originalPath: String) -> String { + let tree = Parser.parse(source: source) + let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) + + // Find the first non-import top-level statement; everything before it that + // is an import gets hoisted, anything before that which is *not* an import + // stays in the body (e.g. a top-level `// comment` is left alone). + var hoisted: [String] = [] + var firstBodyByte: AbsolutePosition? + + for item in tree.statements { + if let importDecl = item.item.as(ImportDeclSyntax.self), + firstBodyByte == nil + { + hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) + continue } - timedOut = true + firstBodyByte = item.position + break + } + + let body: String + let firstBodyLine: Int + if let firstBodyByte { + let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) + body = String(source[start...]) + firstBodyLine = locConverter.location(for: firstBodyByte).line } else { - timedOut = false + body = "" + firstBodyLine = 1 } - } else { - exitSemaphore.wait() - timedOut = false + + let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" + + // #sourceLocation must use a forward-slash path; escape backslashes/quotes + // defensively even though macOS paths shouldn't contain them. + let escapedPath = + originalPath + .replacingOccurrences(of: "\\", with: "\\\\") + .replacingOccurrences(of: "\"", with: "\\\"") + + return """ + import SyntaxKit + \(hoistedBlock) + let __skit_root = Group { + #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) + \(body) + #sourceLocation() + } + + print(__skit_root.generateCode()) + """ } - // Child is dead now — pipes get EOF, drain completes shortly. - group.wait() - if timedOut { - let prefix = Data("skit: timed out after \(timeoutSeconds)s\n".utf8) - let stderr = String(decoding: prefix + errBox.value, as: UTF8.self) - return ProcessResult(exitCode: timeoutExitCode, stdout: outBox.value, stderr: stderr) + // MARK: - Spawning swift + + /// Exit code returned when the spawned `swift` is killed by skit's timeout + /// watchdog. Matches POSIX `timeout(1)`. + private let timeoutExitCode: Int32 = 124 + + /// Bounded output capacity for the spawned `swift` (16 MiB). Generated DSL + /// output above this size is exotic; if we ever hit it we'll see a clear + /// SubprocessError rather than a silent truncation. + private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 + private let stderrLimitBytes: Int = 1 * 1_024 * 1_024 + + private enum SwiftRunOutcome: Sendable { + case completed(exitCode: Int32, stdout: Data, stderr: String) + case timedOut } - return ProcessResult( - exitCode: process.terminationStatus, - stdout: outBox.value, - stderr: String(decoding: errBox.value, as: UTF8.self) - ) -} + private func runSwift( + wrappedPath: String, + libPath: String, + helpers: CompiledHelpers?, + timeoutSeconds: Int + ) async throws -> ProcessResult { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + + var arguments: [String] = [ + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-rpath", "-Xlinker", libPath, + ] + + if let helpers { + let helpersPath = helpers.outputDir.path + arguments.append(contentsOf: [ + "-I", helpersPath, + "-L", helpersPath, + "-l\(helpersModuleName)", + "-Xlinker", "-rpath", "-Xlinker", helpersPath, + ]) + } + + arguments.append(wrappedPath) + let argumentsCopy = arguments + + let invocation: @Sendable () async throws -> SwiftRunOutcome = { + let record = try await run( + .name("swift"), + arguments: Arguments(argumentsCopy), + output: .string(limit: stdoutLimitBytes), + error: .string(limit: stderrLimitBytes) + ) + return .completed( + exitCode: exitCode(from: record.terminationStatus), + stdout: Data((record.standardOutput ?? "").utf8), + stderr: record.standardError ?? "" + ) + } + + let outcome: SwiftRunOutcome + if timeoutSeconds <= 0 { + outcome = try await invocation() + } else { + outcome = try await withThrowingTaskGroup(of: SwiftRunOutcome.self) { group in + group.addTask { try await invocation() } + group.addTask { + try await Task.sleep(for: .seconds(timeoutSeconds)) + return .timedOut + } + let first = try await group.next()! + group.cancelAll() + return first + } + } + + switch outcome { + case .completed(let exitCode, let stdout, let stderr): + return ProcessResult(exitCode: exitCode, stdout: stdout, stderr: stderr) + case .timedOut: + return ProcessResult( + exitCode: timeoutExitCode, + stdout: Data(), + stderr: "skit: timed out after \(timeoutSeconds)s\n" + ) + } + } + + private func exitCode(from status: TerminationStatus) -> Int32 { + switch status { + case .exited(let code): + return Int32(truncatingIfNeeded: code) + #if !os(Windows) + case .signaled(let signal): + // Match shell convention: 128 + signal number. + return 128 + Int32(truncatingIfNeeded: signal) + #endif + } + } -private final class PipeDataBox: @unchecked Sendable { - var value = Data() -} +#endif diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index 710a725..bd594e0 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -27,217 +27,222 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import ArgumentParser -import Foundation -import SyntaxParser - -@main -internal struct Skit: AsyncParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "skit", - abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", - subcommands: [Run.self, Parse.self], - defaultSubcommand: Run.self - ) -} - -// MARK: - skit run - -extension Skit { - internal struct Run: AsyncParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "run", - abstract: "Render SyntaxKit DSL input(s) into Swift source.", - discussion: """ - Wraps each input in a `Group { … }` closure and spawns `swift` to - evaluate it. The rendered output is written to stdout (single-file - mode) or mirrored into an output directory (folder mode). - - Forms: - skit run Input.swift — render to stdout - skit run Input.swift -o Out.swift — render to a file - skit run InputDir/ -o OutDir/ — walk **/*.swift (skipping - files prefixed with '_') - and mirror rendered output - into OutDir/ - """ - ) +#if canImport(Subprocess) - @Argument(help: "Path to a .swift input file or a directory of inputs.") - internal var input: String + import ArgumentParser + import Foundation + import SyntaxParser - @Option( - name: [.short, .customLong("output")], - help: "Output file (single-file mode) or directory (folder mode)." + @main + internal struct Skit: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "skit", + abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", + subcommands: [Run.self, Parse.self], + defaultSubcommand: Run.self ) - internal var output: String? + } - @Option( - name: .customLong("lib"), - help: ArgumentHelp( - "Directory containing libSyntaxKit.dylib + module files.", - discussion: """ - When omitted, skit searches: $SKIT_LIB_DIR, then /lib/, - then /../lib/skit/. Build a self-contained bundle with - Scripts/build-skit-release.sh. - """ - ) - ) - internal var libPath: String? + // MARK: - skit run - @Option( - name: .customLong("helpers"), - help: ArgumentHelp( - "Override Helpers/ directory location.", + extension Skit { + internal struct Run: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "run", + abstract: "Render SyntaxKit DSL input(s) into Swift source.", discussion: """ - By default skit walks up from the input looking for one. Helper - sources are pre-compiled into libSyntaxKitHelpers.dylib and made - importable via `import SyntaxKitHelpers`. + Wraps each input in a `Group { … }` closure and spawns `swift` to + evaluate it. The rendered output is written to stdout (single-file + mode) or mirrored into an output directory (folder mode). + + Forms: + skit run Input.swift — render to stdout + skit run Input.swift -o Out.swift — render to a file + skit run InputDir/ -o OutDir/ — walk **/*.swift (skipping + files prefixed with '_') + and mirror rendered output + into OutDir/ """ ) - ) - internal var helpersDir: String? - @Flag( - name: .customLong("no-helpers"), - help: "Skip helpers discovery entirely." - ) - internal var noHelpers: Bool = false + @Argument(help: "Path to a .swift input file or a directory of inputs.") + internal var input: String - @Flag( - name: .customLong("no-cache"), - help: ArgumentHelp( - "Skip the rendered-output cache (always run swift).", - discussion: """ - The cache lives at /outputs// and is keyed - on input bytes, helpers, swift version, libSyntaxKit stamp, and - SKIT_*/SYNTAXKIT_* env. - """ + @Option( + name: [.short, .customLong("output")], + help: "Output file (single-file mode) or directory (folder mode)." ) - ) - internal var noCache: Bool = false - - @Option( - name: .customLong("timeout"), - help: ArgumentHelp( - "Per-input timeout for the spawned `swift` (seconds).", - discussion: """ - Default 60. On expiry: SIGTERM, then SIGKILL after a 5s grace; the - file exits with code 124. Pass 0 to disable the watchdog. - """ + internal var output: String? + + @Option( + name: .customLong("lib"), + help: ArgumentHelp( + "Directory containing libSyntaxKit.dylib + module files.", + discussion: """ + When omitted, skit searches: $SKIT_LIB_DIR, then /lib/, + then /../lib/skit/. Build a self-contained bundle with + Scripts/build-skit-release.sh. + """ + ) ) - ) - internal var timeoutSeconds: Int = 60 - - @Flag( - name: .customLong("no-toolchain-check"), - help: ArgumentHelp( - "Skip the bundle/local Swift-toolchain comparison.", - discussion: """ - skit compares /swift-version.txt to `swift --version` at - startup and refuses to spawn `swift` on mismatch — swiftmodules - aren't reliably compatible across compiler versions. See issue - #157 for the auto-rebuild plan. - """ + internal var libPath: String? + + @Option( + name: .customLong("helpers"), + help: ArgumentHelp( + "Override Helpers/ directory location.", + discussion: """ + By default skit walks up from the input looking for one. Helper + sources are pre-compiled into libSyntaxKitHelpers.dylib and made + importable via `import SyntaxKitHelpers`. + """ + ) ) - ) - internal var noToolchainCheck: Bool = false + internal var helpersDir: String? - internal func validate() throws { - guard timeoutSeconds >= 0 else { - throw ValidationError( - "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + @Flag( + name: .customLong("no-helpers"), + help: "Skip helpers discovery entirely." + ) + internal var noHelpers: Bool = false + + @Flag( + name: .customLong("no-cache"), + help: ArgumentHelp( + "Skip the rendered-output cache (always run swift).", + discussion: """ + The cache lives at /outputs// and is keyed + on input bytes, helpers, swift version, libSyntaxKit stamp, and + SKIT_*/SYNTAXKIT_* env. + """ ) - } - } + ) + internal var noCache: Bool = false + + @Option( + name: .customLong("timeout"), + help: ArgumentHelp( + "Per-input timeout for the spawned `swift` (seconds).", + discussion: """ + Default 60. On expiry, skit cancels the spawn and exits with code + 124. Pass 0 to disable the watchdog. + """ + ) + ) + internal var timeoutSeconds: Int = 60 + + @Flag( + name: .customLong("no-toolchain-check"), + help: ArgumentHelp( + "Skip the bundle/local Swift-toolchain comparison.", + discussion: """ + skit compares /swift-version.txt to `swift --version` at + startup and refuses to spawn `swift` on mismatch — swiftmodules + aren't reliably compatible across compiler versions. See issue + #157 for the auto-rebuild plan. + """ + ) + ) + internal var noToolchainCheck: Bool = false - internal func run() async throws { - let libPath: String - do { - libPath = try resolveLibPath(override: self.libPath) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - throw ExitCode(2) + internal func validate() throws { + guard timeoutSeconds >= 0 else { + throw ValidationError( + "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + ) + } } - if !noToolchainCheck { - switch toolchainCheck(libPath: libPath) { - case .match, .stampMissing: - break - case .mismatch(let bundle, let local): - FileHandle.standardError.write( - Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) + internal func run() async throws { + let libPath: String + do { + libPath = try resolveLibPath(override: self.libPath) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) throw ExitCode(2) } - } - let helpersOptions: HelpersOptions - if noHelpers { - helpersOptions = .disabled - } else if let dir = helpersDir { - helpersOptions = .explicit(dir) - } else { - helpersOptions = .auto - } + if !noToolchainCheck { + switch await toolchainCheck(libPath: libPath) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + FileHandle.standardError.write( + Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) + throw ExitCode(2) + } + } - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { - throw ValidationError("input does not exist: \(input)") - } + let helpersOptions: HelpersOptions + if noHelpers { + helpersOptions = .disabled + } else if let dir = helpersDir { + helpersOptions = .explicit(dir) + } else { + helpersOptions = .auto + } - if isDirectory.boolValue { - guard let output else { - throw ValidationError("directory inputs require -o ") + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + throw ValidationError("input does not exist: \(input)") + } + + if isDirectory.boolValue { + guard let output else { + throw ValidationError("directory inputs require -o ") + } + let helpers = try await resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + let exitCode = await runDirectory( + inputDir: input, + outputDir: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + throw ExitCode(exitCode) + } else { + let helpers = try await resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + try await runSingleFile( + inputPath: input, + outputPath: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) } - let helpers = try resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - let exitCode = await runDirectory( - inputDir: input, - outputDir: output, - libPath: libPath, - helpers: helpers, - useCache: !noCache, - timeoutSeconds: timeoutSeconds - ) - throw ExitCode(exitCode) - } else { - let helpers = try resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - try runSingleFile( - inputPath: input, - outputPath: output, - libPath: libPath, - helpers: helpers, - useCache: !noCache, - timeoutSeconds: timeoutSeconds - ) } } } -} -// MARK: - skit parse + // MARK: - skit parse -extension Skit { - internal struct Parse: ParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "parse", - abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." - ) + extension Skit { + internal struct Parse: ParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "parse", + abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." + ) - internal func run() throws { - let code = String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" - let treeNodes = SyntaxParser.parse(code: code) - let encoder = JSONEncoder() - let data = try encoder.encode(treeNodes) - let json = String(decoding: data, as: UTF8.self) - print(json) + internal func run() throws { + let code = + String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" + let treeNodes = SyntaxParser.parse(code: code) + let encoder = JSONEncoder() + let data = try encoder.encode(treeNodes) + let json = String(decoding: data, as: UTF8.self) + print(json) + } } } -} + +#endif diff --git a/Sources/skit/SkitStub.swift b/Sources/skit/SkitStub.swift new file mode 100644 index 0000000..5d3a7d1 --- /dev/null +++ b/Sources/skit/SkitStub.swift @@ -0,0 +1,44 @@ +// +// SkitStub.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if !canImport(Subprocess) + + import Foundation + + @main + internal enum SkitStub { + internal static func main() { + FileHandle.standardError.write( + Data("skit: this platform is not supported (no Subprocess backend).\n".utf8) + ) + exit(1) + } + } + +#endif diff --git a/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift b/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift new file mode 100644 index 0000000..ebd083e --- /dev/null +++ b/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift @@ -0,0 +1,101 @@ +// +// SkitSubprocessTimeoutTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + import Subprocess + import Testing + + // Regression test for swift-subprocess #256: + // + // + // The skit timeout watchdog (Sources/skit/Runner.swift) races a Subprocess + // run() call against `Task.sleep(timeout)`. On timeout it calls + // `group.cancelAll()`, which Subprocess turns into a teardown sequence on + // the spawned `swift`. The reported bug: when the spawned child has a + // grandchild that inherited the pipe FDs, the parent's stream read can hang + // waiting for an EOF that won't arrive until the grandchild exits. + // + // skit's real-world trigger is `swift Input.swift`, which fork-exec's the + // Swift frontend + linker as grandchildren. We approximate that here with a + // shell pipeline that forks a background `sleep` holding stderr open. + @Suite("Subprocess timeout-cancel") + internal struct SkitSubprocessTimeoutTests { + @Test( + "cancel-on-timeout completes within a bounded wall-time when grandchildren hold pipe fds" + ) + internal func cancelWithBackgroundedGrandchild() async throws { + let start = ContinuousClock.now + + let outcome: Outcome = try await withThrowingTaskGroup(of: Outcome.self) { group in + group.addTask { + // Outer shell spawns a backgrounded grandchild that holds stderr + // open for 30s, then itself sleeps 30s. Both must be forcibly + // killed by the teardown to free our stream reads. + let record = try await run( + .name("sh"), + arguments: ["-c", "(sleep 30 >/dev/null 2>&1 &) ; sleep 30"], + output: .discarded, + error: .discarded + ) + return .completed(record.terminationStatus) + } + group.addTask { + try await Task.sleep(for: .seconds(1)) + return .timedOut + } + let first = try await group.next()! + group.cancelAll() + return first + } + + let elapsed = ContinuousClock.now - start + + #expect( + outcome == .timedOut, + "timeout task should win the race against a 30s sleep" + ) + // Generous bound: cancellation + Subprocess teardown should complete + // well under 15s. If swift-subprocess #256 triggers, this test fails + // (the run task hangs ≥30s waiting for the grandchild to release the + // stderr pipe). + #expect( + elapsed < .seconds(15), + "timeout-cancel took \(elapsed); possible swift-subprocess #256 regression" + ) + } + + private enum Outcome: Equatable, Sendable { + case completed(TerminationStatus) + case timedOut + } + } + +#endif From 675882212d6b69452f49047e311766e1f52ef006 Mon Sep 17 00:00:00 2001 From: leogdion Date: Wed, 13 May 2026 14:42:04 +0000 Subject: [PATCH 24/56] Fixing CI --- .github/workflows/SyntaxKit.yml | 15 ++++++--------- .github/workflows/swift-source-compat.yml | 1 - 2 files changed, 6 insertions(+), 10 deletions(-) diff --git a/.github/workflows/SyntaxKit.yml b/.github/workflows/SyntaxKit.yml index 5fb86e0..451d097 100644 --- a/.github/workflows/SyntaxKit.yml +++ b/.github/workflows/SyntaxKit.yml @@ -65,7 +65,7 @@ jobs: run: | if [[ "${{ steps.check.outputs.full }}" == "true" ]]; then echo 'ubuntu-os=["noble","jammy"]' >> "$GITHUB_OUTPUT" - echo 'ubuntu-swift=[{"version":"6.0"},{"version":"6.1"},{"version":"6.2"},{"version":"6.3"}]' >> "$GITHUB_OUTPUT" + echo 'ubuntu-swift=[{"version":"6.1"},{"version":"6.2"},{"version":"6.3"}]' >> "$GITHUB_OUTPUT" echo 'ubuntu-type=["","wasm","wasm-embedded"]' >> "$GITHUB_OUTPUT" else echo 'ubuntu-os=["noble"]' >> "$GITHUB_OUTPUT" @@ -79,15 +79,12 @@ jobs: runs-on: ubuntu-latest container: ${{ matrix.swift.nightly && format('swiftlang/swift:nightly-{0}-{1}', matrix.swift.version, matrix.os) || format('swift:{0}-{1}', matrix.swift.version, matrix.os) }} strategy: + fail-fast: false matrix: os: ${{ fromJSON(needs.configure.outputs.ubuntu-os) }} swift: ${{ fromJSON(needs.configure.outputs.ubuntu-swift) }} type: ${{ fromJSON(needs.configure.outputs.ubuntu-type) }} exclude: - - swift: {version: "6.0"} - type: wasm - - swift: {version: "6.0"} - type: wasm-embedded - swift: {version: "6.1"} type: wasm - swift: {version: "6.1"} @@ -237,7 +234,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "iPhone 17 Pro" - osVersion: "26.4" + osVersion: "26.4.1" download-platform: true # watchOS Build Matrix @@ -245,7 +242,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Watch Ultra 3 (49mm)" - osVersion: "26.4" + osVersion: "26.4.1" download-platform: true # tvOS Build Matrix @@ -253,7 +250,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple TV" - osVersion: "26.4" + osVersion: "26.4.1" download-platform: true # visionOS Build Matrix @@ -261,7 +258,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Vision Pro" - osVersion: "26.4" + osVersion: "26.4.1" download-platform: true steps: diff --git a/.github/workflows/swift-source-compat.yml b/.github/workflows/swift-source-compat.yml index a8c05a0..4ab0b69 100644 --- a/.github/workflows/swift-source-compat.yml +++ b/.github/workflows/swift-source-compat.yml @@ -16,7 +16,6 @@ jobs: fail-fast: false matrix: container: - - swift:6.0 - swift:6.1 - swift:6.2 - swift:6.3 From fbcc8fa4566208945b7fdb51eb8499ecf7a05be3 Mon Sep 17 00:00:00 2001 From: leogdion Date: Wed, 13 May 2026 11:20:01 -0400 Subject: [PATCH 25/56] Update OS version to 26.5 in SyntaxKit.yml --- .github/workflows/SyntaxKit.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/SyntaxKit.yml b/.github/workflows/SyntaxKit.yml index 451d097..fb5cd8c 100644 --- a/.github/workflows/SyntaxKit.yml +++ b/.github/workflows/SyntaxKit.yml @@ -242,7 +242,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Watch Ultra 3 (49mm)" - osVersion: "26.4.1" + osVersion: "26.5" download-platform: true # tvOS Build Matrix @@ -250,7 +250,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple TV" - osVersion: "26.4.1" + osVersion: "26.5" download-platform: true # visionOS Build Matrix From 89fad366e2a6edbb1bbe00ede9a4ae6d1cba2220 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 13 May 2026 14:58:01 -0400 Subject: [PATCH 26/56] Fix two PR review regressions in Examples/Completed dsl files [skip ci] - for_loops/dsl.swift: restore `For(_, in:, where:, then:)` with where clause filtering; the earlier rewrite to an inner-body `If` was a semantic change (iterate-all-then-branch vs filter-at-iteration). - errors_async/dsl.swift: restore the original `TupleAssignment(...).async().throwing()` line emitting `let (fetchedData, fetchedPosts) = try await (data, posts)`. - Promote `TupleAssignment` (struct, syntax property, init, `.async()`/`.throwing()`/`.asyncSet()`) to `public` so the example compiles via skit, and switch its `import SwiftSyntax` to `public import SwiftSyntax`. Co-Authored-By: Claude Opus 4.7 (1M context) --- Examples/Completed/errors_async/dsl.swift | 11 ++++------- Examples/Completed/for_loops/dsl.swift | 16 ++++++++++------ .../SyntaxKit/Collections/TupleAssignment.swift | 14 +++++++------- 3 files changed, 21 insertions(+), 20 deletions(-) diff --git a/Examples/Completed/errors_async/dsl.swift b/Examples/Completed/errors_async/dsl.swift index 6dc71f1..aad5087 100644 --- a/Examples/Completed/errors_async/dsl.swift +++ b/Examples/Completed/errors_async/dsl.swift @@ -57,13 +57,10 @@ Do { ParameterExp(name: "id", value: Literal.integer(1)) } }.async() - // The original example used `TupleAssignment([...], equals: Tuple {...})` - // to emit `let (fetchedData, fetchedPosts) = try await (data, posts)`, but - // `TupleAssignment` is internal in the current API. Emit two equivalent - // single-variable bindings instead — same observable behaviour for the - // catch block below. - Variable(.let, name: "fetchedData") { VariableExp("data") } - Variable(.let, name: "fetchedPosts") { VariableExp("posts") } + TupleAssignment(["fetchedData", "fetchedPosts"], equals: Tuple { + VariableExp("data") + VariableExp("posts") + }).async().throwing() } catch: { Catch(EnumCase("fetchError")) { // Example catch for async/await diff --git a/Examples/Completed/for_loops/dsl.swift b/Examples/Completed/for_loops/dsl.swift index 753e78a..74b1d1f 100644 --- a/Examples/Completed/for_loops/dsl.swift +++ b/Examples/Completed/for_loops/dsl.swift @@ -39,12 +39,16 @@ Group { } Variable(.let, name: "numbers", equals: Literal.array([Literal.integer(1), Literal.integer(2), Literal.integer(3), Literal.integer(4), Literal.integer(5), Literal.integer(6), Literal.integer(7), Literal.integer(8), Literal.integer(9), Literal.integer(10)])) - For(VariableExp("number"), in: VariableExp("numbers"), then: { - If(VariableExp("number % 2 == 0"), then: { - Call("print") { - ParameterExp(unlabeled: "\"Even number: \\(number)\"") - } - }) + For(VariableExp("number"), in: VariableExp("numbers"), where: { + Infix("==", + lhs: Infix("%", + lhs: VariableExp("number"), + rhs: Literal.integer(2)), + rhs: Literal.integer(0)) + }, then: { + Call("print") { + ParameterExp(unlabeled: "\"Even number: \\(number)\"") + } }) // MARK: - For-in with Dictionary diff --git a/Sources/SyntaxKit/Collections/TupleAssignment.swift b/Sources/SyntaxKit/Collections/TupleAssignment.swift index fbea0ba..1c30d1c 100644 --- a/Sources/SyntaxKit/Collections/TupleAssignment.swift +++ b/Sources/SyntaxKit/Collections/TupleAssignment.swift @@ -28,10 +28,10 @@ // import Foundation -import SwiftSyntax +public import SwiftSyntax /// A tuple assignment statement for destructuring multiple values. -internal struct TupleAssignment: CodeBlock { +public struct TupleAssignment: CodeBlock { private let elements: [String] private let value: any CodeBlock private var isAsync: Bool = false @@ -39,7 +39,7 @@ internal struct TupleAssignment: CodeBlock { private var isAsyncSet: Bool = false /// The syntax representation of this tuple assignment. - internal var syntax: any SyntaxProtocol { + public var syntax: any SyntaxProtocol { if isAsyncSet { return generateAsyncSetSyntax() } @@ -50,14 +50,14 @@ internal struct TupleAssignment: CodeBlock { /// - Parameters: /// - elements: The names of the variables to destructure into. /// - value: The expression to destructure. - internal init(_ elements: [String], equals value: any CodeBlock) { + public init(_ elements: [String], equals value: any CodeBlock) { self.elements = elements self.value = value } /// Marks this destructuring as async. /// - Returns: A copy of the destructuring marked as async. - internal func async() -> Self { + public func async() -> Self { var copy = self copy.isAsync = true return copy @@ -65,7 +65,7 @@ internal struct TupleAssignment: CodeBlock { /// Marks this destructuring as throwing. /// - Returns: A copy of the destructuring marked as throwing. - internal func throwing() -> Self { + public func throwing() -> Self { var copy = self copy.isThrowing = true return copy @@ -73,7 +73,7 @@ internal struct TupleAssignment: CodeBlock { /// Marks this destructuring as concurrent async (async let set). /// - Returns: A copy of the destructuring marked as async set. - internal func asyncSet() -> Self { + public func asyncSet() -> Self { var copy = self copy.isAsyncSet = true return copy From 824fb3ceb1d7e586b90300ba25002f5eb538e25c Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 13 May 2026 15:25:46 -0400 Subject: [PATCH 27/56] Narrow Subprocess guard in skit and split subcommands into their own files [skip ci] Drop the file-level `#if canImport(Subprocess)` around the entire Skit type and the separate SkitStub `@main`. The single `@main Skit` now covers all platforms; only the body of `Run.run()` is guarded, so `skit parse` works without a Subprocess backend. Each subcommand lives in its own extension file (Skit+Run.swift, Skit+Parse.swift), and Run's options shed their verbose `discussion:` text in favor of ArgumentParser's default `--help` rendering. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../skit/{SkitStub.swift => Skit+Parse.swift} | 31 ++- Sources/skit/Skit+Run.swift | 169 +++++++++++++ Sources/skit/Skit.swift | 230 +----------------- 3 files changed, 199 insertions(+), 231 deletions(-) rename Sources/skit/{SkitStub.swift => Skit+Parse.swift} (64%) create mode 100644 Sources/skit/Skit+Run.swift diff --git a/Sources/skit/SkitStub.swift b/Sources/skit/Skit+Parse.swift similarity index 64% rename from Sources/skit/SkitStub.swift rename to Sources/skit/Skit+Parse.swift index 5d3a7d1..47ad22d 100644 --- a/Sources/skit/SkitStub.swift +++ b/Sources/skit/Skit+Parse.swift @@ -1,5 +1,5 @@ // -// SkitStub.swift +// Skit+Parse.swift // SyntaxKit // // Created by Leo Dion. @@ -27,18 +27,25 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if !canImport(Subprocess) +import ArgumentParser +import Foundation +import SyntaxParser - import Foundation +extension Skit { + internal struct Parse: ParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "parse", + abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." + ) - @main - internal enum SkitStub { - internal static func main() { - FileHandle.standardError.write( - Data("skit: this platform is not supported (no Subprocess backend).\n".utf8) - ) - exit(1) + internal func run() throws { + let code = + String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" + let treeNodes = SyntaxParser.parse(code: code) + let encoder = JSONEncoder() + let data = try encoder.encode(treeNodes) + let json = String(decoding: data, as: UTF8.self) + print(json) } } - -#endif +} diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift new file mode 100644 index 0000000..dff3a80 --- /dev/null +++ b/Sources/skit/Skit+Run.swift @@ -0,0 +1,169 @@ +// +// Skit+Run.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import ArgumentParser +import Foundation + +extension Skit { + internal struct Run: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "run", + abstract: "Render SyntaxKit DSL input(s) into Swift source." + ) + + @Argument(help: "Path to a .swift input file or a directory of inputs.") + internal var input: String + + @Option( + name: [.short, .customLong("output")], + help: "Output file (single-file mode) or directory (folder mode)." + ) + internal var output: String? + + @Option( + name: .customLong("lib"), + help: "Directory containing libSyntaxKit.dylib + module files." + ) + internal var libPath: String? + + @Option( + name: .customLong("helpers"), + help: "Override Helpers/ directory location." + ) + internal var helpersDir: String? + + @Flag( + name: .customLong("no-helpers"), + help: "Skip helpers discovery entirely." + ) + internal var noHelpers: Bool = false + + @Flag( + name: .customLong("no-cache"), + help: "Skip the rendered-output cache (always run swift)." + ) + internal var noCache: Bool = false + + @Option( + name: .customLong("timeout"), + help: "Per-input timeout for the spawned `swift` in seconds (0 disables)." + ) + internal var timeoutSeconds: Int = 60 + + @Flag( + name: .customLong("no-toolchain-check"), + help: "Skip the bundle/local Swift-toolchain comparison." + ) + internal var noToolchainCheck: Bool = false + + internal func validate() throws { + guard timeoutSeconds >= 0 else { + throw ValidationError( + "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + ) + } + } + + internal func run() async throws { + #if canImport(Subprocess) + let libPath: String + do { + libPath = try resolveLibPath(override: self.libPath) + } catch { + FileHandle.standardError.write(Data("\(error)\n".utf8)) + throw ExitCode(2) + } + + if !noToolchainCheck { + switch await toolchainCheck(libPath: libPath) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + FileHandle.standardError.write( + Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) + throw ExitCode(2) + } + } + + let helpersOptions: HelpersOptions + if noHelpers { + helpersOptions = .disabled + } else if let dir = helpersDir { + helpersOptions = .explicit(dir) + } else { + helpersOptions = .auto + } + + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + throw ValidationError("input does not exist: \(input)") + } + + if isDirectory.boolValue { + guard let output else { + throw ValidationError("directory inputs require -o ") + } + let helpers = try await resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + let exitCode = await runDirectory( + inputDir: input, + outputDir: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + throw ExitCode(exitCode) + } else { + let helpers = try await resolveHelpers( + nearInputPath: input, + libPath: libPath, + options: helpersOptions + ) + try await runSingleFile( + inputPath: input, + outputPath: output, + libPath: libPath, + helpers: helpers, + useCache: !noCache, + timeoutSeconds: timeoutSeconds + ) + } + #else + FileHandle.standardError.write( + Data("skit: run is not supported on this platform (no Subprocess backend).\n".utf8) + ) + throw ExitCode(1) + #endif + } + } +} diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index bd594e0..92d8c1a 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -27,222 +27,14 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) - - import ArgumentParser - import Foundation - import SyntaxParser - - @main - internal struct Skit: AsyncParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "skit", - abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", - subcommands: [Run.self, Parse.self], - defaultSubcommand: Run.self - ) - } - - // MARK: - skit run - - extension Skit { - internal struct Run: AsyncParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "run", - abstract: "Render SyntaxKit DSL input(s) into Swift source.", - discussion: """ - Wraps each input in a `Group { … }` closure and spawns `swift` to - evaluate it. The rendered output is written to stdout (single-file - mode) or mirrored into an output directory (folder mode). - - Forms: - skit run Input.swift — render to stdout - skit run Input.swift -o Out.swift — render to a file - skit run InputDir/ -o OutDir/ — walk **/*.swift (skipping - files prefixed with '_') - and mirror rendered output - into OutDir/ - """ - ) - - @Argument(help: "Path to a .swift input file or a directory of inputs.") - internal var input: String - - @Option( - name: [.short, .customLong("output")], - help: "Output file (single-file mode) or directory (folder mode)." - ) - internal var output: String? - - @Option( - name: .customLong("lib"), - help: ArgumentHelp( - "Directory containing libSyntaxKit.dylib + module files.", - discussion: """ - When omitted, skit searches: $SKIT_LIB_DIR, then /lib/, - then /../lib/skit/. Build a self-contained bundle with - Scripts/build-skit-release.sh. - """ - ) - ) - internal var libPath: String? - - @Option( - name: .customLong("helpers"), - help: ArgumentHelp( - "Override Helpers/ directory location.", - discussion: """ - By default skit walks up from the input looking for one. Helper - sources are pre-compiled into libSyntaxKitHelpers.dylib and made - importable via `import SyntaxKitHelpers`. - """ - ) - ) - internal var helpersDir: String? - - @Flag( - name: .customLong("no-helpers"), - help: "Skip helpers discovery entirely." - ) - internal var noHelpers: Bool = false - - @Flag( - name: .customLong("no-cache"), - help: ArgumentHelp( - "Skip the rendered-output cache (always run swift).", - discussion: """ - The cache lives at /outputs// and is keyed - on input bytes, helpers, swift version, libSyntaxKit stamp, and - SKIT_*/SYNTAXKIT_* env. - """ - ) - ) - internal var noCache: Bool = false - - @Option( - name: .customLong("timeout"), - help: ArgumentHelp( - "Per-input timeout for the spawned `swift` (seconds).", - discussion: """ - Default 60. On expiry, skit cancels the spawn and exits with code - 124. Pass 0 to disable the watchdog. - """ - ) - ) - internal var timeoutSeconds: Int = 60 - - @Flag( - name: .customLong("no-toolchain-check"), - help: ArgumentHelp( - "Skip the bundle/local Swift-toolchain comparison.", - discussion: """ - skit compares /swift-version.txt to `swift --version` at - startup and refuses to spawn `swift` on mismatch — swiftmodules - aren't reliably compatible across compiler versions. See issue - #157 for the auto-rebuild plan. - """ - ) - ) - internal var noToolchainCheck: Bool = false - - internal func validate() throws { - guard timeoutSeconds >= 0 else { - throw ValidationError( - "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" - ) - } - } - - internal func run() async throws { - let libPath: String - do { - libPath = try resolveLibPath(override: self.libPath) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - throw ExitCode(2) - } - - if !noToolchainCheck { - switch await toolchainCheck(libPath: libPath) { - case .match, .stampMissing: - break - case .mismatch(let bundle, let local): - FileHandle.standardError.write( - Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) - throw ExitCode(2) - } - } - - let helpersOptions: HelpersOptions - if noHelpers { - helpersOptions = .disabled - } else if let dir = helpersDir { - helpersOptions = .explicit(dir) - } else { - helpersOptions = .auto - } - - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { - throw ValidationError("input does not exist: \(input)") - } - - if isDirectory.boolValue { - guard let output else { - throw ValidationError("directory inputs require -o ") - } - let helpers = try await resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - let exitCode = await runDirectory( - inputDir: input, - outputDir: output, - libPath: libPath, - helpers: helpers, - useCache: !noCache, - timeoutSeconds: timeoutSeconds - ) - throw ExitCode(exitCode) - } else { - let helpers = try await resolveHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - try await runSingleFile( - inputPath: input, - outputPath: output, - libPath: libPath, - helpers: helpers, - useCache: !noCache, - timeoutSeconds: timeoutSeconds - ) - } - } - } - } - - // MARK: - skit parse - - extension Skit { - internal struct Parse: ParsableCommand { - internal static let configuration = CommandConfiguration( - commandName: "parse", - abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." - ) - - internal func run() throws { - let code = - String(data: FileHandle.standardInput.readDataToEndOfFile(), encoding: .utf8) ?? "" - let treeNodes = SyntaxParser.parse(code: code) - let encoder = JSONEncoder() - let data = try encoder.encode(treeNodes) - let json = String(decoding: data, as: UTF8.self) - print(json) - } - } - } - -#endif +import ArgumentParser + +@main +internal struct Skit: AsyncParsableCommand { + internal static let configuration = CommandConfiguration( + commandName: "skit", + abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", + subcommands: [Run.self, Parse.self], + defaultSubcommand: Run.self + ) +} From 74f6c6ac986c32d212a75dee3ca692c138a96f1c Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 13 May 2026 19:44:10 -0400 Subject: [PATCH 28/56] Add lifecycle map and inline phase comments across Sources/skit/ [skip ci] Documents the `skit run` pipeline so a reader can trace it end-to-end from the source files alone: top-of-Runner.swift lifecycle map, numbered phases in Skit.Run.run(), docstrings on every previously undocumented orchestrator, and inline step labels inside processFile, runDirectory, runSwift, wrap, buildHelpers, compileHelpers, outputCacheKey, and storeCachedOutput. No behaviour changes. Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skit/Helpers.swift | 35 +++++++++ Sources/skit/OutputCache.swift | 19 ++++- Sources/skit/Runner.swift | 126 +++++++++++++++++++++++++++++++-- Sources/skit/Skit+Run.swift | 29 ++++++++ Sources/skit/Skit.swift | 6 ++ 5 files changed, 209 insertions(+), 6 deletions(-) diff --git a/Sources/skit/Helpers.swift b/Sources/skit/Helpers.swift index f9511ae..12430a4 100644 --- a/Sources/skit/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -109,13 +109,21 @@ /// Compiles helper sources into a per-key cache directory and returns the /// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. + /// + /// Concurrent invocations are tolerated via the staging-dir + atomic-rename + /// pattern: if two processes race to compile the same key, the loser's + /// rename fails and we keep the winner's artefact. internal func buildHelpers( helpersDir: URL, libPath: String ) async throws -> CompiledHelpers? { + // Collect helper sources. An empty Helpers/ dir is "no helpers" rather + // than an error — the caller will fall back to no-helpers mode. let sources = try collectHelperSources(in: helpersDir) if sources.isEmpty { return nil } + // Compute the content-keyed cache path. The dylib's presence under that + // path is what makes a build "cached". let key = try await helpersCacheKey(sources: sources, libPath: libPath) let cacheRoot = try syntaxKitCacheRoot() .appendingPathComponent("helpers") @@ -124,21 +132,28 @@ cacheRoot .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path + // Cache hit: artefact already present, skip the whole compile. let fm = FileManager.default if fm.fileExists(atPath: dylibPath) { return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) } + // Ensure the parent of the cache key dir exists. We don't create the + // key dir itself — the atomic move below installs it. try fm.createDirectory( at: cacheRoot.deletingLastPathComponent(), withIntermediateDirectories: true ) + // Compile into a per-pid + uuid staging dir, then atomically rename into + // place. This is what lets concurrent skit invocations co-exist safely. let staging = cacheRoot.deletingLastPathComponent() .appendingPathComponent( "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") try fm.createDirectory(at: staging, withIntermediateDirectories: true) + // Run swiftc into the staging dir. Clean up on failure so we don't leak + // half-baked artefacts in the cache root. do { try await compileHelpers(sources: sources, into: staging, libPath: libPath) } catch { @@ -160,11 +175,18 @@ return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) } + /// Invokes `swiftc` to build `sources` into a Swift module + dylib under + /// `outDir`. The dylib is named `lib.{dylib,so}` and the + /// module file is `.swiftmodule`. Output (stdout) + /// is discarded; stderr is captured for the error path. private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) async throws { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path + // Base swiftc arguments: emit a library + module file linking against + // libSyntaxKit, with rpath set so the dylib can find libSyntaxKit at + // load time. var args: [String] = [ "-module-name", helpersModuleName, "-emit-module", @@ -188,8 +210,11 @@ "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", ]) #endif + // Append source files last so the leading flags apply to all of them. args.append(contentsOf: sources.map(\.path)) + // Spawn swiftc. Stderr is captured (1 MiB cap) so a compile failure can + // surface the diagnostic verbatim in a CLIError. let result = try await run( .name("swiftc"), arguments: Arguments(args), @@ -209,16 +234,21 @@ // MARK: - Cache key + /// Content-addressed cache key mixing schema version, each helper source's + /// filename + bytes, `swift --version`, and the libSyntaxKit stamp. private func helpersCacheKey(sources: [URL], libPath: String) async throws -> String { var hasher = ContentHasher() hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) + // Filename matters as well as bytes — two same-content files with + // different names produce different symbols. for source in sources { let data = try Data(contentsOf: source) hasher.update(data: Data(source.lastPathComponent.utf8)) hasher.update(data: data) } + // Toolchain + dylib stamp invalidate on cross-version or in-place rebuild. if let swiftVersion = await captureSwiftVersion() { hasher.update(data: Data(swiftVersion.utf8)) } @@ -229,6 +259,7 @@ return hasher.finalize() } + /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. internal func captureSwiftVersion() async -> String? { let result = try? await run( .name("swift"), @@ -239,6 +270,8 @@ return result?.standardOutput } + /// `/` fingerprint of the bundled libSyntaxKit dylib, or nil + /// if unreadable. Catches in-place rebuilds without a version bump. internal func libStamp(libPath: String) -> String? { let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } @@ -247,6 +280,8 @@ return "\(size)/\(Int(mtime))" } + /// Root for all skit caches. Honours `XDG_CACHE_HOME`, else macOS + /// `~/Library/Caches/...` or Linux `~/.cache/syntaxkit`. internal func syntaxKitCacheRoot() throws -> URL { if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index 47f7972..e68dd84 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -44,23 +44,33 @@ libPath: String ) async -> String { var hasher = ContentHasher() + // Schema version: bump to invalidate every existing cache entry at once. hasher.update(data: Data(outputCacheSchemaVersion.utf8)) + // Input source bytes: the primary driver of the key. hasher.update(data: Data(inputSource.utf8)) + // Helpers fingerprint. The helpers cache dir name *is* the helpers cache + // key (per Helpers.swift), so re-mixing it here cheaply propagates any + // helpers change into this key. if let helpers { - // Helpers cache dir name *is* the helpers cache key (per Helpers.swift). hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) } else { hasher.update(data: Data("no-helpers".utf8)) } + // Toolchain version. Different `swift` builds emit different bytes for + // the same DSL input. if let version = await captureSwiftVersion() { hasher.update(data: Data(version.utf8)) } + // libSyntaxKit stamp. A rebuilt dylib can change the rendered output + // even without a Swift-version bump. if let stamp = libStamp(libPath: libPath) { hasher.update(data: Data(stamp.utf8)) } + // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and + // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. let env = ProcessInfo.processInfo.environment .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } .sorted { $0.key < $1.key } @@ -84,11 +94,15 @@ let final = cacheRoot.appendingPathComponent("output.swift") let fm = FileManager.default + // Ensure the parent of the cache key dir exists. The key dir itself is + // installed by the atomic rename below. try fm.createDirectory( at: cacheRoot.deletingLastPathComponent(), withIntermediateDirectories: true ) + // Stage the payload in a per-pid + uuid sibling dir so it can be renamed + // into place as a single atomic step. let staging = cacheRoot.deletingLastPathComponent() .appendingPathComponent( "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" @@ -96,6 +110,9 @@ try fm.createDirectory(at: staging, withIntermediateDirectories: true) try data.write(to: staging.appendingPathComponent("output.swift")) + // Atomic rename into the cache path. If a peer already populated this + // key, swallow the rename error and drop our staging copy. Re-throw only + // if the destination is still missing afterwards. do { try fm.moveItem(at: staging, to: cacheRoot) } catch { diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index d9407f4..4a8513d 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -34,19 +34,42 @@ import SwiftParser import SwiftSyntax + // Run lifecycle (per `skit run` invocation): + // 1. resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) + // 2. toolchainCheck — compare bundle stamp to `swift --version` + // 3. resolveHelpers — discover + compile Helpers/ (memoised on disk) + // 4. runSingleFile / runDirectory — dispatch to single- or batch-input mode + // 5. processFile (per input) — load → cache lookup → wrap → spawn → cache store + // 6. wrap — hoist imports, wrap body in Group { … }, #sourceLocation + // 7. runSwift — spawn `swift` with timeout watchdog + // See Docs/skit.md for design rationale and trade-offs. + // MARK: - Helpers resolution + /// How `skit run` should treat a `Helpers/` directory for this invocation. internal enum HelpersOptions { + /// Walk up from the input looking for `Helpers/`. The default. case auto + /// Skip helpers discovery entirely (`--no-helpers`). The wrapped input + /// won't be able to `import SyntaxKitHelpers`. case disabled + /// Use the directory at the given path (`--helpers `). Validated as + /// an existing directory in `resolveHelpers`. case explicit(String) } + /// Resolves a `Helpers/` directory and compiles it (or returns the cached + /// build). Returns nil when helpers are disabled, when no `Helpers/` was + /// found in auto mode, or when the directory exists but contains no `.swift` + /// sources. On success, writes a one-line "skit: helpers cached/compiled at + /// " note to stderr so users can see whether the cache hit. internal func resolveHelpers( nearInputPath path: String, libPath: String, options: HelpersOptions ) async throws -> CompiledHelpers? { + // Pick the helpers dir according to the mode: walk up the tree, accept an + // explicit override (after validating it's a directory), or bail out. let helpersDir: URL? switch options { case .disabled: @@ -65,6 +88,8 @@ } guard let helpersDir else { return nil } + // Compile (or reuse the cached build). An empty Helpers/ dir is treated + // as "no helpers" rather than an error. guard let compiled = try await buildHelpers(helpersDir: helpersDir, libPath: libPath) else { return nil } @@ -185,6 +210,11 @@ // MARK: - Single-file mode + /// Runs `processFile` on a single input and writes its rendered Swift to + /// `outputPath` (or stdout when nil). Any stderr from the spawned `swift` + /// is surfaced verbatim. On a non-zero subprocess exit, calls `exit()` + /// directly — the caller in `Skit.Run.run()` won't see a thrown error in + /// that path. internal func runSingleFile( inputPath: String, outputPath: String?, @@ -193,6 +223,8 @@ useCache: Bool, timeoutSeconds: Int ) async throws { + // Render the input. `processFile` may hit the output cache and skip the + // spawn entirely; either way the result has the same shape. let result = try await processFile( inputPath: inputPath, libPath: libPath, @@ -200,12 +232,16 @@ useCache: useCache, timeoutSeconds: timeoutSeconds ) + // Surface diagnostics from the spawned `swift` before deciding success. if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) } + // Non-zero subprocess exit propagates as a process exit. We don't write + // partial output in that case. guard result.exitCode == 0 else { exit(result.exitCode) } + // Deliver the rendered output to file or stdout. if let outputPath { try result.stdout.write(to: URL(fileURLWithPath: outputPath)) } else { @@ -215,6 +251,10 @@ // MARK: - Folder mode + /// Walks `inputDir` for `.swift` inputs, processes them concurrently (up to + /// the active core count), and mirrors the rendered output into `outputDir`. + /// A failure on one input does not abort the batch — successful peers are + /// still written. Returns 0 if every input succeeded, 1 otherwise. internal func runDirectory( inputDir: String, outputDir: String, @@ -226,6 +266,8 @@ let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL + // Phase 1: enumerate inputs. Top-level `Helpers/` is excluded so its + // sources aren't processed as DSL inputs. let inputs: [URL] do { inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) @@ -239,12 +281,15 @@ return 0 } + // Phase 2: bounded-concurrency processing. Cap is the active core count + // so a 200-file batch doesn't fork 200 simultaneous `swift` processes. let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) var outcomes: [FileOutcome] = [] var iterator = inputs.makeIterator() await withTaskGroup(of: FileOutcome.self) { group in + // Seed the group up to the concurrency cap… for _ in 0.. } + /// `processFile` adapter that catches errors into the `FileOutcome` result + /// so a single failure doesn't tear down the surrounding `TaskGroup`. private func runOne( _ input: URL, libPath: String, @@ -348,6 +402,9 @@ return candidate.path } + /// Returns every `.swift` file under `inputDir` (recursive), sorted, with + /// hidden files, files prefixed by `_`, and the `excludedDir` subtree + /// removed. Sorted output keeps batch behaviour deterministic across runs. private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { guard let enumerator = FileManager.default.enumerator( @@ -362,12 +419,16 @@ var result: [URL] = [] for case let url as URL in enumerator { let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + // Directories aren't outputs; if this is the excluded `Helpers/` dir, + // prune the whole subtree. if values.isDirectory == true { if let excludedDir, url.standardizedFileURL.path == excludedDir { enumerator.skipDescendants() } continue } + // Filter for `.swift` regular files, skipping the `_`-prefixed + // convention for "not an input" sources. guard values.isRegularFile == true else { continue } guard url.pathExtension == "swift" else { continue } guard !url.lastPathComponent.hasPrefix("_") else { continue } @@ -378,12 +439,19 @@ // MARK: - Per-file work + /// Raw outcome of rendering one input — what `processFile` returns to its + /// caller. `exitCode == 0` indicates the spawned `swift` succeeded (or that + /// the output cache hit, which is treated identically). private struct ProcessResult: Sendable { let exitCode: Int32 let stdout: Data let stderr: String } + /// The per-input render pipeline: load source → consult the output cache → + /// (on miss) wrap → spawn `swift` → rewrite diagnostics → store the result + /// in the cache. The temp wrapper file is created in a per-run tmp dir and + /// torn down by `defer` whether the spawn succeeded or not. private func processFile( inputPath: String, libPath: String, @@ -391,20 +459,30 @@ useCache: Bool, timeoutSeconds: Int ) async throws -> ProcessResult { + // Load the input source. Anything past this point keys off these bytes. let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL let absoluteInputPath = inputURL.path let source = try String(contentsOf: inputURL, encoding: .utf8) + // Compute the output cache key (skipped under `--no-cache`). Mixes input + // bytes, toolchain version, helpers fingerprint, libSyntaxKit stamp, and + // sorted SKIT_*/SYNTAXKIT_* env vars — see `outputCacheKey`. let cacheKey: String? = useCache ? await outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) : nil + // Cache hit: skip the wrap+spawn entirely and return the stored output. if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { return ProcessResult(exitCode: 0, stdout: cached, stderr: "") } + // Wrap the user's input into a complete Swift program that imports + // SyntaxKit, runs the body inside a Group { … } builder, and prints the + // result. See `wrap` for the exact template. let wrapped = wrap(source: source, originalPath: absoluteInputPath) + // Spill the wrapped program to a per-invocation temp dir. The dir is + // cleaned up unconditionally so a failed spawn doesn't leak files. let tmpDir = FileManager.default.temporaryDirectory .appendingPathComponent("skit-\(UUID().uuidString)") try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) @@ -413,6 +491,8 @@ let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + // Spawn `swift` on the wrapped file (with timeout watchdog). stdout is + // the rendered Swift source; stderr is compiler diagnostics, if any. let raw = try await runSwift( wrappedPath: wrappedURL.path, libPath: libPath, @@ -427,6 +507,8 @@ with: absoluteInputPath ) + // Store on the way out. `try?` is deliberate: a cache write failure is + // not a render failure. The next run will simply miss and re-spawn. if let cacheKey, raw.exitCode == 0 { try? storeCachedOutput(key: cacheKey, data: raw.stdout) } @@ -434,6 +516,8 @@ return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } + /// Throwable error wrapper for skit's user-facing diagnostics. The message + /// is printed verbatim — keep it actionable (path, hint, next step). internal struct CLIError: Error, CustomStringConvertible { let message: String var description: String { message } @@ -447,12 +531,16 @@ /// The body is fenced in `#sourceLocation` directives so compiler diagnostics /// in the body reference the original input file and line numbers. internal func wrap(source: String, originalPath: String) -> String { + // Parse the input with SwiftSyntax. The location converter is needed to + // map the body's starting byte offset back to a 1-based line number for + // the `#sourceLocation` directive. let tree = Parser.parse(source: source) let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) - // Find the first non-import top-level statement; everything before it that - // is an import gets hoisted, anything before that which is *not* an import - // stays in the body (e.g. a top-level `// comment` is left alone). + // Scan top-level statements for hoistable imports. Everything before the + // first non-import statement that *is* an import gets hoisted; anything + // before that which is *not* an import stays in the body (e.g. a top-level + // `// comment` is left alone). var hoisted: [String] = [] var firstBodyByte: AbsolutePosition? @@ -467,6 +555,8 @@ break } + // Compute the body slice (source from the first non-import byte onward) + // and the 1-based line number it lives on in the original file. let body: String let firstBodyLine: Int if let firstBodyByte { @@ -478,6 +568,9 @@ firstBodyLine = 1 } + // Render the hoisted-imports block. Trailing newline only if non-empty so + // the wrapper doesn't grow an extra blank line in the common no-imports + // case. let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" // #sourceLocation must use a forward-slash path; escape backslashes/quotes @@ -487,6 +580,8 @@ .replacingOccurrences(of: "\\", with: "\\\\") .replacingOccurrences(of: "\"", with: "\\\"") + // Build the final wrapper. Layout: SyntaxKit import → hoisted imports → + // Group { #sourceLocation(...) #sourceLocation() } → print. return """ import SyntaxKit \(hoistedBlock) @@ -512,11 +607,19 @@ private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 private let stderrLimitBytes: Int = 1 * 1_024 * 1_024 + /// Either the spawned `swift` ran to completion (success or failure) or + /// the watchdog elapsed first. The completed payload is normalized to the + /// shape callers want regardless of platform. private enum SwiftRunOutcome: Sendable { case completed(exitCode: Int32, stdout: Data, stderr: String) case timedOut } + /// Spawns `swift` (script-mode interpreter) on the wrapped input file, + /// optionally splicing in flags to import a precompiled helpers module. + /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing + /// task group; the loser is cancelled. On timeout, returns exit 124 with + /// a one-line stderr message — matching POSIX `timeout(1)`'s convention. private func runSwift( wrappedPath: String, libPath: String, @@ -525,6 +628,8 @@ ) async throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + // Build the base argument list: link against libSyntaxKit, include the + // CShims headers, set rpath so the dylib loads at runtime. var arguments: [String] = [ "-suppress-warnings", "-I", libPath, @@ -534,6 +639,9 @@ "-Xlinker", "-rpath", "-Xlinker", libPath, ] + // Splice in helpers-module flags only when a compiled helpers dylib is + // available. Skipping these makes `import SyntaxKitHelpers` fail in the + // wrapped input, which is fine when no Helpers/ dir was discovered. if let helpers { let helpersPath = helpers.outputDir.path arguments.append(contentsOf: [ @@ -547,6 +655,8 @@ arguments.append(wrappedPath) let argumentsCopy = arguments + // The actual subprocess call, wrapped in a closure so the task-group race + // below can hold a single Sendable reference to it. let invocation: @Sendable () async throws -> SwiftRunOutcome = { let record = try await run( .name("swift"), @@ -561,6 +671,9 @@ ) } + // Race the invocation against a sleep watchdog; whichever finishes first + // wins, the other is cancelled. `timeoutSeconds <= 0` opts out of the + // race entirely (useful for debugging genuinely long codegen). let outcome: SwiftRunOutcome if timeoutSeconds <= 0 { outcome = try await invocation() @@ -577,6 +690,7 @@ } } + // Normalize both outcomes into a single ProcessResult shape. switch outcome { case .completed(let exitCode, let stdout, let stderr): return ProcessResult(exitCode: exitCode, stdout: stdout, stderr: stderr) @@ -589,6 +703,8 @@ } } + /// Collapses Subprocess's `TerminationStatus` into a single Int32 exit code, + /// using the shell convention (128 + signal number) for signalled deaths. private func exitCode(from status: TerminationStatus) -> Int32 { switch status { case .exited(let code): diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index dff3a80..baf8d7b 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -31,6 +31,12 @@ import ArgumentParser import Foundation extension Skit { + /// Render one or more SyntaxKit DSL files into Swift source. + /// + /// `Run` is the default subcommand. It accepts either a single `.swift` + /// file or a directory of `.swift` files; in directory mode the rendered + /// output is written into a mirrored tree under `-o`. The actual work is + /// delegated to free functions in `Runner.swift`. internal struct Run: AsyncParsableCommand { internal static let configuration = CommandConfiguration( commandName: "run", @@ -92,6 +98,9 @@ extension Skit { internal func run() async throws { #if canImport(Subprocess) + // 1. Resolve the libSyntaxKit bundle dir. Failure here is fatal — we + // can't spawn `swift` without knowing where the dylib + swiftmodules + // live. The error message lists the four lookup paths in priority order. let libPath: String do { libPath = try resolveLibPath(override: self.libPath) @@ -100,6 +109,10 @@ extension Skit { throw ExitCode(2) } + // 2. Compare the bundle's recorded `swift --version` against the local + // one. swiftmodules aren't reliably forward-compatible across compiler + // versions, so a mismatch produces a clear error rather than letting + // the spawned `swift` emit a cryptic module-version diagnostic. if !noToolchainCheck { switch await toolchainCheck(libPath: libPath) { case .match, .stampMissing: @@ -111,6 +124,8 @@ extension Skit { } } + // 3. Decide which helpers-resolution mode this invocation is in. + // The actual discovery / compilation happens later in `resolveHelpers`. let helpersOptions: HelpersOptions if noHelpers { helpersOptions = .disabled @@ -120,6 +135,9 @@ extension Skit { helpersOptions = .auto } + // 4. Stat the input to pick single-file vs. directory mode. Directory + // mode requires an explicit `-o` output dir; single-file mode falls + // back to stdout. var isDirectory: ObjCBool = false guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { throw ValidationError("input does not exist: \(input)") @@ -129,11 +147,16 @@ extension Skit { guard let output else { throw ValidationError("directory inputs require -o ") } + // 5a. Resolve helpers relative to the input root. This is the only + // place we compile `Helpers/`; the result is reused across every + // input file in the directory. let helpers = try await resolveHelpers( nearInputPath: input, libPath: libPath, options: helpersOptions ) + // 6a. Hand off to the directory orchestrator and surface its exit + // code via ExitCode (so a partial-failure batch returns 1). let exitCode = await runDirectory( inputDir: input, outputDir: output, @@ -144,11 +167,15 @@ extension Skit { ) throw ExitCode(exitCode) } else { + // 5b. Resolve helpers relative to this single file's parent. let helpers = try await resolveHelpers( nearInputPath: input, libPath: libPath, options: helpersOptions ) + // 6b. Hand off to the single-file orchestrator. It calls `exit()` + // directly on non-zero subprocess exit, so a thrown ExitCode here + // would be unreachable in that path. try await runSingleFile( inputPath: input, outputPath: output, @@ -159,6 +186,8 @@ extension Skit { ) } #else + // Subprocess is the only backend skit knows how to use to spawn + // `swift`/`swiftc`. Without it (Windows, embedded), `run` cannot work. FileHandle.standardError.write( Data("skit: run is not supported on this platform (no Subprocess backend).\n".utf8) ) diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index 92d8c1a..d5cb936 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -29,6 +29,12 @@ import ArgumentParser +/// The `skit` CLI entry point. +/// +/// `Skit` itself is just an ArgumentParser shell that wires up two subcommands: +/// `Run` (the default, rendering SyntaxKit DSL into Swift source) and `Parse` +/// (the inverse, reading Swift source on stdin and emitting JSON). Their bodies +/// live in `Skit+Run.swift` and `Skit+Parse.swift` respectively. @main internal struct Skit: AsyncParsableCommand { internal static let configuration = CommandConfiguration( From 0e25069d91f02810c28aab3396ee61ed8cd26edd Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Sun, 7 Jun 2026 19:45:55 -0400 Subject: [PATCH 29/56] Document no-globals convention and add globals audit [skip ci] Add a Coding Conventions note to CLAUDE.md explaining the no-global- functions/variables rule, the #if-block caveat, and why it can't be a SwiftLint regex rule (needs an AST-aware check). Add globals-audit.md inventorying current file-scope funcs/vars for review. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 10 ++++++ globals-audit.md | 94 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 globals-audit.md diff --git a/CLAUDE.md b/CLAUDE.md index 008b2dd..57cf92e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,6 +75,16 @@ Sources/SyntaxKit/ 4. Run `./Scripts/lint.sh` to ensure code quality 5. Run `swift test` to verify functionality +### Coding Conventions +- **No global functions or variables.** Every function and `let`/`var` must be + nested inside a type (`struct`/`enum`/`class`) or an `extension`. This includes + declarations that look indented only because they sit inside a `#if … #endif` + conditional-compilation block (e.g. `Sources/skit/Runner.swift`) — those are + still file-scope globals and must be moved into a type. This is not currently + enforced by SwiftLint: a `custom_rules` regex can't reliably distinguish a + global nested in `#if` from a legitimate type member, so it needs an + AST-aware (SwiftSyntax) check to enforce. Until then, follow the rule by hand. + ### Package Dependencies - **SwiftSyntax** (601.0.1+) - Apple's Swift syntax parser - **SwiftOperators** - Operator handling diff --git a/globals-audit.md b/globals-audit.md new file mode 100644 index 0000000..c25f870 --- /dev/null +++ b/globals-audit.md @@ -0,0 +1,94 @@ +# Globals Audit + +File-scope **functions** and **variables** (`func` / `let` / `var` declared directly in a file, not inside a type or extension). Types (`struct`/`enum`/`class`/`actor`/`protocol`) are intentionally **excluded** — the no-globals convention targets funcs and vars only. + +Note: in `Sources/skit/` every listed declaration is indented because it lives inside a top-level `#if canImport(Subprocess)` block — it is still a true file-scope global. This is exactly the case a SwiftLint regex rule cannot detect. + +Clean targets (no global funcs/vars): **SyntaxKit**, **SyntaxParser**, **TokenVisitor**. + +Generated: 2026-06-07. Branch: `research/swift-manifest-codegen`. + +--- + +## Sources/skit/Runner.swift + +All inside `#if canImport(Subprocess)`. + +### Functions +- [ ] L66 — `internal func resolveHelpers(nearInputPath path: String, libPath: String, options: HelpersOptions) async throws -> CompiledHelpers?` +- [ ] L109 — `internal func resolveLibPath(override: String?) throws -> String` +- [ ] L147 — `private func isLibDir(_ path: String) -> Bool` +- [ ] L173 — `internal func toolchainCheck(libPath: String) async -> ToolchainCheckResult` +- [ ] L194 — `internal func toolchainMismatchMessage(bundle: String, local: String) -> String` +- [ ] L218 — `internal func runSingleFile(inputPath:outputPath:libPath:helpers:useCache:timeoutSeconds:) async throws` +- [ ] L258 — `internal func runDirectory(inputDir:outputDir:libPath:helpers:useCache:timeoutSeconds:) async -> Int32` +- [ ] L370 — `private func runOne(_ input: URL, libPath:helpers:useCache:timeoutSeconds:) async -> FileOutcome` +- [ ] L394 — `private func helpersExcludePath(inputDir: URL) -> String?` +- [ ] L408 — `private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL]` +- [ ] L455 — `private func processFile(inputPath:libPath:helpers:useCache:timeoutSeconds:) async throws -> ProcessResult` +- [ ] L533 — `internal func wrap(source: String, originalPath: String) -> String` +- [ ] L623 — `private func runSwift(wrappedPath:libPath:helpers:timeoutSeconds:) async throws -> ProcessResult` +- [ ] L708 — `private func exitCode(from status: TerminationStatus) -> Int32` + +### Variables +- [ ] L157 — `internal let toolchainStampFilename = "swift-version.txt"` +- [ ] L602 — `private let timeoutExitCode: Int32 = 124` +- [ ] L607 — `private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024` +- [ ] L608 — `private let stderrLimitBytes: Int = 1 * 1_024 * 1_024` + +--- + +## Sources/skit/Helpers.swift + +All inside `#if canImport(Subprocess)`. + +### Functions +- [ ] L40 — `internal func dylibFilename(forLibrary name: String) -> String` +- [ ] L66 — `internal func discoverHelpersDir(near inputURL: URL) -> URL?` +- [ ] L86 — `internal func collectHelperSources(in helpersDir: URL) throws -> [URL]` +- [ ] L116 — `internal func buildHelpers(helpersDir: URL, libPath: String) async throws -> CompiledHelpers?` +- [ ] L182 — `private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) async throws` +- [ ] L239 — `private func helpersCacheKey(sources: [URL], libPath: String) async throws -> String` +- [ ] L263 — `internal func captureSwiftVersion() async -> String?` +- [ ] L275 — `internal func libStamp(libPath: String) -> String?` +- [ ] L285 — `internal func syntaxKitCacheRoot() throws -> URL` + +### Variables +- [ ] L37 — `internal let helpersModuleName = "SyntaxKitHelpers"` +- [ ] L49 — `private let helpersCacheSchemaVersion = "v1"` + +--- + +## Sources/skit/OutputCache.swift + +All inside `#if canImport(Subprocess)`. + +### Functions +- [ ] L41 — `internal func outputCacheKey(inputSource: String, helpers: CompiledHelpers?, libPath: String) async -> String` +- [ ] L85 — `internal func lookupCachedOutput(key: String) -> Data?` +- [ ] L92 — `internal func storeCachedOutput(key: String, data: Data) throws` +- [ ] L126 — `private func outputCacheDir(for key: String) throws -> URL` + +### Variables +- [ ] L35 — `private let outputCacheSchemaVersion = "v1"` + +--- + +## Sources/DocumentationHarness/Validator.swift + +### Variables +- [ ] L36 — `private let privateDefaultPathExtensions = ["md"]` + +--- + +## Summary + +| Target | Global funcs | Global vars | +|---|---:|---:| +| skit / Runner.swift | 14 | 4 | +| skit / Helpers.swift | 9 | 2 | +| skit / OutputCache.swift | 4 | 1 | +| DocumentationHarness / Validator.swift | 0 | 1 | +| **Total** | **27** | **8** | + +All `skit` globals are free functions/constants inside `#if canImport(Subprocess)` — a deliberate CLI style. `privateDefaultPathExtensions` is the lone non-skit global (a linter reverted an earlier attempt to nest it). From 358c83bde76357ef53e649c342be563f81259d98 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Sun, 7 Jun 2026 20:30:12 -0400 Subject: [PATCH 30/56] Split skit Runner types into separate files [skip ci] Extract the six types from Runner.swift into their own files (HelpersOptions, ToolchainCheckResult, FileOutcome, ProcessResult, CLIError, SwiftRunOutcome), clearing the one_declaration_per_file warnings. Convert two type-returning functions into initializers on their return type: toolchainCheck -> ToolchainCheckResult.init(libPath:) and resolveHelpers -> CompiledHelpers.init?(nearInputPath:...). Extract CompiledHelpers into its own file to host the initializer, which also clears the pre-existing file_name error on Helpers.swift. Move toolchainStampFilename (used only by the toolchain check) into ToolchainCheckResult as a private static, removing a global. Co-Authored-By: Claude Opus 4.8 (1M context) --- Sources/skit/CLIError.swift | 39 +++++++ Sources/skit/CompiledHelpers.swift | 88 ++++++++++++++++ Sources/skit/FileOutcome.swift | 41 ++++++++ Sources/skit/Helpers.swift | 8 -- Sources/skit/HelpersOptions.swift | 44 ++++++++ Sources/skit/ProcessResult.swift | 43 ++++++++ Sources/skit/Runner.swift | 130 +----------------------- Sources/skit/Skit+Run.swift | 8 +- Sources/skit/SwiftRunOutcome.swift | 42 ++++++++ Sources/skit/ToolchainCheckResult.swift | 77 ++++++++++++++ 10 files changed, 380 insertions(+), 140 deletions(-) create mode 100644 Sources/skit/CLIError.swift create mode 100644 Sources/skit/CompiledHelpers.swift create mode 100644 Sources/skit/FileOutcome.swift create mode 100644 Sources/skit/HelpersOptions.swift create mode 100644 Sources/skit/ProcessResult.swift create mode 100644 Sources/skit/SwiftRunOutcome.swift create mode 100644 Sources/skit/ToolchainCheckResult.swift diff --git a/Sources/skit/CLIError.swift b/Sources/skit/CLIError.swift new file mode 100644 index 0000000..ece6614 --- /dev/null +++ b/Sources/skit/CLIError.swift @@ -0,0 +1,39 @@ +// +// CLIError.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + /// Throwable error wrapper for skit's user-facing diagnostics. The message + /// is printed verbatim — keep it actionable (path, hint, next step). + internal struct CLIError: Error, CustomStringConvertible { + let message: String + var description: String { message } + } + +#endif diff --git a/Sources/skit/CompiledHelpers.swift b/Sources/skit/CompiledHelpers.swift new file mode 100644 index 0000000..ac20724 --- /dev/null +++ b/Sources/skit/CompiledHelpers.swift @@ -0,0 +1,88 @@ +// +// CompiledHelpers.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + + /// A compiled `Helpers/` directory ready to splice into the input spawn. + internal struct CompiledHelpers: Sendable { + /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. + let outputDir: URL + /// Whether the build was reused from cache (false = freshly compiled). + let cacheHit: Bool + } + + extension CompiledHelpers { + /// Resolves a `Helpers/` directory and compiles it (or reuses the cached + /// build). Fails (returns nil) when helpers are disabled, when no `Helpers/` + /// was found in auto mode, or when the directory exists but contains no + /// `.swift` sources. On success, writes a one-line "skit: helpers + /// cached/compiled at " note to stderr so users can see whether the + /// cache hit. + internal init?( + nearInputPath path: String, + libPath: String, + options: HelpersOptions + ) async throws { + // Pick the helpers dir according to the mode: walk up the tree, accept an + // explicit override (after validating it's a directory), or bail out. + let helpersDir: URL? + switch options { + case .disabled: + return nil + case .auto: + helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) + case .explicit(let dir): + let url = URL(fileURLWithPath: dir).standardizedFileURL + var isDir: ObjCBool = false + guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), + isDir.boolValue + else { + throw CLIError(message: "--helpers path is not a directory: \(dir)") + } + helpersDir = url + } + guard let helpersDir else { return nil } + + // Compile (or reuse the cached build). An empty Helpers/ dir is treated + // as "no helpers" rather than an error. + guard let compiled = try await buildHelpers(helpersDir: helpersDir, libPath: libPath) else { + return nil + } + let suffix = compiled.cacheHit ? "cached" : "compiled" + FileHandle.standardError.write( + Data( + "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 + )) + self = compiled + } + } + +#endif diff --git a/Sources/skit/FileOutcome.swift b/Sources/skit/FileOutcome.swift new file mode 100644 index 0000000..f4ff50a --- /dev/null +++ b/Sources/skit/FileOutcome.swift @@ -0,0 +1,41 @@ +// +// FileOutcome.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + + /// Result of processing one input in directory mode. The error case is + /// stored (not thrown) so the batch can keep going. + internal struct FileOutcome: Sendable { + let input: URL + let result: Result + } + +#endif diff --git a/Sources/skit/Helpers.swift b/Sources/skit/Helpers.swift index 12430a4..9d0754d 100644 --- a/Sources/skit/Helpers.swift +++ b/Sources/skit/Helpers.swift @@ -48,14 +48,6 @@ /// Bumped when the cache layout changes in a way that requires invalidation. private let helpersCacheSchemaVersion = "v1" - /// A compiled `Helpers/` directory ready to splice into the input spawn. - internal struct CompiledHelpers: Sendable { - /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. - let outputDir: URL - /// Whether the build was reused from cache (false = freshly compiled). - let cacheHit: Bool - } - // MARK: - Discovery /// Walks up from `inputURL` looking for a `Helpers/` directory. Returns the diff --git a/Sources/skit/HelpersOptions.swift b/Sources/skit/HelpersOptions.swift new file mode 100644 index 0000000..713e1fc --- /dev/null +++ b/Sources/skit/HelpersOptions.swift @@ -0,0 +1,44 @@ +// +// HelpersOptions.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + /// How `skit run` should treat a `Helpers/` directory for this invocation. + internal enum HelpersOptions { + /// Walk up from the input looking for `Helpers/`. The default. + case auto + /// Skip helpers discovery entirely (`--no-helpers`). The wrapped input + /// won't be able to `import SyntaxKitHelpers`. + case disabled + /// Use the directory at the given path (`--helpers `). Validated as + /// an existing directory in `CompiledHelpers.init`. + case explicit(String) + } + +#endif diff --git a/Sources/skit/ProcessResult.swift b/Sources/skit/ProcessResult.swift new file mode 100644 index 0000000..c5178bb --- /dev/null +++ b/Sources/skit/ProcessResult.swift @@ -0,0 +1,43 @@ +// +// ProcessResult.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + + /// Raw outcome of rendering one input — what `processFile` returns to its + /// caller. `exitCode == 0` indicates the spawned `swift` succeeded (or that + /// the output cache hit, which is treated identically). + internal struct ProcessResult: Sendable { + let exitCode: Int32 + let stdout: Data + let stderr: String + } + +#endif diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index 4a8513d..7795b1b 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -36,71 +36,14 @@ // Run lifecycle (per `skit run` invocation): // 1. resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) - // 2. toolchainCheck — compare bundle stamp to `swift --version` - // 3. resolveHelpers — discover + compile Helpers/ (memoised on disk) + // 2. ToolchainCheckResult.init — compare bundle stamp to `swift --version` + // 3. CompiledHelpers.init — discover + compile Helpers/ (memoised on disk) // 4. runSingleFile / runDirectory — dispatch to single- or batch-input mode // 5. processFile (per input) — load → cache lookup → wrap → spawn → cache store // 6. wrap — hoist imports, wrap body in Group { … }, #sourceLocation // 7. runSwift — spawn `swift` with timeout watchdog // See Docs/skit.md for design rationale and trade-offs. - // MARK: - Helpers resolution - - /// How `skit run` should treat a `Helpers/` directory for this invocation. - internal enum HelpersOptions { - /// Walk up from the input looking for `Helpers/`. The default. - case auto - /// Skip helpers discovery entirely (`--no-helpers`). The wrapped input - /// won't be able to `import SyntaxKitHelpers`. - case disabled - /// Use the directory at the given path (`--helpers `). Validated as - /// an existing directory in `resolveHelpers`. - case explicit(String) - } - - /// Resolves a `Helpers/` directory and compiles it (or returns the cached - /// build). Returns nil when helpers are disabled, when no `Helpers/` was - /// found in auto mode, or when the directory exists but contains no `.swift` - /// sources. On success, writes a one-line "skit: helpers cached/compiled at - /// " note to stderr so users can see whether the cache hit. - internal func resolveHelpers( - nearInputPath path: String, - libPath: String, - options: HelpersOptions - ) async throws -> CompiledHelpers? { - // Pick the helpers dir according to the mode: walk up the tree, accept an - // explicit override (after validating it's a directory), or bail out. - let helpersDir: URL? - switch options { - case .disabled: - return nil - case .auto: - helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) - case .explicit(let dir): - let url = URL(fileURLWithPath: dir).standardizedFileURL - var isDir: ObjCBool = false - guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), - isDir.boolValue - else { - throw CLIError(message: "--helpers path is not a directory: \(dir)") - } - helpersDir = url - } - guard let helpersDir else { return nil } - - // Compile (or reuse the cached build). An empty Helpers/ dir is treated - // as "no helpers" rather than an error. - guard let compiled = try await buildHelpers(helpersDir: helpersDir, libPath: libPath) else { - return nil - } - let suffix = compiled.cacheHit ? "cached" : "compiled" - FileHandle.standardError.write( - Data( - "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 - )) - return compiled - } - // MARK: - Resource location /// Resolves the directory containing `libSyntaxKit.dylib` + module files, @@ -153,44 +96,6 @@ // MARK: - Toolchain check - /// Filename for the bundle's recorded build-toolchain version. - internal let toolchainStampFilename = "swift-version.txt" - - internal enum ToolchainCheckResult { - /// Bundle stamp matches the local `swift --version` exactly. - case match - /// `/swift-version.txt` is missing (older bundle that predates - /// the stamp). skit prints a one-line note and proceeds. - case stampMissing - case mismatch(bundle: String, local: String) - } - - /// Compares `/swift-version.txt` to `captureSwiftVersion()`. - /// The swiftmodule format isn't reliably forward-compatible across even - /// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the - /// swiftmodule), so the comparison is exact-string after normalising - /// trailing whitespace. - internal func toolchainCheck(libPath: String) async -> ToolchainCheckResult { - let stampURL = URL(fileURLWithPath: libPath).appendingPathComponent(toolchainStampFilename) - guard let stampData = try? Data(contentsOf: stampURL), - let stampRaw = String(data: stampData, encoding: .utf8) - else { - FileHandle.standardError.write( - Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) - ) - return .stampMissing - } - guard let localRaw = await captureSwiftVersion() else { - FileHandle.standardError.write( - Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) - ) - return .stampMissing - } - let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) - let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) - return bundle == local ? .match : .mismatch(bundle: bundle, local: local) - } - internal func toolchainMismatchMessage(bundle: String, local: String) -> String { """ skit: toolchain mismatch @@ -358,13 +263,6 @@ return failed == 0 ? 0 : 1 } - /// Result of processing one input in directory mode. The error case is - /// stored (not thrown) so the batch can keep going. - private struct FileOutcome: Sendable { - let input: URL - let result: Result - } - /// `processFile` adapter that catches errors into the `FileOutcome` result /// so a single failure doesn't tear down the surrounding `TaskGroup`. private func runOne( @@ -439,15 +337,6 @@ // MARK: - Per-file work - /// Raw outcome of rendering one input — what `processFile` returns to its - /// caller. `exitCode == 0` indicates the spawned `swift` succeeded (or that - /// the output cache hit, which is treated identically). - private struct ProcessResult: Sendable { - let exitCode: Int32 - let stdout: Data - let stderr: String - } - /// The per-input render pipeline: load source → consult the output cache → /// (on miss) wrap → spawn `swift` → rewrite diagnostics → store the result /// in the cache. The temp wrapper file is created in a per-run tmp dir and @@ -516,13 +405,6 @@ return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } - /// Throwable error wrapper for skit's user-facing diagnostics. The message - /// is printed verbatim — keep it actionable (path, hint, next step). - internal struct CLIError: Error, CustomStringConvertible { - let message: String - var description: String { message } - } - // MARK: - Wrapping /// Splits the input into hoisted `import` declarations and a verbatim body, @@ -607,14 +489,6 @@ private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 private let stderrLimitBytes: Int = 1 * 1_024 * 1_024 - /// Either the spawned `swift` ran to completion (success or failure) or - /// the watchdog elapsed first. The completed payload is normalized to the - /// shape callers want regardless of platform. - private enum SwiftRunOutcome: Sendable { - case completed(exitCode: Int32, stdout: Data, stderr: String) - case timedOut - } - /// Spawns `swift` (script-mode interpreter) on the wrapped input file, /// optionally splicing in flags to import a precompiled helpers module. /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index baf8d7b..4c71246 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -114,7 +114,7 @@ extension Skit { // versions, so a mismatch produces a clear error rather than letting // the spawned `swift` emit a cryptic module-version diagnostic. if !noToolchainCheck { - switch await toolchainCheck(libPath: libPath) { + switch await ToolchainCheckResult(libPath: libPath) { case .match, .stampMissing: break case .mismatch(let bundle, let local): @@ -125,7 +125,7 @@ extension Skit { } // 3. Decide which helpers-resolution mode this invocation is in. - // The actual discovery / compilation happens later in `resolveHelpers`. + // The actual discovery / compilation happens later in `CompiledHelpers.init`. let helpersOptions: HelpersOptions if noHelpers { helpersOptions = .disabled @@ -150,7 +150,7 @@ extension Skit { // 5a. Resolve helpers relative to the input root. This is the only // place we compile `Helpers/`; the result is reused across every // input file in the directory. - let helpers = try await resolveHelpers( + let helpers = try await CompiledHelpers( nearInputPath: input, libPath: libPath, options: helpersOptions @@ -168,7 +168,7 @@ extension Skit { throw ExitCode(exitCode) } else { // 5b. Resolve helpers relative to this single file's parent. - let helpers = try await resolveHelpers( + let helpers = try await CompiledHelpers( nearInputPath: input, libPath: libPath, options: helpersOptions diff --git a/Sources/skit/SwiftRunOutcome.swift b/Sources/skit/SwiftRunOutcome.swift new file mode 100644 index 0000000..71cfbec --- /dev/null +++ b/Sources/skit/SwiftRunOutcome.swift @@ -0,0 +1,42 @@ +// +// SwiftRunOutcome.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + + /// Either the spawned `swift` ran to completion (success or failure) or + /// the watchdog elapsed first. The completed payload is normalized to the + /// shape callers want regardless of platform. + internal enum SwiftRunOutcome: Sendable { + case completed(exitCode: Int32, stdout: Data, stderr: String) + case timedOut + } + +#endif diff --git a/Sources/skit/ToolchainCheckResult.swift b/Sources/skit/ToolchainCheckResult.swift new file mode 100644 index 0000000..d3a6fa8 --- /dev/null +++ b/Sources/skit/ToolchainCheckResult.swift @@ -0,0 +1,77 @@ +// +// ToolchainCheckResult.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + + internal enum ToolchainCheckResult { + /// Filename for the bundle's recorded build-toolchain version. + private static let toolchainStampFilename = "swift-version.txt" + + /// Bundle stamp matches the local `swift --version` exactly. + case match + /// `/swift-version.txt` is missing (older bundle that predates + /// the stamp). skit prints a one-line note and proceeds. + case stampMissing + case mismatch(bundle: String, local: String) + } + + extension ToolchainCheckResult { + /// Compares `/swift-version.txt` to `captureSwiftVersion()`. + /// The swiftmodule format isn't reliably forward-compatible across even + /// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the + /// swiftmodule), so the comparison is exact-string after normalising + /// trailing whitespace. + internal init(libPath: String) async { + let stampURL = URL(fileURLWithPath: libPath) + .appendingPathComponent(Self.toolchainStampFilename) + guard let stampData = try? Data(contentsOf: stampURL), + let stampRaw = String(data: stampData, encoding: .utf8) + else { + FileHandle.standardError.write( + Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) + ) + self = .stampMissing + return + } + guard let localRaw = await captureSwiftVersion() else { + FileHandle.standardError.write( + Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) + ) + self = .stampMissing + return + } + let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) + let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) + self = bundle == local ? .match : .mismatch(bundle: bundle, local: local) + } + } + +#endif From 9f6c642f700474a178c577be1b076025eb0c0b73 Mon Sep 17 00:00:00 2001 From: leogdion Date: Mon, 8 Jun 2026 11:09:12 -0400 Subject: [PATCH 31/56] Lift resolveLibPath onto Bundle and isLibDir onto FileManager [skip ci] Replace two file-scope globals in Sources/skit/Runner.swift with extension methods, advancing the no-globals migration tracked in globals-audit.md. Bundle.resolveLibPath(candidates:) takes a variadic list of optional candidate paths so the caller supplies both --lib and $SKIT_LIB_DIR rather than having the function pull env itself. Bundle keeps the adjacent and Homebrew fallbacks because they need executableURL. Co-Authored-By: Claude Opus 4.7 (1M context) --- Sources/skit/Bundle+ResolveLibPath.swift | 82 ++++++++++++++++++++++++ Sources/skit/FileManager+IsLibDir.swift | 45 +++++++++++++ Sources/skit/Runner.swift | 52 +-------------- Sources/skit/Skit+Run.swift | 5 +- globals-audit.md | 6 +- 5 files changed, 134 insertions(+), 56 deletions(-) create mode 100644 Sources/skit/Bundle+ResolveLibPath.swift create mode 100644 Sources/skit/FileManager+IsLibDir.swift diff --git a/Sources/skit/Bundle+ResolveLibPath.swift b/Sources/skit/Bundle+ResolveLibPath.swift new file mode 100644 index 0000000..bcced95 --- /dev/null +++ b/Sources/skit/Bundle+ResolveLibPath.swift @@ -0,0 +1,82 @@ +// +// Bundle+ResolveLibPath.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + + extension Bundle { + /// Resolves a directory containing `libSyntaxKit.dylib` + swiftmodules. + /// + /// Tries each non-nil entry in `candidates` in order; if any non-nil + /// candidate is not a SyntaxKit lib dir, throws `CLIError`. If every + /// candidate is absent, falls back to bundle-relative paths derived + /// from `executableURL`: + /// - `/lib` (adjacent layout) + /// - `/../lib/skit` (Homebrew layout) + internal func resolveLibPath(candidates: String?...) throws -> String { + let fileManager = FileManager.default + + for candidate in candidates { + guard let candidate else { continue } + guard fileManager.isLibDir(candidate) else { + throw CLIError(message: "path does not look like a SyntaxKit lib dir: \(candidate)") + } + return candidate + } + + if let execURL = executableURL?.resolvingSymlinksInPath() { + let execDir = execURL.deletingLastPathComponent() + + let adjacent = execDir.appendingPathComponent("lib").path + if fileManager.isLibDir(adjacent) { + return adjacent + } + + let brewLayout = execDir.deletingLastPathComponent() + .appendingPathComponent("lib/skit").path + if fileManager.isLibDir(brewLayout) { + return brewLayout + } + } + + throw CLIError( + message: """ + Could not locate SyntaxKit lib directory. Looked for: + 1. explicit candidates (none provided or all empty) + 2. /lib/ (not found) + 3. /../lib/skit/ (not found) + Run Scripts/build-skit-release.sh to produce a self-contained + release bundle under .build/skit-release/. + """ + ) + } + } + +#endif diff --git a/Sources/skit/FileManager+IsLibDir.swift b/Sources/skit/FileManager+IsLibDir.swift new file mode 100644 index 0000000..ecfdd3c --- /dev/null +++ b/Sources/skit/FileManager+IsLibDir.swift @@ -0,0 +1,45 @@ +// +// FileManager+IsLibDir.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + + extension FileManager { + /// True if `path` is a directory containing `libSyntaxKit.{dylib,so}`. + internal func isLibDir(_ path: String) -> Bool { + var isDir: ObjCBool = false + guard fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { + return false + } + return fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") + } + } + +#endif diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index 7795b1b..11c46e8 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -35,7 +35,7 @@ import SwiftSyntax // Run lifecycle (per `skit run` invocation): - // 1. resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) + // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) // 2. ToolchainCheckResult.init — compare bundle stamp to `swift --version` // 3. CompiledHelpers.init — discover + compile Helpers/ (memoised on disk) // 4. runSingleFile / runDirectory — dispatch to single- or batch-input mode @@ -44,56 +44,6 @@ // 7. runSwift — spawn `swift` with timeout watchdog // See Docs/skit.md for design rationale and trade-offs. - // MARK: - Resource location - - /// Resolves the directory containing `libSyntaxKit.dylib` + module files, - /// in priority order: explicit flag → env var → adjacent-to-binary - /// (`/lib/`) → Homebrew layout (`/../lib/skit/`). - internal func resolveLibPath(override: String?) throws -> String { - if let override { - guard isLibDir(override) else { - throw CLIError(message: "--lib path does not look like a SyntaxKit lib dir: \(override)") - } - return override - } - - if let env = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"], !env.isEmpty { - guard isLibDir(env) else { - throw CLIError(message: "SKIT_LIB_DIR is set but path is not a lib dir: \(env)") - } - return env - } - - if let execURL = Bundle.main.executableURL?.resolvingSymlinksInPath() { - let execDir = execURL.deletingLastPathComponent() - - let adjacent = execDir.appendingPathComponent("lib").path - if isLibDir(adjacent) { return adjacent } - - let brewLayout = execDir.deletingLastPathComponent() - .appendingPathComponent("lib/skit").path - if isLibDir(brewLayout) { return brewLayout } - } - - throw CLIError( - message: """ - Could not locate SyntaxKit lib directory. Looked for: - 1. --lib (not provided) - 2. $SKIT_LIB_DIR (not set) - 3. /lib/ (not found) - 4. /../lib/skit/ (not found) - Run Scripts/build-skit-release.sh to produce a self-contained - release bundle under .build/skit-release/. - """) - } - - private func isLibDir(_ path: String) -> Bool { - let fm = FileManager.default - var isDir: ObjCBool = false - guard fm.fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } - return fm.fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") - } - // MARK: - Toolchain check internal func toolchainMismatchMessage(bundle: String, local: String) -> String { diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index 4c71246..f444ec7 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -103,7 +103,10 @@ extension Skit { // live. The error message lists the four lookup paths in priority order. let libPath: String do { - libPath = try resolveLibPath(override: self.libPath) + let envLibPath = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"].flatMap { + $0.isEmpty ? nil : $0 + } + libPath = try Bundle.main.resolveLibPath(candidates: self.libPath, envLibPath) } catch { FileHandle.standardError.write(Data("\(error)\n".utf8)) throw ExitCode(2) diff --git a/globals-audit.md b/globals-audit.md index c25f870..d1e81bb 100644 --- a/globals-audit.md +++ b/globals-audit.md @@ -16,8 +16,6 @@ All inside `#if canImport(Subprocess)`. ### Functions - [ ] L66 — `internal func resolveHelpers(nearInputPath path: String, libPath: String, options: HelpersOptions) async throws -> CompiledHelpers?` -- [ ] L109 — `internal func resolveLibPath(override: String?) throws -> String` -- [ ] L147 — `private func isLibDir(_ path: String) -> Bool` - [ ] L173 — `internal func toolchainCheck(libPath: String) async -> ToolchainCheckResult` - [ ] L194 — `internal func toolchainMismatchMessage(bundle: String, local: String) -> String` - [ ] L218 — `internal func runSingleFile(inputPath:outputPath:libPath:helpers:useCache:timeoutSeconds:) async throws` @@ -85,10 +83,10 @@ All inside `#if canImport(Subprocess)`. | Target | Global funcs | Global vars | |---|---:|---:| -| skit / Runner.swift | 14 | 4 | +| skit / Runner.swift | 12 | 4 | | skit / Helpers.swift | 9 | 2 | | skit / OutputCache.swift | 4 | 1 | | DocumentationHarness / Validator.swift | 0 | 1 | -| **Total** | **27** | **8** | +| **Total** | **25** | **8** | All `skit` globals are free functions/constants inside `#if canImport(Subprocess)` — a deliberate CLI style. `privateDefaultPathExtensions` is the lone non-skit global (a linter reverted an earlier attempt to nest it). From 800d49a5011d32e6ab5028e94c191c54d3f04b6f Mon Sep 17 00:00:00 2001 From: leogdion Date: Mon, 8 Jun 2026 11:31:00 -0400 Subject: [PATCH 32/56] Remove Helpers feature from skit [skip ci] The Helpers/ module (compile-once-link-many for shared codegen functions) had no real consumer and added ~250 LOC, a cache layer, 3 CLI flags, and a `swiftc` step. Directory mode (`skit run InputDir/ -o OutDir/`) handles the N-inputs case without it. If shared codegen becomes a real need, a simpler prelude-inline approach can be added later. Deletes: CompiledHelpers.swift, HelpersOptions.swift, the helpers discovery/build/cache pipeline in Helpers.swift, the --helpers/--no-helpers CLI flags, the helpers param threaded through Runner.swift and OutputCache.outputCacheKey, and the Docs/skit.md + Sources/skit/README.md sections about helpers. Renames Helpers.swift -> Toolchain.swift, keeping the surviving toolchain/cache utilities (dylibFilename, captureSwiftVersion, libStamp, syntaxKitCacheRoot) that OutputCache and FileManager +IsLibDir still depend on. Verified end-to-end: single-file mode renders correctly; directory mode renders 3/3 in 0.77s cold, 0.28s warm cache, 0.65s --no-cache; --help confirms helpers flags are gone. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/skit.md | 67 ++----- Sources/skit/CompiledHelpers.swift | 88 --------- Sources/skit/Helpers.swift | 290 ----------------------------- Sources/skit/HelpersOptions.swift | 44 ----- Sources/skit/OutputCache.swift | 16 +- Sources/skit/README.md | 40 +--- Sources/skit/Runner.swift | 88 +++------ Sources/skit/Skit+Run.swift | 45 +---- Sources/skit/Toolchain.swift | 80 ++++++++ globals-audit.md | 63 +++---- 10 files changed, 158 insertions(+), 663 deletions(-) delete mode 100644 Sources/skit/CompiledHelpers.swift delete mode 100644 Sources/skit/Helpers.swift delete mode 100644 Sources/skit/HelpersOptions.swift create mode 100644 Sources/skit/Toolchain.swift diff --git a/Docs/skit.md b/Docs/skit.md index 35baa07..50a2528 100644 --- a/Docs/skit.md +++ b/Docs/skit.md @@ -2,7 +2,7 @@ `skit` is a small CLI that takes a SyntaxKit DSL file as input and writes Swift source out the other side. The vision: pure data — JSON, YAML, your own format — drives a manifest written in the SyntaxKit DSL, and `skit` materializes it into idiomatic Swift you check in alongside everything else. Less hand-maintenance, fewer drift bugs. -This doc walks through how `skit` is built. Two verbs, one wrap-and-spawn pipeline, two layers of cache, and a careful toolchain story underneath. Where there are sharp edges, this doc names them. +This doc walks through how `skit` is built. Two verbs, one wrap-and-spawn pipeline, an output cache, and a careful toolchain story underneath. Where there are sharp edges, this doc names them. ## Two verbs @@ -23,12 +23,10 @@ A `.swift` input file looks like a SyntaxKit DSL expression at the top level: ```swift // Models.swift -import SyntaxKitHelpers // optional — only if a Helpers/ dir is present - -equatableModel("Person", fields: [ - ("name", "String"), - ("age", "Int"), -]) +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} ``` The input is not a complete Swift program. It has no `@main`, no `print`, no `let root = …`. `skit run` adds the boilerplate by wrapping the input in a `Group { … }` builder and a top-level `print` that renders the result: @@ -36,14 +34,13 @@ The input is not a complete Swift program. It has no `@main`, no `print`, no `le ```swift // What `skit run` writes to a temp file before spawning `swift`: import SyntaxKit -import SyntaxKitHelpers // hoisted from the input let __skit_root = Group { #sourceLocation(file: "/path/to/Models.swift", line: 3) -equatableModel("Person", fields: [ - ("name", "String"), - ("age", "Int"), -]) +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} #sourceLocation() } @@ -54,26 +51,19 @@ print(__skit_root.generateCode()) Three things deserve a closer look: -**Imports get hoisted.** The wrapper has to start with `import SyntaxKit` so the DSL types are available. The user's `import`s — typically `import SyntaxKitHelpers` plus anything their Helpers/ module references — need to live at the top of the file, not inside `Group { … }`. `skit` parses the input with SwiftSyntax, peels off the leading `import` declarations, and lifts them into the wrapper preamble. Anything else (declarations, expressions, top-level types) stays in the body. +**Imports get hoisted.** The wrapper has to start with `import SyntaxKit` so the DSL types are available. Any additional `import`s the user writes need to live at the top of the file, not inside `Group { … }`. `skit` parses the input with SwiftSyntax, peels off the leading `import` declarations, and lifts them into the wrapper preamble. Anything else (declarations, expressions, top-level types) stays in the body. **`#sourceLocation` keeps diagnostics readable.** When the spawned `swift` emits a compile error, it reports a line number in the wrapped temp file, which is meaningless to the user. The `#sourceLocation` directive remaps body diagnostics back to the original input path and line. Errors in the wrapper preamble (the `import` block, the `Group { … }` opening) still reference the temp file — `skit` rewrites occurrences of the temp path in stderr to the input path as a fallback, so users see something coherent. **`swift` runs in script mode.** Running `swift Input.swift` invokes the Swift interpreter rather than going through `swiftc` + `ld`. Cold-start is around 700ms on macOS; warm spawns are around 110ms. The CLI's hot path leans into this — for batch input via `skit run InputDir/`, we spawn one `swift` per input file in parallel up to the active core count. -## Caches - -Two layers, both keyed by content hash. They live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. - -| Layer | Path | What it skips on hit | -| ------- | ----------------------------- | --------------------------------------------- | -| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | -| Output | `outputs//output.swift` | the `swift` spawn for an input | +## Output cache -**Helpers cache.** Each project's `Helpers/` directory gets compiled into `libSyntaxKitHelpers.{dylib,so}` once and reused. The cache key is a hash of the helper sources, plus the bundled `libSyntaxKit` stamp, plus `swift --version`. Touching a helper file invalidates one shard; updating the toolchain invalidates everything. Hit on a warm cache: skip the ~1–2s `swiftc` compile entirely. +One layer, keyed by content hash. Lives under `~/Library/Caches/com.brightdigit.SyntaxKit/outputs//output.swift` on macOS, `$XDG_CACHE_HOME/syntaxkit/outputs//output.swift` (or `~/.cache/syntaxkit/outputs//output.swift`) on Linux. On hit, `skit` skips the `swift` spawn for an input entirely. -**Output cache.** The fully-rendered output of an input gets cached by a hash of (input bytes, helpers shard, libSyntaxKit stamp, swift version, sorted `SKIT_*`/`SYNTAXKIT_*` env vars). On hit, `skit` doesn't spawn `swift` at all — total wall time is around 0.14s, dominated by hash + file read. Cold miss matches the warm script-mode baseline (~0.5s). +The fully-rendered output of an input gets cached by a hash of (input bytes, libSyntaxKit stamp, swift version, sorted `SKIT_*`/`SYNTAXKIT_*` env vars). On hit, total wall time is around 0.14s, dominated by hash + file read. Cold miss matches the warm script-mode baseline (~0.5s). -`--no-cache` skips the output cache. There's no flag to skip the helpers cache — invalidate it by touching a helper or bumping the toolchain. +`--no-cache` skips the cache. ## Toolchain stamping @@ -97,29 +87,12 @@ The comparison is exact-string match. Patch-level drift broke the originating bu ## Timeout watchdog -The spawned `swift` is the only unbounded piece of `skit run`'s hot path. The wrap step is microseconds. Helpers compile is cached. The output cache hits or misses in milliseconds. But the spawn itself runs *user code* — and that code is allowed to be arbitrarily slow, recursive, or stuck. +The spawned `swift` is the only unbounded piece of `skit run`'s hot path. The wrap step is microseconds. The output cache hits or misses in milliseconds. But the spawn itself runs *user code* — and that code is allowed to be arbitrarily slow, recursive, or stuck. `skit run` defaults to a 60s per-input timeout. On expiry it sends `SIGTERM`, gives a 5s grace, then `SIGKILL`. The wrapped input exits with code 124 — POSIX `timeout(1)`'s convention. `--timeout ` overrides the default; `--timeout 0` disables the watchdog entirely (useful for debugging genuinely long codegen). The implementation is `DispatchSemaphore.wait(timeout: deadline)` paired with a `process.terminationHandler` that signals on child exit. The Linux Foundation `Process.waitUntilExit()` hangs on already-exited children on some configurations, which is why `skit` uses the semaphore-based wait everywhere. Same story for pipe drains — sequential reads after the child exits can deadlock when either pipe (~64 KB buffer on Linux) fills before exit, so both pipes drain concurrently via `DispatchGroup`. -## Helpers - -Shared codegen utilities live in a `Helpers/` directory. `skit` walks up from the input, finds the nearest `Helpers/`, and compiles its sources into a Swift dylib that the wrapped input can `import SyntaxKitHelpers`. - -``` -project/ -├── Helpers/ -│ └── Models.swift # public func equatableModel(_:fields:) -> any CodeBlock -└── inputs/ - ├── Person.swift # imports SyntaxKitHelpers, calls equatableModel(...) - └── Pet.swift # same -``` - -Files prefixed with `_` are skipped — a convention for private helpers within the helpers module. The module name is hard-coded to `SyntaxKitHelpers`. - -The helpers cache is per-content, so editing a helper triggers a fresh compile but reading the same helper across many inputs hits the cache. - ## Sharp edges ### `if`-in-`Group` crashes the Swift type-checker — [#158](https://github.com/brightdigit/SyntaxKit/issues/158) @@ -134,11 +107,10 @@ let _ = Group { } ``` -This is a Swift compiler bug, not a `skit` bug. The workaround is to hoist the conditional into a helper function that uses **plain Swift `if`/`else`** (not a `Group { if … }` body) to return one of two `CodeBlock`s: +This is a Swift compiler bug, not a `skit` bug. The workaround is to hoist the conditional into a plain Swift function that uses **plain Swift `if`/`else`** (not a `Group { if … }` body) to return one of two `CodeBlock`s: ```swift -// In Helpers/Models.swift -public func optionalDebugField(_ include: Bool) -> any CodeBlock { +func optionalDebugField(_ include: Bool) -> any CodeBlock { if include { return Variable(.let, name: "debug", type: "Bool") } else { @@ -147,14 +119,13 @@ public func optionalDebugField(_ include: Bool) -> any CodeBlock { } } -// In Input.swift Struct("Config") { Variable(.let, name: "name", type: "String") optionalDebugField(buildIsDebug) } ``` -The helper itself can't use `Group { if … }` either — same crash. Plain Swift control flow only. +The helper function itself can't use `Group { if … }` either — same crash. Plain Swift control flow only. ### `@main` and top-level decl attributes don't work @@ -199,7 +170,7 @@ A few things were considered for v1 and explicitly punted: ## Reference -- [`Sources/skit/README.md`](../Sources/skit/README.md) — per-target quick reference (flag table, helpers layout). +- [`Sources/skit/README.md`](../Sources/skit/README.md) — per-target quick reference (flag table). - [`Scripts/build-skit-release.sh`](../Scripts/build-skit-release.sh) — release-bundle builder. - [`Docs/research/tuist-manifest-pipeline.md`](research/tuist-manifest-pipeline.md) — the manifest-pipeline pattern this CLI borrows from. - [Issue #154](https://github.com/brightdigit/SyntaxKit/issues/154) — original tracking issue. diff --git a/Sources/skit/CompiledHelpers.swift b/Sources/skit/CompiledHelpers.swift deleted file mode 100644 index ac20724..0000000 --- a/Sources/skit/CompiledHelpers.swift +++ /dev/null @@ -1,88 +0,0 @@ -// -// CompiledHelpers.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -#if canImport(Subprocess) - - import Foundation - - /// A compiled `Helpers/` directory ready to splice into the input spawn. - internal struct CompiledHelpers: Sendable { - /// Directory containing `libSyntaxKitHelpers.dylib` + `.swiftmodule` files. - let outputDir: URL - /// Whether the build was reused from cache (false = freshly compiled). - let cacheHit: Bool - } - - extension CompiledHelpers { - /// Resolves a `Helpers/` directory and compiles it (or reuses the cached - /// build). Fails (returns nil) when helpers are disabled, when no `Helpers/` - /// was found in auto mode, or when the directory exists but contains no - /// `.swift` sources. On success, writes a one-line "skit: helpers - /// cached/compiled at " note to stderr so users can see whether the - /// cache hit. - internal init?( - nearInputPath path: String, - libPath: String, - options: HelpersOptions - ) async throws { - // Pick the helpers dir according to the mode: walk up the tree, accept an - // explicit override (after validating it's a directory), or bail out. - let helpersDir: URL? - switch options { - case .disabled: - return nil - case .auto: - helpersDir = discoverHelpersDir(near: URL(fileURLWithPath: path).standardizedFileURL) - case .explicit(let dir): - let url = URL(fileURLWithPath: dir).standardizedFileURL - var isDir: ObjCBool = false - guard FileManager.default.fileExists(atPath: url.path, isDirectory: &isDir), - isDir.boolValue - else { - throw CLIError(message: "--helpers path is not a directory: \(dir)") - } - helpersDir = url - } - guard let helpersDir else { return nil } - - // Compile (or reuse the cached build). An empty Helpers/ dir is treated - // as "no helpers" rather than an error. - guard let compiled = try await buildHelpers(helpersDir: helpersDir, libPath: libPath) else { - return nil - } - let suffix = compiled.cacheHit ? "cached" : "compiled" - FileHandle.standardError.write( - Data( - "skit: helpers \(suffix) at \(helpersDir.path)\n".utf8 - )) - self = compiled - } - } - -#endif diff --git a/Sources/skit/Helpers.swift b/Sources/skit/Helpers.swift deleted file mode 100644 index 9d0754d..0000000 --- a/Sources/skit/Helpers.swift +++ /dev/null @@ -1,290 +0,0 @@ -// -// Helpers.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -#if canImport(Subprocess) - - import Foundation - import Subprocess - - /// Hardcoded module name for the user's `Helpers/` compilation output. Inputs - /// reach the compiled helpers via `import SyntaxKitHelpers`. - internal let helpersModuleName = "SyntaxKitHelpers" - - /// Platform-specific shared-library filename for a Swift library product. - internal func dylibFilename(forLibrary name: String) -> String { - #if os(Linux) - return "lib\(name).so" - #else - return "lib\(name).dylib" - #endif - } - - /// Bumped when the cache layout changes in a way that requires invalidation. - private let helpersCacheSchemaVersion = "v1" - - // MARK: - Discovery - - /// Walks up from `inputURL` looking for a `Helpers/` directory. Returns the - /// first one found, or nil if no ancestor contains one. - /// - /// When `inputURL` is a file, the search starts from its parent. When it's a - /// directory, the search starts from the directory itself. - internal func discoverHelpersDir(near inputURL: URL) -> URL? { - let fm = FileManager.default - var isDirectory: ObjCBool = false - let exists = fm.fileExists(atPath: inputURL.path, isDirectory: &isDirectory) - var dir = (exists && isDirectory.boolValue) ? inputURL : inputURL.deletingLastPathComponent() - dir = dir.standardizedFileURL - - while true { - let candidate = dir.appendingPathComponent("Helpers") - var isDir: ObjCBool = false - if fm.fileExists(atPath: candidate.path, isDirectory: &isDir), isDir.boolValue { - return candidate.standardizedFileURL - } - let parent = dir.deletingLastPathComponent().standardizedFileURL - if parent.path == dir.path { return nil } - dir = parent - } - } - - /// Globs `**/*.swift` under `helpersDir`, skipping files prefixed with `_`. - internal func collectHelperSources(in helpersDir: URL) throws -> [URL] { - guard - let enumerator = FileManager.default.enumerator( - at: helpersDir, - includingPropertiesForKeys: [.isRegularFileKey], - options: [.skipsHiddenFiles] - ) - else { - throw CLIError(message: "could not enumerate \(helpersDir.path)") - } - - var result: [URL] = [] - for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey]) - guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } - result.append(url.standardizedFileURL) - } - return result.sorted { $0.path < $1.path } - } - - // MARK: - Build pipeline - - /// Compiles helper sources into a per-key cache directory and returns the - /// directory plus a cache-hit flag. Returns nil when the helpers dir is empty. - /// - /// Concurrent invocations are tolerated via the staging-dir + atomic-rename - /// pattern: if two processes race to compile the same key, the loser's - /// rename fails and we keep the winner's artefact. - internal func buildHelpers( - helpersDir: URL, - libPath: String - ) async throws -> CompiledHelpers? { - // Collect helper sources. An empty Helpers/ dir is "no helpers" rather - // than an error — the caller will fall back to no-helpers mode. - let sources = try collectHelperSources(in: helpersDir) - if sources.isEmpty { return nil } - - // Compute the content-keyed cache path. The dylib's presence under that - // path is what makes a build "cached". - let key = try await helpersCacheKey(sources: sources, libPath: libPath) - let cacheRoot = try syntaxKitCacheRoot() - .appendingPathComponent("helpers") - .appendingPathComponent(key) - let dylibPath = - cacheRoot - .appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path - - // Cache hit: artefact already present, skip the whole compile. - let fm = FileManager.default - if fm.fileExists(atPath: dylibPath) { - return CompiledHelpers(outputDir: cacheRoot, cacheHit: true) - } - - // Ensure the parent of the cache key dir exists. We don't create the - // key dir itself — the atomic move below installs it. - try fm.createDirectory( - at: cacheRoot.deletingLastPathComponent(), - withIntermediateDirectories: true - ) - - // Compile into a per-pid + uuid staging dir, then atomically rename into - // place. This is what lets concurrent skit invocations co-exist safely. - let staging = cacheRoot.deletingLastPathComponent() - .appendingPathComponent( - "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)") - try fm.createDirectory(at: staging, withIntermediateDirectories: true) - - // Run swiftc into the staging dir. Clean up on failure so we don't leak - // half-baked artefacts in the cache root. - do { - try await compileHelpers(sources: sources, into: staging, libPath: libPath) - } catch { - try? fm.removeItem(at: staging) - throw error - } - - // Atomic rename into the cache path. If a peer beat us to it (rename failed - // because the destination now exists), keep theirs and drop ours. - do { - try fm.moveItem(at: staging, to: cacheRoot) - } catch { - try? fm.removeItem(at: staging) - if !fm.fileExists(atPath: dylibPath) { - throw error - } - } - - return CompiledHelpers(outputDir: cacheRoot, cacheHit: false) - } - - /// Invokes `swiftc` to build `sources` into a Swift module + dylib under - /// `outDir`. The dylib is named `lib.{dylib,so}` and the - /// module file is `.swiftmodule`. Output (stdout) - /// is discarded; stderr is captured for the error path. - private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) async throws { - let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - let dylib = outDir.appendingPathComponent(dylibFilename(forLibrary: helpersModuleName)).path - let modulePath = outDir.appendingPathComponent("\(helpersModuleName).swiftmodule").path - - // Base swiftc arguments: emit a library + module file linking against - // libSyntaxKit, with rpath set so the dylib can find libSyntaxKit at - // load time. - var args: [String] = [ - "-module-name", helpersModuleName, - "-emit-module", - "-emit-module-path", modulePath, - "-parse-as-library", - "-emit-library", - "-o", dylib, - "-suppress-warnings", - "-I", libPath, - "-L", libPath, - "-lSyntaxKit", - "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-rpath", "-Xlinker", libPath, - ] - #if !os(Linux) - // @rpath install_name is macOS-only; on Linux SONAME isn't needed because - // we use rpath-based loading and the dylib lives in a cache path that's - // known at link time. - args.append(contentsOf: [ - "-Xlinker", "-install_name", - "-Xlinker", "@rpath/\(dylibFilename(forLibrary: helpersModuleName))", - ]) - #endif - // Append source files last so the leading flags apply to all of them. - args.append(contentsOf: sources.map(\.path)) - - // Spawn swiftc. Stderr is captured (1 MiB cap) so a compile failure can - // surface the diagnostic verbatim in a CLIError. - let result = try await run( - .name("swiftc"), - arguments: Arguments(args), - output: .discarded, - error: .string(limit: 1 * 1_024 * 1_024) - ) - - guard result.terminationStatus.isSuccess else { - let stderr = result.standardError ?? "" - throw CLIError( - message: """ - skit: failed to compile Helpers/ (\(result.terminationStatus)) - \(stderr) - """) - } - } - - // MARK: - Cache key - - /// Content-addressed cache key mixing schema version, each helper source's - /// filename + bytes, `swift --version`, and the libSyntaxKit stamp. - private func helpersCacheKey(sources: [URL], libPath: String) async throws -> String { - var hasher = ContentHasher() - hasher.update(data: Data(helpersCacheSchemaVersion.utf8)) - - // Filename matters as well as bytes — two same-content files with - // different names produce different symbols. - for source in sources { - let data = try Data(contentsOf: source) - hasher.update(data: Data(source.lastPathComponent.utf8)) - hasher.update(data: data) - } - - // Toolchain + dylib stamp invalidate on cross-version or in-place rebuild. - if let swiftVersion = await captureSwiftVersion() { - hasher.update(data: Data(swiftVersion.utf8)) - } - if let stamp = libStamp(libPath: libPath) { - hasher.update(data: Data(stamp.utf8)) - } - - return hasher.finalize() - } - - /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. - internal func captureSwiftVersion() async -> String? { - let result = try? await run( - .name("swift"), - arguments: ["--version"], - output: .string(limit: 4_096), - error: .discarded - ) - return result?.standardOutput - } - - /// `/` fingerprint of the bundled libSyntaxKit dylib, or nil - /// if unreadable. Catches in-place rebuilds without a version bump. - internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" - guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } - let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 - let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 - return "\(size)/\(Int(mtime))" - } - - /// Root for all skit caches. Honours `XDG_CACHE_HOME`, else macOS - /// `~/Library/Caches/...` or Linux `~/.cache/syntaxkit`. - internal func syntaxKitCacheRoot() throws -> URL { - if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { - return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") - } - let home = NSHomeDirectory() - #if os(macOS) - return URL(fileURLWithPath: home) - .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") - #else - return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") - #endif - } - -#endif diff --git a/Sources/skit/HelpersOptions.swift b/Sources/skit/HelpersOptions.swift deleted file mode 100644 index 713e1fc..0000000 --- a/Sources/skit/HelpersOptions.swift +++ /dev/null @@ -1,44 +0,0 @@ -// -// HelpersOptions.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -#if canImport(Subprocess) - - /// How `skit run` should treat a `Helpers/` directory for this invocation. - internal enum HelpersOptions { - /// Walk up from the input looking for `Helpers/`. The default. - case auto - /// Skip helpers discovery entirely (`--no-helpers`). The wrapped input - /// won't be able to `import SyntaxKitHelpers`. - case disabled - /// Use the directory at the given path (`--helpers `). Validated as - /// an existing directory in `CompiledHelpers.init`. - case explicit(String) - } - -#endif diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index e68dd84..9bc1b34 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -34,13 +34,12 @@ /// Bumped when the output cache layout changes in a way that requires invalidation. private let outputCacheSchemaVersion = "v1" - /// 64-bit content hash over (cache schema, input source bytes, helpers key, - /// swift version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). - /// Any change in these inputs produces a fresh key and forces a recompile. + /// 64-bit content hash over (cache schema, input source bytes, swift + /// version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any + /// change in these inputs produces a fresh key and forces a recompile. /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. internal func outputCacheKey( inputSource: String, - helpers: CompiledHelpers?, libPath: String ) async -> String { var hasher = ContentHasher() @@ -49,15 +48,6 @@ // Input source bytes: the primary driver of the key. hasher.update(data: Data(inputSource.utf8)) - // Helpers fingerprint. The helpers cache dir name *is* the helpers cache - // key (per Helpers.swift), so re-mixing it here cheaply propagates any - // helpers change into this key. - if let helpers { - hasher.update(data: Data(helpers.outputDir.lastPathComponent.utf8)) - } else { - hasher.update(data: Data("no-helpers".utf8)) - } - // Toolchain version. Different `swift` builds emit different bytes for // the same DSL input. if let version = await captureSwiftVersion() { diff --git a/Sources/skit/README.md b/Sources/skit/README.md index 630250a..d211161 100644 --- a/Sources/skit/README.md +++ b/Sources/skit/README.md @@ -36,41 +36,17 @@ The bundle is portable: `cp -r .build/skit-release ~/anywhere/` and `~/anywhere/ ```swift // Models.swift -import SyntaxKitHelpers // optional — only if a Helpers/ dir is present - -equatableModel("Person", fields: [ - ("name", "String"), - ("age", "Int"), -]) +Struct("Person") { + Variable(.let, name: "name", type: "String") + Variable(.let, name: "age", type: "Int") +} ``` What *won't* work inside the input: top-level `let`/`var` outside the builder DSL, `@main`, `print` (the wrapper adds its own). Result-builder closure rules apply. -## Helpers - -Shared codegen utilities live in a `Helpers/` directory anywhere up-tree from the input. `skit` walks up from the input file (or directory) looking for one. Sources are pre-compiled into `libSyntaxKitHelpers.{dylib,so}` once and cached by content hash: - -``` -project/ -├── Helpers/ -│ └── Models.swift # public func equatableModel(_:fields:) -> any CodeBlock -└── inputs/ - ├── Person.swift # imports SyntaxKitHelpers, calls equatableModel(...) - └── Pet.swift # same -``` - -Files prefixed with `_` are skipped (convention for private helpers within helpers). The helper module name is hard-coded to `SyntaxKitHelpers`. - -Force-disable: `--no-helpers`. Override location: `--helpers `. - -## Caches - -Two layers, both keyed on content + toolchain + dylib stamp + `SKIT_*`/`SYNTAXKIT_*` env vars. Live under `~/Library/Caches/com.brightdigit.SyntaxKit/` on macOS, `$XDG_CACHE_HOME/syntaxkit` (or `~/.cache/syntaxkit`) on Linux. +## Cache -| Layer | Path | What it skips on hit | -| ------- | ----------------------------- | --------------------------------------------- | -| Helpers | `helpers//` | the `swiftc` compile of `Helpers/*.swift` | -| Output | `outputs//output.swift` | the `swift` spawn for an input | +One layer, keyed on content + toolchain + dylib stamp + `SKIT_*`/`SYNTAXKIT_*` env vars. Lives under `~/Library/Caches/com.brightdigit.SyntaxKit/outputs//output.swift` on macOS, `$XDG_CACHE_HOME/syntaxkit/outputs//output.swift` (or `~/.cache/syntaxkit/outputs//output.swift`) on Linux. On hit, `skit` skips the `swift` spawn for an input entirely. Output cache hit is roughly ~0.14s on macOS (no spawn at all); cold miss matches the warm `swift` script-mode baseline (~0.5s). Force a miss with `--no-cache`. @@ -80,8 +56,6 @@ Output cache hit is roughly ~0.14s on macOS (no spawn at all); cold miss matches | ----------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-o, --output ` | stdout | Output file (single-file mode) or directory (folder mode). | | `--lib ` | auto | Directory containing `libSyntaxKit.{dylib,so}` + module files. Search order when omitted: `$SKIT_LIB_DIR` → `/lib/` → `/../lib/skit/`. | -| `--helpers ` | walk-up | Explicit `Helpers/` directory. | -| `--no-helpers` | (off) | Skip helpers discovery entirely. | | `--no-cache` | (off) | Skip the output cache; always spawn `swift`. | | `--timeout ` | `60` | Per-input timeout for the spawned `swift` (SIGTERM → 5s → SIGKILL). On expiry the file exits with code 124. Pass `0` to disable. | | `--no-toolchain-check` | (off) | Skip the startup check that compares `lib/swift-version.txt` to `swift --version`. swiftmodules aren't reliably compatible across compiler versions; on mismatch skit refuses to spawn `swift` and points at the rebuild script. Auto-rebuild fallback tracked in [#157](https://github.com/brightdigit/SyntaxKit/issues/157). | @@ -92,7 +66,7 @@ Output cache hit is roughly ~0.14s on macOS (no spawn at all); cold miss matches - **Linux** — verified on `swift:6.0-jammy/aarch64`. The Mach-O `install_name` step in `Scripts/build-skit-release.sh` is macOS-specific and skipped on Linux. - **Windows** — not supported. -Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. `Runner.swift` and `Helpers.swift` work around it with `terminationHandler` + `DispatchSemaphore`. +Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. `Runner.swift` works around it with `terminationHandler` + `DispatchSemaphore`. ## Deeper dive diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index 11c46e8..a20bba0 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -37,11 +37,10 @@ // Run lifecycle (per `skit run` invocation): // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) // 2. ToolchainCheckResult.init — compare bundle stamp to `swift --version` - // 3. CompiledHelpers.init — discover + compile Helpers/ (memoised on disk) - // 4. runSingleFile / runDirectory — dispatch to single- or batch-input mode - // 5. processFile (per input) — load → cache lookup → wrap → spawn → cache store - // 6. wrap — hoist imports, wrap body in Group { … }, #sourceLocation - // 7. runSwift — spawn `swift` with timeout watchdog + // 3. runSingleFile / runDirectory — dispatch to single- or batch-input mode + // 4. processFile (per input) — load → cache lookup → wrap → spawn → cache store + // 5. wrap — hoist imports, wrap body in Group { … }, #sourceLocation + // 6. runSwift — spawn `swift` with timeout watchdog // See Docs/skit.md for design rationale and trade-offs. // MARK: - Toolchain check @@ -74,7 +73,6 @@ inputPath: String, outputPath: String?, libPath: String, - helpers: CompiledHelpers?, useCache: Bool, timeoutSeconds: Int ) async throws { @@ -83,7 +81,6 @@ let result = try await processFile( inputPath: inputPath, libPath: libPath, - helpers: helpers, useCache: useCache, timeoutSeconds: timeoutSeconds ) @@ -114,18 +111,16 @@ inputDir: String, outputDir: String, libPath: String, - helpers: CompiledHelpers?, useCache: Bool, timeoutSeconds: Int ) async -> Int32 { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL - // Phase 1: enumerate inputs. Top-level `Helpers/` is excluded so its - // sources aren't processed as DSL inputs. + // Phase 1: enumerate inputs. let inputs: [URL] do { - inputs = try collectInputs(at: inputURL, excluding: helpersExcludePath(inputDir: inputURL)) + inputs = try collectInputs(at: inputURL) } catch { FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) return 1 @@ -149,7 +144,7 @@ guard let next = iterator.next() else { break } group.addTask { await runOne( - next, libPath: libPath, helpers: helpers, + next, libPath: libPath, useCache: useCache, timeoutSeconds: timeoutSeconds ) } @@ -160,7 +155,7 @@ if let next = iterator.next() { group.addTask { await runOne( - next, libPath: libPath, helpers: helpers, + next, libPath: libPath, useCache: useCache, timeoutSeconds: timeoutSeconds ) } @@ -218,7 +213,6 @@ private func runOne( _ input: URL, libPath: String, - helpers: CompiledHelpers?, useCache: Bool, timeoutSeconds: Int ) async -> FileOutcome { @@ -226,7 +220,6 @@ let result = try await processFile( inputPath: input.path, libPath: libPath, - helpers: helpers, useCache: useCache, timeoutSeconds: timeoutSeconds ) @@ -236,24 +229,10 @@ } } - /// Returns the path of a `Helpers/` directory living directly under `inputDir`, - /// so the folder-mode enumerator can skip its descendants. Helpers that live - /// outside the input tree don't need to be excluded (they aren't enumerated). - private func helpersExcludePath(inputDir: URL) -> String? { - let candidate = inputDir.appendingPathComponent("Helpers").standardizedFileURL - var isDir: ObjCBool = false - guard FileManager.default.fileExists(atPath: candidate.path, isDirectory: &isDir), - isDir.boolValue - else { - return nil - } - return candidate.path - } - /// Returns every `.swift` file under `inputDir` (recursive), sorted, with - /// hidden files, files prefixed by `_`, and the `excludedDir` subtree - /// removed. Sorted output keeps batch behaviour deterministic across runs. - private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL] { + /// hidden files and files prefixed by `_` removed. Sorted output keeps + /// batch behaviour deterministic across runs. + private func collectInputs(at inputDir: URL) throws -> [URL] { guard let enumerator = FileManager.default.enumerator( at: inputDir, @@ -267,14 +246,8 @@ var result: [URL] = [] for case let url as URL in enumerator { let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) - // Directories aren't outputs; if this is the excluded `Helpers/` dir, - // prune the whole subtree. - if values.isDirectory == true { - if let excludedDir, url.standardizedFileURL.path == excludedDir { - enumerator.skipDescendants() - } - continue - } + // Directories aren't outputs. + if values.isDirectory == true { continue } // Filter for `.swift` regular files, skipping the `_`-prefixed // convention for "not an input" sources. guard values.isRegularFile == true else { continue } @@ -294,7 +267,6 @@ private func processFile( inputPath: String, libPath: String, - helpers: CompiledHelpers?, useCache: Bool, timeoutSeconds: Int ) async throws -> ProcessResult { @@ -304,11 +276,11 @@ let source = try String(contentsOf: inputURL, encoding: .utf8) // Compute the output cache key (skipped under `--no-cache`). Mixes input - // bytes, toolchain version, helpers fingerprint, libSyntaxKit stamp, and - // sorted SKIT_*/SYNTAXKIT_* env vars — see `outputCacheKey`. + // bytes, toolchain version, libSyntaxKit stamp, and sorted + // SKIT_*/SYNTAXKIT_* env vars — see `outputCacheKey`. let cacheKey: String? = useCache - ? await outputCacheKey(inputSource: source, helpers: helpers, libPath: libPath) + ? await outputCacheKey(inputSource: source, libPath: libPath) : nil // Cache hit: skip the wrap+spawn entirely and return the stored output. if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { @@ -335,7 +307,6 @@ let raw = try await runSwift( wrappedPath: wrappedURL.path, libPath: libPath, - helpers: helpers, timeoutSeconds: timeoutSeconds ) // #sourceLocation maps body diagnostics back to the input file. Errors in @@ -439,46 +410,29 @@ private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 private let stderrLimitBytes: Int = 1 * 1_024 * 1_024 - /// Spawns `swift` (script-mode interpreter) on the wrapped input file, - /// optionally splicing in flags to import a precompiled helpers module. + /// Spawns `swift` (script-mode interpreter) on the wrapped input file. /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing /// task group; the loser is cancelled. On timeout, returns exit 124 with /// a one-line stderr message — matching POSIX `timeout(1)`'s convention. private func runSwift( wrappedPath: String, libPath: String, - helpers: CompiledHelpers?, timeoutSeconds: Int ) async throws -> ProcessResult { let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - // Build the base argument list: link against libSyntaxKit, include the - // CShims headers, set rpath so the dylib loads at runtime. - var arguments: [String] = [ + // Link against libSyntaxKit, include the CShims headers, set rpath so + // the dylib loads at runtime. + let argumentsCopy: [String] = [ "-suppress-warnings", "-I", libPath, "-L", libPath, "-lSyntaxKit", "-Xcc", "-I", "-Xcc", cShimsInclude, "-Xlinker", "-rpath", "-Xlinker", libPath, + wrappedPath, ] - // Splice in helpers-module flags only when a compiled helpers dylib is - // available. Skipping these makes `import SyntaxKitHelpers` fail in the - // wrapped input, which is fine when no Helpers/ dir was discovered. - if let helpers { - let helpersPath = helpers.outputDir.path - arguments.append(contentsOf: [ - "-I", helpersPath, - "-L", helpersPath, - "-l\(helpersModuleName)", - "-Xlinker", "-rpath", "-Xlinker", helpersPath, - ]) - } - - arguments.append(wrappedPath) - let argumentsCopy = arguments - // The actual subprocess call, wrapped in a closure so the task-group race // below can hold a single Sendable reference to it. let invocation: @Sendable () async throws -> SwiftRunOutcome = { diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index f444ec7..dd7f764 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -58,18 +58,6 @@ extension Skit { ) internal var libPath: String? - @Option( - name: .customLong("helpers"), - help: "Override Helpers/ directory location." - ) - internal var helpersDir: String? - - @Flag( - name: .customLong("no-helpers"), - help: "Skip helpers discovery entirely." - ) - internal var noHelpers: Bool = false - @Flag( name: .customLong("no-cache"), help: "Skip the rendered-output cache (always run swift)." @@ -127,18 +115,7 @@ extension Skit { } } - // 3. Decide which helpers-resolution mode this invocation is in. - // The actual discovery / compilation happens later in `CompiledHelpers.init`. - let helpersOptions: HelpersOptions - if noHelpers { - helpersOptions = .disabled - } else if let dir = helpersDir { - helpersOptions = .explicit(dir) - } else { - helpersOptions = .auto - } - - // 4. Stat the input to pick single-file vs. directory mode. Directory + // 3. Stat the input to pick single-file vs. directory mode. Directory // mode requires an explicit `-o` output dir; single-file mode falls // back to stdout. var isDirectory: ObjCBool = false @@ -150,40 +127,24 @@ extension Skit { guard let output else { throw ValidationError("directory inputs require -o ") } - // 5a. Resolve helpers relative to the input root. This is the only - // place we compile `Helpers/`; the result is reused across every - // input file in the directory. - let helpers = try await CompiledHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - // 6a. Hand off to the directory orchestrator and surface its exit + // 4a. Hand off to the directory orchestrator and surface its exit // code via ExitCode (so a partial-failure batch returns 1). let exitCode = await runDirectory( inputDir: input, outputDir: output, libPath: libPath, - helpers: helpers, useCache: !noCache, timeoutSeconds: timeoutSeconds ) throw ExitCode(exitCode) } else { - // 5b. Resolve helpers relative to this single file's parent. - let helpers = try await CompiledHelpers( - nearInputPath: input, - libPath: libPath, - options: helpersOptions - ) - // 6b. Hand off to the single-file orchestrator. It calls `exit()` + // 4b. Hand off to the single-file orchestrator. It calls `exit()` // directly on non-zero subprocess exit, so a thrown ExitCode here // would be unreachable in that path. try await runSingleFile( inputPath: input, outputPath: output, libPath: libPath, - helpers: helpers, useCache: !noCache, timeoutSeconds: timeoutSeconds ) diff --git a/Sources/skit/Toolchain.swift b/Sources/skit/Toolchain.swift new file mode 100644 index 0000000..e1c6f8b --- /dev/null +++ b/Sources/skit/Toolchain.swift @@ -0,0 +1,80 @@ +// +// Toolchain.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + import Subprocess + + /// Platform-specific shared-library filename for a Swift library product. + internal func dylibFilename(forLibrary name: String) -> String { + #if os(Linux) + return "lib\(name).so" + #else + return "lib\(name).dylib" + #endif + } + + /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. + internal func captureSwiftVersion() async -> String? { + let result = try? await run( + .name("swift"), + arguments: ["--version"], + output: .string(limit: 4_096), + error: .discarded + ) + return result?.standardOutput + } + + /// `/` fingerprint of the bundled libSyntaxKit dylib, or nil + /// if unreadable. Catches in-place rebuilds without a version bump. + internal func libStamp(libPath: String) -> String? { + let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" + guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } + let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 + let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 + return "\(size)/\(Int(mtime))" + } + + /// Root for all skit caches. Honours `XDG_CACHE_HOME`, else macOS + /// `~/Library/Caches/...` or Linux `~/.cache/syntaxkit`. + internal func syntaxKitCacheRoot() throws -> URL { + if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { + return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + } + let home = NSHomeDirectory() + #if os(macOS) + return URL(fileURLWithPath: home) + .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") + #else + return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") + #endif + } + +#endif diff --git a/globals-audit.md b/globals-audit.md index d1e81bb..d9ec59c 100644 --- a/globals-audit.md +++ b/globals-audit.md @@ -15,45 +15,32 @@ Generated: 2026-06-07. Branch: `research/swift-manifest-codegen`. All inside `#if canImport(Subprocess)`. ### Functions -- [ ] L66 — `internal func resolveHelpers(nearInputPath path: String, libPath: String, options: HelpersOptions) async throws -> CompiledHelpers?` -- [ ] L173 — `internal func toolchainCheck(libPath: String) async -> ToolchainCheckResult` -- [ ] L194 — `internal func toolchainMismatchMessage(bundle: String, local: String) -> String` -- [ ] L218 — `internal func runSingleFile(inputPath:outputPath:libPath:helpers:useCache:timeoutSeconds:) async throws` -- [ ] L258 — `internal func runDirectory(inputDir:outputDir:libPath:helpers:useCache:timeoutSeconds:) async -> Int32` -- [ ] L370 — `private func runOne(_ input: URL, libPath:helpers:useCache:timeoutSeconds:) async -> FileOutcome` -- [ ] L394 — `private func helpersExcludePath(inputDir: URL) -> String?` -- [ ] L408 — `private func collectInputs(at inputDir: URL, excluding excludedDir: String?) throws -> [URL]` -- [ ] L455 — `private func processFile(inputPath:libPath:helpers:useCache:timeoutSeconds:) async throws -> ProcessResult` -- [ ] L533 — `internal func wrap(source: String, originalPath: String) -> String` -- [ ] L623 — `private func runSwift(wrappedPath:libPath:helpers:timeoutSeconds:) async throws -> ProcessResult` -- [ ] L708 — `private func exitCode(from status: TerminationStatus) -> Int32` +- [ ] `internal func toolchainMismatchMessage(bundle: String, local: String) -> String` +- [ ] `internal func runSingleFile(inputPath:outputPath:libPath:useCache:timeoutSeconds:) async throws` +- [ ] `internal func runDirectory(inputDir:outputDir:libPath:useCache:timeoutSeconds:) async -> Int32` +- [ ] `private func runOne(_ input: URL, libPath:useCache:timeoutSeconds:) async -> FileOutcome` +- [ ] `private func collectInputs(at inputDir: URL) throws -> [URL]` +- [ ] `private func processFile(inputPath:libPath:useCache:timeoutSeconds:) async throws -> ProcessResult` +- [ ] `internal func wrap(source: String, originalPath: String) -> String` +- [ ] `private func runSwift(wrappedPath:libPath:timeoutSeconds:) async throws -> ProcessResult` +- [ ] `private func exitCode(from status: TerminationStatus) -> Int32` ### Variables -- [ ] L157 — `internal let toolchainStampFilename = "swift-version.txt"` -- [ ] L602 — `private let timeoutExitCode: Int32 = 124` -- [ ] L607 — `private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024` -- [ ] L608 — `private let stderrLimitBytes: Int = 1 * 1_024 * 1_024` +- [ ] `private let timeoutExitCode: Int32 = 124` +- [ ] `private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024` +- [ ] `private let stderrLimitBytes: Int = 1 * 1_024 * 1_024` --- -## Sources/skit/Helpers.swift +## Sources/skit/Toolchain.swift All inside `#if canImport(Subprocess)`. ### Functions -- [ ] L40 — `internal func dylibFilename(forLibrary name: String) -> String` -- [ ] L66 — `internal func discoverHelpersDir(near inputURL: URL) -> URL?` -- [ ] L86 — `internal func collectHelperSources(in helpersDir: URL) throws -> [URL]` -- [ ] L116 — `internal func buildHelpers(helpersDir: URL, libPath: String) async throws -> CompiledHelpers?` -- [ ] L182 — `private func compileHelpers(sources: [URL], into outDir: URL, libPath: String) async throws` -- [ ] L239 — `private func helpersCacheKey(sources: [URL], libPath: String) async throws -> String` -- [ ] L263 — `internal func captureSwiftVersion() async -> String?` -- [ ] L275 — `internal func libStamp(libPath: String) -> String?` -- [ ] L285 — `internal func syntaxKitCacheRoot() throws -> URL` - -### Variables -- [ ] L37 — `internal let helpersModuleName = "SyntaxKitHelpers"` -- [ ] L49 — `private let helpersCacheSchemaVersion = "v1"` +- [ ] `internal func dylibFilename(forLibrary name: String) -> String` +- [ ] `internal func captureSwiftVersion() async -> String?` +- [ ] `internal func libStamp(libPath: String) -> String?` +- [ ] `internal func syntaxKitCacheRoot() throws -> URL` --- @@ -62,13 +49,13 @@ All inside `#if canImport(Subprocess)`. All inside `#if canImport(Subprocess)`. ### Functions -- [ ] L41 — `internal func outputCacheKey(inputSource: String, helpers: CompiledHelpers?, libPath: String) async -> String` -- [ ] L85 — `internal func lookupCachedOutput(key: String) -> Data?` -- [ ] L92 — `internal func storeCachedOutput(key: String, data: Data) throws` -- [ ] L126 — `private func outputCacheDir(for key: String) throws -> URL` +- [ ] `internal func outputCacheKey(inputSource: String, libPath: String) async -> String` +- [ ] `internal func lookupCachedOutput(key: String) -> Data?` +- [ ] `internal func storeCachedOutput(key: String, data: Data) throws` +- [ ] `private func outputCacheDir(for key: String) throws -> URL` ### Variables -- [ ] L35 — `private let outputCacheSchemaVersion = "v1"` +- [ ] `private let outputCacheSchemaVersion = "v1"` --- @@ -83,10 +70,10 @@ All inside `#if canImport(Subprocess)`. | Target | Global funcs | Global vars | |---|---:|---:| -| skit / Runner.swift | 12 | 4 | -| skit / Helpers.swift | 9 | 2 | +| skit / Runner.swift | 9 | 3 | +| skit / Toolchain.swift | 4 | 0 | | skit / OutputCache.swift | 4 | 1 | | DocumentationHarness / Validator.swift | 0 | 1 | -| **Total** | **25** | **8** | +| **Total** | **17** | **5** | All `skit` globals are free functions/constants inside `#if canImport(Subprocess)` — a deliberate CLI style. `privateDefaultPathExtensions` is the lone non-skit global (a linter reverted an earlier attempt to nest it). From 5083d2b3869c225ee84d78e8c03aa2f34ff61dfc Mon Sep 17 00:00:00 2001 From: leogdion Date: Mon, 8 Jun 2026 11:57:56 -0400 Subject: [PATCH 33/56] Lift more skit globals into types/extensions and refactor OutputCache [skip ci] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Continues the no-globals migration tracked in globals-audit.md. Three related changes plus a new contributor doc: - OutputCache.swift: collapse 4 free functions + 1 schema-version constant into `internal struct OutputCache`. The `init` resolves `syntaxKitCacheRoot()` once and binds it to `self.root`; per-key directories are derived from it on demand. Methods rename to `key(forInput:libPath:)`, `lookup(key:)`, `store(key:data:)`. Runner.processFile updated; behaviour preserved (`--no-cache` and unresolvable-root both short-circuit via `try? OutputCache()`). - Toolchain.swift: extract two utilities into extensions matching the pattern of Bundle+ResolveLibPath / FileManager+IsLibDir. `dylibFilename(forLibrary:)` becomes `String.dylibFilename` (computed property: `"SyntaxKit".dylibFilename`). `libStamp(libPath:)` becomes `FileManager.libStamp(libPath:)`. Call sites updated in FileManager+IsLibDir and OutputCache. - Runner.swift: `wrap(source:originalPath:)` flipped to `private` — it was the only `internal` symbol with no cross-file callers. - Docs/skit-internals.md (new): per-module reference for Runner, OutputCache, and Toolchain. Linked from Docs/skit.md and Sources/skit/README.md. Globals audit drops from 17 funcs/5 vars to 11 funcs/4 vars. Verified end-to-end: 398 tests pass, --no-cache and cold/warm cache paths render identically on the 3-file batch smoke test (0.76s / 0.78s / 0.28s). Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/skit-internals.md | 110 +++++++++++++++++ Docs/skit.md | 1 + Sources/skit/FileManager+IsLibDir.swift | 2 +- Sources/skit/FileManager+LibStamp.swift | 47 +++++++ Sources/skit/OutputCache.swift | 158 ++++++++++++------------ Sources/skit/README.md | 3 +- Sources/skit/Runner.swift | 21 ++-- Sources/skit/String+DylibFilename.swift | 45 +++++++ Sources/skit/Toolchain.swift | 19 --- globals-audit.md | 53 +++----- 10 files changed, 316 insertions(+), 143 deletions(-) create mode 100644 Docs/skit-internals.md create mode 100644 Sources/skit/FileManager+LibStamp.swift create mode 100644 Sources/skit/String+DylibFilename.swift diff --git a/Docs/skit-internals.md b/Docs/skit-internals.md new file mode 100644 index 0000000..4b20f63 --- /dev/null +++ b/Docs/skit-internals.md @@ -0,0 +1,110 @@ +# `skit` internals: Runner, OutputCache, Toolchain + +A contributor's map of the three modules that do the actual work inside `skit run`. For the **why** (design rationale, trade-offs, sharp edges) see [`Docs/skit.md`](skit.md). For the **what** (flags, behaviour) see [`Sources/skit/README.md`](../Sources/skit/README.md). This doc covers the **how** — what each module needs, what it produces, and how they call each other. + +## The pipeline + +`Skit+Run.swift` is the only caller. Per invocation it does, in order: + +1. **`Bundle.main.resolveLibPath(candidates:)`** (`Sources/skit/Bundle+ResolveLibPath.swift`) — locate the lib bundle (`libSyntaxKit.dylib` + swiftmodules). The caller passes `--lib` and `$SKIT_LIB_DIR` as candidates; Bundle falls back to `/lib/` and `/../lib/skit/`. +2. **`ToolchainCheckResult(libPath:)`** (`Sources/skit/ToolchainCheckResult.swift`) — verify the bundle's recorded `swift --version` matches the local one. Refuses to spawn on mismatch. +3. **`runSingleFile(...)` or `runDirectory(...)`** — dispatch into Runner. Everything downstream (wrap, spawn, output-cache lookup/store) happens inside Runner. + +**Toolchain** is consumed transitively (by `Bundle+ResolveLibPath`, `FileManager+IsLibDir`, and `OutputCache`) — it's never called directly from the CLI driver. **OutputCache** is an implementation detail of `Runner.processFile`, never called from the CLI driver either. The CLI driver only sees two surfaces: `Bundle.main.resolveLibPath` and `runSingleFile`/`runDirectory`. + +## Toolchain (`Toolchain.swift` + extensions) + +**Purpose.** Provide the shared utilities that describe (a) the local `swift` toolchain, (b) the bundled SyntaxKit dylib, and (c) where skit's caches live on disk. These are infrastructure, not features — they exist so the rest of the code doesn't have to re-derive them. + +**Surface area** (all `internal`, all inside `#if canImport(Subprocess)`): + +| Symbol | Defined in | What it returns | Consumers | +|---|---|---|---| +| `String.dylibFilename` | `String+DylibFilename.swift` | `"lib.dylib"` on macOS, `"lib.so"` on Linux | `FileManager.isLibDir`, `FileManager.libStamp` | +| `FileManager.libStamp(libPath:)` | `FileManager+LibStamp.swift` | `"/"` of `libSyntaxKit.{dylib,so}`, or nil | `OutputCache.outputCacheKey` | +| `captureSwiftVersion()` | `Toolchain.swift` | verbatim `swift --version` stdout (≤4 KiB), or nil on spawn failure | `OutputCache.outputCacheKey`, `ToolchainCheckResult` | +| `syntaxKitCacheRoot()` | `Toolchain.swift` | `~/Library/Caches/com.brightdigit.SyntaxKit` (macOS), `$XDG_CACHE_HOME/syntaxkit` or `~/.cache/syntaxkit` (Linux) | `OutputCache.outputCacheDir` | + +**Why these four belong together.** They all answer the same question from different angles: "what state of the world does a cache key depend on?" The Swift toolchain (`captureSwiftVersion`), the runtime dylib (`libStamp` via `String.dylibFilename`), and the on-disk cache layout (`syntaxKitCacheRoot`). Together they give `OutputCache` everything it needs to compute a stable, sound key. The two that fit naturally as instance APIs (`String.dylibFilename`, `FileManager.libStamp`) live in their own extension files; the rest stay as free functions in `Toolchain.swift`. + +## Runner (`Runner.swift`) + +**Purpose.** Orchestrate per-input render: pick single-file vs. directory mode, wrap each input into a complete Swift program, spawn `swift` on it with a timeout watchdog, surface output, and consult the output cache. + +**What it needs from the caller:** +- `libPath: String` — the lib bundle dir from step 1; reused for the `swift` invocation's link/rpath flags. +- `useCache: Bool` — gates OutputCache lookup/store (`--no-cache` sets this to false). +- `timeoutSeconds: Int` — per-input watchdog (`0` opts out; default 60s; on expiry, exit 124 matching POSIX `timeout(1)`). + +**Two entry points:** + +| Function | When | Failure semantics | +|---|---|---| +| `runSingleFile(inputPath:outputPath:libPath:useCache:timeoutSeconds:)` (`Runner.swift:72`) | One input, one output (or stdout) | Calls `exit()` on non-zero subprocess result — caller won't see a thrown error in that path | +| `runDirectory(inputDir:outputDir:libPath:useCache:timeoutSeconds:)` (`Runner.swift:110`) | Walks `**/*.swift` under `inputDir`, bounded concurrency = `ProcessInfo.activeProcessorCount`, mirrors output into `outputDir` | Returns 0/1 instead of `exit()` — partial failure allowed, successful peers still written, one-line summary printed to stderr | + +**The per-input work** (`processFile`, `Runner.swift:267`): + +1. Load source bytes. +2. If `useCache`, compute the output cache key via `outputCacheKey(inputSource:libPath:)` and try a hit. +3. On miss: `wrap` the source (hoist imports, splice body into `Group { … }`, fence with `#sourceLocation` so compiler diagnostics map back to the original file), write to a per-invocation `skit-/` temp dir, spawn `swift`, rewrite stderr to swap the temp path for the original path, store the result via `storeCachedOutput`, return. + +The temp dir is cleaned with `defer { try? removeItem }` so a failed spawn doesn't leak files. + +**The spawn** (`runSwift`, `Runner.swift:417`). Fixed argument list: +``` +swift -suppress-warnings + -I -L -lSyntaxKit + -Xcc -I -Xcc /_SwiftSyntaxCShims-include + -Xlinker -rpath -Xlinker + +``` +Raced against a sleep watchdog in a throwing task group; the loser is cancelled. `timeoutSeconds <= 0` skips the race entirely. + +**Output bounds.** `stdoutLimitBytes = 16 MiB`, `stderrLimitBytes = 1 MiB`. Above either limit, `Subprocess` raises a clear error rather than silently truncating. Timeout exit code is 124 (`timeoutExitCode`), matching POSIX `timeout(1)`. + +## OutputCache (`OutputCache.swift`) + +**Purpose.** Cache the rendered Swift (the stdout of the spawned `swift` for a given input) so re-running with unchanged inputs avoids the spawn entirely. Hit cost ≈ 0.14s on macOS vs. ~0.5s for a cold script-mode `swift` spawn. + +**Shape.** Single `internal struct OutputCache` (`OutputCache.swift:35`), consumed only from `Runner.processFile`. `init()` throws — it resolves the cache root via `Toolchain.syntaxKitCacheRoot()` and binds it to `self.root` (`/outputs/`). The caller wraps the init in `try?` so a non-derivable cache root short-circuits silently (same effective behaviour as `--no-cache` for that invocation). + +**Surface** (all `internal` instance methods): + +- **`key(forInput:libPath:)`** — async, mixes: + - schema version (`Self.schemaVersion = "v1"` — bump to invalidate everything) + - input source bytes (the primary driver) + - `swift --version` (via `captureSwiftVersion`) + - libSyntaxKit `/` stamp (via `FileManager.default.libStamp(libPath:)`) + - sorted, NUL-terminated `SKIT_*` / `SYNTAXKIT_*` env vars +- **`lookup(key:)`** — returns the cached `output.swift` bytes or nil. +- **`store(key:data:)`** — atomic stage+rename: writes into `tmp../output.swift` next to the key dir, then `moveItem` to install. If a concurrent peer beat us to it, swallow the rename error and drop our staging copy; re-throw only if the destination is still missing afterwards. + +Storage is wrapped in `try?` at the caller in `processFile` — a cache *write* failure is never a render failure; the next run just re-spawns. + +**On-disk layout.** `/outputs//output.swift`. The per-key directory is derived by the private `directory(for:)` helper; the root comes from `Toolchain.syntaxKitCacheRoot()` once, at init. + +**Why FNV-1a, not SHA-256.** The cache keys aren't security-critical — there's no adversary trying to forge a collision — so `ContentHasher` uses a 64-bit FNV-1a. Deterministic across processes and platforms (unlike Swift's stdlib `Hasher`, whose seed is per-process randomized), which is what makes the keys usable as on-disk directory names. + +## Data-flow recap + +``` +Skit.Run.run() + ├─ Bundle.main.resolveLibPath(candidates: --lib, $SKIT_LIB_DIR) → libPath + │ └─ FileManager.default.isLibDir → "SyntaxKit".dylibFilename + ├─ ToolchainCheckResult(libPath:) → verify (uses captureSwiftVersion) + └─ runSingleFile / runDirectory(libPath, useCache, timeoutSeconds) + └─ processFile(input) + ├─ try? OutputCache() ← nil under --no-cache (or unresolvable root) + ├─ cache?.key(forInput: source, libPath:) ← captureSwiftVersion + FileManager.default.libStamp + ├─ cache.lookup(key:) ← hit returns immediately + ├─ wrap(source) → temp wrapper.swift + ├─ runSwift(wrappedPath, libPath) ← spawns `swift` linked against libSyntaxKit + └─ try? cache.store(key:, data:) ← on the way out (atomic stage+rename) +``` + +Three coupling facts worth remembering: + +- **Toolchain is shared infrastructure.** `OutputCache` depends on three of its four utilities; the bundle-resolution code depends on `dylibFilename`. None of those callers know or care that the utilities live in `Toolchain.swift` — they just call free functions. +- **Runner is the only place OutputCache is touched.** If you ever want a non-CLI consumer of skit (e.g. a long-running server), you'd either call `processFile` directly or replicate its cache logic. Don't instantiate `OutputCache` from elsewhere. +- **Cache safety under concurrency.** Both the store path (in `OutputCache`) and the bundle-resolution path are designed to be safe under concurrent invocations: write into `tmp../`, then `moveItem` to install. Race-loser swallows the rename error if the destination is now populated by a peer. diff --git a/Docs/skit.md b/Docs/skit.md index 50a2528..72f8099 100644 --- a/Docs/skit.md +++ b/Docs/skit.md @@ -171,6 +171,7 @@ A few things were considered for v1 and explicitly punted: ## Reference - [`Sources/skit/README.md`](../Sources/skit/README.md) — per-target quick reference (flag table). +- [`Docs/skit-internals.md`](skit-internals.md) — per-module reference for Runner, OutputCache, and Toolchain. - [`Scripts/build-skit-release.sh`](../Scripts/build-skit-release.sh) — release-bundle builder. - [`Docs/research/tuist-manifest-pipeline.md`](research/tuist-manifest-pipeline.md) — the manifest-pipeline pattern this CLI borrows from. - [Issue #154](https://github.com/brightdigit/SyntaxKit/issues/154) — original tracking issue. diff --git a/Sources/skit/FileManager+IsLibDir.swift b/Sources/skit/FileManager+IsLibDir.swift index ecfdd3c..bb512c9 100644 --- a/Sources/skit/FileManager+IsLibDir.swift +++ b/Sources/skit/FileManager+IsLibDir.swift @@ -38,7 +38,7 @@ guard fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } - return fileExists(atPath: "\(path)/\(dylibFilename(forLibrary: "SyntaxKit"))") + return fileExists(atPath: "\(path)/\("SyntaxKit".dylibFilename)") } } diff --git a/Sources/skit/FileManager+LibStamp.swift b/Sources/skit/FileManager+LibStamp.swift new file mode 100644 index 0000000..82a4ade --- /dev/null +++ b/Sources/skit/FileManager+LibStamp.swift @@ -0,0 +1,47 @@ +// +// FileManager+LibStamp.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Foundation + + extension FileManager { + /// `/` fingerprint of `libSyntaxKit.{dylib,so}` under + /// `libPath`, or nil if unreadable. Catches in-place rebuilds without a + /// version bump. + internal func libStamp(libPath: String) -> String? { + let dylib = "\(libPath)/\("SyntaxKit".dylibFilename)" + guard let attrs = try? attributesOfItem(atPath: dylib) else { return nil } + let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 + let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 + return "\(size)/\(Int(mtime))" + } + } + +#endif diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index 9bc1b34..1a97885 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -31,92 +31,98 @@ import Foundation - /// Bumped when the output cache layout changes in a way that requires invalidation. - private let outputCacheSchemaVersion = "v1" - - /// 64-bit content hash over (cache schema, input source bytes, swift - /// version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any - /// change in these inputs produces a fresh key and forces a recompile. - /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. - internal func outputCacheKey( - inputSource: String, - libPath: String - ) async -> String { - var hasher = ContentHasher() - // Schema version: bump to invalidate every existing cache entry at once. - hasher.update(data: Data(outputCacheSchemaVersion.utf8)) - // Input source bytes: the primary driver of the key. - hasher.update(data: Data(inputSource.utf8)) - - // Toolchain version. Different `swift` builds emit different bytes for - // the same DSL input. - if let version = await captureSwiftVersion() { - hasher.update(data: Data(version.utf8)) - } - // libSyntaxKit stamp. A rebuilt dylib can change the rendered output - // even without a Swift-version bump. - if let stamp = libStamp(libPath: libPath) { - hasher.update(data: Data(stamp.utf8)) + /// On-disk cache of rendered skit output, content-keyed so a re-run on + /// unchanged input skips the `swift` spawn entirely. + internal struct OutputCache { + /// Bumped when the cache layout changes in a way that requires invalidation. + private static let schemaVersion = "v1" + + /// `/outputs/`. Populated once at init; per-key + /// directories are derived from it on demand. + private let root: URL + + internal init() throws { + self.root = try syntaxKitCacheRoot().appendingPathComponent("outputs") } - // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and - // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. - let env = ProcessInfo.processInfo.environment - .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } - .sorted { $0.key < $1.key } - for (key, value) in env { - hasher.update(data: Data("\(key)=\(value)\0".utf8)) + /// 64-bit content hash over (schema version, input source bytes, swift + /// version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any + /// change in these inputs produces a fresh key and forces a recompile. + /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. + internal func key(forInput source: String, libPath: String) async -> String { + var hasher = ContentHasher() + // Schema version: bump to invalidate every existing cache entry at once. + hasher.update(data: Data(Self.schemaVersion.utf8)) + // Input source bytes: the primary driver of the key. + hasher.update(data: Data(source.utf8)) + + // Toolchain version. Different `swift` builds emit different bytes for + // the same DSL input. + if let version = await captureSwiftVersion() { + hasher.update(data: Data(version.utf8)) + } + // libSyntaxKit stamp. A rebuilt dylib can change the rendered output + // even without a Swift-version bump. + if let stamp = FileManager.default.libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } + + // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and + // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. + let env = ProcessInfo.processInfo.environment + .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } + .sorted { $0.key < $1.key } + for (key, value) in env { + hasher.update(data: Data("\(key)=\(value)\0".utf8)) + } + + return hasher.finalize() } - return hasher.finalize() - } + /// Returns the cached rendered output for `key`, or nil on miss. + internal func lookup(key: String) -> Data? { + try? Data(contentsOf: directory(for: key).appendingPathComponent("output.swift")) + } - /// Returns the cached rendered output for `key`, or nil on miss. - internal func lookupCachedOutput(key: String) -> Data? { - guard let dir = try? outputCacheDir(for: key) else { return nil } - return try? Data(contentsOf: dir.appendingPathComponent("output.swift")) - } + /// Atomically stores `data` under `key`. Concurrent writers race via a + /// `tmp../` staging dir + rename; the loser drops their copy. + internal func store(key: String, data: Data) throws { + let cacheRoot = directory(for: key) + let final = cacheRoot.appendingPathComponent("output.swift") + let fileManager = FileManager.default - /// Atomically stores `data` under `key`. Concurrent writers race via a - /// `tmp../` staging dir + rename; the loser drops their copy. - internal func storeCachedOutput(key: String, data: Data) throws { - let cacheRoot = try outputCacheDir(for: key) - let final = cacheRoot.appendingPathComponent("output.swift") - let fm = FileManager.default - - // Ensure the parent of the cache key dir exists. The key dir itself is - // installed by the atomic rename below. - try fm.createDirectory( - at: cacheRoot.deletingLastPathComponent(), - withIntermediateDirectories: true - ) - - // Stage the payload in a per-pid + uuid sibling dir so it can be renamed - // into place as a single atomic step. - let staging = cacheRoot.deletingLastPathComponent() - .appendingPathComponent( - "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + // Ensure the parent of the cache key dir exists. The key dir itself is + // installed by the atomic rename below. + try fileManager.createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true ) - try fm.createDirectory(at: staging, withIntermediateDirectories: true) - try data.write(to: staging.appendingPathComponent("output.swift")) - - // Atomic rename into the cache path. If a peer already populated this - // key, swallow the rename error and drop our staging copy. Re-throw only - // if the destination is still missing afterwards. - do { - try fm.moveItem(at: staging, to: cacheRoot) - } catch { - try? fm.removeItem(at: staging) - if !fm.fileExists(atPath: final.path) { - throw error + + // Stage the payload in a per-pid + uuid sibling dir so it can be renamed + // into place as a single atomic step. + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent( + "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + ) + try fileManager.createDirectory(at: staging, withIntermediateDirectories: true) + try data.write(to: staging.appendingPathComponent("output.swift")) + + // Atomic rename into the cache path. If a peer already populated this + // key, swallow the rename error and drop our staging copy. Re-throw only + // if the destination is still missing afterwards. + do { + try fileManager.moveItem(at: staging, to: cacheRoot) + } catch { + try? fileManager.removeItem(at: staging) + if !fileManager.fileExists(atPath: final.path) { + throw error + } } } - } - private func outputCacheDir(for key: String) throws -> URL { - try syntaxKitCacheRoot() - .appendingPathComponent("outputs") - .appendingPathComponent(key) + private func directory(for key: String) -> URL { + root.appendingPathComponent(key) + } } #endif diff --git a/Sources/skit/README.md b/Sources/skit/README.md index d211161..5e2e066 100644 --- a/Sources/skit/README.md +++ b/Sources/skit/README.md @@ -70,4 +70,5 @@ Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited ## Deeper dive -For the architecture, design decisions, and trade-offs see [`Docs/skit.md`](../../Docs/skit.md). +- [`Docs/skit.md`](../../Docs/skit.md) — architecture, design decisions, trade-offs. +- [`Docs/skit-internals.md`](../../Docs/skit-internals.md) — per-module reference for Runner, OutputCache, and Toolchain (what each needs, how they call each other). diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index a20bba0..e9f14f5 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -275,15 +275,14 @@ let absoluteInputPath = inputURL.path let source = try String(contentsOf: inputURL, encoding: .utf8) - // Compute the output cache key (skipped under `--no-cache`). Mixes input - // bytes, toolchain version, libSyntaxKit stamp, and sorted - // SKIT_*/SYNTAXKIT_* env vars — see `outputCacheKey`. - let cacheKey: String? = - useCache - ? await outputCacheKey(inputSource: source, libPath: libPath) - : nil + // Compute the output cache key (skipped under `--no-cache`, or if the + // cache root can't be derived). Mixes input bytes, toolchain version, + // libSyntaxKit stamp, and sorted SKIT_*/SYNTAXKIT_* env vars — see + // `OutputCache.key(forInput:libPath:)`. + let cache: OutputCache? = useCache ? try? OutputCache() : nil + let cacheKey: String? = await cache?.key(forInput: source, libPath: libPath) // Cache hit: skip the wrap+spawn entirely and return the stored output. - if let cacheKey, let cached = lookupCachedOutput(key: cacheKey) { + if let cache, let cacheKey, let cached = cache.lookup(key: cacheKey) { return ProcessResult(exitCode: 0, stdout: cached, stderr: "") } @@ -319,8 +318,8 @@ // Store on the way out. `try?` is deliberate: a cache write failure is // not a render failure. The next run will simply miss and re-spawn. - if let cacheKey, raw.exitCode == 0 { - try? storeCachedOutput(key: cacheKey, data: raw.stdout) + if let cache, let cacheKey, raw.exitCode == 0 { + try? cache.store(key: cacheKey, data: raw.stdout) } return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) @@ -333,7 +332,7 @@ /// /// The body is fenced in `#sourceLocation` directives so compiler diagnostics /// in the body reference the original input file and line numbers. - internal func wrap(source: String, originalPath: String) -> String { + private func wrap(source: String, originalPath: String) -> String { // Parse the input with SwiftSyntax. The location converter is needed to // map the body's starting byte offset back to a 1-based line number for // the `#sourceLocation` directive. diff --git a/Sources/skit/String+DylibFilename.swift b/Sources/skit/String+DylibFilename.swift new file mode 100644 index 0000000..3f9047c --- /dev/null +++ b/Sources/skit/String+DylibFilename.swift @@ -0,0 +1,45 @@ +// +// String+DylibFilename.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + extension String { + /// Treats `self` as a Swift library product name and returns the + /// platform-specific shared-library filename (`libFoo.dylib` on macOS, + /// `libFoo.so` on Linux). + internal var dylibFilename: String { + #if os(Linux) + return "lib\(self).so" + #else + return "lib\(self).dylib" + #endif + } + } + +#endif diff --git a/Sources/skit/Toolchain.swift b/Sources/skit/Toolchain.swift index e1c6f8b..de133c8 100644 --- a/Sources/skit/Toolchain.swift +++ b/Sources/skit/Toolchain.swift @@ -32,15 +32,6 @@ import Foundation import Subprocess - /// Platform-specific shared-library filename for a Swift library product. - internal func dylibFilename(forLibrary name: String) -> String { - #if os(Linux) - return "lib\(name).so" - #else - return "lib\(name).dylib" - #endif - } - /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. internal func captureSwiftVersion() async -> String? { let result = try? await run( @@ -52,16 +43,6 @@ return result?.standardOutput } - /// `/` fingerprint of the bundled libSyntaxKit dylib, or nil - /// if unreadable. Catches in-place rebuilds without a version bump. - internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/\(dylibFilename(forLibrary: "SyntaxKit"))" - guard let attrs = try? FileManager.default.attributesOfItem(atPath: dylib) else { return nil } - let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 - let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 - return "\(size)/\(Int(mtime))" - } - /// Root for all skit caches. Honours `XDG_CACHE_HOME`, else macOS /// `~/Library/Caches/...` or Linux `~/.cache/syntaxkit`. internal func syntaxKitCacheRoot() throws -> URL { diff --git a/globals-audit.md b/globals-audit.md index d9ec59c..d9aab19 100644 --- a/globals-audit.md +++ b/globals-audit.md @@ -6,7 +6,7 @@ Note: in `Sources/skit/` every listed declaration is indented because it lives i Clean targets (no global funcs/vars): **SyntaxKit**, **SyntaxParser**, **TokenVisitor**. -Generated: 2026-06-07. Branch: `research/swift-manifest-codegen`. +Generated: 2026-06-08. Branch: `research/swift-manifest-codegen`. --- @@ -15,20 +15,20 @@ Generated: 2026-06-07. Branch: `research/swift-manifest-codegen`. All inside `#if canImport(Subprocess)`. ### Functions -- [ ] `internal func toolchainMismatchMessage(bundle: String, local: String) -> String` -- [ ] `internal func runSingleFile(inputPath:outputPath:libPath:useCache:timeoutSeconds:) async throws` -- [ ] `internal func runDirectory(inputDir:outputDir:libPath:useCache:timeoutSeconds:) async -> Int32` -- [ ] `private func runOne(_ input: URL, libPath:useCache:timeoutSeconds:) async -> FileOutcome` -- [ ] `private func collectInputs(at inputDir: URL) throws -> [URL]` -- [ ] `private func processFile(inputPath:libPath:useCache:timeoutSeconds:) async throws -> ProcessResult` -- [ ] `internal func wrap(source: String, originalPath: String) -> String` -- [ ] `private func runSwift(wrappedPath:libPath:timeoutSeconds:) async throws -> ProcessResult` -- [ ] `private func exitCode(from status: TerminationStatus) -> Int32` +- [ ] L48 — `internal func toolchainMismatchMessage(bundle: String, local: String) -> String` +- [ ] L72 — `internal func runSingleFile(inputPath:outputPath:libPath:useCache:timeoutSeconds:) async throws` +- [ ] L110 — `internal func runDirectory(inputDir:outputDir:libPath:useCache:timeoutSeconds:) async -> Int32` +- [ ] L213 — `private func runOne(_ input: URL, libPath:useCache:timeoutSeconds:) async -> FileOutcome` +- [ ] L235 — `private func collectInputs(at inputDir: URL) throws -> [URL]` +- [ ] L267 — `private func processFile(inputPath:libPath:useCache:timeoutSeconds:) async throws -> ProcessResult` +- [ ] L336 — `private func wrap(source: String, originalPath: String) -> String` +- [ ] L417 — `private func runSwift(wrappedPath:libPath:timeoutSeconds:) async throws -> ProcessResult` +- [ ] L486 — `private func exitCode(from status: TerminationStatus) -> Int32` ### Variables -- [ ] `private let timeoutExitCode: Int32 = 124` -- [ ] `private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024` -- [ ] `private let stderrLimitBytes: Int = 1 * 1_024 * 1_024` +- [ ] L405 — `private let timeoutExitCode: Int32 = 124` +- [ ] L410 — `private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024` +- [ ] L411 — `private let stderrLimitBytes: Int = 1 * 1_024 * 1_024` --- @@ -37,25 +37,8 @@ All inside `#if canImport(Subprocess)`. All inside `#if canImport(Subprocess)`. ### Functions -- [ ] `internal func dylibFilename(forLibrary name: String) -> String` -- [ ] `internal func captureSwiftVersion() async -> String?` -- [ ] `internal func libStamp(libPath: String) -> String?` -- [ ] `internal func syntaxKitCacheRoot() throws -> URL` - ---- - -## Sources/skit/OutputCache.swift - -All inside `#if canImport(Subprocess)`. - -### Functions -- [ ] `internal func outputCacheKey(inputSource: String, libPath: String) async -> String` -- [ ] `internal func lookupCachedOutput(key: String) -> Data?` -- [ ] `internal func storeCachedOutput(key: String, data: Data) throws` -- [ ] `private func outputCacheDir(for key: String) throws -> URL` - -### Variables -- [ ] `private let outputCacheSchemaVersion = "v1"` +- [ ] L36 — `internal func captureSwiftVersion() async -> String?` +- [ ] L48 — `internal func syntaxKitCacheRoot() throws -> URL` --- @@ -71,9 +54,9 @@ All inside `#if canImport(Subprocess)`. | Target | Global funcs | Global vars | |---|---:|---:| | skit / Runner.swift | 9 | 3 | -| skit / Toolchain.swift | 4 | 0 | -| skit / OutputCache.swift | 4 | 1 | +| skit / Toolchain.swift | 2 | 0 | +| skit / OutputCache.swift | 0 | 0 | | DocumentationHarness / Validator.swift | 0 | 1 | -| **Total** | **17** | **5** | +| **Total** | **11** | **4** | All `skit` globals are free functions/constants inside `#if canImport(Subprocess)` — a deliberate CLI style. `privateDefaultPathExtensions` is the lone non-skit global (a linter reverted an earlier attempt to nest it). From 605814518fdceebad95580262212bbb01fc8938c Mon Sep 17 00:00:00 2001 From: leogdion Date: Mon, 8 Jun 2026 12:49:04 -0400 Subject: [PATCH 34/56] Capture swift --version once per skit run [skip ci] Previously `captureSwiftVersion()` was called twice from ToolchainCheckResult.init and from OutputCache.key (per-input). On a 100-file directory batch that's 101 spawns of `swift --version` (~50ms each) for what is, within one invocation, a constant string. Lift the capture to Skit.Run.run and thread the result down: - `ToolchainCheckResult.init(libPath:swiftVersion:)` accepts the string and is now synchronous (caller awaits once). - `OutputCache.init(swiftVersion:fileManager:processInfo:)` stores it, so `OutputCache.key(forInput:libPath:)` is now synchronous too. - Runner's parameter chain swaps `useCache: Bool` for `cache: OutputCache?` across runSingleFile, runDirectory, runOne, and processFile; processFile no longer builds the cache itself. - OutputCache marked `@unchecked Sendable` so the single instance can cross concurrent runOne task boundaries in directory mode. Measured win on the 3-file warm-cache smoke test: 0.28s -> 0.14s wall (the `swift --version` spawn was the dominant cost on cache hits). Cold and --no-cache paths unchanged within noise; 398 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/skit-internals.md | 31 +++++++++++--------- Sources/skit/OutputCache.swift | 39 ++++++++++++++++++------- Sources/skit/Runner.swift | 27 +++++++++-------- Sources/skit/Skit+Run.swift | 26 ++++++++++++----- Sources/skit/ToolchainCheckResult.swift | 14 ++++----- 5 files changed, 85 insertions(+), 52 deletions(-) diff --git a/Docs/skit-internals.md b/Docs/skit-internals.md index 4b20f63..84a4bd3 100644 --- a/Docs/skit-internals.md +++ b/Docs/skit-internals.md @@ -21,9 +21,9 @@ A contributor's map of the three modules that do the actual work inside `skit ru | Symbol | Defined in | What it returns | Consumers | |---|---|---|---| | `String.dylibFilename` | `String+DylibFilename.swift` | `"lib.dylib"` on macOS, `"lib.so"` on Linux | `FileManager.isLibDir`, `FileManager.libStamp` | -| `FileManager.libStamp(libPath:)` | `FileManager+LibStamp.swift` | `"/"` of `libSyntaxKit.{dylib,so}`, or nil | `OutputCache.outputCacheKey` | -| `captureSwiftVersion()` | `Toolchain.swift` | verbatim `swift --version` stdout (≤4 KiB), or nil on spawn failure | `OutputCache.outputCacheKey`, `ToolchainCheckResult` | -| `syntaxKitCacheRoot()` | `Toolchain.swift` | `~/Library/Caches/com.brightdigit.SyntaxKit` (macOS), `$XDG_CACHE_HOME/syntaxkit` or `~/.cache/syntaxkit` (Linux) | `OutputCache.outputCacheDir` | +| `FileManager.libStamp(libPath:)` | `FileManager+LibStamp.swift` | `"/"` of `libSyntaxKit.{dylib,so}`, or nil | `OutputCache.key` | +| `captureSwiftVersion()` | `Toolchain.swift` | verbatim `swift --version` stdout (≤4 KiB), or nil on spawn failure | `Skit.Run.run` (called once per invocation, then threaded into `ToolchainCheckResult.init` and `OutputCache.init`) | +| `syntaxKitCacheRoot()` | `Toolchain.swift` | `~/Library/Caches/com.brightdigit.SyntaxKit` (macOS), `$XDG_CACHE_HOME/syntaxkit` or `~/.cache/syntaxkit` (Linux) | `OutputCache.init` | **Why these four belong together.** They all answer the same question from different angles: "what state of the world does a cache key depend on?" The Swift toolchain (`captureSwiftVersion`), the runtime dylib (`libStamp` via `String.dylibFilename`), and the on-disk cache layout (`syntaxKitCacheRoot`). Together they give `OutputCache` everything it needs to compute a stable, sound key. The two that fit naturally as instance APIs (`String.dylibFilename`, `FileManager.libStamp`) live in their own extension files; the rest stay as free functions in `Toolchain.swift`. @@ -67,15 +67,17 @@ Raced against a sleep watchdog in a throwing task group; the loser is cancelled. **Purpose.** Cache the rendered Swift (the stdout of the spawned `swift` for a given input) so re-running with unchanged inputs avoids the spawn entirely. Hit cost ≈ 0.14s on macOS vs. ~0.5s for a cold script-mode `swift` spawn. -**Shape.** Single `internal struct OutputCache` (`OutputCache.swift:35`), consumed only from `Runner.processFile`. `init()` throws — it resolves the cache root via `Toolchain.syntaxKitCacheRoot()` and binds it to `self.root` (`/outputs/`). The caller wraps the init in `try?` so a non-derivable cache root short-circuits silently (same effective behaviour as `--no-cache` for that invocation). +**Shape.** Single `internal struct OutputCache: @unchecked Sendable`, built once per `skit run` invocation in `Skit.Run.run` and shared across every input. `init(swiftVersion:fileManager:processInfo:)` throws — it resolves the cache root via `Toolchain.syntaxKitCacheRoot()`, binds it to `self.root` (`/outputs/`), and stores `swiftVersion` so key derivation never has to re-spawn `swift`. The caller wraps the init in `try?` so a non-derivable cache root short-circuits silently (same effective behaviour as `--no-cache` for that invocation). + +The `@unchecked Sendable` conformance covers the `FileManager` / `ProcessInfo` stored properties (reference types that don't auto-derive Sendable, but the singletons we use are thread-safe for these operations). Runner shares one `OutputCache?` instance across concurrent `runOne` tasks in directory mode. **Surface** (all `internal` instance methods): -- **`key(forInput:libPath:)`** — async, mixes: +- **`key(forInput:libPath:)`** — synchronous (no `await`), mixes: - schema version (`Self.schemaVersion = "v1"` — bump to invalidate everything) - input source bytes (the primary driver) - - `swift --version` (via `captureSwiftVersion`) - - libSyntaxKit `/` stamp (via `FileManager.default.libStamp(libPath:)`) + - `self.swiftVersion` (captured at init from `Skit.Run.run`) + - libSyntaxKit `/` stamp (via `FileManager.libStamp(libPath:)`) - sorted, NUL-terminated `SKIT_*` / `SYNTAXKIT_*` env vars - **`lookup(key:)`** — returns the cached `output.swift` bytes or nil. - **`store(key:data:)`** — atomic stage+rename: writes into `tmp../output.swift` next to the key dir, then `moveItem` to install. If a concurrent peer beat us to it, swallow the rename error and drop our staging copy; re-throw only if the destination is still missing afterwards. @@ -92,15 +94,16 @@ Storage is wrapped in `try?` at the caller in `processFile` — a cache *write* Skit.Run.run() ├─ Bundle.main.resolveLibPath(candidates: --lib, $SKIT_LIB_DIR) → libPath │ └─ FileManager.default.isLibDir → "SyntaxKit".dylibFilename - ├─ ToolchainCheckResult(libPath:) → verify (uses captureSwiftVersion) - └─ runSingleFile / runDirectory(libPath, useCache, timeoutSeconds) + ├─ captureSwiftVersion() → swiftVersion (spawned exactly once) + ├─ ToolchainCheckResult(libPath:, swiftVersion:) → gate (compare to bundle stamp) + ├─ try? OutputCache(swiftVersion:) → cache (nil under --no-cache or unresolvable root) + └─ runSingleFile / runDirectory(libPath, cache, timeoutSeconds) └─ processFile(input) - ├─ try? OutputCache() ← nil under --no-cache (or unresolvable root) - ├─ cache?.key(forInput: source, libPath:) ← captureSwiftVersion + FileManager.default.libStamp - ├─ cache.lookup(key:) ← hit returns immediately + ├─ cache?.key(forInput: source, libPath:) ← self.swiftVersion + FileManager.default.libStamp + ├─ cache.lookup(key:) ← hit returns immediately ├─ wrap(source) → temp wrapper.swift - ├─ runSwift(wrappedPath, libPath) ← spawns `swift` linked against libSyntaxKit - └─ try? cache.store(key:, data:) ← on the way out (atomic stage+rename) + ├─ runSwift(wrappedPath, libPath) ← spawns `swift` linked against libSyntaxKit + └─ try? cache.store(key:, data:) ← on the way out (atomic stage+rename) ``` Three coupling facts worth remembering: diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index 1a97885..4d13366 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -33,23 +33,43 @@ /// On-disk cache of rendered skit output, content-keyed so a re-run on /// unchanged input skips the `swift` spawn entirely. - internal struct OutputCache { + /// + /// `@unchecked Sendable` because the stored `FileManager` / `ProcessInfo` + /// are reference types that don't auto-derive Sendable, but the default + /// singletons used in production (and the typical test doubles) are + /// thread-safe for the operations we invoke. The single instance is + /// shared across concurrent `runOne` tasks in directory mode. + internal struct OutputCache: @unchecked Sendable { /// Bumped when the cache layout changes in a way that requires invalidation. private static let schemaVersion = "v1" /// `/outputs/`. Populated once at init; per-key /// directories are derived from it on demand. private let root: URL - - internal init() throws { + private let fileManager: FileManager + private let processInfo: ProcessInfo + + /// Verbatim `swift --version` output captured once for the lifetime of + /// this cache, so per-input key derivation doesn't re-spawn `swift`. + /// nil if capture failed before construction. + private let swiftVersion: String? + + internal init( + swiftVersion: String?, + fileManager: FileManager = .default, + processInfo: ProcessInfo = .processInfo + ) throws { self.root = try syntaxKitCacheRoot().appendingPathComponent("outputs") + self.swiftVersion = swiftVersion + self.fileManager = fileManager + self.processInfo = processInfo } /// 64-bit content hash over (schema version, input source bytes, swift /// version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any /// change in these inputs produces a fresh key and forces a recompile. /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. - internal func key(forInput source: String, libPath: String) async -> String { + internal func key(forInput source: String, libPath: String) -> String { var hasher = ContentHasher() // Schema version: bump to invalidate every existing cache entry at once. hasher.update(data: Data(Self.schemaVersion.utf8)) @@ -58,18 +78,18 @@ // Toolchain version. Different `swift` builds emit different bytes for // the same DSL input. - if let version = await captureSwiftVersion() { - hasher.update(data: Data(version.utf8)) + if let swiftVersion { + hasher.update(data: Data(swiftVersion.utf8)) } // libSyntaxKit stamp. A rebuilt dylib can change the rendered output // even without a Swift-version bump. - if let stamp = FileManager.default.libStamp(libPath: libPath) { + if let stamp = fileManager.libStamp(libPath: libPath) { hasher.update(data: Data(stamp.utf8)) } // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. - let env = ProcessInfo.processInfo.environment + let env = processInfo.environment .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } .sorted { $0.key < $1.key } for (key, value) in env { @@ -89,7 +109,6 @@ internal func store(key: String, data: Data) throws { let cacheRoot = directory(for: key) let final = cacheRoot.appendingPathComponent("output.swift") - let fileManager = FileManager.default // Ensure the parent of the cache key dir exists. The key dir itself is // installed by the atomic rename below. @@ -102,7 +121,7 @@ // into place as a single atomic step. let staging = cacheRoot.deletingLastPathComponent() .appendingPathComponent( - "tmp.\(ProcessInfo.processInfo.processIdentifier).\(UUID().uuidString)" + "tmp.\(processInfo.processIdentifier).\(UUID().uuidString)" ) try fileManager.createDirectory(at: staging, withIntermediateDirectories: true) try data.write(to: staging.appendingPathComponent("output.swift")) diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index e9f14f5..34ffc8f 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -73,7 +73,7 @@ inputPath: String, outputPath: String?, libPath: String, - useCache: Bool, + cache: OutputCache?, timeoutSeconds: Int ) async throws { // Render the input. `processFile` may hit the output cache and skip the @@ -81,7 +81,7 @@ let result = try await processFile( inputPath: inputPath, libPath: libPath, - useCache: useCache, + cache: cache, timeoutSeconds: timeoutSeconds ) // Surface diagnostics from the spawned `swift` before deciding success. @@ -111,7 +111,7 @@ inputDir: String, outputDir: String, libPath: String, - useCache: Bool, + cache: OutputCache?, timeoutSeconds: Int ) async -> Int32 { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL @@ -145,7 +145,7 @@ group.addTask { await runOne( next, libPath: libPath, - useCache: useCache, timeoutSeconds: timeoutSeconds + cache: cache, timeoutSeconds: timeoutSeconds ) } } @@ -156,7 +156,7 @@ group.addTask { await runOne( next, libPath: libPath, - useCache: useCache, timeoutSeconds: timeoutSeconds + cache: cache, timeoutSeconds: timeoutSeconds ) } } @@ -213,14 +213,14 @@ private func runOne( _ input: URL, libPath: String, - useCache: Bool, + cache: OutputCache?, timeoutSeconds: Int ) async -> FileOutcome { do { let result = try await processFile( inputPath: input.path, libPath: libPath, - useCache: useCache, + cache: cache, timeoutSeconds: timeoutSeconds ) return FileOutcome(input: input, result: .success(result)) @@ -267,7 +267,7 @@ private func processFile( inputPath: String, libPath: String, - useCache: Bool, + cache: OutputCache?, timeoutSeconds: Int ) async throws -> ProcessResult { // Load the input source. Anything past this point keys off these bytes. @@ -275,12 +275,11 @@ let absoluteInputPath = inputURL.path let source = try String(contentsOf: inputURL, encoding: .utf8) - // Compute the output cache key (skipped under `--no-cache`, or if the - // cache root can't be derived). Mixes input bytes, toolchain version, - // libSyntaxKit stamp, and sorted SKIT_*/SYNTAXKIT_* env vars — see - // `OutputCache.key(forInput:libPath:)`. - let cache: OutputCache? = useCache ? try? OutputCache() : nil - let cacheKey: String? = await cache?.key(forInput: source, libPath: libPath) + // Compute the output cache key (nil under `--no-cache` or when the cache + // root couldn't be derived at startup). Mixes input bytes, toolchain + // version, libSyntaxKit stamp, and sorted SKIT_*/SYNTAXKIT_* env vars + // — see `OutputCache.key(forInput:libPath:)`. + let cacheKey: String? = cache?.key(forInput: source, libPath: libPath) // Cache hit: skip the wrap+spawn entirely and return the stored output. if let cache, let cacheKey, let cached = cache.lookup(key: cacheKey) { return ProcessResult(exitCode: 0, stdout: cached, stderr: "") diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index dd7f764..c9b427c 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -100,12 +100,19 @@ extension Skit { throw ExitCode(2) } - // 2. Compare the bundle's recorded `swift --version` against the local + // 2. Capture local `swift --version` once. Feeds both the toolchain + // check (compares against the bundle stamp) and the output cache key + // (one shard per toolchain). Doing this here means we spawn + // `swift --version` exactly once per `skit run` invocation rather + // than once per input. + let swiftVersion = await captureSwiftVersion() + + // 3. Compare the bundle's recorded `swift --version` against the local // one. swiftmodules aren't reliably forward-compatible across compiler // versions, so a mismatch produces a clear error rather than letting // the spawned `swift` emit a cryptic module-version diagnostic. if !noToolchainCheck { - switch await ToolchainCheckResult(libPath: libPath) { + switch ToolchainCheckResult(libPath: libPath, swiftVersion: swiftVersion) { case .match, .stampMissing: break case .mismatch(let bundle, let local): @@ -115,7 +122,12 @@ extension Skit { } } - // 3. Stat the input to pick single-file vs. directory mode. Directory + // 4. Build the output cache (nil under `--no-cache` or if the cache + // root can't be derived). The captured `swiftVersion` is bound into + // the instance so per-input key derivation doesn't re-spawn `swift`. + let cache: OutputCache? = noCache ? nil : try? OutputCache(swiftVersion: swiftVersion) + + // 5. Stat the input to pick single-file vs. directory mode. Directory // mode requires an explicit `-o` output dir; single-file mode falls // back to stdout. var isDirectory: ObjCBool = false @@ -127,25 +139,25 @@ extension Skit { guard let output else { throw ValidationError("directory inputs require -o ") } - // 4a. Hand off to the directory orchestrator and surface its exit + // 6a. Hand off to the directory orchestrator and surface its exit // code via ExitCode (so a partial-failure batch returns 1). let exitCode = await runDirectory( inputDir: input, outputDir: output, libPath: libPath, - useCache: !noCache, + cache: cache, timeoutSeconds: timeoutSeconds ) throw ExitCode(exitCode) } else { - // 4b. Hand off to the single-file orchestrator. It calls `exit()` + // 6b. Hand off to the single-file orchestrator. It calls `exit()` // directly on non-zero subprocess exit, so a thrown ExitCode here // would be unreachable in that path. try await runSingleFile( inputPath: input, outputPath: output, libPath: libPath, - useCache: !noCache, + cache: cache, timeoutSeconds: timeoutSeconds ) } diff --git a/Sources/skit/ToolchainCheckResult.swift b/Sources/skit/ToolchainCheckResult.swift index d3a6fa8..fc45380 100644 --- a/Sources/skit/ToolchainCheckResult.swift +++ b/Sources/skit/ToolchainCheckResult.swift @@ -44,12 +44,12 @@ } extension ToolchainCheckResult { - /// Compares `/swift-version.txt` to `captureSwiftVersion()`. - /// The swiftmodule format isn't reliably forward-compatible across even - /// patch-level Swift releases (originating bug: 6.3.0 → 6.3.2 rejected the - /// swiftmodule), so the comparison is exact-string after normalising - /// trailing whitespace. - internal init(libPath: String) async { + /// Compares `/swift-version.txt` to the caller-captured + /// `swiftVersion` string. The swiftmodule format isn't reliably + /// forward-compatible across even patch-level Swift releases (originating + /// bug: 6.3.0 → 6.3.2 rejected the swiftmodule), so the comparison is + /// exact-string after normalising trailing whitespace. + internal init(libPath: String, swiftVersion: String?) { let stampURL = URL(fileURLWithPath: libPath) .appendingPathComponent(Self.toolchainStampFilename) guard let stampData = try? Data(contentsOf: stampURL), @@ -61,7 +61,7 @@ self = .stampMissing return } - guard let localRaw = await captureSwiftVersion() else { + guard let localRaw = swiftVersion else { FileHandle.standardError.write( Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) ) From fb22c25ccb570830bde4e7fc1c38b15c00f1df33 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 8 Jun 2026 13:48:50 -0400 Subject: [PATCH 35/56] Move skit Toolchain globals into types/extensions [skip ci] Remove Toolchain.swift, lifting its two file-scope functions: - captureSwiftVersion() becomes a fileprivate method on Skit.Run - syntaxKitCacheRoot() becomes ProcessInfo.syntaxKitCacheRoot(default:), with OutputCache supplying the platform default via a static defaultCacheRoot constant OutputCache.init no longer throws (its only failure source is gone), so the call site drops its try?. Update globals-audit.md to reflect the removed file and lowered global-function count. Co-Authored-By: Claude Opus 4.8 (1M context) --- Sources/skit/OutputCache.swift | 36 +++++++++++++------ ...t => ProcessInfo+SyntaxKitCacheRoot.swift} | 35 ++++++------------ Sources/skit/Skit+Run.swift | 27 +++++++++++--- globals-audit.md | 14 ++++---- 4 files changed, 64 insertions(+), 48 deletions(-) rename Sources/skit/{Toolchain.swift => ProcessInfo+SyntaxKitCacheRoot.swift} (56%) diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift index 4d13366..ccc4b51 100644 --- a/Sources/skit/OutputCache.swift +++ b/Sources/skit/OutputCache.swift @@ -39,14 +39,27 @@ /// singletons used in production (and the typical test doubles) are /// thread-safe for the operations we invoke. The single instance is /// shared across concurrent `runOne` tasks in directory mode. - internal struct OutputCache: @unchecked Sendable { + internal struct OutputCache: Sendable { /// Bumped when the cache layout changes in a way that requires invalidation. private static let schemaVersion = "v1" + /// Home-relative cache root used when `XDG_CACHE_HOME` is unset: macOS + /// `~/Library/Caches/...`, else Linux `~/.cache/syntaxkit`. The home dir is + /// fixed for the process lifetime, so this is computed once. + private static let defaultCacheRoot: URL = { + let home = NSHomeDirectory() + #if os(macOS) + return URL(fileURLWithPath: home) + .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") + #else + return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") + #endif + }() + /// `/outputs/`. Populated once at init; per-key /// directories are derived from it on demand. private let root: URL - private let fileManager: FileManager + private let fileManager: @Sendable () -> FileManager private let processInfo: ProcessInfo /// Verbatim `swift --version` output captured once for the lifetime of @@ -56,10 +69,11 @@ internal init( swiftVersion: String?, - fileManager: FileManager = .default, + fileManager: @autoclosure @escaping @Sendable () -> FileManager = .default, processInfo: ProcessInfo = .processInfo - ) throws { - self.root = try syntaxKitCacheRoot().appendingPathComponent("outputs") + ) { + self.root = processInfo.syntaxKitCacheRoot(default: Self.defaultCacheRoot) + .appendingPathComponent("outputs") self.swiftVersion = swiftVersion self.fileManager = fileManager self.processInfo = processInfo @@ -83,7 +97,7 @@ } // libSyntaxKit stamp. A rebuilt dylib can change the rendered output // even without a Swift-version bump. - if let stamp = fileManager.libStamp(libPath: libPath) { + if let stamp = fileManager().libStamp(libPath: libPath) { hasher.update(data: Data(stamp.utf8)) } @@ -112,7 +126,7 @@ // Ensure the parent of the cache key dir exists. The key dir itself is // installed by the atomic rename below. - try fileManager.createDirectory( + try fileManager().createDirectory( at: cacheRoot.deletingLastPathComponent(), withIntermediateDirectories: true ) @@ -123,17 +137,17 @@ .appendingPathComponent( "tmp.\(processInfo.processIdentifier).\(UUID().uuidString)" ) - try fileManager.createDirectory(at: staging, withIntermediateDirectories: true) + try fileManager().createDirectory(at: staging, withIntermediateDirectories: true) try data.write(to: staging.appendingPathComponent("output.swift")) // Atomic rename into the cache path. If a peer already populated this // key, swallow the rename error and drop our staging copy. Re-throw only // if the destination is still missing afterwards. do { - try fileManager.moveItem(at: staging, to: cacheRoot) + try fileManager().moveItem(at: staging, to: cacheRoot) } catch { - try? fileManager.removeItem(at: staging) - if !fileManager.fileExists(atPath: final.path) { + try? fileManager().removeItem(at: staging) + if !fileManager().fileExists(atPath: final.path) { throw error } } diff --git a/Sources/skit/Toolchain.swift b/Sources/skit/ProcessInfo+SyntaxKitCacheRoot.swift similarity index 56% rename from Sources/skit/Toolchain.swift rename to Sources/skit/ProcessInfo+SyntaxKitCacheRoot.swift index de133c8..230880f 100644 --- a/Sources/skit/Toolchain.swift +++ b/Sources/skit/ProcessInfo+SyntaxKitCacheRoot.swift @@ -1,5 +1,5 @@ // -// Toolchain.swift +// ProcessInfo+SyntaxKitCacheRoot.swift // SyntaxKit // // Created by Leo Dion. @@ -30,32 +30,17 @@ #if canImport(Subprocess) import Foundation - import Subprocess - /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. - internal func captureSwiftVersion() async -> String? { - let result = try? await run( - .name("swift"), - arguments: ["--version"], - output: .string(limit: 4_096), - error: .discarded - ) - return result?.standardOutput - } - - /// Root for all skit caches. Honours `XDG_CACHE_HOME`, else macOS - /// `~/Library/Caches/...` or Linux `~/.cache/syntaxkit`. - internal func syntaxKitCacheRoot() throws -> URL { - if let xdg = ProcessInfo.processInfo.environment["XDG_CACHE_HOME"], !xdg.isEmpty { - return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + extension ProcessInfo { + /// Root for all skit caches: `/syntaxkit` when that env + /// var is set and non-empty, otherwise `defaultRoot` (typically the + /// platform's home-relative cache dir). + internal func syntaxKitCacheRoot(default defaultRoot: URL) -> URL { + if let xdg = environment["XDG_CACHE_HOME"], !xdg.isEmpty { + return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + } + return defaultRoot } - let home = NSHomeDirectory() - #if os(macOS) - return URL(fileURLWithPath: home) - .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") - #else - return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") - #endif } #endif diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index c9b427c..f6fe89e 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -122,10 +122,10 @@ extension Skit { } } - // 4. Build the output cache (nil under `--no-cache` or if the cache - // root can't be derived). The captured `swiftVersion` is bound into - // the instance so per-input key derivation doesn't re-spawn `swift`. - let cache: OutputCache? = noCache ? nil : try? OutputCache(swiftVersion: swiftVersion) + // 4. Build the output cache (nil under `--no-cache`). The captured + // `swiftVersion` is bound into the instance so per-input key derivation + // doesn't re-spawn `swift`. + let cache: OutputCache? = noCache ? nil : OutputCache(swiftVersion: swiftVersion) // 5. Stat the input to pick single-file vs. directory mode. Directory // mode requires an explicit `-o` output dir; single-file mode falls @@ -172,3 +172,22 @@ extension Skit { } } } + +#if canImport(Subprocess) + + import Subprocess + + extension Skit.Run { + /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. + fileprivate func captureSwiftVersion() async -> String? { + let result = try? await Subprocess.run( + .name("swift"), + arguments: ["--version"], + output: .string(limit: 4_096), + error: .discarded + ) + return result?.standardOutput + } + } + +#endif diff --git a/globals-audit.md b/globals-audit.md index d9aab19..03f918d 100644 --- a/globals-audit.md +++ b/globals-audit.md @@ -34,11 +34,9 @@ All inside `#if canImport(Subprocess)`. ## Sources/skit/Toolchain.swift -All inside `#if canImport(Subprocess)`. - -### Functions -- [ ] L36 — `internal func captureSwiftVersion() async -> String?` -- [ ] L48 — `internal func syntaxKitCacheRoot() throws -> URL` +Resolved — file removed. Both globals were lifted into types/extensions: +- [x] `captureSwiftVersion()` → `fileprivate` method on `Skit.Run` in `Skit+Run.swift`. +- [x] `syntaxKitCacheRoot()` → `ProcessInfo.syntaxKitCacheRoot(default:)` in `ProcessInfo+SyntaxKitCacheRoot.swift`; the platform default is now a `private static let defaultCacheRoot` on `OutputCache`. --- @@ -54,9 +52,9 @@ All inside `#if canImport(Subprocess)`. | Target | Global funcs | Global vars | |---|---:|---:| | skit / Runner.swift | 9 | 3 | -| skit / Toolchain.swift | 2 | 0 | +| skit / Toolchain.swift (removed) | 0 | 0 | | skit / OutputCache.swift | 0 | 0 | | DocumentationHarness / Validator.swift | 0 | 1 | -| **Total** | **11** | **4** | +| **Total** | **9** | **4** | -All `skit` globals are free functions/constants inside `#if canImport(Subprocess)` — a deliberate CLI style. `privateDefaultPathExtensions` is the lone non-skit global (a linter reverted an earlier attempt to nest it). +The remaining `skit` globals are free functions/constants inside `#if canImport(Subprocess)` in `Runner.swift` — a deliberate CLI style. `privateDefaultPathExtensions` is the lone non-skit global (a linter reverted an earlier attempt to nest it). From 083058a75dfd50b755e98124dd3e1769c54b8dab Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 8 Jun 2026 13:56:16 -0400 Subject: [PATCH 36/56] Move toolchainMismatchMessage onto Skit.Run [skip ci] Lift toolchainMismatchMessage out of Runner.swift into a fileprivate method on Skit.Run (alongside captureSwiftVersion), its only caller. Update globals-audit.md: Runner.swift drops to 8 global functions. Co-Authored-By: Claude Opus 4.8 (1M context) --- Sources/DocumentationHarness/Validator.swift | 13 +++++++------ Sources/skit/Runner.swift | 19 ------------------- Sources/skit/Skit+Run.swift | 19 +++++++++++++++++++ globals-audit.md | 6 +++--- 4 files changed, 29 insertions(+), 28 deletions(-) diff --git a/Sources/DocumentationHarness/Validator.swift b/Sources/DocumentationHarness/Validator.swift index 8585304..6fc7506 100644 --- a/Sources/DocumentationHarness/Validator.swift +++ b/Sources/DocumentationHarness/Validator.swift @@ -33,12 +33,13 @@ package protocol Validator { func validateFile(at fileURL: URL) throws -> [ValidationResult] } -private let privateDefaultPathExtensions = ["md"] -extension Validator { +private extension [String] { /// Default file extensions for documentation files - package static var defaultPathExtensions: [String] { - privateDefaultPathExtensions - } + static let defaultPathExtensions: [String] = ["md"] + +} + +extension Validator { /// Validates all Swift code examples found in documentation files /// - Parameters: @@ -51,7 +52,7 @@ extension Validator { package func validate( relativePaths: [String], atProjectRoot projectRoot: URL, - withPathExtensions pathExtensions: [String] = Self.defaultPathExtensions, + withPathExtensions pathExtensions: [String] = .defaultPathExtensions, using fileSearcher: any FileSearcher = FileManager.default ) throws -> [ValidationResult] { let documentationFiles = try relativePaths.flatMap { docPath in diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index 34ffc8f..95901dd 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -43,25 +43,6 @@ // 6. runSwift — spawn `swift` with timeout watchdog // See Docs/skit.md for design rationale and trade-offs. - // MARK: - Toolchain check - - internal func toolchainMismatchMessage(bundle: String, local: String) -> String { - """ - skit: toolchain mismatch - bundle: \(bundle) - local: \(local) - The bundle's libSyntaxKit was built against a different `swift` than the - one on your PATH. Swift swiftmodules aren't reliably compatible across - versions, so spawning `swift` would fail with a cryptic module-version - diagnostic. - - Rebuild the bundle with: - Scripts/build-skit-release.sh - Or pass --no-toolchain-check to try anyway. - - """ - } - // MARK: - Single-file mode /// Runs `processFile` on a single input and writes its rendered Swift to diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index f6fe89e..eb91fd6 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -188,6 +188,25 @@ extension Skit { ) return result?.standardOutput } + + /// Human-readable error emitted when the bundle's recorded `swift --version` + /// differs from the local one, explaining why and how to recover. + fileprivate func toolchainMismatchMessage(bundle: String, local: String) -> String { + """ + skit: toolchain mismatch + bundle: \(bundle) + local: \(local) + The bundle's libSyntaxKit was built against a different `swift` than the + one on your PATH. Swift swiftmodules aren't reliably compatible across + versions, so spawning `swift` would fail with a cryptic module-version + diagnostic. + + Rebuild the bundle with: + Scripts/build-skit-release.sh + Or pass --no-toolchain-check to try anyway. + + """ + } } #endif diff --git a/globals-audit.md b/globals-audit.md index 03f918d..aed4aae 100644 --- a/globals-audit.md +++ b/globals-audit.md @@ -15,7 +15,7 @@ Generated: 2026-06-08. Branch: `research/swift-manifest-codegen`. All inside `#if canImport(Subprocess)`. ### Functions -- [ ] L48 — `internal func toolchainMismatchMessage(bundle: String, local: String) -> String` +- [x] `toolchainMismatchMessage(bundle:local:)` → `fileprivate` method on `Skit.Run` in `Skit+Run.swift`. - [ ] L72 — `internal func runSingleFile(inputPath:outputPath:libPath:useCache:timeoutSeconds:) async throws` - [ ] L110 — `internal func runDirectory(inputDir:outputDir:libPath:useCache:timeoutSeconds:) async -> Int32` - [ ] L213 — `private func runOne(_ input: URL, libPath:useCache:timeoutSeconds:) async -> FileOutcome` @@ -51,10 +51,10 @@ Resolved — file removed. Both globals were lifted into types/extensions: | Target | Global funcs | Global vars | |---|---:|---:| -| skit / Runner.swift | 9 | 3 | +| skit / Runner.swift | 8 | 3 | | skit / Toolchain.swift (removed) | 0 | 0 | | skit / OutputCache.swift | 0 | 0 | | DocumentationHarness / Validator.swift | 0 | 1 | -| **Total** | **9** | **4** | +| **Total** | **8** | **4** | The remaining `skit` globals are free functions/constants inside `#if canImport(Subprocess)` in `Runner.swift` — a deliberate CLI style. `privateDefaultPathExtensions` is the lone non-skit global (a linter reverted an earlier attempt to nest it). From a6bb156cb9059b791eb4d1cb29265c1b77cc0148 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 8 Jun 2026 14:33:39 -0400 Subject: [PATCH 37/56] Wrap skit Runner free functions in a Runner type [skip ci] Lift the eight free functions and three constants in Runner.swift into an internal `struct Runner: Sendable` holding the per-invocation config (libPath, cache, timeoutSeconds). Its sole internal entry point is `callAsFunction(input:output:)` (invoked as `runner(input:output:)`), which classifies the input via a new `RunInput` enum (single file vs. directory) and dispatches to the now-private runSingleFile/runDirectory. This empties the last cluster of skit file-scope globals; only DocumentationHarness's privateDefaultPathExtensions remains. Update globals-audit.md and Docs/skit-internals.md accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) --- Docs/skit-internals.md | 67 ++-- Sources/skit/RunInput.swift | 64 +++ Sources/skit/Runner.swift | 769 ++++++++++++++++++------------------ Sources/skit/Skit+Run.swift | 42 +- globals-audit.md | 33 +- 5 files changed, 500 insertions(+), 475 deletions(-) create mode 100644 Sources/skit/RunInput.swift diff --git a/Docs/skit-internals.md b/Docs/skit-internals.md index 84a4bd3..d524cfe 100644 --- a/Docs/skit-internals.md +++ b/Docs/skit-internals.md @@ -1,6 +1,6 @@ -# `skit` internals: Runner, OutputCache, Toolchain +# `skit` internals: Runner, OutputCache, toolchain utilities -A contributor's map of the three modules that do the actual work inside `skit run`. For the **why** (design rationale, trade-offs, sharp edges) see [`Docs/skit.md`](skit.md). For the **what** (flags, behaviour) see [`Sources/skit/README.md`](../Sources/skit/README.md). This doc covers the **how** — what each module needs, what it produces, and how they call each other. +A contributor's map of the modules that do the actual work inside `skit run`. For the **why** (design rationale, trade-offs, sharp edges) see [`Docs/skit.md`](skit.md). For the **what** (flags, behaviour) see [`Sources/skit/README.md`](../Sources/skit/README.md). This doc covers the **how** — what each module needs, what it produces, and how they call each other. ## The pipeline @@ -8,50 +8,52 @@ A contributor's map of the three modules that do the actual work inside `skit ru 1. **`Bundle.main.resolveLibPath(candidates:)`** (`Sources/skit/Bundle+ResolveLibPath.swift`) — locate the lib bundle (`libSyntaxKit.dylib` + swiftmodules). The caller passes `--lib` and `$SKIT_LIB_DIR` as candidates; Bundle falls back to `/lib/` and `/../lib/skit/`. 2. **`ToolchainCheckResult(libPath:)`** (`Sources/skit/ToolchainCheckResult.swift`) — verify the bundle's recorded `swift --version` matches the local one. Refuses to spawn on mismatch. -3. **`runSingleFile(...)` or `runDirectory(...)`** — dispatch into Runner. Everything downstream (wrap, spawn, output-cache lookup/store) happens inside Runner. +3. **`Runner(libPath:cache:timeoutSeconds:)`** then **`runner(input:output:)`** — bind the per-invocation configuration into a `Runner` value and call it (via `callAsFunction`) with the raw input/output paths. The runner classifies single-file vs. directory mode and everything downstream (wrap, spawn, output-cache lookup/store) happens inside `Runner`. -**Toolchain** is consumed transitively (by `Bundle+ResolveLibPath`, `FileManager+IsLibDir`, and `OutputCache`) — it's never called directly from the CLI driver. **OutputCache** is an implementation detail of `Runner.processFile`, never called from the CLI driver either. The CLI driver only sees two surfaces: `Bundle.main.resolveLibPath` and `runSingleFile`/`runDirectory`. +The **toolchain utilities** are consumed transitively (by `Bundle+ResolveLibPath`, `FileManager+IsLibDir`, and `OutputCache`) — they're never called directly from the CLI driver. **OutputCache** is an implementation detail of `Runner.processFile`, never called from the CLI driver either. The CLI driver only sees two surfaces: `Bundle.main.resolveLibPath` and calling the `Runner` value itself (`runner(input:output:)`). -## Toolchain (`Toolchain.swift` + extensions) +## Toolchain utilities (no single file) -**Purpose.** Provide the shared utilities that describe (a) the local `swift` toolchain, (b) the bundled SyntaxKit dylib, and (c) where skit's caches live on disk. These are infrastructure, not features — they exist so the rest of the code doesn't have to re-derive them. +**Purpose.** Provide the shared utilities that describe (a) the local `swift` toolchain, (b) the bundled SyntaxKit dylib, and (c) where skit's caches live on disk. These are infrastructure, not features — they exist so the rest of the code doesn't have to re-derive them. There is no longer a `Toolchain.swift`: each utility was lifted onto the type it naturally belongs to (no file-scope globals, per the project convention). -**Surface area** (all `internal`, all inside `#if canImport(Subprocess)`): +**Surface area** (all inside `#if canImport(Subprocess)`): | Symbol | Defined in | What it returns | Consumers | |---|---|---|---| | `String.dylibFilename` | `String+DylibFilename.swift` | `"lib.dylib"` on macOS, `"lib.so"` on Linux | `FileManager.isLibDir`, `FileManager.libStamp` | | `FileManager.libStamp(libPath:)` | `FileManager+LibStamp.swift` | `"/"` of `libSyntaxKit.{dylib,so}`, or nil | `OutputCache.key` | -| `captureSwiftVersion()` | `Toolchain.swift` | verbatim `swift --version` stdout (≤4 KiB), or nil on spawn failure | `Skit.Run.run` (called once per invocation, then threaded into `ToolchainCheckResult.init` and `OutputCache.init`) | -| `syntaxKitCacheRoot()` | `Toolchain.swift` | `~/Library/Caches/com.brightdigit.SyntaxKit` (macOS), `$XDG_CACHE_HOME/syntaxkit` or `~/.cache/syntaxkit` (Linux) | `OutputCache.init` | +| `Skit.Run.captureSwiftVersion()` | `Skit+Run.swift` (`fileprivate`) | verbatim `swift --version` stdout (≤4 KiB), or nil on spawn failure | `Skit.Run.run` (called once per invocation, then threaded into `ToolchainCheckResult.init` and `OutputCache.init`) | +| `ProcessInfo.syntaxKitCacheRoot(default:)` | `ProcessInfo+SyntaxKitCacheRoot.swift` | `$XDG_CACHE_HOME/syntaxkit` when that env var is set, else the passed-in `default` | `OutputCache.init` | -**Why these four belong together.** They all answer the same question from different angles: "what state of the world does a cache key depend on?" The Swift toolchain (`captureSwiftVersion`), the runtime dylib (`libStamp` via `String.dylibFilename`), and the on-disk cache layout (`syntaxKitCacheRoot`). Together they give `OutputCache` everything it needs to compute a stable, sound key. The two that fit naturally as instance APIs (`String.dylibFilename`, `FileManager.libStamp`) live in their own extension files; the rest stay as free functions in `Toolchain.swift`. +`OutputCache` supplies that `default` via its own `private static let defaultCacheRoot` — `~/Library/Caches/com.brightdigit.SyntaxKit` (macOS) or `~/.cache/syntaxkit` (Linux), computed once since the home dir is fixed for the process lifetime. + +**Why these belong together.** They all answer the same question from different angles: "what state of the world does a cache key depend on?" The Swift toolchain (`captureSwiftVersion`), the runtime dylib (`libStamp` via `String.dylibFilename`), and the on-disk cache layout (`syntaxKitCacheRoot`). Together they give `OutputCache` everything it needs to compute a stable, sound key. Each now lives as a method/extension on the type that owns its data: the dylib helpers on `String`/`FileManager`, the cache-root resolver on `ProcessInfo`, and the `swift --version` spawn on `Skit.Run` (its sole caller). ## Runner (`Runner.swift`) **Purpose.** Orchestrate per-input render: pick single-file vs. directory mode, wrap each input into a complete Swift program, spawn `swift` on it with a timeout watchdog, surface output, and consult the output cache. -**What it needs from the caller:** +**Shape.** `internal struct Runner: Sendable`, constructed once per invocation in `Skit.Run.run` and (being `Sendable`) shared across the concurrent `runOne` tasks in directory mode. It holds the per-invocation configuration as stored properties, so the individual inputs don't re-thread it: - `libPath: String` — the lib bundle dir from step 1; reused for the `swift` invocation's link/rpath flags. -- `useCache: Bool` — gates OutputCache lookup/store (`--no-cache` sets this to false). +- `cache: OutputCache?` — the shared output cache, or nil under `--no-cache`; gates lookup/store. - `timeoutSeconds: Int` — per-input watchdog (`0` opts out; default 60s; on expiry, exit 124 matching POSIX `timeout(1)`). -**Two entry points:** +**Single entry point.** `Runner` is callable: `callAsFunction(input:output:)` is the only `internal` method (besides `init`), so `Skit.Run.run` constructs a `Runner` and invokes it directly — `try await runner(input:, output:)` — with the raw `--input`/`-o` strings. It resolves the input via `RunInput.resolve(input:output:)` (`RunInput.swift`) — a two-case enum (`.singleFile` / `.directory`) that stats the path and enforces existence + the directory `-o` requirement, throwing `ValidationError` otherwise — then dispatches to one of two **private** mode methods: -| Function | When | Failure semantics | +| Method (private) | When | Failure semantics | |---|---|---| -| `runSingleFile(inputPath:outputPath:libPath:useCache:timeoutSeconds:)` (`Runner.swift:72`) | One input, one output (or stdout) | Calls `exit()` on non-zero subprocess result — caller won't see a thrown error in that path | -| `runDirectory(inputDir:outputDir:libPath:useCache:timeoutSeconds:)` (`Runner.swift:110`) | Walks `**/*.swift` under `inputDir`, bounded concurrency = `ProcessInfo.activeProcessorCount`, mirrors output into `outputDir` | Returns 0/1 instead of `exit()` — partial failure allowed, successful peers still written, one-line summary printed to stderr | +| `runSingleFile(inputPath:outputPath:)` | One input, one output (or stdout) | Calls `exit()` on non-zero subprocess result — caller won't see a thrown error in that path | +| `runDirectory(inputDir:outputDir:)` | Walks `**/*.swift` under `inputDir`, bounded concurrency = `ProcessInfo.activeProcessorCount`, mirrors output into `outputDir` | Returns 0/1, which `run` re-throws as `ExitCode` — partial failure allowed, successful peers still written, one-line summary printed to stderr | -**The per-input work** (`processFile`, `Runner.swift:267`): +**The per-input work** (`processFile(inputPath:)`, a `private` instance method): 1. Load source bytes. -2. If `useCache`, compute the output cache key via `outputCacheKey(inputSource:libPath:)` and try a hit. -3. On miss: `wrap` the source (hoist imports, splice body into `Group { … }`, fence with `#sourceLocation` so compiler diagnostics map back to the original file), write to a per-invocation `skit-/` temp dir, spawn `swift`, rewrite stderr to swap the temp path for the original path, store the result via `storeCachedOutput`, return. +2. If `cache` is non-nil, compute the output cache key via `cache.key(forInput:libPath:)` and try a hit. +3. On miss: `wrap` the source (hoist imports, splice body into `Group { … }`, fence with `#sourceLocation` so compiler diagnostics map back to the original file), write to a per-invocation `skit-/` temp dir, spawn `swift`, rewrite stderr to swap the temp path for the original path, store the result via `cache.store(key:data:)`, return. -The temp dir is cleaned with `defer { try? removeItem }` so a failed spawn doesn't leak files. +The temp dir is cleaned with `defer { try? removeItem }` so a failed spawn doesn't leak files. `wrap`, `collectInputs`, and `exitCode` are pure `private static` helpers (no dependency on the stored configuration). -**The spawn** (`runSwift`, `Runner.swift:417`). Fixed argument list: +**The spawn** (`runSwift(wrappedPath:)`). Fixed argument list: ``` swift -suppress-warnings -I -L -lSyntaxKit @@ -67,9 +69,9 @@ Raced against a sleep watchdog in a throwing task group; the loser is cancelled. **Purpose.** Cache the rendered Swift (the stdout of the spawned `swift` for a given input) so re-running with unchanged inputs avoids the spawn entirely. Hit cost ≈ 0.14s on macOS vs. ~0.5s for a cold script-mode `swift` spawn. -**Shape.** Single `internal struct OutputCache: @unchecked Sendable`, built once per `skit run` invocation in `Skit.Run.run` and shared across every input. `init(swiftVersion:fileManager:processInfo:)` throws — it resolves the cache root via `Toolchain.syntaxKitCacheRoot()`, binds it to `self.root` (`/outputs/`), and stores `swiftVersion` so key derivation never has to re-spawn `swift`. The caller wraps the init in `try?` so a non-derivable cache root short-circuits silently (same effective behaviour as `--no-cache` for that invocation). +**Shape.** Single `internal struct OutputCache: Sendable`, built once per `skit run` invocation in `Skit.Run.run` and shared across every input. `init(swiftVersion:fileManager:processInfo:)` is non-throwing — it resolves the cache root via `processInfo.syntaxKitCacheRoot(default: Self.defaultCacheRoot)`, binds it to `self.root` (`/outputs/`), and stores `swiftVersion` so key derivation never has to re-spawn `swift`. The caller constructs it directly (`noCache ? nil : OutputCache(swiftVersion:)`); `--no-cache` is the only path that yields a nil cache. -The `@unchecked Sendable` conformance covers the `FileManager` / `ProcessInfo` stored properties (reference types that don't auto-derive Sendable, but the singletons we use are thread-safe for these operations). Runner shares one `OutputCache?` instance across concurrent `runOne` tasks in directory mode. +Sendability: the `FileManager` is injected as a `@Sendable () -> FileManager` closure (default `.default`) rather than stored directly, so the struct derives `Sendable` without an `@unchecked` escape hatch. Runner shares one `OutputCache?` instance across concurrent `runOne` tasks in directory mode. **Surface** (all `internal` instance methods): @@ -84,7 +86,7 @@ The `@unchecked Sendable` conformance covers the `FileManager` / `ProcessInfo` s Storage is wrapped in `try?` at the caller in `processFile` — a cache *write* failure is never a render failure; the next run just re-spawns. -**On-disk layout.** `/outputs//output.swift`. The per-key directory is derived by the private `directory(for:)` helper; the root comes from `Toolchain.syntaxKitCacheRoot()` once, at init. +**On-disk layout.** `/outputs//output.swift`. The per-key directory is derived by the private `directory(for:)` helper; the root comes from `processInfo.syntaxKitCacheRoot(default:)` once, at init. **Why FNV-1a, not SHA-256.** The cache keys aren't security-critical — there's no adversary trying to forge a collision — so `ContentHasher` uses a 64-bit FNV-1a. Deterministic across processes and platforms (unlike Swift's stdlib `Hasher`, whose seed is per-process randomized), which is what makes the keys usable as on-disk directory names. @@ -94,20 +96,21 @@ Storage is wrapped in `try?` at the caller in `processFile` — a cache *write* Skit.Run.run() ├─ Bundle.main.resolveLibPath(candidates: --lib, $SKIT_LIB_DIR) → libPath │ └─ FileManager.default.isLibDir → "SyntaxKit".dylibFilename - ├─ captureSwiftVersion() → swiftVersion (spawned exactly once) + ├─ self.captureSwiftVersion() → swiftVersion (spawned exactly once) ├─ ToolchainCheckResult(libPath:, swiftVersion:) → gate (compare to bundle stamp) - ├─ try? OutputCache(swiftVersion:) → cache (nil under --no-cache or unresolvable root) - └─ runSingleFile / runDirectory(libPath, cache, timeoutSeconds) - └─ processFile(input) + ├─ OutputCache(swiftVersion:) → cache (nil only under --no-cache) + ├─ Runner(libPath:, cache:, timeoutSeconds:) → runner (holds the config) + └─ runner(input:, output:) → callAsFunction → RunInput.resolve → single-file / directory + └─ processFile(inputPath:) ├─ cache?.key(forInput: source, libPath:) ← self.swiftVersion + FileManager.default.libStamp ├─ cache.lookup(key:) ← hit returns immediately - ├─ wrap(source) → temp wrapper.swift - ├─ runSwift(wrappedPath, libPath) ← spawns `swift` linked against libSyntaxKit + ├─ Self.wrap(source) → temp wrapper.swift + ├─ runSwift(wrappedPath:) ← spawns `swift` linked against libSyntaxKit └─ try? cache.store(key:, data:) ← on the way out (atomic stage+rename) ``` Three coupling facts worth remembering: -- **Toolchain is shared infrastructure.** `OutputCache` depends on three of its four utilities; the bundle-resolution code depends on `dylibFilename`. None of those callers know or care that the utilities live in `Toolchain.swift` — they just call free functions. -- **Runner is the only place OutputCache is touched.** If you ever want a non-CLI consumer of skit (e.g. a long-running server), you'd either call `processFile` directly or replicate its cache logic. Don't instantiate `OutputCache` from elsewhere. +- **The toolchain utilities are shared infrastructure.** `OutputCache` depends on three of the four; the bundle-resolution code depends on `dylibFilename`. They no longer share a `Toolchain.swift` — each lives as a method/extension on the type that owns its data, called like any other instance API. +- **Runner is the only place OutputCache is touched.** If you ever want a non-CLI consumer of skit (e.g. a long-running server), you'd construct a `Runner` and call its entry points (or replicate its cache logic). Don't instantiate `OutputCache` from elsewhere. - **Cache safety under concurrency.** Both the store path (in `OutputCache`) and the bundle-resolution path are designed to be safe under concurrent invocations: write into `tmp../`, then `moveItem` to install. Race-loser swallows the rename error if the destination is now populated by a peer. diff --git a/Sources/skit/RunInput.swift b/Sources/skit/RunInput.swift new file mode 100644 index 0000000..64d1465 --- /dev/null +++ b/Sources/skit/RunInput.swift @@ -0,0 +1,64 @@ +// +// RunInput.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import ArgumentParser + import Foundation + + /// Whether a `skit run` input path resolves to a single `.swift` file or a + /// directory of them — the two modes `Runner` dispatches into. Built by + /// `resolve(input:output:)`, which stats the path and enforces the + /// per-mode output rules so `Skit.Run.run` can `switch` on a settled value + /// instead of juggling an `ObjCBool`. + internal enum RunInput { + /// A single input file rendered to `outputPath`, or stdout when nil. + case singleFile(inputPath: String, outputPath: String?) + /// A directory of inputs mirrored into `outputDir` (always required). + case directory(inputDir: String, outputDir: String) + + /// Classifies `input` by stat: existing directory → `.directory`, + /// existing file → `.singleFile`. Throws a `ValidationError` if the path + /// doesn't exist, or if a directory input wasn't given an explicit `-o`. + internal static func resolve(input: String, output: String?) throws -> RunInput { + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + throw ValidationError("input does not exist: \(input)") + } + if isDirectory.boolValue { + guard let output else { + throw ValidationError("directory inputs require -o ") + } + return .directory(inputDir: input, outputDir: output) + } + return .singleFile(inputPath: input, outputPath: output) + } + } + +#endif diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index 95901dd..59b7274 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -29,6 +29,7 @@ #if canImport(Subprocess) + import ArgumentParser import Foundation import Subprocess import SwiftParser @@ -37,440 +38,430 @@ // Run lifecycle (per `skit run` invocation): // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) // 2. ToolchainCheckResult.init — compare bundle stamp to `swift --version` - // 3. runSingleFile / runDirectory — dispatch to single- or batch-input mode + // 3. Runner.runSingleFile / .runDirectory — single- or batch-input mode // 4. processFile (per input) — load → cache lookup → wrap → spawn → cache store // 5. wrap — hoist imports, wrap body in Group { … }, #sourceLocation // 6. runSwift — spawn `swift` with timeout watchdog // See Docs/skit.md for design rationale and trade-offs. - // MARK: - Single-file mode - - /// Runs `processFile` on a single input and writes its rendered Swift to - /// `outputPath` (or stdout when nil). Any stderr from the spawned `swift` - /// is surfaced verbatim. On a non-zero subprocess exit, calls `exit()` - /// directly — the caller in `Skit.Run.run()` won't see a thrown error in - /// that path. - internal func runSingleFile( - inputPath: String, - outputPath: String?, - libPath: String, - cache: OutputCache?, - timeoutSeconds: Int - ) async throws { - // Render the input. `processFile` may hit the output cache and skip the - // spawn entirely; either way the result has the same shape. - let result = try await processFile( - inputPath: inputPath, - libPath: libPath, - cache: cache, - timeoutSeconds: timeoutSeconds - ) - // Surface diagnostics from the spawned `swift` before deciding success. - if !result.stderr.isEmpty { - FileHandle.standardError.write(Data(result.stderr.utf8)) + /// Renders SyntaxKit DSL inputs into Swift source, holding the per-invocation + /// configuration (`libPath`, `cache`, `timeoutSeconds`) so the individual + /// inputs don't have to thread it through every call. Constructed once per + /// `skit run` in `Skit.Run.run`; `Sendable` so a single value can be shared + /// across the concurrent `runOne` tasks in directory mode. + internal struct Runner: Sendable { + /// Directory holding `libSyntaxKit.{dylib,so}` + swiftmodules; reused for + /// the spawned `swift`'s `-I`/`-L`/`-rpath` flags. + private let libPath: String + /// Output cache shared across every input, or nil under `--no-cache`. + private let cache: OutputCache? + /// Per-input watchdog in seconds; `0` opts out of the timeout race. + private let timeoutSeconds: Int + + internal init(libPath: String, cache: OutputCache?, timeoutSeconds: Int) { + self.libPath = libPath + self.cache = cache + self.timeoutSeconds = timeoutSeconds } - // Non-zero subprocess exit propagates as a process exit. We don't write - // partial output in that case. - guard result.exitCode == 0 else { - exit(result.exitCode) - } - // Deliver the rendered output to file or stdout. - if let outputPath { - try result.stdout.write(to: URL(fileURLWithPath: outputPath)) - } else { - FileHandle.standardOutput.write(result.stdout) - } - } - // MARK: - Folder mode - - /// Walks `inputDir` for `.swift` inputs, processes them concurrently (up to - /// the active core count), and mirrors the rendered output into `outputDir`. - /// A failure on one input does not abort the batch — successful peers are - /// still written. Returns 0 if every input succeeded, 1 otherwise. - internal func runDirectory( - inputDir: String, - outputDir: String, - libPath: String, - cache: OutputCache?, - timeoutSeconds: Int - ) async -> Int32 { - let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL - let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL - - // Phase 1: enumerate inputs. - let inputs: [URL] - do { - inputs = try collectInputs(at: inputURL) - } catch { - FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) - return 1 + // MARK: - Dispatch + + + /// Classifies `input` (single file vs. directory) and renders it. Directory + /// mode surfaces its batch exit code via `ExitCode` (so a partial-failure + /// batch returns 1); single-file mode renders to file/stdout and may call + /// `exit()` directly on a non-zero subprocess result. Throws a + /// `ValidationError` if the path doesn't exist, or if a directory input + /// wasn't given an explicit `-o`. + func callAsFunction(input: String, output: String?) async throws { + switch try RunInput.resolve(input: input, output: output) { + case .directory(let inputDir, let outputDir): + let exitCode = await runDirectory(inputDir: inputDir, outputDir: outputDir) + throw ExitCode(exitCode) + case .singleFile(let inputPath, let outputPath): + try await runSingleFile(inputPath: inputPath, outputPath: outputPath) + } } - if inputs.isEmpty { - FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) - return 0 + // MARK: - Single-file mode + + /// Runs `processFile` on a single input and writes its rendered Swift to + /// `outputPath` (or stdout when nil). Any stderr from the spawned `swift` + /// is surfaced verbatim. On a non-zero subprocess exit, calls `exit()` + /// directly — the caller in `Skit.Run.run()` won't see a thrown error in + /// that path. + private func runSingleFile(inputPath: String, outputPath: String?) async throws { + // Render the input. `processFile` may hit the output cache and skip the + // spawn entirely; either way the result has the same shape. + let result = try await processFile(inputPath: inputPath) + // Surface diagnostics from the spawned `swift` before deciding success. + if !result.stderr.isEmpty { + FileHandle.standardError.write(Data(result.stderr.utf8)) + } + // Non-zero subprocess exit propagates as a process exit. We don't write + // partial output in that case. + guard result.exitCode == 0 else { + exit(result.exitCode) + } + // Deliver the rendered output to file or stdout. + if let outputPath { + try result.stdout.write(to: URL(fileURLWithPath: outputPath)) + } else { + FileHandle.standardOutput.write(result.stdout) + } } - // Phase 2: bounded-concurrency processing. Cap is the active core count - // so a 200-file batch doesn't fork 200 simultaneous `swift` processes. - let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) - - var outcomes: [FileOutcome] = [] - var iterator = inputs.makeIterator() - - await withTaskGroup(of: FileOutcome.self) { group in - // Seed the group up to the concurrency cap… - for _ in 0.. Int32 { + let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL + let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL + + // Phase 1: enumerate inputs. + let inputs: [URL] + do { + inputs = try Self.collectInputs(at: inputURL) + } catch { + FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) + return 1 } - // …then refill one task for every completion until inputs are exhausted. - for await outcome in group { - outcomes.append(outcome) - if let next = iterator.next() { - group.addTask { - await runOne( - next, libPath: libPath, - cache: cache, timeoutSeconds: timeoutSeconds - ) - } - } + + if inputs.isEmpty { + FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) + return 0 } - } - // Phase 3: write outputs and surface diagnostics. Successes are always - // written, even when other files in the batch failed (Tuist-analog batch - // semantics). - var failed = 0 - for outcome in outcomes { - let relative = outcome.input.path.dropFirst(inputURL.path.count + 1) - let destination = outputURL.appendingPathComponent(String(relative)) - - switch outcome.result { - case .failure(let error): - failed += 1 - FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) - case .success(let processResult): - // Per-input stderr is fenced with a header so the batch log stays - // readable when several files emit diagnostics. - if !processResult.stderr.isEmpty { - FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) - FileHandle.standardError.write(Data(processResult.stderr.utf8)) + // Phase 2: bounded-concurrency processing. Cap is the active core count + // so a 200-file batch doesn't fork 200 simultaneous `swift` processes. + let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) + + var outcomes: [FileOutcome] = [] + var iterator = inputs.makeIterator() + + await withTaskGroup(of: FileOutcome.self) { group in + // Seed the group up to the concurrency cap… + for _ in 0.. FileOutcome { - do { - let result = try await processFile( - inputPath: input.path, - libPath: libPath, - cache: cache, - timeoutSeconds: timeoutSeconds - ) - return FileOutcome(input: input, result: .success(result)) - } catch { - return FileOutcome(input: input, result: .failure(error)) + return failed == 0 ? 0 : 1 } - } - /// Returns every `.swift` file under `inputDir` (recursive), sorted, with - /// hidden files and files prefixed by `_` removed. Sorted output keeps - /// batch behaviour deterministic across runs. - private func collectInputs(at inputDir: URL) throws -> [URL] { - guard - let enumerator = FileManager.default.enumerator( - at: inputDir, - includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], - options: [.skipsHiddenFiles] - ) - else { - throw CLIError(message: "could not enumerate \(inputDir.path)") + /// `processFile` adapter that catches errors into the `FileOutcome` result + /// so a single failure doesn't tear down the surrounding `TaskGroup`. + private func runOne(_ input: URL) async -> FileOutcome { + do { + let result = try await processFile(inputPath: input.path) + return FileOutcome(input: input, result: .success(result)) + } catch { + return FileOutcome(input: input, result: .failure(error)) + } } - var result: [URL] = [] - for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) - // Directories aren't outputs. - if values.isDirectory == true { continue } - // Filter for `.swift` regular files, skipping the `_`-prefixed - // convention for "not an input" sources. - guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } - result.append(url.standardizedFileURL) - } - return result.sorted { $0.path < $1.path } - } + /// Returns every `.swift` file under `inputDir` (recursive), sorted, with + /// hidden files and files prefixed by `_` removed. Sorted output keeps + /// batch behaviour deterministic across runs. + private static func collectInputs(at inputDir: URL) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], + options: [.skipsHiddenFiles] + ) + else { + throw CLIError(message: "could not enumerate \(inputDir.path)") + } - // MARK: - Per-file work - - /// The per-input render pipeline: load source → consult the output cache → - /// (on miss) wrap → spawn `swift` → rewrite diagnostics → store the result - /// in the cache. The temp wrapper file is created in a per-run tmp dir and - /// torn down by `defer` whether the spawn succeeded or not. - private func processFile( - inputPath: String, - libPath: String, - cache: OutputCache?, - timeoutSeconds: Int - ) async throws -> ProcessResult { - // Load the input source. Anything past this point keys off these bytes. - let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL - let absoluteInputPath = inputURL.path - let source = try String(contentsOf: inputURL, encoding: .utf8) - - // Compute the output cache key (nil under `--no-cache` or when the cache - // root couldn't be derived at startup). Mixes input bytes, toolchain - // version, libSyntaxKit stamp, and sorted SKIT_*/SYNTAXKIT_* env vars - // — see `OutputCache.key(forInput:libPath:)`. - let cacheKey: String? = cache?.key(forInput: source, libPath: libPath) - // Cache hit: skip the wrap+spawn entirely and return the stored output. - if let cache, let cacheKey, let cached = cache.lookup(key: cacheKey) { - return ProcessResult(exitCode: 0, stdout: cached, stderr: "") + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + // Directories aren't outputs. + if values.isDirectory == true { continue } + // Filter for `.swift` regular files, skipping the `_`-prefixed + // convention for "not an input" sources. + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } } - // Wrap the user's input into a complete Swift program that imports - // SyntaxKit, runs the body inside a Group { … } builder, and prints the - // result. See `wrap` for the exact template. - let wrapped = wrap(source: source, originalPath: absoluteInputPath) - - // Spill the wrapped program to a per-invocation temp dir. The dir is - // cleaned up unconditionally so a failed spawn doesn't leak files. - let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skit-\(UUID().uuidString)") - try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) - defer { try? FileManager.default.removeItem(at: tmpDir) } - - let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") - try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - - // Spawn `swift` on the wrapped file (with timeout watchdog). stdout is - // the rendered Swift source; stderr is compiler diagnostics, if any. - let raw = try await runSwift( - wrappedPath: wrappedURL.path, - libPath: libPath, - timeoutSeconds: timeoutSeconds - ) - // #sourceLocation maps body diagnostics back to the input file. Errors in - // the preamble (lines outside the body) still reference the wrapper — - // rewrite literal occurrences of its path so users see something coherent. - let stderr = raw.stderr.replacingOccurrences( - of: wrappedURL.path, - with: absoluteInputPath - ) - - // Store on the way out. `try?` is deliberate: a cache write failure is - // not a render failure. The next run will simply miss and re-spawn. - if let cache, let cacheKey, raw.exitCode == 0 { - try? cache.store(key: cacheKey, data: raw.stdout) - } + // MARK: - Per-file work + + /// The per-input render pipeline: load source → consult the output cache → + /// (on miss) wrap → spawn `swift` → rewrite diagnostics → store the result + /// in the cache. The temp wrapper file is created in a per-run tmp dir and + /// torn down by `defer` whether the spawn succeeded or not. + private func processFile(inputPath: String) async throws -> ProcessResult { + // Load the input source. Anything past this point keys off these bytes. + let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL + let absoluteInputPath = inputURL.path + let source = try String(contentsOf: inputURL, encoding: .utf8) + + // Compute the output cache key (nil under `--no-cache` or when the cache + // root couldn't be derived at startup). Mixes input bytes, toolchain + // version, libSyntaxKit stamp, and sorted SKIT_*/SYNTAXKIT_* env vars + // — see `OutputCache.key(forInput:libPath:)`. + let cacheKey: String? = cache?.key(forInput: source, libPath: libPath) + // Cache hit: skip the wrap+spawn entirely and return the stored output. + if let cache, let cacheKey, let cached = cache.lookup(key: cacheKey) { + return ProcessResult(exitCode: 0, stdout: cached, stderr: "") + } - return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) - } + // Wrap the user's input into a complete Swift program that imports + // SyntaxKit, runs the body inside a Group { … } builder, and prints the + // result. See `wrap` for the exact template. + let wrapped = Self.wrap(source: source, originalPath: absoluteInputPath) + + // Spill the wrapped program to a per-invocation temp dir. The dir is + // cleaned up unconditionally so a failed spawn doesn't leak files. + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("skit-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + + // Spawn `swift` on the wrapped file (with timeout watchdog). stdout is + // the rendered Swift source; stderr is compiler diagnostics, if any. + let raw = try await runSwift(wrappedPath: wrappedURL.path) + // #sourceLocation maps body diagnostics back to the input file. Errors in + // the preamble (lines outside the body) still reference the wrapper — + // rewrite literal occurrences of its path so users see something coherent. + let stderr = raw.stderr.replacingOccurrences( + of: wrappedURL.path, + with: absoluteInputPath + ) - // MARK: - Wrapping - - /// Splits the input into hoisted `import` declarations and a verbatim body, - /// returning a complete Swift program that runs SyntaxKit on the body. - /// - /// The body is fenced in `#sourceLocation` directives so compiler diagnostics - /// in the body reference the original input file and line numbers. - private func wrap(source: String, originalPath: String) -> String { - // Parse the input with SwiftSyntax. The location converter is needed to - // map the body's starting byte offset back to a 1-based line number for - // the `#sourceLocation` directive. - let tree = Parser.parse(source: source) - let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) - - // Scan top-level statements for hoistable imports. Everything before the - // first non-import statement that *is* an import gets hoisted; anything - // before that which is *not* an import stays in the body (e.g. a top-level - // `// comment` is left alone). - var hoisted: [String] = [] - var firstBodyByte: AbsolutePosition? - - for item in tree.statements { - if let importDecl = item.item.as(ImportDeclSyntax.self), - firstBodyByte == nil - { - hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) - continue + // Store on the way out. `try?` is deliberate: a cache write failure is + // not a render failure. The next run will simply miss and re-spawn. + if let cache, let cacheKey, raw.exitCode == 0 { + try? cache.store(key: cacheKey, data: raw.stdout) } - firstBodyByte = item.position - break - } - // Compute the body slice (source from the first non-import byte onward) - // and the 1-based line number it lives on in the original file. - let body: String - let firstBodyLine: Int - if let firstBodyByte { - let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) - body = String(source[start...]) - firstBodyLine = locConverter.location(for: firstBodyByte).line - } else { - body = "" - firstBodyLine = 1 + return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } - // Render the hoisted-imports block. Trailing newline only if non-empty so - // the wrapper doesn't grow an extra blank line in the common no-imports - // case. - let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" - - // #sourceLocation must use a forward-slash path; escape backslashes/quotes - // defensively even though macOS paths shouldn't contain them. - let escapedPath = - originalPath - .replacingOccurrences(of: "\\", with: "\\\\") - .replacingOccurrences(of: "\"", with: "\\\"") - - // Build the final wrapper. Layout: SyntaxKit import → hoisted imports → - // Group { #sourceLocation(...) #sourceLocation() } → print. - return """ - import SyntaxKit - \(hoistedBlock) - let __skit_root = Group { - #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) - \(body) - #sourceLocation() + // MARK: - Wrapping + + /// Splits the input into hoisted `import` declarations and a verbatim body, + /// returning a complete Swift program that runs SyntaxKit on the body. + /// + /// The body is fenced in `#sourceLocation` directives so compiler diagnostics + /// in the body reference the original input file and line numbers. + private static func wrap(source: String, originalPath: String) -> String { + // Parse the input with SwiftSyntax. The location converter is needed to + // map the body's starting byte offset back to a 1-based line number for + // the `#sourceLocation` directive. + let tree = Parser.parse(source: source) + let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) + + // Scan top-level statements for hoistable imports. Everything before the + // first non-import statement that *is* an import gets hoisted; anything + // before that which is *not* an import stays in the body (e.g. a top-level + // `// comment` is left alone). + var hoisted: [String] = [] + var firstBodyByte: AbsolutePosition? + + for item in tree.statements { + if let importDecl = item.item.as(ImportDeclSyntax.self), + firstBodyByte == nil + { + hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) + continue + } + firstBodyByte = item.position + break } - print(__skit_root.generateCode()) - """ - } + // Compute the body slice (source from the first non-import byte onward) + // and the 1-based line number it lives on in the original file. + let body: String + let firstBodyLine: Int + if let firstBodyByte { + let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) + body = String(source[start...]) + firstBodyLine = locConverter.location(for: firstBodyByte).line + } else { + body = "" + firstBodyLine = 1 + } - // MARK: - Spawning swift - - /// Exit code returned when the spawned `swift` is killed by skit's timeout - /// watchdog. Matches POSIX `timeout(1)`. - private let timeoutExitCode: Int32 = 124 - - /// Bounded output capacity for the spawned `swift` (16 MiB). Generated DSL - /// output above this size is exotic; if we ever hit it we'll see a clear - /// SubprocessError rather than a silent truncation. - private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 - private let stderrLimitBytes: Int = 1 * 1_024 * 1_024 - - /// Spawns `swift` (script-mode interpreter) on the wrapped input file. - /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing - /// task group; the loser is cancelled. On timeout, returns exit 124 with - /// a one-line stderr message — matching POSIX `timeout(1)`'s convention. - private func runSwift( - wrappedPath: String, - libPath: String, - timeoutSeconds: Int - ) async throws -> ProcessResult { - let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - - // Link against libSyntaxKit, include the CShims headers, set rpath so - // the dylib loads at runtime. - let argumentsCopy: [String] = [ - "-suppress-warnings", - "-I", libPath, - "-L", libPath, - "-lSyntaxKit", - "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-rpath", "-Xlinker", libPath, - wrappedPath, - ] - - // The actual subprocess call, wrapped in a closure so the task-group race - // below can hold a single Sendable reference to it. - let invocation: @Sendable () async throws -> SwiftRunOutcome = { - let record = try await run( - .name("swift"), - arguments: Arguments(argumentsCopy), - output: .string(limit: stdoutLimitBytes), - error: .string(limit: stderrLimitBytes) - ) - return .completed( - exitCode: exitCode(from: record.terminationStatus), - stdout: Data((record.standardOutput ?? "").utf8), - stderr: record.standardError ?? "" - ) + // Render the hoisted-imports block. Trailing newline only if non-empty so + // the wrapper doesn't grow an extra blank line in the common no-imports + // case. + let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" + + // #sourceLocation must use a forward-slash path; escape backslashes/quotes + // defensively even though macOS paths shouldn't contain them. + let escapedPath = + originalPath + .replacingOccurrences(of: "\\", with: "\\\\") + .replacingOccurrences(of: "\"", with: "\\\"") + + // Build the final wrapper. Layout: SyntaxKit import → hoisted imports → + // Group { #sourceLocation(...) #sourceLocation() } → print. + return """ + import SyntaxKit + \(hoistedBlock) + let __skit_root = Group { + #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) + \(body) + #sourceLocation() + } + + print(__skit_root.generateCode()) + """ } - // Race the invocation against a sleep watchdog; whichever finishes first - // wins, the other is cancelled. `timeoutSeconds <= 0` opts out of the - // race entirely (useful for debugging genuinely long codegen). - let outcome: SwiftRunOutcome - if timeoutSeconds <= 0 { - outcome = try await invocation() - } else { - outcome = try await withThrowingTaskGroup(of: SwiftRunOutcome.self) { group in - group.addTask { try await invocation() } - group.addTask { - try await Task.sleep(for: .seconds(timeoutSeconds)) - return .timedOut + // MARK: - Spawning swift + + /// Exit code returned when the spawned `swift` is killed by skit's timeout + /// watchdog. Matches POSIX `timeout(1)`. + private static let timeoutExitCode: Int32 = 124 + + /// Bounded output capacity for the spawned `swift` (16 MiB). Generated DSL + /// output above this size is exotic; if we ever hit it we'll see a clear + /// SubprocessError rather than a silent truncation. + private static let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 + private static let stderrLimitBytes: Int = 1 * 1_024 * 1_024 + + /// Spawns `swift` (script-mode interpreter) on the wrapped input file. + /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing + /// task group; the loser is cancelled. On timeout, returns exit 124 with + /// a one-line stderr message — matching POSIX `timeout(1)`'s convention. + private func runSwift(wrappedPath: String) async throws -> ProcessResult { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + + // Link against libSyntaxKit, include the CShims headers, set rpath so + // the dylib loads at runtime. + let argumentsCopy: [String] = [ + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-rpath", "-Xlinker", libPath, + wrappedPath, + ] + + // The actual subprocess call, wrapped in a closure so the task-group race + // below can hold a single Sendable reference to it. + let invocation: @Sendable () async throws -> SwiftRunOutcome = { + let record = try await Subprocess.run( + .name("swift"), + arguments: Arguments(argumentsCopy), + output: .string(limit: Self.stdoutLimitBytes), + error: .string(limit: Self.stderrLimitBytes) + ) + return .completed( + exitCode: Self.exitCode(from: record.terminationStatus), + stdout: Data((record.standardOutput ?? "").utf8), + stderr: record.standardError ?? "" + ) + } + + // Race the invocation against a sleep watchdog; whichever finishes first + // wins, the other is cancelled. `timeoutSeconds <= 0` opts out of the + // race entirely (useful for debugging genuinely long codegen). + let outcome: SwiftRunOutcome + if timeoutSeconds <= 0 { + outcome = try await invocation() + } else { + outcome = try await withThrowingTaskGroup(of: SwiftRunOutcome.self) { group in + group.addTask { try await invocation() } + group.addTask { + try await Task.sleep(for: .seconds(self.timeoutSeconds)) + return .timedOut + } + let first = try await group.next()! + group.cancelAll() + return first } - let first = try await group.next()! - group.cancelAll() - return first } - } - // Normalize both outcomes into a single ProcessResult shape. - switch outcome { - case .completed(let exitCode, let stdout, let stderr): - return ProcessResult(exitCode: exitCode, stdout: stdout, stderr: stderr) - case .timedOut: - return ProcessResult( - exitCode: timeoutExitCode, - stdout: Data(), - stderr: "skit: timed out after \(timeoutSeconds)s\n" - ) + // Normalize both outcomes into a single ProcessResult shape. + switch outcome { + case .completed(let exitCode, let stdout, let stderr): + return ProcessResult(exitCode: exitCode, stdout: stdout, stderr: stderr) + case .timedOut: + return ProcessResult( + exitCode: Self.timeoutExitCode, + stdout: Data(), + stderr: "skit: timed out after \(timeoutSeconds)s\n" + ) + } } - } - /// Collapses Subprocess's `TerminationStatus` into a single Int32 exit code, - /// using the shell convention (128 + signal number) for signalled deaths. - private func exitCode(from status: TerminationStatus) -> Int32 { - switch status { - case .exited(let code): - return Int32(truncatingIfNeeded: code) - #if !os(Windows) - case .signaled(let signal): - // Match shell convention: 128 + signal number. - return 128 + Int32(truncatingIfNeeded: signal) - #endif + /// Collapses Subprocess's `TerminationStatus` into a single Int32 exit code, + /// using the shell convention (128 + signal number) for signalled deaths. + private static func exitCode(from status: TerminationStatus) -> Int32 { + switch status { + case .exited(let code): + return Int32(truncatingIfNeeded: code) + #if !os(Windows) + case .signaled(let signal): + // Match shell convention: 128 + signal number. + return 128 + Int32(truncatingIfNeeded: signal) + #endif + } } } diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index eb91fd6..391f4a3 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -36,7 +36,7 @@ extension Skit { /// `Run` is the default subcommand. It accepts either a single `.swift` /// file or a directory of `.swift` files; in directory mode the rendered /// output is written into a mirrored tree under `-o`. The actual work is - /// delegated to free functions in `Runner.swift`. + /// delegated to a `Runner` value (`Runner.swift`). internal struct Run: AsyncParsableCommand { internal static let configuration = CommandConfiguration( commandName: "run", @@ -127,40 +127,14 @@ extension Skit { // doesn't re-spawn `swift`. let cache: OutputCache? = noCache ? nil : OutputCache(swiftVersion: swiftVersion) - // 5. Stat the input to pick single-file vs. directory mode. Directory - // mode requires an explicit `-o` output dir; single-file mode falls - // back to stdout. - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { - throw ValidationError("input does not exist: \(input)") - } + // 5. Bind the per-invocation configuration into a Runner so the input + // orchestration doesn't have to thread libPath/cache/timeout around. + let runner = Runner(libPath: libPath, cache: cache, timeoutSeconds: timeoutSeconds) - if isDirectory.boolValue { - guard let output else { - throw ValidationError("directory inputs require -o ") - } - // 6a. Hand off to the directory orchestrator and surface its exit - // code via ExitCode (so a partial-failure batch returns 1). - let exitCode = await runDirectory( - inputDir: input, - outputDir: output, - libPath: libPath, - cache: cache, - timeoutSeconds: timeoutSeconds - ) - throw ExitCode(exitCode) - } else { - // 6b. Hand off to the single-file orchestrator. It calls `exit()` - // directly on non-zero subprocess exit, so a thrown ExitCode here - // would be unreachable in that path. - try await runSingleFile( - inputPath: input, - outputPath: output, - libPath: libPath, - cache: cache, - timeoutSeconds: timeoutSeconds - ) - } + // 6. Hand the input off to the runner: it classifies single-file vs. + // directory mode (validating existence and the `-o` requirement) and + // renders accordingly, surfacing batch exit codes via ExitCode. + try await runner(input: input, output: output) #else // Subprocess is the only backend skit knows how to use to spawn // `swift`/`swiftc`. Without it (Windows, embedded), `run` cannot work. diff --git a/globals-audit.md b/globals-audit.md index aed4aae..fa030e3 100644 --- a/globals-audit.md +++ b/globals-audit.md @@ -12,23 +12,16 @@ Generated: 2026-06-08. Branch: `research/swift-manifest-codegen`. ## Sources/skit/Runner.swift -All inside `#if canImport(Subprocess)`. - -### Functions -- [x] `toolchainMismatchMessage(bundle:local:)` → `fileprivate` method on `Skit.Run` in `Skit+Run.swift`. -- [ ] L72 — `internal func runSingleFile(inputPath:outputPath:libPath:useCache:timeoutSeconds:) async throws` -- [ ] L110 — `internal func runDirectory(inputDir:outputDir:libPath:useCache:timeoutSeconds:) async -> Int32` -- [ ] L213 — `private func runOne(_ input: URL, libPath:useCache:timeoutSeconds:) async -> FileOutcome` -- [ ] L235 — `private func collectInputs(at inputDir: URL) throws -> [URL]` -- [ ] L267 — `private func processFile(inputPath:libPath:useCache:timeoutSeconds:) async throws -> ProcessResult` -- [ ] L336 — `private func wrap(source: String, originalPath: String) -> String` -- [ ] L417 — `private func runSwift(wrappedPath:libPath:timeoutSeconds:) async throws -> ProcessResult` -- [ ] L486 — `private func exitCode(from status: TerminationStatus) -> Int32` - -### Variables -- [ ] L405 — `private let timeoutExitCode: Int32 = 124` -- [ ] L410 — `private let stdoutLimitBytes: Int = 16 * 1_024 * 1_024` -- [ ] L411 — `private let stderrLimitBytes: Int = 1 * 1_024 * 1_024` +Resolved — all eight functions and three constants were lifted into a new +`internal struct Runner: Sendable` (in the same file) that holds the +per-invocation configuration (`libPath`, `cache`, `timeoutSeconds`). The lone +`internal` entry point is `callAsFunction(input:output:)` (invoked as +`runner(input:output:)`); everything else is private: +- [x] `runSingleFile` / `runDirectory` → **private** mode methods, dispatched by `callAsFunction` via `RunInput`. +- [x] `runOne` / `processFile` / `runSwift` → config-dependent private instance methods. +- [x] `collectInputs` / `wrap` / `exitCode` → pure `private static` methods. +- [x] `timeoutExitCode` / `stdoutLimitBytes` / `stderrLimitBytes` → `private static let` constants. +- [x] `toolchainMismatchMessage(bundle:local:)` → `fileprivate` method on `Skit.Run` in `Skit+Run.swift` (moved earlier). --- @@ -51,10 +44,10 @@ Resolved — file removed. Both globals were lifted into types/extensions: | Target | Global funcs | Global vars | |---|---:|---:| -| skit / Runner.swift | 8 | 3 | +| skit / Runner.swift | 0 | 0 | | skit / Toolchain.swift (removed) | 0 | 0 | | skit / OutputCache.swift | 0 | 0 | | DocumentationHarness / Validator.swift | 0 | 1 | -| **Total** | **8** | **4** | +| **Total** | **0** | **1** | -The remaining `skit` globals are free functions/constants inside `#if canImport(Subprocess)` in `Runner.swift` — a deliberate CLI style. `privateDefaultPathExtensions` is the lone non-skit global (a linter reverted an earlier attempt to nest it). +All `skit` file-scope globals have been lifted into types/extensions. `privateDefaultPathExtensions` in `DocumentationHarness/Validator.swift` is the lone remaining global (a linter reverted an earlier attempt to nest it). From 2f7cbc3cd5121704c491d6f09953cd863b532af4 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 8 Jun 2026 14:35:31 -0400 Subject: [PATCH 38/56] Remove globals-audit.md [skip ci] All tracked file-scope globals have been lifted into types/extensions: the skit cluster into Runner/OutputCache/Skit.Run/ProcessInfo, and DocumentationHarness's defaultPathExtensions is now a static member of a `private extension [String]`. A fresh scan finds no remaining globals in compiled sources, so the audit has served its purpose. Co-Authored-By: Claude Opus 4.8 (1M context) --- globals-audit.md | 53 ------------------------------------------------ 1 file changed, 53 deletions(-) delete mode 100644 globals-audit.md diff --git a/globals-audit.md b/globals-audit.md deleted file mode 100644 index fa030e3..0000000 --- a/globals-audit.md +++ /dev/null @@ -1,53 +0,0 @@ -# Globals Audit - -File-scope **functions** and **variables** (`func` / `let` / `var` declared directly in a file, not inside a type or extension). Types (`struct`/`enum`/`class`/`actor`/`protocol`) are intentionally **excluded** — the no-globals convention targets funcs and vars only. - -Note: in `Sources/skit/` every listed declaration is indented because it lives inside a top-level `#if canImport(Subprocess)` block — it is still a true file-scope global. This is exactly the case a SwiftLint regex rule cannot detect. - -Clean targets (no global funcs/vars): **SyntaxKit**, **SyntaxParser**, **TokenVisitor**. - -Generated: 2026-06-08. Branch: `research/swift-manifest-codegen`. - ---- - -## Sources/skit/Runner.swift - -Resolved — all eight functions and three constants were lifted into a new -`internal struct Runner: Sendable` (in the same file) that holds the -per-invocation configuration (`libPath`, `cache`, `timeoutSeconds`). The lone -`internal` entry point is `callAsFunction(input:output:)` (invoked as -`runner(input:output:)`); everything else is private: -- [x] `runSingleFile` / `runDirectory` → **private** mode methods, dispatched by `callAsFunction` via `RunInput`. -- [x] `runOne` / `processFile` / `runSwift` → config-dependent private instance methods. -- [x] `collectInputs` / `wrap` / `exitCode` → pure `private static` methods. -- [x] `timeoutExitCode` / `stdoutLimitBytes` / `stderrLimitBytes` → `private static let` constants. -- [x] `toolchainMismatchMessage(bundle:local:)` → `fileprivate` method on `Skit.Run` in `Skit+Run.swift` (moved earlier). - ---- - -## Sources/skit/Toolchain.swift - -Resolved — file removed. Both globals were lifted into types/extensions: -- [x] `captureSwiftVersion()` → `fileprivate` method on `Skit.Run` in `Skit+Run.swift`. -- [x] `syntaxKitCacheRoot()` → `ProcessInfo.syntaxKitCacheRoot(default:)` in `ProcessInfo+SyntaxKitCacheRoot.swift`; the platform default is now a `private static let defaultCacheRoot` on `OutputCache`. - ---- - -## Sources/DocumentationHarness/Validator.swift - -### Variables -- [ ] L36 — `private let privateDefaultPathExtensions = ["md"]` - ---- - -## Summary - -| Target | Global funcs | Global vars | -|---|---:|---:| -| skit / Runner.swift | 0 | 0 | -| skit / Toolchain.swift (removed) | 0 | 0 | -| skit / OutputCache.swift | 0 | 0 | -| DocumentationHarness / Validator.swift | 0 | 1 | -| **Total** | **0** | **1** | - -All `skit` file-scope globals have been lifted into types/extensions. `privateDefaultPathExtensions` in `DocumentationHarness/Validator.swift` is the lone remaining global (a linter reverted an earlier attempt to nest it). From dd2c3b46d719fa894b96d312adaa3bd892f704aa Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 8 Jun 2026 15:45:08 -0400 Subject: [PATCH 39/56] Extract skit Runner wrap, timeout, and swift invocation [skip ci] Lift three inline pieces of Runner.swift into reusable homes: - WrappedSource struct replaces the private static wrap template - Task.timeout generic method replaces the inline task-group race - Subprocess.Configuration.swift(libPath:wrappedPath:) builds the swift invocation and its link/include/rpath flags Co-Authored-By: Claude Opus 4.8 (1M context) --- Sources/skit/Runner.swift | 110 ++-------------- .../skit/Subprocess.Configuration+Swift.swift | 56 ++++++++ Sources/skit/Task+Timeout.swift | 51 ++++++++ Sources/skit/WrappedSource.swift | 122 ++++++++++++++++++ 4 files changed, 237 insertions(+), 102 deletions(-) create mode 100644 Sources/skit/Subprocess.Configuration+Swift.swift create mode 100644 Sources/skit/Task+Timeout.swift create mode 100644 Sources/skit/WrappedSource.swift diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index 59b7274..b959ebd 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -32,8 +32,6 @@ import ArgumentParser import Foundation import Subprocess - import SwiftParser - import SwiftSyntax // Run lifecycle (per `skit run` invocation): // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) @@ -65,7 +63,6 @@ } // MARK: - Dispatch - /// Classifies `input` (single file vs. directory) and renders it. Directory /// mode surfaces its batch exit code via `ExitCode` (so a partial-failure @@ -266,8 +263,8 @@ // Wrap the user's input into a complete Swift program that imports // SyntaxKit, runs the body inside a Group { … } builder, and prints the - // result. See `wrap` for the exact template. - let wrapped = Self.wrap(source: source, originalPath: absoluteInputPath) + // result. See `WrappedSource` for the exact template. + let wrapped = WrappedSource(source: source, originalPath: absoluteInputPath).rendered // Spill the wrapped program to a per-invocation temp dir. The dir is // cleaned up unconditionally so a failed spawn doesn't leak files. @@ -299,78 +296,6 @@ return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) } - // MARK: - Wrapping - - /// Splits the input into hoisted `import` declarations and a verbatim body, - /// returning a complete Swift program that runs SyntaxKit on the body. - /// - /// The body is fenced in `#sourceLocation` directives so compiler diagnostics - /// in the body reference the original input file and line numbers. - private static func wrap(source: String, originalPath: String) -> String { - // Parse the input with SwiftSyntax. The location converter is needed to - // map the body's starting byte offset back to a 1-based line number for - // the `#sourceLocation` directive. - let tree = Parser.parse(source: source) - let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) - - // Scan top-level statements for hoistable imports. Everything before the - // first non-import statement that *is* an import gets hoisted; anything - // before that which is *not* an import stays in the body (e.g. a top-level - // `// comment` is left alone). - var hoisted: [String] = [] - var firstBodyByte: AbsolutePosition? - - for item in tree.statements { - if let importDecl = item.item.as(ImportDeclSyntax.self), - firstBodyByte == nil - { - hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) - continue - } - firstBodyByte = item.position - break - } - - // Compute the body slice (source from the first non-import byte onward) - // and the 1-based line number it lives on in the original file. - let body: String - let firstBodyLine: Int - if let firstBodyByte { - let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) - body = String(source[start...]) - firstBodyLine = locConverter.location(for: firstBodyByte).line - } else { - body = "" - firstBodyLine = 1 - } - - // Render the hoisted-imports block. Trailing newline only if non-empty so - // the wrapper doesn't grow an extra blank line in the common no-imports - // case. - let hoistedBlock = hoisted.isEmpty ? "" : hoisted.joined(separator: "\n") + "\n" - - // #sourceLocation must use a forward-slash path; escape backslashes/quotes - // defensively even though macOS paths shouldn't contain them. - let escapedPath = - originalPath - .replacingOccurrences(of: "\\", with: "\\\\") - .replacingOccurrences(of: "\"", with: "\\\"") - - // Build the final wrapper. Layout: SyntaxKit import → hoisted imports → - // Group { #sourceLocation(...) #sourceLocation() } → print. - return """ - import SyntaxKit - \(hoistedBlock) - let __skit_root = Group { - #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) - \(body) - #sourceLocation() - } - - print(__skit_root.generateCode()) - """ - } - // MARK: - Spawning swift /// Exit code returned when the spawned `swift` is killed by skit's timeout @@ -388,26 +313,14 @@ /// task group; the loser is cancelled. On timeout, returns exit 124 with /// a one-line stderr message — matching POSIX `timeout(1)`'s convention. private func runSwift(wrappedPath: String) async throws -> ProcessResult { - let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" - - // Link against libSyntaxKit, include the CShims headers, set rpath so - // the dylib loads at runtime. - let argumentsCopy: [String] = [ - "-suppress-warnings", - "-I", libPath, - "-L", libPath, - "-lSyntaxKit", - "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-rpath", "-Xlinker", libPath, - wrappedPath, - ] + // Build the `swift` invocation (executable + link/include/rpath flags). + let configuration = Subprocess.Configuration.swift(libPath: libPath, wrappedPath: wrappedPath) // The actual subprocess call, wrapped in a closure so the task-group race // below can hold a single Sendable reference to it. let invocation: @Sendable () async throws -> SwiftRunOutcome = { let record = try await Subprocess.run( - .name("swift"), - arguments: Arguments(argumentsCopy), + configuration, output: .string(limit: Self.stdoutLimitBytes), error: .string(limit: Self.stderrLimitBytes) ) @@ -425,16 +338,9 @@ if timeoutSeconds <= 0 { outcome = try await invocation() } else { - outcome = try await withThrowingTaskGroup(of: SwiftRunOutcome.self) { group in - group.addTask { try await invocation() } - group.addTask { - try await Task.sleep(for: .seconds(self.timeoutSeconds)) - return .timedOut - } - let first = try await group.next()! - group.cancelAll() - return first - } + outcome = + try await Task.timeout(.seconds(timeoutSeconds), operation: invocation) + ?? .timedOut } // Normalize both outcomes into a single ProcessResult shape. diff --git a/Sources/skit/Subprocess.Configuration+Swift.swift b/Sources/skit/Subprocess.Configuration+Swift.swift new file mode 100644 index 0000000..744f164 --- /dev/null +++ b/Sources/skit/Subprocess.Configuration+Swift.swift @@ -0,0 +1,56 @@ +// +// Subprocess.Configuration+Swift.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Subprocess + + extension Subprocess.Configuration { + /// A configuration that runs the `swift` interpreter on `wrappedPath`, + /// linked against `libSyntaxKit` in `libPath`. + /// + /// Builds the full flag set: link against libSyntaxKit, include the CShims + /// headers, and set rpath so the dylib loads at runtime. The executable is + /// resolved by name on `PATH`. + internal static func swift(libPath: String, wrappedPath: String) -> Self { + let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + let arguments: [String] = [ + "-suppress-warnings", + "-I", libPath, + "-L", libPath, + "-lSyntaxKit", + "-Xcc", "-I", "-Xcc", cShimsInclude, + "-Xlinker", "-rpath", "-Xlinker", libPath, + wrappedPath, + ] + return Self(executable: .name("swift"), arguments: Arguments(arguments)) + } + } + +#endif diff --git a/Sources/skit/Task+Timeout.swift b/Sources/skit/Task+Timeout.swift new file mode 100644 index 0000000..b8e156b --- /dev/null +++ b/Sources/skit/Task+Timeout.swift @@ -0,0 +1,51 @@ +// +// Task+Timeout.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +extension Task where Failure == any Error, Success: Sendable { + /// Runs `operation`, returning its value, or `nil` if `duration` elapses + /// first. The operation and a sleep watchdog race in a throwing task group; + /// whichever finishes first wins and the loser is cancelled. + internal static func timeout( + _ duration: Duration, + operation: @escaping @Sendable () async throws -> Success + ) async throws -> Success? { + try await withThrowingTaskGroup(of: Success?.self) { group in + group.addTask { try await operation() } + group.addTask { + // Bare `Task` here means `Task`, which has no + // `sleep`; spell out the never-returning task to reach it. + try await Task.sleep(for: duration) + return nil + } + let first = try await group.next()! + group.cancelAll() + return first + } + } +} diff --git a/Sources/skit/WrappedSource.swift b/Sources/skit/WrappedSource.swift new file mode 100644 index 0000000..d421086 --- /dev/null +++ b/Sources/skit/WrappedSource.swift @@ -0,0 +1,122 @@ +// +// WrappedSource.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import SwiftParser +import SwiftSyntax + +/// A SyntaxKit DSL input split into hoisted `import` declarations and a verbatim +/// body, ready to be `rendered` into a complete Swift program that runs SyntaxKit +/// on the body. +/// +/// The body is fenced in `#sourceLocation` directives so compiler diagnostics in +/// the body reference the original input file and line numbers. +internal struct WrappedSource { + /// The original input path, used for the `#sourceLocation` directive. + private let originalPath: String + /// Top-level `import` declarations hoisted above the wrapper body, each already + /// trimmed of surrounding whitespace. + private let hoistedImports: [String] + /// The input source from the first non-import byte onward, verbatim. + private let body: String + /// The 1-based line number the body starts on in the original file. + private let firstBodyLine: Int + + /// Parses `source`, hoisting leading `import` declarations and capturing the + /// remaining body along with the line it begins on. Everything before the first + /// non-import statement that *is* an import gets hoisted; anything before that + /// which is *not* an import stays in the body (e.g. a leading `// comment`). + internal init(source: String, originalPath: String) { + self.originalPath = originalPath + + // Parse the input with SwiftSyntax. The location converter is needed to + // map the body's starting byte offset back to a 1-based line number for + // the `#sourceLocation` directive. + let tree = Parser.parse(source: source) + let locConverter = SourceLocationConverter(fileName: originalPath, tree: tree) + + // Scan top-level statements for hoistable imports. + var hoisted: [String] = [] + var firstBodyByte: AbsolutePosition? + + for item in tree.statements { + if let importDecl = item.item.as(ImportDeclSyntax.self), + firstBodyByte == nil + { + hoisted.append(importDecl.description.trimmingCharacters(in: .whitespacesAndNewlines)) + continue + } + firstBodyByte = item.position + break + } + + self.hoistedImports = hoisted + + // Compute the body slice (source from the first non-import byte onward) + // and the 1-based line number it lives on in the original file. + if let firstBodyByte { + let start = source.utf8.index(source.utf8.startIndex, offsetBy: firstBodyByte.utf8Offset) + self.body = String(source[start...]) + self.firstBodyLine = locConverter.location(for: firstBodyByte).line + } else { + self.body = "" + self.firstBodyLine = 1 + } + } + + /// A complete Swift program that imports SyntaxKit, runs the body inside a + /// `Group { … }` builder, and prints the generated code. + internal var rendered: String { + // Render the hoisted-imports block. Trailing newline only if non-empty so + // the wrapper doesn't grow an extra blank line in the common no-imports + // case. + let hoistedBlock = hoistedImports.isEmpty ? "" : hoistedImports.joined(separator: "\n") + "\n" + + // #sourceLocation must use a forward-slash path; escape backslashes/quotes + // defensively even though macOS paths shouldn't contain them. + let escapedPath = + originalPath + .replacingOccurrences(of: "\\", with: "\\\\") + .replacingOccurrences(of: "\"", with: "\\\"") + + // Build the final wrapper. Layout: SyntaxKit import → hoisted imports → + // Group { #sourceLocation(...) #sourceLocation() } → print. + return """ + import SyntaxKit + \(hoistedBlock) + let __skit_root = Group { + #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) + \(body) + #sourceLocation() + } + + print(__skit_root.generateCode()) + """ + } +} From e7ced53c400002c999ff021ad721fcd29a8609c8 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 8 Jun 2026 16:48:55 -0400 Subject: [PATCH 40/56] Move skit pure-logic helpers into SyntaxKit; decouple Runner from ArgumentParser [skip ci] Relocate the 14 skit files that import neither Subprocess nor ArgumentParser into Sources/SyntaxKit/Execution/, drop their now-unnecessary `#if canImport(Subprocess)` guards, and expose the cross-target ones at `package` access (matching the DocumentationHarness pattern). skit gains a SyntaxKit dependency and imports it where it consumes the moved types. Also corrects OutputCache's stale "@unchecked Sendable" doc comment. Decouple Runner from the CLI so it can be driven in-process: - New RunError typed enum (invalidInput / renderFailed / batchFailed / unexpected). - Runner drops `import ArgumentParser`; `callAsFunction` is now `throws(RunError)` and no longer throws ExitCode or calls `exit()`. - RunInput.resolve throws RunError instead of ArgumentParser's ValidationError. - Skit+Run maps each RunError case back to ExitCode/ValidationError in a `render(using:)` helper, preserving exact CLI behavior (invalid input -> 64, render failure -> subprocess code, batch failure -> 1). Uses enum-pattern catch clauses to sidestep a Swift 6.3.2 SILGen crash on the always-true `catch let error as RunError` cast under typed throws. All 398 tests pass; Periphery reports no unused code. Co-Authored-By: Claude Opus 4.8 (1M context) --- Package.swift | 1 + .../Execution/Bundle+ResolveLibPath.swift | 78 +++++++++ .../Execution}/CLIError.swift | 16 +- .../SyntaxKit/Execution/ContentHasher.swift | 59 +++++++ .../Execution/FileManager+IsLibDir.swift} | 24 ++- .../Execution}/FileManager+LibStamp.swift | 28 ++- .../Execution}/FileOutcome.swift | 19 ++- Sources/SyntaxKit/Execution/OutputCache.swift | 157 +++++++++++++++++ .../ProcessInfo+SyntaxKitCacheRoot.swift} | 25 ++- .../Execution/ProcessResult.swift} | 29 ++-- .../Execution}/String+DylibFilename.swift | 26 ++- .../Execution}/SwiftRunOutcome.swift | 20 +-- .../Execution}/Task+Timeout.swift | 2 +- .../Execution/ToolchainCheckResult.swift | 73 ++++++++ .../Execution}/WrappedSource.swift | 6 +- Sources/skit/Bundle+ResolveLibPath.swift | 82 --------- Sources/skit/ContentHasher.swift | 63 ------- Sources/skit/OutputCache.swift | 161 ------------------ Sources/skit/RunError.swift | 48 ++++++ Sources/skit/RunInput.swift | 9 +- Sources/skit/Runner.swift | 41 +++-- Sources/skit/Skit+Run.swift | 24 ++- Sources/skit/ToolchainCheckResult.swift | 77 --------- 23 files changed, 556 insertions(+), 512 deletions(-) create mode 100644 Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift rename Sources/{skit => SyntaxKit/Execution}/CLIError.swift (77%) create mode 100644 Sources/SyntaxKit/Execution/ContentHasher.swift rename Sources/{skit/ProcessResult.swift => SyntaxKit/Execution/FileManager+IsLibDir.swift} (74%) rename Sources/{skit => SyntaxKit/Execution}/FileManager+LibStamp.swift (65%) rename Sources/{skit => SyntaxKit/Execution}/FileOutcome.swift (75%) create mode 100644 Sources/SyntaxKit/Execution/OutputCache.swift rename Sources/{skit/FileManager+IsLibDir.swift => SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift} (70%) rename Sources/{skit/ProcessInfo+SyntaxKitCacheRoot.swift => SyntaxKit/Execution/ProcessResult.swift} (67%) rename Sources/{skit => SyntaxKit/Execution}/String+DylibFilename.swift (75%) rename Sources/{skit => SyntaxKit/Execution}/SwiftRunOutcome.swift (75%) rename Sources/{skit => SyntaxKit/Execution}/Task+Timeout.swift (98%) create mode 100644 Sources/SyntaxKit/Execution/ToolchainCheckResult.swift rename Sources/{skit => SyntaxKit/Execution}/WrappedSource.swift (97%) delete mode 100644 Sources/skit/Bundle+ResolveLibPath.swift delete mode 100644 Sources/skit/ContentHasher.swift delete mode 100644 Sources/skit/OutputCache.swift create mode 100644 Sources/skit/RunError.swift delete mode 100644 Sources/skit/ToolchainCheckResult.swift diff --git a/Package.swift b/Package.swift index ae34413..779ef62 100644 --- a/Package.swift +++ b/Package.swift @@ -144,6 +144,7 @@ let package = Package( .executableTarget( name: "skit", dependencies: [ + "SyntaxKit", "SyntaxParser", .product(name: "SwiftSyntax", package: "swift-syntax"), .product(name: "SwiftParser", package: "swift-syntax"), diff --git a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift new file mode 100644 index 0000000..0bc49e2 --- /dev/null +++ b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift @@ -0,0 +1,78 @@ +// +// Bundle+ResolveLibPath.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +package import Foundation + +extension Bundle { + /// Resolves a directory containing `libSyntaxKit.dylib` + swiftmodules. + /// + /// Tries each non-nil entry in `candidates` in order; if any non-nil + /// candidate is not a SyntaxKit lib dir, throws `CLIError`. If every + /// candidate is absent, falls back to bundle-relative paths derived + /// from `executableURL`: + /// - `/lib` (adjacent layout) + /// - `/../lib/skit` (Homebrew layout) + package func resolveLibPath(candidates: String?...) throws -> String { + let fileManager = FileManager.default + + for candidate in candidates { + guard let candidate else { continue } + guard fileManager.isLibDir(candidate) else { + throw CLIError(message: "path does not look like a SyntaxKit lib dir: \(candidate)") + } + return candidate + } + + if let execURL = executableURL?.resolvingSymlinksInPath() { + let execDir = execURL.deletingLastPathComponent() + + let adjacent = execDir.appendingPathComponent("lib").path + if fileManager.isLibDir(adjacent) { + return adjacent + } + + let brewLayout = execDir.deletingLastPathComponent() + .appendingPathComponent("lib/skit").path + if fileManager.isLibDir(brewLayout) { + return brewLayout + } + } + + throw CLIError( + message: """ + Could not locate SyntaxKit lib directory. Looked for: + 1. explicit candidates (none provided or all empty) + 2. /lib/ (not found) + 3. /../lib/skit/ (not found) + Run Scripts/build-skit-release.sh to produce a self-contained + release bundle under .build/skit-release/. + """ + ) + } +} diff --git a/Sources/skit/CLIError.swift b/Sources/SyntaxKit/Execution/CLIError.swift similarity index 77% rename from Sources/skit/CLIError.swift rename to Sources/SyntaxKit/Execution/CLIError.swift index ece6614..a5ee68e 100644 --- a/Sources/skit/CLIError.swift +++ b/Sources/SyntaxKit/Execution/CLIError.swift @@ -27,13 +27,13 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +/// Throwable error wrapper for skit's user-facing diagnostics. The message +/// is printed verbatim — keep it actionable (path, hint, next step). +package struct CLIError: Error, CustomStringConvertible { + package let message: String + package var description: String { message } - /// Throwable error wrapper for skit's user-facing diagnostics. The message - /// is printed verbatim — keep it actionable (path, hint, next step). - internal struct CLIError: Error, CustomStringConvertible { - let message: String - var description: String { message } + package init(message: String) { + self.message = message } - -#endif +} diff --git a/Sources/SyntaxKit/Execution/ContentHasher.swift b/Sources/SyntaxKit/Execution/ContentHasher.swift new file mode 100644 index 0000000..6ec0e3c --- /dev/null +++ b/Sources/SyntaxKit/Execution/ContentHasher.swift @@ -0,0 +1,59 @@ +// +// ContentHasher.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +/// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed +/// cache keys. The cache keys aren't security-critical — there's no +/// adversary trying to forge a collision — so we don't need a cryptographic +/// hash. 64 bits of output gives ~10⁻⁹ collision probability at 10⁶ cache +/// entries, which is well past anything we'll see in practice. +/// +/// FNV-1a is deterministic across processes and platforms (unlike the Swift +/// stdlib `Hasher`, whose seed is randomized per-process) — that +/// determinism is what makes it usable as an on-disk cache key. +internal struct ContentHasher { + private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 + private static let prime: UInt64 = 0x0000_0100_0000_01b3 + + private var state: UInt64 = ContentHasher.offsetBasis + + internal mutating func update(data: Data) { + for byte in data { + state ^= UInt64(byte) + state &*= ContentHasher.prime + } + } + + /// Returns the hash as a 16-char lowercase-hex string suitable for use as + /// a directory name. + internal func finalize() -> String { + String(format: "%016x", state) + } +} diff --git a/Sources/skit/ProcessResult.swift b/Sources/SyntaxKit/Execution/FileManager+IsLibDir.swift similarity index 74% rename from Sources/skit/ProcessResult.swift rename to Sources/SyntaxKit/Execution/FileManager+IsLibDir.swift index c5178bb..44199d3 100644 --- a/Sources/skit/ProcessResult.swift +++ b/Sources/SyntaxKit/Execution/FileManager+IsLibDir.swift @@ -1,5 +1,5 @@ // -// ProcessResult.swift +// FileManager+IsLibDir.swift // SyntaxKit // // Created by Leo Dion. @@ -27,17 +27,15 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +import Foundation - import Foundation - - /// Raw outcome of rendering one input — what `processFile` returns to its - /// caller. `exitCode == 0` indicates the spawned `swift` succeeded (or that - /// the output cache hit, which is treated identically). - internal struct ProcessResult: Sendable { - let exitCode: Int32 - let stdout: Data - let stderr: String +extension FileManager { + /// True if `path` is a directory containing `libSyntaxKit.{dylib,so}`. + internal func isLibDir(_ path: String) -> Bool { + var isDir: ObjCBool = false + guard fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { + return false + } + return fileExists(atPath: "\(path)/\("SyntaxKit".dylibFilename)") } - -#endif +} diff --git a/Sources/skit/FileManager+LibStamp.swift b/Sources/SyntaxKit/Execution/FileManager+LibStamp.swift similarity index 65% rename from Sources/skit/FileManager+LibStamp.swift rename to Sources/SyntaxKit/Execution/FileManager+LibStamp.swift index 82a4ade..b55e159 100644 --- a/Sources/skit/FileManager+LibStamp.swift +++ b/Sources/SyntaxKit/Execution/FileManager+LibStamp.swift @@ -27,21 +27,17 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +import Foundation - import Foundation - - extension FileManager { - /// `/` fingerprint of `libSyntaxKit.{dylib,so}` under - /// `libPath`, or nil if unreadable. Catches in-place rebuilds without a - /// version bump. - internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/\("SyntaxKit".dylibFilename)" - guard let attrs = try? attributesOfItem(atPath: dylib) else { return nil } - let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 - let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 - return "\(size)/\(Int(mtime))" - } +extension FileManager { + /// `/` fingerprint of `libSyntaxKit.{dylib,so}` under + /// `libPath`, or nil if unreadable. Catches in-place rebuilds without a + /// version bump. + internal func libStamp(libPath: String) -> String? { + let dylib = "\(libPath)/\("SyntaxKit".dylibFilename)" + guard let attrs = try? attributesOfItem(atPath: dylib) else { return nil } + let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 + let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 + return "\(size)/\(Int(mtime))" } - -#endif +} diff --git a/Sources/skit/FileOutcome.swift b/Sources/SyntaxKit/Execution/FileOutcome.swift similarity index 75% rename from Sources/skit/FileOutcome.swift rename to Sources/SyntaxKit/Execution/FileOutcome.swift index f4ff50a..cf9a07a 100644 --- a/Sources/skit/FileOutcome.swift +++ b/Sources/SyntaxKit/Execution/FileOutcome.swift @@ -27,15 +27,16 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +package import Foundation - import Foundation +/// Result of processing one input in directory mode. The error case is +/// stored (not thrown) so the batch can keep going. +package struct FileOutcome: Sendable { + package let input: URL + package let result: Result - /// Result of processing one input in directory mode. The error case is - /// stored (not thrown) so the batch can keep going. - internal struct FileOutcome: Sendable { - let input: URL - let result: Result + package init(input: URL, result: Result) { + self.input = input + self.result = result } - -#endif +} diff --git a/Sources/SyntaxKit/Execution/OutputCache.swift b/Sources/SyntaxKit/Execution/OutputCache.swift new file mode 100644 index 0000000..f6c7a32 --- /dev/null +++ b/Sources/SyntaxKit/Execution/OutputCache.swift @@ -0,0 +1,157 @@ +// +// OutputCache.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +package import Foundation + +/// On-disk cache of rendered skit output, content-keyed so a re-run on +/// unchanged input skips the `swift` spawn entirely. +/// +/// `Sendable`: the stored state is a `@Sendable` `FileManager` factory plus a +/// `ProcessInfo` and value types, so the single instance can be shared safely +/// across the concurrent `runOne` tasks in directory mode. The default +/// singletons used in production (and the typical test doubles) are +/// thread-safe for the operations we invoke. +package struct OutputCache: Sendable { + /// Bumped when the cache layout changes in a way that requires invalidation. + private static let schemaVersion = "v1" + + /// Home-relative cache root used when `XDG_CACHE_HOME` is unset: macOS + /// `~/Library/Caches/...`, else Linux `~/.cache/syntaxkit`. The home dir is + /// fixed for the process lifetime, so this is computed once. + private static let defaultCacheRoot: URL = { + let home = NSHomeDirectory() + #if os(macOS) + return URL(fileURLWithPath: home) + .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") + #else + return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") + #endif + }() + + /// `/outputs/`. Populated once at init; per-key + /// directories are derived from it on demand. + private let root: URL + private let fileManager: @Sendable () -> FileManager + private let processInfo: ProcessInfo + + /// Verbatim `swift --version` output captured once for the lifetime of + /// this cache, so per-input key derivation doesn't re-spawn `swift`. + /// nil if capture failed before construction. + private let swiftVersion: String? + + package init( + swiftVersion: String?, + fileManager: @autoclosure @escaping @Sendable () -> FileManager = .default, + processInfo: ProcessInfo = .processInfo + ) { + self.root = processInfo.syntaxKitCacheRoot(default: Self.defaultCacheRoot) + .appendingPathComponent("outputs") + self.swiftVersion = swiftVersion + self.fileManager = fileManager + self.processInfo = processInfo + } + + /// 64-bit content hash over (schema version, input source bytes, swift + /// version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any + /// change in these inputs produces a fresh key and forces a recompile. + /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. + package func key(forInput source: String, libPath: String) -> String { + var hasher = ContentHasher() + // Schema version: bump to invalidate every existing cache entry at once. + hasher.update(data: Data(Self.schemaVersion.utf8)) + // Input source bytes: the primary driver of the key. + hasher.update(data: Data(source.utf8)) + + // Toolchain version. Different `swift` builds emit different bytes for + // the same DSL input. + if let swiftVersion { + hasher.update(data: Data(swiftVersion.utf8)) + } + // libSyntaxKit stamp. A rebuilt dylib can change the rendered output + // even without a Swift-version bump. + if let stamp = fileManager().libStamp(libPath: libPath) { + hasher.update(data: Data(stamp.utf8)) + } + + // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and + // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. + let env = processInfo.environment + .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } + .sorted { $0.key < $1.key } + for (key, value) in env { + hasher.update(data: Data("\(key)=\(value)\0".utf8)) + } + + return hasher.finalize() + } + + /// Returns the cached rendered output for `key`, or nil on miss. + package func lookup(key: String) -> Data? { + try? Data(contentsOf: directory(for: key).appendingPathComponent("output.swift")) + } + + /// Atomically stores `data` under `key`. Concurrent writers race via a + /// `tmp../` staging dir + rename; the loser drops their copy. + package func store(key: String, data: Data) throws { + let cacheRoot = directory(for: key) + let final = cacheRoot.appendingPathComponent("output.swift") + + // Ensure the parent of the cache key dir exists. The key dir itself is + // installed by the atomic rename below. + try fileManager().createDirectory( + at: cacheRoot.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + + // Stage the payload in a per-pid + uuid sibling dir so it can be renamed + // into place as a single atomic step. + let staging = cacheRoot.deletingLastPathComponent() + .appendingPathComponent( + "tmp.\(processInfo.processIdentifier).\(UUID().uuidString)" + ) + try fileManager().createDirectory(at: staging, withIntermediateDirectories: true) + try data.write(to: staging.appendingPathComponent("output.swift")) + + // Atomic rename into the cache path. If a peer already populated this + // key, swallow the rename error and drop our staging copy. Re-throw only + // if the destination is still missing afterwards. + do { + try fileManager().moveItem(at: staging, to: cacheRoot) + } catch { + try? fileManager().removeItem(at: staging) + if !fileManager().fileExists(atPath: final.path) { + throw error + } + } + } + + private func directory(for key: String) -> URL { + root.appendingPathComponent(key) + } +} diff --git a/Sources/skit/FileManager+IsLibDir.swift b/Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift similarity index 70% rename from Sources/skit/FileManager+IsLibDir.swift rename to Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift index bb512c9..59de0b5 100644 --- a/Sources/skit/FileManager+IsLibDir.swift +++ b/Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift @@ -1,5 +1,5 @@ // -// FileManager+IsLibDir.swift +// ProcessInfo+SyntaxKitCacheRoot.swift // SyntaxKit // // Created by Leo Dion. @@ -27,19 +27,16 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +import Foundation - import Foundation - - extension FileManager { - /// True if `path` is a directory containing `libSyntaxKit.{dylib,so}`. - internal func isLibDir(_ path: String) -> Bool { - var isDir: ObjCBool = false - guard fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { - return false - } - return fileExists(atPath: "\(path)/\("SyntaxKit".dylibFilename)") +extension ProcessInfo { + /// Root for all skit caches: `/syntaxkit` when that env + /// var is set and non-empty, otherwise `defaultRoot` (typically the + /// platform's home-relative cache dir). + internal func syntaxKitCacheRoot(default defaultRoot: URL) -> URL { + if let xdg = environment["XDG_CACHE_HOME"], !xdg.isEmpty { + return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") } + return defaultRoot } - -#endif +} diff --git a/Sources/skit/ProcessInfo+SyntaxKitCacheRoot.swift b/Sources/SyntaxKit/Execution/ProcessResult.swift similarity index 67% rename from Sources/skit/ProcessInfo+SyntaxKitCacheRoot.swift rename to Sources/SyntaxKit/Execution/ProcessResult.swift index 230880f..e8a14c5 100644 --- a/Sources/skit/ProcessInfo+SyntaxKitCacheRoot.swift +++ b/Sources/SyntaxKit/Execution/ProcessResult.swift @@ -1,5 +1,5 @@ // -// ProcessInfo+SyntaxKitCacheRoot.swift +// ProcessResult.swift // SyntaxKit // // Created by Leo Dion. @@ -27,20 +27,19 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +package import Foundation - import Foundation +/// Raw outcome of rendering one input — what `processFile` returns to its +/// caller. `exitCode == 0` indicates the spawned `swift` succeeded (or that +/// the output cache hit, which is treated identically). +package struct ProcessResult: Sendable { + package let exitCode: Int32 + package let stdout: Data + package let stderr: String - extension ProcessInfo { - /// Root for all skit caches: `/syntaxkit` when that env - /// var is set and non-empty, otherwise `defaultRoot` (typically the - /// platform's home-relative cache dir). - internal func syntaxKitCacheRoot(default defaultRoot: URL) -> URL { - if let xdg = environment["XDG_CACHE_HOME"], !xdg.isEmpty { - return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") - } - return defaultRoot - } + package init(exitCode: Int32, stdout: Data, stderr: String) { + self.exitCode = exitCode + self.stdout = stdout + self.stderr = stderr } - -#endif +} diff --git a/Sources/skit/String+DylibFilename.swift b/Sources/SyntaxKit/Execution/String+DylibFilename.swift similarity index 75% rename from Sources/skit/String+DylibFilename.swift rename to Sources/SyntaxKit/Execution/String+DylibFilename.swift index 3f9047c..7497e7f 100644 --- a/Sources/skit/String+DylibFilename.swift +++ b/Sources/SyntaxKit/Execution/String+DylibFilename.swift @@ -27,19 +27,15 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) - - extension String { - /// Treats `self` as a Swift library product name and returns the - /// platform-specific shared-library filename (`libFoo.dylib` on macOS, - /// `libFoo.so` on Linux). - internal var dylibFilename: String { - #if os(Linux) - return "lib\(self).so" - #else - return "lib\(self).dylib" - #endif - } +extension String { + /// Treats `self` as a Swift library product name and returns the + /// platform-specific shared-library filename (`libFoo.dylib` on macOS, + /// `libFoo.so` on Linux). + internal var dylibFilename: String { + #if os(Linux) + return "lib\(self).so" + #else + return "lib\(self).dylib" + #endif } - -#endif +} diff --git a/Sources/skit/SwiftRunOutcome.swift b/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift similarity index 75% rename from Sources/skit/SwiftRunOutcome.swift rename to Sources/SyntaxKit/Execution/SwiftRunOutcome.swift index 71cfbec..1e8404c 100644 --- a/Sources/skit/SwiftRunOutcome.swift +++ b/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift @@ -27,16 +27,12 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +package import Foundation - import Foundation - - /// Either the spawned `swift` ran to completion (success or failure) or - /// the watchdog elapsed first. The completed payload is normalized to the - /// shape callers want regardless of platform. - internal enum SwiftRunOutcome: Sendable { - case completed(exitCode: Int32, stdout: Data, stderr: String) - case timedOut - } - -#endif +/// Either the spawned `swift` ran to completion (success or failure) or +/// the watchdog elapsed first. The completed payload is normalized to the +/// shape callers want regardless of platform. +package enum SwiftRunOutcome: Sendable { + case completed(exitCode: Int32, stdout: Data, stderr: String) + case timedOut +} diff --git a/Sources/skit/Task+Timeout.swift b/Sources/SyntaxKit/Execution/Task+Timeout.swift similarity index 98% rename from Sources/skit/Task+Timeout.swift rename to Sources/SyntaxKit/Execution/Task+Timeout.swift index b8e156b..beabdc0 100644 --- a/Sources/skit/Task+Timeout.swift +++ b/Sources/SyntaxKit/Execution/Task+Timeout.swift @@ -31,7 +31,7 @@ extension Task where Failure == any Error, Success: Sendable { /// Runs `operation`, returning its value, or `nil` if `duration` elapses /// first. The operation and a sleep watchdog race in a throwing task group; /// whichever finishes first wins and the loser is cancelled. - internal static func timeout( + package static func timeout( _ duration: Duration, operation: @escaping @Sendable () async throws -> Success ) async throws -> Success? { diff --git a/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift b/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift new file mode 100644 index 0000000..8a235e6 --- /dev/null +++ b/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift @@ -0,0 +1,73 @@ +// +// ToolchainCheckResult.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +package enum ToolchainCheckResult { + /// Filename for the bundle's recorded build-toolchain version. + private static let toolchainStampFilename = "swift-version.txt" + + /// Bundle stamp matches the local `swift --version` exactly. + case match + /// `/swift-version.txt` is missing (older bundle that predates + /// the stamp). skit prints a one-line note and proceeds. + case stampMissing + case mismatch(bundle: String, local: String) +} + +extension ToolchainCheckResult { + /// Compares `/swift-version.txt` to the caller-captured + /// `swiftVersion` string. The swiftmodule format isn't reliably + /// forward-compatible across even patch-level Swift releases (originating + /// bug: 6.3.0 → 6.3.2 rejected the swiftmodule), so the comparison is + /// exact-string after normalising trailing whitespace. + package init(libPath: String, swiftVersion: String?) { + let stampURL = URL(fileURLWithPath: libPath) + .appendingPathComponent(Self.toolchainStampFilename) + guard let stampData = try? Data(contentsOf: stampURL), + let stampRaw = String(data: stampData, encoding: .utf8) + else { + FileHandle.standardError.write( + Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) + ) + self = .stampMissing + return + } + guard let localRaw = swiftVersion else { + FileHandle.standardError.write( + Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) + ) + self = .stampMissing + return + } + let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) + let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) + self = bundle == local ? .match : .mismatch(bundle: bundle, local: local) + } +} diff --git a/Sources/skit/WrappedSource.swift b/Sources/SyntaxKit/Execution/WrappedSource.swift similarity index 97% rename from Sources/skit/WrappedSource.swift rename to Sources/SyntaxKit/Execution/WrappedSource.swift index d421086..af48b2f 100644 --- a/Sources/skit/WrappedSource.swift +++ b/Sources/SyntaxKit/Execution/WrappedSource.swift @@ -37,7 +37,7 @@ import SwiftSyntax /// /// The body is fenced in `#sourceLocation` directives so compiler diagnostics in /// the body reference the original input file and line numbers. -internal struct WrappedSource { +package struct WrappedSource { /// The original input path, used for the `#sourceLocation` directive. private let originalPath: String /// Top-level `import` declarations hoisted above the wrapper body, each already @@ -52,7 +52,7 @@ internal struct WrappedSource { /// remaining body along with the line it begins on. Everything before the first /// non-import statement that *is* an import gets hoisted; anything before that /// which is *not* an import stays in the body (e.g. a leading `// comment`). - internal init(source: String, originalPath: String) { + package init(source: String, originalPath: String) { self.originalPath = originalPath // Parse the input with SwiftSyntax. The location converter is needed to @@ -92,7 +92,7 @@ internal struct WrappedSource { /// A complete Swift program that imports SyntaxKit, runs the body inside a /// `Group { … }` builder, and prints the generated code. - internal var rendered: String { + package var rendered: String { // Render the hoisted-imports block. Trailing newline only if non-empty so // the wrapper doesn't grow an extra blank line in the common no-imports // case. diff --git a/Sources/skit/Bundle+ResolveLibPath.swift b/Sources/skit/Bundle+ResolveLibPath.swift deleted file mode 100644 index bcced95..0000000 --- a/Sources/skit/Bundle+ResolveLibPath.swift +++ /dev/null @@ -1,82 +0,0 @@ -// -// Bundle+ResolveLibPath.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -#if canImport(Subprocess) - - import Foundation - - extension Bundle { - /// Resolves a directory containing `libSyntaxKit.dylib` + swiftmodules. - /// - /// Tries each non-nil entry in `candidates` in order; if any non-nil - /// candidate is not a SyntaxKit lib dir, throws `CLIError`. If every - /// candidate is absent, falls back to bundle-relative paths derived - /// from `executableURL`: - /// - `/lib` (adjacent layout) - /// - `/../lib/skit` (Homebrew layout) - internal func resolveLibPath(candidates: String?...) throws -> String { - let fileManager = FileManager.default - - for candidate in candidates { - guard let candidate else { continue } - guard fileManager.isLibDir(candidate) else { - throw CLIError(message: "path does not look like a SyntaxKit lib dir: \(candidate)") - } - return candidate - } - - if let execURL = executableURL?.resolvingSymlinksInPath() { - let execDir = execURL.deletingLastPathComponent() - - let adjacent = execDir.appendingPathComponent("lib").path - if fileManager.isLibDir(adjacent) { - return adjacent - } - - let brewLayout = execDir.deletingLastPathComponent() - .appendingPathComponent("lib/skit").path - if fileManager.isLibDir(brewLayout) { - return brewLayout - } - } - - throw CLIError( - message: """ - Could not locate SyntaxKit lib directory. Looked for: - 1. explicit candidates (none provided or all empty) - 2. /lib/ (not found) - 3. /../lib/skit/ (not found) - Run Scripts/build-skit-release.sh to produce a self-contained - release bundle under .build/skit-release/. - """ - ) - } - } - -#endif diff --git a/Sources/skit/ContentHasher.swift b/Sources/skit/ContentHasher.swift deleted file mode 100644 index 39227b5..0000000 --- a/Sources/skit/ContentHasher.swift +++ /dev/null @@ -1,63 +0,0 @@ -// -// ContentHasher.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -#if canImport(Subprocess) - - import Foundation - - /// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed - /// cache keys. The cache keys aren't security-critical — there's no - /// adversary trying to forge a collision — so we don't need a cryptographic - /// hash. 64 bits of output gives ~10⁻⁹ collision probability at 10⁶ cache - /// entries, which is well past anything we'll see in practice. - /// - /// FNV-1a is deterministic across processes and platforms (unlike the Swift - /// stdlib `Hasher`, whose seed is randomized per-process) — that - /// determinism is what makes it usable as an on-disk cache key. - internal struct ContentHasher { - private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 - private static let prime: UInt64 = 0x0000_0100_0000_01b3 - - private var state: UInt64 = ContentHasher.offsetBasis - - internal mutating func update(data: Data) { - for byte in data { - state ^= UInt64(byte) - state &*= ContentHasher.prime - } - } - - /// Returns the hash as a 16-char lowercase-hex string suitable for use as - /// a directory name. - internal func finalize() -> String { - String(format: "%016x", state) - } - } - -#endif diff --git a/Sources/skit/OutputCache.swift b/Sources/skit/OutputCache.swift deleted file mode 100644 index ccc4b51..0000000 --- a/Sources/skit/OutputCache.swift +++ /dev/null @@ -1,161 +0,0 @@ -// -// OutputCache.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -#if canImport(Subprocess) - - import Foundation - - /// On-disk cache of rendered skit output, content-keyed so a re-run on - /// unchanged input skips the `swift` spawn entirely. - /// - /// `@unchecked Sendable` because the stored `FileManager` / `ProcessInfo` - /// are reference types that don't auto-derive Sendable, but the default - /// singletons used in production (and the typical test doubles) are - /// thread-safe for the operations we invoke. The single instance is - /// shared across concurrent `runOne` tasks in directory mode. - internal struct OutputCache: Sendable { - /// Bumped when the cache layout changes in a way that requires invalidation. - private static let schemaVersion = "v1" - - /// Home-relative cache root used when `XDG_CACHE_HOME` is unset: macOS - /// `~/Library/Caches/...`, else Linux `~/.cache/syntaxkit`. The home dir is - /// fixed for the process lifetime, so this is computed once. - private static let defaultCacheRoot: URL = { - let home = NSHomeDirectory() - #if os(macOS) - return URL(fileURLWithPath: home) - .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") - #else - return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") - #endif - }() - - /// `/outputs/`. Populated once at init; per-key - /// directories are derived from it on demand. - private let root: URL - private let fileManager: @Sendable () -> FileManager - private let processInfo: ProcessInfo - - /// Verbatim `swift --version` output captured once for the lifetime of - /// this cache, so per-input key derivation doesn't re-spawn `swift`. - /// nil if capture failed before construction. - private let swiftVersion: String? - - internal init( - swiftVersion: String?, - fileManager: @autoclosure @escaping @Sendable () -> FileManager = .default, - processInfo: ProcessInfo = .processInfo - ) { - self.root = processInfo.syntaxKitCacheRoot(default: Self.defaultCacheRoot) - .appendingPathComponent("outputs") - self.swiftVersion = swiftVersion - self.fileManager = fileManager - self.processInfo = processInfo - } - - /// 64-bit content hash over (schema version, input source bytes, swift - /// version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any - /// change in these inputs produces a fresh key and forces a recompile. - /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. - internal func key(forInput source: String, libPath: String) -> String { - var hasher = ContentHasher() - // Schema version: bump to invalidate every existing cache entry at once. - hasher.update(data: Data(Self.schemaVersion.utf8)) - // Input source bytes: the primary driver of the key. - hasher.update(data: Data(source.utf8)) - - // Toolchain version. Different `swift` builds emit different bytes for - // the same DSL input. - if let swiftVersion { - hasher.update(data: Data(swiftVersion.utf8)) - } - // libSyntaxKit stamp. A rebuilt dylib can change the rendered output - // even without a Swift-version bump. - if let stamp = fileManager().libStamp(libPath: libPath) { - hasher.update(data: Data(stamp.utf8)) - } - - // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and - // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. - let env = processInfo.environment - .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } - .sorted { $0.key < $1.key } - for (key, value) in env { - hasher.update(data: Data("\(key)=\(value)\0".utf8)) - } - - return hasher.finalize() - } - - /// Returns the cached rendered output for `key`, or nil on miss. - internal func lookup(key: String) -> Data? { - try? Data(contentsOf: directory(for: key).appendingPathComponent("output.swift")) - } - - /// Atomically stores `data` under `key`. Concurrent writers race via a - /// `tmp../` staging dir + rename; the loser drops their copy. - internal func store(key: String, data: Data) throws { - let cacheRoot = directory(for: key) - let final = cacheRoot.appendingPathComponent("output.swift") - - // Ensure the parent of the cache key dir exists. The key dir itself is - // installed by the atomic rename below. - try fileManager().createDirectory( - at: cacheRoot.deletingLastPathComponent(), - withIntermediateDirectories: true - ) - - // Stage the payload in a per-pid + uuid sibling dir so it can be renamed - // into place as a single atomic step. - let staging = cacheRoot.deletingLastPathComponent() - .appendingPathComponent( - "tmp.\(processInfo.processIdentifier).\(UUID().uuidString)" - ) - try fileManager().createDirectory(at: staging, withIntermediateDirectories: true) - try data.write(to: staging.appendingPathComponent("output.swift")) - - // Atomic rename into the cache path. If a peer already populated this - // key, swallow the rename error and drop our staging copy. Re-throw only - // if the destination is still missing afterwards. - do { - try fileManager().moveItem(at: staging, to: cacheRoot) - } catch { - try? fileManager().removeItem(at: staging) - if !fileManager().fileExists(atPath: final.path) { - throw error - } - } - } - - private func directory(for key: String) -> URL { - root.appendingPathComponent(key) - } - } - -#endif diff --git a/Sources/skit/RunError.swift b/Sources/skit/RunError.swift new file mode 100644 index 0000000..fd62786 --- /dev/null +++ b/Sources/skit/RunError.swift @@ -0,0 +1,48 @@ +// +// RunError.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + /// Typed error surfaced by `Runner`. It decouples the renderer from the CLI: + /// `Runner` reports *what* went wrong, and `Skit.Run.run` decides the process + /// exit code (so the engine can also be driven in-process from a library). + internal enum RunError: Error { + /// The input path was invalid — missing, or a directory given without `-o`. + case invalidInput(String) + /// Single-file mode: the spawned `swift` exited non-zero. Carries that code + /// (e.g. a compile failure, `124` on timeout, `128 + signal`). + case renderFailed(exitCode: Int32) + /// Directory mode: at least one input failed; the batch exit code is `1`. + case batchFailed + /// A wrapped Foundation/Subprocess failure (file read/write, spawn error) + /// that has no dedicated exit-code mapping. + case unexpected(any Error) + } + +#endif diff --git a/Sources/skit/RunInput.swift b/Sources/skit/RunInput.swift index 64d1465..af09086 100644 --- a/Sources/skit/RunInput.swift +++ b/Sources/skit/RunInput.swift @@ -29,7 +29,6 @@ #if canImport(Subprocess) - import ArgumentParser import Foundation /// Whether a `skit run` input path resolves to a single `.swift` file or a @@ -44,16 +43,16 @@ case directory(inputDir: String, outputDir: String) /// Classifies `input` by stat: existing directory → `.directory`, - /// existing file → `.singleFile`. Throws a `ValidationError` if the path + /// existing file → `.singleFile`. Throws `RunError.invalidInput` if the path /// doesn't exist, or if a directory input wasn't given an explicit `-o`. - internal static func resolve(input: String, output: String?) throws -> RunInput { + internal static func resolve(input: String, output: String?) throws(RunError) -> RunInput { var isDirectory: ObjCBool = false guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { - throw ValidationError("input does not exist: \(input)") + throw RunError.invalidInput("input does not exist: \(input)") } if isDirectory.boolValue { guard let output else { - throw ValidationError("directory inputs require -o ") + throw RunError.invalidInput("directory inputs require -o ") } return .directory(inputDir: input, outputDir: output) } diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift index b959ebd..3e1bcdb 100644 --- a/Sources/skit/Runner.swift +++ b/Sources/skit/Runner.swift @@ -29,9 +29,9 @@ #if canImport(Subprocess) - import ArgumentParser import Foundation import Subprocess + import SyntaxKit // Run lifecycle (per `skit run` invocation): // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) @@ -64,19 +64,28 @@ // MARK: - Dispatch - /// Classifies `input` (single file vs. directory) and renders it. Directory - /// mode surfaces its batch exit code via `ExitCode` (so a partial-failure - /// batch returns 1); single-file mode renders to file/stdout and may call - /// `exit()` directly on a non-zero subprocess result. Throws a - /// `ValidationError` if the path doesn't exist, or if a directory input - /// wasn't given an explicit `-o`. - func callAsFunction(input: String, output: String?) async throws { + /// Classifies `input` (single file vs. directory) and renders it, reporting + /// failures via the typed `RunError` so the caller — not this engine — owns + /// the process exit status. Directory mode throws `.batchFailed` on a + /// partial-failure batch; single-file mode throws `.renderFailed` on a + /// non-zero subprocess result. `.invalidInput` propagates from + /// `RunInput.resolve`; any Foundation/Subprocess failure is wrapped in + /// `.unexpected`. On success it returns normally (exit 0). + internal func callAsFunction(input: String, output: String?) async throws(RunError) { switch try RunInput.resolve(input: input, output: output) { case .directory(let inputDir, let outputDir): let exitCode = await runDirectory(inputDir: inputDir, outputDir: outputDir) - throw ExitCode(exitCode) + if exitCode != 0 { + throw RunError.batchFailed + } case .singleFile(let inputPath, let outputPath): - try await runSingleFile(inputPath: inputPath, outputPath: outputPath) + do { + try await runSingleFile(inputPath: inputPath, outputPath: outputPath) + } catch let error as RunError { + throw error + } catch { + throw RunError.unexpected(error) + } } } @@ -84,9 +93,9 @@ /// Runs `processFile` on a single input and writes its rendered Swift to /// `outputPath` (or stdout when nil). Any stderr from the spawned `swift` - /// is surfaced verbatim. On a non-zero subprocess exit, calls `exit()` - /// directly — the caller in `Skit.Run.run()` won't see a thrown error in - /// that path. + /// is surfaced verbatim. On a non-zero subprocess exit, throws + /// `RunError.renderFailed` carrying that code — the caller in `Skit.Run.run` + /// maps it to the process exit. We don't write partial output in that case. private func runSingleFile(inputPath: String, outputPath: String?) async throws { // Render the input. `processFile` may hit the output cache and skip the // spawn entirely; either way the result has the same shape. @@ -95,10 +104,10 @@ if !result.stderr.isEmpty { FileHandle.standardError.write(Data(result.stderr.utf8)) } - // Non-zero subprocess exit propagates as a process exit. We don't write - // partial output in that case. + // Non-zero subprocess exit is reported as a typed failure carrying the + // code; the command layer turns it into the process exit status. guard result.exitCode == 0 else { - exit(result.exitCode) + throw RunError.renderFailed(exitCode: result.exitCode) } // Deliver the rendered output to file or stdout. if let outputPath { diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index 391f4a3..ccd7766 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -29,6 +29,7 @@ import ArgumentParser import Foundation +import SyntaxKit extension Skit { /// Render one or more SyntaxKit DSL files into Swift source. @@ -133,8 +134,9 @@ extension Skit { // 6. Hand the input off to the runner: it classifies single-file vs. // directory mode (validating existence and the `-o` requirement) and - // renders accordingly, surfacing batch exit codes via ExitCode. - try await runner(input: input, output: output) + // renders accordingly, reporting failures via the typed `RunError`. + // The command layer owns the mapping from failure to process exit. + try await render(using: runner, input: input, output: output) #else // Subprocess is the only backend skit knows how to use to spawn // `swift`/`swiftc`. Without it (Windows, embedded), `run` cannot work. @@ -152,6 +154,24 @@ extension Skit { import Subprocess extension Skit.Run { + /// Drives `runner` and translates its typed `RunError` into the process + /// exit behaviour the CLI promises: usage errors surface via ArgumentParser's + /// `ValidationError` (exit 64), render/batch failures via `ExitCode`, and + /// anything unexpected is rethrown for ArgumentParser to print (exit 1). + fileprivate func render(using runner: Runner, input: String, output: String?) async throws { + do { + try await runner(input: input, output: output) + } catch .invalidInput(let message) { + throw ValidationError(message) + } catch .renderFailed(let exitCode) { + throw ExitCode(exitCode) + } catch .batchFailed { + throw ExitCode(1) + } catch .unexpected(let underlying) { + throw underlying + } + } + /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. fileprivate func captureSwiftVersion() async -> String? { let result = try? await Subprocess.run( diff --git a/Sources/skit/ToolchainCheckResult.swift b/Sources/skit/ToolchainCheckResult.swift deleted file mode 100644 index fc45380..0000000 --- a/Sources/skit/ToolchainCheckResult.swift +++ /dev/null @@ -1,77 +0,0 @@ -// -// ToolchainCheckResult.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -#if canImport(Subprocess) - - import Foundation - - internal enum ToolchainCheckResult { - /// Filename for the bundle's recorded build-toolchain version. - private static let toolchainStampFilename = "swift-version.txt" - - /// Bundle stamp matches the local `swift --version` exactly. - case match - /// `/swift-version.txt` is missing (older bundle that predates - /// the stamp). skit prints a one-line note and proceeds. - case stampMissing - case mismatch(bundle: String, local: String) - } - - extension ToolchainCheckResult { - /// Compares `/swift-version.txt` to the caller-captured - /// `swiftVersion` string. The swiftmodule format isn't reliably - /// forward-compatible across even patch-level Swift releases (originating - /// bug: 6.3.0 → 6.3.2 rejected the swiftmodule), so the comparison is - /// exact-string after normalising trailing whitespace. - internal init(libPath: String, swiftVersion: String?) { - let stampURL = URL(fileURLWithPath: libPath) - .appendingPathComponent(Self.toolchainStampFilename) - guard let stampData = try? Data(contentsOf: stampURL), - let stampRaw = String(data: stampData, encoding: .utf8) - else { - FileHandle.standardError.write( - Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) - ) - self = .stampMissing - return - } - guard let localRaw = swiftVersion else { - FileHandle.standardError.write( - Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) - ) - self = .stampMissing - return - } - let bundle = stampRaw.trimmingCharacters(in: .whitespacesAndNewlines) - let local = localRaw.trimmingCharacters(in: .whitespacesAndNewlines) - self = bundle == local ? .match : .mismatch(bundle: bundle, local: local) - } - } - -#endif From f7a144be68e92151307e6db7f061a2ce03ecf315 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Mon, 8 Jun 2026 18:56:57 -0400 Subject: [PATCH 41/56] Move render engine into SyntaxKit; inject Subprocess backend via closure [skip ci] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Relocate Runner, RunInput, and RunError from skit into SyntaxKit/Execution so the DSL-render engine is usable as a library, not locked to the CLI. Runner no longer depends on Subprocess: it takes an injected `run` closure (SwiftInvocation -> ProcessResult) and skit supplies the Subprocess-backed implementation. skit-side Subprocess seam, factored into extensions: - TerminationStatus.exitCode — collapse to one Int32 (128 + signal convention). - Subprocess.Configuration.runSwift(for:) — build the swift config, run it, and map the record into a ProcessResult (the `run` closure handed to Runner). - Skit+Run no longer hand-rolls the spawn; it passes the extension as the backend. SyntaxKit-side cleanup: - New SwiftInvocation value type (libPath + wrappedPath) replaces the generic `Configuration` name that clashed with Subprocess.Configuration; own file. - SwiftRunOutcome simplified to `.completed(ProcessResult)`; dead Foundation import dropped. - Split directory/batch mode into Runner+Directory.swift, bringing Runner.swift back under the file-length limit; removed leftover commented code and fixed spacing. Build clean; all 398 tests pass (incl. SkitSubprocessTimeoutTests). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../Execution}/RunError.swift | 2 +- .../Execution}/RunInput.swift | 0 .../Execution/Runner+Directory.swift | 161 ++++++++ Sources/SyntaxKit/Execution/Runner.swift | 224 ++++++++++ .../SyntaxKit/Execution/SwiftInvocation.swift | 37 ++ .../SyntaxKit/Execution/SwiftRunOutcome.swift | 4 +- Sources/skit/Runner.swift | 383 ------------------ Sources/skit/Skit+Run.swift | 11 +- .../skit/Subprocess.Configuration+Swift.swift | 27 ++ Sources/skit/TerminationStatus+ExitCode.swift | 49 +++ 10 files changed, 508 insertions(+), 390 deletions(-) rename Sources/{skit => SyntaxKit/Execution}/RunError.swift (98%) rename Sources/{skit => SyntaxKit/Execution}/RunInput.swift (100%) create mode 100644 Sources/SyntaxKit/Execution/Runner+Directory.swift create mode 100644 Sources/SyntaxKit/Execution/Runner.swift create mode 100644 Sources/SyntaxKit/Execution/SwiftInvocation.swift delete mode 100644 Sources/skit/Runner.swift create mode 100644 Sources/skit/TerminationStatus+ExitCode.swift diff --git a/Sources/skit/RunError.swift b/Sources/SyntaxKit/Execution/RunError.swift similarity index 98% rename from Sources/skit/RunError.swift rename to Sources/SyntaxKit/Execution/RunError.swift index fd62786..c3f8059 100644 --- a/Sources/skit/RunError.swift +++ b/Sources/SyntaxKit/Execution/RunError.swift @@ -32,7 +32,7 @@ /// Typed error surfaced by `Runner`. It decouples the renderer from the CLI: /// `Runner` reports *what* went wrong, and `Skit.Run.run` decides the process /// exit code (so the engine can also be driven in-process from a library). - internal enum RunError: Error { +package enum RunError: Error { /// The input path was invalid — missing, or a directory given without `-o`. case invalidInput(String) /// Single-file mode: the spawned `swift` exited non-zero. Carries that code diff --git a/Sources/skit/RunInput.swift b/Sources/SyntaxKit/Execution/RunInput.swift similarity index 100% rename from Sources/skit/RunInput.swift rename to Sources/SyntaxKit/Execution/RunInput.swift diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift new file mode 100644 index 0000000..f9686ae --- /dev/null +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -0,0 +1,161 @@ +// +// Runner+Directory.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +extension Runner { + /// Walks `inputDir` for `.swift` inputs, processes them concurrently (up to + /// the active core count), and mirrors the rendered output into `outputDir`. + /// A failure on one input does not abort the batch — successful peers are + /// still written. Returns 0 if every input succeeded, 1 otherwise. + internal func runDirectory(inputDir: String, outputDir: String) async -> Int32 { + let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL + let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL + + // Phase 1: enumerate inputs. + let inputs: [URL] + do { + inputs = try Self.collectInputs(at: inputURL) + } catch { + FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) + return 1 + } + + if inputs.isEmpty { + FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) + return 0 + } + + // Phase 2: bounded-concurrency processing. Cap is the active core count + // so a 200-file batch doesn't fork 200 simultaneous `swift` processes. + let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) + + var outcomes: [FileOutcome] = [] + var iterator = inputs.makeIterator() + + await withTaskGroup(of: FileOutcome.self) { group in + // Seed the group up to the concurrency cap… + for _ in 0.. FileOutcome { + do { + let result = try await processFile(inputPath: input.path) + return FileOutcome(input: input, result: .success(result)) + } catch { + return FileOutcome(input: input, result: .failure(error)) + } + } + + /// Returns every `.swift` file under `inputDir` (recursive), sorted, with + /// hidden files and files prefixed by `_` removed. Sorted output keeps + /// batch behaviour deterministic across runs. + private static func collectInputs(at inputDir: URL) throws -> [URL] { + guard + let enumerator = FileManager.default.enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], + options: [.skipsHiddenFiles] + ) + else { + throw CLIError(message: "could not enumerate \(inputDir.path)") + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + // Directories aren't outputs. + if values.isDirectory == true { continue } + // Filter for `.swift` regular files, skipping the `_`-prefixed + // convention for "not an input" sources. + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } + } +} diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift new file mode 100644 index 0000000..ec794df --- /dev/null +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -0,0 +1,224 @@ +// +// Runner.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +// Run lifecycle (per `skit run` invocation): +// 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) +// 2. ToolchainCheckResult.init — compare bundle stamp to `swift --version` +// 3. Runner.runSingleFile / .runDirectory — single- or batch-input mode +// 4. processFile (per input) — load → cache lookup → wrap → spawn → cache store +// 5. wrap — hoist imports, wrap body in Group { … }, #sourceLocation +// 6. runSwift — spawn `swift` with timeout watchdog +// See Docs/skit.md for design rationale and trade-offs. + +/// Renders SyntaxKit DSL inputs into Swift source, holding the per-invocation +/// configuration (`libPath`, `cache`, `timeoutSeconds`) so the individual +/// inputs don't have to thread it through every call. Constructed once per +/// `skit run` in `Skit.Run.run`; `Sendable` so a single value can be shared +/// across the concurrent `runOne` tasks in directory mode. +package struct Runner: Sendable { + /// Directory holding `libSyntaxKit.{dylib,so}` + swiftmodules; reused for + /// the spawned `swift`'s `-I`/`-L`/`-rpath` flags. + private let libPath: String + /// Output cache shared across every input, or nil under `--no-cache`. + private let cache: OutputCache? + /// Per-input watchdog in seconds; `0` opts out of the timeout race. + private let timeoutSeconds: Int + /// Backend that actually spawns `swift` for one `SwiftInvocation`. Injected + /// by the caller (skit supplies a Subprocess-based implementation). + private let run: @Sendable (SwiftInvocation) async throws -> ProcessResult + + package init( + libPath: String, + cache: OutputCache?, + timeoutSeconds: Int, + run: @Sendable @escaping (SwiftInvocation) async throws -> ProcessResult + ) { + self.libPath = libPath + self.cache = cache + self.timeoutSeconds = timeoutSeconds + self.run = run + } + + // MARK: - Dispatch + + /// Classifies `input` (single file vs. directory) and renders it, reporting + /// failures via the typed `RunError` so the caller — not this engine — owns + /// the process exit status. Directory mode throws `.batchFailed` on a + /// partial-failure batch; single-file mode throws `.renderFailed` on a + /// non-zero subprocess result. `.invalidInput` propagates from + /// `RunInput.resolve`; any Foundation/Subprocess failure is wrapped in + /// `.unexpected`. On success it returns normally (exit 0). + package func callAsFunction(input: String, output: String?) async throws(RunError) { + switch try RunInput.resolve(input: input, output: output) { + case .directory(let inputDir, let outputDir): + let exitCode = await runDirectory(inputDir: inputDir, outputDir: outputDir) + if exitCode != 0 { + throw RunError.batchFailed + } + case .singleFile(let inputPath, let outputPath): + do { + try await runSingleFile(inputPath: inputPath, outputPath: outputPath) + } catch let error as RunError { + throw error + } catch { + throw RunError.unexpected(error) + } + } + } + + // MARK: - Single-file mode + + /// Runs `processFile` on a single input and writes its rendered Swift to + /// `outputPath` (or stdout when nil). Any stderr from the spawned `swift` + /// is surfaced verbatim. On a non-zero subprocess exit, throws + /// `RunError.renderFailed` carrying that code — the caller in `Skit.Run.run` + /// maps it to the process exit. We don't write partial output in that case. + private func runSingleFile(inputPath: String, outputPath: String?) async throws { + // Render the input. `processFile` may hit the output cache and skip the + // spawn entirely; either way the result has the same shape. + let result = try await processFile(inputPath: inputPath) + // Surface diagnostics from the spawned `swift` before deciding success. + if !result.stderr.isEmpty { + FileHandle.standardError.write(Data(result.stderr.utf8)) + } + // Non-zero subprocess exit is reported as a typed failure carrying the + // code; the command layer turns it into the process exit status. + guard result.exitCode == 0 else { + throw RunError.renderFailed(exitCode: result.exitCode) + } + // Deliver the rendered output to file or stdout. + if let outputPath { + try result.stdout.write(to: URL(fileURLWithPath: outputPath)) + } else { + FileHandle.standardOutput.write(result.stdout) + } + } + + // MARK: - Per-file work + + /// The per-input render pipeline: load source → consult the output cache → + /// (on miss) wrap → spawn `swift` → rewrite diagnostics → store the result + /// in the cache. The temp wrapper file is created in a per-run tmp dir and + /// torn down by `defer` whether the spawn succeeded or not. `internal` so + /// the directory-mode extension (`Runner+Directory.swift`) can reuse it. + internal func processFile(inputPath: String) async throws -> ProcessResult { + // Load the input source. Anything past this point keys off these bytes. + let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL + let absoluteInputPath = inputURL.path + let source = try String(contentsOf: inputURL, encoding: .utf8) + + // Compute the output cache key (nil under `--no-cache` or when the cache + // root couldn't be derived at startup). Mixes input bytes, toolchain + // version, libSyntaxKit stamp, and sorted SKIT_*/SYNTAXKIT_* env vars + // — see `OutputCache.key(forInput:libPath:)`. + let cacheKey: String? = cache?.key(forInput: source, libPath: libPath) + // Cache hit: skip the wrap+spawn entirely and return the stored output. + if let cache, let cacheKey, let cached = cache.lookup(key: cacheKey) { + return ProcessResult(exitCode: 0, stdout: cached, stderr: "") + } + + // Wrap the user's input into a complete Swift program that imports + // SyntaxKit, runs the body inside a Group { … } builder, and prints the + // result. See `WrappedSource` for the exact template. + let wrapped = WrappedSource(source: source, originalPath: absoluteInputPath).rendered + + // Spill the wrapped program to a per-invocation temp dir. The dir is + // cleaned up unconditionally so a failed spawn doesn't leak files. + let tmpDir = FileManager.default.temporaryDirectory + .appendingPathComponent("skit-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: tmpDir) } + + let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) + + // Spawn `swift` on the wrapped file (with timeout watchdog). stdout is + // the rendered Swift source; stderr is compiler diagnostics, if any. + let raw = try await runSwift(wrappedPath: wrappedURL.path) + // #sourceLocation maps body diagnostics back to the input file. Errors in + // the preamble (lines outside the body) still reference the wrapper — + // rewrite literal occurrences of its path so users see something coherent. + let stderr = raw.stderr.replacingOccurrences( + of: wrappedURL.path, + with: absoluteInputPath + ) + + // Store on the way out. `try?` is deliberate: a cache write failure is + // not a render failure. The next run will simply miss and re-spawn. + if let cache, let cacheKey, raw.exitCode == 0 { + try? cache.store(key: cacheKey, data: raw.stdout) + } + + return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) + } + + // MARK: - Spawning swift + + /// Exit code returned when the spawned `swift` is killed by skit's timeout + /// watchdog. Matches POSIX `timeout(1)`. + private static let timeoutExitCode: Int32 = 124 + + /// Spawns `swift` (via the injected `run` backend) on the wrapped input file. + /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing + /// task group; the loser is cancelled. On timeout, returns exit 124 with + /// a one-line stderr message — matching POSIX `timeout(1)`'s convention. + private func runSwift(wrappedPath: String) async throws -> ProcessResult { + let invocation = SwiftInvocation(libPath: libPath, wrappedPath: wrappedPath) + + // The actual backend call, wrapped in a closure so the task-group race + // below can hold a single Sendable reference to it. + let operation: @Sendable () async throws -> SwiftRunOutcome = { + .completed(try await self.run(invocation)) + } + + // Race the invocation against a sleep watchdog; whichever finishes first + // wins, the other is cancelled. `timeoutSeconds <= 0` opts out of the + // race entirely (useful for debugging genuinely long codegen). + let outcome: SwiftRunOutcome + if timeoutSeconds <= 0 { + outcome = try await operation() + } else { + outcome = try await Task.timeout(.seconds(timeoutSeconds), operation: operation) ?? .timedOut + } + + // Normalize both outcomes into a single ProcessResult shape. + switch outcome { + case .completed(let result): + return result + case .timedOut: + return ProcessResult( + exitCode: Self.timeoutExitCode, + stdout: Data(), + stderr: "skit: timed out after \(timeoutSeconds)s\n" + ) + } + } +} diff --git a/Sources/SyntaxKit/Execution/SwiftInvocation.swift b/Sources/SyntaxKit/Execution/SwiftInvocation.swift new file mode 100644 index 0000000..71cc05c --- /dev/null +++ b/Sources/SyntaxKit/Execution/SwiftInvocation.swift @@ -0,0 +1,37 @@ +// +// SwiftInvocation.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +/// The inputs needed to invoke `swift` on one wrapped DSL program: the lib +/// directory (for `-I`/`-L`/`-rpath`) and the path to the wrapped source. +/// `Runner` hands this to its `run` closure, which performs the actual spawn — +/// the seam that keeps the engine free of any Subprocess dependency. +package struct SwiftInvocation: Sendable { + package let libPath: String + package let wrappedPath: String +} diff --git a/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift b/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift index 1e8404c..068e4da 100644 --- a/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift +++ b/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift @@ -27,12 +27,10 @@ // OTHER DEALINGS IN THE SOFTWARE. // -package import Foundation - /// Either the spawned `swift` ran to completion (success or failure) or /// the watchdog elapsed first. The completed payload is normalized to the /// shape callers want regardless of platform. package enum SwiftRunOutcome: Sendable { - case completed(exitCode: Int32, stdout: Data, stderr: String) + case completed(ProcessResult) case timedOut } diff --git a/Sources/skit/Runner.swift b/Sources/skit/Runner.swift deleted file mode 100644 index 3e1bcdb..0000000 --- a/Sources/skit/Runner.swift +++ /dev/null @@ -1,383 +0,0 @@ -// -// Runner.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -#if canImport(Subprocess) - - import Foundation - import Subprocess - import SyntaxKit - - // Run lifecycle (per `skit run` invocation): - // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) - // 2. ToolchainCheckResult.init — compare bundle stamp to `swift --version` - // 3. Runner.runSingleFile / .runDirectory — single- or batch-input mode - // 4. processFile (per input) — load → cache lookup → wrap → spawn → cache store - // 5. wrap — hoist imports, wrap body in Group { … }, #sourceLocation - // 6. runSwift — spawn `swift` with timeout watchdog - // See Docs/skit.md for design rationale and trade-offs. - - /// Renders SyntaxKit DSL inputs into Swift source, holding the per-invocation - /// configuration (`libPath`, `cache`, `timeoutSeconds`) so the individual - /// inputs don't have to thread it through every call. Constructed once per - /// `skit run` in `Skit.Run.run`; `Sendable` so a single value can be shared - /// across the concurrent `runOne` tasks in directory mode. - internal struct Runner: Sendable { - /// Directory holding `libSyntaxKit.{dylib,so}` + swiftmodules; reused for - /// the spawned `swift`'s `-I`/`-L`/`-rpath` flags. - private let libPath: String - /// Output cache shared across every input, or nil under `--no-cache`. - private let cache: OutputCache? - /// Per-input watchdog in seconds; `0` opts out of the timeout race. - private let timeoutSeconds: Int - - internal init(libPath: String, cache: OutputCache?, timeoutSeconds: Int) { - self.libPath = libPath - self.cache = cache - self.timeoutSeconds = timeoutSeconds - } - - // MARK: - Dispatch - - /// Classifies `input` (single file vs. directory) and renders it, reporting - /// failures via the typed `RunError` so the caller — not this engine — owns - /// the process exit status. Directory mode throws `.batchFailed` on a - /// partial-failure batch; single-file mode throws `.renderFailed` on a - /// non-zero subprocess result. `.invalidInput` propagates from - /// `RunInput.resolve`; any Foundation/Subprocess failure is wrapped in - /// `.unexpected`. On success it returns normally (exit 0). - internal func callAsFunction(input: String, output: String?) async throws(RunError) { - switch try RunInput.resolve(input: input, output: output) { - case .directory(let inputDir, let outputDir): - let exitCode = await runDirectory(inputDir: inputDir, outputDir: outputDir) - if exitCode != 0 { - throw RunError.batchFailed - } - case .singleFile(let inputPath, let outputPath): - do { - try await runSingleFile(inputPath: inputPath, outputPath: outputPath) - } catch let error as RunError { - throw error - } catch { - throw RunError.unexpected(error) - } - } - } - - // MARK: - Single-file mode - - /// Runs `processFile` on a single input and writes its rendered Swift to - /// `outputPath` (or stdout when nil). Any stderr from the spawned `swift` - /// is surfaced verbatim. On a non-zero subprocess exit, throws - /// `RunError.renderFailed` carrying that code — the caller in `Skit.Run.run` - /// maps it to the process exit. We don't write partial output in that case. - private func runSingleFile(inputPath: String, outputPath: String?) async throws { - // Render the input. `processFile` may hit the output cache and skip the - // spawn entirely; either way the result has the same shape. - let result = try await processFile(inputPath: inputPath) - // Surface diagnostics from the spawned `swift` before deciding success. - if !result.stderr.isEmpty { - FileHandle.standardError.write(Data(result.stderr.utf8)) - } - // Non-zero subprocess exit is reported as a typed failure carrying the - // code; the command layer turns it into the process exit status. - guard result.exitCode == 0 else { - throw RunError.renderFailed(exitCode: result.exitCode) - } - // Deliver the rendered output to file or stdout. - if let outputPath { - try result.stdout.write(to: URL(fileURLWithPath: outputPath)) - } else { - FileHandle.standardOutput.write(result.stdout) - } - } - - // MARK: - Folder mode - - /// Walks `inputDir` for `.swift` inputs, processes them concurrently (up to - /// the active core count), and mirrors the rendered output into `outputDir`. - /// A failure on one input does not abort the batch — successful peers are - /// still written. Returns 0 if every input succeeded, 1 otherwise. - private func runDirectory(inputDir: String, outputDir: String) async -> Int32 { - let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL - let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL - - // Phase 1: enumerate inputs. - let inputs: [URL] - do { - inputs = try Self.collectInputs(at: inputURL) - } catch { - FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) - return 1 - } - - if inputs.isEmpty { - FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) - return 0 - } - - // Phase 2: bounded-concurrency processing. Cap is the active core count - // so a 200-file batch doesn't fork 200 simultaneous `swift` processes. - let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) - - var outcomes: [FileOutcome] = [] - var iterator = inputs.makeIterator() - - await withTaskGroup(of: FileOutcome.self) { group in - // Seed the group up to the concurrency cap… - for _ in 0.. FileOutcome { - do { - let result = try await processFile(inputPath: input.path) - return FileOutcome(input: input, result: .success(result)) - } catch { - return FileOutcome(input: input, result: .failure(error)) - } - } - - /// Returns every `.swift` file under `inputDir` (recursive), sorted, with - /// hidden files and files prefixed by `_` removed. Sorted output keeps - /// batch behaviour deterministic across runs. - private static func collectInputs(at inputDir: URL) throws -> [URL] { - guard - let enumerator = FileManager.default.enumerator( - at: inputDir, - includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], - options: [.skipsHiddenFiles] - ) - else { - throw CLIError(message: "could not enumerate \(inputDir.path)") - } - - var result: [URL] = [] - for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) - // Directories aren't outputs. - if values.isDirectory == true { continue } - // Filter for `.swift` regular files, skipping the `_`-prefixed - // convention for "not an input" sources. - guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } - result.append(url.standardizedFileURL) - } - return result.sorted { $0.path < $1.path } - } - - // MARK: - Per-file work - - /// The per-input render pipeline: load source → consult the output cache → - /// (on miss) wrap → spawn `swift` → rewrite diagnostics → store the result - /// in the cache. The temp wrapper file is created in a per-run tmp dir and - /// torn down by `defer` whether the spawn succeeded or not. - private func processFile(inputPath: String) async throws -> ProcessResult { - // Load the input source. Anything past this point keys off these bytes. - let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL - let absoluteInputPath = inputURL.path - let source = try String(contentsOf: inputURL, encoding: .utf8) - - // Compute the output cache key (nil under `--no-cache` or when the cache - // root couldn't be derived at startup). Mixes input bytes, toolchain - // version, libSyntaxKit stamp, and sorted SKIT_*/SYNTAXKIT_* env vars - // — see `OutputCache.key(forInput:libPath:)`. - let cacheKey: String? = cache?.key(forInput: source, libPath: libPath) - // Cache hit: skip the wrap+spawn entirely and return the stored output. - if let cache, let cacheKey, let cached = cache.lookup(key: cacheKey) { - return ProcessResult(exitCode: 0, stdout: cached, stderr: "") - } - - // Wrap the user's input into a complete Swift program that imports - // SyntaxKit, runs the body inside a Group { … } builder, and prints the - // result. See `WrappedSource` for the exact template. - let wrapped = WrappedSource(source: source, originalPath: absoluteInputPath).rendered - - // Spill the wrapped program to a per-invocation temp dir. The dir is - // cleaned up unconditionally so a failed spawn doesn't leak files. - let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skit-\(UUID().uuidString)") - try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) - defer { try? FileManager.default.removeItem(at: tmpDir) } - - let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") - try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) - - // Spawn `swift` on the wrapped file (with timeout watchdog). stdout is - // the rendered Swift source; stderr is compiler diagnostics, if any. - let raw = try await runSwift(wrappedPath: wrappedURL.path) - // #sourceLocation maps body diagnostics back to the input file. Errors in - // the preamble (lines outside the body) still reference the wrapper — - // rewrite literal occurrences of its path so users see something coherent. - let stderr = raw.stderr.replacingOccurrences( - of: wrappedURL.path, - with: absoluteInputPath - ) - - // Store on the way out. `try?` is deliberate: a cache write failure is - // not a render failure. The next run will simply miss and re-spawn. - if let cache, let cacheKey, raw.exitCode == 0 { - try? cache.store(key: cacheKey, data: raw.stdout) - } - - return ProcessResult(exitCode: raw.exitCode, stdout: raw.stdout, stderr: stderr) - } - - // MARK: - Spawning swift - - /// Exit code returned when the spawned `swift` is killed by skit's timeout - /// watchdog. Matches POSIX `timeout(1)`. - private static let timeoutExitCode: Int32 = 124 - - /// Bounded output capacity for the spawned `swift` (16 MiB). Generated DSL - /// output above this size is exotic; if we ever hit it we'll see a clear - /// SubprocessError rather than a silent truncation. - private static let stdoutLimitBytes: Int = 16 * 1_024 * 1_024 - private static let stderrLimitBytes: Int = 1 * 1_024 * 1_024 - - /// Spawns `swift` (script-mode interpreter) on the wrapped input file. - /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing - /// task group; the loser is cancelled. On timeout, returns exit 124 with - /// a one-line stderr message — matching POSIX `timeout(1)`'s convention. - private func runSwift(wrappedPath: String) async throws -> ProcessResult { - // Build the `swift` invocation (executable + link/include/rpath flags). - let configuration = Subprocess.Configuration.swift(libPath: libPath, wrappedPath: wrappedPath) - - // The actual subprocess call, wrapped in a closure so the task-group race - // below can hold a single Sendable reference to it. - let invocation: @Sendable () async throws -> SwiftRunOutcome = { - let record = try await Subprocess.run( - configuration, - output: .string(limit: Self.stdoutLimitBytes), - error: .string(limit: Self.stderrLimitBytes) - ) - return .completed( - exitCode: Self.exitCode(from: record.terminationStatus), - stdout: Data((record.standardOutput ?? "").utf8), - stderr: record.standardError ?? "" - ) - } - - // Race the invocation against a sleep watchdog; whichever finishes first - // wins, the other is cancelled. `timeoutSeconds <= 0` opts out of the - // race entirely (useful for debugging genuinely long codegen). - let outcome: SwiftRunOutcome - if timeoutSeconds <= 0 { - outcome = try await invocation() - } else { - outcome = - try await Task.timeout(.seconds(timeoutSeconds), operation: invocation) - ?? .timedOut - } - - // Normalize both outcomes into a single ProcessResult shape. - switch outcome { - case .completed(let exitCode, let stdout, let stderr): - return ProcessResult(exitCode: exitCode, stdout: stdout, stderr: stderr) - case .timedOut: - return ProcessResult( - exitCode: Self.timeoutExitCode, - stdout: Data(), - stderr: "skit: timed out after \(timeoutSeconds)s\n" - ) - } - } - - /// Collapses Subprocess's `TerminationStatus` into a single Int32 exit code, - /// using the shell convention (128 + signal number) for signalled deaths. - private static func exitCode(from status: TerminationStatus) -> Int32 { - switch status { - case .exited(let code): - return Int32(truncatingIfNeeded: code) - #if !os(Windows) - case .signaled(let signal): - // Match shell convention: 128 + signal number. - return 128 + Int32(truncatingIfNeeded: signal) - #endif - } - } - } - -#endif diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index ccd7766..5f7487e 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -128,9 +128,14 @@ extension Skit { // doesn't re-spawn `swift`. let cache: OutputCache? = noCache ? nil : OutputCache(swiftVersion: swiftVersion) - // 5. Bind the per-invocation configuration into a Runner so the input - // orchestration doesn't have to thread libPath/cache/timeout around. - let runner = Runner(libPath: libPath, cache: cache, timeoutSeconds: timeoutSeconds) + // 5. Bind the per-invocation configuration into a Runner. skit supplies + // the Subprocess-backed `run` closure — the one seam between the + // platform-agnostic engine in SyntaxKit and the Subprocess backend. + let runner = Runner( + libPath: libPath, + cache: cache, + timeoutSeconds: timeoutSeconds + ) { try await Subprocess.Configuration.runSwift(for: $0) } // 6. Hand the input off to the runner: it classifies single-file vs. // directory mode (validating existence and the `-o` requirement) and diff --git a/Sources/skit/Subprocess.Configuration+Swift.swift b/Sources/skit/Subprocess.Configuration+Swift.swift index 744f164..95e37ca 100644 --- a/Sources/skit/Subprocess.Configuration+Swift.swift +++ b/Sources/skit/Subprocess.Configuration+Swift.swift @@ -29,9 +29,17 @@ #if canImport(Subprocess) + import Foundation import Subprocess + import SyntaxKit extension Subprocess.Configuration { + /// Bounded output capacity for the spawned `swift` (16 MiB stdout / 1 MiB + /// stderr). Output above this is exotic; we'd surface a clear SubprocessError + /// rather than silently truncate. + private static let stdoutLimitBytes = 16 * 1_024 * 1_024 + private static let stderrLimitBytes = 1 * 1_024 * 1_024 + /// A configuration that runs the `swift` interpreter on `wrappedPath`, /// linked against `libSyntaxKit` in `libPath`. /// @@ -51,6 +59,25 @@ ] return Self(executable: .name("swift"), arguments: Arguments(arguments)) } + + /// Spawns `swift` for the render `invocation` and normalizes the result + /// into a `ProcessResult`. This is the Subprocess backend skit hands to + /// `Runner` as its `run` closure — the one seam between the (platform- + /// agnostic) engine in SyntaxKit and the Subprocess implementation. + internal static func runSwift( + for invocation: SyntaxKit.SwiftInvocation + ) async throws -> ProcessResult { + let record = try await Subprocess.run( + .swift(libPath: invocation.libPath, wrappedPath: invocation.wrappedPath), + output: .string(limit: stdoutLimitBytes), + error: .string(limit: stderrLimitBytes) + ) + return ProcessResult( + exitCode: record.terminationStatus.exitCode, + stdout: Data((record.standardOutput ?? "").utf8), + stderr: record.standardError ?? "" + ) + } } #endif diff --git a/Sources/skit/TerminationStatus+ExitCode.swift b/Sources/skit/TerminationStatus+ExitCode.swift new file mode 100644 index 0000000..73bafc3 --- /dev/null +++ b/Sources/skit/TerminationStatus+ExitCode.swift @@ -0,0 +1,49 @@ +// +// Subprocess.TerminationStatus+ExitCode.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + import Subprocess + + extension TerminationStatus { + /// Collapses the termination status into a single exit code, using the + /// shell convention (128 + signal number) for signalled deaths. + internal var exitCode: Int32 { + switch self { + case .exited(let code): + return Int32(truncatingIfNeeded: code) + #if !os(Windows) + case .signaled(let signal): + return 128 + Int32(truncatingIfNeeded: signal) + #endif + } + } + } + +#endif From eb460af4c8920eb9f86061bc2deddec4127df56e Mon Sep 17 00:00:00 2001 From: leogdion Date: Mon, 8 Jun 2026 19:52:14 -0400 Subject: [PATCH 42/56] Reshape Runner into an SDK-style API; move CLI presentation to skit [skip ci] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Runner.callAsFunction (Int32-coded dispatch with embedded stdout/stderr writes) is replaced by two typed methods: renderFile -> SingleFileRender and renderDirectory -> DirectoryRender. Both throw typed RunError; the SDK layer no longer touches FileHandle.standardError/standardOutput, and single-file mode no longer writes its output to disk — the caller does. RunError.renderFailed now carries the toolchain stderr alongside the exit code, and RunError.batchFailed (a CLI-shaped sentinel) is gone in favor of inspecting DirectoryRender.failureCount. DirectoryRender models per-input outcomes (Result); the internal TaskGroup payload is renamed RenderTaskResult, made fileprivate to Runner+Directory.swift, and its error type tightened from any Error to RunError (runOne wraps Foundation/Subprocess throws as .unexpected). Skit+Run.swift takes on the CLI presentation the runner shed: stdout/ stderr writes, fenced per-input diagnostics, batch summary line, walk-failure framing, the "no .swift inputs" warning, and the mapping from per-input failures to ExitCode(1). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../SyntaxKit/Execution/DirectoryRender.swift | 82 ++++++++++++ Sources/SyntaxKit/Execution/RunError.swift | 21 ++-- Sources/SyntaxKit/Execution/RunInput.swift | 20 +-- .../Execution/Runner+Directory.swift | 118 ++++++++++++------ Sources/SyntaxKit/Execution/Runner.swift | 82 +++++------- ...leOutcome.swift => SingleFileRender.swift} | 22 ++-- Sources/skit/Skit+Run.swift | 116 +++++++++++++++-- 7 files changed, 333 insertions(+), 128 deletions(-) create mode 100644 Sources/SyntaxKit/Execution/DirectoryRender.swift rename Sources/SyntaxKit/Execution/{FileOutcome.swift => SingleFileRender.swift} (64%) diff --git a/Sources/SyntaxKit/Execution/DirectoryRender.swift b/Sources/SyntaxKit/Execution/DirectoryRender.swift new file mode 100644 index 0000000..71aca81 --- /dev/null +++ b/Sources/SyntaxKit/Execution/DirectoryRender.swift @@ -0,0 +1,82 @@ +// +// DirectoryRender.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +#if canImport(Subprocess) + + package import Foundation + + /// Result of `Runner.renderDirectory`. Successful rendered outputs are + /// written into the mirrored output tree as the batch progresses; this + /// value is the post-batch summary the caller inspects to decide + /// presentation (logging, exit code, etc.). Per-input failures are + /// captured here, not thrown, so a single bad input doesn't tear the + /// batch down. + package struct DirectoryRender: Sendable { + /// Per-input result. `stderr` carries the (possibly path-rewritten) + /// diagnostics from the spawned `swift`; it may be present in both + /// success and failure cases (e.g. a successful render that emitted + /// warnings). `result` is `.success` when the rendered output was + /// written to `destination`; `.failure` when the input could not be + /// rendered or its output could not be written. + package struct FileOutcome: Sendable { + package let input: URL + package let destination: URL + package let stderr: String + package let result: Result + + package init( + input: URL, + destination: URL, + stderr: String, + result: Result + ) { + self.input = input + self.destination = destination + self.stderr = stderr + self.result = result + } + } + + package let outcomes: [FileOutcome] + + package init(outcomes: [FileOutcome]) { + self.outcomes = outcomes + } + + /// Number of inputs whose `result` is `.failure` — the signal the caller + /// uses to map a partially-failed batch to a non-zero exit (or whatever + /// failure semantics fit the host). + package var failureCount: Int { + outcomes.reduce(into: 0) { count, outcome in + if case .failure = outcome.result { count += 1 } + } + } + } + +#endif diff --git a/Sources/SyntaxKit/Execution/RunError.swift b/Sources/SyntaxKit/Execution/RunError.swift index c3f8059..991d8cf 100644 --- a/Sources/SyntaxKit/Execution/RunError.swift +++ b/Sources/SyntaxKit/Execution/RunError.swift @@ -29,19 +29,20 @@ #if canImport(Subprocess) - /// Typed error surfaced by `Runner`. It decouples the renderer from the CLI: - /// `Runner` reports *what* went wrong, and `Skit.Run.run` decides the process - /// exit code (so the engine can also be driven in-process from a library). + /// Typed error surfaced by `Runner`. It decouples the renderer from any + /// particular caller: `Runner` reports *what* went wrong, and the caller + /// (CLI, build plugin, in-process driver) decides how to present it. package enum RunError: Error { - /// The input path was invalid — missing, or a directory given without `-o`. + /// The input path was invalid — missing, or a directory given without an + /// output directory. case invalidInput(String) - /// Single-file mode: the spawned `swift` exited non-zero. Carries that code - /// (e.g. a compile failure, `124` on timeout, `128 + signal`). - case renderFailed(exitCode: Int32) - /// Directory mode: at least one input failed; the batch exit code is `1`. - case batchFailed + /// Single-file render: the spawned `swift` exited non-zero. Carries that + /// code (e.g. a compile failure, `124` on timeout, `128 + signal`) and + /// the (path-rewritten) stderr the toolchain emitted, so the caller can + /// surface diagnostics without having to fish them out elsewhere. + case renderFailed(exitCode: Int32, stderr: String) /// A wrapped Foundation/Subprocess failure (file read/write, spawn error) - /// that has no dedicated exit-code mapping. + /// that has no dedicated mapping. case unexpected(any Error) } diff --git a/Sources/SyntaxKit/Execution/RunInput.swift b/Sources/SyntaxKit/Execution/RunInput.swift index af09086..24d728b 100644 --- a/Sources/SyntaxKit/Execution/RunInput.swift +++ b/Sources/SyntaxKit/Execution/RunInput.swift @@ -31,21 +31,23 @@ import Foundation - /// Whether a `skit run` input path resolves to a single `.swift` file or a - /// directory of them — the two modes `Runner` dispatches into. Built by - /// `resolve(input:output:)`, which stats the path and enforces the - /// per-mode output rules so `Skit.Run.run` can `switch` on a settled value - /// instead of juggling an `ObjCBool`. - internal enum RunInput { - /// A single input file rendered to `outputPath`, or stdout when nil. + /// Whether an input path resolves to a single `.swift` file or a directory + /// of them — the two modes a `Runner` caller dispatches into. Built by + /// `resolve(input:output:)`, which stats the path and enforces the per-mode + /// output rules so the caller can `switch` on a settled value instead of + /// juggling an `ObjCBool`. + package enum RunInput { + /// A single input file. `outputPath` is whatever the caller intends to + /// do with the rendered bytes (write to a file, ignore, etc.); `Runner` + /// itself does not act on it. case singleFile(inputPath: String, outputPath: String?) /// A directory of inputs mirrored into `outputDir` (always required). case directory(inputDir: String, outputDir: String) /// Classifies `input` by stat: existing directory → `.directory`, /// existing file → `.singleFile`. Throws `RunError.invalidInput` if the path - /// doesn't exist, or if a directory input wasn't given an explicit `-o`. - internal static func resolve(input: String, output: String?) throws(RunError) -> RunInput { + /// doesn't exist, or if a directory input wasn't given an explicit output. + package static func resolve(input: String, output: String?) throws(RunError) -> RunInput { var isDirectory: ObjCBool = false guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { throw RunError.invalidInput("input does not exist: \(input)") diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index f9686ae..bba055f 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -31,35 +31,43 @@ import Foundation extension Runner { /// Walks `inputDir` for `.swift` inputs, processes them concurrently (up to - /// the active core count), and mirrors the rendered output into `outputDir`. - /// A failure on one input does not abort the batch — successful peers are - /// still written. Returns 0 if every input succeeded, 1 otherwise. - internal func runDirectory(inputDir: String, outputDir: String) async -> Int32 { + /// the active core count), and mirrors successfully-rendered outputs into + /// `outputDir`. A failure on one input does not abort the batch — successful + /// peers are still written. Returns a `DirectoryRender` with per-input + /// outcomes; the caller inspects `failureCount` (and per-outcome stderr) to + /// decide presentation. + /// + /// Throws `RunError.unexpected` only for bulk failures the SDK can't + /// recover from (e.g. the input directory can't be enumerated). An empty + /// input set is *not* an error — the result simply has no outcomes. + package func renderDirectory( + inputDir: String, + outputDir: String + ) async throws(RunError) -> DirectoryRender { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL - // Phase 1: enumerate inputs. + // Phase 1: enumerate inputs. A walk failure is a bulk failure: there's + // nothing per-file to report, so it surfaces as a typed throw. let inputs: [URL] do { inputs = try Self.collectInputs(at: inputURL) } catch { - FileHandle.standardError.write(Data("skit: failed to walk \(inputDir): \(error)\n".utf8)) - return 1 + throw RunError.unexpected(error) } if inputs.isEmpty { - FileHandle.standardError.write(Data("skit: no .swift inputs under \(inputDir)\n".utf8)) - return 0 + return DirectoryRender(outcomes: []) } // Phase 2: bounded-concurrency processing. Cap is the active core count // so a 200-file batch doesn't fork 200 simultaneous `swift` processes. let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) - var outcomes: [FileOutcome] = [] + var renderResults: [RenderTaskResult] = [] var iterator = inputs.makeIterator() - await withTaskGroup(of: FileOutcome.self) { group in + await withTaskGroup(of: RenderTaskResult.self) { group in // Seed the group up to the concurrency cap… for _ in 0.. FileOutcome { + /// `processFile`'s heterogeneous Foundation/Subprocess throws are wrapped + /// in `RunError.unexpected` here so the rest of the pipeline sees a single + /// typed error. + private func runOne(_ input: URL) async -> RenderTaskResult { do { let result = try await processFile(inputPath: input.path) - return FileOutcome(input: input, result: .success(result)) + return RenderTaskResult(input: input, result: .success(result)) + } catch let error as RunError { + return RenderTaskResult(input: input, result: .failure(error)) } catch { - return FileOutcome(input: input, result: .failure(error)) + return RenderTaskResult(input: input, result: .failure(.unexpected(error))) } } @@ -159,3 +190,12 @@ extension Runner { return result.sorted { $0.path < $1.path } } } + +/// Payload the per-input render `TaskGroup` yields back to `renderDirectory`. +/// Failures are captured (not thrown) so a single bad input doesn't tear down +/// the group; `processFile`'s heterogeneous Foundation/Subprocess throws are +/// normalized into `RunError` (typically `.unexpected`) by `runOne`. +private struct RenderTaskResult: Sendable { + let input: URL + let result: Result +} diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index ec794df..c9bab7d 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -29,20 +29,23 @@ import Foundation -// Run lifecycle (per `skit run` invocation): +// Render lifecycle (per `Runner` call): // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) // 2. ToolchainCheckResult.init — compare bundle stamp to `swift --version` -// 3. Runner.runSingleFile / .runDirectory — single- or batch-input mode +// 3. Runner.renderFile / .renderDirectory — single- or batch-input mode // 4. processFile (per input) — load → cache lookup → wrap → spawn → cache store // 5. wrap — hoist imports, wrap body in Group { … }, #sourceLocation // 6. runSwift — spawn `swift` with timeout watchdog // See Docs/skit.md for design rationale and trade-offs. -/// Renders SyntaxKit DSL inputs into Swift source, holding the per-invocation -/// configuration (`libPath`, `cache`, `timeoutSeconds`) so the individual -/// inputs don't have to thread it through every call. Constructed once per -/// `skit run` in `Skit.Run.run`; `Sendable` so a single value can be shared -/// across the concurrent `runOne` tasks in directory mode. +/// Renders SyntaxKit DSL inputs into Swift source. `Runner` is the SDK-shaped +/// entry point: methods return rendered data (`SingleFileRender`) or +/// structured per-file outcomes (`DirectoryRender`) and throw typed +/// `RunError`s; callers — CLI, build plugin, in-process driver — decide what +/// to do with stdout, stderr, exit codes, and so on. +/// +/// Constructed once per render session; `Sendable` so a single value can be +/// shared across the concurrent per-input tasks in directory mode. package struct Runner: Sendable { /// Directory holding `libSyntaxKit.{dylib,so}` + swiftmodules; reused for /// the spawned `swift`'s `-I`/`-L`/`-rpath` flags. @@ -67,59 +70,32 @@ package struct Runner: Sendable { self.run = run } - // MARK: - Dispatch - - /// Classifies `input` (single file vs. directory) and renders it, reporting - /// failures via the typed `RunError` so the caller — not this engine — owns - /// the process exit status. Directory mode throws `.batchFailed` on a - /// partial-failure batch; single-file mode throws `.renderFailed` on a - /// non-zero subprocess result. `.invalidInput` propagates from - /// `RunInput.resolve`; any Foundation/Subprocess failure is wrapped in - /// `.unexpected`. On success it returns normally (exit 0). - package func callAsFunction(input: String, output: String?) async throws(RunError) { - switch try RunInput.resolve(input: input, output: output) { - case .directory(let inputDir, let outputDir): - let exitCode = await runDirectory(inputDir: inputDir, outputDir: outputDir) - if exitCode != 0 { - throw RunError.batchFailed - } - case .singleFile(let inputPath, let outputPath): - do { - try await runSingleFile(inputPath: inputPath, outputPath: outputPath) - } catch let error as RunError { - throw error - } catch { - throw RunError.unexpected(error) - } - } - } - // MARK: - Single-file mode - /// Runs `processFile` on a single input and writes its rendered Swift to - /// `outputPath` (or stdout when nil). Any stderr from the spawned `swift` - /// is surfaced verbatim. On a non-zero subprocess exit, throws - /// `RunError.renderFailed` carrying that code — the caller in `Skit.Run.run` - /// maps it to the process exit. We don't write partial output in that case. - private func runSingleFile(inputPath: String, outputPath: String?) async throws { + /// Renders one input and returns the rendered bytes plus any compiler + /// diagnostics. No file IO, no stdout/stderr writes — the caller decides + /// where the result goes. + /// + /// On a non-zero subprocess exit, throws `RunError.renderFailed(exitCode: + /// stderr:)` carrying the toolchain's diagnostic. Any Foundation/Subprocess + /// failure (file read, spawn) is wrapped in `RunError.unexpected`. + package func renderFile(input: String) async throws(RunError) -> SingleFileRender { // Render the input. `processFile` may hit the output cache and skip the // spawn entirely; either way the result has the same shape. - let result = try await processFile(inputPath: inputPath) - // Surface diagnostics from the spawned `swift` before deciding success. - if !result.stderr.isEmpty { - FileHandle.standardError.write(Data(result.stderr.utf8)) + let result: ProcessResult + do { + result = try await processFile(inputPath: input) + } catch let error as RunError { + throw error + } catch { + throw RunError.unexpected(error) } - // Non-zero subprocess exit is reported as a typed failure carrying the - // code; the command layer turns it into the process exit status. + // Non-zero subprocess exit is reported as a typed failure carrying both + // the code and the (path-rewritten) toolchain diagnostic. guard result.exitCode == 0 else { - throw RunError.renderFailed(exitCode: result.exitCode) - } - // Deliver the rendered output to file or stdout. - if let outputPath { - try result.stdout.write(to: URL(fileURLWithPath: outputPath)) - } else { - FileHandle.standardOutput.write(result.stdout) + throw RunError.renderFailed(exitCode: result.exitCode, stderr: result.stderr) } + return SingleFileRender(stdout: result.stdout, stderr: result.stderr) } // MARK: - Per-file work diff --git a/Sources/SyntaxKit/Execution/FileOutcome.swift b/Sources/SyntaxKit/Execution/SingleFileRender.swift similarity index 64% rename from Sources/SyntaxKit/Execution/FileOutcome.swift rename to Sources/SyntaxKit/Execution/SingleFileRender.swift index cf9a07a..6673934 100644 --- a/Sources/SyntaxKit/Execution/FileOutcome.swift +++ b/Sources/SyntaxKit/Execution/SingleFileRender.swift @@ -1,5 +1,5 @@ // -// FileOutcome.swift +// SingleFileRender.swift // SyntaxKit // // Created by Leo Dion. @@ -29,14 +29,18 @@ package import Foundation -/// Result of processing one input in directory mode. The error case is -/// stored (not thrown) so the batch can keep going. -package struct FileOutcome: Sendable { - package let input: URL - package let result: Result +/// Result of `Runner.renderFile`. Both fields may be populated on success — +/// the spawned `swift` can emit warnings to `stderr` alongside a valid +/// `stdout`. The caller decides where the bytes go (file, stdout, in-memory). +package struct SingleFileRender: Sendable { + /// Rendered Swift source produced by the wrapped program. + package let stdout: Data + /// Compiler diagnostics from the spawned `swift`, with the wrapper path + /// rewritten back to the original input. Empty when the toolchain was silent. + package let stderr: String - package init(input: URL, result: Result) { - self.input = input - self.result = result + package init(stdout: Data, stderr: String) { + self.stdout = stdout + self.stderr = stderr } } diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index 5f7487e..2b8d197 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -159,22 +159,122 @@ extension Skit { import Subprocess extension Skit.Run { - /// Drives `runner` and translates its typed `RunError` into the process - /// exit behaviour the CLI promises: usage errors surface via ArgumentParser's - /// `ValidationError` (exit 64), render/batch failures via `ExitCode`, and - /// anything unexpected is rethrown for ArgumentParser to print (exit 1). + /// Classifies `input` and dispatches to the matching `Runner` SDK method. + /// The SDK layer is silent — this function (and its helpers) own all + /// stdout/stderr presentation, file IO for single-file mode, and the + /// mapping from `RunError`/batch failures to the process exit behaviour + /// the CLI promises: `ValidationError` (exit 64) for usage errors, + /// `ExitCode` for render/batch failures, and rethrow for anything + /// unexpected (ArgumentParser prints + exit 1). fileprivate func render(using runner: Runner, input: String, output: String?) async throws { + let resolved: RunInput do { - try await runner(input: input, output: output) + resolved = try RunInput.resolve(input: input, output: output) } catch .invalidInput(let message) { throw ValidationError(message) - } catch .renderFailed(let exitCode) { - throw ExitCode(exitCode) - } catch .batchFailed { + } catch { + // RunInput.resolve only throws .invalidInput; defensive. throw ExitCode(1) + } + + switch resolved { + case .singleFile(let inputPath, let outputPath): + try await renderSingle(runner: runner, input: inputPath, output: outputPath) + case .directory(let inputDir, let outputDir): + try await renderBatch(runner: runner, input: inputDir, output: outputDir) + } + } + + /// Renders one input via `Runner.renderFile`, prints any compiler stderr, + /// and writes the rendered Swift source either to `outputPath` or to + /// stdout. The runner itself does none of that — that's the CLI's job. + fileprivate func renderSingle( + runner: Runner, + input: String, + output: String? + ) async throws { + let rendered: SingleFileRender + do { + rendered = try await runner.renderFile(input: input) + } catch .invalidInput(let message) { + throw ValidationError(message) + } catch .renderFailed(let exitCode, let stderr) { + if !stderr.isEmpty { + FileHandle.standardError.write(Data(stderr.utf8)) + } + throw ExitCode(exitCode) } catch .unexpected(let underlying) { throw underlying } + + if !rendered.stderr.isEmpty { + FileHandle.standardError.write(Data(rendered.stderr.utf8)) + } + if let output { + try rendered.stdout.write(to: URL(fileURLWithPath: output)) + } else { + FileHandle.standardOutput.write(rendered.stdout) + } + } + + /// Renders a directory batch via `Runner.renderDirectory`, prints per-input + /// diagnostics (fenced when several files emit them), surfaces non-render + /// failures, prints a one-line summary, and maps any failures to + /// `ExitCode(1)` — Tuist-analog batch semantics. + fileprivate func renderBatch( + runner: Runner, + input: String, + output: String + ) async throws { + let result: DirectoryRender + do { + result = try await runner.renderDirectory(inputDir: input, outputDir: output) + } catch .invalidInput(let message) { + throw ValidationError(message) + } catch .unexpected(let underlying) { + // `renderDirectory` only wraps directory-walk failures in `.unexpected`; + // surface the original "failed to walk" framing the CLI used to print + // before the SDK split. + FileHandle.standardError.write( + Data("skit: failed to walk \(input): \(underlying)\n".utf8) + ) + throw ExitCode(1) + } catch { + // renderDirectory does not throw .renderFailed — that's a per-file + // outcome. Defensive. + throw ExitCode(1) + } + + if result.outcomes.isEmpty { + FileHandle.standardError.write(Data("skit: no .swift inputs under \(input)\n".utf8)) + return + } + + for outcome in result.outcomes { + if !outcome.stderr.isEmpty { + FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) + FileHandle.standardError.write(Data(outcome.stderr.utf8)) + } + // .renderFailed already had its stderr surfaced above. Other failures + // (process spawn, write) carry the diagnostic in the error itself. + if case .failure(let error) = outcome.result, case .renderFailed = error { + continue + } + if case .failure(let error) = outcome.result { + FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) + } + } + + FileHandle.standardError.write( + Data( + "skit: \(result.outcomes.count - result.failureCount)/\(result.outcomes.count) succeeded\n" + .utf8 + ) + ) + + if result.failureCount > 0 { + throw ExitCode(1) + } } /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. From 3f80dde7d9a3716df44319eb915cc86ce84d6df9 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 9 Jun 2026 00:25:43 -0400 Subject: [PATCH 43/56] Refactor renderDirectory into phase helpers; move collectInputs to FileManager - Move collectInputs onto FileManager as an instance method with typed throws(RunError); call site no longer double-wraps errors. - Extract renderDirectory's bounded-concurrency and per-file write phases into processInputs and writeOutput helpers. - Fix type_contents_order: hoist Runner.timeoutExitCode above instance members; move nested Outcome enum above its test method. - Apply SwiftFormat fixes from the lint pass. Co-Authored-By: Claude Opus 4.8 (1M context) --- Sources/DocumentationHarness/Validator.swift | 6 +- Sources/SyntaxKit/Execution/RunError.swift | 2 +- .../Execution/Runner+Directory.swift | 157 ++++++++++-------- Sources/SyntaxKit/Execution/Runner.swift | 8 +- Sources/skit/TerminationStatus+ExitCode.swift | 2 +- .../SkitSubprocessTimeoutTests.swift | 10 +- 6 files changed, 100 insertions(+), 85 deletions(-) diff --git a/Sources/DocumentationHarness/Validator.swift b/Sources/DocumentationHarness/Validator.swift index 6fc7506..07cc034 100644 --- a/Sources/DocumentationHarness/Validator.swift +++ b/Sources/DocumentationHarness/Validator.swift @@ -33,14 +33,12 @@ package protocol Validator { func validateFile(at fileURL: URL) throws -> [ValidationResult] } -private extension [String] { +extension [String] { /// Default file extensions for documentation files - static let defaultPathExtensions: [String] = ["md"] - + fileprivate static let defaultPathExtensions: [String] = ["md"] } extension Validator { - /// Validates all Swift code examples found in documentation files /// - Parameters: /// - relativePaths: Array of relative paths to search for documentation diff --git a/Sources/SyntaxKit/Execution/RunError.swift b/Sources/SyntaxKit/Execution/RunError.swift index 991d8cf..fe4e1eb 100644 --- a/Sources/SyntaxKit/Execution/RunError.swift +++ b/Sources/SyntaxKit/Execution/RunError.swift @@ -32,7 +32,7 @@ /// Typed error surfaced by `Runner`. It decouples the renderer from any /// particular caller: `Runner` reports *what* went wrong, and the caller /// (CLI, build plugin, in-process driver) decides how to present it. -package enum RunError: Error { + package enum RunError: Error { /// The input path was invalid — missing, or a directory given without an /// output directory. case invalidInput(String) diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index bba055f..025fad1 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -49,22 +49,31 @@ extension Runner { // Phase 1: enumerate inputs. A walk failure is a bulk failure: there's // nothing per-file to report, so it surfaces as a typed throw. - let inputs: [URL] - do { - inputs = try Self.collectInputs(at: inputURL) - } catch { - throw RunError.unexpected(error) + let inputs = try FileManager.default.collectInputs(at: inputURL) + guard !inputs.isEmpty else { + return DirectoryRender(outcomes: []) } - if inputs.isEmpty { - return DirectoryRender(outcomes: []) + // Phase 2: render every input with bounded concurrency. + let renderResults = await processInputs(inputs) + + // Phase 3: write successes and capture a per-input outcome for each. + let outcomes = renderResults.map { + writeOutput(for: $0, inputBase: inputURL, outputBase: outputURL) } - // Phase 2: bounded-concurrency processing. Cap is the active core count - // so a 200-file batch doesn't fork 200 simultaneous `swift` processes. + return DirectoryRender(outcomes: outcomes) + } + + /// Renders every input through `runOne` with bounded concurrency. The cap is + /// the active core count so a 200-file batch doesn't fork 200 simultaneous + /// `swift` processes: the group is seeded up to the cap, then refilled one + /// task per completion until the inputs are exhausted. + private func processInputs(_ inputs: [URL]) async -> [RenderTaskResult] { let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) var renderResults: [RenderTaskResult] = [] + renderResults.reserveCapacity(inputs.count) var iterator = inputs.makeIterator() await withTaskGroup(of: RenderTaskResult.self) { group in @@ -82,67 +91,64 @@ extension Runner { } } - // Phase 3: write outputs and capture per-input outcomes. Successes are - // written even when other files in the batch failed (Tuist-analog batch - // semantics). No diagnostics are printed here; the caller does that. - var outcomes: [DirectoryRender.FileOutcome] = [] - outcomes.reserveCapacity(renderResults.count) - for outcome in renderResults { - let relative = outcome.input.path.dropFirst(inputURL.path.count + 1) - let destination = outputURL.appendingPathComponent(String(relative)) - - switch outcome.result { - case .failure(let error): - outcomes.append( - DirectoryRender.FileOutcome( - input: outcome.input, - destination: destination, - stderr: "", - result: .failure(error) + return renderResults + } + + /// Builds the `FileOutcome` for one render result, writing a successful + /// render's stdout to its mirrored destination under `outputBase`. The write + /// side effect lives here; failures (a non-zero render exit, or a write + /// error) are captured into the returned outcome rather than thrown, so a + /// failing peer doesn't prevent successful files in the batch from being + /// written (Tuist-analog batch semantics). No diagnostics are printed here; + /// the caller does that. + private func writeOutput( + for result: RenderTaskResult, + inputBase: URL, + outputBase: URL + ) -> DirectoryRender.FileOutcome { + let relative = result.input.path.dropFirst(inputBase.path.count + 1) + let destination = outputBase.appendingPathComponent(String(relative)) + + switch result.result { + case .failure(let error): + return DirectoryRender.FileOutcome( + input: result.input, + destination: destination, + stderr: "", + result: .failure(error) + ) + case .success(let processResult): + if processResult.exitCode != 0 { + return DirectoryRender.FileOutcome( + input: result.input, + destination: destination, + stderr: processResult.stderr, + result: .failure( + .renderFailed(exitCode: processResult.exitCode, stderr: processResult.stderr) ) ) - case .success(let processResult): - if processResult.exitCode != 0 { - outcomes.append( - DirectoryRender.FileOutcome( - input: outcome.input, - destination: destination, - stderr: processResult.stderr, - result: .failure( - .renderFailed(exitCode: processResult.exitCode, stderr: processResult.stderr) - ) - ) - ) - continue - } - do { - try FileManager.default.createDirectory( - at: destination.deletingLastPathComponent(), - withIntermediateDirectories: true - ) - try processResult.stdout.write(to: destination) - outcomes.append( - DirectoryRender.FileOutcome( - input: outcome.input, - destination: destination, - stderr: processResult.stderr, - result: .success(()) - ) - ) - } catch { - outcomes.append( - DirectoryRender.FileOutcome( - input: outcome.input, - destination: destination, - stderr: processResult.stderr, - result: .failure(.unexpected(error)) - ) - ) - } + } + do { + try FileManager.default.createDirectory( + at: destination.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + try processResult.stdout.write(to: destination) + return DirectoryRender.FileOutcome( + input: result.input, + destination: destination, + stderr: processResult.stderr, + result: .success(()) + ) + } catch { + return DirectoryRender.FileOutcome( + input: result.input, + destination: destination, + stderr: processResult.stderr, + result: .failure(.unexpected(error)) + ) } } - - return DirectoryRender(outcomes: outcomes) } /// `processFile` adapter that catches errors into the `RenderTaskResult` @@ -160,24 +166,35 @@ extension Runner { return RenderTaskResult(input: input, result: .failure(.unexpected(error))) } } +} +extension FileManager { /// Returns every `.swift` file under `inputDir` (recursive), sorted, with /// hidden files and files prefixed by `_` removed. Sorted output keeps /// batch behaviour deterministic across runs. - private static func collectInputs(at inputDir: URL) throws -> [URL] { + /// + /// Throws `RunError.unexpected` when the directory can't be enumerated or a + /// file's resource values can't be read — both are bulk failures with + /// nothing per-file to report. + internal func collectInputs(at inputDir: URL) throws(RunError) -> [URL] { guard - let enumerator = FileManager.default.enumerator( + let enumerator = enumerator( at: inputDir, includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], options: [.skipsHiddenFiles] ) else { - throw CLIError(message: "could not enumerate \(inputDir.path)") + throw RunError.unexpected(CLIError(message: "could not enumerate \(inputDir.path)")) } var result: [URL] = [] for case let url as URL in enumerator { - let values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + let values: URLResourceValues + do { + values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + } catch { + throw RunError.unexpected(error) + } // Directories aren't outputs. if values.isDirectory == true { continue } // Filter for `.swift` regular files, skipping the `_`-prefixed diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index c9bab7d..2b5640a 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -47,6 +47,10 @@ import Foundation /// Constructed once per render session; `Sendable` so a single value can be /// shared across the concurrent per-input tasks in directory mode. package struct Runner: Sendable { + /// Exit code returned when the spawned `swift` is killed by skit's timeout + /// watchdog. Matches POSIX `timeout(1)`. + private static let timeoutExitCode: Int32 = 124 + /// Directory holding `libSyntaxKit.{dylib,so}` + swiftmodules; reused for /// the spawned `swift`'s `-I`/`-L`/`-rpath` flags. private let libPath: String @@ -158,10 +162,6 @@ package struct Runner: Sendable { // MARK: - Spawning swift - /// Exit code returned when the spawned `swift` is killed by skit's timeout - /// watchdog. Matches POSIX `timeout(1)`. - private static let timeoutExitCode: Int32 = 124 - /// Spawns `swift` (via the injected `run` backend) on the wrapped input file. /// When `timeoutSeconds > 0` the spawn races a sleep task in a throwing /// task group; the loser is cancelled. On timeout, returns exit 124 with diff --git a/Sources/skit/TerminationStatus+ExitCode.swift b/Sources/skit/TerminationStatus+ExitCode.swift index 73bafc3..aa0900e 100644 --- a/Sources/skit/TerminationStatus+ExitCode.swift +++ b/Sources/skit/TerminationStatus+ExitCode.swift @@ -1,5 +1,5 @@ // -// Subprocess.TerminationStatus+ExitCode.swift +// TerminationStatus+ExitCode.swift // SyntaxKit // // Created by Leo Dion. diff --git a/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift b/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift index ebd083e..eba0468 100644 --- a/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift +++ b/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift @@ -48,6 +48,11 @@ // shell pipeline that forks a background `sleep` holding stderr open. @Suite("Subprocess timeout-cancel") internal struct SkitSubprocessTimeoutTests { + private enum Outcome: Equatable, Sendable { + case completed(TerminationStatus) + case timedOut + } + @Test( "cancel-on-timeout completes within a bounded wall-time when grandchildren hold pipe fds" ) @@ -91,11 +96,6 @@ "timeout-cancel took \(elapsed); possible swift-subprocess #256 regression" ) } - - private enum Outcome: Equatable, Sendable { - case completed(TerminationStatus) - case timedOut - } } #endif From 5b06d5f59c30542ae4378b0e5f61f92ebff906ca Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 9 Jun 2026 00:34:26 -0400 Subject: [PATCH 44/56] Move writeOutput onto FileManager; fold render result functionally - Relocate writeOutput to the FileManager extension beside collectInputs, since the directory-create + write are filesystem operations. - Replace the success/failure switch with a functional fold: Result.flatMap maps the render outcome into the write outcome, and the write itself is captured via Result { ... }.mapError(.unexpected). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../Execution/Runner+Directory.swift | 105 ++++++++---------- 1 file changed, 47 insertions(+), 58 deletions(-) diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index 025fad1..d33f360 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -59,7 +59,7 @@ extension Runner { // Phase 3: write successes and capture a per-input outcome for each. let outcomes = renderResults.map { - writeOutput(for: $0, inputBase: inputURL, outputBase: outputURL) + FileManager.default.writeOutput(for: $0, inputBase: inputURL, outputBase: outputURL) } return DirectoryRender(outcomes: outcomes) @@ -94,63 +94,6 @@ extension Runner { return renderResults } - /// Builds the `FileOutcome` for one render result, writing a successful - /// render's stdout to its mirrored destination under `outputBase`. The write - /// side effect lives here; failures (a non-zero render exit, or a write - /// error) are captured into the returned outcome rather than thrown, so a - /// failing peer doesn't prevent successful files in the batch from being - /// written (Tuist-analog batch semantics). No diagnostics are printed here; - /// the caller does that. - private func writeOutput( - for result: RenderTaskResult, - inputBase: URL, - outputBase: URL - ) -> DirectoryRender.FileOutcome { - let relative = result.input.path.dropFirst(inputBase.path.count + 1) - let destination = outputBase.appendingPathComponent(String(relative)) - - switch result.result { - case .failure(let error): - return DirectoryRender.FileOutcome( - input: result.input, - destination: destination, - stderr: "", - result: .failure(error) - ) - case .success(let processResult): - if processResult.exitCode != 0 { - return DirectoryRender.FileOutcome( - input: result.input, - destination: destination, - stderr: processResult.stderr, - result: .failure( - .renderFailed(exitCode: processResult.exitCode, stderr: processResult.stderr) - ) - ) - } - do { - try FileManager.default.createDirectory( - at: destination.deletingLastPathComponent(), - withIntermediateDirectories: true - ) - try processResult.stdout.write(to: destination) - return DirectoryRender.FileOutcome( - input: result.input, - destination: destination, - stderr: processResult.stderr, - result: .success(()) - ) - } catch { - return DirectoryRender.FileOutcome( - input: result.input, - destination: destination, - stderr: processResult.stderr, - result: .failure(.unexpected(error)) - ) - } - } - } - /// `processFile` adapter that catches errors into the `RenderTaskResult` /// so a single failure doesn't tear down the surrounding `TaskGroup`. /// `processFile`'s heterogeneous Foundation/Subprocess throws are wrapped @@ -206,6 +149,52 @@ extension FileManager { } return result.sorted { $0.path < $1.path } } + + /// Builds the `FileOutcome` for one render result, writing a successful + /// render's stdout to its mirrored destination under `outputBase`. The write + /// side effect lives here; failures (a non-zero render exit, or a write + /// error) are folded into the returned outcome rather than thrown, so a + /// failing peer doesn't prevent successful files in the batch from being + /// written (Tuist-analog batch semantics). No diagnostics are printed here; + /// the caller does that. + fileprivate func writeOutput( + for result: RenderTaskResult, + inputBase: URL, + outputBase: URL + ) -> DirectoryRender.FileOutcome { + let relative = result.input.path.dropFirst(inputBase.path.count + 1) + let destination = outputBase.appendingPathComponent(String(relative)) + + // stderr is the toolchain's diagnostics whenever the render produced any — + // i.e. on every successful spawn, regardless of how the write then fares. + let stderr = (try? result.result.get())?.stderr ?? "" + + // Fold the render result into the write result: a render failure passes + // through, a non-zero exit becomes `.renderFailed`, and a clean render is + // committed to disk (capturing any write error as `.unexpected`). + let outcome = result.result.flatMap { processResult -> Result in + guard processResult.exitCode == 0 else { + return .failure( + .renderFailed(exitCode: processResult.exitCode, stderr: processResult.stderr) + ) + } + return Result { + try createDirectory( + at: destination.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + try processResult.stdout.write(to: destination) + } + .mapError(RunError.unexpected) + } + + return DirectoryRender.FileOutcome( + input: result.input, + destination: destination, + stderr: stderr, + result: outcome + ) + } } /// Payload the per-input render `TaskGroup` yields back to `renderDirectory`. From d8d53453fa30e305af16f41c2c169ed4c53b5db0 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 9 Jun 2026 08:19:21 -0400 Subject: [PATCH 45/56] Fix lint: split Skit+Run, drop dead FileOutcome.destination, fix #require - Remove .swift-version pin (unblocks swiftly toolchain resolution) - Move CLI presentation helpers into Skit.Run+Render.swift to satisfy the 300-line file_length cap; add guarded `import Subprocess` back - Drop assign-only `destination` from DirectoryRender.FileOutcome - Use `try #require(...)` form in SkitSubprocessTimeoutTests Co-Authored-By: Claude Opus 4.8 (1M context) --- .swift-version | 1 - .../SyntaxKit/Execution/DirectoryRender.swift | 7 +- .../Execution/Runner+Directory.swift | 1 - Sources/skit/Skit+Run.swift | 160 +-------------- Sources/skit/Skit.Run+Render.swift | 188 ++++++++++++++++++ .../SkitSubprocessTimeoutTests.swift | 2 +- 6 files changed, 195 insertions(+), 164 deletions(-) delete mode 100644 .swift-version create mode 100644 Sources/skit/Skit.Run+Render.swift diff --git a/.swift-version b/.swift-version deleted file mode 100644 index a435f5a..0000000 --- a/.swift-version +++ /dev/null @@ -1 +0,0 @@ -6.1 diff --git a/Sources/SyntaxKit/Execution/DirectoryRender.swift b/Sources/SyntaxKit/Execution/DirectoryRender.swift index 71aca81..92fe1c3 100644 --- a/Sources/SyntaxKit/Execution/DirectoryRender.swift +++ b/Sources/SyntaxKit/Execution/DirectoryRender.swift @@ -42,22 +42,19 @@ /// diagnostics from the spawned `swift`; it may be present in both /// success and failure cases (e.g. a successful render that emitted /// warnings). `result` is `.success` when the rendered output was - /// written to `destination`; `.failure` when the input could not be - /// rendered or its output could not be written. + /// written to its mirrored destination; `.failure` when the input could + /// not be rendered or its output could not be written. package struct FileOutcome: Sendable { package let input: URL - package let destination: URL package let stderr: String package let result: Result package init( input: URL, - destination: URL, stderr: String, result: Result ) { self.input = input - self.destination = destination self.stderr = stderr self.result = result } diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index d33f360..570bf33 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -190,7 +190,6 @@ extension FileManager { return DirectoryRender.FileOutcome( input: result.input, - destination: destination, stderr: stderr, result: outcome ) diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index 2b8d197..d7d076c 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -31,6 +31,10 @@ import ArgumentParser import Foundation import SyntaxKit +#if canImport(Subprocess) + import Subprocess +#endif + extension Skit { /// Render one or more SyntaxKit DSL files into Swift source. /// @@ -153,159 +157,3 @@ extension Skit { } } } - -#if canImport(Subprocess) - - import Subprocess - - extension Skit.Run { - /// Classifies `input` and dispatches to the matching `Runner` SDK method. - /// The SDK layer is silent — this function (and its helpers) own all - /// stdout/stderr presentation, file IO for single-file mode, and the - /// mapping from `RunError`/batch failures to the process exit behaviour - /// the CLI promises: `ValidationError` (exit 64) for usage errors, - /// `ExitCode` for render/batch failures, and rethrow for anything - /// unexpected (ArgumentParser prints + exit 1). - fileprivate func render(using runner: Runner, input: String, output: String?) async throws { - let resolved: RunInput - do { - resolved = try RunInput.resolve(input: input, output: output) - } catch .invalidInput(let message) { - throw ValidationError(message) - } catch { - // RunInput.resolve only throws .invalidInput; defensive. - throw ExitCode(1) - } - - switch resolved { - case .singleFile(let inputPath, let outputPath): - try await renderSingle(runner: runner, input: inputPath, output: outputPath) - case .directory(let inputDir, let outputDir): - try await renderBatch(runner: runner, input: inputDir, output: outputDir) - } - } - - /// Renders one input via `Runner.renderFile`, prints any compiler stderr, - /// and writes the rendered Swift source either to `outputPath` or to - /// stdout. The runner itself does none of that — that's the CLI's job. - fileprivate func renderSingle( - runner: Runner, - input: String, - output: String? - ) async throws { - let rendered: SingleFileRender - do { - rendered = try await runner.renderFile(input: input) - } catch .invalidInput(let message) { - throw ValidationError(message) - } catch .renderFailed(let exitCode, let stderr) { - if !stderr.isEmpty { - FileHandle.standardError.write(Data(stderr.utf8)) - } - throw ExitCode(exitCode) - } catch .unexpected(let underlying) { - throw underlying - } - - if !rendered.stderr.isEmpty { - FileHandle.standardError.write(Data(rendered.stderr.utf8)) - } - if let output { - try rendered.stdout.write(to: URL(fileURLWithPath: output)) - } else { - FileHandle.standardOutput.write(rendered.stdout) - } - } - - /// Renders a directory batch via `Runner.renderDirectory`, prints per-input - /// diagnostics (fenced when several files emit them), surfaces non-render - /// failures, prints a one-line summary, and maps any failures to - /// `ExitCode(1)` — Tuist-analog batch semantics. - fileprivate func renderBatch( - runner: Runner, - input: String, - output: String - ) async throws { - let result: DirectoryRender - do { - result = try await runner.renderDirectory(inputDir: input, outputDir: output) - } catch .invalidInput(let message) { - throw ValidationError(message) - } catch .unexpected(let underlying) { - // `renderDirectory` only wraps directory-walk failures in `.unexpected`; - // surface the original "failed to walk" framing the CLI used to print - // before the SDK split. - FileHandle.standardError.write( - Data("skit: failed to walk \(input): \(underlying)\n".utf8) - ) - throw ExitCode(1) - } catch { - // renderDirectory does not throw .renderFailed — that's a per-file - // outcome. Defensive. - throw ExitCode(1) - } - - if result.outcomes.isEmpty { - FileHandle.standardError.write(Data("skit: no .swift inputs under \(input)\n".utf8)) - return - } - - for outcome in result.outcomes { - if !outcome.stderr.isEmpty { - FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) - FileHandle.standardError.write(Data(outcome.stderr.utf8)) - } - // .renderFailed already had its stderr surfaced above. Other failures - // (process spawn, write) carry the diagnostic in the error itself. - if case .failure(let error) = outcome.result, case .renderFailed = error { - continue - } - if case .failure(let error) = outcome.result { - FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) - } - } - - FileHandle.standardError.write( - Data( - "skit: \(result.outcomes.count - result.failureCount)/\(result.outcomes.count) succeeded\n" - .utf8 - ) - ) - - if result.failureCount > 0 { - throw ExitCode(1) - } - } - - /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. - fileprivate func captureSwiftVersion() async -> String? { - let result = try? await Subprocess.run( - .name("swift"), - arguments: ["--version"], - output: .string(limit: 4_096), - error: .discarded - ) - return result?.standardOutput - } - - /// Human-readable error emitted when the bundle's recorded `swift --version` - /// differs from the local one, explaining why and how to recover. - fileprivate func toolchainMismatchMessage(bundle: String, local: String) -> String { - """ - skit: toolchain mismatch - bundle: \(bundle) - local: \(local) - The bundle's libSyntaxKit was built against a different `swift` than the - one on your PATH. Swift swiftmodules aren't reliably compatible across - versions, so spawning `swift` would fail with a cryptic module-version - diagnostic. - - Rebuild the bundle with: - Scripts/build-skit-release.sh - Or pass --no-toolchain-check to try anyway. - - """ - } - } - -#endif diff --git a/Sources/skit/Skit.Run+Render.swift b/Sources/skit/Skit.Run+Render.swift new file mode 100644 index 0000000..b0a1636 --- /dev/null +++ b/Sources/skit/Skit.Run+Render.swift @@ -0,0 +1,188 @@ +// +// Skit.Run+Render.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import ArgumentParser +import Foundation +import SyntaxKit + +#if canImport(Subprocess) + + import Subprocess + + extension Skit.Run { + /// Classifies `input` and dispatches to the matching `Runner` SDK method. + /// The SDK layer is silent — this function (and its helpers) own all + /// stdout/stderr presentation, file IO for single-file mode, and the + /// mapping from `RunError`/batch failures to the process exit behaviour + /// the CLI promises: `ValidationError` (exit 64) for usage errors, + /// `ExitCode` for render/batch failures, and rethrow for anything + /// unexpected (ArgumentParser prints + exit 1). + internal func render(using runner: Runner, input: String, output: String?) async throws { + let resolved: RunInput + do { + resolved = try RunInput.resolve(input: input, output: output) + } catch .invalidInput(let message) { + throw ValidationError(message) + } catch { + // RunInput.resolve only throws .invalidInput; defensive. + throw ExitCode(1) + } + + switch resolved { + case .singleFile(let inputPath, let outputPath): + try await renderSingle(runner: runner, input: inputPath, output: outputPath) + case .directory(let inputDir, let outputDir): + try await renderBatch(runner: runner, input: inputDir, output: outputDir) + } + } + + /// Renders one input via `Runner.renderFile`, prints any compiler stderr, + /// and writes the rendered Swift source either to `outputPath` or to + /// stdout. The runner itself does none of that — that's the CLI's job. + fileprivate func renderSingle( + runner: Runner, + input: String, + output: String? + ) async throws { + let rendered: SingleFileRender + do { + rendered = try await runner.renderFile(input: input) + } catch .invalidInput(let message) { + throw ValidationError(message) + } catch .renderFailed(let exitCode, let stderr) { + if !stderr.isEmpty { + FileHandle.standardError.write(Data(stderr.utf8)) + } + throw ExitCode(exitCode) + } catch .unexpected(let underlying) { + throw underlying + } + + if !rendered.stderr.isEmpty { + FileHandle.standardError.write(Data(rendered.stderr.utf8)) + } + if let output { + try rendered.stdout.write(to: URL(fileURLWithPath: output)) + } else { + FileHandle.standardOutput.write(rendered.stdout) + } + } + + /// Renders a directory batch via `Runner.renderDirectory`, prints per-input + /// diagnostics (fenced when several files emit them), surfaces non-render + /// failures, prints a one-line summary, and maps any failures to + /// `ExitCode(1)` — Tuist-analog batch semantics. + fileprivate func renderBatch( + runner: Runner, + input: String, + output: String + ) async throws { + let result: DirectoryRender + do { + result = try await runner.renderDirectory(inputDir: input, outputDir: output) + } catch .invalidInput(let message) { + throw ValidationError(message) + } catch .unexpected(let underlying) { + // `renderDirectory` only wraps directory-walk failures in `.unexpected`; + // surface the original "failed to walk" framing the CLI used to print + // before the SDK split. + FileHandle.standardError.write( + Data("skit: failed to walk \(input): \(underlying)\n".utf8) + ) + throw ExitCode(1) + } catch { + // renderDirectory does not throw .renderFailed — that's a per-file + // outcome. Defensive. + throw ExitCode(1) + } + + if result.outcomes.isEmpty { + FileHandle.standardError.write(Data("skit: no .swift inputs under \(input)\n".utf8)) + return + } + + for outcome in result.outcomes { + if !outcome.stderr.isEmpty { + FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) + FileHandle.standardError.write(Data(outcome.stderr.utf8)) + } + // .renderFailed already had its stderr surfaced above. Other failures + // (process spawn, write) carry the diagnostic in the error itself. + if case .failure(let error) = outcome.result, case .renderFailed = error { + continue + } + if case .failure(let error) = outcome.result { + FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) + } + } + + FileHandle.standardError.write( + Data( + "skit: \(result.outcomes.count - result.failureCount)/\(result.outcomes.count) succeeded\n" + .utf8 + ) + ) + + if result.failureCount > 0 { + throw ExitCode(1) + } + } + + /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. + internal func captureSwiftVersion() async -> String? { + let result = try? await Subprocess.run( + .name("swift"), + arguments: ["--version"], + output: .string(limit: 4_096), + error: .discarded + ) + return result?.standardOutput + } + + /// Human-readable error emitted when the bundle's recorded `swift --version` + /// differs from the local one, explaining why and how to recover. + internal func toolchainMismatchMessage(bundle: String, local: String) -> String { + """ + skit: toolchain mismatch + bundle: \(bundle) + local: \(local) + The bundle's libSyntaxKit was built against a different `swift` than the + one on your PATH. Swift swiftmodules aren't reliably compatible across + versions, so spawning `swift` would fail with a cryptic module-version + diagnostic. + + Rebuild the bundle with: + Scripts/build-skit-release.sh + Or pass --no-toolchain-check to try anyway. + + """ + } + } + +#endif diff --git a/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift b/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift index eba0468..0542f92 100644 --- a/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift +++ b/Tests/SyntaxKitTests/Integration/SkitSubprocessTimeoutTests.swift @@ -76,7 +76,7 @@ try await Task.sleep(for: .seconds(1)) return .timedOut } - let first = try await group.next()! + let first = try #require(await group.next()) group.cancelAll() return first } From 5bc67c8a6b5e9edda518417a863f9223dba2d1be Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 9 Jun 2026 09:48:25 -0400 Subject: [PATCH 46/56] Refactor render-engine error plumbing and consolidate FileManager helpers - FileOutcome.result: Result -> (any Error)? (nil == success) - Merge all SyntaxKit FileManager extensions into FileManager+Execution.swift (isLibDir, libStamp, collectInputs, writeOutput); drop the inline extension from Runner+Directory.swift (also clears its file_types_order warnings) - runSwift backend returns SwiftRunOutcome; move .completed wrapping out of Runner - collectInputs throws a dedicated CollectInputsError (.cliError / .resourceValuesFailure), mapped to RunError.unexpected by renderDirectory Co-Authored-By: Claude Opus 4.8 (1M context) --- ...sLibDir.swift => CollectInputsError.swift} | 24 +-- .../SyntaxKit/Execution/DirectoryRender.swift | 16 +- .../Execution/FileManager+Execution.swift | 145 ++++++++++++++++++ .../Execution/FileManager+LibStamp.swift | 43 ------ .../Execution/Runner+Directory.swift | 104 ++----------- Sources/SyntaxKit/Execution/Runner.swift | 9 +- Sources/skit/Skit.Run+Render.swift | 8 +- .../skit/Subprocess.Configuration+Swift.swift | 18 ++- 8 files changed, 199 insertions(+), 168 deletions(-) rename Sources/SyntaxKit/Execution/{FileManager+IsLibDir.swift => CollectInputsError.swift} (64%) create mode 100644 Sources/SyntaxKit/Execution/FileManager+Execution.swift delete mode 100644 Sources/SyntaxKit/Execution/FileManager+LibStamp.swift diff --git a/Sources/SyntaxKit/Execution/FileManager+IsLibDir.swift b/Sources/SyntaxKit/Execution/CollectInputsError.swift similarity index 64% rename from Sources/SyntaxKit/Execution/FileManager+IsLibDir.swift rename to Sources/SyntaxKit/Execution/CollectInputsError.swift index 44199d3..0f849fe 100644 --- a/Sources/SyntaxKit/Execution/FileManager+IsLibDir.swift +++ b/Sources/SyntaxKit/Execution/CollectInputsError.swift @@ -1,5 +1,5 @@ // -// FileManager+IsLibDir.swift +// CollectInputsError.swift // SyntaxKit // // Created by Leo Dion. @@ -27,15 +27,15 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation - -extension FileManager { - /// True if `path` is a directory containing `libSyntaxKit.{dylib,so}`. - internal func isLibDir(_ path: String) -> Bool { - var isDir: ObjCBool = false - guard fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { - return false - } - return fileExists(atPath: "\(path)/\("SyntaxKit".dylibFilename)") - } +/// Typed error surfaced by `FileManager.collectInputs(at:)`. The two cases map +/// to the two distinct ways the input walk can fail, so the caller can tell a +/// directory it couldn't enumerate apart from a file whose resource values it +/// couldn't read. +package enum CollectInputsError: Error { + /// The directory could not be enumerated (`FileManager.enumerator` returned + /// nil). Carries a user-facing `CLIError` describing the path. + case cliError(CLIError) + /// A file's resource values (isRegularFile/isDirectory) could not be read; + /// carries the underlying Foundation error. + case resourceValuesFailure(any Error) } diff --git a/Sources/SyntaxKit/Execution/DirectoryRender.swift b/Sources/SyntaxKit/Execution/DirectoryRender.swift index 92fe1c3..f274189 100644 --- a/Sources/SyntaxKit/Execution/DirectoryRender.swift +++ b/Sources/SyntaxKit/Execution/DirectoryRender.swift @@ -39,20 +39,20 @@ /// batch down. package struct DirectoryRender: Sendable { /// Per-input result. `stderr` carries the (possibly path-rewritten) - /// diagnostics from the spawned `swift`; it may be present in both - /// success and failure cases (e.g. a successful render that emitted - /// warnings). `result` is `.success` when the rendered output was - /// written to its mirrored destination; `.failure` when the input could - /// not be rendered or its output could not be written. + /// diagnostics from the spawned `swift`; it may be present whether or not + /// the input succeeded (e.g. a successful render that emitted warnings). + /// `result` is `nil` when the rendered output was written to its mirrored + /// destination, and carries the error when the input could not be rendered + /// or its output could not be written (typically a `RunError`). package struct FileOutcome: Sendable { package let input: URL package let stderr: String - package let result: Result + package let result: (any Error)? package init( input: URL, stderr: String, - result: Result + result: (any Error)? ) { self.input = input self.stderr = stderr @@ -71,7 +71,7 @@ /// failure semantics fit the host). package var failureCount: Int { outcomes.reduce(into: 0) { count, outcome in - if case .failure = outcome.result { count += 1 } + if outcome.result != nil { count += 1 } } } } diff --git a/Sources/SyntaxKit/Execution/FileManager+Execution.swift b/Sources/SyntaxKit/Execution/FileManager+Execution.swift new file mode 100644 index 0000000..f20da2a --- /dev/null +++ b/Sources/SyntaxKit/Execution/FileManager+Execution.swift @@ -0,0 +1,145 @@ +// +// FileManager+Execution.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +extension FileManager { + /// True if `path` is a directory containing `libSyntaxKit.{dylib,so}`. + internal func isLibDir(_ path: String) -> Bool { + var isDir: ObjCBool = false + guard fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { + return false + } + return fileExists(atPath: "\(path)/\("SyntaxKit".dylibFilename)") + } + + /// `/` fingerprint of `libSyntaxKit.{dylib,so}` under + /// `libPath`, or nil if unreadable. Catches in-place rebuilds without a + /// version bump. + internal func libStamp(libPath: String) -> String? { + let dylib = "\(libPath)/\("SyntaxKit".dylibFilename)" + guard let attrs = try? attributesOfItem(atPath: dylib) else { return nil } + let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 + let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 + return "\(size)/\(Int(mtime))" + } + + /// Returns every `.swift` file under `inputDir` (recursive), sorted, with + /// hidden files and files prefixed by `_` removed. Sorted output keeps + /// batch behaviour deterministic across runs. + /// + /// Throws `CollectInputsError.cliError` when the directory can't be + /// enumerated, or `.resourceValuesFailure` when a file's resource values + /// can't be read — both are bulk failures with nothing per-file to report. + internal func collectInputs(at inputDir: URL) throws(CollectInputsError) -> [URL] { + guard + let enumerator = enumerator( + at: inputDir, + includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], + options: [.skipsHiddenFiles] + ) + else { + throw .cliError(CLIError(message: "could not enumerate \(inputDir.path)")) + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values: URLResourceValues + do { + values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + } catch { + throw .resourceValuesFailure(error) + } + // Directories aren't outputs. + if values.isDirectory == true { continue } + // Filter for `.swift` regular files, skipping the `_`-prefixed + // convention for "not an input" sources. + guard values.isRegularFile == true else { continue } + guard url.pathExtension == "swift" else { continue } + guard !url.lastPathComponent.hasPrefix("_") else { continue } + result.append(url.standardizedFileURL) + } + return result.sorted { $0.path < $1.path } + } + + /// Builds the `FileOutcome` for one render result, writing a successful + /// render's stdout to its mirrored destination under `outputBase`. The write + /// side effect lives here; failures (a non-zero render exit, or a write + /// error) are folded into the returned outcome rather than thrown, so a + /// failing peer doesn't prevent successful files in the batch from being + /// written (Tuist-analog batch semantics). No diagnostics are printed here; + /// the caller does that. + internal func writeOutput( + for result: RenderTaskResult, + inputBase: URL, + outputBase: URL + ) -> DirectoryRender.FileOutcome { + let relative = result.input.path.dropFirst(inputBase.path.count + 1) + let destination = outputBase.appendingPathComponent(String(relative)) + + // stderr is the toolchain's diagnostics whenever the render produced any — + // i.e. on every successful spawn, regardless of how the write then fares. + let stderr = (try? result.result.get())?.stderr ?? "" + + // Fold the render result into the write result: a render failure passes + // through, a non-zero exit becomes `.renderFailed`, and a clean render is + // committed to disk (capturing any write error as `.unexpected`). + let outcome = result.result.flatMap { processResult -> Result in + guard processResult.exitCode == 0 else { + return .failure( + .renderFailed(exitCode: processResult.exitCode, stderr: processResult.stderr) + ) + } + return Result { + try createDirectory( + at: destination.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + try processResult.stdout.write(to: destination) + } + .mapError(RunError.unexpected) + } + + // Collapse the success/failure result into the outcome's optional error + // (nil == success). + let failure: (any Error)? + switch outcome { + case .success: + failure = nil + case .failure(let error): + failure = error + } + + return DirectoryRender.FileOutcome( + input: result.input, + stderr: stderr, + result: failure + ) + } +} diff --git a/Sources/SyntaxKit/Execution/FileManager+LibStamp.swift b/Sources/SyntaxKit/Execution/FileManager+LibStamp.swift deleted file mode 100644 index b55e159..0000000 --- a/Sources/SyntaxKit/Execution/FileManager+LibStamp.swift +++ /dev/null @@ -1,43 +0,0 @@ -// -// FileManager+LibStamp.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -import Foundation - -extension FileManager { - /// `/` fingerprint of `libSyntaxKit.{dylib,so}` under - /// `libPath`, or nil if unreadable. Catches in-place rebuilds without a - /// version bump. - internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/\("SyntaxKit".dylibFilename)" - guard let attrs = try? attributesOfItem(atPath: dylib) else { return nil } - let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 - let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 - return "\(size)/\(Int(mtime))" - } -} diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index 570bf33..6959827 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -48,8 +48,15 @@ extension Runner { let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL // Phase 1: enumerate inputs. A walk failure is a bulk failure: there's - // nothing per-file to report, so it surfaces as a typed throw. - let inputs = try FileManager.default.collectInputs(at: inputURL) + // nothing per-file to report, so it surfaces as a typed throw. The + // collect-specific error is folded into `RunError.unexpected` — the bulk + // channel the caller already presents. + let inputs: [URL] + do { + inputs = try FileManager.default.collectInputs(at: inputURL) + } catch { + throw RunError.unexpected(error) + } guard !inputs.isEmpty else { return DirectoryRender(outcomes: []) } @@ -111,96 +118,13 @@ extension Runner { } } -extension FileManager { - /// Returns every `.swift` file under `inputDir` (recursive), sorted, with - /// hidden files and files prefixed by `_` removed. Sorted output keeps - /// batch behaviour deterministic across runs. - /// - /// Throws `RunError.unexpected` when the directory can't be enumerated or a - /// file's resource values can't be read — both are bulk failures with - /// nothing per-file to report. - internal func collectInputs(at inputDir: URL) throws(RunError) -> [URL] { - guard - let enumerator = enumerator( - at: inputDir, - includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], - options: [.skipsHiddenFiles] - ) - else { - throw RunError.unexpected(CLIError(message: "could not enumerate \(inputDir.path)")) - } - - var result: [URL] = [] - for case let url as URL in enumerator { - let values: URLResourceValues - do { - values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) - } catch { - throw RunError.unexpected(error) - } - // Directories aren't outputs. - if values.isDirectory == true { continue } - // Filter for `.swift` regular files, skipping the `_`-prefixed - // convention for "not an input" sources. - guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } - result.append(url.standardizedFileURL) - } - return result.sorted { $0.path < $1.path } - } - - /// Builds the `FileOutcome` for one render result, writing a successful - /// render's stdout to its mirrored destination under `outputBase`. The write - /// side effect lives here; failures (a non-zero render exit, or a write - /// error) are folded into the returned outcome rather than thrown, so a - /// failing peer doesn't prevent successful files in the batch from being - /// written (Tuist-analog batch semantics). No diagnostics are printed here; - /// the caller does that. - fileprivate func writeOutput( - for result: RenderTaskResult, - inputBase: URL, - outputBase: URL - ) -> DirectoryRender.FileOutcome { - let relative = result.input.path.dropFirst(inputBase.path.count + 1) - let destination = outputBase.appendingPathComponent(String(relative)) - - // stderr is the toolchain's diagnostics whenever the render produced any — - // i.e. on every successful spawn, regardless of how the write then fares. - let stderr = (try? result.result.get())?.stderr ?? "" - - // Fold the render result into the write result: a render failure passes - // through, a non-zero exit becomes `.renderFailed`, and a clean render is - // committed to disk (capturing any write error as `.unexpected`). - let outcome = result.result.flatMap { processResult -> Result in - guard processResult.exitCode == 0 else { - return .failure( - .renderFailed(exitCode: processResult.exitCode, stderr: processResult.stderr) - ) - } - return Result { - try createDirectory( - at: destination.deletingLastPathComponent(), - withIntermediateDirectories: true - ) - try processResult.stdout.write(to: destination) - } - .mapError(RunError.unexpected) - } - - return DirectoryRender.FileOutcome( - input: result.input, - stderr: stderr, - result: outcome - ) - } -} - /// Payload the per-input render `TaskGroup` yields back to `renderDirectory`. /// Failures are captured (not thrown) so a single bad input doesn't tear down /// the group; `processFile`'s heterogeneous Foundation/Subprocess throws are /// normalized into `RunError` (typically `.unexpected`) by `runOne`. -private struct RenderTaskResult: Sendable { - let input: URL - let result: Result +/// `internal` so `FileManager.writeOutput` (in `FileManager+Execution.swift`) +/// can consume it. +internal struct RenderTaskResult: Sendable { + internal let input: URL + internal let result: Result } diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index 2b5640a..5e4c36b 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -60,13 +60,13 @@ package struct Runner: Sendable { private let timeoutSeconds: Int /// Backend that actually spawns `swift` for one `SwiftInvocation`. Injected /// by the caller (skit supplies a Subprocess-based implementation). - private let run: @Sendable (SwiftInvocation) async throws -> ProcessResult + private let run: @Sendable (SwiftInvocation) async throws -> SwiftRunOutcome package init( libPath: String, cache: OutputCache?, timeoutSeconds: Int, - run: @Sendable @escaping (SwiftInvocation) async throws -> ProcessResult + run: @Sendable @escaping (SwiftInvocation) async throws -> SwiftRunOutcome ) { self.libPath = libPath self.cache = cache @@ -170,9 +170,10 @@ package struct Runner: Sendable { let invocation = SwiftInvocation(libPath: libPath, wrappedPath: wrappedPath) // The actual backend call, wrapped in a closure so the task-group race - // below can hold a single Sendable reference to it. + // below can hold a single Sendable reference to it. The backend already + // reports a `SwiftRunOutcome` (always `.completed` on a finished spawn). let operation: @Sendable () async throws -> SwiftRunOutcome = { - .completed(try await self.run(invocation)) + try await self.run(invocation) } // Race the invocation against a sleep watchdog; whichever finishes first diff --git a/Sources/skit/Skit.Run+Render.swift b/Sources/skit/Skit.Run+Render.swift index b0a1636..c42919c 100644 --- a/Sources/skit/Skit.Run+Render.swift +++ b/Sources/skit/Skit.Run+Render.swift @@ -134,10 +134,10 @@ import SyntaxKit } // .renderFailed already had its stderr surfaced above. Other failures // (process spawn, write) carry the diagnostic in the error itself. - if case .failure(let error) = outcome.result, case .renderFailed = error { - continue - } - if case .failure(let error) = outcome.result { + if let error = outcome.result { + if let runError = error as? RunError, case .renderFailed = runError { + continue + } FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) } } diff --git a/Sources/skit/Subprocess.Configuration+Swift.swift b/Sources/skit/Subprocess.Configuration+Swift.swift index 95e37ca..a96438a 100644 --- a/Sources/skit/Subprocess.Configuration+Swift.swift +++ b/Sources/skit/Subprocess.Configuration+Swift.swift @@ -61,21 +61,25 @@ } /// Spawns `swift` for the render `invocation` and normalizes the result - /// into a `ProcessResult`. This is the Subprocess backend skit hands to + /// into a `SwiftRunOutcome`. This is the Subprocess backend skit hands to /// `Runner` as its `run` closure — the one seam between the (platform- - /// agnostic) engine in SyntaxKit and the Subprocess implementation. + /// agnostic) engine in SyntaxKit and the Subprocess implementation. A + /// completed spawn always reports `.completed`; the timeout race that can + /// produce `.timedOut` lives in `Runner`. internal static func runSwift( for invocation: SyntaxKit.SwiftInvocation - ) async throws -> ProcessResult { + ) async throws -> SwiftRunOutcome { let record = try await Subprocess.run( .swift(libPath: invocation.libPath, wrappedPath: invocation.wrappedPath), output: .string(limit: stdoutLimitBytes), error: .string(limit: stderrLimitBytes) ) - return ProcessResult( - exitCode: record.terminationStatus.exitCode, - stdout: Data((record.standardOutput ?? "").utf8), - stderr: record.standardError ?? "" + return .completed( + ProcessResult( + exitCode: record.terminationStatus.exitCode, + stdout: Data((record.standardOutput ?? "").utf8), + stderr: record.standardError ?? "" + ) ) } } From e2cc93412d8f404147a033ef36f6d26d610f9335 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 9 Jun 2026 10:36:06 -0400 Subject: [PATCH 47/56] Extract string constants for skit CLI/swiftc flags and Execution paths Move hardcoded literals into named type-local constants (matching the existing private static let convention), giving the CLI surface, swiftc flag set, and cache/path/env conventions a single source of truth. - skit: command/option/flag names, swift executable name, SKIT_LIB_DIR key, shared "skit: " stderr prefix, --version flag, release script path, and the swiftc flag set (-I/-L/-lSyntaxKit/-Xcc/-Xlinker/-rpath/etc). - Execution: cache subpaths, outputs/output.swift names, staging prefix, SKIT_/SYNTAXKIT_ env prefixes, XDG_CACHE_HOME key, SyntaxKit product name, swift extension, lib/lib-skit dirs, %016x hash format, temp-dir prefix, wrapped-input filename, and dylib prefix/extensions. Platform-only constants are #if-gated so neither is flagged unused. CLI option names and all emitted strings are byte-for-byte unchanged. Also adds Docs/string-literal-audit.md cataloguing the broader audit. swift build + swift test (398 tests) + ./Scripts/lint.sh all pass. Co-Authored-By: Claude Opus 4.8 (1M context) --- Docs/string-literal-audit.md | 173 ++++++++++++++++++ .../Execution/Bundle+ResolveLibPath.swift | 10 +- .../SyntaxKit/Execution/ContentHasher.swift | 5 +- .../Execution/FileManager+Execution.swift | 15 +- Sources/SyntaxKit/Execution/OutputCache.swift | 35 +++- .../ProcessInfo+SyntaxKitCacheRoot.swift | 9 +- Sources/SyntaxKit/Execution/Runner.swift | 9 +- .../Execution/String+DylibFilename.swift | 14 +- Sources/skit/Skit+Parse.swift | 5 +- Sources/skit/Skit+Run.swift | 46 +++-- Sources/skit/Skit.Run+Render.swift | 19 +- Sources/skit/Skit.swift | 9 +- .../skit/Subprocess.Configuration+Swift.swift | 28 ++- 13 files changed, 326 insertions(+), 51 deletions(-) create mode 100644 Docs/string-literal-audit.md diff --git a/Docs/string-literal-audit.md b/Docs/string-literal-audit.md new file mode 100644 index 0000000..1ba3692 --- /dev/null +++ b/Docs/string-literal-audit.md @@ -0,0 +1,173 @@ +# String Literal Audit — SyntaxKit + +**Date:** 2026-06-09 +**Branch:** `research/swift-manifest-codegen` +**Scope:** All 208 `.swift` files across 5 modules (`SyntaxKit`, `skit`, `DocumentationHarness`, `SyntaxParser`, `TokenVisitor`). Markdown samples under `Documentation.docc` were excluded as illustrative, not product code. + +**Goal:** Identify hardcoded string literals that should be moved into named constants, prioritized by extraction value (duplication × cross-module reach × domain meaning). + +--- + +## Summary of counts + +| Module | Files | Non-empty string literals | +|---|---|---| +| SyntaxKit | 150 | 292 | +| DocumentationHarness | 20 | 55 | +| TokenVisitor | 28 | 43 | +| skit | 6 | 41 | +| SyntaxParser | 4 | 19 | + +Hotspots within SyntaxKit: `Expressions/` (80), `Collections/` (78), `Execution/` (47). + +--- + +## Tier 1 — Highest value (duplicated + cross-cutting) + +### 1.1 Empty-identifier fallback `.identifier("")` — ~30+ sites + +The `DeclReferenceExprSyntax(baseName: .identifier(""))` "couldn't-resolve-expression" placeholder is the single most pervasive pattern in the codebase. It is a degenerate placeholder emitted whenever an `ExprSyntax` downcast fails. + +**Recommendation:** Not just a constant — a shared factory (e.g. `ExprSyntax.placeholderReference` / `static let emptyDeclReferenceExpr`) so the fallback behavior lives in one place. + +| Area | Locations (file:line) | +|---|---| +| Expressions (~13) | ConditionalOp.swift:42,52,62; FunctionCallExp.swift:58,136; Literal.swift:136,159,179,182; Literal+ExprCodeBlock.swift:97; NegatedPropertyAccessExp.swift:40; OptionalChainingExp.swift:44; PropertyAccessExp.swift:41; ReferenceExp.swift:47 | +| ControlFlow | Guard.swift:56,67; If+Conditions.swift:69,79; SwitchCase.swift:64; For.swift:47,55; Switch.swift:41 | +| Core / CodeBlocks | Core/CodeBlock.swift:52; CodeBlocks/CodeBlock+Generate.swift:51; CodeBlocks/CodeBlock+ExprSyntax.swift:52 | +| Utilities / Variables | Utilities/EnumCase+Syntax.swift:49; Utilities/Let.swift:50; Variables/Variable.swift:182 | +| Parameters / ErrorHandling | Parameters/ParameterExp.swift:44,53; ErrorHandling/Catch.swift:98; ErrorHandling/Throw.swift:45 | +| Collections | CodeBlock+DictionaryValue.swift:51; TupleAssignment.swift:174 | + +### 1.2 Swift type names — ~35 sites + +Universal Swift type tokens scattered across modules. The canonical inference table is `Expressions/Literal.swift`'s `typeName` switch. + +**Recommendation:** Package-wide `TypeNames` caseless enum. + +| String | Count | Locations (file:line) | +|---|---|---| +| `"Any"` | 18 | Collections/TupleLiteralArray.swift:52-59; Collections/DictionaryLiteral.swift:47,48; Collections/ArrayLiteral.swift:47; Expressions/ClosureParameter.swift:45; Expressions/ClosureType.swift:44; Expressions/Literal.swift:67,80-102 | +| `"Void"` | 3 | Expressions/ClosureType.swift:67,95,127 | +| `"String"` | 4 | Collections/TupleLiteralArray.swift:50; Expressions/Literal.swift:61,73; Variables/Variable+LiteralInitializers.swift:63 | +| `"Int"` | 4 | Collections/TupleLiteralArray.swift:47; Expressions/Literal.swift:63,72; Variables/Variable+LiteralInitializers.swift:80 | +| `"Double"` | 4 | Collections/TupleLiteralArray.swift:48; Expressions/Literal.swift:62,74; Variables/Variable+LiteralInitializers.swift:114 | +| `"Bool"` | 4 | Collections/TupleLiteralArray.swift:49; Expressions/Literal.swift:65,75; Variables/Variable+LiteralInitializers.swift:97 | +| `"Any?"` | 3 | Collections/TupleLiteralArray.swift:51; Expressions/Literal.swift:64,76 | +| `"[Any]"` | 2 | Collections/ArrayLiteral.swift:45; Expressions/Literal.swift:92 | +| `"[Any: Any]"` | 3 | Collections/DictionaryExpr.swift:44; Collections/DictionaryLiteral.swift:45; Expressions/Literal.swift:99 | +| `"[String: Any]"` | 1 | Collections/DictionaryExpr.swift:46 | +| `"[String]"` | 1 | Collections/Array+LiteralValue.swift:34 | +| `"[Int: String]"` | 1 | Collections/Dictionary+LiteralValue.swift:34 | + +### 1.3 Duplicated string-escape block — verbatim copy in 2 files + +`Collections/Array+LiteralValue.swift` and `Collections/Dictionary+LiteralValue.swift` contain an identical escape map: `\` → `\\`, `"` → `\"`, `\n` → `\n`, `\r` → `\r`, `\t` → `\t` (lines 47–51 in each). + +**Recommendation:** One shared `String.escapedForSwiftLiteral()` helper. (Related-but-distinct escape logic also lives in `Execution/WrappedSource.swift:105-106` and `TokenVisitor/TriviaPiece.swift`.) + +### 1.4 Whitespace / separators — most cross-cutting tokens + +| String | Count | Locations | +|---|---|---| +| `"\n"` | 12+ | skit/Skit.Run+Render.swift (stderr writes), skit/Skit+Run.swift:104,153; DocumentationHarness/CodeBlockExtraction.swift:141; TokenVisitor/TriviaPiece.swift:63; Collections/Array+LiteralValue.swift:49, Dictionary+LiteralValue.swift:49; Execution/WrappedSource.swift:99 | +| `", "` | 10+ | Collections (ArrayLiteral, Array+LiteralValue, Dictionary+LiteralValue, TupleLiteralArray, DictionaryLiteral, DictionaryExpr); Expressions (ClosureType.swift:91, Literal.swift:88) | + +--- + +## Tier 2 — Domain literals worth naming + +### Operators (Expressions, Patterns) +| String | Locations | +|---|---| +| `"+="` (defined twice independently) | Expressions/Infix.swift:40; Expressions/PlusAssign.swift:53 | +| `"-="`, `"=="`, `"!="`, `">"`, `"<"` | Expressions/Infix.swift:36-41 | +| `"!"` (prefix negation) | Expressions/NegatedPropertyAccessExp.swift:45 | +| `"..<"`, `"..."` (range operators) | Patterns/Range+PatternConvertible.swift:39,58 | + +### skit CLI args & swiftc flags +| String | Locations | +|---|---| +| `"swift"` (executable name) | skit/Skit.Run+Render.swift:160; skit/Subprocess.Configuration+Swift.swift:60 | +| `"--version"`, `"--timeout"` | skit/Skit.Run+Render.swift:163; skit/Skit+Run.swift:73,87 | +| `"run"`, `"parse"`, `"output"`, `"lib"`, `"no-cache"`, `"no-toolchain-check"` | skit/Skit+Run.swift:47,55,62,67,79; skit/Skit+Parse.swift:37 | +| `"-I"`, `"-L"`, `"-lSyntaxKit"`, `"-Xcc"`, `"-Xlinker"`, `"-rpath"`, `"-suppress-warnings"` | skit/Subprocess.Configuration+Swift.swift:50-57 | +| `"SKIT_LIB_DIR"` (env key) | skit/Skit+Run.swift:99 | +| `"skit: "` (stderr prefix, 5+ sites) | skit/Skit.Run+Render.swift:116,126,147,170; skit/Skit+Run.swift:153; also Execution/Runner.swift:197, ToolchainCheckResult.swift:57,64 | + +### Execution paths / env / formats +| String | Locations | +|---|---| +| `"output.swift"` (3×) | Execution/OutputCache.swift:116,123,139 | +| `"swift"` (file extension filter) | Execution/FileManager+Execution.swift:84 | +| `"SyntaxKit"` (dylib product, 2×) | Execution/FileManager+Execution.swift:39,46 | +| `lib\(self).so` / `lib\(self).dylib` | Execution/String+DylibFilename.swift:36,38 | +| `"XDG_CACHE_HOME"`, `"syntaxkit"` cache dir | Execution/ProcessInfo+SyntaxKitCacheRoot.swift:37,38; OutputCache.swift:53 | +| `"SKIT_"` / `"SYNTAXKIT_"` env prefixes | Execution/OutputCache.swift:105 | +| `"Library/Caches/com.brightdigit.SyntaxKit"`, `".cache/syntaxkit"`, `"outputs"` | Execution/OutputCache.swift:51,53,74 | +| `"lib"`, `"lib/skit"` | Execution/Bundle+ResolveLibPath.swift:55,61 | +| `"%016x"` (hash format), `"swift-version.txt"` | Execution/ContentHasher.swift:57; ToolchainCheckResult.swift:34 | + +### DocumentationHarness markers & fences +| String | Locations | +|---|---| +| ``, ``, ``, `` | CodeBlockExtraction.swift:152-155 | +| ` ```swift ` (open), ` ``` ` (close) | CodeBlockExtraction.swift:94,96 | +| `"md"` (default doc extension) | Validator.swift:38 | +| `"import SyntaxKit"` (2×), `"import Foundation"`, `"Package.swift"`, `"@main"` | CodeSyntaxValidator.swift:44,49,62,63 | +| skip dirs: `".build"`, `"node_modules"`, `".git"`, `".svn"`, `"DerivedData"`, `"build"`, `".swiftpm"` | FileManager+Documentation.swift:38-46 | + +### Comment trivia & misc tokens +| String | Locations | +|---|---| +| `"// "`, `"/// "`, `"///"` | Core/Line+Trivia.swift:39,43,45 | +| `"_"` (wildcard / unnamed label, 8+ sites) | Parameters/Parameter.swift:50,77,110; Expressions/Literal.swift:139; Literal+ExprCodeBlock.swift:72; Collections/TupleLiteralArray.swift:86; CodeBlocks/CodeBlockItemSyntax.Item.swift:53 | +| `"."` (member separator) | ErrorHandling/Catch.swift:48; Utilities/EnumCase.swift:46; TokenVisitor/TreeNodeProtocol+Extensions.swift:189; DocumentationHarness (extension join) | +| `"self"` (capture fallback) | Expressions/CaptureInfo.swift:56,67 | +| `"Syntax"` (suffix stripped) | TokenVisitor/Syntax.swift:54 | +| TriviaPiece chars: space, `\t`, `\n`, `\`, `#` | TokenVisitor/TriviaPiece.swift:57-69 | + +--- + +## Tier 3 — Deprecation messages (dedup only, not reuse) + +Single-purpose but **duplicated verbatim** — collapsing each to one local constant removes the copies. + +| Message | Copies | Location | +|---|---|---| +| `"Use Infix(target, \"+=\", value) instead."` | 6 | Expressions/PlusAssign.swift:71,80,89,97,105,113 | +| `"Use parse(code:) which returns [TreeNode] directly instead of JSON"` | 3 | SyntaxParser/SyntaxResponse.swift:44,51,59 | +| `"Use separate lhs and rhs parameters for compile-time safety"` | 1 | Expressions/Infix.swift:128 | +| `"Use ParameterExp(name:value:) ..."` / `"Use ParameterExp(unlabeled:) ..."` | 1 ea. | Parameters/ParameterExp.swift:82,98 | +| `"Use Parameter(unlabeled:type:) ..."` | 1 | Parameters/Parameter.swift:117 | +| `"Use While(kind:condition:) ..."` / `"Use While(VariableExp(condition)...)"` | 1 ea. | ControlFlow/While.swift:130,151 | + +Plus runtime error messages worth extracting for testability: `Infix` operand errors (Infix.swift:58,60), `Parenthesized` arity (Utilities/Parenthesized.swift:58), Execution path/IO errors (Runner.swift, Bundle+ResolveLibPath.swift:47, FileManager+Execution.swift:68, RunInput.swift:53,57). + +--- + +## Already extracted (no action needed) + +- `SyntaxParser/SyntaxParser.swift` — `fold`, `showMissing` +- `TokenVisitor/StructureProperty.swift` — `element`, `count`, `nilValue` +- `TokenVisitor/String.swift` — `String.empty`, `String.defaultFileName` +- `skit/Subprocess.Configuration+Swift.swift` — `stdoutLimitBytes`, `stderrLimitBytes` + +--- + +## Notable non-findings + +- **Token-kind strings are never hardcoded.** Expected literals like `"keyword"`, `"identifier"`, `"invisible"` come from interpolating `token.tokenKind` — they appear only in doc-comments. No work needed. +- **`DocumentationHarness/PackageValidator.swift` and `CompilationResult.swift`** literals sit in `@available(*, unavailable)` dead code — low priority. + +--- + +## Suggested target structure + +A small set of caseless enums plus one factory helper: + +- **Package-wide** (in `Core/` or `Utilities/`): `TypeNames`, `Operators`, `Keywords`/`Tokens`, `Separators`. +- **`ExprSyntax.placeholderReference`** factory for the Tier 1.1 empty-identifier fallback. +- **Module-local** static constants for Execution paths/env, skit CLI args, and DocumentationHarness markers. + +**Recommended order:** Tier 1 first (biggest payoff, touches the most files), then Tier 2 by module, then Tier 3 dedup. diff --git a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift index 0bc49e2..ccaaea1 100644 --- a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift +++ b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift @@ -30,6 +30,12 @@ package import Foundation extension Bundle { + /// Bundle-relative lib dir name for the adjacent layout (`/lib`). + private static let libDirectoryName = "lib" + /// Bundle-relative lib dir path for the Homebrew layout + /// (`/../lib/skit`). + private static let homebrewLibSkitSubpath = "lib/skit" + /// Resolves a directory containing `libSyntaxKit.dylib` + swiftmodules. /// /// Tries each non-nil entry in `candidates` in order; if any non-nil @@ -52,13 +58,13 @@ extension Bundle { if let execURL = executableURL?.resolvingSymlinksInPath() { let execDir = execURL.deletingLastPathComponent() - let adjacent = execDir.appendingPathComponent("lib").path + let adjacent = execDir.appendingPathComponent(Self.libDirectoryName).path if fileManager.isLibDir(adjacent) { return adjacent } let brewLayout = execDir.deletingLastPathComponent() - .appendingPathComponent("lib/skit").path + .appendingPathComponent(Self.homebrewLibSkitSubpath).path if fileManager.isLibDir(brewLayout) { return brewLayout } diff --git a/Sources/SyntaxKit/Execution/ContentHasher.swift b/Sources/SyntaxKit/Execution/ContentHasher.swift index 6ec0e3c..47ea36e 100644 --- a/Sources/SyntaxKit/Execution/ContentHasher.swift +++ b/Sources/SyntaxKit/Execution/ContentHasher.swift @@ -42,6 +42,9 @@ internal struct ContentHasher { private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 private static let prime: UInt64 = 0x0000_0100_0000_01b3 + /// `String(format:)` specifier for the 16-char lowercase-hex digest. + private static let hexFormat = "%016x" + private var state: UInt64 = ContentHasher.offsetBasis internal mutating func update(data: Data) { @@ -54,6 +57,6 @@ internal struct ContentHasher { /// Returns the hash as a 16-char lowercase-hex string suitable for use as /// a directory name. internal func finalize() -> String { - String(format: "%016x", state) + String(format: Self.hexFormat, state) } } diff --git a/Sources/SyntaxKit/Execution/FileManager+Execution.swift b/Sources/SyntaxKit/Execution/FileManager+Execution.swift index f20da2a..0273f8b 100644 --- a/Sources/SyntaxKit/Execution/FileManager+Execution.swift +++ b/Sources/SyntaxKit/Execution/FileManager+Execution.swift @@ -30,20 +30,27 @@ import Foundation extension FileManager { + /// Library product name whose platform-specific dylib marks a lib dir. + private static let syntaxKitProductName = "SyntaxKit" + /// File extension identifying SyntaxKit DSL input files. + private static let swiftFileExtension = "swift" + /// Filename prefix marking a source as "not an input" (skipped in batches). + private static let nonInputFilePrefix = "_" + /// True if `path` is a directory containing `libSyntaxKit.{dylib,so}`. internal func isLibDir(_ path: String) -> Bool { var isDir: ObjCBool = false guard fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { return false } - return fileExists(atPath: "\(path)/\("SyntaxKit".dylibFilename)") + return fileExists(atPath: "\(path)/\(Self.syntaxKitProductName.dylibFilename)") } /// `/` fingerprint of `libSyntaxKit.{dylib,so}` under /// `libPath`, or nil if unreadable. Catches in-place rebuilds without a /// version bump. internal func libStamp(libPath: String) -> String? { - let dylib = "\(libPath)/\("SyntaxKit".dylibFilename)" + let dylib = "\(libPath)/\(Self.syntaxKitProductName.dylibFilename)" guard let attrs = try? attributesOfItem(atPath: dylib) else { return nil } let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 @@ -81,8 +88,8 @@ extension FileManager { // Filter for `.swift` regular files, skipping the `_`-prefixed // convention for "not an input" sources. guard values.isRegularFile == true else { continue } - guard url.pathExtension == "swift" else { continue } - guard !url.lastPathComponent.hasPrefix("_") else { continue } + guard url.pathExtension == Self.swiftFileExtension else { continue } + guard !url.lastPathComponent.hasPrefix(Self.nonInputFilePrefix) else { continue } result.append(url.standardizedFileURL) } return result.sorted { $0.path < $1.path } diff --git a/Sources/SyntaxKit/Execution/OutputCache.swift b/Sources/SyntaxKit/Execution/OutputCache.swift index f6c7a32..ac6c9f7 100644 --- a/Sources/SyntaxKit/Execution/OutputCache.swift +++ b/Sources/SyntaxKit/Execution/OutputCache.swift @@ -41,6 +41,23 @@ package struct OutputCache: Sendable { /// Bumped when the cache layout changes in a way that requires invalidation. private static let schemaVersion = "v1" + #if os(macOS) + /// macOS home-relative cache subpath (under `~`). + private static let macOSCacheSubpath = "Library/Caches/com.brightdigit.SyntaxKit" + #else + /// Linux home-relative cache subpath (under `~`). + private static let linuxCacheSubpath = ".cache/syntaxkit" + #endif + /// Leaf directory holding the rendered outputs within the cache root. + private static let outputsDirectoryName = "outputs" + /// Filename of the rendered Swift source stored per cache key. + private static let outputFileName = "output.swift" + /// Prefix for the per-writer staging directory used for atomic stores. + private static let stagingDirectoryPrefix = "tmp" + /// Environment-variable prefixes mixed into the cache key. + private static let skitEnvPrefix = "SKIT_" + private static let syntaxKitEnvPrefix = "SYNTAXKIT_" + /// Home-relative cache root used when `XDG_CACHE_HOME` is unset: macOS /// `~/Library/Caches/...`, else Linux `~/.cache/syntaxkit`. The home dir is /// fixed for the process lifetime, so this is computed once. @@ -48,9 +65,9 @@ package struct OutputCache: Sendable { let home = NSHomeDirectory() #if os(macOS) return URL(fileURLWithPath: home) - .appendingPathComponent("Library/Caches/com.brightdigit.SyntaxKit") + .appendingPathComponent(macOSCacheSubpath) #else - return URL(fileURLWithPath: home).appendingPathComponent(".cache/syntaxkit") + return URL(fileURLWithPath: home).appendingPathComponent(linuxCacheSubpath) #endif }() @@ -71,7 +88,7 @@ package struct OutputCache: Sendable { processInfo: ProcessInfo = .processInfo ) { self.root = processInfo.syntaxKitCacheRoot(default: Self.defaultCacheRoot) - .appendingPathComponent("outputs") + .appendingPathComponent(Self.outputsDirectoryName) self.swiftVersion = swiftVersion self.fileManager = fileManager self.processInfo = processInfo @@ -102,7 +119,9 @@ package struct OutputCache: Sendable { // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. let env = processInfo.environment - .filter { $0.key.hasPrefix("SKIT_") || $0.key.hasPrefix("SYNTAXKIT_") } + .filter { + $0.key.hasPrefix(Self.skitEnvPrefix) || $0.key.hasPrefix(Self.syntaxKitEnvPrefix) + } .sorted { $0.key < $1.key } for (key, value) in env { hasher.update(data: Data("\(key)=\(value)\0".utf8)) @@ -113,14 +132,14 @@ package struct OutputCache: Sendable { /// Returns the cached rendered output for `key`, or nil on miss. package func lookup(key: String) -> Data? { - try? Data(contentsOf: directory(for: key).appendingPathComponent("output.swift")) + try? Data(contentsOf: directory(for: key).appendingPathComponent(Self.outputFileName)) } /// Atomically stores `data` under `key`. Concurrent writers race via a /// `tmp../` staging dir + rename; the loser drops their copy. package func store(key: String, data: Data) throws { let cacheRoot = directory(for: key) - let final = cacheRoot.appendingPathComponent("output.swift") + let final = cacheRoot.appendingPathComponent(Self.outputFileName) // Ensure the parent of the cache key dir exists. The key dir itself is // installed by the atomic rename below. @@ -133,10 +152,10 @@ package struct OutputCache: Sendable { // into place as a single atomic step. let staging = cacheRoot.deletingLastPathComponent() .appendingPathComponent( - "tmp.\(processInfo.processIdentifier).\(UUID().uuidString)" + "\(Self.stagingDirectoryPrefix).\(processInfo.processIdentifier).\(UUID().uuidString)" ) try fileManager().createDirectory(at: staging, withIntermediateDirectories: true) - try data.write(to: staging.appendingPathComponent("output.swift")) + try data.write(to: staging.appendingPathComponent(Self.outputFileName)) // Atomic rename into the cache path. If a peer already populated this // key, swallow the rename error and drop our staging copy. Re-throw only diff --git a/Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift b/Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift index 59de0b5..75d93f1 100644 --- a/Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift +++ b/Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift @@ -30,12 +30,17 @@ import Foundation extension ProcessInfo { + /// Environment variable pointing at the XDG cache root, if set. + private static let xdgCacheHomeEnvKey = "XDG_CACHE_HOME" + /// Leaf directory appended to the XDG cache root for skit's caches. + private static let cacheDirectoryName = "syntaxkit" + /// Root for all skit caches: `/syntaxkit` when that env /// var is set and non-empty, otherwise `defaultRoot` (typically the /// platform's home-relative cache dir). internal func syntaxKitCacheRoot(default defaultRoot: URL) -> URL { - if let xdg = environment["XDG_CACHE_HOME"], !xdg.isEmpty { - return URL(fileURLWithPath: xdg).appendingPathComponent("syntaxkit") + if let xdg = environment[Self.xdgCacheHomeEnvKey], !xdg.isEmpty { + return URL(fileURLWithPath: xdg).appendingPathComponent(Self.cacheDirectoryName) } return defaultRoot } diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index 5e4c36b..c6fb5c5 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -51,6 +51,11 @@ package struct Runner: Sendable { /// watchdog. Matches POSIX `timeout(1)`. private static let timeoutExitCode: Int32 = 124 + /// Prefix for the per-invocation temp dir holding the wrapped input. + private static let tempDirectoryPrefix = "skit" + /// Filename of the wrapped Swift program spilled into the temp dir. + private static let wrappedInputFileName = "Input.wrapped.swift" + /// Directory holding `libSyntaxKit.{dylib,so}` + swiftmodules; reused for /// the spawned `swift`'s `-I`/`-L`/`-rpath` flags. private let libPath: String @@ -133,11 +138,11 @@ package struct Runner: Sendable { // Spill the wrapped program to a per-invocation temp dir. The dir is // cleaned up unconditionally so a failed spawn doesn't leak files. let tmpDir = FileManager.default.temporaryDirectory - .appendingPathComponent("skit-\(UUID().uuidString)") + .appendingPathComponent("\(Self.tempDirectoryPrefix)-\(UUID().uuidString)") try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) defer { try? FileManager.default.removeItem(at: tmpDir) } - let wrappedURL = tmpDir.appendingPathComponent("Input.wrapped.swift") + let wrappedURL = tmpDir.appendingPathComponent(Self.wrappedInputFileName) try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) // Spawn `swift` on the wrapped file (with timeout watchdog). stdout is diff --git a/Sources/SyntaxKit/Execution/String+DylibFilename.swift b/Sources/SyntaxKit/Execution/String+DylibFilename.swift index 7497e7f..861440d 100644 --- a/Sources/SyntaxKit/Execution/String+DylibFilename.swift +++ b/Sources/SyntaxKit/Execution/String+DylibFilename.swift @@ -28,14 +28,24 @@ // extension String { + /// Shared-library filename prefix (`lib`). + private static let dylibPrefix = "lib" + #if os(Linux) + /// Linux shared-library filename extension. + private static let linuxDylibExtension = ".so" + #else + /// macOS shared-library filename extension. + private static let macOSDylibExtension = ".dylib" + #endif + /// Treats `self` as a Swift library product name and returns the /// platform-specific shared-library filename (`libFoo.dylib` on macOS, /// `libFoo.so` on Linux). internal var dylibFilename: String { #if os(Linux) - return "lib\(self).so" + return "\(Self.dylibPrefix)\(self)\(Self.linuxDylibExtension)" #else - return "lib\(self).dylib" + return "\(Self.dylibPrefix)\(self)\(Self.macOSDylibExtension)" #endif } } diff --git a/Sources/skit/Skit+Parse.swift b/Sources/skit/Skit+Parse.swift index 47ad22d..f54328b 100644 --- a/Sources/skit/Skit+Parse.swift +++ b/Sources/skit/Skit+Parse.swift @@ -33,8 +33,11 @@ import SyntaxParser extension Skit { internal struct Parse: ParsableCommand { + /// The subcommand name as invoked on the command line. + internal static let commandName = "parse" + internal static let configuration = CommandConfiguration( - commandName: "parse", + commandName: commandName, abstract: "Parse Swift source on stdin into a JSON syntax tree on stdout." ) diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index d7d076c..7aad6c0 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -43,8 +43,30 @@ extension Skit { /// output is written into a mirrored tree under `-o`. The actual work is /// delegated to a `Runner` value (`Runner.swift`). internal struct Run: AsyncParsableCommand { + /// The subcommand name as invoked on the command line. + internal static let commandName = "run" + + /// User-facing option/flag names (the `--long` CLI surface). + internal static let outputOptionName = "output" + internal static let libOptionName = "lib" + internal static let noCacheFlagName = "no-cache" + internal static let timeoutOptionName = "timeout" + internal static let noToolchainCheckFlagName = "no-toolchain-check" + + /// Environment variable holding an override for the libSyntaxKit directory. + internal static let libDirEnvironmentKey = "SKIT_LIB_DIR" + + /// Prefix for skit's own diagnostics written to stderr. + internal static let messagePrefix = "skit: " + + /// Argument passed to `swift` to capture the toolchain version banner. + internal static let versionFlag = "--version" + + /// Path to the script that rebuilds the self-contained release bundle. + internal static let buildReleaseScriptPath = "Scripts/build-skit-release.sh" + internal static let configuration = CommandConfiguration( - commandName: "run", + commandName: commandName, abstract: "Render SyntaxKit DSL input(s) into Swift source." ) @@ -52,31 +74,31 @@ extension Skit { internal var input: String @Option( - name: [.short, .customLong("output")], + name: [.short, .customLong(Run.outputOptionName)], help: "Output file (single-file mode) or directory (folder mode)." ) internal var output: String? @Option( - name: .customLong("lib"), + name: .customLong(Run.libOptionName), help: "Directory containing libSyntaxKit.dylib + module files." ) internal var libPath: String? @Flag( - name: .customLong("no-cache"), + name: .customLong(Run.noCacheFlagName), help: "Skip the rendered-output cache (always run swift)." ) internal var noCache: Bool = false @Option( - name: .customLong("timeout"), + name: .customLong(Run.timeoutOptionName), help: "Per-input timeout for the spawned `swift` in seconds (0 disables)." ) internal var timeoutSeconds: Int = 60 @Flag( - name: .customLong("no-toolchain-check"), + name: .customLong(Run.noToolchainCheckFlagName), help: "Skip the bundle/local Swift-toolchain comparison." ) internal var noToolchainCheck: Bool = false @@ -84,7 +106,8 @@ extension Skit { internal func validate() throws { guard timeoutSeconds >= 0 else { throw ValidationError( - "--timeout expects a non-negative integer (seconds), got: \(timeoutSeconds)" + "--\(Self.timeoutOptionName) expects a non-negative integer (seconds), " + + "got: \(timeoutSeconds)" ) } } @@ -96,7 +119,7 @@ extension Skit { // live. The error message lists the four lookup paths in priority order. let libPath: String do { - let envLibPath = ProcessInfo.processInfo.environment["SKIT_LIB_DIR"].flatMap { + let envLibPath = ProcessInfo.processInfo.environment[Self.libDirEnvironmentKey].flatMap { $0.isEmpty ? nil : $0 } libPath = try Bundle.main.resolveLibPath(candidates: self.libPath, envLibPath) @@ -149,9 +172,10 @@ extension Skit { #else // Subprocess is the only backend skit knows how to use to spawn // `swift`/`swiftc`. Without it (Windows, embedded), `run` cannot work. - FileHandle.standardError.write( - Data("skit: run is not supported on this platform (no Subprocess backend).\n".utf8) - ) + let message = + "\(Self.messagePrefix)run is not supported on this platform " + + "(no Subprocess backend).\n" + FileHandle.standardError.write(Data(message.utf8)) throw ExitCode(1) #endif } diff --git a/Sources/skit/Skit.Run+Render.swift b/Sources/skit/Skit.Run+Render.swift index c42919c..d7fc3e4 100644 --- a/Sources/skit/Skit.Run+Render.swift +++ b/Sources/skit/Skit.Run+Render.swift @@ -113,7 +113,7 @@ import SyntaxKit // surface the original "failed to walk" framing the CLI used to print // before the SDK split. FileHandle.standardError.write( - Data("skit: failed to walk \(input): \(underlying)\n".utf8) + Data("\(Self.messagePrefix)failed to walk \(input): \(underlying)\n".utf8) ) throw ExitCode(1) } catch { @@ -123,7 +123,8 @@ import SyntaxKit } if result.outcomes.isEmpty { - FileHandle.standardError.write(Data("skit: no .swift inputs under \(input)\n".utf8)) + FileHandle.standardError.write( + Data("\(Self.messagePrefix)no .swift inputs under \(input)\n".utf8)) return } @@ -144,8 +145,8 @@ import SyntaxKit FileHandle.standardError.write( Data( - "skit: \(result.outcomes.count - result.failureCount)/\(result.outcomes.count) succeeded\n" - .utf8 + ("\(Self.messagePrefix)\(result.outcomes.count - result.failureCount)" + + "/\(result.outcomes.count) succeeded\n").utf8 ) ) @@ -157,8 +158,8 @@ import SyntaxKit /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. internal func captureSwiftVersion() async -> String? { let result = try? await Subprocess.run( - .name("swift"), - arguments: ["--version"], + .name(Skit.swiftExecutableName), + arguments: [Self.versionFlag], output: .string(limit: 4_096), error: .discarded ) @@ -169,7 +170,7 @@ import SyntaxKit /// differs from the local one, explaining why and how to recover. internal func toolchainMismatchMessage(bundle: String, local: String) -> String { """ - skit: toolchain mismatch + \(Self.messagePrefix)toolchain mismatch bundle: \(bundle) local: \(local) The bundle's libSyntaxKit was built against a different `swift` than the @@ -178,8 +179,8 @@ import SyntaxKit diagnostic. Rebuild the bundle with: - Scripts/build-skit-release.sh - Or pass --no-toolchain-check to try anyway. + \(Self.buildReleaseScriptPath) + Or pass --\(Self.noToolchainCheckFlagName) to try anyway. """ } diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index d5cb936..73c4ee2 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -37,8 +37,15 @@ import ArgumentParser /// live in `Skit+Run.swift` and `Skit+Parse.swift` respectively. @main internal struct Skit: AsyncParsableCommand { + /// The top-level command name as invoked on the command line. + internal static let commandName = "skit" + + /// Name of the `swift` executable resolved on `PATH`. Shared by the + /// toolchain-version capture and the Subprocess `swift` configuration. + internal static let swiftExecutableName = "swift" + internal static let configuration = CommandConfiguration( - commandName: "skit", + commandName: commandName, abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", subcommands: [Run.self, Parse.self], defaultSubcommand: Run.self diff --git a/Sources/skit/Subprocess.Configuration+Swift.swift b/Sources/skit/Subprocess.Configuration+Swift.swift index a96438a..526cf4d 100644 --- a/Sources/skit/Subprocess.Configuration+Swift.swift +++ b/Sources/skit/Subprocess.Configuration+Swift.swift @@ -40,6 +40,18 @@ private static let stdoutLimitBytes = 16 * 1_024 * 1_024 private static let stderrLimitBytes = 1 * 1_024 * 1_024 + /// Subdirectory of the lib dir holding the SwiftSyntax CShims headers. + private static let cShimsIncludeSuffix = "_SwiftSyntaxCShims-include" + + /// swiftc flags used to build the `swift` invocation. + private static let flagSuppressWarnings = "-suppress-warnings" + private static let flagInclude = "-I" + private static let flagLibrarySearchPath = "-L" + private static let flagLinkSyntaxKit = "-lSyntaxKit" + private static let flagPassToClang = "-Xcc" + private static let flagPassToLinker = "-Xlinker" + private static let flagRPath = "-rpath" + /// A configuration that runs the `swift` interpreter on `wrappedPath`, /// linked against `libSyntaxKit` in `libPath`. /// @@ -47,17 +59,17 @@ /// headers, and set rpath so the dylib loads at runtime. The executable is /// resolved by name on `PATH`. internal static func swift(libPath: String, wrappedPath: String) -> Self { - let cShimsInclude = "\(libPath)/_SwiftSyntaxCShims-include" + let cShimsInclude = "\(libPath)/\(cShimsIncludeSuffix)" let arguments: [String] = [ - "-suppress-warnings", - "-I", libPath, - "-L", libPath, - "-lSyntaxKit", - "-Xcc", "-I", "-Xcc", cShimsInclude, - "-Xlinker", "-rpath", "-Xlinker", libPath, + flagSuppressWarnings, + flagInclude, libPath, + flagLibrarySearchPath, libPath, + flagLinkSyntaxKit, + flagPassToClang, flagInclude, flagPassToClang, cShimsInclude, + flagPassToLinker, flagRPath, flagPassToLinker, libPath, wrappedPath, ] - return Self(executable: .name("swift"), arguments: Arguments(arguments)) + return Self(executable: .name(Skit.swiftExecutableName), arguments: Arguments(arguments)) } /// Spawns `swift` for the render `invocation` and normalizes the result From 0565caa45e453d2cf79872e7e0505b970c4db6bc Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 9 Jun 2026 15:04:03 -0400 Subject: [PATCH 48/56] Refactor skit run() error handling and abstract the Subprocess backend Centralize the run command's failure handling and isolate the Subprocess dependency behind an SDK-level protocol so the command logic is platform-clean. - Add Skit.Run.CommandError: a self-describing typed error (diagnostic + terminalError) so run() maps every failure to its stderr + ExitCode/ ValidationError in one place; the render chain is now throws(CommandError) end-to-end. - Move the render-session setup into a SyntaxKit `Runner` initializer (Runner+Session.swift) that resolves the lib dir, gates the toolchain check, builds the cache, and assembles the Runner, throwing a typed Runner.SetupError. swiftVersion + the spawn closure are injected so SyntaxKit stays Subprocess-free. - Add a SwiftBackend protocol in SyntaxKit (capture-version + spawn-swift) with a SubprocessBackend conformance in skit, held in an optional that's non-nil only when canImport(Subprocess). run() guards it (nil -> unsupportedPlatform) and no longer branches on #if. - Ungate RunError/RunInput/DirectoryRender (Subprocess-free; the ungated Runner already depended on them) and the Skit.Run+Render extension; #if now survives only where Subprocess types are genuinely used. Behavior (stderr text + exit codes) is unchanged. swift build, swift test (398 tests), and ./Scripts/lint.sh all pass. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../Execution/Bundle+ResolveLibPath.swift | 2 +- .../SyntaxKit/Execution/DirectoryRender.swift | 80 +++--- Sources/SyntaxKit/Execution/RunError.swift | 36 ++- Sources/SyntaxKit/Execution/RunInput.swift | 58 ++--- .../SyntaxKit/Execution/Runner+Session.swift | 100 +++++++ .../SyntaxKit/Execution/SwiftBackend.swift | 45 ++++ Sources/skit/Skit+Run.swift | 117 ++++----- Sources/skit/Skit.Run+CommandError.swift | 131 ++++++++++ Sources/skit/Skit.Run+Render.swift | 243 ++++++++---------- Sources/skit/SubprocessBackend.swift | 71 +++++ 10 files changed, 584 insertions(+), 299 deletions(-) create mode 100644 Sources/SyntaxKit/Execution/Runner+Session.swift create mode 100644 Sources/SyntaxKit/Execution/SwiftBackend.swift create mode 100644 Sources/skit/Skit.Run+CommandError.swift create mode 100644 Sources/skit/SubprocessBackend.swift diff --git a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift index ccaaea1..f4a3f45 100644 --- a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift +++ b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift @@ -44,7 +44,7 @@ extension Bundle { /// from `executableURL`: /// - `/lib` (adjacent layout) /// - `/../lib/skit` (Homebrew layout) - package func resolveLibPath(candidates: String?...) throws -> String { + package func resolveLibPath(candidates: [String?]) throws -> String { let fileManager = FileManager.default for candidate in candidates { diff --git a/Sources/SyntaxKit/Execution/DirectoryRender.swift b/Sources/SyntaxKit/Execution/DirectoryRender.swift index f274189..4229dd4 100644 --- a/Sources/SyntaxKit/Execution/DirectoryRender.swift +++ b/Sources/SyntaxKit/Execution/DirectoryRender.swift @@ -27,53 +27,49 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +package import Foundation - package import Foundation +/// Result of `Runner.renderDirectory`. Successful rendered outputs are +/// written into the mirrored output tree as the batch progresses; this +/// value is the post-batch summary the caller inspects to decide +/// presentation (logging, exit code, etc.). Per-input failures are +/// captured here, not thrown, so a single bad input doesn't tear the +/// batch down. +package struct DirectoryRender: Sendable { + /// Per-input result. `stderr` carries the (possibly path-rewritten) + /// diagnostics from the spawned `swift`; it may be present whether or not + /// the input succeeded (e.g. a successful render that emitted warnings). + /// `result` is `nil` when the rendered output was written to its mirrored + /// destination, and carries the error when the input could not be rendered + /// or its output could not be written (typically a `RunError`). + package struct FileOutcome: Sendable { + package let input: URL + package let stderr: String + package let result: (any Error)? - /// Result of `Runner.renderDirectory`. Successful rendered outputs are - /// written into the mirrored output tree as the batch progresses; this - /// value is the post-batch summary the caller inspects to decide - /// presentation (logging, exit code, etc.). Per-input failures are - /// captured here, not thrown, so a single bad input doesn't tear the - /// batch down. - package struct DirectoryRender: Sendable { - /// Per-input result. `stderr` carries the (possibly path-rewritten) - /// diagnostics from the spawned `swift`; it may be present whether or not - /// the input succeeded (e.g. a successful render that emitted warnings). - /// `result` is `nil` when the rendered output was written to its mirrored - /// destination, and carries the error when the input could not be rendered - /// or its output could not be written (typically a `RunError`). - package struct FileOutcome: Sendable { - package let input: URL - package let stderr: String - package let result: (any Error)? - - package init( - input: URL, - stderr: String, - result: (any Error)? - ) { - self.input = input - self.stderr = stderr - self.result = result - } + package init( + input: URL, + stderr: String, + result: (any Error)? + ) { + self.input = input + self.stderr = stderr + self.result = result } + } - package let outcomes: [FileOutcome] + package let outcomes: [FileOutcome] - package init(outcomes: [FileOutcome]) { - self.outcomes = outcomes - } + package init(outcomes: [FileOutcome]) { + self.outcomes = outcomes + } - /// Number of inputs whose `result` is `.failure` — the signal the caller - /// uses to map a partially-failed batch to a non-zero exit (or whatever - /// failure semantics fit the host). - package var failureCount: Int { - outcomes.reduce(into: 0) { count, outcome in - if outcome.result != nil { count += 1 } - } + /// Number of inputs whose `result` is `.failure` — the signal the caller + /// uses to map a partially-failed batch to a non-zero exit (or whatever + /// failure semantics fit the host). + package var failureCount: Int { + outcomes.reduce(into: 0) { count, outcome in + if outcome.result != nil { count += 1 } } } - -#endif +} diff --git a/Sources/SyntaxKit/Execution/RunError.swift b/Sources/SyntaxKit/Execution/RunError.swift index fe4e1eb..dcefd2f 100644 --- a/Sources/SyntaxKit/Execution/RunError.swift +++ b/Sources/SyntaxKit/Execution/RunError.swift @@ -27,23 +27,19 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) - - /// Typed error surfaced by `Runner`. It decouples the renderer from any - /// particular caller: `Runner` reports *what* went wrong, and the caller - /// (CLI, build plugin, in-process driver) decides how to present it. - package enum RunError: Error { - /// The input path was invalid — missing, or a directory given without an - /// output directory. - case invalidInput(String) - /// Single-file render: the spawned `swift` exited non-zero. Carries that - /// code (e.g. a compile failure, `124` on timeout, `128 + signal`) and - /// the (path-rewritten) stderr the toolchain emitted, so the caller can - /// surface diagnostics without having to fish them out elsewhere. - case renderFailed(exitCode: Int32, stderr: String) - /// A wrapped Foundation/Subprocess failure (file read/write, spawn error) - /// that has no dedicated mapping. - case unexpected(any Error) - } - -#endif +/// Typed error surfaced by `Runner`. It decouples the renderer from any +/// particular caller: `Runner` reports *what* went wrong, and the caller +/// (CLI, build plugin, in-process driver) decides how to present it. +package enum RunError: Error { + /// The input path was invalid — missing, or a directory given without an + /// output directory. + case invalidInput(String) + /// Single-file render: the spawned `swift` exited non-zero. Carries that + /// code (e.g. a compile failure, `124` on timeout, `128 + signal`) and + /// the (path-rewritten) stderr the toolchain emitted, so the caller can + /// surface diagnostics without having to fish them out elsewhere. + case renderFailed(exitCode: Int32, stderr: String) + /// A wrapped Foundation/Subprocess failure (file read/write, spawn error) + /// that has no dedicated mapping. + case unexpected(any Error) +} diff --git a/Sources/SyntaxKit/Execution/RunInput.swift b/Sources/SyntaxKit/Execution/RunInput.swift index 24d728b..173251d 100644 --- a/Sources/SyntaxKit/Execution/RunInput.swift +++ b/Sources/SyntaxKit/Execution/RunInput.swift @@ -27,39 +27,35 @@ // OTHER DEALINGS IN THE SOFTWARE. // -#if canImport(Subprocess) +import Foundation - import Foundation +/// Whether an input path resolves to a single `.swift` file or a directory +/// of them — the two modes a `Runner` caller dispatches into. Built by +/// `resolve(input:output:)`, which stats the path and enforces the per-mode +/// output rules so the caller can `switch` on a settled value instead of +/// juggling an `ObjCBool`. +package enum RunInput { + /// A single input file. `outputPath` is whatever the caller intends to + /// do with the rendered bytes (write to a file, ignore, etc.); `Runner` + /// itself does not act on it. + case singleFile(inputPath: String, outputPath: String?) + /// A directory of inputs mirrored into `outputDir` (always required). + case directory(inputDir: String, outputDir: String) - /// Whether an input path resolves to a single `.swift` file or a directory - /// of them — the two modes a `Runner` caller dispatches into. Built by - /// `resolve(input:output:)`, which stats the path and enforces the per-mode - /// output rules so the caller can `switch` on a settled value instead of - /// juggling an `ObjCBool`. - package enum RunInput { - /// A single input file. `outputPath` is whatever the caller intends to - /// do with the rendered bytes (write to a file, ignore, etc.); `Runner` - /// itself does not act on it. - case singleFile(inputPath: String, outputPath: String?) - /// A directory of inputs mirrored into `outputDir` (always required). - case directory(inputDir: String, outputDir: String) - - /// Classifies `input` by stat: existing directory → `.directory`, - /// existing file → `.singleFile`. Throws `RunError.invalidInput` if the path - /// doesn't exist, or if a directory input wasn't given an explicit output. - package static func resolve(input: String, output: String?) throws(RunError) -> RunInput { - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { - throw RunError.invalidInput("input does not exist: \(input)") - } - if isDirectory.boolValue { - guard let output else { - throw RunError.invalidInput("directory inputs require -o ") - } - return .directory(inputDir: input, outputDir: output) + /// Classifies `input` by stat: existing directory → `.directory`, + /// existing file → `.singleFile`. Throws `RunError.invalidInput` if the path + /// doesn't exist, or if a directory input wasn't given an explicit output. + package static func resolve(input: String, output: String?) throws(RunError) -> RunInput { + var isDirectory: ObjCBool = false + guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + throw RunError.invalidInput("input does not exist: \(input)") + } + if isDirectory.boolValue { + guard let output else { + throw RunError.invalidInput("directory inputs require -o ") } - return .singleFile(inputPath: input, outputPath: output) + return .directory(inputDir: input, outputDir: output) } + return .singleFile(inputPath: input, outputPath: output) } - -#endif +} diff --git a/Sources/SyntaxKit/Execution/Runner+Session.swift b/Sources/SyntaxKit/Execution/Runner+Session.swift new file mode 100644 index 0000000..669524c --- /dev/null +++ b/Sources/SyntaxKit/Execution/Runner+Session.swift @@ -0,0 +1,100 @@ +// +// Runner+Session.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +extension Runner { + /// Why a render session couldn't be brought up. Decoupled from any caller: + /// the initializer reports *what* failed; the caller (CLI, build plugin, + /// in-process driver) decides how to present it and which exit code to use. + package enum SetupError: Error { + /// The libSyntaxKit directory couldn't be resolved from the supplied + /// candidates or the bundle-relative fallbacks. Carries the underlying + /// `CLIError` describing the lookup. + case libResolutionFailed(any Error) + /// The bundle's recorded `swift --version` differs from the local one, + /// so spawning `swift` would hit a swiftmodule-version mismatch. + case toolchainMismatch(bundle: String, local: String) + } + + /// Brings up a render-ready `Runner`: resolves the libSyntaxKit directory + /// from `libCandidates` (falling back to bundle-relative layouts), optionally + /// gates on the bundle/local toolchain comparison, and wires up the output + /// cache. + /// + /// Platform-agnostic by construction — the two inputs that need a Subprocess + /// backend are injected by the caller: the already-captured `swiftVersion` + /// (`swift --version` output, or nil if capture failed) and the `run` closure + /// that actually spawns `swift` for one `SwiftInvocation`. + /// + /// - Throws: `SetupError` for a lookup or toolchain failure, so the caller + /// owns the presentation and exit mapping. + package init( + libCandidates: [String?], + swiftVersion: String?, + enforceToolchainCheck: Bool, + useCache: Bool, + timeoutSeconds: Int, + run: @Sendable @escaping (SwiftInvocation) async throws -> SwiftRunOutcome + ) throws(SetupError) { + // 1. Resolve the libSyntaxKit bundle dir. Failure is fatal — there's no + // dylib + swiftmodules to link against without it. + let libPath: String + do { + libPath = try Bundle.main.resolveLibPath(candidates: libCandidates) + } catch { + throw SetupError.libResolutionFailed(error) + } + + // 2. Compare the bundle's recorded `swift --version` against the local one. + // swiftmodules aren't reliably forward-compatible across compiler versions, + // so a mismatch is surfaced rather than letting the spawned `swift` emit a + // cryptic module-version diagnostic. + if enforceToolchainCheck { + switch ToolchainCheckResult(libPath: libPath, swiftVersion: swiftVersion) { + case .match, .stampMissing: + break + case .mismatch(let bundle, let local): + throw SetupError.toolchainMismatch(bundle: bundle, local: local) + } + } + + // 3. Build the output cache (nil when disabled). The captured `swiftVersion` + // is bound in so per-input key derivation doesn't re-spawn `swift`. + let cache: OutputCache? = useCache ? OutputCache(swiftVersion: swiftVersion) : nil + + // 4. Delegate to the designated initializer, binding the spawn closure. + self.init( + libPath: libPath, + cache: cache, + timeoutSeconds: timeoutSeconds, + run: run + ) + } +} diff --git a/Sources/SyntaxKit/Execution/SwiftBackend.swift b/Sources/SyntaxKit/Execution/SwiftBackend.swift new file mode 100644 index 0000000..9402fb2 --- /dev/null +++ b/Sources/SyntaxKit/Execution/SwiftBackend.swift @@ -0,0 +1,45 @@ +// +// SwiftBackend.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +/// The two `swift`-toolchain operations a render session needs from a spawn +/// backend: capture the local toolchain version (feeds the toolchain check and +/// the cache key), and run `swift` for one `SwiftInvocation`. +/// +/// SyntaxKit defines the contract; a Subprocess-based conformance lives in the +/// `skit` CLI. `Sendable` so the spawn method can be captured in `Runner`'s +/// `@Sendable` run closure. +package protocol SwiftBackend: Sendable { + /// Verbatim `swift --version` output, or nil if the toolchain couldn't be + /// queried. + func captureSwiftVersion() async -> String? + + /// Spawns `swift` for `invocation` and normalizes the result into a + /// `SwiftRunOutcome`. + func runSwift(for invocation: SwiftInvocation) async throws -> SwiftRunOutcome +} diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index 7aad6c0..a90b2a3 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -31,10 +31,6 @@ import ArgumentParser import Foundation import SyntaxKit -#if canImport(Subprocess) - import Subprocess -#endif - extension Skit { /// Render one or more SyntaxKit DSL files into Swift source. /// @@ -112,72 +108,57 @@ extension Skit { } } - internal func run() async throws { - #if canImport(Subprocess) - // 1. Resolve the libSyntaxKit bundle dir. Failure here is fatal — we - // can't spawn `swift` without knowing where the dylib + swiftmodules - // live. The error message lists the four lookup paths in priority order. - let libPath: String - do { - let envLibPath = ProcessInfo.processInfo.environment[Self.libDirEnvironmentKey].flatMap { - $0.isEmpty ? nil : $0 - } - libPath = try Bundle.main.resolveLibPath(candidates: self.libPath, envLibPath) - } catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - throw ExitCode(2) - } - - // 2. Capture local `swift --version` once. Feeds both the toolchain - // check (compares against the bundle stamp) and the output cache key - // (one shard per toolchain). Doing this here means we spawn - // `swift --version` exactly once per `skit run` invocation rather - // than once per input. - let swiftVersion = await captureSwiftVersion() - - // 3. Compare the bundle's recorded `swift --version` against the local - // one. swiftmodules aren't reliably forward-compatible across compiler - // versions, so a mismatch produces a clear error rather than letting - // the spawned `swift` emit a cryptic module-version diagnostic. - if !noToolchainCheck { - switch ToolchainCheckResult(libPath: libPath, swiftVersion: swiftVersion) { - case .match, .stampMissing: - break - case .mismatch(let bundle, let local): - FileHandle.standardError.write( - Data(toolchainMismatchMessage(bundle: bundle, local: local).utf8)) - throw ExitCode(2) - } - } + internal func execute() async throws(CommandError) { + // The spawn backend is nil only where there's no Subprocess backend + // (Windows, embedded) — there `run` can't spawn `swift`/`swiftc`. + guard let backend = Self.swiftBackend else { + throw CommandError.unsupportedPlatform + } - // 4. Build the output cache (nil under `--no-cache`). The captured - // `swiftVersion` is bound into the instance so per-input key derivation - // doesn't re-spawn `swift`. - let cache: OutputCache? = noCache ? nil : OutputCache(swiftVersion: swiftVersion) - - // 5. Bind the per-invocation configuration into a Runner. skit supplies - // the Subprocess-backed `run` closure — the one seam between the - // platform-agnostic engine in SyntaxKit and the Subprocess backend. - let runner = Runner( - libPath: libPath, - cache: cache, + // Capture the two backend-dependent inputs to the otherwise + // platform-agnostic session setup: the local `swift --version` (feeds + // the toolchain check + the cache key, spawned exactly once per run) + // and the SKIT_LIB_DIR override. + let swiftVersion = await backend.captureSwiftVersion() + let envLibPath = ProcessInfo.processInfo.environment[Self.libDirEnvironmentKey] + .flatMap { $0.isEmpty ? nil : $0 } + + // Resolve lib dir → toolchain-gate → cache → assemble Runner. That + // orchestration lives in SyntaxKit (the `Runner` session initializer); + // skit injects the captured `swiftVersion` and the backend's spawn + // method, and maps the typed setup failure onto the CLI's exit policy. + let runner: Runner + do { + runner = try Runner( + libCandidates: [self.libPath, envLibPath], + swiftVersion: swiftVersion, + enforceToolchainCheck: !noToolchainCheck, + useCache: !noCache, timeoutSeconds: timeoutSeconds - ) { try await Subprocess.Configuration.runSwift(for: $0) } - - // 6. Hand the input off to the runner: it classifies single-file vs. - // directory mode (validating existence and the `-o` requirement) and - // renders accordingly, reporting failures via the typed `RunError`. - // The command layer owns the mapping from failure to process exit. - try await render(using: runner, input: input, output: output) - #else - // Subprocess is the only backend skit knows how to use to spawn - // `swift`/`swiftc`. Without it (Windows, embedded), `run` cannot work. - let message = - "\(Self.messagePrefix)run is not supported on this platform " - + "(no Subprocess backend).\n" - FileHandle.standardError.write(Data(message.utf8)) - throw ExitCode(1) - #endif + ) { try await backend.runSwift(for: $0) } + } catch { + throw CommandError(error) + } + + // Hand the input off to the runner: `render` owns presentation and the + // single-file/directory dispatch, translating render failures into + // `CommandError` for the outer catch to map. + try await render(using: runner, input: input, output: output) + } + + /// Renders the input(s), with a single seam between the pipeline and the + /// process: every step `throw`s a typed `CommandError`, and the outer catch + /// maps that to its stderr diagnostic + terminal `ExitCode`/`ValidationError`. + /// Any non-`CommandError` propagates to ArgumentParser unchanged. + internal func run() async throws { + do { + try await self.execute() + } catch { + if let diagnostic = error.diagnostic { + FileHandle.standardError.write(Data(diagnostic.utf8)) + } + throw error.terminalError + } } } } diff --git a/Sources/skit/Skit.Run+CommandError.swift b/Sources/skit/Skit.Run+CommandError.swift new file mode 100644 index 0000000..0d0feb7 --- /dev/null +++ b/Sources/skit/Skit.Run+CommandError.swift @@ -0,0 +1,131 @@ +// +// Skit.Run+CommandError.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import ArgumentParser +import Foundation +import SyntaxKit + +extension Skit.Run { + /// Typed failures the `run` pipeline can produce. The render steps just + /// `throw` the case that describes *what* failed; `run()` catches these in + /// one place and converts each to its stderr diagnostic + process exit, so + /// the steps stay free of presentation/exit logic. + internal enum CommandError: Error { + /// A usage error (bad input path, or a directory given without `-o`). + /// Surfaced as an ArgumentParser `ValidationError` (exit 64), which prints + /// the message and the command's usage. + case usage(String) + /// The libSyntaxKit directory couldn't be resolved. Exit 2. + case libResolutionFailed(any Error) + /// The bundle's recorded `swift --version` differs from the local one. + /// Exit 2. + case toolchainMismatch(bundle: String, local: String) + /// The spawned `swift` exited non-zero (compile failure, `124` on timeout, + /// `128 + signal`). Carries that code + the toolchain stderr; the code is + /// passed through as the process exit. + case renderFailed(exitCode: Int32, stderr: String) + /// A directory batch couldn't be walked. Exit 1. + case directoryWalkFailed(input: String, underlying: any Error) + /// A render/batch failure whose diagnostics were already surfaced (per-input + /// output, a printed summary, or none). Exit 1 with nothing further to say. + case failed + /// A wrapped Foundation/Subprocess failure with no dedicated mapping; the + /// underlying error is rethrown so ArgumentParser prints it (exit 1). + case unexpected(any Error) + /// `run` was invoked on a platform without a Subprocess backend. Exit 1. + case unsupportedPlatform + + /// Maps a SyntaxKit session-setup failure onto the CLI's exit policy. + internal init(_ error: Runner.SetupError) { + switch error { + case .libResolutionFailed(let underlying): + self = .libResolutionFailed(underlying) + case .toolchainMismatch(let bundle, let local): + self = .toolchainMismatch(bundle: bundle, local: local) + } + } + + /// stderr text to emit before exiting, or nil. `.usage` is printed by + /// `ValidationError`; `.failed`/`.unexpected` print nothing here (their + /// diagnostics were already surfaced, or ArgumentParser prints them). + internal var diagnostic: String? { + switch self { + case .usage, .failed, .unexpected: + return nil + case .libResolutionFailed(let error): + return "\(error)\n" + case .toolchainMismatch(let bundle, let local): + return Self.toolchainMismatchMessage(bundle: bundle, local: local) + case .renderFailed(_, let stderr): + return stderr.isEmpty ? nil : stderr + case .directoryWalkFailed(let input, let underlying): + return "\(Skit.Run.messagePrefix)failed to walk \(input): \(underlying)\n" + case .unsupportedPlatform: + return "\(Skit.Run.messagePrefix)run is not supported on this platform " + + "(no Subprocess backend).\n" + } + } + + /// The terminal error ArgumentParser acts on: a `ValidationError` (usage), + /// an `ExitCode`, or the wrapped underlying error. + internal var terminalError: any Error { + switch self { + case .usage(let message): + return ValidationError(message) + case .libResolutionFailed, .toolchainMismatch: + return ExitCode(2) + case .renderFailed(let exitCode, _): + return ExitCode(exitCode) + case .directoryWalkFailed, .failed, .unsupportedPlatform: + return ExitCode(1) + case .unexpected(let underlying): + return underlying + } + } + + /// Human-readable error explaining why the bundle's recorded + /// `swift --version` differs from the local one, and how to recover. + private static func toolchainMismatchMessage(bundle: String, local: String) -> String { + """ + \(Skit.Run.messagePrefix)toolchain mismatch + bundle: \(bundle) + local: \(local) + The bundle's libSyntaxKit was built against a different `swift` than the + one on your PATH. Swift swiftmodules aren't reliably compatible across + versions, so spawning `swift` would fail with a cryptic module-version + diagnostic. + + Rebuild the bundle with: + \(Skit.Run.buildReleaseScriptPath) + Or pass --\(Skit.Run.noToolchainCheckFlagName) to try anyway. + + """ + } + } +} diff --git a/Sources/skit/Skit.Run+Render.swift b/Sources/skit/Skit.Run+Render.swift index d7fc3e4..3e8a6e5 100644 --- a/Sources/skit/Skit.Run+Render.swift +++ b/Sources/skit/Skit.Run+Render.swift @@ -27,163 +27,132 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import ArgumentParser import Foundation import SyntaxKit -#if canImport(Subprocess) - - import Subprocess - - extension Skit.Run { - /// Classifies `input` and dispatches to the matching `Runner` SDK method. - /// The SDK layer is silent — this function (and its helpers) own all - /// stdout/stderr presentation, file IO for single-file mode, and the - /// mapping from `RunError`/batch failures to the process exit behaviour - /// the CLI promises: `ValidationError` (exit 64) for usage errors, - /// `ExitCode` for render/batch failures, and rethrow for anything - /// unexpected (ArgumentParser prints + exit 1). - internal func render(using runner: Runner, input: String, output: String?) async throws { - let resolved: RunInput - do { - resolved = try RunInput.resolve(input: input, output: output) - } catch .invalidInput(let message) { - throw ValidationError(message) - } catch { - // RunInput.resolve only throws .invalidInput; defensive. - throw ExitCode(1) - } - - switch resolved { - case .singleFile(let inputPath, let outputPath): - try await renderSingle(runner: runner, input: inputPath, output: outputPath) - case .directory(let inputDir, let outputDir): - try await renderBatch(runner: runner, input: inputDir, output: outputDir) - } +extension Skit.Run { + /// Classifies `input` and dispatches to the matching `Runner` SDK method. + /// The SDK layer is silent — this function (and its helpers) own all + /// stdout/stderr presentation and file IO, and translate `RunError`/batch + /// failures into the typed `CommandError` that `run()` maps to a process + /// exit. Success-path output is still written here. + internal func render(using runner: Runner, input: String, output: String?) + async throws(CommandError) + { + let resolved: RunInput + do { + resolved = try RunInput.resolve(input: input, output: output) + } catch .invalidInput(let message) { + throw CommandError.usage(message) + } catch { + // RunInput.resolve only throws .invalidInput; defensive. + throw CommandError.failed } - /// Renders one input via `Runner.renderFile`, prints any compiler stderr, - /// and writes the rendered Swift source either to `outputPath` or to - /// stdout. The runner itself does none of that — that's the CLI's job. - fileprivate func renderSingle( - runner: Runner, - input: String, - output: String? - ) async throws { - let rendered: SingleFileRender - do { - rendered = try await runner.renderFile(input: input) - } catch .invalidInput(let message) { - throw ValidationError(message) - } catch .renderFailed(let exitCode, let stderr) { - if !stderr.isEmpty { - FileHandle.standardError.write(Data(stderr.utf8)) - } - throw ExitCode(exitCode) - } catch .unexpected(let underlying) { - throw underlying - } + switch resolved { + case .singleFile(let inputPath, let outputPath): + try await renderSingle(runner: runner, input: inputPath, output: outputPath) + case .directory(let inputDir, let outputDir): + try await renderBatch(runner: runner, input: inputDir, output: outputDir) + } + } - if !rendered.stderr.isEmpty { - FileHandle.standardError.write(Data(rendered.stderr.utf8)) - } - if let output { - try rendered.stdout.write(to: URL(fileURLWithPath: output)) - } else { - FileHandle.standardOutput.write(rendered.stdout) + /// Renders one input via `Runner.renderFile`, prints any compiler stderr, + /// and writes the rendered Swift source either to `outputPath` or to + /// stdout. The runner itself does none of that — that's the CLI's job. + fileprivate func renderSingle( + runner: Runner, + input: String, + output: String? + ) async throws(CommandError) { + let rendered: SingleFileRender + do { + rendered = try await runner.renderFile(input: input) + } catch { + // `error` is typed `RunError` (renderFile is `throws(RunError)`); the + // switch is exhaustive, so this single catch is both total (satisfies + // `throws(CommandError)`) and free of an unreachable clause. + switch error { + case .invalidInput(let message): + throw CommandError.usage(message) + case .renderFailed(let exitCode, let stderr): + throw CommandError.renderFailed(exitCode: exitCode, stderr: stderr) + case .unexpected(let underlying): + throw CommandError.unexpected(underlying) } } - /// Renders a directory batch via `Runner.renderDirectory`, prints per-input - /// diagnostics (fenced when several files emit them), surfaces non-render - /// failures, prints a one-line summary, and maps any failures to - /// `ExitCode(1)` — Tuist-analog batch semantics. - fileprivate func renderBatch( - runner: Runner, - input: String, - output: String - ) async throws { - let result: DirectoryRender + if !rendered.stderr.isEmpty { + FileHandle.standardError.write(Data(rendered.stderr.utf8)) + } + if let output { do { - result = try await runner.renderDirectory(inputDir: input, outputDir: output) - } catch .invalidInput(let message) { - throw ValidationError(message) - } catch .unexpected(let underlying) { - // `renderDirectory` only wraps directory-walk failures in `.unexpected`; - // surface the original "failed to walk" framing the CLI used to print - // before the SDK split. - FileHandle.standardError.write( - Data("\(Self.messagePrefix)failed to walk \(input): \(underlying)\n".utf8) - ) - throw ExitCode(1) + try rendered.stdout.write(to: URL(fileURLWithPath: output)) } catch { - // renderDirectory does not throw .renderFailed — that's a per-file - // outcome. Defensive. - throw ExitCode(1) - } - - if result.outcomes.isEmpty { - FileHandle.standardError.write( - Data("\(Self.messagePrefix)no .swift inputs under \(input)\n".utf8)) - return + // A write failure has no diagnostic of its own; surface the underlying + // error (ArgumentParser prints it, exit 1). + throw CommandError.unexpected(error) } + } else { + FileHandle.standardOutput.write(rendered.stdout) + } + } - for outcome in result.outcomes { - if !outcome.stderr.isEmpty { - FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) - FileHandle.standardError.write(Data(outcome.stderr.utf8)) - } - // .renderFailed already had its stderr surfaced above. Other failures - // (process spawn, write) carry the diagnostic in the error itself. - if let error = outcome.result { - if let runError = error as? RunError, case .renderFailed = runError { - continue - } - FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) - } - } + /// Renders a directory batch via `Runner.renderDirectory`, prints per-input + /// diagnostics (fenced when several files emit them), surfaces non-render + /// failures, prints a one-line summary, and maps any failures to + /// `ExitCode(1)` — Tuist-analog batch semantics. + fileprivate func renderBatch( + runner: Runner, + input: String, + output: String + ) async throws(CommandError) { + let result: DirectoryRender + do { + result = try await runner.renderDirectory(inputDir: input, outputDir: output) + } catch .invalidInput(let message) { + throw CommandError.usage(message) + } catch .unexpected(let underlying) { + // `renderDirectory` only wraps directory-walk failures in `.unexpected`; + // `CommandError.directoryWalkFailed` carries the original "failed to + // walk" framing the CLI prints. + throw CommandError.directoryWalkFailed(input: input, underlying: underlying) + } catch { + // renderDirectory does not throw .renderFailed — that's a per-file + // outcome. Defensive. + throw CommandError.failed + } + if result.outcomes.isEmpty { FileHandle.standardError.write( - Data( - ("\(Self.messagePrefix)\(result.outcomes.count - result.failureCount)" - + "/\(result.outcomes.count) succeeded\n").utf8 - ) - ) + Data("\(Self.messagePrefix)no .swift inputs under \(input)\n".utf8)) + return + } - if result.failureCount > 0 { - throw ExitCode(1) + for outcome in result.outcomes { + if !outcome.stderr.isEmpty { + FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) + FileHandle.standardError.write(Data(outcome.stderr.utf8)) + } + // .renderFailed already had its stderr surfaced above. Other failures + // (process spawn, write) carry the diagnostic in the error itself. + if let error = outcome.result { + if let runError = error as? RunError, case .renderFailed = runError { + continue + } + FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) } } - /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. - internal func captureSwiftVersion() async -> String? { - let result = try? await Subprocess.run( - .name(Skit.swiftExecutableName), - arguments: [Self.versionFlag], - output: .string(limit: 4_096), - error: .discarded + FileHandle.standardError.write( + Data( + ("\(Self.messagePrefix)\(result.outcomes.count - result.failureCount)" + + "/\(result.outcomes.count) succeeded\n").utf8 ) - return result?.standardOutput - } - - /// Human-readable error emitted when the bundle's recorded `swift --version` - /// differs from the local one, explaining why and how to recover. - internal func toolchainMismatchMessage(bundle: String, local: String) -> String { - """ - \(Self.messagePrefix)toolchain mismatch - bundle: \(bundle) - local: \(local) - The bundle's libSyntaxKit was built against a different `swift` than the - one on your PATH. Swift swiftmodules aren't reliably compatible across - versions, so spawning `swift` would fail with a cryptic module-version - diagnostic. + ) - Rebuild the bundle with: - \(Self.buildReleaseScriptPath) - Or pass --\(Self.noToolchainCheckFlagName) to try anyway. - - """ + if result.failureCount > 0 { + throw CommandError.failed } } - -#endif +} diff --git a/Sources/skit/SubprocessBackend.swift b/Sources/skit/SubprocessBackend.swift new file mode 100644 index 0000000..6ad7093 --- /dev/null +++ b/Sources/skit/SubprocessBackend.swift @@ -0,0 +1,71 @@ +// +// SubprocessBackend.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import SyntaxKit + +#if canImport(Subprocess) + + import Subprocess + + /// Subprocess-backed `SwiftBackend`: the real `swift`-spawning implementation + /// skit hands to the render session. The one seam between the platform- + /// agnostic engine in SyntaxKit and the Subprocess implementation. + internal struct SubprocessBackend: SwiftBackend { + /// Verbatim `swift --version` output, or nil on spawn failure. Capped at 4 KiB. + internal func captureSwiftVersion() async -> String? { + let result = try? await Subprocess.run( + .name(Skit.swiftExecutableName), + arguments: [Skit.Run.versionFlag], + output: .string(limit: 4_096), + error: .discarded + ) + return result?.standardOutput + } + + internal func runSwift( + for invocation: SwiftInvocation + ) async throws -> SwiftRunOutcome { + try await Subprocess.Configuration.runSwift(for: invocation) + } + } + + extension Skit.Run { + /// The active spawn backend, or nil when this platform has no Subprocess + /// backend (Windows, embedded) and `run` therefore can't spawn `swift`. + internal static let swiftBackend: (any SwiftBackend)? = SubprocessBackend() + } + +#else + + extension Skit.Run { + /// No Subprocess backend on this platform — `run` reports `unsupportedPlatform`. + internal static let swiftBackend: (any SwiftBackend)? = nil + } + +#endif From 80c9bb93efcccca3c30bd8e1b46230a66ad32f7f Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Tue, 9 Jun 2026 16:53:49 -0400 Subject: [PATCH 49/56] Fix CI; make Execution engine API public CI: - Remove Android API level 36 from the build matrix (33/34/35 pass; 36 fails). - Drop the `Duration` requirement in the timeout watchdog so the SyntaxKit library compiles on its minimum deployment targets. `Task.timeout` now takes `seconds: Int` and uses `Task.sleep(nanoseconds:)`, fixing the iOS 13 / tvOS 13 / watchOS 6 builds that failed on `'Duration' is only available in iOS 16 / tvOS 16 / watchOS 9`. - Move iOS and visionOS simulators to osVersion 26.5 so all four Apple device platforms run on 26.5; drop the stale 26.4 pin comment. Promote the Execution engine surface (Runner, RunInput, SwiftBackend, and related types) from `package` to `public`. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/SyntaxKit.yml | 9 +++---- .../Execution/Bundle+ResolveLibPath.swift | 4 +-- Sources/SyntaxKit/Execution/CLIError.swift | 11 +++++--- .../Execution/CollectInputsError.swift | 2 +- .../SyntaxKit/Execution/DirectoryRender.swift | 26 ++++++++++++------- Sources/SyntaxKit/Execution/OutputCache.swift | 15 ++++++----- .../SyntaxKit/Execution/ProcessResult.swift | 16 +++++++----- Sources/SyntaxKit/Execution/RunError.swift | 2 +- Sources/SyntaxKit/Execution/RunInput.swift | 4 +-- .../Execution/Runner+Directory.swift | 2 +- .../SyntaxKit/Execution/Runner+Session.swift | 4 +-- Sources/SyntaxKit/Execution/Runner.swift | 10 ++++--- .../Execution/SingleFileRender.swift | 12 +++++---- .../SyntaxKit/Execution/SwiftBackend.swift | 2 +- .../SyntaxKit/Execution/SwiftInvocation.swift | 9 ++++--- .../SyntaxKit/Execution/SwiftRunOutcome.swift | 2 +- .../SyntaxKit/Execution/Task+Timeout.swift | 12 ++++++--- .../Execution/ToolchainCheckResult.swift | 8 ++++-- .../SyntaxKit/Execution/WrappedSource.swift | 6 ++--- 19 files changed, 92 insertions(+), 64 deletions(-) diff --git a/.github/workflows/SyntaxKit.yml b/.github/workflows/SyntaxKit.yml index fb5cd8c..ca966bd 100644 --- a/.github/workflows/SyntaxKit.yml +++ b/.github/workflows/SyntaxKit.yml @@ -226,15 +226,12 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" - # iOS / watchOS / tvOS / visionOS osVersion pinned to 26.4 until the - # macos-26 runner image ships Xcode 26.5 simulators — see #160. - # iOS Build Matrix - type: ios runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "iPhone 17 Pro" - osVersion: "26.4.1" + osVersion: "26.5" download-platform: true # watchOS Build Matrix @@ -258,7 +255,7 @@ jobs: runs-on: macos-26 xcode: "/Applications/Xcode_26.5.app" deviceName: "Apple Vision Pro" - osVersion: "26.4.1" + osVersion: "26.5" download-platform: true steps: @@ -298,7 +295,7 @@ jobs: swift: - version: "6.2" - version: "6.3" - android-api-level: [33, 34, 35, 36] + android-api-level: [33, 34, 35] steps: - uses: actions/checkout@v6 - name: Free disk space diff --git a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift index f4a3f45..e48be61 100644 --- a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift +++ b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -package import Foundation +public import Foundation extension Bundle { /// Bundle-relative lib dir name for the adjacent layout (`/lib`). @@ -44,7 +44,7 @@ extension Bundle { /// from `executableURL`: /// - `/lib` (adjacent layout) /// - `/../lib/skit` (Homebrew layout) - package func resolveLibPath(candidates: [String?]) throws -> String { + public func resolveLibPath(candidates: [String?]) throws -> String { let fileManager = FileManager.default for candidate in candidates { diff --git a/Sources/SyntaxKit/Execution/CLIError.swift b/Sources/SyntaxKit/Execution/CLIError.swift index a5ee68e..f41c3a1 100644 --- a/Sources/SyntaxKit/Execution/CLIError.swift +++ b/Sources/SyntaxKit/Execution/CLIError.swift @@ -29,11 +29,14 @@ /// Throwable error wrapper for skit's user-facing diagnostics. The message /// is printed verbatim — keep it actionable (path, hint, next step). -package struct CLIError: Error, CustomStringConvertible { - package let message: String - package var description: String { message } +public struct CLIError: Error, CustomStringConvertible { + /// The user-facing diagnostic, printed verbatim. + public let message: String + /// The error description, identical to `message`. + public var description: String { message } - package init(message: String) { + /// Creates an error carrying a verbatim user-facing `message`. + public init(message: String) { self.message = message } } diff --git a/Sources/SyntaxKit/Execution/CollectInputsError.swift b/Sources/SyntaxKit/Execution/CollectInputsError.swift index 0f849fe..bb4a36e 100644 --- a/Sources/SyntaxKit/Execution/CollectInputsError.swift +++ b/Sources/SyntaxKit/Execution/CollectInputsError.swift @@ -31,7 +31,7 @@ /// to the two distinct ways the input walk can fail, so the caller can tell a /// directory it couldn't enumerate apart from a file whose resource values it /// couldn't read. -package enum CollectInputsError: Error { +public enum CollectInputsError: Error { /// The directory could not be enumerated (`FileManager.enumerator` returned /// nil). Carries a user-facing `CLIError` describing the path. case cliError(CLIError) diff --git a/Sources/SyntaxKit/Execution/DirectoryRender.swift b/Sources/SyntaxKit/Execution/DirectoryRender.swift index 4229dd4..6342e54 100644 --- a/Sources/SyntaxKit/Execution/DirectoryRender.swift +++ b/Sources/SyntaxKit/Execution/DirectoryRender.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -package import Foundation +public import Foundation /// Result of `Runner.renderDirectory`. Successful rendered outputs are /// written into the mirrored output tree as the batch progresses; this @@ -35,19 +35,23 @@ package import Foundation /// presentation (logging, exit code, etc.). Per-input failures are /// captured here, not thrown, so a single bad input doesn't tear the /// batch down. -package struct DirectoryRender: Sendable { +public struct DirectoryRender: Sendable { /// Per-input result. `stderr` carries the (possibly path-rewritten) /// diagnostics from the spawned `swift`; it may be present whether or not /// the input succeeded (e.g. a successful render that emitted warnings). /// `result` is `nil` when the rendered output was written to its mirrored /// destination, and carries the error when the input could not be rendered /// or its output could not be written (typically a `RunError`). - package struct FileOutcome: Sendable { - package let input: URL - package let stderr: String - package let result: (any Error)? + public struct FileOutcome: Sendable { + /// The input file this outcome describes. + public let input: URL + /// The (possibly path-rewritten) `swift` diagnostics for this input. + public let stderr: String + /// `nil` when the output was written; the error otherwise. + public let result: (any Error)? - package init( + /// Creates an outcome for one rendered (or failed) input. + public init( input: URL, stderr: String, result: (any Error)? @@ -58,16 +62,18 @@ package struct DirectoryRender: Sendable { } } - package let outcomes: [FileOutcome] + /// Per-input outcomes, in the order the batch produced them. + public let outcomes: [FileOutcome] - package init(outcomes: [FileOutcome]) { + /// Creates a batch summary from the collected per-input outcomes. + public init(outcomes: [FileOutcome]) { self.outcomes = outcomes } /// Number of inputs whose `result` is `.failure` — the signal the caller /// uses to map a partially-failed batch to a non-zero exit (or whatever /// failure semantics fit the host). - package var failureCount: Int { + public var failureCount: Int { outcomes.reduce(into: 0) { count, outcome in if outcome.result != nil { count += 1 } } diff --git a/Sources/SyntaxKit/Execution/OutputCache.swift b/Sources/SyntaxKit/Execution/OutputCache.swift index ac6c9f7..6200fcf 100644 --- a/Sources/SyntaxKit/Execution/OutputCache.swift +++ b/Sources/SyntaxKit/Execution/OutputCache.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -package import Foundation +public import Foundation /// On-disk cache of rendered skit output, content-keyed so a re-run on /// unchanged input skips the `swift` spawn entirely. @@ -37,7 +37,7 @@ package import Foundation /// across the concurrent `runOne` tasks in directory mode. The default /// singletons used in production (and the typical test doubles) are /// thread-safe for the operations we invoke. -package struct OutputCache: Sendable { +public struct OutputCache: Sendable { /// Bumped when the cache layout changes in a way that requires invalidation. private static let schemaVersion = "v1" @@ -82,7 +82,10 @@ package struct OutputCache: Sendable { /// nil if capture failed before construction. private let swiftVersion: String? - package init( + /// Creates a cache rooted under the SyntaxKit cache directory, keyed in part + /// by the captured `swiftVersion`. `fileManager`/`processInfo` are injectable + /// for testing. + public init( swiftVersion: String?, fileManager: @autoclosure @escaping @Sendable () -> FileManager = .default, processInfo: ProcessInfo = .processInfo @@ -98,7 +101,7 @@ package struct OutputCache: Sendable { /// version, libSyntaxKit stamp, sorted SKIT_*/SYNTAXKIT_* env vars). Any /// change in these inputs produces a fresh key and forces a recompile. /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. - package func key(forInput source: String, libPath: String) -> String { + public func key(forInput source: String, libPath: String) -> String { var hasher = ContentHasher() // Schema version: bump to invalidate every existing cache entry at once. hasher.update(data: Data(Self.schemaVersion.utf8)) @@ -131,13 +134,13 @@ package struct OutputCache: Sendable { } /// Returns the cached rendered output for `key`, or nil on miss. - package func lookup(key: String) -> Data? { + public func lookup(key: String) -> Data? { try? Data(contentsOf: directory(for: key).appendingPathComponent(Self.outputFileName)) } /// Atomically stores `data` under `key`. Concurrent writers race via a /// `tmp../` staging dir + rename; the loser drops their copy. - package func store(key: String, data: Data) throws { + public func store(key: String, data: Data) throws { let cacheRoot = directory(for: key) let final = cacheRoot.appendingPathComponent(Self.outputFileName) diff --git a/Sources/SyntaxKit/Execution/ProcessResult.swift b/Sources/SyntaxKit/Execution/ProcessResult.swift index e8a14c5..caab235 100644 --- a/Sources/SyntaxKit/Execution/ProcessResult.swift +++ b/Sources/SyntaxKit/Execution/ProcessResult.swift @@ -27,17 +27,21 @@ // OTHER DEALINGS IN THE SOFTWARE. // -package import Foundation +public import Foundation /// Raw outcome of rendering one input — what `processFile` returns to its /// caller. `exitCode == 0` indicates the spawned `swift` succeeded (or that /// the output cache hit, which is treated identically). -package struct ProcessResult: Sendable { - package let exitCode: Int32 - package let stdout: Data - package let stderr: String +public struct ProcessResult: Sendable { + /// Exit status of the spawned `swift`. `0` indicates success (or a cache hit). + public let exitCode: Int32 + /// Bytes the wrapped program emitted to standard output — the rendered source. + public let stdout: Data + /// Compiler diagnostics from the spawned `swift`. Empty when it was silent. + public let stderr: String - package init(exitCode: Int32, stdout: Data, stderr: String) { + /// Creates a result from a finished (or cache-served) render. + public init(exitCode: Int32, stdout: Data, stderr: String) { self.exitCode = exitCode self.stdout = stdout self.stderr = stderr diff --git a/Sources/SyntaxKit/Execution/RunError.swift b/Sources/SyntaxKit/Execution/RunError.swift index dcefd2f..1f96aa1 100644 --- a/Sources/SyntaxKit/Execution/RunError.swift +++ b/Sources/SyntaxKit/Execution/RunError.swift @@ -30,7 +30,7 @@ /// Typed error surfaced by `Runner`. It decouples the renderer from any /// particular caller: `Runner` reports *what* went wrong, and the caller /// (CLI, build plugin, in-process driver) decides how to present it. -package enum RunError: Error { +public enum RunError: Error { /// The input path was invalid — missing, or a directory given without an /// output directory. case invalidInput(String) diff --git a/Sources/SyntaxKit/Execution/RunInput.swift b/Sources/SyntaxKit/Execution/RunInput.swift index 173251d..ee1e72b 100644 --- a/Sources/SyntaxKit/Execution/RunInput.swift +++ b/Sources/SyntaxKit/Execution/RunInput.swift @@ -34,7 +34,7 @@ import Foundation /// `resolve(input:output:)`, which stats the path and enforces the per-mode /// output rules so the caller can `switch` on a settled value instead of /// juggling an `ObjCBool`. -package enum RunInput { +public enum RunInput { /// A single input file. `outputPath` is whatever the caller intends to /// do with the rendered bytes (write to a file, ignore, etc.); `Runner` /// itself does not act on it. @@ -45,7 +45,7 @@ package enum RunInput { /// Classifies `input` by stat: existing directory → `.directory`, /// existing file → `.singleFile`. Throws `RunError.invalidInput` if the path /// doesn't exist, or if a directory input wasn't given an explicit output. - package static func resolve(input: String, output: String?) throws(RunError) -> RunInput { + public static func resolve(input: String, output: String?) throws(RunError) -> RunInput { var isDirectory: ObjCBool = false guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { throw RunError.invalidInput("input does not exist: \(input)") diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index 6959827..e34428e 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -40,7 +40,7 @@ extension Runner { /// Throws `RunError.unexpected` only for bulk failures the SDK can't /// recover from (e.g. the input directory can't be enumerated). An empty /// input set is *not* an error — the result simply has no outcomes. - package func renderDirectory( + public func renderDirectory( inputDir: String, outputDir: String ) async throws(RunError) -> DirectoryRender { diff --git a/Sources/SyntaxKit/Execution/Runner+Session.swift b/Sources/SyntaxKit/Execution/Runner+Session.swift index 669524c..e7ddfdd 100644 --- a/Sources/SyntaxKit/Execution/Runner+Session.swift +++ b/Sources/SyntaxKit/Execution/Runner+Session.swift @@ -33,7 +33,7 @@ extension Runner { /// Why a render session couldn't be brought up. Decoupled from any caller: /// the initializer reports *what* failed; the caller (CLI, build plugin, /// in-process driver) decides how to present it and which exit code to use. - package enum SetupError: Error { + public enum SetupError: Error { /// The libSyntaxKit directory couldn't be resolved from the supplied /// candidates or the bundle-relative fallbacks. Carries the underlying /// `CLIError` describing the lookup. @@ -55,7 +55,7 @@ extension Runner { /// /// - Throws: `SetupError` for a lookup or toolchain failure, so the caller /// owns the presentation and exit mapping. - package init( + public init( libCandidates: [String?], swiftVersion: String?, enforceToolchainCheck: Bool, diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index c6fb5c5..cb710c1 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -46,7 +46,7 @@ import Foundation /// /// Constructed once per render session; `Sendable` so a single value can be /// shared across the concurrent per-input tasks in directory mode. -package struct Runner: Sendable { +public struct Runner: Sendable { /// Exit code returned when the spawned `swift` is killed by skit's timeout /// watchdog. Matches POSIX `timeout(1)`. private static let timeoutExitCode: Int32 = 124 @@ -67,7 +67,9 @@ package struct Runner: Sendable { /// by the caller (skit supplies a Subprocess-based implementation). private let run: @Sendable (SwiftInvocation) async throws -> SwiftRunOutcome - package init( + /// Creates a runner bound to a lib directory, an optional output cache, a + /// per-input timeout, and the backend closure that spawns `swift`. + public init( libPath: String, cache: OutputCache?, timeoutSeconds: Int, @@ -88,7 +90,7 @@ package struct Runner: Sendable { /// On a non-zero subprocess exit, throws `RunError.renderFailed(exitCode: /// stderr:)` carrying the toolchain's diagnostic. Any Foundation/Subprocess /// failure (file read, spawn) is wrapped in `RunError.unexpected`. - package func renderFile(input: String) async throws(RunError) -> SingleFileRender { + public func renderFile(input: String) async throws(RunError) -> SingleFileRender { // Render the input. `processFile` may hit the output cache and skip the // spawn entirely; either way the result has the same shape. let result: ProcessResult @@ -188,7 +190,7 @@ package struct Runner: Sendable { if timeoutSeconds <= 0 { outcome = try await operation() } else { - outcome = try await Task.timeout(.seconds(timeoutSeconds), operation: operation) ?? .timedOut + outcome = try await Task.timeout(seconds: timeoutSeconds, operation: operation) ?? .timedOut } // Normalize both outcomes into a single ProcessResult shape. diff --git a/Sources/SyntaxKit/Execution/SingleFileRender.swift b/Sources/SyntaxKit/Execution/SingleFileRender.swift index 6673934..e6df4fe 100644 --- a/Sources/SyntaxKit/Execution/SingleFileRender.swift +++ b/Sources/SyntaxKit/Execution/SingleFileRender.swift @@ -27,19 +27,21 @@ // OTHER DEALINGS IN THE SOFTWARE. // -package import Foundation +public import Foundation /// Result of `Runner.renderFile`. Both fields may be populated on success — /// the spawned `swift` can emit warnings to `stderr` alongside a valid /// `stdout`. The caller decides where the bytes go (file, stdout, in-memory). -package struct SingleFileRender: Sendable { +public struct SingleFileRender: Sendable { /// Rendered Swift source produced by the wrapped program. - package let stdout: Data + public let stdout: Data /// Compiler diagnostics from the spawned `swift`, with the wrapper path /// rewritten back to the original input. Empty when the toolchain was silent. - package let stderr: String + public let stderr: String - package init(stdout: Data, stderr: String) { + /// Creates a single-file render result from the rendered `stdout` bytes and + /// any `stderr` diagnostics. + public init(stdout: Data, stderr: String) { self.stdout = stdout self.stderr = stderr } diff --git a/Sources/SyntaxKit/Execution/SwiftBackend.swift b/Sources/SyntaxKit/Execution/SwiftBackend.swift index 9402fb2..0a0a917 100644 --- a/Sources/SyntaxKit/Execution/SwiftBackend.swift +++ b/Sources/SyntaxKit/Execution/SwiftBackend.swift @@ -34,7 +34,7 @@ /// SyntaxKit defines the contract; a Subprocess-based conformance lives in the /// `skit` CLI. `Sendable` so the spawn method can be captured in `Runner`'s /// `@Sendable` run closure. -package protocol SwiftBackend: Sendable { +public protocol SwiftBackend: Sendable { /// Verbatim `swift --version` output, or nil if the toolchain couldn't be /// queried. func captureSwiftVersion() async -> String? diff --git a/Sources/SyntaxKit/Execution/SwiftInvocation.swift b/Sources/SyntaxKit/Execution/SwiftInvocation.swift index 71cc05c..45562c6 100644 --- a/Sources/SyntaxKit/Execution/SwiftInvocation.swift +++ b/Sources/SyntaxKit/Execution/SwiftInvocation.swift @@ -31,7 +31,10 @@ /// directory (for `-I`/`-L`/`-rpath`) and the path to the wrapped source. /// `Runner` hands this to its `run` closure, which performs the actual spawn — /// the seam that keeps the engine free of any Subprocess dependency. -package struct SwiftInvocation: Sendable { - package let libPath: String - package let wrappedPath: String +public struct SwiftInvocation: Sendable { + /// Directory holding `libSyntaxKit` + swiftmodules, used for the spawned + /// `swift`'s `-I`/`-L`/`-rpath` flags. + public let libPath: String + /// Path to the wrapped DSL source the spawned `swift` should compile and run. + public let wrappedPath: String } diff --git a/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift b/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift index 068e4da..a7fb2c5 100644 --- a/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift +++ b/Sources/SyntaxKit/Execution/SwiftRunOutcome.swift @@ -30,7 +30,7 @@ /// Either the spawned `swift` ran to completion (success or failure) or /// the watchdog elapsed first. The completed payload is normalized to the /// shape callers want regardless of platform. -package enum SwiftRunOutcome: Sendable { +public enum SwiftRunOutcome: Sendable { case completed(ProcessResult) case timedOut } diff --git a/Sources/SyntaxKit/Execution/Task+Timeout.swift b/Sources/SyntaxKit/Execution/Task+Timeout.swift index beabdc0..91e90ac 100644 --- a/Sources/SyntaxKit/Execution/Task+Timeout.swift +++ b/Sources/SyntaxKit/Execution/Task+Timeout.swift @@ -28,11 +28,15 @@ // extension Task where Failure == any Error, Success: Sendable { - /// Runs `operation`, returning its value, or `nil` if `duration` elapses + /// Runs `operation`, returning its value, or `nil` if `seconds` elapses /// first. The operation and a sleep watchdog race in a throwing task group; /// whichever finishes first wins and the loser is cancelled. - package static func timeout( - _ duration: Duration, + /// + /// Takes seconds rather than a `Duration` so the engine stays buildable on + /// the package's minimum deployment targets — `Duration` requires iOS 16 / + /// tvOS 16 / watchOS 9, whereas `Task.sleep(nanoseconds:)` back-deploys. + public static func timeout( + seconds: Int, operation: @escaping @Sendable () async throws -> Success ) async throws -> Success? { try await withThrowingTaskGroup(of: Success?.self) { group in @@ -40,7 +44,7 @@ extension Task where Failure == any Error, Success: Sendable { group.addTask { // Bare `Task` here means `Task`, which has no // `sleep`; spell out the never-returning task to reach it. - try await Task.sleep(for: duration) + try await Task.sleep(nanoseconds: UInt64(seconds) * 1_000_000_000) return nil } let first = try await group.next()! diff --git a/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift b/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift index 8a235e6..4dc3ac2 100644 --- a/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift +++ b/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift @@ -29,7 +29,10 @@ import Foundation -package enum ToolchainCheckResult { +/// Outcome of comparing the bundle's recorded build toolchain against the +/// local `swift --version`. The swiftmodule format isn't reliably +/// forward-compatible across Swift releases, so a mismatch is worth surfacing. +public enum ToolchainCheckResult { /// Filename for the bundle's recorded build-toolchain version. private static let toolchainStampFilename = "swift-version.txt" @@ -38,6 +41,7 @@ package enum ToolchainCheckResult { /// `/swift-version.txt` is missing (older bundle that predates /// the stamp). skit prints a one-line note and proceeds. case stampMissing + /// The bundle stamp and the local `swift --version` differ. case mismatch(bundle: String, local: String) } @@ -47,7 +51,7 @@ extension ToolchainCheckResult { /// forward-compatible across even patch-level Swift releases (originating /// bug: 6.3.0 → 6.3.2 rejected the swiftmodule), so the comparison is /// exact-string after normalising trailing whitespace. - package init(libPath: String, swiftVersion: String?) { + public init(libPath: String, swiftVersion: String?) { let stampURL = URL(fileURLWithPath: libPath) .appendingPathComponent(Self.toolchainStampFilename) guard let stampData = try? Data(contentsOf: stampURL), diff --git a/Sources/SyntaxKit/Execution/WrappedSource.swift b/Sources/SyntaxKit/Execution/WrappedSource.swift index af48b2f..7efd0e3 100644 --- a/Sources/SyntaxKit/Execution/WrappedSource.swift +++ b/Sources/SyntaxKit/Execution/WrappedSource.swift @@ -37,7 +37,7 @@ import SwiftSyntax /// /// The body is fenced in `#sourceLocation` directives so compiler diagnostics in /// the body reference the original input file and line numbers. -package struct WrappedSource { +public struct WrappedSource { /// The original input path, used for the `#sourceLocation` directive. private let originalPath: String /// Top-level `import` declarations hoisted above the wrapper body, each already @@ -52,7 +52,7 @@ package struct WrappedSource { /// remaining body along with the line it begins on. Everything before the first /// non-import statement that *is* an import gets hoisted; anything before that /// which is *not* an import stays in the body (e.g. a leading `// comment`). - package init(source: String, originalPath: String) { + public init(source: String, originalPath: String) { self.originalPath = originalPath // Parse the input with SwiftSyntax. The location converter is needed to @@ -92,7 +92,7 @@ package struct WrappedSource { /// A complete Swift program that imports SyntaxKit, runs the body inside a /// `Group { … }` builder, and prints the generated code. - package var rendered: String { + public var rendered: String { // Render the hoisted-imports block. Trailing newline only if non-empty so // the wrapper doesn't grow an extra blank line in the common no-imports // case. From a5f20b05d95b5089fc2daf637cc5dd5c729c3792 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 10 Jun 2026 09:29:20 -0400 Subject: [PATCH 50/56] Fix skit render, address review findings, make hashing pluggable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit skit run was non-functional: the `swift` interpreter's JIT can't load the dynamic libSyntaxKit symbols. Switch the render backend to compile the wrapped DSL with `swiftc` to a temp executable and run it (Subprocess.Configuration+Swift). Also addresses code-review findings and follow-ups: - Guard --timeout against the UInt64 nanosecond overflow in validate(). - Make ToolchainCheckResult.init pure (no stderr). Record the outcome as a new Runner.ToolchainVerification, carry it on RunError.renderFailed, and have skit surface a one-line hint only on a render failure — keeping the SDK silent. - ContentHasher: fix 32-bit truncation in finalize() (%016x consumed a 32-bit int, halving the 64-bit digest) by formatting via String(_:radix:). - Replace the in-place Package.swift patch in build-skit-release.sh with a SYNTAXKIT_DYNAMIC_LIB env-var switch read by the manifest. - Introduce a pluggable ContentHashing protocol; OutputCache derives keys through an injectable hasher factory defaulting to ContentHasher (FNV-1a). Add unit tests for ContentHasher, WrappedSource, ToolchainCheckResult, OutputCache (incl. injected-hasher), and Runner render-failure toolchain plumbing. Co-Authored-By: Claude Opus 4.8 (1M context) --- Package.swift | 12 ++ Scripts/build-skit-release.sh | 30 +---- .../SyntaxKit/Execution/ContentHasher.swift | 28 ++-- .../SyntaxKit/Execution/ContentHashing.swift | 52 ++++++++ .../Execution/FileManager+Execution.swift | 9 +- Sources/SyntaxKit/Execution/OutputCache.swift | 14 +- Sources/SyntaxKit/Execution/RunError.swift | 7 +- .../Execution/Runner+Directory.swift | 7 +- .../SyntaxKit/Execution/Runner+Session.swift | 14 +- Sources/SyntaxKit/Execution/Runner.swift | 30 ++++- .../Execution/ToolchainCheckResult.swift | 12 +- Sources/skit/Skit+Run.swift | 12 ++ Sources/skit/Skit.Run+CommandError.swift | 35 ++++- Sources/skit/Skit.Run+Render.swift | 9 +- Sources/skit/Skit.swift | 4 + .../skit/Subprocess.Configuration+Swift.swift | 78 ++++++++--- .../Unit/Execution/ContentHasherTests.swift | 76 +++++++++++ .../Unit/Execution/OutputCacheTests.swift | 126 ++++++++++++++++++ .../Execution/RunnerRenderFailureTests.swift | 98 ++++++++++++++ .../Execution/ToolchainCheckResultTests.swift | 111 +++++++++++++++ .../Unit/Execution/WrappedSourceTests.swift | 105 +++++++++++++++ 21 files changed, 790 insertions(+), 79 deletions(-) create mode 100644 Sources/SyntaxKit/Execution/ContentHashing.swift create mode 100644 Tests/SyntaxKitTests/Unit/Execution/ContentHasherTests.swift create mode 100644 Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift create mode 100644 Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift create mode 100644 Tests/SyntaxKitTests/Unit/Execution/ToolchainCheckResultTests.swift create mode 100644 Tests/SyntaxKitTests/Unit/Execution/WrappedSourceTests.swift diff --git a/Package.swift b/Package.swift index 779ef62..5fcc9db 100644 --- a/Package.swift +++ b/Package.swift @@ -1,8 +1,19 @@ // swift-tools-version: 6.1 // The swift-tools-version declares the minimum version of Swift required to build this package. +import Foundation import PackageDescription +// MARK: - Library Linkage + +// The SyntaxKit library product is normally built with automatic (static) +// linkage. The self-contained skit release bundle (Scripts/build-skit-release.sh) +// needs a dynamic libSyntaxKit.dylib instead, so it sets SYNTAXKIT_DYNAMIC_LIB=1 +// rather than patching this manifest in place. +// swiftlint:disable:next explicit_top_level_acl explicit_acl +let syntaxKitLibraryType: Product.Library.LibraryType? = + ProcessInfo.processInfo.environment["SYNTAXKIT_DYNAMIC_LIB"] == "1" ? .dynamic : nil + // MARK: - Swift Settings Configuration // swiftlint:disable:next explicit_top_level_acl explicit_acl @@ -89,6 +100,7 @@ let package = Package( products: [ .library( name: "SyntaxKit", + type: syntaxKitLibraryType, targets: ["SyntaxKit"] ), .executable( diff --git a/Scripts/build-skit-release.sh b/Scripts/build-skit-release.sh index 78eefe7..66de1a5 100755 --- a/Scripts/build-skit-release.sh +++ b/Scripts/build-skit-release.sh @@ -25,33 +25,15 @@ fi SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" OUTPUT_DIR="$REPO_ROOT/.build/skit-release" -PACKAGE_FILE="$REPO_ROOT/Package.swift" -PACKAGE_BACKUP="$(mktemp)" - -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -old = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -new = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if old not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found") -p.write_text(src.replace(old, new, 1)) -PY cd "$REPO_ROOT" +# Build the SyntaxKit library product as a dynamic libSyntaxKit.dylib. Package.swift +# reads SYNTAXKIT_DYNAMIC_LIB and flips the library product to type: .dynamic, so +# we never mutate the canonical manifest. +echo "==> Building with SYNTAXKIT_DYNAMIC_LIB=1 (dynamic libSyntaxKit)" +export SYNTAXKIT_DYNAMIC_LIB=1 + echo "==> swift build -c release --product skit" swift build -c release --product skit diff --git a/Sources/SyntaxKit/Execution/ContentHasher.swift b/Sources/SyntaxKit/Execution/ContentHasher.swift index 47ea36e..4bef0bb 100644 --- a/Sources/SyntaxKit/Execution/ContentHasher.swift +++ b/Sources/SyntaxKit/Execution/ContentHasher.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +public import Foundation /// Non-cryptographic 64-bit FNV-1a hasher used to derive content-addressed /// cache keys. The cache keys aren't security-critical — there's no @@ -37,26 +37,34 @@ import Foundation /// /// FNV-1a is deterministic across processes and platforms (unlike the Swift /// stdlib `Hasher`, whose seed is randomized per-process) — that -/// determinism is what makes it usable as an on-disk cache key. -internal struct ContentHasher { +/// determinism is what makes it usable as an on-disk cache key. It is the +/// default `ContentHashing` conformer used by `OutputCache`. +public struct ContentHasher: ContentHashing { private static let offsetBasis: UInt64 = 0xcbf2_9ce4_8422_2325 private static let prime: UInt64 = 0x0000_0100_0000_01b3 - /// `String(format:)` specifier for the 16-char lowercase-hex digest. - private static let hexFormat = "%016x" + /// Width of the zero-padded lowercase-hex digest (64 bits → 16 hex chars). + private static let hexWidth = 16 private var state: UInt64 = ContentHasher.offsetBasis - internal mutating func update(data: Data) { + /// Creates a hasher seeded with the FNV-1a offset basis. + public init() {} + + public mutating func update(data: Data) { for byte in data { state ^= UInt64(byte) state &*= ContentHasher.prime } } - /// Returns the hash as a 16-char lowercase-hex string suitable for use as - /// a directory name. - internal func finalize() -> String { - String(format: Self.hexFormat, state) + /// Returns the full 64-bit hash as a 16-char lowercase-hex string suitable + /// for use as a directory name. Built with `String(_:radix:)` rather than + /// `String(format: "%016x", …)` because the `%x` specifier consumes a 32-bit + /// `unsigned int`, which would silently truncate the digest to its low 32 + /// bits and halve the entropy described above. + public func finalize() -> String { + let hex = String(state, radix: 16) + return String(repeating: "0", count: max(0, Self.hexWidth - hex.count)) + hex } } diff --git a/Sources/SyntaxKit/Execution/ContentHashing.swift b/Sources/SyntaxKit/Execution/ContentHashing.swift new file mode 100644 index 0000000..d451a6e --- /dev/null +++ b/Sources/SyntaxKit/Execution/ContentHashing.swift @@ -0,0 +1,52 @@ +// +// ContentHashing.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +public import Foundation + +/// An incremental hasher used to derive `OutputCache` keys. Abstracted so the +/// cache's hashing is pluggable: inject a different conformer via +/// `OutputCache.init(…, makeHasher:)` to swap the algorithm (e.g. a +/// cryptographic digest) without touching the cache. +/// +/// Two contractual requirements callers depend on, which the default +/// `ContentHasher` (FNV-1a) satisfies and a replacement must too: +/// - **Determinism across processes and platforms.** Keys are persisted to +/// disk and compared on later runs, so the same byte stream must always +/// produce the same digest. (The stdlib `Hasher` is unsuitable — it is +/// per-process seeded.) +/// - **A digest usable as a directory name.** `finalize()` returns a string +/// that is safe to use as a path component. +public protocol ContentHashing { + /// Creates an empty hasher, ready to accept `update(data:)` calls. + init() + /// Mixes `data`'s bytes into the running digest. Order-significant. + mutating func update(data: Data) + /// Returns the final digest as a filesystem-safe string. + func finalize() -> String +} diff --git a/Sources/SyntaxKit/Execution/FileManager+Execution.swift b/Sources/SyntaxKit/Execution/FileManager+Execution.swift index 0273f8b..1614431 100644 --- a/Sources/SyntaxKit/Execution/FileManager+Execution.swift +++ b/Sources/SyntaxKit/Execution/FileManager+Execution.swift @@ -105,7 +105,8 @@ extension FileManager { internal func writeOutput( for result: RenderTaskResult, inputBase: URL, - outputBase: URL + outputBase: URL, + toolchain: Runner.ToolchainVerification ) -> DirectoryRender.FileOutcome { let relative = result.input.path.dropFirst(inputBase.path.count + 1) let destination = outputBase.appendingPathComponent(String(relative)) @@ -120,7 +121,11 @@ extension FileManager { let outcome = result.result.flatMap { processResult -> Result in guard processResult.exitCode == 0 else { return .failure( - .renderFailed(exitCode: processResult.exitCode, stderr: processResult.stderr) + .renderFailed( + exitCode: processResult.exitCode, + stderr: processResult.stderr, + toolchain: toolchain + ) ) } return Result { diff --git a/Sources/SyntaxKit/Execution/OutputCache.swift b/Sources/SyntaxKit/Execution/OutputCache.swift index 6200fcf..6e46ff3 100644 --- a/Sources/SyntaxKit/Execution/OutputCache.swift +++ b/Sources/SyntaxKit/Execution/OutputCache.swift @@ -76,6 +76,11 @@ public struct OutputCache: Sendable { private let root: URL private let fileManager: @Sendable () -> FileManager private let processInfo: ProcessInfo + /// Factory for the per-key hasher. Pluggable so the cache's hashing algorithm + /// can be swapped; defaults to `ContentHasher` (FNV-1a). A factory rather than + /// a stored instance because `key(forInput:libPath:)` needs a fresh, empty + /// hasher per call. + private let makeHasher: @Sendable () -> any ContentHashing /// Verbatim `swift --version` output captured once for the lifetime of /// this cache, so per-input key derivation doesn't re-spawn `swift`. @@ -84,17 +89,20 @@ public struct OutputCache: Sendable { /// Creates a cache rooted under the SyntaxKit cache directory, keyed in part /// by the captured `swiftVersion`. `fileManager`/`processInfo` are injectable - /// for testing. + /// for testing; `makeHasher` is injectable to plug in a different + /// `ContentHashing` algorithm (defaults to `ContentHasher`). public init( swiftVersion: String?, fileManager: @autoclosure @escaping @Sendable () -> FileManager = .default, - processInfo: ProcessInfo = .processInfo + processInfo: ProcessInfo = .processInfo, + makeHasher: @escaping @Sendable () -> any ContentHashing = { ContentHasher() } ) { self.root = processInfo.syntaxKitCacheRoot(default: Self.defaultCacheRoot) .appendingPathComponent(Self.outputsDirectoryName) self.swiftVersion = swiftVersion self.fileManager = fileManager self.processInfo = processInfo + self.makeHasher = makeHasher } /// 64-bit content hash over (schema version, input source bytes, swift @@ -102,7 +110,7 @@ public struct OutputCache: Sendable { /// change in these inputs produces a fresh key and forces a recompile. /// See `ContentHasher` for the choice of FNV-1a over a cryptographic hash. public func key(forInput source: String, libPath: String) -> String { - var hasher = ContentHasher() + var hasher = makeHasher() // Schema version: bump to invalidate every existing cache entry at once. hasher.update(data: Data(Self.schemaVersion.utf8)) // Input source bytes: the primary driver of the key. diff --git a/Sources/SyntaxKit/Execution/RunError.swift b/Sources/SyntaxKit/Execution/RunError.swift index 1f96aa1..5c1cc57 100644 --- a/Sources/SyntaxKit/Execution/RunError.swift +++ b/Sources/SyntaxKit/Execution/RunError.swift @@ -37,8 +37,11 @@ public enum RunError: Error { /// Single-file render: the spawned `swift` exited non-zero. Carries that /// code (e.g. a compile failure, `124` on timeout, `128 + signal`) and /// the (path-rewritten) stderr the toolchain emitted, so the caller can - /// surface diagnostics without having to fish them out elsewhere. - case renderFailed(exitCode: Int32, stderr: String) + /// surface diagnostics without having to fish them out elsewhere. Also + /// carries the session's `toolchainVerification`: when it isn't `.verified`, + /// the failure may stem from a Swift-toolchain mismatch the check couldn't + /// rule out, which the caller can hint at. + case renderFailed(exitCode: Int32, stderr: String, toolchain: Runner.ToolchainVerification) /// A wrapped Foundation/Subprocess failure (file read/write, spawn error) /// that has no dedicated mapping. case unexpected(any Error) diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index e34428e..af9fd3c 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -66,7 +66,12 @@ extension Runner { // Phase 3: write successes and capture a per-input outcome for each. let outcomes = renderResults.map { - FileManager.default.writeOutput(for: $0, inputBase: inputURL, outputBase: outputURL) + FileManager.default.writeOutput( + for: $0, + inputBase: inputURL, + outputBase: outputURL, + toolchain: toolchainVerification + ) } return DirectoryRender(outcomes: outcomes) diff --git a/Sources/SyntaxKit/Execution/Runner+Session.swift b/Sources/SyntaxKit/Execution/Runner+Session.swift index e7ddfdd..91abece 100644 --- a/Sources/SyntaxKit/Execution/Runner+Session.swift +++ b/Sources/SyntaxKit/Execution/Runner+Session.swift @@ -75,14 +75,21 @@ extension Runner { // 2. Compare the bundle's recorded `swift --version` against the local one. // swiftmodules aren't reliably forward-compatible across compiler versions, // so a mismatch is surfaced rather than letting the spawned `swift` emit a - // cryptic module-version diagnostic. + // cryptic module-version diagnostic. A *mismatch* fails setup; anything else + // is remembered (no IO here — the SDK stays silent) so a later render + // failure can hint that an unverified toolchain may be the cause. + let verification: ToolchainVerification if enforceToolchainCheck { switch ToolchainCheckResult(libPath: libPath, swiftVersion: swiftVersion) { - case .match, .stampMissing: - break + case .match: + verification = .verified + case .stampMissing: + verification = .unverified case .mismatch(let bundle, let local): throw SetupError.toolchainMismatch(bundle: bundle, local: local) } + } else { + verification = .notChecked } // 3. Build the output cache (nil when disabled). The captured `swiftVersion` @@ -94,6 +101,7 @@ extension Runner { libPath: libPath, cache: cache, timeoutSeconds: timeoutSeconds, + toolchainVerification: verification, run: run ) } diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index cb710c1..57ae60f 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -47,6 +47,23 @@ import Foundation /// Constructed once per render session; `Sendable` so a single value can be /// shared across the concurrent per-input tasks in directory mode. public struct Runner: Sendable { + /// Outcome of the bundle/local Swift-toolchain compatibility check performed + /// when the render session was created. Stored on the `Runner` and carried on + /// `RunError.renderFailed` so a caller can hint that an *unverified* toolchain + /// may be the real cause of a build error — swiftmodules aren't reliably + /// compatible across compiler versions. A confirmed *mismatch* never reaches + /// here; it fails session setup via `SetupError.toolchainMismatch`. + public enum ToolchainVerification: Sendable, Equatable { + /// The bundle's recorded `swift --version` matched the local one. + case verified + /// The caller opted out of the check (`enforceToolchainCheck == false`). + case notChecked + /// The check ran but couldn't compare versions — the bundle had no + /// toolchain stamp, or the local `swift --version` couldn't be captured — + /// so compatibility is unknown. + case unverified + } + /// Exit code returned when the spawned `swift` is killed by skit's timeout /// watchdog. Matches POSIX `timeout(1)`. private static let timeoutExitCode: Int32 = 124 @@ -66,6 +83,11 @@ public struct Runner: Sendable { /// Backend that actually spawns `swift` for one `SwiftInvocation`. Injected /// by the caller (skit supplies a Subprocess-based implementation). private let run: @Sendable (SwiftInvocation) async throws -> SwiftRunOutcome + /// Whether this session's bundle/local Swift toolchain was confirmed + /// compatible. Defaults to `.notChecked` for callers that construct a + /// `Runner` directly; the session initializer (`Runner+Session.swift`) sets + /// the real result. Carried onto `RunError.renderFailed`. + public let toolchainVerification: ToolchainVerification /// Creates a runner bound to a lib directory, an optional output cache, a /// per-input timeout, and the backend closure that spawns `swift`. @@ -73,11 +95,13 @@ public struct Runner: Sendable { libPath: String, cache: OutputCache?, timeoutSeconds: Int, + toolchainVerification: ToolchainVerification = .notChecked, run: @Sendable @escaping (SwiftInvocation) async throws -> SwiftRunOutcome ) { self.libPath = libPath self.cache = cache self.timeoutSeconds = timeoutSeconds + self.toolchainVerification = toolchainVerification self.run = run } @@ -104,7 +128,11 @@ public struct Runner: Sendable { // Non-zero subprocess exit is reported as a typed failure carrying both // the code and the (path-rewritten) toolchain diagnostic. guard result.exitCode == 0 else { - throw RunError.renderFailed(exitCode: result.exitCode, stderr: result.stderr) + throw RunError.renderFailed( + exitCode: result.exitCode, + stderr: result.stderr, + toolchain: toolchainVerification + ) } return SingleFileRender(stdout: result.stdout, stderr: result.stderr) } diff --git a/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift b/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift index 4dc3ac2..2842fbf 100644 --- a/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift +++ b/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift @@ -38,8 +38,10 @@ public enum ToolchainCheckResult { /// Bundle stamp matches the local `swift --version` exactly. case match - /// `/swift-version.txt` is missing (older bundle that predates - /// the stamp). skit prints a one-line note and proceeds. + /// The toolchain check couldn't be performed: either `/swift-version.txt` + /// is missing (older bundle that predates the stamp) or the local + /// `swift --version` couldn't be captured. The caller proceeds; presentation + /// of the skipped-check note belongs to the call site, not this value type. case stampMissing /// The bundle stamp and the local `swift --version` differ. case mismatch(bundle: String, local: String) @@ -57,16 +59,10 @@ extension ToolchainCheckResult { guard let stampData = try? Data(contentsOf: stampURL), let stampRaw = String(data: stampData, encoding: .utf8) else { - FileHandle.standardError.write( - Data("skit: bundle has no toolchain stamp; skipping check\n".utf8) - ) self = .stampMissing return } guard let localRaw = swiftVersion else { - FileHandle.standardError.write( - Data("skit: could not capture local `swift --version`; skipping toolchain check\n".utf8) - ) self = .stampMissing return } diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index a90b2a3..67b91a3 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -61,6 +61,10 @@ extension Skit { /// Path to the script that rebuilds the self-contained release bundle. internal static let buildReleaseScriptPath = "Scripts/build-skit-release.sh" + /// Largest timeout we can safely convert to nanoseconds without overflowing + /// the `UInt64` multiplication in `Task.timeout` (`UInt64(seconds) * 1e9`). + internal static let maxTimeoutSeconds = Int(UInt64.max / 1_000_000_000) + internal static let configuration = CommandConfiguration( commandName: commandName, abstract: "Render SyntaxKit DSL input(s) into Swift source." @@ -106,6 +110,14 @@ extension Skit { + "got: \(timeoutSeconds)" ) } + // Upper bound: the value is later multiplied by 1_000_000_000 into a + // `UInt64` of nanoseconds, which overflows silently past this point. + guard timeoutSeconds <= Self.maxTimeoutSeconds else { + throw ValidationError( + "--\(Self.timeoutOptionName) is too large (max \(Self.maxTimeoutSeconds) seconds), " + + "got: \(timeoutSeconds)" + ) + } } internal func execute() async throws(CommandError) { diff --git a/Sources/skit/Skit.Run+CommandError.swift b/Sources/skit/Skit.Run+CommandError.swift index 0d0feb7..f0277b6 100644 --- a/Sources/skit/Skit.Run+CommandError.swift +++ b/Sources/skit/Skit.Run+CommandError.swift @@ -48,8 +48,10 @@ extension Skit.Run { case toolchainMismatch(bundle: String, local: String) /// The spawned `swift` exited non-zero (compile failure, `124` on timeout, /// `128 + signal`). Carries that code + the toolchain stderr; the code is - /// passed through as the process exit. - case renderFailed(exitCode: Int32, stderr: String) + /// passed through as the process exit. Also carries the session's + /// `toolchainVerification` so an unverified toolchain can be hinted as a + /// possible cause alongside the diagnostics. + case renderFailed(exitCode: Int32, stderr: String, toolchain: Runner.ToolchainVerification) /// A directory batch couldn't be walked. Exit 1. case directoryWalkFailed(input: String, underlying: any Error) /// A render/batch failure whose diagnostics were already surfaced (per-input @@ -82,8 +84,10 @@ extension Skit.Run { return "\(error)\n" case .toolchainMismatch(let bundle, let local): return Self.toolchainMismatchMessage(bundle: bundle, local: local) - case .renderFailed(_, let stderr): - return stderr.isEmpty ? nil : stderr + case .renderFailed(_, let stderr, let toolchain): + let parts = [stderr.isEmpty ? nil : stderr, Self.toolchainHint(toolchain)] + .compactMap { $0 } + return parts.isEmpty ? nil : parts.joined() case .directoryWalkFailed(let input, let underlying): return "\(Skit.Run.messagePrefix)failed to walk \(input): \(underlying)\n" case .unsupportedPlatform: @@ -100,7 +104,7 @@ extension Skit.Run { return ValidationError(message) case .libResolutionFailed, .toolchainMismatch: return ExitCode(2) - case .renderFailed(let exitCode, _): + case .renderFailed(let exitCode, _, _): return ExitCode(exitCode) case .directoryWalkFailed, .failed, .unsupportedPlatform: return ExitCode(1) @@ -109,6 +113,27 @@ extension Skit.Run { } } + /// A one-line note appended to a render failure when the bundle/local Swift + /// toolchain wasn't confirmed compatible — so a module-version error reads + /// as a possible toolchain mismatch rather than a mysterious build failure. + /// `nil` when the toolchain was verified (nothing to add). `internal` so the + /// directory-batch path can surface the same note once. + internal static func toolchainHint(_ verification: Runner.ToolchainVerification) -> String? { + switch verification { + case .verified: + return nil + case .notChecked: + return "\(Skit.Run.messagePrefix)note: the bundle/local Swift-toolchain check was skipped " + + "(--\(Skit.Run.noToolchainCheckFlagName)); if this is a module-version error, a " + + "toolchain mismatch may be the cause.\n" + case .unverified: + return + "\(Skit.Run.messagePrefix)note: couldn't verify the bundle's Swift toolchain against " + + "your local `swift` (no toolchain stamp, or version capture failed); if this is a " + + "module-version error, a toolchain mismatch may be the cause.\n" + } + } + /// Human-readable error explaining why the bundle's recorded /// `swift --version` differs from the local one, and how to recover. private static func toolchainMismatchMessage(bundle: String, local: String) -> String { diff --git a/Sources/skit/Skit.Run+Render.swift b/Sources/skit/Skit.Run+Render.swift index 3e8a6e5..e26738d 100644 --- a/Sources/skit/Skit.Run+Render.swift +++ b/Sources/skit/Skit.Run+Render.swift @@ -75,8 +75,8 @@ extension Skit.Run { switch error { case .invalidInput(let message): throw CommandError.usage(message) - case .renderFailed(let exitCode, let stderr): - throw CommandError.renderFailed(exitCode: exitCode, stderr: stderr) + case .renderFailed(let exitCode, let stderr, let toolchain): + throw CommandError.renderFailed(exitCode: exitCode, stderr: stderr, toolchain: toolchain) case .unexpected(let underlying): throw CommandError.unexpected(underlying) } @@ -152,6 +152,11 @@ extension Skit.Run { ) if result.failureCount > 0 { + // Some inputs failed; if the toolchain couldn't be verified, hint once + // that a Swift-version mismatch may be behind the build errors above. + if let hint = CommandError.toolchainHint(runner.toolchainVerification) { + FileHandle.standardError.write(Data(hint.utf8)) + } throw CommandError.failed } } diff --git a/Sources/skit/Skit.swift b/Sources/skit/Skit.swift index 73c4ee2..d500b9d 100644 --- a/Sources/skit/Skit.swift +++ b/Sources/skit/Skit.swift @@ -44,6 +44,10 @@ internal struct Skit: AsyncParsableCommand { /// toolchain-version capture and the Subprocess `swift` configuration. internal static let swiftExecutableName = "swift" + /// Name of the `swiftc` compiler resolved on `PATH`. Used to compile a + /// wrapped DSL program into a temporary executable before running it. + internal static let swiftcExecutableName = "swiftc" + internal static let configuration = CommandConfiguration( commandName: commandName, abstract: "Render SyntaxKit DSL into Swift source, or parse Swift into JSON.", diff --git a/Sources/skit/Subprocess.Configuration+Swift.swift b/Sources/skit/Subprocess.Configuration+Swift.swift index 526cf4d..b0c1a91 100644 --- a/Sources/skit/Subprocess.Configuration+Swift.swift +++ b/Sources/skit/Subprocess.Configuration+Swift.swift @@ -32,9 +32,10 @@ import Foundation import Subprocess import SyntaxKit + import System extension Subprocess.Configuration { - /// Bounded output capacity for the spawned `swift` (16 MiB stdout / 1 MiB + /// Bounded output capacity for the spawned process (16 MiB stdout / 1 MiB /// stderr). Output above this is exotic; we'd surface a clear SubprocessError /// rather than silently truncate. private static let stdoutLimitBytes = 16 * 1_024 * 1_024 @@ -43,7 +44,10 @@ /// Subdirectory of the lib dir holding the SwiftSyntax CShims headers. private static let cShimsIncludeSuffix = "_SwiftSyntaxCShims-include" - /// swiftc flags used to build the `swift` invocation. + /// Filename prefix for the per-render temporary executable. + private static let renderExecutablePrefix = "skit-render-" + + /// swiftc flags used to build the compile invocation. private static let flagSuppressWarnings = "-suppress-warnings" private static let flagInclude = "-I" private static let flagLibrarySearchPath = "-L" @@ -51,14 +55,21 @@ private static let flagPassToClang = "-Xcc" private static let flagPassToLinker = "-Xlinker" private static let flagRPath = "-rpath" + private static let flagOutput = "-o" - /// A configuration that runs the `swift` interpreter on `wrappedPath`, - /// linked against `libSyntaxKit` in `libPath`. + /// A configuration that compiles `wrappedPath` into an executable at + /// `outputPath`, linked against `libSyntaxKit` in `libPath`. /// /// Builds the full flag set: link against libSyntaxKit, include the CShims - /// headers, and set rpath so the dylib loads at runtime. The executable is - /// resolved by name on `PATH`. - internal static func swift(libPath: String, wrappedPath: String) -> Self { + /// headers, and bake an rpath so the produced binary loads the dylib at + /// runtime. `swiftc` is used rather than the `swift` interpreter because the + /// interpreter's JIT does not load the dynamic libSyntaxKit's symbols. The + /// executable is resolved by name on `PATH`. + internal static func compileSwift( + libPath: String, + wrappedPath: String, + outputPath: String + ) -> Self { let cShimsInclude = "\(libPath)/\(cShimsIncludeSuffix)" let arguments: [String] = [ flagSuppressWarnings, @@ -67,30 +78,61 @@ flagLinkSyntaxKit, flagPassToClang, flagInclude, flagPassToClang, cShimsInclude, flagPassToLinker, flagRPath, flagPassToLinker, libPath, + flagOutput, outputPath, wrappedPath, ] - return Self(executable: .name(Skit.swiftExecutableName), arguments: Arguments(arguments)) + return Self(executable: .name(Skit.swiftcExecutableName), arguments: Arguments(arguments)) } - /// Spawns `swift` for the render `invocation` and normalizes the result - /// into a `SwiftRunOutcome`. This is the Subprocess backend skit hands to + /// Renders the `invocation` by compiling the wrapped DSL program with + /// `swiftc` and then running the produced executable, whose stdout is the + /// generated Swift source. This is the Subprocess backend skit hands to /// `Runner` as its `run` closure — the one seam between the (platform- - /// agnostic) engine in SyntaxKit and the Subprocess implementation. A - /// completed spawn always reports `.completed`; the timeout race that can - /// produce `.timedOut` lives in `Runner`. + /// agnostic) engine in SyntaxKit and the Subprocess implementation. + /// + /// A compile failure short-circuits and is reported with the compiler's + /// exit code + stderr (the render layer maps any non-zero exit to a render + /// failure). A successful compile is followed by running the binary and + /// reporting its result. A completed spawn always reports `.completed`; the + /// timeout race that can produce `.timedOut` lives in `Runner`. internal static func runSwift( for invocation: SyntaxKit.SwiftInvocation ) async throws -> SwiftRunOutcome { - let record = try await Subprocess.run( - .swift(libPath: invocation.libPath, wrappedPath: invocation.wrappedPath), + let outputURL = FileManager.default.temporaryDirectory + .appendingPathComponent("\(renderExecutablePrefix)\(UUID().uuidString)") + defer { try? FileManager.default.removeItem(at: outputURL) } + + // 1. Compile the wrapped program to a temporary executable. + let compile = try await Subprocess.run( + .compileSwift( + libPath: invocation.libPath, + wrappedPath: invocation.wrappedPath, + outputPath: outputURL.path + ), + output: .discarded, + error: .string(limit: stderrLimitBytes) + ) + guard compile.terminationStatus.exitCode == 0 else { + return .completed( + ProcessResult( + exitCode: compile.terminationStatus.exitCode, + stdout: Data(), + stderr: compile.standardError ?? "" + ) + ) + } + + // 2. Run the compiled program; its stdout is the rendered Swift source. + let render = try await Subprocess.run( + .path(FilePath(outputURL.path)), output: .string(limit: stdoutLimitBytes), error: .string(limit: stderrLimitBytes) ) return .completed( ProcessResult( - exitCode: record.terminationStatus.exitCode, - stdout: Data((record.standardOutput ?? "").utf8), - stderr: record.standardError ?? "" + exitCode: render.terminationStatus.exitCode, + stdout: Data((render.standardOutput ?? "").utf8), + stderr: render.standardError ?? "" ) ) } diff --git a/Tests/SyntaxKitTests/Unit/Execution/ContentHasherTests.swift b/Tests/SyntaxKitTests/Unit/Execution/ContentHasherTests.swift new file mode 100644 index 0000000..a80717e --- /dev/null +++ b/Tests/SyntaxKitTests/Unit/Execution/ContentHasherTests.swift @@ -0,0 +1,76 @@ +// +// ContentHasherTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import Testing + +@testable import SyntaxKit + +@Suite internal struct ContentHasherTests { + /// The FNV-1a offset basis, as the 16-char hex an empty hasher must finalize to. + private static let emptyDigest = "cbf29ce484222325" + + private func digest(of data: Data) -> String { + var hasher = ContentHasher() + hasher.update(data: data) + return hasher.finalize() + } + + @Test("Empty input finalizes to the FNV-1a offset basis") + internal func emptyInputDigest() { + let hasher = ContentHasher() + #expect(hasher.finalize() == Self.emptyDigest) + #expect(digest(of: Data()) == Self.emptyDigest) + } + + @Test("Same bytes produce the same digest across fresh hashers") + internal func determinism() { + let data = Data("the quick brown fox".utf8) + #expect(digest(of: data) == digest(of: data)) + } + + @Test("Different inputs produce different digests") + internal func distinctInputs() { + #expect(digest(of: Data("alpha".utf8)) != digest(of: Data("beta".utf8))) + } + + @Test("Byte order is significant") + internal func orderSensitivity() { + #expect(digest(of: Data([0x01, 0x02])) != digest(of: Data([0x02, 0x01]))) + } + + @Test("Chunked updates accumulate identically to a single update") + internal func chunkedAccumulation() { + var chunked = ContentHasher() + chunked.update(data: Data("AB".utf8)) + chunked.update(data: Data("C".utf8)) + + #expect(chunked.finalize() == digest(of: Data("ABC".utf8))) + } +} diff --git a/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift b/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift new file mode 100644 index 0000000..343eb9e --- /dev/null +++ b/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift @@ -0,0 +1,126 @@ +// +// OutputCacheTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import Testing + +@testable import SyntaxKit + +@Suite internal struct OutputCacheTests { + /// A `ProcessInfo` whose environment is fixed, so cache-key derivation can be + /// tested without depending on the real process environment. + private final class FakeProcessInfo: ProcessInfo, Sendable { + private let fixedEnvironment: [String: String] + + init(environment: [String: String]) { + self.fixedEnvironment = environment + super.init() + } + + override var environment: [String: String] { fixedEnvironment } + } + + /// A stub `ContentHashing` that ignores its input and returns a fixed digest, + /// used to prove `OutputCache` derives keys through the injected factory. + private struct StubHasher: ContentHashing { + private static let fixedDigest = "stub-digest" + mutating func update(data: Data) {} + func finalize() -> String { Self.fixedDigest } + } + + private static let libPath = "/nonexistent/lib" + + private func cache( + swiftVersion: String? = "swift 6.1", + environment: [String: String] = [:] + ) -> OutputCache { + OutputCache(swiftVersion: swiftVersion, processInfo: FakeProcessInfo(environment: environment)) + } + + private func key(_ cache: OutputCache, source: String = "Struct(\"Foo\") {}") -> String { + cache.key(forInput: source, libPath: Self.libPath) + } + + @Test("key() derives the digest through the injected hasher factory") + internal func usesInjectedHasher() { + let cache = OutputCache( + swiftVersion: "swift 6.1", + processInfo: FakeProcessInfo(environment: [:]), + makeHasher: { StubHasher() } + ) + #expect(cache.key(forInput: "anything", libPath: Self.libPath) == "stub-digest") + } + + @Test("The same inputs always derive the same key") + internal func stableKey() { + let cache = cache() + #expect(key(cache) == key(cache)) + } + + @Test("Different source bytes derive different keys") + internal func sourceSensitivity() { + let cache = cache() + #expect(key(cache, source: "Struct(\"A\") {}") != key(cache, source: "Struct(\"B\") {}")) + } + + @Test("A different swift version derives a different key") + internal func swiftVersionSensitivity() { + #expect(key(cache(swiftVersion: "swift 6.1")) != key(cache(swiftVersion: "swift 6.2"))) + } + + @Test("SKIT_/SYNTAXKIT_ env vars are mixed into the key") + internal func relevantEnvVarsChangeKey() { + let base = key(cache(environment: [:])) + #expect(key(cache(environment: ["SKIT_FOO": "1"])) != base) + #expect(key(cache(environment: ["SYNTAXKIT_BAR": "1"])) != base) + } + + @Test("Unrelated env vars are ignored by the key") + internal func irrelevantEnvVarsIgnored() { + let base = key(cache(environment: [:])) + #expect(key(cache(environment: ["PATH": "/usr/bin", "HOME": "/root"])) == base) + } + + @Test("store then lookup round-trips the rendered payload") + internal func storeLookupRoundTrip() throws { + let cacheRoot = FileManager.default.temporaryDirectory + .appendingPathComponent("outputcache-\(UUID().uuidString)") + defer { try? FileManager.default.removeItem(at: cacheRoot) } + + // XDG_CACHE_HOME redirects the cache root; it isn't a SKIT_/SYNTAXKIT_ var, + // so it doesn't affect the key itself. + let cache = cache(environment: ["XDG_CACHE_HOME": cacheRoot.path]) + let key = key(cache) + let payload = Data("rendered output".utf8) + + #expect(cache.lookup(key: key) == nil) + try cache.store(key: key, data: payload) + #expect(cache.lookup(key: key) == payload) + } +} diff --git a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift new file mode 100644 index 0000000..cf083a0 --- /dev/null +++ b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift @@ -0,0 +1,98 @@ +// +// RunnerRenderFailureTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import Testing + +@testable import SyntaxKit + +@Suite internal struct RunnerRenderFailureTests { + /// Runs `renderFile` against a throwaway input with a stub backend that + /// always reports a non-zero exit, so the render fails deterministically + /// without spawning `swift`. Returns the thrown `RunError`, or nil if the + /// call unexpectedly succeeded. + private func renderFailure( + toolchain: Runner.ToolchainVerification + ) async -> RunError? { + let input = FileManager.default.temporaryDirectory + .appendingPathComponent("skit-input-\(UUID().uuidString).swift") + try? Data("let x = 1\n".utf8).write(to: input) + defer { try? FileManager.default.removeItem(at: input) } + + let runner = Runner( + libPath: "/does/not/matter", + cache: nil, + timeoutSeconds: 0, + toolchainVerification: toolchain + ) { _ in + .completed(ProcessResult(exitCode: 1, stdout: Data(), stderr: "boom\n")) + } + + do { + _ = try await runner.renderFile(input: input.path) + return nil + } catch { + return error + } + } + + @Test("renderFailed carries the session's toolchain verification and diagnostics") + internal func carriesUnverifiedToolchain() async { + guard + case .renderFailed(let exitCode, let stderr, let toolchain)? = + await renderFailure(toolchain: .unverified) + else { + Issue.record("expected .renderFailed") + return + } + #expect(exitCode == 1) + #expect(stderr == "boom\n") + #expect(toolchain == .unverified) + } + + @Test("renderFailed reflects a verified toolchain unchanged") + internal func reflectsVerifiedToolchain() async { + guard + case .renderFailed(_, _, let toolchain)? = + await renderFailure(toolchain: .verified) + else { + Issue.record("expected .renderFailed") + return + } + #expect(toolchain == .verified) + } + + @Test("A directly-constructed Runner defaults to notChecked") + internal func defaultsToNotChecked() { + let runner = Runner(libPath: "/x", cache: nil, timeoutSeconds: 0) { _ in + .completed(ProcessResult(exitCode: 0, stdout: Data(), stderr: "")) + } + #expect(runner.toolchainVerification == .notChecked) + } +} diff --git a/Tests/SyntaxKitTests/Unit/Execution/ToolchainCheckResultTests.swift b/Tests/SyntaxKitTests/Unit/Execution/ToolchainCheckResultTests.swift new file mode 100644 index 0000000..d3adc96 --- /dev/null +++ b/Tests/SyntaxKitTests/Unit/Execution/ToolchainCheckResultTests.swift @@ -0,0 +1,111 @@ +// +// ToolchainCheckResultTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import Testing + +@testable import SyntaxKit + +@Suite internal struct ToolchainCheckResultTests { + private static let stampFilename = "swift-version.txt" + + /// Creates a fresh temp lib dir, optionally seeded with a `swift-version.txt` + /// stamp, runs `body` against its path, and removes the dir afterward. + private func withLibDir( + stamp: String?, + _ body: (String) throws -> Void + ) throws { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("toolchain-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: dir) } + + if let stamp { + try Data(stamp.utf8).write(to: dir.appendingPathComponent(Self.stampFilename)) + } + try body(dir.path) + } + + @Test("Identical bundle/local versions match") + internal func match() throws { + try withLibDir(stamp: "swift 6.1") { libPath in + let result = ToolchainCheckResult(libPath: libPath, swiftVersion: "swift 6.1") + guard case .match = result else { + Issue.record("expected .match, got \(result)") + return + } + } + } + + @Test("Trailing whitespace and newlines are normalized before comparison") + internal func normalizesWhitespace() throws { + try withLibDir(stamp: "swift 6.1\n ") { libPath in + let result = ToolchainCheckResult(libPath: libPath, swiftVersion: "swift 6.1\n") + guard case .match = result else { + Issue.record("expected .match, got \(result)") + return + } + } + } + + @Test("Differing versions report a mismatch carrying both strings") + internal func mismatch() throws { + try withLibDir(stamp: "swift 6.1") { libPath in + let result = ToolchainCheckResult(libPath: libPath, swiftVersion: "swift 6.2") + guard case .mismatch(let bundle, let local) = result else { + Issue.record("expected .mismatch, got \(result)") + return + } + #expect(bundle == "swift 6.1") + #expect(local == "swift 6.2") + } + } + + @Test("A missing stamp file yields stampMissing") + internal func missingStamp() throws { + try withLibDir(stamp: nil) { libPath in + let result = ToolchainCheckResult(libPath: libPath, swiftVersion: "swift 6.1") + guard case .stampMissing = result else { + Issue.record("expected .stampMissing, got \(result)") + return + } + } + } + + @Test("A nil local version yields stampMissing") + internal func nilLocalVersion() throws { + try withLibDir(stamp: "swift 6.1") { libPath in + let result = ToolchainCheckResult(libPath: libPath, swiftVersion: nil) + guard case .stampMissing = result else { + Issue.record("expected .stampMissing, got \(result)") + return + } + } + } +} diff --git a/Tests/SyntaxKitTests/Unit/Execution/WrappedSourceTests.swift b/Tests/SyntaxKitTests/Unit/Execution/WrappedSourceTests.swift new file mode 100644 index 0000000..eed99de --- /dev/null +++ b/Tests/SyntaxKitTests/Unit/Execution/WrappedSourceTests.swift @@ -0,0 +1,105 @@ +// +// WrappedSourceTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import Testing + +@testable import SyntaxKit + +@Suite internal struct WrappedSourceTests { + private static let path = "/tmp/input.swift" + + private func rendered(_ source: String) -> String { + WrappedSource(source: source, originalPath: Self.path).rendered + } + + /// The wrapper always opens with the SyntaxKit import and emits the + /// `Group { … }` + `print(...)` scaffold, regardless of input. + private func expectScaffold(_ rendered: String) { + #expect(rendered.contains("import SyntaxKit")) + #expect(rendered.contains("let __skit_root = Group {")) + #expect(rendered.contains("print(__skit_root.generateCode())")) + #expect(rendered.contains("#sourceLocation(file: \"\(Self.path)\"")) + } + + @Test("No-import source is left in the body, starting at line 1") + internal func noImports() { + let out = rendered("Struct(\"Foo\") {}") + expectScaffold(out) + #expect(out.contains("Struct(\"Foo\") {}")) + #expect(out.contains("#sourceLocation(file: \"\(Self.path)\", line: 1)")) + } + + @Test("Leading imports are hoisted above the wrapper body") + internal func importsOnly() { + let out = rendered("import Foundation\nimport SwiftSyntax\n") + expectScaffold(out) + #expect(out.contains("import Foundation")) + #expect(out.contains("import SwiftSyntax")) + // Imports-only input has no body, so the fence starts at line 1. + #expect(out.contains("#sourceLocation(file: \"\(Self.path)\", line: 1)")) + } + + @Test("Imports are hoisted and the body fence reports the body's line") + internal func mixedImportsAndBody() { + let out = rendered("import Foundation\nlet x = 1\n") + expectScaffold(out) + #expect(out.contains("import Foundation")) + #expect(out.contains("let x = 1")) + // The body slice starts at the newline that is leading trivia of `let`, so + // the fence reports line 1 and the retained leading newline keeps `let x = 1` + // aligned to its original line 2. + #expect(out.contains("#sourceLocation(file: \"\(Self.path)\", line: 1)")) + + // The hoisted import must sit ahead of the `Group {` scaffold, not inside it. + let importIndex = try? #require(out.range(of: "import Foundation")) + let groupIndex = try? #require(out.range(of: "let __skit_root = Group {")) + if let importIndex, let groupIndex { + #expect(importIndex.lowerBound < groupIndex.lowerBound) + } + } + + @Test("A comment before a non-import statement stays in the body") + internal func leadingCommentBeforeBody() { + let out = rendered("// note\nlet x = 1\n") + expectScaffold(out) + // The comment is leading trivia of the first (non-import) statement, so the + // body slice — and thus the rendered output — retains it. + #expect(out.contains("// note")) + #expect(out.contains("let x = 1")) + #expect(out.contains("#sourceLocation(file: \"\(Self.path)\", line: 1)")) + } + + @Test("Empty source still renders the scaffold") + internal func emptySource() { + let out = rendered("") + expectScaffold(out) + #expect(out.contains("#sourceLocation(file: \"\(Self.path)\", line: 1)")) + } +} From fdbf8da06fffbd8b6d37f22b0d4136e35e518445 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 10 Jun 2026 10:40:29 -0400 Subject: [PATCH 51/56] Split FileManager filesystem helpers from domain logic; type FileOutcome.result Extract domain-agnostic filesystem operations into generic FileManager helpers returning clean Swift types instead of Foundation idioms: - FileManager+FileSystem.swift: PathKind enum (replaces the fileExists + ObjCBool dance), FileFingerprint struct (size + mtime), FileEnumerationError, and pathKind/fingerprint/regularFiles(under:)/writeData methods. The recursive walk and path sort live here so deterministic ordering is guaranteed in one place. - URL+Reroot.swift: rerooted(from:onto:), the input->output path mirroring pulled out of writeOutput. FileManager+Execution.swift becomes thin domain wrappers: isLibDir/libStamp over pathKind/fingerprint, collectInputs over regularFiles (mapping FileEnumerationError -> CollectInputsError) plus the SyntaxKit .swift/_-prefix input convention, and writeOutput keeping only Result-folding + rerooted + writeData. Adopt pathKind at the other fileExists sites (RunInput.resolve, OutputCache.store). Tighten DirectoryRender.FileOutcome.result from (any Error)? to RunError?: the only producer (the SDK) funnels everything through RenderTaskResult.writeOutput, which is throws(RunError). The skit consumer drops its now-redundant `as? RunError` downcast. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../SyntaxKit/Execution/DirectoryRender.swift | 11 +- .../Execution/FileManager+Execution.swift | 100 ++++-------- .../Execution/FileManager+FileSystem.swift | 127 ++++++++++++++++ Sources/SyntaxKit/Execution/OutputCache.swift | 2 +- Sources/SyntaxKit/Execution/RunInput.swift | 10 +- .../Execution/Runner+Directory.swift | 33 ++++ Sources/SyntaxKit/Execution/URL+Reroot.swift | 43 ++++++ Sources/skit/Skit.Run+Render.swift | 2 +- .../FileManagerFileSystemTests.swift | 142 ++++++++++++++++++ .../Unit/Execution/OutputCacheTests.swift | 2 +- 10 files changed, 392 insertions(+), 80 deletions(-) create mode 100644 Sources/SyntaxKit/Execution/FileManager+FileSystem.swift create mode 100644 Sources/SyntaxKit/Execution/URL+Reroot.swift create mode 100644 Tests/SyntaxKitTests/Unit/Execution/FileManagerFileSystemTests.swift diff --git a/Sources/SyntaxKit/Execution/DirectoryRender.swift b/Sources/SyntaxKit/Execution/DirectoryRender.swift index 6342e54..0f5f302 100644 --- a/Sources/SyntaxKit/Execution/DirectoryRender.swift +++ b/Sources/SyntaxKit/Execution/DirectoryRender.swift @@ -40,21 +40,22 @@ public struct DirectoryRender: Sendable { /// diagnostics from the spawned `swift`; it may be present whether or not /// the input succeeded (e.g. a successful render that emitted warnings). /// `result` is `nil` when the rendered output was written to its mirrored - /// destination, and carries the error when the input could not be rendered - /// or its output could not be written (typically a `RunError`). + /// destination, and carries the `RunError` when the input could not be + /// rendered (`.renderFailed`) or its output could not be written + /// (`.unexpected`). public struct FileOutcome: Sendable { /// The input file this outcome describes. public let input: URL /// The (possibly path-rewritten) `swift` diagnostics for this input. public let stderr: String - /// `nil` when the output was written; the error otherwise. - public let result: (any Error)? + /// `nil` when the output was written; the `RunError` otherwise. + public let result: RunError? /// Creates an outcome for one rendered (or failed) input. public init( input: URL, stderr: String, - result: (any Error)? + result: RunError? ) { self.input = input self.stderr = stderr diff --git a/Sources/SyntaxKit/Execution/FileManager+Execution.swift b/Sources/SyntaxKit/Execution/FileManager+Execution.swift index 1614431..37e43a8 100644 --- a/Sources/SyntaxKit/Execution/FileManager+Execution.swift +++ b/Sources/SyntaxKit/Execution/FileManager+Execution.swift @@ -39,11 +39,8 @@ extension FileManager { /// True if `path` is a directory containing `libSyntaxKit.{dylib,so}`. internal func isLibDir(_ path: String) -> Bool { - var isDir: ObjCBool = false - guard fileExists(atPath: path, isDirectory: &isDir), isDir.boolValue else { - return false - } - return fileExists(atPath: "\(path)/\(Self.syntaxKitProductName.dylibFilename)") + guard pathKind(atPath: path) == .directory else { return false } + return pathKind(atPath: "\(path)/\(Self.syntaxKitProductName.dylibFilename)") != .missing } /// `/` fingerprint of `libSyntaxKit.{dylib,so}` under @@ -51,48 +48,40 @@ extension FileManager { /// version bump. internal func libStamp(libPath: String) -> String? { let dylib = "\(libPath)/\(Self.syntaxKitProductName.dylibFilename)" - guard let attrs = try? attributesOfItem(atPath: dylib) else { return nil } - let size = (attrs[.size] as? NSNumber)?.intValue ?? 0 - let mtime = (attrs[.modificationDate] as? Date)?.timeIntervalSince1970 ?? 0 - return "\(size)/\(Int(mtime))" + return fingerprint(atPath: dylib).map { + "\($0.size)/\(Int($0.modificationDate.timeIntervalSince1970))" + } } /// Returns every `.swift` file under `inputDir` (recursive), sorted, with - /// hidden files and files prefixed by `_` removed. Sorted output keeps - /// batch behaviour deterministic across runs. + /// hidden files and files prefixed by `_` removed. The recursive walk and + /// sort come from `regularFiles(under:)`; this method adds only the SyntaxKit + /// input convention (`.swift` extension, skip the `_`-prefixed "not an input" + /// sources). /// /// Throws `CollectInputsError.cliError` when the directory can't be /// enumerated, or `.resourceValuesFailure` when a file's resource values /// can't be read — both are bulk failures with nothing per-file to report. internal func collectInputs(at inputDir: URL) throws(CollectInputsError) -> [URL] { - guard - let enumerator = enumerator( - at: inputDir, - includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], - options: [.skipsHiddenFiles] - ) - else { - throw .cliError(CLIError(message: "could not enumerate \(inputDir.path)")) - } - - var result: [URL] = [] - for case let url as URL in enumerator { - let values: URLResourceValues - do { - values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) - } catch { - throw .resourceValuesFailure(error) + let files: [URL] + do { + files = try regularFiles(under: inputDir) + } catch { + // `error` is typed `FileEnumerationError`; map each case onto the + // collect-specific error the caller already presents. + switch error { + case .notEnumerable(let directory): + throw .cliError(CLIError(message: "could not enumerate \(directory.path)")) + case .resourceValuesUnavailable(_, let underlying): + throw .resourceValuesFailure(underlying) } - // Directories aren't outputs. - if values.isDirectory == true { continue } - // Filter for `.swift` regular files, skipping the `_`-prefixed - // convention for "not an input" sources. - guard values.isRegularFile == true else { continue } - guard url.pathExtension == Self.swiftFileExtension else { continue } - guard !url.lastPathComponent.hasPrefix(Self.nonInputFilePrefix) else { continue } - result.append(url.standardizedFileURL) } - return result.sorted { $0.path < $1.path } + + return + files + .filter { $0.pathExtension == Self.swiftFileExtension } + .filter { !$0.lastPathComponent.hasPrefix(Self.nonInputFilePrefix) } + .map(\.standardizedFileURL) } /// Builds the `FileOutcome` for one render result, writing a successful @@ -108,43 +97,20 @@ extension FileManager { outputBase: URL, toolchain: Runner.ToolchainVerification ) -> DirectoryRender.FileOutcome { - let relative = result.input.path.dropFirst(inputBase.path.count + 1) - let destination = outputBase.appendingPathComponent(String(relative)) + // Mirror the input's location under the output base (generic path math). + let destination = result.input.rerooted(from: inputBase, onto: outputBase) // stderr is the toolchain's diagnostics whenever the render produced any — // i.e. on every successful spawn, regardless of how the write then fares. let stderr = (try? result.result.get())?.stderr ?? "" - // Fold the render result into the write result: a render failure passes - // through, a non-zero exit becomes `.renderFailed`, and a clean render is - // committed to disk (capturing any write error as `.unexpected`). - let outcome = result.result.flatMap { processResult -> Result in - guard processResult.exitCode == 0 else { - return .failure( - .renderFailed( - exitCode: processResult.exitCode, - stderr: processResult.stderr, - toolchain: toolchain - ) - ) - } - return Result { - try createDirectory( - at: destination.deletingLastPathComponent(), - withIntermediateDirectories: true - ) - try processResult.stdout.write(to: destination) + let failure: RunError? + do { + try result.writeOutput(to: destination, toolchain: toolchain) { [self] in + try self.writeData($0.output, to: $0.destination) } - .mapError(RunError.unexpected) - } - - // Collapse the success/failure result into the outcome's optional error - // (nil == success). - let failure: (any Error)? - switch outcome { - case .success: failure = nil - case .failure(let error): + } catch { failure = error } diff --git a/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift b/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift new file mode 100644 index 0000000..086e90c --- /dev/null +++ b/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift @@ -0,0 +1,127 @@ +// +// FileManager+FileSystem.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +/// What exists at a filesystem path — the result of a single existence check, +/// replacing the `fileExists(atPath:isDirectory:)` + `ObjCBool` idiom with a +/// value callers can `switch` over. +internal enum PathKind: Equatable { + /// Nothing exists at the path. + case missing + /// A regular file (or any non-directory) exists at the path. + case file + /// A directory exists at the path. + case directory +} + +/// Size + modification date of a file: the change-detection inputs behind a +/// content fingerprint/stamp. +internal struct FileFingerprint: Equatable { + /// File size in bytes (0 when the attribute is absent). + internal let size: Int + /// Last-modification date (Unix epoch when the attribute is absent). + internal let modificationDate: Date +} + +/// A recursive regular-file enumeration failure, decoupled from any domain so +/// the caller maps it onto its own error type. +internal enum FileEnumerationError: Error { + /// The directory couldn't be enumerated at all. + case notEnumerable(URL) + /// A file's resource values couldn't be read mid-walk. + case resourceValuesUnavailable(URL, any Error) +} + +extension FileManager { + /// Classifies what exists at `path` in a single stat — `.missing`, `.file`, + /// or `.directory`. + internal func pathKind(atPath path: String) -> PathKind { + var isDirectory: ObjCBool = false + guard fileExists(atPath: path, isDirectory: &isDirectory) else { + return .missing + } + return isDirectory.boolValue ? .directory : .file + } + + /// Size + modification date of the file at `path`, or `nil` if its attributes + /// can't be read. Absent individual attributes default to `0` / the Unix + /// epoch so a readable item always yields a fingerprint. + internal func fingerprint(atPath path: String) -> FileFingerprint? { + guard let attributes = try? attributesOfItem(atPath: path) else { + return nil + } + let size = (attributes[.size] as? NSNumber)?.intValue ?? 0 + let modificationDate = + (attributes[.modificationDate] as? Date) + ?? Date(timeIntervalSince1970: 0) + return FileFingerprint(size: size, modificationDate: modificationDate) + } + + /// Every regular (non-directory) file under `directory`, recursively, with + /// hidden files skipped and the result sorted by path. Sorted because callers + /// generally want deterministic ordering, and doing it here keeps that + /// guarantee in one place. Unfiltered by extension — the caller decides what + /// counts as relevant. + internal func regularFiles(under directory: URL) throws(FileEnumerationError) -> [URL] { + guard + let enumerator = enumerator( + at: directory, + includingPropertiesForKeys: [.isRegularFileKey, .isDirectoryKey], + options: [.skipsHiddenFiles] + ) + else { + throw .notEnumerable(directory) + } + + var result: [URL] = [] + for case let url as URL in enumerator { + let values: URLResourceValues + do { + values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) + } catch { + throw .resourceValuesUnavailable(url, error) + } + if values.isDirectory == true { continue } + guard values.isRegularFile == true else { continue } + result.append(url) + } + return result.sorted { $0.path < $1.path } + } + + /// Writes `data` to `destination`, first creating any missing intermediate + /// directories. + internal func writeData(_ data: Data, to destination: URL) throws { + try createDirectory( + at: destination.deletingLastPathComponent(), + withIntermediateDirectories: true + ) + try data.write(to: destination) + } +} diff --git a/Sources/SyntaxKit/Execution/OutputCache.swift b/Sources/SyntaxKit/Execution/OutputCache.swift index 6e46ff3..ff69cfd 100644 --- a/Sources/SyntaxKit/Execution/OutputCache.swift +++ b/Sources/SyntaxKit/Execution/OutputCache.swift @@ -175,7 +175,7 @@ public struct OutputCache: Sendable { try fileManager().moveItem(at: staging, to: cacheRoot) } catch { try? fileManager().removeItem(at: staging) - if !fileManager().fileExists(atPath: final.path) { + if fileManager().pathKind(atPath: final.path) == .missing { throw error } } diff --git a/Sources/SyntaxKit/Execution/RunInput.swift b/Sources/SyntaxKit/Execution/RunInput.swift index ee1e72b..6860882 100644 --- a/Sources/SyntaxKit/Execution/RunInput.swift +++ b/Sources/SyntaxKit/Execution/RunInput.swift @@ -46,16 +46,16 @@ public enum RunInput { /// existing file → `.singleFile`. Throws `RunError.invalidInput` if the path /// doesn't exist, or if a directory input wasn't given an explicit output. public static func resolve(input: String, output: String?) throws(RunError) -> RunInput { - var isDirectory: ObjCBool = false - guard FileManager.default.fileExists(atPath: input, isDirectory: &isDirectory) else { + switch FileManager.default.pathKind(atPath: input) { + case .missing: throw RunError.invalidInput("input does not exist: \(input)") - } - if isDirectory.boolValue { + case .directory: guard let output else { throw RunError.invalidInput("directory inputs require -o ") } return .directory(inputDir: input, outputDir: output) + case .file: + return .singleFile(inputPath: input, outputPath: output) } - return .singleFile(inputPath: input, outputPath: output) } } diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index af9fd3c..30df635 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -132,4 +132,37 @@ extension Runner { internal struct RenderTaskResult: Sendable { internal let input: URL internal let result: Result + + struct OutputDestination: Sendable { + let output: Data + let destination: URL + } + + internal func writeOutput( + to destination: URL, toolchain: Runner.ToolchainVerification, + using writeOutputDestination: (OutputDestination) throws -> Void + ) throws(RunError) { + let result = try self.result.get() + + guard result.exitCode == 0 else { + throw + .renderFailed( + exitCode: result.exitCode, + stderr: result.stderr, + toolchain: toolchain + ) + } + + let outputDestination = OutputDestination(output: result.stdout, destination: destination) + + do { + try writeOutputDestination(outputDestination) + } catch { + throw .unexpected(error) + } + // try writeOutputWith(destination) + // return Result { try writeData(processResult.stdout, to: destination) } + // .mapError(RunError.unexpected) + + } } diff --git a/Sources/SyntaxKit/Execution/URL+Reroot.swift b/Sources/SyntaxKit/Execution/URL+Reroot.swift new file mode 100644 index 0000000..794aa6b --- /dev/null +++ b/Sources/SyntaxKit/Execution/URL+Reroot.swift @@ -0,0 +1,43 @@ +// +// URL+Reroot.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +extension URL { + /// Re-roots this URL from `base` onto `newBase`, preserving the relative + /// subpath — e.g. `/in/sub/a.swift` re-rooted from `/in` onto `/out` becomes + /// `/out/sub/a.swift`. Used to mirror an input tree into an output tree. + /// + /// - Precondition: `self` is located under `base` (its path is prefixed by + /// `base`'s). Callers that enumerate `base` satisfy this by construction. + internal func rerooted(from base: URL, onto newBase: URL) -> URL { + let relative = path.dropFirst(base.path.count + 1) + return newBase.appendingPathComponent(String(relative)) + } +} diff --git a/Sources/skit/Skit.Run+Render.swift b/Sources/skit/Skit.Run+Render.swift index e26738d..701bec0 100644 --- a/Sources/skit/Skit.Run+Render.swift +++ b/Sources/skit/Skit.Run+Render.swift @@ -137,7 +137,7 @@ extension Skit.Run { // .renderFailed already had its stderr surfaced above. Other failures // (process spawn, write) carry the diagnostic in the error itself. if let error = outcome.result { - if let runError = error as? RunError, case .renderFailed = runError { + if case .renderFailed = error { continue } FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) diff --git a/Tests/SyntaxKitTests/Unit/Execution/FileManagerFileSystemTests.swift b/Tests/SyntaxKitTests/Unit/Execution/FileManagerFileSystemTests.swift new file mode 100644 index 0000000..1514363 --- /dev/null +++ b/Tests/SyntaxKitTests/Unit/Execution/FileManagerFileSystemTests.swift @@ -0,0 +1,142 @@ +// +// FileManagerFileSystemTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import Testing + +@testable import SyntaxKit + +@Suite internal struct FileManagerFileSystemTests { + private let fileManager = FileManager.default + + /// Creates a fresh empty temp directory, runs `body` against it, and removes + /// it afterward. + private func withTempDir(_ body: (URL) throws -> Void) throws { + let dir = fileManager.temporaryDirectory + .appendingPathComponent("fs-\(UUID().uuidString)") + try fileManager.createDirectory(at: dir, withIntermediateDirectories: true) + defer { try? fileManager.removeItem(at: dir) } + try body(dir) + } + + // MARK: - pathKind + + @Test("pathKind classifies directory, file, and missing paths") + internal func pathKindClassifies() throws { + try withTempDir { dir in + #expect(fileManager.pathKind(atPath: dir.path) == .directory) + + let file = dir.appendingPathComponent("a.txt") + try Data("x".utf8).write(to: file) + #expect(fileManager.pathKind(atPath: file.path) == .file) + + #expect(fileManager.pathKind(atPath: dir.appendingPathComponent("nope").path) == .missing) + } + } + + // MARK: - fingerprint + + @Test("fingerprint reports byte size for an existing file") + internal func fingerprintReportsSize() throws { + try withTempDir { dir in + let file = dir.appendingPathComponent("a.bin") + try Data("hello".utf8).write(to: file) + + let fingerprint = try #require(fileManager.fingerprint(atPath: file.path)) + #expect(fingerprint.size == 5) + } + } + + @Test("fingerprint is nil for a missing file") + internal func fingerprintMissing() throws { + try withTempDir { dir in + #expect(fileManager.fingerprint(atPath: dir.appendingPathComponent("nope").path) == nil) + } + } + + // MARK: - regularFiles + + @Test("regularFiles walks recursively, skips hidden + directories, sorted by path") + internal func regularFilesWalk() throws { + try withTempDir { dir in + let sub = dir.appendingPathComponent("sub") + try fileManager.createDirectory(at: sub, withIntermediateDirectories: true) + try Data("1".utf8).write(to: dir.appendingPathComponent("b.swift")) + try Data("2".utf8).write(to: dir.appendingPathComponent("a.swift")) + try Data("3".utf8).write(to: sub.appendingPathComponent("c.swift")) + try Data("4".utf8).write(to: dir.appendingPathComponent(".hidden")) + + let files = try fileManager.regularFiles(under: dir) + let names = files.map(\.lastPathComponent) + + // The hidden file and the `sub` directory itself are excluded; nested + // files are included; the result is sorted by full path. + #expect(names == ["a.swift", "b.swift", "c.swift"]) + #expect(files == files.sorted { $0.path < $1.path }) + } + } + + @Test("regularFiles yields nothing for a missing directory") + internal func regularFilesMissingDir() throws { + try withTempDir { dir in + // `enumerator(at:)` returns an empty (non-nil) enumerator for a missing + // directory rather than nil, so the walk yields an empty list rather than + // throwing — matching the directory-mode behavior of treating an empty + // input set as a no-op. + let missing = dir.appendingPathComponent("does-not-exist") + let files = try fileManager.regularFiles(under: missing) + #expect(files.isEmpty) + } + } + + // MARK: - writeData + + @Test("writeData creates intermediate directories") + internal func writeDataCreatesIntermediates() throws { + try withTempDir { dir in + let destination = dir.appendingPathComponent("a/b/c/out.txt") + try fileManager.writeData(Data("payload".utf8), to: destination) + + #expect(fileManager.pathKind(atPath: destination.path) == .file) + #expect(try Data(contentsOf: destination) == Data("payload".utf8)) + } + } + + // MARK: - URL.rerooted + + @Test("rerooted preserves the relative subpath under a new base") + internal func rerootedPreservesSubpath() { + let input = URL(fileURLWithPath: "/in/sub/a.swift") + let rerooted = input.rerooted( + from: URL(fileURLWithPath: "/in"), + onto: URL(fileURLWithPath: "/out") + ) + #expect(rerooted.path == "/out/sub/a.swift") + } +} diff --git a/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift b/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift index 343eb9e..02d7224 100644 --- a/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift +++ b/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift @@ -35,7 +35,7 @@ import Testing @Suite internal struct OutputCacheTests { /// A `ProcessInfo` whose environment is fixed, so cache-key derivation can be /// tested without depending on the real process environment. - private final class FakeProcessInfo: ProcessInfo, Sendable { + private final class FakeProcessInfo: ProcessInfo { private let fixedEnvironment: [String: String] init(environment: [String: String]) { From 57fbfc4c9eb7cb153741f3afa43e4fce9c88c2f0 Mon Sep 17 00:00:00 2001 From: leogdion Date: Wed, 10 Jun 2026 12:18:27 -0400 Subject: [PATCH 52/56] Fix non-Apple build; hoist nested types; unify build scripts - Replace OutputCache's concrete ProcessInfo dep with an EnvironmentProvider protocol so FakeEnvironment (struct) replaces the FakeProcessInfo subclass that broke on swift-corelibs-foundation (final class, environment on an extension). Clears the OutputCacheTests failure on Linux/Windows/Android. - Add swift-system as a conditional dep + canImport(System)/SystemPackage shim so the skit target compiles on Linux/Windows. - Hoist Runner.SetupError -> RunnerSetupError, Runner.ToolchainVerification -> ToolchainVerification, Skit.Run.CommandError -> RunCommandError, DirectoryRender.FileOutcome -> FileOutcome. - Replace DirectoryRender with [FileOutcome] + a failureCount extension. - Lift render/renderSingle/renderBatch out of extension Skit.Run into enum Render (none used self). - Move SKIT_LIB_DIR lookup to ProcessInfo.skitLibPath. - Inject FileManager via @Sendable () -> FileManager on Runner; thread through Runner+Session and Bundle.resolveLibPath. - Delete Docs/skit-internals.md + Docs/string-literal-audit.md. - Unify Scripts/build-skit-{debug,release}.sh into Scripts/build-skit.sh (release default; --debug flag). - Lint cleanups: split FileManager+FileSystem.swift into PathKind / FileFingerprint / FileEnumerationError; extract RenderTaskResult to its own file; reorder type contents (cases/properties/init/methods); fix conditional-returns-on-newline + multiline-arguments-brackets; add doc comment to ContentHasher.update(data:). Lint goes from 21 violations to 5 (the remaining 5 are all pre-existing). Co-Authored-By: Claude Opus 4.7 (1M context) --- Docs/skit-internals.md | 116 ------------ Docs/skit.md | 7 +- Docs/string-literal-audit.md | 173 ------------------ Package.swift | 8 +- Scripts/build-skit-debug.sh | 81 -------- .../{build-skit-release.sh => build-skit.sh} | 72 ++++++-- .../Execution/Bundle+ResolveLibPath.swift | 9 +- .../SyntaxKit/Execution/ContentHasher.swift | 2 + .../SyntaxKit/Execution/DirectoryRender.swift | 82 --------- ...ironmentProvider+SyntaxKitCacheRoot.swift} | 20 +- .../Execution/EnvironmentProvider.swift | 44 +++++ .../Execution/FileEnumerationError.swift | 39 ++++ .../SyntaxKit/Execution/FileFingerprint.swift | 39 ++++ .../Execution/FileManager+Execution.swift | 10 +- .../Execution/FileManager+FileSystem.swift | 30 --- Sources/SyntaxKit/Execution/FileOutcome.swift | 72 ++++++++ Sources/SyntaxKit/Execution/OutputCache.swift | 24 +-- Sources/SyntaxKit/Execution/PathKind.swift | 40 ++++ .../Execution/RenderTaskResult.swift | 73 ++++++++ Sources/SyntaxKit/Execution/RunError.swift | 2 +- .../Execution/Runner+Directory.swift | 63 +------ .../SyntaxKit/Execution/Runner+Session.swift | 35 ++-- Sources/SyntaxKit/Execution/Runner.swift | 33 ++-- .../Execution/RunnerSetupError.swift | 42 +++++ .../Execution/ToolchainCheckResult.swift | 6 +- .../Execution/ToolchainVerification.swift | 45 +++++ .../SyntaxKit/Execution/WrappedSource.swift | 60 +++--- Sources/skit/ProcessInfo+SkitLibPath.swift | 41 +++++ Sources/skit/README.md | 5 +- .../{Skit.Run+Render.swift => Render.swift} | 68 +++---- Sources/skit/RunCommandError.swift | 154 ++++++++++++++++ Sources/skit/Skit+Run.swift | 24 +-- Sources/skit/Skit.Run+CommandError.swift | 156 ---------------- .../skit/Subprocess.Configuration+Swift.swift | 9 +- .../Unit/Execution/OutputCacheTests.swift | 27 +-- .../Execution/RunnerRenderFailureTests.swift | 2 +- 36 files changed, 833 insertions(+), 880 deletions(-) delete mode 100644 Docs/skit-internals.md delete mode 100644 Docs/string-literal-audit.md delete mode 100755 Scripts/build-skit-debug.sh rename Scripts/{build-skit-release.sh => build-skit.sh} (54%) delete mode 100644 Sources/SyntaxKit/Execution/DirectoryRender.swift rename Sources/SyntaxKit/Execution/{ProcessInfo+SyntaxKitCacheRoot.swift => EnvironmentProvider+SyntaxKitCacheRoot.swift} (71%) create mode 100644 Sources/SyntaxKit/Execution/EnvironmentProvider.swift create mode 100644 Sources/SyntaxKit/Execution/FileEnumerationError.swift create mode 100644 Sources/SyntaxKit/Execution/FileFingerprint.swift create mode 100644 Sources/SyntaxKit/Execution/FileOutcome.swift create mode 100644 Sources/SyntaxKit/Execution/PathKind.swift create mode 100644 Sources/SyntaxKit/Execution/RenderTaskResult.swift create mode 100644 Sources/SyntaxKit/Execution/RunnerSetupError.swift create mode 100644 Sources/SyntaxKit/Execution/ToolchainVerification.swift create mode 100644 Sources/skit/ProcessInfo+SkitLibPath.swift rename Sources/skit/{Skit.Run+Render.swift => Render.swift} (70%) create mode 100644 Sources/skit/RunCommandError.swift delete mode 100644 Sources/skit/Skit.Run+CommandError.swift diff --git a/Docs/skit-internals.md b/Docs/skit-internals.md deleted file mode 100644 index d524cfe..0000000 --- a/Docs/skit-internals.md +++ /dev/null @@ -1,116 +0,0 @@ -# `skit` internals: Runner, OutputCache, toolchain utilities - -A contributor's map of the modules that do the actual work inside `skit run`. For the **why** (design rationale, trade-offs, sharp edges) see [`Docs/skit.md`](skit.md). For the **what** (flags, behaviour) see [`Sources/skit/README.md`](../Sources/skit/README.md). This doc covers the **how** — what each module needs, what it produces, and how they call each other. - -## The pipeline - -`Skit+Run.swift` is the only caller. Per invocation it does, in order: - -1. **`Bundle.main.resolveLibPath(candidates:)`** (`Sources/skit/Bundle+ResolveLibPath.swift`) — locate the lib bundle (`libSyntaxKit.dylib` + swiftmodules). The caller passes `--lib` and `$SKIT_LIB_DIR` as candidates; Bundle falls back to `/lib/` and `/../lib/skit/`. -2. **`ToolchainCheckResult(libPath:)`** (`Sources/skit/ToolchainCheckResult.swift`) — verify the bundle's recorded `swift --version` matches the local one. Refuses to spawn on mismatch. -3. **`Runner(libPath:cache:timeoutSeconds:)`** then **`runner(input:output:)`** — bind the per-invocation configuration into a `Runner` value and call it (via `callAsFunction`) with the raw input/output paths. The runner classifies single-file vs. directory mode and everything downstream (wrap, spawn, output-cache lookup/store) happens inside `Runner`. - -The **toolchain utilities** are consumed transitively (by `Bundle+ResolveLibPath`, `FileManager+IsLibDir`, and `OutputCache`) — they're never called directly from the CLI driver. **OutputCache** is an implementation detail of `Runner.processFile`, never called from the CLI driver either. The CLI driver only sees two surfaces: `Bundle.main.resolveLibPath` and calling the `Runner` value itself (`runner(input:output:)`). - -## Toolchain utilities (no single file) - -**Purpose.** Provide the shared utilities that describe (a) the local `swift` toolchain, (b) the bundled SyntaxKit dylib, and (c) where skit's caches live on disk. These are infrastructure, not features — they exist so the rest of the code doesn't have to re-derive them. There is no longer a `Toolchain.swift`: each utility was lifted onto the type it naturally belongs to (no file-scope globals, per the project convention). - -**Surface area** (all inside `#if canImport(Subprocess)`): - -| Symbol | Defined in | What it returns | Consumers | -|---|---|---|---| -| `String.dylibFilename` | `String+DylibFilename.swift` | `"lib.dylib"` on macOS, `"lib.so"` on Linux | `FileManager.isLibDir`, `FileManager.libStamp` | -| `FileManager.libStamp(libPath:)` | `FileManager+LibStamp.swift` | `"/"` of `libSyntaxKit.{dylib,so}`, or nil | `OutputCache.key` | -| `Skit.Run.captureSwiftVersion()` | `Skit+Run.swift` (`fileprivate`) | verbatim `swift --version` stdout (≤4 KiB), or nil on spawn failure | `Skit.Run.run` (called once per invocation, then threaded into `ToolchainCheckResult.init` and `OutputCache.init`) | -| `ProcessInfo.syntaxKitCacheRoot(default:)` | `ProcessInfo+SyntaxKitCacheRoot.swift` | `$XDG_CACHE_HOME/syntaxkit` when that env var is set, else the passed-in `default` | `OutputCache.init` | - -`OutputCache` supplies that `default` via its own `private static let defaultCacheRoot` — `~/Library/Caches/com.brightdigit.SyntaxKit` (macOS) or `~/.cache/syntaxkit` (Linux), computed once since the home dir is fixed for the process lifetime. - -**Why these belong together.** They all answer the same question from different angles: "what state of the world does a cache key depend on?" The Swift toolchain (`captureSwiftVersion`), the runtime dylib (`libStamp` via `String.dylibFilename`), and the on-disk cache layout (`syntaxKitCacheRoot`). Together they give `OutputCache` everything it needs to compute a stable, sound key. Each now lives as a method/extension on the type that owns its data: the dylib helpers on `String`/`FileManager`, the cache-root resolver on `ProcessInfo`, and the `swift --version` spawn on `Skit.Run` (its sole caller). - -## Runner (`Runner.swift`) - -**Purpose.** Orchestrate per-input render: pick single-file vs. directory mode, wrap each input into a complete Swift program, spawn `swift` on it with a timeout watchdog, surface output, and consult the output cache. - -**Shape.** `internal struct Runner: Sendable`, constructed once per invocation in `Skit.Run.run` and (being `Sendable`) shared across the concurrent `runOne` tasks in directory mode. It holds the per-invocation configuration as stored properties, so the individual inputs don't re-thread it: -- `libPath: String` — the lib bundle dir from step 1; reused for the `swift` invocation's link/rpath flags. -- `cache: OutputCache?` — the shared output cache, or nil under `--no-cache`; gates lookup/store. -- `timeoutSeconds: Int` — per-input watchdog (`0` opts out; default 60s; on expiry, exit 124 matching POSIX `timeout(1)`). - -**Single entry point.** `Runner` is callable: `callAsFunction(input:output:)` is the only `internal` method (besides `init`), so `Skit.Run.run` constructs a `Runner` and invokes it directly — `try await runner(input:, output:)` — with the raw `--input`/`-o` strings. It resolves the input via `RunInput.resolve(input:output:)` (`RunInput.swift`) — a two-case enum (`.singleFile` / `.directory`) that stats the path and enforces existence + the directory `-o` requirement, throwing `ValidationError` otherwise — then dispatches to one of two **private** mode methods: - -| Method (private) | When | Failure semantics | -|---|---|---| -| `runSingleFile(inputPath:outputPath:)` | One input, one output (or stdout) | Calls `exit()` on non-zero subprocess result — caller won't see a thrown error in that path | -| `runDirectory(inputDir:outputDir:)` | Walks `**/*.swift` under `inputDir`, bounded concurrency = `ProcessInfo.activeProcessorCount`, mirrors output into `outputDir` | Returns 0/1, which `run` re-throws as `ExitCode` — partial failure allowed, successful peers still written, one-line summary printed to stderr | - -**The per-input work** (`processFile(inputPath:)`, a `private` instance method): - -1. Load source bytes. -2. If `cache` is non-nil, compute the output cache key via `cache.key(forInput:libPath:)` and try a hit. -3. On miss: `wrap` the source (hoist imports, splice body into `Group { … }`, fence with `#sourceLocation` so compiler diagnostics map back to the original file), write to a per-invocation `skit-/` temp dir, spawn `swift`, rewrite stderr to swap the temp path for the original path, store the result via `cache.store(key:data:)`, return. - -The temp dir is cleaned with `defer { try? removeItem }` so a failed spawn doesn't leak files. `wrap`, `collectInputs`, and `exitCode` are pure `private static` helpers (no dependency on the stored configuration). - -**The spawn** (`runSwift(wrappedPath:)`). Fixed argument list: -``` -swift -suppress-warnings - -I -L -lSyntaxKit - -Xcc -I -Xcc /_SwiftSyntaxCShims-include - -Xlinker -rpath -Xlinker - -``` -Raced against a sleep watchdog in a throwing task group; the loser is cancelled. `timeoutSeconds <= 0` skips the race entirely. - -**Output bounds.** `stdoutLimitBytes = 16 MiB`, `stderrLimitBytes = 1 MiB`. Above either limit, `Subprocess` raises a clear error rather than silently truncating. Timeout exit code is 124 (`timeoutExitCode`), matching POSIX `timeout(1)`. - -## OutputCache (`OutputCache.swift`) - -**Purpose.** Cache the rendered Swift (the stdout of the spawned `swift` for a given input) so re-running with unchanged inputs avoids the spawn entirely. Hit cost ≈ 0.14s on macOS vs. ~0.5s for a cold script-mode `swift` spawn. - -**Shape.** Single `internal struct OutputCache: Sendable`, built once per `skit run` invocation in `Skit.Run.run` and shared across every input. `init(swiftVersion:fileManager:processInfo:)` is non-throwing — it resolves the cache root via `processInfo.syntaxKitCacheRoot(default: Self.defaultCacheRoot)`, binds it to `self.root` (`/outputs/`), and stores `swiftVersion` so key derivation never has to re-spawn `swift`. The caller constructs it directly (`noCache ? nil : OutputCache(swiftVersion:)`); `--no-cache` is the only path that yields a nil cache. - -Sendability: the `FileManager` is injected as a `@Sendable () -> FileManager` closure (default `.default`) rather than stored directly, so the struct derives `Sendable` without an `@unchecked` escape hatch. Runner shares one `OutputCache?` instance across concurrent `runOne` tasks in directory mode. - -**Surface** (all `internal` instance methods): - -- **`key(forInput:libPath:)`** — synchronous (no `await`), mixes: - - schema version (`Self.schemaVersion = "v1"` — bump to invalidate everything) - - input source bytes (the primary driver) - - `self.swiftVersion` (captured at init from `Skit.Run.run`) - - libSyntaxKit `/` stamp (via `FileManager.libStamp(libPath:)`) - - sorted, NUL-terminated `SKIT_*` / `SYNTAXKIT_*` env vars -- **`lookup(key:)`** — returns the cached `output.swift` bytes or nil. -- **`store(key:data:)`** — atomic stage+rename: writes into `tmp../output.swift` next to the key dir, then `moveItem` to install. If a concurrent peer beat us to it, swallow the rename error and drop our staging copy; re-throw only if the destination is still missing afterwards. - -Storage is wrapped in `try?` at the caller in `processFile` — a cache *write* failure is never a render failure; the next run just re-spawns. - -**On-disk layout.** `/outputs//output.swift`. The per-key directory is derived by the private `directory(for:)` helper; the root comes from `processInfo.syntaxKitCacheRoot(default:)` once, at init. - -**Why FNV-1a, not SHA-256.** The cache keys aren't security-critical — there's no adversary trying to forge a collision — so `ContentHasher` uses a 64-bit FNV-1a. Deterministic across processes and platforms (unlike Swift's stdlib `Hasher`, whose seed is per-process randomized), which is what makes the keys usable as on-disk directory names. - -## Data-flow recap - -``` -Skit.Run.run() - ├─ Bundle.main.resolveLibPath(candidates: --lib, $SKIT_LIB_DIR) → libPath - │ └─ FileManager.default.isLibDir → "SyntaxKit".dylibFilename - ├─ self.captureSwiftVersion() → swiftVersion (spawned exactly once) - ├─ ToolchainCheckResult(libPath:, swiftVersion:) → gate (compare to bundle stamp) - ├─ OutputCache(swiftVersion:) → cache (nil only under --no-cache) - ├─ Runner(libPath:, cache:, timeoutSeconds:) → runner (holds the config) - └─ runner(input:, output:) → callAsFunction → RunInput.resolve → single-file / directory - └─ processFile(inputPath:) - ├─ cache?.key(forInput: source, libPath:) ← self.swiftVersion + FileManager.default.libStamp - ├─ cache.lookup(key:) ← hit returns immediately - ├─ Self.wrap(source) → temp wrapper.swift - ├─ runSwift(wrappedPath:) ← spawns `swift` linked against libSyntaxKit - └─ try? cache.store(key:, data:) ← on the way out (atomic stage+rename) -``` - -Three coupling facts worth remembering: - -- **The toolchain utilities are shared infrastructure.** `OutputCache` depends on three of the four; the bundle-resolution code depends on `dylibFilename`. They no longer share a `Toolchain.swift` — each lives as a method/extension on the type that owns its data, called like any other instance API. -- **Runner is the only place OutputCache is touched.** If you ever want a non-CLI consumer of skit (e.g. a long-running server), you'd construct a `Runner` and call its entry points (or replicate its cache logic). Don't instantiate `OutputCache` from elsewhere. -- **Cache safety under concurrency.** Both the store path (in `OutputCache`) and the bundle-resolution path are designed to be safe under concurrent invocations: write into `tmp../`, then `moveItem` to install. Race-loser swallows the rename error if the destination is now populated by a peer. diff --git a/Docs/skit.md b/Docs/skit.md index 72f8099..1832afc 100644 --- a/Docs/skit.md +++ b/Docs/skit.md @@ -75,7 +75,7 @@ error: module compiled with Swift 6.3 cannot be imported by the Swift 6.3.2 comp If `skit`'s bundled `lib/SyntaxKit.swiftmodule` doesn't match the user's `swift`, the spawned interpreter emits exactly this diagnostic and refuses to compile the wrapped input. The user is left staring at a cryptic message that doesn't name the actual problem. -`skit` mitigates this by recording the build toolchain at bundle time. `Scripts/build-skit-release.sh` writes `lib/swift-version.txt` containing the output of `swift --version`. On startup, `skit run` reads the stamp and compares it to a freshly-captured local `swift --version`. Three paths: +`skit` mitigates this by recording the build toolchain at bundle time. `Scripts/build-skit.sh` writes `lib/swift-version.txt` containing the output of `swift --version`. On startup, `skit run` reads the stamp and compares it to a freshly-captured local `swift --version`. Three paths: - **Match.** Proceed silently. - **Mismatch.** Print a clear error naming both versions and the rebuild command, exit 2. Skip with `--no-toolchain-check`. @@ -153,7 +153,7 @@ A `#if os(macOS) … #endif` block in the input is evaluated when the wrapped fi **macOS** is the primary target. The build and release flows live in `Scripts/`; the bundle is portable across machines with the same Swift version. -**Linux** is verified on `swift:6.0-jammy/aarch64`. One adjustment compared to macOS: the Mach-O `install_name` rewrite in `Scripts/build-skit-release.sh` is skipped — GNU `ld` doesn't accept the flag. The `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. +**Linux** is verified on `swift:6.0-jammy/aarch64`. One adjustment compared to macOS: the Mach-O `install_name` rewrite in `Scripts/build-skit.sh` is skipped — GNU `ld` doesn't accept the flag. The `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. The Foundation.Process workarounds described earlier are Linux-driven. `Process.waitUntilExit()` blocks indefinitely on already-exited children on `swift:6.0-jammy/aarch64` — `skit` uses `DispatchSemaphore` everywhere a child wait is needed, and drains stdout/stderr pipes concurrently to avoid deadlocks when either pipe buffer fills. @@ -171,8 +171,7 @@ A few things were considered for v1 and explicitly punted: ## Reference - [`Sources/skit/README.md`](../Sources/skit/README.md) — per-target quick reference (flag table). -- [`Docs/skit-internals.md`](skit-internals.md) — per-module reference for Runner, OutputCache, and Toolchain. -- [`Scripts/build-skit-release.sh`](../Scripts/build-skit-release.sh) — release-bundle builder. +- [`Scripts/build-skit.sh`](../Scripts/build-skit.sh) — release-bundle builder (use `--debug` for the fast iteration variant). - [`Docs/research/tuist-manifest-pipeline.md`](research/tuist-manifest-pipeline.md) — the manifest-pipeline pattern this CLI borrows from. - [Issue #154](https://github.com/brightdigit/SyntaxKit/issues/154) — original tracking issue. - [Issue #157](https://github.com/brightdigit/SyntaxKit/issues/157), [Issue #158](https://github.com/brightdigit/SyntaxKit/issues/158) — follow-ups. diff --git a/Docs/string-literal-audit.md b/Docs/string-literal-audit.md deleted file mode 100644 index 1ba3692..0000000 --- a/Docs/string-literal-audit.md +++ /dev/null @@ -1,173 +0,0 @@ -# String Literal Audit — SyntaxKit - -**Date:** 2026-06-09 -**Branch:** `research/swift-manifest-codegen` -**Scope:** All 208 `.swift` files across 5 modules (`SyntaxKit`, `skit`, `DocumentationHarness`, `SyntaxParser`, `TokenVisitor`). Markdown samples under `Documentation.docc` were excluded as illustrative, not product code. - -**Goal:** Identify hardcoded string literals that should be moved into named constants, prioritized by extraction value (duplication × cross-module reach × domain meaning). - ---- - -## Summary of counts - -| Module | Files | Non-empty string literals | -|---|---|---| -| SyntaxKit | 150 | 292 | -| DocumentationHarness | 20 | 55 | -| TokenVisitor | 28 | 43 | -| skit | 6 | 41 | -| SyntaxParser | 4 | 19 | - -Hotspots within SyntaxKit: `Expressions/` (80), `Collections/` (78), `Execution/` (47). - ---- - -## Tier 1 — Highest value (duplicated + cross-cutting) - -### 1.1 Empty-identifier fallback `.identifier("")` — ~30+ sites - -The `DeclReferenceExprSyntax(baseName: .identifier(""))` "couldn't-resolve-expression" placeholder is the single most pervasive pattern in the codebase. It is a degenerate placeholder emitted whenever an `ExprSyntax` downcast fails. - -**Recommendation:** Not just a constant — a shared factory (e.g. `ExprSyntax.placeholderReference` / `static let emptyDeclReferenceExpr`) so the fallback behavior lives in one place. - -| Area | Locations (file:line) | -|---|---| -| Expressions (~13) | ConditionalOp.swift:42,52,62; FunctionCallExp.swift:58,136; Literal.swift:136,159,179,182; Literal+ExprCodeBlock.swift:97; NegatedPropertyAccessExp.swift:40; OptionalChainingExp.swift:44; PropertyAccessExp.swift:41; ReferenceExp.swift:47 | -| ControlFlow | Guard.swift:56,67; If+Conditions.swift:69,79; SwitchCase.swift:64; For.swift:47,55; Switch.swift:41 | -| Core / CodeBlocks | Core/CodeBlock.swift:52; CodeBlocks/CodeBlock+Generate.swift:51; CodeBlocks/CodeBlock+ExprSyntax.swift:52 | -| Utilities / Variables | Utilities/EnumCase+Syntax.swift:49; Utilities/Let.swift:50; Variables/Variable.swift:182 | -| Parameters / ErrorHandling | Parameters/ParameterExp.swift:44,53; ErrorHandling/Catch.swift:98; ErrorHandling/Throw.swift:45 | -| Collections | CodeBlock+DictionaryValue.swift:51; TupleAssignment.swift:174 | - -### 1.2 Swift type names — ~35 sites - -Universal Swift type tokens scattered across modules. The canonical inference table is `Expressions/Literal.swift`'s `typeName` switch. - -**Recommendation:** Package-wide `TypeNames` caseless enum. - -| String | Count | Locations (file:line) | -|---|---|---| -| `"Any"` | 18 | Collections/TupleLiteralArray.swift:52-59; Collections/DictionaryLiteral.swift:47,48; Collections/ArrayLiteral.swift:47; Expressions/ClosureParameter.swift:45; Expressions/ClosureType.swift:44; Expressions/Literal.swift:67,80-102 | -| `"Void"` | 3 | Expressions/ClosureType.swift:67,95,127 | -| `"String"` | 4 | Collections/TupleLiteralArray.swift:50; Expressions/Literal.swift:61,73; Variables/Variable+LiteralInitializers.swift:63 | -| `"Int"` | 4 | Collections/TupleLiteralArray.swift:47; Expressions/Literal.swift:63,72; Variables/Variable+LiteralInitializers.swift:80 | -| `"Double"` | 4 | Collections/TupleLiteralArray.swift:48; Expressions/Literal.swift:62,74; Variables/Variable+LiteralInitializers.swift:114 | -| `"Bool"` | 4 | Collections/TupleLiteralArray.swift:49; Expressions/Literal.swift:65,75; Variables/Variable+LiteralInitializers.swift:97 | -| `"Any?"` | 3 | Collections/TupleLiteralArray.swift:51; Expressions/Literal.swift:64,76 | -| `"[Any]"` | 2 | Collections/ArrayLiteral.swift:45; Expressions/Literal.swift:92 | -| `"[Any: Any]"` | 3 | Collections/DictionaryExpr.swift:44; Collections/DictionaryLiteral.swift:45; Expressions/Literal.swift:99 | -| `"[String: Any]"` | 1 | Collections/DictionaryExpr.swift:46 | -| `"[String]"` | 1 | Collections/Array+LiteralValue.swift:34 | -| `"[Int: String]"` | 1 | Collections/Dictionary+LiteralValue.swift:34 | - -### 1.3 Duplicated string-escape block — verbatim copy in 2 files - -`Collections/Array+LiteralValue.swift` and `Collections/Dictionary+LiteralValue.swift` contain an identical escape map: `\` → `\\`, `"` → `\"`, `\n` → `\n`, `\r` → `\r`, `\t` → `\t` (lines 47–51 in each). - -**Recommendation:** One shared `String.escapedForSwiftLiteral()` helper. (Related-but-distinct escape logic also lives in `Execution/WrappedSource.swift:105-106` and `TokenVisitor/TriviaPiece.swift`.) - -### 1.4 Whitespace / separators — most cross-cutting tokens - -| String | Count | Locations | -|---|---|---| -| `"\n"` | 12+ | skit/Skit.Run+Render.swift (stderr writes), skit/Skit+Run.swift:104,153; DocumentationHarness/CodeBlockExtraction.swift:141; TokenVisitor/TriviaPiece.swift:63; Collections/Array+LiteralValue.swift:49, Dictionary+LiteralValue.swift:49; Execution/WrappedSource.swift:99 | -| `", "` | 10+ | Collections (ArrayLiteral, Array+LiteralValue, Dictionary+LiteralValue, TupleLiteralArray, DictionaryLiteral, DictionaryExpr); Expressions (ClosureType.swift:91, Literal.swift:88) | - ---- - -## Tier 2 — Domain literals worth naming - -### Operators (Expressions, Patterns) -| String | Locations | -|---|---| -| `"+="` (defined twice independently) | Expressions/Infix.swift:40; Expressions/PlusAssign.swift:53 | -| `"-="`, `"=="`, `"!="`, `">"`, `"<"` | Expressions/Infix.swift:36-41 | -| `"!"` (prefix negation) | Expressions/NegatedPropertyAccessExp.swift:45 | -| `"..<"`, `"..."` (range operators) | Patterns/Range+PatternConvertible.swift:39,58 | - -### skit CLI args & swiftc flags -| String | Locations | -|---|---| -| `"swift"` (executable name) | skit/Skit.Run+Render.swift:160; skit/Subprocess.Configuration+Swift.swift:60 | -| `"--version"`, `"--timeout"` | skit/Skit.Run+Render.swift:163; skit/Skit+Run.swift:73,87 | -| `"run"`, `"parse"`, `"output"`, `"lib"`, `"no-cache"`, `"no-toolchain-check"` | skit/Skit+Run.swift:47,55,62,67,79; skit/Skit+Parse.swift:37 | -| `"-I"`, `"-L"`, `"-lSyntaxKit"`, `"-Xcc"`, `"-Xlinker"`, `"-rpath"`, `"-suppress-warnings"` | skit/Subprocess.Configuration+Swift.swift:50-57 | -| `"SKIT_LIB_DIR"` (env key) | skit/Skit+Run.swift:99 | -| `"skit: "` (stderr prefix, 5+ sites) | skit/Skit.Run+Render.swift:116,126,147,170; skit/Skit+Run.swift:153; also Execution/Runner.swift:197, ToolchainCheckResult.swift:57,64 | - -### Execution paths / env / formats -| String | Locations | -|---|---| -| `"output.swift"` (3×) | Execution/OutputCache.swift:116,123,139 | -| `"swift"` (file extension filter) | Execution/FileManager+Execution.swift:84 | -| `"SyntaxKit"` (dylib product, 2×) | Execution/FileManager+Execution.swift:39,46 | -| `lib\(self).so` / `lib\(self).dylib` | Execution/String+DylibFilename.swift:36,38 | -| `"XDG_CACHE_HOME"`, `"syntaxkit"` cache dir | Execution/ProcessInfo+SyntaxKitCacheRoot.swift:37,38; OutputCache.swift:53 | -| `"SKIT_"` / `"SYNTAXKIT_"` env prefixes | Execution/OutputCache.swift:105 | -| `"Library/Caches/com.brightdigit.SyntaxKit"`, `".cache/syntaxkit"`, `"outputs"` | Execution/OutputCache.swift:51,53,74 | -| `"lib"`, `"lib/skit"` | Execution/Bundle+ResolveLibPath.swift:55,61 | -| `"%016x"` (hash format), `"swift-version.txt"` | Execution/ContentHasher.swift:57; ToolchainCheckResult.swift:34 | - -### DocumentationHarness markers & fences -| String | Locations | -|---|---| -| ``, ``, ``, `` | CodeBlockExtraction.swift:152-155 | -| ` ```swift ` (open), ` ``` ` (close) | CodeBlockExtraction.swift:94,96 | -| `"md"` (default doc extension) | Validator.swift:38 | -| `"import SyntaxKit"` (2×), `"import Foundation"`, `"Package.swift"`, `"@main"` | CodeSyntaxValidator.swift:44,49,62,63 | -| skip dirs: `".build"`, `"node_modules"`, `".git"`, `".svn"`, `"DerivedData"`, `"build"`, `".swiftpm"` | FileManager+Documentation.swift:38-46 | - -### Comment trivia & misc tokens -| String | Locations | -|---|---| -| `"// "`, `"/// "`, `"///"` | Core/Line+Trivia.swift:39,43,45 | -| `"_"` (wildcard / unnamed label, 8+ sites) | Parameters/Parameter.swift:50,77,110; Expressions/Literal.swift:139; Literal+ExprCodeBlock.swift:72; Collections/TupleLiteralArray.swift:86; CodeBlocks/CodeBlockItemSyntax.Item.swift:53 | -| `"."` (member separator) | ErrorHandling/Catch.swift:48; Utilities/EnumCase.swift:46; TokenVisitor/TreeNodeProtocol+Extensions.swift:189; DocumentationHarness (extension join) | -| `"self"` (capture fallback) | Expressions/CaptureInfo.swift:56,67 | -| `"Syntax"` (suffix stripped) | TokenVisitor/Syntax.swift:54 | -| TriviaPiece chars: space, `\t`, `\n`, `\`, `#` | TokenVisitor/TriviaPiece.swift:57-69 | - ---- - -## Tier 3 — Deprecation messages (dedup only, not reuse) - -Single-purpose but **duplicated verbatim** — collapsing each to one local constant removes the copies. - -| Message | Copies | Location | -|---|---|---| -| `"Use Infix(target, \"+=\", value) instead."` | 6 | Expressions/PlusAssign.swift:71,80,89,97,105,113 | -| `"Use parse(code:) which returns [TreeNode] directly instead of JSON"` | 3 | SyntaxParser/SyntaxResponse.swift:44,51,59 | -| `"Use separate lhs and rhs parameters for compile-time safety"` | 1 | Expressions/Infix.swift:128 | -| `"Use ParameterExp(name:value:) ..."` / `"Use ParameterExp(unlabeled:) ..."` | 1 ea. | Parameters/ParameterExp.swift:82,98 | -| `"Use Parameter(unlabeled:type:) ..."` | 1 | Parameters/Parameter.swift:117 | -| `"Use While(kind:condition:) ..."` / `"Use While(VariableExp(condition)...)"` | 1 ea. | ControlFlow/While.swift:130,151 | - -Plus runtime error messages worth extracting for testability: `Infix` operand errors (Infix.swift:58,60), `Parenthesized` arity (Utilities/Parenthesized.swift:58), Execution path/IO errors (Runner.swift, Bundle+ResolveLibPath.swift:47, FileManager+Execution.swift:68, RunInput.swift:53,57). - ---- - -## Already extracted (no action needed) - -- `SyntaxParser/SyntaxParser.swift` — `fold`, `showMissing` -- `TokenVisitor/StructureProperty.swift` — `element`, `count`, `nilValue` -- `TokenVisitor/String.swift` — `String.empty`, `String.defaultFileName` -- `skit/Subprocess.Configuration+Swift.swift` — `stdoutLimitBytes`, `stderrLimitBytes` - ---- - -## Notable non-findings - -- **Token-kind strings are never hardcoded.** Expected literals like `"keyword"`, `"identifier"`, `"invisible"` come from interpolating `token.tokenKind` — they appear only in doc-comments. No work needed. -- **`DocumentationHarness/PackageValidator.swift` and `CompilationResult.swift`** literals sit in `@available(*, unavailable)` dead code — low priority. - ---- - -## Suggested target structure - -A small set of caseless enums plus one factory helper: - -- **Package-wide** (in `Core/` or `Utilities/`): `TypeNames`, `Operators`, `Keywords`/`Tokens`, `Separators`. -- **`ExprSyntax.placeholderReference`** factory for the Tier 1.1 empty-identifier fallback. -- **Module-local** static constants for Execution paths/env, skit CLI args, and DocumentationHarness markers. - -**Recommended order:** Tier 1 first (biggest payoff, touches the most files), then Tier 2 by module, then Tier 3 dedup. diff --git a/Package.swift b/Package.swift index 5fcc9db..2020939 100644 --- a/Package.swift +++ b/Package.swift @@ -112,7 +112,8 @@ let package = Package( .package(url: "https://github.com/swiftlang/swift-syntax.git", from: "601.0.1"), .package(url: "https://github.com/swiftlang/swift-docc-plugin", from: "1.4.0"), .package(url: "https://github.com/apple/swift-argument-parser.git", from: "1.3.0"), - .package(url: "https://github.com/swiftlang/swift-subprocess.git", from: "0.4.0") + .package(url: "https://github.com/swiftlang/swift-subprocess.git", from: "0.4.0"), + .package(url: "https://github.com/apple/swift-system.git", from: "1.0.0") ], targets: [ .target( @@ -165,6 +166,11 @@ let package = Package( name: "Subprocess", package: "swift-subprocess", condition: .when(platforms: [.macOS, .linux, .windows]) + ), + .product( + name: "SystemPackage", + package: "swift-system", + condition: .when(platforms: [.linux, .windows]) ) ], swiftSettings: swiftSettings diff --git a/Scripts/build-skit-debug.sh b/Scripts/build-skit-debug.sh deleted file mode 100755 index 6e93156..0000000 --- a/Scripts/build-skit-debug.sh +++ /dev/null @@ -1,81 +0,0 @@ -#!/usr/bin/env bash -# -# Build a self-contained skit DEBUG bundle for fast local iteration. -# -# Identical layout to Scripts/build-skit-release.sh but skips release-mode -# optimization (5-15 minute SwiftSyntax compile → ~10 seconds). Use this when -# you want to exercise the end-to-end DSL→Swift transformation locally; use -# the release script when staging an actual release bundle. -# -# Output: .build/skit-debug/{skit, lib/} - -set -euo pipefail - -if [[ "$(uname -s)" != "Darwin" ]]; then - echo "macOS-only. Linux uses a parallel flow." >&2 - exit 1 -fi - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -OUTPUT_DIR="$REPO_ROOT/.build/skit-debug" -PACKAGE_FILE="$REPO_ROOT/Package.swift" -PACKAGE_BACKUP="$(mktemp)" - -cleanup() { - if [[ -s "$PACKAGE_BACKUP" ]]; then - cp "$PACKAGE_BACKUP" "$PACKAGE_FILE" - fi - rm -f "$PACKAGE_BACKUP" -} -trap cleanup EXIT - -cp "$PACKAGE_FILE" "$PACKAGE_BACKUP" - -echo "==> Flipping SyntaxKit library to type: .dynamic (temporary)" -python3 - "$PACKAGE_FILE" <<'PY' -import sys, pathlib -p = pathlib.Path(sys.argv[1]) -src = p.read_text() -new_block = ' .library(\n name: "SyntaxKit",\n type: .dynamic,\n targets: ["SyntaxKit"]\n ),' -if new_block in src: - print("Package.swift already has type: .dynamic — leaving as-is.") - sys.exit(0) -old_block = ' .library(\n name: "SyntaxKit",\n targets: ["SyntaxKit"]\n ),' -if old_block not in src: - sys.exit("Package.swift: expected SyntaxKit library product block not found") -p.write_text(src.replace(old_block, new_block, 1)) -PY - -cd "$REPO_ROOT" - -echo "==> swift build --product skit" -swift build --product skit - -echo "==> swift build --product SyntaxKit" -swift build --product SyntaxKit - -BUILD_DIR="$(ls -d .build/*-apple-macosx*/debug 2>/dev/null | head -1)" -if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate debug build dir under .build//debug" >&2 - exit 1 -fi - -echo "==> Staging $OUTPUT_DIR" -rm -rf "$OUTPUT_DIR" -mkdir -p "$OUTPUT_DIR/lib" - -cp "$BUILD_DIR/skit" "$OUTPUT_DIR/skit" -cp "$BUILD_DIR/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/" -cp -r "$BUILD_DIR/Modules/." "$OUTPUT_DIR/lib/" -cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ - "$OUTPUT_DIR/lib/_SwiftSyntaxCShims-include" - -install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true - -swift --version > "$OUTPUT_DIR/lib/swift-version.txt" - -echo -echo "==> Debug bundle ready at $OUTPUT_DIR" -echo "==> Try it:" -echo " $OUTPUT_DIR/skit Examples/Completed/card_game/dsl.swift --no-cache" diff --git a/Scripts/build-skit-release.sh b/Scripts/build-skit.sh similarity index 54% rename from Scripts/build-skit-release.sh rename to Scripts/build-skit.sh index 66de1a5..9860ca6 100755 --- a/Scripts/build-skit-release.sh +++ b/Scripts/build-skit.sh @@ -1,18 +1,23 @@ #!/usr/bin/env bash # -# Build a self-contained skit release bundle. +# Build a self-contained skit bundle. # -# Output: .build/skit-release/ +# ./Scripts/build-skit.sh # release bundle → .build/skit-release/ +# ./Scripts/build-skit.sh --debug # debug bundle → .build/skit-debug/ +# +# Output: .build/skit-${CONFIG}/ # skit ← the CLI binary # lib/ -# libSyntaxKit.dylib ← release + strip -x +# libSyntaxKit.dylib ← release: stripped; debug: as-built # *.swiftmodule ← SyntaxKit + transitively re-exported modules # _SwiftSyntaxCShims-include/ ← C-shims headers (module map + .h files) # swift-version.txt ← toolchain stamp for startup check # -# Once produced, the bundle is portable: copy the whole .build/skit-release/ -# directory anywhere, and `./skit-release/skit ` Just Works — no -# flags, no env vars, no SyntaxKit checkout required. +# Once produced, the bundle is portable: copy the whole directory anywhere, +# and `./skit-${CONFIG}/skit ` Just Works — no flags, no env vars, no +# SyntaxKit checkout required. The release variant runs a 5–15 minute +# optimized SwiftSyntax compile; the debug variant skips that and finishes +# in ~10 seconds (use it for local iteration). set -euo pipefail @@ -22,9 +27,42 @@ if [[ "$(uname -s)" != "Darwin" ]]; then exit 1 fi +CONFIG="release" +SWIFT_CONFIG_FLAGS=() +STRIP_DYLIB=1 + +while [[ $# -gt 0 ]]; do + case "$1" in + --debug) + CONFIG="debug" + SWIFT_CONFIG_FLAGS=() + STRIP_DYLIB=0 + ;; + --release) + CONFIG="release" + SWIFT_CONFIG_FLAGS=(-c release) + STRIP_DYLIB=1 + ;; + -h | --help) + sed -n '2,18p' "$0" + exit 0 + ;; + *) + echo "unknown argument: $1" >&2 + exit 2 + ;; + esac + shift +done + +# Default config is release; SWIFT_CONFIG_FLAGS was left empty so set it here. +if [[ "$CONFIG" == "release" && ${#SWIFT_CONFIG_FLAGS[@]} -eq 0 ]]; then + SWIFT_CONFIG_FLAGS=(-c release) +fi + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -OUTPUT_DIR="$REPO_ROOT/.build/skit-release" +OUTPUT_DIR="$REPO_ROOT/.build/skit-${CONFIG}" cd "$REPO_ROOT" @@ -34,18 +72,18 @@ cd "$REPO_ROOT" echo "==> Building with SYNTAXKIT_DYNAMIC_LIB=1 (dynamic libSyntaxKit)" export SYNTAXKIT_DYNAMIC_LIB=1 -echo "==> swift build -c release --product skit" -swift build -c release --product skit +echo "==> swift build ${SWIFT_CONFIG_FLAGS[*]} --product skit" +swift build "${SWIFT_CONFIG_FLAGS[@]}" --product skit # `skit` doesn't depend on SyntaxKit (it spawns swift on user input that # imports SyntaxKit at runtime). Build the library product explicitly so the # .dynamic flip above produces libSyntaxKit.dylib + swiftmodule. -echo "==> swift build -c release --product SyntaxKit" -swift build -c release --product SyntaxKit +echo "==> swift build ${SWIFT_CONFIG_FLAGS[*]} --product SyntaxKit" +swift build "${SWIFT_CONFIG_FLAGS[@]}" --product SyntaxKit -BUILD_DIR="$(ls -d .build/*-apple-macosx*/release 2>/dev/null | head -1)" +BUILD_DIR="$(ls -d .build/*-apple-macosx*/"${CONFIG}" 2>/dev/null | head -1)" if [[ -z "$BUILD_DIR" ]]; then - echo "Could not locate release build dir under .build//release" >&2 + echo "Could not locate ${CONFIG} build dir under .build//${CONFIG}" >&2 exit 1 fi @@ -55,12 +93,14 @@ mkdir -p "$OUTPUT_DIR/lib" cp "$BUILD_DIR/skit" "$OUTPUT_DIR/skit" cp "$BUILD_DIR/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/" -strip -x "$OUTPUT_DIR/lib/libSyntaxKit.dylib" +if [[ "$STRIP_DYLIB" == "1" ]]; then + strip -x "$OUTPUT_DIR/lib/libSyntaxKit.dylib" +fi cp -r "$BUILD_DIR/Modules/." "$OUTPUT_DIR/lib/" cp -r "$REPO_ROOT/.build/checkouts/swift-syntax/Sources/_SwiftSyntaxCShims/include" \ "$OUTPUT_DIR/lib/_SwiftSyntaxCShims-include" -# Ensure the dylib's install_name uses @rpath so it's portable. +# Ensure the dylib's install_name uses @rpath so the bundle is portable. install_name_tool -id "@rpath/libSyntaxKit.dylib" "$OUTPUT_DIR/lib/libSyntaxKit.dylib" 2>/dev/null || true # Stamp the bundle with the build toolchain. `skit` compares this against the @@ -74,7 +114,7 @@ DYLIB_SIZE=$(ls -lh "$OUTPUT_DIR/lib/libSyntaxKit.dylib" | awk '{print $5}') TOTAL_SIZE=$(du -sh "$OUTPUT_DIR" | awk '{print $1}') echo -echo "==> Release bundle ready:" +echo "==> ${CONFIG} bundle ready:" echo " Binary: $BINARY_SIZE" echo " Dylib: $DYLIB_SIZE" echo " Total: $TOTAL_SIZE" diff --git a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift index e48be61..9e7987b 100644 --- a/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift +++ b/Sources/SyntaxKit/Execution/Bundle+ResolveLibPath.swift @@ -44,9 +44,10 @@ extension Bundle { /// from `executableURL`: /// - `/lib` (adjacent layout) /// - `/../lib/skit` (Homebrew layout) - public func resolveLibPath(candidates: [String?]) throws -> String { - let fileManager = FileManager.default - + public func resolveLibPath( + candidates: [String?], + fileManager: FileManager = .default + ) throws -> String { for candidate in candidates { guard let candidate else { continue } guard fileManager.isLibDir(candidate) else { @@ -76,7 +77,7 @@ extension Bundle { 1. explicit candidates (none provided or all empty) 2. /lib/ (not found) 3. /../lib/skit/ (not found) - Run Scripts/build-skit-release.sh to produce a self-contained + Run Scripts/build-skit.sh to produce a self-contained release bundle under .build/skit-release/. """ ) diff --git a/Sources/SyntaxKit/Execution/ContentHasher.swift b/Sources/SyntaxKit/Execution/ContentHasher.swift index 4bef0bb..7411a73 100644 --- a/Sources/SyntaxKit/Execution/ContentHasher.swift +++ b/Sources/SyntaxKit/Execution/ContentHasher.swift @@ -51,6 +51,8 @@ public struct ContentHasher: ContentHashing { /// Creates a hasher seeded with the FNV-1a offset basis. public init() {} + /// Mixes `data` into the running hash state via FNV-1a (XOR each byte into + /// the state, then multiply by the FNV prime). public mutating func update(data: Data) { for byte in data { state ^= UInt64(byte) diff --git a/Sources/SyntaxKit/Execution/DirectoryRender.swift b/Sources/SyntaxKit/Execution/DirectoryRender.swift deleted file mode 100644 index 0f5f302..0000000 --- a/Sources/SyntaxKit/Execution/DirectoryRender.swift +++ /dev/null @@ -1,82 +0,0 @@ -// -// DirectoryRender.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -public import Foundation - -/// Result of `Runner.renderDirectory`. Successful rendered outputs are -/// written into the mirrored output tree as the batch progresses; this -/// value is the post-batch summary the caller inspects to decide -/// presentation (logging, exit code, etc.). Per-input failures are -/// captured here, not thrown, so a single bad input doesn't tear the -/// batch down. -public struct DirectoryRender: Sendable { - /// Per-input result. `stderr` carries the (possibly path-rewritten) - /// diagnostics from the spawned `swift`; it may be present whether or not - /// the input succeeded (e.g. a successful render that emitted warnings). - /// `result` is `nil` when the rendered output was written to its mirrored - /// destination, and carries the `RunError` when the input could not be - /// rendered (`.renderFailed`) or its output could not be written - /// (`.unexpected`). - public struct FileOutcome: Sendable { - /// The input file this outcome describes. - public let input: URL - /// The (possibly path-rewritten) `swift` diagnostics for this input. - public let stderr: String - /// `nil` when the output was written; the `RunError` otherwise. - public let result: RunError? - - /// Creates an outcome for one rendered (or failed) input. - public init( - input: URL, - stderr: String, - result: RunError? - ) { - self.input = input - self.stderr = stderr - self.result = result - } - } - - /// Per-input outcomes, in the order the batch produced them. - public let outcomes: [FileOutcome] - - /// Creates a batch summary from the collected per-input outcomes. - public init(outcomes: [FileOutcome]) { - self.outcomes = outcomes - } - - /// Number of inputs whose `result` is `.failure` — the signal the caller - /// uses to map a partially-failed batch to a non-zero exit (or whatever - /// failure semantics fit the host). - public var failureCount: Int { - outcomes.reduce(into: 0) { count, outcome in - if outcome.result != nil { count += 1 } - } - } -} diff --git a/Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift b/Sources/SyntaxKit/Execution/EnvironmentProvider+SyntaxKitCacheRoot.swift similarity index 71% rename from Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift rename to Sources/SyntaxKit/Execution/EnvironmentProvider+SyntaxKitCacheRoot.swift index 75d93f1..674357c 100644 --- a/Sources/SyntaxKit/Execution/ProcessInfo+SyntaxKitCacheRoot.swift +++ b/Sources/SyntaxKit/Execution/EnvironmentProvider+SyntaxKitCacheRoot.swift @@ -1,5 +1,5 @@ // -// ProcessInfo+SyntaxKitCacheRoot.swift +// EnvironmentProvider+SyntaxKitCacheRoot.swift // SyntaxKit // // Created by Leo Dion. @@ -27,20 +27,26 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +internal import Foundation -extension ProcessInfo { +/// Constants for `EnvironmentProvider.syntaxKitCacheRoot(default:)`. Nested +/// in a private enum to satisfy the "no globals" rule while letting the +/// protocol extension below reference them. +private enum SyntaxKitCacheRootConstants { /// Environment variable pointing at the XDG cache root, if set. - private static let xdgCacheHomeEnvKey = "XDG_CACHE_HOME" + static let xdgCacheHomeEnvKey = "XDG_CACHE_HOME" /// Leaf directory appended to the XDG cache root for skit's caches. - private static let cacheDirectoryName = "syntaxkit" + static let cacheDirectoryName = "syntaxkit" +} +extension EnvironmentProvider { /// Root for all skit caches: `/syntaxkit` when that env /// var is set and non-empty, otherwise `defaultRoot` (typically the /// platform's home-relative cache dir). internal func syntaxKitCacheRoot(default defaultRoot: URL) -> URL { - if let xdg = environment[Self.xdgCacheHomeEnvKey], !xdg.isEmpty { - return URL(fileURLWithPath: xdg).appendingPathComponent(Self.cacheDirectoryName) + if let xdg = environment[SyntaxKitCacheRootConstants.xdgCacheHomeEnvKey], !xdg.isEmpty { + return URL(fileURLWithPath: xdg) + .appendingPathComponent(SyntaxKitCacheRootConstants.cacheDirectoryName) } return defaultRoot } diff --git a/Sources/SyntaxKit/Execution/EnvironmentProvider.swift b/Sources/SyntaxKit/Execution/EnvironmentProvider.swift new file mode 100644 index 0000000..e0132f5 --- /dev/null +++ b/Sources/SyntaxKit/Execution/EnvironmentProvider.swift @@ -0,0 +1,44 @@ +// +// EnvironmentProvider.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +public import Foundation + +/// The slice of `ProcessInfo` `OutputCache` depends on. Abstracted into a +/// protocol so tests can inject a fixed environment without subclassing +/// `ProcessInfo` — `swift-corelibs-foundation` declares `ProcessInfo` as +/// `final` with `environment` on an extension, so subclass-and-override +/// doesn't compile on Linux/Windows. +public protocol EnvironmentProvider: Sendable { + /// Process environment variables. + var environment: [String: String] { get } + /// Identifier of the current process, used to namespace staging dirs. + var processIdentifier: Int32 { get } +} + +extension ProcessInfo: EnvironmentProvider {} diff --git a/Sources/SyntaxKit/Execution/FileEnumerationError.swift b/Sources/SyntaxKit/Execution/FileEnumerationError.swift new file mode 100644 index 0000000..2e68f19 --- /dev/null +++ b/Sources/SyntaxKit/Execution/FileEnumerationError.swift @@ -0,0 +1,39 @@ +// +// FileEnumerationError.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +/// A recursive regular-file enumeration failure, decoupled from any domain so +/// the caller maps it onto its own error type. +internal enum FileEnumerationError: Error { + /// The directory couldn't be enumerated at all. + case notEnumerable(URL) + /// A file's resource values couldn't be read mid-walk. + case resourceValuesUnavailable(URL, any Error) +} diff --git a/Sources/SyntaxKit/Execution/FileFingerprint.swift b/Sources/SyntaxKit/Execution/FileFingerprint.swift new file mode 100644 index 0000000..73ed356 --- /dev/null +++ b/Sources/SyntaxKit/Execution/FileFingerprint.swift @@ -0,0 +1,39 @@ +// +// FileFingerprint.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +/// Size + modification date of a file: the change-detection inputs behind a +/// content fingerprint/stamp. +internal struct FileFingerprint: Equatable { + /// File size in bytes (0 when the attribute is absent). + internal let size: Int + /// Last-modification date (Unix epoch when the attribute is absent). + internal let modificationDate: Date +} diff --git a/Sources/SyntaxKit/Execution/FileManager+Execution.swift b/Sources/SyntaxKit/Execution/FileManager+Execution.swift index 37e43a8..de11ca5 100644 --- a/Sources/SyntaxKit/Execution/FileManager+Execution.swift +++ b/Sources/SyntaxKit/Execution/FileManager+Execution.swift @@ -39,7 +39,9 @@ extension FileManager { /// True if `path` is a directory containing `libSyntaxKit.{dylib,so}`. internal func isLibDir(_ path: String) -> Bool { - guard pathKind(atPath: path) == .directory else { return false } + guard pathKind(atPath: path) == .directory else { + return false + } return pathKind(atPath: "\(path)/\(Self.syntaxKitProductName.dylibFilename)") != .missing } @@ -95,8 +97,8 @@ extension FileManager { for result: RenderTaskResult, inputBase: URL, outputBase: URL, - toolchain: Runner.ToolchainVerification - ) -> DirectoryRender.FileOutcome { + toolchain: ToolchainVerification + ) -> FileOutcome { // Mirror the input's location under the output base (generic path math). let destination = result.input.rerooted(from: inputBase, onto: outputBase) @@ -114,7 +116,7 @@ extension FileManager { failure = error } - return DirectoryRender.FileOutcome( + return FileOutcome( input: result.input, stderr: stderr, result: failure diff --git a/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift b/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift index 086e90c..9663732 100644 --- a/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift +++ b/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift @@ -29,36 +29,6 @@ import Foundation -/// What exists at a filesystem path — the result of a single existence check, -/// replacing the `fileExists(atPath:isDirectory:)` + `ObjCBool` idiom with a -/// value callers can `switch` over. -internal enum PathKind: Equatable { - /// Nothing exists at the path. - case missing - /// A regular file (or any non-directory) exists at the path. - case file - /// A directory exists at the path. - case directory -} - -/// Size + modification date of a file: the change-detection inputs behind a -/// content fingerprint/stamp. -internal struct FileFingerprint: Equatable { - /// File size in bytes (0 when the attribute is absent). - internal let size: Int - /// Last-modification date (Unix epoch when the attribute is absent). - internal let modificationDate: Date -} - -/// A recursive regular-file enumeration failure, decoupled from any domain so -/// the caller maps it onto its own error type. -internal enum FileEnumerationError: Error { - /// The directory couldn't be enumerated at all. - case notEnumerable(URL) - /// A file's resource values couldn't be read mid-walk. - case resourceValuesUnavailable(URL, any Error) -} - extension FileManager { /// Classifies what exists at `path` in a single stat — `.missing`, `.file`, /// or `.directory`. diff --git a/Sources/SyntaxKit/Execution/FileOutcome.swift b/Sources/SyntaxKit/Execution/FileOutcome.swift new file mode 100644 index 0000000..c4357e6 --- /dev/null +++ b/Sources/SyntaxKit/Execution/FileOutcome.swift @@ -0,0 +1,72 @@ +// +// FileOutcome.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +public import Foundation + +/// Per-input result of a directory-mode render. `stderr` carries the (possibly +/// path-rewritten) diagnostics from the spawned `swift`; it may be present +/// whether or not the input succeeded (e.g. a successful render that emitted +/// warnings). `result` is `nil` when the rendered output was written to its +/// mirrored destination, and carries the `RunError` when the input could not +/// be rendered (`.renderFailed`) or its output could not be written +/// (`.unexpected`). +/// +/// `Runner.renderDirectory` returns `[FileOutcome]`; per-input failures are +/// captured here, not thrown, so a single bad input doesn't tear the batch +/// down. +public struct FileOutcome: Sendable { + /// The input file this outcome describes. + public let input: URL + /// The (possibly path-rewritten) `swift` diagnostics for this input. + public let stderr: String + /// `nil` when the output was written; the `RunError` otherwise. + public let result: RunError? + + /// Creates an outcome for one rendered (or failed) input. + public init( + input: URL, + stderr: String, + result: RunError? + ) { + self.input = input + self.stderr = stderr + self.result = result + } +} + +extension Array where Element == FileOutcome { + /// Number of inputs whose `result` is non-nil — the signal the caller uses + /// to map a partially-failed batch to a non-zero exit (or whatever failure + /// semantics fit the host). + public var failureCount: Int { + reduce(into: 0) { count, outcome in + if outcome.result != nil { count += 1 } + } + } +} diff --git a/Sources/SyntaxKit/Execution/OutputCache.swift b/Sources/SyntaxKit/Execution/OutputCache.swift index ff69cfd..575dc2e 100644 --- a/Sources/SyntaxKit/Execution/OutputCache.swift +++ b/Sources/SyntaxKit/Execution/OutputCache.swift @@ -32,10 +32,10 @@ public import Foundation /// On-disk cache of rendered skit output, content-keyed so a re-run on /// unchanged input skips the `swift` spawn entirely. /// -/// `Sendable`: the stored state is a `@Sendable` `FileManager` factory plus a -/// `ProcessInfo` and value types, so the single instance can be shared safely -/// across the concurrent `runOne` tasks in directory mode. The default -/// singletons used in production (and the typical test doubles) are +/// `Sendable`: the stored state is a `@Sendable` `FileManager` factory plus +/// an `EnvironmentProvider` and value types, so the single instance can be +/// shared safely across the concurrent `runOne` tasks in directory mode. The +/// default singletons used in production (and the typical test doubles) are /// thread-safe for the operations we invoke. public struct OutputCache: Sendable { /// Bumped when the cache layout changes in a way that requires invalidation. @@ -75,7 +75,7 @@ public struct OutputCache: Sendable { /// directories are derived from it on demand. private let root: URL private let fileManager: @Sendable () -> FileManager - private let processInfo: ProcessInfo + private let environmentProvider: any EnvironmentProvider /// Factory for the per-key hasher. Pluggable so the cache's hashing algorithm /// can be swapped; defaults to `ContentHasher` (FNV-1a). A factory rather than /// a stored instance because `key(forInput:libPath:)` needs a fresh, empty @@ -88,20 +88,20 @@ public struct OutputCache: Sendable { private let swiftVersion: String? /// Creates a cache rooted under the SyntaxKit cache directory, keyed in part - /// by the captured `swiftVersion`. `fileManager`/`processInfo` are injectable - /// for testing; `makeHasher` is injectable to plug in a different + /// by the captured `swiftVersion`. `fileManager`/`environmentProvider` are + /// injectable for testing; `makeHasher` is injectable to plug in a different /// `ContentHashing` algorithm (defaults to `ContentHasher`). public init( swiftVersion: String?, fileManager: @autoclosure @escaping @Sendable () -> FileManager = .default, - processInfo: ProcessInfo = .processInfo, + environmentProvider: any EnvironmentProvider = ProcessInfo.processInfo, makeHasher: @escaping @Sendable () -> any ContentHashing = { ContentHasher() } ) { - self.root = processInfo.syntaxKitCacheRoot(default: Self.defaultCacheRoot) + self.root = environmentProvider.syntaxKitCacheRoot(default: Self.defaultCacheRoot) .appendingPathComponent(Self.outputsDirectoryName) self.swiftVersion = swiftVersion self.fileManager = fileManager - self.processInfo = processInfo + self.environmentProvider = environmentProvider self.makeHasher = makeHasher } @@ -129,7 +129,7 @@ public struct OutputCache: Sendable { // SKIT_*/SYNTAXKIT_* env vars. Sorted so the cache key is stable, and // NUL-terminated so `"AB=" + "C"` doesn't collide with `"A=" + "BC"`. - let env = processInfo.environment + let env = environmentProvider.environment .filter { $0.key.hasPrefix(Self.skitEnvPrefix) || $0.key.hasPrefix(Self.syntaxKitEnvPrefix) } @@ -163,7 +163,7 @@ public struct OutputCache: Sendable { // into place as a single atomic step. let staging = cacheRoot.deletingLastPathComponent() .appendingPathComponent( - "\(Self.stagingDirectoryPrefix).\(processInfo.processIdentifier).\(UUID().uuidString)" + "\(Self.stagingDirectoryPrefix).\(environmentProvider.processIdentifier).\(UUID().uuidString)" ) try fileManager().createDirectory(at: staging, withIntermediateDirectories: true) try data.write(to: staging.appendingPathComponent(Self.outputFileName)) diff --git a/Sources/SyntaxKit/Execution/PathKind.swift b/Sources/SyntaxKit/Execution/PathKind.swift new file mode 100644 index 0000000..75fb74c --- /dev/null +++ b/Sources/SyntaxKit/Execution/PathKind.swift @@ -0,0 +1,40 @@ +// +// PathKind.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +/// What exists at a filesystem path — the result of a single existence check, +/// replacing the `fileExists(atPath:isDirectory:)` + `ObjCBool` idiom with a +/// value callers can `switch` over. +internal enum PathKind: Equatable { + /// Nothing exists at the path. + case missing + /// A regular file (or any non-directory) exists at the path. + case file + /// A directory exists at the path. + case directory +} diff --git a/Sources/SyntaxKit/Execution/RenderTaskResult.swift b/Sources/SyntaxKit/Execution/RenderTaskResult.swift new file mode 100644 index 0000000..316d46f --- /dev/null +++ b/Sources/SyntaxKit/Execution/RenderTaskResult.swift @@ -0,0 +1,73 @@ +// +// RenderTaskResult.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +/// Payload the per-input render `TaskGroup` yields back to `renderDirectory`. +/// Failures are captured (not thrown) so a single bad input doesn't tear down +/// the group; `processFile`'s heterogeneous Foundation/Subprocess throws are +/// normalized into `RunError` (typically `.unexpected`) by `runOne`. +/// `internal` so `FileManager.writeOutput` (in `FileManager+Execution.swift`) +/// can consume it. +internal struct RenderTaskResult: Sendable { + /// Bundles a successful render's bytes with the destination they should be + /// written to. Nested because it's only ever produced by `writeOutput` to + /// hand off to its caller-supplied writer closure. + internal struct OutputDestination: Sendable { + internal let output: Data + internal let destination: URL + } + + internal let input: URL + internal let result: Result + + internal func writeOutput( + to destination: URL, + toolchain: ToolchainVerification, + using writeOutputDestination: (OutputDestination) throws -> Void + ) throws(RunError) { + let result = try self.result.get() + + guard result.exitCode == 0 else { + throw .renderFailed( + exitCode: result.exitCode, + stderr: result.stderr, + toolchain: toolchain + ) + } + + let outputDestination = OutputDestination(output: result.stdout, destination: destination) + + do { + try writeOutputDestination(outputDestination) + } catch { + throw .unexpected(error) + } + } +} diff --git a/Sources/SyntaxKit/Execution/RunError.swift b/Sources/SyntaxKit/Execution/RunError.swift index 5c1cc57..47110f0 100644 --- a/Sources/SyntaxKit/Execution/RunError.swift +++ b/Sources/SyntaxKit/Execution/RunError.swift @@ -41,7 +41,7 @@ public enum RunError: Error { /// carries the session's `toolchainVerification`: when it isn't `.verified`, /// the failure may stem from a Swift-toolchain mismatch the check couldn't /// rule out, which the caller can hint at. - case renderFailed(exitCode: Int32, stderr: String, toolchain: Runner.ToolchainVerification) + case renderFailed(exitCode: Int32, stderr: String, toolchain: ToolchainVerification) /// A wrapped Foundation/Subprocess failure (file read/write, spawn error) /// that has no dedicated mapping. case unexpected(any Error) diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index 30df635..9b4047e 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -33,17 +33,16 @@ extension Runner { /// Walks `inputDir` for `.swift` inputs, processes them concurrently (up to /// the active core count), and mirrors successfully-rendered outputs into /// `outputDir`. A failure on one input does not abort the batch — successful - /// peers are still written. Returns a `DirectoryRender` with per-input - /// outcomes; the caller inspects `failureCount` (and per-outcome stderr) to - /// decide presentation. + /// peers are still written. Returns the per-input outcomes; the caller + /// inspects `failureCount` (and per-outcome stderr) to decide presentation. /// /// Throws `RunError.unexpected` only for bulk failures the SDK can't /// recover from (e.g. the input directory can't be enumerated). An empty - /// input set is *not* an error — the result simply has no outcomes. + /// input set is *not* an error — the returned array is simply empty. public func renderDirectory( inputDir: String, outputDir: String - ) async throws(RunError) -> DirectoryRender { + ) async throws(RunError) -> [FileOutcome] { let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL @@ -53,28 +52,26 @@ extension Runner { // channel the caller already presents. let inputs: [URL] do { - inputs = try FileManager.default.collectInputs(at: inputURL) + inputs = try fileManager().collectInputs(at: inputURL) } catch { throw RunError.unexpected(error) } guard !inputs.isEmpty else { - return DirectoryRender(outcomes: []) + return [] } // Phase 2: render every input with bounded concurrency. let renderResults = await processInputs(inputs) // Phase 3: write successes and capture a per-input outcome for each. - let outcomes = renderResults.map { - FileManager.default.writeOutput( + return renderResults.map { + fileManager().writeOutput( for: $0, inputBase: inputURL, outputBase: outputURL, toolchain: toolchainVerification ) } - - return DirectoryRender(outcomes: outcomes) } /// Renders every input through `runOne` with bounded concurrency. The cap is @@ -122,47 +119,3 @@ extension Runner { } } } - -/// Payload the per-input render `TaskGroup` yields back to `renderDirectory`. -/// Failures are captured (not thrown) so a single bad input doesn't tear down -/// the group; `processFile`'s heterogeneous Foundation/Subprocess throws are -/// normalized into `RunError` (typically `.unexpected`) by `runOne`. -/// `internal` so `FileManager.writeOutput` (in `FileManager+Execution.swift`) -/// can consume it. -internal struct RenderTaskResult: Sendable { - internal let input: URL - internal let result: Result - - struct OutputDestination: Sendable { - let output: Data - let destination: URL - } - - internal func writeOutput( - to destination: URL, toolchain: Runner.ToolchainVerification, - using writeOutputDestination: (OutputDestination) throws -> Void - ) throws(RunError) { - let result = try self.result.get() - - guard result.exitCode == 0 else { - throw - .renderFailed( - exitCode: result.exitCode, - stderr: result.stderr, - toolchain: toolchain - ) - } - - let outputDestination = OutputDestination(output: result.stdout, destination: destination) - - do { - try writeOutputDestination(outputDestination) - } catch { - throw .unexpected(error) - } - // try writeOutputWith(destination) - // return Result { try writeData(processResult.stdout, to: destination) } - // .mapError(RunError.unexpected) - - } -} diff --git a/Sources/SyntaxKit/Execution/Runner+Session.swift b/Sources/SyntaxKit/Execution/Runner+Session.swift index 91abece..c345a67 100644 --- a/Sources/SyntaxKit/Execution/Runner+Session.swift +++ b/Sources/SyntaxKit/Execution/Runner+Session.swift @@ -27,22 +27,9 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +public import Foundation extension Runner { - /// Why a render session couldn't be brought up. Decoupled from any caller: - /// the initializer reports *what* failed; the caller (CLI, build plugin, - /// in-process driver) decides how to present it and which exit code to use. - public enum SetupError: Error { - /// The libSyntaxKit directory couldn't be resolved from the supplied - /// candidates or the bundle-relative fallbacks. Carries the underlying - /// `CLIError` describing the lookup. - case libResolutionFailed(any Error) - /// The bundle's recorded `swift --version` differs from the local one, - /// so spawning `swift` would hit a swiftmodule-version mismatch. - case toolchainMismatch(bundle: String, local: String) - } - /// Brings up a render-ready `Runner`: resolves the libSyntaxKit directory /// from `libCandidates` (falling back to bundle-relative layouts), optionally /// gates on the bundle/local toolchain comparison, and wires up the output @@ -53,7 +40,7 @@ extension Runner { /// (`swift --version` output, or nil if capture failed) and the `run` closure /// that actually spawns `swift` for one `SwiftInvocation`. /// - /// - Throws: `SetupError` for a lookup or toolchain failure, so the caller + /// - Throws: `RunnerSetupError` for a lookup or toolchain failure, so the caller /// owns the presentation and exit mapping. public init( libCandidates: [String?], @@ -61,15 +48,19 @@ extension Runner { enforceToolchainCheck: Bool, useCache: Bool, timeoutSeconds: Int, + fileManager: @autoclosure @escaping @Sendable () -> FileManager = .default, run: @Sendable @escaping (SwiftInvocation) async throws -> SwiftRunOutcome - ) throws(SetupError) { + ) throws(RunnerSetupError) { // 1. Resolve the libSyntaxKit bundle dir. Failure is fatal — there's no // dylib + swiftmodules to link against without it. let libPath: String do { - libPath = try Bundle.main.resolveLibPath(candidates: libCandidates) + libPath = try Bundle.main.resolveLibPath( + candidates: libCandidates, + fileManager: fileManager() + ) } catch { - throw SetupError.libResolutionFailed(error) + throw RunnerSetupError.libResolutionFailed(error) } // 2. Compare the bundle's recorded `swift --version` against the local one. @@ -86,7 +77,7 @@ extension Runner { case .stampMissing: verification = .unverified case .mismatch(let bundle, let local): - throw SetupError.toolchainMismatch(bundle: bundle, local: local) + throw RunnerSetupError.toolchainMismatch(bundle: bundle, local: local) } } else { verification = .notChecked @@ -94,7 +85,10 @@ extension Runner { // 3. Build the output cache (nil when disabled). The captured `swiftVersion` // is bound in so per-input key derivation doesn't re-spawn `swift`. - let cache: OutputCache? = useCache ? OutputCache(swiftVersion: swiftVersion) : nil + let cache: OutputCache? = + useCache + ? OutputCache(swiftVersion: swiftVersion, fileManager: fileManager()) + : nil // 4. Delegate to the designated initializer, binding the spawn closure. self.init( @@ -102,6 +96,7 @@ extension Runner { cache: cache, timeoutSeconds: timeoutSeconds, toolchainVerification: verification, + fileManager: fileManager(), run: run ) } diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index 57ae60f..cb9f039 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +public import Foundation // Render lifecycle (per `Runner` call): // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) @@ -40,30 +40,13 @@ import Foundation /// Renders SyntaxKit DSL inputs into Swift source. `Runner` is the SDK-shaped /// entry point: methods return rendered data (`SingleFileRender`) or -/// structured per-file outcomes (`DirectoryRender`) and throw typed +/// structured per-file outcomes (`[FileOutcome]`) and throw typed /// `RunError`s; callers — CLI, build plugin, in-process driver — decide what /// to do with stdout, stderr, exit codes, and so on. /// /// Constructed once per render session; `Sendable` so a single value can be /// shared across the concurrent per-input tasks in directory mode. public struct Runner: Sendable { - /// Outcome of the bundle/local Swift-toolchain compatibility check performed - /// when the render session was created. Stored on the `Runner` and carried on - /// `RunError.renderFailed` so a caller can hint that an *unverified* toolchain - /// may be the real cause of a build error — swiftmodules aren't reliably - /// compatible across compiler versions. A confirmed *mismatch* never reaches - /// here; it fails session setup via `SetupError.toolchainMismatch`. - public enum ToolchainVerification: Sendable, Equatable { - /// The bundle's recorded `swift --version` matched the local one. - case verified - /// The caller opted out of the check (`enforceToolchainCheck == false`). - case notChecked - /// The check ran but couldn't compare versions — the bundle had no - /// toolchain stamp, or the local `swift --version` couldn't be captured — - /// so compatibility is unknown. - case unverified - } - /// Exit code returned when the spawned `swift` is killed by skit's timeout /// watchdog. Matches POSIX `timeout(1)`. private static let timeoutExitCode: Int32 = 124 @@ -83,6 +66,10 @@ public struct Runner: Sendable { /// Backend that actually spawns `swift` for one `SwiftInvocation`. Injected /// by the caller (skit supplies a Subprocess-based implementation). private let run: @Sendable (SwiftInvocation) async throws -> SwiftRunOutcome + /// FileManager factory used for tmp-dir creation, batch enumeration, and + /// per-input writes. Injectable so tests can swap in a fake; defaults to + /// `.default` in production. + internal let fileManager: @Sendable () -> FileManager /// Whether this session's bundle/local Swift toolchain was confirmed /// compatible. Defaults to `.notChecked` for callers that construct a /// `Runner` directly; the session initializer (`Runner+Session.swift`) sets @@ -96,12 +83,14 @@ public struct Runner: Sendable { cache: OutputCache?, timeoutSeconds: Int, toolchainVerification: ToolchainVerification = .notChecked, + fileManager: @autoclosure @escaping @Sendable () -> FileManager = .default, run: @Sendable @escaping (SwiftInvocation) async throws -> SwiftRunOutcome ) { self.libPath = libPath self.cache = cache self.timeoutSeconds = timeoutSeconds self.toolchainVerification = toolchainVerification + self.fileManager = fileManager self.run = run } @@ -167,10 +156,10 @@ public struct Runner: Sendable { // Spill the wrapped program to a per-invocation temp dir. The dir is // cleaned up unconditionally so a failed spawn doesn't leak files. - let tmpDir = FileManager.default.temporaryDirectory + let tmpDir = fileManager().temporaryDirectory .appendingPathComponent("\(Self.tempDirectoryPrefix)-\(UUID().uuidString)") - try FileManager.default.createDirectory(at: tmpDir, withIntermediateDirectories: true) - defer { try? FileManager.default.removeItem(at: tmpDir) } + try fileManager().createDirectory(at: tmpDir, withIntermediateDirectories: true) + defer { try? fileManager().removeItem(at: tmpDir) } let wrappedURL = tmpDir.appendingPathComponent(Self.wrappedInputFileName) try wrapped.write(to: wrappedURL, atomically: true, encoding: .utf8) diff --git a/Sources/SyntaxKit/Execution/RunnerSetupError.swift b/Sources/SyntaxKit/Execution/RunnerSetupError.swift new file mode 100644 index 0000000..d8c0931 --- /dev/null +++ b/Sources/SyntaxKit/Execution/RunnerSetupError.swift @@ -0,0 +1,42 @@ +// +// RunnerSetupError.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +/// Why a render session couldn't be brought up. Decoupled from any caller: +/// the `Runner` session initializer reports *what* failed; the caller (CLI, +/// build plugin, in-process driver) decides how to present it and which exit +/// code to use. +public enum RunnerSetupError: Error { + /// The libSyntaxKit directory couldn't be resolved from the supplied + /// candidates or the bundle-relative fallbacks. Carries the underlying + /// `CLIError` describing the lookup. + case libResolutionFailed(any Error) + /// The bundle's recorded `swift --version` differs from the local one, + /// so spawning `swift` would hit a swiftmodule-version mismatch. + case toolchainMismatch(bundle: String, local: String) +} diff --git a/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift b/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift index 2842fbf..30e9fdb 100644 --- a/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift +++ b/Sources/SyntaxKit/Execution/ToolchainCheckResult.swift @@ -33,9 +33,6 @@ import Foundation /// local `swift --version`. The swiftmodule format isn't reliably /// forward-compatible across Swift releases, so a mismatch is worth surfacing. public enum ToolchainCheckResult { - /// Filename for the bundle's recorded build-toolchain version. - private static let toolchainStampFilename = "swift-version.txt" - /// Bundle stamp matches the local `swift --version` exactly. case match /// The toolchain check couldn't be performed: either `/swift-version.txt` @@ -45,6 +42,9 @@ public enum ToolchainCheckResult { case stampMissing /// The bundle stamp and the local `swift --version` differ. case mismatch(bundle: String, local: String) + + /// Filename for the bundle's recorded build-toolchain version. + private static let toolchainStampFilename = "swift-version.txt" } extension ToolchainCheckResult { diff --git a/Sources/SyntaxKit/Execution/ToolchainVerification.swift b/Sources/SyntaxKit/Execution/ToolchainVerification.swift new file mode 100644 index 0000000..55df1fe --- /dev/null +++ b/Sources/SyntaxKit/Execution/ToolchainVerification.swift @@ -0,0 +1,45 @@ +// +// ToolchainVerification.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +/// Outcome of the bundle/local Swift-toolchain compatibility check performed +/// when a render session was created. Stored on the `Runner` and carried on +/// `RunError.renderFailed` so a caller can hint that an *unverified* toolchain +/// may be the real cause of a build error — swiftmodules aren't reliably +/// compatible across compiler versions. A confirmed *mismatch* never reaches +/// here; it fails session setup via `RunnerSetupError.toolchainMismatch`. +public enum ToolchainVerification: Sendable, Equatable { + /// The bundle's recorded `swift --version` matched the local one. + case verified + /// The caller opted out of the check (`enforceToolchainCheck == false`). + case notChecked + /// The check ran but couldn't compare versions — the bundle had no + /// toolchain stamp, or the local `swift --version` couldn't be captured — + /// so compatibility is unknown. + case unverified +} diff --git a/Sources/SyntaxKit/Execution/WrappedSource.swift b/Sources/SyntaxKit/Execution/WrappedSource.swift index 7efd0e3..f99dbb2 100644 --- a/Sources/SyntaxKit/Execution/WrappedSource.swift +++ b/Sources/SyntaxKit/Execution/WrappedSource.swift @@ -48,6 +48,36 @@ public struct WrappedSource { /// The 1-based line number the body starts on in the original file. private let firstBodyLine: Int + /// A complete Swift program that imports SyntaxKit, runs the body inside a + /// `Group { … }` builder, and prints the generated code. + public var rendered: String { + // Render the hoisted-imports block. Trailing newline only if non-empty so + // the wrapper doesn't grow an extra blank line in the common no-imports + // case. + let hoistedBlock = hoistedImports.isEmpty ? "" : hoistedImports.joined(separator: "\n") + "\n" + + // #sourceLocation must use a forward-slash path; escape backslashes/quotes + // defensively even though macOS paths shouldn't contain them. + let escapedPath = + originalPath + .replacingOccurrences(of: "\\", with: "\\\\") + .replacingOccurrences(of: "\"", with: "\\\"") + + // Build the final wrapper. Layout: SyntaxKit import → hoisted imports → + // Group { #sourceLocation(...) #sourceLocation() } → print. + return """ + import SyntaxKit + \(hoistedBlock) + let __skit_root = Group { + #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) + \(body) + #sourceLocation() + } + + print(__skit_root.generateCode()) + """ + } + /// Parses `source`, hoisting leading `import` declarations and capturing the /// remaining body along with the line it begins on. Everything before the first /// non-import statement that *is* an import gets hoisted; anything before that @@ -89,34 +119,4 @@ public struct WrappedSource { self.firstBodyLine = 1 } } - - /// A complete Swift program that imports SyntaxKit, runs the body inside a - /// `Group { … }` builder, and prints the generated code. - public var rendered: String { - // Render the hoisted-imports block. Trailing newline only if non-empty so - // the wrapper doesn't grow an extra blank line in the common no-imports - // case. - let hoistedBlock = hoistedImports.isEmpty ? "" : hoistedImports.joined(separator: "\n") + "\n" - - // #sourceLocation must use a forward-slash path; escape backslashes/quotes - // defensively even though macOS paths shouldn't contain them. - let escapedPath = - originalPath - .replacingOccurrences(of: "\\", with: "\\\\") - .replacingOccurrences(of: "\"", with: "\\\"") - - // Build the final wrapper. Layout: SyntaxKit import → hoisted imports → - // Group { #sourceLocation(...) #sourceLocation() } → print. - return """ - import SyntaxKit - \(hoistedBlock) - let __skit_root = Group { - #sourceLocation(file: "\(escapedPath)", line: \(firstBodyLine)) - \(body) - #sourceLocation() - } - - print(__skit_root.generateCode()) - """ - } } diff --git a/Sources/skit/ProcessInfo+SkitLibPath.swift b/Sources/skit/ProcessInfo+SkitLibPath.swift new file mode 100644 index 0000000..c8beb1e --- /dev/null +++ b/Sources/skit/ProcessInfo+SkitLibPath.swift @@ -0,0 +1,41 @@ +// +// ProcessInfo+SkitLibPath.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation + +extension ProcessInfo { + /// Environment variable holding an override for the libSyntaxKit directory. + internal static let skitLibDirEnvironmentKey = "SKIT_LIB_DIR" + + /// The libSyntaxKit directory specified via `SKIT_LIB_DIR`, or `nil` when + /// the env var is unset or empty. + internal var skitLibPath: String? { + environment[Self.skitLibDirEnvironmentKey].flatMap { $0.isEmpty ? nil : $0 } + } +} diff --git a/Sources/skit/README.md b/Sources/skit/README.md index 5e2e066..23d5c51 100644 --- a/Sources/skit/README.md +++ b/Sources/skit/README.md @@ -15,7 +15,7 @@ skit parse < Input.swift # parse Swift source into a JSON syntax tree ```bash # Build a self-contained release bundle (binary + dylib + swiftmodules). -Scripts/build-skit-release.sh +Scripts/build-skit.sh # → .build/skit-release/{skit, lib/} cat > /tmp/Person.swift <<'SWIFT' @@ -63,7 +63,7 @@ Output cache hit is roughly ~0.14s on macOS (no spawn at all); cold miss matches ## Platform notes - **macOS** — primary target. All build/release/test flows in `Scripts/`. -- **Linux** — verified on `swift:6.0-jammy/aarch64`. The Mach-O `install_name` step in `Scripts/build-skit-release.sh` is macOS-specific and skipped on Linux. +- **Linux** — verified on `swift:6.0-jammy/aarch64`. The Mach-O `install_name` step in `Scripts/build-skit.sh` is macOS-specific and skipped on Linux. - **Windows** — not supported. Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited children on `swift:6.0-jammy/aarch64`. `Runner.swift` works around it with `terminationHandler` + `DispatchSemaphore`. @@ -71,4 +71,3 @@ Known Linux gotcha: `Foundation.Process.waitUntilExit()` hangs on already-exited ## Deeper dive - [`Docs/skit.md`](../../Docs/skit.md) — architecture, design decisions, trade-offs. -- [`Docs/skit-internals.md`](../../Docs/skit-internals.md) — per-module reference for Runner, OutputCache, and Toolchain (what each needs, how they call each other). diff --git a/Sources/skit/Skit.Run+Render.swift b/Sources/skit/Render.swift similarity index 70% rename from Sources/skit/Skit.Run+Render.swift rename to Sources/skit/Render.swift index 701bec0..2021076 100644 --- a/Sources/skit/Skit.Run+Render.swift +++ b/Sources/skit/Render.swift @@ -1,5 +1,5 @@ // -// Skit.Run+Render.swift +// Render.swift // SyntaxKit // // Created by Leo Dion. @@ -30,23 +30,28 @@ import Foundation import SyntaxKit -extension Skit.Run { +/// Render-side IO for the `skit run` subcommand: classifies the input, calls +/// the silent `Runner` SDK, and owns all stdout/stderr presentation and file +/// writes. Hosted in an `enum` namespace because none of these helpers use +/// instance state from `Skit.Run` — extracted from `Skit.Run` so the +/// subcommand struct stays focused on argument parsing. +internal enum Render { /// Classifies `input` and dispatches to the matching `Runner` SDK method. /// The SDK layer is silent — this function (and its helpers) own all /// stdout/stderr presentation and file IO, and translate `RunError`/batch - /// failures into the typed `CommandError` that `run()` maps to a process - /// exit. Success-path output is still written here. - internal func render(using runner: Runner, input: String, output: String?) - async throws(CommandError) + /// failures into the typed `RunCommandError` that `Skit.Run.run()` maps to + /// a process exit. Success-path output is still written here. + internal static func render(using runner: Runner, input: String, output: String?) + async throws(RunCommandError) { let resolved: RunInput do { resolved = try RunInput.resolve(input: input, output: output) } catch .invalidInput(let message) { - throw CommandError.usage(message) + throw RunCommandError.usage(message) } catch { // RunInput.resolve only throws .invalidInput; defensive. - throw CommandError.failed + throw RunCommandError.failed } switch resolved { @@ -60,25 +65,25 @@ extension Skit.Run { /// Renders one input via `Runner.renderFile`, prints any compiler stderr, /// and writes the rendered Swift source either to `outputPath` or to /// stdout. The runner itself does none of that — that's the CLI's job. - fileprivate func renderSingle( + private static func renderSingle( runner: Runner, input: String, output: String? - ) async throws(CommandError) { + ) async throws(RunCommandError) { let rendered: SingleFileRender do { rendered = try await runner.renderFile(input: input) } catch { // `error` is typed `RunError` (renderFile is `throws(RunError)`); the // switch is exhaustive, so this single catch is both total (satisfies - // `throws(CommandError)`) and free of an unreachable clause. + // `throws(RunCommandError)`) and free of an unreachable clause. switch error { case .invalidInput(let message): - throw CommandError.usage(message) + throw RunCommandError.usage(message) case .renderFailed(let exitCode, let stderr, let toolchain): - throw CommandError.renderFailed(exitCode: exitCode, stderr: stderr, toolchain: toolchain) + throw RunCommandError.renderFailed(exitCode: exitCode, stderr: stderr, toolchain: toolchain) case .unexpected(let underlying): - throw CommandError.unexpected(underlying) + throw RunCommandError.unexpected(underlying) } } @@ -91,7 +96,7 @@ extension Skit.Run { } catch { // A write failure has no diagnostic of its own; surface the underlying // error (ArgumentParser prints it, exit 1). - throw CommandError.unexpected(error) + throw RunCommandError.unexpected(error) } } else { FileHandle.standardOutput.write(rendered.stdout) @@ -102,34 +107,35 @@ extension Skit.Run { /// diagnostics (fenced when several files emit them), surfaces non-render /// failures, prints a one-line summary, and maps any failures to /// `ExitCode(1)` — Tuist-analog batch semantics. - fileprivate func renderBatch( + private static func renderBatch( runner: Runner, input: String, output: String - ) async throws(CommandError) { - let result: DirectoryRender + ) async throws(RunCommandError) { + let outcomes: [FileOutcome] do { - result = try await runner.renderDirectory(inputDir: input, outputDir: output) + outcomes = try await runner.renderDirectory(inputDir: input, outputDir: output) } catch .invalidInput(let message) { - throw CommandError.usage(message) + throw RunCommandError.usage(message) } catch .unexpected(let underlying) { // `renderDirectory` only wraps directory-walk failures in `.unexpected`; - // `CommandError.directoryWalkFailed` carries the original "failed to + // `RunCommandError.directoryWalkFailed` carries the original "failed to // walk" framing the CLI prints. - throw CommandError.directoryWalkFailed(input: input, underlying: underlying) + throw RunCommandError.directoryWalkFailed(input: input, underlying: underlying) } catch { // renderDirectory does not throw .renderFailed — that's a per-file // outcome. Defensive. - throw CommandError.failed + throw RunCommandError.failed } - if result.outcomes.isEmpty { + if outcomes.isEmpty { FileHandle.standardError.write( - Data("\(Self.messagePrefix)no .swift inputs under \(input)\n".utf8)) + Data("\(Skit.Run.messagePrefix)no .swift inputs under \(input)\n".utf8) + ) return } - for outcome in result.outcomes { + for outcome in outcomes { if !outcome.stderr.isEmpty { FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) FileHandle.standardError.write(Data(outcome.stderr.utf8)) @@ -146,18 +152,18 @@ extension Skit.Run { FileHandle.standardError.write( Data( - ("\(Self.messagePrefix)\(result.outcomes.count - result.failureCount)" - + "/\(result.outcomes.count) succeeded\n").utf8 + ("\(Skit.Run.messagePrefix)\(outcomes.count - outcomes.failureCount)" + + "/\(outcomes.count) succeeded\n").utf8 ) ) - if result.failureCount > 0 { + if outcomes.failureCount > 0 { // Some inputs failed; if the toolchain couldn't be verified, hint once // that a Swift-version mismatch may be behind the build errors above. - if let hint = CommandError.toolchainHint(runner.toolchainVerification) { + if let hint = RunCommandError.toolchainHint(runner.toolchainVerification) { FileHandle.standardError.write(Data(hint.utf8)) } - throw CommandError.failed + throw RunCommandError.failed } } } diff --git a/Sources/skit/RunCommandError.swift b/Sources/skit/RunCommandError.swift new file mode 100644 index 0000000..a268204 --- /dev/null +++ b/Sources/skit/RunCommandError.swift @@ -0,0 +1,154 @@ +// +// RunCommandError.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import ArgumentParser +import Foundation +import SyntaxKit + +/// Typed failures the `run` pipeline can produce. The render steps just +/// `throw` the case that describes *what* failed; `run()` catches these in +/// one place and converts each to its stderr diagnostic + process exit, so +/// the steps stay free of presentation/exit logic. +internal enum RunCommandError: Error { + /// A usage error (bad input path, or a directory given without `-o`). + /// Surfaced as an ArgumentParser `ValidationError` (exit 64), which prints + /// the message and the command's usage. + case usage(String) + /// The libSyntaxKit directory couldn't be resolved. Exit 2. + case libResolutionFailed(any Error) + /// The bundle's recorded `swift --version` differs from the local one. + /// Exit 2. + case toolchainMismatch(bundle: String, local: String) + /// The spawned `swift` exited non-zero (compile failure, `124` on timeout, + /// `128 + signal`). Carries that code + the toolchain stderr; the code is + /// passed through as the process exit. Also carries the session's + /// `toolchainVerification` so an unverified toolchain can be hinted as a + /// possible cause alongside the diagnostics. + case renderFailed(exitCode: Int32, stderr: String, toolchain: ToolchainVerification) + /// A directory batch couldn't be walked. Exit 1. + case directoryWalkFailed(input: String, underlying: any Error) + /// A render/batch failure whose diagnostics were already surfaced (per-input + /// output, a printed summary, or none). Exit 1 with nothing further to say. + case failed + /// A wrapped Foundation/Subprocess failure with no dedicated mapping; the + /// underlying error is rethrown so ArgumentParser prints it (exit 1). + case unexpected(any Error) + /// `run` was invoked on a platform without a Subprocess backend. Exit 1. + case unsupportedPlatform + + /// stderr text to emit before exiting, or nil. `.usage` is printed by + /// `ValidationError`; `.failed`/`.unexpected` print nothing here (their + /// diagnostics were already surfaced, or ArgumentParser prints them). + internal var diagnostic: String? { + switch self { + case .usage, .failed, .unexpected: + return nil + case .libResolutionFailed(let error): + return "\(error)\n" + case .toolchainMismatch(let bundle, let local): + return Self.toolchainMismatchMessage(bundle: bundle, local: local) + case .renderFailed(_, let stderr, let toolchain): + let parts = [stderr.isEmpty ? nil : stderr, Self.toolchainHint(toolchain)] + .compactMap { $0 } + return parts.isEmpty ? nil : parts.joined() + case .directoryWalkFailed(let input, let underlying): + return "\(Skit.Run.messagePrefix)failed to walk \(input): \(underlying)\n" + case .unsupportedPlatform: + return "\(Skit.Run.messagePrefix)run is not supported on this platform " + + "(no Subprocess backend).\n" + } + } + + /// The terminal error ArgumentParser acts on: a `ValidationError` (usage), + /// an `ExitCode`, or the wrapped underlying error. + internal var terminalError: any Error { + switch self { + case .usage(let message): + return ValidationError(message) + case .libResolutionFailed, .toolchainMismatch: + return ExitCode(2) + case .renderFailed(let exitCode, _, _): + return ExitCode(exitCode) + case .directoryWalkFailed, .failed, .unsupportedPlatform: + return ExitCode(1) + case .unexpected(let underlying): + return underlying + } + } + + /// Maps a SyntaxKit session-setup failure onto the CLI's exit policy. + internal init(_ error: RunnerSetupError) { + switch error { + case .libResolutionFailed(let underlying): + self = .libResolutionFailed(underlying) + case .toolchainMismatch(let bundle, let local): + self = .toolchainMismatch(bundle: bundle, local: local) + } + } + + /// A one-line note appended to a render failure when the bundle/local Swift + /// toolchain wasn't confirmed compatible — so a module-version error reads + /// as a possible toolchain mismatch rather than a mysterious build failure. + /// `nil` when the toolchain was verified (nothing to add). `internal` so the + /// directory-batch path can surface the same note once. + internal static func toolchainHint(_ verification: ToolchainVerification) -> String? { + switch verification { + case .verified: + return nil + case .notChecked: + return "\(Skit.Run.messagePrefix)note: the bundle/local Swift-toolchain check was skipped " + + "(--\(Skit.Run.noToolchainCheckFlagName)); if this is a module-version error, a " + + "toolchain mismatch may be the cause.\n" + case .unverified: + return + "\(Skit.Run.messagePrefix)note: couldn't verify the bundle's Swift toolchain against " + + "your local `swift` (no toolchain stamp, or version capture failed); if this is a " + + "module-version error, a toolchain mismatch may be the cause.\n" + } + } + + /// Human-readable error explaining why the bundle's recorded + /// `swift --version` differs from the local one, and how to recover. + private static func toolchainMismatchMessage(bundle: String, local: String) -> String { + """ + \(Skit.Run.messagePrefix)toolchain mismatch + bundle: \(bundle) + local: \(local) + The bundle's libSyntaxKit was built against a different `swift` than the + one on your PATH. Swift swiftmodules aren't reliably compatible across + versions, so spawning `swift` would fail with a cryptic module-version + diagnostic. + + Rebuild the bundle with: + \(Skit.Run.buildReleaseScriptPath) + Or pass --\(Skit.Run.noToolchainCheckFlagName) to try anyway. + + """ + } +} diff --git a/Sources/skit/Skit+Run.swift b/Sources/skit/Skit+Run.swift index 67b91a3..4a62bfc 100644 --- a/Sources/skit/Skit+Run.swift +++ b/Sources/skit/Skit+Run.swift @@ -49,9 +49,6 @@ extension Skit { internal static let timeoutOptionName = "timeout" internal static let noToolchainCheckFlagName = "no-toolchain-check" - /// Environment variable holding an override for the libSyntaxKit directory. - internal static let libDirEnvironmentKey = "SKIT_LIB_DIR" - /// Prefix for skit's own diagnostics written to stderr. internal static let messagePrefix = "skit: " @@ -59,7 +56,7 @@ extension Skit { internal static let versionFlag = "--version" /// Path to the script that rebuilds the self-contained release bundle. - internal static let buildReleaseScriptPath = "Scripts/build-skit-release.sh" + internal static let buildReleaseScriptPath = "Scripts/build-skit.sh" /// Largest timeout we can safely convert to nanoseconds without overflowing /// the `UInt64` multiplication in `Task.timeout` (`UInt64(seconds) * 1e9`). @@ -120,11 +117,11 @@ extension Skit { } } - internal func execute() async throws(CommandError) { + internal func execute() async throws(RunCommandError) { // The spawn backend is nil only where there's no Subprocess backend // (Windows, embedded) — there `run` can't spawn `swift`/`swiftc`. guard let backend = Self.swiftBackend else { - throw CommandError.unsupportedPlatform + throw RunCommandError.unsupportedPlatform } // Capture the two backend-dependent inputs to the otherwise @@ -132,8 +129,7 @@ extension Skit { // the toolchain check + the cache key, spawned exactly once per run) // and the SKIT_LIB_DIR override. let swiftVersion = await backend.captureSwiftVersion() - let envLibPath = ProcessInfo.processInfo.environment[Self.libDirEnvironmentKey] - .flatMap { $0.isEmpty ? nil : $0 } + let envLibPath = ProcessInfo.processInfo.skitLibPath // Resolve lib dir → toolchain-gate → cache → assemble Runner. That // orchestration lives in SyntaxKit (the `Runner` session initializer); @@ -149,19 +145,19 @@ extension Skit { timeoutSeconds: timeoutSeconds ) { try await backend.runSwift(for: $0) } } catch { - throw CommandError(error) + throw RunCommandError(error) } - // Hand the input off to the runner: `render` owns presentation and the + // Hand the input off to the runner: `Render` owns presentation and the // single-file/directory dispatch, translating render failures into - // `CommandError` for the outer catch to map. - try await render(using: runner, input: input, output: output) + // `RunCommandError` for the outer catch to map. + try await Render.render(using: runner, input: input, output: output) } /// Renders the input(s), with a single seam between the pipeline and the - /// process: every step `throw`s a typed `CommandError`, and the outer catch + /// process: every step `throw`s a typed `RunCommandError`, and the outer catch /// maps that to its stderr diagnostic + terminal `ExitCode`/`ValidationError`. - /// Any non-`CommandError` propagates to ArgumentParser unchanged. + /// Any non-`RunCommandError` propagates to ArgumentParser unchanged. internal func run() async throws { do { try await self.execute() diff --git a/Sources/skit/Skit.Run+CommandError.swift b/Sources/skit/Skit.Run+CommandError.swift deleted file mode 100644 index f0277b6..0000000 --- a/Sources/skit/Skit.Run+CommandError.swift +++ /dev/null @@ -1,156 +0,0 @@ -// -// Skit.Run+CommandError.swift -// SyntaxKit -// -// Created by Leo Dion. -// Copyright © 2026 BrightDigit. -// -// Permission is hereby granted, free of charge, to any person -// obtaining a copy of this software and associated documentation -// files (the “Software”), to deal in the Software without -// restriction, including without limitation the rights to use, -// copy, modify, merge, publish, distribute, sublicense, and/or -// sell copies of the Software, and to permit persons to whom the -// Software is furnished to do so, subject to the following -// conditions: -// -// The above copyright notice and this permission notice shall be -// included in all copies or substantial portions of the Software. -// -// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, -// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES -// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT -// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, -// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -// OTHER DEALINGS IN THE SOFTWARE. -// - -import ArgumentParser -import Foundation -import SyntaxKit - -extension Skit.Run { - /// Typed failures the `run` pipeline can produce. The render steps just - /// `throw` the case that describes *what* failed; `run()` catches these in - /// one place and converts each to its stderr diagnostic + process exit, so - /// the steps stay free of presentation/exit logic. - internal enum CommandError: Error { - /// A usage error (bad input path, or a directory given without `-o`). - /// Surfaced as an ArgumentParser `ValidationError` (exit 64), which prints - /// the message and the command's usage. - case usage(String) - /// The libSyntaxKit directory couldn't be resolved. Exit 2. - case libResolutionFailed(any Error) - /// The bundle's recorded `swift --version` differs from the local one. - /// Exit 2. - case toolchainMismatch(bundle: String, local: String) - /// The spawned `swift` exited non-zero (compile failure, `124` on timeout, - /// `128 + signal`). Carries that code + the toolchain stderr; the code is - /// passed through as the process exit. Also carries the session's - /// `toolchainVerification` so an unverified toolchain can be hinted as a - /// possible cause alongside the diagnostics. - case renderFailed(exitCode: Int32, stderr: String, toolchain: Runner.ToolchainVerification) - /// A directory batch couldn't be walked. Exit 1. - case directoryWalkFailed(input: String, underlying: any Error) - /// A render/batch failure whose diagnostics were already surfaced (per-input - /// output, a printed summary, or none). Exit 1 with nothing further to say. - case failed - /// A wrapped Foundation/Subprocess failure with no dedicated mapping; the - /// underlying error is rethrown so ArgumentParser prints it (exit 1). - case unexpected(any Error) - /// `run` was invoked on a platform without a Subprocess backend. Exit 1. - case unsupportedPlatform - - /// Maps a SyntaxKit session-setup failure onto the CLI's exit policy. - internal init(_ error: Runner.SetupError) { - switch error { - case .libResolutionFailed(let underlying): - self = .libResolutionFailed(underlying) - case .toolchainMismatch(let bundle, let local): - self = .toolchainMismatch(bundle: bundle, local: local) - } - } - - /// stderr text to emit before exiting, or nil. `.usage` is printed by - /// `ValidationError`; `.failed`/`.unexpected` print nothing here (their - /// diagnostics were already surfaced, or ArgumentParser prints them). - internal var diagnostic: String? { - switch self { - case .usage, .failed, .unexpected: - return nil - case .libResolutionFailed(let error): - return "\(error)\n" - case .toolchainMismatch(let bundle, let local): - return Self.toolchainMismatchMessage(bundle: bundle, local: local) - case .renderFailed(_, let stderr, let toolchain): - let parts = [stderr.isEmpty ? nil : stderr, Self.toolchainHint(toolchain)] - .compactMap { $0 } - return parts.isEmpty ? nil : parts.joined() - case .directoryWalkFailed(let input, let underlying): - return "\(Skit.Run.messagePrefix)failed to walk \(input): \(underlying)\n" - case .unsupportedPlatform: - return "\(Skit.Run.messagePrefix)run is not supported on this platform " - + "(no Subprocess backend).\n" - } - } - - /// The terminal error ArgumentParser acts on: a `ValidationError` (usage), - /// an `ExitCode`, or the wrapped underlying error. - internal var terminalError: any Error { - switch self { - case .usage(let message): - return ValidationError(message) - case .libResolutionFailed, .toolchainMismatch: - return ExitCode(2) - case .renderFailed(let exitCode, _, _): - return ExitCode(exitCode) - case .directoryWalkFailed, .failed, .unsupportedPlatform: - return ExitCode(1) - case .unexpected(let underlying): - return underlying - } - } - - /// A one-line note appended to a render failure when the bundle/local Swift - /// toolchain wasn't confirmed compatible — so a module-version error reads - /// as a possible toolchain mismatch rather than a mysterious build failure. - /// `nil` when the toolchain was verified (nothing to add). `internal` so the - /// directory-batch path can surface the same note once. - internal static func toolchainHint(_ verification: Runner.ToolchainVerification) -> String? { - switch verification { - case .verified: - return nil - case .notChecked: - return "\(Skit.Run.messagePrefix)note: the bundle/local Swift-toolchain check was skipped " - + "(--\(Skit.Run.noToolchainCheckFlagName)); if this is a module-version error, a " - + "toolchain mismatch may be the cause.\n" - case .unverified: - return - "\(Skit.Run.messagePrefix)note: couldn't verify the bundle's Swift toolchain against " - + "your local `swift` (no toolchain stamp, or version capture failed); if this is a " - + "module-version error, a toolchain mismatch may be the cause.\n" - } - } - - /// Human-readable error explaining why the bundle's recorded - /// `swift --version` differs from the local one, and how to recover. - private static func toolchainMismatchMessage(bundle: String, local: String) -> String { - """ - \(Skit.Run.messagePrefix)toolchain mismatch - bundle: \(bundle) - local: \(local) - The bundle's libSyntaxKit was built against a different `swift` than the - one on your PATH. Swift swiftmodules aren't reliably compatible across - versions, so spawning `swift` would fail with a cryptic module-version - diagnostic. - - Rebuild the bundle with: - \(Skit.Run.buildReleaseScriptPath) - Or pass --\(Skit.Run.noToolchainCheckFlagName) to try anyway. - - """ - } - } -} diff --git a/Sources/skit/Subprocess.Configuration+Swift.swift b/Sources/skit/Subprocess.Configuration+Swift.swift index b0c1a91..403b0df 100644 --- a/Sources/skit/Subprocess.Configuration+Swift.swift +++ b/Sources/skit/Subprocess.Configuration+Swift.swift @@ -32,7 +32,14 @@ import Foundation import Subprocess import SyntaxKit - import System + + // System (FilePath) ships in the Apple SDK on Darwin; non-Apple platforms + // need swift-system's `SystemPackage` product to get the same `FilePath`. + #if canImport(System) + import System + #else + import SystemPackage + #endif extension Subprocess.Configuration { /// Bounded output capacity for the spawned process (16 MiB stdout / 1 MiB diff --git a/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift b/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift index 02d7224..6136161 100644 --- a/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift +++ b/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift @@ -33,17 +33,15 @@ import Testing @testable import SyntaxKit @Suite internal struct OutputCacheTests { - /// A `ProcessInfo` whose environment is fixed, so cache-key derivation can be - /// tested without depending on the real process environment. - private final class FakeProcessInfo: ProcessInfo { - private let fixedEnvironment: [String: String] - - init(environment: [String: String]) { - self.fixedEnvironment = environment - super.init() - } - - override var environment: [String: String] { fixedEnvironment } + /// A value-type `EnvironmentProvider` whose environment is fixed, so + /// cache-key derivation can be tested without depending on the real process + /// environment. A value type rather than a `ProcessInfo` subclass because + /// `swift-corelibs-foundation` (Linux/Windows) declares `ProcessInfo` as + /// `final` with `environment` on an extension — subclass-and-override + /// doesn't compile there. + private struct FakeEnvironment: EnvironmentProvider { + let environment: [String: String] + let processIdentifier: Int32 = 1 } /// A stub `ContentHashing` that ignores its input and returns a fixed digest, @@ -60,7 +58,10 @@ import Testing swiftVersion: String? = "swift 6.1", environment: [String: String] = [:] ) -> OutputCache { - OutputCache(swiftVersion: swiftVersion, processInfo: FakeProcessInfo(environment: environment)) + OutputCache( + swiftVersion: swiftVersion, + environmentProvider: FakeEnvironment(environment: environment) + ) } private func key(_ cache: OutputCache, source: String = "Struct(\"Foo\") {}") -> String { @@ -71,7 +72,7 @@ import Testing internal func usesInjectedHasher() { let cache = OutputCache( swiftVersion: "swift 6.1", - processInfo: FakeProcessInfo(environment: [:]), + environmentProvider: FakeEnvironment(environment: [:]), makeHasher: { StubHasher() } ) #expect(cache.key(forInput: "anything", libPath: Self.libPath) == "stub-digest") diff --git a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift index cf083a0..ba6d706 100644 --- a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift +++ b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift @@ -38,7 +38,7 @@ import Testing /// without spawning `swift`. Returns the thrown `RunError`, or nil if the /// call unexpectedly succeeded. private func renderFailure( - toolchain: Runner.ToolchainVerification + toolchain: ToolchainVerification ) async -> RunError? { let input = FileManager.default.temporaryDirectory .appendingPathComponent("skit-input-\(UUID().uuidString).swift") From f1b98b645fbb1135bb183e61ff97f0367952ac97 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 10 Jun 2026 14:44:40 -0400 Subject: [PATCH 53/56] Fix Windows + WASM CI test failures (filesystem semantics) regularFiles relied solely on .skipsHiddenFiles, which keys off the dot-prefix on Unix but the FILE_ATTRIBUTE_HIDDEN attribute on Windows, so dot-prefixed entries leaked through there. Filter dot-prefixed path components explicitly for consistent hidden-file semantics on every platform. The two RunnerRenderFailure render tests drive real temp-file IO that is unreliable on WASI (and rendering can't spawn swift there anyway), so disable them on WASI via a new Platform.isWASI test-trait helper rather than #if. Also clear adjacent lint findings surfaced along the way: drop the group.next()! force-unwrap in Task.timeout for a for-await loop guarded by assertionFailure, and bind operands separately in the determinism tests to avoid identical_operands. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../Execution/FileManager+FileSystem.swift | 19 ++++++++ .../SyntaxKit/Execution/Task+Timeout.swift | 12 ++++-- Tests/SyntaxKitTests/Platform.swift | 43 +++++++++++++++++++ .../Unit/Execution/ContentHasherTests.swift | 4 +- .../Unit/Execution/OutputCacheTests.swift | 4 +- .../Execution/RunnerRenderFailureTests.swift | 10 ++++- 6 files changed, 85 insertions(+), 7 deletions(-) create mode 100644 Tests/SyntaxKitTests/Platform.swift diff --git a/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift b/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift index 9663732..e9047b0 100644 --- a/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift +++ b/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift @@ -72,6 +72,11 @@ extension FileManager { var result: [URL] = [] for case let url as URL in enumerator { + // `.skipsHiddenFiles` keys off the dot-prefix convention on Unix but the + // `FILE_ATTRIBUTE_HIDDEN` attribute on Windows, so a dot-prefixed entry + // slips through there. Filter dot-prefixed components explicitly to keep + // the same hidden-file semantics on every platform. + if hasHiddenComponent(url, under: directory) { continue } let values: URLResourceValues do { values = try url.resourceValues(forKeys: [.isRegularFileKey, .isDirectoryKey]) @@ -85,6 +90,20 @@ extension FileManager { return result.sorted { $0.path < $1.path } } + /// True when `url` lies under a dot-prefixed path component relative to + /// `directory`. Mirrors `.skipsHiddenFiles` on platforms (Windows) where that + /// option keys off the hidden *attribute* rather than the dot-prefix + /// convention — and matches Unix's behavior of not descending into hidden + /// directories by also excluding files nested under them. + private func hasHiddenComponent(_ url: URL, under directory: URL) -> Bool { + let base = directory.standardizedFileURL.pathComponents + let full = url.standardizedFileURL.pathComponents + guard full.count > base.count else { + return false + } + return full[base.count...].contains { $0.hasPrefix(".") } + } + /// Writes `data` to `destination`, first creating any missing intermediate /// directories. internal func writeData(_ data: Data, to destination: URL) throws { diff --git a/Sources/SyntaxKit/Execution/Task+Timeout.swift b/Sources/SyntaxKit/Execution/Task+Timeout.swift index 91e90ac..de27c77 100644 --- a/Sources/SyntaxKit/Execution/Task+Timeout.swift +++ b/Sources/SyntaxKit/Execution/Task+Timeout.swift @@ -47,9 +47,15 @@ extension Task where Failure == any Error, Success: Sendable { try await Task.sleep(nanoseconds: UInt64(seconds) * 1_000_000_000) return nil } - let first = try await group.next()! - group.cancelAll() - return first + for try await first in group { + group.cancelAll() + return first + } + // Unreachable: two child tasks were added above, so the group always + // yields at least one result before draining. Trip loudly in debug if + // that invariant is ever broken; fall back to `nil` in release. + assertionFailure("task group drained without yielding a result") + return nil } } } diff --git a/Tests/SyntaxKitTests/Platform.swift b/Tests/SyntaxKitTests/Platform.swift new file mode 100644 index 0000000..022a8cf --- /dev/null +++ b/Tests/SyntaxKitTests/Platform.swift @@ -0,0 +1,43 @@ +// +// Platform.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +/// Compile-time platform facts surfaced as runtime booleans for use with +/// Swift Testing's `.enabled(if:)` / `.disabled(if:)` traits. +internal enum Platform { + /// True on WebAssembly/WASI. The render pipeline relies on host filesystem + /// behavior (atomic temp writes) and ultimately spawning `swift`, neither of + /// which is available there, so render tests are disabled on this platform. + internal static let isWASI: Bool = { + #if os(WASI) + return true + #else + return false + #endif + }() +} diff --git a/Tests/SyntaxKitTests/Unit/Execution/ContentHasherTests.swift b/Tests/SyntaxKitTests/Unit/Execution/ContentHasherTests.swift index a80717e..5e20f43 100644 --- a/Tests/SyntaxKitTests/Unit/Execution/ContentHasherTests.swift +++ b/Tests/SyntaxKitTests/Unit/Execution/ContentHasherTests.swift @@ -52,7 +52,9 @@ import Testing @Test("Same bytes produce the same digest across fresh hashers") internal func determinism() { let data = Data("the quick brown fox".utf8) - #expect(digest(of: data) == digest(of: data)) + let first = digest(of: data) + let second = digest(of: data) + #expect(first == second) } @Test("Different inputs produce different digests") diff --git a/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift b/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift index 6136161..d24da0e 100644 --- a/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift +++ b/Tests/SyntaxKitTests/Unit/Execution/OutputCacheTests.swift @@ -81,7 +81,9 @@ import Testing @Test("The same inputs always derive the same key") internal func stableKey() { let cache = cache() - #expect(key(cache) == key(cache)) + let firstKey = key(cache) + let secondKey = key(cache) + #expect(firstKey == secondKey) } @Test("Different source bytes derive different keys") diff --git a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift index ba6d706..0b4e619 100644 --- a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift +++ b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift @@ -62,7 +62,10 @@ import Testing } } - @Test("renderFailed carries the session's toolchain verification and diagnostics") + @Test( + "renderFailed carries the session's toolchain verification and diagnostics", + .disabled(if: Platform.isWASI, "Render needs host filesystem/subprocess; not on WASI") + ) internal func carriesUnverifiedToolchain() async { guard case .renderFailed(let exitCode, let stderr, let toolchain)? = @@ -76,7 +79,10 @@ import Testing #expect(toolchain == .unverified) } - @Test("renderFailed reflects a verified toolchain unchanged") + @Test( + "renderFailed reflects a verified toolchain unchanged", + .disabled(if: Platform.isWASI, "Render needs host filesystem/subprocess; not on WASI") + ) internal func reflectsVerifiedToolchain() async { guard case .renderFailed(_, _, let toolchain)? = From 5a1d9b8e24202b08ff15ea5f5fde8287b10282d7 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 10 Jun 2026 14:58:43 -0400 Subject: [PATCH 54/56] Map render RunErrors via RunCommandError initializers; cut skit complexity Add RunCommandError(renderFileError:) and RunCommandError(renderDirectoryError:input:) initializers that translate the RunError thrown by Runner.renderFile / renderDirectory onto the CLI's exit policy, mirroring the existing init(_:RunnerSetupError). Render.renderSingle/renderBatch now use a single total catch each instead of an inline switch / multiple pattern-matched catches, and the per-outcome batch diagnostics move into a reportOutcomeDiagnostics helper. Clears both Render.swift cyclomatic_complexity violations; behavior (error->exit mapping and stderr output) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) --- Sources/skit/Render.swift | 61 +++++++++++++----------------- Sources/skit/RunCommandError.swift | 30 +++++++++++++++ 2 files changed, 57 insertions(+), 34 deletions(-) diff --git a/Sources/skit/Render.swift b/Sources/skit/Render.swift index 2021076..a6c43ec 100644 --- a/Sources/skit/Render.swift +++ b/Sources/skit/Render.swift @@ -75,16 +75,8 @@ internal enum Render { rendered = try await runner.renderFile(input: input) } catch { // `error` is typed `RunError` (renderFile is `throws(RunError)`); the - // switch is exhaustive, so this single catch is both total (satisfies - // `throws(RunCommandError)`) and free of an unreachable clause. - switch error { - case .invalidInput(let message): - throw RunCommandError.usage(message) - case .renderFailed(let exitCode, let stderr, let toolchain): - throw RunCommandError.renderFailed(exitCode: exitCode, stderr: stderr, toolchain: toolchain) - case .unexpected(let underlying): - throw RunCommandError.unexpected(underlying) - } + // initializer's switch is exhaustive, so this single catch is total. + throw RunCommandError(renderFileError: error) } if !rendered.stderr.isEmpty { @@ -115,17 +107,10 @@ internal enum Render { let outcomes: [FileOutcome] do { outcomes = try await runner.renderDirectory(inputDir: input, outputDir: output) - } catch .invalidInput(let message) { - throw RunCommandError.usage(message) - } catch .unexpected(let underlying) { - // `renderDirectory` only wraps directory-walk failures in `.unexpected`; - // `RunCommandError.directoryWalkFailed` carries the original "failed to - // walk" framing the CLI prints. - throw RunCommandError.directoryWalkFailed(input: input, underlying: underlying) } catch { - // renderDirectory does not throw .renderFailed — that's a per-file - // outcome. Defensive. - throw RunCommandError.failed + // `error` is typed `RunError` (renderDirectory is `throws(RunError)`); + // the initializer's switch is exhaustive, so this single catch is total. + throw RunCommandError(renderDirectoryError: error, input: input) } if outcomes.isEmpty { @@ -135,20 +120,7 @@ internal enum Render { return } - for outcome in outcomes { - if !outcome.stderr.isEmpty { - FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) - FileHandle.standardError.write(Data(outcome.stderr.utf8)) - } - // .renderFailed already had its stderr surfaced above. Other failures - // (process spawn, write) carry the diagnostic in the error itself. - if let error = outcome.result { - if case .renderFailed = error { - continue - } - FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) - } - } + reportOutcomeDiagnostics(outcomes) FileHandle.standardError.write( Data( @@ -166,4 +138,25 @@ internal enum Render { throw RunCommandError.failed } } + + /// Prints per-input diagnostics for a batch: each file's compiler stderr + /// (fenced with a `---- path ----` header), then for any non-`.renderFailed` + /// failure (whose stderr was just surfaced) a one-line `path: error`. Spawn/ + /// write failures carry their diagnostic in the error itself. + private static func reportOutcomeDiagnostics(_ outcomes: [FileOutcome]) { + for outcome in outcomes { + if !outcome.stderr.isEmpty { + FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) + FileHandle.standardError.write(Data(outcome.stderr.utf8)) + } + // .renderFailed already had its stderr surfaced above. Other failures + // (process spawn, write) carry the diagnostic in the error itself. + if let error = outcome.result { + if case .renderFailed = error { + continue + } + FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) + } + } + } } diff --git a/Sources/skit/RunCommandError.swift b/Sources/skit/RunCommandError.swift index a268204..bc075c3 100644 --- a/Sources/skit/RunCommandError.swift +++ b/Sources/skit/RunCommandError.swift @@ -112,6 +112,36 @@ internal enum RunCommandError: Error { } } + /// Maps the `RunError` from `Runner.renderFile` onto the CLI's exit policy. + /// `renderFile`'s error is exhaustive over `RunError`, so every case has a + /// direct translation. + internal init(renderFileError error: RunError) { + switch error { + case .invalidInput(let message): + self = .usage(message) + case .renderFailed(let exitCode, let stderr, let toolchain): + self = .renderFailed(exitCode: exitCode, stderr: stderr, toolchain: toolchain) + case .unexpected(let underlying): + self = .unexpected(underlying) + } + } + + /// Maps the `RunError` from `Runner.renderDirectory` onto the CLI's exit + /// policy. Unlike the single-file path, `renderDirectory` only wraps + /// directory-walk failures in `.unexpected` (carrying `input` for the + /// "failed to walk" framing), and never throws `.renderFailed` — that's a + /// per-file outcome — so that case is defensive. + internal init(renderDirectoryError error: RunError, input: String) { + switch error { + case .invalidInput(let message): + self = .usage(message) + case .unexpected(let underlying): + self = .directoryWalkFailed(input: input, underlying: underlying) + case .renderFailed: + self = .failed + } + } + /// A one-line note appended to a render failure when the bundle/local Swift /// toolchain wasn't confirmed compatible — so a module-version error reads /// as a possible toolchain mismatch rather than a mysterious build failure. From 49130b9aea7c7c60a4de7c759b55e6e87ad0cd31 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 10 Jun 2026 15:32:12 -0400 Subject: [PATCH 55/56] Make SyntaxKit render methods filesystem-free; move IO into skit (#163) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The public Runner render API no longer reads user inputs or writes user outputs. `renderFile(input:)` becomes `render(source:originalPath:)` and `renderDirectory(inputDir:outputDir:)` becomes `render(sources: [RenderInput]) -> [FileOutcome]`, where FileOutcome now carries the rendered `stdout`. The internal temp-wrapper write that swiftc compiles stays in the SDK (purity is at the API boundary, not zero IO). The reusable filesystem primitives stay as public SDK utilities the CLI composes — `FileManager.collectInputs`, `URL.rerooted`, `FileManager.writeData` are now public; the output-mirroring orchestration (`FileManager.writeOutput`, `RenderTaskResult.writeOutput`/`OutputDestination`) is deleted. skit's Render/Render+Batch now own walking, reading, and writing: collect inputs → read into RenderInput → render in-memory → mirror each stdout via rerooting, folding read/write failures into per-file outcomes so one bad input doesn't abort the batch. RunnerRenderFailureTests now drives in-memory source (no temp input file); adds RunnerBatchRenderTests covering empty/all-success/isolated-failure batch semantics with a stub backend. Verified end-to-end: single-file render to stdout, directory render mirrors a nested tree (2/2 succeeded, exit 0), and a bad input is diagnosed + skipped while peers are written (1/2 succeeded, exit 1). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../Execution/FileManager+Execution.swift | 44 +---- .../Execution/FileManager+FileSystem.swift | 7 +- Sources/SyntaxKit/Execution/FileOutcome.swift | 24 +-- Sources/SyntaxKit/Execution/RenderInput.swift | 50 ++++++ .../Execution/RenderTaskResult.swift | 39 +---- .../Execution/Runner+Directory.swift | 89 +++++----- Sources/SyntaxKit/Execution/Runner.swift | 46 +++--- Sources/SyntaxKit/Execution/URL+Reroot.swift | 7 +- Sources/skit/Render+Batch.swift | 154 ++++++++++++++++++ Sources/skit/Render.swift | 104 ++++-------- Sources/skit/RunCommandError.swift | 18 +- .../Execution/RunnerBatchRenderTests.swift | 124 ++++++++++++++ .../Execution/RunnerRenderFailureTests.swift | 15 +- 13 files changed, 469 insertions(+), 252 deletions(-) create mode 100644 Sources/SyntaxKit/Execution/RenderInput.swift create mode 100644 Sources/skit/Render+Batch.swift create mode 100644 Tests/SyntaxKitTests/Unit/Execution/RunnerBatchRenderTests.swift diff --git a/Sources/SyntaxKit/Execution/FileManager+Execution.swift b/Sources/SyntaxKit/Execution/FileManager+Execution.swift index de11ca5..51b61bf 100644 --- a/Sources/SyntaxKit/Execution/FileManager+Execution.swift +++ b/Sources/SyntaxKit/Execution/FileManager+Execution.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +public import Foundation extension FileManager { /// Library product name whose platform-specific dylib marks a lib dir. @@ -64,7 +64,10 @@ extension FileManager { /// Throws `CollectInputsError.cliError` when the directory can't be /// enumerated, or `.resourceValuesFailure` when a file's resource values /// can't be read — both are bulk failures with nothing per-file to report. - internal func collectInputs(at inputDir: URL) throws(CollectInputsError) -> [URL] { + /// + /// `public` so callers (e.g. the skit CLI) can collect inputs explicitly and + /// feed the in-memory sources to `Runner.render(sources:)`. + public func collectInputs(at inputDir: URL) throws(CollectInputsError) -> [URL] { let files: [URL] do { files = try regularFiles(under: inputDir) @@ -85,41 +88,4 @@ extension FileManager { .filter { !$0.lastPathComponent.hasPrefix(Self.nonInputFilePrefix) } .map(\.standardizedFileURL) } - - /// Builds the `FileOutcome` for one render result, writing a successful - /// render's stdout to its mirrored destination under `outputBase`. The write - /// side effect lives here; failures (a non-zero render exit, or a write - /// error) are folded into the returned outcome rather than thrown, so a - /// failing peer doesn't prevent successful files in the batch from being - /// written (Tuist-analog batch semantics). No diagnostics are printed here; - /// the caller does that. - internal func writeOutput( - for result: RenderTaskResult, - inputBase: URL, - outputBase: URL, - toolchain: ToolchainVerification - ) -> FileOutcome { - // Mirror the input's location under the output base (generic path math). - let destination = result.input.rerooted(from: inputBase, onto: outputBase) - - // stderr is the toolchain's diagnostics whenever the render produced any — - // i.e. on every successful spawn, regardless of how the write then fares. - let stderr = (try? result.result.get())?.stderr ?? "" - - let failure: RunError? - do { - try result.writeOutput(to: destination, toolchain: toolchain) { [self] in - try self.writeData($0.output, to: $0.destination) - } - failure = nil - } catch { - failure = error - } - - return FileOutcome( - input: result.input, - stderr: stderr, - result: failure - ) - } } diff --git a/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift b/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift index e9047b0..96af336 100644 --- a/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift +++ b/Sources/SyntaxKit/Execution/FileManager+FileSystem.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +public import Foundation extension FileManager { /// Classifies what exists at `path` in a single stat — `.missing`, `.file`, @@ -105,8 +105,9 @@ extension FileManager { } /// Writes `data` to `destination`, first creating any missing intermediate - /// directories. - internal func writeData(_ data: Data, to destination: URL) throws { + /// directories. `public` so a caller writing a rendered batch (e.g. the skit + /// CLI) can mirror outputs into a destination tree. + public func writeData(_ data: Data, to destination: URL) throws { try createDirectory( at: destination.deletingLastPathComponent(), withIntermediateDirectories: true diff --git a/Sources/SyntaxKit/Execution/FileOutcome.swift b/Sources/SyntaxKit/Execution/FileOutcome.swift index c4357e6..491111a 100644 --- a/Sources/SyntaxKit/Execution/FileOutcome.swift +++ b/Sources/SyntaxKit/Execution/FileOutcome.swift @@ -29,32 +29,36 @@ public import Foundation -/// Per-input result of a directory-mode render. `stderr` carries the (possibly -/// path-rewritten) diagnostics from the spawned `swift`; it may be present -/// whether or not the input succeeded (e.g. a successful render that emitted -/// warnings). `result` is `nil` when the rendered output was written to its -/// mirrored destination, and carries the `RunError` when the input could not -/// be rendered (`.renderFailed`) or its output could not be written -/// (`.unexpected`). +/// Per-input result of a batch render. `stdout` is the rendered Swift source +/// (empty on failure). `stderr` carries the (possibly path-rewritten) +/// diagnostics from the spawned `swift`; it may be present whether or not the +/// input succeeded (e.g. a successful render that emitted warnings). `result` +/// is `nil` when the input rendered successfully (so `stdout` is valid and the +/// caller may write it), and carries the `RunError` when the input could not be +/// rendered (`.renderFailed`/`.unexpected`). /// -/// `Runner.renderDirectory` returns `[FileOutcome]`; per-input failures are +/// `Runner.render(sources:)` returns `[FileOutcome]`; per-input failures are /// captured here, not thrown, so a single bad input doesn't tear the batch -/// down. +/// down. The SDK does not write `stdout` anywhere — the caller owns that. public struct FileOutcome: Sendable { /// The input file this outcome describes. public let input: URL + /// The rendered Swift source for this input; empty when `result` is non-nil. + public let stdout: Data /// The (possibly path-rewritten) `swift` diagnostics for this input. public let stderr: String - /// `nil` when the output was written; the `RunError` otherwise. + /// `nil` when the input rendered successfully; the `RunError` otherwise. public let result: RunError? /// Creates an outcome for one rendered (or failed) input. public init( input: URL, + stdout: Data, stderr: String, result: RunError? ) { self.input = input + self.stdout = stdout self.stderr = stderr self.result = result } diff --git a/Sources/SyntaxKit/Execution/RenderInput.swift b/Sources/SyntaxKit/Execution/RenderInput.swift new file mode 100644 index 0000000..5df61e5 --- /dev/null +++ b/Sources/SyntaxKit/Execution/RenderInput.swift @@ -0,0 +1,50 @@ +// +// RenderInput.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +public import Foundation + +/// One in-memory input for a batch render: the source bytes paired with the +/// URL that identifies them. `Runner.render(sources:)` never opens `url` — it +/// is a *label*, used for `#sourceLocation` mapping and stderr path-rewriting, +/// and carried back on the matching `FileOutcome` so the caller can map the +/// rendered output to a destination (e.g. by rerooting `url` onto an output +/// tree). The caller is responsible for reading the file into `source`. +public struct RenderInput: Sendable { + /// The URL identifying this input. Used as a diagnostic label and as the + /// key the caller reroots when writing output; never opened by the SDK. + public let url: URL + /// The input's source bytes, already read into memory by the caller. + public let source: String + + /// Pairs an identifying `url` with its already-loaded `source`. + public init(url: URL, source: String) { + self.url = url + self.source = source + } +} diff --git a/Sources/SyntaxKit/Execution/RenderTaskResult.swift b/Sources/SyntaxKit/Execution/RenderTaskResult.swift index 316d46f..dbdc4bf 100644 --- a/Sources/SyntaxKit/Execution/RenderTaskResult.swift +++ b/Sources/SyntaxKit/Execution/RenderTaskResult.swift @@ -29,45 +29,12 @@ import Foundation -/// Payload the per-input render `TaskGroup` yields back to `renderDirectory`. +/// Payload the per-input render `TaskGroup` yields back to `render(sources:)`. /// Failures are captured (not thrown) so a single bad input doesn't tear down /// the group; `processFile`'s heterogeneous Foundation/Subprocess throws are -/// normalized into `RunError` (typically `.unexpected`) by `runOne`. -/// `internal` so `FileManager.writeOutput` (in `FileManager+Execution.swift`) -/// can consume it. +/// normalized into `RunError` (typically `.unexpected`) by `runOne`. `Runner` +/// then folds each value into a public `FileOutcome`. internal struct RenderTaskResult: Sendable { - /// Bundles a successful render's bytes with the destination they should be - /// written to. Nested because it's only ever produced by `writeOutput` to - /// hand off to its caller-supplied writer closure. - internal struct OutputDestination: Sendable { - internal let output: Data - internal let destination: URL - } - internal let input: URL internal let result: Result - - internal func writeOutput( - to destination: URL, - toolchain: ToolchainVerification, - using writeOutputDestination: (OutputDestination) throws -> Void - ) throws(RunError) { - let result = try self.result.get() - - guard result.exitCode == 0 else { - throw .renderFailed( - exitCode: result.exitCode, - stderr: result.stderr, - toolchain: toolchain - ) - } - - let outputDestination = OutputDestination(output: result.stdout, destination: destination) - - do { - try writeOutputDestination(outputDestination) - } catch { - throw .unexpected(error) - } - } } diff --git a/Sources/SyntaxKit/Execution/Runner+Directory.swift b/Sources/SyntaxKit/Execution/Runner+Directory.swift index 9b4047e..3d34b13 100644 --- a/Sources/SyntaxKit/Execution/Runner+Directory.swift +++ b/Sources/SyntaxKit/Execution/Runner+Directory.swift @@ -30,47 +30,52 @@ import Foundation extension Runner { - /// Walks `inputDir` for `.swift` inputs, processes them concurrently (up to - /// the active core count), and mirrors successfully-rendered outputs into - /// `outputDir`. A failure on one input does not abort the batch — successful - /// peers are still written. Returns the per-input outcomes; the caller - /// inspects `failureCount` (and per-outcome stderr) to decide presentation. - /// - /// Throws `RunError.unexpected` only for bulk failures the SDK can't - /// recover from (e.g. the input directory can't be enumerated). An empty - /// input set is *not* an error — the returned array is simply empty. - public func renderDirectory( - inputDir: String, - outputDir: String - ) async throws(RunError) -> [FileOutcome] { - let inputURL = URL(fileURLWithPath: inputDir).standardizedFileURL - let outputURL = URL(fileURLWithPath: outputDir).standardizedFileURL - - // Phase 1: enumerate inputs. A walk failure is a bulk failure: there's - // nothing per-file to report, so it surfaces as a typed throw. The - // collect-specific error is folded into `RunError.unexpected` — the bulk - // channel the caller already presents. - let inputs: [URL] - do { - inputs = try fileManager().collectInputs(at: inputURL) - } catch { - throw RunError.unexpected(error) - } - guard !inputs.isEmpty else { + /// Renders a batch of in-memory inputs concurrently (up to the active core + /// count) and returns a per-input `FileOutcome` carrying the rendered + /// `stdout`, diagnostics, and any `RunError`. The SDK reads nothing and + /// writes nothing — the caller supplies the sources (e.g. via + /// `FileManager.collectInputs` + reads) and writes each successful + /// `stdout` wherever it likes (e.g. by rerooting `outcome.input`). A failure + /// on one input does not affect its peers; inspect `failureCount` (and + /// per-outcome `stderr`) to decide presentation. An empty input set yields an + /// empty array. + public func render(sources: [RenderInput]) async -> [FileOutcome] { + guard !sources.isEmpty else { return [] } - // Phase 2: render every input with bounded concurrency. - let renderResults = await processInputs(inputs) + // Render every input with bounded concurrency, then fold each raw result + // into the public per-input outcome. + let renderResults = await processInputs(sources) + return renderResults.map(outcome(for:)) + } - // Phase 3: write successes and capture a per-input outcome for each. - return renderResults.map { - fileManager().writeOutput( - for: $0, - inputBase: inputURL, - outputBase: outputURL, - toolchain: toolchainVerification + /// Folds one raw `RenderTaskResult` into a public `FileOutcome`: a zero-exit + /// render yields its `stdout` with `result == nil`; a non-zero exit becomes + /// `.renderFailed` (carrying the session toolchain); a captured throw is + /// surfaced as-is. No file IO — writing is the caller's job. + private func outcome(for taskResult: RenderTaskResult) -> FileOutcome { + switch taskResult.result { + case .success(let process) where process.exitCode == 0: + return FileOutcome( + input: taskResult.input, + stdout: process.stdout, + stderr: process.stderr, + result: nil + ) + case .success(let process): + return FileOutcome( + input: taskResult.input, + stdout: Data(), + stderr: process.stderr, + result: .renderFailed( + exitCode: process.exitCode, + stderr: process.stderr, + toolchain: toolchainVerification + ) ) + case .failure(let error): + return FileOutcome(input: taskResult.input, stdout: Data(), stderr: "", result: error) } } @@ -78,7 +83,7 @@ extension Runner { /// the active core count so a 200-file batch doesn't fork 200 simultaneous /// `swift` processes: the group is seeded up to the cap, then refilled one /// task per completion until the inputs are exhausted. - private func processInputs(_ inputs: [URL]) async -> [RenderTaskResult] { + private func processInputs(_ inputs: [RenderInput]) async -> [RenderTaskResult] { let maxConcurrent = max(1, ProcessInfo.processInfo.activeProcessorCount) var renderResults: [RenderTaskResult] = [] @@ -108,14 +113,14 @@ extension Runner { /// `processFile`'s heterogeneous Foundation/Subprocess throws are wrapped /// in `RunError.unexpected` here so the rest of the pipeline sees a single /// typed error. - private func runOne(_ input: URL) async -> RenderTaskResult { + private func runOne(_ input: RenderInput) async -> RenderTaskResult { do { - let result = try await processFile(inputPath: input.path) - return RenderTaskResult(input: input, result: .success(result)) + let result = try await processFile(source: input.source, originalPath: input.url.path) + return RenderTaskResult(input: input.url, result: .success(result)) } catch let error as RunError { - return RenderTaskResult(input: input, result: .failure(error)) + return RenderTaskResult(input: input.url, result: .failure(error)) } catch { - return RenderTaskResult(input: input, result: .failure(.unexpected(error))) + return RenderTaskResult(input: input.url, result: .failure(.unexpected(error))) } } } diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index cb9f039..5bd0a71 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -32,8 +32,8 @@ public import Foundation // Render lifecycle (per `Runner` call): // 1. Bundle.main.resolveLibPath — find lib/ (explicit flag → env → adjacent → brew) // 2. ToolchainCheckResult.init — compare bundle stamp to `swift --version` -// 3. Runner.renderFile / .renderDirectory — single- or batch-input mode -// 4. processFile (per input) — load → cache lookup → wrap → spawn → cache store +// 3. Runner.render(source:) / .render(sources:) — single- or batch-input mode +// 4. processFile (per input) — cache lookup → wrap → spawn → cache store // 5. wrap — hoist imports, wrap body in Group { … }, #sourceLocation // 6. runSwift — spawn `swift` with timeout watchdog // See Docs/skit.md for design rationale and trade-offs. @@ -96,19 +96,21 @@ public struct Runner: Sendable { // MARK: - Single-file mode - /// Renders one input and returns the rendered bytes plus any compiler - /// diagnostics. No file IO, no stdout/stderr writes — the caller decides - /// where the result goes. - /// - /// On a non-zero subprocess exit, throws `RunError.renderFailed(exitCode: - /// stderr:)` carrying the toolchain's diagnostic. Any Foundation/Subprocess - /// failure (file read, spawn) is wrapped in `RunError.unexpected`. - public func renderFile(input: String) async throws(RunError) -> SingleFileRender { - // Render the input. `processFile` may hit the output cache and skip the - // spawn entirely; either way the result has the same shape. + /// Renders one in-memory input and returns the rendered bytes plus any + /// compiler diagnostics. The SDK reads no input and writes no output; + /// `originalPath` is only a diagnostic label (`#sourceLocation` + stderr + /// path-rewriting), and the caller decides where the result goes. + /// On a non-zero subprocess exit, throws `RunError.renderFailed` carrying the + /// toolchain's diagnostic. Any Foundation/Subprocess failure (the internal + /// temp-wrapper write, spawn) is wrapped in `RunError.unexpected`. + public func render( + source: String, + originalPath: String + ) async throws(RunError) -> SingleFileRender { + // `processFile` may hit the output cache and skip the spawn; same shape. let result: ProcessResult do { - result = try await processFile(inputPath: input) + result = try await processFile(source: source, originalPath: originalPath) } catch let error as RunError { throw error } catch { @@ -128,16 +130,14 @@ public struct Runner: Sendable { // MARK: - Per-file work - /// The per-input render pipeline: load source → consult the output cache → - /// (on miss) wrap → spawn `swift` → rewrite diagnostics → store the result - /// in the cache. The temp wrapper file is created in a per-run tmp dir and - /// torn down by `defer` whether the spawn succeeded or not. `internal` so - /// the directory-mode extension (`Runner+Directory.swift`) can reuse it. - internal func processFile(inputPath: String) async throws -> ProcessResult { - // Load the input source. Anything past this point keys off these bytes. - let inputURL = URL(fileURLWithPath: inputPath).standardizedFileURL - let absoluteInputPath = inputURL.path - let source = try String(contentsOf: inputURL, encoding: .utf8) + /// The per-input render pipeline: consult the output cache → (on miss) wrap → + /// spawn `swift` → rewrite diagnostics → store in the cache. `source` comes + /// from the caller; `originalPath` is only a `#sourceLocation`/stderr label + /// (never opened). The temp wrapper is created in a per-run tmp dir and torn + /// down by `defer`. `internal` so `Runner+Directory.swift` can reuse it. + internal func processFile(source: String, originalPath: String) async throws -> ProcessResult { + // The label diagnostics map back to; standardized to match `#sourceLocation`. + let absoluteInputPath = URL(fileURLWithPath: originalPath).standardizedFileURL.path // Compute the output cache key (nil under `--no-cache` or when the cache // root couldn't be derived at startup). Mixes input bytes, toolchain diff --git a/Sources/SyntaxKit/Execution/URL+Reroot.swift b/Sources/SyntaxKit/Execution/URL+Reroot.swift index 794aa6b..d307548 100644 --- a/Sources/SyntaxKit/Execution/URL+Reroot.swift +++ b/Sources/SyntaxKit/Execution/URL+Reroot.swift @@ -27,7 +27,7 @@ // OTHER DEALINGS IN THE SOFTWARE. // -import Foundation +public import Foundation extension URL { /// Re-roots this URL from `base` onto `newBase`, preserving the relative @@ -36,7 +36,10 @@ extension URL { /// /// - Precondition: `self` is located under `base` (its path is prefixed by /// `base`'s). Callers that enumerate `base` satisfy this by construction. - internal func rerooted(from base: URL, onto newBase: URL) -> URL { + /// + /// `public` so a caller mirroring a rendered batch (e.g. the skit CLI) can + /// map each `FileOutcome.input` onto its destination tree. + public func rerooted(from base: URL, onto newBase: URL) -> URL { let relative = path.dropFirst(base.path.count + 1) return newBase.appendingPathComponent(String(relative)) } diff --git a/Sources/skit/Render+Batch.swift b/Sources/skit/Render+Batch.swift new file mode 100644 index 0000000..4be2669 --- /dev/null +++ b/Sources/skit/Render+Batch.swift @@ -0,0 +1,154 @@ +// +// Render+Batch.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import SyntaxKit + +extension Render { + /// Collects a directory's `.swift` inputs, reads them, renders them via the + /// filesystem-free `Runner.render(sources:)`, mirrors each rendered output + /// into `output`, prints per-input diagnostics (fenced when several files + /// emit them), prints a one-line summary, and maps any failures to + /// `ExitCode(1)` — Tuist-analog batch semantics. + internal static func renderBatch( + runner: Runner, + input: String, + output: String + ) async throws(RunCommandError) { + let inputURL = URL(fileURLWithPath: input).standardizedFileURL + let outputURL = URL(fileURLWithPath: output).standardizedFileURL + + let inputs: [URL] + do { + inputs = try FileManager.default.collectInputs(at: inputURL) + } catch { + throw RunCommandError(collectInputsError: error, input: input) + } + + if inputs.isEmpty { + FileHandle.standardError.write( + Data("\(Skit.Run.messagePrefix)no .swift inputs under \(input)\n".utf8) + ) + return + } + + // Read inputs into memory (read failures become per-file outcomes so a bad + // input doesn't abort the batch), render in-memory, then mirror each + // successful render into the output tree. + let (sources, readFailures) = loadSources(inputs) + let rendered = await runner.render(sources: sources) + let written = rendered.map { writeRendered($0, inputBase: inputURL, outputBase: outputURL) } + let outcomes = readFailures + written + + reportOutcomeDiagnostics(outcomes) + + FileHandle.standardError.write( + Data( + ("\(Skit.Run.messagePrefix)\(outcomes.count - outcomes.failureCount)" + + "/\(outcomes.count) succeeded\n").utf8 + ) + ) + + if outcomes.failureCount > 0 { + // Some inputs failed; if the toolchain couldn't be verified, hint once + // that a Swift-version mismatch may be behind the build errors above. + if let hint = RunCommandError.toolchainHint(runner.toolchainVerification) { + FileHandle.standardError.write(Data(hint.utf8)) + } + throw RunCommandError.failed + } + } + + /// Prints per-input diagnostics for a batch: each file's compiler stderr + /// (fenced with a `---- path ----` header), then for any non-`.renderFailed` + /// failure (whose stderr was just surfaced) a one-line `path: error`. Spawn/ + /// write failures carry their diagnostic in the error itself. + private static func reportOutcomeDiagnostics(_ outcomes: [FileOutcome]) { + for outcome in outcomes { + if !outcome.stderr.isEmpty { + FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) + FileHandle.standardError.write(Data(outcome.stderr.utf8)) + } + // .renderFailed already had its stderr surfaced above. Other failures + // (process spawn, write) carry the diagnostic in the error itself. + if let error = outcome.result { + if case .renderFailed = error { + continue + } + FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) + } + } + } + + /// Reads every input into a `RenderInput`. A read failure does not abort the + /// batch — it becomes a per-file `.unexpected` `FileOutcome`, returned + /// alongside the successfully-loaded sources. + private static func loadSources( + _ inputs: [URL] + ) -> (sources: [RenderInput], failures: [FileOutcome]) { + var sources: [RenderInput] = [] + var failures: [FileOutcome] = [] + for url in inputs { + do { + let source = try String(contentsOf: url, encoding: .utf8) + sources.append(RenderInput(url: url, source: source)) + } catch { + failures.append( + FileOutcome(input: url, stdout: Data(), stderr: "", result: .unexpected(error)) + ) + } + } + return (sources, failures) + } + + /// Mirrors a successful render's `stdout` to its destination under + /// `outputBase`, folding any write error into the returned outcome. Already- + /// failed outcomes pass through untouched. + private static func writeRendered( + _ outcome: FileOutcome, + inputBase: URL, + outputBase: URL + ) -> FileOutcome { + guard outcome.result == nil else { + return outcome + } + let destination = outcome.input.rerooted(from: inputBase, onto: outputBase) + do { + try FileManager.default.writeData(outcome.stdout, to: destination) + return outcome + } catch { + return FileOutcome( + input: outcome.input, + stdout: outcome.stdout, + stderr: outcome.stderr, + result: .unexpected(error) + ) + } + } +} diff --git a/Sources/skit/Render.swift b/Sources/skit/Render.swift index a6c43ec..f224ea9 100644 --- a/Sources/skit/Render.swift +++ b/Sources/skit/Render.swift @@ -62,101 +62,53 @@ internal enum Render { } } - /// Renders one input via `Runner.renderFile`, prints any compiler stderr, - /// and writes the rendered Swift source either to `outputPath` or to - /// stdout. The runner itself does none of that — that's the CLI's job. + /// Reads `input`, renders it via the filesystem-free `Runner.render(source:)`, + /// prints any compiler stderr, and writes the rendered Swift source either to + /// `output` or to stdout. The runner reads and writes nothing — that's the + /// CLI's job. private static func renderSingle( runner: Runner, input: String, output: String? ) async throws(RunCommandError) { + let inputURL = URL(fileURLWithPath: input).standardizedFileURL + let source: String + do { + source = try String(contentsOf: inputURL, encoding: .utf8) + } catch { + throw RunCommandError.unexpected(error) + } + let rendered: SingleFileRender do { - rendered = try await runner.renderFile(input: input) + rendered = try await runner.render(source: source, originalPath: inputURL.path) } catch { - // `error` is typed `RunError` (renderFile is `throws(RunError)`); the + // `error` is typed `RunError` (render is `throws(RunError)`); the // initializer's switch is exhaustive, so this single catch is total. throw RunCommandError(renderFileError: error) } + try emit(rendered, to: output) + } + + /// Prints any diagnostics, then writes the rendered source to `output` (or + /// stdout when `output` is nil). A write failure has no diagnostic of its own, + /// so the underlying error is surfaced (ArgumentParser prints it, exit 1). + private static func emit( + _ rendered: SingleFileRender, + to output: String? + ) throws(RunCommandError) { if !rendered.stderr.isEmpty { FileHandle.standardError.write(Data(rendered.stderr.utf8)) } - if let output { - do { - try rendered.stdout.write(to: URL(fileURLWithPath: output)) - } catch { - // A write failure has no diagnostic of its own; surface the underlying - // error (ArgumentParser prints it, exit 1). - throw RunCommandError.unexpected(error) - } - } else { + guard let output else { FileHandle.standardOutput.write(rendered.stdout) + return } - } - - /// Renders a directory batch via `Runner.renderDirectory`, prints per-input - /// diagnostics (fenced when several files emit them), surfaces non-render - /// failures, prints a one-line summary, and maps any failures to - /// `ExitCode(1)` — Tuist-analog batch semantics. - private static func renderBatch( - runner: Runner, - input: String, - output: String - ) async throws(RunCommandError) { - let outcomes: [FileOutcome] do { - outcomes = try await runner.renderDirectory(inputDir: input, outputDir: output) + try rendered.stdout.write(to: URL(fileURLWithPath: output)) } catch { - // `error` is typed `RunError` (renderDirectory is `throws(RunError)`); - // the initializer's switch is exhaustive, so this single catch is total. - throw RunCommandError(renderDirectoryError: error, input: input) - } - - if outcomes.isEmpty { - FileHandle.standardError.write( - Data("\(Skit.Run.messagePrefix)no .swift inputs under \(input)\n".utf8) - ) - return - } - - reportOutcomeDiagnostics(outcomes) - - FileHandle.standardError.write( - Data( - ("\(Skit.Run.messagePrefix)\(outcomes.count - outcomes.failureCount)" - + "/\(outcomes.count) succeeded\n").utf8 - ) - ) - - if outcomes.failureCount > 0 { - // Some inputs failed; if the toolchain couldn't be verified, hint once - // that a Swift-version mismatch may be behind the build errors above. - if let hint = RunCommandError.toolchainHint(runner.toolchainVerification) { - FileHandle.standardError.write(Data(hint.utf8)) - } - throw RunCommandError.failed - } - } - - /// Prints per-input diagnostics for a batch: each file's compiler stderr - /// (fenced with a `---- path ----` header), then for any non-`.renderFailed` - /// failure (whose stderr was just surfaced) a one-line `path: error`. Spawn/ - /// write failures carry their diagnostic in the error itself. - private static func reportOutcomeDiagnostics(_ outcomes: [FileOutcome]) { - for outcome in outcomes { - if !outcome.stderr.isEmpty { - FileHandle.standardError.write(Data("---- \(outcome.input.path) ----\n".utf8)) - FileHandle.standardError.write(Data(outcome.stderr.utf8)) - } - // .renderFailed already had its stderr surfaced above. Other failures - // (process spawn, write) carry the diagnostic in the error itself. - if let error = outcome.result { - if case .renderFailed = error { - continue - } - FileHandle.standardError.write(Data("\(outcome.input.path): \(error)\n".utf8)) - } + throw RunCommandError.unexpected(error) } } } diff --git a/Sources/skit/RunCommandError.swift b/Sources/skit/RunCommandError.swift index bc075c3..55db1f9 100644 --- a/Sources/skit/RunCommandError.swift +++ b/Sources/skit/RunCommandError.swift @@ -126,19 +126,15 @@ internal enum RunCommandError: Error { } } - /// Maps the `RunError` from `Runner.renderDirectory` onto the CLI's exit - /// policy. Unlike the single-file path, `renderDirectory` only wraps - /// directory-walk failures in `.unexpected` (carrying `input` for the - /// "failed to walk" framing), and never throws `.renderFailed` — that's a - /// per-file outcome — so that case is defensive. - internal init(renderDirectoryError error: RunError, input: String) { + /// Maps the `CollectInputsError` from `FileManager.collectInputs` onto the + /// CLI's exit policy. Both cases are bulk directory-walk failures, carrying + /// `input` for the "failed to walk" framing the CLI prints. + internal init(collectInputsError error: CollectInputsError, input: String) { switch error { - case .invalidInput(let message): - self = .usage(message) - case .unexpected(let underlying): + case .cliError(let underlying): + self = .directoryWalkFailed(input: input, underlying: underlying) + case .resourceValuesFailure(let underlying): self = .directoryWalkFailed(input: input, underlying: underlying) - case .renderFailed: - self = .failed } } diff --git a/Tests/SyntaxKitTests/Unit/Execution/RunnerBatchRenderTests.swift b/Tests/SyntaxKitTests/Unit/Execution/RunnerBatchRenderTests.swift new file mode 100644 index 0000000..c29920f --- /dev/null +++ b/Tests/SyntaxKitTests/Unit/Execution/RunnerBatchRenderTests.swift @@ -0,0 +1,124 @@ +// +// RunnerBatchRenderTests.swift +// SyntaxKit +// +// Created by Leo Dion. +// Copyright © 2026 BrightDigit. +// +// Permission is hereby granted, free of charge, to any person +// obtaining a copy of this software and associated documentation +// files (the “Software”), to deal in the Software without +// restriction, including without limitation the rights to use, +// copy, modify, merge, publish, distribute, sublicense, and/or +// sell copies of the Software, and to permit persons to whom the +// Software is furnished to do so, subject to the following +// conditions: +// +// The above copyright notice and this permission notice shall be +// included in all copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, +// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +// OTHER DEALINGS IN THE SOFTWARE. +// + +import Foundation +import Testing + +@testable import SyntaxKit + +/// Exercises the filesystem-free batch entry point `Runner.render(sources:)`: +/// it reads nothing and writes nothing, returning a `FileOutcome` per input +/// carrying the rendered `stdout` (or a `RunError`). Disabled on WASI because +/// the internal temp-wrapper write the render still performs is unreliable +/// there (see `RunnerRenderFailureTests`). +@Suite internal struct RunnerBatchRenderTests { + /// Marker a source carries to make the stub backend fail that input. + private static let failMarker = "// FAIL" + + /// A runner whose stub backend reads the wrapped program and reports a + /// non-zero exit for any input whose source contains `failMarker`, and exit 0 + /// (with a fixed rendered payload) otherwise — so a single call can mix + /// successes and failures without spawning `swift`. + private func runner(toolchain: ToolchainVerification = .verified) -> Runner { + Runner( + libPath: "/does/not/matter", + cache: nil, + timeoutSeconds: 0, + toolchainVerification: toolchain + ) { invocation in + let wrapped = (try? String(contentsOfFile: invocation.wrappedPath, encoding: .utf8)) ?? "" + if wrapped.contains(Self.failMarker) { + return .completed(ProcessResult(exitCode: 1, stdout: Data(), stderr: "boom\n")) + } + return .completed(ProcessResult(exitCode: 0, stdout: Data("RENDERED".utf8), stderr: "")) + } + } + + private func input(_ name: String, _ source: String) -> RenderInput { + RenderInput(url: URL(fileURLWithPath: "/in/\(name)"), source: source) + } + + @Test("An empty input set yields no outcomes without touching the backend") + internal func emptyYieldsEmpty() async { + let outcomes = await runner().render(sources: []) + #expect(outcomes.isEmpty) + } + + @Test( + "Every successful input returns its rendered stdout with a nil result", + .disabled(if: Platform.isWASI, "Render needs host filesystem/subprocess; not on WASI") + ) + internal func successesCarryStdout() async { + let outcomes = await runner().render(sources: [ + input("a.swift", "let a = 1\n"), + input("b.swift", "let b = 2\n"), + ]) + + #expect(outcomes.count == 2) + #expect(outcomes.failureCount == 0) + for outcome in outcomes { + #expect(outcome.result == nil) + #expect(outcome.stdout == Data("RENDERED".utf8)) + } + // The input URLs are preserved so the caller can reroot each output. + #expect(Set(outcomes.map(\.input.lastPathComponent)) == ["a.swift", "b.swift"]) + } + + @Test( + "A failing input is isolated: peers still render, failureCount reflects it", + .disabled(if: Platform.isWASI, "Render needs host filesystem/subprocess; not on WASI") + ) + internal func failureIsIsolated() async { + let outcomes = await runner().render(sources: [ + input("ok.swift", "let ok = 1\n"), + input("bad.swift", "\(Self.failMarker)\nlet bad = 2\n"), + ]) + + let byName = Dictionary( + uniqueKeysWithValues: outcomes.map { ($0.input.lastPathComponent, $0) } + ) + + #expect(outcomes.count == 2) + #expect(outcomes.failureCount == 1) + + let okOutcome = try? #require(byName["ok.swift"]) + #expect(okOutcome?.result == nil) + #expect(okOutcome?.stdout == Data("RENDERED".utf8)) + + let badOutcome = try? #require(byName["bad.swift"]) + #expect(badOutcome?.stdout.isEmpty == true) + guard case .renderFailed(let exitCode, let stderr, let toolchain)? = badOutcome?.result else { + Issue.record("expected .renderFailed for the marked input") + return + } + #expect(exitCode == 1) + #expect(stderr == "boom\n") + #expect(toolchain == .verified) + } +} diff --git a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift index 0b4e619..f4c18c8 100644 --- a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift +++ b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift @@ -33,18 +33,13 @@ import Testing @testable import SyntaxKit @Suite internal struct RunnerRenderFailureTests { - /// Runs `renderFile` against a throwaway input with a stub backend that - /// always reports a non-zero exit, so the render fails deterministically - /// without spawning `swift`. Returns the thrown `RunError`, or nil if the - /// call unexpectedly succeeded. + /// Runs `render(source:originalPath:)` against in-memory source with a stub + /// backend that always reports a non-zero exit, so the render fails + /// deterministically without spawning `swift`. Returns the thrown `RunError`, + /// or nil if the call unexpectedly succeeded. private func renderFailure( toolchain: ToolchainVerification ) async -> RunError? { - let input = FileManager.default.temporaryDirectory - .appendingPathComponent("skit-input-\(UUID().uuidString).swift") - try? Data("let x = 1\n".utf8).write(to: input) - defer { try? FileManager.default.removeItem(at: input) } - let runner = Runner( libPath: "/does/not/matter", cache: nil, @@ -55,7 +50,7 @@ import Testing } do { - _ = try await runner.renderFile(input: input.path) + _ = try await runner.render(source: "let x = 1\n", originalPath: "/tmp/input.swift") return nil } catch { return error From 6d6301b9f3810a46c69a7c1c6e9ec5d0d194aeb1 Mon Sep 17 00:00:00 2001 From: Leo Dion Date: Wed, 10 Jun 2026 18:01:42 -0400 Subject: [PATCH 56/56] Make Runner.render(source:) originalPath optional for anonymous snippets originalPath is only a diagnostic label (#sourceLocation + stderr path-rewriting), so default it to nil and resolve the no-path case to a synthetic "source.swift" label in processFile. Lets a developer render a bare string of SyntaxKit DSL without inventing a path. RunnerRenderFailureTests now exercises the no-originalPath call. Follow-up #164 tracks revisiting the synthetic-label choice. Co-Authored-By: Claude Opus 4.8 (1M context) --- Sources/SyntaxKit/Execution/Runner.swift | 18 +++++++++--------- .../Execution/RunnerRenderFailureTests.swift | 3 ++- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/Sources/SyntaxKit/Execution/Runner.swift b/Sources/SyntaxKit/Execution/Runner.swift index 5bd0a71..6da6170 100644 --- a/Sources/SyntaxKit/Execution/Runner.swift +++ b/Sources/SyntaxKit/Execution/Runner.swift @@ -36,7 +36,6 @@ public import Foundation // 4. processFile (per input) — cache lookup → wrap → spawn → cache store // 5. wrap — hoist imports, wrap body in Group { … }, #sourceLocation // 6. runSwift — spawn `swift` with timeout watchdog -// See Docs/skit.md for design rationale and trade-offs. /// Renders SyntaxKit DSL inputs into Swift source. `Runner` is the SDK-shaped /// entry point: methods return rendered data (`SingleFileRender`) or @@ -76,8 +75,8 @@ public struct Runner: Sendable { /// the real result. Carried onto `RunError.renderFailed`. public let toolchainVerification: ToolchainVerification - /// Creates a runner bound to a lib directory, an optional output cache, a - /// per-input timeout, and the backend closure that spawns `swift`. + /// Creates a runner bound to a lib dir, optional output cache, per-input + /// timeout, and the backend closure that spawns `swift`. public init( libPath: String, cache: OutputCache?, @@ -99,15 +98,14 @@ public struct Runner: Sendable { /// Renders one in-memory input and returns the rendered bytes plus any /// compiler diagnostics. The SDK reads no input and writes no output; /// `originalPath` is only a diagnostic label (`#sourceLocation` + stderr - /// path-rewriting), and the caller decides where the result goes. + /// path-rewriting) — pass `nil` for an anonymous snippet. /// On a non-zero subprocess exit, throws `RunError.renderFailed` carrying the /// toolchain's diagnostic. Any Foundation/Subprocess failure (the internal /// temp-wrapper write, spawn) is wrapped in `RunError.unexpected`. public func render( source: String, - originalPath: String + originalPath: String? = nil ) async throws(RunError) -> SingleFileRender { - // `processFile` may hit the output cache and skip the spawn; same shape. let result: ProcessResult do { result = try await processFile(source: source, originalPath: originalPath) @@ -135,9 +133,11 @@ public struct Runner: Sendable { /// from the caller; `originalPath` is only a `#sourceLocation`/stderr label /// (never opened). The temp wrapper is created in a per-run tmp dir and torn /// down by `defer`. `internal` so `Runner+Directory.swift` can reuse it. - internal func processFile(source: String, originalPath: String) async throws -> ProcessResult { - // The label diagnostics map back to; standardized to match `#sourceLocation`. - let absoluteInputPath = URL(fileURLWithPath: originalPath).standardizedFileURL.path + internal func processFile(source: String, originalPath: String?) async throws -> ProcessResult { + // Diagnostics label: standardize a given path, else a synthetic snippet name. + let absoluteInputPath = + originalPath.map { URL(fileURLWithPath: $0).standardizedFileURL.path } + ?? "source.swift" // Compute the output cache key (nil under `--no-cache` or when the cache // root couldn't be derived at startup). Mixes input bytes, toolchain diff --git a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift index f4c18c8..e6aac68 100644 --- a/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift +++ b/Tests/SyntaxKitTests/Unit/Execution/RunnerRenderFailureTests.swift @@ -50,7 +50,8 @@ import Testing } do { - _ = try await runner.render(source: "let x = 1\n", originalPath: "/tmp/input.swift") + // Omit `originalPath` — exercises the optional anonymous-snippet path. + _ = try await runner.render(source: "let x = 1\n") return nil } catch { return error