-
Notifications
You must be signed in to change notification settings - Fork 2
skit: SyntaxKit codegen CLI for v0.0.5 #156
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
56 commits
Select commit
Hold shift + click to select a range
67107ea
Add research docs for SyntaxKit-driven codegen CLI (#154)
leogdion e492503
POC step 1: wrap+spawn flow works at ~720ms cold (#154)
leogdion cf9f02a
Add POC step 1 reproducer script (#154)
leogdion 840cb0c
Release-config dylib size: 9.3 MB stripped (#154)
leogdion 9e5affa
POC step 2: skitrun CLI wraps + spawns SyntaxKit DSL inputs (#154)
leogdion dd209c9
POC step 3: skitrun folder mode (#154)
leogdion c6daced
POC step 4: self-contained skitrun release bundle (#154)
leogdion 3d3b631
POC step 5: Helpers/ discovery + per-toolchain cache (#154)
leogdion 2adc3cd
POC step 6: rendered-output cache + --no-cache (#154)
leogdion 68e49f0
Note web-server form as a post-CLI follow-up (#154)
leogdion 65f2103
POC step 7: Linux smoke test + Foundation.Process workarounds (#154)
leogdion 2836cf7
Add skitrun README scoped to the target dir (#154)
leogdion c6d9e7d
Fix skitrun release bundle and update blackjack example
leogdion 84b497b
Update Examples/Completed/*/dsl.swift to current API + skitrun shape
leogdion 207cb7e
Add --timeout watchdog to skitrun
leogdion a7d8876
Stamp + detect toolchain mismatch at skitrun startup
leogdion 47e5be8
Update 3 more Examples/Completed/*/dsl.swift to current API
leogdion b0008fc
Move enum_generator out of Examples/Completed/
leogdion 18f7097
Unify skit + skitrun into one binary with ArgumentParser subcommands
leogdion f3a1912
Productize Sources/skit/README.md for v0.0.5
leogdion 05e8502
Replace POC research log with Docs/skit.md
leogdion e3e243f
Replace swift-crypto/SHA-256 with pure-Swift FNV-1a content hash
leogdion 61349da
Move skit to swift-subprocess; revert simulator OS pin to 26.4
leogdion 6758822
Fixing CI
leogdion fbcc8fa
Update OS version to 26.5 in SyntaxKit.yml
leogdion 89fad36
Fix two PR review regressions in Examples/Completed dsl files [skip ci]
leogdion 824fb3c
Narrow Subprocess guard in skit and split subcommands into their own …
leogdion 74f6c6a
Add lifecycle map and inline phase comments across Sources/skit/ [ski…
leogdion 0e25069
Document no-globals convention and add globals audit [skip ci]
leogdion 358c83b
Split skit Runner types into separate files [skip ci]
leogdion 9f6c642
Lift resolveLibPath onto Bundle and isLibDir onto FileManager [skip ci]
leogdion 800d49a
Remove Helpers feature from skit [skip ci]
leogdion 5083d2b
Lift more skit globals into types/extensions and refactor OutputCache…
leogdion 6058145
Capture swift --version once per skit run [skip ci]
leogdion fb22c25
Move skit Toolchain globals into types/extensions [skip ci]
leogdion 083058a
Move toolchainMismatchMessage onto Skit.Run [skip ci]
leogdion a6bb156
Wrap skit Runner free functions in a Runner type [skip ci]
leogdion 2f7cbc3
Remove globals-audit.md [skip ci]
leogdion dd2c3b4
Extract skit Runner wrap, timeout, and swift invocation [skip ci]
leogdion e7ced53
Move skit pure-logic helpers into SyntaxKit; decouple Runner from Arg…
leogdion f7a144b
Move render engine into SyntaxKit; inject Subprocess backend via clos…
leogdion eb460af
Reshape Runner into an SDK-style API; move CLI presentation to skit […
leogdion 3f80dde
Refactor renderDirectory into phase helpers; move collectInputs to Fi…
leogdion 5b06d5f
Move writeOutput onto FileManager; fold render result functionally
leogdion d8d5345
Fix lint: split Skit+Run, drop dead FileOutcome.destination, fix #req…
leogdion 5bc67c8
Refactor render-engine error plumbing and consolidate FileManager hel…
leogdion e2cc934
Extract string constants for skit CLI/swiftc flags and Execution paths
leogdion 0565caa
Refactor skit run() error handling and abstract the Subprocess backend
leogdion 80c9bb9
Fix CI; make Execution engine API public
leogdion a5f20b0
Fix skit render, address review findings, make hashing pluggable
leogdion fdbf8da
Split FileManager filesystem helpers from domain logic; type FileOutc…
leogdion 57fbfc4
Fix non-Apple build; hoist nested types; unify build scripts
leogdion f1b98b6
Fix Windows + WASM CI test failures (filesystem semantics)
leogdion 5a1d9b8
Map render RunErrors via RunCommandError initializers; cut skit compl…
leogdion 49130b9
Make SyntaxKit render methods filesystem-free; move IO into skit (#163)
leogdion 6d6301b
Make Runner.render(source:) originalPath optional for anonymous snippets
leogdion File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,7 +16,6 @@ jobs: | |
| fail-fast: false | ||
| matrix: | ||
| container: | ||
| - swift:6.0 | ||
| - swift:6.1 | ||
| - swift:6.2 | ||
| - swift:6.3 | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,177 @@ | ||
| # `skit` — a SyntaxKit CLI for config-driven Swift codegen | ||
|
|
||
| `skit` is a small CLI that takes a SyntaxKit DSL file as input and writes Swift source out the other side. The vision: pure data — JSON, YAML, your own format — drives a manifest written in the SyntaxKit DSL, and `skit` materializes it into idiomatic Swift you check in alongside everything else. Less hand-maintenance, fewer drift bugs. | ||
|
|
||
| This doc walks through how `skit` is built. Two verbs, one wrap-and-spawn pipeline, an output cache, and a careful toolchain story underneath. Where there are sharp edges, this doc names them. | ||
|
|
||
| ## Two verbs | ||
|
|
||
| ``` | ||
| skit run Input.swift # SyntaxKit DSL → Swift source on stdout | ||
| skit run Input.swift -o Out.swift | ||
| skit run InputDir/ -o OutDir/ # walk **/*.swift and mirror rendered output | ||
| skit parse < Input.swift # Swift source → JSON syntax tree | ||
| ``` | ||
|
|
||
| `run` is the default subcommand. `skit Input.swift` is shorthand for `skit run Input.swift`. `parse` is a one-shot for the inverse direction (Swift source → SwiftSyntax tree as JSON) — useful for tooling that wants to introspect existing code. | ||
|
|
||
| The remainder of this doc is about `run`, which is where the interesting design decisions live. | ||
|
|
||
| ## How `skit run` works | ||
|
|
||
| A `.swift` input file looks like a SyntaxKit DSL expression at the top level: | ||
|
|
||
| ```swift | ||
| // Models.swift | ||
| Struct("Person") { | ||
| Variable(.let, name: "name", type: "String") | ||
| Variable(.let, name: "age", type: "Int") | ||
| } | ||
| ``` | ||
|
|
||
| The input is not a complete Swift program. It has no `@main`, no `print`, no `let root = …`. `skit run` adds the boilerplate by wrapping the input in a `Group { … }` builder and a top-level `print` that renders the result: | ||
|
|
||
| ```swift | ||
| // What `skit run` writes to a temp file before spawning `swift`: | ||
| import SyntaxKit | ||
|
|
||
| let __skit_root = Group { | ||
| #sourceLocation(file: "/path/to/Models.swift", line: 3) | ||
| Struct("Person") { | ||
| Variable(.let, name: "name", type: "String") | ||
| Variable(.let, name: "age", type: "Int") | ||
| } | ||
| #sourceLocation() | ||
| } | ||
|
|
||
| print(__skit_root.generateCode()) | ||
| ``` | ||
|
|
||
| `skit` then spawns `swift` (the script-mode interpreter, not `swiftc`) on this wrapped file, captures stdout, and writes the result to the user's chosen output target. | ||
|
|
||
| Three things deserve a closer look: | ||
|
|
||
| **Imports get hoisted.** The wrapper has to start with `import SyntaxKit` so the DSL types are available. Any additional `import`s the user writes need to live at the top of the file, not inside `Group { … }`. `skit` parses the input with SwiftSyntax, peels off the leading `import` declarations, and lifts them into the wrapper preamble. Anything else (declarations, expressions, top-level types) stays in the body. | ||
|
|
||
| **`#sourceLocation` keeps diagnostics readable.** When the spawned `swift` emits a compile error, it reports a line number in the wrapped temp file, which is meaningless to the user. The `#sourceLocation` directive remaps body diagnostics back to the original input path and line. Errors in the wrapper preamble (the `import` block, the `Group { … }` opening) still reference the temp file — `skit` rewrites occurrences of the temp path in stderr to the input path as a fallback, so users see something coherent. | ||
|
|
||
| **`swift` runs in script mode.** Running `swift Input.swift` invokes the Swift interpreter rather than going through `swiftc` + `ld`. Cold-start is around 700ms on macOS; warm spawns are around 110ms. The CLI's hot path leans into this — for batch input via `skit run InputDir/`, we spawn one `swift` per input file in parallel up to the active core count. | ||
|
|
||
| ## Output cache | ||
|
|
||
| One layer, keyed by content hash. Lives under `~/Library/Caches/com.brightdigit.SyntaxKit/outputs/<sha>/output.swift` on macOS, `$XDG_CACHE_HOME/syntaxkit/outputs/<sha>/output.swift` (or `~/.cache/syntaxkit/outputs/<sha>/output.swift`) on Linux. On hit, `skit` skips the `swift` spawn for an input entirely. | ||
|
|
||
| The fully-rendered output of an input gets cached by a hash of (input bytes, libSyntaxKit stamp, swift version, sorted `SKIT_*`/`SYNTAXKIT_*` env vars). On hit, total wall time is around 0.14s, dominated by hash + file read. Cold miss matches the warm script-mode baseline (~0.5s). | ||
|
|
||
| `--no-cache` skips the cache. | ||
|
|
||
| ## Toolchain stamping | ||
|
|
||
| Pre-compiled Swift modules (`SyntaxKit.swiftmodule`) are tightly coupled to the compiler that built them. Even patch-level differences fail to load: | ||
|
|
||
| ``` | ||
| error: module compiled with Swift 6.3 cannot be imported by the Swift 6.3.2 compiler | ||
| ``` | ||
|
|
||
| If `skit`'s bundled `lib/SyntaxKit.swiftmodule` doesn't match the user's `swift`, the spawned interpreter emits exactly this diagnostic and refuses to compile the wrapped input. The user is left staring at a cryptic message that doesn't name the actual problem. | ||
|
|
||
| `skit` mitigates this by recording the build toolchain at bundle time. `Scripts/build-skit.sh` writes `lib/swift-version.txt` containing the output of `swift --version`. On startup, `skit run` reads the stamp and compares it to a freshly-captured local `swift --version`. Three paths: | ||
|
|
||
| - **Match.** Proceed silently. | ||
| - **Mismatch.** Print a clear error naming both versions and the rebuild command, exit 2. Skip with `--no-toolchain-check`. | ||
| - **Missing stamp.** Print a one-line note and proceed. Older bundles built before this check existed shouldn't break. | ||
|
|
||
| The comparison is exact-string match. Patch-level drift broke the originating bug, so anything less strict would just defer the failure to the cryptic message we're trying to avoid. | ||
|
|
||
| **What this doesn't do**: it doesn't fix the mismatch, just surfaces it. The fix is to re-run the release script with the user's current toolchain. The auto-rebuild path — bundle SyntaxKit sources alongside the prebuilt module and recompile transparently when the stamp doesn't match — is tracked as [issue #157](https://github.com/brightdigit/SyntaxKit/issues/157). | ||
|
|
||
| ## Timeout watchdog | ||
|
|
||
| The spawned `swift` is the only unbounded piece of `skit run`'s hot path. The wrap step is microseconds. The output cache hits or misses in milliseconds. But the spawn itself runs *user code* — and that code is allowed to be arbitrarily slow, recursive, or stuck. | ||
|
|
||
| `skit run` defaults to a 60s per-input timeout. On expiry it sends `SIGTERM`, gives a 5s grace, then `SIGKILL`. The wrapped input exits with code 124 — POSIX `timeout(1)`'s convention. `--timeout <s>` overrides the default; `--timeout 0` disables the watchdog entirely (useful for debugging genuinely long codegen). | ||
|
|
||
| The implementation is `DispatchSemaphore.wait(timeout: deadline)` paired with a `process.terminationHandler` that signals on child exit. The Linux Foundation `Process.waitUntilExit()` hangs on already-exited children on some configurations, which is why `skit` uses the semaphore-based wait everywhere. Same story for pipe drains — sequential reads after the child exits can deadlock when either pipe (~64 KB buffer on Linux) fills before exit, so both pipes drain concurrently via `DispatchGroup`. | ||
|
|
||
| ## Sharp edges | ||
|
|
||
| ### `if`-in-`Group` crashes the Swift type-checker — [#158](https://github.com/brightdigit/SyntaxKit/issues/158) | ||
|
|
||
| `CodeBlockBuilderResult` declares all the methods needed for `if`/`else` in a result builder (`buildEither`, `buildOptional`, `buildArray`), but using them surfaces a Swift type-checker bug: | ||
|
|
||
| ```swift | ||
| let _ = Group { | ||
| if true { // ← error: failed to produce diagnostic for expression | ||
| Struct("A") { … } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| This is a Swift compiler bug, not a `skit` bug. The workaround is to hoist the conditional into a plain Swift function that uses **plain Swift `if`/`else`** (not a `Group { if … }` body) to return one of two `CodeBlock`s: | ||
|
|
||
| ```swift | ||
| func optionalDebugField(_ include: Bool) -> any CodeBlock { | ||
| if include { | ||
| return Variable(.let, name: "debug", type: "Bool") | ||
| } else { | ||
| return Group {} // empty Group as the "no-op" branch — there's no | ||
| // public EmptyCodeBlock type | ||
| } | ||
| } | ||
|
|
||
| Struct("Config") { | ||
| Variable(.let, name: "name", type: "String") | ||
| optionalDebugField(buildIsDebug) | ||
| } | ||
| ``` | ||
|
|
||
| The helper function itself can't use `Group { if … }` either — same crash. Plain Swift control flow only. | ||
|
|
||
| ### `@main` and top-level decl attributes don't work | ||
|
|
||
| `@main` and other decl attributes (like top-level `@available`) at the start of the input fail to compile. The wrapper places the user's body inside `Group { … }`, where these attributes try to bind to a function-call expression rather than a declaration. Examples: | ||
|
|
||
| ```swift | ||
| @main // ❌ error: expected declaration | ||
| Struct("Person") { … } | ||
|
|
||
| @available(iOS 17, *) // ❌ error: expected declaration | ||
| Struct("ModernView") { … } | ||
| ``` | ||
|
|
||
| The DSL has its own mechanism for attribute attachment — call `.attribute("Published")`, `.attribute("available", arguments: ["iOS 17", "*"])` on the `Variable`/`Struct`/etc. — which renders the attribute in the *output*, not on the DSL expression itself. That's the path users should take. | ||
|
|
||
| ### Comments don't carry through | ||
|
|
||
| Swift comments in the input file (`// MARK: - Models`, block comments, inline `//` after a DSL line) don't render to the output. The input's comments are *only* for the input's author. To emit a comment in the rendered code, attach it to a `CodeBlock` via `.comment { Line(.doc, "…") }`. | ||
|
|
||
| ### `#if os(...)` is gen-time, not output | ||
|
|
||
| A `#if os(macOS) … #endif` block in the input is evaluated when the wrapped file compiles, controlling whether the enclosed DSL expressions are part of the builder. It does *not* emit a `#if os(macOS)` directive into the rendered Swift output. If you want compile-time-conditional output, emit the `#if` lines as raw text via `VariableExp("#if os(macOS)")` (or write a small helper). | ||
|
|
||
| ## Platform notes | ||
|
|
||
| **macOS** is the primary target. The build and release flows live in `Scripts/`; the bundle is portable across machines with the same Swift version. | ||
|
|
||
| **Linux** is verified on `swift:6.0-jammy/aarch64`. One adjustment compared to macOS: the Mach-O `install_name` rewrite in `Scripts/build-skit.sh` is skipped — GNU `ld` doesn't accept the flag. The `-rpath` injection (which is what actually locates the dylib at runtime) works on both platforms. | ||
|
|
||
| The Foundation.Process workarounds described earlier are Linux-driven. `Process.waitUntilExit()` blocks indefinitely on already-exited children on `swift:6.0-jammy/aarch64` — `skit` uses `DispatchSemaphore` everywhere a child wait is needed, and drains stdout/stderr pipes concurrently to avoid deadlocks when either pipe buffer fills. | ||
|
|
||
| **Windows** is not supported. | ||
|
|
||
| ## What's deferred | ||
|
|
||
| A few things were considered for v1 and explicitly punted: | ||
|
|
||
| - **Auto-rebuild on toolchain mismatch** — [#157](https://github.com/brightdigit/SyntaxKit/issues/157). Today the user gets a clear error and a rebuild command; tomorrow `skit` should rebuild `libSyntaxKit` from bundled sources transparently and cache the result per Swift version. The stamp-and-detect path shipped in this release is the foundation; the rebuild fallback is the natural next step. | ||
| - **Multi-file output from a single input** — out of scope. The folder mode handles N-inputs → N-outputs; if you need fan-out from one logical generator, split it into multiple input files. | ||
| - **Sandboxing the spawned `swift`** — out of scope. The threat model is "you ran your own code", same as `swift Input.swift` by hand. | ||
| - **HTTP / web-server form** — out of scope for the CLI. A long-lived server could share a warm interpreter and the caches across requests, but that's a different shape with its own isolation questions. Revisit later. | ||
|
|
||
| ## Reference | ||
|
|
||
| - [`Sources/skit/README.md`](../Sources/skit/README.md) — per-target quick reference (flag table). | ||
| - [`Scripts/build-skit.sh`](../Scripts/build-skit.sh) — release-bundle builder (use `--debug` for the fast iteration variant). | ||
| - [`Docs/research/tuist-manifest-pipeline.md`](research/tuist-manifest-pipeline.md) — the manifest-pipeline pattern this CLI borrows from. | ||
| - [Issue #154](https://github.com/brightdigit/SyntaxKit/issues/154) — original tracking issue. | ||
| - [Issue #157](https://github.com/brightdigit/SyntaxKit/issues/157), [Issue #158](https://github.com/brightdigit/SyntaxKit/issues/158) — follow-ups. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,10 @@ | ||
| Class("Foo") { | ||
| Variable(.var, name: "bar", type: "String", defaultValue: "bar").attribute("Published") | ||
| Variable(.var, name: "bar", type: "String", equals: "bar").attribute("Published") | ||
| Function("bar") { | ||
| print("bar") | ||
| Call("print") { | ||
| ParameterExp(unlabeled: Literal.string("bar")) | ||
| } | ||
| }.attribute("available", arguments: ["iOS 17.0", "*"]) | ||
| Function("baz") { | ||
| }.attribute("objc")}.attribute("objc") | ||
| }.attribute("objc") | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could this be .build*