Portable targets: newline-only lexer mode (env-spec use case)#59
Open
johnsoncodehk wants to merge 2 commits into
Open
Portable targets: newline-only lexer mode (env-spec use case)#59johnsoncodehk wants to merge 2 commits into
johnsoncodehk wants to merge 2 commits into
Conversation
Wire engine-emitted NEWLINE tokens through IR and target lexers; add envspec fixture to portable-targets gate. Reject indent+newline combos; reorder rd alts longest-first for newline grammars to match interpreter prefix ambiguity. Co-authored-by: Cursor <cursoragent@cursor.com>
The interpreter's newline-only line scanner treats only LF/CR as line terminators (a bare LS/PS hits the unexpected-character throw); the portable boundary/ws code paths accepted U+2028/U+2029 as boundaries, diverging on inputs like 'A=1\u2028B=2' (oracle reject, portable accept). Planner adversarial probe caught it; all three targets now fall through to the standard throw.
Owner
Author
|
Follow-up |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
newline: NewlineConfigsupport to the portable emit targets (TS/Go/Rust), so newline-sensitive grammars (dotenv / env-spec flavour, issue #10 lineage) can now be emitted — previously all three targets threwpattern 'never' unsupportedon the engine-emitted NEWLINE token.ParserIR.newlineCfgcarries{ token, flowOpen, flowClose, comment }; thenever()NEWLINE token is excluded from the lexer match list (it is engine-emitted, never text-matched). Grammars declaringindentstill throw (out of scope).gen-lexer.tsnewline-only semantics — exactly one NEWLINE per significant boundary, leading-boundary suppression, blank-line folding, comment-only lines skipped, flow( … )suspension viaflowDepth.IdentvsIdent '(' … ')'); the sharedrdRuletemplate is untouched.test/fixtures/envspec.tsfixture + 12 cases (8 accept / 4 reject) intest/portable-targets.ts, green on all three targets.Grammars without
newlineare provably unaffected: emitted calc sources at this commit are byte-identical tof6fcccefor all three targets.Validation
node test/portable-targets.ts— envspec 8/8 + 4/4 on ts/go/rust; javascript 42/42 + 11/11, typescript 13/13 + 4/4, calc 21/21 + 7/7 baselines unchangednode test/check.ts emit-parity— 5/5node test/newline-mode.ts— 29/29 (interpreter-side regression)npx tsc --noEmit— clean