Skip to content

Portable targets: newline-only lexer mode (env-spec use case)#59

Open
johnsoncodehk wants to merge 2 commits into
masterfrom
portable-newline-mode
Open

Portable targets: newline-only lexer mode (env-spec use case)#59
johnsoncodehk wants to merge 2 commits into
masterfrom
portable-newline-mode

Conversation

@johnsoncodehk

Copy link
Copy Markdown
Owner

Summary

Adds newline: NewlineConfig support to the portable emit targets (TS/Go/Rust), so newline-sensitive grammars (dotenv / env-spec flavour, issue #10 lineage) can now be emitted — previously all three targets threw pattern 'never' unsupported on the engine-emitted NEWLINE token.

  • IR: ParserIR.newlineCfg carries { token, flowOpen, flowClose, comment }; the never() NEWLINE token is excluded from the lexer match list (it is engine-emitted, never text-matched). Grammars declaring indent still throw (out of scope).
  • Lexers (×3, symmetrical): a line-boundary state machine mirroring gen-lexer.ts newline-only semantics — exactly one NEWLINE per significant boundary, leading-boundary suppression, blank-line folding, comment-only lines skipped, flow ( … ) suspension via flowDepth.
  • rd alt ordering: for newline grammars only, prefix-ambiguous rd alts are reordered longest-first at the IR level so first-match backtracking agrees with the interpreter's longest-match (e.g. Ident vs Ident '(' … ')'); the shared rdRule template is untouched.
  • Gate: new test/fixtures/envspec.ts fixture + 12 cases (8 accept / 4 reject) in test/portable-targets.ts, green on all three targets.

Grammars without newline are provably unaffected: emitted calc sources at this commit are byte-identical to f6fccce for all three targets.

Validation

  • node test/portable-targets.ts — envspec 8/8 + 4/4 on ts/go/rust; javascript 42/42 + 11/11, typescript 13/13 + 4/4, calc 21/21 + 7/7 baselines unchanged
  • node test/check.ts emit-parity — 5/5
  • node test/newline-mode.ts — 29/29 (interpreter-side regression)
  • npx tsc --noEmit — clean
  • Adversarial parity sweep (24 extra inputs beyond the gate: CRLF, bare CR, nested/unbalanced flow, comment/blank lines inside flow, EOF-boundary shapes) — oracle ≡ emitted TS on all

johnsoncodehk and others added 2 commits July 11, 2026 04:57
Wire engine-emitted NEWLINE tokens through IR and target lexers; add envspec
fixture to portable-targets gate. Reject indent+newline combos; reorder rd
alts longest-first for newline grammars to match interpreter prefix ambiguity.

Co-authored-by: Cursor <cursoragent@cursor.com>
The interpreter's newline-only line scanner treats only LF/CR as line
terminators (a bare LS/PS hits the unexpected-character throw); the
portable boundary/ws code paths accepted U+2028/U+2029 as boundaries,
diverging on inputs like 'A=1\u2028B=2' (oracle reject, portable accept).
Planner adversarial probe caught it; all three targets now fall through
to the standard throw.
@johnsoncodehk

Copy link
Copy Markdown
Owner Author

Follow-up 4193aa6: planner adversarial review caught an LS/PS divergence — the executor's boundary/ws scanner treated U+2028/U+2029 as line terminators, but the interpreter's newline-only mode recognises only LF/CR (LS/PS hits the unexpected-character throw), so A=1\u2028B=2 was oracle-reject / portable-accept. All three targets now match (probe re-run: parity on all 24 adversarial inputs + 3 LS/PS witnesses; full gate, emit-parity 5/5, newline-mode 29/29, tsc, and calc byte-identity all re-verified green).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant