Diagnose excessive expression nesting before the compiler stack overflows

[LunaStev]

Problem

wavec check aborts on a small source containing deeply nested parentheses instead of producing a diagnostic. This terminates the compiler process and prevents structured error reporting. Start in front/parser/src/expr/primary.rs and front/parser/src/expr/mod.rs.

Reproduction

Audited on canonical master b384e64c21257a87a1afcd8b43ba28b8c03dcfe4 (2026-09-30). The local checkout used for reproductions has the identical tracked tree. Linux amd64, Python 3.14.7.

Executed after LLVM_SYS_211_PREFIX=/usr/lib64/llvm21 cargo +1.89 build --locked --jobs 2 --bin wavec (Rust 1.89, LLVM 21). Disable core dumps for the reproduction:

ulimit -c 0
python3 - <<'REPRO'
from pathlib import Path
n = 1000
Path('/tmp/nested.wave').write_text('fun main() -> i32 { return ' + '(' * n + '0' + ')' * n + '; }')
REPRO
target/debug/wavec check /tmp/nested.wave --std-root "$PWD/std"

100 parentheses passed. At 1,000 and 5,000 the subprocess exited with SIGABRT (returncode=-6) and thread 'main' has overflowed its stack. The failure was reproduced with the debug compiler; release-profile thresholds were not measured.

Scope

Bound recursive frontend work or make it iterative. Preserve valid expression semantics and source locations; cover cleanup/AST traversal as well as initial parsing. Difficulty: hard.

Completion criteria

• Adversarial nesting ends with a normal, located compiler diagnostic instead of stack overflow or an abort.
• Regression tests run the compiler as a subprocess so a crash cannot terminate the test harness.
• Cover parentheses, unary nesting and related expression forms near the chosen limit; ordinary inputs retain behavior and JSON diagnostics remain valid.

Design decision needed: If introducing a user-visible nesting limit, agree on the limit and diagnostic contract before implementation.

Please mention @LunaStev to discuss the design before starting implementation.