Repository navigation
[Enhancement] Split oversized UCI generated compilation units safely #92
Description
Activity
8 GiB WSL bounded compile follow-up (2026-09-28)
WSL reported 7.7 GiB total, about 5.3 GiB available at test start, and 2 GiB swap. Commands ran serially in
systemd-run --user --scopewithMemoryMax=3500M,MemorySwapMax=0, and a 10-minute timeout. The cap held; host swap remained essentially unused.- Corrected four-schema UCI C++ header: 4,013,901 bytes.
g++ -std=c++20 -fsyntax-only -fmax-errors=5passed in 33 s, peak RSS 2,780,252 KiB. An actualg++ -std=c++20 -O0 -cobject build also passed in 40 s, peak RSS 2,489,260 KiB. This compiled an includer TU for the full generated header. - Rust models-only (
codecs = false,zero_copy = false): 5,098,673 bytes. Serialcargo checkreached the 3.5 GiB cgroup cap and was killed after 73 s. Retrying withCARGO_PROFILE_DEV_DEBUG=0andCARGO_INCREMENTAL=0also reached the cap after 74 s. No generated-code diagnostic appeared before the cap. - Rust default codec output: 34,293,538 bytes / 804,623 lines. Serial
cargo checkreached the same cap after 42 s. The temporary test crate also lacked apolyxmldependency, so its early unresolved-import diagnostic is a harness setup issue; this run cannot establish whether the generated Rust compiles.
Practical conclusion for this 7.7 GiB WSL setup: keep serial compiler jobs at or below about 3.5 GiB (roughly 45% of RAM) with swap disabled for their cgroup. C++ is confirmed to compile at that cap. Rust needs source splitting or a larger isolated machine before we can claim a full UCI compile passes. The bound prevented a host OOM in these runs.
- Corrected four-schema UCI C++ header: 4,013,901 bytes.
- added 2 commits that reference this issue
on Sep 30, 2026 Resolved in commits
629917cand50702ed.Implementation Summary
-
Topological SCC Condensation Chunker (
crates/polyxml-core/src/ir/chunker.rs):- Uses Tarjan's strongly connected components algorithm to collapse any cyclic type dependencies into atomic super-nodes.
- Constructs the condensation DAG and performs deterministic Kahn topological sort.
- Partitions types into bounded chunks (
chunk_00.rsthroughchunk_22.rsfor UCI 2.5) guaranteeing forward-only dependency flow (a type in chunk$p$ only references types in chunk$q$ where$q \le p$ ).
-
Codegen & Zero Cyclic Glob Resolution:
- Emits unidirectional DAG module imports (
use super::chunk_XX::*;) strictly from prior chunks. - Parent
mod.rsre-exports all chunks (pub use chunk_XX::*;), ensuring 100% public API backward compatibility. - Codec helpers (
skip_xml_element,read_element_text) are scopedpub(crate)inmod.rsand cleanly accessible by all chunks.
- Emits unidirectional DAG module imports (
-
Config & CLI Options:
- Added
--split-unitsand--chunk-size <N>topolyxml generateand manifestsplit_units = true/chunk_size = 250. - Automatically splits oversized schemas with >400 types.
- Added
-
Verification:
- Workspace test suite and clippy pass with 0 warnings.
- End-to-end integration test
test_cli_rust_split_units_and_chunkingvalidates chunk generation and downstreamcargo checkcompilation.
-
Topological SCC Condensation Chunker (
Completed via automatic topological SCC compilation unit chunking.
Observed
The four-schema UCI 2.5
defense_ucimodule generates very large single compilation units: Rust 11,797,134 bytes / 220,638 lines; Python 2,785,167 bytes; C# 2,121,694 bytes; Go 1,316,503 bytes; C++ 872,711 bytes; TypeScript 574,685 bytes. Java emits separate type files, butAboutType.javaalone is 690,302 bytes and appears to contain unrelated fields; investigate that as part of the shared IR issue before drawing compile-cost conclusions.After fixing the parser depth bug locally, Rust output grew to 34,293,538 bytes / 804,623 lines; Python grew to 8,954,411 bytes / 209,144 lines and passes
py_compile(about 597 MB peak RSS). A cappedrustfmt --checkon the Rust file aborted while constructing its diff (memory allocation of 2971725166368 bytes failed; process peak RSS about 1.0 GB). Full Rust compile time and peak memory have not been established. A full Rust consumer build may be unsafe on a small-memory workstation. Existing #55 covers a separate 600/1500-element phf benchmark; this issue tracks practical source splitting for a real multi-schema module.Done when
scripts/memcap.shor equivalent).Corpus compile gate: polyxml/polyxml-schema-corpus#1
Additional bounded runs (local, not merged)
codecs = false,zero_copy = false): 5.1 MB / 117,062 lines. A serialcargo checkreached about 2.2 GiB RSS after 42 seconds; it was stopped before the 2.2 GiB cgroup cap.g++ -fsyntax-only -fmax-errors=5produced no new diagnostic by about 2.5 GiB RSS and was stopped before the cap.Related draft PRs