Incremental GC with automatic pacing and optional scheduling hooks #372

Major collections currently stop application threads while tracing the live heap. Parallel marking reduces the duration, but larger live graphs can still produce visible hitches. The next collector should spread marking across short pauses and work automatically in ordinary hxcpp applications, including games whose frame loop is owned by another library.

The immediate priority is incremental marking. Concurrent marking in #327 is deferred while this is developed and measured.

Scope

• Add an opt-in HXCPP_GC_INCREMENTAL mode. Keep the current collector as the default during development and validation.
• Implement this in libhxcpp without requiring a Haxe compiler or standard library patch. Audit the existing generated barriers and runtime store paths before relying on them.
• Trace the actual heap while application threads are stopped, retaining marking progress between slices. Do not build and trace a second graph of copied object references.
• Keep new concurrent marking and concurrent sweeping outside this first change. Existing background allocator work remains subject to its current ownership rules.

Collection and mutation rules

1. Start a major cycle before free allocation space is exhausted. Preserve marking queues, liveness state and unprocessed work between slices.
2. Stop application threads for a bounded amount of tracing work, then resume them. Check elapsed time between small work units rather than only between entire large objects.
3. Make pointer arrays, maps and other large containers resumable. Keep owner and mutation-version information so resizing or rehashing cannot leave a saved iterator pointing into replaced storage.
4. Record reference changes between slices. Cover generated setters, array and map operations, Dynamic and reflection writes, closures and supported native stores. Handle allocations made during a cycle and references from new objects into older graphs.
5. Do not reclaim or reuse storage while unfinished marking work can still refer to it. Define early buffer release and resize behavior explicitly.
6. Preserve weak-map closure, zombies, weak references and finalizer ordering. Observable lifecycle processing happens once, after reachability is complete.
7. Define how nursery collections and allocation interact with an unfinished major cycle. Keep existing generational and moving builds as controls, and validate supported combinations individually.
8. Preserve the completion contracts of explicit collection, compaction, disabling collection, diagnostics and shutdown.

Root capture, thread-stop waits, weak processing, finalizers and finishing work must be included in the reported interruptions. A requested slice duration is a scheduling target until every participating operation has a resumable path. It is not a hard pause guarantee.

Automatic pacing

No frame loop is required. Allocation accounting, heap reserve and observed collection progress should determine when work is due. Advance the cycle at suitable allocator slow paths and safepoints, starting early enough to avoid routinely forcing a full collection under pressure.

Pace work over elapsed time as well as allocated bytes so small slices do not simply accumulate into another long interruption. If reserve is exhausted, complete safely and record the synchronous fallback and its full duration.

Without an engine hint the runtime cannot know frame boundaries or the time left before presentation. Automatic pacing must still work for servers, tools and games with no integration.

Optional library API

Provide Haxe wrappers owned by libhxcpp, backed by C++ runtime functions. No change to cpp.vm.Gc is required. Proposed operations:

• step(budgetMs): request incremental work during spare application time and report whether work remains.
• Configure the automatic slice target and collection pacing.
• Read cycle progress, pause distributions, stopped time, allocation stalls, fallback reasons and memory overhead.

Names and signatures can be settled during implementation. A game engine can spend spare frame time through these hooks; calling them is optional. Native integrations such as Lime/OpenFL need explicit barrier and lifetime coverage as part of the supported configuration.

Tests and delivery

• Add conventional semantic tests first, using test/regression/Name/{Main.hx,build.hxml,stdout.txt}. Reuse existing coverage where possible.
• Cover graph rewiring, new-to-old references, container resize and rehash, weak closure, roots, thread attachment and detachment, native writes and explicit GC lifecycle operations.
• Add deterministic slice controls and tests that mutate an already scanned object between slices. Use acknowledgements rather than timing sleeps.
• Add an independent synchronous reachability verifier before reclamation. Partial incremental marks must not hide missed objects from the verifier.
• Implement resumable marking and safe failure paths before enabling automatic scheduling.
• Add define-matrix coverage, GC stress modes, sanitizer runs and CI configuration required by CONTRIBUTING.md.
• Add game-style frame workloads, large stable graphs, large containers, burst allocation, high mutation and memory-pressure cases. Include fresh and warmed heaps and long enough runs to expose late fallbacks.
• Document supported configurations, API semantics, budgets and fallback behavior.

Record matched payload work, compiler, runtime revision, collector settings, hardware and raw samples. Compare end-to-end GC interruptions, frame p99.9 and maximum latency, total stopped time, throughput and peak memory. Include thread-stop waits, allocator assists, retries and fallbacks.

Initial targets to evaluate on declared game workloads and hardware are p99.9 GC interruptions below 1 ms, observed maxima below 2 ms and at most 10 percent throughput loss against the existing collector, with an explicit memory budget. These are proposed acceptance targets, not measured results or guarantees.

Only consider making this the default after correctness and performance gates pass across the supported configurations. Keep an explicit option for the existing collector.

Related: #327 for deferred concurrent marking and #258 for benchmark and performance regression tracking.