Skip to content

fix(runtime): recover retained native pools and graph owners - #122

Merged
roodboi merged 66 commits into
nextfrom
codex/same-boot-refresh-recovery
Oct 3, 2026
Merged

roodboi merged 66 commits into
nextfrom
codex/same-boot-refresh-recovery

Conversation

@roodboi

@roodboi roodboi commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Retained native projects can become unable to restart when cleanup acknowledgement, foreground ownership or recovery history outlives its active generation. This change provides selected recovery while preserving persistent data and refusing changed or ambiguous ownership. A failed first container start that leaves created / exit 128 can now be shut down with its running siblings through ordinary retaining down, then restarted normally.

Head a774b8fe19fb52a06eb08918050e8cfa50ce241f targets protected next. All eight required exact-head CI checks pass. A fresh signed bundle passed actual retaining cleanup and ordinary restart of a retained 14-service QA project on Apple Silicon macOS, followed by status, logs, exec, one-off run, Redis retention, normal-origin Google sign-in, data-backed search and deep-scroll images on two result pages. This PR is ready for review; it does not publish a release.

Behavior

  • Adds separate same-boot interrupted-start cleanup with a per-step journal. Each effect binds the selected dead foreground/relay, resource generation and exact coordinator attempt. A normally removed owner/socket pair uses a distinct witness bound to the retained operation lock, unchanged private parent, dead process and selected coordinator. Present pins remain strict; mixed/replaced paths refuse. Confirmation reacquires Engine → relay → Coordinator and rechecks publication immediately before ACK. Uncertain effects are never replayed. Completion requires fresh compute/helper/probe/environment absence, cleanup acknowledgement, publisher retirement and dependency reservation release. Frontend down and failed startup independently verify retention/finalization; a failed original startup remains an error.
  • Preserves shared dependency volumes through exact existing cache binding and provenance checks, including Engine creation identity and guest directory device/inode. Those checks run at every cleanup fence, completion, retry and selected Confirmed/ACK transition. The cleanup journal contains no volume deletion or release step.
  • Recovers exact previous-boot shared HTTPS ownership through selected archival after completed graph cleanup, retired publication, immediate boot succession, executable/process absence, free port and unchanged full owner/lease/CA proof. Original bytes/inodes remain preserved. Interrupted archival retains its admission barrier, newer owners block replay and legacy executables require quiescence.
  • Gives validated current completed dead-owner proof precedence over historical live-owner sidecars; superseded completed proofs are archived only after bounded history validation. Retires exited reservation-fenced bridge helpers with current run/container/network and allocation/socket proof, aligns cleanup capacity with allocation, and preserves required-listener exit evidence.

Validation

  • Full default Rust: 1,059 passed / 62 ignored; all features: 1,152 passed / 85 ignored; zero failures. Strict formatting and default/all-feature Clippy pass. Staged-source privacy and clean signed packaging pass; all six manifest entries and CLI/native signatures were independently verified.
  • Frontend evidence from the unchanged TypeScript implementation: full Bun 1,802 passed / 67 skipped / zero failures, focused recovery/down/start 86 passed / 669 assertions, typecheck and lint. These were not rerun solely for the two Rust-only increments; current hosted CI independently passes.
  • Native controls cover coordinator replacement, malformed probe state, failed-container drift, uncertain stops, marker/coordinator ACK pairings, retained-volume exclusion, cache incarnation drift, journal completion ordering and actual normal ControlListener drop. Independent review found no remaining effect-safety blocker within this bounded scope. The ACK negative checks the witness after a synthetic confirmed write, rather than a full paused ACK transition. The per-step loop has no fake-Engine end-to-end fixture; actual native qualification is recorded separately.
  • The maintained TLA+ suite passes, including 92 shared-HTTPS states and three guard-removal controls. This is abstract state evidence, not process/durability or whole-product proof.
  • Actual macOS retaining down completed all 382 journal steps, acknowledged cleanup, retired publication and released the dependency reservation. Independent readback confirmed compute absence, all four original volumes (including the cache) present with unchanged names, and unchanged sibling receipt, VM disk identities, CA and boot.
  • Ordinary retained restart then reached ready-observed with a verified live foreground owner; 12 long-running services run and both completed jobs remain exited. Normal status/logs, populated dependency-volume exec/run and synthetic Redis data readback pass. System-trusted port 443 returned all 16 sign-in assets as HTTP 200 over HTTP/2. Fresh Chrome Google sign-in, QA-backed Boston search, direct page-2 navigation with different results and all 27 observed images on each page pass after deep scrolling.

Limits and remaining work

Pending guest-script cleanup advances only on verified complete absence and can remain fenced after partial deletion. Completed interrupted-start journals remain evidence: a later resource generation cannot reuse them and returns graph_interrupted_start_cleanup_stale; verified restore-boundary archival is a separate follow-up. No manual journal removal is required or authorized here.

The earlier pinned containerd descriptor-initialization panic did not recur in this ordinary restart. Its cause remains unproven; matching engine digests and ten non-reproducing initialization probes do not prove a crash fix. The earlier dependency-socket fixture flake likewise remains unproven. Bun 1.4.2 is compatibility-qualified, not a proven crash fix.

The human-activated loopback relay qualifies this normal app origin, not installed-global routing. Combined merged-next integration/CI, protected prerelease approval, actual published-artifact installation, full-v5 performance/scale and separate domain/CA qualification remain independent gates. PR #123 supplies the separate prerelease preparation/install/rollback channel.

Release signal: fix. No automatic migration, application tracked-source/config change, stable replacement, tag or publication.

hack-cli-tests added 30 commits September 29, 2026 21:41
Stand-in invocations ran concurrently (a foreground `up` while the driver
polls `ps`) and each did an unlocked read-modify-write of state.json. A
`ps` that loaded the state before the second `up` saved its foreground
token then saved its stale copy over it, so `up` saw its token gone and
exited 0 before readiness; lost call records failed the acceptance test
as well. That failed 26 of 40 local runs, and Runtime state models on CI.

Each invocation now holds an exclusive flock across its read-modify-write
and releases it only before its long waits (the foreground loop and the
ps-hang fault). 40 of 40 local runs pass.
@roodboi
roodboi marked this pull request as ready for review October 3, 2026 01:13
@roodboi
roodboi merged commit cf4b6e9 into next Oct 3, 2026
10 checks passed
@roodboi
roodboi deleted the codex/same-boot-refresh-recovery branch October 3, 2026 03:24
roodboi added a commit that referenced this pull request Oct 3, 2026
Stable release automation accepts every `v*` tag, while native candidate
bundles require manual selection. Add an explicit `5.0.0-next.N` channel
that prepares a complete versioned macOS ARM64 bundle by default and
publishes only after exact protected-`next` CI and human environment
approval. Prereleases remain outside GitHub latest and the stable
Homebrew formula.

Add an opt-in `hack-next` installer with verified provenance, checksums
and signatures, immutable per-version bundles/homes, quiescence-gated
upgrades, retained rollback, and explicit return to stable. Switching
software does not migrate v4 Docker or v5 VM data. Product guides cover
dispatch, pinned installation, provider setup and recovery.

Validation: 16 focused Bun tests pass, including 23 Python installer
controls on macOS Python 3.9; full CLI suite 1,719 passed, 67 skipped,
zero failures; typecheck, lint, shell syntax and staged privacy checks
pass. The DB package gates used the existing shared cache. On the M3,
two real signed bundles from 8821f93 passed 12
install/upgrade/rollback/stable-switch checks using macOS Python 3.9 and
the real native executor. Both versions retained separate homes; failed
upgrade and active-launcher controls preserved selection; the installed
stable executable remained unchanged. The macOS metadata archive
regression reproduces 16 entries without suppression and verifies
exactly eight regular payload files after the fix using an independent
tar reader. All eight required hosted CI checks pass on exact head
8821f93 (optional Codesmith skipped). Published-asset download
verification remains a post-publication gate.

Release signal: optional candidate distribution capability. No release
tag or publication has occurred. Before first publication, merge the
candidate fixes and this channel into `next`, configure
reviewer-protected `v5-prerelease`, and finish candidate application
acceptance tracked in #122.

---------

Co-authored-by: hack-cli-tests <tests@hack>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant