Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
10697ae
fix: recover missing native provider HOME aliases
Sep 30, 2026
8f99174
fix: explicitly recover legacy host filesystem device identities
Sep 30, 2026
4b5bdff
fix: fence replaced lock paths before filesystem recovery
Sep 30, 2026
3266931
fix: explicitly recover retained graph host pins after reboot
Sep 30, 2026
2404e43
test(runtime): model absent publication recovery safety
Sep 30, 2026
b3a9246
fix(runtime): recover retained graphs after lost reboot publications
Sep 30, 2026
18fc123
fix(runtime): release recovery VM locks with inherited descriptors
Sep 30, 2026
54f5541
test(runtime): preserve private recovery inspection selection
Sep 30, 2026
8e76aa5
test(runtime): isolate concurrent recovery fixture roots
Sep 30, 2026
318b39c
fix(runtime): restore retained source after host device migration
Sep 30, 2026
b843857
test(runtime): isolate publisher acknowledgement transport
Sep 30, 2026
015e00d
test(runtime): qualify bind-only source recovery invocation
Sep 30, 2026
0f3b971
fix(runtime): prefer verified current cleanup over historical enrollment
Sep 30, 2026
9da7305
test(runtime): stabilize source recovery lifecycle fixtures
Sep 30, 2026
dddebc8
fix(cli): preserve legacy namespaces during retained branch startup
Sep 30, 2026
46fe63e
fix(cli): select retained routing before branch normalization
Sep 30, 2026
87f1239
fix(cli): verify active legacy branch restart identity
Sep 30, 2026
5535152
fix(cli): retain content IDs when restoring unchanged image declarations
Sep 30, 2026
f90bdd8
fix(runtime): recover selected dependency sockets in a quiescent live…
Sep 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 136 additions & 0 deletions docs/guides/native-candidate.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,25 @@ still apply. A same-checkout branch namespace does not create an isolated source
tree. These planning contracts do not qualify multiple independent source roots
inside one VM pool.

An existing frontend branch mapping may retain a graph admitted before native
branch namespaces. If its current scoped review differs, the frontend first
checks native selection against that exact saved run, owner, namespace and plan:
restore selection for a stopped graph, or authenticated service selection for an
active graph. A native unbranched review of the same canonical project and unchanged
original Compose must then return the saved namespace. Only that retained run
continues with unbranched review and execution arguments and keeps its original
adapted route labels, before branch hostname rewriting. This preserves legacy
source contracts that did not enroll hostname changes. The selected generation,
restore selection and shared-source semantic compatibility are checked again
before admission; mappings, graph receipts and ownership namespaces are not
rewritten. Active legacy restart also rechecks its service, container, boot and
receipt generation after compatibility review, before cleanup eligibility. A
changed selection refuses before stopping the graph. This preflight check does
not make cleanup atomic with that generation; existing ownership and frontend
finalization checks still apply. Fresh and already branch-scoped graphs keep
their normal branch namespace. This compatibility path does not authorize
adopting an unrelated branch or project.

## Manual candidate upgrade and rollback

Candidate bundles are selected by their full path. There is no automatic candidate
Expand All @@ -118,6 +137,117 @@ complete bundle to change the candidate.
binary refusing newer state is an unsupported downgrade, not a successful rollback.
Preserve that state; do not rewrite receipts or adopt another home's data.

Host cleanup or a reboot can remove the temporary `/private/tmp/hkl-<owner>` HOME
alias while the private candidate home and disks remain intact. Normal commands
report `provider_home_missing`; they do not initialize another pool or recreate the
alias during observation. Explicit `runtime recover --json` can restore only that
absent exact alias, after verifying the receipt-bound dead provider, both recorded
disks, free VM lock, closed disk handles, and no active provider command. Existing
files, directories, foreign links, a live or reused PID, and uncertain ownership
refuse without replacement. Recovery rechecks the receipt before exclusive creation,
then validates socket absence and flushes the disks before recording
`recovered-unclean`. If those final checks fail, it reports
`provider_home_restored_recovery_incomplete`, keeps the exact owned alias and retained
data, and requires inspection before retry. No guest is started by alias recovery.
Interrupted starts without a recorded process or both identified disks remain
outside this repair path.

A physical macOS reboot may also renumber the mounted filesystem device. Strict
disk and source checks still refuse a changed device number; missing-HOME recovery
does not waive them. For an offline **stock pool**, inspect the separate migration:

```sh
./hack-native --candidate-root /absolute/private/candidate-home runtime host-filesystem-recovery --json
./hack-native --candidate-root /absolute/private/candidate-home runtime recover-host-filesystem --expect-sha256 <selection_sha256> --accept-legacy-device-rebind --json
./hack-native --candidate-root /absolute/private/candidate-home runtime recover --json
```

Review the inspection before selecting its hash. This explicit legacy migration
requires an absent recorded provider whose start predates the current host boot,
no active provider commands, exclusive existing operation/VM locks and closed disk
handles. Both disk inodes, sizes and ext4 UUIDs, the exact source path/inode and its
ownership must be unchanged; only one common old-to-new device-number change is
allowed. The inspection is read-only. Publication atomically changes only the
owner's disk and source device numbers, retaining its phase and process record.
Normal commands keep their strict identity checks.

Legacy receipts have no original host boot UUID or filesystem volume UUID. Calendar
timestamps corroborate a reboot, and matching retained file identities constrain
the migration, but neither proves original volume continuity. The opt-in explicitly
accepts that limitation; copied or relocated pools are outside this procedure.
Prepared-base pools and pending owner/network/activation updates require separate
recovery and are refused. A torn owner publication preserves `owner.pending` and
blocks another migration; do not delete or adopt that file manually.

This does not start the VM, restore HOME, retire stale sockets or rewrite historical
graph receipts. Use ordinary recovery afterward. Historical shared-source graphs
retain the prior identity and may require verified cleanup plus a new generation;
successful metadata migration alone does not establish application recovery.

After that explicit provider recovery and one audited VM boot, a retained graph
from the immediately preceding guest boot can still carry old host device numbers
in its publisher, relay-control, and dependency-socket receipts. Inspect and select
one run's host-pin recovery separately:

```sh
./hack-native --candidate-root /absolute/private/candidate-home graph inspect-host-pin-recovery --run-id <run-id> --json
./hack-native --candidate-root /absolute/private/candidate-home graph recover-host-pins --run-id <run-id> --expect-selection <selection_sha256> --accept-legacy-device-rebind --json
./hack-native --candidate-root /absolute/private/candidate-home graph recover-cleanup --run-id <run-id> --expect-receipt <original_receipt_sha256> --json
./hack-native --candidate-root /absolute/private/candidate-home graph retire-recovered-publisher --run-id <run-id> --expect-owner <owner> --json
```

The first command only inspects. The second publishes a private, exact-run
witness before cleanup; it changes no graph, publisher, control, or dependency
receipt. It requires the provider's repaired current disks/source, unchanged
recorded inodes and raw receipts, dead owners whose recorded starts precede the
current physical host boot, refused socket listeners, and the immediate guest
boot transition. It also binds retained volume names and actual labels. Cleanup
and retirement can use the witness only for those selected old pins; ordinary
reads and later publisher generations remain strict. Missing or foreign pins,
active listeners, changed resources, pending journals, or further guest boots
refuse without adopting another run. A completed older cleanup journal is
retained as history and does not itself block a later selected generation.

Legacy receipts do not identify the original APFS volume. This explicit
device-number rebind cannot prove pre-reboot volume continuity. Completing these
commands retains the old run's data and proves cleanup of its dead generation;
it does not migrate `graph.source.shared`, restore the application, establish
route readiness, or claim overall v5 acceptance. Same-run source continuity
requires a separate explicit witness-bound transition after cleanup and
publisher retirement. If macOS removed the foreground or relay-control
directory itself during reboot, this command refuses: the old pin receipts no
longer exist, and absent pathnames cannot stand in for their recorded owner
identities. That case requires a separate selected absence-recovery procedure.

When a **physical host reboot** removed both the deterministic foreground
publication root and this run's relay-control root, inspect the distinct
absence path with the private original provider Owner and its exact
pre-migration host-filesystem inspection:

```sh
./hack-native --candidate-root /absolute/private/candidate-home graph inspect-absent-publication-cleanup --run-id <run-id> --original-owner-file <private-original-owner.json> --host-inspection-file <private-inspection.json> --json
./hack-native --candidate-root /absolute/private/candidate-home graph recover-absent-publication-cleanup --run-id <run-id> --original-owner-file <private-original-owner.json> --host-inspection-file <private-inspection.json> --expect-selection <selection_sha256> --retain-data --accept-unpinned-post-reboot --json
```

Inspection does not create a publication root. A failed selected action can
leave only its private lock reservation; inspection recognizes that exact
lock-only state. Recovery locks the foreground publication before acquiring
the VM lease, then durably records the selected absence before any cleanup.
It retains volumes, verifies the stopped receipt, and records a separate
absent-publisher retirement. Ordinary publication and cleanup cannot infer
ownership from missing paths. The old foreground PID and original physical
volume are not proved by legacy graph receipts; this path requires explicit
acceptance of that post-reboot limitation and refuses a changed boot, present
or foreign publication, stale selection, pending state, or changed resources.
An interrupted recovery can resume only its exact selected intent on the same
host and immediate guest boot. A later successful restore treats this witness
as history; it never grants cleanup of the new generation.

This operation does not change historical shared-source device identity.
Source-mounted projects require the separate selected source-continuity step
before normal same-run restore. The command's stopped/data-retained result is
not proof of application startup or routing.

Compatibility is qualified for specific bundle hashes and state formats. The
displayed version alone does not establish frontend/executor or downgrade
compatibility. V4 and candidate homes remain separate; this procedure does not
Expand Down Expand Up @@ -230,6 +360,12 @@ and route selections.
Normal foreground `hack restart` performs this selection and retained-data restore.
It verifies the old containers and networks are absent and the retained volumes
still have their recorded identities; it does not silently create replacement data.
When the original Compose file is unchanged, retained startup also selects the
authenticated container image IDs from that stopped graph. Mutable tags are not
resolved again for those services. Missing images still refuse native admission;
this selection never replaces generation, source or resource checks. Changing the
original Compose file uses normal image resolution and the existing compatibility
rules, rather than silently adopting an old image for a new declaration.
For a shared-source graph admitted with a retained compatibility contract, ordinary
source-content edits may change the reviewed plan ID: restart checks the stable
execution, exclusion, mount and dependency-cache inputs before cleanup and again
Expand Down
50 changes: 50 additions & 0 deletions packages/runtime-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,21 @@ state or changed provider identity refuses recovery. VM disks and graph data rem
This is cooperative same-user recovery for a stopped pool, not adoption of an
unreceipted active relay or proof of application readiness.

For an already-running VM whose retained graphs are all stopped, the separate
`hack-local runtime quiescent-dependency-socket-recovery --json` command selects
legacy unreceipted socket inodes without rebooting. Explicitly apply its selection
with `hack-local runtime recover-quiescent-dependency-sockets --expect-sha256 <hash> --json`.
This mode holds a pool publication gate before foreground retirement locks and the
provider lease. It pins the exact provider receipt, host/guest boot, every retained
graph receipt and volume identity, and requires absent compute, no foreground
publisher, no dependency assignment and no untracked guest containers. It rechecks
that proof before each inode-selected unlink and journals the selection separately
from stopped-pool recovery so partial retries retain their original scope.
An incomplete journal, live listener, replacement path or changed proof refuses
recovery. Owner bytes, source-rebind witnesses, retained data and the running VM
remain unchanged. This is explicit cooperative legacy cleanup, not proof of the
original socket creator or a security boundary against another same-user process.

TERM and INT are checked during initial graph startup, including readiness waits
and before new service effects. Cancellation enters owned cleanup while preserving
persistent data. Checks occur between bounded operations; an in-flight operation
Expand Down Expand Up @@ -803,3 +818,38 @@ before fresh dependency ownership is admitted. This does not enable file-based
`graph restart`/`restore` or normalized live-owner redelivery, and does not replay
initializer cache-release effects. Native same-run restoration and cross-boot
volume continuity require separate runtime qualification.

An explicitly recovered post-reboot graph may retain a stopped shared-source
receipt whose `source.shared.device` names the prior host filesystem device.
After **completed** selected absent-publication cleanup and separate publisher
retirement, inspect that same run with `graph inspect-source-device-rebind
--run-id RUN --json`. Review the returned original stopped-receipt hash and
qualification, then commit only that selection with `graph
recover-source-device-rebind --run-id RUN --expect-selection SHA
--accept-legacy-device-rebind --json`. This private candidate command changes no
stopped receipt, cleanup proof, retirement proof or named volume. It writes one
immutable witness for the current Owner share, requiring the same canonical
project path, inode, guest path and unfiltered access, with only the host device
number changed. The current guest's writable virtiofs mount and selected volume
identities are checked independently. The explicit acceptance records that
legacy receipts cannot establish original physical volume continuity across the
host reboot.

An exact completed cleanup and retirement for the current stopped receipt takes
precedence over older cleanup records. Pending operations, unresolved initializer
effects and mismatched receipt or retirement identities still refuse recovery;
historical records cannot authorize a later generation.

`restore-selection` then binds the witness's raw hash to the selected generation.
The source checks use only an in-memory device projection; original retention and
restore history consume the unchanged stopped receipt. The new receipt receives
the projected source after history retention, before the first new-attempt write.
The witness file remains pinned and is rechecked before that write. Once the
first new generation replaces the stopped receipt, the witness is audit history
and grants no authority to later ordinary down/up cycles. An incomplete
`source-device-rebind.pending` file, including an empty or partial write, is
preserved and refuses both explicit recovery and a new retired-publisher bind;
there is no automatic prefix-based adoption or deletion. The ignored native
synthetic-device fixture checks this same-run transition and later retained
marker reads, but only an actual host-reboot application run can qualify the
physical continuity claim.
Loading
Loading