Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
10697ae
fix: recover missing native provider HOME aliases
Sep 30, 2026
8f99174
fix: explicitly recover legacy host filesystem device identities
Sep 30, 2026
4b5bdff
fix: fence replaced lock paths before filesystem recovery
Sep 30, 2026
3266931
fix: explicitly recover retained graph host pins after reboot
Sep 30, 2026
2404e43
test(runtime): model absent publication recovery safety
Sep 30, 2026
b3a9246
fix(runtime): recover retained graphs after lost reboot publications
Sep 30, 2026
18fc123
fix(runtime): release recovery VM locks with inherited descriptors
Sep 30, 2026
54f5541
test(runtime): preserve private recovery inspection selection
Sep 30, 2026
8e76aa5
test(runtime): isolate concurrent recovery fixture roots
Sep 30, 2026
318b39c
fix(runtime): restore retained source after host device migration
Sep 30, 2026
b843857
test(runtime): isolate publisher acknowledgement transport
Sep 30, 2026
015e00d
test(runtime): qualify bind-only source recovery invocation
Sep 30, 2026
0f3b971
fix(runtime): prefer verified current cleanup over historical enrollment
Sep 30, 2026
9da7305
test(runtime): stabilize source recovery lifecycle fixtures
Sep 30, 2026
dddebc8
fix(cli): preserve legacy namespaces during retained branch startup
Sep 30, 2026
46fe63e
fix(cli): select retained routing before branch normalization
Sep 30, 2026
87f1239
fix(cli): verify active legacy branch restart identity
Sep 30, 2026
5535152
fix(cli): retain content IDs when restoring unchanged image declarations
Sep 30, 2026
f90bdd8
fix(runtime): recover selected dependency sockets in a quiescent live…
Sep 30, 2026
7dc0811
fix(runtime): preserve witnessed dependency cache scope on restore
Sep 30, 2026
ee4bf3a
fix(runtime): archive retired dependency rebind before restore
Sep 30, 2026
ffdabaf
chore: reconcile retained recovery with latest next
Sep 30, 2026
2320bf9
test(native): cover retired dependency journal across restored genera…
Sep 30, 2026
0b80466
fix: retire acknowledged stopped graph publishers explicitly
Sep 30, 2026
9c78243
test(native): refresh only the live dependency service
Sep 30, 2026
ff48ed7
test: serialize the frontend acceptance stand-in's shared state
Sep 30, 2026
65a7cc8
test(native): use pinned Bun HTTP fixture for endpoint proof
Sep 30, 2026
df82fde
fix(runtime): release acknowledged dead dependency claims without los…
Sep 30, 2026
b49fbda
refactor(runtime): name the observed dependency record tuple
Sep 30, 2026
e372326
fix: keep a published Unix socket's identity through startup failure
Sep 30, 2026
01424a9
test: observe the MCP socket's private mode where it is now set
Sep 30, 2026
45fb8f3
fix: never remove or chmod an unproven staging entry when publishing …
Sep 30, 2026
fba8691
fix: move an unproven staging entry aside without overwriting a holdi…
Sep 30, 2026
2fe54aa
fix(runtime): archive exact quiescent HTTPS owner evidence
Sep 30, 2026
7718f2a
fix(runtime): select legacy HTTPS device migration explicitly
Sep 30, 2026
b34fba3
fix(runtime): budget retained live pools without duplicate disk alloc…
Sep 30, 2026
de2017c
fix(native): preflight explicit recovery of stopped retained graphs
Sep 30, 2026
f88c422
fix(native): retain admitted images in stopped restart preflight
Sep 30, 2026
4a9ac29
fix(runtime): confirm acknowledged publisher retirement during restart
Sep 30, 2026
1b9b896
fix(runtime): recover same-boot owners after completed dependency ref…
Oct 1, 2026
a330a73
fix(runtime): admit restore with current completed retention proof
Oct 1, 2026
fd284a4
chore: reconcile retained recovery candidate with next
Oct 1, 2026
b357e03
chore: reconcile retired rebind qualification with current candidate
Oct 1, 2026
efdcdf6
test(native): align synthetic prior-boot reservation evidence
Oct 1, 2026
bf8c7f2
test(native): preserve retired journal qualification failures
Oct 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 154 additions & 2 deletions docs/guides/native-candidate.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,25 @@ still apply. A same-checkout branch namespace does not create an isolated source
tree. These planning contracts do not qualify multiple independent source roots
inside one VM pool.

An existing frontend branch mapping may retain a graph admitted before native
branch namespaces. If its current scoped review differs, the frontend first
checks native selection against that exact saved run, owner, namespace and plan:
restore selection for a stopped graph, or authenticated service selection for an
active graph. A native unbranched review of the same canonical project and unchanged
original Compose must then return the saved namespace. Only that retained run
continues with unbranched review and execution arguments and keeps its original
adapted route labels, before branch hostname rewriting. This preserves legacy
source contracts that did not enroll hostname changes. The selected generation,
restore selection and shared-source semantic compatibility are checked again
before admission; mappings, graph receipts and ownership namespaces are not
rewritten. Active legacy restart also rechecks its service, container, boot and
receipt generation after compatibility review, before cleanup eligibility. A
changed selection refuses before stopping the graph. This preflight check does
not make cleanup atomic with that generation; existing ownership and frontend
finalization checks still apply. Fresh and already branch-scoped graphs keep
their normal branch namespace. This compatibility path does not authorize
adopting an unrelated branch or project.

## Manual candidate upgrade and rollback

Candidate bundles are selected by their full path. There is no automatic candidate
Expand All @@ -118,6 +137,117 @@ complete bundle to change the candidate.
binary refusing newer state is an unsupported downgrade, not a successful rollback.
Preserve that state; do not rewrite receipts or adopt another home's data.

Host cleanup or a reboot can remove the temporary `/private/tmp/hkl-<owner>` HOME
alias while the private candidate home and disks remain intact. Normal commands
report `provider_home_missing`; they do not initialize another pool or recreate the
alias during observation. Explicit `runtime recover --json` can restore only that
absent exact alias, after verifying the receipt-bound dead provider, both recorded
disks, free VM lock, closed disk handles, and no active provider command. Existing
files, directories, foreign links, a live or reused PID, and uncertain ownership
refuse without replacement. Recovery rechecks the receipt before exclusive creation,
then validates socket absence and flushes the disks before recording
`recovered-unclean`. If those final checks fail, it reports
`provider_home_restored_recovery_incomplete`, keeps the exact owned alias and retained
data, and requires inspection before retry. No guest is started by alias recovery.
Interrupted starts without a recorded process or both identified disks remain
outside this repair path.

A physical macOS reboot may also renumber the mounted filesystem device. Strict
disk and source checks still refuse a changed device number; missing-HOME recovery
does not waive them. For an offline **stock pool**, inspect the separate migration:

```sh
./hack-native --candidate-root /absolute/private/candidate-home runtime host-filesystem-recovery --json
./hack-native --candidate-root /absolute/private/candidate-home runtime recover-host-filesystem --expect-sha256 <selection_sha256> --accept-legacy-device-rebind --json
./hack-native --candidate-root /absolute/private/candidate-home runtime recover --json
```

Review the inspection before selecting its hash. This explicit legacy migration
requires an absent recorded provider whose start predates the current host boot,
no active provider commands, exclusive existing operation/VM locks and closed disk
handles. Both disk inodes, sizes and ext4 UUIDs, the exact source path/inode and its
ownership must be unchanged; only one common old-to-new device-number change is
allowed. The inspection is read-only. Publication atomically changes only the
owner's disk and source device numbers, retaining its phase and process record.
Normal commands keep their strict identity checks.

Legacy receipts have no original host boot UUID or filesystem volume UUID. Calendar
timestamps corroborate a reboot, and matching retained file identities constrain
the migration, but neither proves original volume continuity. The opt-in explicitly
accepts that limitation; copied or relocated pools are outside this procedure.
Prepared-base pools and pending owner/network/activation updates require separate
recovery and are refused. A torn owner publication preserves `owner.pending` and
blocks another migration; do not delete or adopt that file manually.

This does not start the VM, restore HOME, retire stale sockets or rewrite historical
graph receipts. Use ordinary recovery afterward. Historical shared-source graphs
retain the prior identity and may require verified cleanup plus a new generation;
successful metadata migration alone does not establish application recovery.

After that explicit provider recovery and one audited VM boot, a retained graph
from the immediately preceding guest boot can still carry old host device numbers
in its publisher, relay-control, and dependency-socket receipts. Inspect and select
one run's host-pin recovery separately:

```sh
./hack-native --candidate-root /absolute/private/candidate-home graph inspect-host-pin-recovery --run-id <run-id> --json
./hack-native --candidate-root /absolute/private/candidate-home graph recover-host-pins --run-id <run-id> --expect-selection <selection_sha256> --accept-legacy-device-rebind --json
./hack-native --candidate-root /absolute/private/candidate-home graph recover-cleanup --run-id <run-id> --expect-receipt <original_receipt_sha256> --json
./hack-native --candidate-root /absolute/private/candidate-home graph retire-recovered-publisher --run-id <run-id> --expect-owner <owner> --json
```

The first command only inspects. The second publishes a private, exact-run
witness before cleanup; it changes no graph, publisher, control, or dependency
receipt. It requires the provider's repaired current disks/source, unchanged
recorded inodes and raw receipts, dead owners whose recorded starts precede the
current physical host boot, refused socket listeners, and the immediate guest
boot transition. It also binds retained volume names and actual labels. Cleanup
and retirement can use the witness only for those selected old pins; ordinary
reads and later publisher generations remain strict. Missing or foreign pins,
active listeners, changed resources, pending journals, or further guest boots
refuse without adopting another run. A completed older cleanup journal is
retained as history and does not itself block a later selected generation.

Legacy receipts do not identify the original APFS volume. This explicit
device-number rebind cannot prove pre-reboot volume continuity. Completing these
commands retains the old run's data and proves cleanup of its dead generation;
it does not migrate `graph.source.shared`, restore the application, establish
route readiness, or claim overall v5 acceptance. Same-run source continuity
requires a separate explicit witness-bound transition after cleanup and
publisher retirement. If macOS removed the foreground or relay-control
directory itself during reboot, this command refuses: the old pin receipts no
longer exist, and absent pathnames cannot stand in for their recorded owner
identities. That case requires a separate selected absence-recovery procedure.

When a **physical host reboot** removed both the deterministic foreground
publication root and this run's relay-control root, inspect the distinct
absence path with the private original provider Owner and its exact
pre-migration host-filesystem inspection:

```sh
./hack-native --candidate-root /absolute/private/candidate-home graph inspect-absent-publication-cleanup --run-id <run-id> --original-owner-file <private-original-owner.json> --host-inspection-file <private-inspection.json> --json
./hack-native --candidate-root /absolute/private/candidate-home graph recover-absent-publication-cleanup --run-id <run-id> --original-owner-file <private-original-owner.json> --host-inspection-file <private-inspection.json> --expect-selection <selection_sha256> --retain-data --accept-unpinned-post-reboot --json
```

Inspection does not create a publication root. A failed selected action can
leave only its private lock reservation; inspection recognizes that exact
lock-only state. Recovery locks the foreground publication before acquiring
the VM lease, then durably records the selected absence before any cleanup.
It retains volumes, verifies the stopped receipt, and records a separate
absent-publisher retirement. Ordinary publication and cleanup cannot infer
ownership from missing paths. The old foreground PID and original physical
volume are not proved by legacy graph receipts; this path requires explicit
acceptance of that post-reboot limitation and refuses a changed boot, present
or foreign publication, stale selection, pending state, or changed resources.
An interrupted recovery can resume only its exact selected intent on the same
host and immediate guest boot. A later successful restore treats this witness
as history; it never grants cleanup of the new generation.

This operation does not change historical shared-source device identity.
Source-mounted projects require the separate selected source-continuity step
before normal same-run restore. The command's stopped/data-retained result is
not proof of application startup or routing.

Compatibility is qualified for specific bundle hashes and state formats. The
displayed version alone does not establish frontend/executor or downgrade
compatibility. V4 and candidate homes remain separate; this procedure does not
Expand Down Expand Up @@ -230,6 +360,12 @@ and route selections.
Normal foreground `hack restart` performs this selection and retained-data restore.
It verifies the old containers and networks are absent and the retained volumes
still have their recorded identities; it does not silently create replacement data.
When the original Compose file is unchanged, retained startup also selects the
authenticated container image IDs from that stopped graph. Mutable tags are not
resolved again for those services. Missing images still refuse native admission;
this selection never replaces generation, source or resource checks. Changing the
original Compose file uses normal image resolution and the existing compatibility
rules, rather than silently adopting an old image for a new declaration.
For a shared-source graph admitted with a retained compatibility contract, ordinary
source-content edits may change the reviewed plan ID: restart checks the stable
execution, exclusion, mount and dependency-cache inputs before cleanup and again
Expand Down Expand Up @@ -436,7 +572,16 @@ requires ownership inspection rather than automatic removal. Restart does not
implicitly migrate a shared pool's network policy or interrupt other projects.
When an interrupted frontend cannot write that final acknowledgement, an explicit
`restart --recover-frontend --expect-finalization-attempt <32-hex>` can resume
the saved, already-cleaned restart intent. New attempts retain their frontend
the saved, already-cleaned restart intent. It can also preflight an exact graph
that was stopped before the frontend saved an intent: fresh native observations
must prove absent containers/networks and present retained volumes, and repeat
that proof after review before cleanup. The normal intent, retaining down,
frontend recovery and startup sequence still runs. An active graph keeps its
authenticated service selection and live-listener checks; uncertain observations
cannot select the stopped path. Stopped preflight retains the admitted
image IDs when the original Compose input is unchanged, matching retained startup
instead of resolving mutable tags again. Changed original input keeps normal
image resolution and the existing compatibility checks. New attempts retain their frontend
PID and HTTPS port in the private token. Older v1 attempts additionally require
`--expect-frontend-pid <previously-observed-pid>`; a guessed PID is not recovery
evidence. Recovery requires the exact stopped graph receipt with its volumes
Expand Down Expand Up @@ -927,8 +1072,15 @@ operation stops verified guest dependency listeners, removes owned containers an
network resources, retains persistent data, retires stale foreground and relay
publications, and releases the graph's dependency reservation before unlocking the
pool. It does not restart the pool. Pending startup, one-off or dependency
rebind state is refused. Completion records owner-death evidence independently of
rebind state is refused. A completed dependency refresh is accepted only for the
current boot and exact ready receipt generation, with matching terminal services
and helper markers. After owned absence is independently verified, its journal is
archived before stale publications are retired. An interrupted archive resumes
from the committed cleanup proof and exact journal bytes; it never replays refresh.
Completion records owner-death evidence independently of
the live relay acknowledgement protocol.
When a restored publisher is admitted, older absence-retirement records remain
validated history; the exact current cleanup proof supplies retention authority.

After completion, retained restore can create a fresh owner;
`graph retire-recovered-publisher --run-id RUN --expect-owner OWNER` remains an
Expand Down
Loading
Loading