Skip to content

macOS guests 1/6: native VZ runtime and instance lifecycle - #498

Draft
chruffins wants to merge 16 commits into
mainfrom
spike/macos-guests
Draft

chruffins wants to merge 16 commits into
mainfrom
spike/macos-guests

Conversation

@chruffins

@chruffins chruffins commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Experimental, administrator-provisioned macOS guests on local Apple silicon, using the normal Hypeman image storage and instance create/list/get/start/stop/delete APIs. This is a spike checkpoint, not production-ready macOS support.

  • Add Mac boot/platform identity to shared VM configuration and VZ shim translation; retain the Linux boot path and reject Mac boot in QEMU.
  • Add an offline cmd/import-macos importer for a stopped macvm bundle: APFS-clone disk/aux, hash content/config, publish a local ready darwin/arm64 image. No HTTP host-path import or OCI macOS pull.
  • Clone writable boot disk and auxiliary storage per instance; use template CPU/RAM defaults, preserved NAT MAC and observed VZ DHCP leases.
  • Skip Linux kernel/initrd/config-disk/agent readiness and ballooning for Mac guests.
  • Prevent concurrent use of preserved Mac identity through manager admission and a process-lifetime shim lock.
  • Explicitly reject unsupported lifecycle/features rather than route them through Linux implementations. Exec rejects before WebSocket upgrade.
  • Add opt-in macos_only startup, which rejects Linux guests and skips Linux downloads, builders and Caddy/DNS; add a configurable HTTP listen address for loopback-only use.
  • Include a standalone provisioning/save-restore experiment harness and docs/macos-experimental.md.

Runtime validation

Performed sequentially on an Apple M5 / 16 GiB host with an installed, configured macOS template (4 CPUs / 8 GiB, 64 GiB logical boot disk). VM bundles, passwords, SSH keys, JWT credentials and local logs/screenshots are not checked in.

  • Imported image appears ready through GET /images.
  • POST /instances returns 201, darwin/arm64, VZ and Running.
  • Authenticated desktop, Chrome 155 CDP navigation, screenshot and Apple Paravirtual WebGL pass.
  • Duplicate active template identity returns 409; unsupported standby returns 400; exec returns 501 before upgrade.
  • Stop -> Stopped; start -> Running; desktop and Chrome CDP pass after cold restart.
  • API-process restart recovers the live guest without rebooting it; Chrome/tabs continue.
  • Delete -> 204, storage removed, subsequent GET -> 404; imported image remains ready.
  • New create containing only name/image inherits template CPU/RAM and cached native platform.
  • Separate direct-shim experiment preserves Chrome/tabs through fresh-process restore of a matching disk/aux/state clone. This does not establish API snapshot support.

These are smoke observations, not repeated boot-to-Chrome-ready benchmarks.

Tests / review

  • Focused macOS/import/platform/image resolver/identity/DHCP/VZ/QEMU rejection/API mapping/schema-default/exec unit tests pass, using containers_image_openpgp.
  • API/importer and signed VZ shim builds pass; git diff --check passes.
  • Broad suite previously timed out; no full-suite or Linux runtime regression-pass claim.
  • Structured autoreview (--mode local) was retried before this commit, but Codex/OpenAI authentication fails with 401. No external clean-review verdict.

Explicit limitations / follow-ups

  • Running means VMM state, not SSH/desktop/application readiness. No automatic Chrome launch.
  • No Darwin guest agent: SSH is the smoke transport. Stop/delete shuts down the VMM and does not guarantee orderly guest OS/application shutdown.
  • API snapshot/fork/standby/restore, instance updates/volumes, env/commands/credentials/policies, passthrough and shaping remain unsupported for Mac guests.
  • Identity, MAC, host keys and user secrets are preserved. One active instance per identifier; external source/clone concurrency must also be avoided. Rekey/provisioning is required before multi-tenant use.
  • No macOS image tagging/promotion or registry pulls. Imported image files must remain immutable.
  • Mixed/full-mode ingress startup was not validated; initial Caddy startup failed with a storage-converter panic. The opt-in Mac-only mode deliberately skips ingress startup and rejects ingress creation/builds.
  • System-launchd/TCC context, vsock, identity rekey, resource-pressure/concurrency and >=5 cold/warm readiness measurements remain follow-ups.

Next slices: Darwin desktop-aware readiness/launch/clean shutdown; consistent API snapshot bundles; clone identity/credential rekey; full Linux regression and platform measurements.


Note

High Risk
Introduces a new VZ macOS boot path, identity admission, and broad instance/API branching; misconfiguration or identity collisions could affect VM lifecycle on Apple silicon hosts.

Overview
Adds experimental local macOS guests on Apple silicon via VZ: offline import of stopped macvm bundles, normal instance create/start/stop/delete, and a dedicated Mac boot path in the hypervisor and vz-shim (hardware model, machine ID, auxiliary storage). Linux OCI pulls, macOS image tagging, and registry workflows stay blocked; instances clone writable boot disk + aux per guest, use template CPU/RAM defaults, preserved NAT MAC, and DHCP-derived IP.

API and lifecycle: macOS instances skip Linux config disk, guest-agent readiness, proportional I/O/network shaping defaults, and TAP-based networking. Unsupported operations (snapshot, fork, standby, restore, update, volumes, vsock/exec) fail explicitly; exec returns 501 before WebSocket upgrade. macos_only config skips Linux kernel/initrd downloads, builders, and ingress/Caddy startup; rejects Linux instance creates and CreateBuild / CreateIngress. HTTP listen_address allows loopback binding.

Safety: Concurrent use of the same Mac machine identifier is blocked in the instance manager and with a process-lifetime flock in vz-shim. OpenAPI/schema copy and defaults defer Linux-centric defaults when creating from imported macOS images.

Also includes cmd/import-macos, docs/macos-experimental.md, and a standalone spike/macvm provisioning harness (not wired into the main API).

Reviewed by Cursor Bugbot for commit 4849059. Configure here.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
-->

✱ stlc build

✅ go code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

116 files generated at a1e7731 (pushed)

go get github.com/kernel/hypeman-go-staging@a1e77312412da281257c31a1383d3949f7252747
✅ python code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

232 files generated at f666c03 (pushed)

✅ typescript code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

138 files generated at feee2d3 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_76WGdh5n-wise-ledge
Timestamp2026-10-09T21:57:21.858Z
stlc8413509
Spec hash7b3ae593d3cc
Config hash659c3687c3f0

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-10-09 21:57:45 UTC

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4849059. Configure here.

Comment thread lib/instances/macos.go
meta, err := m.loadMetadata(entry.Name())
if err != nil {
return fmt.Errorf("check Mac identity admission: %w", err)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Identity admission fails on metadata errors

Medium Severity

checkMacOSIdentityAvailable treats any loadMetadata failure as fatal, including missing metadata for an in-progress create. A concurrent Linux or macOS create that has already made a guest directory but not yet written metadata causes an unrelated macOS start or create to fail admission.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 4849059. Configure here.

}
if exit, ok := e.(*exec.ExitError); !ok || exit.ExitCode() != 1 {
return nil, fmt.Errorf("cannot verify storage is closed: %v", e)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lsof probe leaks process groups

Medium Severity

The import lsof probe runs through exec.CommandContext without a dedicated process group and without SIGKILLing that group on every completion path. Context cancel only stops the wrapper, so lsof descendants can leak after success, ErrWaitDelay, or a wrapper that exits first.

Fix in Cursor Fix in Web

Triggered by learned rule: Subprocess binary probes must use process groups and kill on all completion paths

Reviewed by Cursor Bugbot for commit 4849059. Configure here.

@chruffins chruffins changed the title Experimental local macOS guests through VZ and instance API macOS guests 1/6: native VZ runtime and instance lifecycle Oct 8, 2026
- Skip directories without instance metadata during identity admission so
  an in-progress create or leftover directory cannot block a macOS start.
- Fail create when the image lookup errors instead of silently applying
  Linux disk and network defaults to a macOS guest.
- Take the identity lock from a fixed /tmp path so separate server
  processes with different TMPDIR values contend on the same lock.
- Resolve disk and aux storage from one image layout via GetBootStorage.
- Remove the destination directory when a spike clone fails partway, so a
  retry is not blocked by a partial bundle.
macOS instances own independent disk and auxiliary storage cloned at create
time and keep their identity in metadata, so start no longer resolves the
template image. Deleting an imported image leaves existing instances
startable.

Reject --with-state combined with --rekey or --new-mac in the spike clone
command: saved state is only restorable into the configuration it was saved
from.
The image lookup used for instance defaults returned 404 for an uncached
image, which skipped the auto-pull path in resolveImageForCreate. Treat
not-found as no macOS defaults and let create resolve and pull the image.
legacyImageExists rebuilt the legacy layout and re-read its metadata for a
path the caller had already resolved.
The API looked up the image by tag to decide whether to apply Linux disk and
network defaults, while the manager resolved the image it actually creates
from. A tag that moved, or an uncached image, could make the defaults describe
a different guest.

Move the Linux defaults (vCPUs, disk I/O, network bandwidth) into the manager,
applied after image resolution and before resource reservation. macOS guests
keep taking CPU and memory from the image and continue to reject shaping. The
API no longer resolves images at all.
@chruffins

Copy link
Copy Markdown
Contributor Author

Live QA of the reviewed combined PR1–3 sources plus PR4 patches on Apple silicon passed normal authenticated HTTP create from a digest-pinned, previously HTTP-pulled installed macOS OCI artifact (registry offline). Verified actual macOS 27.0.1/26A434 cold boot over pinned-key SSH, inherited 4CPU/8GiB template defaults, duplicate-identity HTTP409, live-guest API restart recovery with unchanged guest boot time, HTTP stop/start with fresh boot time and durable guest-only marker, and unchanged full SHA256 of the immutable materialized base disk+aux afterward. Stop/start here used system-agent-disabled fallback, not verified graceful root-agent shutdown. QA used an isolated data dir/slot; original instance/API/storage were untouched and the QA VM/API ended stopped.

Runtime issue found: the long QA data directory generated an overlong AF_UNIX socket path. VZ reported VM started, then control listener failed listen unix .../vz.sock: bind: invalid argument, resulting in HTTP500 after ~30s. VMM exit and complete create rollback were confirmed. Retrying via a short private data-dir alias passed. Recommend validating the resulting control/vsock socket lengths before VM launch and returning a clear admission error rather than starting then failing. This is not fixed by the workaround and may affect other Unix-socket consumers too.

chruffins and others added 9 commits October 9, 2026 18:59
- Move the macOS config-disk skip to the start and create call sites instead
  of checking inside createConfigDisk, so the Linux-only step is decided once
  by its caller.
- Read the locally imported raw disk with a stat in legacyLayout instead of
  parsing metadata on every layout lookup.
- Return real image lookup errors from the macOS-only pre-check rather than
  reporting them as "not imported".
- Skip the Linux overlay limit for macOS boot disks, whose size comes from the
  imported image.
- Return 501 from the instance stat endpoint for macOS instances before
  dialing the guest, matching exec.
- Remove redundant macOS gates in stop, a stale fixture, a no-op TMPDIR setup
  in the identity lock test, and a misplaced doc comment.
MacOSImage.Validate holds the field checks and the 6-byte Ethernet MAC rule.
The local importer previously accepted EUI-64 and InfiniBand addresses that
VZ rejects at boot. The OCI machine-image parser will use the same method.
- macOSNetworkConfig pins a networked macOS guest to its preserved MAC and
  returns its network config, replacing the copies in create and start.
- Move the vmnet lease parser out of the darwin-only file. It only reads
  /var/db/dhcpd_leases, which is absent off macOS, so the non-darwin stub
  is no longer needed.
- Drop the macOS guards on attach and detach volume. Both return "not yet
  implemented" for every instance, so the guard only added a metadata read.
RestoreInstance rejects macOS guests with ErrInvalidRequest but had no case
for it, so the request fell through to the 500 default. Map it to 400 and
declare that response in the spec.
… place

- Split prepareMacOSRequest into validateMacOSCreate, a pure check over the
  request, the image and the backend's capabilities, and applyMacOSDefaults.
  The check no longer reads runtime.GOOS or the backend name, so the rejection
  list runs on every host with fixture capabilities.
- Add Capabilities.SupportsMacOSBoot, set by the vz backend on arm64.
- Drop the six pre-lock macOS rejections. Each operation checks the record it
  already loads for the work, so the instance is read once instead of twice.
macOSNetworkConfig already returns nil for guests without a network, so the
callers assign the result directly instead of testing it into a var first.
@chruffins

Copy link
Copy Markdown
Contributor Author

Simplification update at e6e6ab7: shared bounded complete-machine bundle decoding and one macOS request-default preparation; shared concrete create/start network + boot-config preparation. macOS preserves template MAC and avoids Linux allocation/proxy/config-disk dependencies; Linux rollback and request preservation have synthetic tests. Focused instance/API/image race tests passed 3 runs. Identity leases and stopped-storage/export requirements remain.

All PRs remain draft. This is source-level/synthetic validation, not a new live boot or production build proof. Default Codex independent review was attempted but is still blocked by authentication (HTTP401). Published atomically after checking reviewer heads with explicit per-ref force-with-lease; prior heads preserved locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant