diff --git a/CHANGELOG.md b/CHANGELOG.md index 9eac2ff63..c7dfde709 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added ### Changed +* **Breaking:** Snapshots use ABI 7 and config schema v4. Existing snapshots + must be regenerated. +* **Breaking:** `PtRootFinder` receives a `SnapshotMemoryReader` and returns a + `Result>`. Read snapshot bytes with `snapshot.read_exact(gpa, buffer)?` + and return roots with `Ok(roots)`. +* Related snapshots share immutable memory layers. This reduces capture time + and memory use. A guest can read data it overwrote before a snapshot from the + older layers of that snapshot. +* KVM: deleting a memory slot flushes only that slot's guest mappings on Linux + 6.12 and later. This speeds up the first guest call after a snapshot, + restore, or unmap changes memory slots. +* **Breaking:** `Sandbox::snapshot` installs the captured snapshot into the + sandbox and removes `map_region` and `map_file_cow` regions. Pages of these + regions that the guest has not mapped into its page tables are lost. +* `vmem::walk_va_spaces` merges adjacent mappings of the same kind into one + mapping. ### Removed diff --git a/docs/paging-development-notes.md b/docs/paging-development-notes.md index 7d2bbbdcd..edcdde009 100644 --- a/docs/paging-development-notes.md +++ b/docs/paging-development-notes.md @@ -116,10 +116,10 @@ the metadata at the top of the scratch region and grows downward. When the host takes a snapshot of a guest, it will traverse the guest page tables, collecting every (non-page-table) physical page that is -mapped (outside of the scratch map) in the guest. It will write out a -new compacted snapshot with precisely those pages in order, and a new -set of page tables which produce precisely the same virtual memory -layout, except for the scratch map. +mapped (outside of the scratch map) in the guest. It will reuse pages +unchanged since earlier snapshots, write the rest in order to a new +compacted blob, and build a new set of page tables which produce +precisely the same virtual memory layout, except for the scratch map. ### Pre-sizing the scratch region diff --git a/docs/snapshot-oci-format.md b/docs/snapshot-oci-format.md index 98c9918c5..479e44341 100644 --- a/docs/snapshot-oci-format.md +++ b/docs/snapshot-oci-format.md @@ -22,24 +22,28 @@ path/ blobs/sha256/ OCI image manifest JSON Hyperlight config JSON - raw memory bytes - (`memory_size` bytes) + one raw blob per snapshot blob: + guest data, then page tables canonical virtqueue rings ``` Four blob kinds per tag: * **manifest** (`application/vnd.oci.image.manifest.v1+json`). Tiny JSON - pointer record selected via `index.json`. References one config and - two layers by digest. -* **config** (`application/vnd.hyperlight.snapshot.config.v3+json`). The + pointer record selected via `index.json`. References one config, + the snapshot blobs, and one transport layer by digest. +* **config** (`application/vnd.hyperlight.snapshot.config.v4+json`). The snapshot descriptor: arch, hypervisor, CPU vendor, ABI version, resume address and captured registers, memory and transport layout, - registered host functions, snapshot generation counter, and namespaced - application metadata. Loaded eagerly and fully parsed. -* **layer / memory** (`application/vnd.hyperlight.snapshot.memory.v1`). - The raw guest memory image, exactly `memory_size` bytes. mmap'd on - restore. + registered host functions, the start address, data length, and page-table + length of each blob, the blob with the page tables for a restore, snapshot + generation counter, and namespaced application metadata. + Loaded eagerly and fully parsed. +* **layers / data** (`application/vnd.hyperlight.snapshot.memory.v2`). + One raw, host-page-aligned blob per snapshot blob, in config order. Each + blob holds guest data followed by the page-table tree of the snapshot that + created it. Each blob is mmap'd on restore, and the guest maps only its + data. * **layer / transport** (`application/vnd.hyperlight.snapshot.transport.v1`). A bounded binary image of the canonical G2H and H2G rings. Loading checks ring @@ -48,6 +52,8 @@ Four blob kinds per tag: The runtime queue protocol and canonical checkpoint are described in [Virtqueue host and guest communication](./virtio-host-guest-communication.md). +Manifest layers are ordered data, then transport. + Blob filenames are the sha256 of the blob bytes, so identical blobs across tags are stored once. @@ -95,7 +101,7 @@ A single saved `Snapshot` consists of exactly: config blob for tooling visibility, * one **manifest** blob (referenced by that index entry), * one **config** blob (referenced by the manifest's `config` field), -* one memory **layer** blob, +* one or more data **layer** blobs, * one transport **layer** blob. Saving two snapshots under different tags into the same `path` @@ -138,12 +144,12 @@ podman), `go-containerregistry` (crane), and `regclient`. ## Read semantics `Snapshot::load(path, reference)` reads a snapshot. It does not check -the manifest, config, memory, or transport blobs against their sha256 digests. +the manifest, config, data, or transport blobs against their sha256 digests. `reference` is an [`OciReference`], either a tag that matches the `org.opencontainers.image.ref.name` annotation or the manifest digest returned by `save`. `Snapshot::checked_load` adds the digest -check on all four blobs, catching accidental corruption on disk. +check on every blob, catching accidental corruption on disk. Both run every other check (OCI structure, descriptor sizes, schema versions, arch / hypervisor / CPU vendor / ABI tags, layout bounds, entrypoint bounds). The caller is responsible for trusting the source. diff --git a/docs/snapshot-versioning.md b/docs/snapshot-versioning.md index b86f72ca0..c704c311c 100644 --- a/docs/snapshot-versioning.md +++ b/docs/snapshot-versioning.md @@ -18,8 +18,8 @@ A snapshot carries four independently evolvable version markers: size), and the calling convention for guest function entry. A change to any of these breaks older snapshots unless the loader adds a compat path. -* **Snapshot blob encoding**, `MT_SNAPSHOT_V1` - (`application/vnd.hyperlight.snapshot.memory.v1`), aliased as +* **Snapshot blob encoding**, `MT_SNAPSHOT_V2` + (`application/vnd.hyperlight.snapshot.memory.v2`), aliased as `MT_SNAPSHOT_CURRENT`. This is the on-wire format of the snapshot blob: framing, section ordering, alignment, dirty/zero-page elision, anything about how the bytes are packed inside the OCI layer. @@ -27,19 +27,20 @@ A snapshot carries four independently evolvable version markers: (`application/vnd.hyperlight.snapshot.transport.v1`), aliased as `MT_TRANSPORT_CURRENT`. This is the binary encoding of canonical virtqueue state stored outside the memory layer. -* **Config schema**, `MT_CONFIG_V3` - (`application/vnd.hyperlight.snapshot.config.v3+json`), aliased as +* **Config schema**, `MT_CONFIG_V4` + (`application/vnd.hyperlight.snapshot.config.v4+json`), aliased as `MT_CONFIG_CURRENT`. This is the JSON shape of the config blob: field names, types, required vs optional, the descriptors the loader needs in order to reconstruct the sandbox (memory sizes, buffer sizes, `abi_version`, `hyperlight_version`, etc.). Renaming a field, changing its type, or adding a required field is a schema change and - bumps this constant. Version 3 describes the virtqueue-only memory layout - and requires a transport layer. Config v1 and v2 are incompatible with - the current ABI. + bumps this constant. Version 4 describes snapshot blobs, the blob with + the page tables, and transport geometry, and requires a transport + layer. Config v1, v2, and v3 are incompatible with the current ABI. ABI 6 requires checkpoint readiness after guest initialization and permits -H2G prefill to omit retained slots. Earlier snapshots must be regenerated. +H2G prefill to omit retained slots. ABI 7 stores snapshot memory as +layers. Earlier snapshots must be regenerated. The `OCI_LAYOUT_VERSION` constant is pinned by the OCI image-layout spec at `1.0.0`. diff --git a/proposals/0003-incremental-snapshots/README.md b/proposals/0003-incremental-snapshots/README.md index 0499e5e71..63e1b4962 100644 --- a/proposals/0003-incremental-snapshots/README.md +++ b/proposals/0003-incremental-snapshots/README.md @@ -10,12 +10,11 @@ pages since the parent snapshot. This is: 2. memory intensive, since unmodified data is duplicated across snapshots Incremental snapshots correct this. A snapshot instead holds a list of -immutable layers. Each layer has one shared read-only blob, and the live guest -address ranges that the layer gives. +immutable layers. Each layer is one shared read-only blob of guest data. An incremental snapshot points at the blobs of the parent and adds one new -layer. That layer holds the data that changed, and the page tables for a -restore. Sharing a blob costs one cloned `Arc`, because a blob is read-only. +layer. That layer holds the data that changed. The snapshot also holds its +page tables. Sharing a blob costs one cloned `Arc`, because a blob is read-only. Thus a snapshot stores the changes only, which lowers memory use and makes a snapshot faster to take. @@ -47,35 +46,32 @@ and snapshot B is taken. ```mermaid flowchart LR subgraph A["Snapshot A"] - LA["layer
live p0 p1 p2 p3"] + LA["layer 0"] end subgraph B["Snapshot B"] - LB0["layer
live p0 p1 p3"] - LB1["layer
live p2'"] + LB0["layer 0"] + LB1["layer 1"] end - BLOB0["blob 0
data p0 p1 p2 p3
page tables"] - BLOB1["blob 1
data p2'
page tables"] + BLOB0["blob 0
data p0 p1 p2 p3
page tables A"] + BLOB1["blob 1
data p2'
page tables B"] LA --> BLOB0 LB0 --> BLOB0 LB1 --> BLOB1 ``` -Snapshot B keeps the layer of snapshot A, but the live ranges of that layer do -not hold page 2. Both snapshots share blob 0. Only page 2 is a copy. +Snapshot B keeps the layer of snapshot A. Both snapshots share blob 0. Only +page 2 is a copy. The page tables of snapshot B point page 2 at blob 1, so page +2 and page tables A in blob 0 are stale for snapshot B. * A `SnapshotMemory` has the layers of one snapshot, in the sequence of their - guest physical address. It also has the index of the layer that gives the - page tables for a restore. -* A `SnapshotLayer` has one blob and the ranges in that blob that the layer - gives. Each layer has its own ranges. Layers in different snapshots can share - a blob. -* A `SnapshotBlob` is one immutable, contiguous block of host memory. It has two - parts, guest data and page tables. All snapshots that use this data share the - blob. A blob has no data - when every mapped page is already in a live range, for example when the host - calls `map_region` or `map_file_cow` and then takes a snapshot. Every layer - contains page tables, but only the last layer's page tables are needed for a + guest physical address. It also names the blob with the page tables for a restore. +* A `SnapshotBlob` is one immutable, contiguous block of host memory with guest + data followed by the page tables of the snapshot that created it. A blob with + data is a layer. All snapshots that keep the layer share the blob. When every + mapped page is already in a layer, for example when the host calls + `map_region` or `map_file_cow` and then takes a snapshot, the new blob holds + only page tables and is not a layer. The data ranges of the blobs do not overlap, thus at most one layer holds each address. The layers are sorted by the start address of their data range. A @@ -86,29 +82,48 @@ The snapshots make a tree. A sandbox can restore any snapshot and then make more snapshots from it. Snapshots with the same parent share the blobs of that parent. +### Page tables + +Each snapshot stores its page tables after the guest data of its new blob. The +guest mapping of the blob covers only the data. Data pages are shared across +snapshots, and every snapshot rebuilds its page tables. A kept layer also keeps +the page tables of the snapshot that created it. + +A restore copies the page tables into writable scratch. The guest writes them on +copy on write faults, and x86-64 processors set accessed and dirty flags in +them. Blobs are read-only. Capture builds the tables for their scratch address. + ### Taking a Snapshot of a Sandbox 1. Find the pages that this snapshot must save. Read the guest page tables to - get all the mapped pages. A page whose guest physical address is in a live - range of a layer is already in a previous snapshot, and this snapshot shares - it. All other pages must be saved. + get all the mapped pages. A page whose guest physical address is in a layer + is already in a previous snapshot, and this snapshot shares it. All other + pages must be saved. 2. Find a place in the guest address space for the new pages. The new blob must not overlap the layers that this snapshot keeps. Use the first unused part that is large enough. -3. Save the new pages in a new blob. Copy them into the blob, then write the - page tables after them. A saved page gets a new address in the blob, thus - the page tables are built again. A shared page keeps its address. -4. Make the list of layers. The new blob is one layer, and it gives the page - tables for a restore. This snapshot keeps each parent layer that still gives - at least one page. Such a layer keeps its blob, and its live ranges lose the - pages that moved into the new blob. This snapshot drops a parent layer that - gives no page. +3. Save the new pages in a new blob. A saved page gets a new address in the + blob, thus the page tables are built again, and saved after the pages. A + shared page keeps its address. +4. Make the list of layers. The new blob is one layer when it holds new pages. + This snapshot keeps each parent layer that still gives at least one page, + whole. This snapshot drops a parent layer that gives no page. +5. Install the snapshot into the sandbox. A later snapshot saves only the pages + changed after this one. A page the guest wrote before this snapshot faults + again on its next write. + +### Regions from `map_region` and `map_file_cow` + +A snapshot saves the pages of these regions that the guest page tables map. +Step 5 removes the regions, and the sandbox stops using their memory. A page +the guest has not mapped is lost. A later snapshot can place a blob at its +guest physical address. ### Restoring a Sandbox to a Snapshot Only the mapping step changes. Before, a restore replaced one memory region with the flat memory of the snapshot. Now it compares the current VM mappings -with the live ranges of the snapshot. It unmaps the regions that the snapshot +with the layers of the snapshot. It unmaps the regions that the snapshot does not have, and maps the regions that the VM does not have. Mappings that both have stay in place, thus a restore between snapshots that share layers moves only the difference. @@ -119,46 +134,51 @@ page tables of the snapshot into scratch, and resets the vCPU. ### Limits -A snapshot has one VM memory mapping for each live range of each of its layers. -Mapping or unmapping one is a hypervisor call. A restore changes only the -mappings that differ, but that can be all of them, so an unbounded number of -live ranges would make a restore arbitrarily slow. A snapshot therefore has a -fixed cap on its total mappings, and taking a snapshot above the cap fails. -Every layer except the one with the restore page tables must give at least one -live range, so the cap bounds the layer count too. +A snapshot has one VM memory mapping for each of its layers. Mapping or +unmapping one is a hypervisor call. A restore changes only the mappings that +differ, but that can be all of them, so an unbounded number of layers would make +a restore arbitrarily slow. A snapshot therefore has a fixed cap on its layers, +and taking a snapshot above the cap fails. -The page tables in the layers before the last waste space, in memory and on -disk, unless the snapshot that created them is also kept. +### Stale pages + +The VM maps each layer whole, including its stale pages. The guest page tables +do not reach them, but the guest writes its own page tables and can map any +guest physical address. A guest can therefore read data that it overwrote +before a snapshot, from any layer of that snapshot. Zeroing a secret before a +snapshot does not remove it from the snapshot. ### API -No public API changes, except `PtRootFinder`. It received the flat snapshot +No public API changes, except `PtRootFinder` and `Sandbox::snapshot`. +`PtRootFinder` received the flat snapshot buffer, which no longer exists, so it now receives a reader that takes a guest -physical address. +physical address. `Sandbox::snapshot` installs the snapshot into the sandbox +and removes the regions of `map_region` and `map_file_cow`. ### Snapshots on Disk This builds on the OCI image format that snapshots already use. The image has -one layer file per blob, named by its sha256. The config lists the layers, -their live ranges, and which layer gives the restore page tables. The writer -emits the version 2 config and memory media types. The loader still accepts -version 1 images, and makes their single blob one layer. Thus old snapshots -still load. +one layer file per blob, named by its sha256. The config lists the guest +physical address, data length, and page-table length of each blob, and which +blob has the page tables for a restore. A transport layer follows the blobs. +The writer emits the version 4 config and version 2 memory media types. The +loader rejects older config versions. Older snapshots must be regenerated. Each snapshot is one manifest that lists every blob it needs by digest. Two snapshots that share a blob name the same digest. The layout stores that file once. -Snapshot A and snapshot B saved to one directory: +Snapshot A and snapshot B saved to one directory, transport layers omitted: ``` index.json tag a -> manifest A, tag b -> manifest B blobs/sha256/ manifest A: config , layers blobs/sha256/ manifest B: config , layers -blobs/sha256/ config A: layer 0 live p0 p1 p2 p3 -blobs/sha256/ config B: layer 0 live p0 p1 p3, layer 1 live p2' -blobs/sha256/ blob 0 -blobs/sha256/ blob 1 +blobs/sha256/ config A: blob 0, page tables in blob 0 +blobs/sha256/ config B: blobs 0 and 1, page tables in blob 1 +blobs/sha256/ blob 0: data, page tables A +blobs/sha256/ blob 1: data, page tables B ``` Deleting tag a leaves blob 0 in place, because manifest B still names it. @@ -188,26 +208,22 @@ Deleting tag a leaves blob 0 in place, because manifest B still names it. restores with scratch empty. A diff holds a scratch region and restores on top of a base. -## Open questions - -Should one blob hold both the guest data and the page tables? +* Store the page tables of each snapshot in a separate blob. Kept layers then + hold no stale page tables. Rejected because a platform that runs the guest on + the snapshot page tables in place can map them with the data of the new + layer in one mapping. A separate blob needs a mapping of its own. -For separate blobs: A snapshot carries only the page tables it restores from, -so it drops the page tables that its earlier layers waste. That saves memory -and disk, though page tables are small next to the data. Guest data and page -tables also become separate concerns with separate lifetimes, which makes the -code simpler. The layer types lose the index of the page table layer, and the -rule that every other layer must stay live. +* Map only the pages each layer still gives, and leave stale pages unmapped. + Stale data is then out of reach of the guest. Rejected because: -For one blob: A future plan is for the guest to read the page tables from the -mapped blob, skipping the copy that a restore puts in scratch. A separate blob -would then need its own mapping, which costs one more mapping per snapshot and -two hypercalls per restore. This cost is hypothetical, because nothing maps the -page tables yet. + * Each stale page splits the mapping of its layer, so the mapping count + depends on the guest write pattern. A heap allocator that writes across the + heap needs many mappings, and every restore pays for them. + * Layers need per-snapshot page ranges, and the file format stores them. ## Future work An API to compact a snapshot would rebuild it as a single layer. The new layer -holds only the live pages, so the snapshot drops its dead pages and its extra -page tables. It also needs one mapping instead of many, which gives a long -chain room under the cap. This is out of scope for this HIP. +holds only the pages in use, so the snapshot drops its stale pages and their +memory. It also needs one mapping instead of many, which gives a long chain +room under the cap. This is out of scope for this HIP. diff --git a/src/hyperlight_common/src/arch/aarch64/vmem.rs b/src/hyperlight_common/src/arch/aarch64/vmem.rs index 1eec351f7..15ce844dc 100644 --- a/src/hyperlight_common/src/arch/aarch64/vmem.rs +++ b/src/hyperlight_common/src/arch/aarch64/vmem.rs @@ -589,7 +589,13 @@ pub unsafe fn walk_va_spaces( ) -> Vec<(SpaceId, Vec)> { unsafe { internal_walk_va_spaces(&op, roots.iter().cloned(), true, address, len) - .map(|(id, mappings)| (id, mappings.collect::>())) + .map(|(id, mappings)| { + let mut collected = Vec::new(); + for mapping in mappings { + crate::vmem::push_space_mapping(&mut collected, mapping); + } + (id, collected) + }) .collect::>() } } diff --git a/src/hyperlight_common/src/arch/amd64/vmem.rs b/src/hyperlight_common/src/arch/amd64/vmem.rs index 7f989df58..5de9cddb0 100644 --- a/src/hyperlight_common/src/arch/amd64/vmem.rs +++ b/src/hyperlight_common/src/arch/amd64/vmem.rs @@ -360,7 +360,10 @@ pub unsafe fn walk_va_spaces( for r in iter { if let Some(mapping) = unsafe { read_and_decode_pte(op, &r) } { - mappings.push(crate::vmem::SpaceAwareMapping::ThisSpace(mapping)); + crate::vmem::push_space_mapping( + &mut mappings, + crate::vmem::SpaceAwareMapping::ThisSpace(mapping), + ); } } diff --git a/src/hyperlight_common/src/vmem.rs b/src/hyperlight_common/src/vmem.rs index d20652cc4..d6925ae00 100644 --- a/src/hyperlight_common/src/vmem.rs +++ b/src/hyperlight_common/src/vmem.rs @@ -521,6 +521,27 @@ pub enum SpaceAwareMapping { AnotherSpace(SpaceReferenceMapping), } +/// Adds a mapping to a page-table walk result. +/// +/// Combines `ThisSpace` mappings when their physical and virtual ranges are +/// adjacent and their mapping kinds are equal. This reduces the result size. +pub(crate) fn push_space_mapping( + mappings: &mut alloc::vec::Vec, + mapping: SpaceAwareMapping, +) { + if let SpaceAwareMapping::ThisSpace(next) = &mapping + && let Some(SpaceAwareMapping::ThisSpace(previous)) = mappings.last_mut() + && previous.kind == next.kind + && previous.phys_base.checked_add(previous.len) == Some(next.phys_base) + && previous.virt_base.checked_add(previous.len) == Some(next.virt_base) + && let Some(len) = previous.len.checked_add(next.len) + { + previous.len = len; + return; + } + mappings.push(mapping); +} + /// Counterpart of [`walk_va_spaces`]'s `AnotherSpace` entries on the /// write side: installs a link in `op`'s root PT tree at `ref_map.our_va` /// that points at whatever intermediate table the owning space ended @@ -554,3 +575,94 @@ pub use arch::space_aware_map; /// Same invariants as [`virt_to_phys`]. Callers must ensure the page /// tables are not being mutated concurrently. pub use arch::walk_va_spaces; + +#[cfg(test)] +mod tests { + use super::*; + + fn basic_mapping(phys_base: u64, virt_base: u64, len: u64) -> SpaceAwareMapping { + SpaceAwareMapping::ThisSpace(Mapping { + phys_base, + virt_base, + len, + kind: MappingKind::Basic(BasicMapping { + readable: true, + writable: false, + executable: true, + }), + }) + } + + #[test] + fn push_space_mapping_coalesces_adjacent_mappings() { + let mut mappings = alloc::vec![basic_mapping(0x1000, 0x2000, PAGE_SIZE as u64)]; + + push_space_mapping( + &mut mappings, + basic_mapping( + 0x1000 + PAGE_SIZE as u64, + 0x2000 + PAGE_SIZE as u64, + PAGE_SIZE as u64, + ), + ); + + assert_eq!(mappings.len(), 1); + let SpaceAwareMapping::ThisSpace(mapping) = &mappings[0] else { + panic!("expected a local mapping"); + }; + assert_eq!(mapping.len, 2 * PAGE_SIZE as u64); + } + + #[test] + fn push_space_mapping_preserves_non_adjacent_mappings() { + for next in [ + basic_mapping(0x3000, 0x3000, PAGE_SIZE as u64), + basic_mapping(0x2000, 0x4000, PAGE_SIZE as u64), + SpaceAwareMapping::ThisSpace(Mapping { + phys_base: 0x2000, + virt_base: 0x3000, + len: PAGE_SIZE as u64, + kind: MappingKind::Cow(CowMapping { + readable: true, + executable: true, + }), + }), + ] { + let mut mappings = alloc::vec![basic_mapping(0x1000, 0x2000, PAGE_SIZE as u64)]; + push_space_mapping(&mut mappings, next); + assert_eq!(mappings.len(), 2); + } + } + + #[test] + fn push_space_mapping_preserves_space_reference_boundaries() { + let reference = SpaceAwareMapping::AnotherSpace(SpaceReferenceMapping { + depth: 1, + space: 0x1000, + our_va: 0x2000, + their_va: 0x2000, + }); + let mut mappings = alloc::vec![basic_mapping(0x1000, 0x2000, PAGE_SIZE as u64)]; + + push_space_mapping(&mut mappings, reference); + push_space_mapping( + &mut mappings, + basic_mapping( + 0x1000 + PAGE_SIZE as u64, + 0x2000 + PAGE_SIZE as u64, + PAGE_SIZE as u64, + ), + ); + + assert_eq!(mappings.len(), 3); + } + + #[test] + fn push_space_mapping_preserves_overflowing_mappings() { + let mut mappings = alloc::vec![basic_mapping(0, 0, u64::MAX)]; + + push_space_mapping(&mut mappings, basic_mapping(u64::MAX, u64::MAX, 1)); + + assert_eq!(mappings.len(), 2); + } +} diff --git a/src/hyperlight_host/benches/benchmarks.rs b/src/hyperlight_host/benches/benchmarks.rs index 791018ae6..6d161158a 100644 --- a/src/hyperlight_host/benches/benchmarks.rs +++ b/src/hyperlight_host/benches/benchmarks.rs @@ -301,6 +301,46 @@ fn bench_snapshot_restore(b: &mut criterion::Bencher, size: SandboxSize) { }); } +/// Includes the call, which pays for any memory remapping done by the previous snapshot. +fn bench_snapshot_call_and_create(b: &mut criterion::Bencher, size: SandboxSize) { + b.iter_custom(|iters| { + let mut sbox = create_multiuse_sandbox_with_size(size); + let mut total_duration = Duration::ZERO; + + for _ in 0..iters { + let start = Instant::now(); + sbox.call::("Echo", "hello\n".to_string()).unwrap(); + let snapshot = sbox.snapshot().unwrap(); + total_duration += start.elapsed(); + + std::hint::black_box(snapshot); + } + + total_duration + }); +} + +/// Restores alternate between two snapshots, so the target memory is never the current one. +fn bench_snapshot_restore_alternate(b: &mut criterion::Bencher, size: SandboxSize) { + b.iter_custom(|iters| { + let mut sbox = create_multiuse_sandbox_with_size(size); + let first = sbox.snapshot().unwrap(); + sbox.call::("Echo", "hello\n".to_string()).unwrap(); + let snapshots = [first, sbox.snapshot().unwrap()]; + let mut total_duration = Duration::ZERO; + + for i in 0..iters { + sbox.call::("Echo", "hello\n".to_string()).unwrap(); + + let start = Instant::now(); + sbox.restore(snapshots[(i % 2) as usize].clone()).unwrap(); + total_duration += start.elapsed(); + } + + total_duration + }); +} + fn bench_sandbox_from_snapshot(b: &mut criterion::Bencher, size: SandboxSize) { use hyperlight_host::HostFunctions; use hyperlight_host::sandbox::snapshot::{OciTag, Snapshot}; @@ -338,6 +378,18 @@ fn snapshots_benchmark(c: &mut Criterion) { }); } + for size in SandboxSize::all() { + group.bench_function(format!("call_and_create/{}", size.name()), |b| { + bench_snapshot_call_and_create(b, size) + }); + } + + for size in SandboxSize::all() { + group.bench_function(format!("restore_alternate/{}", size.name()), |b| { + bench_snapshot_restore_alternate(b, size) + }); + } + group.finish(); } diff --git a/src/hyperlight_host/build.rs b/src/hyperlight_host/build.rs index bd9f15b1d..10b045a25 100644 --- a/src/hyperlight_host/build.rs +++ b/src/hyperlight_host/build.rs @@ -160,20 +160,6 @@ fn main() -> Result<()> { // the gdb feature (only temporarily!) needs to use // writable/un-shared snapshot memories. unshared_snapshot_mem: { gdb }, - // The `ReadableSharedMemory` trait in `mem::layout` is only - // needed in two situations: - // 1. The `gdb` debug path reads guest memory through it - // (`ResolvedGpa::copy_to_slice`), which only exists under - // the `gdb` cfg (gdb feature + debug build). - // 2. The `mem_profile` stack unwinder reads guest memory - // through it — but ONLY when snapshots are shared/read-only - // (`not(unshared_snapshot_mem)`). When the gdb feature - // forces un-shared snapshots, `mem_profile` instead reads - // via the inherent `HostSharedMemory::copy_to_slice`, so - // the trait is genuinely unused. - // Gating the trait (and its impls) on this exact condition means - // we never have to `#[allow(dead_code)]` it. - readable_shared_mem: { any(gdb, all(feature = "mem_profile", not(unshared_snapshot_mem))) }, } #[cfg(feature = "build-metadata")] diff --git a/src/hyperlight_host/src/hypervisor/gdb/mod.rs b/src/hyperlight_host/src/hypervisor/gdb/mod.rs index 79ca6e083..8e5280be2 100644 --- a/src/hyperlight_host/src/hypervisor/gdb/mod.rs +++ b/src/hyperlight_host/src/hypervisor/gdb/mod.rs @@ -23,9 +23,8 @@ use super::regs::CommonRegisters; use crate::HyperlightError; use crate::hypervisor::regs::CommonFpu; use crate::hypervisor::virtual_machine::{HypervisorError, RegisterError, VirtualMachine}; -use crate::mem::layout::BaseGpaRegion; use crate::mem::memory_region::MemoryRegion; -use crate::mem::mgr::SandboxMemoryManager; +use crate::mem::mgr::{BackingSource, GuestPhysicalMemoryView, SandboxMemoryManager}; use crate::mem::shared_mem::HostSharedMemory; #[derive(Debug, Error)] @@ -110,13 +109,15 @@ impl<'a> DebugMemoryView<'a> { data: &mut [u8], gpa: u64, ) -> std::result::Result<(), DebugMemoryAccessError> { - self.mem_mgr - .layout - .resolve_gpa(gpa, &self.guest_mmap_regions) - .ok_or(DebugMemoryAccessError::TranslateGuestAddress(gpa))? - .with_memories(&self.mem_mgr.shared_mem, &self.mem_mgr.scratch_mem) - .copy_to_slice(data) - .map_err(|e| DebugMemoryAccessError::CopyFailed(Box::new(e))) + GuestPhysicalMemoryView::with_dynamic( + &self.mem_mgr.shared_mem, + &self.mem_mgr.scratch_mem, + &self.guest_mmap_regions, + self.mem_mgr.layout, + ) + .read(gpa, data) + .map(|_| ()) + .map_err(|e| DebugMemoryAccessError::CopyFailed(Box::new(e))) } /// Writes memory from the guest's address space with a maximum length of a PAGE_SIZE @@ -132,27 +133,29 @@ impl<'a> DebugMemoryView<'a> { data: &[u8], gpa: u64, ) -> std::result::Result<(), DebugMemoryAccessError> { - let resolved = self - .mem_mgr - .layout - .resolve_gpa(gpa, &self.guest_mmap_regions) - .ok_or(DebugMemoryAccessError::TranslateGuestAddress(gpa))?; + let backing = GuestPhysicalMemoryView::with_dynamic( + &self.mem_mgr.shared_mem, + &self.mem_mgr.scratch_mem, + &self.guest_mmap_regions, + self.mem_mgr.layout, + ) + .resolve(gpa, data.len()) + .ok_or(DebugMemoryAccessError::TranslateGuestAddress(gpa))?; // We can only safely write (without causing UB in the host // process) if the address is in the scratch region - match resolved.base { - #[cfg(unshared_snapshot_mem)] - BaseGpaRegion::Snapshot(()) => self + match backing.source { + BackingSource::Snapshot(layer_index) => self .mem_mgr .shared_mem - .copy_from_slice(data, resolved.offset) - .map_err(|e| DebugMemoryAccessError::CopyFailed(Box::new(e.into()))), - BaseGpaRegion::Scratch(()) => self + .write_layer(layer_index, backing.offset, data) + .map_err(|e| DebugMemoryAccessError::CopyFailed(Box::new(e))), + BackingSource::Scratch => self .mem_mgr .scratch_mem - .copy_from_slice(data, resolved.offset) + .copy_from_slice(data, backing.offset) .map_err(|e| DebugMemoryAccessError::CopyFailed(Box::new(e.into()))), - _ => Err(DebugMemoryAccessError::WriteToReadOnly), + BackingSource::Dynamic(_) => Err(DebugMemoryAccessError::WriteToReadOnly), } } } @@ -503,7 +506,7 @@ mod tests { #[test] fn test_mem_access_write_single_byte() -> crate::Result<()> { let memory = TestMemory::new()?; - let scratch_gpa = memory.mem_mgr.layout.get_first_free_scratch_gpa(); + let scratch_gpa = memory.mem_mgr.first_free_scratch_gpa(); let write_data = [0xCCu8; 1]; memory.access().write(&write_data, scratch_gpa).unwrap(); @@ -518,7 +521,7 @@ mod tests { #[test] fn test_mem_access_write_multiple_bytes() -> crate::Result<()> { let memory = TestMemory::new()?; - let scratch_gpa = memory.mem_mgr.layout.get_first_free_scratch_gpa(); + let scratch_gpa = memory.mem_mgr.first_free_scratch_gpa(); let write_data = [0xAAu8; 16]; memory.access().write(&write_data, scratch_gpa).unwrap(); diff --git a/src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs b/src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs index 0c6870f23..11b68b492 100644 --- a/src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs +++ b/src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs @@ -30,11 +30,11 @@ use crate::hypervisor::virtual_machine::{ HypervisorType, RegisterError, ResetVcpuError, VirtualMachine, VmError, get_available_hypervisor, }; -use crate::mem::mgr::{SandboxMemoryManager, SnapshotSharedMemory}; +use crate::mem::mgr::SandboxMemoryManager; use crate::mem::shared_mem::{GuestSharedMemory, HostSharedMemory}; use crate::sandbox::SandboxConfiguration; use crate::sandbox::host_funcs::FunctionRegistry; -use crate::sandbox::snapshot::NextAction; +use crate::sandbox::snapshot::{NextAction, SnapshotMemoryBacking}; #[cfg(feature = "mem_profile")] use crate::sandbox::trace::MemTraceInfo; #[cfg(crashdump)] @@ -44,7 +44,7 @@ impl HyperlightVm { #[allow(clippy::too_many_arguments)] #[cfg_attr(target_os = "macos", allow(unused))] pub(crate) fn new( - snapshot_mem: SnapshotSharedMemory, + snapshot_mem: SnapshotMemoryBacking, scratch_mem: GuestSharedMemory, root_pt_addr: u64, next_action: NextAction, @@ -83,8 +83,7 @@ impl HyperlightVm { let interrupt_handle: Arc = Arc::new(WindowsInterruptHandle::new(vm.partition_handle())); - let snapshot_slot = 0u32; - let scratch_slot = 1u32; + let scratch_slot = 0u32; let vm_can_reset_vcpu = vm.can_reset_vcpu(); let mut ret = Self { vm, @@ -96,7 +95,7 @@ impl HyperlightVm { next_slot: scratch_slot + 1, freed_slots: Vec::new(), - snapshot_slot, + snapshot_mappings: Vec::new(), snapshot_memory: None, scratch_slot, scratch_memory: None, diff --git a/src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs b/src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs index 9d6157364..894c6f4e2 100644 --- a/src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs +++ b/src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs @@ -36,12 +36,12 @@ use crate::hypervisor::virtual_machine::{ }; use crate::hypervisor::{InterruptHandle, InterruptHandleImpl}; use crate::mem::memory_region::{MemoryRegion, MemoryRegionFlags, MemoryRegionType}; -use crate::mem::mgr::{SandboxMemoryManager, SnapshotSharedMemory}; +use crate::mem::mgr::SandboxMemoryManager; use crate::mem::shared_mem::{GuestSharedMemory, HostSharedMemory, SharedMemory}; use crate::metrics::{METRIC_ERRONEOUS_VCPU_KICKS, METRIC_GUEST_CANCELLATION}; use crate::sandbox::host_funcs::FunctionRegistry; use crate::sandbox::outb::{HandleOutbError, handle_outb}; -use crate::sandbox::snapshot::NextAction; +use crate::sandbox::snapshot::{NextAction, SnapshotMemoryBacking}; #[cfg(feature = "mem_profile")] use crate::sandbox::trace::MemTraceInfo; #[cfg(crashdump)] @@ -370,10 +370,11 @@ pub(crate) struct HyperlightVm { pub(super) next_slot: u32, // Monotonically increasing slot number pub(super) freed_slots: Vec, // Reusable slots from unmapped regions - pub(super) snapshot_slot: u32, + // Hypervisor slot and mapping for each snapshot layer. + pub(super) snapshot_mappings: Vec<(u32, MemoryRegion)>, // The current snapshot region, used to keep it alive as long as // it is used & when unmapping - pub(super) snapshot_memory: Option>, + pub(super) snapshot_memory: Option>, pub(super) scratch_slot: u32, // The slot number used for the scratch region // The current scratch region, used to keep it alive as long as it // is used & when unmapping @@ -406,6 +407,17 @@ pub(crate) struct HyperlightVm { } impl HyperlightVm { + /// Returns an unused hypervisor memory slot. + fn allocate_slot(&mut self) -> u32 { + if let Some(slot) = self.freed_slots.pop() { + slot + } else { + let slot = self.next_slot; + self.next_slot += 1; + slot + } + } + /// Map a region of host memory into the sandbox. /// /// Safety: The caller must ensure that the region points to valid memory and @@ -448,10 +460,7 @@ impl HyperlightVm { // Check against the snapshot region if let Some(ref snapshot) = self.snapshot_memory { let snap_start = crate::mem::layout::SandboxMemoryLayout::BASE_ADDRESS; - #[cfg(not(unshared_snapshot_mem))] - let snap_end = snap_start + snapshot.guest_mapped_size(); - #[cfg(unshared_snapshot_mem)] - let snap_end = snap_start + snapshot.mem_size(); + let snap_end = snap_start + snapshot.gpa_span_len(); if new_start < snap_end && new_end > snap_start { return Err(MapRegionError::Overlapping { new_start, @@ -478,13 +487,7 @@ impl HyperlightVm { } // Try to reuse a freed slot first, otherwise use next_slot - let slot = if let Some(freed_slot) = self.freed_slots.pop() { - freed_slot - } else { - let slot = self.next_slot; - self.next_slot += 1; - slot - }; + let slot = self.allocate_slot(); // Safety: slots are unique. It's up to caller to ensure that the region is valid unsafe { self.vm.map_memory((slot, region))? }; @@ -515,23 +518,42 @@ impl HyperlightVm { self.mmap_regions.iter().map(|(_, region)| region) } - /// Update the snapshot mapping to point to a new GuestSharedMemory + /// Updates the VM mappings to match the specified snapshot. + /// + /// This method keeps shared mappings. It removes obsolete mappings and + /// adds new mappings. pub(crate) fn update_snapshot_mapping( &mut self, - snapshot: SnapshotSharedMemory, + snapshot: SnapshotMemoryBacking, ) -> Result<(), UpdateRegionError> { - let guest_base = crate::mem::layout::SandboxMemoryLayout::BASE_ADDRESS as u64; - let rgn = snapshot.mapping_at(guest_base, MemoryRegionType::Snapshot); + let target_mappings = snapshot.mappings(); - // Keep the backing memory alive until its VM mapping is removed. - if let Some(old_snapshot) = self.snapshot_memory.as_ref() { - let old_rgn = old_snapshot.mapping_at(guest_base, MemoryRegionType::Snapshot); - self.vm.unmap_memory((self.snapshot_slot, &old_rgn))?; + for index in (0..self.snapshot_mappings.len()).rev() { + let (slot, current) = &self.snapshot_mappings[index]; + if target_mappings.contains(current) { + continue; + } + self.vm.unmap_memory((*slot, current))?; + let (slot, _) = self.snapshot_mappings.remove(index); + self.freed_slots.push(slot); } - self.snapshot_memory = None; - unsafe { self.vm.map_memory((self.snapshot_slot, &rgn))? }; + self.snapshot_memory = Some(snapshot); + for mapping in target_mappings { + if self + .snapshot_mappings + .iter() + .any(|(_, current)| current == &mapping) + { + continue; + } + let slot = self.allocate_slot(); + // SAFETY: `self.snapshot_memory` owns the backing for `mapping`. + unsafe { self.vm.map_memory((slot, &mapping))? }; + self.snapshot_mappings.push((slot, mapping)); + } + Ok(()) } diff --git a/src/hyperlight_host/src/hypervisor/hyperlight_vm/test_support.rs b/src/hyperlight_host/src/hypervisor/hyperlight_vm/test_support.rs index e410d6266..79e60ed7b 100644 --- a/src/hyperlight_host/src/hypervisor/hyperlight_vm/test_support.rs +++ b/src/hyperlight_host/src/hypervisor/hyperlight_vm/test_support.rs @@ -23,7 +23,6 @@ pub(crate) enum VmOperation { SetDebugRegs, #[cfg(target_arch = "x86_64")] ResetXsave, - #[cfg(target_arch = "x86_64")] SetSregs, #[cfg(target_arch = "x86_64")] SetMsrs, @@ -163,7 +162,6 @@ impl VirtualMachine for FaultInjectingVirtualMachine { &mut self, sregs: &CommonSpecialRegisters, ) -> std::result::Result<(), RegisterError> { - #[cfg(target_arch = "x86_64")] if self.should_fail(VmOperation::SetSregs) { return Err(RegisterError::SetSregs(Self::injected_error())); } @@ -294,11 +292,12 @@ impl HyperlightVm { } #[allow(clippy::type_complexity, reason = "test-only mapping state")] - pub(crate) fn base_mapping_state(&self) -> (Option<(usize, usize)>, Option<(usize, usize)>) { + pub(crate) fn base_mapping_state(&self) -> (Vec<(u32, MemoryRegion)>, Option<(usize, usize)>) { let snapshot = self - .snapshot_memory - .as_ref() - .map(|memory| (memory.base_addr(), memory.mem_size())); + .snapshot_mappings + .iter() + .map(|(slot, mapping)| (*slot, mapping.clone())) + .collect(); let scratch = self .scratch_memory .as_ref() diff --git a/src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs b/src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs index 8ca3024fc..452f8f508 100644 --- a/src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs +++ b/src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs @@ -41,7 +41,7 @@ use crate::mem::ptr::RawPtr; use crate::mem::shared_mem::{GuestSharedMemory, HostSharedMemory}; use crate::sandbox::SandboxConfiguration; use crate::sandbox::host_funcs::FunctionRegistry; -use crate::sandbox::snapshot::NextAction; +use crate::sandbox::snapshot::{NextAction, SnapshotMemoryBacking}; #[cfg(feature = "mem_profile")] use crate::sandbox::trace::MemTraceInfo; #[cfg(crashdump)] @@ -57,7 +57,7 @@ impl HyperlightVm { #[instrument(err(Debug), skip_all, parent = Span::current(), level = "Trace")] #[allow(clippy::too_many_arguments)] pub(crate) fn new( - snapshot_mem: SnapshotSharedMemory, + snapshot_mem: SnapshotMemoryBacking, scratch_mem: GuestSharedMemory, _root_pt_addr: u64, next_action: NextAction, @@ -105,8 +105,7 @@ impl HyperlightVm { }) .map_err(VmError::Register)?; - let snapshot_slot = 0u32; - let scratch_slot = 1u32; + let scratch_slot = 0u32; #[cfg_attr(not(gdb), allow(unused_mut))] let mut ret = Self { vm, @@ -118,7 +117,7 @@ impl HyperlightVm { next_slot: scratch_slot + 1, freed_slots: Vec::new(), - snapshot_slot, + snapshot_mappings: Vec::new(), snapshot_memory: None, scratch_slot, scratch_memory: None, @@ -877,7 +876,7 @@ mod tests { use crate::mem::memory_region::MemoryRegionFlags; use crate::mem::mgr::{GuestPageTableBuffer, SandboxMemoryManager}; use crate::mem::ptr::RawPtr; - use crate::mem::shared_mem::{ExclusiveSharedMemory, ReadonlySharedMemory}; + use crate::mem::shared_mem::ExclusiveSharedMemory; use crate::sandbox::SandboxConfiguration; use crate::sandbox::host_funcs::FunctionRegistry; #[cfg(any(crashdump, gdb))] @@ -1426,7 +1425,7 @@ mod tests { fn create_test_layout(code_size: usize) -> (SandboxMemoryLayout, Box<[u8]>) { let config = SandboxConfiguration::default(); - let mut layout = SandboxMemoryLayout::new(config, code_size, 4096, None).unwrap(); + let layout = SandboxMemoryLayout::new(config, code_size, 4096, None).unwrap(); let pt_base_gpa = layout.get_pt_base_gpa(); let pt_buf = GuestPageTableBuffer::new(pt_base_gpa as usize); @@ -1464,7 +1463,7 @@ mod tests { unsafe { vmem::map(&pt_buf, scratch_mapping) }; let pt_bytes = pt_buf.into_bytes(); - layout.set_pt_size(pt_bytes.len()).unwrap(); + layout.ensure_page_tables_fit(pt_bytes.len()).unwrap(); (layout, pt_bytes) } @@ -1479,18 +1478,20 @@ mod tests { let mem_size = layout.get_memory_size().unwrap(); let mut snapshot_contents = vec![0u8; mem_size]; - let snapshot_pt_start = mem_size - layout.get_pt_size(); - snapshot_contents[snapshot_pt_start..].copy_from_slice(&pt_bytes); snapshot_contents[layout.guest_code_offset()..layout.guest_code_offset() + code.len()] .copy_from_slice(code); layout.write_peb(&mut snapshot_contents).unwrap(); - let ro_mem = - ReadonlySharedMemory::from_bytes(&snapshot_contents, snapshot_pt_start).unwrap(); + let snapshot_memory = crate::sandbox::snapshot::memory::tests::flat_memory( + &snapshot_contents, + &pt_bytes, + SandboxMemoryLayout::BASE_ADDRESS as u64, + hyperlight_common::layout::scratch_base_gpa(config.get_scratch_size()), + ); let scratch_mem = ExclusiveSharedMemory::new(config.get_scratch_size()).unwrap(); let mem_mgr = SandboxMemoryManager::new( layout, - ro_mem.to_mgr_snapshot_mem().unwrap(), + SnapshotMemoryBacking::from_snapshot(Arc::new(snapshot_memory)).unwrap(), scratch_mem, NextAction::Initialise(layout.get_guest_code_gva() as u64), ); @@ -2143,11 +2144,11 @@ mod tests { const CODE_SIZE_BOUND: usize = 512; - let (layout, _) = create_test_layout(CODE_SIZE_BOUND); + let (layout, pt_bytes) = create_test_layout(CODE_SIZE_BOUND); let scratch_size = layout.get_scratch_size(); let scratch_base_gpa = hyperlight_common::layout::scratch_base_gpa(scratch_size); let scratch_base_gva = hyperlight_common::layout::scratch_base_gva(scratch_size); - let fxsave_gpa = layout.get_first_free_scratch_gpa(); + let fxsave_gpa = layout.get_pt_base_gpa() + pt_bytes.len() as u64; let fxsave_offset = usize::try_from(fxsave_gpa - scratch_base_gpa).unwrap(); let fxsave_gva = scratch_base_gva + fxsave_offset as u64; let flag_gva = fxsave_gva + 512; @@ -2210,7 +2211,7 @@ mod tests { // Reuse common test setup - initialise() will run the code let ctx = create_test_vm_context(&code); - assert_eq!(ctx.hshm.layout.get_first_free_scratch_gpa(), fxsave_gpa); + assert_eq!(ctx.hshm.first_free_scratch_gpa(), fxsave_gpa); FxsaveTestContext { ctx, fxsave_offset } } diff --git a/src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs b/src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs index e352ab20c..9bd348647 100644 --- a/src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs +++ b/src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs @@ -7,8 +7,8 @@ use hyperlight_common::outb::VmAction; #[cfg(gdb)] use kvm_bindings::kvm_guest_debug; use kvm_bindings::{ - Msrs, kvm_debugregs, kvm_fpu, kvm_msr_entry, kvm_regs, kvm_sregs, kvm_userspace_memory_region, - kvm_xsave, + KVM_CAP_DISABLE_QUIRKS2, KVM_X86_QUIRK_SLOT_ZAP_ALL, Msrs, kvm_debugregs, kvm_enable_cap, + kvm_fpu, kvm_msr_entry, kvm_regs, kvm_sregs, kvm_userspace_memory_region, kvm_xsave, }; use kvm_ioctls::Cap::UserMemory; use kvm_ioctls::{ @@ -175,6 +175,29 @@ impl KvmVm { .create_vm_with_type(0) .map_err(|e| CreateVmError::CreateVmFd(e.into()))?; + // Deleting a memslot flushes the guest mappings of every memslot unless + // this quirk is disabled. KVM keeps the flush as a workaround for + // assigned GPUs. Hyperlight assigns no devices. + let quirks = vm_fd.check_extension_raw(KVM_CAP_DISABLE_QUIRKS2.into()); + // `quirks` has one bit per quirk this kernel can disable. + if quirks <= 0 || quirks & KVM_X86_QUIRK_SLOT_ZAP_ALL as i32 == 0 { + tracing::debug!( + "Host kernel cannot disable KVM_X86_QUIRK_SLOT_ZAP_ALL. Each memory slot \ + deletion invalidates guest mappings in all slots, which adds latency to the \ + first guest call after a snapshot, restore, or unmap. Host kernel version \ + 6.12 or later enables per-slot invalidation." + ); + } else { + let cap = kvm_enable_cap { + cap: KVM_CAP_DISABLE_QUIRKS2, + args: [KVM_X86_QUIRK_SLOT_ZAP_ALL.into(), 0, 0, 0], + ..Default::default() + }; + vm_fd + .enable_cap(&cap) + .map_err(|e| CreateVmError::InitializeVm(e.into()))?; + } + #[cfg(feature = "hw-interrupts")] let timer_irq_eventfd = Self::setup_irqfd(&vm_fd)?; diff --git a/src/hyperlight_host/src/mem/layout.rs b/src/hyperlight_host/src/mem/layout.rs index 6d22ef5f9..443c9c4b3 100644 --- a/src/hyperlight_host/src/mem/layout.rs +++ b/src/hyperlight_host/src/mem/layout.rs @@ -57,184 +57,13 @@ use tracing::{Span, instrument}; use super::memory_region::MemoryRegionType::{Code, Heap, InitData, Peb}; use super::memory_region::{ - DEFAULT_GUEST_BLOB_MEM_FLAGS, GuestMemoryRegion, MemoryRegion, MemoryRegion_, - MemoryRegionFlags, MemoryRegionVecBuilder, + DEFAULT_GUEST_BLOB_MEM_FLAGS, GuestMemoryRegion, MemoryRegion_, MemoryRegionFlags, + MemoryRegionVecBuilder, }; -#[cfg(readable_shared_mem)] -use super::shared_mem::HostSharedMemory; -use super::shared_mem::{ExclusiveSharedMemory, ReadonlySharedMemory}; use crate::error::HyperlightError::{MemoryRequestTooBig, MemoryRequestTooSmall}; use crate::sandbox::SandboxConfiguration; use crate::{Result, new_error}; -pub(crate) enum BaseGpaRegion { - Snapshot(Sn), - Scratch(Sc), - Mmap(MemoryRegion), -} - -// It's an invariant of this type, checked on creation, that the -// offset is in bounds for the base region. -pub(crate) struct ResolvedGpa { - pub(crate) offset: usize, - pub(crate) base: BaseGpaRegion, -} - -impl AsRef<[u8]> for ExclusiveSharedMemory { - fn as_ref(&self) -> &[u8] { - self.as_slice() - } -} -impl AsRef<[u8]> for ReadonlySharedMemory { - fn as_ref(&self) -> &[u8] { - self.as_slice() - } -} - -impl ResolvedGpa { - pub(crate) fn with_memories(self, sn: Sn2, sc: Sc2) -> ResolvedGpa { - ResolvedGpa { - offset: self.offset, - base: match self.base { - BaseGpaRegion::Snapshot(_) => BaseGpaRegion::Snapshot(sn), - BaseGpaRegion::Scratch(_) => BaseGpaRegion::Scratch(sc), - BaseGpaRegion::Mmap(r) => BaseGpaRegion::Mmap(r), - }, - } - } -} -impl<'a> BaseGpaRegion<&'a [u8], &'a [u8]> { - pub(crate) fn as_ref<'b>(&'b self) -> &'a [u8] { - match self { - BaseGpaRegion::Snapshot(sn) => sn, - BaseGpaRegion::Scratch(sc) => sc, - BaseGpaRegion::Mmap(r) => unsafe { - #[allow(clippy::useless_conversion)] - let host_region_base: usize = r.host_region.start.into(); - #[allow(clippy::useless_conversion)] - let host_region_end: usize = r.host_region.end.into(); - let len = host_region_end - host_region_base; - std::slice::from_raw_parts(host_region_base as *const u8, len) - }, - } - } -} -impl<'a> ResolvedGpa<&'a [u8], &'a [u8]> { - pub(crate) fn as_ref<'b>(&'b self) -> &'a [u8] { - let base = self.base.as_ref(); - if self.offset > base.len() { - return &[]; - } - &self.base.as_ref()[self.offset..] - } -} -/// A read-only abstraction over the different kinds of backing memory -/// a [`ResolvedGpa`] can point at (the host snapshot mapping, the -/// scratch mapping, or a raw `&[u8]` view of either), letting callers -/// copy guest bytes out without caring which concrete memory type they -/// hold. -/// -/// This trait only exists in builds that actually read guest memory -/// through it — see the `readable_shared_mem` cfg alias in `build.rs` -/// for the exact conditions (the `gdb` debug path and the -/// shared-snapshot `mem_profile` path). In every other configuration it -/// is compiled out entirely, so there is no dead code to `#[allow]`. -#[cfg(readable_shared_mem)] -pub(crate) trait ReadableSharedMemory { - fn copy_to_slice(&self, slice: &mut [u8], offset: usize) -> Result<()>; -} -#[cfg(readable_shared_mem)] -impl ReadableSharedMemory for &HostSharedMemory { - fn copy_to_slice(&self, slice: &mut [u8], offset: usize) -> Result<()> { - Ok(HostSharedMemory::copy_to_slice(self, slice, offset)?) - } -} -/// Coherence workaround for the blanket impl below. -/// -/// We want `ReadableSharedMemory` for both `&HostSharedMemory` (above) -/// and for any `T: AsRef<[u8]>` (so that `ExclusiveSharedMemory` / -/// `ReadonlySharedMemory` and their references are covered by a single -/// impl). A naive `impl> ReadableSharedMemory for T` -/// would *overlap* the `&HostSharedMemory` impl — the compiler can't -/// prove `&HostSharedMemory` never implements `AsRef<[u8]>` — and is -/// rejected with E0119. -/// -/// To break the overlap we introduce a private marker trait and -/// implement it *only* for the specific types we want the blanket impl -/// to cover (deliberately excluding `&HostSharedMemory`). The blanket -/// impl is then bounded on this marker rather than on `AsRef<[u8]>` -/// directly, so the two impls provably never overlap. -#[cfg(readable_shared_mem)] -mod coherence_hack { - use super::{ExclusiveSharedMemory, ReadonlySharedMemory}; - // Used only as a bound on the blanket impl below, so the name reads - // as unused even though removing it breaks compilation. - #[allow(unused)] - pub(super) trait SharedMemoryAsRefMarker: AsRef<[u8]> {} - impl SharedMemoryAsRefMarker for ExclusiveSharedMemory {} - impl SharedMemoryAsRefMarker for &ExclusiveSharedMemory {} - impl SharedMemoryAsRefMarker for ReadonlySharedMemory {} - impl SharedMemoryAsRefMarker for &ReadonlySharedMemory {} -} -#[cfg(readable_shared_mem)] -impl ReadableSharedMemory for T { - fn copy_to_slice(&self, slice: &mut [u8], offset: usize) -> Result<()> { - let ss: &[u8] = self.as_ref(); - let end = offset + slice.len(); - if end > ss.len() { - return Err(new_error!( - "Attempt to read up to {} in memory of size {}", - offset + slice.len(), - self.as_ref().len() - )); - } - slice.copy_from_slice(&ss[offset..end]); - Ok(()) - } -} -/// Copy `slice.len()` bytes out of the resolved guest region. -/// -/// Only the `gdb` debug path uses this one-argument convenience (it -/// already carries the offset inside `self`); `mem_profile` reads via -/// the two-argument inherent methods instead. Hence it is gated on the -/// `gdb` cfg alone, even though the [`ReadableSharedMemory`] trait it -/// relies on is available slightly more widely. -#[cfg(gdb)] -impl ResolvedGpa { - pub(crate) fn copy_to_slice(&self, slice: &mut [u8]) -> Result<()> { - match &self.base { - BaseGpaRegion::Snapshot(sn) => sn.copy_to_slice(slice, self.offset), - BaseGpaRegion::Scratch(sc) => sc.copy_to_slice(slice, self.offset), - BaseGpaRegion::Mmap(r) => unsafe { - #[allow(clippy::useless_conversion)] - let host_region_base: usize = r.host_region.start.into(); - #[allow(clippy::useless_conversion)] - let host_region_end: usize = r.host_region.end.into(); - let len = host_region_end - host_region_base; - // Safety: it's a documented invariant of MemoryRegion - // that the memory must remain alive as long as the - // sandbox is alive, and the way this code is used, - // the lifetimes of the snapshot and scratch memories - // ensure that the sandbox is still alive. This could - // perhaps be cleaned up/improved/made harder to - // misuse significantly, but it would require a much - // larger rework. - let ss = std::slice::from_raw_parts(host_region_base as *const u8, len); - let end = self.offset + slice.len(); - if end > ss.len() { - return Err(new_error!( - "Attempt to read up to {} in memory of size {}", - self.offset + slice.len(), - ss.len() - )); - } - slice.copy_from_slice(&ss[self.offset..end]); - Ok(()) - }, - } - } -} - #[derive(Copy, Clone)] pub(crate) struct SandboxMemoryLayout { /// The heap size of this sandbox. @@ -259,16 +88,6 @@ pub(crate) struct SandboxMemoryLayout { h2g_buffer_size: usize, /// Fixed ring and pool placement within scratch. transport_arena: TransportArena, - /// Size of the primary guest memory region at `BASE_ADDRESS` - /// (code, PEB, heap, init data). For a snapshot-backed layout - /// this is also the guest-visible prefix of the host snapshot - /// mapping. - snapshot_size: usize, - /// Size of the page-table region. Sits at the tail of the host - /// snapshot mapping but is never mapped to the guest from there. - /// On restore the host copies it into scratch, where the guest - /// sees it at `SNAPSHOT_PT_GVA`. `None` until page tables are built. - pt_size: Option, } impl Debug for SandboxMemoryLayout { @@ -292,8 +111,6 @@ impl Debug for SandboxMemoryLayout { .field("H2G Buffer Size", &self.h2g_buffer_size) .field("G2H Pool Pages", &self.get_g2h_pool_pages()) .field("H2G Pool Pages", &self.get_h2g_pool_pages()) - .field("Snapshot Size", &format_args!("{:#x}", self.snapshot_size)) - .field("PT Size", &format_args!("{:#x}", self.pt_size.unwrap_or(0))) .field( "Guest Code Offset", &format_args!("{:#x}", self.guest_code_offset()), @@ -361,22 +178,20 @@ impl SandboxMemoryLayout { return Err(MemoryRequestTooSmall(scratch_size, min_scratch_size)); } - let mut ret = Self { + let ret = Self { heap_size, code_size, code_gva: Self::BASE_ADDRESS, init_data_size, init_data_permissions, - pt_size: None, scratch_size, g2h_dims, h2g_dims, g2h_buffer_size, h2g_buffer_size, transport_arena, - snapshot_size: 0, }; - ret.set_snapshot_size(ret.get_memory_size()?); + ret.get_memory_size()?; Ok(ret) } @@ -432,42 +247,16 @@ impl SandboxMemoryLayout { self.h2g_dims } - /// Guest-visible prefix size of the snapshot blob. - pub(crate) fn snapshot_size(&self) -> usize { - self.snapshot_size - } - - /// Recorded page-table tail size, `None` until page tables are built. - pub(crate) fn pt_size(&self) -> Option { - self.pt_size - } - - /// Page-table tail size, or 0 if page tables are not yet built. - pub(crate) fn get_pt_size(&self) -> usize { - self.pt_size.unwrap_or(0) - } - - /// Record the size of the page-table tail appended to the - /// snapshot blob. The PT bytes live at the end of the blob and - /// the host mapping, outside the guest mapping of the snapshot - /// region, and are copied into the scratch region on restore. - /// `snapshot_size` (the guest-visible prefix of the blob) is an - /// independent field and must be set separately. - pub(crate) fn set_pt_size(&mut self, size: usize) -> Result<()> { + pub(crate) fn ensure_page_tables_fit(&self, page_table_len: usize) -> Result<()> { let min_fixed_scratch = hyperlight_common::layout::min_scratch_size(self.transport_arena.size()); - let min_scratch = min_fixed_scratch.saturating_add(size); + let min_scratch = min_fixed_scratch.saturating_add(page_table_len); if self.scratch_size < min_scratch { return Err(MemoryRequestTooSmall(self.scratch_size, min_scratch)); } - self.pt_size = Some(size); Ok(()) } - pub(crate) fn set_snapshot_size(&mut self, new_size: usize) { - self.snapshot_size = new_size; - } - /// Returns the memory regions associated with this memory layout, /// suitable for passing to a hypervisor for mapping into memory pub(crate) fn get_memory_regions(&self) -> Result>> { @@ -656,37 +445,6 @@ impl SandboxMemoryLayout { Ok(()) } - - /// Determine what region this gpa is in, and its offset into that region - pub(crate) fn resolve_gpa( - &self, - gpa: u64, - mmap_regions: &[MemoryRegion], - ) -> Option> { - let scratch_base = hyperlight_common::layout::scratch_base_gpa(self.scratch_size); - if gpa >= scratch_base && gpa < scratch_base + self.scratch_size as u64 { - return Some(ResolvedGpa { - offset: (gpa - scratch_base) as usize, - base: BaseGpaRegion::Scratch(()), - }); - } else if gpa >= SandboxMemoryLayout::BASE_ADDRESS as u64 - && gpa < SandboxMemoryLayout::BASE_ADDRESS as u64 + self.snapshot_size as u64 - { - return Some(ResolvedGpa { - offset: gpa as usize - SandboxMemoryLayout::BASE_ADDRESS, - base: BaseGpaRegion::Snapshot(()), - }); - } - for rgn in mmap_regions { - if gpa >= rgn.guest_region.start as u64 && gpa < rgn.guest_region.end as u64 { - return Some(ResolvedGpa { - offset: gpa as usize - rgn.guest_region.start, - base: BaseGpaRegion::Mmap(rgn.clone()), - }); - } - } - None - } } /// Changes to the below methods is part of Snapshot ABI, and @@ -741,11 +499,6 @@ impl SandboxMemoryLayout { self.transport_arena.end_addr() } - /// First GPA available to the guest scratch allocator. - pub(crate) fn get_first_free_scratch_gpa(&self) -> u64 { - self.get_pt_base_gpa() + self.pt_size.unwrap_or(0) as u64 - } - /// Exact transport placement in the fixed scratch prefix. pub(crate) fn get_transport_arena(&self) -> TransportArena { self.transport_arena @@ -809,7 +562,7 @@ mod tests { #[test] fn transport_arena_starts_at_scratch_base() { let cfg = SandboxConfiguration::default(); - let mut layout = SandboxMemoryLayout::new(cfg, 4096, 0, None).unwrap(); + let layout = SandboxMemoryLayout::new(cfg, 4096, 0, None).unwrap(); let arena = layout.get_transport_arena(); let scratch_base = hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()); @@ -817,13 +570,9 @@ mod tests { assert_eq!(layout.get_pt_base_gpa(), arena.end_addr()); assert_eq!(layout.get_pt_base_scratch_offset(), arena.size()); - layout.set_pt_size(PAGE_SIZE).unwrap(); + layout.ensure_page_tables_fit(PAGE_SIZE).unwrap(); assert_eq!(layout.get_transport_arena(), arena); - assert_eq!( - layout.get_first_free_scratch_gpa(), - arena.end_addr() + PAGE_SIZE as u64 - ); } #[test] @@ -833,10 +582,10 @@ mod tests { let minimum = hyperlight_common::layout::min_scratch_size(layout.get_transport_arena().size()); cfg.set_scratch_size(minimum); - let mut layout = SandboxMemoryLayout::new(cfg, 4096, 0, None).unwrap(); + let layout = SandboxMemoryLayout::new(cfg, 4096, 0, None).unwrap(); assert!(matches!( - layout.set_pt_size(PAGE_SIZE), + layout.ensure_page_tables_fit(PAGE_SIZE), Err(MemoryRequestTooSmall(..)) )); } @@ -980,8 +729,6 @@ mod tests { pin_eq!(layout.get_memory_size().unwrap(), 0x4000); pin_eq!(layout.get_scratch_size(), 0x30000); - pin_eq!(layout.get_pt_size(), 0); - pin_eq!(layout.get_pt_base_scratch_offset(), 0x15000); let arena = layout.get_transport_arena(); @@ -1000,13 +747,6 @@ mod tests { layout.get_pt_base_gpa() - hyperlight_common::layout::scratch_base_gpa(0x30000), 0x15000 ); - // pt_size is zero here, so the first free scratch GPA equals - // the page table base. - pin_eq!( - layout.get_first_free_scratch_gpa(), - layout.get_pt_base_gpa() - ); - // A second snapshot layout keeps the transport prefix fixed // relative to its scratch base. let mut cfg = SandboxConfiguration::default(); @@ -1025,8 +765,6 @@ mod tests { ); pin_eq!(layout.get_scratch_size(), 0x40000); - pin_eq!(layout.get_pt_size(), 0); - pin_eq!(layout.get_pt_base_scratch_offset(), 0x15000); let arena = layout.get_transport_arena(); @@ -1042,9 +780,5 @@ mod tests { layout.get_pt_base_gpa() - hyperlight_common::layout::scratch_base_gpa(0x40000), 0x15000 ); - pin_eq!( - layout.get_first_free_scratch_gpa(), - layout.get_pt_base_gpa() - ); } } diff --git a/src/hyperlight_host/src/mem/mgr.rs b/src/hyperlight_host/src/mem/mgr.rs index 970525211..50a5c759f 100644 --- a/src/hyperlight_host/src/mem/mgr.rs +++ b/src/hyperlight_host/src/mem/mgr.rs @@ -15,15 +15,13 @@ use hyperlight_common::vmem::{BasicMapping, MappingKind}; use tracing::{Span, instrument}; use super::layout::SandboxMemoryLayout; -use super::shared_mem::{ - ExclusiveSharedMemory, GuestSharedMemory, HostSharedMemory, ReadonlySharedMemory, SharedMemory, -}; +use super::shared_mem::{ExclusiveSharedMemory, GuestSharedMemory, HostSharedMemory, SharedMemory}; use super::virtq::{self, G2hConsumer, H2gConsumer, VirtqSnapshot}; use crate::hypervisor::regs::CommonSpecialRegisters; use crate::mem::memory_region::MemoryRegion; #[cfg(crashdump)] use crate::mem::memory_region::{CrashDumpRegion, MemoryRegionFlags, MemoryRegionType}; -use crate::sandbox::snapshot::{NextAction, Snapshot}; +use crate::sandbox::snapshot::{NextAction, Snapshot, SnapshotMemoryBacking}; use crate::{HyperlightError, Result, new_error}; #[cfg(crashdump)] @@ -97,41 +95,265 @@ fn try_coalesce_region( false } -// It would be nice to have a simple type alias -// `SnapshotSharedMemory` that abstracts over the -// fact that the snapshot shared memory is `ReadonlySharedMemory` -// normally, but there is (temporary) support for writable -// `GuestSharedMemory` with `#[cfg(gdb)]`. Unfortunately, rustc gets -// annoyed about an unused type parameter, unless one goes to a little -// bit of effort to trick it... -mod unused_hack { - #[cfg(not(unshared_snapshot_mem))] - use crate::mem::shared_mem::ReadonlySharedMemory; - use crate::mem::shared_mem::SharedMemory; - pub trait SnapshotSharedMemoryT { - type T; - } - pub struct SnapshotSharedMemory_; - impl SnapshotSharedMemoryT for SnapshotSharedMemory_ { - #[cfg(not(unshared_snapshot_mem))] - type T = ReadonlySharedMemory; - #[cfg(unshared_snapshot_mem)] - type T = S; - } - pub type SnapshotSharedMemory = ::T; +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum BackingSource { + /// Snapshot layer index. + Snapshot(usize), + Scratch, + /// Dynamic region index. + Dynamic(usize), } -impl ReadonlySharedMemory { - pub(crate) fn to_mgr_snapshot_mem( - &self, - ) -> Result> { - #[cfg(not(unshared_snapshot_mem))] - let ret = self.clone(); - #[cfg(unshared_snapshot_mem)] - let ret = self.copy_to_writable()?; - Ok(ret) + +/// A guest physical range resolved to its backing memory. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct GuestBacking { + pub(crate) source: BackingSource, + pub(crate) offset: usize, +} + +pub(crate) struct GuestPhysicalMemoryView<'a> { + snapshot: &'a SnapshotMemoryBacking, + scratch: &'a HostSharedMemory, + dynamic: &'a [MemoryRegion], + layout: SandboxMemoryLayout, +} + +impl<'a> GuestPhysicalMemoryView<'a> { + #[cfg(any(test, feature = "mem_profile", feature = "trace_guest"))] + pub(crate) fn new( + snapshot: &'a SnapshotMemoryBacking, + scratch: &'a HostSharedMemory, + layout: SandboxMemoryLayout, + ) -> Self { + Self { + snapshot, + scratch, + dynamic: &[], + layout, + } + } + + /// Build a view that includes the sandbox's dynamic mappings. + pub(crate) fn with_dynamic( + snapshot: &'a SnapshotMemoryBacking, + scratch: &'a HostSharedMemory, + dynamic: &'a [MemoryRegion], + layout: SandboxMemoryLayout, + ) -> Self { + Self { + snapshot, + scratch, + dynamic, + layout, + } + } + + pub(crate) fn resolve(&self, gpa: u64, len: usize) -> Option { + let end = gpa.checked_add(u64::try_from(len).ok()?)?; + let scratch_start = + hyperlight_common::layout::scratch_base_gpa(self.layout.get_scratch_size()); + let scratch_end = scratch_start.checked_add(self.layout.get_scratch_size() as u64)?; + if scratch_start <= gpa && end <= scratch_end { + return Some(GuestBacking { + source: BackingSource::Scratch, + offset: usize::try_from(gpa - scratch_start).ok()?, + }); + } + if let Some((layer_index, offset)) = self.snapshot.resolve(gpa, len) { + return Some(GuestBacking { + source: BackingSource::Snapshot(layer_index), + offset, + }); + } + self.dynamic + .iter() + .enumerate() + .find_map(|(region_index, region)| { + let start = u64::try_from(region.guest_region.start).ok()?; + let region_end = u64::try_from(region.guest_region.end).ok()?; + if start <= gpa && end <= region_end { + Some(GuestBacking { + source: BackingSource::Dynamic(region_index), + offset: usize::try_from(gpa - start).ok()?, + }) + } else { + None + } + }) + } + + pub(crate) fn read(&self, gpa: u64, destination: &mut [u8]) -> Result { + let backing = self + .resolve(gpa, destination.len()) + .ok_or_else(|| new_error!("GPA range is not backed: {gpa:#x}"))?; + self.read_backing(backing, destination)?; + Ok(backing) + } + + pub(crate) fn read_backing(&self, backing: GuestBacking, destination: &mut [u8]) -> Result<()> { + let offset = backing.offset; + match backing.source { + BackingSource::Snapshot(layer_index) => { + self.snapshot + .copy_layer_to_slice(layer_index, destination, offset)? + } + BackingSource::Scratch => { + self.scratch.copy_to_slice(destination, offset)?; + } + BackingSource::Dynamic(region_index) => { + let region = &self.dynamic[region_index]; + #[allow(clippy::useless_conversion)] + let host_start: usize = region.host_region.start.into(); + #[allow(clippy::useless_conversion)] + let host_end: usize = region.host_region.end.into(); + let source_end = offset + .checked_add(destination.len()) + .ok_or_else(|| new_error!("dynamic memory read range overflows"))?; + let host_len = host_end + .checked_sub(host_start) + .ok_or_else(|| new_error!("dynamic memory host range is invalid"))?; + if source_end > host_len { + return Err(new_error!("dynamic memory read range is out of bounds")); + } + let source_start = host_start + .checked_add(offset) + .ok_or_else(|| new_error!("dynamic memory host address overflows"))?; + // SAFETY: Dynamic regions come from `Sandbox::map_region`, whose + // contract keeps them valid and unmodified for the sandbox's + // lifetime. The checks above bound this read. + unsafe { + std::ptr::copy( + source_start as *const u8, + destination.as_mut_ptr(), + destination.len(), + ) + }; + } + } + Ok(()) + } + + #[cfg(crashdump)] + fn host_range(&self, gpa: u64, len: usize) -> Result> { + let backing = self + .resolve(gpa, len) + .ok_or_else(|| new_error!("GPA range is not backed: {gpa:#x}"))?; + let offset = backing.offset; + let (base, backing_len) = match backing.source { + BackingSource::Snapshot(layer_index) => self.snapshot.host_range(layer_index)?, + BackingSource::Scratch => (self.scratch.base_addr(), self.scratch.mem_size()), + BackingSource::Dynamic(region_index) => { + let region = &self.dynamic[region_index]; + #[allow(clippy::useless_conversion)] + let start: usize = region.host_region.start.into(); + #[allow(clippy::useless_conversion)] + let end: usize = region.host_region.end.into(); + (start, end - start) + } + }; + let end_offset = offset + .checked_add(len) + .filter(|end| *end <= backing_len) + .ok_or_else(|| new_error!("resolved GPA range is out of bounds"))?; + let start = base + .checked_add(offset) + .ok_or_else(|| new_error!("resolved host address overflows"))?; + let end = base + .checked_add(end_offset) + .ok_or_else(|| new_error!("resolved host address overflows"))?; + Ok(start..end) + } +} +#[cfg(any(feature = "mem_profile", feature = "trace_guest", test))] +pub(crate) struct GuestVirtualMemoryReader<'a> { + memory: GuestPhysicalMemoryView<'a>, + root_pt: u64, + cached_page: Option<(u64, u64)>, +} + +#[cfg(any(feature = "mem_profile", feature = "trace_guest", test))] +impl<'a> GuestVirtualMemoryReader<'a> { + fn new( + snapshot: &'a SnapshotMemoryBacking, + scratch: &'a HostSharedMemory, + layout: SandboxMemoryLayout, + root_pt: u64, + ) -> Self { + Self { + memory: GuestPhysicalMemoryView::new(snapshot, scratch, layout), + root_pt, + cached_page: None, + } + } + + fn translate_page(&mut self, page_gva: u64) -> Result { + if let Some((cached_gva, cached_gpa)) = self.cached_page + && cached_gva == page_gva + { + return Ok(cached_gpa); + } + + use crate::sandbox::snapshot::PageTableReader; + + let pt_buf = PageTableReader::new(&self.memory, self.root_pt); + // SAFETY: The vCPU is stopped for this outb, so its page tables cannot + // change during the walk. + let mapping = unsafe { + hyperlight_common::vmem::virt_to_phys( + &pt_buf, + page_gva, + hyperlight_common::vmem::PAGE_SIZE as u64, + ) + } + .next(); + // A suppressed read leaves a not-present entry behind, so ask before + // blaming the guest for an unmapped page. + pt_buf.finish()?; + let mapping = mapping.ok_or_else(|| new_error!("GVA page is not mapped: {page_gva:#x}"))?; + let mapping_offset = page_gva + .checked_sub(mapping.virt_base) + .ok_or_else(|| new_error!("page-table mapping starts after requested GVA"))?; + if mapping + .len + .checked_sub(mapping_offset) + .is_none_or(|len| len < hyperlight_common::vmem::PAGE_SIZE as u64) + { + return Err(new_error!( + "page-table mapping does not cover requested GVA page" + )); + } + let page_gpa = mapping + .phys_base + .checked_add(mapping_offset) + .ok_or_else(|| new_error!("guest physical address overflows"))?; + self.cached_page = Some((page_gva, page_gpa)); + Ok(page_gpa) + } + + pub(crate) fn read_exact(&mut self, gva: u64, destination: &mut [u8]) -> Result<()> { + gva.checked_add(u64::try_from(destination.len())?) + .ok_or_else(|| new_error!("guest virtual address overflows"))?; + let mut copied = 0usize; + while copied < destination.len() { + let current_gva = gva + .checked_add(u64::try_from(copied)?) + .ok_or_else(|| new_error!("guest virtual address overflows"))?; + let page_gva = current_gva / hyperlight_common::vmem::PAGE_SIZE as u64 + * hyperlight_common::vmem::PAGE_SIZE as u64; + let page_offset = usize::try_from(current_gva - page_gva)?; + let chunk_len = + (destination.len() - copied).min(hyperlight_common::vmem::PAGE_SIZE - page_offset); + let gpa = self + .translate_page(page_gva)? + .checked_add(u64::try_from(page_offset)?) + .ok_or_else(|| new_error!("guest physical address overflows"))?; + self.memory + .read(gpa, &mut destination[copied..copied + chunk_len])?; + copied += chunk_len; + } + Ok(()) } } -pub(crate) use unused_hack::SnapshotSharedMemory; /// Guest operations the host accepts during a VM entry. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -146,7 +368,7 @@ pub(crate) enum ExecutionMode { /// for a given `Sandbox`. pub(crate) struct SandboxMemoryManager { /// Shared memory for the Sandbox - pub(crate) shared_mem: SnapshotSharedMemory, + pub(crate) shared_mem: SnapshotMemoryBacking, /// Scratch memory for the Sandbox pub(crate) scratch_mem: S, /// The memory layout of the underlying shared memory @@ -311,11 +533,16 @@ impl SandboxMemoryManager where S: SharedMemory, { + /// First GPA available to the guest scratch allocator. + pub(crate) fn first_free_scratch_gpa(&self) -> u64 { + self.layout.get_pt_base_gpa() + self.shared_mem.page_table_len() as u64 + } + /// Create a new `SandboxMemoryManager` with the given parameters #[instrument(skip_all, parent = Span::current(), level= "Trace")] pub(crate) fn new( layout: SandboxMemoryLayout, - shared_mem: SnapshotSharedMemory, + shared_mem: SnapshotMemoryBacking, scratch_mem: S, next_action: NextAction, ) -> Self { @@ -343,7 +570,7 @@ where impl SandboxMemoryManager { pub(crate) fn from_snapshot(s: &Snapshot) -> Result { let layout = *s.layout(); - let shared_mem = s.memory().to_mgr_snapshot_mem()?; + let shared_mem = SnapshotMemoryBacking::from_snapshot(s.snapshot_memory().clone())?; let scratch_mem = ExclusiveSharedMemory::new(s.layout().get_scratch_size())?; let next_action = s.next_action(); let mut mgr = Self::new(layout, shared_mem, scratch_mem, next_action); @@ -415,6 +642,11 @@ impl SandboxMemoryManager { } impl SandboxMemoryManager { + #[cfg(any(feature = "mem_profile", feature = "trace_guest", test))] + pub(crate) fn guest_virtual_memory_reader(&self, root_pt: u64) -> GuestVirtualMemoryReader<'_> { + GuestVirtualMemoryReader::new(&self.shared_mem, &self.scratch_mem, self.layout, root_pt) + } + /// Create a snapshot with the given mapped regions. #[allow(clippy::too_many_arguments)] pub(crate) fn snapshot( @@ -434,8 +666,8 @@ impl SandboxMemoryManager { }; let snapshot = Snapshot::new( - &mut self.shared_mem, - &mut self.scratch_mem, + &self.shared_mem, + &self.scratch_mem, self.layout, crate::mem::exe::LoadInfo::dummy(), mapped_regions, @@ -688,12 +920,66 @@ impl SandboxMemoryManager { } } + fn replace_snapshot_memory( + &mut self, + snapshot: &Snapshot, + ) -> Result> { + let new_snapshot_mem = + SnapshotMemoryBacking::from_snapshot(snapshot.snapshot_memory().clone())?; + let (hsnapshot, gsnapshot) = new_snapshot_mem.build(); + self.shared_mem = hsnapshot; + Ok(gsnapshot) + } + + fn apply_snapshot_metadata(&mut self, snapshot: &Snapshot) { + self.layout = *snapshot.layout(); + // Inherit the snapshot's own generation number — the + // guest-visible counter reflects "which snapshot is the + // sandbox currently a clone of", not "how many restores have + // happened into this (possibly-reused) partition". + self.snapshot_count = snapshot.snapshot_generation(); + // Carry the guest ELF entry point across restore so crashdumps + // report the restored image's entry. + self.original_entrypoint = snapshot.original_entrypoint(); + } + + /// Installs memory produced by a capture without restoring runtime state. + pub(crate) fn install_captured_snapshot( + &mut self, + snapshot: &Snapshot, + ) -> Result> { + let allocator_offset = self.scratch_mem.mem_size() + - hyperlight_common::layout::SCRATCH_TOP_ALLOCATOR_OFFSET as usize; + let used_scratch_end = self.scratch_mem.read::(allocator_offset)?; + let gsnapshot = self.replace_snapshot_memory(snapshot)?; + self.apply_snapshot_metadata(snapshot); + self.update_snapshot_scratch_bookkeeping()?; + self.zero_freed_scratch(used_scratch_end)?; + Ok(gsnapshot) + } + + /// Zeroes the scratch pages freed by resetting the allocator. + fn zero_freed_scratch(&mut self, used_end_gpa: u64) -> Result<()> { + let scratch_base = hyperlight_common::layout::scratch_base_gpa(self.scratch_mem.mem_size()); + // The guest controls the allocator value. + let end = used_end_gpa.min(hyperlight_common::layout::scratch_allocator_limit_gpa()); + let start = self.first_free_scratch_gpa(); + if end <= start { + return Ok(()); + } + let start = usize::try_from(start - scratch_base)?; + let end = usize::try_from(end - scratch_base)?; + self.scratch_mem + .with_exclusivity(|scratch| scratch.as_mut_slice()[start..end].fill(0))?; + Ok(()) + } + /// Restore base memory after the caller checks snapshot compatibility. pub(crate) fn restore_snapshot( &mut self, snapshot: &Snapshot, ) -> Result<( - Option>, + SnapshotMemoryBacking, Option, )> { let virtq = snapshot.virtq(); @@ -711,21 +997,7 @@ impl SandboxMemoryManager { // A panic during restricted dispatch can skip the mode reset. self.execution_mode = ExecutionMode::Unrestricted; - let gsnapshot = if *snapshot.memory() == self.shared_mem { - // If the snapshot memory is already the correct memory, - // which is readonly, don't bother with restoring it, - // since its contents must be the same. Note that in the - // #[cfg(unshared_snapshot_mem)] case, this condition will - // never be true, since even immediately after a restore, - // self.shared_mem is a (writable) copy, not the original - // shared_mem. - None - } else { - let new_snapshot_mem = snapshot.memory().to_mgr_snapshot_mem()?; - let (hsnapshot, gsnapshot) = new_snapshot_mem.build(); - self.shared_mem = hsnapshot; - Some(gsnapshot) - }; + let gsnapshot = self.replace_snapshot_memory(snapshot)?; let new_scratch_size = snapshot.layout().get_scratch_size(); let gscratch = if new_scratch_size == self.scratch_mem.mem_size() { // zero_or_replace picks the fastest zeroing strategy for @@ -742,15 +1014,7 @@ impl SandboxMemoryManager { self.scratch_mem = hscratch; Some(gscratch) }; - self.layout = *snapshot.layout(); - // Inherit the snapshot's own generation number — the - // guest-visible counter reflects "which snapshot is the - // sandbox currently a clone of", not "how many restores have - // happened into this (possibly-reused) partition". - self.snapshot_count = snapshot.snapshot_generation(); - // Carry the guest ELF entry point across restore so crashdumps - // report the restored image's entry. - self.original_entrypoint = snapshot.original_entrypoint(); + self.apply_snapshot_metadata(snapshot); self.update_scratch_bookkeeping()?; if let Some(virtq) = virtq { self.restore_virtq(virtq)?; @@ -797,9 +1061,16 @@ impl SandboxMemoryManager { use hyperlight_common::layout::*; let scratch_size = self.scratch_mem.mem_size(); self.update_scratch_bookkeeping_item(SCRATCH_TOP_SIZE_OFFSET, scratch_size as u64)?; + self.update_snapshot_scratch_bookkeeping()?; + + Ok(()) + } + + fn update_snapshot_scratch_bookkeeping(&mut self) -> Result<()> { + use hyperlight_common::layout::*; self.update_scratch_bookkeeping_item( SCRATCH_TOP_ALLOCATOR_OFFSET, - self.layout.get_first_free_scratch_gpa(), + self.first_free_scratch_gpa(), )?; // Record the GPA of the snapshot's copy of the page tables. // The copy lives at the tail of the snapshot blob; we copy it @@ -855,23 +1126,10 @@ impl SandboxMemoryManager { // `Snapshot::new`). Keeping them outside the KVM slot avoids // overlapping with `map_file_cow` regions installed // immediately after the snapshot in the guest PA space. - let snapshot_pt_end = self.shared_mem.mem_size(); - let snapshot_pt_size = self.layout.get_pt_size(); - let snapshot_pt_start = - snapshot_pt_end - snapshot_pt_size.next_multiple_of(page_size::get()); - self.scratch_mem.with_exclusivity(|scratch| { - #[cfg(not(unshared_snapshot_mem))] - let bytes = &self.shared_mem.as_slice()[snapshot_pt_start..snapshot_pt_end]; - #[cfg(unshared_snapshot_mem)] - let bytes = { - let mut bytes = vec![0u8; snapshot_pt_size]; - self.shared_mem - .copy_to_slice(&mut bytes, snapshot_pt_start)?; - bytes - }; - #[allow(clippy::needless_borrow)] - scratch.copy_from_slice(&bytes, self.layout.get_pt_base_scratch_offset()) - })??; + let page_tables = self.shared_mem.page_table_bytes(); + let offset = self.layout.get_pt_base_scratch_offset(); + self.scratch_mem + .with_exclusivity(|scratch| scratch.copy_from_slice(page_tables, offset))??; Ok(()) } @@ -886,53 +1144,57 @@ impl SandboxMemoryManager { root_pt: u64, mmap_regions: &[MemoryRegion], ) -> Result> { - use crate::sandbox::snapshot::SharedMemoryPageTableBuffer; + use crate::sandbox::snapshot::PageTableReader; let len = hyperlight_common::layout::SCRATCH_TOP_GVA; - - let regions = self.shared_mem.with_contents(|snapshot| { - self.scratch_mem.with_contents(|scratch| { - let pt_buf = - SharedMemoryPageTableBuffer::new(snapshot, scratch, self.layout, root_pt); - - let mappings: Vec<_> = - unsafe { hyperlight_common::vmem::virt_to_phys(&pt_buf, 0, len as u64) } - .collect(); - - if mappings.is_empty() { - return Err(new_error!("No page table mappings found (len {len})",)); - } - - let mut regions: Vec = Vec::new(); - for mapping in &mappings { - let virt_base = mapping.virt_base as usize; - let virt_end = (mapping.virt_base + mapping.len) as usize; - - if let Some(resolved) = self.layout.resolve_gpa(mapping.phys_base, mmap_regions) - { - let (flags, region_type) = mapping_kind_to_flags(&mapping.kind); - let resolved = resolved.with_memories(snapshot, scratch); - let contents = resolved.as_ref(); - let host_base = contents.as_ptr() as usize; - let host_len = (mapping.len as usize).min(contents.len()); - - if try_coalesce_region(&mut regions, virt_base, virt_end, host_base, flags) - { - continue; - } - - regions.push(CrashDumpRegion { - guest_region: virt_base..virt_end, - host_region: host_base..host_base + host_len, - flags, - region_type, - }); - } + let memory = GuestPhysicalMemoryView::with_dynamic( + &self.shared_mem, + &self.scratch_mem, + mmap_regions, + self.layout, + ); + let pt_buf = PageTableReader::new(&memory, root_pt); + let mut regions: Vec = Vec::new(); + let mut saw_mapping = false; + // SAFETY: Crash handling runs with the vCPU stopped, so its page tables + // cannot change during the walk. + for mapping in unsafe { hyperlight_common::vmem::virt_to_phys(&pt_buf, 0, len as u64) } { + saw_mapping = true; + let (flags, region_type) = mapping_kind_to_flags(&mapping.kind); + let mut mapped = 0usize; + let mapping_len = usize::try_from(mapping.len)?; + while mapped < mapping_len { + let chunk_len = (mapping_len - mapped).min(hyperlight_common::vmem::PAGE_SIZE); + let gpa = mapping + .phys_base + .checked_add(u64::try_from(mapped)?) + .ok_or_else(|| new_error!("crash-dump GPA overflows"))?; + let Some(host_region) = memory.host_range(gpa, chunk_len).ok() else { + mapped += chunk_len; + continue; + }; + let virt_base = usize::try_from(mapping.virt_base)? + .checked_add(mapped) + .ok_or_else(|| new_error!("crash-dump GVA overflows"))?; + let virt_end = virt_base + .checked_add(chunk_len) + .ok_or_else(|| new_error!("crash-dump GVA range overflows"))?; + + if !try_coalesce_region(&mut regions, virt_base, virt_end, host_region.start, flags) + { + regions.push(CrashDumpRegion { + guest_region: virt_base..virt_end, + host_region, + flags, + region_type, + }); } - - Ok(regions) - }) - })???; + mapped += chunk_len; + } + } + if !saw_mapping { + return Err(new_error!("No page table mappings found (len {len})",)); + } Ok(regions) } @@ -951,109 +1213,29 @@ impl SandboxMemoryManager { /// * `root_pt` - The root page table physical address (CR3) #[cfg(feature = "trace_guest")] pub(crate) fn read_guest_memory_by_gva( - &mut self, + &self, gva: u64, len: usize, root_pt: u64, ) -> Result> { - use hyperlight_common::vmem::PAGE_SIZE; - - use crate::sandbox::snapshot::{SharedMemoryPageTableBuffer, access_gpa}; - - self.shared_mem.with_contents(|snap| { - self.scratch_mem.with_contents(|scratch| { - let pt_buf = SharedMemoryPageTableBuffer::new(snap, scratch, self.layout, root_pt); - - // Walk page tables to get all mappings that cover the GVA range - let mappings: Vec<_> = unsafe { - hyperlight_common::vmem::virt_to_phys(&pt_buf, gva, len as u64) - } - .collect(); - - if mappings.is_empty() { - return Err(new_error!( - "No page table mappings found for GVA {:#x} (len {})", - gva, - len, - )); - } - - // Resulting vector of bytes to return - let mut result = Vec::with_capacity(len); - let mut current_gva = gva; - - for mapping in &mappings { - // The page table walker should only return valid mappings - // that cover our current read position. - if mapping.virt_base > current_gva { - return Err(new_error!( - "Page table walker returned mapping with virt_base {:#x} > current read position {:#x}", - mapping.virt_base, - current_gva, - )); - } - - // Calculate the offset within this page where to start copying - let page_offset = (current_gva - mapping.virt_base) as usize; - - let bytes_remaining = len - result.len(); - let available_in_page = PAGE_SIZE - page_offset; - let bytes_to_copy = bytes_remaining.min(available_in_page); - - // Translate the GPA to host memory - let gpa = mapping.phys_base + page_offset as u64; - let (mem, offset) = access_gpa(snap, scratch, self.layout, gpa) - .ok_or_else(|| { - new_error!( - "Failed to resolve GPA {:#x} to host memory (GVA {:#x})", - gpa, - gva - ) - })?; - - let slice = mem - .get(offset..offset + bytes_to_copy) - .ok_or_else(|| { - new_error!( - "GPA {:#x} resolved to out-of-bounds host offset {} (need {} bytes)", - gpa, - offset, - bytes_to_copy - ) - })?; - - result.extend_from_slice(slice); - current_gva += bytes_to_copy as u64; - } - - if result.len() != len { - tracing::error!( - "Page table walker returned mappings that don't cover the full requested length: got {}, expected {}", - result.len(), - len, - ); - return Err(new_error!( - "Could not read full GVA range: got {} of {} bytes {:?}", - result.len(), - len, - mappings - )); - } - - Ok(result) - }) - })?? + let mut result = vec![0; len]; + self.guest_virtual_memory_reader(root_pt) + .read_exact(gva, &mut result)?; + Ok(result) } } #[cfg(test)] mod tests { + use std::sync::Arc; + use hyperlight_common::flatbuffer_wrappers::function_call::FunctionCallType; use hyperlight_common::flatbuffer_wrappers::function_types::{ParameterValue, ReturnType}; use hyperlight_common::transport::{ MsgHeader, SIZE_PREFIX_LEN, size_prefix_payload_len, size_prefixed_len, }; use hyperlight_common::virtq::DescFlags; + use hyperlight_common::vmem::{self, BasicMapping, Mapping, MappingKind, PAGE_SIZE}; #[cfg(target_arch = "x86_64")] use hyperlight_testing::sandbox_sizes::{LARGE_HEAP_SIZE, MEDIUM_HEAP_SIZE, SMALL_HEAP_SIZE}; #[cfg(target_arch = "x86_64")] @@ -1062,26 +1244,32 @@ mod tests { use super::*; #[cfg(target_arch = "x86_64")] use crate::GuestBinary; + use crate::mem::layout::SandboxMemoryLayout; + use crate::mem::memory_region::MemoryRegionType; + use crate::mem::shared_mem::{ExclusiveSharedMemory, ReadonlySharedMemory}; use crate::mem::virtq::tests::{H2G_BUFFER_SIZE, SCRATCH_SIZE, TestCase, memory_layout}; - #[cfg(target_arch = "x86_64")] use crate::sandbox::SandboxConfiguration; + #[cfg(target_arch = "x86_64")] + use crate::sandbox::snapshot::Snapshot; + use crate::sandbox::snapshot::memory::tests::flat_memory; + use crate::sandbox::snapshot::{SnapshotBlob, SnapshotMemory}; fn manager(case: &TestCase) -> SandboxMemoryManager { let host_page_size = page_size::get(); - - #[cfg(not(unshared_snapshot_mem))] - let shared_mem = - ReadonlySharedMemory::from_bytes(&vec![0; host_page_size], host_page_size).unwrap(); - - #[cfg(unshared_snapshot_mem)] - let shared_mem = ExclusiveSharedMemory::new(host_page_size) - .unwrap() - .build() - .0; + let layout = memory_layout(); + let snapshot = Arc::new(flat_memory( + &vec![0; host_page_size], + &[0; PAGE_SIZE], + SandboxMemoryLayout::BASE_ADDRESS as u64, + hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()), + )); let mut mgr = SandboxMemoryManager::new( - memory_layout(), - shared_mem, + layout, + SnapshotMemoryBacking::from_snapshot(snapshot) + .unwrap() + .build() + .0, case.scratch.clone(), NextAction::None, ); @@ -1390,4 +1578,257 @@ mod tests { assert!(mgr.g2h_consumer.is_some()); assert!(mgr.h2g_consumer.is_some()); } + + #[test] + fn snapshot_memory_backing_reads_across_layers() { + let host_page_size = page_size::get(); + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let scratch = base + (4 * host_page_size) as u64; + let layer = |gpa_start: u64, value: u8| { + let data_len = host_page_size; + let mut storage = ExclusiveSharedMemory::new(data_len).unwrap(); + storage + .copy_from_slice(&vec![value; host_page_size], 0) + .unwrap(); + let storage = storage.freeze(data_len).unwrap(); + Arc::new(SnapshotBlob::new(storage, gpa_start, 0, scratch).unwrap()) + }; + let first = layer(base, 0x11); + let second = layer(base + host_page_size as u64, 0x22); + let memory = Arc::new( + SnapshotMemory::new( + Box::new([first, second]), + crate::sandbox::snapshot::memory::tests::stub_page_tables(), + ) + .unwrap(), + ); + let (snapshot, _) = SnapshotMemoryBacking::from_snapshot(memory) + .unwrap() + .build(); + let mut bytes = [0; 2]; + + snapshot + .read_snapshot_gpa(base + host_page_size as u64 - 1, &mut bytes) + .unwrap(); + + assert_eq!(bytes, [0x11, 0x22]); + assert!( + snapshot + .read_snapshot_gpa(base + 2 * host_page_size as u64 - 1, &mut bytes) + .is_err() + ); + } + + fn virtual_reader_manager() -> (SandboxMemoryManager, u64, u64) { + let cfg = SandboxConfiguration::default(); + let layout = SandboxMemoryLayout::new(cfg, PAGE_SIZE, 0, None).unwrap(); + let host_page_size = page_size::get(); + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let second_gpa = base + host_page_size as u64; + let scratch_base = hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()); + let scratch_gpa = scratch_base + layout.get_scratch_size() as u64 - 0x10000; + let pt_base = layout.get_pt_base_gpa(); + let pt_buf = GuestPageTableBuffer::new(pt_base as usize); + let kind = MappingKind::Basic(BasicMapping { + readable: true, + writable: false, + executable: false, + }); + // SAFETY: The local page-table buffer has no concurrent writers. + unsafe { + vmem::map( + &pt_buf, + Mapping { + phys_base: base, + virt_base: base, + len: PAGE_SIZE as u64, + kind, + }, + ); + vmem::map( + &pt_buf, + Mapping { + phys_base: second_gpa, + virt_base: base + PAGE_SIZE as u64, + len: PAGE_SIZE as u64, + kind, + }, + ); + vmem::map( + &pt_buf, + Mapping { + phys_base: scratch_gpa, + virt_base: base + 2 * PAGE_SIZE as u64, + len: PAGE_SIZE as u64, + kind, + }, + ); + } + let page_tables = pt_buf.into_bytes(); + layout.ensure_page_tables_fit(page_tables.len()).unwrap(); + + let first_storage = + ReadonlySharedMemory::from_bytes(&vec![0x11; host_page_size], host_page_size).unwrap(); + let first = Arc::new(SnapshotBlob::new(first_storage, base, 0, scratch_base).unwrap()); + + let second_storage = + ReadonlySharedMemory::from_bytes(&vec![0x22; host_page_size], host_page_size).unwrap(); + let second = + Arc::new(SnapshotBlob::new(second_storage, second_gpa, 0, scratch_base).unwrap()); + let memory = Arc::new( + SnapshotMemory::new( + Box::new([first, second]), + crate::sandbox::snapshot::memory::tests::page_tables_from_bytes(&page_tables), + ) + .unwrap(), + ); + let manager = SandboxMemoryManager::new( + layout, + SnapshotMemoryBacking::from_snapshot(memory).unwrap(), + ExclusiveSharedMemory::new(layout.get_scratch_size()).unwrap(), + NextAction::None, + ); + let (manager, _) = manager.build().unwrap(); + manager + .scratch_mem + .copy_from_slice( + &vec![0x33; PAGE_SIZE], + (scratch_gpa - scratch_base) as usize, + ) + .unwrap(); + (manager, pt_base, base) + } + + #[test] + fn virtual_reader_reads_across_snapshot_layers() { + let (manager, pt_base, base) = virtual_reader_manager(); + let mut reader = manager.guest_virtual_memory_reader(pt_base); + let mut bytes = [0u8; 2]; + + reader + .read_exact(base + PAGE_SIZE as u64 - 1, &mut bytes) + .unwrap(); + + assert_eq!(bytes, [0x11, 0x22]); + } + + #[test] + fn virtual_reader_reads_snapshot_and_cow_scratch_across_pages() { + let (manager, root, base) = virtual_reader_manager(); + let mut bytes = [0; 2]; + manager + .guest_virtual_memory_reader(root) + .read_exact(base + 2 * PAGE_SIZE as u64 - 1, &mut bytes) + .unwrap(); + assert_eq!(bytes, [0x22, 0x33]); + } + + #[test] + fn virtual_reader_rejects_unmapped_and_overflowing_ranges() { + let (manager, root, base) = virtual_reader_manager(); + let mut reader = manager.guest_virtual_memory_reader(root); + assert!( + reader + .read_exact(base + 3 * PAGE_SIZE as u64, &mut [0]) + .is_err() + ); + assert!(reader.read_exact(u64::MAX, &mut [0; 2]).is_err()); + assert!(reader.read_exact(u64::MAX, &mut []).is_ok()); + } + + #[test] + fn virtual_reader_reports_invalid_page_tables() { + let (manager, _, base) = virtual_reader_manager(); + let error = manager + .guest_virtual_memory_reader(u64::MAX) + .read_exact(base, &mut [0]) + .unwrap_err(); + assert!(error.to_string().contains("unbacked")); + } + + #[test] + fn guest_physical_memory_view_resolves_scratch_and_dynamic_edges() { + let host_page_size = page_size::get(); + let layout = + SandboxMemoryLayout::new(SandboxConfiguration::default(), PAGE_SIZE, 0, None).unwrap(); + let scratch_start = hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()); + let snapshot = Arc::new(flat_memory( + &[0u8; PAGE_SIZE], + &[0; PAGE_SIZE], + SandboxMemoryLayout::BASE_ADDRESS as u64, + scratch_start, + )); + let (snapshot, _) = SnapshotMemoryBacking::from_snapshot(snapshot) + .unwrap() + .build(); + + let mut scratch = ExclusiveSharedMemory::new(layout.get_scratch_size()).unwrap(); + scratch.copy_from_slice(&[0x11], 0).unwrap(); + scratch + .copy_from_slice(&[0x22], layout.get_scratch_size() - 1) + .unwrap(); + let (scratch, _) = scratch.build(); + + let dynamic_gpa = SandboxMemoryLayout::BASE_ADDRESS as u64 + host_page_size as u64; + let mut dynamic = ExclusiveSharedMemory::new(host_page_size).unwrap(); + dynamic.copy_from_slice(&[0x33], 0).unwrap(); + dynamic + .copy_from_slice(&[0x44], host_page_size - 1) + .unwrap(); + let (dynamic, dynamic_guest) = dynamic.build(); + let dynamic_region = dynamic_guest.mapping_at(dynamic_gpa, MemoryRegionType::Scratch); + let dynamic_regions = [dynamic_region]; + let view = + GuestPhysicalMemoryView::with_dynamic(&snapshot, &scratch, &dynamic_regions, layout); + let mut byte = [0u8; 1]; + + assert_eq!( + view.read(scratch_start, &mut byte).unwrap(), + GuestBacking { + source: BackingSource::Scratch, + offset: 0, + } + ); + assert_eq!(byte, [0x11]); + assert_eq!( + view.read( + scratch_start + layout.get_scratch_size() as u64 - 1, + &mut byte, + ) + .unwrap(), + GuestBacking { + source: BackingSource::Scratch, + offset: layout.get_scratch_size() - 1, + } + ); + assert_eq!(byte, [0x22]); + assert_eq!( + view.read(dynamic_gpa, &mut byte).unwrap(), + GuestBacking { + source: BackingSource::Dynamic(0), + offset: 0, + } + ); + assert_eq!(byte, [0x33]); + assert_eq!( + view.read(dynamic_gpa + host_page_size as u64 - 1, &mut byte) + .unwrap(), + GuestBacking { + source: BackingSource::Dynamic(0), + offset: host_page_size - 1, + } + ); + assert_eq!(byte, [0x44]); + assert!( + view.resolve(dynamic_gpa + host_page_size as u64 - 1, 2) + .is_none() + ); + assert!( + view.resolve(scratch_start + layout.get_scratch_size() as u64 - 1, 2,) + .is_none() + ); + + drop(dynamic); + drop(dynamic_guest); + } } diff --git a/src/hyperlight_host/src/mem/shared_mem.rs b/src/hyperlight_host/src/mem/shared_mem.rs index e6ff17b14..42d01fac8 100644 --- a/src/hyperlight_host/src/mem/shared_mem.rs +++ b/src/hyperlight_host/src/mem/shared_mem.rs @@ -107,6 +107,10 @@ pub enum SharedMemoryError { #[error("Cannot take exclusive access to a ReadonlySharedMemory")] ReadonlySharedMemoryExclusiveRequest, + /// Calling code attempted to freeze memory that still has writable aliases. + #[error("Cannot freeze shared memory that still has other owners")] + SharedMemoryNotExclusive, + /// An error was encountered when trying to convert a slice of raw /// bytes into some logical data #[error("Error reading slice {0}")] @@ -867,7 +871,7 @@ impl ExclusiveSharedMemory { /// the safety documentation of pointer::offset. /// /// This is ensured by a check in ::new() - pub(super) fn as_mut_slice(&mut self) -> &mut [u8] { + pub(crate) fn as_mut_slice(&mut self) -> &mut [u8] { unsafe { std::slice::from_raw_parts_mut(self.base_ptr(), self.mem_size()) } } @@ -948,6 +952,30 @@ impl ExclusiveSharedMemory { ) } + /// Convert this allocation to immutable memory without copying it. + pub(crate) fn freeze(mut self, guest_mapped_size: usize) -> Result { + if guest_mapped_size > self.mem_size() + || !guest_mapped_size.is_multiple_of(page_size::get()) + { + return Err(SharedMemoryError::MemoryRequest( + guest_mapped_size, + 0, + self.mem_size(), + page_size::get(), + )); + } + if Arc::get_mut(&mut self.region).is_none() { + return Err(SharedMemoryError::SharedMemoryNotExclusive); + } + + // This is the only Arc left. To make another one you have to clone this + // one, and it lives in `self`, which this function owns. + Ok(ReadonlySharedMemory { + region: self.region, + guest_mapped_size, + }) + } + /// Gets the file handle of the shared memory region for this Sandbox #[cfg(target_os = "windows")] pub fn get_mmap_file_handle(&self) -> HANDLE { @@ -1010,7 +1038,7 @@ impl GuestSharedMemory { } #[allow(clippy::panic)] // This will not ever actually panic: the only places this - // is called are HyperlightVm::update_snapshot_mapping and + // is called are SnapshotMemoryBacking::mappings and // HyperlightVm::update_scratch_mapping. The latter // statically uses the Scratch region type, and the former // does not use this at all when the unshared_snapshot_mem @@ -1502,7 +1530,6 @@ pub struct ReadonlySharedMemory { /// exposes to the guest. Production callers pass the size of the /// guest-visible prefix of the snapshot blob; the remainder of /// the blob is the page-table tail that lives only host-side. - #[cfg_attr(unshared_snapshot_mem, allow(dead_code))] guest_mapped_size: usize, } // Safety: HostMapping is only non-Send/Sync (causing @@ -1520,9 +1547,7 @@ unsafe impl Sync for ReadonlySharedMemory {} impl ReadonlySharedMemory { pub(crate) fn from_bytes(contents: &[u8], guest_mapped_size: usize) -> Result { - if guest_mapped_size == 0 - || guest_mapped_size > contents.len() - || !guest_mapped_size.is_multiple_of(page_size::get()) + if guest_mapped_size > contents.len() || !guest_mapped_size.is_multiple_of(page_size::get()) { return Err(SharedMemoryError::MemoryRequest( guest_mapped_size, @@ -1541,7 +1566,6 @@ impl ReadonlySharedMemory { } /// The number of bytes that should be mapped into guest PA space. - #[cfg(not(unshared_snapshot_mem))] pub(crate) fn guest_mapped_size(&self) -> usize { self.guest_mapped_size } @@ -1549,8 +1573,8 @@ impl ReadonlySharedMemory { /// Create a `ReadonlySharedMemory` backed by a file on disk. /// /// The file's length must be a non-zero multiple of `PAGE_SIZE`. - /// `guest_mapped_size` must be a non-zero multiple of `PAGE_SIZE` - /// no greater than the file's length. + /// `guest_mapped_size` must be a multiple of `PAGE_SIZE` no greater + /// than the file's length. pub(crate) fn from_file(file: &std::fs::File, guest_mapped_size: usize) -> Result { let len: usize = file .metadata() @@ -1568,10 +1592,7 @@ impl ReadonlySharedMemory { )); } - if guest_mapped_size == 0 - || guest_mapped_size > len - || !guest_mapped_size.is_multiple_of(page_size::get()) - { + if guest_mapped_size > len || !guest_mapped_size.is_multiple_of(page_size::get()) { return Err(SharedMemoryError::MemoryRequest( guest_mapped_size, 0, @@ -1741,14 +1762,16 @@ impl ReadonlySharedMemory { #[cfg(unshared_snapshot_mem)] pub(crate) fn copy_to_writable(&self) -> Result { - let mut writable = ExclusiveSharedMemory::new(self.mem_size())?; - writable.copy_from_slice(self.as_slice(), 0)?; + let mut writable = ExclusiveSharedMemory::new(self.guest_mapped_size)?; + writable.copy_from_slice(&self.as_slice()[..self.guest_mapped_size], 0)?; Ok(writable) } #[cfg(not(unshared_snapshot_mem))] - pub(crate) fn build(self) -> (Self, Self) { - (self.clone(), self) + pub(crate) fn copy_to_slice(&self, slice: &mut [u8], offset: usize) -> Result<()> { + bounds_check!(offset, slice.len(), self.mem_size()); + slice.copy_from_slice(&self.as_slice()[offset..offset + slice.len()]); + Ok(()) } #[cfg(not(unshared_snapshot_mem))] @@ -1759,7 +1782,7 @@ impl ReadonlySharedMemory { ) -> MemoryRegion { #[allow(clippy::panic)] // This will not ever actually panic: the only place this is - // called is HyperlightVm::update_snapshot_mapping, which + // called is SnapshotMemoryBacking::mappings, which // always calls it with the Snapshot region type. if region_type != MemoryRegionType::Snapshot { panic!("ReadonlySharedMemory::mapping_at should only be used for Snapshot regions"); @@ -1833,7 +1856,7 @@ mod tests { #[cfg(not(miri))] use super::HostSharedMemory; - use super::{ExclusiveSharedMemory, Result, SharedMemory}; + use super::{ExclusiveSharedMemory, Result, SharedMemory, SharedMemoryError}; #[cfg(not(miri))] use crate::mem::shared_mem_tests::read_write_test_suite; @@ -1944,6 +1967,43 @@ mod tests { .unwrap(); } + #[test] + fn freeze_transfers_mapping_without_copying() { + let page_size = page_size::get(); + let mut memory = ExclusiveSharedMemory::new(2 * page_size).unwrap(); + memory.copy_from_slice(&[1, 2, 3, 4], page_size).unwrap(); + let base_addr = memory.base_addr(); + + let frozen = memory.freeze(page_size).unwrap(); + + assert_eq!(frozen.base_addr(), base_addr); + assert_eq!(frozen.guest_mapped_size(), page_size); + assert_eq!(&frozen.as_slice()[page_size..page_size + 4], &[1, 2, 3, 4]); + } + + #[test] + fn freeze_rejects_writable_aliases() { + let memory = ExclusiveSharedMemory::new(page_size::get()).unwrap(); + let alias = ExclusiveSharedMemory { + region: memory.region.clone(), + }; + + assert!(matches!( + memory.freeze(0), + Err(SharedMemoryError::SharedMemoryNotExclusive) + )); + drop(alias); + } + + #[test] + fn freeze_rejects_invalid_guest_mapped_size() { + let page_size = page_size::get(); + let freeze = |size| ExclusiveSharedMemory::new(page_size).unwrap().freeze(size); + + assert!(freeze(page_size + 1).is_err()); + assert!(freeze(2 * page_size).is_err()); + } + #[test] fn copy_into_from() -> Result<()> { let mem_size: usize = page_size::get(); @@ -2567,14 +2627,6 @@ mod tests { ))); } - #[test] - fn from_file_rejects_zero_guest_mapped_size() { - let tmp = make_temp_file(page_size::get()); - let err = ReadonlySharedMemory::from_file(tmp.as_file(), 0) - .expect_err("zero guest_mapped_size should be rejected"); - assert!(format!("{}", err).contains("0x0 < 0x0")); - } - #[test] fn from_file_rejects_unaligned_guest_mapped_size() { let tmp = make_temp_file(2 * page_size::get()); diff --git a/src/hyperlight_host/src/mem/virtq/tests.rs b/src/hyperlight_host/src/mem/virtq/tests.rs index 0048fb79c..277f9a20b 100644 --- a/src/hyperlight_host/src/mem/virtq/tests.rs +++ b/src/hyperlight_host/src/mem/virtq/tests.rs @@ -135,7 +135,7 @@ fn snapshots_and_restores_rings() { let captured = VirtqSnapshot::capture(&layout, &case.scratch).unwrap(); let restored = host_scratch(); - let allocator = layout.get_first_free_scratch_gpa(); + let allocator = layout.get_pt_base_gpa(); let allocator_offset = restored.mem_size() - SCRATCH_TOP_ALLOCATOR_OFFSET as usize; let mbx_offset = layout.get_transport_arena().mbx_offset(); @@ -154,6 +154,12 @@ fn snapshots_and_restores_rings() { assert_eq!(restored_snapshot, captured); assert_eq!(restored.read::(allocator_offset).unwrap(), allocator); + let arena_gpa_offset = restored.mem_size() + - hyperlight_common::layout::SCRATCH_TOP_TRANSPORT_ARENA_GPA_OFFSET as usize; + assert_eq!( + restored.read::(arena_gpa_offset).unwrap(), + layout.get_transport_arena().base_addr() + ); assert_eq!(restored.read::<[u8; 16]>(spare_offset).unwrap(), [0; 16]); assert_eq!(pool_bytes, [0; 16]); @@ -293,33 +299,6 @@ fn rejects_h2g_snapshot_chain_shape() { ); } -#[test] -fn restores_with_finalized_layout() { - let case = TestCase::new(); - let layout = memory_layout(); - let snapshot = VirtqSnapshot::capture(&layout, &case.scratch).unwrap(); - - let mut grown_layout = layout; - - grown_layout - .set_pt_size(layout.get_pt_size() + vmem::PAGE_SIZE) - .unwrap(); - grown_layout.set_snapshot_size(layout.snapshot_size() + page_size::get()); - let restored = host_scratch(); - - snapshot.restore(&grown_layout, &restored).unwrap(); - assert_eq!( - VirtqSnapshot::capture(&grown_layout, &restored).unwrap(), - snapshot - ); - let arena_gpa_offset = restored.mem_size() - - hyperlight_common::layout::SCRATCH_TOP_TRANSPORT_ARENA_GPA_OFFSET as usize; - assert_eq!( - restored.read::(arena_gpa_offset).unwrap(), - grown_layout.get_transport_arena().base_addr() - ); -} - #[test] fn uses_scratch_payloads_outside_pools() { let case = TestCase::new(); diff --git a/src/hyperlight_host/src/sandbox/initialized.rs b/src/hyperlight_host/src/sandbox/initialized.rs index 37ab42084..ca128d811 100644 --- a/src/hyperlight_host/src/sandbox/initialized.rs +++ b/src/hyperlight_host/src/sandbox/initialized.rs @@ -1,6 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // Copyright 2025 The Hyperlight Authors. +use std::collections::BTreeSet; use std::path::Path; #[cfg(crashdump)] use std::path::PathBuf; @@ -26,6 +27,7 @@ use crate::mem::shared_mem::{HostSharedMemory, SharedMemory as _}; use crate::metrics::{ METRIC_GUEST_ERROR, METRIC_GUEST_ERROR_LABEL_CODE, maybe_time_and_emit_guest_call, }; +use crate::sandbox::snapshot::SnapshotMemoryBacking; use crate::{HyperlightError, Result, log_then_return}; /// The lifecycle state of a [`Sandbox`]. @@ -100,17 +102,37 @@ pub struct Sandbox { #[deprecated(since = "0.18.0", note = "use Sandbox")] pub type MultiUseSandbox = Sandbox; +/// Read-only access to live snapshot memory by guest physical address. +pub struct SnapshotMemoryReader<'a> { + memory: &'a SnapshotMemoryBacking, +} + +impl SnapshotMemoryReader<'_> { + /// Copy live snapshot bytes at `gpa` into `destination`. + pub fn read_exact(&self, gpa: u64, destination: &mut [u8]) -> Result<()> { + self.memory.read_snapshot_gpa(gpa, destination) + } +} + /// Callback for discovering page table roots from guest memory. /// /// Called during [`Sandbox::snapshot`] with: -/// - `snapshot_mem` - the sandbox's snapshot (shared) memory as a byte slice +/// - `snapshot_mem` - live snapshot memory accessed by GPA /// - `scratch_mem` - the sandbox's scratch memory as a byte slice /// - `root_pt_gpa` - the root page table GPA of the currently-executing /// address space /// -/// Returns a list of root page table GPAs to walk. If the list is -/// empty, only `root_pt_gpa` is used. -pub type PtRootFinder = Box Vec + Send>; +/// Returns additional root page table GPAs to walk. `root_pt_gpa` is always +/// walked first, ignoring duplicates. An error aborts the capture. +pub type PtRootFinder = + Box, &[u8], u64) -> Result> + Send>; + +fn normalize_page_table_roots(roots: Vec, active_root: u64) -> Vec { + let mut seen = BTreeSet::from([active_root]); + let mut normalized = vec![active_root]; + normalized.extend(roots.into_iter().filter(|root| seen.insert(*root))); + normalized +} impl Sandbox { fn check_ready(&self) -> Result<()> { @@ -165,7 +187,7 @@ impl Sandbox { /// Set a callback that discovers page table roots from guest memory. /// The callback receives (snapshot_mem, scratch_mem, cr3) and returns - /// the list of root GPAs to walk during snapshot creation. + /// additional root GPAs to walk during snapshot creation. /// /// The callback must support every guest restored into this sandbox. pub fn set_pt_root_finder(&mut self, finder: PtRootFinder) { @@ -380,6 +402,28 @@ impl Sandbox { /// Creates a snapshot of the sandbox's current memory state. /// + /// If the sandbox was created from or restored to a snapshot, this capture can + /// reuse memory that has not changed. A successful capture becomes the basis + /// for later captures. This avoids copying the full sandbox each time, reducing + /// capture time and memory use. + /// + /// Taking another snapshot after changing the sandbox creates a child of the + /// previous snapshot. Restoring an earlier snapshot, changing the sandbox, and + /// taking another creates a separate branch. Snapshots therefore form a + /// conceptual tree of states. Calling this method again without changing the + /// sandbox returns the same snapshot. + /// + /// A snapshot holds at most 30 memory layers. A capture that changes memory adds + /// one layer. A layer drops out once later changes replace all of its pages. + /// Creating a snapshot fails when it would exceed this limit or finds no room + /// for the new layer. This check leaves the sandbox unchanged and ready. + /// + /// Memory added through [`Self::map_region`] and [`Self::map_file_cow`] is + /// copied into the snapshot only for pages the guest has mapped into its page + /// tables. Capture then removes these regions from this sandbox. Pages the + /// guest has not mapped are lost, and their guest physical addresses can be + /// reused by later snapshots. + /// /// The returned snapshot can be applied to any /// [`Sandbox`] whose registered host functions are a /// superset of those registered here at the time of capture. See @@ -395,7 +439,8 @@ impl Sandbox { /// /// This method returns [`crate::HyperlightError::PoisonedSandbox`] when the /// sandbox is poisoned and [`crate::HyperlightError::UnrecoverableSandbox`] - /// when it is unrecoverable. + /// when it is unrecoverable. An unexpected hypervisor or platform error during + /// capture may leave the sandbox poisoned or unrecoverable. /// /// # Examples /// @@ -433,12 +478,14 @@ impl Sandbox { .map_err(|e| HyperlightError::HyperlightVmError(e.into()))?; // Use the callback if set, otherwise just CR3 let root_pt_gpas = if let Some(finder) = &self.pt_root_finder { - let roots = self.mem_mgr.shared_mem.with_contents(|snap| { - self.mem_mgr - .scratch_mem - .with_contents(|scratch| finder(snap, scratch, cr3)) - })??; - if roots.is_empty() { vec![cr3] } else { roots } + let snapshot = SnapshotMemoryReader { + memory: &self.mem_mgr.shared_mem, + }; + let roots = self + .mem_mgr + .scratch_mem + .with_contents(|scratch| finder(&snapshot, scratch, cr3))??; + normalize_page_table_roots(roots, cr3) } else { vec![cr3] }; @@ -470,6 +517,31 @@ impl Sandbox { host_functions, )?; let snapshot = Arc::new(memory_snapshot); + + if let Err(error) = self.remove_dynamic_mappings() { + self.status = SandboxStatus::Poisoned; + self.snapshot = None; + return Err(error); + } + let snapshot_mem = match self.mem_mgr.install_captured_snapshot(&snapshot) { + Ok(snapshot_mem) => snapshot_mem, + Err(error) => { + self.status = SandboxStatus::Unrecoverable; + self.snapshot = None; + return Err(error); + } + }; + if let Err(error) = self.vm.update_snapshot_mapping(snapshot_mem) { + self.status = SandboxStatus::Unrecoverable; + self.snapshot = None; + return Err(HyperlightVmError::UpdateRegion(error).into()); + } + if let Err(error) = self.vm.apply_sregs(snapshot.root_pt_gpa(), &sregs) { + self.status = SandboxStatus::Poisoned; + self.snapshot = None; + return Err(HyperlightVmError::Restore(error.into()).into()); + } + self.snapshot = Some(snapshot.clone()); Ok(snapshot) } @@ -486,13 +558,21 @@ impl Sandbox { }) } - fn restore_memory_and_mappings(&mut self, snapshot: &Snapshot) -> Result<()> { - let (snapshot_mem, scratch_mem) = self.mem_mgr.restore_snapshot(snapshot)?; - if let Some(snapshot_mem) = snapshot_mem { + fn remove_dynamic_mappings(&mut self) -> Result<()> { + let regions = self.vm.get_mapped_regions().cloned().collect::>(); + for region in ®ions { self.vm - .update_snapshot_mapping(snapshot_mem) - .map_err(HyperlightVmError::UpdateRegion)?; + .unmap_region(region) + .map_err(HyperlightVmError::UnmapRegion)?; } + Ok(()) + } + + fn restore_memory_and_mappings(&mut self, snapshot: &Snapshot) -> Result<()> { + let (snapshot_mem, scratch_mem) = self.mem_mgr.restore_snapshot(snapshot)?; + self.vm + .update_snapshot_mapping(snapshot_mem) + .map_err(HyperlightVmError::UpdateRegion)?; if let Some(scratch_mem) = scratch_mem { self.vm .update_scratch_mapping(scratch_mem) @@ -522,6 +602,8 @@ impl Sandbox { /// ## Status after restore /// /// A successful restore sets the status to [`Ready`](SandboxStatus::Ready). + /// The restored snapshot becomes the basis for later calls to + /// [`Self::snapshot`]. /// The restored state includes snapshot and scratch memory, vCPU state, /// stack state, the next VM action, captured MSRs on x86_64, and the removal /// of dynamic memory mappings. This discards leaked allocations, restores @@ -646,12 +728,7 @@ impl Sandbox { self.status = SandboxStatus::Poisoned; self.snapshot = None; - let current_regions: Vec = self.vm.get_mapped_regions().cloned().collect(); - for region in ¤t_regions { - self.vm - .unmap_region(region) - .map_err(HyperlightVmError::UnmapRegion)?; - } + self.remove_dynamic_mappings()?; if let Err(error) = self.restore_memory_and_mappings(&snapshot) { self.status = SandboxStatus::Unrecoverable; @@ -852,6 +929,10 @@ impl Sandbox { /// (typically page-aligned). The `region_type` field is ignored as guest /// page table entries are not created. /// + /// [`Self::snapshot`] copies only the pages of this region that the guest has + /// mapped into its page tables, then removes the region. Pages the guest has + /// not mapped are lost. + /// /// ## Poisoned Sandbox /// /// This method will return [`crate::HyperlightError::PoisonedSandbox`] if the sandbox @@ -881,6 +962,10 @@ impl Sandbox { /// /// Returns the length of the mapping in bytes. /// + /// [`Self::snapshot`] copies only the pages of this region that the guest has + /// mapped into its page tables, then removes the region. Pages the guest has + /// not mapped are lost. + /// /// ## Poisoned Sandbox /// /// This method will return [`crate::HyperlightError::PoisonedSandbox`] if the sandbox @@ -894,7 +979,7 @@ impl Sandbox { // Validate that the full mapped range doesn't overlap the // sandbox's primary shared memory region. - let shared_size = self.mem_mgr.shared_mem.mem_size() as u64; + let shared_size = self.mem_mgr.shared_mem.gpa_span_len() as u64; let base_addr = crate::mem::layout::SandboxMemoryLayout::BASE_ADDRESS as u64; let shared_end = base_addr.checked_add(shared_size).ok_or_else(|| { crate::HyperlightError::Error("shared memory end overflow".to_string()) @@ -1263,6 +1348,7 @@ mod tests { use hyperlight_testing::sandbox_sizes::{LARGE_HEAP_SIZE, MEDIUM_HEAP_SIZE, SMALL_HEAP_SIZE}; use hyperlight_testing::{c_simple_guest_as_pathbuf, simple_guest_as_pathbuf}; + use super::normalize_page_table_roots; use crate::func::host_functions::Registerable; #[cfg(not(gdb))] use crate::hypervisor::hyperlight_vm::test_support::VmOperation; @@ -1558,12 +1644,9 @@ mod tests { let generation = sandbox.mem_mgr.snapshot_count; let ring = sandbox.mem_mgr.layout.get_transport_arena().g2h_ring_addr(); - let offset = sandbox - .mem_mgr - .layout - .resolve_gpa(ring, &[]) - .unwrap() - .offset; + let scratch_base = + hyperlight_common::layout::scratch_base_gpa(sandbox.mem_mgr.layout.get_scratch_size()); + let offset = usize::try_from(ring - scratch_base).unwrap(); sandbox.mem_mgr.scratch_mem.write::(offset, 1).unwrap(); @@ -1933,7 +2016,9 @@ mod tests { } fn allocate_guest_memory() -> GuestSharedMemory { - page_aligned_memory(b"test data for snapshot") + let mut data = vec![0; 2 * page_size::get()]; + data[..b"test data for snapshot".len()].copy_from_slice(b"test data for snapshot"); + page_aligned_memory(&data) } #[test] @@ -1966,7 +2051,7 @@ mod tests { // 3. Take snapshot 2 with 1 region mapped let snapshot2 = sbox.snapshot().unwrap(); - assert_eq!(sbox.vm.get_mapped_regions().count(), 1); + assert_eq!(sbox.vm.get_mapped_regions().count(), 0); // 4. Re(store to snapshot 1 (should unmap the region) sbox.restore(snapshot1.clone()).unwrap(); @@ -1999,6 +2084,59 @@ mod tests { assert_eq!(new_read, orig_read); } + /// `map_region` invalidates the cached snapshot without running the guest, + /// and its pages are not in guest VA space until a later call maps them. So + /// the next capture walks the same page tables the previous one built, finds + /// every page already backed by a layer, and materializes nothing. + #[test] + #[cfg(not(unshared_snapshot_mem))] + fn capture_without_guest_progress_reuses_every_layer() { + let mut sbox = SandboxBuilder::from_file(simple_guest_as_pathbuf()) + .build() + .unwrap(); + + let first = sbox.snapshot().unwrap(); + assert!( + !first.snapshot_memory().layers().is_empty(), + "the first capture materializes the initialization delta" + ); + + let map_mem = allocate_guest_memory(); + let region = region_for_memory(&map_mem, 0x200000000_usize, MemoryRegionFlags::READ); + // SAFETY: `map_mem` remains alive until capture removes the mapping. + unsafe { sbox.map_region(®ion).unwrap() }; + + let second = sbox.snapshot().unwrap(); + assert!( + !Arc::ptr_eq(&first, &second), + "map_region must invalidate the cached snapshot" + ); + assert_eq!(sbox.vm.get_mapped_regions().count(), 0); + + // Every data-bearing layer is shared with the previous snapshot. + for layer in second.snapshot_memory().layers().iter() { + assert!( + first + .snapshot_memory() + .layers() + .iter() + .any(|previous| Arc::ptr_eq(previous, layer)), + "capture copied a page it could have shared" + ); + } + + assert_eq!( + second.snapshot_memory().layers().len(), + first.snapshot_memory().layers().len() + ); + + sbox.restore(second).unwrap(); + assert_eq!( + sbox.call::("Echo", "hello".to_string()).unwrap(), + "hello" + ); + } + /// Compaction copies mapped-region pages into the snapshot blob, /// so cross-instance restore preserves their contents without the /// target ever mapping the region. @@ -2098,6 +2236,37 @@ mod tests { assert_eq!(sandbox.call::("GetStatic", ()).unwrap(), 0); } + #[test] + #[cfg(not(gdb))] + fn snapshot_capture_mapping_failure_is_unrecoverable() { + let mut sandbox = SandboxBuilder::from_file(simple_guest_as_pathbuf()) + .build() + .unwrap(); + let recovery_snapshot = sandbox.snapshot().unwrap(); + sandbox.call::("AddToStatic", 42i32).unwrap(); + let fault_plan = sandbox + .vm + .inject_vm_faults([VmOperation::Map(MemoryRegionType::Snapshot)]); + + let error = sandbox.snapshot().err().unwrap(); + + assert!(matches!(error, HyperlightError::HyperlightVmError(_))); + assert!(fault_plan.is_consumed()); + assert_eq!(sandbox.status(), SandboxStatus::Unrecoverable); + assert!(matches!( + sandbox.restore(recovery_snapshot), + Err(HyperlightError::UnrecoverableSandbox) + )); + assert!(matches!( + sandbox.call::("GetStatic", ()), + Err(HyperlightError::UnrecoverableSandbox) + )); + assert!(matches!( + sandbox.snapshot(), + Err(HyperlightError::UnrecoverableSandbox) + )); + } + #[test] #[cfg(not(gdb))] fn snapshot_restore_mapping_failure_is_unrecoverable() { @@ -2122,7 +2291,7 @@ mod tests { let error = target.restore(snapshot.clone()).unwrap_err(); assert!(matches!(error, HyperlightError::HyperlightVmError(_))); assert_eq!(target.status(), SandboxStatus::Unrecoverable); - assert_eq!(target.vm.base_mapping_state(), (None, mappings.1)); + assert_eq!(target.vm.base_mapping_state(), (Vec::new(), mappings.1)); assert!(fault_plan.is_consumed()); assert!(matches!( @@ -2273,6 +2442,36 @@ mod tests { std::fs::remove_file(mapped_path).unwrap(); } + #[test] + #[cfg(not(gdb))] + fn snapshot_capture_dynamic_unmapping_failure_is_recoverable() { + let path = simple_guest_as_pathbuf(); + let mut sandbox = UninitializedSandbox::new(GuestBinary::FilePath(path), None) + .unwrap() + .evolve() + .unwrap(); + let recovery_snapshot = sandbox.snapshot().unwrap(); + sandbox.call::("AddToStatic", 42i32).unwrap(); + + let map_mem = allocate_guest_memory(); + let region = region_for_memory(&map_mem, 0x200000000_usize, MemoryRegionFlags::READ); + // SAFETY: `map_mem` remains alive until restore removes the mapping. + unsafe { sandbox.map_region(®ion).unwrap() }; + let fault_plan = sandbox + .vm + .inject_vm_faults([VmOperation::Unmap(MemoryRegionType::Heap)]); + + let error = sandbox.snapshot().err().unwrap(); + assert!(matches!(error, HyperlightError::HyperlightVmError(_))); + assert!(sandbox.status().is_poisoned()); + assert!(fault_plan.is_consumed()); + + sandbox.restore(recovery_snapshot).unwrap(); + assert_eq!(sandbox.status(), SandboxStatus::Ready); + assert_eq!(sandbox.vm.get_mapped_regions().count(), 0); + assert_eq!(sandbox.call::("GetStatic", ()).unwrap(), 0); + } + #[test] #[cfg(not(gdb))] fn snapshot_restore_vcpu_reset_failure_is_recoverable() { @@ -2306,18 +2505,14 @@ mod tests { assert!(matches!(error, HyperlightError::HyperlightVmError(_))); assert!(target.status().is_poisoned()); assert!(fault_plan.is_consumed()); + let (snapshot_mappings, scratch_mapping) = target.vm.base_mapping_state(); + assert!(!snapshot_mappings.is_empty()); assert_eq!( - target.vm.base_mapping_state(), - ( - Some(( - target.mem_mgr.shared_mem.base_addr(), - target.mem_mgr.shared_mem.mem_size(), - )), - Some(( - target.mem_mgr.scratch_mem.base_addr(), - target.mem_mgr.scratch_mem.mem_size(), - )), - ) + scratch_mapping, + Some(( + target.mem_mgr.scratch_mem.base_addr(), + target.mem_mgr.scratch_mem.mem_size(), + )) ); target.restore(snapshot.clone()).unwrap(); @@ -2326,6 +2521,176 @@ mod tests { } } + #[test] + #[cfg(not(gdb))] + fn snapshot_capture_page_table_switch_failure_is_recoverable() { + let path = simple_guest_as_pathbuf(); + let mut sandbox = UninitializedSandbox::new(GuestBinary::FilePath(path), None) + .unwrap() + .evolve() + .unwrap(); + let recovery_snapshot = sandbox.snapshot().unwrap(); + sandbox.call::("AddToStatic", 42i32).unwrap(); + + let fault_plan = sandbox.vm.inject_vm_faults([VmOperation::SetSregs]); + + let error = sandbox.snapshot().err().unwrap(); + assert!(matches!(error, HyperlightError::HyperlightVmError(_))); + assert!(sandbox.status().is_poisoned()); + assert!(fault_plan.is_consumed()); + + sandbox.restore(recovery_snapshot).unwrap(); + assert_eq!(sandbox.status(), SandboxStatus::Ready); + assert_eq!(sandbox.call::("GetStatic", ()).unwrap(), 0); + } + + #[test] + #[cfg(not(gdb))] + fn snapshot_capture_does_not_reset_vcpu() { + #[cfg(target_arch = "x86_64")] + let reset_operations = [ + VmOperation::SetRegs, + VmOperation::SetDebugRegs, + VmOperation::ResetXsave, + VmOperation::SetMsrs, + ]; + #[cfg(target_arch = "aarch64")] + let reset_operations = [VmOperation::ResetVcpu]; + + for reset_operation in reset_operations { + let path = simple_guest_as_pathbuf(); + let mut sandbox = UninitializedSandbox::new(GuestBinary::FilePath(path), None) + .unwrap() + .evolve() + .unwrap(); + sandbox.call::("AddToStatic", 42i32).unwrap(); + sandbox.checkpoint_transport_for_snapshot().unwrap(); + let fault_plan = sandbox.vm.inject_vm_faults([reset_operation]); + + sandbox.snapshot().unwrap(); + + assert!(!fault_plan.is_consumed()); + assert_eq!(sandbox.status(), SandboxStatus::Ready); + } + } + + #[test] + fn snapshot_capture_reclaims_internal_scratch_pages() { + let path = simple_guest_as_pathbuf(); + let mut sandbox = UninitializedSandbox::new(GuestBinary::FilePath(path), None) + .unwrap() + .evolve() + .unwrap(); + sandbox.call::("AddToStatic", 42i32).unwrap(); + + sandbox.snapshot().unwrap(); + + let allocator_offset = sandbox.mem_mgr.scratch_mem.mem_size() + - hyperlight_common::layout::SCRATCH_TOP_ALLOCATOR_OFFSET as usize; + assert_eq!( + sandbox + .mem_mgr + .scratch_mem + .read::(allocator_offset) + .unwrap(), + sandbox.mem_mgr.first_free_scratch_gpa() + ); + } + + #[test] + fn snapshot_capture_zeroes_reclaimed_scratch() { + let path = simple_guest_as_pathbuf(); + let mut sandbox = UninitializedSandbox::new(GuestBinary::FilePath(path), None) + .unwrap() + .evolve() + .unwrap(); + let page_gva = 0x4000_0000_u64; + let page_size = hyperlight_common::vmem::PAGE_SIZE; + assert!(sandbox.call::("MapZeroInitPage", page_gva).unwrap()); + let allocator_offset = sandbox.mem_mgr.scratch_mem.mem_size() + - hyperlight_common::layout::SCRATCH_TOP_ALLOCATOR_OFFSET as usize; + let used_end = sandbox + .mem_mgr + .scratch_mem + .read::(allocator_offset) + .unwrap(); + let before = sandbox + .mem_mgr + .scratch_mem + .with_contents(|scratch| scratch.to_vec()) + .unwrap(); + sandbox.snapshot().unwrap(); + + let scratch_base = + hyperlight_common::layout::scratch_base_gpa(sandbox.mem_mgr.scratch_mem.mem_size()); + let free_start = + usize::try_from(sandbox.mem_mgr.first_free_scratch_gpa() - scratch_base).unwrap(); + let free_end = usize::try_from(used_end - scratch_base).unwrap(); + let after = sandbox + .mem_mgr + .scratch_mem + .with_contents(|scratch| scratch.to_vec()) + .unwrap(); + assert!(before[free_start..free_end].iter().any(|&byte| byte != 0)); + assert!(after[free_start..free_end].iter().all(|&byte| byte == 0)); + + let page = sandbox + .call::>("ReadMappedBuffer", (page_gva, page_size as u64, false)) + .unwrap(); + assert_eq!(page, vec![0; page_size]); + } + + /// The guest writes the allocator value that bounds install's scratch zeroing. + /// Out-of-range values must not zero memory outside free scratch. + #[test] + fn snapshot_capture_clamps_guest_allocator_value() { + use hyperlight_common::layout::{ + SCRATCH_TOP_ALLOCATOR_OFFSET, SCRATCH_TOP_SIZE_OFFSET, scratch_base_gpa, + }; + + for allocator in [0, u64::MAX] { + let mut sbox = SandboxBuilder::from_file(simple_guest_as_pathbuf()) + .build() + .unwrap(); + sbox.snapshot().unwrap(); + let scratch_size = sbox.mem_mgr.scratch_mem.mem_size(); + let first_free = sbox.mem_mgr.first_free_scratch_gpa(); + let marker_offset = + usize::try_from(first_free - scratch_base_gpa(scratch_size)).unwrap(); + let marker = 0x1234_5678_9abc_def0_u64; + let scratch = &sbox.mem_mgr.scratch_mem; + scratch.write(marker_offset, marker).unwrap(); + scratch + .write( + scratch_size - SCRATCH_TOP_ALLOCATOR_OFFSET as usize, + allocator, + ) + .unwrap(); + + // `map_region` drops the cached snapshot without running the guest. + let map_mem = allocate_guest_memory(); + let region = region_for_memory(&map_mem, 0x200000000_usize, MemoryRegionFlags::READ); + // SAFETY: `map_mem` remains alive until capture removes the mapping. + unsafe { sbox.map_region(®ion).unwrap() }; + sbox.snapshot().unwrap(); + + let scratch = &sbox.mem_mgr.scratch_mem; + assert_eq!(sbox.mem_mgr.first_free_scratch_gpa(), first_free); + let expected = if allocator == 0 { marker } else { 0 }; + assert_eq!(scratch.read::(marker_offset).unwrap(), expected); + assert_eq!( + scratch + .read::(scratch_size - SCRATCH_TOP_SIZE_OFFSET as usize) + .unwrap(), + scratch_size as u64 + ); + assert_eq!( + sbox.call::("Echo", "hello\n".to_string()).unwrap(), + "hello\n" + ); + } + } + #[test] #[cfg(all(target_arch = "x86_64", not(gdb)))] fn snapshot_restore_msr_failure_is_recoverable() { @@ -2348,18 +2713,14 @@ mod tests { assert!(matches!(error, HyperlightError::HyperlightVmError(_))); assert!(target.status().is_poisoned()); assert!(fault_plan.is_consumed()); + let (snapshot_mappings, scratch_mapping) = target.vm.base_mapping_state(); + assert!(!snapshot_mappings.is_empty()); assert_eq!( - target.vm.base_mapping_state(), - ( - Some(( - target.mem_mgr.shared_mem.base_addr(), - target.mem_mgr.shared_mem.mem_size(), - )), - Some(( - target.mem_mgr.scratch_mem.base_addr(), - target.mem_mgr.scratch_mem.mem_size(), - )), - ) + scratch_mapping, + Some(( + target.mem_mgr.scratch_mem.base_addr(), + target.mem_mgr.scratch_mem.mem_size(), + )) ); target.restore(snapshot).unwrap(); @@ -2607,7 +2968,7 @@ mod tests { assert_eq!(source.call::("StackAllocate", 256i32).unwrap(), 256); assert_eq!(target.call::("AddToStatic", 17i32).unwrap(), 17); - target.set_pt_root_finder(Box::new(|_, _, root| vec![root])); + target.set_pt_root_finder(Box::new(|_, _, root| Ok(vec![root]))); assert!(target.pt_root_finder.is_some()); assert_ne!( @@ -2635,6 +2996,74 @@ mod tests { )); } + #[test] + fn active_page_table_root_is_first() { + const ACTIVE: u64 = 0x1000; + const OTHER: u64 = 0x2000; + + assert_eq!( + normalize_page_table_roots(vec![OTHER, ACTIVE, ACTIVE], ACTIVE), + vec![ACTIVE, OTHER] + ); + assert_eq!( + normalize_page_table_roots(vec![OTHER, OTHER], ACTIVE), + vec![ACTIVE, OTHER] + ); + assert_eq!( + normalize_page_table_roots(vec![OTHER], ACTIVE), + vec![ACTIVE, OTHER] + ); + assert_eq!(normalize_page_table_roots(Vec::new(), ACTIVE), vec![ACTIVE]); + } + + #[test] + fn page_table_root_finder_reads_snapshot_memory() { + use std::sync::atomic::{AtomicBool, Ordering}; + + let mut sandbox = SandboxBuilder::from_file(simple_guest_as_pathbuf()) + .build() + .unwrap(); + let base = crate::mem::layout::SandboxMemoryLayout::BASE_ADDRESS as u64; + let mut expected = [0; 16]; + sandbox + .mem_mgr + .shared_mem + .read_snapshot_gpa(base, &mut expected) + .unwrap(); + let invoked = Arc::new(AtomicBool::new(false)); + let callback_invoked = invoked.clone(); + sandbox.set_pt_root_finder(Box::new(move |snapshot, _, root| { + let mut bytes = [0; 16]; + snapshot.read_exact(base, &mut bytes)?; + assert_eq!(bytes, expected); + callback_invoked.store(true, Ordering::Relaxed); + Ok(vec![root]) + })); + + sandbox.snapshot().unwrap(); + assert!(invoked.load(Ordering::Relaxed)); + } + + #[test] + fn page_table_root_finder_error_aborts_snapshot() { + let mut sandbox = SandboxBuilder::from_file(simple_guest_as_pathbuf()) + .build() + .unwrap(); + sandbox.set_pt_root_finder(Box::new(|_, _, _| { + Err(crate::new_error!("failed to discover page-table roots")) + })); + + let error = sandbox.snapshot().err().unwrap(); + + assert!( + error + .to_string() + .contains("failed to discover page-table roots") + ); + assert_eq!(sandbox.status(), SandboxStatus::Ready); + assert!(sandbox.snapshot.is_none()); + } + #[test] fn snapshot_restore_replaces_c_guest_with_rust_guest() { let mut source = @@ -2810,7 +3239,7 @@ mod tests { )); assert!(Arc::ptr_eq(&target.snapshot().unwrap(), &cached_snapshot)); - assert_eq!(target.vm.get_mapped_regions().count(), 1); + assert_eq!(target.vm.get_mapped_regions().count(), 0); assert!( target .call::("CheckMapped", guest_base as u64) @@ -2868,11 +3297,13 @@ mod tests { .unwrap() .evolve() .unwrap(); - assert!(snapshot.memory().mem_size() > target.mem_mgr.shared_mem.mem_size()); + assert!( + snapshot.snapshot_memory().gpa_span_len() > target.mem_mgr.shared_mem.gpa_span_len() + ); let map_mem = allocate_guest_memory(); let guest_base = crate::mem::layout::SandboxMemoryLayout::BASE_ADDRESS - + target.mem_mgr.shared_mem.mem_size(); + + target.mem_mgr.shared_mem.gpa_span_len(); let region = region_for_memory(&map_mem, guest_base, MemoryRegionFlags::READ); // SAFETY: `map_mem` is page-aligned and outlives every use of `target`. unsafe { target.map_region(®ion).unwrap() }; @@ -3209,7 +3640,7 @@ mod tests { use std::io::Write; let page_size = page_size::get(); - let padded_len = content.len().max(page_size).div_ceil(page_size) * page_size; + let padded_len = content.len().max(page_size).div_ceil(page_size) * page_size + page_size; let mut padded = vec![0u8; padded_len]; padded[..content.len()].copy_from_slice(content); diff --git a/src/hyperlight_host/src/sandbox/mod.rs b/src/hyperlight_host/src/sandbox/mod.rs index ca2c94b8e..cdc4c920d 100644 --- a/src/hyperlight_host/src/sandbox/mod.rs +++ b/src/hyperlight_host/src/sandbox/mod.rs @@ -45,7 +45,7 @@ pub use config::SandboxConfiguration; #[allow(deprecated)] pub use initialized::MultiUseSandbox; /// Re-export for the `Sandbox` type -pub use initialized::{PtRootFinder, Sandbox, SandboxStatus}; +pub use initialized::{PtRootFinder, Sandbox, SandboxStatus, SnapshotMemoryReader}; /// Re-export for `GuestBinary` type pub use uninitialized::GuestBinary; /// Re-export for `UninitializedSandbox` type diff --git a/src/hyperlight_host/src/sandbox/snapshot/file/config.rs b/src/hyperlight_host/src/sandbox/snapshot/file/config.rs index 1427c53a5..4012b01b4 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/file/config.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/file/config.rs @@ -2,6 +2,7 @@ // Copyright 2025 The Hyperlight Authors. use std::collections::BTreeMap; +use std::sync::Arc; use hyperlight_common::flatbuffer_wrappers::function_types::{ParameterType, ReturnType}; use hyperlight_common::flatbuffer_wrappers::host_function_definition::HostFunctionDefinition; @@ -13,6 +14,7 @@ use crate::hypervisor::regs::CommonSpecialRegisters; #[cfg(target_arch = "x86_64")] use crate::hypervisor::regs::MsrEntry; use crate::mem::layout::SandboxMemoryLayout; +use crate::sandbox::snapshot::memory::SnapshotBlob; // --- Arch and hypervisor identifiers -------------------------------- @@ -161,11 +163,11 @@ impl CpuVendor { /// Top-level Hyperlight snapshot config JSON. Lives at /// `blobs/sha256/` with media type -/// `application/vnd.hyperlight.snapshot.config.v3+json`. +/// `application/vnd.hyperlight.snapshot.config.v4+json`. /// /// In OCI terms this is the "image config" blob that the manifest's /// `config` descriptor points to. It describes the accompanying -/// memory layer (the snapshot bytes) and everything the loader needs +/// data, page-table and transport layers and everything the loader needs /// to reconstruct a runnable `Snapshot`. #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -179,6 +181,8 @@ pub(super) struct OciSnapshotConfig { pub(super) hypervisor: Hypervisor, /// CPU vendor captured at snapshot time. Checked on load. pub(super) cpu_vendor: CpuVendor, + /// Host page size at capture. Blob storage is padded to it. + pub(super) host_page_size: usize, /// Top of the guest stack, in guest virtual address space. pub(super) stack_top_gva: u64, /// Guest virtual address the loader resumes the paused call at. @@ -196,9 +200,10 @@ pub(super) struct OciSnapshotConfig { #[cfg(target_arch = "x86_64")] pub(super) msrs: Vec, pub(super) layout: MemoryLayout, - /// Total size of the memory blob in bytes (including the guest - /// page-table tail, if any). Equal to `self.memory.mem_size()`. - pub(super) memory_size: u64, + /// Snapshot blobs, in manifest descriptor order. + pub(super) layers: Vec, + /// Index into `layers` of the blob with the page tables of this snapshot. + pub(super) page_table_layer: usize, /// Names and signatures of host functions registered when this /// snapshot was taken. Validated against the loader's registry. pub(super) host_functions: Vec, @@ -233,8 +238,26 @@ pub(super) struct MemoryLayout { pub(super) h2g_buffer_size: usize, pub(super) g2h_pool_pages: usize, pub(super) h2g_pool_pages: usize, - pub(super) snapshot_size: usize, - pub(super) pt_size: Option, +} + +/// One snapshot blob: guest data mapped at `gpa_start`, followed by page +/// tables. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct OciSnapshotLayer { + pub(super) gpa_start: u64, + pub(super) data_len: usize, + pub(super) page_table_len: usize, +} + +impl From<&Arc> for OciSnapshotLayer { + fn from(layer: &Arc) -> Self { + Self { + gpa_start: layer.gpa_start(), + data_len: layer.data_len(), + page_table_len: layer.page_table_len(), + } + } } /// Name and signature of one host function registered when the @@ -450,65 +473,17 @@ impl OciSnapshotConfig { self.hyperlight_version )); } - // Bound memory size early so the subsequent file-size check - // does not have to deal with absurd values. - if self.memory_size == 0 || self.memory_size > SandboxMemoryLayout::MAX_MEMORY_SIZE as u64 { - return Err(crate::new_error!( - "snapshot memory_size ({}) is out of range", - self.memory_size - )); - } - if !(self.memory_size as usize).is_multiple_of(PAGE_SIZE) { + if self.host_page_size != page_size::get() { return Err(crate::new_error!( - "snapshot memory_size ({}) is not a multiple of PAGE_SIZE", - self.memory_size - )); - } - // `snapshot_size` is the guest-visible prefix of the blob, - // mapped at `BASE_ADDRESS`. `pt_size` is the page-table tail - // after it, present in the blob and host mapping but outside - // the guest mapping. They sum to `memory_size`. - if self.layout.snapshot_size == 0 { - return Err(crate::new_error!("snapshot snapshot_size must be nonzero")); - } - if !self.layout.snapshot_size.is_multiple_of(PAGE_SIZE) { - return Err(crate::new_error!( - "snapshot snapshot_size ({}) is not a multiple of PAGE_SIZE", - self.layout.snapshot_size - )); - } - let pt = self - .layout - .pt_size - .ok_or_else(|| crate::new_error!("snapshot pt_size is missing"))?; - if pt == 0 { - return Err(crate::new_error!("snapshot pt_size must be nonzero")); - } - if !pt.is_multiple_of(PAGE_SIZE) { - return Err(crate::new_error!( - "snapshot pt_size ({}) is not a multiple of PAGE_SIZE", - pt - )); - } - // The total memory size might be bigger because it has to - // take into account the host page size, as well as the guest - // page size. - let total_size = (self.layout.snapshot_size as u64) - .saturating_add(pt as u64) - .next_multiple_of(page_size::get() as u64); - if total_size != self.memory_size { - return Err(crate::new_error!( - "snapshot snapshot_size ({}) + pt_size ({}), rounded to {}, does not equal memory_size ({})", - self.layout.snapshot_size, - pt, - total_size, - self.memory_size + "snapshot host page size mismatch: file uses {}, current host uses {}", + self.host_page_size, + page_size::get() )); } // Cap each layout field at `MAX_MEMORY_SIZE` so the later // size and offset sums in `SandboxMemoryLayout` cannot // overflow `u64`. Whether the regions fit the snapshot is - // checked against `snapshot_size` in `load_inner`. + // checked against the snapshot GPA span in `load_inner`. let max_region = SandboxMemoryLayout::MAX_MEMORY_SIZE; for (name, value) in [ ("heap_size", self.layout.heap_size), @@ -840,6 +815,7 @@ mod tests { abi_version: SNAPSHOT_ABI_VERSION, hypervisor: Hypervisor::Mshv, cpu_vendor: CpuVendor::current(), + host_page_size: page_size::get(), stack_top_gva: 0x2000, entrypoint_addr: SandboxMemoryLayout::BASE_ADDRESS as u64, original_entrypoint_addr: SandboxMemoryLayout::BASE_ADDRESS as u64, @@ -859,10 +835,9 @@ mod tests { h2g_buffer_size: PAGE_SIZE, g2h_pool_pages: 8, h2g_pool_pages: 4, - snapshot_size: PAGE_SIZE, - pt_size: None, }, - memory_size: PAGE_SIZE as u64, + layers: Vec::new(), + page_table_layer: 0, host_functions: Vec::new(), snapshot_generation: 0, metadata: BTreeMap::new(), @@ -941,9 +916,10 @@ mod schema_pin { const PINNED_CALL: &str = r#"{ "hyperlight_version": "x.y.z", "arch": "x86_64", - "abi_version": 6, + "abi_version": 7, "hypervisor": "mshv", "cpu_vendor": "intel", + "host_page_size": 4096, "stack_top_gva": 3735928559, "entrypoint_addr": 8192, "original_entrypoint_addr": 4096, @@ -1111,11 +1087,16 @@ mod schema_pin { "g2h_buffer_size": 4096, "h2g_buffer_size": 4096, "g2h_pool_pages": 8, - "h2g_pool_pages": 4, - "snapshot_size": 9, - "pt_size": null + "h2g_pool_pages": 4 }, - "memory_size": 65536, + "layers": [ + { + "gpa_start": 16384, + "data_len": 65536, + "page_table_len": 4096 + } + ], + "page_table_layer": 0, "host_functions": [ { "function_name": "fn_void", @@ -1132,9 +1113,10 @@ mod schema_pin { const PINNED_CALL: &str = r#"{ "hyperlight_version": "x.y.z", "arch": "aarch64", - "abi_version": 6, + "abi_version": 7, "hypervisor": "mshv", "cpu_vendor": "intel", + "host_page_size": 16384, "stack_top_gva": 3735928559, "entrypoint_addr": 8192, "original_entrypoint_addr": 4096, @@ -1158,11 +1140,16 @@ mod schema_pin { "g2h_buffer_size": 4096, "h2g_buffer_size": 4096, "g2h_pool_pages": 8, - "h2g_pool_pages": 4, - "snapshot_size": 9, - "pt_size": null + "h2g_pool_pages": 4 }, - "memory_size": 65536, + "layers": [ + { + "gpa_start": 16384, + "data_len": 65536, + "page_table_len": 4096 + } + ], + "page_table_layer": 0, "host_functions": [ { "function_name": "fn_void", diff --git a/src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs b/src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs index e745ef2af..9eedbb6e8 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs @@ -10,10 +10,12 @@ pub(in crate::sandbox::snapshot) const MT_CONFIG_V2: &str = "application/vnd.hyperlight.snapshot.config.v2+json"; pub(in crate::sandbox::snapshot) const MT_CONFIG_V3: &str = "application/vnd.hyperlight.snapshot.config.v3+json"; -pub(in crate::sandbox::snapshot) const MT_CONFIG_CURRENT: &str = MT_CONFIG_V3; -pub(in crate::sandbox::snapshot) const MT_SNAPSHOT_V1: &str = - "application/vnd.hyperlight.snapshot.memory.v1"; -pub(in crate::sandbox::snapshot) const MT_SNAPSHOT_CURRENT: &str = MT_SNAPSHOT_V1; +pub(in crate::sandbox::snapshot) const MT_CONFIG_V4: &str = + "application/vnd.hyperlight.snapshot.config.v4+json"; +pub(in crate::sandbox::snapshot) const MT_CONFIG_CURRENT: &str = MT_CONFIG_V4; +pub(in crate::sandbox::snapshot) const MT_SNAPSHOT_V2: &str = + "application/vnd.hyperlight.snapshot.memory.v2"; +pub(in crate::sandbox::snapshot) const MT_SNAPSHOT_CURRENT: &str = MT_SNAPSHOT_V2; pub(in crate::sandbox::snapshot) const MT_TRANSPORT_V1: &str = "application/vnd.hyperlight.snapshot.transport.v1"; pub(in crate::sandbox::snapshot) const MT_TRANSPORT_CURRENT: &str = MT_TRANSPORT_V1; @@ -21,7 +23,7 @@ pub(in crate::sandbox::snapshot) const MT_TRANSPORT_CURRENT: &str = MT_TRANSPORT /// ABI version for the snapshot memory blob. Bumped when the /// host-guest contract for the snapshot bytes changes. See /// docs/snapshot-versioning.md. -pub(in crate::sandbox::snapshot) const SNAPSHOT_ABI_VERSION: u32 = 6; +pub(in crate::sandbox::snapshot) const SNAPSHOT_ABI_VERSION: u32 = 7; /// OCI standard annotation key for a manifest's tag inside an image /// index. Set on the manifest descriptor in `index.json`, not on the diff --git a/src/hyperlight_host/src/sandbox/snapshot/file/mod.rs b/src/hyperlight_host/src/sandbox/snapshot/file/mod.rs index b336907f5..3fa493fab 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/file/mod.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/file/mod.rs @@ -12,26 +12,29 @@ pub(crate) mod reference; mod transport; use std::path::{Path, PathBuf}; +use std::sync::Arc; use hyperlight_common::flatbuffer_wrappers::host_function_details::HostFunctionDetails; -use hyperlight_common::vmem::PAGE_SIZE; use oci_spec::image::{ Descriptor, DescriptorBuilder, ImageIndex, ImageIndexBuilder, ImageManifest, ImageManifestBuilder, MediaType, SCHEMA_VERSION, }; -use self::config::{Arch, CpuVendor, HostFunction, Hypervisor, MemoryLayout, OciSnapshotConfig}; +use self::config::{ + Arch, CpuVendor, HostFunction, Hypervisor, MemoryLayout, OciSnapshotConfig, OciSnapshotLayer, +}; use self::digest::{Digest256, oci_digest, parse_oci_digest, verify_blob_bytes, verify_blob_file}; use self::fsutil::{put_blob, put_blob_if_absent, read_bounded, replace_file_atomic}; use self::media_types::{ ANNOTATION_ARCH, ANNOTATION_CPU, ANNOTATION_HYPERVISOR, ANNOTATION_REF_NAME, }; pub(super) use self::media_types::{ - MT_CONFIG_CURRENT, MT_CONFIG_V1, MT_CONFIG_V2, MT_CONFIG_V3, MT_SNAPSHOT_CURRENT, - MT_SNAPSHOT_V1, MT_TRANSPORT_CURRENT, MT_TRANSPORT_V1, SNAPSHOT_ABI_VERSION, + MT_CONFIG_CURRENT, MT_CONFIG_V1, MT_CONFIG_V2, MT_CONFIG_V3, MT_CONFIG_V4, MT_SNAPSHOT_CURRENT, + MT_SNAPSHOT_V2, MT_TRANSPORT_CURRENT, MT_TRANSPORT_V1, SNAPSHOT_ABI_VERSION, }; use self::reference::{OciDigest, OciReference, OciTag}; -use super::{NextAction, Snapshot}; +use super::memory::{MAX_SNAPSHOT_LAYERS, host_page_size}; +use super::{NextAction, Snapshot, SnapshotBlob, SnapshotMemory}; use crate::mem::layout::SandboxMemoryLayout; use crate::mem::memory_region::MemoryRegionFlags; use crate::mem::shared_mem::{ReadonlySharedMemory, SharedMemory}; @@ -227,28 +230,16 @@ fn load_config( Ok(cfg) } -fn open_snapshot_blob( - blobs_dir: &Path, - snap_desc: &Descriptor, - expected_blob_len: u64, - verify_blobs: bool, -) -> crate::Result { +fn open_snapshot_blob(blobs_dir: &Path, snap_desc: &Descriptor) -> crate::Result { let snap_hex = parse_oci_digest(snap_desc.digest())?; let snap_path = blobs_dir.join(&snap_hex); - let mut snap_file = self::fsutil::open_no_follow(&snap_path)?; + let snap_file = self::fsutil::open_no_follow(&snap_path)?; let snap_file_len = snap_file .metadata() .map_err(|e| crate::new_error!("failed to stat snapshot blob: {}", e))? .len(); - if snap_file_len != expected_blob_len { - return Err(crate::new_error!( - "snapshot blob size mismatch: file is {} bytes, expected {} (memory_size)", - snap_file_len, - expected_blob_len, - )); - } if snap_file_len != snap_desc.size() { return Err(crate::new_error!( "snapshot blob size {} disagrees with OCI descriptor size {}", @@ -256,12 +247,30 @@ fn open_snapshot_blob( snap_desc.size() )); } - if verify_blobs { - verify_blob_file("snapshot", &mut snap_file, &snap_hex)?; - } Ok(snap_file) } +/// The size validation in `open_snapshot_blob` stats the file before +/// mapping. Nothing prevents the file from being truncated between that +/// stat and the mmap, which would leave the mapping shorter than the +/// descriptor claims and make restore read past the end. Compare the mapped +/// length against the descriptor to reject a file mutated under us. +fn map_snapshot_blob( + snap_file: &std::fs::File, + snap_desc: &Descriptor, + data_len: usize, +) -> crate::Result { + let memory = ReadonlySharedMemory::from_file(snap_file, data_len)?; + if memory.mem_size() as u64 != snap_desc.size() { + return Err(crate::new_error!( + "mapped snapshot size ({}) does not match descriptor size ({}); the blob may have changed during loading", + memory.mem_size(), + snap_desc.size() + )); + } + Ok(memory) +} + /// Read a bounded OCI blob and check its descriptor size and optional digest. fn load_blob( label: &str, @@ -285,6 +294,15 @@ fn load_blob( Ok(bytes) } +fn blob_descriptor(media_type: &str, digest: &Digest256, size: usize) -> crate::Result { + DescriptorBuilder::default() + .media_type(MediaType::Other(media_type.to_string())) + .digest(oci_digest(digest)?) + .size(size as u64) + .build() + .map_err(|e| crate::new_error!("failed to build {media_type} descriptor: {e}")) +} + impl Snapshot { /// Save this snapshot into an OCI Image Layout directory on disk. /// The saved snapshot can be loaded later with @@ -497,23 +515,22 @@ impl Snapshot { cfg: &OciSnapshotConfig, cfg_bytes: &[u8], ) -> crate::Result { - let memory_bytes = self.state.memory.as_slice(); - let memory_size = memory_bytes.len(); - if memory_size == 0 || !memory_size.is_multiple_of(PAGE_SIZE) { - return Err(crate::new_error!( - "snapshot memory size {} must be a non-zero multiple of PAGE_SIZE", - memory_size - )); - } - let blobs_dir = dir.join("blobs").join("sha256"); std::fs::create_dir_all(&blobs_dir).map_err(|e| { crate::new_error!("failed to create OCI blobs dir {:?}: {}", blobs_dir, e) })?; - // Snapshot blob: the raw memory bytes. - let snapshot_digest = Digest256::from_bytes(memory_bytes); - put_blob_if_absent(&blobs_dir, &snapshot_digest, memory_bytes)?; + let mut descriptors = self + .state + .memory + .blobs() + .map(|blob| { + let bytes = blob.memory().as_slice(); + let digest = Digest256::from_bytes(bytes); + put_blob_if_absent(&blobs_dir, &digest, bytes)?; + blob_descriptor(MT_SNAPSHOT_CURRENT, &digest, bytes.len()) + }) + .collect::>>()?; // Transport blob: the canonical ring image omitted from memory. let transport = self.state.virtq.as_ref().ok_or_else(|| { @@ -522,30 +539,18 @@ impl Snapshot { let transport_bytes = transport::encode(transport)?; let transport_digest = Digest256::from_bytes(&transport_bytes); put_blob(&blobs_dir, &transport_digest, &transport_bytes)?; + descriptors.push(blob_descriptor( + MT_TRANSPORT_CURRENT, + &transport_digest, + transport_bytes.len(), + )?); // Config blob. let cfg_digest = Digest256::from_bytes(cfg_bytes); put_blob(&blobs_dir, &cfg_digest, cfg_bytes)?; // Manifest blob. - let config_descriptor = DescriptorBuilder::default() - .media_type(MediaType::Other(MT_CONFIG_CURRENT.to_string())) - .digest(oci_digest(&cfg_digest)?) - .size(cfg_bytes.len() as u64) - .build() - .map_err(|e| crate::new_error!("failed to build config descriptor: {}", e))?; - let snapshot_descriptor = DescriptorBuilder::default() - .media_type(MediaType::Other(MT_SNAPSHOT_CURRENT.to_string())) - .digest(oci_digest(&snapshot_digest)?) - .size(memory_size as u64) - .build() - .map_err(|e| crate::new_error!("failed to build snapshot descriptor: {}", e))?; - let transport_descriptor = DescriptorBuilder::default() - .media_type(MediaType::Other(MT_TRANSPORT_CURRENT.to_string())) - .digest(oci_digest(&transport_digest)?) - .size(transport_bytes.len() as u64) - .build() - .map_err(|e| crate::new_error!("failed to build transport descriptor: {}", e))?; + let config_descriptor = blob_descriptor(MT_CONFIG_CURRENT, &cfg_digest, cfg_bytes.len())?; // `artifactType` is set equal to `config.mediaType` per OCI // image-spec "Guidelines for Artifact Usage". Registries // surface this on the distribution-spec referrers API. Tools @@ -555,7 +560,7 @@ impl Snapshot { .media_type(MediaType::ImageManifest) .artifact_type(MediaType::Other(MT_CONFIG_CURRENT.to_string())) .config(config_descriptor) - .layers(vec![snapshot_descriptor, transport_descriptor]) + .layers(descriptors) .build() .map_err(|e| crate::new_error!("failed to build OCI manifest: {}", e))?; let manifest_bytes = serde_json::to_vec_pretty(&manifest) @@ -624,6 +629,7 @@ impl Snapshot { hypervisor: Hypervisor::current() .ok_or_else(|| crate::new_error!("no hypervisor available to tag snapshot"))?, cpu_vendor: CpuVendor::current(), + host_page_size: page_size::get(), stack_top_gva: self.state.stack_top_gva, entrypoint_addr, original_entrypoint_addr: self.state.original_entrypoint, @@ -648,10 +654,19 @@ impl Snapshot { h2g_buffer_size: l.get_h2g_buffer_size(), g2h_pool_pages: l.get_g2h_pool_pages(), h2g_pool_pages: l.get_h2g_pool_pages(), - snapshot_size: l.snapshot_size(), - pt_size: l.pt_size(), }, - memory_size: self.state.memory.mem_size() as u64, + layers: self + .state + .memory + .blobs() + .map(OciSnapshotLayer::from) + .collect(), + page_table_layer: self + .state + .memory + .blobs() + .position(|blob| Arc::ptr_eq(blob, self.state.memory.page_table_blob())) + .ok_or_else(|| crate::new_error!("snapshot page-table blob is missing"))?, host_functions, snapshot_generation: self.state.snapshot_generation, metadata: self.metadata.clone(), @@ -774,8 +789,8 @@ impl Snapshot { // Loader dispatch on config media type. let cfg_media = cfg_desc.media_type().to_string(); match cfg_media.as_str() { - MT_CONFIG_V3 => {} - MT_CONFIG_V1 | MT_CONFIG_V2 => { + MT_CONFIG_V4 => {} + MT_CONFIG_V1 | MT_CONFIG_V2 | MT_CONFIG_V3 => { return Err(crate::new_error!( "snapshot config media type {:?} is incompatible with snapshot ABI {}", cfg_media, @@ -786,7 +801,7 @@ impl Snapshot { return Err(crate::new_error!( "unexpected config media type {:?} (supported: {:?})", other, - MT_CONFIG_V3 + MT_CONFIG_V4 )); } } @@ -813,25 +828,26 @@ impl Snapshot { } } let layers = manifest.layers(); - if layers.len() != 2 { + let (data_descs, tail) = layers.split_at(layers.len().saturating_sub(1)); + let [transport_desc] = tail else { return Err(crate::new_error!( - "expected exactly two OCI layers (memory and transport), found {}", + "expected snapshot layers and one transport layer, found {} layers", layers.len() )); - } - let snap_desc = &layers[0]; - let snap_media = snap_desc.media_type().to_string(); - match snap_media.as_str() { - MT_SNAPSHOT_V1 => {} - other => { - return Err(crate::new_error!( - "unexpected snapshot layer media type {:?} (supported: {:?})", - other, - MT_SNAPSHOT_V1 - )); + }; + for snap_desc in data_descs { + let snap_media = snap_desc.media_type().to_string(); + match snap_media.as_str() { + MT_SNAPSHOT_V2 => {} + other => { + return Err(crate::new_error!( + "unexpected snapshot layer media type {:?} (supported: {:?})", + other, + MT_SNAPSHOT_V2 + )); + } } } - let transport_desc = &layers[1]; let transport_media = transport_desc.media_type().to_string(); match transport_media.as_str() { MT_TRANSPORT_V1 => {} @@ -846,11 +862,70 @@ impl Snapshot { // 4. config blob let cfg = load_config(&blobs_dir, cfg_desc, verify_blobs)?; + // Each data layer has one mapping, and one more blob may hold only page + // tables. Reject before opening blobs. + if cfg.layers.len() > MAX_SNAPSHOT_LAYERS + 1 { + return Err(crate::new_error!( + "snapshot has {} data layers, limit is {}", + cfg.layers.len(), + MAX_SNAPSHOT_LAYERS + 1 + )); + } + if data_descs.len() != cfg.layers.len() { + return Err(crate::new_error!( + "expected {} snapshot layers and one transport layer, found {} layers", + cfg.layers.len(), + layers.len() + )); + } + if cfg.page_table_layer >= cfg.layers.len() { + return Err(crate::new_error!( + "snapshot page_table_layer {} is out of range", + cfg.page_table_layer + )); + } + + // Reject blob sizes no valid snapshot has before opening any blob. + // Guest data must fit between BASE_ADDRESS and scratch without + // overlapping. Page tables must fit in scratch, which is at most + // MAX_MEMORY_SIZE. + let data_space = hyperlight_common::layout::scratch_base_gpa(cfg.layout.scratch_size) + .saturating_sub(SandboxMemoryLayout::BASE_ADDRESS as u64); + let mut data_bytes = 0u64; + for (index, (layer, desc)) in cfg.layers.iter().zip(data_descs).enumerate() { + if layer.data_len == 0 && index != cfg.page_table_layer { + return Err(crate::new_error!("snapshot layer {} has no data", index)); + } + let blob_len = layer + .data_len + .checked_add(layer.page_table_len) + .filter(|_| layer.page_table_len <= SandboxMemoryLayout::MAX_MEMORY_SIZE) + .and_then(|len| len.checked_next_multiple_of(host_page_size())); + if blob_len.is_none_or(|len| len as u64 != desc.size()) { + return Err(crate::new_error!( + "snapshot layer {} size {} does not match its data and page-table lengths", + index, + desc.size() + )); + } + data_bytes = data_bytes + .checked_add(layer.data_len as u64) + .filter(|bytes| *bytes <= data_space) + .ok_or_else(|| { + crate::new_error!( + "snapshot data exceeds the {} bytes below scratch", + data_space + ) + })?; + } - // 5. snapshot blob: open once, hash and mmap the same + // 5. snapshot blobs: open each once, hash and mmap the same // handle so an attacker cannot swap the file between // verification and mapping. - let snap_file = open_snapshot_blob(&blobs_dir, snap_desc, cfg.memory_size, verify_blobs)?; + let mut snap_files = data_descs + .iter() + .map(|desc| open_snapshot_blob(&blobs_dir, desc)) + .collect::>>()?; let transport_bytes = load_blob( "transport", &blobs_dir, @@ -890,53 +965,58 @@ impl Snapshot { cfg.layout.code_virt_base }; layout.set_code_gva(code_gva)?; - // `snapshot_size` and `pt_size` are independent fields. - if let Some(pt) = cfg.layout.pt_size { - layout.set_pt_size(pt)?; - } - layout.set_snapshot_size(cfg.layout.snapshot_size); - - // `snapshot_size` is the guest-visible prefix mapped into the - // snapshot region. It must cover at least the regions the - // layout fields describe (code, PEB, heap, init data), - // otherwise the guest mapping is too short to back them. The - // `snapshot_size + pt_size == memory_size` invariant alone - // does not bound `snapshot_size` from below, since a smaller - // `snapshot_size` can be offset by a larger `pt_size`. - let required_memory_size = layout.get_memory_size()? as u64; - if (layout.snapshot_size() as u64) < required_memory_size { - return Err(crate::new_error!( - "snapshot snapshot_size ({}) is smaller than the layout size ({})", - layout.snapshot_size(), - required_memory_size - )); - } + layout.ensure_page_tables_fit(cfg.layers[cfg.page_table_layer].page_table_len)?; let virtq = transport::decode(&layout, &transport_bytes)?; - // 7. mmap the snapshot blob (file-backed CoW). The blob is - // the raw memory image. `ReadonlySharedMemory::from_file` + // 7. mmap the snapshot blobs (file-backed CoW). Each blob is + // a raw memory image. `ReadonlySharedMemory::from_file` // surrounds it with host guard pages. The guest mapping - // of the snapshot region covers only the data prefix - // (`snapshot_size`). The PT tail sits past that prefix - // in the host mapping and is copied into the scratch - // region on restore. Keeping it out of the guest mapping - // of the snapshot region avoids overlap with - // `map_file_cow` regions installed immediately after the - // snapshot in guest PA space. - let memory = ReadonlySharedMemory::from_file(&snap_file, layout.snapshot_size())?; - - // The size validation in `open_snapshot_blob` stats the file - // before mapping. Nothing prevents the file from being - // truncated between that stat and the mmap, which would leave - // the mapping shorter than the config claims and make restore - // read past the end. Compare the mapped length against - // `memory_size` to reject a file mutated under us. - if memory.mem_size() as u64 != cfg.memory_size { + // of each blob covers only the data prefix (`data_len`) + // at its layer's GPA. The PT tail sits past that prefix + // in the host mapping. Restore copies the tail of the + // page-table layer into the scratch region. Keeping PT + // tails out of the guest mapping avoids overlap with the + // next layer and with `map_file_cow` regions installed + // immediately after the snapshot in guest PA space. + let scratch_base = hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()); + let blobs = cfg + .layers + .iter() + .zip(data_descs) + .zip(&snap_files) + .map(|((layer, desc), file)| { + Ok(Arc::new(SnapshotBlob::new( + map_snapshot_blob(file, desc, layer.data_len)?, + layer.gpa_start, + layer.page_table_len, + scratch_base, + )?)) + }) + .collect::>>()?; + let page_tables = blobs[cfg.page_table_layer].clone(); + let data_layers = blobs + .into_iter() + .filter(|blob| blob.data_len() > 0) + .collect(); + let memory = Arc::new(SnapshotMemory::new(data_layers, page_tables)?); + // Hash after validating the layers so malformed snapshots fail + // cheaply. + if verify_blobs { + for (desc, file) in data_descs.iter().zip(&mut snap_files) { + verify_blob_file("snapshot", file, &parse_oci_digest(desc.digest())?)?; + } + } + + // The snapshot GPA span must cover at least the regions the + // layout fields describe (code, PEB, heap, init data), + // otherwise the guest mapping is too short to back them. + let required_memory_size = layout.get_memory_size()?; + if memory.gpa_span_len() < required_memory_size { return Err(crate::new_error!( - "mapped snapshot size ({}) does not match config memory_size ({}); the blob may have changed during loading", - memory.mem_size(), - cfg.memory_size + "snapshot GPA span ({}) is smaller than the layout size ({})", + memory.gpa_span_len(), + required_memory_size )); } diff --git a/src/hyperlight_host/src/sandbox/snapshot/file_tests.rs b/src/hyperlight_host/src/sandbox/snapshot/file_tests.rs index 7e801ea92..7b40c45e1 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/file_tests.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/file_tests.rs @@ -13,8 +13,10 @@ use serde_json::Value; use sha2::{Digest as _, Sha256}; use crate::func::Registerable; -use crate::mem::shared_mem::SharedMemory; -use crate::sandbox::snapshot::{OciDigest, OciReference, OciTag, Snapshot}; +use crate::mem::shared_mem::{ReadonlySharedMemory, SharedMemory as _}; +use crate::sandbox::snapshot::{ + OciDigest, OciReference, OciTag, Snapshot, SnapshotBlob, SnapshotMemory, +}; use crate::{GuestBinary, HostFunctions, Sandbox, SandboxBuilder, UninitializedSandbox}; fn create_test_sandbox() -> Sandbox { @@ -106,7 +108,7 @@ fn find_snapshot_blob(oci_dir: &std::path::Path) -> std::path::PathBuf { oci_dir.join("blobs").join("sha256").join(snap_digest) } -/// Locate the transport (layer 1) blob inside `oci_dir`. +/// Locate the transport (last layer) blob inside `oci_dir`. fn find_transport_blob(oci_dir: &std::path::Path) -> std::path::PathBuf { let index: Value = serde_json::from_slice(&std::fs::read(oci_dir.join("index.json")).unwrap()).unwrap(); @@ -117,7 +119,7 @@ fn find_transport_blob(oci_dir: &std::path::Path) -> std::path::PathBuf { .unwrap(); let manifest_path = oci_dir.join("blobs").join("sha256").join(manifest_digest); let manifest: Value = serde_json::from_slice(&std::fs::read(&manifest_path).unwrap()).unwrap(); - let transport_digest = manifest["layers"][1]["digest"] + let transport_digest = manifest["layers"].as_array().unwrap().last().unwrap()["digest"] .as_str() .unwrap() .strip_prefix("sha256:") @@ -241,6 +243,10 @@ fn snapshot_metadata_round_trip() { .unwrap(), Some(metadata) ); + assert_eq!( + loaded.layout().get_guest_code_gva(), + snapshot.layout().get_guest_code_gva() + ); } #[test] @@ -260,6 +266,167 @@ fn snapshot_without_metadata_omits_config_field() { assert_eq!(loaded.metadata::("missing").unwrap(), None); } +#[test] +fn layered_writer_preserves_blobs_and_transport_geometry() { + let snapshot = create_snapshot(); + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("snap"); + snapshot + .save(&path, &OciTag::new("latest").unwrap()) + .unwrap(); + + let manifest: Value = + serde_json::from_slice(&std::fs::read(manifest_path(&path)).unwrap()).unwrap(); + let config: Value = + serde_json::from_slice(&std::fs::read(find_config_blob(&path)).unwrap()).unwrap(); + let layers = manifest["layers"].as_array().unwrap(); + assert_eq!( + manifest["config"]["mediaType"], + "application/vnd.hyperlight.snapshot.config.v4+json" + ); + // `artifactType` mirrors `config.mediaType` so registries that surface + // the distribution-spec referrers API report a useful type, and tooling + // that falls back to `config.mediaType` sees the same value. + assert_eq!( + manifest["artifactType"], + "application/vnd.hyperlight.snapshot.config.v4+json" + ); + assert_eq!(config["abi_version"], 7); + let blobs = snapshot.snapshot_memory().blobs().collect::>(); + assert_eq!(layers.len(), blobs.len() + 1); + for (index, blob) in blobs.iter().enumerate() { + assert_eq!( + layers[index]["mediaType"], + "application/vnd.hyperlight.snapshot.memory.v2" + ); + let bytes = std::fs::read(blob_path(&path, &layers[index])).unwrap(); + assert_eq!(bytes, blob.memory().as_slice()); + assert_eq!(config["layers"][index]["data_len"], blob.data_len()); + assert_eq!( + config["layers"][index]["page_table_len"], + blob.page_table_len() + ); + } + let page_table_layer = config["page_table_layer"].as_u64().unwrap() as usize; + assert!(Arc::ptr_eq( + blobs[page_table_layer], + snapshot.snapshot_memory().page_table_blob() + )); + assert_eq!( + layers[blobs.len()]["mediaType"], + super::file::MT_TRANSPORT_V1 + ); + + let loaded = Snapshot::checked_load(&path, OciTag::new("latest").unwrap()).unwrap(); + assert_eq!( + loaded.snapshot_memory().layers().len(), + snapshot.snapshot_memory().layers().len() + ); + assert_eq!( + loaded.layout().get_g2h_queue_size(), + snapshot.layout().get_g2h_queue_size() + ); + assert_eq!( + loaded.layout().get_h2g_queue_size(), + snapshot.layout().get_h2g_queue_size() + ); + assert_eq!( + loaded.layout().get_g2h_buffer_size(), + snapshot.layout().get_g2h_buffer_size() + ); + assert_eq!( + loaded.layout().get_h2g_buffer_size(), + snapshot.layout().get_h2g_buffer_size() + ); + assert_eq!( + loaded.layout().get_g2h_pool_pages(), + snapshot.layout().get_g2h_pool_pages() + ); + assert_eq!( + loaded.layout().get_h2g_pool_pages(), + snapshot.layout().get_h2g_pool_pages() + ); +} + +#[test] +fn layered_snapshot_recapture_round_trip() { + let mut snapshot = Arc::try_unwrap(create_snapshot()).ok().unwrap(); + let source = &snapshot.state.memory.layers()[0]; + let data = source.memory().as_slice(); + let data_len = source.data_len(); + let page_size = page_size::get(); + let boundary = (data_len / 2 / page_size) * page_size; + assert!(boundary > 0 && boundary < data_len); + let base = source.gpa_start(); + let scratch_base = + hyperlight_common::layout::scratch_base_gpa(snapshot.layout().get_scratch_size()); + let mut layers = [(0, boundary), (boundary, data_len)] + .map(|(start, end)| { + let blob = SnapshotBlob::new( + ReadonlySharedMemory::from_bytes(&data[start..end], end - start).unwrap(), + base + start as u64, + 0, + scratch_base, + ) + .unwrap(); + Arc::new(blob) + }) + .to_vec(); + layers.extend_from_slice(&snapshot.state.memory.layers()[1..]); + let page_tables = + super::memory::tests::page_tables_from_bytes(snapshot.state.memory.page_table_bytes()); + Arc::get_mut(&mut snapshot.state).unwrap().memory = + Arc::new(SnapshotMemory::new(layers.into_boxed_slice(), page_tables).unwrap()); + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("snap"); + snapshot + .save(&path, &OciTag::new("latest").unwrap()) + .unwrap(); + let manifest: Value = + serde_json::from_slice(&std::fs::read(manifest_path(&path)).unwrap()).unwrap(); + let layers = manifest["layers"].as_array().unwrap(); + let data_layers = snapshot.state.memory.layers().len(); + // The data layers, then a blob with only page tables, then transport. + assert_eq!(layers.len(), data_layers + 2); + for layer in &layers[..data_layers + 1] { + assert_eq!(layer["mediaType"], super::file::MT_SNAPSHOT_V2); + } + assert_eq!( + layers[data_layers + 1]["mediaType"], + super::file::MT_TRANSPORT_V1 + ); + let loaded = Snapshot::checked_load(&path, OciTag::new("latest").unwrap()).unwrap(); + assert_eq!(loaded.snapshot_memory().layers().len(), data_layers); + let mut sandbox = + Sandbox::from_snapshot(Arc::new(loaded), HostFunctions::default(), None).unwrap(); + assert_eq!( + sandbox + .call::("Echo", "layers".to_string()) + .unwrap(), + "layers" + ); + let initial = sandbox.call::("GetStatic", ()).unwrap(); + assert_eq!( + sandbox.call::("AddToStatic", 42).unwrap(), + initial + 42 + ); + let captured = sandbox.snapshot().unwrap(); + captured + .save(&path, &OciTag::new("captured").unwrap()) + .unwrap(); + assert_eq!(sandbox.call::("AddToStatic", 1).unwrap(), initial + 43); + + let loaded = Snapshot::checked_load(&path, OciTag::new("captured").unwrap()).unwrap(); + let mut restored = + Sandbox::from_snapshot(Arc::new(loaded), HostFunctions::default(), None).unwrap(); + assert_eq!(restored.call::("GetStatic", ()).unwrap(), initial + 42); + assert_eq!( + restored.call::("AddToStatic", 1).unwrap(), + initial + 43 + ); +} + /// A pre-existing snapshot blob with the right length but wrong /// bytes (corruption, partial copy, foreign tool) must be detected /// and replaced by `save`, not silently trusted. @@ -297,16 +464,19 @@ fn save_self_heals_same_length_wrong_content_snapshot_blob() { #[test] fn snapshot_and_pt_size_round_trip() { let snap = create_snapshot(); - let original_snapshot_size = snap.layout().snapshot_size(); - let original_pt_size = snap.layout().pt_size(); + let original_snapshot_size = snap.snapshot_memory().gpa_span_len(); + let original_pt_size = snap.snapshot_memory().page_table_len(); let dir = tempfile::tempdir().unwrap(); let path = dir.path().join("running"); snap.save(&path, &OciTag::new("latest").unwrap()).unwrap(); let loaded = Snapshot::checked_load(&path, OciTag::new("latest").unwrap()).unwrap(); - assert_eq!(loaded.layout().snapshot_size(), original_snapshot_size); - assert_eq!(loaded.layout().pt_size(), original_pt_size); + assert_eq!( + loaded.snapshot_memory().gpa_span_len(), + original_snapshot_size + ); + assert_eq!(loaded.snapshot_memory().page_table_len(), original_pt_size); } #[test] @@ -539,7 +709,7 @@ fn restore_missing_transport_preserves_target() { // Seed guest state and read the mapped file before caching the snapshot. let file = tempfile::NamedTempFile::new().unwrap(); - std::fs::write(file.path(), vec![0x5a; page_size::get()]).unwrap(); + std::fs::write(file.path(), vec![0x5a; page_size::get() * 2]).unwrap(); let mut target = create_test_sandbox(); target.call::("AddToStatic", 5i32).unwrap(); @@ -568,7 +738,12 @@ fn restore_missing_transport_preserves_target() { assert!(Arc::ptr_eq(target.snapshot.as_ref().unwrap(), &cached)); assert_eq!(target.mem_mgr.snapshot_count, generation); assert_eq!( - target.call::>("ReadMappedBuffer", args).unwrap(), + target + .call::>( + "ReadMappedBuffer", + (guest_base, hyperlight_common::vmem::PAGE_SIZE as u64, false), + ) + .unwrap(), expected ); assert_eq!(target.call::("GetStatic", ()).unwrap(), 5); @@ -590,7 +765,7 @@ fn load_noncanonical_transport_preserves_target() { // Seed guest state and read the mapped file before caching the snapshot. let file = tempfile::NamedTempFile::new().unwrap(); - std::fs::write(file.path(), vec![0x5a; page_size::get()]).unwrap(); + std::fs::write(file.path(), vec![0x5a; page_size::get() * 2]).unwrap(); let mut target = create_test_sandbox(); target.call::("AddToStatic", 5i32).unwrap(); @@ -620,7 +795,12 @@ fn load_noncanonical_transport_preserves_target() { assert_eq!(target.mem_mgr.snapshot_count, generation); assert_eq!( - target.call::>("ReadMappedBuffer", args).unwrap(), + target + .call::>( + "ReadMappedBuffer", + (guest_base, hyperlight_common::vmem::PAGE_SIZE as u64, false), + ) + .unwrap(), expected ); assert_eq!(target.call::("GetStatic", ()).unwrap(), 5); @@ -1168,168 +1348,62 @@ fn snapshot_blob_size_mismatch_rejected() { } #[test] -fn snapshot_layout_snapshot_size_zero_rejected() { - let snapshot = create_snapshot(); - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("snap"); - snapshot - .save(&path, &OciTag::new("latest").unwrap()) - .unwrap(); - rewrite_config(&path, |cfg| { - cfg["layout"]["snapshot_size"] = Value::from(0u64); - }); - let err = unwrap_err_snapshot(Snapshot::checked_load( - &path, - OciTag::new("latest").unwrap(), - )); - let msg = format!("{}", err); - assert!( - msg.contains("snapshot_size"), - "expected snapshot_size error, got: {}", - msg - ); -} - -#[test] -fn snapshot_layout_snapshot_size_unaligned_rejected() { - let snapshot = create_snapshot(); - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("snap"); - snapshot - .save(&path, &OciTag::new("latest").unwrap()) - .unwrap(); - rewrite_config(&path, |cfg| { - let s = cfg["layout"]["snapshot_size"].as_u64().unwrap(); - cfg["layout"]["snapshot_size"] = Value::from(s + 1); - }); - let err = unwrap_err_snapshot(Snapshot::checked_load( - &path, - OciTag::new("latest").unwrap(), - )); - let msg = format!("{}", err); - assert!( - msg.contains("PAGE_SIZE") || msg.contains("multiple"), - "expected page alignment error, got: {}", - msg - ); -} - -#[test] -fn snapshot_layout_snapshot_size_must_match_memory_size() { - let snapshot = create_snapshot(); - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("snap"); - snapshot - .save(&path, &OciTag::new("latest").unwrap()) - .unwrap(); - let page = hyperlight_common::vmem::PAGE_SIZE as u64; - rewrite_config(&path, |cfg| { - let m = cfg["memory_size"].as_u64().unwrap(); - cfg["layout"]["snapshot_size"] = Value::from(m + page); - }); - let err = unwrap_err_snapshot(Snapshot::checked_load( - &path, - OciTag::new("latest").unwrap(), - )); - let msg = format!("{}", err); - assert!( - msg.contains("does not equal memory_size"), - "expected snapshot_size + pt_size != memory_size error, got: {}", - msg - ); -} - -#[test] -fn snapshot_size_smaller_than_layout_rejected() { - // Shrinking `snapshot_size` while growing `pt_size` by the same - // amount preserves `snapshot_size + pt_size == memory_size` and the - // blob length, yet leaves the guest mapping too short to back the - // regions the layout describes. The loader must compare - // `snapshot_size` against the size the layout fields imply. - let snapshot = create_snapshot(); - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("snap"); - snapshot - .save(&path, &OciTag::new("latest").unwrap()) - .unwrap(); - let page = hyperlight_common::vmem::PAGE_SIZE as u64; - // Size the layout fields imply. The guest-visible prefix must - // cover at least this much. - let required = snapshot.layout().get_memory_size().unwrap() as u64; +fn snapshot_layer_gpa_unaligned_rejected() { + let (_dir, path) = save_for_mutation(); rewrite_config(&path, |cfg| { - let mem = cfg["memory_size"].as_u64().unwrap(); - // One page short of the required size, with the page-table - // tail absorbing the rest so `memory_size` (and the blob - // length) stay constant. - let short = required - page; - cfg["layout"]["snapshot_size"] = Value::from(short); - cfg["layout"]["pt_size"] = Value::from(mem - short); - // Grow scratch to cover the larger pt tail so the scratch - // bound is not what trips. - cfg["layout"]["scratch_size"] = Value::from(mem + page); + let gpa = cfg["layers"][0]["gpa_start"].as_u64().unwrap(); + cfg["layers"][0]["gpa_start"] = Value::from(gpa + 1); }); let err = unwrap_err_snapshot(Snapshot::checked_load( &path, OciTag::new("latest").unwrap(), )); - assert_err_contains(err, "is smaller than the layout size"); + assert_err_contains(err, "not aligned"); } #[test] -fn snapshot_layout_pt_size_unaligned_rejected() { - let snapshot = create_snapshot(); - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("snap"); - snapshot - .save(&path, &OciTag::new("latest").unwrap()) - .unwrap(); +fn snapshot_page_table_len_unaligned_rejected() { + let (_dir, path) = save_for_mutation(); rewrite_config(&path, |cfg| { - if let Some(p) = cfg["layout"]["pt_size"].as_u64() { - cfg["layout"]["pt_size"] = Value::from(p + 1); - } else { - cfg["layout"]["pt_size"] = Value::from(1u64); - } + let layer = cfg["page_table_layer"].as_u64().unwrap() as usize; + let len = cfg["layers"][layer]["page_table_len"].as_u64().unwrap(); + cfg["layers"][layer]["page_table_len"] = Value::from(len + 1); }); let err = unwrap_err_snapshot(Snapshot::checked_load( &path, OciTag::new("latest").unwrap(), )); - let msg = format!("{}", err); - assert!( - msg.contains("pt_size") || msg.contains("PAGE_SIZE") || msg.contains("multiple"), - "expected pt_size validation error, got: {}", - msg - ); + assert_err_contains(err, "size"); } #[test] -fn snapshot_layout_missing_pt_size_rejected() { +fn snapshot_missing_page_table_len_rejected() { let (_dir, path) = save_for_mutation(); rewrite_config(&path, |cfg| { - let memory_size = cfg["memory_size"].as_u64().unwrap(); - cfg["layout"]["snapshot_size"] = Value::from(memory_size); - cfg["layout"]["pt_size"] = Value::Null; + cfg["layers"][0] + .as_object_mut() + .unwrap() + .remove("page_table_len"); }); let err = unwrap_err_snapshot(Snapshot::checked_load( &path, OciTag::new("latest").unwrap(), )); - assert_err_contains(err, "pt_size"); + assert_err_contains(err, "page_table_len"); } #[test] -fn snapshot_layout_zero_pt_size_rejected() { +fn snapshot_page_table_len_zero_rejected() { let (_dir, path) = save_for_mutation(); rewrite_config(&path, |cfg| { - let memory_size = cfg["memory_size"].as_u64().unwrap(); - cfg["layout"]["snapshot_size"] = Value::from(memory_size); - cfg["layout"]["pt_size"] = Value::from(0u64); + let layer = cfg["page_table_layer"].as_u64().unwrap() as usize; + cfg["layers"][layer]["page_table_len"] = Value::from(0u64); }); let err = unwrap_err_snapshot(Snapshot::checked_load( &path, OciTag::new("latest").unwrap(), )); - assert_err_contains(err, "pt_size"); + assert_err_contains(err, "does not match its data and page-table lengths"); } #[test] @@ -1613,7 +1687,7 @@ fn save_same_tag_same_content_is_idempotent() { ); } -/// Two tags written from one in-memory snapshot share all four blobs. +/// Two tags written from one in-memory snapshot share all blobs. #[test] fn save_shares_blobs_across_tags_with_identical_content() { let snap = create_snapshot(); @@ -1627,7 +1701,12 @@ fn save_shares_blobs_across_tags_with_identical_content() { .unwrap() .filter_map(|e| e.ok().map(|e| e.file_name())) .collect(); - assert_eq!(blobs.len(), 4, "expected 4 deduped blobs, got {:?}", blobs); + assert_eq!( + blobs.len(), + snap.snapshot_memory().blobs().count() + 3, + "expected deduped blobs, got {:?}", + blobs + ); } /// Replacing one tag in a three-tag layout keeps the other two @@ -1869,6 +1948,218 @@ fn save_for_mutation() -> (tempfile::TempDir, std::path::PathBuf) { (dir, path) } +fn blob_path(path: &std::path::Path, descriptor: &Value) -> std::path::PathBuf { + path.join("blobs").join("sha256").join( + descriptor["digest"] + .as_str() + .unwrap() + .strip_prefix("sha256:") + .unwrap(), + ) +} + +fn write_layered_blob(path: &std::path::Path, bytes: &[u8], descriptor: &mut Value) { + let digest = sha256_hex(bytes); + std::fs::write(path.join("blobs").join("sha256").join(&digest), bytes).unwrap(); + descriptor["digest"] = Value::from(format!("sha256:{digest}")); + descriptor["size"] = Value::from(bytes.len() as u64); +} + +fn make_layered_layout(path: &std::path::Path, split_data: bool) { + if !split_data { + return; + } + let cfg: Value = + serde_json::from_slice(&std::fs::read(find_config_blob(path)).unwrap()).unwrap(); + let image = std::fs::read(find_snapshot_blob(path)).unwrap(); + let first = &cfg["layers"][0]; + let data_len = first["data_len"].as_u64().unwrap() as usize; + let page_size = page_size::get(); + let boundary = (data_len / 2 / page_size) * page_size; + assert!(boundary > 0 && boundary < data_len); + let base = first["gpa_start"].as_u64().unwrap(); + // The second part keeps the page tables that follow the data. + let parts = [ + (&image[..boundary], 0, boundary, 0), + ( + &image[boundary..], + boundary, + data_len - boundary, + first["page_table_len"].as_u64().unwrap(), + ), + ]; + let mut layers = Vec::new(); + let mut data_descriptors = Vec::new(); + let manifest: Value = + serde_json::from_slice(&std::fs::read(manifest_path(path)).unwrap()).unwrap(); + for (bytes, start, part_data_len, page_table_len) in parts { + layers.push(serde_json::json!({ + "gpa_start": base + start as u64, + "data_len": part_data_len, + "page_table_len": page_table_len + })); + let mut descriptor = manifest["layers"][0].clone(); + descriptor["mediaType"] = Value::from(super::file::MT_SNAPSHOT_V2); + write_layered_blob(path, bytes, &mut descriptor); + data_descriptors.push(descriptor); + } + layers.extend_from_slice(&cfg["layers"].as_array().unwrap()[1..]); + data_descriptors.extend_from_slice(&manifest["layers"].as_array().unwrap()[1..]); + // The split puts one more blob before every later blob. The page tables of + // the first blob move to its second part. + let page_table_layer = match cfg["page_table_layer"].as_u64().unwrap() { + 0 => 1, + index => index + 1, + }; + + rewrite_config(path, |cfg| { + cfg["layers"] = Value::Array(layers); + cfg["page_table_layer"] = Value::from(page_table_layer); + }); + rewrite_manifest(path, |manifest| { + manifest["layers"] = Value::Array(data_descriptors); + }); +} + +#[test] +fn layered_oci_load_restores_data_page_tables_and_transport() { + let (_dir, path) = save_for_mutation(); + let config: Value = + serde_json::from_slice(&std::fs::read(find_config_blob(&path)).unwrap()).unwrap(); + let data_layers = config["layers"] + .as_array() + .unwrap() + .iter() + .filter(|layer| layer["data_len"].as_u64().unwrap() > 0) + .count(); + make_layered_layout(&path, true); + let loaded = Snapshot::checked_load(&path, OciTag::new("latest").unwrap()).unwrap(); + assert_eq!(loaded.snapshot_memory().layers().len(), data_layers + 1); + let mut sandbox = + Sandbox::from_snapshot(Arc::new(loaded), HostFunctions::default(), None).unwrap(); + let result: String = sandbox.call("Echo", "layered".to_string()).unwrap(); + assert_eq!(result, "layered"); +} + +#[test] +fn layered_oci_rejects_missing_descriptor() { + let (_dir, path) = save_for_mutation(); + make_layered_layout(&path, false); + rewrite_manifest(&path, |manifest| { + manifest["layers"].as_array_mut().unwrap().remove(0); + }); + let err = unwrap_err_snapshot(Snapshot::checked_load( + &path, + OciTag::new("latest").unwrap(), + )); + assert_err_contains(err, "snapshot layers and one transport layer"); +} + +#[test] +fn layered_oci_rejects_wrong_descriptor_media() { + for (media_type, expected_error) in [ + ( + super::file::MT_SNAPSHOT_V2, + "unexpected snapshot layer media type", + ), + ( + super::file::MT_TRANSPORT_V1, + "unexpected transport layer media type", + ), + ] { + let (_dir, path) = save_for_mutation(); + rewrite_manifest(&path, |manifest| { + let descriptor = manifest["layers"] + .as_array_mut() + .unwrap() + .iter_mut() + .find(|descriptor| descriptor["mediaType"] == media_type) + .unwrap(); + descriptor["mediaType"] = Value::from("application/vnd.example.other"); + }); + let err = unwrap_err_snapshot(Snapshot::checked_load( + &path, + OciTag::new("latest").unwrap(), + )); + assert_err_contains(err, expected_error); + } +} + +#[test] +fn layered_oci_rejects_mismatched_storage_size_before_opening() { + let (_dir, path) = save_for_mutation(); + rewrite_manifest(&path, |manifest| { + manifest["layers"][0]["size"] = Value::from(u64::MAX / 2); + }); + let err = unwrap_err_snapshot(Snapshot::checked_load( + &path, + OciTag::new("latest").unwrap(), + )); + assert_err_contains(err, "does not match its data and page-table lengths"); +} + +#[test] +fn layered_oci_rejects_overlapping_data_layers() { + let (_dir, path) = save_for_mutation(); + make_layered_layout(&path, true); + rewrite_config(&path, |cfg| { + cfg["layers"][1]["gpa_start"] = cfg["layers"][0]["gpa_start"].clone(); + }); + let err = unwrap_err_snapshot(Snapshot::checked_load( + &path, + OciTag::new("latest").unwrap(), + )); + assert_err_contains(err, "overlap"); +} + +#[test] +fn layered_oci_rejects_too_many_data_layers() { + let (_dir, path) = save_for_mutation(); + rewrite_config(&path, |cfg| { + let layer = cfg["layers"][0].clone(); + cfg["layers"] = Value::from(vec![layer; super::memory::MAX_SNAPSHOT_LAYERS + 2]); + }); + let err = unwrap_err_snapshot(Snapshot::checked_load( + &path, + OciTag::new("latest").unwrap(), + )); + assert_err_contains(err, "data layers, limit is"); +} + +#[test] +fn layered_oci_rejects_data_beyond_scratch_before_opening() { + let (_dir, path) = save_for_mutation(); + let data_len = 1u64 << 40; + let mut page_table_len = 0; + rewrite_config(&path, |cfg| { + cfg["layers"][0]["data_len"] = Value::from(data_len); + page_table_len = cfg["layers"][0]["page_table_len"].as_u64().unwrap(); + }); + rewrite_manifest(&path, |manifest| { + manifest["layers"][0]["size"] = + Value::from((data_len + page_table_len).next_multiple_of(page_size::get() as u64)); + }); + let err = unwrap_err_snapshot(Snapshot::checked_load( + &path, + OciTag::new("latest").unwrap(), + )); + assert_err_contains(err, "snapshot data exceeds"); +} + +#[test] +fn layered_oci_rejects_mismatched_artifact_type() { + let (_dir, path) = save_for_mutation(); + make_layered_layout(&path, false); + rewrite_manifest(&path, |manifest| { + manifest["artifactType"] = Value::from("application/vnd.example.other"); + }); + let err = unwrap_err_snapshot(Snapshot::checked_load( + &path, + OciTag::new("latest").unwrap(), + )); + assert_err_contains(err, "does not match config media type"); +} + fn assert_err_contains(err: crate::HyperlightError, needle: &str) { let msg = format!("{}", err); assert!( @@ -2082,6 +2373,7 @@ fn legacy_config_versions_rejected() { for media_type in [ "application/vnd.hyperlight.snapshot.config.v1+json", "application/vnd.hyperlight.snapshot.config.v2+json", + "application/vnd.hyperlight.snapshot.config.v3+json", ] { let (_dir, path) = save_for_mutation(); rewrite_manifest(&path, |m| { @@ -2091,7 +2383,7 @@ fn legacy_config_versions_rejected() { &path, OciTag::new("latest").unwrap(), )); - assert_err_contains(err, "incompatible with snapshot ABI 6"); + assert_err_contains(err, "incompatible with snapshot ABI 7"); } } @@ -2139,7 +2431,8 @@ fn unknown_snapshot_layer_media_type_rejected() { fn unknown_transport_layer_media_type_rejected() { let (_dir, path) = save_for_mutation(); rewrite_manifest(&path, |m| { - m["layers"][1]["mediaType"] = Value::from("application/vnd.example.unknown.v1"); + let descriptor = m["layers"].as_array_mut().unwrap().last_mut().unwrap(); + descriptor["mediaType"] = Value::from("application/vnd.example.unknown.v1"); }); let err = unwrap_err_snapshot(Snapshot::checked_load( &path, @@ -2218,8 +2511,13 @@ fn config_blob_size_descriptor_mismatch_rejected() { fn transport_blob_size_descriptor_mismatch_rejected() { let (_dir, path) = save_for_mutation(); rewrite_manifest(&path, |manifest| { - let size = manifest["layers"][1]["size"].as_u64().unwrap(); - manifest["layers"][1]["size"] = Value::from(size + 1); + let descriptor = manifest["layers"] + .as_array_mut() + .unwrap() + .last_mut() + .unwrap(); + let size = descriptor["size"].as_u64().unwrap(); + descriptor["size"] = Value::from(size + 1); }); let error = unwrap_err_snapshot(Snapshot::load(&path, OciTag::new("latest").unwrap())); @@ -2243,39 +2541,6 @@ fn malformed_config_json_rejected() { assert_err_contains(err, "config JSON"); } -#[test] -fn memory_size_zero_rejected() { - let (_dir, path) = save_for_mutation(); - rewrite_config(&path, |cfg| { - cfg["memory_size"] = Value::from(0u64); - }); - let err = unwrap_err_snapshot(Snapshot::checked_load( - &path, - OciTag::new("latest").unwrap(), - )); - assert_err_contains(err, "memory_size"); -} - -#[test] -fn memory_size_unaligned_rejected() { - let (_dir, path) = save_for_mutation(); - rewrite_config(&path, |cfg| { - let sz = cfg["memory_size"].as_u64().unwrap(); - cfg["memory_size"] = Value::from(sz + 1); - }); - let err = unwrap_err_snapshot(Snapshot::checked_load( - &path, - OciTag::new("latest").unwrap(), - )); - let msg = format!("{}", err); - // Either the page-alignment check or the file-size check trips. - assert!( - msg.contains("memory_size") || msg.contains("PAGE_SIZE") || msg.contains("size"), - "expected memory_size rejection, got: {}", - msg - ); -} - #[test] fn bad_init_data_permissions_rejected() { let (_dir, path) = save_for_mutation(); @@ -2588,36 +2853,6 @@ fn manifest_descriptor_non_image_manifest_rejected() { ); } -#[test] -fn manifest_uses_correct_config_and_layer_media_types() { - let snap = create_snapshot(); - let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("layout"); - snap.save(&path, &OciTag::new("latest").unwrap()).unwrap(); - let manifest: Value = - serde_json::from_slice(&std::fs::read(manifest_path(&path)).unwrap()).unwrap(); - assert_eq!( - manifest["config"]["mediaType"].as_str().unwrap(), - "application/vnd.hyperlight.snapshot.config.v3+json" - ); - assert_eq!(manifest["layers"].as_array().unwrap().len(), 2); - assert_eq!( - manifest["layers"][0]["mediaType"].as_str().unwrap(), - "application/vnd.hyperlight.snapshot.memory.v1" - ); - assert_eq!( - manifest["layers"][1]["mediaType"].as_str().unwrap(), - "application/vnd.hyperlight.snapshot.transport.v1" - ); - // `artifactType` mirrors `config.mediaType` so registries that surface - // the distribution-spec referrers API report a useful type, and tooling - // that falls back to `config.mediaType` sees the same value. - assert_eq!( - manifest["artifactType"].as_str().unwrap(), - "application/vnd.hyperlight.snapshot.config.v3+json" - ); -} - #[test] fn manifest_missing_artifact_type_rejected() { let snap = create_snapshot(); @@ -2811,7 +3046,12 @@ fn transport_blob_at_size_limit_reaches_decoder() { let bytes = vec![0; 2 * 1024 * 1024]; std::fs::write(transport_path, &bytes).unwrap(); rewrite_manifest(&path, |manifest| { - manifest["layers"][1]["size"] = Value::from(bytes.len() as u64); + let descriptor = manifest["layers"] + .as_array_mut() + .unwrap() + .last_mut() + .unwrap(); + descriptor["size"] = Value::from(bytes.len() as u64); }); let error = unwrap_err_snapshot(Snapshot::load(&path, OciTag::new("latest").unwrap())); @@ -2825,7 +3065,12 @@ fn transport_blob_too_large_rejected() { let bytes = vec![0; 2 * 1024 * 1024 + 1]; std::fs::write(transport_path, &bytes).unwrap(); rewrite_manifest(&path, |manifest| { - manifest["layers"][1]["size"] = Value::from(bytes.len() as u64); + let descriptor = manifest["layers"] + .as_array_mut() + .unwrap() + .last_mut() + .unwrap(); + descriptor["size"] = Value::from(bytes.len() as u64); }); let error = unwrap_err_snapshot(Snapshot::load(&path, OciTag::new("latest").unwrap())); @@ -2922,20 +3167,6 @@ fn config_blob_too_large_on_write_rejected() { assert_err_contains(err, "config blob"); } -#[test] -fn memory_size_too_large_rejected() { - let (_dir, path) = save_for_mutation(); - rewrite_config(&path, |cfg| { - // 16 GiB exceeds MAX_MEMORY_SIZE. - cfg["memory_size"] = Value::from(16u64 * 1024 * 1024 * 1024); - }); - let err = unwrap_err_snapshot(Snapshot::checked_load( - &path, - OciTag::new("latest").unwrap(), - )); - assert_err_contains(err, "memory_size"); -} - #[test] fn layout_field_too_large_rejected() { let (_dir, path) = save_for_mutation(); @@ -3065,21 +3296,18 @@ fn save_replaces_symlink_snapshot_blob_with_regular_file() { assert_eq!(result, "hello\n"); } -/// A snapshot descriptor claiming a size different from the blob file is -/// rejected before mmap. +/// A blob file whose size differs from its descriptor is rejected before +/// mmap. #[test] fn snapshot_descriptor_size_disagrees_with_file_rejected() { let (_dir, path) = save_for_mutation(); - rewrite_manifest(&path, |m| { - let sz = m["layers"][0]["size"].as_u64().unwrap(); - m["layers"][0]["size"] = Value::from(sz + 1); - }); + let mut blob = std::fs::OpenOptions::new() + .append(true) + .open(find_snapshot_blob(&path)) + .unwrap(); + std::io::Write::write_all(&mut blob, &[0]).unwrap(); let err = unwrap_err_snapshot(Snapshot::load(&path, OciTag::new("latest").unwrap())); - let msg = format!("{err}"); - assert!( - msg.contains("snapshot blob size"), - "expected snapshot-blob descriptor disagreement error, got: {msg}" - ); + assert_err_contains(err, "disagrees with OCI descriptor size"); } // `load` runs every non-digest validator. The unverified @@ -3513,7 +3741,10 @@ fn save_new_tag_into_loaded_layout_preserves_live_mapping() { // Record the full mapped image and every on-disk blob before the // second save, so any byte change is caught. - let mapping_before = loaded_a.memory().as_slice().to_vec(); + let mapping_before = loaded_a.state.memory.layers()[0] + .memory() + .as_slice() + .to_vec(); let blobs_dir = path.join("blobs").join("sha256"); let blobs_before = read_blob_dir(&blobs_dir); @@ -3526,7 +3757,7 @@ fn save_new_tag_into_loaded_layout_preserves_live_mapping() { // The live mapping is unchanged, byte for byte. assert_eq!( - loaded_a.memory().as_slice(), + loaded_a.state.memory.layers()[0].memory().as_slice(), mapping_before.as_slice(), "live snapshot mapping changed after a new tag was written" ); diff --git a/src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs b/src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs new file mode 100644 index 000000000..68be2c1bc --- /dev/null +++ b/src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright 2026 The Hyperlight Authors. + +#[cfg(not(unshared_snapshot_mem))] +use std::marker::PhantomData; +use std::sync::Arc; + +use super::{SnapshotBlob, SnapshotMemory}; +use crate::mem::memory_region::{MemoryRegion, MemoryRegionType}; +use crate::mem::shared_mem::{ + ExclusiveSharedMemory, GuestSharedMemory, HostSharedMemory, SharedMemory, +}; +use crate::{Result, new_error}; + +/// Memory for one snapshot with one or more layers. +/// Standard builds use each layer's read-only blob. GDB builds use one writable +/// copy of each layer's data. +pub(crate) struct SnapshotMemoryBacking { + memory: Arc, + /// One writable copy for each layer in `memory`. + #[cfg(unshared_snapshot_mem)] + backings: Box<[S]>, + #[cfg(not(unshared_snapshot_mem))] + _phase: PhantomData S>, +} + +impl Clone for SnapshotMemoryBacking { + fn clone(&self) -> Self { + Self { + memory: self.memory.clone(), + #[cfg(unshared_snapshot_mem)] + backings: self.backings.clone(), + #[cfg(not(unshared_snapshot_mem))] + _phase: PhantomData, + } + } +} + +impl SnapshotMemoryBacking { + pub(crate) fn from_snapshot(memory: Arc) -> Result { + #[cfg(unshared_snapshot_mem)] + let backings = memory + .layers() + .iter() + .map(|layer| layer.memory().copy_to_writable()) + .collect::, _>>()? + .into_boxed_slice(); + Ok(Self { + memory, + #[cfg(unshared_snapshot_mem)] + backings, + #[cfg(not(unshared_snapshot_mem))] + _phase: PhantomData, + }) + } + + pub(crate) fn build( + self, + ) -> ( + SnapshotMemoryBacking, + SnapshotMemoryBacking, + ) { + #[cfg(unshared_snapshot_mem)] + let (host_backings, guest_backings) = self + .backings + .into_vec() + .into_iter() + .map(ExclusiveSharedMemory::build) + .unzip::<_, _, Vec<_>, Vec<_>>(); + let memory = self.memory; + ( + SnapshotMemoryBacking { + memory: memory.clone(), + #[cfg(unshared_snapshot_mem)] + backings: host_backings.into_boxed_slice(), + #[cfg(not(unshared_snapshot_mem))] + _phase: PhantomData, + }, + SnapshotMemoryBacking { + memory, + #[cfg(unshared_snapshot_mem)] + backings: guest_backings.into_boxed_slice(), + #[cfg(not(unshared_snapshot_mem))] + _phase: PhantomData, + }, + ) + } +} + +impl SnapshotMemoryBacking { + #[cfg(all(test, not(unshared_snapshot_mem)))] + pub(crate) fn layers(&self) -> &[Arc] { + self.memory.layers() + } + + pub(crate) fn resolve(&self, gpa: u64, len: usize) -> Option<(usize, usize)> { + self.memory.resolve(gpa, len) + } + + pub(crate) fn gpa_span_len(&self) -> usize { + self.memory.gpa_span_len() + } + + pub(crate) fn page_table_len(&self) -> usize { + self.memory.page_table_len() + } +} + +impl SnapshotMemoryBacking { + /// One mapping per layer, covering its whole data. + pub(crate) fn mappings(&self) -> Vec { + #[cfg(not(unshared_snapshot_mem))] + let backings = self.memory.layers().iter().map(|layer| layer.memory()); + #[cfg(unshared_snapshot_mem)] + let backings = self.backings.iter(); + self.memory + .layers() + .iter() + .zip(backings) + .map(|(layer, backing)| { + backing.mapping_at(layer.gpa_start(), MemoryRegionType::Snapshot) + }) + .collect() + } +} + +impl SnapshotMemoryBacking { + pub(crate) fn reusable_layers(&self) -> Option<&[Arc]> { + #[cfg(not(unshared_snapshot_mem))] + { + Some(self.memory.layers()) + } + #[cfg(unshared_snapshot_mem)] + { + None + } + } + + pub(crate) fn copy_layer_to_slice( + &self, + layer_index: usize, + slice: &mut [u8], + offset: usize, + ) -> Result<()> { + #[cfg(not(unshared_snapshot_mem))] + let backing = self + .memory + .layers() + .get(layer_index) + .map(|layer| layer.memory()); + #[cfg(unshared_snapshot_mem)] + let backing = self.backings.get(layer_index); + backing + .ok_or_else(|| new_error!("snapshot layer index is out of bounds"))? + .copy_to_slice(slice, offset) + .map_err(Into::into) + } + + pub(crate) fn read_snapshot_gpa(&self, gpa: u64, slice: &mut [u8]) -> Result<()> { + let mut copied = 0usize; + while copied < slice.len() { + let current_gpa = gpa + .checked_add(u64::try_from(copied)?) + .ok_or_else(|| new_error!("snapshot GPA range overflows"))?; + let (layer_index, offset, available) = self + .memory + .resolve_chunk(current_gpa) + .ok_or_else(|| new_error!("snapshot GPA range is not mapped: {current_gpa:#x}"))?; + let chunk_len = available.min(slice.len() - copied); + self.copy_layer_to_slice(layer_index, &mut slice[copied..copied + chunk_len], offset)?; + copied += chunk_len; + } + Ok(()) + } + + /// Writes into a snapshot layer resolved by [`Self::resolve`]. + #[cfg(gdb)] + pub(crate) fn write_layer( + &self, + layer_index: usize, + offset: usize, + slice: &[u8], + ) -> Result<()> { + self.backings + .get(layer_index) + .ok_or_else(|| new_error!("snapshot layer index is out of bounds"))? + .copy_from_slice(slice, offset) + .map_err(Into::into) + } + + pub(crate) fn page_table_bytes(&self) -> &[u8] { + self.memory.page_table_bytes() + } + + #[cfg(crashdump)] + pub(crate) fn host_range(&self, layer_index: usize) -> Result<(usize, usize)> { + #[cfg(not(unshared_snapshot_mem))] + let backing = self + .memory + .layers() + .get(layer_index) + .map(|layer| layer.memory()); + #[cfg(unshared_snapshot_mem)] + let backing = self.backings.get(layer_index); + let backing = backing.ok_or_else(|| new_error!("snapshot layer index is out of bounds"))?; + Ok((backing.base_addr(), backing.mem_size())) + } +} diff --git a/src/hyperlight_host/src/sandbox/snapshot/memory/mod.rs b/src/hyperlight_host/src/sandbox/snapshot/memory/mod.rs new file mode 100644 index 000000000..8aea74eca --- /dev/null +++ b/src/hyperlight_host/src/sandbox/snapshot/memory/mod.rs @@ -0,0 +1,471 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright 2026 The Hyperlight Authors. + +//! Snapshot memory model: +//! +//! ```text +//! Snapshot +//! `-- memory: Arc +//! |-- layers: Box<[Arc]> +//! | |-- SnapshotBlob 0 +//! | | |-- memory: ReadonlySharedMemory [data | page tables] +//! | | |-- gpa_start: u64 +//! | | `-- page_table_len: usize +//! | `-- SnapshotBlob 1 +//! | `-- ... +//! `-- page_tables: Arc +//! ``` + +use std::ops::Range; +use std::sync::Arc; + +use hyperlight_common::vmem::PAGE_SIZE; + +use crate::mem::layout::SandboxMemoryLayout; +use crate::mem::shared_mem::{ReadonlySharedMemory, SharedMemory}; +use crate::{Result, new_error}; + +mod backing; +pub(crate) use backing::SnapshotMemoryBacking; + +// Arbitrary cap on the layers of one snapshot. Each layer is one mapping, which +// costs a map/unmap when a sandbox switches snapshots, so this bounds restore +// latency. +pub(crate) const MAX_SNAPSHOT_LAYERS: usize = 30; + +/// Memory state assembled from immutable snapshot layers. +/// +/// Layer ranges never overlap. +#[derive(Debug)] +pub(crate) struct SnapshotMemory { + /// Blobs with guest data, sorted by start GPA, which lookups binary + /// search. + layers: Box<[Arc]>, + /// The blob whose page tables this snapshot restores into scratch memory. + /// It is one of `layers` when it also holds guest data. + page_tables: Arc, +} + +/// Immutable memory one snapshot creates: guest data followed by the page +/// tables of that snapshot. The VM maps the data whole at `gpa_start`, +/// including pages a later layer replaced. The page tables are never mapped. +/// Snapshots that reuse the data share the blob. +#[derive(Debug)] +pub(crate) struct SnapshotBlob { + /// Its `guest_mapped_size` is the data length. + memory: ReadonlySharedMemory, + gpa_start: u64, + page_table_len: usize, +} + +impl SnapshotBlob { + /// `gpa_start` is unused when the blob has no data. + pub(crate) fn new( + memory: ReadonlySharedMemory, + gpa_start: u64, + page_table_len: usize, + scratch_base_gpa: u64, + ) -> Result { + let len = memory.mem_size(); + let data_len = memory.guest_mapped_size(); + let expected_len = data_len + .checked_add(page_table_len) + .and_then(|len| len.checked_next_multiple_of(host_page_size())); + if !is_page_aligned(data_len as u64) + || !page_table_len.is_multiple_of(PAGE_SIZE) + || len == 0 + || expected_len != Some(len) + { + return Err(new_error!("snapshot layer size is invalid")); + } + if data_len > 0 { + if !is_page_aligned(gpa_start) { + return Err(new_error!("snapshot layer GPA is not aligned")); + } + if gpa_start < SandboxMemoryLayout::BASE_ADDRESS as u64 { + return Err(new_error!("snapshot layer starts below BASE_ADDRESS")); + } + let gpa_end = gpa_start + .checked_add(data_len as u64) + .ok_or_else(|| new_error!("snapshot layer GPA range overflows"))?; + if gpa_end > scratch_base_gpa { + return Err(new_error!("snapshot layer data overlaps scratch memory")); + } + } + Ok(Self { + memory, + gpa_start, + page_table_len, + }) + } + + pub(crate) fn memory(&self) -> &ReadonlySharedMemory { + &self.memory + } + + pub(crate) fn gpa_start(&self) -> u64 { + self.gpa_start + } + + /// Length of the guest data, which the VM maps. + pub(crate) fn data_len(&self) -> usize { + self.memory.guest_mapped_size() + } + + pub(crate) fn page_table_len(&self) -> usize { + self.page_table_len + } + + pub(crate) fn gpa_range(&self) -> Range { + self.gpa_start..self.gpa_start + self.data_len() as u64 + } + + fn page_table_bytes(&self) -> &[u8] { + let data_len = self.data_len(); + &self.memory.as_slice()[data_len..data_len + self.page_table_len] + } +} + +impl SnapshotMemory { + pub(crate) fn new( + mut layers: Box<[Arc]>, + page_tables: Arc, + ) -> Result { + if layers.is_empty() { + return Err(new_error!("snapshot has no layers")); + } + if layers.len() > MAX_SNAPSHOT_LAYERS { + return Err(new_error!( + "snapshot layer count {} exceeds {}", + layers.len(), + MAX_SNAPSHOT_LAYERS + )); + } + if layers.iter().any(|layer| layer.data_len() == 0) { + return Err(new_error!("snapshot layer has no data")); + } + layers.sort_by_key(|layer| layer.gpa_start); + // Layers sit between BASE_ADDRESS and scratch, so disjoint layers also + // bound the total layer size. Overlap also rejects a blob referenced + // twice. + if layers + .windows(2) + .any(|pair| pair[0].gpa_range().end > pair[1].gpa_start) + { + return Err(new_error!("snapshot blob GPA ranges overlap")); + } + if page_tables.page_table_len == 0 { + return Err(new_error!("snapshot page tables are empty")); + } + if page_tables.data_len() > 0 + && !layers.iter().any(|layer| Arc::ptr_eq(layer, &page_tables)) + { + return Err(new_error!("snapshot page-table blob data is not a layer")); + } + + Ok(Self { + layers, + page_tables, + }) + } + + /// Builds a one-layer snapshot from `memory`, which holds guest data + /// followed by `page_table_len` bytes of page tables. + pub(crate) fn from_flat( + memory: ReadonlySharedMemory, + gpa_start: u64, + page_table_len: usize, + scratch_base_gpa: u64, + ) -> Result { + let blob = Arc::new(SnapshotBlob::new( + memory, + gpa_start, + page_table_len, + scratch_base_gpa, + )?); + Self::new(Box::new([blob.clone()]), blob) + } + + pub(crate) fn layers(&self) -> &[Arc] { + &self.layers + } + + /// Every blob this snapshot holds: the layers, then the page-table blob + /// when it holds no data. + pub(crate) fn blobs(&self) -> impl Iterator> { + let page_table_only = (self.page_tables.data_len() == 0).then_some(&self.page_tables); + self.layers.iter().chain(page_table_only) + } + + pub(crate) fn page_table_blob(&self) -> &Arc { + &self.page_tables + } + + pub(crate) fn page_table_bytes(&self) -> &[u8] { + self.page_tables.page_table_bytes() + } + + pub(crate) fn gpa_span_len(&self) -> usize { + // Sorted, disjoint layers put the highest end in the last one. + self.layers.last().map_or(0, |last| { + // Blobs end below the scratch base, so the offset fits in usize. + (last.gpa_start - SandboxMemoryLayout::BASE_ADDRESS as u64) as usize + last.data_len() + }) + } + + pub(crate) fn page_table_len(&self) -> usize { + self.page_tables.page_table_len + } + + pub(crate) fn resolve(&self, gpa: u64, len: usize) -> Option<(usize, usize)> { + let (index, offset, available) = self.resolve_chunk(gpa)?; + (len <= available).then_some((index, offset)) + } + + /// Returns the layer index, the offset into its blob, and the data bytes + /// left in the blob from `gpa`. + pub(crate) fn resolve_chunk(&self, gpa: u64) -> Option<(usize, usize, usize)> { + let index = self + .layers + .partition_point(|layer| layer.gpa_start <= gpa) + .checked_sub(1)?; + let layer = &self.layers[index]; + let offset = usize::try_from(gpa - layer.gpa_start).ok()?; + let data_len = layer.data_len(); + (offset < data_len).then(|| (index, offset, data_len - offset)) + } + + pub(crate) fn read_page_tables( + &self, + pt_gpa_base: u64, + gpa: u64, + destination: &mut [u8], + ) -> Result<()> { + let source = gpa + .checked_sub(pt_gpa_base) + .and_then(|offset| usize::try_from(offset).ok()) + .and_then(|offset| { + self.page_table_bytes() + .get(offset..offset.checked_add(destination.len())?) + }) + .ok_or_else(|| new_error!("page-table read range is out of bounds"))?; + destination.copy_from_slice(source); + Ok(()) + } +} + +pub(crate) fn host_page_size() -> usize { + PAGE_SIZE.max(page_size::get()) +} + +/// Guest and host pages are powers of two, so the larger one implies both. +fn is_page_aligned(value: u64) -> bool { + value.is_multiple_of(host_page_size() as u64) +} + +#[cfg(test)] +pub(crate) mod tests { + use super::*; + use crate::mem::shared_mem::ExclusiveSharedMemory; + + const SCRATCH_BASE_GPA: u64 = 0x1_0000_0000; + + /// Copies guest data and page tables into one blob. The data is padded to a + /// host page. + pub(crate) fn blob_bytes(data: &[u8], page_tables: &[u8]) -> Result { + let data_len = data.len().next_multiple_of(host_page_size()); + let mut bytes = vec![0u8; data_len + page_tables.len()]; + bytes[..data.len()].copy_from_slice(data); + bytes[data_len..].copy_from_slice(page_tables); + Ok(ReadonlySharedMemory::from_bytes(&bytes, data_len)?) + } + + pub(crate) fn flat_memory( + data: &[u8], + page_tables: &[u8], + gpa_start: u64, + scratch_base_gpa: u64, + ) -> SnapshotMemory { + SnapshotMemory::from_flat( + blob_bytes(data, page_tables).unwrap(), + gpa_start, + page_tables.len(), + scratch_base_gpa, + ) + .unwrap() + } + + pub(crate) fn page_tables_from_bytes(bytes: &[u8]) -> Arc { + Arc::new(SnapshotBlob::new(blob_bytes(&[], bytes).unwrap(), 0, bytes.len(), 0).unwrap()) + } + + pub(crate) fn stub_page_tables() -> Arc { + page_tables_from_bytes(&vec![0u8; PAGE_SIZE]) + } + + fn blob(data_pages: usize, gpa_start: u64) -> Arc { + let data_len = data_pages * page_size::get(); + let storage = ExclusiveSharedMemory::new(data_len) + .unwrap() + .freeze(data_len) + .unwrap(); + Arc::new(SnapshotBlob::new(storage, gpa_start, 0, SCRATCH_BASE_GPA).unwrap()) + } + + #[test] + fn blob_rejects_overflowing_gpa_range() { + let host_page_size = page_size::get(); + let storage = ExclusiveSharedMemory::new(host_page_size) + .unwrap() + .freeze(host_page_size) + .unwrap(); + let gpa_start = u64::MAX - host_page_size as u64 + 1; + + assert!(SnapshotBlob::new(storage, gpa_start, 0, u64::MAX).is_err()); + } + + #[test] + fn blob_rejects_sizes_that_do_not_fill_it() { + let host_page_size = page_size::get(); + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let storage = |data_len| { + ExclusiveSharedMemory::new(2 * host_page_size) + .unwrap() + .freeze(data_len) + .unwrap() + }; + + assert!(SnapshotBlob::new(storage(host_page_size), base, 0, SCRATCH_BASE_GPA).is_err()); + assert!(SnapshotBlob::new(storage(0), base, 0, SCRATCH_BASE_GPA).is_err()); + assert!(SnapshotBlob::new(storage(host_page_size), base, 1, SCRATCH_BASE_GPA).is_err()); + SnapshotBlob::new( + storage(host_page_size), + base, + host_page_size, + SCRATCH_BASE_GPA, + ) + .unwrap(); + } + + #[test] + fn page_tables_follow_the_data_and_stay_unmapped() { + let host_page_size = page_size::get(); + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let memory = SnapshotMemory::from_flat( + blob_bytes(&vec![0x11; host_page_size], &vec![0x22; PAGE_SIZE]).unwrap(), + base, + PAGE_SIZE, + SCRATCH_BASE_GPA, + ) + .unwrap(); + + assert_eq!(memory.page_table_bytes(), vec![0x22; PAGE_SIZE]); + assert_eq!(memory.gpa_span_len(), host_page_size); + assert_eq!(memory.resolve(base + host_page_size as u64, 1), None); + assert_eq!(memory.blobs().count(), 1); + } + + #[test] + fn memory_rejects_page_table_blob_data_outside_the_layers() { + let host_page_size = page_size::get(); + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let page_tables = Arc::new( + SnapshotBlob::new( + blob_bytes(&vec![0; host_page_size], &vec![0; PAGE_SIZE]).unwrap(), + base + host_page_size as u64, + PAGE_SIZE, + SCRATCH_BASE_GPA, + ) + .unwrap(), + ); + + assert!(SnapshotMemory::new(Box::new([blob(1, base)]), page_tables.clone()).is_err()); + SnapshotMemory::new(Box::new([blob(1, base), page_tables.clone()]), page_tables).unwrap(); + } + + #[test] + fn reads_page_tables_from_page_table_only_blob() { + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let page_tables = page_tables_from_bytes(&vec![0x33; PAGE_SIZE]); + let memory = SnapshotMemory::new(Box::new([blob(1, base)]), page_tables).unwrap(); + let mut byte = [0u8; 1]; + + memory.read_page_tables(0x8000, 0x8fff, &mut byte).unwrap(); + + assert_eq!(byte, [0x33]); + assert_eq!(memory.resolve(base, PAGE_SIZE), Some((0, 0))); + } + + #[test] + fn memory_rejects_overlapping_blob_gpas() { + let host_page_size = page_size::get(); + let gpa = SandboxMemoryLayout::BASE_ADDRESS as u64; + let layers = Box::new([blob(2, gpa), blob(1, gpa + host_page_size as u64)]); + + assert!(SnapshotMemory::new(layers, stub_page_tables()).is_err()); + } + + #[test] + fn memory_canonicalizes_layers() { + let host_page_size = page_size::get(); + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let layers = Box::new([blob(1, base + host_page_size as u64), blob(1, base)]); + + let memory = SnapshotMemory::new(layers, stub_page_tables()).unwrap(); + + assert_eq!(memory.layers()[0].gpa_start(), base); + } + + #[test] + fn resolve_covers_whole_blob_and_rejects_past_end() { + let host_page_size = page_size::get(); + let gpa = SandboxMemoryLayout::BASE_ADDRESS as u64; + let memory = SnapshotMemory::new(Box::new([blob(3, gpa)]), stub_page_tables()).unwrap(); + + assert_eq!(memory.resolve(gpa, 1), Some((0, 0))); + assert_eq!( + memory.resolve(gpa + host_page_size as u64, host_page_size), + Some((0, host_page_size)) + ); + assert_eq!( + memory.resolve(gpa + (3 * host_page_size - 1) as u64, 1), + Some((0, 3 * host_page_size - 1)) + ); + assert_eq!( + memory.resolve(gpa + (3 * host_page_size - 1) as u64, 2), + None + ); + assert_eq!(memory.resolve(gpa - 1, 1), None); + } + + #[test] + fn reads_reject_addresses_past_the_blob() { + let host_page_size = page_size::get(); + let gpa = SandboxMemoryLayout::BASE_ADDRESS as u64; + let memory = SnapshotMemory::new(Box::new([blob(3, gpa)]), stub_page_tables()).unwrap(); + let (backing, _) = SnapshotMemoryBacking::from_snapshot(Arc::new(memory)) + .unwrap() + .build(); + let mut byte = [0u8; 1]; + + backing + .read_snapshot_gpa(gpa + host_page_size as u64 + 1, &mut byte) + .unwrap(); + assert!( + backing + .read_snapshot_gpa(gpa + 3 * host_page_size as u64 + 1, &mut byte) + .is_err() + ); + } + + #[test] + fn memory_rejects_excess_layers() { + let host_page_size = page_size::get() as u64; + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let layers = (0..=MAX_SNAPSHOT_LAYERS as u64) + .map(|index| blob(1, base + index * host_page_size)) + .collect(); + + assert!(SnapshotMemory::new(layers, stub_page_tables()).is_err()); + } +} diff --git a/src/hyperlight_host/src/sandbox/snapshot/mod.rs b/src/hyperlight_host/src/sandbox/snapshot/mod.rs index 197edadf0..a8cfc388a 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/mod.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/mod.rs @@ -3,9 +3,12 @@ mod file; mod file_tests; +pub(crate) mod memory; mod tripwires; -use std::collections::{BTreeMap, HashMap}; +use std::cell::{Cell, RefCell}; +use std::collections::BTreeMap; +use std::ops::Range; use std::sync::Arc; pub(crate) use file::host_cpu_vendor_golden_tag; @@ -18,6 +21,7 @@ use hyperlight_common::vmem::{ }; use tracing::{Span, instrument}; +pub(crate) use self::memory::{SnapshotBlob, SnapshotMemory, SnapshotMemoryBacking}; use crate::Result; use crate::hypervisor::regs::CommonSpecialRegisters; #[cfg(target_arch = "x86_64")] @@ -25,8 +29,8 @@ use crate::hypervisor::regs::MsrEntry; use crate::mem::exe::{ExeInfo, LoadInfo}; use crate::mem::layout::SandboxMemoryLayout; use crate::mem::memory_region::{MemoryRegion, MemoryRegionFlags}; -use crate::mem::mgr::{GuestPageTableBuffer, SnapshotSharedMemory}; -use crate::mem::shared_mem::{ReadonlySharedMemory, SharedMemory}; +use crate::mem::mgr::{BackingSource, GuestBacking, GuestPageTableBuffer, GuestPhysicalMemoryView}; +use crate::mem::shared_mem::{ExclusiveSharedMemory, HostSharedMemory, ReadonlySharedMemory}; use crate::mem::virtq::VirtqSnapshot; use crate::sandbox::SandboxConfiguration; use crate::sandbox::uninitialized::{GuestBinary, GuestEnvironment}; @@ -70,7 +74,7 @@ struct SnapshotState { /// on?). layout: crate::mem::layout::SandboxMemoryLayout, /// Memory of the sandbox at the time this snapshot was taken - memory: ReadonlySharedMemory, + memory: Arc, /// Extra debug information about the binary in this snapshot, /// from when the binary was first loaded into the snapshot. /// @@ -137,17 +141,20 @@ impl hyperlight_common::vmem::TableReadOps for Snapshot { addr + offset } unsafe fn read_entry(&self, addr: u64) -> vmem::PageTableEntry { - let addr = addr as usize; - let Some(pte_bytes) = self.state.memory.as_slice().get(addr..addr + PTE_SIZE) else { + let mut pte_bytes = [0u8; PTE_SIZE]; + if self + .state + .memory + .read_page_tables(self.state.layout.get_pt_base_gpa(), addr, &mut pte_bytes) + .is_err() + { // Attacker-controlled data pointed out-of-bounds. We'll // default to returning 0 in this case, which, for most // architectures (including x86-64 and arm64, the ones we // care about presently) will be a not-present entry. return 0; - }; - // The `get()` above ensures exactly PTE_SIZE bytes. - #[allow(clippy::unwrap_used)] - vmem::PageTableEntry::from_le_bytes(pte_bytes.try_into().unwrap()) + } + vmem::PageTableEntry::from_le_bytes(pte_bytes) } #[allow(clippy::unnecessary_cast)] fn to_phys(addr: u64) -> vmem::PhysAddr { @@ -162,54 +169,67 @@ impl hyperlight_common::vmem::TableReadOps for Snapshot { } } -pub(crate) fn access_gpa<'a>( - snap: &'a [u8], - scratch: &'a [u8], - layout: SandboxMemoryLayout, - gpa: u64, -) -> Option<(&'a [u8], usize)> { - let resolved = layout.resolve_gpa(gpa, &[])?.with_memories(snap, scratch); - Some((resolved.base.as_ref(), resolved.offset)) -} - -pub(crate) struct SharedMemoryPageTableBuffer<'a> { - snap: &'a [u8], - scratch: &'a [u8], - layout: SandboxMemoryLayout, +pub(crate) struct PageTableReader<'a> { + memory: &'a GuestPhysicalMemoryView<'a>, root: u64, + failure: Cell>, + /// Address and copy of the last table page read. A walk reads the + /// entries of one table in sequence. + cache: RefCell<(u64, [u8; PAGE_SIZE])>, } -impl<'a> SharedMemoryPageTableBuffer<'a> { - pub(crate) fn new( - snap: &'a [u8], - scratch: &'a [u8], - layout: SandboxMemoryLayout, - root: u64, - ) -> Self { +impl<'a> PageTableReader<'a> { + pub(crate) fn new(memory: &'a GuestPhysicalMemoryView<'a>, root: u64) -> Self { Self { - snap, - scratch, - layout, + memory, root, + failure: Cell::new(None), + // An unaligned address never matches a table page. + cache: RefCell::new((u64::MAX, [0; PAGE_SIZE])), } } + + /// Returns suppressed page-table read failures. + pub(crate) fn finish(&self) -> Result<()> { + match self.failure.get() { + Some(failure) => Err(crate::new_error!("{}", failure)), + None => Ok(()), + } + } + + fn read_cached(&self, addr: u64) -> Option { + let page = addr & !(PAGE_SIZE as u64 - 1); + let offset = usize::try_from(addr - page).ok()?; + let mut cache = self.cache.borrow_mut(); + if cache.0 != page { + cache.0 = u64::MAX; + self.memory.read(page, &mut cache.1).ok()?; + cache.0 = page; + } + let entry = cache.1.get(offset..offset + PTE_SIZE)?; + Some(vmem::PageTableEntry::from_le_bytes(entry.try_into().ok()?)) + } } -impl<'a> hyperlight_common::vmem::TableReadOps for SharedMemoryPageTableBuffer<'a> { +impl<'a> hyperlight_common::vmem::TableReadOps for PageTableReader<'a> { type TableAddr = u64; fn entry_addr(addr: u64, offset: u64) -> u64 { addr + offset } unsafe fn read_entry(&self, addr: u64) -> vmem::PageTableEntry { - let memoff = access_gpa(self.snap, self.scratch, self.layout, addr); - let Some(pte_bytes) = memoff.and_then(|(mem, off)| mem.get(off..off + PTE_SIZE)) else { + if let Some(entry) = self.read_cached(addr) { + return entry; + } + let mut pte_bytes = [0u8; PTE_SIZE]; + if self.memory.read(addr, &mut pte_bytes).is_err() { // Attacker-controlled data pointed out-of-bounds. We'll // default to returning 0 in this case, which, for most // architectures (including x86-64 and arm64, the ones we // care about presently) will be a not-present entry. + // `finish` reports the failure. + self.failure + .set(Some("snapshot page-table walk accessed unbacked memory")); return 0; - }; - // The `get()` above ensures exactly PTE_SIZE bytes. - #[allow(clippy::unwrap_used)] - vmem::PageTableEntry::from_le_bytes(pte_bytes.try_into().unwrap()) + } + vmem::PageTableEntry::from_le_bytes(pte_bytes) } #[allow(clippy::unnecessary_cast)] fn to_phys(addr: u64) -> vmem::PhysAddr { @@ -223,11 +243,12 @@ impl<'a> hyperlight_common::vmem::TableReadOps for SharedMemoryPageTableBuffer<' self.root } } -impl<'a> core::convert::AsRef> for SharedMemoryPageTableBuffer<'a> { +impl<'a> core::convert::AsRef> for PageTableReader<'a> { fn as_ref(&self) -> &Self { self } } + /// Return true if `virt_base` is a VA we must not preserve into the /// rebuilt snapshot page tables: it is either part of the scratch /// region (re-mapped freshly by `map_specials`) or, on amd64, part of @@ -244,27 +265,211 @@ fn skip_virt(virt_base: u64, scratch_gva: u64) -> bool { false } -/// Find the contents of the page which starts at gpa in guest physical -/// memory, taking into account excess host->guest regions -/// -/// # Safety -/// The host side of the regions identified by MemoryRegion must be -/// alive and must not be mutated by any other thread: references to -/// these regions may be created and live for `'a`. -unsafe fn guest_page<'a>( - snap: &'a [u8], - scratch: &'a [u8], - regions: &[MemoryRegion], - layout: SandboxMemoryLayout, - gpa: u64, -) -> Option<&'a [u8]> { - let resolved = layout - .resolve_gpa(gpa, regions)? - .with_memories(snap, scratch); - if resolved.as_ref().len() < PAGE_SIZE { +fn for_each_mapping_page( + mapping: &Mapping, + mut visitor: impl FnMut(u64, u64, MappingKind) -> Result<()>, +) -> Result<()> { + if !validate_mapping(mapping)? { + return Ok(()); + } + let mut offset = 0u64; + while offset < mapping.len { + let phys = mapping + .phys_base + .checked_add(offset) + .ok_or_else(|| crate::new_error!("guest physical mapping overflows"))?; + let virt = mapping + .virt_base + .checked_add(offset) + .ok_or_else(|| crate::new_error!("guest virtual mapping overflows"))?; + visitor(phys, virt, mapping.kind)?; + offset = offset + .checked_add(PAGE_SIZE as u64) + .ok_or_else(|| crate::new_error!("guest mapping offset overflows"))?; + } + Ok(()) +} + +fn validate_mapping(mapping: &Mapping) -> Result { + if mapping.kind == MappingKind::Unmapped { + return Ok(false); + } + if mapping.len == 0 + || !mapping.phys_base.is_multiple_of(PAGE_SIZE as u64) + || !mapping.virt_base.is_multiple_of(PAGE_SIZE as u64) + || !mapping.len.is_multiple_of(PAGE_SIZE as u64) + { + return Err(crate::new_error!("guest mapping is not page aligned")); + } + Ok(true) +} + +fn mapping_has_skipped_pages(mapping: &Mapping, scratch_gva: u64) -> Result { + let last_page = mapping + .virt_base + .checked_add(mapping.len - PAGE_SIZE as u64) + .ok_or_else(|| crate::new_error!("guest virtual mapping overflows"))?; + let snapshot_pt_start = hyperlight_common::layout::SNAPSHOT_PT_GVA_MIN as u64; + let snapshot_pt_end = hyperlight_common::layout::SNAPSHOT_PT_GVA_MAX as u64; + Ok(last_page >= scratch_gva + || (mapping.virt_base <= snapshot_pt_end && snapshot_pt_start <= last_page)) +} + +#[derive(Clone, Copy)] +struct NewPage { + source_gpa: u64, + backing: GuestBacking, +} + +/// Marks reused layers, or queues pages for copying into the new layer. +fn record_backed_pages( + backing: GuestBacking, + source_gpa: u64, + len: usize, + reused_layers: Option<&mut [bool]>, + new_pages: &mut Vec, +) -> Result<()> { + if let BackingSource::Snapshot(layer_index) = backing.source + && let Some(reused_layers) = reused_layers + { + *reused_layers + .get_mut(layer_index) + .ok_or_else(|| crate::new_error!("snapshot layer index is out of bounds"))? = true; + return Ok(()); + } + + for offset in (0..len).step_by(PAGE_SIZE) { + new_pages.push(NewPage { + source_gpa: source_gpa + .checked_add(u64::try_from(offset)?) + .ok_or_else(|| crate::new_error!("snapshot source GPA overflows"))?, + backing: GuestBacking { + offset: backing + .offset + .checked_add(offset) + .ok_or_else(|| crate::new_error!("snapshot source backing offset overflows"))?, + ..backing + }, + }); + } + Ok(()) +} + +struct NewPageRun { + source: Range, + destination_start: u64, +} + +struct NewPageCopy { + backing: GuestBacking, + destination: Range, +} + +fn new_page_copies(pages: &[NewPage]) -> Result> { + let mut copies = Vec::::new(); + for (index, page) in pages.iter().enumerate() { + let start = index + .checked_mul(PAGE_SIZE) + .ok_or_else(|| crate::new_error!("snapshot page offset overflows"))?; + let end = start + .checked_add(PAGE_SIZE) + .ok_or_else(|| crate::new_error!("snapshot page range overflows"))?; + if let Some(copy) = copies.last_mut() + && copy.backing.source == page.backing.source + && copy.backing.offset.checked_add(copy.destination.len()) == Some(page.backing.offset) + { + copy.destination.end = end; + } else { + copies.push(NewPageCopy { + backing: page.backing, + destination: start..end, + }); + } + } + Ok(copies) +} + +fn new_page_runs(pages: &[NewPage], destination_start: u64) -> Result> { + let mut runs = Vec::::new(); + for (index, page) in pages.iter().enumerate() { + let source_gpa = page.source_gpa; + let destination = destination_start + .checked_add(u64::try_from(index.checked_mul(PAGE_SIZE).ok_or_else( + || crate::new_error!("snapshot page offset overflows"), + )?)?) + .ok_or_else(|| crate::new_error!("snapshot page GPA overflows"))?; + if let Some(run) = runs.last_mut() + && run.source.end == source_gpa + { + run.source.end = run + .source + .end + .checked_add(PAGE_SIZE as u64) + .ok_or_else(|| crate::new_error!("snapshot source page run overflows"))?; + } else { + let source_end = source_gpa + .checked_add(PAGE_SIZE as u64) + .ok_or_else(|| crate::new_error!("snapshot source page overflows"))?; + runs.push(NewPageRun { + source: source_gpa..source_end, + destination_start: destination, + }); + } + } + Ok(runs) +} + +fn page_destination(runs: &[NewPageRun], source_gpa: u64) -> Option { + let index = runs.partition_point(|run| run.source.end <= source_gpa); + let run = runs.get(index)?; + let offset = source_gpa.checked_sub(run.source.start)?; + (source_gpa < run.source.end).then(|| run.destination_start.checked_add(offset))? +} + +fn packed_data_len(page_count: usize, host_page_size: usize) -> Result { + let page_bytes = page_count + .checked_mul(PAGE_SIZE) + .ok_or_else(|| crate::new_error!("snapshot data size overflows"))?; + page_bytes + .checked_next_multiple_of(host_page_size) + .ok_or_else(|| crate::new_error!("snapshot data padding overflows")) +} + +fn first_fit_blob_gpa( + data_len: usize, + layers: &[Arc], + scratch_base: u64, +) -> Option { + if data_len == 0 { return None; } - Some(&resolved.as_ref()[..PAGE_SIZE]) + let data_len = u64::try_from(data_len).ok()?; + let mut candidate = SandboxMemoryLayout::BASE_ADDRESS as u64; + for range in layers.iter().map(|layer| layer.gpa_range()) { + let end = candidate.checked_add(data_len)?; + if end <= range.start { + return Some(candidate); + } + candidate = candidate.max(range.end); + } + (candidate.checked_add(data_len)? <= scratch_base).then_some(candidate) +} + +fn snapshot_mapping_kind(kind: MappingKind) -> Option { + match kind { + MappingKind::Cow(mapping) => Some(MappingKind::Cow(mapping)), + MappingKind::Basic(mapping) if mapping.writable => Some(MappingKind::Cow(CowMapping { + readable: mapping.readable, + executable: mapping.executable, + })), + MappingKind::Basic(mapping) => Some(MappingKind::Basic(BasicMapping { + readable: mapping.readable, + writable: false, + executable: mapping.executable, + })), + MappingKind::ZeroInit(mapping) => Some(MappingKind::ZeroInit(mapping)), + MappingKind::Unmapped => None, + } } fn map_specials(pt_buf: &GuestPageTableBuffer, scratch_size: usize) { @@ -391,8 +596,7 @@ impl Snapshot { map_specials(&pt_buf, layout.get_scratch_size()); let pt_bytes = pt_buf.into_bytes(); - layout.set_pt_size(pt_bytes.len())?; - memory.extend(&pt_bytes); + layout.ensure_page_tables_fit(pt_bytes.len())?; let exn_stack_top_gva = hyperlight_common::layout::SCRATCH_TOP_GVA as u64 - hyperlight_common::layout::SCRATCH_TOP_EXN_STACK_OFFSET @@ -410,9 +614,22 @@ impl Snapshot { .checked_add(entrypoint_offset) .ok_or_else(|| crate::new_error!("ELF entrypoint GVA overflows"))?; + // The page tables follow the data, which is padded to a host page. + let data_len = memory + .len() + .next_multiple_of(self::memory::host_page_size()); + memory.resize(data_len, 0); + memory.extend_from_slice(&pt_bytes); + let memory = Arc::new(SnapshotMemory::from_flat( + ReadonlySharedMemory::from_bytes(&memory, data_len)?, + SandboxMemoryLayout::BASE_ADDRESS as u64, + pt_bytes.len(), + scratch_base_gpa(layout.get_scratch_size()), + )?); + Ok(Self { state: Arc::new(SnapshotState { - memory: ReadonlySharedMemory::from_bytes(&memory, layout.snapshot_size())?, + memory, layout, load_info, stack_top_gva: exn_stack_top_gva, @@ -440,11 +657,12 @@ impl Snapshot { /// Capture memory with optional validated transport images. /// /// Page-table and snapshot sizes do not affect transport geometry. + /// `root_pt_gpas` must be distinct. #[instrument(err(Debug), skip_all, parent = Span::current(), level= "Trace")] - pub(crate) fn new( - shared_mem: &mut SnapshotSharedMemory, - scratch_mem: &mut S, - mut layout: SandboxMemoryLayout, + pub(crate) fn new( + shared_mem: &SnapshotMemoryBacking, + scratch_mem: &HostSharedMemory, + layout: SandboxMemoryLayout, load_info: LoadInfo, regions: Vec, root_pt_gpas: &[u64], @@ -457,166 +675,247 @@ impl Snapshot { host_functions: HostFunctionDetails, virtq: Option, ) -> Result { - let mut phys_seen = HashMap::::new(); + if root_pt_gpas.is_empty() { + return Err(crate::new_error!("snapshot has no page-table roots")); + } let scratch_gva = scratch_base_gva(layout.get_scratch_size()); - let memory = shared_mem.with_contents(|snap_c| { - scratch_mem.with_contents(|scratch_c| { - // Phase 1: walk every PT root together. This detects - // aliased intermediate tables (e.g. Nanvix's kernel- - // half PTs, which multiple process PDs share by - // pointing at the same PT page). The walker emits - // `ThisSpace(leaf)` for private leaves and - // `AnotherSpace(ref)` for sub-trees that were already - // seen via an earlier root. Results are returned in - // `root_pt_gpas` order — which is also the topological - // order of the `AnotherSpace` references — so - // processing in iteration order is safe. - let op = SharedMemoryPageTableBuffer::new( - snap_c, - scratch_c, - layout, - root_pt_gpas.first().copied().unwrap_or(0), - ); - let walk = unsafe { - vmem::walk_va_spaces( - &op, - root_pt_gpas, - 0, - hyperlight_common::layout::SCRATCH_TOP_GVA as u64, - ) + let memory_view = + GuestPhysicalMemoryView::with_dynamic(shared_mem, scratch_mem, ®ions, layout); + // Unbacked roots fail through `PageTableReader::finish`. + if let Some(root) = root_pt_gpas + .iter() + .find(|root| !root.is_multiple_of(PAGE_SIZE as u64)) + { + return Err(crate::new_error!( + "snapshot page-table root {root:#x} is not aligned" + )); + } + // Phase 1: walk every PT root together. This detects + // aliased intermediate tables (e.g. Nanvix's kernel- + // half PTs, which multiple process PDs share by + // pointing at the same PT page). The walker emits + // `ThisSpace(leaf)` for private leaves and + // `AnotherSpace(ref)` for sub-trees that were already + // seen via an earlier root. Results are returned in + // `root_pt_gpas` order — which is also the topological + // order of the `AnotherSpace` references — so + // processing in iteration order is safe. + let op = PageTableReader::new(&memory_view, root_pt_gpas[0]); + // SAFETY: Snapshot capture runs with the vCPU stopped, so the source + // page tables cannot change during the walk. + let walk = unsafe { + vmem::walk_va_spaces( + &op, + root_pt_gpas, + 0, + hyperlight_common::layout::SCRATCH_TOP_GVA as u64, + ) + }; + op.finish()?; + let source_layers = shared_mem.reusable_layers(); + let mut reused_layers = source_layers.map(|layers| vec![false; layers.len()]); + let mut new_pages = Vec::new(); + for (_, mappings) in &walk { + for mapping in mappings { + let SpaceAwareMapping::ThisSpace(mapping) = mapping else { + continue; }; - - // Phase 2: rebuild each space's page tables, compacting - // `ThisSpace` leaves into a dense snapshot blob and - // linking `AnotherSpace` entries to already-built - // spaces' tables. - // TODO: Look for opportunities to hugepage map - let mut snapshot_memory: Vec = Vec::new(); - let pt_buf = GuestPageTableBuffer::new(layout.get_pt_base_gpa() as usize); - // Allocate one root table per space and remember the - // addresses returned by `alloc_table` instead of - // assuming the buffer's physical layout. - let mut root_addrs: Vec = Vec::with_capacity(root_pt_gpas.len()); - root_addrs.push(pt_buf.initial_root()); - for _ in 1..root_pt_gpas.len() { - root_addrs.push(unsafe { pt_buf.alloc_table() }); + if validate_mapping(mapping)? && !mapping_has_skipped_pages(mapping, scratch_gva)? { + if matches!(mapping.kind, MappingKind::ZeroInit(_)) { + continue; + } + let mapping_len = usize::try_from(mapping.len)?; + if let Some(backing) = memory_view.resolve(mapping.phys_base, mapping_len) { + record_backed_pages( + backing, + mapping.phys_base, + mapping_len, + reused_layers.as_deref_mut(), + &mut new_pages, + )?; + continue; + } } + for_each_mapping_page(mapping, |source_gpa, virt_gva, kind| { + if skip_virt(virt_gva, scratch_gva) || matches!(kind, MappingKind::ZeroInit(_)) + { + return Ok(()); + } + let backing = memory_view.resolve(source_gpa, PAGE_SIZE).ok_or_else(|| { + crate::new_error!("snapshot leaf names unbacked GPA {source_gpa:#x}") + })?; + record_backed_pages( + backing, + source_gpa, + PAGE_SIZE, + reused_layers.as_deref_mut(), + &mut new_pages, + ) + })?; + } + } + + new_pages.sort_unstable_by_key(|page| page.source_gpa); + new_pages.dedup_by_key(|page| page.source_gpa); + + let mut layers = match (source_layers, reused_layers) { + (Some(source_layers), Some(reused_layers)) => source_layers + .iter() + .zip(reused_layers) + .filter(|(_, reused)| *reused) + .map(|(layer, _)| layer.clone()) + .collect(), + _ => Vec::new(), + }; + let host_page_size = self::memory::host_page_size(); + let data_len = packed_data_len(new_pages.len(), host_page_size)?; + let new_blob_gpa = if data_len == 0 { + None + } else { + Some( + first_fit_blob_gpa( + data_len, + &layers, + scratch_base_gpa(layout.get_scratch_size()), + ) + .ok_or_else(|| { + crate::new_error!("snapshot has no address space for a new layer") + })?, + ) + }; + let new_page_runs = match new_blob_gpa { + Some(destination_start) => new_page_runs(&new_pages, destination_start)?, + None => Vec::new(), + }; - let mut built_roots: BTreeMap = BTreeMap::new(); - for (root_idx, (space_id, mappings)) in walk.into_iter().enumerate() { - pt_buf.set_root(root_addrs[root_idx]); - built_roots.insert(space_id, root_addrs[root_idx]); - - for sam in mappings { - match sam { - SpaceAwareMapping::ThisSpace(mapping) => { - // Drop the scratch region and (on - // amd64) the snapshot's own PT - // self-map; both are re-mapped - // freshly by `map_specials`. - if skip_virt(mapping.virt_base, scratch_gva) { - continue; - } - - // Writable pages become CoW in the - // rebuilt snapshot; read-only pages - // stay read-only. - let (kind, has_contents) = match mapping.kind { - MappingKind::Cow(cm) => (MappingKind::Cow(cm), true), - MappingKind::Basic(bm) if bm.writable => ( - MappingKind::Cow(CowMapping { - readable: bm.readable, - executable: bm.executable, - }), - true, - ), - MappingKind::Basic(bm) => ( - MappingKind::Basic(BasicMapping { - readable: bm.readable, - writable: false, - executable: bm.executable, - }), - true, - ), - MappingKind::Unmapped => continue, - MappingKind::ZeroInit(bm) => (MappingKind::ZeroInit(bm), false), - }; - let new_gpa = if has_contents { - let Some(contents) = (unsafe { - guest_page( - snap_c, - scratch_c, - ®ions, - layout, - mapping.phys_base, - ) - }) else { - continue; - }; - Some(*phys_seen.entry(mapping.phys_base).or_insert_with(|| { - let new_offset = snapshot_memory.len(); - snapshot_memory.extend(contents); - new_offset + SandboxMemoryLayout::BASE_ADDRESS - })) - } else { - None - }; - - let compacted = Mapping { - phys_base: new_gpa.unwrap_or(0) as u64, - virt_base: mapping.virt_base, + // Phase 2: rebuild each space's page tables, compacting + // `ThisSpace` leaves into a dense snapshot blob and + // linking `AnotherSpace` entries to already-built + // spaces' tables. + // TODO: Look for opportunities to hugepage map + let pt_buf = GuestPageTableBuffer::new(layout.get_pt_base_gpa() as usize); + // Allocate one root table per space and remember the + // addresses returned by `alloc_table` instead of + // assuming the buffer's physical layout. + let mut root_addrs = Vec::with_capacity(root_pt_gpas.len()); + root_addrs.push(pt_buf.initial_root()); + for _ in 1..root_pt_gpas.len() { + // SAFETY: `pt_buf` is local to this capture and accessed serially. + root_addrs.push(unsafe { pt_buf.alloc_table() }); + } + + let mut built_roots: BTreeMap = BTreeMap::new(); + for (root_index, (space_id, mappings)) in walk.into_iter().enumerate() { + pt_buf.set_root(root_addrs[root_index]); + built_roots.insert(space_id, root_addrs[root_index]); + let mut pending_mapping: Option = None; + let flush_mapping = |pending: &mut Option| { + if let Some(mapping) = pending.take() { + // SAFETY: The mapping is page-aligned and `pt_buf` is local + // to this capture. + unsafe { vmem::map(&pt_buf, mapping) }; + } + }; + for mapping in mappings { + match mapping { + SpaceAwareMapping::ThisSpace(mapping) => { + for_each_mapping_page(&mapping, |source_gpa, virt_gva, kind| { + // Drop the scratch region and (on + // amd64) the snapshot's own PT + // self-map; both are re-mapped + // freshly by `map_specials`. + if skip_virt(virt_gva, scratch_gva) { + flush_mapping(&mut pending_mapping); + return Ok(()); + } + // Writable pages become CoW in the + // rebuilt snapshot; read-only pages + // stay read-only. + let kind = snapshot_mapping_kind(kind).ok_or_else(|| { + crate::new_error!("snapshot walker returned an unmapped leaf") + })?; + let destination_gpa = + page_destination(&new_page_runs, source_gpa).unwrap_or(source_gpa); + if let Some(pending) = pending_mapping.as_mut() + && pending.kind == kind + && pending.phys_base.checked_add(pending.len) + == Some(destination_gpa) + && pending.virt_base.checked_add(pending.len) == Some(virt_gva) + { + pending.len = + pending.len.checked_add(PAGE_SIZE as u64).ok_or_else(|| { + crate::new_error!("snapshot mapping length overflows") + })?; + } else { + flush_mapping(&mut pending_mapping); + pending_mapping = Some(Mapping { + phys_base: destination_gpa, + virt_base: virt_gva, len: PAGE_SIZE as u64, kind, - }; - unsafe { vmem::map(&pt_buf, compacted) }; - } - SpaceAwareMapping::AnotherSpace(ref_map) => { - // Link to the owning space's already- - // rebuilt intermediate table — this - // is what preserves Nanvix's - // kernel-half-shared invariant across - // process PDs after relocation. - unsafe { - vmem::space_aware_map(&pt_buf, ref_map, &built_roots); - } + }); } - } + Ok(()) + })?; + } + SpaceAwareMapping::AnotherSpace(reference) => { + flush_mapping(&mut pending_mapping); + // Link to the owning space's already- + // rebuilt intermediate table — this + // is what preserves Nanvix's + // kernel-half-shared invariant across + // process PDs after relocation. + // SAFETY: The reference came from the source walk and + // `pt_buf` is local to this capture. The walk returns + // spaces in topological order, so the owner named by + // `reference` already has its root in `built_roots`. + unsafe { vmem::space_aware_map(&pt_buf, reference, &built_roots) }; } } + } + flush_mapping(&mut pending_mapping); + } - // Phase 3: Map the scratch region into each root. - for &root_addr in &root_addrs { - pt_buf.set_root(root_addr); - map_specials(&pt_buf, layout.get_scratch_size()); - } - pt_buf.set_root(pt_buf.initial_root()); - - snapshot_memory.resize( - snapshot_memory.len().next_multiple_of(page_size::get()), - 0u8, - ); - - // Phase 4: finalize PT bytes. - let pt_data = pt_buf.into_bytes(); - layout.set_pt_size(pt_data.len())?; - snapshot_memory.extend(&pt_data); - Ok::<_, crate::HyperlightError>(snapshot_memory) - }) - })???; - // Only the data prefix is exposed to the guest. The PT tail - // sits past it in the host mapping and is copied into the - // scratch region on restore. Keeping it out of the guest - // mapping of the snapshot region avoids overlap with - // `map_file_cow` regions installed immediately after the - // snapshot in guest PA space. - let guest_visible_size = memory.len() - layout.get_pt_size(); - debug_assert!(guest_visible_size.is_multiple_of(page_size::get())); - layout.set_snapshot_size(guest_visible_size); + // Phase 3: Map the scratch region into each root. + for &root_addr in &root_addrs { + pt_buf.set_root(root_addr); + map_specials(&pt_buf, layout.get_scratch_size()); + } + pt_buf.set_root(pt_buf.initial_root()); + // Phase 4: write the new blob, the new pages followed by the page + // tables. A capture with no new pages still needs a blob for its + // page tables. + let page_tables = pt_buf.into_bytes(); + layout.ensure_page_tables_fit(page_tables.len())?; + let blob_len = data_len + .checked_add(page_tables.len()) + .and_then(|len| len.checked_next_multiple_of(host_page_size)) + .ok_or_else(|| crate::new_error!("snapshot blob size overflows"))?; + let mut allocation = ExclusiveSharedMemory::new(blob_len)?; + for copy in new_page_copies(&new_pages)? { + let destination = allocation + .as_mut_slice() + .get_mut(copy.destination) + .ok_or_else(|| crate::new_error!("snapshot destination page is out of bounds"))?; + memory_view.read_backing(copy.backing, destination)?; + } + allocation.copy_from_slice(&page_tables, data_len)?; + let blob = Arc::new(SnapshotBlob::new( + allocation.freeze(data_len)?, + new_blob_gpa.unwrap_or_default(), + page_tables.len(), + scratch_base_gpa(layout.get_scratch_size()), + )?); + if data_len > 0 { + layers.push(blob.clone()); + } + // Validates the layer limit. + let memory = Arc::new(SnapshotMemory::new(layers.into_boxed_slice(), blob)?); Ok(Self { state: Arc::new(SnapshotState { layout, - memory: ReadonlySharedMemory::from_bytes(&memory, guest_visible_size)?, + memory, load_info, stack_top_gva, sregs: Some(sregs), @@ -726,7 +1025,7 @@ impl Snapshot { /// Return the main memory contents of the snapshot #[instrument(skip_all, parent = Span::current(), level= "Trace")] - pub(crate) fn memory(&self) -> &ReadonlySharedMemory { + pub(crate) fn snapshot_memory(&self) -> &Arc { &self.state.memory } @@ -841,27 +1140,148 @@ impl Snapshot { #[cfg(test)] mod tests { + use std::sync::Arc; + use hyperlight_common::flatbuffer_wrappers::host_function_details::HostFunctionDetails; - use hyperlight_common::vmem::{self, BasicMapping, Mapping, MappingKind, PAGE_SIZE}; + use hyperlight_common::vmem::{ + self, BasicMapping, Mapping, MappingKind, PAGE_SIZE, PAGE_TABLE_SIZE, + }; + use super::SnapshotMemoryBacking; use crate::hypervisor::regs::CommonSpecialRegisters; use crate::mem::exe::LoadInfo; use crate::mem::layout::SandboxMemoryLayout; - use crate::mem::mgr::{GuestPageTableBuffer, SandboxMemoryManager, SnapshotSharedMemory}; - use crate::mem::shared_mem::{ - ExclusiveSharedMemory, HostSharedMemory, ReadonlySharedMemory, SharedMemory, + use crate::mem::mgr::{ + BackingSource, GuestBacking, GuestPageTableBuffer, GuestPhysicalMemoryView, + SandboxMemoryManager, }; + use crate::mem::shared_mem::{ExclusiveSharedMemory, HostSharedMemory}; fn default_sregs() -> CommonSpecialRegisters { CommonSpecialRegisters::default() } - fn simple_pt_base() -> usize { - page_size::get() + SandboxMemoryLayout::BASE_ADDRESS + fn new_page(source_gpa: u64, offset: usize) -> super::NewPage { + super::NewPage { + source_gpa, + backing: GuestBacking { + source: BackingSource::Snapshot(0), + offset, + }, + } + } + + #[test] + fn page_runs_map_each_source_to_its_destination() { + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let page = PAGE_SIZE as u64; + let destination = base + 0x10_0000; + // Two contiguous pages, then a gap, then one more. + let pages = [ + new_page(base, 0), + new_page(base + page, PAGE_SIZE), + new_page(base + 8 * page, 8 * PAGE_SIZE), + ]; + + let runs = super::new_page_runs(&pages, destination).unwrap(); + assert_eq!(runs.len(), 2); + + // Destinations are packed in index order regardless of source gaps. + assert_eq!(super::page_destination(&runs, base), Some(destination)); + assert_eq!( + super::page_destination(&runs, base + page), + Some(destination + page) + ); + assert_eq!( + super::page_destination(&runs, base + 8 * page), + Some(destination + 2 * page) + ); + // An address in the source gap belongs to no run. + assert_eq!(super::page_destination(&runs, base + 2 * page), None); + assert_eq!(super::page_destination(&runs, base + 9 * page), None); + } + + #[test] + fn page_copies_merge_only_contiguous_backings() { + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let page = PAGE_SIZE as u64; + // Contiguous source addresses, but the third page comes from a + // backing offset that does not follow the second. + let pages = [ + new_page(base, 0), + new_page(base + page, PAGE_SIZE), + new_page(base + 2 * page, 9 * PAGE_SIZE), + ]; + + let copies = super::new_page_copies(&pages).unwrap(); + assert_eq!(copies.len(), 2); + assert_eq!(copies[0].destination, 0..2 * PAGE_SIZE); + assert_eq!(copies[1].destination, 2 * PAGE_SIZE..3 * PAGE_SIZE); + } + + #[test] + fn blob_placement_respects_the_scratch_boundary() { + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let scratch_base = base + 2 * PAGE_SIZE as u64; + + // An empty delta needs no address. + assert_eq!(super::first_fit_blob_gpa(0, &[], scratch_base), None); + // With nothing to reuse the delta lands at the base. + assert_eq!( + super::first_fit_blob_gpa(PAGE_SIZE, &[], scratch_base), + Some(base) + ); + // Exactly filling the space below scratch is allowed. + assert_eq!( + super::first_fit_blob_gpa(2 * PAGE_SIZE, &[], scratch_base), + Some(base) + ); + // One page more is not. + assert_eq!( + super::first_fit_blob_gpa(3 * PAGE_SIZE, &[], scratch_base), + None + ); } - fn make_simple_pt_mem(contents: &[u8]) -> SnapshotSharedMemory { - let pt_buf = GuestPageTableBuffer::new(simple_pt_base()); + #[test] + fn snapshot_page_policy_reuses_shared_and_copies_private_backings() { + let source_gpa = SandboxMemoryLayout::BASE_ADDRESS as u64; + let backing = GuestBacking { + source: BackingSource::Snapshot(0), + offset: 0, + }; + let mut reused = vec![false]; + let mut copied = Vec::new(); + + super::record_backed_pages( + backing, + source_gpa, + 2 * PAGE_SIZE, + Some(reused.as_mut_slice()), + &mut copied, + ) + .unwrap(); + assert_eq!(reused, [true]); + assert!(copied.is_empty()); + + reused[0] = false; + super::record_backed_pages(backing, source_gpa, 2 * PAGE_SIZE, None, &mut copied).unwrap(); + assert_eq!(reused, [false]); + assert_eq!(copied.len(), 2); + assert_eq!(copied[0].source_gpa, source_gpa); + assert_eq!(copied[0].backing, backing); + assert_eq!(copied[1].source_gpa, source_gpa + PAGE_SIZE as u64); + assert_eq!( + copied[1].backing, + GuestBacking { + source: BackingSource::Snapshot(0), + offset: PAGE_SIZE, + } + ); + } + + fn make_simple_pt_memory(contents: &[u8], pt_base: u64) -> super::SnapshotMemory { + let pt_buf = GuestPageTableBuffer::new(pt_base as usize); let mapping = Mapping { phys_base: SandboxMemoryLayout::BASE_ADDRESS as u64, virt_base: SandboxMemoryLayout::BASE_ADDRESS as u64, @@ -876,26 +1296,189 @@ mod tests { super::map_specials(&pt_buf, PAGE_SIZE); let pt_bytes = pt_buf.into_bytes(); - let mut snapshot_mem = vec![0u8; page_size::get() + pt_bytes.len()]; - snapshot_mem[0..page_size::get()].copy_from_slice(contents); - snapshot_mem[page_size::get()..].copy_from_slice(&pt_bytes); - ReadonlySharedMemory::from_bytes(&snapshot_mem, page_size::get()) - .unwrap() - .to_mgr_snapshot_mem() + super::memory::tests::flat_memory( + contents, + &pt_bytes, + SandboxMemoryLayout::BASE_ADDRESS as u64, + hyperlight_common::layout::scratch_base_gpa(PAGE_SIZE), + ) + } + + fn make_simple_pt_host_mem( + contents: &[u8], + pt_base: u64, + ) -> SnapshotMemoryBacking { + SnapshotMemoryBacking::from_snapshot(Arc::new(make_simple_pt_memory(contents, pt_base))) .unwrap() + .build() + .0 } fn make_simple_pt_mgr() -> (SandboxMemoryManager, u64) { let cfg = crate::sandbox::SandboxConfiguration::default(); let scratch_mem = ExclusiveSharedMemory::new(cfg.get_scratch_size()).unwrap(); + let layout = SandboxMemoryLayout::new(cfg, 4096, 0x3000, None).unwrap(); + let pt_base = layout.get_pt_base_gpa(); + let memory = make_simple_pt_memory(&vec![0u8; page_size::get()], pt_base); + layout + .ensure_page_tables_fit(memory.page_table_len()) + .unwrap(); let mgr = SandboxMemoryManager::new( - SandboxMemoryLayout::new(cfg, 4096, 0x3000, None).unwrap(), - make_simple_pt_mem(&vec![0u8; page_size::get()]), + layout, + SnapshotMemoryBacking::from_snapshot(Arc::new(memory)).unwrap(), scratch_mem, super::NextAction::None, ); let (mgr, _) = mgr.build().unwrap(); - (mgr, simple_pt_base() as u64) + (mgr, pt_base) + } + + #[test] + fn capture_and_restore_preserve_multiple_page_table_roots() { + let cfg = crate::sandbox::SandboxConfiguration::default(); + let layout = SandboxMemoryLayout::new(cfg, 4096, 0x3000, None).unwrap(); + let pt_base = layout.get_pt_base_gpa(); + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let pt_buf = GuestPageTableBuffer::new(pt_base as usize); + let first_root = pt_buf.initial_root(); + // SAFETY: The local buffer owns its page tables and has no concurrent access. + let second_root = unsafe { ::alloc_table(&pt_buf) }; + let kind = MappingKind::Basic(BasicMapping { + readable: true, + writable: false, + executable: false, + }); + // SAFETY: Both roots belong to this local buffer and the mappings are page-aligned. + unsafe { + vmem::map( + &pt_buf, + Mapping { + phys_base: base, + virt_base: base, + len: PAGE_SIZE as u64, + kind, + }, + ); + pt_buf.set_root(second_root); + vmem::map( + &pt_buf, + Mapping { + phys_base: base + PAGE_SIZE as u64, + virt_base: base + PAGE_SIZE as u64, + len: PAGE_SIZE as u64, + kind, + }, + ); + } + pt_buf.set_root(first_root); + let page_tables = pt_buf.into_bytes(); + layout.ensure_page_tables_fit(page_tables.len()).unwrap(); + + let data_len = 2 * page_size::get(); + let mut bytes = vec![0u8; data_len]; + bytes[..PAGE_SIZE].fill(0x11); + bytes[PAGE_SIZE..2 * PAGE_SIZE].fill(0x22); + let memory = super::memory::tests::flat_memory( + &bytes, + &page_tables, + base, + hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()), + ); + let manager = SandboxMemoryManager::new( + layout, + SnapshotMemoryBacking::from_snapshot(Arc::new(memory)).unwrap(), + ExclusiveSharedMemory::new(layout.get_scratch_size()).unwrap(), + super::NextAction::None, + ); + let (mut manager, _) = manager.build().unwrap(); + + let snapshot = super::Snapshot::new( + &manager.shared_mem, + &manager.scratch_mem, + manager.layout, + LoadInfo::dummy(), + Vec::new(), + &[first_root, second_root], + 0, + default_sregs(), + #[cfg(target_arch = "x86_64")] + Vec::new(), + super::NextAction::None, + 0, + 1, + HostFunctionDetails::default(), + None, + ) + .unwrap(); + let rebuilt_first_root = snapshot.root_pt_gpa(); + let rebuilt_second_root = rebuilt_first_root + PAGE_TABLE_SIZE as u64; + let roots = [rebuilt_first_root, rebuilt_second_root]; + // SAFETY: The immutable snapshot owns both roots throughout the walk. + let walked = unsafe { vmem::walk_va_spaces(&snapshot, &roots, base, 2 * PAGE_SIZE as u64) }; + assert_eq!(walked.len(), 2); + assert!(walked[0].1.iter().any(|mapping| matches!( + mapping, + vmem::SpaceAwareMapping::ThisSpace(mapping) if mapping.virt_base == base + ))); + assert!(walked[1].1.iter().any(|mapping| matches!( + mapping, + vmem::SpaceAwareMapping::ThisSpace(mapping) + if mapping.virt_base == base + PAGE_SIZE as u64 + ))); + + manager.restore_snapshot(&snapshot).unwrap(); + let memory = + GuestPhysicalMemoryView::new(&manager.shared_mem, &manager.scratch_mem, manager.layout); + for (root, gva, expected) in [ + (rebuilt_first_root, base, 0x11), + (rebuilt_second_root, base + PAGE_SIZE as u64, 0x22), + ] { + let reader = super::PageTableReader::new(&memory, root); + // SAFETY: The reader borrows the stopped manager's mapped memory. + let mapping = unsafe { vmem::virt_to_phys(&reader, gva, 1) } + .next() + .unwrap(); + let gpa = mapping.phys_base + gva - mapping.virt_base; + let mut byte = [0u8; 1]; + memory.read(gpa, &mut byte).unwrap(); + assert_eq!(byte, [expected]); + } + } + + #[cfg(target_arch = "x86_64")] + #[test] + fn capture_rejects_unbacked_intermediate_page_table() { + let (manager, pt_base) = make_simple_pt_mgr(); + let scratch_base = + hyperlight_common::layout::scratch_base_gpa(manager.layout.get_scratch_size()); + let root_offset = usize::try_from(pt_base - scratch_base).unwrap(); + let invalid_entry = (0xdead_0000 | vmem::PAGE_PRESENT).to_le_bytes(); + manager + .scratch_mem + .copy_from_slice(&invalid_entry, root_offset) + .unwrap(); + + let result = super::Snapshot::new( + &manager.shared_mem, + &manager.scratch_mem, + manager.layout, + LoadInfo::dummy(), + Vec::new(), + &[pt_base], + 0, + default_sregs(), + Vec::new(), + super::NextAction::None, + 0, + 1, + HostFunctionDetails::default(), + None, + ); + let error = match result { + Ok(_) => panic!("capture unexpectedly accepted an unbacked page table"), + Err(error) => error, + }; + assert!(error.to_string().contains("accessed unbacked memory")); } #[test] @@ -904,9 +1487,10 @@ mod tests { // Create first snapshot with pattern A let pattern_a = vec![0xAA; page_size::get()]; + let pattern_a_memory = make_simple_pt_host_mem(&pattern_a, pt_base); let snapshot_a = super::Snapshot::new( - &mut make_simple_pt_mem(&pattern_a).build().0, - &mut mgr.scratch_mem, + &pattern_a_memory, + &mgr.scratch_mem, mgr.layout, LoadInfo::dummy(), Vec::new(), @@ -922,12 +1506,14 @@ mod tests { None, ) .unwrap(); + assert_eq!(snapshot_a.snapshot_memory().layers().len(), 1); // Create second snapshot with pattern B let pattern_b = vec![0xBB; page_size::get()]; + let pattern_b_memory = make_simple_pt_host_mem(&pattern_b, pt_base); let snapshot_b = super::Snapshot::new( - &mut make_simple_pt_mem(&pattern_b).build().0, - &mut mgr.scratch_mem, + &pattern_b_memory, + &mgr.scratch_mem, mgr.layout, LoadInfo::dummy(), Vec::new(), @@ -946,14 +1532,462 @@ mod tests { // Restore snapshot A mgr.restore_snapshot(&snapshot_a).unwrap(); + let mut restored = vec![0u8; pattern_a.len()]; mgr.shared_mem - .with_contents(|contents| assert_eq!(&contents[0..pattern_a.len()], &pattern_a[..])) + .read_snapshot_gpa(SandboxMemoryLayout::BASE_ADDRESS as u64, &mut restored) .unwrap(); + assert_eq!(restored, pattern_a); // Restore snapshot B mgr.restore_snapshot(&snapshot_b).unwrap(); + restored.fill(0); mgr.shared_mem - .with_contents(|contents| assert_eq!(&contents[0..pattern_b.len()], &pattern_b[..])) + .read_snapshot_gpa(SandboxMemoryLayout::BASE_ADDRESS as u64, &mut restored) + .unwrap(); + assert_eq!(restored, pattern_b); + } + + #[test] + fn page_table_reader_reports_unbacked_memory() { + let (manager, pt_base) = make_simple_pt_mgr(); + let memory = crate::mem::mgr::GuestPhysicalMemoryView::new( + &manager.shared_mem, + &manager.scratch_mem, + manager.layout, + ); + + let reader = super::PageTableReader::new(&memory, pt_base); + assert!(reader.finish().is_ok()); + + // A read off every backing is recorded rather than returned, so the + // walk keeps going against not-present entries until finish() asks. + // SAFETY: the reader borrows `memory`, which outlives this call. + let entry = unsafe { + ::read_entry( + &reader, + !(PAGE_SIZE as u64 - 1), + ) + }; + assert_eq!(entry, 0); + let error = reader.finish().unwrap_err().to_string(); + assert!(error.contains("accessed unbacked memory"), "{error}"); + } + + #[test] + fn page_table_reader_switches_cached_table_pages() { + let (manager, pt_base) = make_simple_pt_mgr(); + let first = 8; + let second = PAGE_SIZE + 16; + manager.scratch_mem.write::(first, 0x1111).unwrap(); + manager.scratch_mem.write::(second, 0x2222).unwrap(); + let scratch_base = + hyperlight_common::layout::scratch_base_gpa(manager.layout.get_scratch_size()); + let memory = crate::mem::mgr::GuestPhysicalMemoryView::new( + &manager.shared_mem, + &manager.scratch_mem, + manager.layout, + ); + let reader = super::PageTableReader::new(&memory, pt_base); + // SAFETY: the reader borrows `memory`, which outlives these calls. + let read = |offset: usize| unsafe { + ::read_entry( + &reader, + scratch_base + offset as u64, + ) + }; + + assert_eq!(read(first), 0x1111); + assert_eq!(read(second), 0x2222); + assert_eq!(read(first), 0x1111); + assert!(reader.finish().is_ok()); + } + + #[test] + fn capture_rejects_invalid_page_table_roots() { + let (manager, root) = make_simple_pt_mgr(); + for roots in [ + Vec::new(), + vec![root + 1], + vec![u64::MAX - (PAGE_SIZE as u64 - 1)], + ] { + let result = super::Snapshot::new( + &manager.shared_mem, + &manager.scratch_mem, + manager.layout, + LoadInfo::dummy(), + Vec::new(), + &roots, + 0, + default_sregs(), + #[cfg(target_arch = "x86_64")] + Vec::new(), + super::NextAction::None, + 0, + 1, + HostFunctionDetails::default(), + None, + ); + assert!(result.is_err(), "invalid roots accepted: {roots:?}"); + } + } + + #[test] + fn capture_reuses_shared_pages_and_materializes_private_pages() { + let host_page_size = page_size::get(); + let cfg = crate::sandbox::SandboxConfiguration::default(); + let layout = SandboxMemoryLayout::new(cfg, 4096, 0x3000, None).unwrap(); + let pt_base = layout.get_pt_base_gpa(); + let scratch_offset = layout.get_scratch_size() / 2; + let scratch_gpa = hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()) + + scratch_offset as u64; + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + + let pt_buf = GuestPageTableBuffer::new(pt_base as usize); + // SAFETY: The page-aligned mappings are written serially to this local + // page-table buffer. + unsafe { + vmem::map( + &pt_buf, + Mapping { + phys_base: base, + virt_base: base, + len: host_page_size as u64, + kind: MappingKind::Basic(BasicMapping { + readable: true, + writable: false, + executable: true, + }), + }, + ); + vmem::map( + &pt_buf, + Mapping { + phys_base: scratch_gpa, + virt_base: base + (2 * host_page_size) as u64, + len: host_page_size as u64, + kind: MappingKind::Basic(BasicMapping { + readable: true, + writable: true, + executable: false, + }), + }, + ); + vmem::map( + &pt_buf, + Mapping { + phys_base: scratch_gpa, + virt_base: base + host_page_size as u64, + len: host_page_size as u64, + kind: MappingKind::Basic(BasicMapping { + readable: true, + writable: true, + executable: false, + }), + }, + ); + } + super::map_specials(&pt_buf, layout.get_scratch_size()); + let page_tables = pt_buf.into_bytes(); + layout.ensure_page_tables_fit(page_tables.len()).unwrap(); + + let mut source_bytes = vec![0u8; 2 * host_page_size]; + source_bytes[..host_page_size].fill(0x11); + source_bytes[host_page_size..2 * host_page_size].fill(0x22); + let source = super::memory::tests::flat_memory( + &source_bytes, + &page_tables, + base, + hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()), + ); + let source_blob = source.layers()[0].clone(); + let managed = SnapshotMemoryBacking::from_snapshot(Arc::new(source)).unwrap(); + let scratch = ExclusiveSharedMemory::new(layout.get_scratch_size()).unwrap(); + let manager = SandboxMemoryManager::new(layout, managed, scratch, super::NextAction::None); + let (manager, _) = manager.build().unwrap(); + let reuses_source_layers = manager.shared_mem.reusable_layers().is_some(); + #[cfg(gdb)] + { + let (layer_index, offset) = manager.shared_mem.resolve(base, host_page_size).unwrap(); + manager + .shared_mem + .write_layer(layer_index, offset, &vec![0x44; host_page_size]) + .unwrap(); + } + let mut source_page = vec![0u8; host_page_size]; + manager + .shared_mem + .read_snapshot_gpa(base, &mut source_page) + .unwrap(); + manager + .scratch_mem + .copy_from_slice(&vec![0x33; host_page_size], scratch_offset) + .unwrap(); + + let captured = super::Snapshot::new( + &manager.shared_mem, + &manager.scratch_mem, + manager.layout, + LoadInfo::dummy(), + Vec::new(), + &[pt_base], + 0, + default_sregs(), + #[cfg(target_arch = "x86_64")] + Vec::new(), + super::NextAction::None, + 0, + 1, + HostFunctionDetails::default(), + None, + ) + .unwrap(); + + let layers = captured.snapshot_memory().layers(); + let delta_gpa = if reuses_source_layers { + assert_eq!(layers.len(), 2); + assert!(std::sync::Arc::ptr_eq(&layers[0], &source_blob)); + assert_eq!(layers[1].data_len(), host_page_size); + assert_eq!(layers[1].gpa_start(), base + (2 * host_page_size) as u64); + base + (2 * host_page_size) as u64 + } else { + assert_eq!(layers.len(), 1); + assert!(!std::sync::Arc::ptr_eq(&layers[0], &source_blob)); + assert_eq!(layers[0].data_len(), 2 * host_page_size); + assert_eq!(layers[0].gpa_start(), base); + base + host_page_size as u64 + }; + + // SAFETY: `captured` is immutable for the duration of this walk. + let mappings = unsafe { vmem::virt_to_phys(&captured, base, 3 * host_page_size as u64) } + .step_by(host_page_size / PAGE_SIZE) + .collect::>(); + assert_eq!(mappings.len(), 3); + assert_eq!(mappings[0].phys_base, base); + assert_eq!(mappings[1].phys_base, delta_gpa); + assert_eq!(mappings[2].phys_base, mappings[1].phys_base); + assert_eq!( + mappings[0].kind, + MappingKind::Basic(BasicMapping { + readable: true, + writable: false, + executable: true, + }) + ); + assert_eq!( + mappings[1].kind, + MappingKind::Cow(hyperlight_common::vmem::CowMapping { + readable: true, + executable: false, + }) + ); + let (delta_layer, delta_offset) = captured + .snapshot_memory() + .resolve(mappings[1].phys_base, host_page_size) + .unwrap(); + assert_eq!( + layers[delta_layer].memory().as_slice()[delta_offset..delta_offset + host_page_size], + vec![0x33; host_page_size] + ); + let (kept_layer, kept_offset) = captured + .snapshot_memory() + .resolve(base, host_page_size) + .unwrap(); + assert_eq!( + layers[kept_layer].memory().as_slice()[kept_offset..kept_offset + host_page_size], + source_page + ); + // The reused layer stays mapped whole, including the page the guest + // replaced. + let (replaced_layer, replaced_offset) = captured + .snapshot_memory() + .resolve(base + host_page_size as u64, host_page_size) + .unwrap(); + let replaced = vec![if reuses_source_layers { 0x22 } else { 0x33 }; host_page_size]; + assert_eq!( + layers[replaced_layer].memory().as_slice() + [replaced_offset..replaced_offset + host_page_size], + replaced + ); + } + + #[test] + fn capture_rejects_leaf_into_page_table_tail() { + let host_page_size = page_size::get(); + let cfg = crate::sandbox::SandboxConfiguration::default(); + let layout = SandboxMemoryLayout::new(cfg, 4096, 0x3000, None).unwrap(); + let pt_base = layout.get_pt_base_gpa(); + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let pt_buf = GuestPageTableBuffer::new(pt_base as usize); + // SAFETY: The page-aligned mapping is written to a local page-table + // buffer with no concurrent access. + unsafe { + vmem::map( + &pt_buf, + Mapping { + phys_base: base + host_page_size as u64, + virt_base: base, + len: PAGE_SIZE as u64, + kind: MappingKind::Basic(BasicMapping { + readable: true, + writable: false, + executable: false, + }), + }, + ); + } + super::map_specials(&pt_buf, layout.get_scratch_size()); + let page_tables = pt_buf.into_bytes(); + layout.ensure_page_tables_fit(page_tables.len()).unwrap(); + + let bytes = vec![0u8; host_page_size]; + let source = super::memory::tests::flat_memory( + &bytes, + &page_tables, + base, + hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()), + ); + let manager = SandboxMemoryManager::new( + layout, + SnapshotMemoryBacking::from_snapshot(Arc::new(source)).unwrap(), + ExclusiveSharedMemory::new(layout.get_scratch_size()).unwrap(), + super::NextAction::None, + ); + let (manager, _) = manager.build().unwrap(); + + let result = super::Snapshot::new( + &manager.shared_mem, + &manager.scratch_mem, + manager.layout, + LoadInfo::dummy(), + Vec::new(), + &[pt_base], + 0, + default_sregs(), + #[cfg(target_arch = "x86_64")] + Vec::new(), + super::NextAction::None, + 0, + 1, + HostFunctionDetails::default(), + None, + ); + let error = match result { + Ok(_) => panic!("capture unexpectedly accepted a page-table-tail leaf"), + Err(error) => error, + }; + + assert!( + error + .to_string() + .contains("snapshot leaf names unbacked GPA") + ); + } + + #[test] + #[cfg(not(unshared_snapshot_mem))] + fn capture_rejects_delta_exceeding_layer_limit() { + let host_page_size = page_size::get(); + let cfg = crate::sandbox::SandboxConfiguration::default(); + let layout = SandboxMemoryLayout::new(cfg, 4096, 0x3000, None).unwrap(); + let pt_base = layout.get_pt_base_gpa(); + let scratch_offset = layout.get_scratch_size() / 2; + let scratch_gpa = hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()) + + scratch_offset as u64; + let base = SandboxMemoryLayout::BASE_ADDRESS as u64; + let max_layers = super::memory::MAX_SNAPSHOT_LAYERS; + + let pt_buf = GuestPageTableBuffer::new(pt_base as usize); + // One page in each of `max_layers` source layers, then one scratch page + // that needs a new layer. + for index in 0..=max_layers { + let phys_base = if index < max_layers { + base + (index * host_page_size) as u64 + } else { + scratch_gpa + }; + // SAFETY: The page-aligned mapping is written to a local page-table + // buffer with no concurrent access. + unsafe { + vmem::map( + &pt_buf, + Mapping { + phys_base, + virt_base: base + (index * host_page_size) as u64, + len: host_page_size as u64, + kind: MappingKind::Basic(BasicMapping { + readable: true, + writable: index == max_layers, + executable: false, + }), + }, + ); + } + } + super::map_specials(&pt_buf, layout.get_scratch_size()); + let page_tables = pt_buf.into_bytes(); + layout.ensure_page_tables_fit(page_tables.len()).unwrap(); + + let scratch_base = hyperlight_common::layout::scratch_base_gpa(layout.get_scratch_size()); + let source_layers = (0..max_layers) + .map(|index| { + let storage = crate::mem::shared_mem::ReadonlySharedMemory::from_bytes( + &vec![index as u8; host_page_size], + host_page_size, + ) + .unwrap(); + Arc::new( + super::SnapshotBlob::new( + storage, + base + (index * host_page_size) as u64, + 0, + scratch_base, + ) + .unwrap(), + ) + }) + .collect::>(); + let source = super::SnapshotMemory::new( + source_layers.clone(), + super::memory::tests::page_tables_from_bytes(&page_tables), + ) + .unwrap(); + let managed = SnapshotMemoryBacking::from_snapshot(Arc::new(source)).unwrap(); + let scratch = ExclusiveSharedMemory::new(layout.get_scratch_size()).unwrap(); + let manager = SandboxMemoryManager::new(layout, managed, scratch, super::NextAction::None); + let (mut manager, _) = manager.build().unwrap(); + manager + .scratch_mem + .copy_from_slice(&vec![0x55; host_page_size], scratch_offset) .unwrap(); + + let result = manager.snapshot( + Vec::new(), + &[pt_base], + 0, + default_sregs(), + #[cfg(target_arch = "x86_64")] + Vec::new(), + super::NextAction::None, + HostFunctionDetails::default(), + ); + let error = match result { + Ok(_) => panic!("capture unexpectedly exceeded the layer limit"), + Err(error) => error, + }; + + assert!(error.to_string().contains("layer count 31 exceeds 30")); + assert_eq!(manager.snapshot_count, 0); + assert!( + manager + .shared_mem + .layers() + .iter() + .zip(&source_layers) + .all(|(layer, source)| Arc::ptr_eq(layer, source)) + ); + assert_eq!( + manager.scratch_mem.read::(scratch_offset).unwrap(), + 0x55 + ); } } diff --git a/src/hyperlight_host/src/sandbox/snapshot/tripwires.rs b/src/hyperlight_host/src/sandbox/snapshot/tripwires.rs index 46302ed05..bc5e2f937 100644 --- a/src/hyperlight_host/src/sandbox/snapshot/tripwires.rs +++ b/src/hyperlight_host/src/sandbox/snapshot/tripwires.rs @@ -16,9 +16,9 @@ use super::file::{ SNAPSHOT_ABI_VERSION, }; -const EXPECTED_ABI_VERSION: u32 = 6; -const EXPECTED_MT_CONFIG: &str = "application/vnd.hyperlight.snapshot.config.v3+json"; -const EXPECTED_MT_SNAPSHOT: &str = "application/vnd.hyperlight.snapshot.memory.v1"; +const EXPECTED_ABI_VERSION: u32 = 7; +const EXPECTED_MT_CONFIG: &str = "application/vnd.hyperlight.snapshot.config.v4+json"; +const EXPECTED_MT_SNAPSHOT: &str = "application/vnd.hyperlight.snapshot.memory.v2"; const EXPECTED_MT_TRANSPORT: &str = "application/vnd.hyperlight.snapshot.transport.v1"; const EXPECTED_OCI_LAYOUT_VERSION: &str = "1.0.0"; diff --git a/src/hyperlight_host/src/sandbox/trace/mem_profile.rs b/src/hyperlight_host/src/sandbox/trace/mem_profile.rs index f4c92a6de..4b33e06cf 100644 --- a/src/hyperlight_host/src/sandbox/trace/mem_profile.rs +++ b/src/hyperlight_host/src/sandbox/trace/mem_profile.rs @@ -7,9 +7,6 @@ use fallible_iterator::FallibleIterator; use framehop::Unwinder; use crate::hypervisor::regs::CommonRegisters; -#[cfg(not(unshared_snapshot_mem))] -use crate::mem::layout::ReadableSharedMemory; -use crate::mem::layout::SandboxMemoryLayout; use crate::mem::mgr::SandboxMemoryManager; use crate::mem::shared_mem::HostSharedMemory; use crate::sandbox::outb::HandleOutbError; @@ -88,10 +85,7 @@ impl MemTraceInfo { let mut buf: [u8; 8] = [0u8; 8]; mem_mgr .shared_mem - .copy_to_slice( - &mut buf, - (addr - SandboxMemoryLayout::BASE_ADDRESS as u64) as usize, - ) + .read_snapshot_gpa(addr, &mut buf) .map_err(|_| ())?; Ok(u64::from_ne_bytes(buf)) }; diff --git a/src/hyperlight_host/src/sandbox/uninitialized.rs b/src/hyperlight_host/src/sandbox/uninitialized.rs index 267f87325..785e00a78 100644 --- a/src/hyperlight_host/src/sandbox/uninitialized.rs +++ b/src/hyperlight_host/src/sandbox/uninitialized.rs @@ -18,7 +18,7 @@ use crate::func::{ParameterTuple, SupportedReturnType}; use crate::log_build_details; use crate::mem::memory_region::{DEFAULT_GUEST_BLOB_MEM_FLAGS, MemoryRegionFlags}; use crate::mem::mgr::SandboxMemoryManager; -use crate::mem::shared_mem::{ExclusiveSharedMemory, SharedMemory}; +use crate::mem::shared_mem::ExclusiveSharedMemory; use crate::sandbox::SandboxConfiguration; use crate::{Result, Sandbox, new_error}; @@ -269,6 +269,10 @@ impl UninitializedSandbox { /// the sandbox's primary shared memory region (`BASE_ADDRESS` to /// `BASE_ADDRESS + shared_mem_size`). /// + /// [`Sandbox::snapshot`] copies only the pages of this region that the + /// guest has mapped into its page tables, during initialization or a later + /// call, then removes the region. Pages the guest has not mapped are lost. + /// /// Returns the length of the mapping in bytes. #[instrument(err(Debug), skip(self, file_path, guest_base), parent = Span::current())] pub fn map_file_cow( @@ -278,7 +282,7 @@ impl UninitializedSandbox { ) -> crate::Result { // Validate that guest_base is outside the sandbox's primary memory slot. // (Full range check happens after prepare_file_cow when we know the mapped size.) - let shared_size = self.mgr.shared_mem.mem_size() as u64; + let shared_size = self.mgr.shared_mem.gpa_span_len() as u64; let base_addr = crate::mem::layout::SandboxMemoryLayout::BASE_ADDRESS as u64; let prepared = super::file_mapping::prepare_file_cow(file_path, guest_base)?; @@ -332,7 +336,7 @@ impl UninitializedSandbox { /// This is useful for placing file mappings at guest physical addresses /// that don't overlap the primary shared memory slot. pub fn shared_mem_size(&self) -> usize { - self.mgr.shared_mem.mem_size() + self.mgr.shared_mem.gpa_span_len() } /// Sets the maximum log level for guest code execution. diff --git a/src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs b/src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs index 164a276c8..91ce709f9 100644 --- a/src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs +++ b/src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs @@ -8,7 +8,7 @@ //! publish. See `docs/snapshot-versioning.md`. /// Goldens version, a `vMAJOR.MINOR` string. -pub(crate) const GOLDENS_VERSION: &str = "v6.0"; +pub(crate) const GOLDENS_VERSION: &str = "v7.0"; /// Old majors kept loadable through a compatibility path, verified /// alongside `GOLDENS_VERSION`. A backwards-compatible break (Option 2) diff --git a/src/tests/rust_guests/simpleguest/src/main.rs b/src/tests/rust_guests/simpleguest/src/main.rs index 3753b276e..66fa2709e 100644 --- a/src/tests/rust_guests/simpleguest/src/main.rs +++ b/src/tests/rust_guests/simpleguest/src/main.rs @@ -1149,6 +1149,25 @@ fn read_from_user_memory(num: u64, expected: Vec) -> Result> { Ok(bytes) } +#[guest_function("MapZeroInitPage")] +fn map_zero_init_page(base: u64) -> bool { + // SAFETY: The test supplies an unused, page-aligned GVA. + unsafe { + hyperlight_guest_bin::paging::map_region( + 0, + base as *mut u8, + PAGE_SIZE as u64, + MappingKind::ZeroInit(BasicMapping { + readable: true, + writable: true, + executable: false, + }), + ); + hyperlight_guest_bin::paging::barrier::first_valid_same_ctx(); + } + true +} + #[guest_function("ReadMappedBuffer")] fn read_mapped_buffer(base: u64, len: u64, do_map: bool) -> Vec { let base = base as usize as *const u8;