feat: preserve virtual queue buffers across snapshots by andreiltd · Pull Request #1891 · hyperlight-dev/hyperlight

andreiltd · 2026-10-06T10:17:53Z

This patch lets the guest keep referencing virtqueue data after a snapshot. Retained data is any guest value that keeps a transport pool slot alive:

H2G ByteChunks, which hold a host request
host replies, which become owner-backed guest Bytes

During normal execution:

the guest reads through the alias, which maps to the pool memory in scratch
dropping the last owner frees the slot for reuse

At checkpoint (prepare_snapshot):

the guest reclaims finished G2H work and resets both producers. After that, only retained slots are still live
prune_pool records the live slot ranges and unmaps every alias page they don't touch
H2G is refilled from free slots up to the queue size, including slots that share a page with retained ones

At snapshot:

the snapshot walker skips the scratch map but includes the aliases. Retained pages are therefore captured as ordinary memory

After checkpoint, the first transport entry (maybe_refresh):

maps missing alias pages to scratch
after restore, copies only the recorded retained bytes from snapshot alias pages into scratch, then remaps those pages to scratch. This preserves host writes in neighboring slots, such as the first request
copies nothing when no buffers are retained

Closes #1884

Copilot

Copilot review overview

🟡 Changes recommended

Alias page-table sizing and zero-capacity checkpoint cases can prevent supported sandboxes from initializing or continuing after restore.

Review effort: Balanced
Findings: 2 · 1

Open (3)

Reserve scratch space for alias page tables · New
Preserve control-call capacity when pruning H2G slots · New
Fix typo in snapshot failure message · New

What changed in this PR

Preserves transport-backed guest buffers across snapshot capture and restore using stable aliases and pinned pages.

Changes:

Adds alias mapping, pruning, and mailbox checkpoint states.
Allows partially prefilled H2G rings in snapshots.
Adds restore tests, benchmarks, ABI 6, and documentation.

File	Description
`src/tests/rust_guests/simpleguest/src/main.rs`	Adds retained buffer readers.
`src/tests/rust_guests/Cargo.lock`	Locks the new dependency.
`src/tests/c_guests/c_simpleguest/main.c`	Adds C retention fixtures.
`src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs`	Advances goldens to v6.
`src/hyperlight_host/tests/sandbox_host_tests.rs`	Exercises repeated fragmented calls.
`src/hyperlight_host/tests/integration_test.rs`	Tests retained data restoration.
`src/hyperlight_host/src/sandbox/uninitialized_evolve.rs`	Completes initial checkpoints.
`src/hyperlight_host/src/sandbox/snapshot/tripwires.rs`	Pins ABI 6.
`src/hyperlight_host/src/sandbox/snapshot/file/transport.rs`	Updates malformed transport testing.
`src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs`	Bumps the snapshot ABI.
`src/hyperlight_host/src/sandbox/snapshot/file/config.rs`	Updates pinned schemas.
`src/hyperlight_host/src/sandbox/snapshot/file_tests.rs`	Updates ABI rejection assertions.
`src/hyperlight_host/src/sandbox/initialized.rs`	Enables retained-buffer snapshots.
`src/hyperlight_host/src/sandbox/config.rs`	Clarifies scratch usage.
`src/hyperlight_host/src/mem/virtq/tests.rs`	Tests mailbox and partial prefill restore.
`src/hyperlight_host/src/mem/virtq/mod.rs`	Accepts bounded H2G prefixes.
`src/hyperlight_host/src/mem/mgr.rs`	Implements mailbox checkpoint states.
`src/hyperlight_host/benches/benchmarks.rs`	Benchmarks retained-buffer restores.
`src/hyperlight_guest/src/transport/mod.rs`	Exposes refresh and checkpoint entry points.
`src/hyperlight_guest/src/transport/mem.rs`	Maps completed buffers through aliases.
`src/hyperlight_guest/src/transport/context.rs`	Prunes and refreshes pool aliases.
`src/hyperlight_guest/src/transport/backing.rs`	Implements alias paging and pinning.
`src/hyperlight_guest/Cargo.toml`	Adds fixed bitset support.
`src/hyperlight_guest_bin/src/transport.rs`	Updates transport imports.
`src/hyperlight_guest_bin/src/lib.rs`	Checkpoints after guest initialization.
`src/hyperlight_guest_bin/src/guest_function/call.rs`	Refreshes aliases before dispatch.
`src/hyperlight_common/src/virtq/ring/canonical.rs`	Adds prefix validation.
`src/hyperlight_common/src/virtq/pool/tests.rs`	Tests slot exclusion behavior.
`src/hyperlight_common/src/virtq/pool/slot.rs`	Adds persistent slot exclusions.
`src/hyperlight_common/src/virtq/pool/fuzz.rs`	Removes obsolete free-slot checks.
`src/hyperlight_common/src/transport.rs`	Defines mailbox wire values.
`docs/virtio-host-guest-communication.md`	Documents retained-buffer snapshots.
`docs/snapshot-versioning.md`	Documents ABI 6.
`CHANGELOG.md`	Records the feature and ABI break.
`Cargo.lock`	Locks the guest dependency.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ludfjig

lgtm. Does this impact the performance of the first guest call after a restore? Do you happen to have any perf numers?

syntactically

This looks pretty good, but I think it could be simplified a lot by removing the invariant that the whole "pool" range is 1:1 mapped early, and instead doing the mappings when necessary. (There is a slight problem in ensuring that dropping a view/"returning" a buffer after snapshot does not actually return said buffer, since it is not scratch backed anymore---but this I think can be solved by keeping track of the generation of the view and no-op'ing it (or rather, changing it to just unmap the alias pages) if the generation is different). Then there is no need for the prune_pool/map_pool dance across restore.

I guess that lazily allocating the aliases (and only to guest-owned/used buffers) like I suggest could be a tiny bit worse for bytechunks performance, but because of the relatively light tlb invalidations needed, I would guess it would not really be super noticeable in the current workload, and it would trade off by needing a lot less work for initialisation. So i would expect that to be an overall win, probably.

syntactically · 2026-10-08T09:31:09Z


+### Retained virtual addresses
+
+Each pool has a stable virtual alias range, mapped eagerly at initialization.


Why map eagerly instead of when actually using a buffer? I think if doing the latter one can avoid the need to explicitly look for and unmap the buffers on checkpoint. And, it means we do not need to spend the memory on page tables (alias_table_len) unless references are actually created for a given buffer (which I think only happens in the bytechunks case right now iirc?)

Perf being main reason, although please see the numbers below.

syntactically · 2026-10-08T09:39:28Z

+
+/// Translate a scratch address into its guest physical address.
+const fn scratch_gpa(addr: u64) -> u64 {
+    addr - (SCRATCH_TOP_GVA - SCRATCH_TOP_GPA) as u64


Do we need to make this 1:1 assumption here (i.e. is walking the range too slow)? I don't think any other code makes that assumption, so maybe mention it in paging or layout docs if we are going to start making that an invariant.

I can try walking here, yes.

syntactically · 2026-10-08T09:47:53Z

+
+    /// Prepare pool aliases after checkpointing or a generation change.
+    ///
+    /// After restore, retained slots are copied from captured memory into


Why is this copying back into the original scratch pool? The guest users of the page are already accessing it through their alias mappings.

I envision this functionality being useful mostly to allow the lifecycle of large data blobs to be independent from snapshot/restore (for example to allow replacing the init data section and as a more portable alternative to loading wasm binaries via mapping before the first snapshot). But, I think this large copy on restore would be quite bad for performance in that case?

Pool slots that are less then page size would make neigbor slots unusable if they share a fixed alias page.

I'm still not clear why it is helpful to copy here. Everything accessing the retained data is doing so through the old alias which has now been disassociated from the buffer pool backing---why do we want to re-associate them?

andreiltd · 2026-10-08T14:08:10Z

Hey @syntactically ! I agree that the lazy model is a bit cleaner: GuestMapping has clear responsibility, own an alias while bytes exist, then release it. Restored value remains on snapshotted memory. Compared to eager approach the cost of mapping is per call rather than once on restore. It scales with buffer size. Here are some numbers:

+-------------------------------------------------+
¦ buffer size ¦ eager (us) ¦ lazy (us) ¦ overhead ¦
+-------------+------------+-----------+----------¦
¦ 64 B        ¦      14.19 ¦     14.51 ¦    +2.2% ¦
+-------------+------------+-----------+----------¦
¦ 4 KiB       ¦      15.11 ¦     15.59 ¦    +3.2% ¦
+-------------+------------+-----------+----------¦
¦ 12 KiB      ¦      16.81 ¦     17.50 ¦    +4.1% ¦
+-------------+------------+-----------+----------¦
¦ 64 KiB      ¦      26.39 ¦     28.96 ¦    +9.7% ¦
+-------------+------------+-----------+----------¦
¦ 1 MiB       ¦     201.53 ¦    241.06 ¦   +19.6% ¦
+-------------------------------------------------+

I think it could be simplified a lot by removing the invariant that the whole "pool" range is 1:1

I think I disagree. The complexity will move but I don't see how lazy mapping will simplify the implementation drastically:

we need alias allocator machinery for finding unused alias for new mapping because alias can already be occupied by retained buffer. The annoying part is that we cannot get away with simple bump pointer, because unmapping does not reclaim page tables so touching new virtual regions grows page tables and we will run out of scratch way before we exhaust virtual address space. The allocator need to maintain the list of freed regions, coalesce them and also maintain the next pointer. Nothing too crazy, I think we can even use vm-allocator crate for that but eager mapping avoid that all together.
pool and mapping need to gain epoch semantics. Again nothing too crazy but we need a mechanism to detach old leases from the pool free list so that retained buffers that already have backing from snapshot do not take space in the pool.

That being said, I we think the numbers are acceptable, I'm happy to give the lazy mapping a go?

syntactically · 2026-10-08T16:48:34Z

Hey @syntactically ! I agree that the lazy model is a bit cleaner: GuestMapping has clear responsibility, own an alias while bytes exist, then release it. Restored value remains on snapshotted memory. Compared to eager approach the cost of mapping is per call rather than once on restore. It scales with buffer size. Here are some numbers:
+-------------------------------------------------+
¦ buffer size ¦ eager (us) ¦ lazy (us) ¦ overhead ¦
+-------------+------------+-----------+----------¦
¦ 64 B        ¦      14.19 ¦     14.51 ¦    +2.2% ¦
+-------------+------------+-----------+----------¦
¦ 4 KiB       ¦      15.11 ¦     15.59 ¦    +3.2% ¦
+-------------+------------+-----------+----------¦
¦ 12 KiB      ¦      16.81 ¦     17.50 ¦    +4.1% ¦
+-------------+------------+-----------+----------¦
¦ 64 KiB      ¦      26.39 ¦     28.96 ¦    +9.7% ¦
+-------------+------------+-----------+----------¦
¦ 1 MiB       ¦     201.53 ¦    241.06 ¦   +19.6% ¦
+-------------------------------------------------+

I am surprised the overhead is quite so large, but for a lot of the use cases we have that are just restore+call I think it is just shifting the overhead from restore to call?

I think it could be simplified a lot by removing the invariant that the whole "pool" range is 1:1

I think I disagree. The complexity will move but I don't see how lazy mapping will simplify the implementation drastically:

we need alias allocator machinery for finding unused alias for new mapping because alias can already be occupied by retained buffer. The annoying part is that we cannot get away with simple bump pointer, because unmapping does not reclaim page tables so touching new virtual regions grows page tables and we will run out of scratch way before we exhaust virtual address space. The allocator need to maintain the list of freed regions, coalesce them and also maintain the next pointer. Nothing too crazy, I think we can even use vm-allocator crate for that but eager mapping avoid that all together.

Good point about the lack of reclamation for the old page tables limiting the bump allocator semantics. They do get reclaimed on snapshot, but I could certainly see it being a problem in some workloads. We can't just try to switch to reclaiming them easily either, unless we change the whole physical page allocator to have a freelist, hm..

I see what you mean about how this basically makes the allocation for VA space used for data copies 1:1 with the buffer allocation itself, simplifying that problem, but I think it is quite undesirable to have the old buffers taking up space in the new pool. Apart from everything else, this means if you do send data + snapshot retaining data + send data you have to size the buffer pool appropriately for 2 rounds of data, which increases the size of the scratch region, which is quite unpleasant (it's important to minimize the size of the scratch region for a given workload, since the scratch region has to be zeroed on restore and directly contributes to restore latency).

pool and mapping need to gain epoch semantics. Again nothing too crazy but we need a mechanism to detach old leases from the pool free list so that retained buffers that already have backing from snapshot do not take space in the pool.

I think this is important to have anyway. In fact it seems like a big /dis/advantage to me that in the current PR we end up taking up space in the live descriptor pool for data that was read more than a snapshot ago (which I imagine is likely to survive for some time, a la the generational hypothesis). And, having that association means that we end up having the corresponding scratch pages which there is nothing obvious to do with, so we end up (I think unnecessarily?) copying the data into them just to keep them in sync. I do think it is pretty important for perf to avoid that extra copy that is happening on restore and I don't think should be necessary.

andreiltd · 2026-10-08T21:38:49Z

I believe last commit addresses all the issues.

Keep transport pools and aliases stable. Pin pages backing live buffers and exclude overlapping slots until a checkpoint finds those pages empty. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

hyperlight-gh-bot · 2026-10-12T13:46:14Z

Benchmark Results

Measured commit: 0ca6f1d32b0e
Baseline commit: dcb53c04a5a7

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`760.33 ns` (➖ 1.00x faster)
`vec_bytes`	`594.51 ns` (➖ 1.03x slower)
		`379.41 µs` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`524.25 ns` (➖ 1.00x faster)
`65536`	`141.57 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`82.09 ms` (➖ 1.01x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.71 ns` (➖ 1.02x faster)	`7.75 ns` (➖ 1.00x slower)	`7.72 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`93.08 µs` (➖ 1.00x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.25 µs` (➖ 1.03x slower)	`7.22 µs` (➖ 1.14x faster)
`65536`	`2.06 µs` (➖ 1.06x slower)	`1.94 µs` (➖ 1.08x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`6.35 µs` (➖ 1.01x faster)	`6.27 µs` (➖ 1.00x slower)
`8192`	`1.12 µs` (➖ 1.11x slower)	`1.09 µs` (➖ 1.01x slower)
`262144`		`27.99 µs` (➖ 1.02x slower)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`812.57 ns` (➖ 1.02x slower)
`vec_bytes`	`624.83 ns` (➖ 1.10x faster)
		`750.27 µs` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`637.92 ns` (➖ 1.07x slower)
`65536`	`159.66 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`85.41 ms` (➖ 1.03x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`8.26 ns` (➖ 1.00x slower)	`8.21 ns` (➖ 1.00x slower)	`8.21 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`52.60 µs` (➖ 1.03x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.90 µs` (➖ 1.00x slower)	`8.88 µs` (➖ 1.00x slower)
`65536`	`2.54 µs` (➖ 1.01x slower)	`2.53 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.42 µs` (➖ 1.01x faster)	`8.28 µs` (➖ 1.01x faster)
`8192`	`907.21 ns` (➖ 1.00x slower)	`910.13 ns` (➖ 1.08x slower)
`262144`		`34.32 µs` (➖ 1.06x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`950.85 ns` (➖ 1.01x faster)
`vec_bytes`	`709.12 ns` (➖ 1.03x slower)
		`316.00 µs` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`675.98 ns` (➖ 1.01x slower)
`65536`	`188.54 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`60.40 ms` (➖ 1.01x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.25 ns` (➖ 1.03x slower)	`9.93 ns` (➖ 1.00x slower)	`9.91 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`85.98 µs` (➖ 1.01x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`9.09 µs` (➖ 1.02x faster)	`9.00 µs` (➖ 1.01x faster)
`65536`	`2.65 µs` (➖ 1.02x faster)	`2.45 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.22 µs` (➖ 1.00x faster)	`8.05 µs` (➖ 1.01x faster)
`8192`	`1.47 µs` (➖ 1.10x slower)	`1.35 µs` (➖ 1.02x faster)
`262144`		`35.32 µs` (➖ 1.01x slower)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`911.24 ns` (➖ 1.03x faster)
`vec_bytes`	`651.79 ns` (➖ 1.01x faster)
		`719.66 µs` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`645.35 ns` (➖ 1.04x slower)
`65536`	`170.01 ns` (➖ 1.03x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`70.60 ms` (➖ 1.06x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.42 ns` (➖ 1.02x faster)	`9.18 ns` (➖ 1.04x slower)	`9.60 ns` (➖ 1.09x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`45.99 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.94 µs` (➖ 1.00x slower)	`7.95 µs` (➖ 1.00x slower)
`65536`	`2.27 µs` (➖ 1.02x faster)	`2.25 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.55 µs` (➖ 1.01x slower)	`7.51 µs` (➖ 1.00x faster)
`8192`	`942.23 ns` (➖ 1.00x faster)	`920.83 ns` (➖ 1.02x slower)
`262144`		`40.77 µs` (➖ 1.04x slower)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.14 µs` (➖ 1.01x slower)
`vec_bytes`	`839.16 ns` (➖ 1.11x slower)
		`1.87 ms` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`801.12 ns` (➖ 1.02x slower)
`65536`	`229.66 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`80.02 ms` (➖ 1.02x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`11.80 ns` (➖ 1.11x slower)	`12.46 ns` (➖ 1.14x slower)	`10.97 ns` (➖ 1.04x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`656.26 µs` (➖ 1.22x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.97 µs` (➖ 1.03x slower)	`8.79 µs` (➖ 1.02x slower)
`65536`	`2.54 µs` (➖ 1.00x faster)	`2.60 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.78 µs` (➖ 1.02x faster)	`8.82 µs` (➖ 1.02x faster)
`8192`	`1.43 µs` (➖ 1.06x faster)	`1.43 µs` (➖ 1.04x slower)
`262144`		`45.57 µs` (➖ 1.21x slower)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.20 µs` (➖ 1.00x slower)
`vec_bytes`	`774.27 ns` (➖ 1.03x faster)
		`3.71 ms` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`734.04 ns` (➖ 1.04x faster)
`65536`	`211.66 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`120.77 ms` (➖ 1.19x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.97 ns` (➖ 1.03x faster)	`9.98 ns` (➖ 1.14x faster)	`9.95 ns` (➖ 1.06x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`505.04 µs` (➖ 1.08x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.80 µs` (➖ 1.01x slower)	`7.90 µs` (➖ 1.02x slower)
`65536`	`2.36 µs` (➖ 1.02x slower)	`2.37 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.55 µs` (➖ 1.04x faster)	`8.51 µs` (➖ 1.13x faster)
`8192`	`1.12 µs` (➖ 1.14x faster)	`1.11 µs` (➖ 1.03x slower)
`262144`		`42.42 µs` (➖ 1.06x faster)

_{Reported by cargo ci bench-report --candidate run:38252109488 --config-file bench_report.toml.}

andreiltd · 2026-10-12T14:37:51Z

There is an unrelated issue that was causing macos tests to hang in this PR. As a workaround I had to increase the page reservation for exception handler. See #1925.

syntactically

@andreiltd It looks to me like this makes allocation lazy, but keeps the 1:1 mapping between the alias GVA layout and the buffer pool layout, rather than adding a GVA allocator for the alias space as I think you mentioned in the previous comment? Correct me if I'm wrong.

I think the most important reason to be lazy at all is in order to avoid that 1:1 invariant so that we don't have to have idle buffer slots sitting around consuming scarce, expensive, scratch memory when they are totally useless (since they are really just acting as reservations in another address space).

Since all the buffer sizes are fixed right now, if you want that allocator to be simple, I think you can use the same approach as for the main pool of dividing the relevant alias gva space into buffer-sized chunks and having a bitmap track usage of them, but just have that bitmap be independent of the usage in the main pool?

syntactically · 2026-10-12T16:42:56Z

+    ) -> Result<(), AllocError> {
+        let generation = self.generation.checked_add(1).ok_or(AllocError::Overflow)?;
+
+        for addr in reserved.clone() {


What is this doing?

Checking if reservation addresses passed are valid with allocation_len(addr)? . Just reused this function as it already do all the checking, although I can inline the checks here to make this code more obvious.

Maybe I misunderstood what these reservations were being used for, sorry. Does this check need to happen always or should it be e.g. a debug_assert!?

andreiltd · 2026-10-12T21:36:45Z

It looks to me like this makes allocation lazy, but keeps the 1:1 mapping between the alias GVA layout and the buffer pool layout, rather than adding a GVA allocator for the alias space as I think you mentioned in the previous comment?

Hey @syntactically. No, that's not how it works. The alias layout is now independent of pool and it is no longer a fixed offset. The lazy implementation reserves a separate page aligned range for each GuestMapping and restore gives the retained buffer backing independent of scratch. Pool and retained buffers lifetimes are separated.

I think the most important reason to be lazy at all is in order to avoid that 1:1 invariant so that we don't have to have idle buffer slots sitting around consuming scarce.

Yes, there is a test specifically checking if queue is still at full capacity when retained buffers are alive:
https://github.com/hyperlight-dev/hyperlight/pull/1891/changes#diff-0653ec70e87c8e9f58e66156f5d49924fe8ffe4501aafe6b99de943adb77363cR1003

So the code recovers usable virtq pool capacity, rather than shrinking the scratch.

Since all the buffer sizes are fixed right now, if you want that allocator to be simple, I think you can use the same approach as for the main pool of dividing the relevant alias gva space into buffer-sized chunks and having a bitmap track usage of them, but just have that bitmap be independent of the usage in the main pool?

I think using fixedbitset here is an alternative, I don't necessarily think it is simpler though. Buffer sizes are configurable. The bitset allocator will need to find contiguous runs for multi page buffers. I also don't think current allocator is logic is too crazy with just two concise functions? I'm open to try it but most likely in a follow up PR.

syntactically

Hey @syntactically. No, that's not how it works. The alias layout is now independent of pool and it is no longer a fixed offset. The lazy implementation reserves a separate page aligned range for each GuestMapping and restore gives the retained buffer backing independent of scratch. Pool and retained buffers lifetimes are separated.

Oh, whoops, I must have misread something, not sure how I missed that. I see it now, sorry!! In that case, I just have one more minor question (see inline).

Yes, there is a test specifically checking if queue is still at full capacity when retained buffers are alive: https://github.com/hyperlight-dev/hyperlight/pull/1891/changes#diff-0653ec70e87c8e9f58e66156f5d49924fe8ffe4501aafe6b99de943adb77363cR1003

Perfect, thanks!

I think using fixedbitset here is an alternative, I don't necessarily think it is simpler though. Buffer sizes are configurable. The bitset allocator will need to find contiguous runs for multi page buffers. I also don't think current allocator is logic is too crazy with just two concise functions? I'm open to try it but most likely in a follow up PR.

No this seems totally reasonable, just wanted to throw it out there as an alternative.

syntactically · 2026-10-12T22:11:48Z

+            lower.allocated.clear();
+        }
+
+        for addr in reserved {


What are these reservations needed for?

syntactically · 2026-10-12T22:14:25Z

+before guest entry. Pre-initialization snapshots use normal guest startup.

-## Retention mailbox
+The first request fits the H2G capacity posted at checkpoint. Retained slots


This comment is not true anymore since the retained slots do not reduce buffer pool capacity anymore I think?

syntactically · 2026-10-12T22:17:21Z

+    ) -> Result<(), AllocError> {
+        let generation = self.generation.checked_add(1).ok_or(AllocError::Overflow)?;
+
+        for addr in reserved.clone() {


Maybe I misunderstood what these reservations were being used for, sorry. Does this check need to happen always or should it be e.g. a debug_assert!?

andreiltd added this pull request to stack #1892 October 6, 2026 10:17

andreiltd added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. regen-goldens Regenerate snapshot golden fixtures labels Oct 6, 2026

This comment has been minimized.

Sign in to view

andreiltd force-pushed the virtq-lazy-pool-generations branch from f718561 to 36eda9c Compare October 6, 2026 11:29

andreiltd marked this pull request as ready for review October 6, 2026 11:45

Copilot AI balanced review requested due to automatic review settings October 6, 2026 11:45

andreiltd requested review from danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners October 6, 2026 11:45

Copilot started reviewing on behalf of andreiltd October 6, 2026 11:48 View session

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Comment thread src/hyperlight_guest/src/transport/backing.rs Outdated

Comment thread src/hyperlight_guest/src/transport/backing.rs Outdated

Comment thread src/hyperlight_guest_bin/src/guest_function/call.rs Outdated

This comment has been minimized.

Sign in to view

andreiltd marked this pull request as draft October 6, 2026 14:38

andreiltd force-pushed the virtq-lazy-pool-generations branch from 36eda9c to 534cad5 Compare October 6, 2026 16:10

andreiltd marked this pull request as ready for review October 6, 2026 16:22

This comment has been minimized.

Sign in to view

ludfjig previously approved these changes Oct 7, 2026

View reviewed changes

andreiltd dismissed ludfjig’s stale review via 9395642 October 7, 2026 09:12

andreiltd force-pushed the virtq-lazy-pool-generations branch from 534cad5 to 9395642 Compare October 7, 2026 09:12

andreiltd removed this pull request from stack #1892 October 7, 2026 09:26

andreiltd force-pushed the virtq-lazy-pool-generations branch from 9395642 to d2751f4 Compare October 7, 2026 09:29

andreiltd changed the base branch from virtq-unmap-page to main October 7, 2026 09:29

andreiltd force-pushed the virtq-lazy-pool-generations branch from d2751f4 to 255b9ab Compare October 7, 2026 09:46

andreiltd force-pushed the virtq-lazy-pool-generations branch from ad8fc12 to 9de91ff Compare October 8, 2026 08:39

This comment has been minimized.

Sign in to view

syntactically requested changes Oct 8, 2026

View reviewed changes

andreiltd dismissed ludfjig’s stale review via 3930fd8 October 8, 2026 21:12

andreiltd force-pushed the virtq-lazy-pool-generations branch 2 times, most recently from f1dd2a9 to 0d0576f Compare October 8, 2026 21:35

This comment has been minimized.

Sign in to view

ludfjig previously approved these changes Oct 9, 2026

View reviewed changes

Comment thread src/hyperlight_common/src/layout.rs Outdated

Comment thread src/hyperlight_guest/Cargo.toml Outdated

andreiltd dismissed ludfjig’s stale review via 49e1c32 October 9, 2026 06:13

andreiltd force-pushed the virtq-lazy-pool-generations branch 3 times, most recently from db7b79c to f9fb01b Compare October 9, 2026 06:30

This comment has been minimized.

Sign in to view

andreiltd added 2 commits October 10, 2026 00:25

feat: preserve virtual queue buffers across snapshots

1600b4b

Keep transport pools and aliases stable. Pin pages backing live buffers and exclude overlapping slots until a checkpoint finds those pages empty. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat: map virtq buffers lazily

d1e7001

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd force-pushed the virtq-lazy-pool-generations branch from f9fb01b to d1e7001 Compare October 9, 2026 22:25

This comment has been minimized.

Sign in to view

andreiltd force-pushed the virtq-lazy-pool-generations branch 2 times, most recently from e63a419 to 86ee0c2 Compare October 12, 2026 12:15

fix: increase exception stack pages reservation

0ca6f1d

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

andreiltd force-pushed the virtq-lazy-pool-generations branch from 86ee0c2 to 0ca6f1d Compare October 12, 2026 12:55

syntactically requested changes Oct 12, 2026

View reviewed changes

syntactically approved these changes Oct 12, 2026

View reviewed changes

syntactically reviewed Oct 12, 2026

View reviewed changes


		### Retained virtual addresses

		Each pool has a stable virtual alias range, mapped eagerly at initialization.

Conversation

andreiltd commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

syntactically left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andreiltd Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andreiltd commented Oct 8, 2026

Uh oh!

syntactically commented Oct 8, 2026

Uh oh!

andreiltd commented Oct 8, 2026

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Oct 12, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

andreiltd commented Oct 6, 2026 •

edited

Loading

andreiltd Oct 8, 2026 •

edited

Loading

andreiltd commented Oct 12, 2026 •

edited

Loading