Skip to content

Write initial snapshot memory in place - #1927

Open
ludfjig wants to merge 1 commit into
ll/1-layered-snapshotsfrom
ll/2-lazy-sandbox-memory
Open

ludfjig wants to merge 1 commit into
ll/1-layered-snapshotsfrom
ll/2-lazy-sandbox-memory

Conversation

@ludfjig

@ludfjig ludfjig commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

New sandboxes skip copying their initial memory. Snapshot::from_env writes the guest code, PEB, init data and page tables in place, and the host populates the rest, mostly the heap, on first guest touch. Sandbox creation gets 33 to 88% faster for small to large heaps.

Benchmarks

Compared to the PR below.

  • MSHV: Linux 6.6 mshv.
  • KVM: Linux 6.17.
Benchmark MSHV KVM
sandboxes/create_initialized/small 4.6 ms → 3.09 ms (−33%) 5.12 ms → 2.92 ms (−43%)
sandboxes/create_initialized/medium 11.8 ms → 5.93 ms (−50%) 9.81 ms → 3.11 ms (−68%)
sandboxes/create_initialized/large 38.5 ms → 11 ms (−71%) 28.8 ms → 3.59 ms (−88%)
snapshot_files/cold_start_via_evolve/large 38.8 ms → 10.7 ms (−72%) 29.1 ms → 4.24 ms (−85%)

No benchmark regresses.

Full results

All benchmarks on both hosts (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change. A change needs agreement between a 0.5 s and a 3 s run, or a 3 s run alone.

snapshots

Benchmark Size MSHV KVM
call_and_create default 378 µs → 366 µs (−3%) 360 µs → 379 µs (~)
call_and_create small 425 µs → 411 µs (~) 417 µs → 462 µs (~)
call_and_create medium 789 µs → 777 µs (~) 769 µs → 769 µs (~)
call_and_create large 1.84 ms → 1.82 ms (~) 1.85 ms → 1.87 ms (~)
create default 249 µs → 234 µs (−6%) 262 µs → 254 µs (~)
create small 291 µs → 282 µs (~) 324 µs → 313 µs (~)
create medium 606 µs → 612 µs (~) 606 µs → 629 µs (~)
create large 1.68 ms → 1.68 ms (~) 1.7 ms → 1.68 ms (~)
restore default 69 µs → 71.4 µs (~) 22.2 µs → 24.7 µs (~)
restore small 68.7 µs → 73.9 µs (~) 26.7 µs → 23.5 µs (~)
restore medium 72.3 µs → 77.2 µs (~) 30.8 µs → 28 µs (~)
restore large 113 µs → 110 µs (~) 85.2 µs → 61.7 µs (~)
restore_alternate default 78.6 µs → 82.5 µs (~) 71.8 µs → 92.1 µs (~)
restore_alternate small 82.9 µs → 82.2 µs (~) 93.4 µs → 94.4 µs (~)
restore_alternate medium 86.8 µs → 85.6 µs (~) 97.9 µs → 98.8 µs (~)
restore_alternate large 123 µs → 125 µs (~) 158 µs → 129 µs (~)

guest_calls

Benchmark Size MSHV KVM
call default 45.2 µs → 45.4 µs (~) 17.1 µs → 16.9 µs (~)
call small 45.2 µs → 45.2 µs (~) 17.5 µs → 18.0 µs (~)
call medium 45.4 µs → 45.0 µs (~) 17.5 µs → 17.5 µs (~)
call large 44.7 µs → 45.2 µs (~) 17.6 µs → 17.5 µs (~)
call_with_host_function default 75.2 µs → 73.5 µs (~) 34.5 µs → 31.9 µs (~)
call_with_host_function small 74.1 µs → 74 µs (~) 34.7 µs → 33.4 µs (~)
call_with_host_function medium 75.2 µs → 75.7 µs (~) 32.8 µs → 34.2 µs (~)
call_with_host_function large 75.2 µs → 75.8 µs (~) 32.7 µs → 33.6 µs (~)
call_with_restore default 133 µs → 148 µs (~) 62.9 µs → 62.5 µs (~)
call_with_restore small 140 µs → 139 µs (~) 63.8 µs → 71.9 µs (~)
call_with_restore medium 138 µs → 141 µs (~) 67.9 µs → 67.4 µs (~)
call_with_restore large 166 µs → 166 µs (~) 89.1 µs → 89.8 µs (~)
different_thread - 74 µs → 72.3 µs (~) 42.9 µs → 42.3 µs (~)
interrupt_latency - 48.9 µs → 64.5 µs (~) 35.9 µs → 39.2 µs (~)

guest_functions_with_large_parameters

Benchmark Size MSHV KVM
guest_call_with_large_parameters - 163 ms → 172 ms (~) 916 ms → 890 ms (~)

sample_workloads

Benchmark Size MSHV KVM
24K_in_8K_out_c - 56.5 µs → 57.5 µs (~) 27.6 µs → 27.6 µs (~)
24K_in_8K_out_rust - 56.1 µs → 55.3 µs (~) 27.9 µs → 27.8 µs (~)

sandboxes

Benchmark Size MSHV KVM
create_initialized default 1.72 ms → 1.49 ms (−13%) 2.8 ms → 2.83 ms (~)
create_initialized small 4.6 ms → 3.09 ms (−33%) 5.12 ms → 2.92 ms (−43%)
create_initialized medium 11.8 ms → 5.93 ms (−50%) 9.81 ms → 3.11 ms (−68%)
create_initialized large 38.5 ms → 11 ms (−71%) 28.8 ms → 3.59 ms (−88%)
create_initialized_and_drop default 21.4 ms → 24.3 ms (~) 14.1 ms → 11 ms (~)
create_initialized_and_drop small 28.9 ms → 25.9 ms (~) 13.3 ms → 15.7 ms (~)
create_initialized_and_drop medium 31.5 ms → 28.3 ms (−10%) 24.3 ms → 14.8 ms (−39%)
create_initialized_and_drop large 57.9 ms → 30.7 ms (−47%) 39.6 ms → 12.4 ms (−69%)
sandbox_from_snapshot default 665 µs → 631 µs (−5%) 971 µs → 947 µs (~)
sandbox_from_snapshot small 812 µs → 805 µs (~) 978 µs → 971 µs (~)
sandbox_from_snapshot medium 2.02 ms → 2.04 ms (~) 1.01 ms → 999 µs (~)
sandbox_from_snapshot large 6.06 ms → 6.16 ms (~) 1.22 ms → 1.21 ms (~)

snapshot_files

Benchmark Size MSHV KVM
cold_start_via_evolve default 1.61 ms → 1.58 ms (~) 2.96 ms → 3.05 ms (~)
cold_start_via_evolve small 4.55 ms → 3.22 ms (~) 5.76 ms → 3.14 ms (~)
cold_start_via_evolve medium 12.3 ms → 5.31 ms (~) 11 ms → 3.52 ms (~)
cold_start_via_evolve large 38.8 ms → 10.7 ms (~) 29.1 ms → 4.24 ms (~)
cold_start_via_snapshot default 1.97 ms → 2.01 ms (~) 2.85 ms → 2.73 ms (~)
cold_start_via_snapshot small 8.93 ms → 9.13 ms (~) 8.95 ms → 8.9 ms (~)
cold_start_via_snapshot medium 53.8 ms → 54 ms (~) 50.9 ms → 50.5 ms (~)
cold_start_via_snapshot large 205 ms → 205 ms (~) 194 ms → 196 ms (~)
cold_start_via_snapshot_unverified default 1.4 ms → 1.43 ms (~) 2.2 ms → 1.95 ms (−11%)
cold_start_via_snapshot_unverified small 2.08 ms → 2.21 ms (~) 2.17 ms → 2.3 ms (~)
cold_start_via_snapshot_unverified medium 3.59 ms → 3.52 ms (~) 2.25 ms → 2.1 ms (~)
cold_start_via_snapshot_unverified large 7.73 ms → 7.67 ms (~) 2.59 ms → 2.34 ms (~)
load_snapshot default 701 µs → 703 µs (~) 701 µs → 692 µs (~)
load_snapshot small 6.71 ms → 6.63 ms (~) 6.68 ms → 6.56 ms (~)
load_snapshot medium 50.4 ms → 50.3 ms (~) 48.8 ms → 48.3 ms (~)
load_snapshot large 198 ms → 197 ms (~) 192 ms → 192 ms (~)
load_snapshot_unverified default 112 µs → 116 µs (~) 123 µs → 130 µs (~)
load_snapshot_unverified small 113 µs → 112 µs (~) 119 µs → 116 µs (~)
load_snapshot_unverified medium 114 µs → 123 µs (~) 120 µs → 123 µs (~)
load_snapshot_unverified large 118 µs → 122 µs (~) 125 µs → 127 µs (~)
save_snapshot default 941 µs → 943 µs (~) 43.4 ms → 39.5 ms (~)
save_snapshot small 9.09 ms → 9.11 ms (~) 61.5 ms → 52.2 ms (~)
save_snapshot medium 69.6 ms → 67.7 ms (~) 402 ms → 396 ms (~)
save_snapshot large 271 ms → 266 ms (~) 1.75 s → 1.74 s (~)

@ludfjig ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 12, 2026
@ludfjig
ludfjig added this pull request to stack #1929 October 12, 2026 20:01
@hyperlight-gh-bot

This comment has been minimized.

Copilot AI balanced review requested due to automatic review settings October 13, 2026 18:56
@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 96fab64 to ab775e0 Compare October 13, 2026 18:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Lazy no-reserve allocations need aggregate admission or commitment accounting to prevent deferred host-wide OOM.

1 open finding
What changed in this PR

Optimizes sandbox creation by constructing initial snapshot memory directly in shared memory and relying on lazy page population.

Changes:

  • Writes guest initialization data in place.
  • Freezes shared memory without copying.
  • Documents the performance improvement.
File Description
src/​hyperlight_host/​src/​sandbox/​snapshot/​mod.rs Builds snapshots directly in shared memory.
src/​hyperlight_host/​src/​mem/​shared_mem.rs Restricts the copying helper to tests and HVF.
CHANGELOG.md Documents lazy memory population.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs
@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from ab775e0 to 107a858 Compare October 13, 2026 21:01
@hyperlight-gh-bot

This comment has been minimized.

@ludfjig
ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 107a858 to aa55340 Compare October 13, 2026 23:00
@hyperlight-gh-bot

This comment has been minimized.

andreiltd
andreiltd previously approved these changes Oct 14, 2026
Snapshot::from_env writes the guest code, PEB and init data straight
into shared memory and freezes it. The host populates the remaining
pages on first guest touch, so sandbox creation cost stays independent
of heap size.

Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
@hyperlight-gh-bot

Copy link
Copy Markdown

Benchmark Results

Measured commit: 421a4538a965
Baseline commit: fd719b974c9d

kvm / amd (Linux) (🌟 6.46x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 6.46x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 760.73 ns (➖ 1.00x faster)
vec_bytes 589.73 ns (➖ 1.01x slower)
373.00 µs (➖ 1.00x slower)

payload_allocation

slot_pool_segmented
262144 528.41 ns (➖ 1.01x slower)
65536 142.08 ns (➖ 1.03x faster)

sandboxes

create_initialized_and_drop
medium 13.14 ms (🌟 6.46x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
7.88 ns (➖ 1.02x slower) 7.97 ns (➖ 1.03x slower) 7.89 ns (➖ 1.01x slower)

snapshot_files

load_snapshot_unverified
small 117.79 µs (➖ 1.03x slower)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 7.19 µs (➖ 1.02x faster) 7.27 µs (➖ 1.02x faster)
65536 2.06 µs (➖ 1.08x slower) 1.92 µs (➖ 1.09x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 6.36 µs (➖ 1.03x faster) 6.28 µs (➖ 1.03x faster)
8192 1.12 µs (➖ 1.07x slower) 1.06 µs (➖ 1.04x faster)
262144 27.73 µs (➖ 1.02x slower)
kvm / intel (Linux) (🌟 5.49x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 5.49x faster
  • payload_allocation/slot_pool_segmented/262144 — ✅ 1.57x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 679.60 ns (➖ 1.14x faster)
vec_bytes 515.95 ns (➖ 1.06x faster)
682.47 µs (➖ 1.08x faster)

payload_allocation

slot_pool_segmented
262144 512.98 ns (✅ 1.57x faster)
65536 138.62 ns (➖ 1.06x faster)

sandboxes

create_initialized_and_drop
medium 14.77 ms (🌟 5.49x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
7.05 ns (➖ 1.07x faster) 7.00 ns (➖ 1.04x faster) 7.01 ns (➖ 1.05x faster)

snapshot_files

load_snapshot_unverified
small 47.20 µs (➖ 1.20x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 7.65 µs (➖ 1.14x faster) 7.72 µs (➖ 1.13x faster)
65536 2.19 µs (➖ 1.10x faster) 2.19 µs (➖ 1.10x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 7.24 µs (➖ 1.14x faster) 7.15 µs (➖ 1.13x faster)
8192 782.08 ns (➖ 1.09x faster) 733.02 ns (➖ 1.12x faster)
262144 30.36 µs (➖ 1.10x faster)
mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 926.92 ns (➖ 1.00x faster)
vec_bytes 700.64 ns (➖ 1.01x faster)
357.34 µs (➖ 1.33x slower)

payload_allocation

slot_pool_segmented
262144 717.29 ns (➖ 1.01x faster)
65536 202.80 ns (➖ 1.03x slower)

sandboxes

create_initialized_and_drop
medium 65.48 ms (➖ 1.12x slower)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
9.66 ns (➖ 1.01x slower) 9.39 ns (➖ 1.00x faster) 9.42 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 104.72 µs (➖ 1.02x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 9.98 µs (➖ 1.03x faster) 9.71 µs (➖ 1.00x slower)
65536 2.45 µs (➖ 1.00x faster) 2.22 µs (➖ 1.01x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 9.58 µs (➖ 1.01x faster) 8.20 µs (➖ 1.25x faster)
8192 1.25 µs (➖ 1.04x faster) 1.25 µs (➖ 1.01x slower)
262144 36.28 µs (➖ 1.07x faster)
mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 917.59 ns (➖ 1.01x slower)
vec_bytes 655.18 ns (➖ 1.00x faster)
756.69 µs (➖ 1.19x slower)

payload_allocation

slot_pool_segmented
262144 648.01 ns (➖ 1.00x slower)
65536 171.69 ns (➖ 1.00x faster)

sandboxes

create_initialized_and_drop
medium 68.08 ms (➖ 1.07x slower)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
9.42 ns (➖ 1.00x faster) 9.50 ns (➖ 1.00x slower) 9.28 ns (➖ 1.00x faster)

snapshot_files

load_snapshot_unverified
small 53.15 µs (➖ 1.01x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 7.91 µs (➖ 1.00x slower) 7.97 µs (➖ 1.01x slower)
65536 2.25 µs (➖ 1.00x slower) 2.26 µs (➖ 1.00x slower)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 7.74 µs (➖ 1.03x slower) 7.45 µs (➖ 1.02x faster)
8192 1.01 µs (➖ 1.02x slower) 944.35 ns (➖ 1.03x slower)
262144 37.68 µs (➖ 1.05x faster)
hyperv-ws2025 / amd (Windows) (🌟 8.51x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 8.51x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.20 µs (➖ 1.01x faster)
vec_bytes 793.63 ns (➖ 1.01x slower)
1.84 ms (➖ 1.16x faster)

payload_allocation

slot_pool_segmented
262144 783.49 ns (➖ 1.10x faster)
65536 235.04 ns (➖ 1.01x faster)

sandboxes

create_initialized_and_drop
medium 11.42 ms (🌟 8.51x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
11.22 ns (➖ 1.08x slower) 10.58 ns (➖ 1.04x slower) 10.69 ns (➖ 1.03x slower)

snapshot_files

load_snapshot_unverified
small 1.00 ms (➖ 1.21x slower)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 9.21 µs (➖ 1.03x faster) 9.05 µs (➖ 1.03x faster)
65536 2.37 µs (➖ 1.01x faster) 2.34 µs (➖ 1.02x faster)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.69 µs (➖ 1.02x faster) 8.93 µs (➖ 1.01x slower)
8192 1.32 µs (➖ 1.04x faster) 1.29 µs (➖ 1.01x faster)
262144 38.44 µs (➖ 1.03x faster)
hyperv-ws2025 / intel (Windows) (🌟 7.34x)

Top improvements

  • sandboxes/create_initialized_and_drop/medium — 🌟 7.34x faster

Benchmark Results

function_call_codec

encode_control decode_vec_bytes_copy
byte_chunks 1.18 µs (➖ 1.04x faster)
vec_bytes 762.53 ns (➖ 1.05x faster)
2.80 ms (➖ 1.01x faster)

payload_allocation

slot_pool_segmented
262144 723.49 ns (➖ 1.03x faster)
65536 212.78 ns (➖ 1.01x slower)

sandboxes

create_initialized_and_drop
medium 13.57 ms (🌟 7.34x faster)

slot_pool

alloc_dealloc_1500 alloc_dealloc_4096 alloc_dealloc_128
9.96 ns (➖ 1.01x slower) 9.89 ns (➖ 1.11x faster) 10.05 ns (➖ 1.02x slower)

snapshot_files

load_snapshot_unverified
small 587.84 µs (➖ 1.16x faster)

virtq_readonly

slot_pool_segmented_fragmented slot_pool_segmented
262144 7.82 µs (➖ 1.03x slower) 7.76 µs (➖ 1.00x slower)
65536 2.54 µs (➖ 1.10x slower) 2.34 µs (➖ 1.00x slower)

virtq_readwrite

slot_pool_segmented_fragmented slot_pool_segmented
65536 8.49 µs (➖ 1.05x faster) 8.43 µs (➖ 1.01x faster)
8192 1.12 µs (➖ 1.12x faster) 1.12 µs (➖ 1.03x slower)
262144 42.01 µs (➖ 1.01x faster)

Reported by cargo ci bench-report --candidate run:38814039501 --config-file bench_report.toml.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/enhancement For PRs adding features, improving functionality, docs, tests, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants