From 9bcaccc4933f941aa528fcfafb7caacd2c125eb8 Mon Sep 17 00:00:00 2001
From: AdaAibaby <8872412+adababys@users.noreply.github.com>
Date: Thu, 29 Oct 2026 10:09:41 +0800
Subject: [PATCH] docs(embed/gcp): note memory-snapshot resume is unsupported
 on nested virt
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

GCE nested-virtualization VMs expose a KVM that cannot reliably restore
Firecracker device state, so memory-snapshot resume (the normal
POST /sandboxes path) yields a running guest whose virtio-net is dead: envd
is unreachable and the sandbox is marked stopping ~15s after create.
Cold boot (filesystem_boot) works on the same node because it initialises
devices from scratch rather than restoring them.

This is a platform (nested KVM) limitation, not a runtime bug — pristine
upstream Firecracker on the same GCE instance fails even earlier, exiting the
moment a VM is paused. Document it in the GCP module README so operators reach
for cold boot (or a bare-metal instance) instead of chasing a resume that
cannot work there.

Refs #3745.
---
 embed/terraform/gcp/README.md | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/embed/terraform/gcp/README.md b/embed/terraform/gcp/README.md
index 786b2b3c0d..482a7ca037 100644
--- a/embed/terraform/gcp/README.md
+++ b/embed/terraform/gcp/README.md
@@ -154,6 +154,26 @@ sudo journalctl -u google-startup-scripts -f
 The `ssh_command` output is a shell snippet that looks the current instance
 name up, which is why it is used through `eval`.
 
+### Limitations on nested virtualization
+
+This module runs Firecracker inside a GCE nested-virtualization VM, where the
+host KVM is itself exposed by Google's hypervisor. **Cold-boot sandboxes
+(`filesystem_boot`) work; memory-snapshot resume does not.** On this platform
+Firecracker's snapshot/restore path cannot reliably restore device state:
+the guest resumes with a dead `virtio-net` (its vCPU and lazily-faulted memory
+are fine, but `envd` is unreachable, so the sandbox fails its health check and
+is marked stopping about 15 seconds after create). This is a limitation of the
+nested KVM, not of the runtime — pristine upstream Firecracker on the same
+instance type fails even earlier, exiting the moment a VM is paused.
+
+The normal `POST /sandboxes` path resumes from a memory snapshot and is
+therefore affected. If you need working sandboxes on GCE nested virt, use the
+cold-boot path (gRPC `SandboxService.Create` with `filesystem_boot: true`),
+which does not restore device state. For the full memory-resume economics
+(sub-second resume, shared guest memory), run on a bare-metal instance whose
+`/dev/kvm` is real hardware rather than a nested VM — GCE's metal machine
+types, for example.
+
 ### Secrets
 
 `ADMIN_TOKEN`, `SANDBOX_ACCESS_TOKEN_HASH_SEED` and the team API key are all