[live-migration] Preserve host-side GCS logging across migration#2826
[live-migration] Preserve host-side GCS logging across migration#2826rawahars wants to merge 4 commits into
Conversation
13a58f5 to
96c7500
Compare
By default vsockexec dials the host once, hands the socket to the child
as its stdio, and execs into it. If the host listener is absent when the
guest starts, or the connection later drops, the child's writes fail and
it exits. Because the child runs under PID 1 in the UVM, its death makes
PID 1 exit and the guest kernel panics, taking the VM down.
Add a new -r flag that keeps vsockexec running as a relay between the
child and the host instead of exec'ing into it:
- the child writes to a pipe (which never breaks), not to the socket;
- vsockexec keeps the host connection itself and forwards the pipe to it;
- it waits and retries if the host is not listening yet (late connect);
- it re-dials and resends if the connection drops (reconnect);
- the child's exit status is propagated, so PID 1 still sees the real
exit code and shutdown semantics are unchanged.
Without -r the behavior is byte-for-byte identical to before, so no
existing caller is affected. stdout and stderr sharing one port continue
to share a single host connection.
Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
Wire the reconnect mode (-r) added in the previous commit into the LCOW
live-migration path so live-migratable pods keep host-side GCS logging
instead of forgoing it.
Before, these pods dropped GCS log forwarding entirely: a plain
vsockexec would block on its outbound connect and stall guest init when
the host log listener was absent, so the wrapper and listener were
skipped. Reconnect mode removes that constraint:
- buildGCSCommand emits /bin/vsockexec -r for live-migratable
sandboxes, so the guest tolerates a missing listener and re-dials
LinuxLogVsockPort after the connection drops;
- StartWithMigrationOptions arms a fresh listener on the destination
before start, so the resumed guest can reconnect and resume
streaming;
- setupLoggingListener no longer early-returns and Resume no longer
force-closes logOutputDone.
Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
78a3546 to
3d5ef13
Compare
|
No functional changes since last review. Rebased onto mainline to resolve conflicts. |
Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
marma-dev
left a comment
marma-dev
left a comment
There was a problem hiding this comment.
Overall looks good. I need a bit of clarity on the destination dial-timing, and a minor nit on string construction.
Optional: Would be great to add a Go unit test for the new gate (VM whose cmdline lacks -r → InitializeLiveMigrationOnSource rejects) and, if feasible, an integration test that drops/re-arms the host listener mid-stream to exercise the reconnect path.
| @@ -127,6 +157,12 @@ func (c *Controller) StartWithMigrationOptions(ctx context.Context, config *hcs. | |||
|
|
|||
| // Watch for VM exit in the background. | |||
| go c.waitForVMExit(ctx) | |||
|
|
|||
| // Collect any errors from establishing the log connection. | |||
| if err := g.Wait(); err != nil { | |||
| return err | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
Does the destination guest dial LinuxLogVsockPort during StartWithMigrationOptions, or only at resume?
setupLoggingListener dispatches the accept onto g, and that goroutine blocks in [AcceptConnection] until a connection arrives or the VM exits. So g.Wait() here can't return until the guest dials. If the destination guest only executes at resume (precopy), this blocks until VM exit and hangs the migration
Since Listen errors are already returned synchronously by setupLoggingListener, g.Wait() only ever waits on the accept, which we don't want to block on. I would suggest dropping the g.Wait() (and the defer g.Wait() and letting the accept run in the background — it closes logOutputDone on VM exit. If I'm wrong about the destination dial timing, please confirm and disregard.
| g, gctx := errgroup.WithContext(ctx) | ||
| defer func() { | ||
| _ = g.Wait() | ||
| }() | ||
|
|
||
| if err := c.setupLoggingListener(gctx, g); err != nil { | ||
| return fmt.Errorf("re-arm logging listener on resume: %w", err) | ||
| } | ||
|
|
||
| // Collect any errors from establishing the log connection. | ||
| if err := g.Wait(); err != nil { | ||
| return err |
There was a problem hiding this comment.
Same pattern here, and this one can stall even when the guest is alive: the reconnect relay only re-dials on its next write to the dead socket. If GCS is briefly idle, no write → no reconnect → the accept never completes → g.Wait() blocks until GCS next logs (or VM exit).
This also contradicts the adjacent comment ("the accept runs in the background … so a slow guest re-dial cannot stall resume"). I would recommend dropping g.Wait() and relying on the background accept.
| if !strings.Contains(kernelCmdLine, fmt.Sprintf("/bin/vsockexec -r -e %d", vmutils.LinuxLogVsockPort)) { | ||
| return fmt.Errorf("cannot initialize live migration on source: VM was not created with live-migration support enabled") | ||
| } |
There was a problem hiding this comment.
Two things on the LM-eligibility check:
- It matches
fmt.Sprintf("/bin/vsockexec -r -e %d", …)as a substring, but the builder constructs that string independently in buildGCSCommand. Any formatting drift (flag order/spacing) silently breaks the gate. Could we extract a shared helper/const referenced by both the builder and here so they can't desync? c.hcsDocument.VirtualMachine.Chipsetis dereferenced with a nil check on Chipset but not on VirtualMachine (orhcsDocument). Please add the guard for parity.
There was a problem hiding this comment.
Pull request overview
This PR aims to preserve host-side GCS log forwarding for live-migratable LCOW pods by making the guest-side vsock log forwarder resilient to missing/dropped host listeners (via a new vsockexec -r reconnect/relay mode) and ensuring the destination host re-arms the log listener before resuming a migrated VM.
Changes:
- Add
-rreconnect/relay mode tovsockexecto tolerate missing/dropped host listeners without surfacing disconnects to the child process. - Re-enable/arm the host-side LCOW GCS log listener on migration destination start and on source rollback resume.
- Remove
LiveMigrationSupportEnabledfromSandboxOptionsand instead wire reconnect-mode vsockexec directly when building the GCS command line; update tests accordingly.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| vsockexec/vsockexec.c | Adds reconnect/relay mode (-r) for resilient stdio forwarding over vsock. |
| internal/controller/vm/vm_migration.go | Ensures destination arms a log listener before start; gates LM init on reconnect-mode kernel cmdline. |
| internal/controller/vm/vm_lcow.go | Removes LM “skip logs” behavior; adjusts log listener completion signaling to be robust to re-arming. |
| internal/controller/vm/save_lcow.go | Re-arms log listener on source rollback resume after blackout. |
| internal/builder/vm/lcow/specs.go | Removes LM flag threading into kernel args parsing/build path. |
| internal/builder/vm/lcow/specs_test.go | Updates LM wiring tests to expect reconnect-mode wrapper when LM is enabled. |
| internal/builder/vm/lcow/sandbox_options.go | Removes LiveMigrationSupportEnabled from SandboxOptions. |
| internal/builder/vm/lcow/kernel_args.go | Always wraps GCS with vsockexec; uses -r when LM annotation is enabled. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // Open the connection to the host, waiting for the host if it isn't ready. | ||
| int sock = dial_retry(port); | ||
|
|
| static int run_relay(unsigned int ports[3], char** child_argv) { | ||
| // In reconnect mode we forward the program's OUTPUT (stdout and/or stderr) | ||
| // to one host port. Use whichever output port was requested. | ||
| unsigned int port = ports[2] != 0 ? ports[2] : ports[1]; | ||
| if (port == 0) { | ||
| // Nothing to forward: reconnect mode is only meaningful with -o or -e. | ||
| fprintf(stderr, "vsockexec: -r requires -o or -e\n"); | ||
| return 1; | ||
| } | ||
|
|
| g, gctx := errgroup.WithContext(ctx) | ||
| defer func() { | ||
| _ = g.Wait() | ||
| }() | ||
|
|
| // The blackout also dropped the source's GCS log connection, which tore | ||
| // down its listener and closed logOutputDone. Install a fresh signal and | ||
| // re-arm the listener so the resumed guest's reconnect-mode vsockexec can | ||
| // reconnect and host-side logs resume. The accept runs in the background | ||
| // (WithoutCancel so it outlives this call; AcceptConnection still returns | ||
| // on VM exit) so a slow guest re-dial cannot stall resume. | ||
| c.logOutputDone = make(chan struct{}) | ||
| g, gctx := errgroup.WithContext(ctx) | ||
| defer func() { | ||
| _ = g.Wait() | ||
| }() | ||
|
|
||
| if err := c.setupLoggingListener(gctx, g); err != nil { | ||
| return fmt.Errorf("re-arm logging listener on resume: %w", err) | ||
| } | ||
|
|
||
| // Collect any errors from establishing the log connection. | ||
| if err := g.Wait(); err != nil { | ||
| return err | ||
| } |
Today a live-migratable LCOW pod gives up host-side GCS logging entirely, because the guest→host log connection is host-local state that migration doesn't carry: after a move there's no listener to reconnect to, and a guest that depended on the log socket would stall on boot. This PR makes that log path survive migration instead of disabling it.
It does so in two steps:
Make the forwarder resilient —
vsockexecgets an opt-in reconnect mode (-r) that runs as a relay between the child and the host: it tolerates the host listener being absent at boot, and re-dials and resumes if the connection drops. Default behavior is unchanged.Use it on the migration path — live-migratable pods now run
vsockexec -r, and the destination host arms a fresh log listener before resume, so the migrated guest simply reconnects and keeps streaming GCS logs.Net result: GCS logs keep flowing before, during, and after a live migration, with no change for non-migratable pods.