Skip to content

[LXC] State-aware lifecycle management#633

Open
dhoehna wants to merge 8 commits into
microsoft:mainfrom
dhoehna:user/dahoehna/lxc-state-aware-lifecycle
Open

[LXC] State-aware lifecycle management#633
dhoehna wants to merge 8 commits into
microsoft:mainfrom
dhoehna:user/dahoehna/lxc-state-aware-lifecycle

Conversation

@dhoehna

@dhoehna dhoehna commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Linked work item: AB#62953349 — [LXC] State-aware lifecycle management

Summary

Implements the existing StatefulSandboxBackend trait for the LXC backend, modeled on IsolationSessionRunner.

  • New state_aware module in the LXC common crate; provision / start / exec / stop / deprovision fully wired to the LXC CLI wrappers.
  • exec reuses the lxc-attach PTY streaming path; the returned ExecHandle waiter yields the captured exit code.
  • deprovision performs best-effort iptables cleanup.
  • Registered BACKEND_KEY = "lxc" in the Rust dispatcher/parser and the SDK prefix map (+ SDK state-aware types).

Validation

  • cargo fmt --all -- --check; cargo clippy --target x86_64-unknown-linux-gnu -p lxc_common -D warnings — clean
  • cargo test -p wxc_common — 397 passed; cargo test -p mxc-sdk — 32 passed
  • cargo check --target x86_64-unknown-linux-gnu -p lxc_common --tests — pass

Coupling

Adds an SDK state-aware backend key and touches the shared dispatcher/parser, so merge-time reconciliation with the schema PR (AB#62830582) is expected.

Microsoft Reviewers: Open in CodeFlow

Copilot AI review requested due to automatic review settings July 10, 2026 22:56
@dhoehna
dhoehna requested a review from a team as a code owner July 10, 2026 22:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds state-aware lifecycle support for the Linux LXC backend by implementing StatefulSandboxBackend in lxc_common, wiring dispatch from lxc-exec, and exposing the new backend key/prefix in both Rust parsing/dispatch and the TypeScript SDK state-aware types/helpers.

Changes:

  • Register lxc as a state-aware backend prefix/key in the Rust dispatcher/config parser (with new unit tests).
  • Add LxcStateAwareRunner implementing provision/start/exec/stop/deprovision for LXC containers (best-effort iptables cleanup on teardown).
  • Extend the SDK state-aware type system + prefix map to support lxc, including per-phase config/metadata typings.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/core/wxc_common/src/state_aware_dispatch.rs Adds lxc sandboxId prefix routing + a unit test.
src/core/wxc_common/src/config_parser.rs Accepts experimental.lxc as a known experimental backend key and adjusts key matching for containment: "lxc" (with test).
src/core/lxc/src/main.rs Switches to load_mxc_request and routes state-aware requests to LxcStateAwareRunner.
src/core/lxc/Cargo.toml Adds serde_json dependency for error envelope printing.
src/Cargo.lock Lockfile update reflecting the new serde_json dependency.
src/backends/lxc/common/src/state_aware.rs New LXC state-aware lifecycle implementation.
src/backends/lxc/common/src/lib.rs Exposes the new state_aware module.
sdk/src/state-aware-types.ts Adds lxc to state-aware backend union and defines LXC per-phase config + metadata types.
sdk/src/state-aware-helper.ts Adds lxc prefix mapping and hoists containerId as a cross-cutting field.

Comment thread src/backends/lxc/common/src/state_aware.rs
Comment thread sdk/src/state-aware-types.ts Outdated
…) (AB#62953349)

Provision/start/exec/stop/deprovision for the LXC backend, modeled on IsolationSessionRunner; reuses lxc CLI wrappers and one-shot lxc-attach PTY streaming. Registers the lxc wire key in Rust dispatch/parser and SDK state-aware routing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3b78bec0-e139-4cfd-9c10-092ef986d4f4
@dhoehna
dhoehna force-pushed the user/dahoehna/lxc-state-aware-lifecycle branch from ded0b0d to fabdfd0 Compare July 13, 2026 16:45
dhoehna and others added 4 commits July 13, 2026 14:52
… + narrow LXC start network type

Restrict is_valid_container_name to the same character set and length bound
(<=20 chars, alphanumeric/-/_) that NetworkIptablesManager::new uses to derive
the per-container iptables chain name. This makes the container-name ->
chain-name mapping an identity on valid names, so distinct names (e.g. 'a.b'
vs 'ab', or names differing only past the 20th char) can no longer collide onto
the same firewall chain and cross-tear-down each other's rules.

Narrow LxcStartConfig.network to Omit<NetworkConfig, 'proxy'> so the SDK rejects
network.proxy at compile time, matching the Rust runner which rejects it at
start (apply_network_policy). Adds Rust + TypeScript tests for both.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…aware_provision.json)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
Resolved conflicts:
- src/core/lxc/src/main.rs: kept state-aware imports; dropped now-unused ScriptRunner
- src/Cargo.lock: took main's lock, reconciled via cargo metadata

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
Re-run GitHub Actions after a transient Hyperlight E2E network flake (hyperlight_networking live-HTTP cases timed out after 30s). No source changes; this empty commit only re-fires the pull_request workflows.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
Comment thread src/backends/lxc/common/src/state_aware.rs
Comment thread src/backends/lxc/common/src/state_aware.rs
Comment thread src/backends/lxc/common/src/state_aware.rs
Comment thread src/core/lxc/src/main.rs Outdated
- Route Lxc state-aware dispatch through mxc_engine::run_state_aware so the
  lxc binary stays a thin CLI shim instead of hand-rolling the backend match.
- Stop/destroy the container before tearing down its iptables rules in stop(),
  deprovision(), and the start() rollback, discovering the veth first so the
  FORWARD hook rule can still be deleted after the device is gone. Closes an
  unrestricted-egress window during teardown.
- Clear lxc.mount.entry before reapplying filesystem mounts so a restart with a
  tightened policy no longer inherits the previous run's bind mounts (new
  LxcContainer::clear_config_item).
- Kill the timed-out child's whole process group and bound the output drain in
  mxc_pty::run_with_pty so a leaked in-container process holding the pty open can
  no longer hang exec forever (new join_with_timeout helper).

Adds unit/regression tests for each fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dhoehna

dhoehna commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

🧪 Local test re-verification — 2026-07-17

Re-ran the test suites locally at branch tip f84cf2b on a dev workstation. All green (0 failures).

Windows host (x86_64-pc-windows-msvc, cargo 1.96.1):

  • cargo test -p wxc_common396 passed, 0 failed
  • cargo test -p mxc-sdk11 passed, 0 failed (+4 runtime-ignored — require an elevated, host-prepped Windows host per docs/host-prep.md)

Linux (WSL2 Ubuntu-24.04, x86_64-unknown-linux-gnu, cargo 1.97.0, isolated CARGO_TARGET_DIR):

  • cargo test -p lxc_common62 passed, 0 failed — the state-aware LXC backend ran on Linux (the original Validation only cargo check-compiled it for the Linux target).

Note: the wxc_common count differs slightly from the original Validation because the branch advanced since it was written.

Resolve conflict in wxc_common/src/state_aware_dispatch.rs: register both the `lxc` (this PR) and `wsb` (upstream microsoft#578) state-aware backend prefixes in backend_from_prefix, and keep both resolve_backend unit tests. Added `correlation_vector: None` to the lxc test to match the ParsedStateAwareRequest field introduced upstream (microsoft#624).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dhoehna

dhoehna commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

🔀 Merge-conflict resolution — synced with main (2026-07-17)

Merged microsoft/mxc main (d271ac4) into this branch to clear the dirty (unmergeable) state. This is the merge-time reconciliation flagged in the Coupling section of the description — it reconciles this PR with the Windows Sandbox state-aware work (#578) and the telemetry correlationVector work (#624) that landed on main after this PR was opened.

Only one conflict, in src/core/wxc_common/src/state_aware_dispatch.rs — both sides registered a new state-aware backend in the same place:

Everything else auto-merged (incl. the mxc_engine::state_aware::run_state_aware backend arms and the SDK state-aware-*.ts types).

Local validation at merge tip dd29002 (Windows workstation):

  • cargo test -p wxc_common -> 455 passed, 0 failed
  • cargo fmt -p wxc_common -- --check -> clean
  • cargo build -p wxc (compiles the merged mxc_engine) -> success
  • SDK npm run build + npm test -> 189 passed, 0 failed, 4 skipped

(Linux lxc_common compile/tests are exercised by CI on this commit.)

container
.start()
.map_err(|e| MxcError::backend_error(format!("Failed to start container: {e}")))?;
if let Err(e) = apply_network_policy(&container, request, &mut logger) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apply_network_policy

Container egress is still not filtered — the network policy fails open for the container's entire runtime. This is separate from the teardown-ordering fix in this update (which correctly closed the stop/deprovision window); this one is about the direction the chain is hooked, and it is untouched.

start() applies network enforcement here via apply_network_policy() -> NetworkIptablesManager::apply_firewall_rules(). The per-container chain filters on destination (-d ACCEPT/DROP for allowed/blocked hosts, then a default DROP/ACCEPT), but it is hooked into FORWARD with:

-I FORWARD -o <veth> -j <chain>   (src/backends/lxc/common/src/network_iptables.rs:237)

In a bridged veth setup, container->internet egress enters the host FORWARD chain as -i , while internet->container replies are -o . Hooking on -o therefore applies the chain only to traffic heading INTO the container, never to traffic originating from it:

  • Outbound SYN (-i veth): not matched -> leaves unfiltered; blocked_hosts, allowed_hosts, and the default DROP are all bypassed.
  • Return SYN-ACK (-o veth): matched, but it's ESTABLISHED,RELATED -> ACCEPTed; and -d can't match a remote host on the return path anyway.

Net effect: default_network_policy=Block, blocked_hosts, and allowed_hosts are ineffective for egress; a container started with a deny-by-default policy can reach any destination for its whole lifetime, not just during teardown.

Two related fail-open paths in the same function: if discover_veth_interface() returns None it logs "Skipping FORWARD hook" but still sets rules_applied=true / returns Ok(true), so start() treats an unhooked container as success; and a non-Firewall/Both enforcement mode returns Ok(true) without applying anything.

Suggested fix: hook the chain on -i (network_iptables.rs:237) and change the cleanup -D FORWARD -o (network_iptables.rs:264) to -i, so the destination-based rules apply to container egress. Fail start (return Ok(false)/error) when the veth can't be discovered or the hook can't be installed, instead of proceeding with no enforcement. Add a test that sets deny-by-default and asserts the container cannot reach a non-allowlisted host.

@microsoft-github-policy-service microsoft-github-policy-service Bot added the Needs-Author-Feedback Issue needs attention from issue or PR author label Jul 17, 2026
…tate-aware-lifecycle

# Conflicts:
#	sdk/node/src/state-aware-helper.ts
#	sdk/node/src/state-aware-types.ts
#	sdk/node/tests/unit/state-aware-types.test.ts
#	src/backends/lxc/common/src/filesystem_mounts.rs
#	src/core/wxc_common/src/state_aware_backend.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Needs-Author-Feedback Issue needs attention from issue or PR author

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants