Skip to content

Firecracker runs unjailed — integrate the jailer to contain a compromised VMM #83

Description

@CMGS

Problem

cocoon spawns firecracker directly, as the invoking user (root in typical deployments), in the host mount/PID/network namespaces, with no cgroup confinement. The only containment layer active today is Firecracker's built-in seccomp filters.

For workloads that run untrusted code in the guest, this leaves defense-in-depth incomplete: a single guest→VMM escape (device-model bug in virtio-blk/vsock) yields code execution as root on the host — able to read every VM's images, snapshots and sockets under /var/lib/cocoon and take over the node. With the jailer, the same escape lands as a dedicated unprivileged uid inside an empty chroot with namespace isolation and cgroup limits; harming the host then requires chaining a second (kernel privilege-escalation) vulnerability.

What the jailer provides

Firecracker's production baseline (how AWS runs it): per-VM chroot containing only hard-linked resources, mount/PID (and optionally network) namespace isolation, drop to a per-VM uid:gid, cgroup attachment, resource limits — on top of the built-in seccomp.

Integration work items

  1. Path model: every file the VMM opens must live inside the chroot — kernel/initrd, EROFS layer blobs (content-addressed and shared across VMs: hard links require same-filesystem, else bind mounts), COW file, snapshot mem/state on restore. The API socket and hybrid-vsock UDS move into the chroot; VM records (socket_path, vsock_socket in vm list) must keep reporting host-resolvable paths so existing consumers stay untouched.
  2. uid/gid allocation per VM + ownership/permission scheme for shared read-only blobs.
  3. cgroups: jailer cgroup integration, including behavior when cocoon itself runs inside a container.
  4. Snapshot/clone/restore: golden mem/state hard-linked into each clone's chroot; latency must be A/B-measured (restore paths are tens of ms; target overhead ≤ ~5ms).
  5. Process-tree awareness: jailer forks the VMM, so stop/delete/orphan-scan logic must follow the child, not the jailer pid.
  6. Network namespace: NIC-less VMs (--nics 0) work trivially in an empty netns — for those, jailing is pure win with no TAP plumbing.

Interim mitigation (cheap, immediate)

Run cocoon/VMMs as a dedicated non-root user and tighten /var/lib/cocoon to 0700 — turns "escape = root" into "escape = confined user" even before jailer lands.

Acceptance

  • Running FC VMs launch via jailer with per-VM uid + chroot + cgroup.
  • Restore/clone latency regression ≤ ~5ms in A/B.
  • vm list consumers unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions