Skip to content

device-hotplug.md

Latest commit

 

History

History
132 lines (100 loc) · 4.45 KB

device-hotplug.md

File metadata and controls

132 lines (100 loc) · 4.45 KB
Raw

Device Hotplugging [Developer Preview]

Warning

This feature is currently in Developer Preview. It may have limitations, and its API or behavior may change in future releases.

Device hotplugging allows attaching and detaching PCI virtio devices to a running microVM without requiring a reboot. Supported device types are:

  • virtio-block
  • virtio-pmem
  • virtio-net

Prerequisites

  • PCI transport enabled: Firecracker must be started with the --enable-pci flag. Device hotplugging is not supported with MMIO transport.
  • PCIe hot-plug ports configured: pcie_hotplug_ports must be set in the machine configuration. It defaults to 0, which disables hotplugging.
  • Guest kernel with PCI and PCIe hot-plug support: The guest kernel must have PCI and the native PCI Express hot-plug driver (pciehp) enabled (CONFIG_HOTPLUG_PCI=y, CONFIG_HOTPLUG_PCI_PCIE=y, CONFIG_PCIEPORTBUS=y).

How it works

Firecracker puts a number of PCI Express root ports on the root bus, each with one hot-plug capable slot. A hotplugged device goes into a free slot, and the root port raises a native PCI Express hot-plug interrupt. The guest's pciehp driver handles it and binds or unbinds the device by itself. Each root port occupies one slot on the root bus and starts one secondary bus, which is where its device appears.

Reserving hot-plug ports

Ports are reserved at boot through the pcie_hotplug_ports machine configuration option, which is the maximum number of devices that can be hotplugged at any one time. It cannot be changed after boot.

socket_location=/run/firecracker.socket

curl --unix-socket $socket_location -i \
    -X PUT 'http://localhost/machine-config' \
    -H 'Content-Type: application/json' \
    -d '{
        "vcpu_count": 2,
        "mem_size_mib": 1024,
        "pcie_hotplug_ports": 4
    }'

There's a slight boot time overhead for each root port added due to the additional bus enumeration by the guest (this does not affect snapshot restore). It is recommended that you only reserve the number of ports needed. The maximum is 31.

Limitations

  • vhost-user devices need shareable guest memory: a vhost-user backend maps guest memory by file descriptor, which Firecracker only provides when a vhost-user device is configured before boot. Hotplugging one to a VM booted without any, or to a VM restored from a snapshot file, is rejected.

Hotplugging a device

Hotplugging uses the same API endpoints used for pre-boot device configuration. The only difference is that the request is issued after the VM has started.

socket_location=/run/firecracker.socket

curl --unix-socket $socket_location -i \
    -X PUT 'http://localhost/drives/block1' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "drive_id": "block1",
        "path_on_host": "/path/to/block.ext4",
        "is_root_device": false,
        "is_read_only": false
    }'

Firecracker sends a notification to the guest and the guest discovers the new device.

Hotplugging latency

Linux waits for 120 ms after receiving a hotplug notification. This is due to delays mandated by the PCIe specification. That means that the device will show up in the guest in a bit more than 120 ms.

If this latency is unacceptable for your use case one workaround is triggering a manual bus rescan from inside the guest:

echo 1 > /sys/bus/pci/rescan

Hot-unplugging a device

Only devices behind root ports can be unplugged. Issue a DELETE request to the device's endpoint:

curl --unix-socket $socket_location -i \
    -X DELETE 'http://localhost/drives/block1'

This starts a graceful removal. Firecracker asks the guest to release the device and the call returns immediately. The guest's pciehp driver then quiesces the device and powers the slot off at which point Firecracker frees the backing resources. Linux waits five seconds before releasing the device.

You can poll GET /vm/config to find out when the device is actually gone. Until then the device ID and its root port stay in use.

Forcing a removal

If the guest does not respond, force removes the device immediately without waiting:

curl --unix-socket $socket_location -i \
    -X DELETE 'http://localhost/drives/block1' \
    -H 'Content-Type: application/json' \
    -d '{ "force": true }'

This might have unpredictable consequences for the guest.