Repository navigation
feat(firecracker): hot-plug behind PCIe root ports and wait for device removal - #290
Merged
Merged
Conversation
…e removal
Upstream Firecracker now places every post-boot PCI device behind a free
PCIe root port and fails the hot-plug when machine-config sets none, which
is the default. A DELETE of a device behind a root port is now graceful:
it returns at once, and the device stays until the guest releases it.
Give each --pci VM 8 root ports. The field stays off the wire for MMIO VMs,
because Firecracker rejects root ports there. After a DELETE, poll
/vm/config until the device is gone, with the same 30s bound as the Cloud
Hypervisor eject wait. A NIC the guest never releases returns
ErrEjectPending, as on Cloud Hypervisor. The released Firecracker build
removes the device at once, so the first poll returns there. A DELETE
body ({"force": true}) is not an option: that build rejects any body.
…onfig, method-aware test route Trim waitDeviceRemoved and pcieHotplugPorts comments to the fact the code cannot carry, drop two comments that restate type names, capitalize own-line comments in start.go, rename machineConfigFor to buildMachineConfig, stop the removal closures shadowing cfg, and register the stub route as GET /vm/config.
…ot-bus builds only The removal wait completes only when the guest kernel runs pciehp, and the printed guest steps are needed only when Firecracker plugs devices onto the root bus.
- createSnapshotFC and loadSnapshotFC call sendJSONOnce on their long-timeout
socket client instead of hand-rolling marshal plus DoAPIOnce. Same client,
no retry, same 204; only the marshal-error text changes.
- putEntropy calls putJSON with struct{}{}: same retry, PUT, 204 and {} body.
- LiveNICs drops the idx reset after Sscanf: a failed Sscanf never writes idx
(checked for EOF, non-integer, mismatch and overflow inputs).
- Drop the goroutine that closed the PTY master after a failed console relay.
It only mattered in a long-lived process that launches VMs; the daemon
launches none and no sibling repo imports cocoon/hypervisor, so the CLI
exit closes the fd.
Existing tests cover LiveNICs; the other three rest on the helper source.
The relay 5s timeout stays: it bounds a hang under the ops lock.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream Firecracker changes how PCI hot-plug works, and the current cocoon Firecracker backend breaks on it:
pcie_hotplug_ports, which defaults to 0, so every hot-plug fails withNoFreeRootPort. This hitsvm disk attach, NIC resize, and clone-time--data-disk/--nicson--pciVMs.This PR:
--pciVM 8 root ports. MMIO VMs do not get the field, because Firecracker rejects root ports without PCI.RemoveNICandDiskDetachpoll/vm/configafter the DELETE until the device is gone, with the same 30s bound as the Cloud Hypervisor eject wait. A NIC the guest never releases returnsErrEjectPending, as on Cloud Hypervisor.docs/devices.mdanddocs/firecracker.md, and makes the printed guest hints say they apply only when Firecracker plugs devices onto the root bus.The guest kernel needs PCIe hot-plug (
CONFIG_HOTPLUG_PCI_PCIE) on a root-port build. Without it the guest never releases a device, so every detach waits 30s and fails.Compatibility with the Firecracker build that
cocoon-check --upgradeinstalls today (fork dev74a2b360b): it already acceptspcie_hotplug_portsand removes devices synchronously, so the first poll returns there. It rejects any DELETE body, so{"force": true}is not an option.Order: merge this before the Firecracker fork dev moves to the new upstream. Snapshots keep the root port count of their VM, so bases made before this change have none. On the new Firecracker, clones of those bases cannot hot-plug until the base is created again.
Tested:
GOWORK=off make lintandmake fmt-check(linux and darwin),asl -forwarder=false ./...(linux and darwin),make vet,go test -raceonhypervisor/...,extend/...,cmd/vm/...TestRemoveNICWaitsForGuestRelease,TestRemoveNICReportsAnUnreleasedDevice,TestMachineConfigFor. The first two fail against the oldextend.go.Not tested:
--pciVM with a very large layer count could run out of slots.