Skip to content

windows.md

Latest commit

 

History

History
217 lines (191 loc) · 14.9 KB

windows.md

File metadata and controls

217 lines (191 loc) · 14.9 KB
Raw

Windows guests: how they work

The README's Windows guests section says how to build, run and reach a Windows machine, and examples/windows builds whole labs. This page is the other half: how vk boots and drives Windows, what each piece does, and what is known not to work yet.

A Windows machine is a UEFI guest on virtio devices, in the same embedded VMM as vk's Linux guests (no QEMU, no AHCI emulation, no other VMM on the host), driven through its qemu-ga agent instead of vk-agent.

Overview

 Dockerfile ─► vk build ─► winiso install (cached) ─► RUN/COPY layers (qcow2, cached) ─► bundle
                                                                                          │
 compose file ─► vk run --compose ─► manager ─► winsvc: boot, wait, secrets, provisioning │
                                       │                                                  ▼
                                       └──────────────► libkrun VMM (boot child) ◄── vm.json
                                                         ├ UEFI firmware (embedded)
                                                         ├ virtio-pci: blk, net, console, rng
                                                         ├ uefi-vars: UEFI variables (vk-uefi-vars)
                                                         ├ TPM 2.0 CRB: vk-tpm
                                                         └ ACPI, SMBIOS, Hyper-V, pvpanic
 vk exec / cp / healthcheck ─► qemu-ga over virtio-serial (qga.sock in the run dir)

Firmware

The firmware is edk2's OvmfPkg/CloudHv platform, built by ./build-firmware.sh from the nixpkgs revision the build image pins (the firmware output of .devcontainer/nix/flake.nix), with Secure Boot and the TPM on and without SMM, which libkrun does not emulate. build.sh embeds it into vk (VIRTKIT_UEFI_FIRMWARE names another at run time); libkrun loads it as a PVH ELF, like a Linux kernel. Two variable store templates come with it: an empty one, and one with Microsoft's Secure Boot keys enrolled (firmware=uefi-secboot).

One local patch, .devcontainer/nix/firmware/cloudhv-host-variables.patch, changes the build's configuration only:

  • The UEFI variables are the host's. edk2's variable driver for a variable service outside the guest (VirtMmCommunicationDxe with VariableSmmRuntimeDxe, written for QEMU's uefi-vars device) replaces the platform's own, which kept them in the guest and checked signed writes there. vk's vk-uefi-vars is that service: it keeps the variables in uefi-vars.fd, checks PK, KEK, db and dbx updates against the keys above them, and enforces the firmware's variable policies, out of the guest kernel's reach. The variable driver's runtime cache is off, as that client requires (it stops the boot otherwise). See the design.
  • edk2's TCG MOR driver is in, so the firmware has the MemoryOverwriteRequestControl variable BitLocker looks for with a TPM. MorLock is not supported.

VMM

libkrun is vendored under third_party/libkrun; its VENDOR.md lists every local patch with its rationale and test. For Windows they add:

  • the platform devices Windows and the firmware expect: an ACPI PM timer, a PCI host bridge at 00:00.0 that the CloudHv firmware identifies, a CMOS real-time clock, COM1/COM2, ACPI processor objects, a FACS in the XSDT, a 32-bit PCI hole the firmware assigns BARs in;
  • virtio-pci with MSI-X and QEMU's subsystem IDs, which the virtio-win drivers bind to, and device reset (Windows' drivers reset their devices as they start; virtio-net keeps its backend across a reset, since it owns the tap or socket and must not open it twice);
  • the UEFI variable service's device (uefi-vars, edk2's QemuUefiVars register interface, DMA transfers) and the fw_cfg file the firmware finds it through, in front of vk-uefi-vars; a snapshot taken before it, with the CFI flash the variables used to live on, restores with the flash;
  • a TPM 2.0 CRB device backed by vk-tpm, a TPM 2.0 in Rust inside vk (no swtpm on the host), its state in tpm-state;
  • SMBIOS tables with a per-machine UUID, a VM generation ID, KVM's Hyper-V enlightenments, pvpanic (a guest crash is logged to the run's console.vmm.log);
  • on a host whose KVM delivers an int1 onto itself (Hyper-V's nested SVM on AMD: WSL2, Azure), which the VMM probes in a throwaway VM, the guest's debug exceptions taken by the VMM, which steps over int1: the guest ran it again forever, and Windows' PatchGuard runs one now and then;
  • pause and resume, and snapshot and restore of memory, CPU and device state.

Driver (vk-driver)

Module Role
winiso The unattended install: the FAT32 install disk (made in a Linux helper VM), Setup, the first boots' servicing, the settle step.
winbuild Windows Dockerfile stages: the RUN/COPY layers through qemu-ga, sysprep, the bundle.
wincache Layers shared through the build cache registry, and the host's Windows cache sweep.
uefi Booting a UEFI guest: its spec, the firmware, its run directory, waiting for Windows to finish starting, stopping it.
qga The qemu-ga client: one connection at a time, resynchronized after a guest reboot.
relay The boot child's sockets: qemu-ga shared among its clients, COM1's input.
winexec Commands through qemu-ga: a script staged in the guest, its output streamed back, its exit code; file copies.
winsvc A Windows compose service: secrets, provisioning with restarts, stop.
wintap A guest on a host tap: its static address and the run's names, set over qemu-ga.
wineval How long an image built from evaluation media keeps working, as Windows reports it.
snapshot vk snapshot of a machine or a run, and the snapshot bundle.
vmmctl The run's vmm.sock: pause, resume, snapshot, quit.
console vk console on the serial console.

The run directory of a Windows machine holds its overlays (disk0.qcow2), uefi-vars.fd, tpm-state, system-uuid, vmgenid, the qemu-ga socket (qga.sock), the control socket (vmm.sock), the serial console (console.log, console.sock) and the VMM's own log (console.vmm.log).

A stop presses the ACPI power button and, if Windows is still up 10 seconds later, asks qemu-ga to shut it down. Three minutes after the button (ten for a build step's guest), vk asks the VMM to quit, which flushes the disks (a kill would leave a qcow2 overlay's cached metadata unwritten, and the next run of a kept disk would corrupt it), and kills it only if it is still running 10 seconds later. vk stop waits 210 seconds by default before it reports a run as stuck; a VM left behind by a vk that crashed ends itself the same way 11 minutes after the button.

Sharing builds between hosts

With a build cache registry server ([build] cache_registry, or --cache-registry), a Windows build shares its layers as Linux build steps do (wincache):

  • Push. The install, once settled, and each RUN/COPY layer go to build-cache:win-<key>, chunked by content (FastCDC, zstd), each chunk stored once registry wide. A layer whose parent is in the registry as this host has it is pushed as a diff: the parent's chunks are reused and only the ranges the step wrote are chunked again. The install is pushed whole.
  • Pull. A host missing a layer pulls it as a .vk_ro_img view: its chunks are fetched, then decompressed as Windows reads them, and the next step's overlay or a bundle builds on it. A pulled layer stands alone; it needs none of the layers under it, so a build takes only the registry's record of each layer it finds there (its password, its licensing) and pulls the one a step builds on or the bundle boots: a host whose target is cached downloads that layer only, never the install under it. The record's manifest digest pins what is pulled later. A failed lookup makes the layer here; a layer found but then not pulled (its manifest gone since) starts the build over taking no layer from the registry; a failed push leaves it local.
  • Once. While a host makes a layer it holds the registry's build lock on its key; the others wait for its push rather than install Windows again (up to three hours, longer than a Windows 11 install; then they build too, and say so).
  • Product keys. A --product-key stays in the install and every derived layer, including registry copies. Use a retail or MAK key only where every host and puller may use it. On the building host, the install medium containing it is accessible only to its owner.
  • Passwords. Each layer's entry records its install's Administrator password. Concurrent installs of the same key get different passwords, so a pulled layer must carry its own. These passwords are stored in plaintext: anyone who can pull from build-cache can read them, so restrict pull access to the build hosts.
  • Media. vk registry put <iso>... stores install media in the registry's files/ area, by digest (files/sha256:<hex>). A winiso: stage whose file is not in its build context fetches the one it pins from there (into the host's Windows cache), so neither the repository nor every host holds the ISOs. Each build that uses an install marks its media as used, so the registry's retention keeps them as long as the install is in use, unless they were put with --ttl, which drops them on time. vk registry put --keep keeps them regardless.

A store on disk — the builtin one, or a cache_registry naming a path — is not used: it would only hold a second copy of the host's own layer cache, and --cache-registry naming one is refused. A --reinstall never pulls the install: its keys count this host's reinstalls, which another host's count may match, so it installs again and pushes over what that host pushed.

Reclaiming. A CI host sweeps its Windows cache after each Windows build: an entry nobody references and nobody used for [executor] windows_cache_idle_secs (7 days by default; 0 is refused, as a build drops its reference to the install under a layer it is publishing) goes (layers, installs, media, fetched ISOs), never one a kept layer stands on, then the chunks no remaining pulled layer reads, and what a process that is gone left half made and has not touched for as long. The sweep first locks every idle entry nobody references, and keeps the chain of every other one. A build references each entry before it reads it, waiting behind the sweep's lock, and fails with "gone from the Windows build cache; build again" when the sweep took it; it holds its references to the end, and a CI job holds its service's whole chain for as long as it runs. A pull holds the chunk store until its layer is in place, and the sweep leaves the chunks alone while one does. A bundle vk build wrote holds nothing once the build ends: on a CI host it stops booting once its layers are reclaimed.

Tests

  • tests/windows-ad-e2e.sh: the AD lab (a DC, a Linux job creating its users, two members, RDP sign-ins), then a snapshot of the lab and its restore.
  • tests/windows-rdp-e2e.sh: a full Remote Desktop session into a standalone server.
  • tests/windows-ci-e2e.sh: a CI job's Windows service built from a Dockerfile, with ISOs from REGISTRY or a local build cache registry filled from ISO_DIR.
  • tests/int1-e2e.sh: a guest's int1 traps once, with the VMM stepping over it; says whether the host has the nested-SVM fault without it.
  • tests/release-e2e.sh runs the Windows tests when ISO_DIR names where the ISOs are, and int1-e2e.sh always.

Known limits

  • A domain controller ignores the ACPI power button: its winlogon waits on the Group Policy client, which on a domain controller started as a new machine can last as long as it runs. vk then asks qemu-ga after 10 seconds, and Active Directory takes most of a minute to stop.
  • Windows expects things a VM does not have, which it logs at each first boot: TPM-WMI 519 ("TPM has been cleared", Windows taking ownership of a new TPM), 1040 and 1041 (no EK certificate, so the device cannot be attested), and with Secure Boot off, 1796 (the Secure Boot update task cannot apply its updates).
  • A guest on a host tap gets its static address over qemu-ga, once it answers, about a minute after it starts; until then the adapter has what DHCP on the tap gave it, or nothing.
  • A Windows compose service also has an address on the run's switch, from the top of the subnet down (.254, .253, ...), like any compose service, besides its tap.
  • Build steps run without a TPM: a key sealed to an ephemeral TPM would leave the layer unbootable.
  • No hardware breakpoints in a Windows guest where the host delivers int1 onto itself (Hyper-V's nested SVM on AMD: WSL2, Azure): the VMM takes the guest's debug exceptions there to step over int1 (see VMM above), so a kernel debugger's hardware breakpoints and watchpoints in the guest do not fire. tests/int1-e2e.sh says whether a host has the fault.
  • Secure Boot without SMM guards the boot chain: the variables are out of the guest's reach now, but the firmware itself runs unprotected from the guest's kernel while it boots.
  • Restarting Windows during BitLocker's first encryption on a saturated host disk can leave it unbootable (open, under investigation): with the host disk taking tens of seconds per write (a QLC SSD past its cache, under several guests), a Windows 11 guest restarted mid-conversion (Enable-BitLocker -UsedSpaceOnly with a TPM protector) often does not boot again, with NTFS_FILE_SYSTEM, UNMOUNTABLE_BOOT_VOLUME or CRITICAL_SERVICE_FAILED (0xC0000428). It predates the host-kept variables and the flushing stop (it reproduces on 0.86.0). The 255 s disk timeout vk sets at install (Disk\TimeOutValue) stops the I/O errors and the bug check during the conversion, but not this failure. A restart under the same load without the conversion boots, and a machine encrypted with the earlier firmware keeps booting on this one.

Secure Boot database updates

Windows' Secure-Boot-Update task writes authenticated variables (db, dbx, KEK) at run time. While the firmware checked those writes itself, the first one stopped the guest with bug check 0x1E: a memory image of the faulting guest (vk snapshot taken just before, the guest-physical fault address from its crash dump, and the bytes there matched against the firmware modules' .debug files) put the fault in CRYPTO_new_ex_data, OpenSSL linked into VariableRuntimeDxe, following a pointer into the crypto heap by its physical address. The crypto state built during boot keeps physical addresses, and Windows, unlike Linux, maps the runtime services only at the virtual addresses it gives them. The variables are now the host's (above), and no signature is checked in the guest any more: the same update on Windows 11 applies (event 1034, dbx from 76 to 14072 bytes) and dbx keeps it across a restart.