GitLab CI executor

virtkit ships a GitLab custom executor that runs each CI job in a throwaway microVM: a fresh guest per job, destroyed when the job ends, with its own kernel and no shared state between jobs. Concurrent jobs each get their own VM. This guide covers configuring the runner, selecting the job image, sizing the guest, keeping the host inside its memory, controlling egress, and attaching services.

For the host-side setup mechanics (runner wiring, state layout, exit codes) see also the driver README; the full config reference is config.example.toml.

Runner wiring

Point the runner's custom-executor stages at vk in /etc/gitlab-runner/config.toml:

[[runners]]
  [runners.custom]
    config_exec  = "/usr/local/bin/vk"
    config_args  = ["gitlab", "config"]
    prepare_exec = "/usr/local/bin/vk"
    prepare_args = ["gitlab", "prepare"]
    run_exec     = "/usr/local/bin/vk"
    run_args     = ["gitlab", "run"]
    cleanup_exec = "/usr/local/bin/vk"
    cleanup_args = ["gitlab", "cleanup"]

The executor reads its host configuration from the usual virtkit config file ($VIRTKIT_CONFIG or /etc/virtkit/config.toml). Everything below that a job controls is set through .gitlab-ci.yml job variables.

Nested virtualization

Job VMs mask VMX/SVM, so a job cannot boot microVMs of its own. A runner that hosts jobs which build or test virtkit itself grants them that:

[executor.vm]
nested = true

It applies to the job VM whatever MICROVM_IMAGE names. In a compose: fleet it also unlocks the per-service marker, so the nesting guest can be a sibling rather than the primary:

services:
  builder:
    image: virtkit/build
    x-virtkit: { nested: true }

Without the grant that marker is refused when the fleet loads rather than cleared — a fleet that asked for a nesting builder must not look like it got one. Nesting reaches host KVM's nested paths, so the grant is the host admin's to make and there is deliberately no job variable for it.

The grant is per host and unconditional: every job VM this runner boots gets VMX/SVM, asked for or not. Set it on a runner dedicated to trusted pipelines, not one that also runs fork merge requests.

The host must be loaded with kvm_intel.nested=1 / kvm_amd.nested=1; vk check --feature gitlab reports when it is not, and prepare refuses the job rather than booting a guest that cannot nest.

CI tools

[executor] tools_dir names a host directory of static tools — git, git-remote-http(s), git-lfs, gitlab-runner — shared read-only into every job VM; the guest links each one its image lacks onto its PATH. On a host that runs a gitlab-runner of its own — one vk node manages, or one on systemd's PATH — vk check --feature gitlab fails without a gitlab-runner there: that runner's jobs transfer no artifacts, caches or dotenv reports without one. Jobs a hub places need only git. The directory is resolved at each job's start, so pointing a link at a new one changes the tools of the jobs that start after it.

On a fleet node the hub can provision it: vk-hub tools add holds a build context whose tools stage holds the tools — a definition need only provide git — and vk-hub nodes tools has nodes build it and install the result as <state_dir>/tools/current (CI tools). Once the node's first build is done, as vk-hub nodes shows, point the executor there; before that the link does not exist, and jobs given it fail to boot:

[executor]
tools_dir = "/var/lib/virtkit/tools/current"   # <state_dir>/tools/current

Host-side checkouts

A host-side checkout — which dockerfile: and compose: job images need — clones the sources on the runner instead of in the guest. Point checkout_dir at the runner's builds tmpfs when the state disk is too slow:

[executor]
host_checkout = true
checkout_dir = "/builds"
checkout_cache_idle_secs = 1800

A tree is reused by concurrent slot and project, so a second job on the same slot fetches rather than re-clones. Before each checkout, and during vk gc, trees unused for longer than this are removed — which is what keeps a tmpfs checkout_dir from filling with the repositories of jobs that have moved on. Unset, checkouts inherit image_cache_idle_secs (30 minutes by default). A tree a prepare or a running guest is using is never removed.

vk gc also collects the host's local build cache: entries unused for 14 days, then the least recently used until it fits 10% of its filesystem, at most 50 GiB. A runner that relies on a large local build cache sets [gc] build_cache_retention_days = 0 and build_cache_max = "none" to keep it whole.

Under an explicit checkout_dir, virtkit keeps its trees in <checkout_dir>/vk/<slot>/<project>, so the sweep walks only its own — a sibling tree from another GitLab executor is outside the root it reads, and only trees virtkit created are eligible within it. Each tree also records which directory it is, rechecked immediately before removal, so bookkeeping left behind by a tree removed from outside cannot stand in for whatever appears at that path next. The configured root must still be writable only by the runner user: anyone who can replace the private subtree can interfere with the runner's checkouts.

Releases up to 0.31.0 put their checkouts directly under an explicit checkout_dir, which leaves them outside the new root, so an upgrade neither migrates nor reclaims them: remove the old <checkout_dir>/<slot> directories once no runner from before the upgrade is using them. The first job per slot and project after the upgrade clones instead of fetching, since its tree has moved. Under the default <state_dir>/checkouts root the trees keep their paths and are still fetched into, but carry no identity record until the same slot and project is checked out again, which is when they become reclaimable.

Job variables

Job variables are read once per job, at the job level. GitLab passes them to the executor prefixed with CUSTOM_ENV_; set them under a job's (or the pipeline's) variables:. An empty string is treated exactly like an unset variable — it does not override the host default.

Variable	Effect
`MICROVM_IMAGE`	Guest image (prefix-based source; see below). Unset → `local/default`.
`MICROVM_CPUS`	vCPU count, clamped to the host `[executor.vm] max_cpus` ceiling.
`MICROVM_MEM`	Guest RAM as `<n>G`, clamped to `[executor.vm] max_mem`.
`MICROVM_USER`	User to run the job as inside the guest.
`MICROVM_EGRESS_ALLOW_IP` / `_ALLOW_NAME` / `_AUDIT` / `_DRY_RUN`	Narrow / audit / dry-run the run-phase egress cap (see Egress).
`MICROVM_BUILD_EGRESS_ALLOW_IP` / `_ALLOW_NAME` / `_AUDIT`	Narrow the build-phase egress cap.
`MICROVM_USAGE_REPORT`	End this job's trace with what every job of its project has been using (see Sizing a project).

Image selection

MICROVM_IMAGE is prefix-based — the prefix names the source (when unset, the job's plain GitLab image: is read the same way):

unset → local/default (the baked default bundle);
local/<name> — a bundle directory under [local] dir;
virtkit/<name>[:tag|@sha256:…] — a bundle in the [registry] repo;
docker/<name>[:tag|@sha256:…] — an OCI image from the [docker] repo, booted directly (embedded kernel + agent). A reference [docker] does not route — and every one of them when that section is absent — is offered first to the vk-registry this runner already uses ([build] cache_registry/[registry]), which relays and caches the upstreams it is configured for, and pulled from the registry it names otherwise;
dockerfile:<path>[?context=<dir>&buildcontext=NAME=DIR&arg=NAME=VALUE][#<stage>] — a git-defined image: virtkit builds the Dockerfile from the job's own checkout (each RUN in a microVM, no Docker involved) and boots the result, so the job image lives in the repo instead of a registry. <path> is relative to the repo root; the build context defaults to the Dockerfile's directory (?context=. for the repo root), ?arg= supplies a --build-arg (repeatable), ?buildcontext=NAME=DIR (repeatable) declares an extra repo-root-relative directory a COPY --from=NAME or RUN --mount=…,from=NAME may read, and #<stage> selects a stage. Every path stays inside the checkout. Built images are cached and shared across jobs and runners. Requires [executor] host_checkout;
compose:<file>#<primary> — a whole fleet from a compose file in the checkout: boots <primary> as the job VM and the other services (built or pulled the same way) as siblings on the job network. Same host_checkout requirement. A service sizes its own guest with an x-virtkit: { cpus:, mem: } marker (default 2 vCPUs / 1G), clamped to the host [executor.vm] max_cpus/max_mem ceilings like the job's own MICROVM_CPUS/MICROVM_MEM; the primary keeps following those job variables. x-virtkit: { nested: true } needs the runner to have set [executor.vm] nested and is refused when the fleet loads otherwise: nesting reaches host KVM, so it is the runner's decision rather than a job's, the same way the executor never hands a job the guest PMU. A service's persistent state — x-virtkit: { persist_root: true }, overlay,persist volumes (see the README's Volumes and persistent state) — lives under .virtkit/ beside the compose file, inside the checkout that every job starts by cleaning: it lasts for the job, across a service being stopped and started within it, and never from one job to the next. The job's script runs commands in its services with vk-agent exec --service NAME [-b] [-t] [--user U] [--dir D] [--env K=V] [--clear-env] -- CMD…, with vk exec's flags: the job VM's agent (/run/vk/bin/vk-agent, its directory last on the PATH) passes it to the host over the VM's host command channel, which runs vk exec --service with it, into the job's own services, and refuses anything else — no other host command, no other job. An image's own full vk keeps vk exec --service for the VMs it runs itself. It does grant the job more than its network does: each request starts a host vk process (at most 4 at once), the command runs as any user of the service it names, root included, and a service's egress is the job's to use. [executor] guest_service_exec = false turns it off.

my-job:
  variables:
    MICROVM_IMAGE: virtkit/myimage      # :tag (default latest) or @sha256:…

test-in-repo-image:
  variables:
    MICROVM_IMAGE: dockerfile:ci/Dockerfile#test   # built from this checkout, then booted

Resource usage

Each phase reports what it cost the runner. A job that builds its own image (image: dockerfile:… / compose:…) gets the build figures as the build ends, next to its timing breakdown — a job whose image is already built runs no build and reports none:

virtkit: build resource usage: cpu 8m12s, peak memory 3.2 GiB (largest process 1.2 GiB), read 1.8 GiB, written 4.1 GiB, sent 12 MiB, received 2.7 GiB

and the run figures come at the very end of the trace:

virtkit: job resource usage: cpu 2m14s, peak memory 1.6 GiB, overlay 1.2 GiB of 16.0 GiB, read 3.4 GiB, written 812 MiB, sent 3 MiB, received 941 MiB

cpu is all the CPU time the phase burned on the host, the guests' own execution included — for the build, the stage guests plus vk's own work assembling and caching the image; for the run, the job's microVM plus the host processes around it (the switch, the virtio-fs daemons, any service VM). Against the phase's own duration it gives the parallelism it reached: a ceiling for sizing one guest's vCPUs, since the total also carries the host helpers and every sibling VM.

When the guest kernel's OOM killer terminates a process, a line identifies each victim before the figures above:

virtkit: guest OOM: the kernel killed cc1plus (pid 1234, 1.9 GiB RSS) at +48s (raise MICROVM_MEM)

The size is the victim's reclaimed anonymous RSS, and +48s is its guest uptime at death. A step that exits with signal 9 and no message often indicates an OOM kill; raise MICROVM_MEM. No line means either that no process was killed or that the guest was too old or unresponsive to report it.

When the VMM detects and retries a virtio-fs request with a missing notification, a line near the end of the trace reports the warning count and quotes the first warning from the VMM log:

virtkit: virtio-fs stall (2 times): virtio-fs /builds/acme/web: queue 1: 1 completion(s) past used_event for 10s (…): suspected lost interrupt, signalling again

Under a fleet node the job also reports it to vk-hub among its events (see fleet-prototype.md).

peak memory is the most the phase held at one time, not the memory left at the end: a guest hands freed RAM straight back to the host, so its live figure says little about the peak it passed through. A build reports its largest single process as well, because the two size different knobs:

the total is what the host had to have free while the phase ran — for a build, several stage guests at once, so lower it with [build] jobs; for a job, the run VM and its helpers together, i.e. MICROVM_MEM plus the VMM's own overhead.
the largest process — a stage guest, or vk itself — is what [build] mem has to cover for one stage.

The two are measured as their processes allow, and err in opposite directions. A build's guests are gone by the time it reports, so both its memory figures are sampled while it runs — a spike shorter than a tenth of a second can be missed, and longer than that on a host where reading the process tree is expensive, since the sampler backs off from its own cost. Each sweep totals and ranks the same reads, so the total always covers the largest process reported beside it. A job's VM is read in one pass instead, summing the high-water mark of every process still running at the end: marks that never coincided both count, while a helper that peaked and exited earlier contributes its CPU time but no memory. Read either figure as a ceiling — pages the VMM shares with its virtio-fs daemons count in each — and for a job with services it spans every service VM, not just the one the stages run in.

Either line is omitted rather than guessed at when the figures cannot be had: the run ones for a job whose guest died and took the supervisor with it, the build ones for a build that shared the host process with another (both would be charged for each other's guests) or that the host could spare no sampler thread for.

overlay is how full the job filled its writable layer, against what that layer held. With [executor] checkout_overlay (the default) a job builds on an overlay above its checkout whose upper layer is a tmpfs inside the VM, so every write under CI_PROJECT_DIR — the build tree, a package cache pointed there, an unpacked archive — is guest RAM, capped at [executor] checkout_overlay_size (80% of the VM's memory by default). That cap is a wall a job can hit: it fails with ENOSPC while every disk on the host sits empty, and the written figure beside it says the job wrote nothing at all, because none of those pages ever reached a block device. Read the pair as the room the job had left — 4.2 GiB of 16.0 GiB has plenty, 15.9 GiB of 16.0 GiB is a job about to fail on space — and raise MICROVM_MEM when a build tree needs more, since the capacity follows it.

Because that checkout is read-only for the job's life, the guest caches its paths, attributes, misses and directory listings for the whole job: git status or a build tool's dependency check walks the tree against the host once, and every later pass is answered from the guest's own cache. With checkout_overlay = false the share is read-write and keeps close-to-open consistency (attributes re-fetched after 5 s). Its tree, .git included, is then the guest's to rewrite, so the next job on the slot clones afresh rather than running the host's git in it.

Caching the paths is not the same as caching the files: a file the job reads still crosses virtio-fs once, and a tree of 100k small files costs about 20 s per pass that way. So with [executor] checkout_tmpfs (the default alongside checkout_overlay) the executor packs the worktree — everything but .git — into one tar at boot, which the guest streams through the share's DAX window and unpacks onto the overlay's tmpfs upper in well under a second. Every file then reads from guest RAM; only .git is still served by the lower, and its packfiles are large enough to be DAX-mapped. The end-of-job checkout seeded into the guest tmpfs trace line reports the tar's size and host packing time; the guest's unpack time is in its console log (vk-agent init: seeded the overlay …). The tree's size is part of the overlay figure and of what checkout_overlay_size must hold, so a small MICROVM_MEM on a large repository is where to look if the layer fills sooner than expected. checkout_tmpfs = false leaves the lower to serve files on demand.

The figure is the high-water mark, not what the layer held at the end: a job that unpacks an archive and deletes it would otherwise read as having needed nothing. It comes from the guest, which is the only place a tmpfs can be seen from, so a job with no overlaid checkout reports none — as does one on a guest whose agent is older than the figure.

Sizing the writable layer

[executor] checkout_overlay_size is where the two failures either side of that wall are traded off, and the mark above is what to size it from:

too low and a job dies for want of a partition on a VM that had the memory for it — the build tree hits the cap while gigabytes of guest RAM sit unused;
too near 100% and the layer starves the job's own processes instead. Compilers spike late (linking), so the failure lands as an OOM kill of whatever the kernel picks — far harder to read than a filesystem reporting itself full.

It costs nothing below the cap: tmpfs pages are allocated on use, so a job that never fills the layer is unaffected by how large it was allowed to grow. Nor does raising it commit host memory — [executor.schedule] admission reserves MICROVM_MEM either way, so the host is already sized for a job that uses all of its VM.

The default is 80% rather than the kernel's own 50% tmpfs default. That default exists to keep an unevictable tmpfs from starving the long-lived services of a general-purpose machine; a one-shot job guest has no such services, and the tree it is there to build is the legitimate main consumer of its memory. Set "50%" to restore the kernel's behaviour, or an absolute "12G" where every job on the runner is the same shape.

read and written are what the phase cost the runner's storage, not what its programs asked for: read is what was actually fetched from the block layer, so a guest re-reading a file it has already pulled into its own page cache costs nothing here, and written is what was handed over to be written out. Two consequences are worth expecting rather than discovering:

work that never reaches a disk is not counted, because there is nothing to count: a [executor] checkout_dir pointed at the runner's builds tmpfs, or a guest write absorbed by its overlay, is RAM, and a tmpfs page never reaches the block layer;
a job re-run on a warm host reads far less than the same job on a cold one, which is the cache doing its job rather than the measurement wavering.

sent and received are what the phase's guests moved in and out of the host, counted by the switch that forwards it — payload from the guests' side, so the framing around it and any retransmit under it are the host's traffic rather than the job's. What the guests send each other is switched between them without ever being forwarded, so it is not in the figure. Each phase counts its own: what a dockerfile: build's RUN steps fetched is on the build line, the packages the job's steps installed and the artifacts they uploaded on the job line. Only what a guest pulled is in either: vk resolves and pulls a base image itself, on the host and outside the switch, so those layers are in neither figure. A job on net.mode = "tap" has no switch in its path, so it reports no network at all rather than zero.

A measured zero is printed — read 0 B says this job touched no disk, which is a fact about the job. The clause is left off only where the figure could not be taken at all: a kernel built without CONFIG_TASK_IO_ACCOUNTING, or a network nothing counted. The history keeps the two apart as well, so a host that cannot measure never leaves a job remembered as having moved nothing, and vk check says whether this kernel accounts block I/O:

ok   usage    block I/O accounted, process tree from the kernel's child lists

What the job did, second by second

The figures above are one number per phase. For the shape of a job over time — which step saturated the CPUs, when memory climbed, what process was running while nothing else moved — each job's guest also records itself: every 10 seconds it samples its own /proc and appends a sample to a log the host keeps after the VM is gone.

<state_dir>/atop/2026-08-11/42137-acme-web-test_unit/atop.log

Once the job's VM is gone, the host compresses the log with zstd to atop.log.zst beside it, about twenty times smaller; a runner that died before its job's cleanup leaves the log plain, and a later day's first job compresses it — one over 512 MiB stays plain. vk atop reads either.

Beside it, vmm-threads.txt records what the threads of the job VM's VMM were doing when the host stopped it: each thread's name, state, wait channel and the syscall it was blocked in, with the file its first argument names when that is an fd. A job that hung on a guest request the host never answered (e.g. an fs worker thread stuck in a syscall) ends at its timeout, and this is what is left of the VMM. Kernel stacks are included only when the runner runs as root. The record is kept only where the guest's statistics are, and a runner that dies before the job's cleanup loses it.

Every job's trace ends with the account of it, in a section GitLab's web UI shows folded — one line to open when a job is worth looking into, and out of the way when it is not:

▶ what the job's guest did (vk atop 42137)

Opened, it holds exactly what --summary prints below. A raw trace (the artifact, curl of the trace API, a runner log) has no UI to fold it, so it shows the same lines unfolded between the section markers — which is what a reader piping a trace through grep wants anyway. A job whose guest recorded nothing gets no section at all rather than an empty one.

On the runner, vk atop finds one again from a job id, or from any part of a recorded job's name — the newest run answering — printing just the path, so it composes with whatever reads it. Nothing matching means an empty stdout and a non-zero exit:

vk atop 42137                    # that run
zstdless $(vk atop test_unit)    # the last run of this job, compressed or not

The name to give is the one in the directory, which is the job's name with anything outside [A-Za-z0-9._-] replaced — a test:unit job is recorded as test_unit. A job id matches only the id a directory name leads with, so 42 never answers for job 42137.

--summary reads the log for you instead, accounting the whole job — what its guest did with its processors and memory, what it moved, where it was held up, and which of its processes the time went to:

$ vk atop 42137 --summary
virtkit: 42137-acme-web-test_unit — what its guest did:
  recorded         2026/08/12 07:21:49 → 07:21:59 UTC (10s), 11 samples at 1s, the first
                   covering the guest's boot (counted in the totals, not the rates)
  guest            runner on 2 cpus, 988 MiB memory, no swap
  cpu              1.7s of cpu time — 0.2s user, 0.9s system, 0.6s stolen by the host; 0.1s
                   waiting for disk; 16% busy at peak, 5% on average, cpu 1 busiest at 22%
                   of its own time
  load             0.00 at peak, 0.00 / 0.00 / 0.00 at the end (1m / 5m / 15m); 285 context
                   switches a second at peak
  memory           63 MiB held at peak (6% of the VM), 146 MiB of cache at peak
  pressure         cpu 182ms, 0.9% at 07:21:51; io 50ms, 0.3% at 07:21:55
  disk             2 MiB read, 92 KiB written — vda busiest, 96ms busy
  network          eth0 received 42 B, sent 714 B
  cpu over time    ▁▁▁▁▁▁▁▁▁▂  (peak 16%)
  memory over time ▁▁▁▁▁▁▁▁▁▁  (peak 6%)

  what ran — the 10 of 61 that used the most cpu
  command                                   pid   cpu  peak rss    read  written
  cc1plus ×1184 (2 failed)                    -  6m12s   184 MiB   1 GiB   96 MiB
  /init                                       1  0.3s     11 MiB   2 MiB   20 KiB
  …

(Wrapped here to fit the page — the real output is one line per label.)

The commands a job ran over and over are one row each, with the number of runs and how many of them failed: a build that forks a thousand compilers is a thousand processes no sweep ever saw, and one row of cc1plus ×1184 says what a thousand rows of one run each would bury. A process the sampler did watch keeps its own row, however many namesakes came and went around it.

A multi-threaded process appears in the stalled section after the table if it did some work, then spent at least five minutes in the same set of kernel wait channels, using under 10% of a cpu with no disk I/O. The set is what counts, not how many threads wait in each channel: threads polling a stuck worker (a 10 ms sleep loop, say) are caught running now and then, which changes the counts every few samples, and cost a hung process a few percent of a cpu. A process that has done no work since guest boot is excluded. Other idle daemons in a fixed set of channels (a JVM or Gradle daemon, a dockerd the job started) are listed too. Processes the job started come first, then the longest stalls. The section lists wait channels and available kernel stacks, one line per distinct stack:

  stalled — its threads in the same wait channels, under 10% of a cpu and no disk i/o, for 5m00s or more
  ruff check . (pid 412, 9 threads) for 11h58m from 07:22:10 UTC
    waiting: 1 in futex_do_wait, 1 in request_wait_answer, 7 in hrtimer_nanosleep
    stacks at 07:27:10 UTC:
      1 thread (412): futex_do_wait < __futex_wait < …
      1 thread (420): request_wait_answer < fuse_simple_request < …
      7 threads (413 414 415 …): hrtimer_nanosleep < do_nanosleep < …

The guest applies the same test to the same samples and writes those stacks once per process, at its first stall and for its first 256 threads; the stacks at line says when, and how many threads they cover when that is not all of them. The frames need a guest kernel built with CONFIG_STACKTRACE; without it, each thread shows only its wait channel and (no stack). A thread stuck in request_wait_answer or another fuse_* function is waiting on a virtio-fs request; one in futex_* alone is waiting on a lock in the tool itself.

--view walks the recording a sample at a time in a full-screen panel — the guest's processors (each core its own bar), memory and swap, pressure, disks and interfaces, then the processes ordered by what they were using at that moment:

42137-acme-web-test_unit — sample 11/11 at 07:21:59 +1s
cpu    16% [██          ]  user   3%  sys  10%  wait   0%  steal   3%  0.2s of cpu time
cores 0:[▊       ]  10%  1:[█▊      ]  22%
mem     6% [▊           ]  63 MiB of 988 MiB  cache 146 MiB  no swap
psi   cpu  0.4%  mem  0.0%  mem-full  0.0%  io  0.0%  io-full  0.0%
disk  vda read 0 B/s write 4 KiB/s busy 4ms
net   eth0 in 0 B/s out 42 B/s   0 tcp connections, 0 resent

    pid  st      >cpu     memory       disk  command
     65   S      0.1s      1 MiB      8 KiB  sh
    700   E      0.0s    512 KiB      1 KiB  true
      1   S      0.0s     11 MiB        0 B  /init
…
←/→ step  home/end jump  c/m/d sort by cpu  a whole job  / filter  q quit

The st column is the state each task was in, E marking one that ended during the sample. ←/→ step through the samples and Home/End jump to either end; c, m and d order the processes by processor time, memory or disk; a swaps each process's activity in this sample for its whole-job totals; / filters the table to the commands matching what you type; q leaves. The first sample is marked, because its counters cover the guest's boot rather than an interval. The filter matches on ASCII.

--follow is the same panel over a job that is still running: samples appear as the guest commits them, and while the view sits on the last one it moves with them. Stepping back holds it still — the header says [paused] — until End, or stepping forward onto the last sample, takes the live tail up again. Both need a terminal on stdin and stdout; without one (a pipe, or TERM=dumb) they say so and point at --summary. A process's disk figures read - where the guest's kernel accounted none, as in the summary.

Totals cover every sample, including the first — a job's VM boot is part of what the job cost. A figure computed over an interval — the cpu percentages, the rate of context switching, the sparklines — leaves that first sample out, because it covers however long the guest had been up rather than one interval, and averaging a boot into the picture flattens everything after it. A figure that was simply the reading at a moment — the load, the memory held, a pressure average — comes from every sample, the first included, so a peak the guest reached while booting is still reported as one. pressure is the line to read when a job is slow for no visible reason: it is time the guest spent waiting for a processor, for memory or for its disk, which no total of what it used can show. A figure the kernel could not measure prints as -, never as a zero.

--json writes the samples themselves, one object per line, for anything that would rather compute than read. The units are the log's own: pages, with their pagesize beside them; ticks, with their hertz beside them; 512-byte sectors; and KiB for a process's resident size (rsize_kib). A counter the guest's kernel does not have is null rather than a zero — as is a scale whose record a sample did not carry, so check hertz and pagesize before dividing by one. Each process has threads and threads_running, the thread count and how many threads were running when the sample was taken. A multi-threaded one also has wchans, its non-running threads counted by kernel wait channel as last recorded, and wchans_since, the epoch at which that histogram was recorded; both are null when none is recorded (always for a single-threaded one). stacks is the one-time dump of a stalled process's per-thread kernel stacks (tid, wchan, frames), present in the one sample that recorded it and [] everywhere else, so it shows that moment, not the process's current state. A log with no complete sample writes nothing and exits 0, where --summary reports that there is nothing to account yet:

vk atop 42137 --json | jq -c '{t: .epoch, user_sys: (.cpu.user + .cpu.system)}'
vk atop 42137 --json | jq '.procs | max_by(.rsize_kib) | {name, rsize_kib}'

The format is the text atop -P prints, pinned to the field order of atop 2.8.1 (what Debian 12 ships), so anything that already reads that — a parser, or plain grep/awk — reads these logs. Each line is one record: a label, this guest's name, the epoch, the date and time, the seconds the sample covers, then the label's own fields. A SEP line closes each sample, and a RESET line opens the first one, whose counters cover the guest's whole boot. Counter labels carry per-interval differences; size labels carry the value as it stood.

The system labels are CPU, cpu (per processor), CPL, MEM, SWP, PAG, PSI, DSK and NET; every process gets a PRG, PRC, PRM and PRD line, and so does every task that ended during the interval — those carry state E, the status they exited with and how long they lived. So the busiest samples of a job are one sort away (zstdcat passes a running job's plain atop.log through unchanged):

zstdcat atop.log.zst | awk '$1 == "CPU" { print $5, $9 + $10 }' | sort -k2 -n | tail   # time, busy ticks
zstdcat atop.log.zst | grep '^PRM ' | sort -k12 -n | tail   # largest processes (column 12 only
                                                            # while no process name holds a
                                                            # space — see below)

Two labels are virtkit's own, which atop-format parsers skip as unknown. PRW lists a multi-threaded process's non-running threads by wait channel (<pid> (<name>) futex_do_wait:1 hrtimer_nanosleep:7 …). It is written only when that histogram changes, so the latest line stands until the next one. PRK holds one thread's kernel stack (<pid> (<name>) <tid> <wchan> <frame>;<frame>;…, frames - without CONFIG_STACKTRACE) and is written once for each thread of a stalled process.

Worth knowing before reading a log:

it is the job's own VM, not the fleet: a compose: job's service VMs are not sampled;
a process that starts and exits between two samples is still recorded — the guest's kernel reports every task as it dies, and those land in the sample covering the exit with state E, their whole life on them. Two things follow: a dead process has no command line left to read, so it appears under the name the kernel keeps (cc1plus, not the arguments it was given), and a guest exiting tasks faster than the sampler can read them loses some — the log says so on the guest console, and it takes extreme churn;
a thread exiting is not recorded on its own: its time is charged to the process it belongs to, which is what the process's own record carries;
there are no per-process network figures (PRN), which real atop needs a kernel module for. What the job moved in and out of the host is already on its resource line above;
a disk that moved nothing in a sample gets no DSK line in it: the guest kernel carries sixteen ramdisks and eight loop devices that never see a sector;
a field this guest cannot source carries the value atop itself prints when it has no answer — CPU frequency, cgroup limits and the proportional set size are the ones to ignore rather than read;
a process's name and command line are parenthesised and may hold spaces, so column numbers shift on the PR* labels: awk/sort -k are safe on the system labels, and a PRG/PRM line has to be split on its parentheses first;
the timestamps and the date directory are UTC.

Recording is not free, and it is on by default. Each job's guest gets a read-write virtio-fs share of its own archive directory — the one directory it can write, and it can write anything and any amount into it until the job ends — and it boots with psi=1, which its own scheduler pays for in exchange for the PSI label. atop = false gives up the recording and the share; the guest still boots with psi=1, since idle page-cache trimming asks for the same figures — only turning that off as well ([executor.vm] reclaim or [executor.vm] balloon) gives the scheduler it back.

Set [executor] atop_interval_secs for a finer or coarser resolution, and atop = false to record nothing:

[executor]
atop = true                # default
atop_interval_secs = 10    # default; at least 1
atop_retention_days = 14   # default; 0 keeps only today's jobs

The archive is kept for atop_retention_days and then reclaimed: each day past the window is dropped whole by the first job recorded the following day — once a day, not once a job — so the directory stays bounded on a runner nobody visits. The window is counted in whole UTC days beside today's, so 14 leaves fifteen date directories. Only a directory whose name is a date in that YYYY-MM-DD form is ever dropped: a file, a symlink, or anything else an operator leaves in the archive is never touched, so a log worth keeping can simply be renamed.

Two things to know before setting it. A day goes even if a guest is still recording into it, so a job that outlives the window has its log unlinked while it is still writing — the guest keeps appending to a file nobody can open again. With 0 that is any job running past midnight, and with a short window any job outliving it. And the reclamation belongs to the recording: atop = false stops both, so an archive already on disk stays at the size it reached and has to be removed by hand.

Sizing the two phases

The phases are sized independently, and by different people: the run VM by the job, the build by the host.

	Run phase	Build phase
vCPUs	`[executor.vm] cpus`, per job `MICROVM_CPUS` (capped by `[executor.vm] max_cpus`)	`[build] cpus`, per stage guest — unset = the host's CPUs shared out among the stages running at once, between 2 and 4 each
RAM	`[executor.vm] mem`, per job `MICROVM_MEM` (capped by `[executor.vm] max_mem`)	`[build] mem`, per stage guest — unset = `4G`; a stage overrides it with `# vk: mem=…` or `--stage-mem`
Concurrency	one VM per job (the runner's own `concurrent`)	`[build] jobs` stages at once — unset = as many of this build's stages, smallest first, as fit in 80% of host `MemTotal`, capped at 16

Build sizing is host configuration only — there are no MICROVM_* equivalents, because built images are cached and shared across jobs and runners, so no single job owns the build guest it happens to trigger. Raising [build] mem lowers the derived jobs count unless you also set it: per-stage headroom and stage concurrency trade against each other.

jobs is a ceiling, not a promise. It is derived from the memory the host has, so that a build's width does not depend on the minute it started — but a host also carries job VMs, other builds and whatever else it runs, so a stage additionally reserves its guest RAM against what is free before it boots, and waits when the host is full. The build announces both, and a stage parked on memory says so in the trace:

virtkit: build: up to 6 stage(s) at once (from host memory), each cpus=12, mem=4G
virtkit: build: host memory 31241 MiB, 15180 MiB in use elsewhere, 3124 MiB held back — 12937 MiB free for stage guests now

With stages sized individually the ceiling is not a division: it is as many of this build's stages as fit, smallest first, in the same 80% — and never more stages than the build has. The stages that differ are named on the line, and the gate's own reading is in MiB, since a build of one 24G stage and four 2G ones has no single number of stages its free memory is worth:

virtkit: build: up to 4 stage(s) at once (from host memory), each cpus=12, mem=4G; sized apart: compile mem=8G cpus=16

One stage often needs far more than the rest — the compile, the link — and sizing every stage for it wastes both memory and concurrency. A # vk: comment above a stage's FROM sizes that stage alone:

# vk: mem=8G cpus=16
FROM rust:1.90 AS compile

mem and cpus are the keys; either may be omitted, and a stage without the comment takes [build] mem / [build] cpus as before. vk build --stage-mem compile=8G --stage-cpus compile=16 says the same thing for one run without editing the file — the flags win over the comment, field by field, and name a stage by its AS name, or stage<N> by position without one. In a build of several units (--compose, or several --target) a name sizes that stage in every unit declaring it. A flag naming a stage no unit declares fails the build, as a hint that cannot mean anything does. It rides a comment so the Dockerfile still builds with docker build, and so the size stays out of the instruction stream cache keys are computed from: a stage built at 2G and the same stage built at 8G produce the same artifact under the same key, so tuning one throws no cache away. A hint larger than the host can give one stage is held to what the host has, with a line in the build log — the 24G a stage wants on the build server must not make it unbuildable on a laptop. Anything the hint cannot mean — an unknown key, an unreadable size, a # vk: line that precedes no FROM — fails the build rather than being ignored.

Each stage reports its peak guest memory when it finishes and in the final timing breakdown, alongside the guest size. The guest measures MemTotal - MemAvailable, excluding reclaimable page cache that a host-side VMM measurement would include. The figure is read from each of a stage's guests as it goes, so a stage killed out of memory prints no completion line, and the final breakdown carries only what its earlier guests had reported — a lower bound on what it needed, or nothing at all if it died in its first.

Back to the gate: a build's own guests are charged at their declared size from the moment they are admitted, never at what they have faulted in so far — a guest that booted seconds ago has touched almost none of its RAM, and a gate that believed MemAvailable would wave the next stages in against memory the first one is about to take. Everything else on the host is measured afresh each time a stage is about to be admitted, so a build narrows when a job VM starts beside it and widens again when one ends. A stage with nothing else of this build's live is always admitted, whatever the host looks like: a build with no way to make room should be slow, not stuck. Stages that do wait are admitted oldest first, so a large stage is not overtaken indefinitely by the smaller ones queued behind it.

The measurement is of memory actually in use, so it lags a job VM that [executor.schedule] mem_budget has just granted but which has not faulted its RAM in yet — the same trap the gate avoids for the build's own guests, one level out. Where jobs and builds share a host, that is a reason to bound both up front rather than to rely on the gate alone. Set [build] no_mem_gate to switch the gate off entirely and let jobs bound the build by itself.

Giving idle memory back

A guest holds every file page it ever read until something in it needs the memory, so a job that untarred a source tree and built it goes on holding that tree long after the stage that read it — duplicated in the host's own page cache, or, on a busy runner, in its swap. Pages a guest frees already come back through the balloon's free-page reporting; the file cache does not.

[executor.vm] reclaim makes it come back too:

[executor.vm]
reclaim = "auto"      # the default

auto evicts, whenever the guest is not under memory pressure, the file cache it has not touched for a minute or two — by age, through the kernel's multi-gen LRU, so what a build is still re-reading stays and what it read once goes. A size ("512M", "2G") or a share of the guest's RAM ("5%") keeps that much cache as a fixed floor instead, and "off" leaves the guest alone. Compose services follow the job VM's setting unless they declare their own x-virtkit.reclaim.

It rides on the balloon, so [executor.vm] balloon = false stops the job VM trimming: without free-page reporting the trimmed pages never reach the host, and the job would lose its cache for nothing. Compose services attach a balloon of their own whatever [executor.vm] balloon says, so they go on trimming under the setting they inherited.

Sharing the host page cache with the guests

Job virtio-fs shares — the checkout (host_checkout), tools tree and [executor.share] — use DAX windows to read the host page cache directly. Previously, each guest cached its own copy: ten concurrent job VMs reading a tools tree held eleven copies on the host. [executor.vm] dax sets the window size:

[executor.vm]
dax = "8G"            # per share; the default: files of 1M and more through the window
# dax = "8G:always"   # every file through the window
# dax = "8G:inode=64K" # another size floor

The window reserves guest address space, not memory, and costs nothing until mapped. "off" disables it. Each mapping costs the host an mmap and the guest an EPT invalidation per 2 MiB range whatever the file's size, which a source tree's small files never repay: by default only regular files of 1M and more are mapped (dax=inode; the host marks them), the rest read through the guest page cache as without DAX. Each guest supports 64G of windows — eight at the default size, with further shares served without DAX. Guests with more than 63.25G of RAM have no room for windows and receive none. Mappings are 4 KiB-granular, so the benefit is avoiding a tools tree's memory cost per VM, not per-fault latency. Compose services inherit the job VM's setting unless they declare x-virtkit.dax.

Keeping the host inside its memory

gitlab-runner decides how many jobs to take with concurrent, a count that knows nothing about what those jobs boot. Past the host's RAM the OOM killer arbitrates — it takes a VMM, and that job dies mid-stage with no explanation in its trace.

Set a memory budget and a job instead claims the guest RAM it is about to boot, and waits when the host is full:

[executor.schedule]
mem_budget = "50%"        # total guest RAM admitted at once; exact "48G" also works
wait_timeout_secs = 600   # then the job gives up

A percentage is resolved from this host's MemTotal, so one config scales across runners of different sizes. It is rounded up to the whole-GiB unit job sizes come in — a runner's MemTotal always reads somewhat under its nominal size, and rounding down there would cost a whole job's worth of budget — and any percentage is capped at the whole GiB the host actually reports, so 100% cannot round past the machine. A host whose /proc/meminfo cannot be read fails a percentage budget rather than guessing one.

The budget stays a guest RAM ceiling either way, not a host one: leave enough of the host outside it for the VMMs, a tmpfs-backed checkout, and anything else the box runs. Unset, admission is disabled (the default).

A job that waits says so in its trace, and says so again when it gets in:

virtkit: waiting for 8192 MiB of the host's 49152 MiB memory budget (45056 MiB reserved, 1 job(s) asked first)
virtkit: admitted after waiting 34s for memory

Claims are held for the job's whole life and released at cleanup — or by the kernel, if the job dies, so a crash frees its memory rather than leaking the budget. Jobs are admitted oldest request first, so a large job cannot be starved by a stream of small ones; the cost is that a small job may queue behind a large one it would have fit alongside. Several runner processes on one host share the ledger as long as they share a state_dir and run as the same user.

A job that gives up waiting exits a system failure, which GitLab retries only for jobs that ask for it:

heavy-job:
  retry:
    max: 2
    when: runner_system_failure

A MICROVM_MEM above the whole budget is clamped to it, the same way [executor.vm] max_mem clamps one: a job asking for more than the host can ever admit would otherwise fail every attempt. Keep [executor.vm] mem itself at or under the budget, though — a default no job can fit in makes every job on the host fail admission.

By default the budget counts what a job declares (MICROVM_MEM, or [executor.vm] mem), which is nearly always far more than it uses — so a runner gated this way runs fewer jobs than it could. A job's services count too, each at its x-virtkit.mem, else a Windows Dockerfile's # vk: mem=, else the default, held under the same ceilings; a job and services that together exceed the whole budget are refused. Under from_history the job's history already holds its services, whose VMs are the supervisor's children, so it is capped at the guest's and the services' declared sizes together. With host_checkout and net.mode = "switch", a compose: job's admission runs after the checkout, which holds its services: the clone itself is not admitted.

Reserving what jobs actually use

Turn on from_history and a job is instead admitted against its own recent peaks — the same figures its trace reports:

[executor.schedule]
mem_budget = "50%"
from_history = true

virtkit: reserving 531 MiB from what this job has been using (it declares 2048 MiB)

Runs are remembered per project and job name, under <state_dir>/history/<project id>-<slug>/<job name>-<digest>, so test:unit is predicted from test:unit, not from the heavy build beside it. The project's own id scopes it, since two projects can share a slug; the digest of the job's name as written keeps two names that reduce to the same filename — test:unit and test/unit — from sharing a history.

Every job records its run, whether or not this setting is on, so turning it on has something to work from. Each file keeps at most its thousand most recent runs, but there is one file per job name a project has ever run and nothing prunes them, so the directory grows slowly with the job names a host has seen. It holds no state anything else depends on: delete it, or any file in it, whenever you like — the jobs it covered fall back to their declared size for one run and start again.

A reservation is the largest run of the last 14 days plus 25% — the largest, because a job that peaks one run in five needs room for that run — never below 512 MiB and never above what the job declares. A job with no history yet falls back to its declared size, as does a job whose name changed.

A job's runs also count only while its memory ceiling is the one they ran under. Raise MICROVM_MEM from 4G to 16G and the job is treated as unknown again for one run: what it reached pressed against 4 GiB says nothing about what it reaches given 16, and predicting the new job from the old one is how a host ends up overcommitted the moment someone widens a job that needed it. Lowering the ceiling starts it again for the same reason, and putting a ceiling back finds the earlier runs still there — they are set aside, not discarded, for as long as the file's thousand-run cap has not pushed them out behind the newer ceiling's runs.

The window is days rather than a count of runs, because what changes a job's appetite — a dependency, a fixture, the code — changes on calendar time, while the same number of runs spans half an hour on a busy merge queue and most of a year on a release job. A job too quiet to have anything inside the window keeps its last five runs however old they are, so a monthly release job is still estimated from what it did rather than from what it asks for. A spike therefore stops being believed by ageing out, not by being outvoted.

Every job trace says where it stands, whether or not the host reserves this way — reading it is how you decide to:

virtkit: job resource usage: cpu 2m14s, peak memory 1.6 GiB, overlay 1.2 GiB of 16.0 GiB, read 3.4 GiB, written 812 MiB, sent 3 MiB, received 941 MiB
virtkit: most this job has used lately: memory 2.1 GiB, overlay 1.9 GiB of 16.0 GiB, job dir 6.4 GiB, read 12.0 GiB, written 3.1 GiB, sent 40 MiB, received 4.2 GiB over 37 runs; the next run reserves 2.6 GiB

Only the memory and the job dir are reserved against — the job dir by disk admission (see Keeping the job dirs' disk from filling); the writable layer and the traffic ride along because a job that fills its overlay or pulls 4 GiB in and out of the host every run is a fact about the host worth knowing. Each figure is its own maximum over the window, so they need not all come from the same run.

The run count is what the estimate rests on: the runs of the last 14 days, or the last five however old for a job too quiet to have that many. The ; the next run reserves … clause appears only where the host really does reserve that way.

The VM is still given the memory it declares; only the reservation shrinks. That is the point: guest RAM is faulted in on use and handed back when freed, so a generous declared size costs nothing at runtime, while an honest reservation is what lets the host run more than a handful of jobs.

The trade is real, though: a job that suddenly needs much more than it ever has — a new dependency, a bigger fixture — is admitted against the old figure, and the host can be overcommitted for that one run. The 25% headroom and the fortnight window absorb drift, not a step change. Leave it off until a few pipelines have been measured.

Sizing a project

One job's trace says what that job needs. vk gitlab usage says what a whole project does — every job this host remembers, heaviest first, with what its next run would reserve and what the lot would reserve if they all ran at once, which is the figure [executor.schedule] mem_budget has to cover:

$ vk gitlab usage acme
virtkit: 42-acme — what its jobs have been using lately:
  job         memory            overlay  job dir  ceiling  reserves  runs     read  written   sent  received
  build      5.9 GiB  3.9 GiB / 4.0 GiB  6.4 GiB  8.0 GiB   7.3 GiB    24  3.4 GiB  812 MiB  3 MiB   941 MiB
  test_unit  500 MiB                  -  1.1 GiB  2.0 GiB   625 MiB    37        -        -  2 MiB    88 MiB
virtkit: 2 jobs; all at once they would reserve 7.9 GiB, against a budget of 16.0 GiB

The argument is any part of a project's <id>-<slug> directory name, so the slug alone will do; without one it reports every project on the host. reserves is what each job's next run would claim: its declared size, or — with [executor.schedule] from_history on, as above — what its history says it needs. overlay is the writable layer against its capacity, the one column holding a figure a job can fail against rather than merely be sized by: build above has 150 MiB of room left. job dir is the most the job's dir under <state_dir>/jobs has held on the host by the end of a run — its rootfs overlay, above all. A - is a figure no run could measure — an unaudited kernel for the disk columns, a net.mode = "tap" job for the network ones, a job with no overlaid checkout for overlay, runs recorded by an older vk for job dir — which is not the same as a job that moved nothing.

A job can ask for its own project's report and get it in its trace, for an operator with the GitLab UI but no shell on the runner:

project-sizing:
  when: manual
  variables:
    MICROVM_USAGE_REPORT: "1"           # 1/true/yes/on
  script:
    - echo sizing

It reports that job's project and no other, whatever else this host has run — matched exactly, so a project whose directory name merely contains another's is not confused with it. Which project a job belongs to is taken from the runner's own account of the job, not from the CI_* variables beside it, so it is not something a job can name for itself — and on a runner old enough not to write that account, the report is refused rather than answered from what the job claims.

Placing a job on a memory node

On a multi-socket runner, guest RAM that is scattered across the sockets costs the job for its whole life: each page lands on whichever node the thread that first touched it happened to be running on, and the scheduler then moves the vCPU threads away from it. A VM placed on one node — its vCPUs pinned there and its memory preferring that node — pays none of that.

Placement is on by default and does nothing on a single-node host, which is most of them. It is host-wide, not a runner setting: the same key places vk run guests, build stage guests and compose services on the machine.

[numa]
mode = "auto"         # the default; "off" or "interleave" instead

auto puts each VM on the node with the most room left that it fits on, and interleaves one that fits nowhere — more RAM than the node has room for, or more vCPUs than the node has CPUs — across every node instead, which at least spreads it evenly rather than piling it onto whichever node booted it. The node is a preference for memory and not a cap: a guest that grows past what its node has free spills onto another node instead of driving that node into reclaim and an out-of-memory kill, so a placement the host outgrows costs locality and nothing more. Each VM says which it got:

virtkit: NUMA: node 1 (cpus 24-47) of 2
virtkit: NUMA: interleaved across 2 nodes (the VM does not fit one node)

With mem_budget set, placement uses the admission ledger. Each job's node is recorded beside its claim, so later jobs count what others were granted, including RAM they have yet to fault in. Without a budget, jobs use live per-node free memory plus this process's outstanding allocations, like other VMs. Live memory figures lag newly admitted VMs; setting a budget accounts for those grants across runners on a busy host.

off places nothing and leaves it to the kernel. interleave spreads every VM across all nodes without choosing one, for a host whose jobs are all larger than a node, or one where something outside virtkit is taking memory the ledger cannot see.

Placement is best-effort: it is applied to the VMM process before it starts, and a host that refuses it — a cpuset that excludes the chosen node, a kernel built without NUMA — gets a line in the log and a VM that boots unplaced, never a failed job.

To measure what it is worth on a given host, run the same job (or the same vk build) twice, once with mode = "off" and once with mode = "auto", and compare the wall time the trace reports. The kernel's own counters say whether the difference is the placement: while the VM runs, numastat -p <vmm> breaks that VMM's resident memory down per node. -p takes a process-name pattern as well as a pid, which is the easier handle on a job: the job's VMM is named vk:<hostname> ([executor.vm] hostname). numastat with no arguments counts the allocations that did not land on the node that asked, numa_miss and other_node. A placed VM concentrates its pages on one node and stops growing other_node; an interleaved one splits them evenly. Check the virtkit: NUMA: line first when a run measures the same either way: a VM wider than a node is interleaved in both, and interleaving is not what the comparison is testing.

Keeping the job dirs' disk from filling

Each job's dir under <state_dir>/jobs holds its rootfs overlay — every write the guest makes outside its checkout lands there — along with the checkout packed for the guest and the logs. Overlays grow while jobs run, so a filesystem that fits every job at boot can be full an hour later, and then every job on it fails.

So a job also claims room there before it boots, and waits for it as it waits for memory:

[executor.schedule]
disk_admission = true     # the default; false turns it off
disk_default = "8G"       # what a job with no history of its own is expected to write;
                          # unset, 8G or the filesystem's size, whichever is smaller

A job is expected to need the most its dir has held at the end of its recent runs plus 25%, under the same window and ceiling rule as a memory reservation from history (whether or not from_history is on), or disk_default until it has a history. It is admitted once the filesystem's free space covers that plus what the jobs already admitted there have yet to write: each one's expectation less what its dir holds now, which is already gone from the free space. A job with no other claim on the filesystem never waits for disk, whatever it expects, and beside other claims its own test asks for no more than could come free (what is free now plus what the other admitted jobs' dirs hold): a job that last filled the whole filesystem would otherwise wait for room that can never appear. Once admitted it still claims its full expectation on the ledger, so the jobs after it are held back by all it is expected to write. It needs no mem_budget, and shares the oldest-first queue and wait_timeout_secs with memory admission:

virtkit: waiting for 25.0 GiB of room in /var/lib/virtkit/jobs (31.2 GiB free, 18.5 GiB still to be written by the jobs admitted there, 0 job(s) asked first)
virtkit: admitted after waiting 412s for disk space

Pulled and built images on the same filesystem that no job, VM or service holds give way to a job short of room: it evicts them, least recently used first and whatever image_cache_idle_secs says, until its shortfall is free, and checks again. The shortfall is what would admit its whole expectation: what the admitted jobs have yet to write plus what it expects, less what is free. Registry chunks no cached image references go before any image. Two kinds of image stay: those the job's last run booted, since a job is admitted before it resolves its image and those are what it is about to use, and those used in the last 10 minutes (or image_cache_idle_secs, if shorter), which a job may have resolved without yet holding. Only a job's own images are kept: past the grace, one job's admission can evict the images of a job admitted before it that has yet to resolve them. Eviction is all or nothing: when what it could free falls short, nothing goes and the job waits as above, asking again once its shortfall is back within what could have been freed, and otherwise about once a minute. Its trace says what went; a job expecting 25.0 GiB, with 31.2 GiB free and 18.5 GiB still to be written by the jobs admitted there, is 12.3 GiB short:

virtkit: evicted 3 unused image(s) and 41 unreferenced cache chunk file(s) under /var/lib/virtkit for room in /var/lib/virtkit/jobs: 12.9 GiB freed of the 12.3 GiB needed

Host checkouts and the Windows build cache are not evicted this way; checkout_cache_idle_secs and windows_cache_idle_secs alone bound them.

A disk_default set larger than the filesystem can never be admitted, and fails every job without a history at once. Left unset it is capped at the filesystem's size, as an expectation from history is, so a host whose job dirs sit on a small filesystem runs its new jobs fewer at a time rather than not at all.

Only what lands in a job's own dir, <state_dir>/jobs/<job id>, is claimed. Anything else a job writes to the same filesystem — a host checkout without checkout_overlay, the atop archive, when state_dir puts them there — is seen only as free space already gone when the next job is checked. So is anything outside virtkit.

This bounds what the expectations say, not what jobs do: a job that writes far more than it ever has can still fill the filesystem, as can the first runs of jobs this host has no history of. It counts bytes, not inodes: a job that creates millions of small files can run the filesystem out of inodes with space to spare. A prepare that finds it full fails naming the filesystem and how much of it is used.

While a host is upgraded, a job prepared by an older vk neither claims disk nor waits for it; disk admission covers the host once every job it has running was prepared by this version.

What admission does not do

A waiting job has already been assigned by GitLab: it holds a concurrent slot and its own timeout runs while it waits. Admission keeps a host from overcommitting; it does not keep work out of the runner. For that, see below.

Nor does it cover everything a job boots. The claim is taken at the start of prepare, so a job that builds its own image holds its full guest RAM across that build — and the build's own stage guests ([build] mem × [build] jobs) are outside the budget entirely. A host has to leave room for both. An auto [build] jobs does not close that gap on its own: it is a fixed share of the whole machine, not of what is left of it. A stage does also wait for the memory it is about to use (see Sizing the two phases), so a build narrows while jobs hold the host — but that waiting measures memory already faulted in, not what the job ledger has granted, so it lags a job VM that has only just been admitted. Set [build] jobs by hand to bound the build up front as well.

Throttling a busy runner

concurrent decides how many jobs gitlab-runner accepts. Sized by hand it is a guess that has to hold for the worst pipeline, so it is usually set low — and a job it turns away stays pending in GitLab, holding no slot, running no timeout, free to land on another runner. That is the one thing admission cannot do for a job already assigned.

gitlab-runner re-reads config.toml when it changes, so concurrent can follow the host. Two pieces do that, split along the privilege line:

	Runs as	Does
`vk tune`	the runner user	Reads the ledger, works out how many jobs fit, capped at `max_concurrency`, writes that one number to `<state_dir>/schedule/desired-concurrency`
`vk-runnerctl`	root	Reads that number, clamps it into a range it configures, edits `concurrent`, puts the file back atomically

The split is the point: config.toml is root's, and granting vk the right to write it would grant root outright — vk run boots VMs with arbitrary mounts. vk-runnerctl instead takes no arguments and no paths; everything it touches is named in its own root-owned config. The worst an attacker who owns the runner user can do is ask for a concurrency an administrator already allowed.

# /etc/virtkit/runnerctl.toml — root-owned, 0644
runner_config = "/etc/gitlab-runner/config.toml"
desired_file  = "/var/lib/virtkit/schedule/desired-concurrency"
min = 1
max = 12                  # never exceeded, whatever is requested
cooldown_secs = 60        # shortest interval between two writes
stale_secs = 300          # a request older than this is treated as gone
# reload_command = ["systemctl", "kill", "-s", "HUP", "gitlab-runner"]

Run the measuring half from a user timer every half minute (vk tune), and the privileged half either from a root timer — nothing is granted to anyone — or from the runner user through a sudoers rule that allows no arguments:

gitlab-runner ALL=(root) NOPASSWD: /usr/local/lib/vk/vk-runnerctl ""

What it does with the file is deliberately narrow: it rewrites the single concurrent line and proves the result differs from the original at that key alone before installing it, so comments, key order and registration tokens survive untouched, and the original is kept once as config.toml.vk-orig. Lowering the number never disturbs a running job — it only stops new ones being taken.

The number itself is "the jobs running now, plus what both the budget and the host still have room for", at the size a job on this host typically reserves:

virtkit: runner concurrency 2 (estimate 2, ceiling none); 6144 of 8192 MiB committed by 1 job(s), typical job 2048 MiB, 23040 of 32768 MiB host memory available

It falls the moment the host fills and climbs back one step at a time, because a job that has just started has not yet reached its real size. Alongside the ledger, MemAvailable limits how many typical jobs fit after keeping 15% of physical RAM free. A tmpfs-backed repository checkout therefore lowers the offered slots by its actual allocated size, as does an unrelated service on the box, without a guessed per-repository reserve; reclaimable page cache does not, since MemAvailable already counts it as available. Below that 15% floor the runner takes nothing beyond what it is already running, whatever the ledger says. One slot is always offered even then, since concurrent = 0 is not a setting gitlab-runner has and a runner that accepted nothing at all would never pick up again. And if vk tune stops writing, the runner is not left throttled: vk-runnerctl walks concurrent back up to max a step at a time.

A host whose /proc/meminfo cannot be read reports host memory unreadable and schedules on the budget alone.

That estimate is capped by [executor.schedule] max_concurrency, and the smaller of the two wins. With max_concurrency alone and no mem_budget, vk tune holds the runner at that ceiling. It is the runner user's own cap on what it asks for; vk-runnerctl's min and max are the administrator's range and still clamp the result. While the ceiling binds, the reported estimate is at most one above it: it climbs from the number last written.

On a fleet node, vk node run makes the same decision itself every half minute and whenever its hub's desired state changes, with the hub's ceiling as a third term: the smallest of the estimate, the hub's ceiling and max_concurrency wins, so the hub can lower the number but never raise it past the node's own. Only the half-minute pass raises it, by the estimate's one step when there is a memory budget. The node keeps the hub's ceiling across restarts and while the hub is unreachable. vk tune stands aside while vk node run is up, and otherwise applies the ceiling the node last received.

A runner that runs as the same user as vk, with its own ~/.gitlab-runner/config.toml, needs no vk-runnerctl: name the file as [node] runner_config and vk tune sets its concurrent itself, with the same one-line edit and the same proof that nothing else changed. It refuses a file another user owns.

Getting the number wrong is cheap on purpose. It decides what the runner accepts, never what is committed: too high and the extra jobs queue at the admission gate exactly as before, too low and the host idles until the next run. The gate is the guarantee; this is the throughput.

Egress control

A job's networking is governed by the host [egress] configuration. virtkit runs one userspace switch per job; the booted job guest and every service VM share that switch, each with its own address. They are all held to the run policy, and a service may narrow itself further with its own allowlist (see per-service egress) — the switch enforces the policy per source VM, so one VM cannot use another's network policy. A job that runs commands in a service with vk-agent exec --service does reach what that service may; see per-service egress.

There are two independent phases, each with its own host cap:

[egress] — the run phase: the booted job guest and its service VMs.
[egress.build] — the build phase: the RUN steps of a git-defined image (image: dockerfile:…) or a compose build: service. Absent by default, meaning the build runs unrestricted, like docker build.

A job a hub placed on a vk node may also be held to the hub's egress rules, chosen by project and ref, and a job a node's gitlab-runner runs to the hub's Fallback rule; see The hub's rules.

Host caps

Each phase has two dimensions — allow_ip (direct, non-DNS egress) and allow_name (DNS suffixes) — set in the host config:

[egress]
# DNS suffixes, dot-anchored: "corp.example.com" also matches *.corp.example.com
allow_name = ["corp.example.com", "crates.io", "debian.org"]
# Direct-IP egress as CIDRs, optionally port-scoped as CIDR:port (else any port).
# Needed only for a destination a job reaches by literal IP (no DNS name).
allow_ip = ["10.10.140.49/32"]

The list semantics are the load-bearing part:

an absent list is unrestricted for that dimension;
an explicit empty list (allow_name = []) denies that dimension;
a phase is unrestricted only when both its lists are absent;
consequently, once either dimension is present the phase is a restricted allowlist and the other, omitted dimension is deny-all. For example, with allow_name set but allow_ip absent, all direct-IP egress is denied — a job reaching a host by literal IP needs that IP added to allow_ip;
every entry is checked as every allowlist in vk is (vk run --allow-*, --build-allow-*, vk dev, the switch): a name is a bare DNS suffix, never a URL, a pattern, a port or a fully qualified name with its trailing dot; an IP entry is an IPv4 CIDR, IPv6 refused. A bad entry in the config, or in a job's (or its services') MICROVM_EGRESS_ALLOW_* / MICROVM_BUILD_EGRESS_ALLOW_*, fails the job by its config key or variable.

Under an allowlist, a name outside allow_name resolves to NXDOMAIN, reverse (PTR) lookups included, and a query the gateway cannot parse is REFUSED; the job's own service aliases still resolve, as do reverse lookups of the job's own addresses. Listing in-addr.arpa or ip6.arpa in allow_name reopens reverse lookups as a way to carry data out in the names queried. The gateway rebuilds forwarded queries from their question, keeping only the id, RD, AD and CD flags and a bare EDNS record. Queries it cannot rebuild are REFUSED, or NOTIMP for a DNS update or notify.

An allowed name is reached by address: resolving it through the gateway opens, to that VM alone, the IPv4 addresses its answer gives for it (directly or through its CNAME aliases, which need not be allowed names), for the answer's TTL (at least 30 s) plus 60 s. Filtering is by address, not by name, so the limits are those of the address:

an opened address is reachable on any port but DNS's 53, and for every site it serves: on shared hosting or a CDN, the other sites behind it too;
a DNS-over-HTTPS or DNS-over-TLS resolver behind an opened address resolves names past the allowlist, though only listed ranges and addresses opened by allowed names can be reached;
an address stays open for that whole time, even if the name's records change meanwhile;
a job resolving very many names keeps only part of them open: past 64 names on one address of one VM, or 16384 (VM, address, name) entries across the job's VMs, the entries soonest to expire close, and their addresses are refused until resolved again.

Narrowing from a job

A job may narrow its egress with lists of its own, never widen it: the switch checks them on their own when the job runs, after the host cap (and the hub's rules), so a destination is reachable only if each allows it. Each variable is a list — one or more entries separated by spaces, commas, tabs, or newlines (empty entries are ignored). A # begins an end-of-line comment, so a block-scalar list can document each entry inline:

least-privilege-job:
  variables:
    # Any of these separators work; mix freely.
    MICROVM_EGRESS_ALLOW_NAME: "crates.io debian.org github.com"
    # equivalently: "crates.io,debian.org,github.com"
    MICROVM_EGRESS_ALLOW_IP: "10.0.0.0/8 192.168.5.10/32"

The value is always a string, never a YAML sequence — a GitLab variable value cannot be a list. Use the separators above, or a block scalar (|) to keep a long list readable one-per-line (newlines are valid separators):

  variables:
    MICROVM_EGRESS_ALLOW_NAME: |      # a string, split on newlines — not a YAML list
      crates.io          # Rust registry
      static.crates.io   # crate downloads
      index.crates.io    # sparse index
      debian.org         # apt mirrors (also covers deb./security.)

  variables:
    MICROVM_EGRESS_ALLOW_NAME:        # ✗ invalid — a variable value cannot be a sequence
      - crates.io
      - debian.org

(A genuine list is only used in the host [egress] config — allow_name = ["crates.io", "debian.org"] — not in a job variable.)

You can also reference other CI/CD variables — GitLab expands $VAR / ${VAR} in the value before the executor sees it, so a shared list defined once expands into the list virtkit parses:

variables:
  COMMON_EGRESS: "crates.io debian.org github.com"   # define once (group/project/pipeline)

rust-job:
  variables:
    # Reference the shared list, and add a job-specific host.
    MICROVM_EGRESS_ALLOW_NAME: "$COMMON_EGRESS internal.corp.example.com"

The lists read as the host cap's do: with neither variable set the job restricts nothing of its own; once either is set, the dimension not set allows none. MICROVM_EGRESS_ALLOW_NAME alone reaches no literal IP but the addresses its names resolve to; under a cap that lists only allow_ip, a job's names resolve to nothing, since the cap allows no name.
No entry is checked against the cap: an entry the cap leaves nothing of — a name no allow_name entry is under or over, a range no allow_ip range overlaps — is kept, and the trace warns of it at prepare: virtkit: warning: MICROVM_EGRESS_ALLOW_NAME pypi.org: outside the node's [egress] allow_name, its lookups will be refused. Such a name reaches nothing; such a range reaches only the addresses of names every layer allows (reachable only as an address of a name it allows), nothing when the cap lists no name. Under dry-run the warning says the layer only reports. An entry only partly within the cap reaches the part the cap allows.
An invalid entry (not a DNS suffix, not an IPv4 CIDR) fails the job with a job-visible error.

The same applies to the build phase via MICROVM_BUILD_EGRESS_ALLOW_IP / _ALLOW_NAME.

The hub's rules

A hub that has egress rules (Job egress from the hub) sends a placed job, with its start, the lists its rules allow in each phase. They are one more layer, checked after the node's cap and before the job's own narrowing:

the node's [egress] or [egress.build];
the hub's rules for the job;
the job's MICROVM_EGRESS_* or MICROVM_BUILD_EGRESS_*;
a service's own MICROVM_EGRESS_ALLOW_*, for that service's VM, on top of the job's.

A destination must pass each in turn; the switch never merges them into one list, nor checks one layer's lists against another's. Each reads its own lists as the host cap's read, and an address a lookup opened passes a layer only if that layer allows the name looked up. A phase the hub's rules leave unrestricted is the node's alone, as without a hub. A job's or a service's MICROVM_*EGRESS_ALLOW_* entry outside the hub's lists is kept, and the trace warns of it naming the rules (outside the hub's egress rules (wab, wab-branches), its lookups will be refused). The hub can only narrow what the node's cap allows, never widen it.

The node's own dry-run, [egress] dry_run or MICROVM_EGRESS_DRY_RUN, never relaxes the hub's rules: what they refuse is refused, its names resolve to NXDOMAIN, and queries the gateway cannot rebuild are refused, dry-run or not. A rule the hub has in dry-run blocks nothing, in either phase: what it would block is reported to the hub under its name, and listed in the trace for the run phase. Only the per-job switch enforces the hub's rules, so a node whose [net] mode is tap or pool takes no job they restrict, and fails one whose run phase they restrict. A Windows image does not build under rules that restrict the build phase, since its build steps cannot be held to them, and a dry-run rule says nothing of them.

The trace names the rules and each phase's lists, layer by layer:

virtkit: egress under the hub's rules: wab, wab-branches (dry-run, not enforced: wab-release)
virtkit:   run: node cap unrestricted; hub names corp.example.com, ips 10.10.140.49/32; job not narrowed
virtkit:   build: node cap names debian.org crates.io, ips none; hub names debian.org, ips none; job not narrowed

and each stage's denials under what decided them: egress blocked by the allowlist (the node's cap, the job's or a service's), egress blocked by the hub's egress rules (…), egress the allowlist would block (dry-run, not enforced), and egress the hub's dry-run rule … would block (not enforced).

A job run by the node's own gitlab-runner gets the hub's Fallback rule alone, default: its project and ref come from variables the job can set, so no rule by project or ref may apply to it. The node keeps the Fallback the hub last sent in its desired state, across hub outages; a job takes it at prepare and keeps it to the end, whatever the hub sends meanwhile, and is held to it as a placed job is to its rules. Its trace names the rule the same way. Only the per-job switch enforces it, so under tap or pool networking the node does not tell the hub it can, and the hub sends it none.

Per-service egress

A services: entry can carry its own MICROVM_EGRESS_ALLOW_IP / MICROVM_EGRESS_ALLOW_NAME in its variables: to narrow its own VM further: the switch holds it to its own lists and the job's, each checked on its own, so it reaches only what both allow, within the host cap and the hub's rules. The rules are those of a job's lists, with one addition: for a service, a present-but-empty value denies that dimension (e.g. MICROVM_EGRESS_ALLOW_NAME: "" gives the service no external DNS egress). A service that sets neither variable is held to the job's lists alone.

A per-service value is the same kind of list (space/comma/newline-separated) and supports the same $VAR references:

variables:
  MIRROR_EGRESS: "registry.corp.example.com mirror.corp.example.com"

job-with-services:
  services:
    - name: postgres:16
      alias: db
      variables:
        MICROVM_EGRESS_ALLOW_NAME: ""            # db gets no external egress
    - name: registry-proxy:latest
      alias: proxy
      variables:
        # A list, via a shared variable plus an extra host.
        MICROVM_EGRESS_ALLOW_NAME: "$MIRROR_EGRESS cache.corp.example.com"
    - name: fetcher:latest
      alias: fetch
      variables:
        MICROVM_EGRESS_ALLOW_NAME: "crates.io,pypi.org,files.pythonhosted.org"

The switch enforces each policy against the flow's source VM — authenticated by the per-VM socket it arrives on, not by any address the guest writes into its packets — so a service cannot borrow another VM's (looser) policy. Requests are checked at job prepare: an invalid entry fails the job with a visible error before anything boots, and one the host cap, the hub's rules or the job's own lists leave nothing of is kept, and warned of.

A job's script can still act through a service: vk-agent exec --service NAME runs a command in it (see image selection), under the service's policy — the job's lists and its own, so never more than the job may reach itself; set [executor] guest_service_exec = false to refuse it.

What a job contacts

Every job trace names what that job is known to reach out to, just before what it cost:

virtkit: names this job has contacted under unrestricted egress: deb.debian.org, github.com

This is the standing list, not this run's: each run adds what its guests resolved in the run phase to what the job has resolved before, so a nightly step's host is on it even in a pipeline that did not run one. Write an allow_name from it and the job keeps working — bar any entry showing ?, which is a name the guest sent outside the characters a hostname is spelled with, kept only to show that it happened. A dockerfile: build's own resolutions are not on it — those are the build phase's, and [egress.build] is a separate cap with its own audit.

The list belongs to the policy, not to the job. Narrowing an allowlist starts it again and the line then names the policy the list was gathered under (under egress policy a379a6f61c4b8e02) — what a job reached while it could reach anything says nothing about the same job once an allowlist is in force, and those names would otherwise linger as if it still needed them. It is kept per job under <state_dir>/sites/, keyed exactly as the run history is, and capped: past a couple of hundred names the line says how many more it is not showing, because a list that long is no longer an allowlist anyone would paste.

Audit mode

Audit mode adds the counts for one run: every resolution that run made, plus the external IPs it dialed without a resolution behind them, printed where the job can see it (at the end of the job trace for the run phase, after the build for the build phase). Turn it on when the standing list above is not enough — to see how often a name is reached, or to catch a step talking to a literal IP.

[egress]
audit = true          # host-wide, for every job's run phase

[egress.build]
audit = true          # for the build phase

A job can enable it for itself even when the host default is off:

discover-egress-job:
  variables:
    MICROVM_EGRESS_AUDIT: "1"           # 1/true/yes/on
    MICROVM_BUILD_EGRESS_AUDIT: "1"

Dry-run

Audit lists the contacts the allowlist allows; dry-run lists the ones it would deny. It covers the run phase only: the would-be denials appear in an "egress the allowlist would block (dry-run, not enforced)" block in the trace, but nothing is blocked; a denied name still resolves, so the job runs unchanged. To roll an allowlist out, turn it on, watch a few pipelines, and add whatever the trace flags before enforcement bites.

Dry-run covers the service VMs on the job's LAN too, including a service's own MICROVM_EGRESS_ALLOW_* lists. It never relaxes the hub's rules, whose own dry-run covers both phases.

[egress]
allow_name = ["corp.example.com"]   # the allowlist under test
dry_run = true                      # evaluate it, report, but don't block
audit = true                        # (optional) also list what it allowed

A job can turn it on for itself with MICROVM_EGRESS_DRY_RUN — but, unlike audit, only when the host set no run-phase allow_ip/allow_name of its own: dry-run suspends enforcement, and a job must not relax a cap the host enforces. Where the host set a cap, only the host [egress] dry_run toggle takes effect.

rollout-egress-job:
  variables:
    MICROVM_EGRESS_ALLOW_NAME: "corp.example.com"   # job-defined allowlist
    MICROVM_EGRESS_DRY_RUN: "1"                      # 1/true/yes/on — observe, don't block

On the command line

Outside CI, the same controls are CLI flags on vk run / vk build: --audit-egress audits the booted guest and --build-audit-egress a build's RUN steps (mirroring the --net / --build-net phase split). vk run's --allow-name / --allow-ip restrict the booted guest's egress, and --egress-restrict alone denies it all. Dry-run is CI-only.

Reported to the hub

A job a hub placed on a vk node also reports its egress to the hub, while it runs and with its result: each destination refused, or flagged under dry-run, in both phases, with the VM it came from, the layer that decided it (the node's cap, the hub's rules, the job's variables or a service's) and when, and for a hub rule in dry-run its name; the hub rules the job ran under; what the job's guests reached; the MICROVM_*EGRESS_ALLOW_* entries that failed it, or that a layer above leaves nothing of; and the bytes of each phase. The build phase is recorded whether or not audit is on, which only adds the summaries above to the trace. Once a build's or the run's log of denials or of contacts reaches 8 MiB, the switch only counts what else it sees, for the trace's lines and summaries too. While the job runs, the node sends what it recorded so far at each stage's start and every few seconds when it changed, and again after a reconnect, so the hub's pages follow it; a hub older than the node gets the report with the result alone. Jobs run by gitlab-runner report nothing. See Job egress from the hub.

The node also reports its own caps in its inventory, each phase as it holds jobs to it: [egress] under [net] mode = "switch" (nothing without a network, nothing enforced over a tap), with [egress] dry_run, and [egress.build] whatever the mode — normalized and fingerprinted, so that the hub tells a changed cap from the one a job ran under. A list of more than 256 entries is reported as too large to show. The hub shows them beside its own rules, evaluates them in Test a job, and names the entries its jobs show no use of; it cannot widen them. The node reads its config once, its executors each time they start, so a job may be held to caps edited since the node last reported them: each phase of the job's report names the caps it was held to, by their fingerprint, and the hub counts the phase toward those.

Services

CI services: run as sibling microVMs on the per-job switch, each resolvable by its alias over the switch's DNS. They need the per-job LAN, so the job must set [net] mode = "switch"; vk pulls each service image host-side (registry credentials never enter a guest) into the same digest-keyed cache the job's image: uses — an OCI ref (mysql:8, redis:7) under <state_dir>/docker, a virtkit/ bundle under <state_dir>/registry.

integration-test:
  services:
    - name: mysql:8
      alias: db
    - name: redis:7
      alias: cache
  script:
    - ./run-tests.sh                    # reaches the DB at host `db`

A service reaches the job VM back as build, the alias GitLab's docker executor gives the job container on a per-build network, or by [executor.vm] hostname; a service aliased either name keeps that name.

Windows services

A compose: job's service whose build: Dockerfile installs Windows (FROM winiso:, see Windows guests) runs beside the job VM, which stays Linux:

# ci/lab.compose.yaml
services:
  job:
    build: { context: ., dockerfile: job.Dockerfile }
  dc:
    build: { context: windows, dockerfile: lab.Dockerfile, target: dc }
    x-virtkit: { cpus: 2, mem: 3G }

ad-tests:
  variables:
    MICROVM_IMAGE: compose:ci/lab.compose.yaml#job
  script:
    - ./tests/ad.sh dc            # the DC by its service name

Prepare builds it with vk build's Windows path into the job dir. Its layers come from the host's Windows cache or, with [build] cache_registry on a registry server, from the hosts that built them (sharing builds); the ISOs a winiso: stage pins come from that registry too (vk registry put), so they need not be in the repository. A first install takes 20 minutes or more, once for the fleet.
The supervisor boots it on the job's LAN under its service name and runs its provisioning (CMD, compose environment); prepare waits for that, up to 30 minutes, before the script starts, and fails the job as soon as the supervisor is gone.
All of it happens inside prepare: raise gitlab-runner's prepare_exec_timeout (custom executor settings) to cover a first install plus provisioning.
Its size is the compose x-virtkit one, else its Dockerfile's # vk: line, else the runner's [build] guest size, held under the runner's vm.max_cpus/vm.max_mem and mem_budget like any service; its build guests are held under them too, and less than the 2G Windows needs, asked for or left by a ceiling, fails the build up front. Its memory counts in the job's admission, as every service's does; its build guests and its disk growth do not yet.
A step's network (RUN --network=default) follows [egress.build]'s allowlist as a Linux step's does; its audit does not cover a Windows step. Under an allowlist Windows reports no Internet, so BITS or winget may refuse to run unless the allowlist lets its connectivity check through.
Its layers, installs and media are reclaimed once no build or job used them for [executor] windows_cache_idle_secs (7 days by default, at least 1; reclaiming an install means installing Windows again).
The Dockerfile's winiso: paths (the medium, its --drivers) must stay inside the build context: an absolute path, a .. or a symlink out is refused. A stage builds on a winiso: medium or an earlier stage, never on a bundle: a bundle names its layer by key alone, which would hand the job any layer of the host's cache or the registry.
One Dockerfile per service, no build args and no additional contexts, one NIC; a bundle image: path is refused, and the job VM cannot be Windows.

Per-service networking

Each service is a real VM with its own address on the shared switch, so a service can carry its own egress allowlist in its variables: — see per-service egress for the semantics and an example. A service that sets no egress variable shares the job's run policy.

Navigation Menu

Repository navigation

gitlab-ci.md

gitlab-ci.md

GitLab CI executor

Runner wiring

Nested virtualization

CI tools

Host-side checkouts

Job variables

Image selection

Resource usage

Sizing the writable layer

What the job did, second by second

Sizing the two phases

Giving idle memory back

Sharing the host page cache with the guests

Keeping the host inside its memory

Reserving what jobs actually use

Sizing a project

Placing a job on a memory node

Keeping the job dirs' disk from filling

What admission does not do

Throttling a busy runner

Egress control

Host caps

Narrowing from a job

The hub's rules

Per-service egress

What a job contacts

Audit mode

Dry-run

On the command line

Reported to the hub

Services

Windows services

Per-service networking

FilesExpand file tree

gitlab-ci.md

Latest commit

History

gitlab-ci.md

File metadata and controls

GitLab CI executor

Runner wiring

Nested virtualization

CI tools

Host-side checkouts

Job variables

Image selection

Resource usage

Sizing the writable layer

What the job did, second by second

Sizing the two phases

Giving idle memory back

Sharing the host page cache with the guests

Keeping the host inside its memory

Reserving what jobs actually use

Sizing a project

Placing a job on a memory node

Keeping the job dirs' disk from filling

What admission does not do

Throttling a busy runner

Egress control

Host caps

Narrowing from a job

The hub's rules

Per-service egress

What a job contacts

Audit mode

Dry-run

On the command line

Reported to the hub

Services

Windows services

Per-service networking