Status: experimental; vk-gitlab, vk-hub and vk node implement it. vk-hub-proto
defines the wire types: client for the hub's client API, dispatch for session job
messages, and job for the job spec. Serde tests pin their JSON. See
GitLab jobs for the design and rationale.
Three parties take part:
vk-gitlab— a daemon that is gitlab-runner's GitLab-facing side: it holds the runner tokens, asks GitLab for jobs, writes their traces and states, and submits each job to the hub as a job spec.vk-hub— reserves capacity on nodes, places jobs, relays and stores their output, and reports how they ended.vk node— runs each job in microVMs, stage by stage, and talks to GitLab directly for the job's own transfers: the clone, dependency artifacts, artifact uploads.
| Credential | Held by | Used for |
|---|---|---|
runner authentication token (glrt-…) |
vk-gitlab alone |
POST /api/v4/jobs/request, /runners/verify |
relay key (ed25519, <state_dir>/hub/key.pk8) |
vk-gitlab alone |
signing every client API request; its session with the hub |
CI_JOB_TOKEN (the job response's token) |
vk-gitlab; the job's node, in the spec |
the daemon: trace, state; the node: clone, artifact upload |
dependency tokens (dependencies[].token) |
the job's node | downloading that dependency's artifacts |
registry credentials (credentials) |
the job's node | pulling the job's images |
| node key | the node | its session, nothing else |
gitlab-runner sends the job token, not the runner token, on every per-job endpoint —
PUT /jobs/:id, PATCH /jobs/:id/trace, POST /jobs/:id/artifacts — so a node holding
the job token holds nothing the job itself does not. The hub sees a spec's secrets in
transit and keeps them in memory only, until the node has accepted the job; its records
and audit log hold the spec redacted (CiJob::redacted).
vk-gitlabasks the hub whether the runner's placement has room (POST /v1/capacity), long-polling until it does.- It reserves one envelope on a node (
POST /v1/reservations); the hub offers it to nodes until one accepts. - It asks GitLab for a job (
POST /api/v4/jobs/request), renewing the reservation while the request is outstanding. On 204 it renews and asks again. - On 201 it writes the job to its state file, translates the response into a spec and
submits it on the reservation (
POST /v1/jobs). - The hub sends the node
start; the node journals the job, converts the reservation into the job's ledger entry and answersaccepted. - Seeing the job accepted,
vk-gitlabsends GitLabPUT /jobs/:idstate=running— the commit of a job taken undertwo_phase_job_commit. - The node runs the stages, streaming masked output to the hub, which stores it and acks.
vk-gitlabreads it from its GitLab offset and patches the trace. - The node uploads the artifacts to GitLab itself, then sends its result once its output is all acked.
vk-gitlabpatches the rest of the trace, sends the finalPUT /jobs/:id, settles the job with the hub (POST /v1/jobs/<id>/settle) and drops it from its state file.
GitLab hands a runner a job only when the runner asks, and a job it has handed over cannot move to another runner. So the daemon holds capacity before it asks, never after:
- Capacity first.
POST /v1/capacityis advisory: the hub's count, from heartbeats less the reservations and starting jobs they do not show yet, of how many envelopes of the placement its ready nodes could take, each node at most what its cap — the smaller of its concurrency ceiling and its ownmax_concurrency— leaves past the reservations and jobs it holds. The daemon does not poll GitLab for a runner while it is 0, so a fleet with no room leaves the job pending in GitLab for another runner. - Then a reservation. Before each job request the daemon holds a reservation. The node decides it through its admission ledger at once — a reservation never waits in the ledger's queue — and counts it as granted memory and disk while it lasts.
- Then the job request. The reservation's lease covers the request: a lease of
lease_secs(default 90) against a job request the daemon times out at 60 seconds, renewed everylease_secs / 3while the request is outstanding. A request in flight is never aborted: GitLab may already have assigned the job. - Then the job. The job is submitted on the reservation at once. The node sizes the job by
its own rules —
MICROVM_CPUS,MICROVM_MEM, the script's# vk:hint, history — and returns what the job does not need to the ledger; a job needing more than its reservation waits for the rest in the ledger, as an executor job waits today.
Idle reservations are bounded: one per outstanding job request, so one per runner at the
default request_concurrency of 1, each held for at most one request plus a renewal. A
reservation that ends mid-request — its node lost, or the node's clock ran its lease out —
leaves a job that arrives meanwhile without one; the hub then places it afresh within its
place_within_secs (vk-gitlab asks for 300 seconds), and failing that ends it no_capacity,
which the daemon reports as runner_system_failure. That window is the only way a job taken
from GitLab waits for capacity.
The lease is a duration on the node's monotonic clock from its acceptance; no wall-clock time crosses the wire. The daemon counts it from when it sent its request, so it always believes the lease ends earlier than the node does.
A runner in vk-gitlab's configuration names its placement; every runner of one vk-gitlab
takes jobs from the one GitLab server its [gitlab] url names:
[gitlab]
url = "https://gitlab.example.com"
[[runners]]
name = "ci"
token_file = "/etc/vk-gitlab/runner-ci.token" # glrt-…
pool = "ci"
labels = ["large-memory"]
envelope = { mem = "16G", cpus = 8, disk = "16G" }
request_concurrency = 1GitLab's job response does not carry the job's tags, and GitLab only gives a runner jobs
whose tags it has, so placement is per runner: one runner per class of work, with the tags
of that class registered in GitLab and its pool and labels here. A pool is a set of nodes the
hub's operator names (vk-hub nodes pools <id> <a,b>); a label is one a node declares in its
own configuration and reports in its inventory's labels, absent from an older node's. The
placement stays in the daemon's configuration, which it reports to the hub; the hub sets the
relay no ceiling on it.
A node's concurrency ceiling (vk-hub nodes ceiling) and its own executor limit
([executor.schedule] max_concurrency, which the node reports) cap its placed work at the
smaller limit. The hub offers and starts nothing on a node whose reservations and unfinished
jobs reach the cap. The node enforces it independently, refusing ceiling or concurrency
by its own count. Running jobs above a lowered ceiling carry on.
A host runs its own gitlab-runner with the vk executor or takes placed jobs, never both. A node
that finds such a runner on its host says so in its report and refuses with runner every new
offer and every start without a reservation (a reservation it already holds is renewed and
started on); the hub places nothing on it and counts it in no capacity.
A host that takes placed jobs sets [node] runner = "none" (unset gives none only where no
gitlab-runner unit or binary shows), so the node drains, resets and updates it as one whose
runner it supervises.
Among the nodes with room, the hub offers a reservation, or starts a job without one, on the
least loaded first, then the one with the most room, then by node ID. A node's load is the
largest of: its placed work against that work plus its room (the envelopes it still fits);
the vCPUs of its placed work per CPU; and its heartbeat's 1-minute load average per CPU, left
out for a node too old to report one. Work therefore spreads across nodes even at moderate
load — concurrent builds, and the traffic they put on shared services such as the registry,
are not stacked on the one node with slightly more room, a host busy with work of its own
takes new work last, and losing a node loses fewer jobs. Memory, disk and the cap
bound each node, and capacity (fits) is the sum of every node's room.
A job whose image its node builds from the checkout (dockerfile: or compose:, its own or a
service's) goes first to a node that ran a job of the same project with the same such images
within that node's image_cache_idle_secs, while that node is lightly loaded: its 1-minute
load average, plus the vCPUs of jobs sent to it in the last minute and of this job, per CPU,
below the hub's image_affinity_max_load (default 0.7), and without this job no more than
image_affinity_max_extra_load (default 0.25) above the lowest such load among candidates.
That node boots it in seconds where another spends a minute or two building the image. A job
submitted on a reservation on a node without the image then starts on the warm node without it,
and the hub releases the reservation. Under heavier load the order above holds.
HTTP/1.1 and JSON over TLS 1.3, on the hub's node listener (addr), under /v1/. Bodies are
vk_hub_proto::client's. vk-gitlab is an enrolled relay (GitLab relay
enrollment): vk-hub token create --kind gitlab | vk-gitlab join <hub-url> --token - [--ca FILE] pins its key on the hub and keeps it, the hub's URL, its CA and
the relay's ID under <state_dir>/hub/ (0700); vk-gitlab leave deletes them. Both are
refused while a state file holds a job.
Authentication. Each request is signed by the relay's key, as a node signs a release
download: vk-relay carries the relay's ID, vk-time the time, and vk-signature its
ed25519 signature over client_request_message — the relay ID, method, path and query, the
body's SHA-256, the time, and the TLS connection's exporter. The hub checks that the relay is
enrolled and not removed, the time within DOWNLOAD_SKEW_SECS, and the signature. A signature
is good on its own connection only, so the daemon opens one connection per request and signs
a retry again; a TLS-terminating proxy in front of the hub breaks the binding. A relay is
held to no pool or envelope ceiling: it already holds the runner tokens and sees every job's
secrets.
Session. Besides the client API, vk-gitlab run holds a session with the hub at
/v1/gitlab/session, as a node does, redialing it when lost: it reports its version, host,
system ID, concurrent, its GitLab server's URL and the SHA-256 of its certificate's public
key, and each runner's name, GitLab runner ID, placement, limit and whether GitLab took its
token; every heartbeat, each runner's job requests outstanding, jobs held and last GitLab
answer; and each runner's state (polling, waiting_capacity, server_pending,
unhealthy, token_refused). The hub answers with its status of the server. A runner asks
GitLab for jobs only while that status is approved: not before the first status, nor while
the server is pending, rejected or the relay rebound, nor once the hub refused the relay for
good. Jobs it already holds run either way.
| Request | Body | Answer |
|---|---|---|
POST /v1/capacity |
CapacityRequest |
200 Capacity |
POST /v1/reservations |
ReservationRequest |
201 ReservationGrant |
POST /v1/reservations/<id>/renew |
RenewRequest |
200 ReservationGrant |
DELETE /v1/reservations/<id> |
— | 204 |
POST /v1/jobs |
JobSubmission |
201 JobView |
GET /v1/jobs/<id>?after=<revision>&wait=<secs> |
— | 200 JobView |
GET /v1/jobs/<id>/output?offset=<n>&wait=<secs> |
— | 200 bytes |
POST /v1/jobs/<id>/cancel |
CancelRequest |
202 JobView |
POST /v1/jobs/<id>/settle |
— | 204 |
POST /v1/clients |
ClientHello |
204 (API keys of other clients; a relay is refused) |
Long polls. wait on the two GETs, and wait_secs in a capacity request, hold a
request for at most 60 seconds: a capacity answer until its revision passes after, a job
view until its revision passes after, an output read until there are bytes past offset
or the output is complete. A long poll that times out answers as a plain request would.
Capacity. {placement, after, wait_secs} → {revision, fits}. fits counts envelopes,
not nodes: a node with room for two counts two.
Reservations. {request_id, placement, lease_secs, wait_secs} → {reservation, node, envelope, lease_secs}. The hub filters the pool's nodes on labels, readiness and headroom, and
offers the envelope to one at a time, least loaded first (Placement), until one
accepts or wait_secs runs out (503 no_capacity). renew extends a lease from now and
answers the lease the node granted; a reservation that lapsed, was released or whose node was
lost answers 410 reservation_gone. DELETE releases it, and answers 204 for one already gone.
Jobs. {request_id, placement, reservation, place_within_secs, spec} → JobView. The hub
starts the job on the reservation's node — unless a lightly loaded node holds the job's
built image (placement), when it releases the reservation and starts the job
there; with no reservation, or one that is gone, it places the job as it places a reservation,
for up to place_within_secs. A node that refuses the start leaves the job free to place
again; a start the node may have taken is never placed twice — see failures. The
spec is at most 512 KiB serialized (413 too_large).
JobView is {id, revision, state, node, stage, output_len, cancel, result}: state is
queued, starting, running or finished, and result is present once finished, with the
node's usage when it sent one (see Result).
revision moves with every change but the output's length.
Output. The answer's body is the output from offset, at most 1 MiB, with
vk-output-offset (its first byte's offset), vk-output-length (the output's length so
far) and vk-output-complete: true when the job has ended and the body reaches its end. An
offset past the end answers 416 with vk-output-length. The hub keeps a job's whole
output until it is settled, so the daemon can read from any offset — whatever GitLab says it
holds.
Cancellation. {mode}: graceful or immediate. A later immediate overrides a graceful;
cancelling a finished job changes nothing and answers its view.
Settle. The daemon has delivered the outcome to GitLab; the hub drops the job's output and keeps its record.
Idempotency and retries. request_id — 16 random bytes, hex — makes a create
idempotent for a day: the same ID with the same body answers the first answer again, or 410
not_found once the hub has dropped that job from its history; with another body, 409
conflict. Renew, release, cancel and settle are idempotent by nature.
The daemon retries transport errors and every retryable code with backoff from 1 to 30
seconds and jitter, honouring retry_after_secs, and never retries another 4xx.
| Status | code |
Meaning | Retry |
|---|---|---|---|
| 400 | invalid |
unparseable body, malformed ID, a job without a project | no |
| 401 | unauthorized |
no signature, a bad one or its time too far off, or a relay the hub does not know or removed | no |
| 403 | forbidden |
a job of another GitLab server than the relay's, of a relay told rebound, or an API key's GitLab job |
no |
| 403 | approval_pending |
a job of a project, or of a server, an operator has yet to approve on the hub | no: the job fails |
| 403 | project_rejected |
a job of a project, or of a server, an operator rejected | no: the job fails |
| 404 | not_found |
no such job or reservation of this relay's | no |
| 409 | conflict |
request_id reused with another body |
no |
| 410 | reservation_gone |
lapsed, released, or its node lost | reserve again |
| 410 | not_found |
a create retried after its job left the hub's history | no |
| 413 | too_large |
spec over 512 KiB | no |
| 500 | internal |
— | yes |
| 503 | no_capacity |
no node accepted in time | yes, after retry_after_secs |
| 503 | unavailable |
hub busy or starting | yes, after retry_after_secs |
An error body is {error, code, retry_after_secs?}; a code the client does not know reads as
other.
A job refused approval_pending or project_rejected is not held: approval is a human step.
vk-gitlab releases its reservation and fails it in GitLab, uncommitted, as
runner_configuration_error (script_failure where GitLab does not know it), its trace ending
with what to do, as for a project pending:
ERROR: project acme/web (ID 12) is waiting for approval on the fleet hub https://hub.example.com.
Retry this job once an operator has approved it (Connections > GitLab > Pending).
GitLab's retry: when: runner_system_failure does not retry that reason, so a pending project
does not loop. The hub keeps the refusal as the answer to the submission's request_id, so it
counts once.
Version 3 (JOBS) adds one message type each way to the existing session, {"type": "job", "kind": …}, whose payloads are vk_hub_proto::dispatch's. A job message in a session below 3
is a protocol error. PROTOCOL stays at 1 to 2; each side that implements version 3 negotiates
from a range of its own reaching it: the hub accepts 1 to 3, so a node offering 3 gets it, and
one offering at most 2 is steered and never offered a job.
| Hub → node | Answer | Meaning |
|---|---|---|
offer {reservation, envelope, lease_secs} |
offer_reply |
set the envelope aside |
renew {reservation, lease_secs} |
lease |
extend the lease from now |
release {reservation} |
lease gone |
give it back |
start {job, reservation?, envelope, spec} |
job |
run a job |
output_ack {job, offset} |
— | output stored durably up to offset |
cancel {job, mode} |
job, then result |
stop a job |
recorded {job} |
— | result stored; stop repeating it |
| Node → hub | Meaning |
|---|---|
held {reservations, jobs} |
everything the node holds, once per session after its report |
offer_reply {reservation, reply} |
accepted {lease_secs} or refused {reason, message?} |
lease {reservation, state} |
held {remaining_secs} or gone {why}: expired, released, started, unknown |
job {job, state} |
accepted, refused {reason, message?}, running {stage, activity?, guest? {cpus, mem_mib}}, finished |
output {job, offset, data} |
output bytes from offset, base64, at most 256 KiB |
result {job, result} |
how the job ended; repeated until recorded |
egress {job, report} |
from version 6: a running job's egress report so far, when it changes and after a reconnect |
A running state's guest is the size of the job's VM, once its prepare has sized it; a hub
older than it ignores it.
An egress report is cumulative, with a revision: the hub keeps the highest, so one sent again
replaces the counts rather than adding to them, and the result's report, the last, has the
highest. The node takes it at each stage's start and every 5 seconds, at most once every 2, and
sends it when it changed; a session below version 6 gets the result's alone.
Refusal reasons are memory, disk, cpus, not_ready (draining, drained, quarantined, in
maintenance), policy, no_reservation, invalid, ceiling (the node's placed jobs not
finished and reservations held reach the hub's ceiling, as the node last applied it),
concurrency (the same, at the node's own max_concurrency) and runner (the host runs its own
gitlab-runner with the vk executor); one the hub does not know reads as other, as ceiling,
concurrency and runner do on a hub older than them.
Reservations. A node answers an offer at once from its ledger: granted, or refused with
the resource that is short. It holds the entry in the ledger itself, with no job process
behind it, and drops it when the lease runs out on its monotonic clock, telling the hub
lease gone expired. A lease is at most 600 seconds. A drain refuses new offers and lets
held reservations lapse; a quarantine releases them.
Start. The node journals the job before answering accepted, so a start redelivered after
a reconnect is answered from the journal and not run twice. A start naming a reservation the
node does not hold is admitted like any other ask, without waiting: refused no_reservation
when it does not fit.
Output. Offsets count bytes of the output as GitLab will hold it: masked, with section
markers, timestamped (unless the job sets FF_TIMESTAMPS false), cut at the trace limit. A
stamp is when the node read the line (its start, for a line that spans reads), however late
the daemon reads it from the hub. The node sends at most 4 MiB past the last ack, keeps every
byte past it on disk, and after a reconnect resends from the offset the hub acked. The hub
appends each chunk to the job's output file and acks once it is synced; a chunk overlapping
what it holds is trimmed, one leaving a gap is a protocol error.
Result. {failure?, exit_code?, message?, output_len, artifacts, usage?}, sent once every
byte up to output_len is acked. artifacts gives each upload's outcome: uploaded, skipped,
too_large or failed. usage is what the job used on the node, {wall_ms, cpu_ms?, peak_mem_bytes?, cpus?, mem_mib?}:
wall_ms: from the driver's start to the job's end, cleanup included;cpu_msandpeak_mem_bytes: the job's VM supervisor and every process under it — the VMM and its vCPUs, service VMs, the switch, the forwards — read from/procafter the last stage and before cleanup, using the same measurement as the executor'sjob resource usagetrace. CPU time is user plus system, guest execution included. Peak memory is each process'sVmHWMsummed: an upper bound where they did not peak together. Host-side cache and artifact transfers, and a host checkout, are not counted. Both are absent when no VM was up to read;cpusandmem_mib: the guest's size, from the job'sMICROVM_CPUSandMICROVM_MEMclamped by the node's ceilings.
usage is optional, as are all its fields except wall_ms.
Reconnects. After its report, a version-3 node sends held. The hub releases each
reservation it does not know, cancels immediate each job it has given up on, and answers
each job it follows with output_ack at the offset it holds.
vk-gitlab speaks GitLab's runner API as gitlab-runner 19.5 does
(network/gitlab.go, network/trace.go):
- Runners.
POST /api/v4/runners/verifyat start with theglrt-token and asystem_idof the daemon's own, persisted. - Job requests.
POST /api/v4/jobs/requestwithRUNNER-TOKEN,infoandlast_update— theX-GitLab-Last-Updateof the previous answer, which lets GitLab hold the request until its queue changes. 201 is a job, 204 none, 403 a token GitLab refuses (the runner stops), 429 and 503 back off asRetry-Aftersays. - Features.
info.featuresadvertisesvariables,image,services,artifacts,cache,fallback_cache_keys,upload_multiple_artifacts,upload_raw_artifacts,refspecs,masking,raw_variables,artifacts_exclude,multi_build_steps,trace_reset,trace_checksum,trace_size,cancelable,cancel_gracefully,return_exit_code,service_variablesandtwo_phase_job_commit. Notsession,terminal,proxy,shared,vault_secrets,service_multiple_aliases(a node's executor takes one alias per service),image_executor_opts,service_executor_opts,native_steps_integrationorjob_inputs.info.executorisvk,info.shellbash. - Commit.
PUT /jobs/:idstate=runningonce the node has accepted the job. A job that cannot be placed is failed instead (runner_system_failure). - Trace.
PATCH /jobs/:id/trace?debug_trace=<CI_DEBUG_TRACE>withJOB-TOKENandContent-Range: <start>-<end>(inclusive, no unit), at most 1 MiB a patch, everyX-GitLab-Trace-Update-Intervalseconds (3 by default, capped at 15 minutes). 202 moves the offset on; 416 moves it to the end of the response'sRange: 0-<n>, which is also how a restarted daemon finds where to resume; 404 or 403 abort the job. The trace preserves the node's output byte for byte. Errors for jobs the daemon refuses or cannot decode, submit or place use gitlab-runner's logger timestamp format (stream00, stdout), unless the job'sFF_TIMESTAMPSis false. The state file preserves this flag across restarts. - Keep-alive. With no new output for 30 seconds,
PUT /jobs/:idstate=runningwith the trace'scrc32:<hex>checksum and byte size. - Final update. Once the hub reports the job finished and its output complete and every
byte is patched:
PUT /jobs/:idwithstatesuccessorfailed,failure_reason,exit_codeandoutput {checksum, bytesize}. 202 and 412 are asked again after the update interval, with backoff up to an hour, as gitlab-runner's final update does. - Cancellation. Every PATCH and PUT answer carries
Job-Status.cancelingasks the hub for a graceful cancel;canceledorfailed, or a 403, an immediate one, after which the daemon writes nothing more for the job.
The daemon keeps one state file per runner, 0600, recording each job it has taken — GitLab
job ID, job token, hub job ID, request_ids — before it submits or commits it. A restarted
daemon resumes each from the hub's view and GitLab's trace offset; a job it had not yet
submitted is submitted with the same request_id.
The state file is <state_dir>/<runner name>.json. state_dir defaults to
/var/lib/vk-gitlab and is created 0700; the daemon refuses to start when group or others
can access it. A daemon run as an unprivileged service user cannot create the default, so
create it for that user before the first start:
install -d -m 0700 -o <user> -g <group> /var/lib/vk-gitlab. A record stays until the hub
has settled its job, including a job already reported to GitLab or abandoned before
submission. A runner's name defaults to its token's short form, so a runner whose token may
change needs an explicit name, or the new token starts an empty file and the old one's jobs
are never resumed; the daemon warns at start about state files no configured runner owns.
Hub jobs and reservations belong to the relay that created them, and the hub answers
not_found for them to any other. The state file records the relay's ID, and a daemon
enrolled as another relay refuses to start while the file holds jobs, as it does a file of a
vk-gitlab from before relays. Before enrolling again (join --replace), stop the daemon
(SIGTERM) and let its running jobs finish within shutdown_timeout; join and leave are
refused while a job is still recorded, until it ends or the file is removed, which gives the
job up to GitLab's timeout.
The daemon does not rotate runner tokens. It verifies the token at start and daily, warns
from a week before GitLab's token_expires_at, and logs an error once it has expired.
The hub never retries a GitLab job. It places a job again only while no node can have
started it: after a refused start, or when the reservation it was sent to is gone and no
start went out. A start whose answer was lost waits for the node's held to say whether it
took it.
| How it ended | JobResult.failure |
GitLab failure_reason |
Older GitLab |
|---|---|---|---|
| success | — | — | — |
| a step's non-zero exit | script |
script_failure |
|
past timeout_secs |
timeout |
job_execution_timeout |
|
| image or service image not pulled or built | image_pull |
image_pull_failure |
runner_system_failure |
| unsupported job, refused by node policy | configuration |
runner_configuration_error |
script_failure |
| clone, artifact or cache transfer failed | external_dependency |
runner_external_dependency_failure |
runner_system_failure |
| VM, agent or host failure | system |
runner_system_failure |
|
| node stopped or drained hard mid-job | interrupted |
runner_interrupted |
unknown_failure |
| canceled | canceled |
unknown_failure: GitLab keeps the job canceled |
|
not placed within place_within_secs |
no_capacity |
runner_system_failure |
|
| node lost | lost |
runner_system_failure |
"Older GitLab" applies when the job's features.failure_reasons lacks the reason; the
fallback follows gitlab-runner's common/failure_reason_mapper.go
(job::gitlab_failure_reason). GitLab's retry: rules then decide whether the job runs
again, on whichever runner asks.
A node is lost for a job once it has been unreachable for job_lost_after (default 5
minutes) past the hub's unreachable threshold. The hub ends the job lost; the daemon
reports it; when the node comes back, its held names the job and the hub cancels it
immediate, and its later uploads fail on a token GitLab has invalidated. A node out of
touch for less keeps running the job and buffering its output, which the trace catches up
on.
A hub restart keeps job records, outputs and offsets, and loses specs and reservations:
a job not yet accepted by a node ends lost, and a reservation the hub no longer knows is
released when its node reports it. A daemon restart loses nothing it had written to its
state file.
Artifact uploads go from the node to GitLab, with the job token, as gitlab-runner's
artifacts-uploader does from inside the build environment. Relaying them through the hub
and the daemon would put every byte through two more hops, through a session that carries
control and telemetry only and whose messages are capped at 1 MiB, for no gain in trust: the
node already holds the token that authorizes the upload. The cost is that the node, not the
daemon, sees GitLab's answer, so it reports each upload's outcome in the result and in the
trace. The node follows gitlab-runner's upload rules: POST /jobs/:id/artifacts multipart
with the archive in field file, query artifact_format, artifact_type and expire_in,
a 307's Location followed, 413 not retried, 503 retried after Retry-After. Unlike
gitlab-runner, it refuses a 307 from https to http and sends the job token after a 307 only
to GitLab's own origin.
Dependency artifacts come from GitLab to the node: GET /jobs/<dependency id>/artifacts
with that dependency's token.
Caches go to the registry the node already uses. A cache is an OCI artifact with one
tar+zstd layer, in repository ci-cache/<gitlab host>[/<gitlab path>]/<project id>,
tagged with the sha256 of its key and the protection of the job's ref, so a protected and an
unprotected job never share one, as with gitlab-runner. The last archive written wins; content addressing
makes an unchanged cache's upload a blob probe. Caches are not shared with gitlab-runner's
cache.zips: a project moving to the fleet starts cold.
A job with a registry token. When the hub mints a token for the job's registry
(step 4), the job's start carries it,
with the repository its ci-cache namespace resolves to. The node then keeps the job's
caches in exactly that repository, an absolute path on the registry beside the node's own
[registry] repo prefix, with the job's token, each tagged with the sha256 of its key alone:
the repository names the protection. It skips the caches, with a warning, when the
repository is not a repository name or the token has expired. A job whose start names no
cache repository keeps today's caches with the node's credential, except under [registry] guest_credential = "job", where it runs without caches, with a warning in its trace,
rather than fall back to the node's credential. The token is masked in the job's trace, and
dropped from its journaled start once its result is written. The node reports its
[registry] base URL in its inventory, which the hub matches against its registries to
mint the token for it.
Under guest_credential = "job" the guest's registry proxy (registry.vk) runs for every
job and injects the job's token, never the node's credential. A job with no token gets a
proxy that refuses everything; so does every job a local gitlab-runner gives the node, which
under "job" reaches the registry only by its own means.
With a token, the proxy forwards the lock API (/lock/) and plain files (/dav/files/) as
well as /v2/; the registry enforces the token's grants. Under /dav/files/ only GET,
HEAD, PROPFIND, PUT, MKCOL, DELETE and OPTIONS pass: MOVE and COPY would name the proxy in
their Destination. The job's steps find the proxy in VK_REGISTRY (http://registry.vk)
and their grants in VK_REGISTRY_NAMESPACES, space-separated <prefix>=<rights> with r,
w and l for read, write and lock. The proxy needs net.mode = "switch", the only network
that serves registry.vk; under any other, a job gets neither the proxy nor these variables.
Archiving and extracting run in the guest, in vk-agent, on the job's tree as the job left
it, with the bytes streamed over vsock; the node does the network transfers. No archiver
needs to be in the job's image. On the node, an archive is converted as it streams; only
what a format needs whole touches the job's disk — a downloaded zip, a cache layer, an
artifact's archive before its upload — each capped at 10 GiB, so a job cannot fill the disk
other jobs share. A request to the cache's registry is bounded by CACHE_REQUEST_TIMEOUT
(minutes, 10 by default), as gitlab-runner bounds its cache transfers.
JobSpec is tagged by kind; gitlab_ci is the only kind. vk-gitlab translates
GitLab's job response into a CiJob, normalizing what the node would otherwise interpret:
defaults are filled in, cache keys are expanded and sanitized (cache/cachekey), aliases are
split, empty formats become zip. What only concerns the conversation with GitLab stays with
the daemon.
Every field of the response, from gitlab-runner 19.5's common/spec/spec.go:
| GitLab field | In the spec | Notes |
|---|---|---|
id |
job.id |
|
token |
token |
|
allow_git_fetch |
sources.allow_fetch |
|
job_info.name, .stage, .pipeline_id, .project_id, .project_name, .project_full_path, .namespace_id, .root_namespace_id, .user_id |
job.* |
project_full_path as project_path |
job_info.organization_id, .instance_id, .instance_uuid, .scoped_user_id |
— | ignored: already among the job's variables where the job needs them |
job_info.time_in_queue_seconds, .project_jobs_running_on_instance_runners_count, .queue_size, .queue_depth |
— | daemon only: queue metrics |
git_info.repo_url |
sources.repo_url |
credentials stripped; the node adds the job token through a credential helper |
git_info.repo_object_format, .ref, .sha, .before_sha, .ref_type, .refspecs, .depth, .protected |
sources.* |
|
runner_info.timeout |
timeout_secs |
|
runner_info.uuid |
— | daemon only: logs |
inputs |
— | job_inputs is not advertised |
variables[] key, value, public, file, masked, raw |
variables |
the daemon appends the runner's own (CI_RUNNER_VERSION, …); the node adds the environment's (CI_BUILDS_DIR, CI_PROJECT_DIR, CI_JOB_STATUS, CI_SERVER_TLS_CA_FILE, …) |
steps[] name, script, timeout, when, allow_failure |
steps |
|
image name, alias, command, entrypoint, ports, variables, pull_policy |
image |
resolved under the node's image rules |
image.executor_opts.docker.platform, .docker.user, .kubernetes.user |
image.platform, image.user |
image_executor_opts is not advertised; when GitLab sends them anyway, these are honoured |
services[] |
services |
as image; run as a compose group beside the job VM, on one node |
artifacts[] name, untracked, paths, exclude, when, artifact_type, artifact_format, expire_in |
artifacts |
|
cache[] key, untracked, policy, paths, when, fallback_keys |
caches |
keys expanded against the job's and the runner's variables, as gitlab-runner expands them; variables only the node sets (CI_PROJECT_DIR, CI_CONCURRENT_ID, …) expand to nothing |
credentials[] of type registry |
registry_credentials |
other types are ignored, as by gitlab-runner |
dependencies[] id, token, name, artifacts_file |
dependencies |
|
features.trace_sections, .token_mask_prefixes |
trace |
trace.limit_bytes is the runner's output_limit, 4 MiB by default |
features.failure_reasons |
— | daemon only: failure mapping |
features.tracing |
— | ignored: no OpenTelemetry export |
secrets |
— | vault_secrets is not advertised; a job with secrets fails runner_configuration_error |
hooks[] |
hooks |
pre_get_sources_script and post_get_sources_script, run in the guest around the checkout |
run |
— | native_steps_integration is not advertised; a job with run fails runner_configuration_error |
policy_options |
— | the daemon writes gitlab-runner's "Job triggered by policy" line to the trace |
suspend_options |
— | ignored: no suspended environments |
the runner's tls_ca_file |
server_ca_pem |
its contents, when set, for CI_SERVER_TLS_CA_FILE; without it CI_SERVER_TLS_CA_FILE is not set |
server_url is the runner's configured URL; job.runner_id the ID /runners/verify
returned.
The node runs gitlab-runner's stage order (common/build.go): prepare_executor (admission,
image, boot, services), prepare_script, get_sources, restore_cache,
download_artifacts, each step_<name>, after_script, archive_cache or
archive_cache_on_failure, upload_artifacts_on_success or _on_failure, then cleanup.
get_sources uses the host-side checkout the executor already has, honouring GIT_STRATEGY,
GIT_DEPTH, GIT_SUBMODULE_STRATEGY, GIT_CHECKOUT and GIT_CLEAN_FLAGS; the other stages
reuse the executor's VM, exec and cleanup code. Step scripts are generated as gitlab-runner's
bash shell generates them (shells/abstract.go, shells/bash.go), and output is masked and
timestamped as its common/buildlogger masks and stamps it: each stream on its own — the
node's lines and vk gitlab prepare as stream 00, vk gitlab run as 01, stdout O and
stderr E apart — masked before it is stamped, the stamps counted in the trace limit.
FF_TIMESTAMPS is read from the job's variables only, on by default as in gitlab-runner 19.5;
a runner's [runners.feature_flags] has no equivalent here. Those ports keep gitlab-runner's
MIT notice.
Ported from gitlab-runner v19.5.0 (MIT; the notice is in NOTICE), as tests of the
vk-gitlab crate unless noted:
| Upstream | Ported as |
|---|---|
network/gitlab_test.go getRequestJobResponse |
a JobResponse JSON fixture; parse it and translate it to a CiJob, checking every field the table above maps |
common/support.go GetSuccessfulBuild, GetRemoteSuccessfulMultistepBuild, GetFailedBuild, getBuildResponse, getStepsBuildResponse |
JobResponse fixtures for multi-step, failing and run jobs |
common/network_test.go Test_Image_ExecutorOptions_UnmarshalJSON, TestFeaturesInfo_JSONMarshaling |
executor options and the info.features body |
network/gitlab_test.go TestGitLabClient_RequestJob, TestSetLastUpdate, …_TransmitsTwoPhaseJobCommit, TestVerifyRunner, TestTokenIsCreatedRunnerToken |
request, long-poll and verify against a fake GitLab |
TestUpdateJob, TestUpdateJobAsKeepAlive |
state updates and Job-Status handling |
TestPatchTrace, TestRangeMismatchPatchTrace, TestPatchTraceContentRangeAndLength, TestPatchTraceContentRangeHeaderValues, TestPatchTraceUrlParams, TestUpdateIntervalHeaderHandling, TestAbortedPatchTrace, TestJobFailedStatePatchTrace |
trace patching, offsets and intervals |
network/trace_test.go (TestJobFinishTraceUpdateRetry, TestJobDelayedTraceProcessingWithRejection, TestJobMaxTracePatchSize, TestCancelingJobIncrementalUpdate, TestJobChecksum, TestJobBytesize) |
the trace loop and final update |
checkTestArtifactsUploadHandlerContent, TestArtifactsUpload, TestArtifactsDownload |
upload and download expectations, in vk-driver's node job tests |
common/failure_reason_mapper_test.go |
vk-hub-proto job::tests (built) |
cache/cachekey tests |
key sanitizing, in vk-gitlab |
helpers/archives zip tests, helpers/trace masker tests |
archive and masking, in vk-driver |
- Uncommitted jobs. What GitLab does with a job taken under
two_phase_job_committhat is never committed — whether it returns it to the queue, and after how long — decides whether a job the fleet cannot place should be failed or left uncommitted. - Registry scopes. A cache write needs a token scoped to the project's cache repository;
vk-registrygrants per-account read and write, not per-repository scopes, so caches start with the node's own registry credential. - Secrets through the hub. The hub sees a spec's secrets in memory. Encrypting them to the reserved node's pinned key would keep them from the hub, at the cost of the hub no longer placing a job on another node than the one reserved.