Repository navigation
fix(keploy-ci): start-docker starts containerd itself and waits on each daemon while it works - #33
Merged
Merged
Conversation
slayerjain
force-pushed
the
fix/start-docker-own-containerd
branch
3 times, most recently
from
October 19, 2026 06:43
18492a3 to
c35b0bb
Compare
…ch daemon while it works
Enterprise lanes failed before any of their own code ran. start-docker's dockerd quit with "failed to start containerd: timeout waiting for containerd to start": pipeline 11771 (go-docker-timefreeze #214), and four steps of 11931 (194, 369, 402 and 418).
- dockerd starts the containerd it manages and gives it a fixed 15 s (moby's startupTimeout, in 29.1.5 daemon/internal/libcontainerd/supervisor/remote_daemon.go).
- On those starved runners containerd took longer. In 11771 it was silent for 11.8 s or more after its bolt mount manager loaded. In 11931 it sat 15 s or more in its bolt metadata store's open, logging "waiting for response from boltdb open".
- start-docker's own wait, 60 tries a second apart, was a clock too.
start-docker now:
- starts containerd itself, with the layout dockerd gives the containerd it manages: root and state under /var/lib/docker and /var/run/docker (so on the image's /var/lib/docker volume), the CRI plugin off, its socket at /var/run/docker/containerd/containerd.sock.
- containerd runs in a session of its own (setsid), as moby starts it, so it outlives a terminal that ends.
- It logs at dockerd's level, as moby gives the containerd it manages: debug with --debug (-D, or any of Go's true spellings), else dockerd's --log-level (-l, either form), info by default.
- It is waited for until `ctr version` answers.
- then starts dockerd with --containerd on that socket, and waits until `docker info` answers.
- waits on each for as long as it is alive and its log grows:
- it fails at once, with both daemons' logs, when the daemon exits;
- it fails only when the daemon has neither answered nor logged for START_DOCKER_SILENCE_CHECKS looks in a row (600 by default, a hang guard).
- A look is a second. The probe is never cut short: one runs in the background at a time, each look sees whether it has finished, and another starts once it has answered no. dockerd listens on its socket before its init and answers only after, and `docker info` asks containerd, the daemon that starves, for its system info, so a probe may take long. A probe killed at a bound would be killed on every look, and a start that would have answered would fail. A slow init that goes on logging is waited for, however long, and only a daemon that stops progressing ends the wait (and then the probe in flight is killed).
- A stall is said as it goes on: every START_DOCKER_NOTICE_LOOKS quiet looks (30), with the daemon's last log line, so a step that a CI cap ends still says why.
- START_DOCKER_PROBE_SECONDS (10 s) bounds only the one look start-docker takes at a socket no pidfile names, which it has nothing else to watch on. What answers there is reused; a socket that refuses (nothing listens) is taken over; one that listens and does not answer within the bound fails the start, said: its daemon may be slow, not gone, and a second one is not started on the same root. The probes name the socket (`docker -H unix:///var/run/docker.sock`), whatever DOCKER_HOST or the docker context say.
- restarts containerd when it exits, as dockerd's own supervisor did: a watcher, in its own session, starts it again and says so in /var/log/containerd.log, and dockerd reconnects.
- runs one start at a time (flock on /var/run/start-docker.lock, closed for every process it starts, so a daemon does not hold it); another start waits for it.
- run again, reuses what is there:
- a dockerd or containerd an earlier start left running (their pidfiles, written as each is started; the watcher keeps containerd's), waited for until it answers however far it got, as after a start killed mid-wait, and not started twice on one root;
- a containerd its watcher is starting again, once it has;
- a dockerd or containerd that answers but that no pidfile names;
- the watcher an earlier start left (its pidfile, checked to name the watcher and not some other sh): one watcher at a time.
- refuses DOCKERD_FLAGS, and /etc/docker/daemon.json keys, that move what it fixes (--data-root, --exec-root, --pidfile, --host, --containerd, --config-file, short forms run together as pflag reads them, such as -DH and -Htcp://...), since containerd's root and socket and dockerd's pidfile and socket are fixed here. DOCKERD_FLAGS is split into words, not globbed.
- when a start fails, kills what it started, dockerd, the watcher, then containerd, and reaps each, so nothing of it is left, not even a zombie, and the next begins clean. What an earlier start left, and this one only waited for, is not this one's to kill.
- The watcher is sent SIGTERM, on which it kills the last containerd it started ($!, as a TERM can come between that start and its pid=$!) and reaps what it started.
- With the trap killing $pid, 40 runs in 300 of a watcher-alike TERMed at a random moment under dash left a child; with $!, none in 600.
- appends to both logs, not truncating them under a daemon still writing them.
With --containerd, dockerd still gives up about a second after containerd stops answering it ("failed to get containerd plugins"); that is not a start's wait, and it is left as it is.
Tests: keploy-ci/start-docker-test.sh runs each case in a fresh privileged container of the keploy-ci image, with this checkout's script. In CI (docker-build.yml) that is the image this commit builds, so an engine or containerd bump is tested with start-docker before it is released. Each case's container runs detached, known by its id, and is waited for under a hang guard (CASE_GUARD, 900 s), which ends a case that hangs, fails it and removes its container: a guard around `docker run` would only signal the client, which hands the signal to the container's PID 1 and waits on. The wait runs in the background, so an INT or TERM to the run is acted on at once. A container goes with its volume (the image's /var/lib/docker), and the one running goes too when the run is ended. Stand-ins first on PATH make containerd slow, dead or silent, and dockerd slow, wedged or failing. The cases:
- the guard ends a case that hangs and removes its container;
- settings that are not whole numbers, 1 or more, and DOCKERD_FLAGS or daemon.json keys that move what start-docker fixes (in each spelling above), are refused (2), and flags that do not (-ldebug, --dns) are not;
- it starts both, with DOCKERD_FLAGS reaching dockerd (a registry mirror, as the lanes give it, and --debug). containerd and its watcher each lead a session of their own, and containerd logs at debug. It builds and runs a container (an image of the engine's static docker-init, nothing pulled);
- a second start finds the first's dockerd (its pidfile) and returns once it answers, and with --log-level=debug containerd logs at debug; the first runs with DOCKER_HOST pointing elsewhere;
- the failure fixed, replicated: dockerd starting a containerd that takes 20 s quits with "timeout waiting for containerd to start";
- a containerd that takes 75 s, logging as it loads, is waited for with 60 looks of silence allowed (more than a real daemon here keeps silent: 15 s on a starved runner): it is the growing log that keeps the wait going;
- a containerd silent for 20 s, then up, is waited for within its silence checks;
- a containerd that exits fails the start at once (1), with its log;
- a containerd that neither answers nor logs fails after its silence checks (1, the start's own status, not the guard's);
- a dockerd that exits (a bad flag) fails the start at once. Run under a subreaper that reaps nothing, the failed start leaves nothing behind, running or a zombie. The next start comes up with one containerd, and both logs keep the first start's lines. With -l debug, containerd logs at debug;
- a start that fails after its watcher started containerd again leaves nothing either (and with --log-level debug, containerd logs at debug);
- a dockerd stopped once its socket is listening (its probes block) that goes on logging for 75 s is waited for with 60 looks of silence allowed. One that logs nothing is said to stall at 2 quiet looks and fails at 4, said ("dockerd has not answered, and its log has not grown in 4 looks");
- a docker that takes 8 s to answer, with 30 looks of silence allowed and the one-shot bound at 2 s, is waited for and its answer taken;
- a socket no pidfile names that refuses is taken over, and the start goes on; one that listens and does not answer fails the start, said, and nothing is started over it;
- a start killed after it started dockerd, before dockerd wrote its own pidfile, leaves its pid: the next start waits for that dockerd;
- a watcher pidfile that names some other sh is no watcher: containerd gets one of its own;
- a start while containerd's watcher is starting it again waits for the watcher, and reuses the containerd it starts;
- a second start while the first's containerd is still starting waits for the first (the lock), then for its dockerd, and nothing is started twice: one containerd, one watcher;
- a start after one killed while containerd was starting waits for that containerd and gives it a watcher; one after a start killed in dockerd's init waits for that dockerd until it answers;
- a containerd killed under a running dockerd (--debug=1, containerd at debug too) is started again, said in its log. dockerd says it is back on it (its event stream processed again), and docker runs a container;
- a start after dockerd stopped reuses the containerd that answers, starts no second one, and appends to dockerd's log; containerd is at info by default.
Each of 33 mutants of start-docker fails the cases, among them: the probe in a wait bounded (a slow docker info cut every look), the one-shot look at an unknown socket unbounded, a mute unknown socket taken over, progress ignored, no lock, dockerd holding the lock, a dockerd or containerd an earlier start left not waited for, no dockerd pid recorded, a restarting containerd not waited for, a second watcher, the watcher known by comm sh, the logs truncated, the watcher in the caller's session, dockerd's level not given to containerd, Go's bool spellings or -l VALUE ignored, the settings, fixed flags, run-together short flags or daemon.json not checked, docker info following DOCKER_HOST, no stall notice, a failed start that does not reap, a failed start that keeps what it started, a watcher killed instead of stopped, and a watcher that leaves its containerd.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
slayerjain
force-pushed
the
fix/start-docker-own-containerd
branch
from
October 19, 2026 10:55
c35b0bb to
58e504c
Compare
slayerjain
marked this pull request as ready for review
October 19, 2026 11:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
start-dockerno longer lets dockerd start containerd. dockerd gives the containerd it starts a fixed 15 s and then quits with "timeout waiting for containerd to start". Enterprise pipelines 11771 and 11931 (four steps) failed that way on starved runners.start-dockernow:--containerdpointing at it.START_DOCKER_PROBE_SECONDS, 10 s). A probe that blocks counts as no answer that look; it does not end the wait.START_DOCKER_SILENCE_CHECKSquiet looks (600). That is a hang guard, not a time budget.START_DOCKER_NOTICE_LOOKS), so a CI cap that kills the step first still shows why.--data-root,-H,-DH,--config-file, and the rest).keploy-ci/start-docker-test.shhas 24 cases. Each runs in a fresh privileged container of the image this commit builds (thestart-dockerjob in docker-build.yml), under a working per-case hang guard, and the container is removed with its volume. The cases cover:All 33 mutants are caught. The watcher-trap window was shown separately: 40 of 300 runs left a child before the fix, 0 of 600 after.
After merge, a keploy-ci image release and a pin bump in enterprise's
.woodpeckertake it to the lanes.