Running parallel instances using namespaces

[BsAtHome]

The list of tests is getting very long. We can either change the code to allow port/key/address shifting or, for example, Linux namespaces (using unshare).

@hdiethelm has tested a bit already. Any suggestions and links are welcome.

[BsAtHome]

See also #2722

[hdiethelm]

What I managed is to get two linuxcnc instances running using:
sudo unshare --net --ipc su $USER -c "../scripts/linuxcnc"
and removing the lockfile check in scripts/linuxcnc.in.

--net separates the network namespaces and also the named fifo in rtapi_app. The named fifo can also be set with RTAPI_FIFO_PATH
--ipc separates the shared memory namespaces.

Interestingly, if I close the second axis gui, the first one closes also. There is some hidden connection somewhere, might be close window by name or a PID file.

Using overlayfs and mount namespaces, it should be possible to reuse existing files but new files like the lockfile and possibly others are volatile and per instance. But I did not continue investigating future.

[BsAtHome]

The real problem is the sudo. I never allow a build/test process to use sudo.

The linuxcnc script kills processes by name and kills everything it finds. No process currently creates PID files. That was one thing I was working on because you only want to kill what you created.

[hdiethelm]

The real problem is the sudo. I never allow a build/test process to use sudo.

You can get rid of the sudo by creating a user namespace where you are root inside and from there creating the net and ipc namespace. This is how podman works. I need to dig up the doc again.

The linuxcnc script kills processes by name and kills everything it finds. No process currently creates PID files. That was one thing I was working on because you only want to kill what you created.

This can be isolated with a pid namespace.

So it should all be possible somehow to create container like separation for the tests but it will be a bit complex. Also test scheduling and result retrieval needs some effort. That's why I did not continue.

Finalizing #2722 sounds also not simple but it could help for more than only the tests.

[rmu75]

That sounds a bit like what docker is doing, maybe that could be a way forward that doesn't involve dealing with all the details of namespace handling?

[grandixximo]

I've been running parallel instances locally for a while to test different branches side by side. I use bubblewrap, which works as a normal user with no sudo:

#!/bin/bash
exec bwrap --dev-bind / / --unshare-ipc --unshare-pid --unshare-net \
    --proc /proc --tmpfs /tmp --tmpfs /dev/shm --die-with-parent -- "$@"

Then sandbox.sh linuxcnc -r config.ini or sandbox.sh halrun ....

• --unshare-ipc plus a private /dev/shm separates the HAL/RTAPI shared memory.
• --unshare-net separates the rtapi_app FIFO and any abstract sockets.
• --unshare-pid means the kill-by-name in the linuxcnc script only sees its own processes. That should also cover the "closing one AXIS closes the other" effect.
• A private /tmp gives each instance its own lockfile, so scripts/linuxcnc.in needs no changes.

Limits: my runs are headless. The private /tmp hides /tmp/.X11-unix, so GUI tests would need that socket bound back in or an xvfb inside the sandbox. It isn't realtime either, because setuid gets no privileges inside. It also needs unprivileged user namespaces, which some distros restrict (Ubuntu 24.04 does it through AppArmor).

[hdiethelm]

That sounds a bit like what docker is doing, maybe that could be a way forward that doesn't involve dealing with all the details of namespace handling?

Exactly, but with docker, you can not share the root file system. Normally, this is what you want. But with linuxcnc and all the dependency's, the image is ~2.5GB in size, so it could be slower at the end instead of faster due to the image has to be built or downloaded in CI.

@grandixximo Thanks for the tip with bwrap, it's way easier than using unshare.

The following command does something similar with unshare:
unshare --user --map-users=1000,1000,1 --map-groups=1000,1000,1 --fork --pid --net --ipc --mount-proc ../scripts/linuxcnc

By using user namespace, sudo is not needed. With overlays, tmp can be isolated for new files but shared for existing ones. This works with X11 and the lockfile:
bwrap --dev-bind / / --overlay-src /tmp --tmp-overlay /tmp --dev-bind /tmp/.X11-unix /tmp/.X11-unix --unshare-ipc --unshare-pid --unshare-net --proc /proc --die-with-parent -- ../scripts/linuxcnc

Somehow, the second linuxcnc still manages to kill the first one. There is still something interfering.

[hdiethelm]

Found it: axis-remote --quit called from linuxcnc at shutdown goes trough the X server. This can not be isolated or there is just no GUI.

For tests, this should be no issue, X must be isolated anyway to use an xvfb for the gui tests.

But it is annoying otherwise, it kills just an axis gui running on the same X server. Sometimes the right one, sometimes the wrong one.

[BsAtHome]

Found it: axis-remote --quit called from linuxcnc at shutdown goes trough the X server. This can not be isolated or there is just no GUI.
But it is annoying otherwise, it kills just an axis gui running on the same X server. Sometimes the right one, sometimes the wrong one.

It is actually wrong in a parallel program context way. This should connect to a socket for the instance you want to talk to and not the container.

For tests, this should be no issue, X must be isolated anyway to use an xvfb for the gui tests.

Yes, for tests it works fine because they use a dedicated xvfb instance, but otherwise, it needs to be separated out.

[hdiethelm]

Found it: axis-remote --quit called from linuxcnc at shutdown goes trough the X server. This can not be isolated or there is just no GUI.
But it is annoying otherwise, it kills just an axis gui running on the same X server. Sometimes the right one, sometimes the wrong one.

It is actually wrong in a parallel program context way. This should connect to a socket for the instance you want to talk to and not the container.

Indeed, it just operates on the window named axis trough X11. If I have 2 running and change to axis #2 in axis-remote, it closes the second window.

xwininfo -tree -root | grep axis
0x5200019 "axis.ngc - AXIS 2.10.0~pre1 auf LinuxCNC-HAL-SIM-AXIS": ("axis #2" "Axis")  847x752+5+29  +3169+391
0x4c00019 "axis.ngc - AXIS 2.10.0~pre1 auf LinuxCNC-HAL-SIM-AXIS": ("axis" "Axis")  847x752+5+29  +2463+386

Looks like linuxcnc is designed from the ground up to run one instance. Using bwrap should work but it feels more like a workaround than a real solution.

It's probably still worth integrating if there are no other show stoppers going trough some files. The parallel test run will need more or less the same framework, independent of the bwrap isolation. As soon as LINUXCNC_INSTANCE is available, the bwrap isolation can just be removed.

[hdiethelm]

So, first prototype: https://github.com/hdiethelm/linuxcnc-fork/tree/tests_parallel

time scripts/runtests -p -j 128
...
real	1m12.127s
user	0m1.634s
sys	0m1.694s

Most tests spend most time waiting, so it's a massive speedup and given enough ram, all tests can pretty much be run in parallel...
It seams to work. However, I had to isolate $HOME. If I don't do this, tests randomly fail. There are probably some files there also. Only isolating the linuxcnc-src folder is not enough.

Still missing: A way to collect all the results to create a summary and some cleanup. It is just a quick&dirty POC for now.