Skip to content

[RF] Reimplement RooFit::MultiProcess IPC without ZeroMQ - #23343

Merged
guitargeek merged 2 commits into
root-project:masterfrom
guitargeek:roofit-no-zmq
Sep 12, 2026
Merged

[RF] Reimplement RooFit::MultiProcess IPC without ZeroMQ#23343
guitargeek merged 2 commits into
root-project:masterfrom
guitargeek:roofit-no-zmq

Conversation

@guitargeek

Copy link
Copy Markdown
Contributor

Replace the ZeroMQ-based interprocess communication in RooFit::MultiProcess with a direct implementation on top of plain socketpair() pipes, and remove the ZeroMQ and cppzmq dependencies together with the RooFitZMQ wrapper library.

The new transport lives in three small classes in the multiprocess package:

  • Message: a byte buffer replacing zmq::message_t, including in the Job::receive_task_result_on_master() interface.
  • Channel: one end of an AF_UNIX socketpair with framed whole-message send/receive on top of the byte stream (8-byte header carrying the payload size and a "more" bit for multipart messages). Sends never block: bytes the kernel buffer does not accept are kept in a per-channel pending buffer that is flushed whenever any channel in the process waits for input, mimicking the previous unlimited high-water-mark ZeroMQ setup and making send-send deadlocks between processes impossible. Multipart messages are flushed on their final frame only, so a k-frame state update costs one send() system call per receiver instead of k.
  • Poller: an index-stable replacement for the ZeroMQPoller.

The socketpairs are created in the ProcessManager before forking, so every process inherits its ends of the already-connected channels; the IPC socket files in /tmp and the PUB-SUB subscriber handshake are gone. All descriptors are opened close-on-exec so they cannot leak into programs executed by user code. One full-duplex channel per link replaces the previous socket pairs: master-queue, queue-worker, and master-worker, where the latter carries both the state updates previously published over PUB-SUB and the task results previously pushed to the master's PULL socket. The master receives results round-robin over the ready worker channels, like the fair queuing of a ZeroMQ PULL socket, and keeps multipart messages together.

SIGTERM handling no longer needs the sigprocmask/ppoll dance (whose zmq_ppoll needed the ZeroMQ draft API, and which plain ppoll would not cover on macOS): the signal handler now writes to a self-pipe that every poll watches, which closes the same check-then-block race. Benign signal interruptions (profilers, SIGCHLD, debuggers) are retried inside the wait primitive instead of surfacing to the event loops, so they can no longer desynchronize multi-frame message sequences; only a SIGTERM leaves a blocking call exceptionally, which also made the old EINTR-retry ladders at the call sites collapse.

During JobManager shutdown the ProcessManager now terminates the child processes before the Messenger closes the channels, so no process sees a closed connection during a normal shutdown. An unexpectedly closed connection fails fast instead of hanging in a poll that can never return, and exceptions on the forked child processes are caught in JobManager::activate() so they exit in an orderly way rather than unwinding into the master-side stack inherited through fork.

Since the feature no longer needs external dependencies, it is now always built on non-Windows platforms. The roofit_multiprocess build option therefore has no effect anymore: it is deprecated with a warning and will be removed one release cycle later.

🤖 Done with the help of AI

The worker-side update wrote the received offsets into the vector with
reserve() plus std::copy to begin(), which is undefined behavior
whenever the vector is shorter than the message: the copy writes into
raw reserved storage past end() and the vector size stays stale, so
the received offsets are silently invisible afterwards. The mainstream
fit flows are only safe because LikelihoodJob::update_state (which
uses resize(), correctly) happens to run first and size the shared
vector. Use resize() here too, like in LikelihoodJob.

🤖 Done with the help of AI
Replace the ZeroMQ-based interprocess communication in
RooFit::MultiProcess with a direct implementation on top of plain
socketpair() pipes, and remove the ZeroMQ and cppzmq dependencies
together with the RooFitZMQ wrapper library.

The new transport lives in three small classes in the multiprocess
package:

- Message: a byte buffer replacing zmq::message_t, including in the
  Job::receive_task_result_on_master() interface.
- Channel: one end of an AF_UNIX socketpair with framed whole-message
  send/receive on top of the byte stream (8-byte header carrying the
  payload size and a "more" bit for multipart messages). Sends never
  block: bytes the kernel buffer does not accept are kept in a
  per-channel pending buffer that is flushed whenever any channel in
  the process waits for input, mimicking the previous unlimited
  high-water-mark ZeroMQ setup and making send-send deadlocks between
  processes impossible. Multipart messages are flushed on their final
  frame only, so a k-frame state update costs one send() system call
  per receiver instead of k.
- Poller: an index-stable replacement for the ZeroMQPoller.

The socketpairs are created in the ProcessManager before forking, so
every process inherits its ends of the already-connected channels; the
IPC socket files in /tmp and the PUB-SUB subscriber handshake are gone.
All descriptors are opened close-on-exec so they cannot leak into
programs executed by user code. One full-duplex channel per link
replaces the previous socket pairs: master-queue, queue-worker, and
master-worker, where the latter carries both the state updates
previously published over PUB-SUB and the task results previously
pushed to the master's PULL socket. The master receives results
round-robin over the ready worker channels, like the fair queuing of a
ZeroMQ PULL socket, and keeps multipart messages together.

SIGTERM handling no longer needs the sigprocmask/ppoll dance (whose
zmq_ppoll needed the ZeroMQ draft API, and which plain ppoll would not
cover on macOS): the signal handler now writes to a self-pipe that
every poll watches, which closes the same check-then-block race.
Benign signal interruptions (profilers, SIGCHLD, debuggers) are
retried inside the wait primitive instead of surfacing to the event
loops, so they can no longer desynchronize multi-frame message
sequences; only a SIGTERM leaves a blocking call exceptionally, which
also made the old EINTR-retry ladders at the call sites collapse.

During JobManager shutdown the ProcessManager now terminates the child
processes before the Messenger closes the channels, so no process sees
a closed connection during a normal shutdown. An unexpectedly closed
connection fails fast instead of hanging in a poll that can never
return, and exceptions on the forked child processes are caught in
JobManager::activate() so they exit in an orderly way rather than
unwinding into the master-side stack inherited through fork.

Since the feature no longer needs external dependencies, it is now
always built on non-Windows platforms. The roofit_multiprocess build
option therefore has no effect anymore: it is deprecated with a
warning and will be removed one release cycle later.

🤖 Done with the help of AI
@github-actions

Copy link
Copy Markdown

Test Results

    23 files      23 suites   3d 19h 39m 50s ⏱️
 3 874 tests  3 874 ✅ 0 💤 0 ❌
79 120 runs  79 120 ✅ 0 💤 0 ❌

Results for commit 02f098f.

@guitargeek
guitargeek merged commit e6fc92a into root-project:master Sep 12, 2026
41 of 43 checks passed
@guitargeek
guitargeek deleted the roofit-no-zmq branch September 12, 2026 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants