[ru551n]

Adds -j/--num-compile-threads, the number of files compiled in parallel, independent of -p, so files can be compiled with many threads while tests run with few, for example when the simulator takes a license only to simulate. A file is compiled as soon as the files it depends on are compiled. -j 0 uses all logical CPUs. The default, -j 1, compiles one file at a time as before. --num-sim-threads is added as an alias of -p/--num-threads.

What was added

• Scheduler: SimulatorInterface.compile_source_files compiles each file once its dependencies are compiled, using a thread pool. As before, the dependents of a failed file are skipped, and --keep-compiling keeps going. Without it, no new files start after a failure.

• Library rule: GHDL and Riviera-PRO update a library's index without locking it.

  • Two compiles into the same library lose units. Riviera-PRO even reports success for every compile.
  • A compile reading a library while GHDL rewrites its index fails with "cannot find resource library", because GHDL deletes the index before renaming the new one into place.

  Unless a simulator sets supports_concurrent_compiles_into_library, a file is therefore not compiled while its own library is compiled into or read, or while a library it reads is compiled into. The libraries a file reads are those of its dependencies and those named in its library clauses. NVC sets the flag, because it holds a reader/writer lock on its libraries (see NVC performance with Vunit #1036).

• Library clauses: a library clause fails, in GHDL for example, on a library nothing has been compiled into yet. VUnit now parses library clauses, and a file waits for the first file before it in compile order of each other library it names, so the library exists as it does in a serial compile. A library that comes after the file in compile order fails as it does serially.

• GHDL: with gcc coverage, GHDL compiles one file at a time, because the .gcno files land in the working directory.

• Output: with -j > 1, each file's output is written once its compile finishes, so outputs don't mix.

Measured

A clean compile (--compile --clean -j N) of hdl-modules, 330 files, wall time including VUnit's start-up, median of 3 runs:

-j	NVC	GHDL	Riviera-PRO*
master	5.88 s	7.51 s	11.85 s
1	5.95 s	7.51 s	11.90 s
4	2.51 s	5.62 s	7.39 s
16	1.65 s	5.63 s	7.24 s

* With --keep-compiling, because hdl-modules does not compile completely with Riviera-PRO on master either ('latency attribute in sine_calculator.vhd). Every run compiled the same 327 files.

GHDL and Riviera-PRO level off early because almost every file reads vunit_lib, so the library rule serialises most of the compile. The follow-up #1236, built on this PR, addresses that with --batch-compile, which compiles the files of a library that are ready at the same time in one call, the automatic form of the soft file groups discussed in #1088. With both flags, the clean compile of hdl-modules takes 1.42 s with NVC, 1.64 s with GHDL and 2.66 s with Riviera-PRO.

Tested

• New unit tests for compiling different libraries concurrently, one file per library at a time, the read rule, concurrent readers of a library, waiting for dependencies, skipping dependents and writing each file's output in one piece. They fail on master.
• UI tests: files are compiled with one thread by default, whatever -p is, and with -j threads otherwise; --num-sim-threads sets -p.
• The unit and lint tests pass.
• GHDL stress test: 10 clean hdl-modules compiles each at 4, 16 and 32 compile threads. All 29 runs that reached the compile compiled all 330 files. The 30th failed earlier, on a duplicate configuration name in hdl-modules' own randomised test setup.

A news fragment is included.

🤖 Generated with Claude Code

[LarsAsplund]

After fixing this PR, we should continue with the batch compile PR before merging anything. I haven't reviewed that yet and I suspect that there may be some dependencies with this PR that has to be sorted out.

[ru551n]

Posted by Claude on behalf of ru551n.

OK, #1236 next. I'll rebase it on these changes, and it will pick up -j instead of --parallel-compile.

[LarsAsplund]

runs in parallel -> that runs in parallel