atomic_queue:tests timeouts when machine has a single CPU

[sanvila]

Hello. When I build this package on AWS machines of type m7a.medium or r7a.medium, which incidentally have a single CPU, the tests do not pass. This is an example for version 1.7.2:

==================================== 1/2 =====================================
test:         atomic_queue:tests
start time:   04:41:28
duration:     30.04s
result:       killed by signal 15 SIGTERM
command:      MESON_TEST_ITERATION=1 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MALLOC_PERTURB_=217 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/tests
==============================================================================

==================================== 2/2 =====================================
test:         atomic_queue:example
start time:   04:41:58
duration:     11.73s
result:       exit status 0
command:      MESON_TEST_ITERATION=1 MALLOC_PERTURB_=160 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/example
==============================================================================


Summary of Failures:

1/2 atomic_queue:tests   TIMEOUT        30.04s   killed by signal 15 SIGTERM

Ok:                1   
Fail:              0   
Timeout:           1   

Can I assume that atomic_queue:tests is not meant to be run on single-cpu systems, and if so, would it make sense if the test itself could detect that and be skipped?

(Note: This is usually reproducible by booting with GRUB_CMDLINE_LINUX="nr_cpus=1"

Thanks.

[max0x7ba]

Hello, Santiago,

Can I assume that atomic_queue:tests is not meant to be run on single-cpu systems,

While some of the unit-tests don't need more than a single CPU to run, the most important unit-tests do require at least 2 CPUs to be able to complete. (In the past, the GitHub CI runners had 2 CPUs, nowadays they have 4 CPUs.)

would it make sense if the test itself could detect that and be skipped?

The most important unit-tests do require at least 2 CPUs to run. Skipping any of these most important unit-test makes running the rest of the unit-tests pretty much worthless, I am afraid.

---

You should upgrade from v1.7.2 to the latest version (currently v1.9.4) for the best performance. This won't change the minimum 2-CPU requirement for the unit-tests, though.

[max0x7ba]

This is usually reproducible by booting with GRUB_CMDLINE_LINUX="nr_cpus=1"

The unit-tests getting stuck forever on a single-CPU-system is reproducible without rebooting by using taskset -c 0, e.g.:

chrt -f 1 taskset -c 0 make -C ~/src/atomic_queue -R run_tests

[sanvila]

Ok, a little bit of background: I'm building the 41000 packages in Debian unstable on single-CPU systems. As of today, only 20 of them do not build ok, and we don't have any debian/control field to indicate "this package will only build on systems with 2 or more CPUs". Debian packages (.deb) can be created in a machine to be used in another machine, so being able to build the package is always useful.

One possible course of action here would be to check nproc in debian/rules and skip all the tests if it's 1.

If this is what you recommend (not running the tests at all when it's already known that they will not pass), I guess this issue can be closed.

Thanks.

[max0x7ba]

Well, I just double-checked, and

taskset -c 0 make -C ~/src/atomic_queue -R run_tests

does complete the unit-tests, but takes much longer. Because the spin-waiting thread gets preempted.

whereas:

chrt -f 1 taskset -c 0 /tmp/b1/release/gcc/tests --log_level=unit_scope --report_level=short

gets stuck as expected. Because the spin-waiting thread cannot get preempted (when the rt throttling is disabled).

---

With many CPUs:

$ /bin/time chrt -f 1 /tmp/b1/release/gcc/tests --log_level=unit_scope --report_level=short
...
Test module "atomic_queue" has passed with:
  30 test cases out of 30 passed
  42676 assertions out of 42676 passed

Elapsed time 0:01.81, user 10.41s, system 0.00s, CPU 575%, RSS 4896kB, exit-status 0.

45× slower with one CPU:

$ /bin/time taskset -c 0 /tmp/b1/release/gcc/tests --log_level=unit_scope --report_level=short
...
Test module "atomic_queue" has passed with:
  28 test cases out of 30 passed
  2 test cases out of 30 passed with warnings
  42674 assertions out of 42674 passed
  2 failed warnings

Elapsed time 1:28.28, user 88.16s, system 0.00s, CPU 99%, RSS 5408kB, exit-status 0.

[max0x7ba]

Ok, a little bit of background: I'm building the 41000 packages in Debian unstable on single-CPU systems. As of today, only 20 of them do not build ok, and we don't have any debian/control field to indicate "this package will only build on systems with 2 or more CPUs". Debian packages (.deb) can be created in a machine to be used in another machine, so being able to build the package is always useful.

I understand your predicament now.

Albeit running the unit-tests with one CPU is somewhat possible, it cannot possibly test the scenarios when multiple threads contentend on the queue data members or its elements simultaneously. Which is the most important unit-test.

---

tests.cc is intended to be portable by not using anything beyond std:: and boost:: libraries.

std::thread::hardware_concurrency() reports the total number of cores, rather than available. I am not sure about how to detect the number of available CPU cores from C++ code portably.

---

The Makefile uses nproc to detect the number of available CPUs, but the Makefile is Linux-only. CMake and Meson aren't restricted to Linux and, hence, cannot easily invoke nproc.

Setting an environment variable to tell the unit-tests to skip some tests is easily possible.

---

Well, yet another option:

$ /tmp/b1/release/gcc/tests --help=run_test
  run_test
    Filters which tests to execute.
    --run_test=<test unit filter>
    -t <test unit filter>

  Filters which test units to execute. The framework supports both 'selection
  filters', which allow to select which test units to enable from the set of
  available test units, and 'disabler filters', which allow to disable some test
  units. Boost.test also supports enabling/disabling test units at compile time.
  These settings identify the default set of test units to run. Parameter run_test
  is used to change this default. This parameter is repeatable, so you can specify
  more than one filter if necessary.

I should find out what to pass into --run_test= option to disable running the multi-core unit-tests...

[sanvila]

Ok, I was actually ready to use the nproc trick in debian/rules, but given that you are willing to find something better (even if only for a small margin), I will be more than glad to test whatever fix you can propose. Thanks a lot!

[max0x7ba]

I should find out what to pass into --run_test= option to disable running the multi-core unit-tests...

Setting environment variable BOOST_TEST_RUN_FILTERS='!stress*' disables running the stress family of unit-tests, which spawn threads and require more than one CPU.

Would this option solve this issue for you, Santiago?

[sanvila]

Yes, it's perfect, and the best part is that this is already implemented and does not require any changes from your side.
I guess this can be closed now.

[max0x7ba]

the best part is that this is already implemented and does not require any changes from your side.

Exactly.

But, please, do package the latest version of the library (v1.9.4 as of today). v1.7.2 had a lot of unintended pessimizations, so that v1.9.4 is a bit of next level performance.

v1.7.2 and v1.9.4 have the same API and ABI, so that a v1.7.2 queue object can be operated upon with v1.9.4 code, and vice versa. v1.9.4 code-paths are much better optimized.

I guess this can be closed now.

Up to you. Whenever you've packaged v1.9.4 and are happy with it 😇

[sanvila]

Hi. The package is team-maintained. I tried upgrading to 1.9.4 but had a little problem with autopkgtest, so I decided to fix the single-cpu issue for now and documented the autopkgtest problem here for my team fellows to see:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1148713

So yes, we will eventually upgrade to 1.9.4.

Thanks.

[max0x7ba]

Hi. The package is team-maintained. I tried upgrading to 1.9.4 but had a little problem with autopkgtest, so I decided to fix the single-cpu issue for now and documented the autopkgtest problem here for my team fellows to see:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1148713

There was a change removing -I. preprocessor option because double-quote #include "x" (like in #include "benchmarks.h") must first search the directory of the source file with #include "x" directive, so that -I. seemed superfluous.

Whereas angle-quote #include <x> doesn't search the directory of the source file with #include <x> directive, and this requires using -I. preprocessor option.

---

Using c++ instead of g++ does often result in unexpected compiler errors. This can be the root cause of the error you refer to because the CI builds with g++ and clang++ from scratch just fine, e.g. the latest CI build: https://github.com/max0x7ba/atomic_queue/actions/runs/35728700941

[max0x7ba]

Hi. The package is team-maintained. I tried upgrading to 1.9.4 but had a little problem with autopkgtest, so I decided to fix the single-cpu issue for now and documented the autopkgtest problem here for my team fellows to see:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1148713

So yes, we will eventually upgrade to 1.9.4.

Would you take v1.9.5 which fixes this bug-report? 😁