Skip to content

src/client/linux/minidump_writer: make minidump stack-skipping test more deterministic - #48

Open
illwieckz wants to merge 1 commit into
masterfrom
illwieckz/threads
Open

illwieckz wants to merge 1 commit into
masterfrom
illwieckz/threads

Conversation

@illwieckz

Copy link
Copy Markdown
Member

So, for this I may need some expertise by people like @slipher.

Basically, there is a test that can randomly fail in CI. It's related to threading. It's not just related to threading inside the test, but it is subject to the generic threading pressure on the build machine. For example on the CI using make -j1 may not reproduce the failure, while make -j4 would. It depends on the machine. For example on my end I got it with make -j32 once.

At some point while adding some printf for debugging purpose made the bug disappear, and after removing them and adding other printf elsewhere, a single CI run reproduced it on both linux-amd64-clang, linux-arm64-gcc and linux-i686-gcc but not linux-amd64-gcc… Adding such printf may have impacted some thread synchronisation as a side effect and may reduce or increase the chance of reproducing the bug.

By luck, the CI reproduces the failure very reliably with the linux-i686 platform, so I plumbed that platform in the CI the proper way to track down it:

My own machine was unreliable to reproduce the bug even with the linux-i686 platform though I managed to catch it sometime, demonstrating it could be reproduced outside the CI. So I did all the debugging investigation by pushing debugging and diagnostic code to the CI, reading the log and doing back and forth implementing diagnostic code and attempted fixes followed by test runs this way.

I'm not super familiar with threading so I got assisted by an LLM. I hope the purpose of the test has not been modified in the process.

With that patch, all the CI runs are passing reliably on that previously-unreliable test, on all platforms I tested (they all build and run all the checks):

  • Linux amd64 GCC
  • Linux i686 GCC
  • Linux arm64 GCC
  • Linux amd64 Clang
  • Windows amd64 MinGW
  • macOS amd64 AppleClang
  • macOS arm64 AppleClang

I also tested a FreeBSD amd64 Clang run on my end and everything passed.


Make minidump stack-skipping test more deterministic.

  • Check pthread attribute initialization, configuration, and thread creation errors.
  • Give the helper thread a dedicated and zero-initialized stack, to make the test less dependent on residual stack contents when checking whether stacks referencing the executable are skipped.

The test checks that a helper thread that runs in libc and doesn't reference the main executable is excluded from the minidump. Initializing the dedicated stack should reduce the chance that stale pointers into the executable mapping cause the stack to be included.

This changes a bit the original conditions of the test, since residual stack contents that could affect the result are eliminated. But it should make the test more deterministic.

This test has been observed to sometimes fail in CI, reporting more threads with dumped stacks than expected.

…ore deterministic

- Check pthread attribute initialization, configuration, and thread
  creation errors.
- Give the helper thread a dedicated and zero-initialized stack, to make
  the test less dependent on residual stack contents when checking whether
  stacks referencing the executable are skipped.

The test checks that a helper thread that runs in libc and doesn't
reference the main executable is excluded from the minidump.
Initializing the dedicated stack should reduce the chance that stale
pointers into the executable mapping cause the stack to be included.

This changes a bit the original conditions of the test, since residual
stack contents that could affect the result are eliminated. But it
should make the test more deterministic.

This test has been observed to sometimes fail in CI, reporting
more threads with dumped stacks than expected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant