Conversation
Port of the budget selection from upstream (1fd4b6e, 7918dfd, 5e152c3), reduced to the part that chooses the per-PID timeout: - `LLDB_TIMEOUT` (30 s) for release, `LLDB_SLOW_BUILD_TIMEOUT` (120 s) for debug, sanitizer and coverage servers, read from `args.build_flags`, or from the binary via `clickhouse local` (`is_slow_build_binary`) when the server never started and the flags were not collected. - `print_c_stacktraces` takes `per_pid_timeout` and `total_timeout`, clamps each attach to what is left of the aggregate ceiling and names the pids it skipped. The per-test timeout handler, which runs inside a fired one-shot alarm, keeps the tight pair (30 s per pid, 60 s total); the abort paths take the flavor budget. Not ported: the stacktrace log files, the truncation marker and `keep_output_on_error` (a separate upstream change), and the removal of the on-demand lldb install, which the fork's images still need. Tests mirror upstream's, run without a server. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`get_stacktraces_from_lldb` killed lldb on the budget and assumed the server was untouched. It is not: lldb attaches with ptrace, which stops the inferior, and loads symbols afterwards; on a large sanitizer binary that alone outlasts the budget. When the tracer is killed while the inferior sits in that stop, the kernel detaches it but leaves it in group-stop, and nothing continues it. Seen on an MSan CAS-S3 stateless shard: six lldb attaches all hit the budget; the one aimed at the server left it in state T (the watchdog logged `Child process was stopped by signal 19`), and from then on the server ignored SIGTERM and SIGTRAP, `system flush logs` and `clickhouse stop` timed out, and the system-table dump failed on the server's own status-file lock. Run lldb in its own process group, kill the whole group on timeout, then send SIGCONT to the target, which is a no-op for a process that was never stopped. The test stops a target by hand, substitutes a never-returning command for lldb, and checks the target is running again after the timeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
k-morozov
self-requested a review
September 16, 2026 12:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes to
tests/clickhouse-test's lldb stack collection, each its own commit:1fd4b6eea445,7918dfd548d4,5e152c370237, reduced to the timeout selection): 30 s per pid for release, 120 s for debug/sanitizer/coverage, read fromargs.build_flagsor from the binary viaclickhouse localwhen the server never started;print_c_stacktracesgets an aggregate deadline and names the pids it skips; the per-test timeout handler keeps the tight 30 s / 60 s pair. Not ported: the stacktrace log files and truncation marker, and the removal of the on-demand lldb install (the fork's images still need it).Tests in
ci/tests/test_print_stacktraces.py: six ported budget tests plus one that stops a target by hand, substitutes a never-returning command for lldb, and checks the target runs again after the timeout. All nine tests in the file pass locally against a standalone server.Related: #2298
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
CI: the test runner gives lldb 120 s instead of 30 s to collect stacks from debug and sanitizer servers, and resumes a server that a timed-out lldb attach left stopped.
Documentation entry for user-facing changes
CI/CD Options
Exclude tests:
Regression jobs to run:
🤖 Generated with Claude Code
https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx