Repository navigation
Upgrade the recommendation service's Python agent to 1.3.0 to stop its gunicorn workers deadlocking - #316
Merged
Conversation
…s gunicorn workers deadlocking Agent 1.2.0 started under `sw-python run -p` runs a full agent, gRPC channel included, in the gunicorn master. A worker forked while the master's gRPC threads are busy deadlocks before it boots, gunicorn aborts it after 30s (WORKER TIMEOUT, code 134), and the replacement hangs the same way, so the service never answers. Each abort also writes a ~200 MB core dump into the container, which filled the showcase node's disk and evicted the other pods (DiskPressure). Agent 1.3.0 (apache/skywalking-python#409) only instruments the master and opens the gRPC channel in each worker after fork. It is built from the sdist because the PyPI 1.3.0 wheel lacks the generated skywalking.protocol package, and the 1.3.0 image still ships the 1.2.0 wheel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the live demo the
recommendationservice never answers a request, and the node keeps going into DiskPressure, evicting the other pods.sw-python run -pstarts a full agent, gRPC channel included, in the gunicorn master.fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers) deadlock before they boot. Gunicorn aborts each one after 30s (WORKER TIMEOUT,code 134), and the replacement hangs the same way.core.gunicorn.*dump into the container's root. One pod reached ~52 GiB, which put the node into DiskPressure and left 156 Evicted pods.This started with #308, which made the agent actually start (before it, the agent failed on
No module named 'skywalking.protocol', so the master had no gRPC threads).Fix
Install agent 1.3.0, which includes apache/skywalking-python#409: the master is only instrumented, and each worker opens its own gRPC channel after fork. 1.3.0 requires grpcio >= 1.83 (1.85.3 gets installed).
It is installed from the sdist (
--no-binary) because of two packaging problems in the 1.3.0 release:skywalking/protocolfiles (the 1.2.0 wheel has 73; the sdist has them).apache/skywalking-python:1.3.0-*images still contain the 1.2.0 wheel.Verification (local, docker)
Old and new images side by side, against a working gRPC collector and against one that accepts TCP but never answers:
skipping fork() handlers, 2 worker timeouts,/healthfailing at first/health200 throughoutWith 1.3.0 both workers register as
agent::recommendation, and the collector receives instance properties, keep-alives, profile queries, trace segments and meters./rcmditself wasn't exercised locally, since it needssongsandrating.