Skip to content

Upgrade the recommendation service's Python agent to 1.3.0 to stop its gunicorn workers deadlocking - #316

Merged
wankai123 merged 1 commit into
mainfrom
fix/recommendation-python-agent-fork-deadlock
Nov 20, 2026
Merged

wankai123 merged 1 commit into
mainfrom
fix/recommendation-python-agent-fork-deadlock

Conversation

@wankai123

Copy link
Copy Markdown
Member

Problem

On the live demo the recommendation service never answers a request, and the node keeps going into DiskPressure, evicting the other pods.

  • Agent 1.2.0 under sw-python run -p starts a full agent, gRPC channel included, in the gunicorn master.
  • Workers forked while the master's gRPC threads are busy (fork_posix.cc:75] Other threads are currently calling into gRPC, skipping fork() handlers) deadlock before they boot. Gunicorn aborts each one after 30s (WORKER TIMEOUT, code 134), and the replacement hangs the same way.
  • Every abort writes a ~200 MB core.gunicorn.* dump into the container's root. One pod reached ~52 GiB, which put the node into DiskPressure and left 156 Evicted pods.

This started with #308, which made the agent actually start (before it, the agent failed on No module named 'skywalking.protocol', so the master had no gRPC threads).

Fix

Install agent 1.3.0, which includes apache/skywalking-python#409: the master is only instrumented, and each worker opens its own gRPC channel after fork. 1.3.0 requires grpcio >= 1.83 (1.85.3 gets installed).

It is installed from the sdist (--no-binary) because of two packaging problems in the 1.3.0 release:

  • The PyPI 1.3.0 wheel has no skywalking/protocol files (the 1.2.0 wheel has 73; the sdist has them).
  • The apache/skywalking-python:1.3.0-* images still contain the 1.2.0 wheel.

Verification (local, docker)

Old and new images side by side, against a working gRPC collector and against one that accepts TCP but never answers:

Working collector Collector that never answers
1.2.0 (before) healthy skipping fork() handlers, 2 worker timeouts, /health failing at first
1.3.0 (after) healthy no timeouts, /health 200 throughout

With 1.3.0 both workers register as agent::recommendation, and the collector receives instance properties, keep-alives, profile queries, trace segments and meters. /rcmd itself wasn't exercised locally, since it needs songs and rating.

…s gunicorn workers deadlocking

Agent 1.2.0 started under `sw-python run -p` runs a full agent, gRPC channel included, in the gunicorn
master. A worker forked while the master's gRPC threads are busy deadlocks before it boots, gunicorn aborts it
after 30s (WORKER TIMEOUT, code 134), and the replacement hangs the same way, so the service never answers.
Each abort also writes a ~200 MB core dump into the container, which filled the showcase node's disk and
evicted the other pods (DiskPressure).

Agent 1.3.0 (apache/skywalking-python#409) only instruments the master and opens the gRPC channel in each
worker after fork. It is built from the sdist because the PyPI 1.3.0 wheel lacks the generated
skywalking.protocol package, and the 1.3.0 image still ships the 1.2.0 wheel.
@wankai123
wankai123 merged commit a151753 into main Nov 20, 2026
@wankai123
wankai123 deleted the fix/recommendation-python-agent-fork-deadlock branch November 20, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant