Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/SERVER_LOCK_AUDIT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Server responsiveness under contention

The November 2026 live-process sample showed the ASGI main thread waiting for a Python lock while background workers continued. Two independent health requests timed out. Native sampling could not identify the precise Python lock owner; attaching a Python stack reader required unavailable elevated access. This audit fixes the independently reproducible blocking paths rather than assuming one lock explains every incident.

## Audited boundaries

| Area | Blocking risk | Treatment |
| --- | --- | --- |
| Repository setup, fork, push and sync | Job reservations acquired a thread RLock on the event loop; sync validation could enter progress import and database locks | Whole reservations run on one control worker, retaining atomic duplicate-job checks. Validation reads a progress snapshot; the job performs synchronization. |
| Import, restore, SDK/tool/agent installs and fork setup | `Jobs.start`, admission, logging and executor setup could wait synchronously | Job admission runs on reserved control workers. Actual work keeps its own job thread and progress/cancellation. |
| Repair replay/recovery | Duplicate checks and job creation held the job lock on the event loop | Each atomic reservation stays on a single worker. |
| Job cancellation and settings job discovery | Used the same generic worker capacity as long operations, or acquired a lock directly | Independent control capacity, separate from builds and UI snapshots. Large job JSON is encoded off the event loop. |
| Progress activity | Even an in-memory status call acquired the activity mutex | Activity status is read on a worker. Project GET routes retain existing snapshot-only contexts. |
| Browser bridge and login/OAuth | Capability/session locks and settings file locks could block HTTP dispatch | Locking operations are offloaded. Cookie authorization reads the current bounded session map without waiting for a writer, and still checks expiry. OAuth state consumption remains atomic. |
| Agent refresh, prompt previews, tool removal, background task admission | Synchronous helper calls could reach mutexes, subprocess or disk work | Offload at the request boundary. |
| Database/process writer gates | Waiting can be legitimate during transactions and must not be bypassed | Preserve writer ordering, transaction scopes, rollback and verification. Keep these operations off the event loop; a regression holds a real writer while serving health. |
| Read caches and map snapshots | Cache builders/consumers can wait for another worker; recursive snapshot waits could wait on themselves forever | Existing transaction cache bypass remains. Snapshot consumers have a bounded wait; recursive builders fail immediately. Successful builders remain valid after a consumer timeout. |
| Diagnostics | The stall logger itself could wait for a logging lock on the event loop; a completely blocked loop could not report itself | Log from workers. An independent heartbeat watchdog captures bounded thread code locations during a persistent stall. |
| Room host and MCP dispatch | Database, claim, compiler and member/auth operations | Checked the existing worker boundaries; synchronous room business logic and MCP tool/auth/discovery work already run on workers. Shared UI room admission now uses the control workers too. |
| Shutdown | Waiting for open requests can precede cleanup | Existing immediate signal-driven bounded shutdown remains. Do not force-release locks or commit a half-finished transaction. |

## Diagnostics and regression gates

A development or packaged runtime writes `.decomp/logs/event-loop-stacks.log` if its event loop misses heartbeats for five seconds. One bounded report is replaced at most every 30 seconds. It contains thread names and Python file/function/line locations, not locals, arguments, source lines, SQL bindings or request URLs. The watchdog stops with the server lifespan.

Tests use one shared ASGI event loop, deliberately hold backend admission, job reservation, progress and database writer paths, and require concurrent `/health` responses. The database test checks that the delayed settings write commits and SQLite integrity remains valid. Separate tests cover reserved capacity, authorization expiry, recursive/bounded cache waits, watchdog capture and an AST gate against direct blocking lock/job/database admissions in asynchronous routes.

Hardware exhaustion, the interpreter or a native extension can still prevent scheduling. This does not promise that every possible future hang is impossible. These boundaries prevent the reproduced backend lock waits from freezing the server and make another stall diagnosable without an elevated debugger.
2 changes: 1 addition & 1 deletion src/decompnet/app/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ def _start(args,home,internal,state_file,log,restarted,headless):
mcp, manager, security, quit_app = build(net, internal, port, secret, **({"headless":True} if headless else {}))
app = mcp.streamable_http_app(transport_security=security, max_request_body_size=REQUEST_BODY_LIMIT, host="127.0.0.1")
from ..ui_diagnostics import Diagnostics
app = Diagnostics(app)
app = Diagnostics(app, stack_path=internal / "logs" / "event-loop-stacks.log")
# with a tray icon, helpers kept working without the window have a way back to it
window.IN_TRAY = False if headless else tray.start(window.open_app, quit_app)
print(f"{edition.NAME} {__import__('decompnet').__version__}\nyour projects: {hostenv.shown_path(home)}"
Expand Down
5 changes: 5 additions & 0 deletions src/decompnet/app/routes/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ class App:
fail: Callable[..., Any] # fail(error, code=400): the error as JSON
ui_call: Callable[..., Any] | None = None # reserved capacity for snapshot-only UI reads
fast_call: Callable[..., Any] | None = None # lightweight counters, isolated from diff/map work
control_call: Callable[..., Any] | None = None # job admission/cancellation cannot queue behind builds
updater: updates.Updater | None = None
busy: Callable[[], bool] | None = None # whether helpers or a job are working
quit_app: Callable[[], None] | None = None # quit: the window, every helper, the server
Expand All @@ -58,6 +59,10 @@ class App:
autopilot: Callable[[str], None] | None = None # state: start autopilot on a project
with_project: Callable[..., Any] | None = None # projects: a project route from fn(slug, request, data)

async def control(self, fn, *args):
"""Short locking operations use reserved workers, never the ASGI thread."""
return await (self.control_call or self.call)(fn, *args)

async def read(self, fn, *args):
"""Read current state in a worker without waiting for saved-progress verification."""
def snapshot():
Expand Down
12 changes: 6 additions & 6 deletions src/decompnet/app/routes/browser_bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,15 @@ async def authorize(request):
body = await data(request)
slug = str(body.get('slug') or '')
if body.get('action') == 'revoke':
return JSONResponse(bridge.revoke(slug))
return JSONResponse(await app.control(lambda: bridge.revoke(slug)))
return JSONResponse(await app.call(bridge.create_code, slug, str(body.get('origin') or '')))
except (ValueError, NetworkError, OSError) as error:
return app.fail(error)

@app.mcp.custom_route('/api/browser-bridge/{action}', methods=['POST', 'OPTIONS'])
async def browser(request):
origin = request.headers.get('origin', '')
if request.headers.get('host') not in app.allowed_hosts or not bridge.allowed(origin):
if request.headers.get('host') not in app.allowed_hosts or not await app.control(lambda: bridge.allowed(origin)):
return JSONResponse({'error':'This browser origin has not been paired in the desktop app.'}, status_code=403)
headers = {'Access-Control-Allow-Origin':origin, 'Vary':'Origin', 'Cache-Control':'no-store',
'Access-Control-Allow-Methods':'POST, OPTIONS', 'Access-Control-Allow-Headers':'Authorization, Content-Type',
Expand All @@ -53,11 +53,11 @@ async def browser(request):
body = await data(request)
action = request.path_params['action']
if action == 'pair':
return JSONResponse(bridge.pair(origin, body.get('code', '')), headers=headers)
return JSONResponse(await app.control(lambda: bridge.pair(origin, body.get('code', ''))), headers=headers)
auth = request.headers.get('authorization', '')
if not auth.startswith('Bearer '):
raise PermissionError('Pair this browser first.')
key, session = bridge.auth(origin, auth[7:])
key, session = await app.control(lambda: bridge.auth(origin, auth[7:]))
if action == 'functions':
result = await app.call(bridge.listing, session, body.get('search', ''))
elif action == 'task':
Expand All @@ -73,9 +73,9 @@ async def browser(request):
variant_id=body.get('variantId')
if variant_id is not None and (not isinstance(variant_id, str) or len(variant_id)>64):
raise ValueError('Choose a host compiler variation.')
task = bridge.consume(key, session, str(body.get('taskId') or ''))
task = await app.control(lambda: bridge.consume(key, session, str(body.get('taskId') or '')))
try:
jid = app.jobs.start('browser-check', lambda log: bridge.submit(key, session, task, source, variant_id))
jid = await app.control(lambda: app.jobs.start('browser-check', lambda log: bridge.submit(key, session, task, source, variant_id)))
except Exception:
session['busy'] = False
raise
Expand Down
22 changes: 11 additions & 11 deletions src/decompnet/app/routes/forks.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ def work(log):
settings.data["last_project"] = slug
settings.save()
return {"slug": slug}
return JSONResponse({"job": jobs.start("fork", work)})
return JSONResponse({"job": await app.control(lambda: jobs.start("fork", work))})

@mcp.custom_route("/api/fork/local/scan", methods=["POST"])
async def api_local_scan(request: Request) -> Response:
Expand All @@ -61,7 +61,7 @@ def work(log):
result = net.forks.scan_local(hostenv.local_path(folder))
log(f"Found {len(result['projects'])} repositories.")
return result
return JSONResponse({"job": jobs.start("local-scan", work)})
return JSONResponse({"job": await app.control(lambda: jobs.start("local-scan", work))})

@mcp.custom_route("/api/fork/local/import", methods=["POST"])
async def api_local_import(request: Request) -> Response:
Expand All @@ -75,7 +75,7 @@ async def api_local_import(request: Request) -> Response:
return fail(ValueError("Choose the folder and at least one repository from its scan."))
def work(log):
return net.forks.import_local(hostenv.local_path(folder), paths, copies=data.get("copies") is not False, log=log)
return JSONResponse({"job": jobs.start("local-import", work)})
return JSONResponse({"job": await app.control(lambda: jobs.start("local-import", work))})

def fork_of(slug: str):
if not net.fork_slug(slug):
Expand Down Expand Up @@ -118,7 +118,7 @@ def work(log):
settings.save()
autopilot(active)
return res
return JSONResponse({"job": jobs.start("fork-setup", work)})
return JSONResponse({"job": await app.control(lambda: jobs.start("fork-setup", work))})

@mcp.custom_route("/api/fork/find-rom", methods=["POST"])
async def api_fork_find_rom(request: Request) -> Response:
Expand All @@ -135,7 +135,7 @@ def work(log):

_, project, _ = net.forks.get(slug)
return find(project, net.forks.needs(slug), str(settings.data.get("rom_library") or ""), log)
return JSONResponse({"job": jobs.start("find-rom", work)})
return JSONResponse({"job": await app.control(lambda: jobs.start("find-rom", work))})

@mcp.custom_route("/api/fork/setup", methods=["POST"])
async def api_fork_setup(request: Request) -> Response:
Expand Down Expand Up @@ -163,7 +163,7 @@ def work(log):
settings.save()
autopilot(active)
return res
return JSONResponse({"job": jobs.start("fork-setup", work)})
return JSONResponse({"job": await app.control(lambda: jobs.start("fork-setup", work))})

@mcp.custom_route("/api/fork/check", methods=["POST"])
async def api_fork_check(request: Request) -> Response:
Expand All @@ -176,7 +176,7 @@ async def api_fork_check(request: Request) -> Response:
await app.read(fork_of, slug)
except NetworkError as e:
return fail(e)
return JSONResponse({"job": jobs.start("fork-check", lambda log: net.forks.check(slug, log=log, build=True))})
return JSONResponse({"job": await app.control(lambda: jobs.start("fork-check", lambda log: net.forks.check(slug, log=log, build=True)))})

@mcp.custom_route("/api/fork/fix-setup", methods=["GET", "POST"])
async def api_fork_fix_setup(request: Request) -> Response:
Expand All @@ -194,8 +194,8 @@ async def api_fork_fix_setup(request: Request) -> Response:
folder = Path(proj["root"])
except (NetworkError, ValueError) as e:
return fail(e)
return JSONResponse({"job": jobs.start("fix-setup", lambda log: setup_fix.launch(
app.home, folder, str(data.get("error") or ""), kind, log))})
return JSONResponse({"job": await app.control(lambda: jobs.start("fix-setup", lambda log: setup_fix.launch(
app.home, folder, str(data.get("error") or ""), kind, log)))})

@mcp.custom_route("/api/fork/rom-check", methods=["POST"])
async def api_fork_rom_check(request: Request) -> Response:
Expand All @@ -207,7 +207,7 @@ async def api_fork_rom_check(request: Request) -> Response:
await app.read(fork_of, slug)
except NetworkError as e:
return fail(e)
return JSONResponse({"job": jobs.start("rom-check", lambda log: net.forks.check_rom(slug, log=log))})
return JSONResponse({"job": await app.control(lambda: jobs.start("rom-check", lambda log: net.forks.check_rom(slug, log=log)))})

@mcp.custom_route("/api/fork/start-files", methods=["POST"])
async def api_fork_start_files(request: Request) -> Response:
Expand All @@ -219,6 +219,6 @@ async def api_fork_start_files(request: Request) -> Response:
await app.read(fork_of, slug)
except NetworkError as e:
return fail(e)
return JSONResponse({"job": jobs.start("fork-start-files", lambda log: net.forks.start_files(slug, log=log))})
return JSONResponse({"job": await app.control(lambda: jobs.start("fork-start-files", lambda log: net.forks.start_files(slug, log=log)))})

mcp.custom_route("/api/fork/needs", methods=["GET"])(with_project(lambda s, r, d: net.forks.needs(fork_of(s)["slug"])))
4 changes: 2 additions & 2 deletions src/decompnet/app/routes/life.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,8 @@ async def api_log(request: Request) -> Response:
if (bad := guard(request, write=True)):
return bad
data = await body(request)
page_log.warning("page: %s (%s)\n%s", str(data.get("message", ""))[:PAGE_ERROR_CHARS],
str(data.get("where", ""))[:PAGE_ERROR_CHARS], str(data.get("stack", ""))[:PAGE_STACK_CHARS])
await app.control(lambda: page_log.warning("page: %s (%s)\n%s", str(data.get("message", ""))[:PAGE_ERROR_CHARS],
str(data.get("where", ""))[:PAGE_ERROR_CHARS], str(data.get("stack", ""))[:PAGE_STACK_CHARS]))
return JSONResponse({"ok": True})

@mcp.custom_route("/api/quit", methods=["POST"])
Expand Down
2 changes: 1 addition & 1 deletion src/decompnet/app/routes/projects.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ def respond():
path = request.url.path
if request.method == 'GET' and elapsed >= 0.5 and started - slow_requests.get(path, float('-inf')) >= 60:
slow_requests[path] = started
jobs.logger.warning('Slow UI request %s %.3fs (includes worker queue and response encoding)', path, elapsed)
await app.control(lambda: jobs.logger.warning('Slow UI request %s %.3fs (includes worker queue and response encoding)', path, elapsed))
return handler

def g_boundary_evidence(slug, request, data):
Expand Down
30 changes: 17 additions & 13 deletions src/decompnet/app/routes/repair_feedback.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,14 @@ def select_examples():
ids = await app.read(select_examples)

if not ids:raise NetworkError('No examples match these filters.')
with app.jobs._lock:
if any(j['kind']=='repair-replay' and j['state']=='running' for j in app.jobs.jobs.values()):
raise NetworkError('A repair replay is already running. Stop it or wait before starting another.')
job=app.jobs.start('repair-replay',lambda log:repair_replay_jobs.run(feedback,slug,ids,trials,seconds,log,minimize=minimize,diagnostics=diagnostics,combinations=combinations),
cancel_message='Repair replay stopped. No game progress or review states changed.')
return JSONResponse({'job':job})
def start_replay():
with app.jobs._lock:
if any(j['kind']=='repair-replay' and j['state']=='running' for j in app.jobs.jobs.values()):
raise NetworkError('A repair replay is already running. Stop it or wait before starting another.')
job=app.jobs.start('repair-replay',lambda log:repair_replay_jobs.run(feedback,slug,ids,trials,seconds,log,minimize=minimize,diagnostics=diagnostics,combinations=combinations),
cancel_message='Repair replay stopped. No game progress or review states changed.')
return JSONResponse({'job':job})
return await app.control(start_replay)
except (NetworkError,ValueError,KeyError,TypeError) as error:
return app.fail(error)

Expand Down Expand Up @@ -140,12 +142,14 @@ async def recovery(request):
retry_failed = data.get('retry_failed', False)
if not isinstance(retry_failed, bool):
raise NetworkError('Retry failed measurements must be true or false.')
with app.jobs._lock:
if any(j['kind'] == 'candidate-recovery' and j['state'] == 'running' for j in app.jobs.jobs.values()):
raise NetworkError('A recovery batch is running. Stop it or wait before starting another.')
job = app.jobs.start('candidate-recovery', lambda log: candidate_rechecks.run(app.net, slug, limit, log, retry_failed),
cancel_message='Recovery stopped. Finished measurements are saved; selected C is unchanged.')
return JSONResponse({'job':job})
def start_recovery():
with app.jobs._lock:
if any(j['kind'] == 'candidate-recovery' and j['state'] == 'running' for j in app.jobs.jobs.values()):
raise NetworkError('A recovery batch is running. Stop it or wait before starting another.')
job = app.jobs.start('candidate-recovery', lambda log: candidate_rechecks.run(app.net, slug, limit, log, retry_failed),
cancel_message='Recovery stopped. Finished measurements are saved; selected C is unchanged.')
return JSONResponse({'job':job})
return await app.control(start_recovery)
except (NetworkError, ValueError, KeyError, TypeError) as error:
return app.fail(error)

Expand Down Expand Up @@ -198,7 +202,7 @@ async def investigate(request):
kind = str(data.get('agent') or '')
if not any(a['id'] == kind and a['installed'] for a in await app.call(repair_investigations.choices)):
raise NetworkError('Choose an installed coding CLI for this investigation.')
job = app.jobs.start('repair investigation', lambda log: repair_investigations.launch(app.home, example, kind, log))
job = await app.control(lambda: app.jobs.start('repair investigation', lambda log: repair_investigations.launch(app.home, example, kind, log)))
return JSONResponse({'job': job})
except (NetworkError, ValueError, KeyError, TypeError) as error:
return app.fail(error)
Expand Down
Loading
Loading