Skip to content

RAD-16862 Stop stale Android wakes from starving the upload queue - #57

Merged
arnonate merged 1 commit into
masterfrom
RAD-16862-stale-wakes
Oct 28, 2026
Merged

arnonate merged 1 commit into
masterfrom
RAD-16862-stale-wakes

Conversation

@elliottkember

@elliottkember elliottkember commented Oct 28, 2026 •

Copy link
Copy Markdown

Summary

Android uploads can stop for good. A request that gets a 401 parks, and the library schedules a wake at its expiry. Diana's requests expire in 10 years. A fresh token unparks the request, but the wake stays.

Each delayed WorkManager row holds one of the app's 20 JobScheduler slots until it runs. WorkManager's in-process scheduler reads only the 200 oldest enqueued rows. Enough stale wakes fill both, and then no queued work starts. This matches RAD-16862: 534 entries, all queued, with no activity for hours on a working network.

Before: a parked entry always gets a wake at its expiry, and the wake stays after the entry unparks.

After: a parked entry gets a wake only when it expires within 24 hours. When an entry is scheduled again, the library cancels any wake it no longer needs. The boot sweep does this too, so phones that are stuck now recover on the first launch after the update.

Evidence from the stuck phone

  • The queue export shows all 352 session rows moving to queued at the same millisecond. That was Oct 21, 09:29:04.333 UTC. After that, no row was attempted again. Rows added the next day were never attempted at all.
  • Sentry shows a 401 on the user-data call 4 seconds earlier. The app then tried a token refresh.
  • The app process had run since 05:54 that day, so the boot sweep did not make that change. In a running process, only updateHeaders() moves rows that are not paused back to queued with one timestamp. It does that only to parked rows. So the rows were parked on a 401, each with a wake 10 years ahead.
  • No log records a 401 on an upload request itself. The queue's attempt log on the phone was empty.

What to look at

  • QueueController.scheduleRun(): the two places that cancel a stale wake.
  • WorkManagerScheduler.cancelWake(): it checks for an unfinished wake before it cancels.

Decisions and trade-offs

  • 24-hour horizon. A parked entry with a far expiry has no wake. It settles expired at the first launch after its expiry, not exactly on time.
  • No cancel while the entry's worker runs. That worker can be the wake itself.
  • cancelWake() reads WorkManager first. A WorkManager cancel reschedules all pending work even when the name has no rows. The boot sweep calls this for every queued entry.
  • Backoff wakes from release() are unchanged. They run within 2 hours, so they slow the queue but do not stop it.

Test Plan

  • ./gradlew :react-native-background-upload:testDebugUnitTest from example/RNBGUExample/android: 311 tests, 0 failures.
  • Four of the new tests fail when the source changes are removed. They cover the sweep cleanup, the cancel on unpark, and park with a far expiry. A fifth test checks that a running worker's wake stays.
  • Pre-commit hooks: tsc and jest (190 tests) pass.

What's required for testing (prerequisites)?

An Android phone on 11.0.1 with a few hundred queued uploads that parked on a 401.

What are the steps to reproduce (after prerequisites)?

  1. Refresh the auth token so the entries unpark.
  2. Expected on 11.0.1: the queue does not start.
  3. Install this branch and launch the app. Expected: the queue drains.

Compatibility

OS Implemented
iOS ❌
Android ✅

iOS uses URLSession, not WorkManager, so this problem does not apply there.

Checklist

  • I have tested this on a device and a simulator
  • I added the documentation in README.md
  • I updated the typed files (TS and Flow)
  • I've added Detox End-to-End Test(s)
  • I've created a snack to demonstrate the changes: LINK HERE

🤖 Generated with Claude Code

A parked entry got a wake at its expiry, which is years ahead for a
long-lived request. Unparking it left that wake in place. Each delayed
WorkManager row holds a JobScheduler slot, and the in-process scheduler
reads only the 200 oldest enqueued rows. Enough stale wakes filled both,
so no queued work started.

A parked entry now gets an expiry wake only when the expiry is within
24 hours. scheduleRun() cancels a wake that is no longer needed, so the
boot sweep frees the slots on phones that are already stuck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@arnonate arnonate left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😵

@arnonate
arnonate marked this pull request as ready for review October 28, 2026 16:55
@arnonate
arnonate merged commit d1ae136 into master Oct 28, 2026
5 checks passed
@arnonate arnonate mentioned this pull request Oct 28, 2026
arnonate added a commit that referenced this pull request Oct 28, 2026
Jira: [RAD-16862](https://openspaceai.atlassian.net/browse/RAD-16862)

Bumps the version to 11.0.2 and adds the CHANGELOG note for
#57, which had none. Master
is protected, so the release commit needs a PR this time.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

[RAD-16862]:
https://openspaceai.atlassian.net/browse/RAD-16862?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
arnonate added a commit that referenced this pull request Oct 29, 2026
…nd a boot snapshot of WorkManager runs (#58)

Jira: [RAD-17288](https://openspaceai.atlassian.net/browse/RAD-17288),
follow-up to
[RAD-16862](https://openspaceai.atlassian.net/browse/RAD-16862)

On Android, the library's warnings and errors only reach logcat. When a
customer's queue stalled for six days, support's exported logs couldn't
show why. A stuck phone gave us no WorkManager state, and we had to
infer stale wakes from timestamps.

This adds `takeDiagnostics()`. Android writes each warning and error to
a small file, keeping the newest 50. That includes runs in the
background with no JS. Each time the module starts, which is once per
launch plus once per JS reload, it also writes one `info` record. It
describes the entries the boot sweep schedules, as the sweep found them,
and every unfinished WorkManager run of the library before the sweep
cleans up. That covers how many runs there are by state, how many are
delayed wakes, how many of those are due more than 24 hours out, the
highest run attempt count, and how many queued or running entries had no
run at all. JS reads and clears the file with `takeDiagnostics()`. iOS
returns an empty list for now.

## What to look at

- `DiagnosticLog.kt`: the file format, the cap, and how failures are
swallowed.
- `QueueController.sweep()`: the snapshot is read before #57's wake
cleanup, so it shows what the earlier process left. It keeps each entry
as found, so an entry that was running when its process died still
counts as running with its old `updatedAt`.

## Decisions and trade-offs

- **A file, not a live event.** A headless worker has no JS, and the
boot sweep runs before JS subscribes, so a live event would lose both.
- **One WorkManager query per launch.** `getWorkInfosByTag(WORK_TAG)`
reads every library row at once. A query per entry would run hundreds of
times on the queue's only thread.
- **Wakes are told apart by their initial delay.** WorkInfo doesn't
expose the unique work name, and wakes and main chains share their tags.
A wake due already has no delay and counts as a main-chain run.
- **Drain on read.** Each call clears the log, so a consumer that fails
to report the records loses them. This keeps the API to one call.
- **The expected "no event emitter" warning stays in logcat only.** It
fires once per dropped event, so persisting it could push the snapshot
and real errors out of the 50-record cap.
- **A snapshot on every module start with queued entries, even healthy
ones.** The consumer decides what is worth reporting, for example when
`delayedPastHorizon` is above 0 or `oldestUpdatedAt` is old.

## Test Plan

- `./gradlew :react-native-background-upload:testDebugUnitTest`: 322
tests, 0 failures. New tests cover the log's cap, drain, corrupt-file
handling, and a fresh instance reading what the last one left, Diag
routing, and the snapshot's fields, including stale wakes with no entry
left.
- `yarn typecheck`, `yarn test` (192 pass) and `yarn lint:ci` pass.
- I didn't build the iOS example app. The iOS change is a stub that
resolves an empty list.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


[RAD-16862]:
https://openspaceai.atlassian.net/browse/RAD-16862?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ



[RAD-17288]:
https://openspaceai.atlassian.net/browse/RAD-17288?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants