Skip to content

fix(appuplink): join the new hour's relay handle when the hour turns - #1605

Merged
frahlg merged 1 commit into
masterfrom
claude/relay-rotation-gap
Oct 13, 2026
Merged

frahlg merged 1 commit into
masterfrom
claude/relay-rotation-gap

Conversation

@frahlg

@frahlg frahlg commented Oct 13, 2026

Copy link
Copy Markdown
Member

Problem

A phone that opens the app in the first minutes of an hour can wait up to five minutes before it reaches the box over the relay.

Seen on 13 Oct 2026 in the FTW Cloud C3 end-to-end test (local Core on claude/cloud-c3-core, production relay, iOS simulator). The app dialled the new hour's handle from 09:00. The box logged relay epoch rotated ... epoch=497743 at 09:03:18. For those three minutes the band said "Reconnecting to your box", and the box saw no app session. The app got ready at 09:03:23. Cold starts outside the window took 399–426 ms.

Cause

Three rules, each fine alone, leave a hole together:

  • The relay turns its epoch at its first 15-second heartbeat after the hour. From then on it refuses the old epoch with 4409 (relay/src/server.ts:445, "exact, not tolerant").
  • It closes each old room only on that room's turn in a five-minute spread, rotationOffsetMs(handle, 300_000) (server.ts:641).
  • The box moved only when the relay closed it with 4410, then waited 0–5 s.

The app picks its epoch from the wall clock on every dial, and iOS dials on every foreground. So an app opened after the relay turns, and before the box's room gets its turn, waits alone in the new room.

Fix

When its clock passes the hour, the box also joins the new hour's handle, after 0–5 s of jitter. It keeps the old handle until the relay closes that room, so a phone already talking to the box keeps it, and a phone that dials the new handle finds it.

  • A 4409 that names the old epoch means the relay has not read its clock yet. The box treats that as "not yet": it keeps the old room, tries again every 5 s, at most 4 times, and does not take it as a clock correction. After that it falls back to today's behaviour and moves on 4410.
  • Uplink.live now holds every connection that is up (one, or two in the overlap). A revoke drops sessions on both. Dead-man claims go on the newest socket; the relay counts claims per socket, so closing the old one releases nothing the new one holds.
  • No change to the app or the relay. Web, iOS and Android all gain from it.

I chose the box over the two other places it could live:

  • The relay could accept the previous epoch while the old room is still open. But the app dials the wall-clock epoch, so the app would also need a fallback to the old handle, in three clients (web, iOS, Android), and the relay has an open PR on server.ts (feat(stats): restore GitHub history and track FTW Cloud energy usage ftw-webapp#50).
  • The box could leave the old room on the hour. Then phones already in the old room would lose the box until the relay closes the room. That moves the gap; it does not remove it.

Numbers

Simulation of 400,000 app opens at random moments in the hour, with the relay's 15 s heartbeat, its 300 s spread and the box's 0–5 s jitter:

App opens that wait Mean wait when waiting Longest wait
Before 4.5 % 104 s 304 s
After 0.07 % 1.8 s 5 s

The cost: each box holds a second socket for up to five minutes each hour. While the relay lags behind the hour, it refuses on average 1.5 early tries per box per hour.

Privacy: the relay could already link the two handles by IP and by the reconnect seconds later; the relay README says rotation does not hide either. Holding both sockets for a few minutes does not add a link it lacked. The new handle now appears with every other box's in the first seconds of the hour, not alone just after the old one closes.

Tests

go/internal/appuplink/rotation_test.go adds roomRelay, a fake of server.ts with a room per handle, the exact epoch rule and per-room rotation. The test is the relay's clock and moves the box's clock by hand.

  • TestAPhoneOpenedAfterTheHourFindsTheBox: the case from 13 Oct. It fails on master ("a phone on the new hour's handle found no box until the relay rotated the old room") and passes here.
  • TestAPhoneConnectedAcrossTheHourKeepsTheBox: a session in the old room still answers after the hour; after the room's turn the phone finds the box on the new handle.
  • TestTheBoxWaitsForARelayThatHasNotTurned: a refused early try leaves the box in the old room, and it joins once the relay turns.

go test -race -count=5 ./internal/appuplink/ and make verify pass.

Follow-ups, not in this PR

  • The relay should turn its epoch when a join arrives after the hour, not on the next heartbeat. That removes the refused early tries. It belongs in srcfl/ftw-webapp relay/src/server.ts, where Add configreload watcher tests + fix Stop() double-close panic #50 is open.
  • server.ts:513 counts a deadman <id> claim each time a socket says it, but server.ts:556 releases it once per socket. A box claims again on the same socket after a subscription change, so the switch may never arm after that socket closes. I have not tested this.

🤖 Generated with Claude Code

The relay refuses the old epoch from its first heartbeat after the hour,
but closes each old room only on its turn in a five-minute spread. The
box moved only when closed, so a phone that opened the app in between
waited in an empty room for up to five minutes (seen 09:00-09:03:18 on
13 Oct 2026).

The box now joins the new hour's handle when its clock turns, after
0-5 s of jitter, and keeps the old one until the relay closes it. A 4409
naming the old epoch means the relay has not turned yet: the box keeps
the old room and tries again, at most four times.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frahlg
frahlg marked this pull request as ready for review October 13, 2026 07:54
@frahlg
frahlg merged commit 22a8f32 into master Oct 13, 2026
17 checks passed
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T08:05:13.264352Z 034be96 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 034be96dce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +171 to +172
if n := len(u.live); n > 0 {
text = u.live[n-1].text

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid reclaiming deadman rows on the surviving socket

When the relay has not advanced its epoch yet, each speculative link is immediately rejected with 4409, but its serve goroutine still launches ResyncDeadman; after the HTTP posts finish and the rejected link has been removed, this lookup selects the old surviving socket and sends every deadman <id> claim on it again. The relay increments repeated claims but releases each ID only once when that socket closes, so after rotation these stale claims can prevent an armed dead-man notification from ever firing. Bind the resync to the connection that initiated it, and only claim after that connection has been accepted, rather than retargeting it to whichever socket is newest.

AGENTS.md reference: AGENTS.md:L22-L23

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant