Skip to content

Add a network watchdog that recovers a WiFi link that stopped carrying traffic - #1282

Merged
gskjold merged 1 commit into
mainfrom
network-watchdog
Oct 17, 2026
Merged

gskjold merged 1 commit into
mainfrom
network-watchdog

Conversation

@gskjold

@gskjold gskjold commented Oct 16, 2026

Copy link
Copy Markdown
Member

Why

Two HAN-powered Pow-U+ units (ESP32-S2, v2.5.7) at the same site went silent while still running. One had a long uptime but was unreachable on the LAN and sent nothing to the cloud. It came back the moment it was physically moved 50 cm. The second went silent a few days later in what looks like the same state.

The firmware only trusts WiFi.status(). The driver keeps reporting WL_CONNECTED as long as it hears the AP, even when none of the device's own frames get through, which happens on a marginal link with 802.11b disabled. The only existing recovery is the MQTT auto-reboot, and MQTT wasn't configured on these units, so nothing ever acted.

What it does

  • Probe: ping the gateway once a minute (3 echo requests, one reply is enough). It's a local check, so networks with no internet access are unaffected.
  • Escalate when the gateway stops answering:
    • restart WiFi after 3 minutes, then again at 6, 12, 24 and 48 minutes without a reply;
    • a restart is a full disconnect(), so the next connect does an all-channel scan and skips the pinned last-AP retry from Reduce WiFi connect current draw on bus powered devices #1272, because the current AP is the suspect;
    • on ESP32 a watchdog restart also allows 802.11b until the next reboot (WiFiClientConnectionHandler::allow11b()), since those rates are more robust on a marginal link;
    • reboot after 60 minutes, saving data storage first, with the new REBOOT_CAUSE_NETWORK_UNREACHABLE (14).
  • Safety:
    • it only arms once the gateway has answered after boot, so a gateway that blocks ping, or a network that never came up, can't cause restarts or reboot loops;
    • after a watchdog reboot it has to arm again before it can act;
    • it only runs in WiFi client mode, and is paused during firmware upgrades.
  • Diagnostics are kept in ResetDataContainer (no-init RAM, so they survive software reboots on ESP32): watchdog restarts, the longest outage, and the AP, channel and RSSI when the gateway first stopped answering. They're exposed in sysinfo.json next to the current AP and channel:
    "wifi": { "bssid": "…", "ch": 6, "wd": { "r": 0, "lo": 0, "fb": "…", "fc": 0, "fr": 0 } }

Structure

  • NetworkWatchdog: the policy only (timing, arming, escalation, stats). No Arduino or ping code; unit tested on native.
  • GatewayPing: a non-blocking ping using ping_sock on ESP32 and the SDK ping_start() on ESP8266.
  • handleNetworkWatchdog() in the main sketch wires the two together and does the logging and the actions.

Notes for review

  • ResetDataContainer gained fields after magic, and the magic is bumped from 0x4a to 0x4b so leftovers from older firmware are cleared. As a result, the first boot after upgrading reports no previous reboot cause.
  • The 802.11b fallback is ESP32 only. On ESP8266 the rate limits are set through the SDK in connect(), and I haven't verified how to undo them safely.
  • Ethernet is not covered.
  • If the AP is down for a long time, an armed device now reboots once an hour after it was last reachable; before, it retried forever without rebooting.
  • ESP8266 flash goes from 98.0% to 98.4% (+4.1 KB); about 16 KB of headroom remains.
  • The UI shows ex_cause as a raw number, so the new cause 14 needs no UI or translation change. sysinfo.json gained a field, which may matter for the public API docs.

Testing

  • pio test -e native_watchdog: 7 tests, covering never arming, the probe interval, a healthy network, the 3/6/12/24/48 → 60 min schedule, the reset on recovery, and outage recording. It's added to both CI workflows.
  • pio test -e native and native_config still pass.
  • pio run for esp32s2, esp32 and esp8266 builds.
  • On device: pull the AP's uplink, or block the device's MAC at the AP while it stays associated. Expect "Gateway did not answer ping" each minute, a WiFi restart at 3 min, and wifi.wd in sysinfo.json filling in.
  • On device: a gateway that blocks ICMP never arms the watchdog (no "network watchdog is active" in the log, and no restarts).
  • On device, ESP32: after a watchdog restart, 802.11b rates are in use (negotiated phy mode).
  • Leave it running for an hour unreachable: it reboots, and sysinfo shows ex_cause 14 with the stats preserved.

🤖 Generated with Claude Code

…g traffic

A device can stay associated to the AP (WiFi.status() == WL_CONNECTED)
while nothing it sends gets through, for instance on a marginal link
where beacons are still heard but data frames are not. Nothing noticed
this state, so a device could stay unreachable for days with a long
uptime, until it was physically moved.

Ping the gateway once a minute (3 echo requests). When it stops
answering, restart WiFi with a full scan after 3 minutes, then with a
doubling backoff (6, 12, 24, 48 minutes), and reboot after an hour
with the new reboot cause REBOOT_CAUSE_NETWORK_UNREACHABLE. On ESP32 a
watchdog restart also allows 802.11b rates until the next reboot,
which are more robust on a marginal link.

The watchdog is only armed once the gateway has answered after boot,
so a gateway that blocks ping or a network that never comes up can not
cause restarts or reboot loops. It only runs in WiFi client mode and
is paused during firmware upgrades.

The number of restarts, the longest outage, and the AP, channel and
RSSI when the gateway stopped answering are kept in the reset data
container (survives software reboots on ESP32) and exposed in
sysinfo.json together with the current AP and channel.

The policy is a plain class tested on native (pio test -e
native_watchdog); the ping is a thin wrapper around ping_sock on ESP32
and the SDK ping on ESP8266.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🔧 PR Build Artifacts

Version: 37b89c8

All environments built successfully. Download the zip files:

Artifacts expire after 7 days. View workflow run

@ArnieO

ArnieO commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Flott forbedring! Regner med dette er hos Goman.

@gskjold
gskjold merged commit d25d6c1 into main Oct 17, 2026
8 checks passed
@gskjold
gskjold deleted the network-watchdog branch October 17, 2026 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants