[waddell]

#33 (Rhys Williams) fixed the HTTP 406 on the main Overpass request by sending a User-Agent and switching to https. This follows up on the rest of that code path.

What this changes

• get_pause_duration() still called http://overpass-api.de/api/status with no User-Agent, so it got the 406 page and always fell back to 10 s. It now uses https and the same User-Agent.
• The User-Agent is now a module constant, osmnet/<version> (+https://github.com/UDST/osmnet), built from osmnet.__version__ and used by both requests. To make that importable, __version__ is now set in osmnet/__init__.py before from .load import *.
• The status parser read line index 3, but the page now has Rate limit and Announced endpoint lines, so that index no longer holds the slot line. It now finds the line by content: N slots available, then Slot available after: <time> (earliest one), then Currently running queries.
• Retries are capped. overpass_request() used to recurse forever on 429/504; it now loops with a new max_retries=5 keyword and raises OverpassRetryError when the retries run out. get_pause_duration() used to recurse forever while the server was busy; it now takes max_retries=5 and falls back to default_duration. Existing call signatures are unchanged. One side effect: an explicit error_pause_duration now applies to every retry, not just the first. The status request also gets a 30 s timeout.

Live /api/status page fetched on 2026-10-04, also saved as a test fixture:

Connected as: 843828827
Current time: 2026-10-04T12:18:14Z
Announced endpoint: gall.openstreetmap.de/
Rate limit: 2
2 slots available now.
Currently running queries (pid, space limit, time limit, start time):

Tests and CI

• New osmnet/tests/test_overpass.py (17 tests, requests mocked, no network). It covers the live layout, the old layout, Slot available after, Currently running, the 406 HTML page, the User-Agent on both requests, and retry exhaustion. 15 of the 17 fail against current dev.
• The tests in test_load.py that query Overpass are skipped when OSMNET_SKIP_NETWORK_TESTS=1. There was no existing marker. Local runs are unchanged unless the variable is set.
• New .github/workflows/ci.yml runs the offline tests on Python 3.10–3.13 (ubuntu-latest) on push and pull_request to dev. CI has effectively been dead: .travis.yml and appveyor.yml are still in the repo but no longer run. I left them in place.

Local results on Python 3.13 (pandas 3.0.6, geopandas 1.2.0):

• With OSMNET_SKIP_NETWORK_TESTS=1: 24 passed, 14 skipped.
• Full suite against the live API: 32 passed, 6 failed. The 6 failures are test_make_osm_query, test_parse_network_osm_query, test_intersection_nodes2, test_node_pairs_two_way, test_node_pairs_one_way and test_custom_query_pass. They assert hard-coded element counts that no longer match current OSM data, and they fail the same way on unmodified dev.
• pycodestyle --max-line-length=100 osmnet is clean.

Not done here

• No version bump. CHANGELOG.rst has a new "Unreleased" entry that credits Fix Overpass API 406 errors by setting request headers #33. A 0.1.8 release should follow this.
• The deprecated unary_union and the stale counts in the live tests are untouched.

[smmaurer]

Thanks Paul, this looks good! I pushed a couple of follow-up commits and retargeted the PR to main, which we'll make the default branch as part of the new release.

Retry backoff. With the status request working, get_pause_duration() returns 0 whenever slots are free, including while the server is overloaded and returning 504s. So the 5 retries ran back to back. I saw this in a live test run: test_column_names raised OverpassRetryError after several "Re-trying request in 0.00 seconds" messages. Before this PR, the 406 on the status request meant the pause always fell back to 10 s. Now, unless error_pause_duration is given, each retry waits at least 5 s, doubling up to 60 s, or longer if the status page asks for it. In a live run today the server returned four 504s in a row; the retries waited 5, 10, 20 and 40 s and the fifth attempt succeeded.

Smaller items:

• Consistent domains for interpreter request and status request.
• Server remarks are logged at WARNING. The existing code passed level= to str.format() instead of log().
• The pause_duration docstring now clarifies that the argument is unused.
• ci.yml now runs on main.

Four new offline tests cover these changes, and there are three changelog bullets. Offline: 28 passed, 14 skipped; pycodestyle is clean. Live, only the six tests with outdated OSM element counts still fail. A packaging/CI PR will follow and run those tests against saved Overpass responses.