Skip to content

www: request only the count files a run publishes, and survive a 5xx - #219

Merged
lukekarrys merged 1 commit into
mainfrom
lk/site-build-missing-files
Nov 18, 2026
Merged

lukekarrys merged 1 commit into
mainfrom
lk/site-build-missing-files

Conversation

@lukekarrys

@lukekarrys lukekarrys commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Problem

Every Vercel deploy of the site, production included (main at 6146dbc, 2026-11-18 05:58; also #217 and #218), fails while prerendering:

⨯ Error: Request for https://benchmarks-data.vlt.io/2026-11-18/vue-registry-clean-process-count.json failed with status 503
> 108 |     throw new Error(`Request for ${url} failed with status ${response.status}`);

Root cause

Two things combined:

  1. The build requests files the data says do not exist. loadFixture (apps/www/src/lib/load.ts) fetched a package-count and a process-count file for every (fixture, variation) it rendered, and relied on a 404 to say "none". But registry-* variations publish no process count and build publishes no count at all (docs/METHODOLOGY.md). Eight requests per build were for files that are not there.
  2. GitHub Pages answers some of those missing paths with 503 instead of 404, especially under a burst, and the same path flips between the two minutes apart. cachedGet (apps/www/src/lib/data.ts) treated a 404 as missing but threw on anything else — and that throw is inside "use cache", so the prerender fails the page on it even though the only caller (probe) catches it. One 503 on an optional file fails the whole build.

Fix

  • Stop requesting what the data says does not exist. loadFixture now reads the run's manifest shard (manifest/{date}.json, the DirectoryEntry every caller already fetches) and requests only the count files its packageCounts / processCounts list. loadScenario takes the shard as an entry option; report.ts and about/components.tsx pass the one they have, and it is fetched when a caller does not. No special-casing of registry or build anywhere — the shard is the authority, as METHODOLOGY's "Discover, never enumerate" asks.
  • Retry transient 5xx. fetchRetrying retries a 5xx up to two more times with a short backoff (250 ms, 500 ms), alongside the existing 429 handling.
  • Optional vs required. A 5xx that outlasts the retries is returned from cachedGet like a 404 (not thrown inside the cache); DataClient turns it into a DataFetchError. probe — which wraps every fetch a page can render without (count files, versions.json, summaries, chart-data) — reports it as missing and logs a server-side console.warn. Data a page cannot render without (the index, a run's shard) is never probed, so its DataFetchError still throws and a real outage still fails the build instead of publishing an empty site.
  • apps/www/test/report.test.ts: the trend-budget test now excludes the shown run's shard from the files it counts as the trend's (it is the shown run's, like its files and summary).

Tests

New apps/www/test/data.test.ts (mocked fetch):

  • fetchRetrying retries a 5xx and returns the answer that follows; gives up after three attempts on a persistent 5xx; does not retry a 404.
  • Through dataClient(): a persistent 503 on probed data is failed with one warning; a 404 is absent with no warning; a 503 on getIndex() (required) rejects with DataFetchError after its retries.
  • loadScenario with a shard listing package counts but no process counts for registry-clean, and nothing for build: no process-count request for registry-clean, no count request at all for build, both where the shard lists both, and the shard is fetched when the caller did not pass it.

Verification

  • vlt install at the root (changed vlt-lock.json, not committed); vlr format:check, vlr lint:check, vlr typecheck clean at the root; cd apps/www && vlr test: 13 files, 277 tests pass.
  • Real next build against https://benchmarks-data.vlt.io with every fetch logged (a --required preload appending status + URL), .next/cache/fetch-cache cleared before each run so nothing was served from the build cache. Five runs on this branch: all 40 pages generated every time, every request a 200, zero 404s or 5xx; two exited 0 (the other three were stopped after page generation by Next's unrelated "known security vulnerability" upgrade notice, which fires on alternate runs and is not acted on here).
  • Baseline on origin/main the same way: all 40 pages, 126×200 and 8×404 (it happened to get 404s rather than 503s this time, which is the flip described above).

Requested URLs, before minus after — exactly the eight files that do not exist, and nothing added:

2026-11-18/astro-registry-clean-process-count.json
2026-11-18/babylon-registry-clean-process-count.json
2026-11-18/large-registry-clean-process-count.json
2026-11-18/next-build-package-count.json
2026-11-18/next-build-process-count.json
2026-11-18/next-registry-clean-process-count.json
2026-11-18/svelte-registry-clean-process-count.json
2026-11-18/vue-registry-clean-process-count.json

The 18 count files still requested are the ones the shard lists: {astro,babylon,large,next,svelte,vue}-clean-{package,process}-count.json and *-registry-clean-package-count.json.

Not changed

  • loadTrend's per-fixture package-count fallback for a run without a summary still relies on 404s — but every one of the 399 published dates has a summary, so that path never runs in production; left alone to keep this focused.
  • No Next.js upgrade; the "SecurityFatalError" upgrade notice seen during builds is unrelated to this failure.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Temporary server errors are retried, helping results load more reliably when services are briefly unavailable.
    • Scenario data now requests package and process counts only when they’re available for that run. Missing optional counts no longer prevent other scenario data from loading.
    • When optional data remains unavailable after retries, it’s reported as failed; required data errors continue to prevent the build from succeeding.

Every Vercel deploy, production included, has been failing while prerendering:
a request for a count file that does not exist (registry-* publishes no process
count, build no count at all) came back from GitHub Pages as a 503 rather than a
404, and `cachedGet` threw on it.

Fix at the source: `loadFixture` reads the run's manifest shard, which every
caller already fetches, and requests only the count files it lists. The shard is
passed to `loadScenario` as an option and fetched there when a caller has not.

Make fetches robust too: `fetchRetrying` retries a 5xx twice with a short
backoff, and a 5xx that outlasts the retries is returned from the cache like a
404 rather than thrown, so `probe` can treat optional data as missing (with a
server-side warning) while required data (the index, a shard) still throws its
`DataFetchError` and fails the build loudly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 18, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
benchmarks Ready Ready Preview Nov 18, 2026 6:50am UTC

Request Review

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

Walkthrough

This change updates data fetch retry handling for 5xx responses and changes scenario loading to use directory entry metadata when available. It also updates callers and tests to match the new fetch and loading paths.

Changes: Scenario data loading and fetch handling

Layer / File(s) Summary
Fetch retry and probe results
apps/www/src/lib/data.ts, apps/www/test/data.test.ts
fetchRetrying now retries 5xx responses with a separate retry limit and shared backoff logic. cachedGet allows exhausted 5xx responses through the non-OK path instead of throwing immediately. probe now warns before it returns failed for non-404 fetch errors. Tests cover 5xx retry success, persistent 5xx failure, 404 non-retry, probe warning behavior, and getIndex error handling.
Directory-entry-guided scenario loading
apps/www/src/lib/load.ts, apps/www/test/data.test.ts
loadScenario now accepts an optional entry value or promise and fetches the directory entry only when the caller does not supply one. loadFixture now uses that entry to request package-count and process-count files only when the manifest lists the fixture variation. Tests verify selective count-file requests and manifest fetching when no entry is provided.
Caller updates and report test alignment
apps/www/src/app/about/components.tsx, apps/www/src/lib/report.ts, apps/www/test/report.test.ts
LatestResults and the non-averaged report path now pass their existing directory entry into loadScenario. The trend fetch-budget test now excludes manifest/ paths from the trend-file set.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Suggested reviewers: ejkorol

Merge Risk: Minimal · up to 07321

This change reduces prerender failures caused by requests for unpublished count files and by transient 5xx responses. No actionable merge-blocking risk was identified, and the change is easy to revert.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@lukekarrys

Copy link
Copy Markdown
Member Author

Reviewed and merging: this is the fix for every Vercel deploy failing since 2026-11-18 (production included), and this PR's own Vercel build passes.

  • Requests: loadFixture now requests only the count files the run's manifest shard lists, which removes exactly the 8 requests for files that don't exist (*-registry-clean-process-count.json, next-build-*-count.json).
  • Retries: 5xx responses are retried twice with backoff. getIndex and getDirectoryEntry aren't probed, so a real outage of required data still fails the build.

One trade-off to know about (not new code, but its effect changes): benchmark result files are also read through probe. A real result file that keeps returning 5xx after the retries now renders as missing (with a server warning) instead of failing the build. With the retries, and the 503s only ever seen on paths that don't exist, a briefly missing fixture beats a stale site, but it's the place to look if a fixture ever vanishes from a page.

🤖 Generated with Claude Code

@lukekarrys
lukekarrys merged commit 48920d2 into main Nov 18, 2026
11 of 12 checks passed
@lukekarrys
lukekarrys deleted the lk/site-build-missing-files branch November 18, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant