Skip to content

Stop reporting routine 4xx to Sentry, and report the DB errors we swallowed - #684

Open
AchrafReyani wants to merge 1 commit into
OpenFn:mainfrom
AchrafReyani:fix/sentry-4xx-and-swallowed-db-errors
Open

AchrafReyani wants to merge 1 commit into
OpenFn:mainfrom
AchrafReyani:fix/sentry-4xx-and-swallowed-db-errors

Conversation

@AchrafReyani

Copy link
Copy Markdown

Short Description

Reports the three database errors the client-auth path was swallowing, and stops sending routine 4xx to Sentry.

Fixes #554

Implementation Details

Both halves live in the same two files, as the issue notes.

1. The three swallowed DB errors. Each console.errord and moved on; each now captures next to the log it already wrote, tagged with a reason so the three are distinguishable in Sentry:

site reason
platform/src/server.ts — runMigrations() failure migrations-failed
platform/src/auth/instance-auth.ts — startup DB probe db-unreachable
platform/src/auth/instance-auth.ts — runtime client lookup client-lookup-error

The lookup site is the one worth calling out: the 503 it leads to is already captured as client-store-unavailable-503, but that capture carries only the token hash — never the error the database actually returned. captureException was already imported in both files, so no new imports were needed (server.ts has had it since the hook was added, so that part of the acceptance criteria was already satisfied).

2. Skipping 4xx in the onError hook. The one thing here that isn't a one-liner is reading the status, because Elysia puts it in a different place depending on where the error came from. I probed each case rather than guessing:

error error.status code set.status
NOT_FOUND 404 "NOT_FOUND" 200 (untouched)
VALIDATION 422 "VALIDATION" 422
PARSE 400 "PARSE" 400
ApolloThrowable(429, …) — 429 500
bare throw new Error(…) — "UNKNOWN" 500

So errorStatus takes the first number among error.status, code, set.status, and the hook skips when that is a 4xx. Note the NOT_FOUND row — set.status alone would read as 200, which is why the order matters. An error whose status can't be read is by definition unexpected, so it is still reported.

errorStatus and isClientError are exported from server.ts purely so the tests can drive them against real Elysia contexts instead of hand-written stand-ins; happy to make them module-private if you'd rather.

No HTTP response changes: the hook still returns nothing, so Elysia produces its normal body and status. Verified below.

Verification

Environment: Windows, bun 1.4.2, Postgres 16 in Docker and poetry 2.3.2 / Python 3.11 set up to match the bun CI job.

Reproduced first, on main. A local HTTP server standing in for Sentry (SENTRY_DSN=http://…@127.0.0.1:9999/1), counting envelopes. Five requests — GET /favicon.ico, GET /wp-admin, POST /services/test_errors {"trigger":"RATE_LIMIT"}, {"trigger":"UNEXPECTED"}, and a malformed JSON body:

favicon.ico -> 404   wp-admin -> 404   429 -> 429   500 -> 500   parse -> 400
=== SENTRY EVENTS (main) ===
count: 2
 - Error       | NOT_FOUND
 - SyntaxError | JSON Parse error: Expected '}'

Same five requests on this branch — identical status codes, nothing reported:

favicon.ico -> 404   wp-admin -> 404   429 -> 429   500 -> 500   parse -> 400
=== SENTRY EVENTS (this branch) ===
count: 0

The DB captures, end to end. Booted with APOLLO_CLIENTS_DB_URL=postgres://apollo:apollo@127.0.0.1:1/nope so the migration run and the startup probe both fail:

=== server log ===
Apollo migrations failed to run. …
Apollo instance auth: the database could not be reached …
=== SENTRY EVENTS ===
count: 2
 - PostgresError | Failed to connect | extra: {"reason":"migrations-failed"}
 - PostgresError | Failed to connect | extra: {"reason":"db-unreachable"}

On main the same boot reports nothing.

Tests. Six added — an errorStatus/isClientError table built from contexts captured off a real Elysia onError (so it's Elysia's behaviour, not my memory of it), the unreadable-status case, a 404 through the real server proving the hook consults the guard, the client-lookup-error capture, and the two startup failures. All six fail on main and pass here.

bun test platform/test, with Postgres and poetry wired up as in CI:

main         122 pass, 4 fail   (126 tests)
this branch  128 pass, 4 fail   (132 tests)

The 4 failures are the same on both and are this machine, not the change: they shell out to pkill, which Windows doesn't have (error: Executable not found in $PATH: "pkill").

AI Usage

  • Yes, I have used AI
  • No, I have not used AI

🤖 Generated with Claude Code

…llowed

Three database failures in the client-auth path were logged with
console.error and then forgotten, so production never saw them: the
migration run in server.ts, the startup reachability probe, and the
runtime client lookup. Each now captures alongside the log it already
wrote, tagged with a reason. The lookup site matters most — the 503 it
leads to is already reported, but with only the token hash, never the
error the database actually returned.

The onError hook, meanwhile, reported everything it saw, including the
NOT_FOUND from bots hitting /favicon.ico and the parse error from a
malformed body. Those are caller errors and they bury the ones we want.
Elysia puts an error's status in a different place depending on where it
came from — `status` on its own errors, a numeric `code` on an
ApolloThrowable, and otherwise only what it has already put on `set` —
so errorStatus reads all three in that order and the hook skips a 4xx.
An error whose status cannot be read is unexpected, so it is still
reported rather than dropped.

No change to any HTTP response: the hook still returns nothing and
Elysia produces its normal error body and status.

Fixes OpenFn#554

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Send swallowed auth/migration DB errors to Sentry, and stop reporting routine 4xx

1 participant