Issues

flake: TestFireWSLargeEcho_SwappedEchoesAreReorder failed once in CI (reorder=0: swapped echoes were not detected)

goceleris/probatorium#553 · opened

on Oct 25, 2026

The uws column dies at start (ERR_MODULE_NOT_FOUND on uWebSockets.js 20.71 under Node 24.21) and is dropped from the results: 10 cells per arch missing, no dnf, run concludes success

goceleris/probatorium#519 · opened

on Oct 16, 2026

amd64 bandwidth rows plateau at 9.1-9.25 Gbit/s for every server that can reach it (post-4k, post-4k-h2, ws-large-echo; 13.6-18.4 before 2026-10-04), and env.json records no bond, link speed, governor or IRQ state and gates nothing

goceleris/probatorium#518 · opened

on Oct 16, 2026

1

I-WS-STALL dossier: the goroutine dump runs inside the stall it records (2067 ms on msr1, soak 37140355449) and can extend it, and the dossier has no per-thread state, so a process-wide stall cannot be placed (runtime, kernel or scheduler)

goceleris/probatorium#494 · opened

on Oct 4, 2026

host-sampler: refapp_pid and refapp_rss_kb are 0 in every sample on every host, because it looks the refapp up as `pgrep -x probatorium-runner` (an 18-character name that can never match a 15-character comm)

goceleris/probatorium#493 · opened

on Oct 4, 2026

ValidateGate has no row for a refapp that died: a cell with status=failed and an I-LIVENESS failure_reason reaches the gate table only through its side effects, and the liveness tail calls a Go panic "no crash signature"

goceleris/probatorium#492 · opened

on Oct 4, 2026

fieldguard: TestGuardControlReadSide's parallel package loads peak at 5-7 GiB RSS, so a 7.75 GiB container run is OOM-killed when another container runs

goceleris/probatorium#489 · opened

on Oct 3, 2026

debugvars: no guard checks that a publish line reads its own EngineMetrics field, so a key wired to the wrong field passes all of them

goceleris/probatorium#488 · opened

on Oct 3, 2026

kitchen_sink: the walker's /healthz and /livez states are 404s; healthcheck serves /livez, /readyz, /startupz, and a Use mount never runs for unrouted paths (celeris#852)

goceleris/probatorium#487 · opened

on Oct 3, 2026

No tier observes a refapp graceful shutdown: Tier 1 SIGKILLs at the deadline and Tier 3 drops output and exit status after SIGTERM, so a data race, checkptr report or crash in a shutdown path cannot fail the race or checkptr tier

goceleris/probatorium#482 · opened

on Oct 2, 2026

Race tier: auth_session_ratelimit/adaptive/amd64 fails I-WS-ECHO (ws_echo_timeout=1) in 4 of 7 runs, whenever its route stays inline after the switch on the 2-worker ring; celeris fairness or a 2 s budget too tight for -race is unclassified

goceleris/probatorium#481 · opened

on Oct 2, 2026

1

cluster-forensics ships the last soak's host-sampler series as this run's on every tier without a sampler (nightly 37028784671 carries weekend-36433207097's samples.jsonl/kernel.log, hiding a swap/OOM day)

goceleris/probatorium#479 · opened

on Oct 2, 2026

I-MEM-1 under #470's bound misses leaks under ~3 KB/s where heap troughs scatter, celeris#573's 2.2 KB/s in auth_session/io_uring included, and late-onset leaks

goceleris/probatorium#478 · opened

on Oct 2, 2026

3

BenchTier has no host sampler during the bench and runs columns in one alphabetical block per arch: msr1's +1.09M bond0 rx drops cannot be placed, and engine and time window are confounded

goceleris/probatorium#477 · opened

on Oct 2, 2026

driver-pg-write: the postgres store fails within one second mid-cell and every later request errors (6 columns, both hosts, 3 suspect cells in bench 36441790519), taking the cell's rated passes with it

goceleris/probatorium#476 · opened

on Oct 2, 2026

network_bound can never see request bytes (isNetworkBound uses loadgen throughput_bps = response bodies): it flags 0 cells in bench 36441790519 while amd64 post-4k sits at the 2x10G client link

goceleris/probatorium#475 · opened

on Oct 2, 2026

scenario_resources on a rated cell spans the saturation pass and the whole rated sweep (~290 s), so SUT CPU per saturation request read from it is understated by a median 17% (amd64) / 11% (arm64)

goceleris/probatorium#474 · opened

on Oct 2, 2026

msr1 (arm64 SUT): a leftover 'rvtest' process (since ~2026-09-11, 12 SO_REUSEPORT listeners) and idle booked as iowait (IO PSI some 99%): every arm64 mean_cpu_pct reads ~100; clear it before the v1.6.0 release bench

goceleris/probatorium#473 · opened

on Oct 2, 2026

5

Follow-ups from #464: GO-2026-5024 in the tools module's x/sys, an unchecked Dependabot allow list, no PR tier for later mage bumps, an inline govulncheck pin, a stale queue pointer

goceleris/probatorium#472 · opened

on Oct 2, 2026

Tier-1 Markov walker counts a gated requests_error but discards the error, so a requests_error failure cannot be attributed (the #327 class)

goceleris/probatorium#471 · opened

on Oct 2, 2026

1

I-PANIC cannot see a panic that escapes to the engine's async-handler recover ("async handler panicked", the celeris#791 path): panic_count counts only middleware/recovery's records

goceleris/probatorium#468 · opened

on Oct 2, 2026

1

I-PANIC sees an unexpected panic only in idle windows on the observability refapp: the property loop reads panic_count before ExpectedPanics, and designed panics landing between the reads hide it

goceleris/probatorium#467 · opened

on Oct 2, 2026

I-MEM-1's noise floor models only the GC sawtooth, so the ws_echo walker's 0.6-1.6 MB trough scatter fails early windows by chance (soak 36433207097, 354 arm64 rows)

goceleris/probatorium#466 · opened

on Oct 2, 2026

6

validation: the RFC scraper reads a chunked body after a HEAD response, so a correct header-only HEAD with Transfer-Encoding: chunked would count as MissingChunkEnd

goceleris/probatorium#463 · opened

on Oct 2, 2026

Follow-ups from #460 (celeris fe9264f pin): merged before review, PR-body scripts and counts, the container commit stamp, what the local suites cannot check

goceleris/probatorium#462 · opened

on Oct 2, 2026

Navigation Menu

Repository navigation

Issues

flake: TestFireWSLargeEcho_SwappedEchoesAreReorder failed once in CI (reorder=0: swapped echoes were not detected)

The uws column dies at start (ERR_MODULE_NOT_FOUND on uWebSockets.js 20.71 under Node 24.21) and is dropped from the results: 10 cells per arch missing, no dnf, run concludes success

amd64 bandwidth rows plateau at 9.1-9.25 Gbit/s for every server that can reach it (post-4k, post-4k-h2, ws-large-echo; 13.6-18.4 before 2026-10-04), and env.json records no bond, link speed, governor or IRQ state and gates nothing

I-WS-STALL dossier: the goroutine dump runs inside the stall it records (2067 ms on msr1, soak 37140355449) and can extend it, and the dossier has no per-thread state, so a process-wide stall cannot be placed (runtime, kernel or scheduler)

host-sampler: refapp_pid and refapp_rss_kb are 0 in every sample on every host, because it looks the refapp up as `pgrep -x probatorium-runner` (an 18-character name that can never match a 15-character comm)

ValidateGate has no row for a refapp that died: a cell with status=failed and an I-LIVENESS failure_reason reaches the gate table only through its side effects, and the liveness tail calls a Go panic "no crash signature"

fieldguard: TestGuardControlReadSide's parallel package loads peak at 5-7 GiB RSS, so a 7.75 GiB container run is OOM-killed when another container runs

debugvars: no guard checks that a publish line reads its own EngineMetrics field, so a key wired to the wrong field passes all of them

kitchen_sink: the walker's /healthz and /livez states are 404s; healthcheck serves /livez, /readyz, /startupz, and a Use mount never runs for unrouted paths (celeris#852)

No tier observes a refapp graceful shutdown: Tier 1 SIGKILLs at the deadline and Tier 3 drops output and exit status after SIGTERM, so a data race, checkptr report or crash in a shutdown path cannot fail the race or checkptr tier

Race tier: auth_session_ratelimit/adaptive/amd64 fails I-WS-ECHO (ws_echo_timeout=1) in 4 of 7 runs, whenever its route stays inline after the switch on the 2-worker ring; celeris fairness or a 2 s budget too tight for -race is unclassified

cluster-forensics ships the last soak's host-sampler series as this run's on every tier without a sampler (nightly 37028784671 carries weekend-36433207097's samples.jsonl/kernel.log, hiding a swap/OOM day)

I-MEM-1 under #470's bound misses leaks under ~3 KB/s where heap troughs scatter, celeris#573's 2.2 KB/s in auth_session/io_uring included, and late-onset leaks

BenchTier has no host sampler during the bench and runs columns in one alphabetical block per arch: msr1's +1.09M bond0 rx drops cannot be placed, and engine and time window are confounded

driver-pg-write: the postgres store fails within one second mid-cell and every later request errors (6 columns, both hosts, 3 suspect cells in bench 36441790519), taking the cell's rated passes with it

network_bound can never see request bytes (isNetworkBound uses loadgen throughput_bps = response bodies): it flags 0 cells in bench 36441790519 while amd64 post-4k sits at the 2x10G client link

scenario_resources on a rated cell spans the saturation pass and the whole rated sweep (~290 s), so SUT CPU per saturation request read from it is understated by a median 17% (amd64) / 11% (arm64)

msr1 (arm64 SUT): a leftover 'rvtest' process (since ~2026-09-11, 12 SO_REUSEPORT listeners) and idle booked as iowait (IO PSI some 99%): every arm64 mean_cpu_pct reads ~100; clear it before the v1.6.0 release bench

Follow-ups from #464: GO-2026-5024 in the tools module's x/sys, an unchecked Dependabot allow list, no PR tier for later mage bumps, an inline govulncheck pin, a stale queue pointer

Tier-1 Markov walker counts a gated requests_error but discards the error, so a requests_error failure cannot be attributed (the #327 class)

I-PANIC cannot see a panic that escapes to the engine's async-handler recover ("async handler panicked", the celeris#791 path): panic_count counts only middleware/recovery's records

I-PANIC sees an unexpected panic only in idle windows on the observability refapp: the property loop reads panic_count before ExpectedPanics, and designed panics landing between the reads hide it

I-MEM-1's noise floor models only the GC sawtooth, so the ws_echo walker's 0.6-1.6 MB trough scatter fails early windows by chance (soak 36433207097, 354 arm64 rows)

validation: the RFC scraper reads a chunked body after a HEAD response, so a correct header-only HEAD with Transfer-Encoding: chunked would count as MissingChunkEnd

Follow-ups from #460 (celeris fe9264f pin): merged before review, PR-body scripts and counts, the container commit stamp, what the local suites cannot check