Repository navigation
[Quality][P0] Add deterministic reference-scene visual and GPU performance qualification #128
Description
Activity
Implementation progress and Metal qualification evidence (2026-07-23)
Implemented the deterministic qualification stack:
- Versioned seven-case corpus: high/constrained PBR spheres, Damaged Helmet, Sponza, a pinned deterministic 96-mesh Bistro derivative, skinned Fox plus alpha-tested foliage motion, and a 10,240-mesh/192-light stress scene.
- Fixed-step uncapped warm-up/measurement/capture lifecycle. Sustained windows are now 240–300 measured frames after 120–180 warm-up frames; capture/readback is outside the measured interval.
- Native adapter/backend/capability and active renderer-path reporting. Qualified local adapter: Apple M1 Max / Metal / high-end-rt-bindless / hardware ray query / timestamp query.
- One dormant debug-capture API producing final plus
hdr-scene,scene-depth, and all three shadow cascades. Every one of the seven current cases produced all five required intermediates. - JSON plus Markdown/HTML summaries, strict visual and hard-machine performance gating, explicit report-only mode, baseline review/install with stale-review and path containment protection, reproducibility comparison, and five seeded fault controls.
- Scheduled/manual self-hosted Metal and Vulkan CI, a constrained-tier hard gate, compact job summaries, and always-uploaded success/failure evidence.
- Migration documentation:
tools/qualityis canonical;tools/validate.shis retained only as legacy macOS exploration.
Measured evidence:
- Reproducibility retry: PASS with identical stable metadata.
- PBR and Helmet final plus all intermediates: byte-identical across repeated runs.
- Sponza final temporal variance: SSIM 0.9999991, luminance RMSE 0.0000724; all intermediates within the stricter reproducibility limits.
- Seeded controls: BRDF energy, shadow placement, GI leakage, motion history, and texture orientation all DETECTED.
- Post-refactor comparison: PBR and Helmet byte-identical; Sponza SSIM 0.9999971 / RMSE 0.0001277.
- Native shared tests: 131 unit pass, 10 ordinary GPU goldens pass, 3 render-target tests pass; 0 failures.
bloom-diff: 3 tests pass.bloom-reference: 2 tests pass.- macOS/shared release checks pass.
- FFI validation: 0 failures, 0 warnings across macOS/Linux/Windows/Android/iOS/tvOS/watchOS/web.
- File-line ratchet, actionlint, Python compile/tests, and
git diff --check: PASS.
Important current-quality findings are now locked into review evidence rather than hidden: PBR material lobes are largely flat, and the final Sponza/Bistro output has strong temporal/post blur while the captured HDR scene is substantially cleaner. These are quality-roadmap inputs, not claims of UE-level output.
Governance blocker before closing: approved baselines do not yet exist. A no-write initial review bundle is ready at
tools/quality/out/baseline-review-current/review.json; it recordsbaseline_state: absentfor all seven cases and includes 35 named intermediates. Per the acceptance contract, a human must inspect it and explicitly runbaseline-install --approved-by NAME. I have not self-approved or silently installed those pixels.Cross-backend status: Metal is locally qualified. The Vulkan/RTX 4080 and DX12 hardware evidence remains pending execution on labelled hardware; #140 tracks the broader Vulkan/DX12 RT qualification dependency.
Review UX follow-up: the human-review bundle now has a self-contained gallery at
tools/quality/out/baseline-review-human/review.htmlplusreview.mdand the authoritativereview.json. The gallery shows each proposed final, applicable before/diff evidence, CPU/GPU p95, and expandable named HDR/depth/shadow intermediates. It remains no-write; no baseline has been installed.Temporal-evidence qualification checkpoint (commit 011f5c8, pushed to draft PR #147):
- The manifest now requires the complete capture-only TAA, SSR, and SSGI evidence set for every quality-preset 3 case. A renderer or capture regression that drops one of these outputs now fails qualification instead of silently yielding an incomplete bundle.
- The runner validates all current named diagnostic outputs, including the realtime PT names, so a future PT corpus case can use the same contract without another runner change.
- The constrained preset metadata now truthfully records TAA as disabled and does not require diagnostics for disabled temporal systems.
- A unit contract pins the distinction between High/stress preset-3 cases and the constrained case.
- This is qualification-only: no production render pass, resource, shader, pixel, or measured-frame cost changed. Diagnostics remain confined to the post-measurement capture frame.
Representative native Metal proof:
python3 tools/quality/run.py run quick --case pbr-spheres-high --report-only --out /tmp/bloom-quality-temporal-evidenceThe 420-frame case emitted all 16 required PNGs. TAA maps were 512x512; SSR and SSGI temporal maps were 128x128. The resolved SSGI, filtered SSR, and raw SSR metric sidecars each reported zero non-finite pixels. The only failure was the existing missing approved baseline; there were no missing-intermediate failures.
Additional validation: 8/8 quality-runner unit tests pass, Python compilation passes, manifest parsing reports all 9 cases, and
git diff --checkpasses.The remaining closure blockers are unchanged: a human must inspect and explicitly approve/install baselines, and the Vulkan/RTX 4080 hardware bundle remains pending. No baseline was self-approved or installed.
Progress landed in PR #147 at 56bf87c and is qualified by docs/evidence/issue-128-bistro-ssr-normal-v1.{md,json}.
The governed Bistro exterior case (96 camera-visible meshes, 180 warm-up + 240 measured frames, high/native, TAA + SSR + HW-ray-query SSGI + bloom/fog/shafts + transparency/transmission) exposed a real SSR defect. Fragment Y increases downward, but SSR reconstructed the view-space derivative normal with the opposite cross-product order. Camera-visible normals therefore faced away, NdotV collapsed to zero, and Schlick Fresnel became 1.0 across most geometry. That was the pale, globally washed frame.
The shared ordinary/layered SSR source now constructs a view-facing normal and has a Naga/source-contract regression test. SSR mean luminance fell from 0.266009541 to 0.011289671 (23.56x), with zero non-finite pixels/outliers. The SSR pass stayed effectively flat: 2.639690 ms to 2.643431 ms (+0.003741 ms / +0.14%), with no passes, textures, buffers, bindings, or steady-state resource creation added.
Release qualification is green: 357 unit tests, 59 runnable GPU goldens, 4 render-target tests, device construction, strict lint/format, Wasm, FFI/schema, quality governance, cooker tests, and example inventory. The Bistro report still fails only because the human-approved baseline is intentionally missing; the corrected candidate was not auto-approved.
Bistro qualification progress in commit 40ff96e:
- The rich launcher and default demo now select
assets/bistrox.gltf, the complete expanded-material asset with all 2,909 placements and 551 shared source meshes. - The sibling
bistro.gltfusesMSFT_texture_ddsindirection and produced a black-frame diagnostic result in the current path, so it remains explicitly available for importer diagnostics rather than being the user-facing quality scene. - A rebuilt bounded launch confirmed
bistro_meshes=2909,bistro_placements_attached=2909, and a valid warmed rendered frame. - The same commit restores per-pixel specular/glossiness maps for 109 Bistro materials, making this reference scene a more truthful material-fidelity test.
- The rich launcher and default demo now select
Checkpoint 922aff2 fixes a deterministic-quality qualification hole.
The runner already exported BLOOM_QUALITY_FIXED_TIMESTEP, but EngineState ignored it for material animation, so wind/procedural captures advanced on wall time. The engine now activates a positive finite fixed renderer/callback clock only when requested, while public getTime remains wall elapsed and performance telemetry reports the clock actually consumed.
Evidence:
- procedural foliage replays the complete repeated 16-frame fractional sequence byte-for-byte
- governed quick/pbr-spheres-high reports active timestep 0.016666667 and a truthful 3069.967 ms wall measurement for 300 frames
- four alternating same-binary normal/fixed runs produced byte-identical final images (SHA-256 1929adb79316e90085e4ee0c1fd164e2008ed3c673b9b1c59cd99df436b9bf07)
- wall and GPU deltas change sign between pairs; CPU means and GPU p95 do not regress consistently, classified as no measurable regression
- all 12 quality-runner governance unit tests pass
Full artifact: docs/evidence/issue-149-fractional-dynamic-native-match-v1.md and .json.
The governed report still names two pre-existing setup failures: missing taa-thin-feature-confidence intermediate and no installed approved portable pbr-spheres-high baseline. Issue remains open.
Follow-up checkpoints 1938e86 and 75773a7 are pushed to both active branches.
Root cause: 2e9e744 intentionally replaced the obsolete capture-only taa-thin-feature-confidence classifier with the production-relevant taa-detail-lock diagnostic, but the quality manifest and Python allowlist still required the old filename.
The governed contract now requires the renderer's actual output. Exact quick/pbr-spheres-high result:
- all 30 named intermediates emitted, including taa-detail-lock
- no missing-intermediate failure
- active fixed timestep 0.016666667
- 300 measured frames; wall mean 5.734647 ms
- CPU mean/p95 2.031954/2.384414 ms
- GPU mean/p95 12.569622/23.123957 ms
- only remaining failure: approved portable baseline is absent
A non-mutating initial review bundle was generated at /private/tmp/bloom-quality-detail-lock-review-v2 with baseline_state absent and all 30 intermediates. No baseline was installed because installation requires an independent human reviewer name.
Evidence: docs/evidence/issue-128-governed-detail-lock-v1.md and .json. Issue remains open for baseline approval and remaining corpus cases.
Checkpoint
08d0010extends the deterministic quality contract with an exact frame-indexed Sponza motion consumer. The runner forcesBLOOM_QUALITY_FIXED_TIMESTEP=0.016666666667, pixel-exact headless dimensions, exact numbered capture, and explicit synchronous completion before comparing native 1.0, fractional 0.75, and an independent repeat.The 32-frame 1600x900 run passes its new tight reference gates (
0.013151819 <= 0.0133frame RMSE;0.009808212 <= 0.0100motion-derivative RMSE). Independent Metal hardware-ray variation remains far inside the existing governed reproducibility bounds (worst luma RMSE0.000188266, minimum SSIM0.999986172).Evidence:
docs/evidence/issue-149-sponza-motion-native-match-v1.md/.json. No baseline was installed and no production shader or graph policy changed in this checkpoint.Qualification checkpoint
26b1893adds governed Sponza owner isolation and stricter perceptual acceptance gates.The runner now supports an explicit
--ssgi 0|1control and optional mean luma RMSE, SSIM, OKLab, and edge-delta limits in addition to native frame and motion-derivative RMSE. The accepted 32-frame 1600x900 candidate passes the stricter gate set; the exact base fails it. Independent repeats remain inside the hardware-ray reproducibility envelope.Quality telemetry now reports the moving-detail policy, base and locked strengths, classifier, and zero additional samples. No baseline is silently installed.
Evidence and commands: https://github.com/Bloom-Engine/engine/blob/26b1893/docs/evidence/issue-149-detail-locked-moving-residual-v1.md
Integrated deterministic browser quality evidence at
45d3300(d8d47faproduction checkpoint).The Web lane now goes beyond a flat known-color smoke: real Chrome imports the canonical Damaged Helmet, runs matched native 1.0 / fractional 0.75 TAA / fractional 0.75 no-TAA motion sequences, rejects validation errors, incomplete/static/flat output, excessive spatial or motion-derivative error, unexpected hardware RT, and an inactive software SSGI fallback.
result.jsonrecords governed limits, measured metrics, adapter/tier, runtime paths, material residency, and relative frame directories.CI uploads
target/ci/web-smokeon success and failure for 30 days. Temporary site/WASM copies are removed first, so the artifact contains the compact report, logs, known frame, and 24 numbered quality frames.The new gate immediately found a real defect hidden by the old 2D smoke (
ssr_temporal_pipelineinvalid under WebGPU uniform-control rules), and it passes after the bounded shader fix. Exact base/candidate native A/B profiling shows no measurable performance or resource regression.Evidence:
docs/evidence/issue-149-web-fallback-tsr-v1.mdand.json. Broader hardware/scene qualification remains open.Integrated a machine-readable fractional/native performance qualification component at
b272dca(test/CI commit27217f3)../scripts/ci-check.sh --hardware --component fractional-native-throughputnow runs three alternating 0.75/native-1.0 pairs at quality preset 4, with 120 warm-up and 600 measured frames per run. It fails closed unless both median end-to-end wall time and median timestamped GPU work retain at least a 5% advantage.result.jsonandsummary.mdare written before assertions so failed hardware qualification remains diagnosable.Metal qualification passed twice. The exact accepted run measured 12.37% wall-time and 22.91% GPU-work advantage; steady-state resource-churn counters remained zero. The component is wired into both Metal and Vulkan hardware jobs and uploaded with their quality artifacts. The hosted quality-contract job passes at the integrated SHA (run 33296876642); hardware jobs correctly skip on pull requests.
Evidence:
issue-149-fractional-native-throughput-v1.mdand JSON.This advances the deterministic performance/evidence contract but does not close #128: a hosted Vulkan result and the remaining representative-scene/tier bundles still need to be produced. The repository currently reports zero registered self-hosted Actions runners.
Fail-closed bootstrap matrix checkpoint is integrated at
f101159(runner fix465c5f5).The initial nine-case capture audit found that
run_casereturned immediately when an approved baseline was absent, before applying hard CPU/GPU budgets. Report-only/bootstrap results could therefore hide a real performance failure behind the expected baseline failure. The runner now finalizes telemetry, steady-state resources, required intermediates, and applicable performance budgets first; baseline absence remains an independent failure and explicit human review/install remains mandatory.The real PBR gate now reports CPU and GPU overruns plus the missing baseline together. A focused regression test covers the same ordering with a missing required intermediate. Hosted renderer-quality CI passes at the exact integrated SHA (run 33299848026); the complete local quality contract passes 48 Python governance tests, three
bloom-difftests, and 48 cooker tests.The clean exact-commit Metal bootstrap captured all nine manifest cases with every required intermediate and zero steady-state bind-group/graph/pipeline/transient-resource churn. No baseline was installed. The newly visible applicable budget state is:
- pass: Damaged Helmet, masked-alpha coverage;
- fail: PBR high (CPU + GPU), Sponza (GPU), skinned-alpha motion (CPU), weighted transparency (GPU);
- constrained tier: three-run median passes (
4.552 msCPU p95 vs5.0;7.099 msGPU p95 vs8.0), with one CPU sample at5.094 ms; - Bistro and draw/light stress remain report-only on Metal because their hard budgets belong to RTX 4080 / Vulkan.
The dominant timestamped work in the remaining GPU failures is bloom (PBR
10.192 ms, Sponza25.669 ms; weighted transparency9.218 msplus4.045 mscomposition), which is the next investigation. No budget or reproducibility bound was weakened.Evidence:
issue-128-bootstrap-matrix-failclosed-v1.mdand JSON.#128 remains open: the initial candidate images still require human approval, the exposed Metal budget failures must be resolved or independently qualified, and Vulkan hardware evidence is still absent.
Bloom pass-tail performance checkpoint is integrated at
9cae4cd(renderer change5e543a0).The initial four-bilinear-fetch upsample experiment passed static image comparison but was rejected: balanced exact-commit repeats were neutral (
+2.2%PBR bloom median,-0.9%Sponza). Per-level timestamps instead showed a ~2–2.5 ms Metal cost floor even for the smallest 90x50 / 45x25 passes, identifying render-pass/store-load overhead as the useful target.The retained change keeps the original governed nine-tap filter, reduces the chain from five mips/nine render passes to four mips/seven passes, and adds stable per-level profiler labels. Three-run exact-commit medians versus
f101159:- PBR: bloom
9.891 -> 7.836 ms(-20.8%), total GPU mean-12.7%, GPU p95-6.0%, wall mean-3.1%; - Sponza: bloom
27.431 -> 20.890 ms(-23.8%), total GPU mean-16.5%, GPU p95-24.0%, wall mean-10.2%; - weighted transparency: neutral inside the governed noise envelope (GPU p95
31.007 -> 30.854 ms).
Exact base/candidate images pass: PBR and weighted transparency are pixel-identical; Sponza is SSIM
0.999998748, luma RMSE0.000070276, with zero pixels above the0.02tolerance.The clean exact-candidate full corpus completed all nine cases, every required intermediate, and zero steady-state bind-group/graph/pipeline/transient-resource churn. Applicable GPU p95 gates still open are PBR
23.461/20, Sponza56.685/55, weighted transparency30.841/30, plus a noisy Damaged Helmet sample at12.181/12 ms. No budget/noise bound was weakened, and no baseline was installed.Validation: 70 Python quality-governance tests, three
bloom-difftests, and 27 focused shader tests pass. Hosted quality governance is green at the exact integrated SHA (run 33303552632); its hardware jobs are skipped, so this is not Vulkan qualification.Evidence:
issue-128-bloom-pass-tail-v1.mdand JSON.#128 remains open for remaining Metal budget work, explicit human baseline/motion approval, and hosted Vulkan evidence.
- PBR: bloom
Verified Metal checkpoint pushed.
Renderer commit: 09ad0b7
Evidence commit: 4d5631a
Evidence: https://github.com/Bloom-Engine/engine/blob/4d5631abd6ccee29f7179b133112130f55b1b65a/docs/evidence/issue-128-ssgi-resolve-compute-v1.mdThe half-resolution SSGI resolve is now an 8x8 compute storage write instead of a fullscreen raster pass. Pixel-center UVs, the resolve filter, history ownership, and the RGBA16F target are preserved.
Balanced Apple M1 Max / Metal medians:
- PBR resolve 2.798 -> 0.052 ms (-98.2%); total GPU p95 23.374 -> 15.921 ms (-31.9%).
- Sponza resolve 5.068 -> 0.056 ms (-98.9%); total GPU p95 59.994 -> 41.746 ms (-30.4%).
The clean exact nine-case corpus now clears every applicable Metal GPU p95 budget: PBR 15.408/20, Helmet 8.425/12, Sponza 41.824/55, skinned 19.830/30, weighted 23.648/30, and masked 14.886/30 ms. Three host-CPU outliers in the long run clear on exact three-run medians (2.605, 7.159, and 5.240 ms).
Visual/resource gates:
- 9/9 final-image diffs pass; worst SSIM 0.999997616.
- 8/8 direct SSGI-intermediate diffs pass; worst SSIM 0.999997973 and zero pixels above tolerance.
- zero steady-state resource churn in all nine cases; one frame-submission encoder.
- 483 release library tests and 86 GPU golden tests pass.
Remaining gates are explicit human baseline installation and hosted RTX 4080 / Vulkan qualification; neither was weakened or bypassed.
9 remaining items
The next renderer fix is in draft #159: correct SSGI depth receivers and normal reconstruction. The original 16-frame HD startup limits now pass on Radeon/Vulkan, verified Radeon/DX12, and hosted Metal; the new analytic-plane regression rejects either old calculation.
At
d610d6a1ad4ab0ea9c7768c1a30acd2cce3fc2e4, all nine Radeon images pass in three complete runs. Runs 2 and 3 pass all configured checks, including host load, and reproduce all 257 compared artifacts byte-for-byte. Their metadata and timing differences also meet the unchanged reproducibility limits. Run 1 retains its background-load failures. RTX-specific budgets and named physical-hardware acceptance remain unperformed.Report, captures, logs, source, and checksum receipts are published. ZIP SHA-256:
0ea0d9ff4769c7ee9eaed2ed977bf2205f6eda52d328120819743037a5f5645a.Next-work audit found seven canonical Windows link/compile failures (six outdated palette uses and one unavailable Perry UI function). It also found that hosted Windows CI reports green without Cargo output because its Bash entry point is invoked through PowerShell without an explicit Bash shell. The published local Windows tests were executed directly and remain valid; the hosted invocation and example matrix need correction. The broader engine plan remains open and no PR is merged.
Windows build/example follow-up is published in draft PRs #160 and #161. Windows CI now actually executes Bash/Cargo and requires execution summaries; the corrected hosted native build and all 20 canonical native example links pass. Exact local source
59244b9also passes 20/20. The pinned matching Perry source build resolves the release-library and per-app profile link failures; all failed profiles remain in the evidence.The Windows shared suite is still failing: the latest hosted run finishes with 486 passed, two DX12 GPU failures, and one ignored test. Both failures reproduce on the Radeon 760M and Windows' software adapter; Vulkan passes. The preceding access violation remains unresolved. These findings are under separate investigation, and startup/clean-install qualification remains next.
Report and evidence ZIP. ZIP: 9,940,053 bytes; SHA-256
2ebcb258a72e473723331dcd9fbaad1781b36d0ea70b0e95a3bcfd90fcf0a9b9. All 287 payload hashes and archive CRCs were verified, along with the uploaded asset digest and tag source. No unavailable RTX qualification or completed broader engine acceptance is claimed.Draft #162 and its published report/evidence preserve DX12 indirect draw identities through the existing bounded fallback and restore the visibility reconstruction GPU oracle without changing its pixel limits. The original two oracles pass in five Radeon/WARP backend/compiler configurations; the new production virtual-visibility check passes on Radeon Vulkan, Radeon DX12 and WARP. The complete local Vulkan shared CI component passes at
ab8019b.Hosted Windows native building and all 20 example links pass again. The Windows shared process still crashes with STATUS_ACCESS_VIOLATION (exit 139) before completing the suite; this remains unresolved. The expanded local DX12 golden suite also exposes a camera-cut equality failure that reproduces on unchanged parent
59244b9. Camera-history diagnosis and the hosted crash are being handled separately; neither failure is waived.ZIP: 2,006,103 bytes, 201 verified payloads. SHA-256
ff50b4653062f2070508b825fc47f032e5907d5a054285e193ff33d2cb74975e. The uploaded digest and exact source tag were verified. No full platform/hardware acceptance or merge is claimed.Published the camera-history reset evidence and draft PR #163, stacked on #162.
The unchanged shader leaves visible residue after 40 old-history frames (about 16,900 pixels, maximum channel error 232). Guarding old color/confidence reads with the existing history predicate restores exact fresh/reset equality at both eight and 40 frames under Radeon Vulkan, DX12/FXC and DX12/DXC. All six captured RGBA pairs match byte for byte; thresholds and baselines are unchanged.
The complete local DX12 and Vulkan shared suites pass, including all 93 goldens. Hosted macOS passes the same expanded camera-cut check and 93 goldens; real Windows native builds and all 20 example links pass. Hosted Windows shared tests still crash with
0xc0000005before goldens. A local WARP shader-optimizer crash is now under investigation; #163 does not claim to fix it. Named RTX/M1 Max full-quality jobs remain skipped.Report and evidence ZIP —
camera-history-c643b85-windows-metal-20260911-v1.zip(3,052,783 bytes), SHA-256c03b55eafd7c8d2cf17c08c958440395d4aa65fd37f9beeef1241a53e247d4d7. Every payload hash and ZIP CRC was verified, and GitHub asset digests match. The archive includes original failures and exact source/hosted-tree receipts. No merge or unavailable-hardware result is claimed.Draft PR #164 and its evidence report are published. Hosted Tests run 34552411807 passes all 22 jobs, including actual Windows shared execution, native engine compilation and all 20 example links.
The Windows lane now explicitly uses DX12/FXC. Original WARP/DXIL controls reproduce an access violation during concurrent traversal shader compilation; the debugger stack and failed controls are retained. All ten hierarchy tests pass serially with DXC, while both four-thread DXC controls crash. FXC passes the concurrent controls. The underlying WARP/DXIL defect remains open.
Complete local Radeon DX12/DXC and Vulkan shared components pass all 93 goldens and remaining integration checks. Hosted Windows goldens skip the CPU adapter: their nominal 93 "ok" results do not qualify physical images. The separate physical Radeon FXC run still has four layered-material pipeline compilation failures (X3570/X3511); an explicit-LOD correction is being validated without changing assertions or baselines.
Archive
warp-compiler-f96af2e-local-hosted-20260911-v1.zip(1,980,437 bytes), SHA-2560c181559edb073296bd74367c865f457b067bdfafc0436dbda48c25b32adf407. ZIP CRC, every payload hash and both GitHub asset digests are verified. Source and hosted merge trees match. Native/browser starter startup, installed-package usability, wider engine requirements and named hardware acceptance remain open; no PR is merged.Draft PR #165 and its evidence report are published. The sheen lookup now samples explicit level zero, fixing four DX12/FXC material pipeline compilation failures (X3570/X3511) without changing rendering assertions or baselines.
The complete local FXC shared suite passes, including all 93 goldens; DX12/DXC and Vulkan each pass all 93 goldens too. Hosted Tests run 34555112391 passes all 22 jobs. Windows library tests pass 489 with one ignored; its goldens skip the CPU adapter and are not physical-image qualification. Hosted Metal supplies additional rendering evidence. The native engine build and all 20 example links pass.
Archive
fxc-layered-5d99fdd-local-hosted-20260911-v1.zip(1,847,832 bytes), SHA-2568fc1a6af07b975a14ecc4d3b09b35c3db1967038b590d8ebca869528a9810b7b. Every payload hash, ZIP CRC and both GitHub asset digests are verified. The committed shader matches the tested candidate; the hosted merge and PR head have identical trees.The WARP/DXIL concurrency defect remains open. Starter rendering, installed native startup, broader engine/performance acceptance, unavailable hardware qualification and draft review remain separate work. No PR is merged.
Draft #166 and its evidence report are published. The installed web command now runs through Node on Windows, preserves compiler failures and requires fresh build outputs. The original Bash-shim failure and initial hosted directory-copy failure are retained.
The final asynchronous-copy implementation passes all nine command regressions and a fresh installed Perry-plus-engine WASM build locally. Hosted Tests run 34556841600 passes all 22 jobs, including the actual Windows pack/install/help check on Node 22.23.2, native engine build and all 20 example links. PR head and hosted merge checkout have identical Git trees.
Archive
portable-web-f30ebd0-local-hosted-20260911-v1.zip(1,805,739 bytes), SHA-256f0d74f30f4ff5c04b9a3a4c5e64d33c8b8f5765f264a9cd7fed34328ef7a54a2. All payload hashes, ZIP CRC and both uploaded asset digests are verified.This is installed command/build evidence. Compiled-game browser rendering, the full starter and shared lifecycle remain open; native installed startup is covered separately by draft #167. Stable npm 0.4.16 still has the old entry point. No package is published to npm and no PR is merged. Radeon work continues; RTX-specific qualification remains unperformed.
Draft #167 and its evidence report are published. The installed Windows package now finds Jolt's published win32-x64 archives and omits redundant final-link metadata that pointed to an absent CMake directory.
A fresh actual package installs, links, runs a falling Jolt body and captures all 16,384 expected pixels on Radeon DX12 and Vulkan. Both PNGs are byte-identical. Hosted Tests run 34557843801 passes all 22 jobs, including the fresh installed native physics/image check on Windows, native engine build and all 20 example links. Hosted source trees match the PR head. The hosted startup uses Microsoft Basic Render Driver; it is not physical-GPU performance evidence.
Archive
installed-native-73b1e42-local-hosted-20260911-v1.zip(2,363,657 bytes), SHA-2562b53e68cf459bb5d8c6efb74384c0791f11795e43e97fba36ac3cf7f808b8f0a. Every payload hash, ZIP CRC and both uploaded asset digests are verified. Earlier path, harness, link and missing-capture failures remain explicitly distinguished in the report.The acceptance is native headless startup using the qualified Perry 0.5.1220 runtime profile. Visible presentation, packaged DXC, general long paths and the complete native/browser starter remain open. Direct-2D capture is handled in draft #168. No PR is merged or npm package released; RTX-specific qualification remains unperformed.
Draft #168 and its evidence report are published. Direct native frames now service queued output captures using the existing readback implementation. Render-target-only frames keep the request pending; scene diagnostic captures wait for a scene frame.
A fresh package at dc77ba5 renders the exact Jolt physics fixture through scene and direct-2D paths on Radeon DX12 and Vulkan. All four PNGs match all 16,384 expected pixels and are byte-identical. Both backends also pass all five render-target tests and all 93 goldens with original assertions. The initial WASM module-guard error is corrected and its failed/passing checks are retained.
Hosted Tests run 34559712338 passes all 22 jobs, including actual installed scene/direct-2D captures on Microsoft Basic Render Driver, the native build and all 20 example links. The hosted merge checkout and PR head have identical Git trees. Windows goldens skip the CPU adapter; its installed captures are software-rendering startup evidence, not physical-GPU performance results.
Archive
direct-capture-dc77ba5-local-hosted-20260911-v1.zip(2,770,207 bytes), SHA-256f0f2a03b41d1f926c900a9901831f2a96a5311cab55f22c63e60f0b7f542dd34. All payload hashes, ZIP CRC and both uploaded asset digests are verified.Visible presentation, browser startup/capture, packaged DXC and full starter/lifecycle acceptance remain separate. Draft #169 advances shared cleanup and canonical example startup. No PR is merged or npm package released; RTX-specific qualification remains unperformed.
Draft #169 and its evidence report are published.
runGame(updateAndDraw, cleanup)now supports final cleanup on native and web. Pong uses that shared loop and edge-triggered pause input. All 51 invalid palette references in six examples are corrected and checked against the exported palette.A fresh installed package simulates Jolt, produces all 16,384 expected pixels and records exactly one cleanup in scene/direct-2D modes on Radeon DX12 and Vulkan. A real callback-only native compatibility check passes. Compiled Pong records
RPRPRPRPRwith the old held-key pause check andRPPPPPPPPwith the correction. All 20 native examples link; corrected Pong and the engine complete the web build. The initial palette crash and intermediate input-staging audit mistake remain in the archive.Hosted Tests run 34561587869 passes all 22 jobs. Both installed Windows rendering modes produce the exact image and one cleanup on Microsoft Basic Render Driver. The merge checkout and PR head have identical Git trees. Windows goldens skip the CPU adapter; installed captures prove software-rendered startup, not physical-GPU performance.
Archive
starter-lifecycle-b43ea2c-local-hosted-20260911-v1.zip(4,558,218 bytes), SHA-256db70fbe196446d3648c238c5e770ceaeabcb8b28282e8012f08c3cdf447e928d. All payload hashes, ZIP CRC and uploaded asset digests are verified.Draft #170 adds actual Perry-compiled browser acceptance; its result is still pending and its separate Windows shared-test job has an access violation despite the FXC setting. This publication reports #169's exact successful run, not a resolution of that continuing instability. Browser rendering, visible native presentation, packaged DXC, fixed updates and the complete starter remain open. No PR is merged or npm package released; RTX-specific qualification remains unperformed.
Draft #170 now passes all 22 hosted Tests jobs at
9e6f4ab82f50db97d66385adf4c67f4e08fdae07; its report and evidence are published.The real Perry-compiled game renders eight frames, records one cleanup and produces all 16,384 expected pixels in hosted macOS Chrome. A compiled FFI call with an explicitly injected startup error is correctly rejected. The PR head and hosted checkout have identical Git trees. Physics is omitted from this rendering fixture; starter assets/text, browser physics and the full example runtime matrix still need acceptance.
The archive retains the initial compiled-game timeout caused by unresolved imports despite a successful compiler exit, plus the original Windows shared-test access violation despite FXC. The corrected preparer verifies exact dependency resolution and actual WASM imports. The serial Windows library passes 489 tests with one existing ignored test; all 20 native examples link and both fresh installed Windows rendering modes produce the exact frame and one cleanup on Microsoft Basic Render Driver. Windows raster goldens skip the CPU adapter. The serial setting is a qualified CI mitigation, not an identified crash root cause. General Perry throw propagation remains a separately observed limitation.
Archive
compiled-web-9e6f4ab-local-hosted-20260911-v1.zip(3,943,175 bytes), SHA-256d8989d608b8fb5d790fdd1088bdfedc662b511e56a10163e58d6c495551dd507. ZIP CRC, every included payload hash and uploaded asset digests are verified. No PR is merged or npm package released; RTX-specific qualification remains unperformed.Draft #171 passes all 22 hosted Tests jobs at
55b6cf96bbe6eff3fd2280126c1a7fd0a913f966; its report and evidence are published.The installed package now provides
bloom new, native build/run, and web build/serve with one source template and an asset example. Default creation carries the exact engine archive into the new project, avoiding an older registry package with the same version. Hosted creation verifies installed source identity and rejects missing-compiler and unsupported-target controls. PR head and hosted checkout have identical Git trees.Local candidate acceptance builds the unmodified native starter, runs a bounded copy through
npm start, renders its greeting and square at 800x450, reads the asset and cleans up once on Radeon DX12. Restoring the unmodified source completes the full web build. Separate local default-creation and managed-cache checks pass. The retained evidence distinguishes those candidate runs from the final committed hosted packaging result.Hosted native engine build, all 20 example links, both installed startup captures and the minimal #170 compiled-game browser check pass. Full starter asset/text browser rendering, visible native presentation, fresh-machine setup through this CLI, other native starter platforms, fixed updates and distributable shader runtimes remain open at this source. The new commands are draft package content, not a stable npm release.
Archive
starter-cli-55b6cf9-local-hosted-20260911-v1.zip(3,145,956 bytes), SHA-2568051a5bb6df184b611fc03b9be94c7f81c664a3e70d9fede12b38ca97a27f678. ZIP CRC, every included payload hash and uploaded asset digests are verified. No PR is merged or npm package released; RTX-specific qualification remains unperformed.Draft #172 passes all 22 hosted Tests jobs at
eca01e39e476e191ce8cd29bf130ec194d09fa02. Its report and evidence are published.The optional fixed lifecycle provides bounded simulation ticks, variable updates, interpolation and one-time cleanup. Actual native and WASM execution agrees on timing, hook order, invalid/overflowing inputs, early stop and cleanup. The original callback-only loop stays supported.
Hosted fresh installed scene/direct-2D/lifecycle modes each pass all 16,384 expected pixels and clean up once. The real compiled browser fixture passes its exact image, eight updates/draws, advancing fixed ticks, one cleanup and explicit FFI startup-fault control. All 20 examples compile/link, and installed default starter creation plus setup-error controls pass. Local Radeon DX12/Vulkan lifecycle captures and the revised starter's bounded native render/asset/cleanup run also pass.
The PR and hosted checkout have identical Git trees. The native package report preserves
source_dirty: true; its seven reported installed runtime hashes match the PR with Windows newline conversion. The report does not claim a wholly clean native build checkout. Hosted software DX12 rendering remains distinct from physical Radeon measurements.The full starter's successful web build in #172 did not prove execution: later runtime work found a named-void-callback conversion failure addressed by draft #173. Full starter browser acceptance, example runtime coverage, pause/focus/device-loss behavior, components, visible presentation and wider packaging/graphics qualification remain open. No PR is merged or npm package released; RTX-specific acceptance remains unperformed.
Archive
fixed-game-lifecycle-eca01e3-local-hosted-20260911-v1.zip(3,492,597 bytes), SHA-25662842982376bc923a0ed04ca2df6af49c86141119602bbf57f56a7a9afe056a2. ZIP CRC, every included payload hash and uploaded asset digests are verified.Draft #173 now passes all 22 hosted Tests jobs at
955ac5b92baa09ab4ba3ba39bc71e646f578b2b1. The complete installed starter report and evidence are published.The exact 11-file website generated by the installed
bloom run --webcommand passes real Chrome execution: successful welcome-asset fetch/read, eight opaque-black clears and white text/square draws, the expected 800x450 capture, normal stop and one cleanup. The existing compiled-game image/fault control, installed native scene/direct/lifecycle checks and all 20 native example links also pass.The evidence retains the original named-void callback failure, omitted-hidden-file rejection and invalid-color first-frame failure. The final bootstrap preserves actual void returns, the starter uses explicit RGBA literals, and its verifier checks real draw/clear arguments. Local Radeon native rendering and the corrected installed web command also pass.
Hosted and PR Git trees match. Installed web/native reports keep their dirty-source flags; eleven web and seven native reported installed sources match the PR with recorded Windows newline/npm shebang conversion. This does not establish a wholly clean native build checkout or clean-machine distribution.
General public-constant and canonical-example runtime corrections are being qualified in #174. Full gameplay, browser physics, pause/focus, resize/device loss, visible native presentation, wider packaging/graphics acceptance and named RTX hardware requirements remain open. No PR is merged or npm package released.
Archive
installed-starter-browser-955ac5b-local-hosted-20260911-v1.zip(5,500,913 bytes), SHA-2565a8e79c4373fc5f16950bd9204f9078fdfcd902814378a001b6cd20d99cd0caa. ZIP CRC, all 491 payload hashes and uploaded asset digests are verified.Draft #174 now passes all 22 hosted Tests jobs at
c414595e016b37e8087928d6c2b37d282497bf47. The portable example native/browser report and evidence are published.All six games pass real browser content rendering, eight frames, normal stop and one cleanup. All six native content/capture/cleanup checks and all 20 native links pass locally on Radeon 760M/DX12 and in hosted Windows software rendering. Public native/WASM constants and alias contracts agree, and the pure-WASM guard rejects an actual engine call. Installed native and browser starter checks also pass.
The evidence preserves earlier failures. Cross-drive fixture placement produced colliding Perry globals despite successful MSVC linking; same-drive projects and a duplicate-global rejection gate correct that path. Pointer-lock denial no longer stops Voxel Sandbox, while retry and cleanup remain covered. Hosted/PR Git trees match, and reported files are hash-verified with recorded newline conversions; actual dirty-source flags remain visible.
Full gameplay/input interaction, the broader canonical runtime matrix, graphics reconstruction, visible presentation, clean distribution and wider API/streaming/components/UI work remain open. The separate local line-rendering candidate still fails quality bounds. Named RTX 4080 and physical constrained-adapter acceptance remain unperformed. No PR is merged or npm package released.
Archive
portable-examples-c414595-local-hosted-20260911-v1.zip(10,532,340 bytes), SHA-256dae76a5658444ae6416410cbeb5c7e85d7f97b44407ac0f39e15b6133dcc06a9. ZIP CRC, all 766 payload hashes and uploaded asset digests are verified.
Parent: #126
Related: #24
Problem
Bloom has useful pieces—golden tests,
tools/bloom-reference,tools/cycles_reference, realtime examples, andtools/validate.sh—but not one repeatable qualification workflow.tools/validate.shis report-oriented, reference cameras/configuration are not a versioned contract, and performance measurements can be hidden by vsync or warm-up behavior.Without a stable corpus, “UE-quality,” “looks better,” and “costs 1 ms” are not independently verifiable claims.
Outcome
A deterministic command and machine-readable result format that compares realtime Bloom output against approved baselines/reference renders across fixed scenes, cameras, quality tiers, motion sequences, and adapters.
Required design
Manifest
Add a versioned manifest (for example
tools/quality/scenes.toml) with, per case:Corpus
Start with:
examples/pbr-spheresfor BRDF/material lobes;examples/sponzafor interior GI, shadowing, leakage, and temporal behavior;examples/bistrofor scale, sun/sky, streaming pressure, and exterior materials;DamagedHelmet.glbfor canonical glTF material fidelity;Every scene needs at least one still pose and relevant features need a short motion sequence. Assets that cannot live in Git must be fetched by content hash with a license/source manifest.
Runner and outputs
result.json, final/intermediate PNGs, heatmaps, and a compact HTML/Markdown summary.--report-onlyexplicitly for local exploration.Baseline governance
Acceptance criteria
tools/validate.sh, Lumen follow-up: benchmark harness for SSGI quality / convergence #24, and existing reference tools are reused or superseded with migration notes rather than left as competing workflows.Likely files
tools/validate.shtools/bloom-reference/tools/cycles_reference/tools/quality/native/shared/src/renderer/.github/workflows/test.ymlor a dedicated scheduled/hardware workflowVerification
The PR must attach one complete result bundle from each available backend and demonstrate an intentional failure.
Non-goals
Dependencies