Repository navigation
Compare LLVM and Whale results with shared semantic fixtures #773
Description
Motivation
PR #859 added the opt-in --whale IR adapter. PR #860 adds checked numeric lowering and serialized IR verification. These tests do not establish execution parity. Existing semantic execution fixtures primarily exercise LLVM; the full shared comparison harness remains unimplemented. The newer upstream bounded integer/tracked-memory oracle enables a narrower adapter test step tracked in #869, after dependency integration #868.
Starting points: src/cli.rs, tests/fixtures/typed_conversions/main.wave, tests/codegen_regressions.rs.
Scope
Design a harness that builds/runs the same source, input and target through backend adapters and compares defined stdout, exit and memory results. An LLVM baseline and explicit missing-backend outcomes can be implemented first; actual two-backend execution depends on a Whale adapter. Keep this scoped to language semantics, separate from ABI fixture design (#368) and the wider migration tracker (#93).
Difficulty: very hard.
Completion criteria
- Run identical fixtures under the same selected semantic/target contract.
- Cover conversions, aggregates and control flow.
- Distinguish unsupported capability, missing backend, compilation failure and semantic mismatch.
- Retain a minimal reproducible fixture for discrepancies; do not report skipped Whale runs as parity.
Design decision needed: Agree on adapter interfaces, common target/feature coverage and comparison rules before implementation.
Please mention @LunaStev to discuss the design before starting implementation.
Related implementation: #861, #862, #863 and #864. Keep this issue open until the complete shared comparison scope passes.
Activity