Skip to content

Compare LLVM and Whale results with shared semantic fixtures #773

Open

Description

@LunaStev

Motivation

PR #859 added the opt-in --whale IR adapter. PR #860 adds checked numeric lowering and serialized IR verification. These tests do not establish execution parity. Existing semantic execution fixtures primarily exercise LLVM; the full shared comparison harness remains unimplemented. The newer upstream bounded integer/tracked-memory oracle enables a narrower adapter test step tracked in #869, after dependency integration #868.

Starting points: src/cli.rs, tests/fixtures/typed_conversions/main.wave, tests/codegen_regressions.rs.

Scope

Design a harness that builds/runs the same source, input and target through backend adapters and compares defined stdout, exit and memory results. An LLVM baseline and explicit missing-backend outcomes can be implemented first; actual two-backend execution depends on a Whale adapter. Keep this scoped to language semantics, separate from ABI fixture design (#368) and the wider migration tracker (#93).

Difficulty: very hard.

Completion criteria

  • Run identical fixtures under the same selected semantic/target contract.
  • Cover conversions, aggregates and control flow.
  • Distinguish unsupported capability, missing backend, compilation failure and semantic mismatch.
  • Retain a minimal reproducible fixture for discrepancies; do not report skipped Whale runs as parity.

Design decision needed: Agree on adapter interfaces, common target/feature coverage and comparison rules before implementation.

Please mention @LunaStev to discuss the design before starting implementation.

Related implementation: #861, #862, #863 and #864. Keep this issue open until the complete shared comparison scope passes.

Activity

  1. added
    enhancementA request for a new feature or improvement.
    help wantedThe issue requires extra attention or help from others.
    discussionIssues that require a technical or community discussion.
    needs testingIssues that require additional testing or verification.
    cross-platformIssues related to making Wave work seamlessly across multiple operating systems.
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cross-platformIssues related to making Wave work seamlessly across multiple operating systems.discussionIssues that require a technical or community discussion.enhancementA request for a new feature or improvement.help wantedThe issue requires extra attention or help from others.needs testingIssues that require additional testing or verification.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions