fix(preprocess): attribute macro-expanded tokens to the use site

[e54-bot]

Summary

Fixes #506.

expand_macro re-lexes replacement text, then shifted every generated token's span by the invocation's start position. A replacement wider than its invocation produced token spans ending past the authored line, and multi-line expansions attributed generated tokens to unrelated authored lines. Those fabricated provenance spans surfaced downstream as invalid-span validation failures in Wright lint-fix flows on real projects (composition/bootstrap.opy:64, overwatch-ai-pve/utilities/hero_switch.opy:9).

Since expanded text does not exist in authored source, the authored macro name is the only honest provenance: every expanded token — in object/function macros and script macros alike — now carries the use-site span.

diag::shift_position remains in use where positions genuinely correspond to text (includes, scripts, settings blocks) and is untouched.

Test plan

• New: expanded_tokens_carry_the_use_site_span — a replacement far wider than T attributes every expanded token to the authored use-site span 3:9-3:10, which stays within the authored line.
• Updated: multiline_define_preserves_relative_expansion_spans → multiline_define_expanded_tokens_attribute_to_the_use_site — the old assertions pinned the buggy shifted coordinates (second expansion token on line 4 for a one-line invocation on line 3); now pins both tokens to the invocation span.
• cargo fmt --all -- --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, cargo test --workspace --all-targets --all-features (all green), python3 -m unittest discover -s tools/overpy/tests (40 tests OK), differential corpus native_and_reference_agree_on_the_corpus passes.

[e54-bot]

Independent review of the diff against issue #506. The use-site direction matches the documented contract (docs/opy/provider.md: expanded code maps to the invocation site), all shift_expansion_spans call sites are migrated, and the tests do discriminate against the old behavior. Two findings:

1. Blocking — uniform spans destroy the layout the indentation-sensitive parser requires. Token.span carries two roles: provenance and the line/column layout that block parsing reads (parser.rs:519-531 block_indent, parser/statements.rs:4-25 parse_block, parser/statements.rs:348-368 parse_colon_body, parser/definitions.rs:57-72, parser/declarations.rs:100,205). The old shifted spans preserved the expansion's relative layout — which is why expand_macro/expand_script re-indent continuation lines (macros.rs:284-288, scripts.rs:78-79). Attributing every token to use_site collapses all columns to one value, so multi-line #!define and __script__ expansions containing indented blocks regress, verified against base 12f3b5d:

   • #!define check() if A == 1:\ + B = 1\ + C = 2 invoked at rule-body indent: base → if body has 2 stmts, 1 action; this PR → if body has 1 stmt and C = 2 escapes as an unconditional sibling action (silent semantic miscompile).
   • #!define gen() for i in range(0, 3):\ + B = i → parse-error: expected an indented block after ':' (base parses through to lowering).
   • #!define makerule() rule "generated":\ + @Event global\ + A = 1 → same hard parse error (base compiles to a rule).
   • Same for script macros: __script__ returning "if A == 1:\n B = 1\n C = 2" → if body loses C = 2.
     for/while/do/switch/rule/def/enum bodies hard-fail via expect_block_indent; if/elif/else silently mis-parse via the same-indent inline-statement path at statements.rs:353. Multi-line defines preserving statement boundaries are a supported contract (compiler/tests/preprocessing.rs:55-86). Required correction: keep use-site provenance while preserving expansion layout for the parser (e.g., a per-token layout position separate from the provenance span). Note the issue's own 'bound to the use-site span' direction would hit the same collapse, so the representation likely needs an owner/architect decision.

2. Residual same-class site — f-string interpolations inside expanded strings still fabricate positions past the authored line. parser/expressions.rs:700-717 computes interpolation origin/error spans as string_span.start.col + index and parser.rs:559-593 (parse_expression_fragment) shifts inner expression tokens from there. With string_span now the use-site span, #!define M f"hp: {B +}" invoked at 3:9-3:10 reports expected an expression but found '' at 3:17 — outside both the use-site span and the 9-column authored line. Pre-existing under the shifted regime, but it's the same invalid-span class #506 targets and survives this PR; bounding those derived spans to string_span would be a no-op for authored strings and fixes the expanded case.

[e54-bot]

Reworked in 430547d per the blocking finding — the layout/provenance split the review suggested:

• Token now carries span (authored provenance; the use-site span for expanded tokens) and a crate-internal layout (the shifted coordinates the parser reads for indentation and statement-boundary decisions). Parser decision reads — block_indent, expect_statement_end, newline_lines, colon-body/switch/case/enum/subroutine indent checks, the if line check — read layout; AST spans and diagnostics keep reading span, so expanded constructs still report the authored use site.
• layout anchors at the invocation token's own layout position, so nested expansions keep relative structure.
• The f-string finding is fixed in the same commit: parse_expression_fragment clamps derived provenance into the string span (no-op for authored strings) — #!define M f"hp: {B +}" now reports inside 6:9–6:10.

Verified against your repros: check()/gen()/makerule() all keep block structure (new tests: multiline_function_define_preserves_block_structure, ..._for_block, script_macro_multiline_output_preserves_block_structure with a blockgen.js fixture, expanded_f_string_interpolation_errors_stay_inside_the_use_site). Full suite green: fmt/clippy, cargo test --workspace --all-targets --all-features, python oracle suite (40), differential corpus.

[e54-bot]

Re-review of 430547d. The span/layout split is the right shape and both prior findings are verified fixed end-to-end against base 12f3b5d:

• #!define check()/gen()/makerule() multi-line defines keep block structure (if body keeps both statements; for/rule/elif/else/switch/case/default parse and emit identically to base); __script__ blockgen output preserves the if body; nested INNER()-inside-OUTER() anchors at the invocation's virtual position correctly.
• Provenance: expanded tokens' span is the use site — AST spans on expanded constructs report the authored invocation (check() nodes all report 11:5–11:10), an error inside an expansion reports the macro name (8:5), and the f-string interpolation error clamps into the use site (8:10) instead of fabricating column 17.
• The layout == span invariant for authored tokens holds at every construction/mutation site: Token::new, attribute_expansion_spans (nesting anchors at use_site.layout — correct), expand_settings, parse_expression_fragment, RulePrefixMarker. All other .span position reads are either provenance construction or authored-domain-only (token_is_at_or_after, scope_col, is_multiline, raw_source_text — whose re-lex check correctly falls back for expanded args).
• Test discrimination: the span/f-string tests fail on base 12f3b5d, and the block-structure tests fail on the first (uniform-span) approach 59e87fa.

One new finding — a residual mixed-domain read of the same class the split was meant to fix:

1. crates/opy-rs/src/parser/expressions.rs:16-17 — the conditional-expression guard compares peek().layout.start.line (layout domain) against then_value.span().end.line (provenance domain) and feeds that same provenance line into newline_after_line, whose entries are layout lines. For an expanded then_value, the provenance end is the use-site line while an adjacent if sits on the expansion's virtual line, so the guard wrongly fires and the if is reparsed as a statement if. Verified repro:

   #!define E() A = 0\
       B = 0\
       y = B if C else D
   
   rule "r":
       @Event global
       E()

   Base 12f3b5d compiles this to Set Global Variable(y, If-Then-Else(Global.C, Global.B, Global.D)); 430547d hard-fails with expected ':' after the if condition at the E() use site (13:5). Same failure for the nested-tail variant (B TAIL() where TAIL() expands to if C else D) and for the bare-statement form B if C else D on expansion line ≥3. The trigger is an unparenthesized X if C else D on a virtual expansion line below the use-site line with an earlier expansion newline already consumed — a plausible multi-line define shape, and a parse regression on input base accepted.

   Correction: evaluate the guard entirely in layout coordinates. self.tokens[self.pos - 1].layout.end.line is exactly the value base compared — then_value.span().end.line equals the token adjacent to if — for both the != check and the newline_after_line line argument (the file argument is unaffected since layout.file == span.file for every token). This is also the only remaining mixed-domain site; the audit found no others. Add a regression test for a ternary on a later line of a multi-line #!define — none of the new tests cover this guard path.

[e54-bot]

Fixed in 161e8a0: the ternary statement-boundary guard now reads the just-consumed token's layout on both sides (the pre-split span carried exactly that), covering both the != line check and the newline_after_line argument. Verified the reviewer's repro fails on 430547d (expected ':' after the if condition at the use site) and compiles on the fix; regression test added (multiline_function_define_preserves_conditional_expressions). Workspace tests + fmt re-verified.

[e54-bot]

Re-review of the rework at head 161e8a0 (430547d + the conditional-guard fix). I audited every .span/ .layout read in the parser and preprocessor and ran ~20 adversarial compiles through opy-cli against both commits. Verdict: the rework is correct; approve-quality (self-approve blocked by identity).

Finding (already resolved at head): at 430547d, parse_expr_inner's statement-boundary guard mixed coordinate spaces — peek().layout.start.line vs then_value.span().end.line. Since expanded AST spans collapse to the use site, any conditional expression on a non-first expansion line misfired: a #!define producing y = B if C else D on its third line failed with expected ':' after the if condition (reproduced on a clean 430547d build). 161e8a0 fixes it correctly by comparing the just-consumed token's .layout on both sides — verified the same case now lowers to If-Then-Else, and the regression test covers it. This was the only span/layout mix I could find in structural code.

Audit coverage:

• Every parser structural read now uses .layout: block_indent, newline_lines recording, expect_statement_end's continued-line check, if/elif/else/else-if columns, open_if_indents, switch case/default cols, for/while/do/do-while headers, enum/macro/subroutine/rule body indents, and (post-161e8a0) the conditional guard.
• Token construction sites all initialize layout correctly: lexer Token::new (layout = span), attribute_expansion_spans (layout = shifted coords anchored at use_site.layout.start, so nested expansions preserve structure), expand_settings re-lexed tokens (layout = shifted span), parse_expression_fragment (layout shifted, provenance clamped by bounded_span), RulePrefixMarker (inherits).
• The span.file == layout.file invariant holds at every site, so newline_lines.entry(token.span.file) and fragment file keys are safe.
• Deliberate .span reads are in the right domain: token_is_at_or_after only sees authored pre-expansion input; raw_source_text slices authored source and its lexical verification falls back safely for nested-expansion args; optimization_state_at evaluating directive state at the use-site span is semantically the right anchor (arguably better than the pre-PR shifted positions).
• bounded_span clamps through Position: Ord (line-major); monotone, cannot invert a span.

Falsification evidence at 161e8a0: multi-line if/elif/else/for/while/do/switch defines, rule/enum/macro-decl generation, __script__ macros (including nested inside a text-macro expansion), 2- and 3-level nested defines with ternaries on later expansion lines, mid-line multi-line invocations (x = M + y splits identically to textual substitution), f-string interpolations containing ternaries inside expanded strings, #!include-sourced defines, settings-block define expansion, #!defineMember. Diagnostics inside multi-line expansions report the authored use site — B = nosuchvar on expansion line 3 reports at the invocation (10:5), not fabricated coordinates. 9/13 real-world corpus projects compile; the 4 failures (6v6-adjustments, overpy-meipocalypse, overpy-zencopter, ow1-emulator) are recorded known gaps in fixture.json (nativeStatus: failure) — ow1-emulator's Found 'if', but no 'else' is the designed continued-inline-if rejection, identical pre-fix.

cargo test -p opy-rs all green; fmt/clippy clean. Non-blocking nit: TokenKind::Indent is a dead variant (pre-existing, unrelated to this change). The public contract in docs/opy/tooling-api.md ('expansion stamps expanded tokens with the use-site span') is unchanged; layout is crate-internal, so no doc update is needed.