Rotation fixes: ?rotmg negative-D1 hang, ?rotg/?rotmg NaN canonicalization, -O1 test bug

[sigilante]

Fixes the three open issues, all found by the RustFloat differential fuzzer.

#33 — ?rotmg loops forever when d1 goes negative

The first rescale loop's guard, d1 != 0 && d1 <= rgamsq, never goes false for a negative d1: multiplying by gamsq keeps it negative and pins it at -inf, which is still <= rgamsq. A negative d1 is reachable from valid-looking inputs — a negative d2 whose q2 = (d2*y1)*y1 underflows to -0 slips past the q2 < 0 singularity test, and the flag = 1 branch then swaps it into d1.

The guard is now 0 < d1, matching the Rust port, so the loop terminates however d1 became negative. All four precisions.

#34 — ?rotg and ?rotmg do not nan_unify their outputs

These were the only routines writing outputs without nan_unify_*, so a NaN came back carrying SoftFloat's propagated payload and sign instead of SINGNAN/DOUBNAN/HALFNAN/QUADNAN. The stores are now unified, ?rotm-style.

• ?rotmg unifies only the values it actually writes: the p2 == 0 early return still leaves D1/D2/X1 untouched, and only the P[1..4] entries the flag selects are written.
• ?rotg unifies c and s before a and b, preserving the previous store order for an aliasing caller.

One correction to the issue. The specific drotg sign leak it reports (a = 0xfff8000000000000) does not reproduce on an 8086-specialization build, which is what this repo builds on both CI platforms: f_abs clears the NaN's sign before it reaches the arithmetic, so that input canonicalizes without the fix. The payload leak reported for srotg does reproduce. The ?rotg tests here therefore use signaling NaNs, which leak on this build (0x7ff4… came back as 0x7ffc…, payload intact). The sign leak was presumably observed against the ARM-VFPv2 build.

#35 — test_cdotu_rounding_modes reads uninitialized stack at -O1

An in-loop compound literal's lifetime ends with the iteration, and clang at -O1 sank the initializing stores for the second cvec argument past the loop's back-edge, so the first iteration multiplied by stack garbage. The four operand literals are hoisted out as static arrays, as the rest of the file does.

Testing

• 325 of 325 pass at -O0, -O1 and -O2 (313 before, 12 new). Registration lint passes.
• All 12 new tests fail against the pre-fix sources, confirmed by linking the original eight files from master: the eight NaN tests fail, and the four ?rotmg termination tests hang until killed.
• Both defects were reproduced before being fixed — the srotmg example from ?rotmg loops forever when d1 goes negative (reachable via negative d2 with q2 underflowing to -0) #33 was still running after 3 s, and the -O1 assertion failed on every run.

#33 and #34 share a commit because they touch overlapping hunks in the same four ?rotmg files.

🤖 Generated with Claude Code