s_ibm_correct_state writes q_prim inside the loop that reads q_prim at image-point stencils: read-after-write race across ghost points

[sbryngelson]

Symptom

A moving immersed boundary evolves nondeterministically run to run, independently of #1886. Two identical restarts from the same checkpoint diverge.

Cause

The ghost-cell reconstruction in s_ibm_correct_state (src/simulation/m_ibm.fpp:148) writes the corrected primitive variables into q_prim_vf inside the same parallel loop that reads q_prim_vf at each ghost point's image-point stencil. Those stencils are not guaranteed to avoid other ghost cells: s_compute_interpolation_coeffs has an all-body fallback that can select a stencil containing them.

So whether ghost point A reads the pre- or post-correction value at a cell that ghost point B is writing depends on the order the two are scheduled. That is a race across ghost points, and the ordering is not fixed.

This is separate from #1886 (the stale device copy of num_gps). Fixing #1886 removes one source of run-to-run difference; this one remains.

Fix

Write the reconstruction's output to a per-ghost-point buffer — alpha_rho(1:nf), alpha(1:nf), p, c — and copy the buffer into q_prim_vf in a second pass after the loop, so no ghost point can read another's write.

Test that would catch it

Run any moving-IB case twice from the same checkpoint and compare the conservative variables. Bit-exactness is the right expectation; the existing GPU tests use translating bodies whose ghost-point sets barely change, which is presumably why neither this nor #1886 was visible to CI.

Found with Claude Code on OLCF Frontier.