Symptom
Two failures on the gain side of an IB ownership handoff, on Frontier with CCE OpenMP offload:
- the force relay's absorb kernel faults, and
- when it does not fault, the body's recorded force is identically zero for the steps after the handoff.
Cause
Both are the same shape: small host-side bookkeeping is kept on the device and the device copy is stale on a rank that has just started tracking a patch.
The relay. s_communicate_ib_forces (src/simulation/m_ibm.fpp:1384) is MPI plus O(num_ibs) arithmetic, but runs its absorb on the device, with a section update of the received ids and forces before each stage. On a rank whose device copy had never been written — one that just gained the patch — that update was not visible to the kernel, and the stale id faulted the lookup. A loud bounds check on the received ids is what showed it: the device copy did not hold what the section update had just written.
The lookup. s_update_ib_lookup (m_ibm.fpp:1657) has the same problem from the other direction: the read-back left ib_gbl_idx_lookup at -1 after a handoff, s_get_neighborhood_idx then returned -1, and the reduction's if (j > 0) silently dropped that rank's contribution — zero force for the body.
Fix
Neither belongs on the device. Build the lookup on the host and push it once; run the relay on the host and apply the summed forces to the device afterwards. What the relay was doing on the device every stage — three 2.6 MB updates, 36 kernels, 36 copy-backs — is pure overhead for arithmetic that is O(num_ibs).
Worth adding a consistency check per relay call while there: the failure mode is silent in one direction and a nil-pointer fault in the other.
Part of the moving-IB-across-ranks set filed today; see #1895, #1896, #1897, #1898.
Found with Claude Code on OLCF Frontier.
Symptom
Two failures on the gain side of an IB ownership handoff, on Frontier with CCE OpenMP offload:
Cause
Both are the same shape: small host-side bookkeeping is kept on the device and the device copy is stale on a rank that has just started tracking a patch.
The relay.
s_communicate_ib_forces(src/simulation/m_ibm.fpp:1384) is MPI plusO(num_ibs)arithmetic, but runs its absorb on the device, with a section update of the received ids and forces before each stage. On a rank whose device copy had never been written — one that just gained the patch — that update was not visible to the kernel, and the stale id faulted the lookup. A loud bounds check on the received ids is what showed it: the device copy did not hold what the section update had just written.The lookup.
s_update_ib_lookup(m_ibm.fpp:1657) has the same problem from the other direction: the read-back leftib_gbl_idx_lookupat-1after a handoff,s_get_neighborhood_idxthen returned-1, and the reduction'sif (j > 0)silently dropped that rank's contribution — zero force for the body.Fix
Neither belongs on the device. Build the lookup on the host and push it once; run the relay on the host and apply the summed forces to the device afterwards. What the relay was doing on the device every stage — three 2.6 MB updates, 36 kernels, 36 copy-backs — is pure overhead for arithmetic that is
O(num_ibs).Worth adding a consistency check per relay call while there: the failure mode is silent in one direction and a nil-pointer fault in the other.
Part of the moving-IB-across-ranks set filed today; see #1895, #1896, #1897, #1898.
Found with Claude Code on OLCF Frontier.