Skip to content

Fix Hermitian upper-triangle handling in native ELPA and cuSolver - #7923

Open
MagTheoryLab wants to merge 7 commits into
deepmodeling:developfrom
MagTheoryLab:codex/fix-lcao-eigensolver-hermitian
Open

Fix Hermitian upper-triangle handling in native ELPA and cuSolver#7923
MagTheoryLab wants to merge 7 commits into
deepmodeling:developfrom
MagTheoryLab:codex/fix-lcao-eigensolver-hermitian

Conversation

@MagTheoryLab

Copy link
Copy Markdown

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue or explained why this PR does not need one.
  • I have added adequate unit tests and/or case tests, or explained why not.
  • I have listed the exact verification commands run and their results.
  • I have described user-visible behavior changes, including INPUT parameter changes.
  • I have explained core-module impact for ESolver, HSolver, ElecState, Hamilt, Operator, Psi, or other source/ changes.
  • I have requested any needed governance exception below.

Linked Issue

Related to #7914. This PR addresses the same authoritative-upper-triangle matrix contract in the separate native elpa and cusolver implementations, so no separate issue is required.

Dependency and attribution: the current branch also includes the two genelpa commits from #7914 (e27ef6629938d04ea1649d4779ad13a3931bc6eb and 53208bf2713db64359315d3f0359adfefbbb4a38). Their original author, TaoXia, author dates, and cherry-pick source trailers are preserved. The Pauli/y change from #7914 is not included. These imported changes belong to #7914; this branch should be rebased after that PR merges so they are not reviewed or merged twice. The native ELPA/cuSolver fixes and the LAPACK test-workspace correction are authored by Chen Chengbing.

Unit Tests and/or Case Tests for my changes

Reproduction: preserve the authoritative upper triangles and deliberately overwrite lower-triangle entries. Compare eigenvalues against LAPACK using an independent, unchanged fixture. For the before/after controls, link the same regression tests with either the previous solver object from develop (fb9ce1da4) or the corrected solver object; other objects within each comparison remain unchanged.

Result summary:

Test Previous implementation Corrected implementation
Complex native ELPA, block sizes 1/2/3, one rank 3/3 failed 3/3 passed
Real and complex cuSolver 2/2 failed 2/2 passed
ScaLAPACK controls in the cuSolver test group 4/4 passed 4/4 passed

Corrected native ELPA also passed all three tests with four MPI ranks. Its tests check eigenvalues to 1e-10 and verify that caller-owned H/S buffers remain unchanged across two consecutive solves. After importing the genelpa commits, native ELPA again passed 3/3 on one rank and 3/3 on four ranks, and the cuSolver group passed 6/6. The imported genelpa upper-triangle regression passed on one rank.

Commands run: SOURCE_DIR denotes the source snapshot, BUILD_DIR the configured CUDA build, and NEGATIVE_BUILD_DIR the separate directory containing the old-solver control executables.

export OMP_NUM_THREADS=1
cmake --build "$BUILD_DIR" -j8 --target MODULE_HSOLVER_LCAO MODULE_HSOLVER_LCAO_cusolver abacus_basic_gpu
"$BUILD_DIR/abacus_basic_gpu" --version
cd "$SOURCE_DIR/source/source_hsolver/test"

"$BUILD_DIR/source/source_hsolver/test/MODULE_HSOLVER_LCAO" --gtest_filter='BlockSizes/DiagoElpaNativeUpperTest.*'
mpirun -np 4 "$BUILD_DIR/source/source_hsolver/test/MODULE_HSOLVER_LCAO" --gtest_filter='BlockSizes/DiagoElpaNativeUpperTest.*'
"$BUILD_DIR/source/source_hsolver/test/MODULE_HSOLVER_LCAO_cusolver"
"$BUILD_DIR/source/source_hsolver/test/MODULE_HSOLVER_LCAO" --gtest_filter='DiagoElpaComplexTest.*'

"$NEGATIVE_BUILD_DIR/native-old" --gtest_filter='BlockSizes/DiagoElpaNativeUpperTest.*'
"$NEGATIVE_BUILD_DIR/cusolver-old"

The build succeeded and reported ABACUS v3.11.0-beta9. Positive filtered tests returned 0; both negative-control executables returned the expected GTest failure status 1. Environment: Tesla V100-SXM2-32GB, GNU C++, CUDA 12.9.1, OpenMPI 5.0.10, ELPA 2026.02.001, and LibXC 7.0.0.

Additional checks and limitations:

  • git diff --check passed. Governance checks reported no errors; documentation/test-evidence warnings are addressed by this body.
  • The complete MODULE_HSOLVER_LCAO group and the imported genelpa regression on four MPI ranks were attempted but hit an ELPA two-stage CPU-path failure. An independent program containing no ABACUS objects reproduced the failure. Crash addresses map to CUDA-aware MPI branches passing uninitialized device-buffer pointers during CPU execution. A diagnostic one-stage variant passed the full group on one rank (10/10) and four ranks (9 passed, 1 existing LAPACK test skipped). This variant is not part of the PR; the installed library has not been repaired.
  • A supplied Fe2 LCAO/SOC case with default GGA exhausted 200 SCF iterations for native ELPA CPU, native ELPA GPU, and cuSolver GPU. Final density residuals were 4.5907e-5, 1.7279e-5, and 1.7026e-5, above scf_thr=1e-6. This PR establishes the matrix-contract correction; it does not claim to resolve that remaining SCF issue.
  • Full repository testing was not run because this change is scoped to dense eigensolver input handling and its regression helpers. Test fixtures and tolerances were not relaxed.

What's changed?

LCAO supplies authoritative upper triangles for Hermitian H/S matrices. Previously, native ELPA could consume stale lower entries from the original full complex buffers, while cuSolver explicitly selected the lower triangle. Consequently, otherwise valid upper-triangle inputs could produce incorrect eigenvalues.

  • Reconstruct private Hermitian H/S copies from the upper triangles before complex native ELPA solves, preserving caller buffers and avoiding reuse of a decomposition of temporary overlap storage.
  • Select the upper triangle for real and complex cuSolver solves.
  • Add poisoned-lower-triangle regression tests and allocate the LAPACK test workspaces to match their advertised lwork.
  • Include TaoXia's two genelpa fixes as the dependency described above.

No gga_grad functionality or GPU performance refactor is included.

Governance Notes

  • INPUT/docs changes: no parameter names, defaults, formats, or public interfaces change. Parameter documentation updates are not required for this correction of the existing matrix contract.
  • Core module impact: HSolver native ELPA input preparation, cuSolver triangle selection, the imported genelpa transform, and colocated test helpers. ESolver, ElecState, Hamilt, Operator, and Psi interfaces are unchanged. The real native ELPA decomposition-reuse behavior is unchanged.
  • Exceptions requested: none. The changes add no GlobalV, GlobalC, or PARAM dependencies and introduce no new direct MPI calls.

@mohanchen mohanchen added Feature Discussed The features will be discussed first but will not be implemented soon Refactor Refactor ABACUS codes labels Sep 8, 2026
mohanchen and others added 3 commits September 8, 2026 09:22
Project multi-k density matrices before star restoration to preserve the symmetry assumed by reduced EXX contractions. Add complex-density and spin-channel regressions and update the Si HSE reference after independent solver and full-contraction checks.

@mohanchen mohanchen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will handle this PR later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature Discussed The features will be discussed first but will not be implemented soon Refactor Refactor ABACUS codes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants