feat(cuda): probe cuda13 native libraries before cuda12 - #1440
Open
modelpath-dev wants to merge 2 commits into
Open
modelpath-dev wants to merge 2 commits into
modelpath-dev wants to merge 2 commits into
Conversation
Extend the cublas matrix and Gather Binaries layout so a Compile run can produce cu13.0.0 artifacts next to cu12.4.0 without changing the managed loader or NuGet packages yet.
Teach NativeLibraryWithCuda to try major 13 when SkipCheck is on or the detected major is 13, and cover the path order with unit tests. Binaries and NuGet packages stay out of this change.
This was referenced Sep 18, 2026
modelpath-dev
force-pushed
the
feat/1360-cuda13-loader
branch
from
September 26, 2026 11:56
82b1527 to
ac99b14
Compare
Member
|
Started a build action for version 815a2a5915f22ce6a760c676389c5dfe8535c08f (the current version of llama.cpp that we support). Once that's done I'll merge a PR to use those binaries. See: https://github.com/SciSharp/LLamaSharp/actions/runs/36282985190 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Part 2 of the CUDA 13 split for #1360. Once part 1 has produced
cu13.0.0binaries for development, the managed loader can prefer cuda13 before cuda12 without shipping new NuGet packages yet.Depends on #1439.
Scope
LLama/Native/Load/NativeLibraryWithCuda.cs: when SkipCheck is on (or major 13 is detected), try major 13 before 12 and 11.LLama.Unittest/NativeLibraryWithCudaTests.cs: cover unknown-major SkipCheck order, detected 13, detected 12, and SkipCheck off with unknown major.Out of scope: compile workflow, nuspecs,
LLamaSharp.Runtime.targets, docs.Tradeoffs
Stacked on the build branch so the history stays ordered. Against
mastertoday this PR also contains the compile.yml commit from #1439. After #1439 merges, rebase this branch and the reviewable diff is only the two loader files.Blast Radius
Changes which native CUDA directory
Prepare()tries first when probing. Without cuda13 binaries present, the new probe returns null and falls through to cuda12 the same as before.Verification
NativeLibraryWithCuda.cs,NativeLibraryWithCudaTests.cs.cuda13thencuda12thencuda11under SkipCheck). Could not rundotnet testlocally (SDK missing); CI on this PR is the live run.