Support for CUDA Tile C++ kernels - #404
Open
benvanwerkhoven wants to merge 4 commits into
Open
Conversation
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request adds support for tuning CUDA Tile C++ kernels using the cuda-python backend. CUDA Tile C++ was introduced in CUDA 13.3 and requires a C++ compiler that supports C++20.
For a while I have been trying to get it to work with NVRTC, but without much luck unfortunately. The kernels do compile with NVRTC and produce no compiler errors, but result in all kinds of errors when you try to load the compiled CUBIN or TileIR binaries, or when you try to retrieve the mangled/lowered/nonmangled name from the compiled program. When I finally managed to retrieve the NVRTC compiled kernels (either as cuFunction or cuKernel), I got segmentation faults when trying to launch those kernels (either with cuLaunchKernel or cuLaunchKernelEx).
After trying many different flavors of retrieving and launching the NVRTC compiled kernels, I decided to go for the NVCC route, which is also taken by Nvidia's Tile-Gym (https://github.com/NVIDIA/TileGym/blob/ff2a52ec4895764cb48473312164471fd031021d/src/tilegym/ops/tilecpp/utils/_cuda_utils.py#L249). This route also uses the cuda.core Python bindings, which are more high-level than the cuda.bindings APIs that we typically use.
It all feels a bit like a workaround, because it really seems that it should be possible to compile tile kernels with NVRTC. It is also quite a bit slower to dump the kernel to a file, call nvcc in a subprocess, and read the tilecubin back in. However, for now it seems to be the only route that actually works. Perhaps in the future we can simplify the code again when NVRTC and its Python bindings fully support tile kernels.
The entire use case of using Python to compile CUDA Tile C++ kernels is likely something that is not the typical path that CUDA/NVRTC developers had in mind. Also the documentation seems to direct Python users to cuTile rather than CUDA Tile C++ kernels. This is understandable and indeed for most Python users it is probably more convenient to just use cuTile. However, we're specifically interested in compiling and calling CUDA Tile C++ kernels from Python, which is a bit more of a hassle for now.
The current state of this pull is that it works on my system, but needs to be tested on other systems. There is an example kernel and script that tunes it for now. However, I need to add more tests and also check how errors like 'too much shared memory' etc can be captured when compiling this type of kernel.