hexagon: add an experimental Qualcomm FP8 DiT and VAE path

[happyyzy]

Motivation

We are building high-performance DiT inference on Qualcomm devices, covering both Adreno GPUs and Hexagon NPUs, for Local Dream.

Our Adreno work is already being contributed to llama.cpp: #22755, #26085, and #26331 are merged; Fused QKNorm-RoPE #28446 is currently open.

This PR provides the integrated Hexagon NPU path for stable-diffusion.cpp.

Implementation

The v79+ path includes standard F8_E4M3 safetensor loading; HMX 32x32 set_tensor repacking; FP8 W8A16 GEMM; non-causal HMX FA; fused Q/K RMSNorm-RoPE; segmented-K Klein linear2; FP16 VAE activations; direct implicit-GEMM convolution; fused GroupNorm-affine-SiLU; and fused upscale-convolution. Qwen3-4B uses the existing generic Hexagon Q4 kernels. This path supports end-to-end image generation at resolutions from 1K up to 2K, amazing for a phone!

The FP8 GEMM uses the v79 activation.hf / weight.f8 HMX path with FP32 accumulation. HMX cvt.hf = acc(2) applies the architectural 1/256 conversion; the HVX output epilogue compensates by 256 and folds the model's FP32 weight scale and any graph scalar scale into the same store pass.

Temporary integration

The FP8 plumbing required for full llama.cpp integration has not yet been accepted into llama.cpp's embedded ggml, for reasons that remain unclear to us. This PR temporarily uses happyyzy/ggml, branch work/qualcomm-hexagon-optimizations, commit 8121decc678bfb9f02a0f7190bce143bfb1599fa.

The generic pieces are being upstreamed into llama.cpp incrementally. This is a temporary integration vehicle for Local Dream until that work is complete.

End-to-end results

Device: Snapdragon 8 Elite / SM8750 / HTP v79. TE, DiT, and VAE all run on HTP. Prompt is a lovely cat, Euler, CFG 1, seed 42. ``E2Eisgenerate_image completed`, including live TE, all sampling steps, and VAE decode. None of these runs uses tiled VAE.

Model	Size	Steps	Upstream Q4_0 warm DiT	FP8 warm DiT	VAE	E2E
Z-Image	1024x1024	8	91.03 s/it	10.26 s/it mean (9.95-10.35)	2.47s	100.54s
FLUX.2/Klein 4B	1024x1024	4	79.42 s/it	8.54 s/it mean (8.43-8.69)	2.14s	49.89s
Z-Image	1536x1536	8	OOM	32.91 s/it mean before thermal throttling (32.86-32.96)	9.08s	306.03s
FLUX.2/Klein 4B	1536x1536	4	OOM	22.42 s/it mean before thermal throttling (22.40-22.44)	5.44s	111.68s

During the long 1536 runs, Z-Image iterations 5-8 rose to 36.16-38.37 s/it and the final Klein iteration rose to 25.24 s/it. The E2E numbers above include that thermal throttling.

This puts phone-class Snapdragon inference close to mainstream NVIDIA RTX GPU latency for these distilled pipelines.

Commands

export LD_LIBRARY_PATH="$PWD" ADSP_LIBRARY_PATH="$PWD"
LLM=/data/local/tmp/sd_bench_clean/qwen_3_4b-Q4_0.gguf
ZM="$PWD/z-image-turbo_fp8_scaled_e4m3fn_KJ.safetensors"
KM="$PWD/flux-2-klein-4b-fp8.safetensors"
ZV=/data/local/tmp/sd_bench_clean/ae.safetensors
KV=/data/local/tmp/validate6_model_pack_20260412_aux/flux2-vae.safetensors
run() { ./sd-cli --log-level verbose --diffusion-model "$1" --llm "$LLM" --vae "$2" --backend diffusion=HTP0,te=HTP0,vae=HTP0 --fa --vae-conv-direct -t "$5" -p "a lovely cat" --cfg-scale 1 --steps "$4" --sampling-method euler -W "$3" -H "$3" --seed 42 -o "$6"; }
run "$ZM" "$ZV" 1024 8 4 zimage_1024_s8.png
run "$KM" "$KV" 1024 4 8 klein_1024_s4.png
run "$ZM" "$ZV" 1536 8 8 zimage_1536_s8.png
run "$KM" "$KV" 1536 4 8 klein_1536_s4.png

Images

Z-Image 1024, 8 steps

Klein 1024, 4 steps

Z-Image 1536, 8 steps

Klein 1536, 4 steps

FP8 vs Upstream Q4_0/Q8_0

Compared with upstream Hexagon using standard Q4_0/Q8_0 GGUF weights, FP8 is 8.87x faster for Z-Image and 9.30x faster for FLUX.2/Klein at 1K, while also delivering visibly better image quality.

• Resolution: 1024x1024
• Sampling steps: 8
• Prompt:

雨夜的未来上海外滩，镜头前是一辆旧式有轨电车穿过积水街道，街边霓虹牌同时写着“欢迎光临”“火锅”“Open 24 Hours”，远处玻璃摩天楼与石库门老建筑并列，空中漂浮无人机广告屏，屏幕上有清晰汉字“春风
得意”，画面里有穿风衣的人群、红色雨伞、湿漉漉的柏油路反射青蓝与橙红灯光，构图复杂、层次深、电影感、超细节

Q4_0 + Q8_0

F8_E4M3

[happyyzy]

Clarification: this PR is not intended to be merged into master; its purpose is to establish a temporary Qualcomm optimization branch. I would be happy to maintain this branch if granted write access, or alternatively, @leejet could create the Qualcomm branch from master and retarget #1970 to it.

[leejet]

For ggml-related changes, please submit a PR to ggml-org/ggml or ggml-org/llama.cpp first. If the changes are merged upstream, I will sync them here.

As for the FP8-related changes, I previously submitted a PR to ggml-org/llama.cpp, but it has not been merged yet.

[happyyzy]

Yes, getting first-class FP8 support merged into llama.cpp has proven difficult. The standalone ggml repository is not the primary development path; it is periodically synchronized from the copy embedded in llama.cpp. Several FP8 proposals remain open or in draft form:

• #10055: FP8 support for GGUF/llama
• #25336: E4M3 FP8 CPU quantization type
• #27512: OCP FP8 E4M3 support
• #28485: FP8 types and CPU conversions

Our Hexagon work also includes FP8 GEMM, flash attention, fused operations, VAE kernels, runtime integration, and tests. Upstreaming the complete stack incrementally will likely take weeks or even months.

Meanwhile, there is clear demand from Qualcomm users who want to run diffusion models on their devices:

• #1416: Hexagon backend support
• #1039: NPU backend support
• #747: Adreno 830 VAE crashes and performance
• #876: SDXL on Snapdragon X Elite
• #584: Snapdragon 870 Android/Vulkan support

Our Local Dream users also want to run newer DiT models beyond SDXL on-device.

This is intended to be a Qualcomm-wide optimization branch, covering both the Hexagon NPU and the Adreno OpenCL GPU path. We will also integrate the existing and follow-up Adreno OpenCL work, including #22755, #26085, #26331, and #28446, as those changes land upstream.

For this reason, I believe it would be practical to create a temporary qualcomm branch. I can maintain the branch while the generic components are upstreamed, progressively remove the local implementations as their upstream replacements land, and eventually retire the branch once the work has converged.

[iwr-redmond]

Yes, getting first-class FP8 support merged into llama.cpp has proven difficult.

You may wish to reach out to the GenieX team at Qualcomm. An earlier (pre-acquisition) version of what is now GenieX included stable-diffusion.cpp and they have regular meetings with the llama.cpp maintainer responsible for Hexagon.

[vmobilis]

@happyyzy, hello, maybe you could try to contact @rmatif, he attempted to use Hexagon (#670, comment) in rmatif/Local-Diffusion (but did't publish releases with QNN support).