fix: handle multi-dim activations in LoKr forward - #1980
Closed
undefeaterpeter wants to merge 1 commit into
Closed
undefeaterpeter wants to merge 1 commit into
undefeaterpeter wants to merge 1 commit into
Conversation
ggml_ext_lokr_forward's non-conv branch assumed a 2-D activation [q, batch] and took batch = ne[1]. Model blocks that run linear layers on 3-D activations [features, L, N] (e.g. Krea-2 attention with N > 1) hit the first split reshape with a mismatched element count, and the GGML_ASSERT inside ggml_reshape_3d aborted the process (SIGABRT) whenever a LoKr LoRA was active on such a layer. Fold every trailing dim into the batch (nelements / q) and reshape the result back to the activation's original trailing dims, so the out-diff adds element-wise onto the linear output instead of relying on a flat 2-D shape. The conv branch already batches on ne[3] and is unchanged.
Author
|
Withdrawn. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Applying a LoKr LoRA (
lokr_w1/lokr_w2factors) to a model whose linear layers run on 3-D activations aborts the process with SIGABRT. Crash stack:Root cause
The non-conv branch of
ggml_ext_lokr_forwardassumes a 2-D activation[q, batch]and takesbatch = h->ne[1].Krea2::KreaAttention(and other batched attention paths) feed linears 3-D activations[features, L, N]; wheneverN > 1,ggml_nelements(h) = q·L·N ≠ vq·uq·L, so the first split reshape's element-countGGML_ASSERTfires and aborts. WithN == 1every reshape happens to pass, which is why LoKr works on many models and the gap went unnoticed.Fix
batch = ggml_nelements(h) / q_actual— fold all trailing dims into the batch (the existing reshapes already assume a contiguous activation).[up·vp, ne[1], ne[2], ne[3]]instead of flat 2-D, soforward_with_lora'sggml_add_inplaceadds the diff element-wise onto the linear output rather than depending on a shape that no longer matches.ne[3]and is unchanged. The batch dim stays outermost through every reshape/transpose in the pipeline, so folding L·N and restoring at the end is index-exact; the Kronecker row convention (out = u·vp + p) is unchanged.Verification
lokr_w1/lokr_w2factors, OneTrainer/diffusers export) — 100% SIGABRT at the stack above before the fix.src/model/adapter/is identical between that commit and current master.🤖 Generated with Claude Code — reviewed and submitted by the repo owner per the AI-assisted contribution policy.