GLM 5.2: CUDA backend support#517
Open
giannisanni wants to merge 1 commit into
Open
Conversation
Implements the CUDA side of GLM 5.2 inference on top of the glm5.2
branch engine: MLA attention, DSA indexer, compact KV, routed MoE with
IQ2_XXS gate/up/down dp4a kernels, and SSD expert streaming (io_uring
O_DIRECT fetch engine, host LFU expert cache, parallel fetch backfill).
Engine changes are minimal:
- allow DS4_BACKEND_CUDA through the GLM backend gate,
- gate the split value-projection path on the f16 compact cache: it
assumes the Metal f16 layout and fails the batch attention-lora encode
on CUDA's f32 cache.
The CUDA backend replaces the placeholder stubs with real
implementations and adds the GLM kernel implementations in
ds4_cuda_glm_{kv,indexer,attn,moe,stubs}.inc.
Includes the batch expert-tile LUT initialization fix also submitted
against main as antirez#513 (models with n_embd > 4096).
Tested on RTX 4060 Ti 16GB (sm_89), 32GB RAM host, Linux, against the
official GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf with
--ssd-streaming: greedy decode correct at ~0.3-0.4 t/s, 200-token batch
prefill correct at ~3.2 t/s, 600-token at ~6.5 t/s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As discussed in #458: CUDA backend support for GLM 5.2, on top of the glm5.2 branch engine.
What it adds
The CUDA implementations of everything the GLM engine needs: MLA attention, DSA indexer, compact KV, routed MoE with IQ2_XXS gate/up/down dp4a kernels, and SSD expert streaming (io_uring O_DIRECT fetch engine with a pread fallback, host LFU expert cache, parallel fetch backfill). The placeholder stubs in ds4_cuda.cu become real implementations; GLM-specific kernels live in ds4_cuda_glm_{kv,indexer,attn,moe,stubs}.inc.
Engine changes (kept minimal)
Testing
RTX 4060 Ti 16GB (sm_89), 32GB RAM host, Linux, nvcc 12.0, against the official GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf with --ssd-streaming:
Note: this ds4_cuda.cu includes the batch expert-tile LUT initialization fix I also submitted against main as #513 (the tile kernels loaded the IQ2 dequant tables only for n_embd <= 4096; GLM is 7168). Whichever lands first, the other becomes a trivial rebase.
More background on the streaming behavior at these sizes is in #495. Happy to adjust anything to fit the branch's direction.