POWER10/11 MMA acceleration for all quantized formats (silicon-validated)#100
Open
mavin2009 wants to merge 21 commits into
Open
POWER10/11 MMA acceleration for all quantized formats (silicon-validated)#100mavin2009 wants to merge 21 commits into
mavin2009 wants to merge 21 commits into
Conversation
MMA outer-product kernels for the 1-bit and ternary Bonsai formats, with a select-and-sum GEMV that reads raw 1/2-bit weights for token generation. Dispatch via llamafile_sgemm behind #if __MMA__.
K-quant GEMM kernels: raw unsigned codes ride the unsigned GER operand, offsets handled algebraically (separable rank-one update from the activation quantizer's existing sums).
Q3_K completes the K-quants; the IQ4 family flips operand orientation (signed codebook values on the signed side, activations XOR-flipped) with a per-row 128*codesum correction at repack.
The Q4_1/Q5_1 min term reduces to one multiply-add because Q8_1 blocks already store the scaled activation sum.
TQ1_0, TQ2_0, IQ2_XXS/XS/S, IQ3_XXS/S, IQ1_S/M collapse into shared 32-deep and 16-deep chunk kernels plus per-format scalar decoders; decode runs at repack time, off the hot path.
Replaces silent skips in the one-shot drivers with GGML_ABORT.
First llamafile_sgemm call for a tensor packs the whole matrix once; subsequent calls reuse it for the model lifetime. Capacity-bounded (PPC_MMA_PACK_CACHE_MB).
…shared B-pack FNV-1a content fingerprint detects model reloads at the same address; admission-without-eviction makes the cache monotonic; column-partitioned activation packing removes per-thread duplicate pack work on cache hits.
Two accumulator sets alternate so the next chunk's GERs issue before the previous chunk drains. On POWER10 silicon: pp -1..-5%, tg +3-4% pre-0015; standard remains the default.
Replaces tinyBLAS_Q0_PPC for these types; small-n shapes stay on vec_dot pending GEMV forwarding.
The 128*W correction is exactly representable and subtracted from the exact integer accumulator before scaling, matching ggml's scalar rounding order. Field fix from POWER10 validation.
n too small to feed every thread by columns (worst case n=1) now row-partitions with the cached pack. Field fix, Q4_K tg32.
The packed path reads int8-expanded weights (1.8-3x native bytes) and loses generation to vec_dot below one column tile; measured on POWER10 (tg 4.18 -> 35.7 t/s on a 1.5B IQ2_M after this change plus the cache fix).
128 fixed slots lost to the ~197 tensors of a 28-layer model -- a third of the weights re-packed every call. Slot count can no longer bind; the byte cap is the only bound and crossing it prints what it costs. Stats API added.
These formats' vec_dot runs at 2-3% of the memory wall on POWER10; the packed path wins despite the int8 expansion (tg +50%/+98%).
…flag dcbt TH=8 stream hints measured +13% pp128 on IQ4_XS silicon and become that kernel's default (PPC_DCBT_LINES restores line touches); neutral on K-quants. lxvp vector-pair loads measured -1..-22% and stay behind IQGRID_LXVP as documented negative space.
All threads of the op pack disjoint row-tile slices; the last one publishes. Cold start 4.4s -> 1.05s on a 1.5B IQ2_M, +6% steady pp from NUMA-friendly first touch.
Built to test the stream-parallelism hypothesis at n=1; measured 27-44% slower than vec_dot -- n=1 is issue-rate-bound on POWER10. Kernel and float64 tests remain for wider silicon to re-run.
…asurement Each xvi8ger4pp retires 4 rows x 4 depth over tiles already decoded in the pack cache. On POWER10: TQ1_0 3.1x over vec_dot, IQ2_XXS +42%, IQ1_S +23%, IQ3_XXS/S +38%, IQ1_M +58%; TQ2_0/IQ2_S/IQ2_XS keep vec_dot, which measured better for them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
MMA (Matrix-Multiply Assist) acceleration for every quantized format this fork ships on POWER10/POWER11 — the Bonsai formats (Q1_0/Q2_0) plus the K-quants, the IQ grid codebooks, the legacy types, TQ1_0/TQ2_0, IQ4_NL/XS and MXFP4/NVFP4. Today only Q4_0/Q8_0 and the float types ride MMA; everything else falls back to scalar/VSX vec_dot.
Integration is via
llamafile_sgemmbehind#if defined(__MMA__): zero impact on any other architecture. The series adds ten kernel translation units, a tensor-keyed weight-pack cache (packs once per tensor per model lifetime, parallel first-touch across the op's threads), and small-n dispatch policy that routes each format to whichever path measured faster on silicon.The branch is 21 commits, one per patch of the numbered series it was developed as; each commit message carries its measured result.
Measured on POWER10
IBM 9105-42A LPAR (4 cores/SMT2), RHEL 9.7, GCC 11.5. Baseline = the same tree built
-mcpu=power9(MMA compiled out), same machine, minutes apart.Prompt processing: 3.7×–48× across every format tested. Highlights (pp128 @8t): Q2_0 1.7B 8.7→416 t/s (48×), IQ3_XS 1.5B 33→331 t/s (10×), TQ1_0 13→332 t/s (26×), Q4_K 27B 5.9→21.9 t/s (3.7×).
Token generation: parity or better everywhere. Q2_0 4.6× (its fallback is scalar); GER-GEMV over the cached tiles buys TQ1_0 3.1×, IQ2_XXS +42%, IQ3_XXS/S +38%, IQ1_S +23%, IQ1_M +58%; formats whose vec_dot measured better (TQ2_0, IQ2_S/XS, K-quants, IQ4, legacy) keep vec_dot at small n by explicit dispatch policy.
How it was verified
-mcpu=power8control build containing none of this code).dequantize_row_*— every grid/sign table compared exhaustively, ~7.6M random dequantized elements at maxrel = 0.Full methodology, the validation harness, benchmark data and the complete engineering log (including the experiments that lost and stayed in-tree as documented negative space) are in the companion repo: https://github.com/mavin2009/ppc-mma-kernels — start with
docs/VALIDATION-POWER10.mdanddocs/REVIEW.md.Caveats, stated plainly
scripts/validate-on-power.shin the companion repo runs the whole protocol against a checkout of this branch and emits a paste-ready report — independent results very welcome.PPC_MMA_PACK_CACHE_MBto change, refusals print loudly). Migrating it into the repack buffer-type machinery is designed but deliberately sequenced later; the design note explains why.Happy to split this into smaller PRs (kernels first, cache second, dispatch policy third) if that's easier to review.