We’ve been working on a compressed-weight matrix kernel for the Radeon RX 9060 XT 16 GB. Our best measured case went from 9.30 ms to 2.55 ms, about 3.65× faster than our previous fused kernel.
The useful change was how often we decoded the weights. The earlier path decoded the same fragment again for each 16-column activation tile. The new path decodes it once and reuses it across several tiles, keeping the fragments and RDNA4 WMMA accumulators in registers.
Here are the recorded October 6 results:
| Token-like activation columns | Previous fused kernel | Register-reuse kernel | Speedup |
|---|---|---|---|
| 32 | 2.206 ms | 1.204 ms | 1.83× |
| 64 | 4.545 ms | 1.296 ms | 3.51× |
| 128 | 9.300 ms | 2.550 ms | 3.65× |
This was a synthetic 14,336 × 4,096 matrix using CodeGEMM’s additive-v8 format. The dimensions match a Llama feed-forward layer, but the weights weren’t taken from a pretrained checkpoint. The device was gfx1200, running Windows OpenCL—not a ROCm/HIP full-model run.
We used four rotating fixtures, three warmup cycles and 21 measured cycles per cell, randomized the variant order and waited explicitly for each kernel. The times above are complete-cycle GPU medians divided by four fixtures; compilation and allocation aren’t included. A separate process run gave similar ratios.
The selected path uses about 14.88 MiB of packed codes, scales and codebooks without materializing a 112 MiB expanded FP16 weight matrix. The compressed representation comes from CodeGEMM. Our work is the RDNA execution path and exact-rounding register reuse.
All tested large-fixture FP32 outputs matched the frozen fused baseline bit for bit. That establishes agreement with the represented compressed weights, not quality against the original FP16 model.
There are limits worth keeping visible. Small matrices got slower, and 8–16 activation columns showed no consistent gain. The baseline is our own kernel, not NVIDIA, llama.cpp, tuned BLAS or production CodeGEMM. We haven’t shown that the whole AI runs 3.65× faster, or that text tokenization is faster. We also haven’t isolated an AMD-specific architectural cause for the gain.
Next is a dispatcher that chooses a path by matrix shape, then real-model integration and comparisons with optimized runtimes. Quality, memory use and latency need to sit beside the speed numbers. That work is still ahead of us. Source and raw logs are retained locally; a public reproduction bundle hasn’t been published yet.
For anyone working on compressed inference or RDNA kernels: which real-model shapes and baseline runtimes would you test next?