I’d like to hear about your experiences with custom optimisations - what you’ve worked on, what you’ve learned, what surprised you. It’d help me and others make better decisions and build better solutions.
At the moment I’m studying MFMA and GEMM operations on AMD MI300X (CDNA 3) on AMD dev cloud and trying to apply them to specific contexts: inference workloads, device utilisation, and broader computer design. It can be difficult to choose the right path - which examples to study, which fundamentals to ground oneself in first.
Here’s a concrete fork I’ve been wrestling with: do I bootstrap from Composable Kernel examples and refactor inward to understand each layer? Or write my own small kernels from scratch first: GEMV, reductions to internalise wavefront and MFMA mechanics before touching production-grade code?
Curious how others approached the same decision and chose their path.
Beyond that, any war stories about kernel optimisation on AMD: what surprised you / where the steep learning curves were - all welcome!
Definitely start by writing your own small kernels from scratch. Composable Kernel is great for finished products. The biggest surprise you will run into on the MI300X is how tricky memory management is. The math cores are so incredibly fast that your main problem won’t be the actual calculations. Your biggest headache will be moving the data into the right place fast enough to keep the cores busy without making the system crash or stall out. Also, look into using Triton because it handles a lot of that frustrating memory balancing for you automatically.
Thank you for you advise, triton is a really good tool. Currently I’m learning HIP kernels, but I can’t understand the line between triton and HIP kernels application. Can you provide your view about the difference between them?
hey, think of Triton as Python for writing fast deep learning kernels, while HIP is like raw C++ for maximum hardware control. it is much easier to write and automatically handles complex memory layout optimizations.
Working on MI300X optimization here too — one thing that surprised us
significantly: __shfl_down / __shfl_up silently lower to ds_bpermute
(LDS round-trip) on CDNA3. If you have any warp-level reductions inside
your GEMM or attention kernels (softmax, layer norm, dot products),
they’re likely hitting LDS on every cross-lane step without any warning.
We characterized the full-DPP wave64 geometry that avoids this entirely
— rocprofv3 measures SQ_INSTS_LDS = 0, and it runs 1.79× faster than
hipCUB WarpReduce on MI300X. Might be directly relevant to your
inference workload reductions.
GitHub: github . com / jpcpol / AMD-Incstinct-Labs
Paper + benchmark results (Zenodo, MIT): doi . org / 10.5281 / zenodo.20700838
Curious what MFMA tile sizes you’re finding work best for your inference
shapes — that’s an area we’re heading into next (MLIR auto-tiling).
Happy to share — this has been my computational acceleration rabbit hole for the past couple months.
Started from a narrow finding: wave::reduce_sum and similar cross-lane primitives on MI300X (CDNA3) were running ~2.4× slower than hipCUB’s equivalents. Root cause was that the shuffle-down pattern lowers to ds_bpermute (an LDS round-trip) instead of native DPP instructions — a compiler lowering gap, not a hardware limitation. Rewriting with full-DPP primitives closed that gap (1.79× over hipCUB).
From there I built up: a Flash Attention kernel combining DPP softmax with DME (Data Movement Engine) async K/V prefetch, tested MFMA matrix cores (helped at D=128, not at D=64 — occupancy issue, not a primitive one), and eventually a full native HIP LLM runtime running Qwen2.5-0.5B end-to-end on MI300X with token-level correctness validation against HuggingFace.
Biggest lesson: throughput numbers alone hid a real correctness bug (a transposed GEMM) that only token-level validation caught. Worth building that check in early rather than trusting benchmark numbers alone.
Everything’s public — code and paper/benchmarks on Zenodo (DOI 10.5281/zenodo.20700838) if useful as a reference point.
