# Three different levers for squeezing more throughput out of LLM serving on AMD Instinct MI355X **URL:** \ **Category:** Generative AI\ **Tags:** rocm, performance, llm, amd-instinct, mi355x\ **Created:** [October 2, 2026, 1:39pm UTC](https://devcommunity.amd.com/t/three-different-levers-for-squeezing-more-throughput-out-of-llm-serving-on-amd-instinct-mi355x/1129 "2026-10-02T13:39:45Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![nirth\_AMD](https://dub1.discourse-cdn.com/amd/user_avatar/devcommunity.amd.com/nirth_amd/32/623_2.png) [@nirth\_AMD](https://devcommunity.amd.com/u/nirth_AMD)\ **Post date:** [October 2, 2026, 1:39pm UTC](https://devcommunity.amd.com/t/three-different-levers-for-squeezing-more-throughput-out-of-llm-serving-on-amd-instinct-mi355x/1129/1 "2026-10-02T13:39:46Z")
Three different levers for squeezing more throughput out of LLM serving on AMD Instinct MI355X - speculative decoding, parallelism strategy, and KV-cache quantization. Each tackles the problem from a different angle, and each comes with its own real benchmark numbers. * * * **Quick reference table** | Blog | Key result | Who it’s for | | --- | --- | --- | | [DFlash Speculative Decoding on AMD Instinct MI355X: Up to 5× Faster Qwen3.5 Inference](https://rocm.blogs.amd.com/artificial-intelligence/dflash-on-MI355x/README.html "https://rocm.blogs.amd.com/artificial-intelligence/dflash-on-mi355x/readme.html") | Up to 5.02x single-request throughput on Qwen3.5-27B vs. standard decoding, plus 10-20% more when the target model is MXFP4-quantized | Anyone serving large models who’s hit a single-request latency wall | | [Serving GLM-5.2-MXFP4 on AMD Instinct™ MI355X: When Prefill Context Parallelism Pays](https://rocm.blogs.amd.com/artificial-intelligence/glm52-mxfp4-mi355/README.html "https://rocm.blogs.amd.com/artificial-intelligence/glm52-mxfp4-mi355/readme.html") | Maps exactly where prefill context parallelism helps vs. hurts by prompt length on 4x MI355X - same checkpoint, same hardware, CP on vs. off | Teams tuning multi-GPU serving configs for long-context prompts | | [UltraQuant on AMD Instinct: More Efficient Agentic Serving for Qwen3.8-MXFP4](https://rocm.blogs.amd.com/software-tools-optimization/ultraquant-kv4/README.html "https://rocm.blogs.amd.com/software-tools-optimization/ultraquant-kv4/readme.html") | 29% more tokens/sec and 24% lower per-token latency vs. 8-bit KV at concurrency 32, with 1.7x faster first token; accuracy within noise on GPQA-Diamond/SWE-bench Lite | Teams needing more concurrency headroom without an accuracy hit | * * * Start with DFlash if your bottleneck is single-request decode latency, bring in the GLM-5.2 prefill CP findings once you’re scaling serving across GPUs, then layer UltraQuant’s KV-cache quantization on top once you need more concurrency without burning accuracy. * * * ### **[DFlash Speculative Decoding on AMD Instinct MI355X: Up to 5× Faster Qwen3.5 Inference](https://rocm.blogs.amd.com/artificial-intelligence/dflash-on-MI355x/README.html "https://rocm.blogs.amd.com/artificial-intelligence/dflash-on-mi355x/readme.html")** ![artificial-intelligence-dflash-on-MI355x-images-dflash_thumbnail](https://global.discourse-cdn.com/amd/original/2X/6/63ab694aacc310c0aacd6b4464757cb1cf02850b.webp) DFlash replaces the usual autoregressive speculative drafter with a block-diffusion model that denoises an entire draft block in a single forward pass instead of token-by-token. On Qwen3.5-27B this delivers up to 5.02x single-request throughput over standard decoding, with consistent wins over Multi-Token Prediction (MTP), and an extra 10-20% when the target model itself is MXFP4-quantized. * * * ### **[Serving GLM-5.2-MXFP4 on AMD Instinct™ MI355X: When Prefill Context Parallelism Pays](https://rocm.blogs.amd.com/artificial-intelligence/glm52-mxfp4-mi355/README.html "https://rocm.blogs.amd.com/artificial-intelligence/glm52-mxfp4-mi355/readme.html")** ![artificial-intelligence-glm52-mxfp4-mi355-images-glm52-mxfp4-cp-thumbnail](https://global.discourse-cdn.com/amd/original/2X/a/a00ecea1be798c6768f37e2a2f700345a59906cb.webp) Splitting prompt tokens across GPU ranks during prefill (context parallelism) isn’t a universal win - this piece pins down exactly where it helps versus hurts when serving GLM-5.2-MXFP4 on 4x MI355X, same checkpoint and hardware, CP on vs. off. It maps where the tradeoff sign flips by prompt length, and where single-stream serving already breaks even. * * * ### **[UltraQuant on AMD Instinct: More Efficient Agentic Serving for Qwen3.8-MXFP4](https://rocm.blogs.amd.com/software-tools-optimization/ultraquant-kv4/README.html "https://rocm.blogs.amd.com/software-tools-optimization/ultraquant-kv4/readme.html")** ![software-tools-optimization-ultraquant-kv4-images-ultraquant-kv4](https://global.discourse-cdn.com/amd/original/2X/a/aa943e678bae34a367403e6f259591c2a19d80f8.webp) Pairs MXFP4-quantized Qwen3.8-2.4T-A95B with UltraQuant’s 4-bit KV cache quantization on vLLM. At concurrency 32 it serves 29% more tokens/sec at 24% lower per-token latency than the fastest 8-bit KV backend, with 1.7x faster time-to-first-token, and accuracy on GPQA-Diamond and SWE-bench Lite holds within noise of the 8-bit baseline. Multi-Token Prediction stacks on top for further gains at moderate concurrency. Running any of these on your own workloads? Drop your numbers below! ---
**Author:** ![GERALD\_BUSTILLA](https://dub1.discourse-cdn.com/amd/user_avatar/devcommunity.amd.com/gerald_bustilla/32/631_2.png) [@GERALD\_BUSTILLA](https://devcommunity.amd.com/u/GERALD_BUSTILLA)\ **Post date:** [October 3, 2026, 12:35am UTC](https://devcommunity.amd.com/t/three-different-levers-for-squeezing-more-throughput-out-of-llm-serving-on-amd-instinct-mi355x/1129/2 "2026-10-03T00:35:02Z")
Yes it is great especially the ultraquant and the dflash, now i know how to create a proper documentations thanks 😆 😆