Serving LLMs for Agents on AMD Instinct: Long Context, Fast Tokens, Real Workloads

Agents put pressure on LLM serving from three sides at once. Prompts get very long, every step waits on the next token, and the real workload looks nothing like a clean single-request benchmark. These four recent deep dives each take on one of those pressures on AMD Instinct GPUs, with the deployment setup and the numbers to back it up.


Quick reference table

Blog Key result Who it’s for
Long Context, One Node: Serving GLM-5.3-FP8 on MI300X with DP8+EP8 On one 8x MI300X node, DP8+EP8 delivers 147% higher output throughput than TP=8 on a 96K-in / 32K-out workload. TP=8 still wins on short 8K/1K requests Anyone serving long-context traffic on a single node and picking a parallelism layout
Bringing Ultra-Fast, Interactive LLM Inference to AMD Instinct MI355X GPUs 1.9x-3.3x faster batch-1 decode on a single MI355X (about 1,100 tokens/s for Qwen3.8-27B), open-sourced via ATOM, SGLang and vLLM Builders of latency-sensitive apps where one user waits on every token
FlyDSL Mono Kernels for High-Interactivity LLM Serving in ATOM MiniMax-M3 batch-1 step time cut from 11.47 ms to 6.26 ms (1.83x) on MI355X, and DeepSeek-V4.1-Flash per-user interactivity up 1.35x in vLLM Kernel and inference engineers squeezing the low-concurrency decode path
GLM-5.2 AgentX on AMD Instinct MI355X: Boosting Throughput and Interactivity On the InferenceX AgentX coding-agent benchmark, peak per-GPU throughput up 1.72x and peak P90 interactivity up 2.46x with GLM-5.2-MXFP4 in SGLang Teams running agentic coding workloads in production

Read together, they cover the whole agent-serving picture. Pick the layout that fits your context length, fuse the decode path so each token arrives faster, then check it all against a real agent benchmark instead of a synthetic one.


Long Context, One Node: Serving GLM-5.3-FP8 on MI300X with DP8+EP8

This post serves a 744B-parameter MoE on a single 8-GPU MI300X node with vLLM and compares DP8+EP8 against plain tensor parallelism. The honest result is also the useful one. DP8+EP8 wins big under long-context load (1,066 vs 432 output tokens/s on 96K/32K), while TP=8 stays faster on short 8K/1K requests. So the right layout depends on what your agents actually send.


Bringing Ultra-Fast, Interactive LLM Inference to AMD Instinct MI355X GPUs

AMD is open-sourcing latency-optimized inference paths for Qwen3.8-27B, Qwen3.8-Flash-Next and GPT-OSS-120B (preview). They combine persistent megakernels, speculative decoding, low-precision data types and die-aware data placement on MI355X. If your app has one user waiting on every token, this is where the batch-1 speedups come from, and the post includes full reproduction steps.


FlyDSL Mono Kernels for High-Interactivity LLM Serving in ATOM

This post takes a second route to fast decode: fused “mono” kernels written in FlyDSL and integrated into ATOM, with paths into vLLM and SGLang. On MI355X it nearly halves MiniMax-M3’s batch-1 step time once indexer context parallelism is added, and it lifts DeepSeek-V4.1-Flash’s per-user P90 interactivity from 318 to 429 tokens/s at concurrency 1. Read it if you want to see how the kernel-level wins are built, not just what they deliver.


GLM-5.2 AgentX on AMD Instinct MI355X: Boosting Throughput and Interactivity

This post puts it all to a real test: GLM-5.2-MXFP4 in SGLang on the InferenceX AgentX benchmark, which models multi-turn coding-agent traffic. Peak per-GPU throughput rises from 10,625 to 18,327 tokens/s, and peak P90 interactivity from 110 to 271 tokens/s per user. The authors note this isn’t a controlled A/B, since the GPU allocation also changed, so treat it as an end-to-end tuning story rather than a single-knob comparison.

Drop questions or your own numbers below - it’s always good to see how these setups hold up under your own agent traffic.