Which numeric formats an MI300X actually executes — measured on a Developer Cloud droplet

Artificial Intelligence (AI) Cloud & Edge AI AMD Developer Cloud
, , , ,
1

I spent a few days driving a single MI300X droplet on AMD Developer Cloud entirely
through an MCP server, and the part worth sharing here isn’t the tooling — it’s what
the card said when I stopped reading spec tables and measured it.

8192³ matmul, 30 iterations after 5 warm-up, torch 2.12.0+rocm10.0.0, gfx942
(MI300X VF, 304 CUs, 191.69 GiB). The card was concurrently serving, so every
absolute rate sits near half of peak — the ratios are the result, not the
TFLOP/s:

dtype TFLOP/s vs bf16
bf16 664.3 1.00x
fp16 662.5 1.00x
fp8 e4m3fnuz 1172.5 1.77x
int8 455.6 0.69x

Three of those four contradict something plausible.

int8 measured slower than bf16, and the spec says it shouldn’t. AMD rates int8
at 2614.9 TOPS — identical to fp8, double bf16. On that basis int8 looks like the
better-supported of the two. Measured, it reached 17.4% of its own peak. I read this
as a kernel gap rather than a silicon one: torch._int_mm isn’t finding a tuned path
on this stack. If someone has hit a library or hand-written MFMA route that closes
it, I’d like to hear about it — I didn’t attempt one.

fp8 here is e4m3fnuz, not e4m3fn, and that bites harder than a format note
suggests. Same byte, different value:

bit pattern 0b01000000 as float8_e4m3fn   -> 2.0
bit pattern 0b01000000 as float8_e4m3fnuz -> 1.0

float8_e4m3fn   round-trip [1.0, 2.0, 240.0, 448.0] -> [1.0, 2.0, 240.0, 448.0]
float8_e4m3fnuz round-trip [1.0, 2.0, 240.0, 448.0] -> [1.0, 2.0, 240.0, nan]

448 is an ordinary weight on an H100 and NaN here. Asking for the NVIDIA flavour at
least fails loudly rather than silently — RuntimeError: CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED
— and vLLM agrees from its own side (is_fp8_fnuz() keys on "gfx94"). So an fp8
checkpoint from the Hub is not drop-in; quantize online from the bf16 weights and the
scales get derived on the machine that will run them.

fp4 isn’t on gfx942 at all, and the capability list says otherwise.
supported_quantization includes mxfp4, but that’s a generic registry, not a
hardware claim — supports_mx() gates on gfx95/gfx1250. Rather than refusing,
the stack falls back to an emulation kernel that dequantizes the weights back to bf16
on every forward pass and runs an ordinary bf16 F.linear. Bf16 speed, no resident
saving during compute, dequant overhead on top, plus the full fp4 error. Useful for
answering “would this model survive fp4” before buying MI355X; not a serving path.

Net: on this card it’s fp8 e4m3fnuz, quantized online, and nothing else. Three
separate sources pointed elsewhere to get there.

Two operational notes from the same box, in case they save anyone the hour they cost
me:

  • rocm-smi and amd-smi exit 0 when they fail. With the driver uninitialised,
    rocm-smi printed “Driver not initialized” to stderr, nothing to stdout, and exited
    0. Parse the output; never branch on the exit status.
  • A freshly provisioned droplet needs one reboot. amdgpu fails to bind the VF
    and unloads, so lspci sees the card and rocminfo reports no GPU agent. The
    reliable test is whether rocminfo reports a gfx target — /dev/kfd can be
    present on a card that never bound.

Full write-up with the inventory tables, the vLLM memory accounting and the scope
caveats: https://xbill999.medium.com/managing-amd-mi300x-deployments-with-mcp-318d0ab116c8

Tooling is Apache-2.0 here, if the MCP side is of interest:

Happy to run a follow-up measurement on this droplet if there’s a shape or dtype
people want checked.

Ten weight formats of Gemma 4 E2B on one MI300X: fp8 is the only one at bf16's pace
2

Asking rather than answering here, to be upfront: I’m coming from the CUDA side and don’t have an MI300X under me yet, so none of the below is tested.

Taking you up on the follow-up offer if it’s cheap. The int8 row is the one I’d most like to see re-run with the dtypes interleaved round-robin across iterations rather than as sequential blocks. You flag that the card was concurrently serving and treat the ratios rather than the absolutes as the result, which is fair, but int8 landing at 17.4% of its own peak while bf16 sits near 50% is a wide enough gap that contention and kernel gap are hard to separate if those runs were sequential. Interleaving costs nothing and would make the int8 finding much harder to argue with.

On the library or hand-written MFMA route you asked about, three places I would look, in rough order of effort. hipBLASLt’s own tuning rather than whatever _int_mm resolves to by default, since a shape outside the tuned set quietly falls back to something generic. Composable Kernel, which carries int8 GEMM templates for gfx942. And Triton, if you want the hand-written route without writing assembly: tl.dot on int8 inputs should lower to the MFMA int8 instruction on gfx942, and a tile-tuned GEMM is a few dozen lines. That last one would settle the question cleanly, because if Triton reaches a sane fraction of peak on the same shape then it’s a dispatch and tuning gap in the torch path rather than anything about the silicon. Pointers, not results.

Separately, your two operational notes were worth more to me than the table. I’ve been writing a probe that reports what a box actually is before you trust it, and it was using the presence of /dev/kfd as a liveness signal. Your point that /dev/kfd can be present on a card that never bound means that check was worse than useless, since it would report healthy on precisely the failure it exists to catch. Switching it to whether rocminfo reports a gfx target. The probe already parses rocm-smi output rather than branching on its exit status, but I hadn’t known there was a real reason for that until your note.

Repo, if any of it is useful: GitHub - Lingikaushikreddy/rocm-devops-starter: Diagnostic probe, smoke-test training run, ROCm container and CI for getting PyTorch workloads running on AMD GPUs. · GitHub

3

I have another article almost done that includes a full
run of vLLM against other platforms like CUDA and TPU.

I didn’t dwell on the hardware issue- as I used a debian trixie image
with some script customizations to keep the OS the same across rigs.
The trixie base image did not come with AMD drivers- so I had to stage it.

I will check out your repo.

thanks

4

Thanks, and the driver staging is the useful bit for me. That is the part I was expecting to trip over.

Which way did you go: bake the amdgpu and ROCm userspace into the image, or stage it at first boot from a script? Keeping one OS across rigs is exactly the constraint that makes the container story awkward, and I would rather not rediscover your answer the slow way.

Interested in the vLLM across CUDA and TPU piece when it lands.

5

Taking the offer: the same sweep at M=1 and M=8, with N and K at model dimensions rather than 8192³. That’s the friendliest shape there is and decode never sees it. AMD’s own low-latency GEMM writeup tabulates the real decode grid and shows small-M large-K falling off hard. Your dtype ranking may not survive the move: Accelerating LLM Inference on AMD GPUs with Low-Latency GEMMs — ROCm Blogs

7

Good call, thanks.

You were right that my test used the wrong shape.

I re-ran it with the sizes the model actually uses when it writes text, and fp8 came out about 1.5x faster, not 1.8x.

The int8 option couldn’t run at those sizes at all.

Then I served the whole model with fp8 turned on and measured real output speed.
It was 20–47% slower than leaving it off, at every load I tried.

The faster math is there, but converting to fp8 on every step costs more than it saves.

This typically happens on chips when they have to convert the datatypes to the best format
on that particular hardware- and sometimes the conversion costs more than the savings.

Full results:
https://github.com/xbill9/gemma4-dev/blob/main/gpu-vllm-mi300x-2b/benchmarks/runs/2026-09-18-vllm-sweep-mi300x-v0191-fp8/REPORT.md

The layer-by-layer test and the script, if you want to run it yourself:
https://github.com/xbill9/gemma4-dev/tree/main/gpu-vllm-mi300x-2b/benchmarks/runs/2026-09-18-gemm-decode-shapes-mi300x