Hi AMD Developer Community! 
I’m sharing AMD LLM Lab, an independent open-source project I’ve been developing to explore and predict Large Language Model (LLM) inference performance on AMD hardware.
I conducted 57 benchmark configurations across Qwen2.5 models ranging from 0.5B to 32B parameters on an AMD Instinct MI300X, examining GPU memory consumption, generation throughput, context length, and quantization trade-offs.
What I explored
-
GPU memory: How model size and configuration affect VRAM consumption.
-
Inference performance: Measuring generated tokens per second.
-
Context scaling: Understanding how longer contexts affect memory and throughput.
-
Quantization: Comparing BF16, FP16, INT8, INT4, and INT2 configurations.
-
Performance prediction: Using regression models to estimate memory requirements and throughput for unbenchmarked configurations.
One particularly interesting observation was that lower numerical precision did not consistently result in lower measured memory consumption or faster inference. The results depend on the implementation, backend, and workload, making empirical benchmarking essential.
Explore the project
The project also distinguishes between measured, estimated, and interpolated results to make the evidence behind performance predictions more transparent.
I’d appreciate your feedback
I’d particularly like to hear from developers working with AMD Instinct GPUs, ROCm, LLM inference, and quantization:
-
What benchmarking practices would make these results more reproducible?
-
Which ROCm optimization techniques should I investigate next?
-
How would you improve prediction reliability across unseen model sizes and configurations?
If you find the project useful, please consider starring the GitHub repository. It helps other developers discover the project and supports continued open-source development.
I’m looking forward to learning from the community and improving the project through technical feedback and collaboration.
Note: AMD LLM Lab is an independent open-source research project, not an official AMD product.
4 Likes
For PyTorch-backed runs, one useful reproducibility detail would be separate columns for tensor memory (memory_allocated) and memory held by the caching allocator (memory_reserved), plus their peak values. On ROCm, unused allocator cache can still appear as used in amd-smi. These counters use torch.cuda even on HIP. PyTorch HIP memory management
After model loading and warmup have completed (torch.cuda.synchronize()), record the baseline values, call reset_peak_memory_stats(), run a fixed prompt/output workload, wait for GPU work to finish, and record the peaks. That defines a warmed-generation window; I would report the model-loading peak separately. Peak memory tracking
I would also use a fresh process per precision/backend configuration so allocator state from an earlier cell does not carry into the next comparison. Which memory counter and measurement window produced your VRAM figures?
3 Likes
Thanks for this detailed and valuable feedback. You’re right that distinguishing allocated memory from reserved memory is important for reproducible PyTorch/ROCm benchmarking, especially when the caching allocator can affect how memory usage appears in external monitoring tools.
My current results need a more explicit description of the memory measurement method and measurement window. I’ll audit the benchmark implementation to establish exactly how the reported VRAM figures were collected, then work toward recording baseline and peak allocated/reserved memory separately, synchronizing GPU operations, and using a fixed generation workload after warmup.
I also agree that running each precision/backend configuration in a fresh process would help avoid allocator state affecting comparisons. I’ll document model-loading memory separately from warmed-generation memory where the measurement setup supports it.
Thanks for highlighting these details. Improving measurement provenance and reproducibility is an important next step for AMD LLM Lab.
2 Likes
Thanks, Vedant — I appreciate the thoughtful follow-up. Making the measurement method explicit will be a useful improvement. One small addition to that plan could be a compact table with post-warmup baseline, absolute peak, and peak minus baseline for both allocated and reserved memory in the same warmed-generation window.
I would label those as PyTorch allocator metrics and keep device-wide VRAM measurements separate, since reserved memory describes the allocator’s pool. Keeping the model-loading peak separate also makes sense. I’d be glad to read the updated methodology when it’s ready.
3 Likes
In your 7B sweep, INT8 peaks at 27.0 GB and INT4 at 24.4 GB, above BF16’s 18.4 GB. The quantized weights alone can’t explain that, so the peak probably includes copies made while loading. Recording load peak, steady and per-run peak for every row, as your rocm_bench.py already does for BF16 and FP16, would make the comparison fair. The slower INT runs likely come from quanto itself, which only lists fast kernels for CUDA and is in maintenance mode, so they reflect quanto more than the MI300X.
2 Likes
Thanks for the valuable feedback! I agree that separating loading peak, steady-state memory, and per-run peak would make comparisons more reliable. I’ll audit the benchmark implementation and investigate the impact of Quanto’s ROCm kernel support before drawing conclusions about quantization performance.
1 Like