While running AMD’s chatbot_vllm.ipynb, I recorded a useful distinction for other beginners: model-weight memory is only one part of inference memory.
In this run, the notebook used Qwen/Qwen3-4B-Instruct-2507 with vLLM 0.16.0, BF16 and tensor parallel size 1. The model-loading log reported 7.67 GiB. A separate log reported 158.58 GiB available for KV cache. These numbers describe different things: neither is a measurement of peak GPU memory for the whole application.
For a deployment estimate, I now separate:
- Model weights, using a rough parameter-count × bytes-per-weight estimate.
- KV cache, which depends on model architecture, context length and concurrent sequences.
- Activations, runtime workspaces, graph capture and framework overhead.
- Headroom for the workload being tested.
Two practical lessons from this notebook:
- An interactive chat loop must be exited before the kernel can execute the next cell. In this notebook, typing
exitreturned the kernel to the next task. - Save and reload the notebook to verify outputs persist. I checked this after running the custom Chinese assistant and the ipywidgets chat UI.
The notebook described the platform as MI300 with 192 GB memory; I did not independently query the device model. These are learning observations from one run, not a hardware benchmark or a throughput claim.
Which measurements do you normally record when checking whether a vLLM workload fits: peak allocated memory, reserved memory, KV-cache utilization, or all three?