vLLM notebook notes: model weights are not total GPU memory

While running AMD’s chatbot_vllm.ipynb, I recorded a useful distinction for other beginners: model-weight memory is only one part of inference memory.

In this run, the notebook used Qwen/Qwen3-4B-Instruct-2507 with vLLM 0.16.0, BF16 and tensor parallel size 1. The model-loading log reported 7.67 GiB. A separate log reported 158.58 GiB available for KV cache. These numbers describe different things: neither is a measurement of peak GPU memory for the whole application.

For a deployment estimate, I now separate:

  • Model weights, using a rough parameter-count × bytes-per-weight estimate.
  • KV cache, which depends on model architecture, context length and concurrent sequences.
  • Activations, runtime workspaces, graph capture and framework overhead.
  • Headroom for the workload being tested.

Two practical lessons from this notebook:

  1. An interactive chat loop must be exited before the kernel can execute the next cell. In this notebook, typing exit returned the kernel to the next task.
  2. Save and reload the notebook to verify outputs persist. I checked this after running the custom Chinese assistant and the ipywidgets chat UI.

The notebook described the platform as MI300 with 192 GB memory; I did not independently query the device model. These are learning observations from one run, not a hardware benchmark or a throughput claim.

Which measurements do you normally record when checking whether a vLLM workload fits: peak allocated memory, reserved memory, KV-cache utilization, or all three?

Hello, for me it’s really just two lines that vLLM already prints. At startup you get “Maximum concurrency for N tokens per request: X.XXx”, which tells you how many full length requests actually fit. Then while it runs the log shows “GPU KV cache usage: X%”. Torch peak and reserved don’t tell you much here, since vLLM grabs its pool up front and just holds it.