Hi everyone, I’m Kaushik. Just joined the AI Developer Program and figured I should introduce myself before I start asking questions.
My background is mostly ML infra and DevOps - training pipelines, Docker, CI/CD, that kind of thing. Lately I’ve been contributing to NVIDIA NeMo, so pretty much all my GPU work so far has been on the CUDA side. I joined here because I want to find out how much of that actually carries over to ROCm and MI300X.
First thing I’m going to try is getting a normal PyTorch training job running on Developer Cloud and writing down everything that breaks along the way. If it turns into something useful I’ll post it here.
Two things I wanted to ask:
- for someone coming from CUDA, what’s the one thing you wish somebody had told you on day one?
- is there anything on the ROCm side that’s under-documented right now and worth writing about? happy to take it on since I’ll be learning it anyway.
Thanks, looking forward to being around here.
Hello great questions asked. More carries over than you’d expect, and that’s the day one thing. PyTorch training scripts usually run unmodified, you keep writing torch.cuda because the ROCm build reuses that namespace instead of inventing a parallel one, and HIP mirrors the CUDA runtime call for call. Your NeMo work transfers further than you think. Where the time goes instead is proving the box came up right before trusting a number, which is why your probe is the useful idea and the doc worth writing.
Thanks, that is the reassurance I was after on the porting side.
The torch.cuda point is the one the probe already leans on. It reads torch.version.hip to tell a ROCm build from a CPU-only wheel, precisely because the shared namespace means a pip install torch that cannot see the card looks identical to a working one until you ask it directly.
On proving the box came up right: the numeric-formats thread cost me one of my own checks this week. I was using the presence of /dev/kfd as the liveness signal, and it can be present on a card whose driver never bound, so it reported healthy on exactly the failure it existed to catch. It is rocminfo resolving a gfx target now.
The other thing it picked up, which you may already know: e4m3fnuz has no infinity, so a -inf mask fill lands as NaN, and its max is 240 rather than 448. Both are testable on a laptop with no GPU, and it is the same root cause as vLLM’s fp8 path refusing to start on gfx942.
Writing the doc.
On the documentation side, it’d be interesting having a practical CUDA → ROCm troubleshooting guide. Real examples of what breaks when moving an existing training pipeline, including dependencies, container setup, PyTorch compatibility, profiling, and performance tuning.
Welcome to the community!