Training an LLM from scratch on 8x AMD Instinct MI325X: progress update 🚀

I’m a solo founder from India building CodeSymb-1B, a 1.18B-parameter coding model trained from scratch. Thanks to the Supermicro JumpStart program, it’s training on an H14 8U system with 8x AMD Instinct MI325X (256 GB HBM3E each), and the session was just extended to Nov 27.

Where it stands right now:

  • 62B tokens trained, heading to 265B
  • ~505K tokens/s across 8 GPUs = 35% MFU (FLOPs counted honestly, embedding lookup excluded)
  • BF16, PyTorch 2.9 + ROCm 6.3, DDP + torch.compile
  • Every pretraining token is unique: a single pass over ~269B tokens of code, math and educational web text
  • 20.5B-token instruction-tuning set ready, built from 17 open datasets with 3.3M exact duplicates removed

What moved the needle on ROCm:

  1. TunableOp ON (PYTORCH_TUNABLEOP_ENABLED=1): ~475K → ~495K tok/s. Pay the one-time tuning cost once, save the CSVs, reuse them on every restart.
  2. Compiling the trunk + output head + cross-entropy as one graph: peak memory 208 → 153 GiB per GPU.
  3. Native GQA in SDPA flash attention (enable_gqa), verified forward + backward against the reference path.
  4. Removing per-step GPU→CPU syncs (loss read every 20 steps, pinned async batch uploads).
  5. Training data in RAM (tmpfs) instead of NVMe.
    Together: ~505K tok/s and a steady 35% MFU on MI325X.

Next: SFT, GRPO and DPO, 10K LoRA adapters, and a public benchmark plus technical report with the full ROCm training recipe once training completes.

Checkpoints and data: NiftyOracleAI/CodeSymb-1B-MI325X · Hugging Face

Huge thanks to AMD and the Supermicro JumpStart team.

Hello, thanks for sharing this. It’s really nice to see a from-scratch run holding 35% MFU on MI325X, and your notes on TunableOp and compiling everything as one graph will help other people training on these cards. One thing caught my eye in your screenshot. Your learning rate is still going up around step 80K (1.79e-4 to 2.62e-4 in about a thousand steps), and the loss creeps from about 1.49 to 1.52 at the same time. If you restarted with a fresh warmup on purpose after the extension, all good. If not, the scheduler state might not have loaded from the checkpoint, so it’s worth a quick look before it hits peak LR. Good luck with the rest of the run!

Thanks for the sharp eye, Derenik! Yes, that’s intentional. When the JumpStart session was extended, I raised the pretraining target from 100B to 265B tokens. That re-anchors the cosine schedule, so the LR has to come back up. To avoid a sudden jump, I glide it from 1.04e-4 to the new cosine value (~2.66e-4) over 2,000 steps rather than restarting warmup from zero. The small loss bump (~1.49 → 1.52) is the usual cost of re-warming, plus the run now trains only on never-seen data. I’ll post an update once it settles back down. Appreciate you looking!

Thanks for explaining. Gliding to the new cosine value instead of restarting warmup from zero is a great way to handle the extension. At the rate in your screenshot, the glide should finish about 50 steps after step 80307, so the loss should settle soon. Looking forward to the update and the technical report.

Thanks Derenik! Exactly: the glide ran 2,000 steps from the restart at step 78,287 and topped out at 2.62e-4. Loss plateaued about +0.05 above where it was and is now starting to settle as the LR decays. I’ll post an update with the recovery curve once it’s back below the pre-extension level. Appreciate you following the run!

Thanks for the details. Looking forward to the recovery curve and the technical report, and good luck with the rest of the run!

1 Like

Update: 151B tokens on 8× MI325X — numbers, lessons, and what’s next

Quick progress report on CodeSymb-1B, a 1.18B-parameter code model trained from scratch on a Supermicro H14 8U with 8× AMD Instinct MI325X (JumpStart program). All numbers below come from the run’s own logs (9,933 metric records, one every 20 steps).

Results

  • 151.28B tokens trained from scratch (loss 5.33 → 1.065)
  • 488–508K tokens/s median by phase, 34.6% median MFU (35.5% best), BF16
  • 89.7 h of training inside a ~111 h window (81%)
  • Stack: Ubuntu 24.04, ROCm/HIP 6.3, PyTorch 2.9.1+rocm6.3, DDP + torch.compile — PyTorch’s built-in kernels, tuned with TunableOp

What mattered most on MI325X

  1. TunableOp is the single biggest lever. Without it: 300–408K tok/s. With tuned GEMMs: ~505K. First tuning took ~26 min; saving the per-GPU result files and reusing them cut startup to ~20 s. Save these files next to your checkpoints.
  2. 256 GB HBM3e changes the batch math. 48 sequences × 2,048 tokens per GPU fit with no gradient accumulation (786K tokens per step).
  3. The host CPUs can run your whole data pipeline beside training. On 2× EPYC 9575F we pretokenized 260B tokens (30.8 h), deduplicated 247M documents (14 min) and built a 22.5B-token SFT set (6.2 h) while training stayed at 488–496K tok/s.
  4. Make stops lossless. A stop file triggers a full save (weights, optimizer, data position); full checkpoints every 30 min, milestones every 10B tokens. Every restart was cheap, including a clean stop 54 min before the Nov 20 maintenance cutoff.

A data fix worth sharing: our SentencePiece tokenizer (NFKC, no byte fallback) silently deleted newlines and indentation — fatal for Python. Instead of retraining the tokenizer, we mapped 7 token IDs that never occurred in the data to newline/tab/space runs. Same vocabulary, same model; at 130.7B tokens we switched to 189.5B never-seen, deduplicated tokens with formatting kept.

Honest notes: median throughput after one mid-run restart was 2–4% lower than before it, and we haven’t isolated why (GPUs were running near their ~1,000 W limit). The base model is not instruction-tuned or evaluated yet.

What’s next: pretraining to 300B, then SFT (22.48B tokens, 4,096 context, reasoning + grounded RAG answers with citations), GRPO + DPO, and HumanEval/MBPP evals — about 5 days of compute once access resumes.

Full benchmark report (PDF): https://huggingface.co/NiftyOracleAI/CodeSymb-1B-MI325X/resolve/main/CodeSymb_MI325X_Benchmark_Report.pdf
Model card and checkpoints: NiftyOracleAI/CodeSymb-1B-MI325X · Hugging Face

Thanks to Supermicro and the JumpStart team for the access, and to AMD for the MI325X stack. Happy to answer questions on the ROCm setup — ask away.