Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus

DeepSeek-AI released the DeepSeek-V4 series on April 24, 2026: a preview pair of MIT-licensed Mixture-of-Experts models, with DeepSeek-V4-Flash at 284B total parameters (13B activated) and a one-million-token context window. Flash pushes sparse attention further than any open-weight model before it: its 43 transformer layers interleave three different attention types, each layer sits inside a hyper-connection block rather than a plain residual, and every token is routed through 256 experts. Each of those choices breaks an assumption baked into stock Megatron-LM training code.

This blog walks you through enabling end-to-end DeepSeek-V4-Flash pretraining in Primus on AMD Instinct™ MI355X GPUs. You will learn what the architecture looks like layer by layer, which knobs Primus exposes to configure it, and — where most of the engineering went — the kernel work that took the model from it runs to it runs fast. All of it is BF16 training — see the endnotes for what that leaves out. Every config, launch script, and benchmark referenced here ships in the open-source Primus repository, so you can reproduce the run yourself.

The DeepSeek-V4-Flash architecture

DeepSeek-V4-Flash keeps the skeleton you already know from DeepSeek-V3 — a Transformer stack with DeepSeekMoE feed-forward layers and a Multi-Token Prediction head. What changed sits underneath: how attention reads the KV cache, and how residual connections carry signal between blocks. Pretraining also moves to the Muon optimizer for most parameters, keeping AdamW for the embedding, the prediction head, and the RMSNorm weights.

DeepSeek-V4-Flash architecture showing one transformer block with its four mHC mixers, plus detail panels for the three attention types, the MoE layer, and manifold-constrained hyper-connections

Figure 1: DeepSeek-V4-Flash architecture — one transformer block, with detail panels for the three attention types, the MoE layer, and mHC.

Figure 1 shows a single block. The overall shape is familiar: 43 layers, hidden size 4,096, a 129,280-token vocabulary, and a MoE layer in every block with one shared expert alongside 256 routed experts, six of which activate per token. Two details already depart from V3 — every sub-layer is wrapped in manifold-constrained hyper-connections (mHC) rather than a plain residual, and the first three MoE layers route tokens by hash instead of through the learned router.

The attention module is where most of the change lives. All 64 query heads read a single 512-dimensional latent that serves as both key and value, making the layer multi-query rather than multi-head. Queries arrive through a low-rank path of rank 1,024 that the sparse-selection indexer shares. And because 64 heads of 512 dimensions is a wide tensor to project back down to 4,096, the output projection splits into 8 groups that each pass through a 1,024-dimensional bottleneck.

Read more Enabling DeepSeek-V4-Flash Training on AMD Instinct MI355X GPUs with Primus — ROCm Blogs