# I’m building ASEMA — an inference architecture for running 763B+ parameter sparse MoE models on 32GB RAM + 8GB VRAM

**URL:** <https://devcommunity.amd.com/t/i-m-building-asema-an-inference-architecture-for-running-763b-parameter-sparse-moe-models-on-32gb-ram-8gb-vram/1144>\
**Category:** Artificial Intelligence (AI)\
**Tags:** ai, gpu, developer-program\
**Created:** [October 5, 2026, 1:45pm UTC](https://devcommunity.amd.com/t/i-m-building-asema-an-inference-architecture-for-running-763b-parameter-sparse-moe-models-on-32gb-ram-8gb-vram/1144 "2026-10-05T13:45:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sajeev\_ajay](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@sajeev\_ajay](https://devcommunity.amd.com/u/sajeev_ajay)\
**Post date:** [October 5, 2026, 1:45pm UTC](https://devcommunity.amd.com/t/i-m-building-asema-an-inference-architecture-for-running-763b-parameter-sparse-moe-models-on-32gb-ram-8gb-vram/1144/1 "2026-10-05T13:45:44Z")

</div>

I’m building ASEMA — an inference architecture for running 763B+ parameter sparse MoE models on 32GB RAM + 8GB VRAM

Hi AMD community,

I’m a student developer from India, and I’ve been working on something that started as a simple question:

What happens if you try to run an extremely large sparse Mixture-of-Experts model on hardware that is nowhere near large enough to hold the entire model in RAM or VRAM?

Instead of shrinking the model, pruning experts, distilling it, or sending inference to a cloud API, I decided to attack the memory and execution problem itself.

The result is ASEMA:

Adaptive Sparse-Expert Memory Architecture.

The core idea is to treat storage, system memory, GPU memory, and compute as a coordinated hierarchy.

NVMe storage  
↓  
Expert paging / asynchronous I/O  
↓  
RAM resident cache  
↓  
GPU resident working set  
↓  
MoE routing  
↓  
GPU expert execution  
↓  
Token generation

My current experimental platform is:

• AMD Radeon RX 580 8GB  
• AMD Ryzen 7 5700X  
• 32GB system RAM  
• Windows 11  
• Dual NVMe storage

The model I have been working with is DeepSeek-V4.1-Flash, a very large sparse MoE checkpoint with a total footprint vastly beyond the memory available on this machine.

The architecture I have been implementing includes:

• Real model tensor loading  
• Sparse MoE routing  
• Top-k expert selection  
• FP4 expert execution  
• Expert caching  
• Asynchronous storage I/O  
• Prefetching  
• RAM/VRAM residency management  
• CPU/GPU execution  
• Persistent GPU resources  
• KV-cache infrastructure  
• Telemetry and profiling  
• Memory-bounded execution  
• Correctness verification against reference calculations

One of the most interesting constraints is that I am deliberately NOT trying to load the entire model into RAM or VRAM.

The goal is to keep the complete model represented on storage while dynamically bringing the tensors required for the current computation into the active memory hierarchy.

This turns the problem from:

“How do I fit a gigantic model into memory?”

into:

“How do I design an inference runtime that intelligently moves only what computation needs, exactly when it is needed?”

That is the problem ASEMA is trying to solve.

I have also been experimenting with custom GPU execution on the RX 580 rather than treating the GPU simply as a conventional inference backend.

My previous experiments demonstrated GPU-accelerated FP4 expert execution and storage-to-GPU execution pipelines under strict memory constraints.

The next stage is much more ambitious.

I want to make ASEMA model-independent and investigate whether the same architecture can be extended from the current 763B-class experiment toward trillion-parameter and multi-trillion-parameter sparse MoE models.

I recently joined the AMD AI Developer Program, and I would especially like feedback from the AMD/ROCm community on the architecture.

In particular, I would love feedback on:

1. Which parts of the storage → RAM → VRAM pipeline would you redesign for AMD hardware?

2. What would be the best ROCm/HIP architecture for persistent expert execution and asynchronous expert movement?

3. Which AMD profiling tools should I use to identify the largest bottlenecks?

4. Are there AMD-specific approaches for improving sparse MoE expert execution that I should investigate?

5. Would this architecture be interesting as an open-source AMD/ROCm project?

I’m planning to open-source ASEMA once the current implementation reaches a reproducible state with proper benchmarks and documentation.

I’m not looking for hype — I’m looking for technically serious feedback.

If anyone here works with ROCm, GPU kernels, inference systems, sparse MoE models, or large-model serving, I would genuinely appreciate your thoughts.

Project: ASEMA  
Focus: Sparse MoE inference / memory hierarchy / expert paging / GPU acceleration  
Hardware: Ryzen 7 5700X + Radeon RX 580 8GB + 32GB RAM

This started as an experiment on a consumer GPU.

I’m now trying to turn it into a real inference architecture.

---

<div class="post-metadata">

**Author:** ![Dorothy\_Biswas](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@Dorothy\_Biswas](https://devcommunity.amd.com/u/Dorothy_Biswas)\
**Post date:** [October 6, 2026, 7:01am UTC](https://devcommunity.amd.com/t/i-m-building-asema-an-inference-architecture-for-running-763b-parameter-sparse-moe-models-on-32gb-ram-8gb-vram/1144/2 "2026-10-06T07:01:12Z")

</div>

Hi @sajeev_ajay,

Impressive work on ASEMA! Dynamically paging sparse MoE weights across storage, RAM, and VRAM is a solid approach for running large models under tight memory constraints.

---

<div class="post-metadata">

**Author:** ![Hakob\_Arzumanyan](https://dub1.discourse-cdn.com/amd/user_avatar/devcommunity.amd.com/hakob_arzumanyan/32/214_2.png) [@Hakob\_Arzumanyan](https://devcommunity.amd.com/u/Hakob_Arzumanyan)\
**Post date:** [October 6, 2026, 11:07am UTC](https://devcommunity.amd.com/t/i-m-building-asema-an-inference-architecture-for-running-763b-parameter-sparse-moe-models-on-32gb-ram-8gb-vram/1144/3 "2026-10-06T11:07:33Z")

</div>

Hi,  
Thank you for sharing your project,  
Your pipeline question has a neat answer for this particular model: the 196.6B of Engram tables are n-gram lookups, so they don’t need your paging hierarchy at all. Engines for Qwen3.8-Flash-Next already leave its similar n-gram table on NVMe and read only the rows each token needs. That leaves the routed experts as the real problem: V4.1 Flash activates about 16B parameters per output token, so your expert-cache hit rate will decide your speed: [deepseek-ai/DeepSeek-V4.1-Flash | vLLM Recipes](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash) On HIP, ROCm doesn’t support the RX 580 on Windows, so Vulkan compute is the GPU path there, and your Developer Program credits can cover HIP experiments on an MI300X. Looking forward to the open-source release 😃

---

<div class="post-metadata">

**Author:** ![Athos\_Felipe](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@Athos\_Felipe](https://devcommunity.amd.com/u/Athos_Felipe)\
**Post date:** [October 8, 2026, 2:10pm UTC](https://devcommunity.amd.com/t/i-m-building-asema-an-inference-architecture-for-running-763b-parameter-sparse-moe-models-on-32gb-ram-8gb-vram/1144/4 "2026-10-08T14:10:11Z")

</div>

For a HIP prototype on supported hardware, I would test a bounded pinned staging pool and two device slots.

Keep a host staging slot unchanged until its H2D copy completes; the CPU must check or wait for completion before refilling it. With separate copy and compute streams, record an event after the copy and make compute wait before reading that device slot. Also record an event after its last consuming kernel and make the copy stream wait before overwriting the same slot (`hipStreamWaitEvent`). Host and device reuse need separate protection.

`asyncEngineCount > 0` indicates overlap capability, not proof that transfers and kernels overlap in this pipeline. AMD describes these mechanisms in its [HIP asynchronous execution guide](https://rocm.docs.amd.com/projects/HIP/en/docs-7.0.0/how-to/hip_runtime_api/asynchronous.html).

I would time NVMe reads on the host, inspect transfers and kernels in a GPU trace, and compare the same expert sequence with sequential and pipelined execution.

How much of the current per-token latency comes from NVMe reads, H2D transfers and expert execution?
