Hi AMD Developer Community! ![]()
I want to share GRETA Core, an AI inference and training engine that I am building from scratch in C++20. It has been designed from day one to run natively and efficiently on AMD hardware (both Ryzen CPUs and Radeon GPUs via HIP/ROCm and Vulkan 1.4).
What is GRETA Core?
GRETA Core is a sovereign AI engine powered by a ~1.2 billion parameter Decoupled Mixture of Experts (MoE) architecture, featuring an original routing mechanism called 4D-Lattice Routing. Unlike traditional monolithic models, GRETA splits its knowledge base into 320 specialized experts organized in a 3-tier hierarchy (64 Master + 192 Sub + 64 Micro). These experts are hot-swapped dynamically from disk to memory depending on the router’s real-time trajectory.
Model Specifications:
| Parameter | Value |
|---|---|
| Total Parameters | ~1,228M (~1.2B) |
| Transformer Layers | 6 |
| Hidden Dimension | 512 |
| FFN Dimension | 2,048 (SwiGLU) |
| Total MoE Experts | 320 (3 tiers) |
| Attention Heads | 8 (GQA 4:1, 2 KV heads) |
| Vocabulary | 16,384 tokens (Proprietary morpho-semantic tokenizer) |
| Max Context | 4,096 tokens |
| Cognitive Domains | 16 |
Key Advantages of GRETA: From PC to Cloud Servers
GRETA Core’s decentralized architecture offers distinct, high-impact advantages depending on where it is deployed:
-
For Personal Use / Consumer Hardware:
- Extreme Memory Efficiency: By decoupling and hot-swapping experts from disk instead of holding the entire model in RAM, users can run and train a ~1.2B parameter engine on standard desktop PCs or laptops with just 16 GB of RAM.
- Uncompromising Local Performance: Optimizations like CCX thread affinity bypass local processor latency penalties, getting the maximum performance from Ryzen CPUs and Radeon GPUs without relying on cloud-based APIs.
-
For Enterprise / Security and Modularization:
- Absolute Data Sovereignty: Native C++ local execution guarantees zero external API calls or third-party cloud dependencies. Sensitive enterprise data never leaves your secure corporate network.
- Modular “Engram” Architecture: Organizations can create and plug in modular specialized experts (e.g., tailored for legal, finance, coding, or customer support) as standalone
.gcorefiles, without needing to retrain the underlying base model.
-
For Cloud Deployments and Scalability:
- Multi-backend Portability: The combination of HIP Late Binding and Vulkan compute shaders allows the exact same compiled binary to deploy instantly on any cloud node (CPU-only, hybrid, or dedicated GPU clusters) without container orchestration overhead or complex ROCm driver version locks.
- Drastically Lower Infrastructure Costs: GRETA’s lightweight memory footprint allows for hosting significantly more concurrent engine instances per server, directly translating to reduced hardware resource costs and lower memory bandwidth requirements for cloud providers.
Why is this relevant to the AMD Community?
GRETA Core has been optimized specifically for AMD systems. Here are the core hardware-aligned features implemented so far:
1. AMD HIP Late Binding (No ROCm Build-Time Dependency)
To avoid distribution headaches, I implemented a Late Binding system that detects amdhip64.dll (or libamdhip64.so) at runtime using LoadLibrary/dlopen and resolves HIP entry points dynamically (hipInit, hipMalloc, hipModuleLaunchKernel, etc.).
This approach brings key benefits:
- A single binary runs on systems with or without an AMD GPU — no recompilation needed.
- Developers do not need the full ROCm SDK installed just to compile the project.
- CPU fallback is transparent and automatic if no AMD GPU is detected.
cpp
// Late Binding: Dynamic loading of HIP runtime
hip_lib = LoadLibraryW(L"amdhip64.dll");
hipInit_ptr = (fn_hipInit)GetProcAddress(hip_lib, “hipInit”);
hipMalloc_ptr = (fn_hipMalloc)GetProcAddress(hip_lib, “hipMalloc”);
// … all HIP symbols are resolved dynamically at runtime
2. Specialized HIP Kernels for AMD GPUs
When HIP is loaded, GRETA dispatches performance-critical operations directly to Radeon hardware:
- GEMM (
launch_hip_gemm) — General matrix multiplication. - Causal Attention (
launch_hip_attention) — Multi-head attention with GQA support. - MoE Routing (
launch_hip_moe_routing) — Top-K expert selection offloaded to GPU. - RMSNorm (
launch_hip_rmsnorm) - Softmax (
launch_hip_softmax) - Stochastic Rounding FP4 (
stochastic_round_fp4.hip) — Low-precision training quantization.
3. Ryzen CCX & SMT Thread Affinity Optimization
To maximize CPU cache efficiency on Ryzen architectures, the engine implements core affinity mapping to prevent L3 cache thrashing across CCX boundaries:
// Affinity Mask 0x5555: Selects even physical cores only (bypassing SMT threads)
// This keeps execution localized within a single CCX
DWORD_PTR affinityMask = 0x5555;
SetThreadAffinityMask(GetCurrentThread(), affinityMask);
By keeping high-frequency worker threads on physical cores within the same CCX, we achieved a ~30-35% reduction in latency during the forward pass by eliminating inter-CCX cache invalidations.
4. A Suite of 17 Vulkan Compute Shaders (SPIR-V)
For cross-platform GPU acceleration, GRETA uses Vulkan 1.4 compute pipelines. The suite includes:
| Shader | Purpose |
|---|---|
fp4_decompress.comp |
FP4 → FP16 decompression using dynamic LUTs |
fused_attention.comp |
Causal multi-head self-attention |
mlp_forward.comp / mlp_backward.comp |
Forward/backward passes for SwiGLU FFNs |
moe_routing.comp |
MoE routing pipeline executed on GPU |
muon_step.comp |
Weight updates via the Muon optimizer |
paged_attention.comp |
PagedKVCache for efficient long-context management |
qat_forward.comp |
Quantization-Aware Training simulation |
grouped_matmul.comp |
Batched matrix multiplication for grouped experts |
rmsnorm.comp / softmax.comp |
Layer normalizations and probability scaling |
vit_forward.comp / mamba_forward.comp |
Multimodal and sequential state-space layers |
| + 4 more | Projectors, resampling, and quantized matmuls |
5. Cache-Blocked CPU Backend with AVX2 & FMA3
For the CPU execution path, the matrix multiplication kernel utilizes:
- 32×32×64 cache blocking to maximize L1/L2 cache hit rates.
- AVX2 FMA instructions (
_mm256_fmadd_ps) achieving 8 FLOPs per cycle. - 4× Loop unrolling to saturate execution ports.
- OpenMP thread-level parallelism across matrix columns.
4D-Lattice Routing Architecture
The MoE routing pipeline is highly specialized, driving tokens through a 10-stage process:
- RMSNorm of the input token vector.
- 3D Projection mapping the token to lattice coordinates (cx, cy, cz).
- Discrete Gating using learnable router weights.
- Anisotropic RBF calculating distance using learnable per-expert sigmas.
- Expert Identity Embeddings incorporating learned expert semantics.
- Trajectory-Aware Routing adding temporal velocity biases.
- Hierarchical Region Routing filtering routing targets space-wise.
- Memory-Routing Fusion integrating recalled regions from the memory fabric.
- Entropy Floor ensuring diversity (capped at a ~4.16 bits minimum).
- Stickiness / Grouping preserving expert continuity across sequential tokens.
Current Training Status
The model is currently training locally on CPU (AMD Ryzen) on a bilingual (English/Spanish) dataset using the following settings:
- Strict Low-Memory Mode: Capped to 2 active transformer layers out of 6, and Top-K=1 gating, fitting easily within a 16 GB RAM envelope.
- Corpus: 292 sequences, running 2 training epochs.
- Optimizer: Muon with cosine learning rate annealing.
- Gradient Accumulation: 4 steps.
Engine Tech Stack
- Language Standard: C++20 (MSVC v143 / GCC 11+)
- Build System: CMake 3.22+
- GPU APIs: Vulkan 1.4 (SPIR-V) + AMD HIP (Late Binding) + OpenCL (fallback)
- CPU Instruction Sets: AVX2/FMA3 + OpenMP
- Target OS: Windows 10/11, Linux (Ubuntu 22.04+)
Roadmap
- Complete training of the bilingually aligned base model.
- Enable full 6-layer execution and Top-K=2 routing with expanded memory/GPU configurations.
- Publish comparative performance benchmarks on AMD consumer hardware (Ryzen 7 / Radeon RX 7000 series).
- Launch the Engram Marketplace: modular expert files (
.gcoreformat) that can be shared and hot-swapped dynamically.
I would love to get your thoughts on this architecture, especially:
- Experiences with HIP Late Binding vs. static compilation against the ROCm toolkit.
- L3 cache optimization strategies on Zen 4/5 architectures.
- Best practices for Vulkan compute shader tuning on AMD RDNA architectures.
Thanks for reading! ![]()
— Leandro Emanuel Timberini
Creator and Lead Architect of GRETA Core