I’m Deepakkumar Malkar, solo founder of
NiftyOracle from Vaijapur, Maharashtra,
India. I’m building CodeSymb — a 5 Trillion
parameter Mixture-of-Experts coding AI
from scratch, with zero funding and zero team.
What I’ve Built So Far (All Free Hardware):
Published on HuggingFace:
1.49M coding samples dataset (55+ downloads)
100,000 unique error-fix pairs (no equivalent exists publicly)
50,000 code execution traces
Custom 50K vocab BPE tokenizer (Hindi + English)
4 model repositories ready for training
Active Training:
CodeSymb-1B (1.08B params) training on
Kaggle T4 — step 8,500+ with stable loss
Architecture Designed:
5T MoE: 96 layers, 64 experts/layer
2 active experts per token (156B active)
GQA attention, YaRN RoPE (128K context)
DeepSpeed ZeRO-3 + CPU offload ready
Why AMD MI300X is Perfect for CodeSymb:
The AMD MI300X with 192GB HBM3 is
architecturally ideal for MoE models:
I’ve already verified our DeepSpeed config
loads correctly. The architecture is proven.
We just need the compute.
What I’m Asking:
Access to AMD MI300X through the
Developer Program for CodeSymb 5T training.
This would make CodeSymb the first
frontier-scale AI model trained on AMD
hardware in India — a story AMD and
India’s tech community would both celebrate.
In return: Full ROCm compatibility benchmarks published AMD credited in all publications Performance comparison vs NVIDIA A100 Open source training code on GitHub HuggingFace model card credits AMD
Hello, heads up on the memory maths, since it changes the ask a lot. The 156B active is a compute number, not a memory one. Every expert still has to be resident, since any token can route to any of them and all of them get gradient updates. So training holds all 5T.
At roughly 16 bytes per param for ZeRO-3 mixed precision, weights plus grads plus fp32 master plus two Adam states, 5T comes to around 80TB. One MI300X is 192GB, so that is hundreds of cards before activations.
The datasets and the tokenizer are genuinely good work though, and that is the part worth leading with. Developer Cloud does free MI300X hours, and joining the AMD Developer Program to finish your 1B and publish clean ROCm numbers is something worth to do. I will leave the link to the program AMD AI Developer Program - AMD Developer Portal
Thank you for the correction — that is
genuinely useful and I appreciate you
taking the time.
You are right. I conflated compute
routing with memory residency. With
all experts live for gradient updates,
the full 80TB figure is what matters.
That changes the hardware ask completely.
The immediate and realistic goal is
finishing CodeSymb-1B with clean ROCm
numbers — that is actually achievable
on Developer Cloud.
Two questions if you do not mind:
How do I access the free MI300X
hours on AMD Developer Cloud?
Is the AMD AI Developer Program
the right entry point for this?
The dataset and tokenizer work is
already live at
if useful for reference.
Thank you again — this is exactly the
kind of feedback that saves months
of wasted effort.
Hi @CodeSymb_AI, impressive ambition! I’m still trying to wrap my head around all the technical details and math behind training a 5T MoE model—learning a lot from this thread. Best of luck with the project!