Building CodeSymb — India's First 5T MoE Coding AI on AMD MI300X

:india: Building India’s First 5T MoE Coding AI

on AMD Hardware — Seeking MI300X Access

Hi AMD Community!

I’m Deepakkumar Malkar, solo founder of
NiftyOracle from Vaijapur, Maharashtra,
India. I’m building CodeSymb — a 5 Trillion
parameter Mixture-of-Experts coding AI
from scratch, with zero funding and zero team.


What I’ve Built So Far (All Free Hardware):

Published on HuggingFace:

  • 1.49M coding samples dataset (55+ downloads)
  • 100,000 unique error-fix pairs (no equivalent exists publicly)
  • 50,000 code execution traces
  • Custom 50K vocab BPE tokenizer (Hindi + English)
  • 4 model repositories ready for training

Active Training:

  • CodeSymb-1B (1.08B params) training on
    Kaggle T4 — step 8,500+ with stable loss

Architecture Designed:

  • 5T MoE: 96 layers, 64 experts/layer
  • 2 active experts per token (156B active)
  • GQA attention, YaRN RoPE (128K context)
  • DeepSpeed ZeRO-3 + CPU offload ready

Why AMD MI300X is Perfect for CodeSymb:

The AMD MI300X with 192GB HBM3 is
architecturally ideal for MoE models:

  • Active parameters per token: ~156B
  • Fits in single MI300X with ZeRO-3!
  • ROCm + PyTorch = our stack is compatible
  • 5.3 TB/s memory bandwidth handles
    MoE expert switching perfectly

I’ve already verified our DeepSpeed config
loads correctly. The architecture is proven.
We just need the compute.


What I’m Asking:

Access to AMD MI300X through the
Developer Program for CodeSymb 5T training.

This would make CodeSymb the first
frontier-scale AI model trained on AMD
hardware in India
— a story AMD and
India’s tech community would both celebrate.

In return:
:white_check_mark: Full ROCm compatibility benchmarks published
:white_check_mark: AMD credited in all publications
:white_check_mark: Performance comparison vs NVIDIA A100
:white_check_mark: Open source training code on GitHub
:white_check_mark: HuggingFace model card credits AMD


My Background:

  • Live production fintech app on Google Play
  • LFX Mentee at Linux Foundation (CNCF Meshery)
  • Google Cloud Gen AI Academy — APAC Cohort 3

HuggingFace: NiftyOracleAI (NiftyOracle)
Email: niftyoracle.in@gmail.com

Nothing is impossible. :fire:

— Deepakkumar Malkar | NiftyOracle :india:

Hello, heads up on the memory maths, since it changes the ask a lot. The 156B active is a compute number, not a memory one. Every expert still has to be resident, since any token can route to any of them and all of them get gradient updates. So training holds all 5T.

At roughly 16 bytes per param for ZeRO-3 mixed precision, weights plus grads plus fp32 master plus two Adam states, 5T comes to around 80TB. One MI300X is 192GB, so that is hundreds of cards before activations.

The datasets and the tokenizer are genuinely good work though, and that is the part worth leading with. Developer Cloud does free MI300X hours, and joining the AMD Developer Program to finish your 1B and publish clean ROCm numbers is something worth to do. I will leave the link to the program AMD AI Developer Program - AMD Developer Portal

1 Like

Thank you for the correction — that is
genuinely useful and I appreciate you
taking the time.

You are right. I conflated compute
routing with memory residency. With
all experts live for gradient updates,
the full 80TB figure is what matters.
That changes the hardware ask completely.

The immediate and realistic goal is
finishing CodeSymb-1B with clean ROCm
numbers — that is actually achievable
on Developer Cloud.

Two questions if you do not mind:

  1. How do I access the free MI300X
    hours on AMD Developer Cloud?
  2. Is the AMD AI Developer Program
    the right entry point for this?

The dataset and tokenizer work is
already live at

if useful for reference.

Thank you again — this is exactly the
kind of feedback that saves months
of wasted effort.

— Deepakkumar | NiftyOracle :india:

Hi Hakob Arzumanyan,

Quick update — I just earned the
AMD ROCm Certified Associate certification
from AMD AI Academy today.

Credential ID: e1f2a811-e682-4812-a284-349893a59730

I wanted you to know because our
conversation inspired me to complete
the certification properly before
requesting MI300X access.

I believe this demonstrates I understand
the ROCm stack at a technical level
and can use MI300X access productively.

Could you point me toward the right
contact for the Developer Cloud
free hours you mentioned?

Thank you again for your guidance.

— Dev Malkar
AMD ROCm Certified Associate
NiftyOracle :india:

Hello, congratulations on completing the certification : D

Here is the link to join the AMD AI Developer Program AMD AI Developer Program

Hi @CodeSymb_AI, impressive ambition! I’m still trying to wrap my head around all the technical details and math behind training a 5T MoE model—learning a lot from this thread. Best of luck with the project!

@Dorothy_Biswas Thank you so much!

Quick update since you’ve been
following — plan evolved a bit!

WHAT’S REAL TODAY:
:white_check_mark: 1B model fully trained (loss 1.17)
:white_check_mark: 413 Rank-100 Expert LoRAs live
:white_check_mark: codesymb.com launching now
:white_check_mark: Built entirely from a phone! :mobile_phone:

MASSIVE NEWS just happened:
AMD + Supermicro gave us scheduled
access to an 8× MI325X cluster
(Nov 16-20)!

288GB HBM3E × 8 GPUs = 2.3TB!

With this:
→ 10,000 LoRAs in one day
→ 10B model by end of week

The 5T MoE is still the ultimate
vision — but we’re building the
foundation first, brick by brick.

Nothing is impossible.
Just a matter of time! :fire:

codesymb.com :globe_showing_americas: