Thanks for sharing TensorFly. On the 300 CTAs, grid.sync() needs a hipLaunchCooperativeKernel launch, and the runtime then checks the grid against its occupancy limit and returns hipErrorCooperativeLaunchTooLarge instead of hanging. So size it from hipOccupancyMaxActiveBlocksPerMultiprocessor times the CU count rather than a fixed 300. I couldn’t find where tf_persistent_decode_kernel gets launched in the v3.0 zip, so that’s where it would go. On vmcnt(0), LLVM’s gfx942 memory model already emits s_waitcnt vmcnt(0) and a cache invalidate after an acquire load like yours, so the inline asm is redundant. One small thing I noticed: benchmark() compares a 50 MB hipMemcpy with a cache hit that’s only a map lookup, so the 82x is a copy versus no copy. Your cold_ms is the like-for-like number. Good luck with the runs!
Thank you so much for this detailed
review — exactly the kind of feedback
I needed from someone with real gfx942
knowledge!
You’re absolutely right on all three
points:
On 300 CTAs:
I’ll replace the hardcoded 300 with:
```cpp
int safe_ctas;
hipOccupancyMaxActiveBlocksPerMultiprocessor(
&safe_ctas,
tf_persistent_decode_kernel,
TF_THREADS_PER_CTA, 0);
safe_ctas *= props.multiProcessorCount;
```
Much cleaner and hardware-adaptive!
Will push this fix today.
On vmcnt(0) asm:
Didn’t know LLVM already handles this
for gfx942 — removing the redundant
inline asm now. Good to know!
On the 82x benchmark:
You’re 100% correct. The fair
comparison is cold_ms (hipMemcpy)
vs cold_ms (TensorFly first load).
I’ll fix benchmark() to compare
like-for-like properly.
The tf_persistent_decode_kernel launch
is currently a placeholder — I’m
implementing the full cooperative
launch on MI325X × 8 starting Nov 16!
Will share real benchmark results
here on Nov 21 with the corrected
methodology.
Thank you for taking the time to
review this! Really appreciate having
AMD engineers look at the code.