AI
Bringing up a 1.58-bit (BitNet) LLM conversion on a Ryzen AI Max+ 395
An independent developer successfully used a Nimo AI Mini PC, powered by an AMD Ryzen AI Max+ 395 with 128 GB of unified memory, as a local machine to retrain
Key takeaways
- The Nimo AI Mini PC's 128 GB unified memory successfully runs 7B-scale LLM training experiments locally.
- Teacher-student distillation at 7B scale peaks at 87 GB of memory, which does not fit on consumer discrete GPUs.
- Stock ROCm segfaults on the Ryzen AI Max+ 395's gfx1151 architecture on first dispatch, requiring AMD's TheRock wheel.
- The fp32 matrix path on RDNA 3.5 is roughly 100 times slower than bf16, making bf16 mandatory for training.
- Teacher-free quantization-aware training has a negative size-scaling problem at 7B scale, whereas distillation holds performance flat.
An independent developer successfully used a Nimo AI Mini PC, powered by an AMD Ryzen AI Max+ 395 with 128 GB of unified memory, as a local machine to retrain and convert Qwen2.5-7B into a 1.58-bit ternary model (BitNet b1.58). The project demonstrated that the 128 GB unified memory can handle memory-intensive teacher-student distillation workloads peaking at 87 GB, which exceed the capacity of consumer discrete GPUs. However, the developer noted that stock ROCm software segfaults on the hardware, requiring AMD's TheRock wheel, and that using bf16 is mandatory to avoid a massive performance penalty on the RDNA 3.5 architecture.
By the numbers
- 128 GB
- Unified memory capacity of the mini PC
- 87 GB
- Peak memory footprint for teacher-student distillation at 7B
- 100x
- Performance penalty of fp32 matmul vs bf16
- 67 GB
- Peak memory footprint for teacher-free 7B training
- ~256 GB/s
- Unified memory bandwidth of the system
Want this on autopilot?
We file the AI news, make the video, and post it for you.
Common questions
- marketing.blog.post.faq.what_happened
- The Nimo AI Mini PC's 128 GB unified memory successfully runs 7B-scale LLM training experiments locally.
- marketing.blog.post.faq.source
- marketing.blog.post.faq.source_answer
