Hey, I also have a Strix Halo box. Which model are you running exactly? The Q8 and BF16 are half the size of Qwen3-Coder-Next-UD-Q8 I have been running before. I tested 3.8-BF16 and UD-Q8 but both run at abysmal speed in comparison to Coder Next (around 8 t/s during chat). Which one gives you best results?
You need to use MTP (Multi token prediction) to get acceptable speeds on a Strix Halo. On llama.cpp you just need to add --spec-type draft-mtp --spec-draft-n-max3 -np 1 to your arguments.
You should see around 15-20 t/s in coding tasks at Q8, which will still be much slower than Qwen3-Coder-Next-UD-Q8 since it’s 27b active parameters instead of 3b.
As for Q8 vs BF16, it usually makes barely to no difference in quality. Speed is higher on Q8 though. Still testing with 3.8 since it just released.
27b fits easily into 128 unified RAM, even unquantized. So something was definitely wrong.
Running 3.8-27b on my Strix Halo box right now.
Hey, I also have a Strix Halo box. Which model are you running exactly? The Q8 and BF16 are half the size of Qwen3-Coder-Next-UD-Q8 I have been running before. I tested 3.8-BF16 and UD-Q8 but both run at abysmal speed in comparison to Coder Next (around 8 t/s during chat). Which one gives you best results?
You need to use MTP (Multi token prediction) to get acceptable speeds on a Strix Halo. On llama.cpp you just need to add
--spec-type draft-mtp --spec-draft-n-max 3 -np 1to your arguments.You should see around 15-20 t/s in coding tasks at Q8, which will still be much slower than Qwen3-Coder-Next-UD-Q8 since it’s 27b active parameters instead of 3b.
As for Q8 vs BF16, it usually makes barely to no difference in quality. Speed is higher on Q8 though. Still testing with 3.8 since it just released.
Thank you, I will check this out. So what I got from this is that the speed may be slower but results should be more accurate?
Yes, 27b is going to produce much better results.
On a DGX with 128G unified I could push Qwen3.6-27B to 8-bit and it had pretty decent response. It just croaked with MLX on the Mac.
Waiting to see what Apple announces next month. Judging by RAM prices, it’ll be waaay outside my budget.