Glad they set a time rather than leaving it open
I got a 7900 XTX a while back to run 3.6 27B assuming it might be a while before there’s anything better, but the 3.8 announcement is a pleasant surprise. 3.6 27B is more or less on par with Claude Sonnet 4.6 for coding according to benchmarks and my own experience and I’m hoping 3.8 will be more like Sonnet 5.
Incidentally, Newegg isn’t the greatest, but I did get a new 7900 XTX for like $730 after trading in my old RTX 3070. With used 3090s going for $1200 / $2k new or 4090s going for $3500 new, I think the 7900 XTX is really the only sanely priced 24GB GPU left. ROCm is slower than CUDA due to less maturity and you might be on your own getting certain things working that only support CUDA, but man does llama.cpp work well. With 3.6 27B, I’m getting 200k context, 700 t/s prefill on average (1000+ with small context) and 20 t/s decode without MTP or much optimization (hoping to get like 40-50 decode after spending more time).
Those prices make me mad. I will never spend more than $300 on a gpu.
In what year’s money?
I really hope the 27b does not lack non-thinking mode and the vision encoder. That would be a huge downgrade over 3.6.
How much does it even cost to run that stuff? Do you need your own private data center?
Nope. For 27B a 16GB video card (8 in a pinch) + 16 (32 better) GB RAM will work for reasonable context, if not screamingly fast. Any M series Macbook with 32 Gb+ (24 in a pinch). Hell, there are setups to run even larger models of a nvme drive if you have enough patience.
Smaller models like 8B are fine on a CPU, or even a phone.
I have my eyes on old 32gb amd instinct cards. One of those from ali will cost you ~450-500 euro, which is a lot for old unefficient card, but does provide amazing bandwidth and 32gb of vram. With just one you can run those models almost with full 256k context (or whatever the number there is). Just 4 of those and you can even run models that come close to things like claude and have 1m token context window
Sadly, I can’t justify it — my rig already have 7900xtx + 4060ti (so 32gb vram in total) and realistically I know that there no reason to buy it right now in memory crisis. In couple of years either Chinese manufacturers will catch up ram wise or someone will come up with dedicated device/accelerator for half the price. Or done other similar breakthrough will happen, maybe 4/8 channels of ram on regular desktop will become a thing, instead of being HEDT/server only
I don’t think you can run 27B on 16GB without degrading its quality. You’d have to drop to lower quantization. Is there some new magic?
35B A3B runs well on 8 and 16GB GPUs.
IQ3s perform very well these days.
Everything runs well if one is patient enough 😂
But yeah, with a smaller context and carefully offloading some of the layers to the CPU, it’ll run just not very fast. And definitely using Q4 quants.
Everything runs well if one is patient enough 😂
I mean, if you’re really patient you can even run with CPU only as long as you have enough RAM… My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K – my current default model). That pp tok/s is painfully slow… but it still works!
Same computer, same model but using a discrete GPU that can fit all the weights on the card – ~900 tok/s prompt processing, ~70 tok/s decode.
Yeah, down to 4bit, which works OK, but 35BA3B is much more performant. Sometimes you want the monolithic quality and damn the speed…
I have been running Qwen 3.6 for a while now on a Mac Mini and it’s been really nice. It is responding to changes requested in a code review right now and it’s doing a pretty good job of it.
qwen 3.6 27b runs on an AMD AI MAX 395+ fairly well.
I have one in a laptop with 64gb of ram that I bought before things went too crazy for $2200. Not sure how much one is now.
Anyway, you don’t need a datacenter and that’s why I personally think the current AI “boom” is a bubble around a nascent tech that’s not really ready for lots of the situations it’s been sold for. 🤷♂️
qwen 3.6 27b runs on an AMD AI MAX 395+ fairly well.
Try Qwen 3.6 35B-A3B on your hardware instead; it’s much faster.
I’ll check it out!
27b is much better though…
And it still gets 15-20 t/s with MTP which is plenty fast.
27b is much better though…
I haven’t really found that to be the case for my uses, but getting much faster decode is really useful for me… Have you tried running the MoE model at a higher quant? I usually use Q6_K.
I run both of them at Q8, unquantized KV cache and same settings.
35b does really well at most tasks but once the dependencies get complicated and the tool calls reach into the hundreds, 35b struggles where 27b still excels.
So generally I use 27b unless I need it fast, then I swap to 35b or 122b depending on what I need.
why I personally think the current AI “boom” is a bubble around a nascent tech that’s not really ready for lots of the situations it’s been sold for
Basically this. On top of that, we not far off from the point where you can run something like claude localy. Obviously we don’t have access to claude itself, but we do have access to local big models that come close to it in performance, like kimi-k3 ~500gb on average, deepseek ~1tb or glm5.2 (smallest quant starts from 228gb)
And it may sound like a lot, but it isn’t in reality — most amd cpus from am4 able to address 128gb of ram, while new am5 can do 256gb. Hell, my gaming PC have 128gb of DDR5. So we already technically can run some of those models on a regular office PC. Obviously, due to ram crisis, it will cost you a fortune, but it’s not like it impossible. Furthermore, demand for gpu with a lot of vram will sooner or later be satisfied, so it just matter of time when average consumer will be able to do it. And well, why would we need dedicated data centers then?
A 3090 works great. Any 24-32 GPU would work well.
No you don’t need a data center, this can run on any current gaming PC with a GPU that has 16GB+ VRAM.
Although to get good token generation speed and full context you need at least 24GB VRAM.
deleted by creator
