Nope. For 27B a 16GB video card (8 in a pinch) + 16 (32 better) GB RAM will work for reasonable context, if not screamingly fast. Any M series Macbook with 32 Gb+ (24 in a pinch). Hell, there are setups to run even larger models of a nvme drive if you have enough patience.
Smaller models like 8B are fine on a CPU, or even a phone.
I have my eyes on old 32gb amd instinct cards. One of those from ali will cost you ~450-500 euro, which is a lot for old unefficient card, but does provide amazing bandwidth and 32gb of vram. With just one you can run those models almost with full 256k context (or whatever the number there is). Just 4 of those and you can even run models that come close to things like claude and have 1m token context window
Sadly, I can’t justify it — my rig already have 7900xtx + 4060ti (so 32gb vram in total) and realistically I know that there no reason to buy it right now in memory crisis. In couple of years either Chinese manufacturers will catch up ram wise or someone will come up with dedicated device/accelerator for half the price. Or done other similar breakthrough will happen, maybe 4/8 channels of ram on regular desktop will become a thing, instead of being HEDT/server only
I mean, if you’re really patient you can even run with CPU only as long as you have enough RAM… My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K – my current default model). That pp tok/s is painfully slow… but it still works!
Same computer, same model but using a discrete GPU that can fit all the weights on the card – ~900 tok/s prompt processing, ~70 tok/s decode.
Nope. For 27B a 16GB video card (8 in a pinch) + 16 (32 better) GB RAM will work for reasonable context, if not screamingly fast. Any M series Macbook with 32 Gb+ (24 in a pinch). Hell, there are setups to run even larger models of a nvme drive if you have enough patience.
Smaller models like 8B are fine on a CPU, or even a phone.
I have my eyes on old 32gb amd instinct cards. One of those from ali will cost you ~450-500 euro, which is a lot for old unefficient card, but does provide amazing bandwidth and 32gb of vram. With just one you can run those models almost with full 256k context (or whatever the number there is). Just 4 of those and you can even run models that come close to things like claude and have 1m token context window
Sadly, I can’t justify it — my rig already have 7900xtx + 4060ti (so 32gb vram in total) and realistically I know that there no reason to buy it right now in memory crisis. In couple of years either Chinese manufacturers will catch up ram wise or someone will come up with dedicated device/accelerator for half the price. Or done other similar breakthrough will happen, maybe 4/8 channels of ram on regular desktop will become a thing, instead of being HEDT/server only
I don’t think you can run 27B on 16GB without degrading its quality. You’d have to drop to lower quantization. Is there some new magic?
35B A3B runs well on 8 and 16GB GPUs.
Everything runs well if one is patient enough 😂
But yeah, with a smaller context and carefully offloading some of the layers to the CPU, it’ll run just not very fast. And definitely using Q4 quants.
I mean, if you’re really patient you can even run with CPU only as long as you have enough RAM… My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K – my current default model). That pp tok/s is painfully slow… but it still works!
Same computer, same model but using a discrete GPU that can fit all the weights on the card – ~900 tok/s prompt processing, ~70 tok/s decode.
IQ3s perform very well these days.
Yeah, down to 4bit, which works OK, but 35BA3B is much more performant. Sometimes you want the monolithic quality and damn the speed…