I mean, if you’re really patient you can even run with CPU only as long as you have enough RAM… My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K – my current default model). That pp tok/s is painfully slow… but it still works!
Same computer, same model but using a discrete GPU that can fit all the weights on the card – ~900 tok/s prompt processing, ~70 tok/s decode.
I mean, if you’re really patient you can even run with CPU only as long as you have enough RAM… My ~decade old computer with DDR3 RAM gets 14 tok/s prompt processing and ~5 tok/s decode on MoE models (e.g. a Qwen 3.6 35B-A3B variant at Q6_K – my current default model). That pp tok/s is painfully slow… but it still works!
Same computer, same model but using a discrete GPU that can fit all the weights on the card – ~900 tok/s prompt processing, ~70 tok/s decode.