Everyone was desperate to be first because capitalism.
But we are getting good models without the insane build out requirement. Which will be hilarious to leave the cunts holding the bag. Not that the planet is better for it in the end.
~1 token per second (storage bound gen4 nvme)… Some of us have places to be.
Don’t get me wrong. Its impressive that it can run at all, but honestly the usecase is exceedingly narrow. You’d have better results with a structured quantized gpu-only gemma or qwen workflow. Quality over quantity, rely on validation and a structured process: lots of cross-model review and iteration loops with spec and test driven dev. You could probably get a working alpha by the time colibri set up the environment.
Yeah I’m just beginning my local AI journey on a 5080, tried Qwen3.6 27b Q4 and was getting like 1tps because of the vram overflow. Ran it over night at it was still chewing on generating a prompt for a sub agent when I got up in the middle of the night until it simply ended in some kind of “fetch failure” lol. I think I gave it something too large to tackle, but either way 1tps is kinda garbage.
It has been some time since my initial comment so at the time I was mainly using LM studio. Qwen 3.6 a3b is the MOE and it does work well on my card, but the dense model that is more intelligent/capable is the Qwen 3.6 27b which doesn’t fit on the card and does get offloaded, but offloading cuts the speed down to like 1/tps.
I have since found a version of the 27b model that is “quantized,” for lack of a better term, differently and has to be run through TabbyAPI which gets back to 30ish tps. It can’t offload so it must fit fully on the card which keeps the speed high. Might be worth a look if you’re interested, the only downside is that with my 16gb card the context limit has to be kept pretty low ~40k if I remember correctly
But it’s so good at programming if you already know how to program! Surely that’s worth burning the planet and crashing the world economy??
Actually still no
https://github.com/JustVugg/colibri
Everyone was desperate to be first because capitalism. But we are getting good models without the insane build out requirement. Which will be hilarious to leave the cunts holding the bag. Not that the planet is better for it in the end.
~1 token per second (storage bound gen4 nvme)… Some of us have places to be.
Don’t get me wrong. Its impressive that it can run at all, but honestly the usecase is exceedingly narrow. You’d have better results with a structured quantized gpu-only gemma or qwen workflow. Quality over quantity, rely on validation and a structured process: lots of cross-model review and iteration loops with spec and test driven dev. You could probably get a working alpha by the time colibri set up the environment.
Yeah I’m just beginning my local AI journey on a 5080, tried Qwen3.6 27b Q4 and was getting like 1tps because of the vram overflow. Ran it over night at it was still chewing on generating a prompt for a sub agent when I got up in the middle of the night until it simply ended in some kind of “fetch failure” lol. I think I gave it something too large to tackle, but either way 1tps is kinda garbage.
Is that quantized? 4 bit Qwen 3.6 can get 22tps on a 1060.
It’s the q4 quantization, but it requires 20+GB vram and my 5080 only has 16
What are you using to run the model? Llama.cpp will automatically split the model between your system ram and graphics card’s vram.
Qwen 3.6 is a mixture of experts model with only 3B parameters active at a time. Even without quantization your card could easily run that.
It has been some time since my initial comment so at the time I was mainly using LM studio. Qwen 3.6 a3b is the MOE and it does work well on my card, but the dense model that is more intelligent/capable is the Qwen 3.6 27b which doesn’t fit on the card and does get offloaded, but offloading cuts the speed down to like 1/tps.
I have since found a version of the 27b model that is “quantized,” for lack of a better term, differently and has to be run through TabbyAPI which gets back to 30ish tps. It can’t offload so it must fit fully on the card which keeps the speed high. Might be worth a look if you’re interested, the only downside is that with my 16gb card the context limit has to be kept pretty low ~40k if I remember correctly