Hey guys,
What’s currently the best LLM for low-VRAM machines with only 6 GB VRAM? I’ve got 32GB RAM as well.
I’m experimenting a little with SillyTavern and I’m curious which model gets the most out of my setup. Should be multilingual and suitable for “casual chatting”.
I know I will probably not get very far with this, but I’m still interested in how far we’ve already come.
(Using KoboldCPP if that matters).
~sp3ctre
You must log in or register to comment.
On my MacBook Air m2, I’m currently using Qwen 3.5 4b with 8 bit quantisation, and even at its maximum context length, multiple web search RAGs, and the model being built for vision and reasoning, it only ever hits 4.3gb of memory tops.
I run it though LM Studio, so paired with the fact it’s a Mac, your mileage may vary in terms of how much memory it uses, but it does have [from my experience] an output quality a bit over ChatGPT 4o, and is actually really solid for research purposes if that’s what you’re looking for.
I mean, do you need it to be fast? You could probably run a pretty decent 20b model if you are okay with the speed of offloading.
Doesn’t necessarily need to be very fast, but I don’t plan to wait a minute for one simple sentence as well :)
Is that possible without tinkering too much?
I have a Qwen3.6-35b-a3b model running on a dated desktop machine with 4GB VRAM.
I use 8-bit-quant, but also have 48GB normal RAM.
Delivers ~7tk/s, which is already totally usable for most things.
Tried it on my recent Core-i7 company laptop with 8GB VRAM and got 20tk/s.
Oh, and I am also using KoboldCPP (on a Linux foundation).
I have been using qwen-3.5-9b as a general purpose LLM that I can still load up while gaming (on a 16GB card). I never have issues if I’m under 10ish gigs of VRAM usage for the game, so I imagine it should work for your use case.
I’ve been generally happy with the results on everyday reasoning tasks and programming questions.
Late to the party, but this was just released: LiquidAI/LFM2.5-8B-A1B-GGUF
I guess you could fully fit it at Q4 with a little context if you need all the speed you can get, or offload the experts to RAM if you prefer higher quality and/or more context.
I’m running Qwen 3.5 4B Q4 on 16GB RAM. Yup, no VRAM haha. 5-6tk/s. Llama.cpp
I’m running gemma-4-e4b on my 8GB machine. I’ll drop down to e2b on CPU. It’s probably the best you’ll get. 140 languages, vision, decent at agentic work. Not great at code.
