Discussion
I'm REALLY trying to move off of foundational LLM models and would like to run something locally. For the moment, that's just coding. I've heard that Qwen2.5-Coder or CodeGemma are SLMs that are targeted on just coding so are much smaller, and in theory anyone can run on a 16GB Mac.
Has anyone tried this? Are there better alternatives?
RE: https://toot.cafe/@peter/116170457727792391
@scottjenson I think an open model in the cloud might be a better approach? Local models demand so much hardware. With prices and things shifting constantly is it a wise investment?
@peter pointed out he runs his on a "green" service, which is a nice touch:
@scottjenson You can probably get a lot of automation done with local models but they're not very knowledgeable. It can to some extent be improved with tool calling. But that required longer context length which consumes a lot of memory. Then there is the security aspect of running them locally with full access to your computer. On Mac M-series for instance the gpu cant be virtualized for containers.
I recommend using llama.cpp over ollama. It has builtin webserver for interactive.
@scottjenson I have been testing some different models on my 128GB M1 Ultra and had some decent results. Slow for sure. Some results take 15 minutes, which would be 90 seconds on Claude Sonnet. Holding out hope that the Google+Apple models can be used for non-Xcode coding tools when they get released. Sounds like Apple is spending some effort on optimizing some for Macs specifically.
@trode That's my hope: EVENTUALLY this will be possible locally but I'm guessing it'll be a bit longer....
@scottjenson I've been reasonably happy with the 4 bits version of this one https://huggingface.co/unsloth/Qwen3.6-27B-GGUF slightly to large to fit in 16G, but the 2bits should work. I think it is around the level of a frontier model from a year ago for coding specifically, for more general stuff the big models have more breath of knowledge.
@scottjenson qwen3-coder with Q_4_KM should work locally
@scottjenson
I would go for Gemma 4 12B QAT or Qwen 3.5 9B fine tuned for coding.
For Mac you can even try Bonsai 27B (which is basically Qwen 3.6 27B quantized to Q2_0)
@nicolaottomano Helpful thank you! A previous reply said that whle they work, they are significantly slower. I can happily accept 2-5x slow down speed but if it's 20x slower, then it feels like I'd need to wait.
In my experience running models locally, the smaller models often run much faster than the bigger ones (as there is so much less swapping) I'm just hoping a small model gets "good enough"
Can you comment on speed? I'm running an M2 mac but I'm willing to get a new M5.
@scottjenson
I'm on Windows, so I have no first hand experience on Mac, but many praise Mac for the MLX speed and capabilities.
On Windows with a Nvidia RTX 4050 6 GB VRAM + 16 GB of system RAM the said models run quite fast with MLT (speculative decoding) enabled. It's around 20 tokens/s. Of course YMMV.
@scottjenson Yeah, it's VERY slow. Like literally unusable slow. If you're trying to get off the Claude/Codex train I'd look at GLM/Kimi but it's still subscription.
Really good coding agents aren't YET self hostable on "consumer" devices. And if somebody says it's not true they're either lying or have 196GB RAM Mac and are trying to say it's consumer electronic because you can buy it in the store.
@pawel Very helpful. thanks
@scottjenson You can always also take a look at https://ollama.com/ and get bigger model but in the cloud?
@pawel I actually have an Ollama account but I'm considering switching to llama.cpp but my understanding is that the two or roughly similar
Install bonfire.cafe
Get the full app experience
