What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?
trollbridge 2 minutes ago [-]
If you read to the bottom of the page, it says they're funded by a few people, and one of them is Samsung. I'm betting Samsung wants to be able to ship a capable AI system on a future model of their phone so they can compete with Apple.
I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
trollbridge 1 minutes ago [-]
Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.
The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows.
I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.
liuliu 1 hours ago [-]
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
liuliu 1 hours ago [-]
You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
syntaxing 52 minutes ago [-]
For those curious about their demo, I’m pretty sure it’s using Locally AI (iOS only) that lmstudio acquired/aquihired a couple months ago.
luckystarr 19 minutes ago [-]
Tried it on Android and got "!!!!!!!!!!!!!" for answers.
syntaxing 38 minutes ago [-]
I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.
pulse7 21 minutes ago [-]
Most probably not optimized yet for this model...
alvatech 2 hours ago [-]
TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
NitpickLawyer 1 hours ago [-]
There's two variants of this (or, as the joke goes, for very big values of bit):
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
PcChip 3 minutes ago [-]
this is a really dumb question, but how is -1 represented?
is it a float? if so, how many bits is the float?
I've never heard of a bit ever having more than two possible values
bensyverson 1 hours ago [-]
Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
thomasjb 33 minutes ago [-]
I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)
erelong 52 minutes ago [-]
I was trying Ornith 9B locally (it's up on Ollama) which claims:
> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.
Only tried it so much so far; it did a little better than Qwen 9B
liuliu 50 minutes ago [-]
Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).
janalsncm 37 minutes ago [-]
Is that a 1-bit LLM? I don’t understand the connection with this article.
erelong 8 minutes ago [-]
Oh, I don't actually know the difference if you want to explain it
The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?
xyzsparetimexyz 1 hours ago [-]
That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?
Catloafdev 27 minutes ago [-]
Doing some naive math, the F16 filesize is ~53.8gb, the 1-bit version is ~3.8gb, about 7% of the original size. The F16 size is roughly 2x param count, so that gives a rough ballpark of ~110B.
I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
is it a float? if so, how many bits is the float?
I've never heard of a bit ever having more than two possible values
> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.
https://deep-reinforce.com/ornith_1_0.html
Only tried it so much so far; it did a little better than Qwen 9B
The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?
I can just see their image tool on the app store
Available on HuggingFace: https://huggingface.co/collections/prism-ml/bonsai-27b