While hosted mega models make headlines for doing cool stuff, the vast majority of applications for AI simply don't need all that power, and thus cost. That’s a big part of why businesses are screaming that there’s no ROI from AI.
Brining this tech down into small local models is likely where this all converges for the vast majority of use cases and what solves the present ROI crisis for LLM-based AI.
scotty79 5 hours ago [-]
If you are into small local models I highly recommend vibe thinker. It's a model trained specifically for reasoning. Basically a problem solver. When compared with other models, on math problems benchmarks, it's closer to models hundred times its size than ten times its size which it beats comfortably.
It supports long contexts on limited VRAM and is blazing fast.
Wat. Those are some crazy benchmark scores for a 3B model
MakazhanAlpamys 2 hours ago [-]
[dead]
user_7832 1 hours ago [-]
Tangential/meta: Holy shit, I've never seen a thread where almost half the comments are dead (and LLM written), especially for a post that's (currently) at 86 points and 20 comments (4x ratio is "pretty good quality" post signal generally for me).
MakazhanAlpamys 1 hours ago [-]
Those are mine. I used an LLM for my replies and that was a bad call, I said so further down.
Writing them myself now.
fintuner 5 hours ago [-]
I run a fine-tuned 4B for AML compliance at community banks — the ROI math is exactly this
wonger_ 2 hours ago [-]
Could you describe more? Your process, resources, use cases, user feedback
MakazhanAlpamys 2 hours ago [-]
[flagged]
victor106 4 hours ago [-]
How much data do you need to fine tune a model?
MakazhanAlpamys 49 minutes ago [-]
Depends what you change. Format or style, few hundred rows is often enough. A task
the model already half knows, few thousand.
New facts is where people waste a week. The model comes back wrong in a new way.
Use RAG for facts.
MakazhanAlpamys 2 hours ago [-]
[flagged]
fintuner 3 hours ago [-]
[flagged]
kamranjon 6 hours ago [-]
This seems really interesting - I was curious about this line from the website.
“The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.”
How does soup auto tune the hyper parameters and make some of these more complex training decisions?
Does this mean that this will not be free at some point?
The website is difficult to read (gray on black doesn't work well for me).
MakazhanAlpamys 2 hours ago [-]
[dead]
4 hours ago [-]
bookmon 2 hours ago [-]
This looks cool - what 4 GB GPU laptop do you recommend?
MakazhanAlpamys 40 minutes ago [-]
[dead]
MakazhanAlpamys 2 hours ago [-]
[flagged]
ranger_danger 5 hours ago [-]
Why is there still a hard VRAM requirement that's dependent on the model size? Isn't that exactly what this project is supposed to solve?
MakazhanAlpamys 44 minutes ago [-]
[dead]
MakazhanAlpamys 2 hours ago [-]
[flagged]
MakazhanAlpamys 7 hours ago [-]
Author here.
The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model.
Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division.
Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning.
Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux.
While hosted mega models make headlines for doing cool stuff, the vast majority of applications for AI simply don't need all that power, and thus cost. That’s a big part of why businesses are screaming that there’s no ROI from AI.
Brining this tech down into small local models is likely where this all converges for the vast majority of use cases and what solves the present ROI crisis for LLM-based AI.
It supports long contexts on limited VRAM and is blazing fast.
https://github.com/WeiboAI/VibeThinker
Writing them myself now.
New facts is where people waste a week. The model comes back wrong in a new way. Use RAG for facts.
“The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.”
How does soup auto tune the hyper parameters and make some of these more complex training decisions?
I have a couple of comments about https://trysoup.dev
> Get Started for Free
Does this mean that this will not be free at some point?
The website is difficult to read (gray on black doesn't work well for me).
The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model.
Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division.
Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning.
Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux.
Measurement records, including the ones I threw away: https://github.com/MakazhanAlpamys/Soup/tree/main/benchmarks
Write-up: https://doi.org/10.5281/zenodo.21771064
Happy to answer anything about the scheduler or the correctness protocol.