I'm curious how your project compares to plain mmap!
Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).
It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.
Catloafdev 15 minutes ago [-]
Ya I'd be interested to see a comparison of using llamacpp with ssd offloading to compare real speeds.
mmastrac 2 minutes ago [-]
I have a project that's almost ready to run DiffusionGemma as well. The two project might potentially work well together. I'm getting ~20tok/s on a 36GB M3 and there's strong possibility we might be able to crib faster kernels from each other.
Feel free to reach out.
addaon 55 minutes ago [-]
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
GeekyBear 39 minutes ago [-]
The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.
> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.
My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads
If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
afzalive 50 minutes ago [-]
The M5 MBP has 24GB of RAM, more context in RAM perhaps?
gitpusher42 24 minutes ago [-]
The process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood
gitpusher42 26 minutes ago [-]
[dead]
hsienchuc 29 minutes ago [-]
I've run local video generation models on an 8GB graphics card and know firsthand that nothing runs smoothly when memory is insufficient. So seeing 14GB of weights crammed into 2GB of RAM is impressive.
If running continuously for over an hour (like an overnight batch task), will a fanless MacBook Air overheat and throttle? Can the SSD handle the continuous weight reads and sustained output speeds?
Great work, congratulations on the release!
m00x 11 minutes ago [-]
This is where MoEs shine though. You don't need all experts in memory at once. Diffusion inference doesn't have sparse inference.
greggh 24 minutes ago [-]
It does exactly what it says it does. On my Mac mini M4 with 16GB of ram it is running at just over 5 tok/s. That jump from M4 to M5 is crazy.
gitpusher42 4 minutes ago [-]
What exact specs do you have? It might be because it's the 256 GB version. afaik, those versions have much slower memory bandwidth than the 512 GB models
My friend tried it on an M4 MacBook Pro and got 25–27 tok/s
mxmlnkn 52 minutes ago [-]
This sounds really cool. My intuition was that the selected experts might change heavily for each token, resulting in slow SSD loads for each token. This seems to be wrong. Did you create some statistics on how often the experts need to be changed? What is the longest token run without any expert change? What does such a token run look like? In which cases do experts change frequently?
gitpusher42 7 minutes ago [-]
The full route changes almost every token. The cache works through partial reuse, about 40% of experts repeat on the next token and 57% within two tokens, cutting I/O from 166 to 88 ms/token on M2 Mac.
The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly
yakupov_bulat 56 minutes ago [-]
Wow, amazing!
What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.
gitpusher42 45 minutes ago [-]
It depends on the use case.
I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.
The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.
If you need maximum speed and flexibility just use MLX
0gs 53 minutes ago [-]
you could use mine ... github.com/0gsd/enough (it has other stuff too)
WithinReason 48 minutes ago [-]
Nice job implementing expert caching!
gitpusher42 39 minutes ago [-]
Thank you!
Under good conditions it achieves approx a 67% cache hit rate with 16 expert slots
touwer 40 minutes ago [-]
Cool! Is there any info on this doing harm to the SSD? (Or other parts?)
wtallis 38 minutes ago [-]
Reads don't wear out flash memory to any meaningful extent.
gitpusher42 16 minutes ago [-]
AFAIK it should not because it is only reading
h2aichat 47 minutes ago [-]
Hope you can do it for Windows users also (and small graphics cards). Thanks
gitpusher42 36 minutes ago [-]
Uh, I’m afraid it is Apple only. It is written using Apple’s GPU language, Metal, and heavily relies on the Apples’s shared memory architecture
Windows PCs would require a completely different approach
znpy 51 minutes ago [-]
I wonder if i can run this on my MacBook Neo!
gitpusher42 42 minutes ago [-]
I haven't tried it but it should work! You can try it and share your results, it would be really appreciated
I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s
Also, it must be easy to adjust for iPhones and iPads in theory
sscarduzio 36 minutes ago [-]
How does this compare to DwarfStar4?
Rendered at 16:24:18 GMT+0000 (Coordinated Universal Time) with Vercel.
Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled).
It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.
Feel free to reach out.
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.
https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...
If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
If running continuously for over an hour (like an overnight batch task), will a fanless MacBook Air overheat and throttle? Can the SSD handle the continuous weight reads and sustained output speeds?
Great work, congratulations on the release!
My friend tried it on an M4 MacBook Pro and got 25–27 tok/s
The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly
What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.
I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.
The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.
If you need maximum speed and flexibility just use MLX
Windows PCs would require a completely different approach
I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s
Also, it must be easy to adjust for iPhones and iPads in theory