I've been using Cactus for a few months, great product!
Makes it really easy to plug and play different models on my phone.
If anybody is curious what a Pixel 9 Pro is capable of:
Tokens: 277- TTFT: 1609ms 9 tok/sec
qwen2.5 1.5b instruct q6_k
Sure, here's a simple implementation of the Bubble Sort algorithm in Python:
def bubble_sort(arr):
n = len(arr)
for i in range(n):
# Flag to detect any swap in current pass
swapped = False
for j in range(0, n-i-1):
# Swap if the element found is greater than the next element
if arr[j] > arr[j+1]:
arr[j], arr[j+1] = arr[j+1], arr[j]
swapped = True
# If no swap occurs in the inner loop, the array is already sorted
if not swapped:
break
This function sorts the array in ascending order using the Butbble Sort algorithm. The outer loop runs n times, where n is the length of the array. The inner loop runs through the array, comparing adjacent elements and swapping them if they are in the wrong order. The swapped flag is used to detect if any elements were swapped in the current pass, which would indicate that the array is already sorted and can be exited early.
HenryNdubuaku 142 days ago [-]
Thanks for the kind words, we’ve improved performance now actually, follow the instructions on the core repo.
Same model should run 3x faster on the same phone.
These improvements are still being pushed to the SDKs though.
cco 142 days ago [-]
Wow! 3x is huge.
I've had great experiences with gpt-oss20b on my laptop, a genuinely useful local model.
3x probably doesn't get my Pixel Pro 9 to being able to run 20b models, but its getting close!
HenryNdubuaku 141 days ago [-]
Although GPT OSS 20B has 1.7B activated parameters which will be fast, 20B weights is a lot for developers to bundle or consumers to download. That’s the actual problem.
MrDrMcCoy 141 days ago [-]
Question: can this utilize multiple forms of compute at once? Many phones have both GPUs that are capable of doing compute as well as NPUs, and that number will only increase. I'm sure it would be challenging, but that's a lot of performance to leave on the table if it can't do so already.
I am very curious what could be done with your impressive optimization on an rk3588, since it has pretty decent bits in all 3 categories, and am now seriously considering a Radxa Orion to play with this on :)
One more if you have a moment: will this be limited to text generation, or will it have audio and image capabilities as well? Would be neat to enable not only image generation, but also explore voice recognition, translation, computer vision, as well as image editing and enhancement features in mobile apps beyond what the big players daign to give us :)
rshemet 141 days ago [-]
Yes! Cactus is optimized for mobile CPU inference and we're finishing internal testing of hybrid kernels that use the NPU, as well other chips.
We don't advise using GPUs on smartphones, since they're very energy-inefficient. Mobile GPU inference is actually the main driver behind the stereotype that "mobile inference drains your battery and heats up your phone".
Wrt to your last question – the short answer is yes, we'll have multimodal support. We currently support voice transcription and image understanding. We'll be expanding these capabilities to add more models, voice synthesis, and much more.
MrDrMcCoy 141 days ago [-]
Very exciting, thanks!
ks2048 141 days ago [-]
This paragraph is a bit confusing:
> While Cactus can be used for all Apple devices including Macbooks due to their design, for computers/AMD/Intel/Nvidia generally, please use HuggingFace, Llama.cpp, Ollama, vLLM, MLX. They're built for those, support x86, and are all great!
It reads like you're saying for all Apple devices (which would include iOS), use these other things.(?) For iOS, are you trying to beat performance of other options? If so, it would be helpful to include comparison benchmarks.
HenryNdubuaku 141 days ago [-]
We are focused on phones and we did add some benchmarks and will add more. However, anyone can see performance for themselves with the repo directly.
pzo 142 days ago [-]
FWIW They change license 2 weeks ago from Apache 2.0 to non commercial. Understand they need to pay the bills but lost trust with such move. Will stick with react-native-ai [0] that is extension of vercel aisdk but with also local inference on edge devices
[0] react-native-ai.dev
observationist 142 days ago [-]
Open source for the PR, then switching to non-open licensing is a cowardly, bullshit move.
Use open source and stick with it, or don't touch it at all, and tell any VC shitheels saying otherwise to pound sand.
If your business is so fragile or unoriginal that it can't survive being open source, then it will fail anyway. If you make it open source, embrace the ethos and build community, then your product or service will be stronger for it. If the big players clone your work, you get instant underdog credibility and notoriety.
ls-a 141 days ago [-]
Chatgpt could spit me the same optimizations they're doing in a few minutes. They're very generic optimizations that anyone who wants to work on mobile should do. Looks like they're planning to troll the competition with lawsuits using this license.
HenryNdubuaku 142 days ago [-]
Thanks for sharing your thoughts. Honestly, I’d be annoyed too and it might sound like an excuse, but our circumstance was quite unique, it was a difficult decision at that time being an open-source contributor myself.
It’s still free for the community, just that corporations need a license. Should we make this clearer in the license?
typpilol 142 days ago [-]
Yes.
Just say that in the license.
HenryNdubuaku 142 days ago [-]
Done, thanks, let us know anything else.
typpilol 141 days ago [-]
Nice job on taking feedback.
pzo 141 days ago [-]
They updated but not to something they write here sugarcoating like they only try to limit corporations abuse. It’s not that paid license is for corporations only, it’s still non commercial for everyone including community.
HenryNdubuaku 142 days ago [-]
Understandable, though to explain, Cactus is still free for personal & small projects if you fall into that category. We’re early and would definitely consider your concerns on license in our next steps, thanks.
mdaniel 142 days ago [-]
For fear of having dang show up and scold me, I'll just add the factual statement that I will never ever believe any open source claim in any Launch HN ever. I can now save myself the trouble of checking, because I can be certain it's untrue
I already knew to avoid "please share your thoughts," although I guess I am kind of violating that one by even commenting
theturtletalks 142 days ago [-]
I agree, I've seen so many products start open source to gain traction, get into YC, and then either go closed source or change the license. That's a bait and switch and I appreciate the comment pointing it out.
I downloaded Cactus a couple months back because I saw a comment, but bait and switch like this makes we want to look for an actual open source solution.
HenryNdubuaku 142 days ago [-]
The license change doesn’t affect you based on your explanation actually, the licence has been updated with clearer words. We really appreciate you as a user, please share any more feedback you have, thanks.
theturtletalks 142 days ago [-]
I don’t appreciate you dismissing my claim. When I installed Cactus chat months ago, the company was claiming that Cactus chat would allow users to connect to other apps on their device and allow them to be controlled by AI.
Your license change goes against that. You say it’s free for personal use but how many times do people create something for personal use and monetize it later? What if I use Cactus chat to control a commercial app? Does that make Cactus chat use “commercial”?
HenryNdubuaku 142 days ago [-]
It’s absolutely fine to share your thoughts, that’s the point of this post, we want to understand where people’s heads are at, it’s what determines our next decisions. What do you really think? I’m genuinely asking so I don’t think mods will react.
trollbridge 141 days ago [-]
Here’s an example of what I want to do: ship our application entirely open source/free (AGPL3), but with options for interested parties who want to pay us for support/consulting to do so. Likewise, we want interested parties who want to build their own proprietary app on top of our stack to be able to do so.
Mixing in a “you have to pay if you’re a corporation” licence makes this difficult if not impossible, particularly if we wanted deep integration with eg Cactus. We don’t want to police a “corporation” who wants to use our open source software.
HenryNdubuaku 141 days ago [-]
Thanks for pointing this out, another factor for us to figure out. We waive the license for such cases, wanna get in touch? I don’t think your consumers have to worry about the license.
mritchie712 142 days ago [-]
how many GB does an app packaged with Qwen3 600m + Cactus take up?
e.g. if I built a basic LLM chat app with Qwen3 600m + Cactus, whats the total app size?
HenryNdubuaku 142 days ago [-]
400mb if you ship the model as an asset. However, you can also build the app to download the model post-install, Cactus SDKs support this, as well as agentic workflows you’d need.
asdfrgtfhgnjn 141 days ago [-]
Wow made an account just for this! I was using cactus for a paid app i soft launched recently. Does that mean that now i cannot update this dependency? What is your pricing? I do not see that anywhere. If i did not notice this and pulled the updated version, would i be liable to be sued? Also, I implemented cactus on the good faith assumption that i was going to receive the updates in your roadmap, as a proper apache project. I must admit this is quite the move guys
HenryNdubuaku 141 days ago [-]
Apologies for this, but you have nothing to worry about, no one is suing you. We are experimenting with the license and monetisation for corporations not indie developers. Please keep using Cactus the way you want, take this response as explicit permission while we go away and chew on your feedback.
cientifico 141 days ago [-]
Can you clarify the following sentence:
> We are open-source (https://github.com/cactus-compute/cactus). Cactus is free for hobbyists and personal projects, with a paid license required for commercial use.
If it is open-source, one is free to distribute even for commercial use by definition. Which one is correct and what's your business model?
kvakkefly 141 days ago [-]
Why do you believe open source means free to use and distribute commercially?
Cheer2171 141 days ago [-]
Are you joking or just new? This is a foundational, bedrock principal of open source.
Because that’s literally the definition of open source:
> Open-source software is software released under a license where the copyright holder grants users the rights to use, study, change, and distribute the software and its source code, for any purpose.
That’s the first result you get on Google—and it’s exactly why so many companies relicensed their projects (Redis, HashiCorp, Elasticsearch, MongoDB…).
If it’s open source, you can sell it, host it, or give it away for free. The only difference is which obligations the license attaches:
GPL → you must keep the license.
AGPL → you must keep it and extend it to hosted services.
BSD/MIT → do almost whatever you want.
But the core right is always the same: distribute, host, and sell. Courts have even confirmed this is the accepted definition of “open source.”
VladVladikoff 142 days ago [-]
How does this startup plan to make money?
HenryNdubuaku 142 days ago [-]
Cactus is free for hobbyists and personal projects, but we charge a tiny fee for commercial use which comes with more features that are relevant for enterprises.
binary132 141 days ago [-]
I couldn’t find a pricing page on your site. How tiny is tiny?
HenryNdubuaku 141 days ago [-]
It’s custom for now as we are calibrating to see what works for everyone, wanna get in touch?
binary132 141 days ago [-]
I’m not in the market, just curious.
ivape 141 days ago [-]
Are you saying this would be more performant than Apple’s on device LLM/inferencing?
HenryNdubuaku 140 days ago [-]
Valid question. Our perspective is that there can be multiple players, there are 7B devices to power, everyone will get a slice.
elpakal 141 days ago [-]
Came here to ask about how they view Apple Foundation Models as a threat.
> guarantees privacy by default, works offline, and doesn't rack up a massive API bill at scale.
I’ve been really interested in on-device ML for most of my career, and now I wonder how valuable these benefits really are. LLM vendor APIs are pretty performant these days, security is security, and with an on-device model you have to provide updates every time a new model comes out.
HenryNdubuaku 140 days ago [-]
You don’t have to bundle the weights as an asset, you can do over-the-air updates, new weights are simply downloaded.
elpakal 139 days ago [-]
Neat, but not really addressing my point. My point is that you still need to roll out changes and LLM ApIs just work.
nicktay 141 days ago [-]
I built apps using Flutter and this project seems to make it possible to use models directly in app instead of cloud APIs. Curious about the commercial license here. How is the trade off between pricing and performance?
rshemet 141 days ago [-]
indeed, this is exactly the goal! The license grants rights to commercial use, unlocks additional hardware acceleration, includes cloud telemetry, and offers significant savings over using cloud APIs.
In our deployments, we've seen open source models rival and even outperform lower-tier cloud counterparts. Happy to share some benchmarks if you like.
Our pricing is on a per-monthly-active-device basis, regardless of utilization. For voice-agent workflows, you typically hit savings as soon as you process over ≈2min of daily inference.
nextworddev 142 days ago [-]
Will this drain my battery
HenryNdubuaku 142 days ago [-]
This was one of the issues we set out to solve, so not as much as you’d expect.
giveita 141 days ago [-]
Tried the android app but model download froze. Are you using the same docker-style repositories as Ollama. Because they suck. If you do I suggest use your own s3 instead.
HenryNdubuaku 141 days ago [-]
We host on HuggingFace, were you able to get it to work eventually?
dcreater 142 days ago [-]
Does it incorporate web search tool?
HenryNdubuaku 142 days ago [-]
It can incorporate any tool you want at all. This company’s app use exactly that feature, you can download and get a sense of it before digging in. https://anythingllm.com/mobile
ApolloRising 141 days ago [-]
Would you consider adding a mode where it could go online if the user instructed it to?
The first picture on the android app store page shows Claude Haiku as the model
HenryNdubuaku 142 days ago [-]
Thanks for noticing! The app is just a demo for the framework, so devs can compare the open-source models against frontier Cloud models and make a decision. We removed the comparison now so those screenshots indeed has to be updated.
joseph4521 141 days ago [-]
AI Dungeon should contact you to make an offline mode again.
HenryNdubuaku 141 days ago [-]
Ok, looking forward to it!
apwell23 141 days ago [-]
curious. what are the usecases for <100ms latency ?
HenryNdubuaku 141 days ago [-]
Real-time video and audio inference.
potato-peeler 141 days ago [-]
Can it be fine tuned for a specific task?
HenryNdubuaku 141 days ago [-]
Yes, you can fine-tune a model for any task, what do you have in mind?
hzbxxjfn 136 days ago [-]
[dead]
negro_jj 134 days ago [-]
Up date
Rendered at 07:22:43 GMT+0000 (Coordinated Universal Time) with Vercel.
Makes it really easy to plug and play different models on my phone.
If anybody is curious what a Pixel 9 Pro is capable of:
Tokens: 277- TTFT: 1609ms 9 tok/sec
qwen2.5 1.5b instruct q6_k
Sure, here's a simple implementation of the Bubble Sort algorithm in Python:
def bubble_sort(arr): n = len(arr) for i in range(n): # Flag to detect any swap in current pass swapped = False for j in range(0, n-i-1): # Swap if the element found is greater than the next element if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[j] swapped = True # If no swap occurs in the inner loop, the array is already sorted if not swapped: break
# Example usage: arr = [64, 34, 25, 12, 22, 11, 90] bubble_sort(arr) print("Sorted array is:", arr)
This function sorts the array in ascending order using the Butbble Sort algorithm. The outer loop runs n times, where n is the length of the array. The inner loop runs through the array, comparing adjacent elements and swapping them if they are in the wrong order. The swapped flag is used to detect if any elements were swapped in the current pass, which would indicate that the array is already sorted and can be exited early.
Same model should run 3x faster on the same phone.
These improvements are still being pushed to the SDKs though.
I've had great experiences with gpt-oss20b on my laptop, a genuinely useful local model.
3x probably doesn't get my Pixel Pro 9 to being able to run 20b models, but its getting close!
I am very curious what could be done with your impressive optimization on an rk3588, since it has pretty decent bits in all 3 categories, and am now seriously considering a Radxa Orion to play with this on :)
One more if you have a moment: will this be limited to text generation, or will it have audio and image capabilities as well? Would be neat to enable not only image generation, but also explore voice recognition, translation, computer vision, as well as image editing and enhancement features in mobile apps beyond what the big players daign to give us :)
We don't advise using GPUs on smartphones, since they're very energy-inefficient. Mobile GPU inference is actually the main driver behind the stereotype that "mobile inference drains your battery and heats up your phone".
Wrt to your last question – the short answer is yes, we'll have multimodal support. We currently support voice transcription and image understanding. We'll be expanding these capabilities to add more models, voice synthesis, and much more.
> While Cactus can be used for all Apple devices including Macbooks due to their design, for computers/AMD/Intel/Nvidia generally, please use HuggingFace, Llama.cpp, Ollama, vLLM, MLX. They're built for those, support x86, and are all great!
It reads like you're saying for all Apple devices (which would include iOS), use these other things.(?) For iOS, are you trying to beat performance of other options? If so, it would be helpful to include comparison benchmarks.
[0] react-native-ai.dev
https://github.com/cactus-compute/cactus/commit/b1b5650d1132...
Use open source and stick with it, or don't touch it at all, and tell any VC shitheels saying otherwise to pound sand.
If your business is so fragile or unoriginal that it can't survive being open source, then it will fail anyway. If you make it open source, embrace the ethos and build community, then your product or service will be stronger for it. If the big players clone your work, you get instant underdog credibility and notoriety.
It’s still free for the community, just that corporations need a license. Should we make this clearer in the license?
Just say that in the license.
I already knew to avoid "please share your thoughts," although I guess I am kind of violating that one by even commenting
I downloaded Cactus a couple months back because I saw a comment, but bait and switch like this makes we want to look for an actual open source solution.
Your license change goes against that. You say it’s free for personal use but how many times do people create something for personal use and monetize it later? What if I use Cactus chat to control a commercial app? Does that make Cactus chat use “commercial”?
Mixing in a “you have to pay if you’re a corporation” licence makes this difficult if not impossible, particularly if we wanted deep integration with eg Cactus. We don’t want to police a “corporation” who wants to use our open source software.
e.g. if I built a basic LLM chat app with Qwen3 600m + Cactus, whats the total app size?
> We are open-source (https://github.com/cactus-compute/cactus). Cactus is free for hobbyists and personal projects, with a paid license required for commercial use.
If it is open-source, one is free to distribute even for commercial use by definition. Which one is correct and what's your business model?
https://opensource.org/faq#commercial
> Open-source software is software released under a license where the copyright holder grants users the rights to use, study, change, and distribute the software and its source code, for any purpose.
That’s the first result you get on Google—and it’s exactly why so many companies relicensed their projects (Redis, HashiCorp, Elasticsearch, MongoDB…).
If it’s open source, you can sell it, host it, or give it away for free. The only difference is which obligations the license attaches:
GPL → you must keep the license.
AGPL → you must keep it and extend it to hosted services.
BSD/MIT → do almost whatever you want.
But the core right is always the same: distribute, host, and sell. Courts have even confirmed this is the accepted definition of “open source.”
> guarantees privacy by default, works offline, and doesn't rack up a massive API bill at scale.
I’ve been really interested in on-device ML for most of my career, and now I wonder how valuable these benefits really are. LLM vendor APIs are pretty performant these days, security is security, and with an on-device model you have to provide updates every time a new model comes out.
In our deployments, we've seen open source models rival and even outperform lower-tier cloud counterparts. Happy to share some benchmarks if you like.
Our pricing is on a per-monthly-active-device basis, regardless of utilization. For voice-agent workflows, you typically hit savings as soon as you process over ≈2min of daily inference.