Once you hit the memory you need, generation speed is mainly set by bandwidth, and every Ultra from M1 thru M3 has ~800 GB/s. IMO best ROI for most people is 'cheapest used Ultra with enough RAM'
I setup an eBay alert and picked up a used M2 Ultra that has delivered good ROI (at least, far better than 15k for comparable-for-my-use-case performance)
simonw 3 hours ago [-]
The numbers I was most interested in are tucked away in a chart towards the bottom - the speed comparison of the Mac Studios v.s. a RTX 5090:
Those RTX 5090 numbers are bad. You can get over 200 tps with ninfer using NVFP4 and MTP.
beastman82 2 hours ago [-]
can confirm.
I dont' know why people spend huge money on these and Spark. The 5090 is running qwen 3.8 at 200+ tps!! That's 1-2 orders of magnitude faster.
tomega2134 6 minutes ago [-]
Is a 5090 still cost efficent when it is (currently) unobtainable? Or when obtainable only at current prices (min. $6500 USD)?
_hugerobots_ 1 hours ago [-]
Have a 5090, and yes it's very fast. But it's like the worst ADHD team member and requires constant supervision and review from larger models. It's context size on-card is good for super, suuuuuper shallow precision work.
The gb10/spark on top of it, that thing can refactor enormous monorepo architecture. The time it takes the 5090 to compact, reiterate and execute a plan is often the same time as the gb10.
nacs 2 hours ago [-]
People don't buy Sparks and M5 Ultras to run a 27B model - you buy it to run an MoE model like Qwen Next which this M5 excelled at.
ProllyInfamous 3 minutes ago [-]
Exactly; when I first got my RTX 5070 Ti (to game with!!!), I loaded then-latest Qwen3.6 (~30B, cannot remember exactly).
My technical-expert twin played around with that LLM for about an hour, and then correctly reasoned "it's able to be WRONG, faster."
This seems apt.
throwaway27448 1 hours ago [-]
A) the macos value add is enormous if you have any investment in the ecosystem, B) for me at least a GPU is completely useless for anything but being a token generator.
bigyabai 1 hours ago [-]
> for me at least a GPU is completely useless for anything but being a token generator.
No thanks to the "macos value add" that forces you to use Metal while Valve customers frolick in Protonland.
throwaway27448 15 minutes ago [-]
> No thanks to the "macos value add" that forces you to use Metal while Valve customers frolick in Protonland.
Crossover works on macos, too. So does moltenvk, so does vanilla wine, etc etc. You can run most games without a hitch these days (allegedly, according to /r/macgaming). But I don't play video games so a GPU would probably be better off in some kid's computer.
Eisenstein 1 hours ago [-]
A 5090 has a 1.79TB/s memory bandwidth. Qwen 3.8 27B NVFP4 is 22GB. You cannot generate tokens faster than the weights can traverse the GPU memory, so that makes max generation speed without MTP to be 81T/s. Say MTP is giving you 0.5 acceptance rate (very good), that is 1.5 * 81 is 121T/s. Even with a perfect acceptance rate you would only get 162T/s.
beastman82 1 hours ago [-]
Off the top of my head, I'm guessing we're missing sparse attention. But I'll run your challenge through and see where the gaps are. I promise I'm telling the truth :)
1 hours ago [-]
mathisfun123 2 hours ago [-]
same reason they spend huge amounts of money on rolexes when seikos work better (the tech crowd isn't immune from vanity).
throwaway27448 1 hours ago [-]
If you seriously think apple products are nothing but a status item, you're deluding yourself and probably have been for decades.
_hugerobots_ 1 hours ago [-]
This 1000%. Data centres don't equate to medium sized labs and businesses. A stack of Macs is up and running without digging trenches, an electrician on staff and a department of PhDs to justify the spend.
bigyabai 1 hours ago [-]
It's likely that a stack of Macs will draw more power for slower prefill/decode than equivalently priced Nvidia GPUs. If power efficient inference is the goal, Macs are a non-starter.
_hugerobots_ 33 minutes ago [-]
So if it isn't a comparative ability, now it's a power cost issue? This reads like goal post moving.
1 hours ago [-]
mathisfun123 51 minutes ago [-]
If you seriously think apple cares about anything other than cell phones, you're deluding yourself and probably have been for decades.
8 minutes ago [-]
throwaway27448 18 minutes ago [-]
...did you mean profit? I don't think they're manufacturing iphones just on the hope they delight you. This is also true of Google et al.
I don't get these weird parasocial emotional attachments/beefs people have with brands. Talk to a therapist.
tom_ 24 minutes ago [-]
They've been selling phones for less than 20 years at this point? Though I suppose 1.9 is not equal to 1, so it gets the plural.
liuliu 2 hours ago [-]
Both are probably single-token decode performance, which is reasonable to show. Otherwise agree RTX 5090 should shinebetter with NVFP4.
ActorNightly 24 minutes ago [-]
When I say that Apple is astroturfing HN, this is what I mean. Anyone can easily verify that even a 3090 will blow any Mac out of the water in terms of Tok/sec. Somehow its ok to just post outright lies when it comes to Apple product comparison.
bee_rider 8 minutes ago [-]
I guess it is possible, but Apple has had very vocal fans for decades. I suspect, rather than astroturfing, it is just people who are in their ecosystem.
nacs 2 hours ago [-]
That's a dense model. Of course it will do worse.
Now try running that Qwen 3.8 Next model on the 5090 and tell me what TPS you get (hint: it's near 0 since it doesnt fit the 32GB VRAM on 5090 vs the 256 in OPs M5).
peri-cl 2 hours ago [-]
Surprisingly, the Reddit crowd are reporting 50–60 tokens/s (for the 32 GiB 5090 + 128 GiB RAM)—on par with the M5 Ultra benchmarks, despite both the PCIe bottleneck and much smaller DDR5 bandwidth,
That's with CPU offload to a DDR5 6000 RAM though which is around $3-4k at least.
redox99 3 hours ago [-]
A dense 27B doesn't really make sense for the Mac. A MoE makes way more sense when you have modest bandwidth but lots of memory.
tcdent 49 minutes ago [-]
A dense model (up to the amount of memory available) actually does make the most sense on unified memory architectures. But when you hit the limit of what you can hold in memory, you reach the limitation of the platform.
Whereas a hybrid architecture with distinct DRAM and VRAM with sparse MoE, you can leverage two different bit rates depending on the actual need for constant access to common layers versus sparse access to infrequent layers and arbitrage the difference in cost for each of those in distinct classes of hardware.
peri-cl 3 hours ago [-]
They do MoE. They benchmarked GLM 5.3-flash (320B / 18B), and Qwen 3.8-flash-next (125B / 6B). The dense Qwen is only focused (I assume) because it's about the only thing that fits on a 5090, that they can compare the two heads on.
alex7o 34 minutes ago [-]
On my m5 max 27b model does 75tps on 256k ctx and starts at 80 on the 8k ctx when you add https://huggingface.co/collections/z-lab/dflash-2 to it. So yeah base might be 30tps (I used iq4) but mtp or dflash help a lot and should be used when checking what is useful and what is not for running models as it is not fare to judge without them.
peri-cl 3 hours ago [-]
Those are some incredible graphs, that leap in prompt processing going from M3 to M5.
Also: ~30 token/s on GLM 5.3-flash, locally. (That's roughly Opus 4.8-tier. I think).
/meta Here's a CSS filter that stops those nuisance chart animations,
Thank you for this. I wish Apple focused their silicon design on improving the TTFT metrics but coming from an M3 Pro, it still looks laggard compared to Nvidia's TensorCores in the 5090.
Maybe Apple is an acquisition away from changing that balance.
wlesieutre 3 hours ago [-]
The rumor on Apple's processor roadmap is that they're skipping other M6 variations (all previous generations had Pro and Max, a few had Ultra) in order to focus on the M7 generation for AI reasons. What exactly the M7 improvements are who knows.
I think that comes down to TSMC. Nvidia apparently booked out the whole A18 or 16 node. Apple is on 2nm right now and M7 will jump right to A14. According to my quick AI research anyway.
dagmx 2 hours ago [-]
That sounds a lot like AI fantasy slop.
Apple just shifted to N2. They’re not going to be doing another major shift right away.
I assume those are non-batched. I think the M series GPU can do 4X to 8X depending on model quant, which means if you can batch queries you'll get almost 4X to 8X performance.
jmyeet 3 hours ago [-]
The selling point of the M5 Ultra Mac Studio is that you can run much larger models that the 5090 can't without swapping. NVidia aggressively segments the market on VRAM for this reason. That's why a 5090 has an MSRP of ~$2k (but good luck getting one for less than $4k) while a 6000 Pro, which is basically a 5090 with 96GB of RAM has now soared beyond $15k where 3-6 months ago it was more like $10-11k. A 6000 Pro has the same memory bandwidth but slightly more CUDA units (IIRC ~24k vs ~21k).
This advantage won't be apparent with a 27B model. The 256GB MS can probably run the newer Flash models locally, something you can't do on a 5090.
I don't think we'll get a successor to the 5090 until late 2028, maybe even 2029. I'm basing this on the launch date of the 5000 series and that we haven't got a midcycle refresh yet. Rumor has it the chips are ready but the 3GB RAM modules are 3-4x the price of the 2GB modules used on the current cards.
Apple should see a Mac Studio major update in 2028. That might even force NVidia's hand. But it's really impossible to say what the state of the market will be 2-3 years from now. It may have completely crashed. I suspect not however.
The interesting thing will be when the bandwidth demands start forcing HBM memory onto these home/enthusiast solutions.
pama 2 hours ago [-]
But what about builds that combine 8 of the 5090 with infiniband between boxes? Wouldn't that be comparable to the mac in terms of price and potentially beat it by a lot in terms of performance for the large MoE? I understand the space/heat/noise considerations, but price wise it may still not make as much sense as people think. (Agreed that it is hard to get the NVIDIA hardware and the 6000 pro are priced less competitively).
wmf 2 hours ago [-]
No, $40K is not comparable to $10K.
kridsdale1 2 hours ago [-]
While that sounds super awesome,
How many people are actually going to build and maintain that vs a box you can grab at the mall that fits in a lunchbox?
prmoustache 2 hours ago [-]
Sounds like nice utility bill in the making.
jmyeet 1 hours ago [-]
I can't speak to Infiniband pricing for something like that. It seems like the cheap option is 56/100Gbps with used Enterprise equipment. You'd need 8 HCAs, DAC cabling and a switch but even then you're into thousands of dollars. If you want 200Gbps+ it gets into the tens of thousands (AFAICT).
Each PC is probably going to cost ~$6k and you're talking about 8000W of electricity draw. That's going to consume multiple 20A circuits even at 240V. And the electricity ain't free either. A Mac Studio seems to draw ~500W max.
Oh and the Mac Studio has an upgrade route to run 1T+ models too by chaining them together with TB5 chaining. OSX supports RDMA this way. That's comparable bandwidth to the 100Gbps Infiniband option.
So you're talking about $50-60k of hardware and more power draw and more heat for something that will I'm sure beat the MS M5U option but at huge cost. Also, at that kind of price point, I'm likely to get a workstation PC and put 2 (or possibly 3) 6000 Pros in it.
3 hours ago [-]
traceroute66 2 hours ago [-]
Not forgetting of course that an RTX5090 is what 600W+ ? And the Mac is probably half that at most ?
washadjeffmad 42 minutes ago [-]
Certainly not forgetting wattage. A 5090 is 575W. The M5 Ultra Studio is 480W.
nvidia-smi -pl 450 for like a 4% reduction in throughput. I tend to set it around 350W because it's a comfortable temperature blowing on my legs under the desk without warming my office in the summer.
I put together this system two years ago, so it's a little out of date, but it only cost $3000 for the same performance and capability as an Ultra. I don't think I would spend $7000 to save 100W, though.
beastman82 2 hours ago [-]
sure. so is 2x power worth 10x perf? I think it is in most cases.
ActorNightly 21 minutes ago [-]
When you are doing matrix math, compute is compute. Apple cant be more efficient due to physics. The only reason Macs are more efficient in general is that they have tightly bundled hw and sw for specific tasks.
GeekyBear 17 minutes ago [-]
The next Ultra, supposedly on deck in 2028:
> Apple's planned M7 Ultra chip is being designed to support up to 1.5 TB of unified memory and to push AI performance toward the class of Nvidia's Blackwell accelerators, according to a new Bloomberg report published by Mark Gurman...
Apple plans to release a base M6 chip this fall for entry-level Macs... a base M7 in the first half of 2027, M7 Pro and M7 Max at the end of 2027, and the M7 Ultra in 2028.
This is great as a first look, but the author is not a developer, so we don't yet know whether a dev can be as productive with local models on M5 Mac Studio compared to a 20x subscription plan.
I'm also curious about any new low hanging optimization opportunities in the kernels for this new hardware.
It's already clear to me that M5 Mac Studio is more cost-effective than anything you can run on open router, assuming decent utilization.
The M5 Mac Studio will be the most cost effective way to run uncensored cyber capable open agents.
An exciting tipping point will be if programmers can get an Astra-Ultra like experience all week with this hardware. That would be a real sense where this hardware exceeds the value of even 20x cloud subscriptions.
zozbot234 2 hours ago [-]
Astra-Ultra? Even the largest open model to date (Kimi K3) is nowhere close to Astra level, and it will be quite slow even on the highest-spec M5 Ultra, with achievable speeds of about 0.5 tok/s at most due to having to stream weights from SSD (~13 GB/s on the highest storage capacity M5 Max machines so far). This is OK for doing simple Q&A in the background but it's far from a genuine coding experience. You'd have to test batching of multiple thinking streams in order to try and raise overall tok/s via layer-wise reuse of the streamed weights (and this is where the "Ultra" part sort of becomes relevant; Kimi series models have good support for agent swarms) but this would decrease single-session performance even further. It would only be usable for background jobs, though the hardware would then have a chance of paying for itself if it was fully used on a 24/7 basis.
slowin 2 hours ago [-]
> This is great as a first look, but the author is not a developer, so we don't yet know whether a dev can be as productive with local models on M5 Mac Studio compared to a 20x subscription plan.
Local models are definitely not as productive as SOTA, sadly it's not close yet. I do think someday they will be "good enough" to use, but they aren't today. Even the SOTA models barely code well, with Opus 4.5 being the first, good coding model.
That being said, I think it's absolutely imperative that we keep pushing local model performance. We need to continue to advance technology there and ensure that the model labs don't do regulatory capture in the name of "safety" (or anything else).
nowittyusername 1 hours ago [-]
With the latest codex (weekly quota burn) fiasco I tried open weight alternatives for the first time. And tyeah... open weight models cant compete with likes of astra yet. But, my hope is that by the time I get my Mac studio at end of november an open weight models would have closed the gap (which i think is realistic at the speed of progress). Now its true a better gpt version will also be available then but it also seems the gap is shrinking with time so theres that.
_hugerobots_ 1 hours ago [-]
Local models can be widely used as productive assets. Yes the infrastructure of SOTA API models is engineered specifically for you to be that utility, but the blanket statement that local isn't up to par is intensely short sighted. Billions of tokens per month on local pays for the hardware when compared to sota costs per month.
slowin 52 minutes ago [-]
I believe they can currently be used productively for non-coding tasks (classification, light summary)... but they definitely are not even close to SOTA when it comes to software development.
_hugerobots_ 38 minutes ago [-]
Defining productivity is a use-case scenario, and a wildly generalized assumption for most people in this argument. Local infrastructure doesn't need to be sota for absolutely every single need for a dev lab, but it absolutely can be delivered with non-api frontier class models.
slowin 21 minutes ago [-]
Just to be clear, I'm specifically talking about coding. I think local models can help with productivity today, just not coding.
I'm also a huge fan of local models and think it's absolutely imperative that they continue to advance so we can move off of the Anthropic/OpenAI hosted models. It's important to accurately asses where we are in that journey though.
srcreigh 18 minutes ago [-]
I think the issue is generalization, if you were more specific about which local models aren’t good enough for which tasks compared to which frontier models in your experience, it’d be a lot more informative
sajithdilshan 3 hours ago [-]
On Apple website it says 512GB memory option is available in October. I guess bumping to that one would cost additional 4-6k US$. So an Ultra with 2TB storage would be north of 15k US$.
That’s like 12 years worth of OpenAI Pro subscriptions
112233 3 hours ago [-]
Hard to guess, it can go either way. If you will need to be in a syndicate to use non-sterilized models, that mac makes sense. But if there is mandatory registration of personal cyberarms, you risk going to mines once they check you purchases. You could try to play normie and pretend you simply wanted to show off, by keeping your actual work on external disk, but that leaves traces on system. Counting on someone in the Gap renting you gray iron works as long as you can swap credits. Still, this gear is tiny. Put it in your e-car, with uplink, and leave it at uncle's farm. Discreet.
Razengan 2 hours ago [-]
I gotta have some of what you had :)
kridsdale1 2 hours ago [-]
I thought it was a fun bit of cyberpunk fiction. Those who downvoted him seem to have taken it at face value?
I appreciate the reference to RUSH: Red Barchetta in the final line.
nowittyusername 60 minutes ago [-]
512 option isnt worth it imo, you get severe slowdowns when weights are that large. 256 is the sweet spot, you can run large open weight models at decent speeds for full private inference.
geodel 3 hours ago [-]
Agreed.
Specially since one can pay half right now to OpenAI and sign a 12 year iron clad contract for uninterrupted service delivery of OpenAI Pro.
Kurtz79 3 hours ago [-]
I think we all expect the heavy subsidized subscriptions to end or significantly increase in price at some point, but it could be years from now and I'd rather spend a similar figure on an hypotetical Mac Studio M8 Ultra, or whatever more advanced competitor that will have likley appeared by that time.
A more apples-to-apples comparison would be with API cost in OpenRouter at the same tok/s rate for the same models that you can run locally, maybe.
BatFastard 56 minutes ago [-]
>A more apples-to-apples comparison
Don't you mean an Apple to NVidea comparison?
vardump 3 hours ago [-]
I hope that was sarcasm.
cyclopeanutopia 3 hours ago [-]
"iron clad" :)
prmoustache 1 hours ago [-]
Censorship included.
patrickmcnamara 2 hours ago [-]
HN always has these completely contrived counterarguments. What is actually going to realistically happen that will prevent use of an LLM provider? Did you think that the OP literally meant the 12 years or maybe it was just to show how expensive using a Mac Mini as an alternative is?
geodel 1 hours ago [-]
> how expensive using a Mac Mini as an alternative is?
I think it goes without saying. And it is eminently evident over last couple of decades that from compute to storage to meals 3rd part providers have saved billions upon billions of dollars to enterprises and individuals alike by providing these essential services.
kridsdale1 2 hours ago [-]
Mass revolts of the peasantry burning down data centers and cutting fiber lines.
geodel 2 hours ago [-]
Yes, it feels like that. Whereas frontier labs are pushing the frontier of human knowledge, selflessly working towards pulling humanity from dark ages. Ignorant peasants trying to burn the modern civilization down. Don't they know data centers and fiber lines are lifeline of modern economy?
ericmay 2 hours ago [-]
Just commenting here because you're discussing hardware: I thought the test results from the SSD published in this article [1] were pretty interesting. Maybe that's old news though.
Yeah, anyone who thinks local AI is going to save them money is likely to be disappointed, at least if they want to run models that are even remotely capable.
Plenty of other reasons to get excited about local AI, but I don't think cost is one of them.
criddell 3 hours ago [-]
Maybe you are using a local model to go after some Millennium Prize problem and you don't want OpenAI to take your work and use it to win the prize for themselves? $15k might be a bargain.
And, yes, I know a current local model wasn't going to solve the Navier-Stokes problem, but I'm just using it as an example where privacy might be valuable.
simonw 2 hours ago [-]
Agreed, plenty of other reasons to get excited about local AI.
hgoel 2 hours ago [-]
Despite being on a site called Hacker News, we seem to often overlook the simple aspect of wanting local AI hardware to hack (not necessarily in the cybersecurity sense) with. I got my local AI hardware because it's an enjoyable hobby for me.
ionwake 2 minutes ago [-]
apparently if you ever point out HN starts for hackernews and thus expect related attitudes you get downvoted by shocked ( what I guess are zoomers and not bots ) that desperately opine the name is a random abberation doesn't mean anything and one should not deviate from our corporate overlods in any manner.
tempoponet 3 hours ago [-]
While I know it's not apples to apples, the target comparison right now is 2x DGX Sparks. Similar price, 256gb. The conversation has focused on memory bandwidth vs. compute in agentic loops, so for most people the raw numbers will mean less than the "time per task" in coding benchmarks.
This is a great article and bodes well for the M5, but we should expect more like this comparing to other platforms before we truly understand where it fits.
_hugerobots_ 45 minutes ago [-]
Speed vs task-completion is a new conversation and a great point. Whereas the cost to compute doesn't exist in a vacuum, making mistakes costs less, is easier to maintain with granularity and a whole host of other factors when you own the lab.
liuliu 2 hours ago [-]
When people benchmark MLX related quant models, they really need to publish numbers on benchmarks. You cannot take this as it is what you get of the original models. MLX uses pretty simple quantization methods so at lower bits without QAT, it is just not as good quality as llama.cpp ones.
akozak 2 hours ago [-]
"a total cost of $0" Uhh ... how much is that hardware?
novaleaf 1 hours ago [-]
Another comment approximates at around USD$15k, so yeah, not zero.
ApolloFortyNine 3 hours ago [-]
The model being tested is 18k as configured.
I didn't expect this to make the 5090 to look like a good deal.
nacs 2 hours ago [-]
5090 has 32GB VRAM.
It'd be silly to buy the 18k model to run a tiny model like Qwen 27B. You use models like GLM Flash and Qwen Next which won't fit on a single 5090.
15 minutes ago [-]
asimovDev 5 minutes ago [-]
can run multiple subagents of Qwen 27B though, right? Unless I am fundamentally misunderstanding how VRAM constraints work
orsorna 1 hours ago [-]
Is it that silly? You could run multiple 27B models in parallel.
peri-cl 1 hours ago [-]
You actually don't need more RAM to batch multiple inference tasks of the same model.
(Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated).
crossroadsguy 2 hours ago [-]
My mac is 5 years old. I don't think I can comfortably buy a new one right now. It has a 16GB unified RAM. Honestly that would be enough for so many local models that I want to use but can't use. Because RAM usage (even with literally every single user installed app quit/stopped) the RAM usage is very high that I can barely safely get 6-7 GB (I am supposed to get ~10 GB, but it goes up and down real fast!). That's a shame. If only I could install an alternative OS that uses very little amount of RAM :-)
SamuelAdams 2 hours ago [-]
I think Apple is really sleeping on making this run a Linux server. These things are very capable and draw very little wattage when idle. It would make an excellent homelab device, but MacOS currently holds it back in this regard.
What are good options to run local models nowadays? Something good for coding and personal assistant kind of things
saejox 29 minutes ago [-]
i can buy a house with that amount of money. it used to be car money.
theplumber 2 hours ago [-]
At this point I think I will get the DGX gb300 workstation though I will wait a bit more for the cold season. It is double the price but at least is the real thing
addaon 2 hours ago [-]
Ordered one for OpenFOAM. Excited for it. Will be nice to not have my laptop running CFD 24 hours a day, but my M1 Max is currently my fastest machine… I’m expecting about 3.5x from the M5 Ultra.
kokonokko1337 3 hours ago [-]
> "It also happens to be a Mac, with an operating system that looks nice and doesn’t suck"
Yes Apple has some of the best hardware out there, albeit overpriced. But the software is such a hindrance and I can't take anyone that states otherwise seriously. If only it had proper Linux support (and the Asahi people do an amazing job but you can reverse-engineer only so many stuff with limited funding, and then you have to do it again for new models). MacOS is good if you just want to have a standard experience, which to be fair is most people. It's good for just setting up an LLM server I guess since the hardware is a perfect fit. I wouldn't touch it otherwise.
steve1977 1 hours ago [-]
What exactly is missing from macOS that makes you feel the need for Linux?
I get it on Windows systems, at least when someone wants to use Linux-type tooling. But macOS already supports pretty much all of that natively?
RunSet 31 minutes ago [-]
> What exactly is missing from macOS that makes you feel the need for Linux?
For starters, the source code.
steve1977 7 minutes ago [-]
And why would you need that to run LLMs?
Apart from that, for the UNIX part, the source is available for quite a few components:
If you are buying expensive hardware to run LLMs "on your own machine" you will soon find your ladder is on the wrong wall.
devy 3 hours ago [-]
This dream machine costs over $15k (not including the Apple Studio Display)? Nah, that dream is SO OUT OF TOUCH!
slashtom 45 minutes ago [-]
Fantastic review, this is how it should be done with local AI.
12kaj2 2 hours ago [-]
The Year Of Local AI will be here no later than 2040, coinciding with the Year Of The Linux Desktop.
prmoustache 1 hours ago [-]
The year of linux on the Desktop was 26 years ago for me.
snarfy 3 hours ago [-]
$12,299
andrekandre 2 hours ago [-]
5 years of (200/month) tokens at that price, meanwhile an rtx 5090 pc is about half that… hmm
but i wonder how much these token costs are sustainable or not, it may be in the long term cheaper to have your own hardware if token costs go up (and hopefully hardware gets cheaper again)
chasd00 2 hours ago [-]
The token price isn't the only reason to run a model locally though. You can do additional training to specialize or remove censorship that may be a no-no per TOS with cloud GPUs.
cptskippy 1 hours ago [-]
I think we'll eventually get to the point where folks will have a local AI agent but I think people need to temper their expectations to a degree. You aren't going to have data center level tok/s from a box sitting under your desk and you don't need instantaneous responses for many workloads. Having a local agent that can execute tasks over a couple days with your supervision that might otherwise take you weeks is perfectly acceptable.
However I also think that Agentic AI is very much not an out-of-the-box solution, local or otherwise, and it takes a high level of technical knowledge to create an effective AI agent. And there's a problem now where most orchestration is fixed on what models are used for what tasks with no ability to weight constraints like cost, speed, and security.
villgax 1 hours ago [-]
Lol, try generation of images & videos on these, they ought to improve perf on Deep learning not just llms
WarmWash 3 hours ago [-]
>Let’s address the elephant in the room first: why bother with local AI at all when cloud frontier models are better and often faster?
Ehh, the actual elephant in the room is:
"why bother with local AI at all when you can lease a GPU for $5/hr?"
To which the answer is you shouldn't bother, unless you have a bunch of money to throw at hobby projects.
Youden 4 minutes ago [-]
$5/hr = $3600/mo.
Unless you only need the AI available some of the time, $5/hr is pretty expensive. That's an RTX Pro twice a year.
If you're using it for discrete sessions of coding or something, that might make sense for you but if you're using it for an always-on assistant, that pricing kinda sucks.
chasd00 2 hours ago [-]
> unless you have a bunch of money to throw at hobby projects.
there are lots of people with very expensive hobbies, see sailboat racing for example.
WarmWash 56 minutes ago [-]
"Lots" of people are billionaires too, just like "lots" of people like to watch TV in their free time.
sghiassy 3 hours ago [-]
Imagine spending a trillion dollars on data centers and then reading this article. Nightmare fuel for OpenAI
CamperBob2 11 minutes ago [-]
And nightmare fuel is just what they'll be selling at the UN this week, for this very reason.
Sam's address will probably be more riveting, imaginative, and terrifying than the last couple of Terminator screenplays. Legislators will lobby him to write the laws for them, and the ghost of Harlan Ellison will threaten to sue him.
whalesalad 3 hours ago [-]
For 99.99% of people, spending 15 grand on a Mac Studio just to run Qwen 3.8 locally is a non starter.
jmull 2 hours ago [-]
It's not the M5 Ultra itself, but the M7s or M9s that will do the damage.
99% of people will use whatever AI is free. The sophisticated, heavy users that are willing and able to pay a lot of money the ones that will be interested in controlling their inference bills.
Today, the sweet spot where an M5 Ultra makes sense is tiny. But we might expect that to grow a lot.
BatFastard 51 minutes ago [-]
Anthropic is reporting 100 Billion ARR.
Even if you could get a frontier model, you would not be able to run it on any Mac. So speculating on what M7 or M9 will achieve in 5 years (if we even still exist) seems pointless.
sghiassy 43 minutes ago [-]
Do you need a frontier model to write emails, check your calendar, search the web?
I don’t think Apple is going to lie down and cede AI to the cloud.
sghiassy 1 hours ago [-]
Yes, but in 7 years?
whalesalad 46 minutes ago [-]
In 7 years we will probably all be living under ground fighting skynet with plasma rifles made out of old microwave parts
fragmede 40 minutes ago [-]
You will. Some of us are going to be already ground into dust that the microwave parts are made out of. Others will be locked into our communism cubes with our daily allotment of entertainment and sustinece. let out into the sunlight for only 30 minutes per day.
beastman82 2 hours ago [-]
at 15 tok/s
rwissinger 4 hours ago [-]
[dead]
Rendered at 17:58:55 GMT+0000 (Coordinated Universal Time) with Vercel.
I setup an eBay alert and picked up a used M2 Ultra that has delivered good ROI (at least, far better than 15k for comparable-for-my-use-case performance)
I dont' know why people spend huge money on these and Spark. The 5090 is running qwen 3.8 at 200+ tps!! That's 1-2 orders of magnitude faster.
My technical-expert twin played around with that LLM for about an hour, and then correctly reasoned "it's able to be WRONG, faster."
This seems apt.
No thanks to the "macos value add" that forces you to use Metal while Valve customers frolick in Protonland.
Crossover works on macos, too. So does moltenvk, so does vanilla wine, etc etc. You can run most games without a hitch these days (allegedly, according to /r/macgaming). But I don't play video games so a GPU would probably be better off in some kid's computer.
I don't get these weird parasocial emotional attachments/beefs people have with brands. Talk to a therapist.
Now try running that Qwen 3.8 Next model on the 5090 and tell me what TPS you get (hint: it's near 0 since it doesnt fit the 32GB VRAM on 5090 vs the 256 in OPs M5).
https://old.reddit.com/r/LocalLLaMA/comments/1wl06np/qwen38f...
(Note it's a sparse MoE with only 6B active).
That's with CPU offload to a DDR5 6000 RAM though which is around $3-4k at least.
Whereas a hybrid architecture with distinct DRAM and VRAM with sparse MoE, you can leverage two different bit rates depending on the actual need for constant access to common layers versus sparse access to infrequent layers and arbitrage the difference in cost for each of those in distinct classes of hardware.
Also: ~30 token/s on GLM 5.3-flash, locally. (That's roughly Opus 4.8-tier. I think).
/meta Here's a CSS filter that stops those nuisance chart animations,
Maybe Apple is an acquisition away from changing that balance.
https://www.macrumors.com/2026/06/25/2027-macs-m7-chips/
Apple just shifted to N2. They’re not going to be doing another major shift right away.
And TSMCs own roadmap would put your hallucination years away at best for a a product that follows a roughly annual cadence https://www.tomshardware.com/tech-industry/semiconductors/ts...
This advantage won't be apparent with a 27B model. The 256GB MS can probably run the newer Flash models locally, something you can't do on a 5090.
I don't think we'll get a successor to the 5090 until late 2028, maybe even 2029. I'm basing this on the launch date of the 5000 series and that we haven't got a midcycle refresh yet. Rumor has it the chips are ready but the 3GB RAM modules are 3-4x the price of the 2GB modules used on the current cards.
Apple should see a Mac Studio major update in 2028. That might even force NVidia's hand. But it's really impossible to say what the state of the market will be 2-3 years from now. It may have completely crashed. I suspect not however.
The interesting thing will be when the bandwidth demands start forcing HBM memory onto these home/enthusiast solutions.
Each PC is probably going to cost ~$6k and you're talking about 8000W of electricity draw. That's going to consume multiple 20A circuits even at 240V. And the electricity ain't free either. A Mac Studio seems to draw ~500W max.
Oh and the Mac Studio has an upgrade route to run 1T+ models too by chaining them together with TB5 chaining. OSX supports RDMA this way. That's comparable bandwidth to the 100Gbps Infiniband option.
So you're talking about $50-60k of hardware and more power draw and more heat for something that will I'm sure beat the MS M5U option but at huge cost. Also, at that kind of price point, I'm likely to get a workstation PC and put 2 (or possibly 3) 6000 Pros in it.
nvidia-smi -pl 450 for like a 4% reduction in throughput. I tend to set it around 350W because it's a comfortable temperature blowing on my legs under the desk without warming my office in the summer.
I put together this system two years ago, so it's a little out of date, but it only cost $3000 for the same performance and capability as an Ultra. I don't think I would spend $7000 to save 100W, though.
> Apple's planned M7 Ultra chip is being designed to support up to 1.5 TB of unified memory and to push AI performance toward the class of Nvidia's Blackwell accelerators, according to a new Bloomberg report published by Mark Gurman...
Apple plans to release a base M6 chip this fall for entry-level Macs... a base M7 in the first half of 2027, M7 Pro and M7 Max at the end of 2027, and the M7 Ultra in 2028.
https://www.tomshardware.com/tech-industry/semiconductors/ap...
I'm also curious about any new low hanging optimization opportunities in the kernels for this new hardware.
It's already clear to me that M5 Mac Studio is more cost-effective than anything you can run on open router, assuming decent utilization.
The M5 Mac Studio will be the most cost effective way to run uncensored cyber capable open agents.
An exciting tipping point will be if programmers can get an Astra-Ultra like experience all week with this hardware. That would be a real sense where this hardware exceeds the value of even 20x cloud subscriptions.
Local models are definitely not as productive as SOTA, sadly it's not close yet. I do think someday they will be "good enough" to use, but they aren't today. Even the SOTA models barely code well, with Opus 4.5 being the first, good coding model.
That being said, I think it's absolutely imperative that we keep pushing local model performance. We need to continue to advance technology there and ensure that the model labs don't do regulatory capture in the name of "safety" (or anything else).
I'm also a huge fan of local models and think it's absolutely imperative that they continue to advance so we can move off of the Anthropic/OpenAI hosted models. It's important to accurately asses where we are in that journey though.
That’s like 12 years worth of OpenAI Pro subscriptions
I appreciate the reference to RUSH: Red Barchetta in the final line.
Specially since one can pay half right now to OpenAI and sign a 12 year iron clad contract for uninterrupted service delivery of OpenAI Pro.
A more apples-to-apples comparison would be with API cost in OpenRouter at the same tok/s rate for the same models that you can run locally, maybe.
Don't you mean an Apple to NVidea comparison?
I think it goes without saying. And it is eminently evident over last couple of decades that from compute to storage to meals 3rd part providers have saved billions upon billions of dollars to enterprises and individuals alike by providing these essential services.
[1] https://www.macworld.com/article/3238319/mac-studio-m5-max-r...
Plenty of other reasons to get excited about local AI, but I don't think cost is one of them.
And, yes, I know a current local model wasn't going to solve the Navier-Stokes problem, but I'm just using it as an example where privacy might be valuable.
This is a great article and bodes well for the M5, but we should expect more like this comparing to other platforms before we truly understand where it fits.
I didn't expect this to make the 5090 to look like a good deal.
It'd be silly to buy the 18k model to run a tiny model like Qwen 27B. You use models like GLM Flash and Qwen Next which won't fit on a single 5090.
(Each task needs its own context, but the (e.g.) 27B of constant parameters isn't duplicated).
Yes Apple has some of the best hardware out there, albeit overpriced. But the software is such a hindrance and I can't take anyone that states otherwise seriously. If only it had proper Linux support (and the Asahi people do an amazing job but you can reverse-engineer only so many stuff with limited funding, and then you have to do it again for new models). MacOS is good if you just want to have a standard experience, which to be fair is most people. It's good for just setting up an LLM server I guess since the hardware is a perfect fit. I wouldn't touch it otherwise.
I get it on Windows systems, at least when someone wants to use Linux-type tooling. But macOS already supports pretty much all of that natively?
For starters, the source code.
Apart from that, for the UNIX part, the source is available for quite a few components:
https://github.com/apple-oss-distributions
notably also the kernel
https://github.com/apple-oss-distributions/xnu
but i wonder how much these token costs are sustainable or not, it may be in the long term cheaper to have your own hardware if token costs go up (and hopefully hardware gets cheaper again)
However I also think that Agentic AI is very much not an out-of-the-box solution, local or otherwise, and it takes a high level of technical knowledge to create an effective AI agent. And there's a problem now where most orchestration is fixed on what models are used for what tasks with no ability to weight constraints like cost, speed, and security.
Ehh, the actual elephant in the room is:
"why bother with local AI at all when you can lease a GPU for $5/hr?"
To which the answer is you shouldn't bother, unless you have a bunch of money to throw at hobby projects.
Unless you only need the AI available some of the time, $5/hr is pretty expensive. That's an RTX Pro twice a year.
If you're using it for discrete sessions of coding or something, that might make sense for you but if you're using it for an always-on assistant, that pricing kinda sucks.
there are lots of people with very expensive hobbies, see sailboat racing for example.
Sam's address will probably be more riveting, imaginative, and terrifying than the last couple of Terminator screenplays. Legislators will lobby him to write the laws for them, and the ghost of Harlan Ellison will threaten to sue him.
99% of people will use whatever AI is free. The sophisticated, heavy users that are willing and able to pay a lot of money the ones that will be interested in controlling their inference bills.
Today, the sweet spot where an M5 Ultra makes sense is tiny. But we might expect that to grow a lot.
Even if you could get a frontier model, you would not be able to run it on any Mac. So speculating on what M7 or M9 will achieve in 5 years (if we even still exist) seems pointless.
I don’t think Apple is going to lie down and cede AI to the cloud.