> optimizations applied at different levels of the stack
That's because it's just "guess and check" not some deep universal insight.
freakynit 29 minutes ago [-]
Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?
rf15 1 minutes ago [-]
Ok, I'll bite: no, considering these are very different domains and you don't get system access by getting the wrong speculative branch for your next text token, you just get a slightly different (but probably still related enough) text.
arjie 27 minutes ago [-]
You know what I'm curious about? Whether you have brand guidelines inside the company, a Claude skillset, or the blog post author makes the charts in line with the brand colours and so on.
brrrrrm 4 hours ago [-]
this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time!
the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there
philipkiely 4 hours ago [-]
I think the biggest net new recent technique is P/D disaggregation. And that spec dec is very different now especially post DSpark/DFlash.
But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.
brrrrrm 3 hours ago [-]
perhaps its unfair to say this in hindsight, but it's a fairly straightforward application of little's law that's been around for some time
Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate.
This is a tautology. You can say that with anything. Gastronomy techniques will make a previous recipe better, or create a new recipe better than others, or a mix of both.
Ifkaluva 2 hours ago [-]
The point is to classify them into two kinds. The kind that shifts the frontier is more powerful, since improves capabilities without incurring tradeoffs.
philipkiely 2 hours ago [-]
I also wrote this as somewhat of a defense of the techniques that don't move the frontier -- there is a lot of value in being able to pick a point on the curve.
fsckboy 21 minutes ago [-]
"the efficient frontier" is an important landmark of (investment) portfolio theory. It proves/explains/illustrates how you can combine selections from a diffuse cloud of individual investments and still land on a frontier that is better than any of your individual choices. It's the entire basis of "diversify your portfolio".
The efficient frontier of LLM inference is a line, not a frontier.
no matter how good is something a smart person writes down, a pleb will come along and try to hang on its coattails.
If you want to steal an idea for this, steal indifference curves, they'd make more sense.
calclavia 4 hours ago [-]
good recap on the recent inference techniques!
yeasin-arafat 4 minutes ago [-]
[flagged]
paidx 4 hours ago [-]
[flagged]
jing09928 3 hours ago [-]
[dead]
nedo_var 4 hours ago [-]
[dead]
killerdog10 3 hours ago [-]
[dead]
datadrivenangel 4 hours ago [-]
The author does not deeply mention that quality/intelligence is a third dimension here in addition to throughput and latency, and the frontier is jagged so quality and intelligence require bespoke benchmarks to evaluate tradeoffs for speed and cost.
philipkiely 4 hours ago [-]
These are both good points that I attempted to cover, quotes:
> In practice, the efficient frontier is very jagged. Rather than a smooth, continuous line between outcomes, small changes can have big impacts. These cutoff points are often unintuitive and must be discovered empirically through sweeps.
> However, quantization introduces a new set of tradeoffs between quality and serving efficiency. This is a particularly jagged frontier, where a large degree of improvement to serving efficiency is possible with little-to-no reduction in model quality, especially when using microscaling floating-point number formats like MXFP4 and NVFP4.
Would appreciate ideas on how to explain in greater depth
Rendered at 04:51:21 GMT+0000 (Coordinated Universal Time) with Vercel.
As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.
Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.
Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].
Everything old is new again (:
[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...
[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf
[2] https://www.microsoft.com/en-us/research/publication/distrib...
That's because it's just "guess and check" not some deep universal insight.
the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there
But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.
https://arxiv.org/html/2401.09670v2
The efficient frontier of LLM inference is a line, not a frontier.
this is a frontier:
https://upload.wikimedia.org/wikipedia/commons/e/e1/Markowit...
no matter how good is something a smart person writes down, a pleb will come along and try to hang on its coattails.
If you want to steal an idea for this, steal indifference curves, they'd make more sense.
> In practice, the efficient frontier is very jagged. Rather than a smooth, continuous line between outcomes, small changes can have big impacts. These cutoff points are often unintuitive and must be discovered empirically through sweeps.
> However, quantization introduces a new set of tradeoffs between quality and serving efficiency. This is a particularly jagged frontier, where a large degree of improvement to serving efficiency is possible with little-to-no reduction in model quality, especially when using microscaling floating-point number formats like MXFP4 and NVFP4.
Would appreciate ideas on how to explain in greater depth