It seems implementing transformers is the new "hello world" of AI assisted coding.
datadrivenangel 1 hours ago [-]
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
risyachka 1 hours ago [-]
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it
datadrivenangel 1 hours ago [-]
it only cost them $10,000 in Claude API usage. :)
hmate9 1 hours ago [-]
You get that much usage in a monthly $200 subscription.
dvh 45 minutes ago [-]
So who's paying the cost?
stronglikedan 4 minutes ago [-]
Everyone else that doesn't use the entirety of the plan they pay for.
ralph84 12 minutes ago [-]
API pricing vs. individual plan pricing is just price discrimination. They're both positive gross margin.
surgical_fire 5 minutes ago [-]
People believe in some wild stuff.
exe34 43 minutes ago [-]
Corporate customers
dofm 1 hours ago [-]
And a kitten.
codekansas 1 hours ago [-]
This is cool
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
chalkycrimp 54 minutes ago [-]
Good chance I'm not the only one thinking this: share the code
spwa4 4 minutes ago [-]
I must say I do see one application for this, that you can't really match reasonably: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
saejox 45 minutes ago [-]
i wonder if we can emulate an FPGA on a rtx 3090? then test it on an AWS EC2 F2 few hours a day.
It seems implementing transformers is the new "hello world" of AI assisted coding.
Zero surprise it did it
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
https://architectlabs.com/blog/frostbite