For people interested in this: I've been working on an open-source libtpu implementation for https://tenstorrent.com/ accelerators as a side project, it can already run most of torchtpu, see https://github.com/pcmoritz/libtt (based on the tenstorrent software stack). It shows all the things involved in lowering torch to tile based accelerators.
Thanks for sharing, this is super cool! Right now my approach to correctness it to just run as many tests as possible and fix them (torchtpu greatly expands the amount of available tests), but formal verification is also super cool. Having a look at your book https://lean.brettkoonce.com/blueprint.pdf
Thanks for the encouragement, if you have the hardware and end up trying it out I'd love to get your feedback (feel free to drop an issue in the repo) :)
yhjc2692 53 minutes ago [-]
I would love to do that, but unfortunately don’t have one on hand (yet?). In fact, I’m curious - did you buy a TT device to work on this, or are you using a cloud provider like Koyeb?
pcmoritz 49 minutes ago [-]
I first built my own to try it out (the exact hardware is described in https://relativity.cc/hardware/ under "one Blackhole card") and when I liked the hardware I got a quietbox 2. It is very high quality but a little expensive (but inexpensive compared to other LLM hardware and it has excellent interconnects which nobody else in the consumer/dev space has).
nico 27 minutes ago [-]
Pardon my ignorance, but who else besides Google has TPUs? Are they commercially available somewhere? What’s their main application right now outside of Google?
cavisne 16 minutes ago [-]
You can rent them on cloud + they sell directly to Anthropic. Its pretty well known that Anthropic is a heavy user of TPU's (along with Trainium & GPU's)
teaearlgraycold 17 minutes ago [-]
They sell IoT-grade ones
npalli 3 hours ago [-]
Interesting, Amazon doing the same for AWS Trainium.
> How does Cerebras WSE-3 with 44GB of 'L2' on-chip SRAM compare to Google's TPUs, Tesla's TPUs, NorthPole, Groq LPU, Tenstorrent's, and AMD's NPU designs?
There are TPUs made out of silicon carbide with standard fab.
There is in-flight tensor processing with photons, but that's not quite a TPU.
hackernudes 2 hours ago [-]
I think the commenter was asking if Google sells TPU chip hardware to use outside of Google's own infrastructure and cloud.
tuna74 32 minutes ago [-]
Is the Linux driver available?
23agt 3 hours ago [-]
Google is selling TPU shovels via Cloud services and also physically to select partners. So this makes sense.
There will be an abundance of shovels. Nebius already crashed by almost half from its hype peak.
airza 2 hours ago [-]
it doesn't seem like there's an abundance of shovels coming anytime soon
yipinwong 3 hours ago [-]
Dumb question from non-AI engineer.
Is this somehow Google's answer to today's best DeepSeek open-sourcing inferencing engine?
bitpush 3 hours ago [-]
I dont think big companies like Google react to news like that. Very likely this was meant to go out on Tuesday for weeks now. And it just happened to have coincided with another news.
JacobAsmuth 2 hours ago [-]
Yeah they read the news this morning and whipped up a quick Torch->XLA translation layer and coordinated with four different inference engines (who also all wrote their own blog posts) and shoved this out the door quick to stay on top of the news cycle
parthdesai 2 hours ago [-]
Why be rude on purpose? OP clearly prefaced by saying it was a dumb question
caaqil 1 hours ago [-]
Welcome to the internet, great to have you here.
mathisfun123 2 hours ago [-]
they're releasing this today because the pytorch conf is this week...
geremiiah 2 hours ago [-]
No, these are two very different objectives.
Tanjreeve 3 hours ago [-]
They do different things and sit in different parts of the stack (training vs inference). TorchTPU seems more like a play to get more people familiar with Googles TPU chips as they want to sell them to people outside of google.
beardedetim 2 hours ago [-]
Yeah I'd assume this is to allow us to train on their TPUs with torch instead of needing to rent a GPU
yipinwong 1 hours ago [-]
ty. that helps a lot and know where to research next.
so if the moat isn't the model and the moat isn't the inference layer, i guess the moat is the hardware now? what does this mean for frontier labs?
badrequest 2 hours ago [-]
hardware has always been the moat in the AI space
AnotherGoodName 46 minutes ago [-]
Yeah a lot of people bought into the idea of CUDA being a moat but we’re well past the point where it’s pretty trivial to jump to another equivalent stack on the software side.
coredog64 2 minutes ago [-]
Even if it's allegedly possible, the devil has always been in the details. It always sucks to find out that there's some quirk of your model architecture that breaks an implicit assumption that underpins the export process.
jacquesm 31 minutes ago [-]
Those are the two main interfaces: the inference hardware and the endpoints. Endpoints are pretty much standardized to the point that you can switch them out in seconds. CUDA still has an edge but every release like this chips away at that. Even so, TPUs are unobtanium so while I appreciate Google's effort here I also recognize that it is not something that will be useful to me.
In that sense AMD is still better positioned to give NVidia a run for their money but given the family connection and AMD's underdog status I am not very hopeful that we'll see meaningful competition in inference hardware any time soon.
NVidia is a much harder nut to crack than OpenAI or Anthropic. They are easily a decade ahead of viable competition. Check out the GB300 if you don't believe that.
AndrewKemendo 58 minutes ago [-]
Will always be true
JacobAsmuth 2 hours ago [-]
Products can also be a moat.
Androcles 3 hours ago [-]
[flagged]
karansuthar 2 hours ago [-]
[dead]
Rendered at 20:00:51 GMT+0000 (Coordinated Universal Time) with Vercel.
The Tenstorrent ISA is documented btw https://github.com/tenstorrent/tt-isa-documentation, I wonder if one could do the verification end-to-end with the hardware.
https://github.com/aws-neuron/torch-neuronx
Also I guess this is the final nail in the coffin of Tensorflow.
Since HW specifics live mostly in the kernel layer (Pallas or CUDA), I imagine the Jax vs PyTorch question is more of a style preference now?
"No, but I have seen the house that the project built, and it is terrific." ~ some Google director
"OpenTPU: Open-Source Reimplementation of Google Tensor Processing Unit (TPU)" https://news.ycombinator.com/item?id=44111452
From https://news.ycombinator.com/item?id=45848676 :
> How does Cerebras WSE-3 with 44GB of 'L2' on-chip SRAM compare to Google's TPUs, Tesla's TPUs, NorthPole, Groq LPU, Tenstorrent's, and AMD's NPU designs?
There are TPUs made out of silicon carbide with standard fab.
There is in-flight tensor processing with photons, but that's not quite a TPU.
There will be an abundance of shovels. Nebius already crashed by almost half from its hype peak.
Is this somehow Google's answer to today's best DeepSeek open-sourcing inferencing engine?
In that sense AMD is still better positioned to give NVidia a run for their money but given the family connection and AMD's underdog status I am not very hopeful that we'll see meaningful competition in inference hardware any time soon.
NVidia is a much harder nut to crack than OpenAI or Anthropic. They are easily a decade ahead of viable competition. Check out the GB300 if you don't believe that.