I'd recommend having a "gemm with a twist" [0] example in the README.md instead of having an element-wise example. It's pretty hard to evaluate how helpful this is for AI otherwise.
[0] For example, gemm but the lhs is in fp8 e4m3 and rhs is in bf16 and we want fp32 accumulation, output to bf16 after applying GELU.
wingertge 289 days ago [-]
We don't yet support newer types like fp8 and fp4, that's actually my next project. I'm the only contributor with the hardware to actually use the new types, so it's a bit bottlenecked on a single person right now. But yes, the example is rather simplistic, should probably work on that some time once I'm done updating the feature set to Blackwell.
lostmsu 289 days ago [-]
Isn't there a CPU-based "emulator" in Nvidia dev tools?
wingertge 289 days ago [-]
From what I can tell it's not accurate enough to catch a lot of errors in the real world. Maybe an illegal instruction, but not a race condition from a missing sync or a warp divergence on a uniform instruction or other potential issues like that.
ashvardanian 289 days ago [-]
Agreed! I was looking through the summation example < https://github.com/tracel-ai/cubecl/blob/main/examples/sum_t...> and it seems like the primary focus is on the more traditional pre-2018 GPU programming without explicit warp-level operations, asynchrony, atomics, barriers, or countless tensor-core operations.
The project feels very nice and it would be great to have more notes in the README on the excluded functionality to better scope its applicability in more advanced GPGPU scenarios.
nathanielsimard 289 days ago [-]
We support warp operations, barriers for Cuda, atomics for most backends, tensor cores instructions as well. It's just not well documented on the readme!
ashvardanian 289 days ago [-]
Amazing! Would love to try them! If possible, would also ask for a table translating between CubeCL and CUDA terminology. It seems like CUDA Warps are called Planes in CubeCL, and it’s probably not the only difference.
0x7cfe 289 days ago [-]
CubeCL is the computation backend for Burn (https://burn.dev/) - ML framework done by the same team which does all the tensor magic like autodiff, op fusion and dynamic graphs.
nathanielsimard 289 days ago [-]
One of the main author here, the readme isn't really well up-to-date. We have our own gemm implementation based on CubeCL. It's still moving a lot, but we support tensor cores, use warp operations (Plane Operations in CubeCL), we even added TMA instructions for CUDA.
In Halide, the concept was great, yet the problems in kernel development were moved to the side of "scheduling", i.e. determining tiling/vectorization/parallellization for the kernel runs.
the__alchemist 289 days ago [-]
Love it. I've been using cudarc lately; would love to try this since it looks like it can share data structures between host and device (?). I infer that this is a higher-level abstraction.
zekrioca 289 days ago [-]
Very interesting project! I am wondering how it compare against OpenCL, which I think adopts the same fundamental idea (write once, run everywhere)? Is it about CUbeCL's internal optimization for Rust that happens at compile time?
fc417fc802 289 days ago [-]
This appears to be single source which would make it similar to SYCL.
Given that it can target WGPU I'm really wondering why OpenCL isn't included as a backend. One of my biggest complaints about GPGPU stuff is that so many of the solutions are GPU only, and often only target the vendor compute APIs (CUDA, ROCm) which have much narrower ecosystem support (versus an older core vulkan profile for example).
It's desirable to be able to target CPU for compatibility, debugging, and also because it can be nice to have a single solution for parallelizing all your data heavy work. The latter reduces mental overhead and permits more code reuse.
wingertge 289 days ago [-]
There's infrastructure in the SPIR-V compiler to be able to target both OpenCL and Vulkan, but we don't currently use it because OpenCL would require a new runtime, while Vulkan can simply use the existing wgpu runtime and pass raw SPIR-V shaders.
One thing I've never investigated is how performance OpenCL actually is for CPU. Do you happen to have any resources comparing it to a more native CPU implementation?
fc417fc802 289 days ago [-]
Sorry my interest there is debugging and I'm not immediately coming across good benchmarks. PoCL [0] seems to have added a TBB backend [1] so I'd expect it to be reasonable (otherwise why bother) but I haven't tested it.
It isn't really related to your question but I think the FluidX3D benchmarks [2] illustrate that OpenCL is at least viable across a wide variety of hardware.
As far as targeting CPUs in a release build it's not a particular backend that's important to me. The issue is at the source code level. Having single source is nice but you're still stuck with these two very different approaches. It means that the code is still clearly segmented and thus retargeting any given task (at least nontrivial ones) involves rewriting it to at least some extent.
Contrast that with a model like OpenMP where the difference between CPU and GPU is marking the relevant segment for offload. Granted that you'll often need to change algorithms when switching to achieve reasonable performance but it's still a really nice quality of life feature not to have to juggle more paradigms and libraries.
Makes sense. And indeed, having OpenCL as a backend would be a very interesting extension.
ttoinou 289 days ago [-]
Who would use the OpenCL backend rather than the others targets provided ?
nathanielsimard 289 days ago [-]
A lot of things happen at compile time, but you can execute arbitrary code in your kernel that executes at compile time, similar to generics, but with more flexibility. It's very natural to branch on a comptime config to select an algorithm.
unit149 289 days ago [-]
[dead]
gitroom 289 days ago [-]
Gotta say, the constant dance between all these GPU frameworks kinda wears me out sometimes - always chasing that better build, you know?
nathanielsimard 289 days ago [-]
The need to build CubeCL came from the Burn deep learning framework (https://github.com/tracel-ai/burn), where we want to easily build algorithms like in CUDA with a real programming language, while also being able to integrate those algorithms inside a compiler at runtime to fuse dynamic graphs.
Since we don't want to rewrite everything multiple times, it also has to be multi-platform and optimal, so the feature set must be per-device, not per-language. I'm not aware of a tool that does that, especially in Rust (which Burn is written in).
fc417fc802 289 days ago [-]
> I'm not aware of a tool that does that
Jax? But then you're stuck in python. SYCL?
But yeah not for Rust. This project is filling a prominent hole IMO.
rowanG077 288 days ago [-]
Futhark immediately came to mind. It's designed to be able to be trivially integrated into a package.
I don't think that fits; that's a ML framework. The others in the link are general GPU frameworks.
bionhoward 289 days ago [-]
Praying to the kernel gods for some Rust FP8 training
adastra22 289 days ago [-]
Where is the Metal love…
Almondsetat 289 days ago [-]
Why would anyone love something born out of pure spite for industry standards?
m-schuetz 289 days ago [-]
To be fair, the industry standards all suck except for CUDA.
pjmlp 289 days ago [-]
For the same reason CUDA and ROCm are supported.
miohtama 289 days ago [-]
Apple is known to be not that great contributor to open source, unlike Nvidia, AMD, Intel.
pjmlp 289 days ago [-]
You should check Linus opinion on those.
Also, to whom do you have to thank LLVM exists in first place, and has not fizzled out as yet another university compiler research project?
289 days ago [-]
syl20bnr 289 days ago [-]
It also compiles directly to MSL, it is just missing from the post title.
adastra22 289 days ago [-]
No it compiles indirectly through wgpu, which means it doesn’t have access to any Metal extensions not exposed by the wgpu interface.
syl20bnr 289 days ago [-]
I am the coder of the MSL dialect for the CubeCL CPP compiler. Since 0.5 release it directly compiles to MSL and support simdgroup matrix functions for instance. It does use wgpu for the runtime but without naga as we added msl pass through to wgpu just for this.
adastra22 289 days ago [-]
You should update the README.
syl20bnr 289 days ago [-]
You are right we just released Burn and updated its readme, we were not thinking that CubeCL would be the one that could be featured. ^_^
grovesNL 289 days ago [-]
wgpu has some options to access backend-specific types and shader passthrough (i.e., you provide your own shader for a backend directly).
Generally wgpu is open to supporting any Metal extensions you need. There's usually an analogous extension in one of the other backends (e.g., Vulkan, DX12) anyway.
moffkalast 289 days ago [-]
From the moment I understood the weakness of my flesh, it disgusted me. I craved the strength and certainty of steel. I aspired to the purity of the Blessed Machine.
DarkmSparks 289 days ago [-]
wow, what's the downsides to this? It feels like it could be one of the biggest leaps in programming in a long time, does it keep rusts safety aspects? How does it compare with say openCL?
nathanielsimard 289 days ago [-]
We have safe and unsafe version for launching kernels where we can ensure that a kernel won't corrupt data elsewhere (and therefore won't create memory error or segfaults). But within a kernel ressources are mutable and shared between GPU cores, since that's how GPUs work.
289 days ago [-]
Rendered at 09:46:22 GMT+0000 (Coordinated Universal Time) with Vercel.
[0] For example, gemm but the lhs is in fp8 e4m3 and rhs is in bf16 and we want fp32 accumulation, output to bf16 after applying GELU.
The project feels very nice and it would be great to have more notes in the README on the excluded functionality to better scope its applicability in more advanced GPGPU scenarios.
In Halide, the concept was great, yet the problems in kernel development were moved to the side of "scheduling", i.e. determining tiling/vectorization/parallellization for the kernel runs.
Given that it can target WGPU I'm really wondering why OpenCL isn't included as a backend. One of my biggest complaints about GPGPU stuff is that so many of the solutions are GPU only, and often only target the vendor compute APIs (CUDA, ROCm) which have much narrower ecosystem support (versus an older core vulkan profile for example).
It's desirable to be able to target CPU for compatibility, debugging, and also because it can be nice to have a single solution for parallelizing all your data heavy work. The latter reduces mental overhead and permits more code reuse.
One thing I've never investigated is how performance OpenCL actually is for CPU. Do you happen to have any resources comparing it to a more native CPU implementation?
It isn't really related to your question but I think the FluidX3D benchmarks [2] illustrate that OpenCL is at least viable across a wide variety of hardware.
As far as targeting CPUs in a release build it's not a particular backend that's important to me. The issue is at the source code level. Having single source is nice but you're still stuck with these two very different approaches. It means that the code is still clearly segmented and thus retargeting any given task (at least nontrivial ones) involves rewriting it to at least some extent.
Contrast that with a model like OpenMP where the difference between CPU and GPU is marking the relevant segment for offload. Granted that you'll often need to change algorithms when switching to achieve reasonable performance but it's still a really nice quality of life feature not to have to juggle more paradigms and libraries.
[0] https://github.com/pocl/pocl
[1] https://portablecl.org/docs/html/drivers.html
[2] https://github.com/ProjectPhysX/FluidX3D
Since we don't want to rewrite everything multiple times, it also has to be multi-platform and optimal, so the feature set must be per-device, not per-language. I'm not aware of a tool that does that, especially in Rust (which Burn is written in).
Jax? But then you're stuck in python. SYCL?
But yeah not for Rust. This project is filling a prominent hole IMO.
Also, to whom do you have to thank LLVM exists in first place, and has not fizzled out as yet another university compiler research project?
Generally wgpu is open to supporting any Metal extensions you need. There's usually an analogous extension in one of the other backends (e.g., Vulkan, DX12) anyway.