> Modern RDMA for Ethernet and IP – Supporting intelligent, low-latency transport for high-throughput environments.
> Open Standards and Interoperability – Avoids vendor lock-in while accelerating ecosystem-wide innovation.
> End-to-End Scalability – From routing and provisioning to operations and testing, UEC scales to millions of endpoints.
frantathefranta 241 days ago [-]
Skimming this, it looks like it's trying compete with Infiniband, but combining it with plain Ethernet?
jauntywundrkind 241 days ago [-]
That so much of the document is about how UltaEthernet maps to Libfabric rather confirms that premise! There's also a credit based flow system, and connection manager roles, which are also easily identifiable Infiniband concepts.
I'm interested to see what the optional hardware features are. And what their relationship is to the different UE profiles (AI Base, AI Full, and HPC).
> The Ultra Ethernet Transport (UET) layer is designed to handle the most challenging application scale,
deliver packets reliably and securely, manage and avoid congestion within the network, and react to
contention at the endpoints. Its goals are minimal tail latency and highest network utilization. At the
same time, UET is designed to enable simple hardware and software implementations – such as what
might be required for accelerator-integrated endpoints. UET can be programmed through the OFI
libfabric standard interface. It sets out to address the shortcomings of RoCEv2, specifically its
semantics, transport layer, wire operations, implementation complexities, and scale limits
deaddodo 241 days ago [-]
Self-admittedly less knowledgeable about this subject, but how does this differ from IBoE?
jauntywundrkind 241 days ago [-]
The end of the quote I provided does some very high level contrasting of the well known RoCE v2 RDMA over converged Ethernet, which is the most popular IBoE like thing.
I'm sure there's a lot more nuanced to it all. But I think predictability/utilization/latency in RoCE are worse, that it relies on Explicit Congestion Notification more for flow control. Where-as UE is using Infiniband style credit based flow control, which should insure that any data sent has sufficient throughout allocated to it to be received.
UE seems to be a more direct creation of an Infiniband like network, atop Ethernet but where all players are agreeing to behave in an Infiniband like way with Infiniband predictability, where-as RoCE encapsulates Infiniband data but still behaves more like an Ethernet network lacking the coordination of Infiniband. I'm far from certain; it'd be so fun to have some extensive material to go over to really find out.
bobmcnamara 241 days ago [-]
Seems like it. Sorta how storage grade Ethernet took out fiber channel. Probably shares a lot with it in some of the lower layers.
241 days ago [-]
241 days ago [-]
anonymousDan 241 days ago [-]
So how long before hardware is available that supports this spec? Would the kit likely be cheaper than infiniband (or even ROCE) equivalents?
bgnn 241 days ago [-]
It seems there isn't much difference on physical layer with standard 802.3 100Gb/lane. So in principle this is a software addition. Hardware supporting it should be available quickly.
I'm not familiar with infiniband, so can't comment on the pricing.
throw0101d 241 days ago [-]
While currently it seems like the only (?) InfiniBand vendor is Nvida/Mellanox, there used to be more folks selling the gear. For example Intel used to sell switches (had some for an Isilon backend):
> Intel's Omni-Path was very similar but a little different from Infiniband.
While Intel may have had the 'IB-adjacent' Omni-Path, they also had plain-old IB as well. I ran these/similar switches for an Isilon back-end network (starting when Isilon was still an independent company, pre-EMC buyout, pre-Dell buyout, in the OneFS 6.x days):
Intel bought Qlogic's IB division when they sold it off, in 2012; I believe at one point the former QLogic parts were branded Omni-Path before it diverged. (You can still see the DNA of that in how the Omni-Path driver stack is a relative of the Qlogic drivers, last I knew.)
Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space.
So at this point, I believe NVIDIA's Mellanox devices are the only people selling IB chips these days, and glancing at the TOP500 from June, seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.
throw0101d 240 days ago [-]
> […] seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.
I've done HPC for the last little while, but mostly Eth-based. Lately I have done a smattering more of IB, and (AFIACT) the main advantage IB gets you is lower latency (both 'innately' at the network layer, and with more 'out-of-box' RDMA).
rincebrain 238 days ago [-]
One reason we preferred Ethernet, when I last cared an age ago, was that we had a longstanding investment in building a lot of Windows-based bespoke software, and Infiniband support on Windows might now be up to merely 'iffy' instead of "requires a cold boot after you try using RDMA". So after we tried using IB once and got burned by being promised the software would be there Soon(tm), we went with then-10GbE for the portion we managed.
Infiniband is, if not purpose-built for HPC, at least more specifically suited for it. Ethernet mostly gets to be usable for HPC setups, IIRC, by implementing IB verbs over Ethernet and some Ethernet extensions that let it pretend _really_ hard that it's a guaranteed delivery transport.
But yes, I think IB's main advantage these days is reliably lower latency (which some kinds of HPC workloads certainly care more about). I believe you also used to get better performance on large switch fabrics when you started saturating significant fractions of the links, but I don't know that that's true any more, since some markets of Ethernet switch started caring about that and listing it as a bullet point.
241 days ago [-]
lousken 241 days ago [-]
This is cool, but I am more curious about what happens to all the cat 6a cabling in offices, where do we go after 10GBASE-T
crote 241 days ago [-]
Desktop computer are barely on 2.5GBASE-T, and there's still 5G in-between. I reckon it'll be 5-10 years until we'll see widespread 10G rollout, and it'll easily last another 10 before we need something better. The bigger issue is wireless access points. High-end ones are already on 10G today, and unlike desktops PoE is quite critical here.
25GBASE-T and 40GBASE-T are dead-on-arrival: the spec exists, but nobody ever bothered to actually ship it. It requires yet another cable upgrade and the maximum distance is too short to be usable. And that's before we even looked at power consumption.
The obvious answer is fiber, but that's orders of magnitude more complicated to roll out. The fiber itself is a massive pain to install when you want to make it to-length on site, the currently-popular connectors require a lot of babysitting, it can't be backwards compatible with the old RJ45 stuff, and it can't do PoE. Pulling a standard single-mode fiber pair with LC connectors to every cubicle and access point? Not exactly an attractive option.
I personally think we're probably going to see some kind of Frankensteined (think GG45-like) RJ45-with-fiber pop up, but that won't be any time soon.
phonon 240 days ago [-]
The Realtek RTL8127 chip for 10GbE will cost $10, draw 1.95 W and is designed for motherboards. So I think you're a bit pessimistic on timelines...
Change is coming! I feel like 5Gbit is mostly useful for switches, that a lot of other products are gonna jump direct to 10GBase-T.
25GBase-T or 40GBase-T is probably going to be a bit more energy intensive for a while, and 30m is ok but not great range if you're trying to span an office.
Ideally I'd really like to see optics costs fall, for consumers to start picking it up. Maybe the future is SFP everywhere and perhaps that makes sense (swap out transciever if one fails! Direct Attach Copper maybe or Base-T for convenience). But I really hope we see some real inroads, some of this separation between consumer & commercial go away.
I got a 150' 40Gbps fiber optic display port cable for ~$50, over half a decade ago. Unidirectional (I believe) but still just totally reset my expectations of what stuff really has to cost! Seemed too good to be true but still using it! We're spending more and more energy and effort trying to squeeze bits down a difficult copper channel, but optics right there, getting easier and easier.
I wish Intel hadn't given up on on-package interconnects (available on package on Xeon & Phi, 2016-2019). High end at the time. But now PCIe, USB, and DisplayPort all have optical capable specs, and the energy cost and distance of copper keeps making less and less sense. We gotta go forward, and the break from scant tiny consumer bandwidth to something much different feels like a potential dam burst.
10GBASE-T dates back to 2006, and we've been seeing it on the odd prosumer workstation / elite gamer motherboard for quite a while now. But those markets aren't going to lead to a full office rewiring.
The entry-level market is more interesting: when will we see it on motherboards that retail for $100? When will we see it on the average Dell Optiplex? When will your ISP router come with 10G ports?
Will some people be using 10G before that? Definitely, and I'll be one of them. Will it be something to consider for the average office environment? Nah.
lousken 234 days ago [-]
as someone with a homelab i disagree, i dont want to throw away my cat 6a cabling here, however i would like to see a bit more than i currently have with my 10Gbps, so maybe a middleground?
I would assume that max distance wouldn't be 100meters, but even if 30, that's more than plenty for lots of use cases
And as you mentioned, poe devices will be a huge pain that will have to be resolved
BizarroLand 241 days ago [-]
10g is going to remain sufficient for office use for a long time.
Ultra Ethernet will be primarily for cluster/AI/Super computing, trading, server, and backplane scenarios where high speed low-latency throughput turns into money or bang for your buck.
Debbie in accounting will practically never need more than 10g networking to do her job.
crote 241 days ago [-]
> Debbie in accounting will practically never need more than 10g networking to do her job.
And 640K of RAM ought to be enough for anyone. Who knows, perhaps in 2050 Edward will be mailing Debbie iterations of their 500GB AI fraud model for local inference?
The question isn't if 10G will need to be replaced with something better, but when.
BizarroLand 241 days ago [-]
Unless there are sea changes in the global infrastructure of computing in the next 25 years, that 500gb AI fraud model would be hosted on a server and made available to debbie through a web connection.
Further, almost every person on an internal network still uses 1g, which has been going strong for 25 years so far and will likely only cease being the standard when it's no longer economical for manufacturers to produce 1g hardware.
bgnn 241 days ago [-]
There's already 25G BASE-T over CAT6 available. I worked on and taped out a 25G PHY in 2019. Cable reach is 30m instead of 100m though. There's work going on 40G for 10m reach.
MrMorden 240 days ago [-]
If by "available" you mean that there's a standard. There may be PHY designs that exist but they're not available for purchase, probably because there's no identifiable market. Any potential customer who for some reason wanted to build an AP1000 reactor unit for their datacenter and a second unit for that datacenter's PHYs would need to design their own networking equipment. Which might use 40G but at this point it's questionable — that's the sort of slow equipment you might see in a rack-scale on-premises installation.
bgnn 237 days ago [-]
The engineering samples are available and if a network equipment manufacturer or a hyperscaler chooses to use it, it can be in production quite fast.
It's a bad idea though. This is the reason it's not adopted in any product.
datadrivenangel 241 days ago [-]
Ethernet but for AI!
jpgvm 241 days ago [-]
UltraEthernet predates the AI boom. I don't blame them for adding the AI buzzwords, for this to succeed people need to think about it in the same vein as IB/RoCE/etc and a lot of the decision makers are unfortunately not technical enough to understand what this is or does.
Ultimately it's just a different (IMO slightly more refined) look at how to support RDMA on Ethernet vs RoCE which is a more ham fisted implementation.
RoCE took an encapsulation approach that has some drawbacks (namely it's reliance on PFC/ECN for congestion management).
This takes a different approach that attempts to actually do first-class re-implementations of Infiniband-ish congestion control with end-to-end credit based flow control similar to Infiniband virtual lanes.
The writing was on the wall by 2015-2017, earlier still if you're the smart/attentive type.
jpgvm 239 days ago [-]
Yeah I may have underestimated how recent the formation of UEC was but IMO the efforts started around the period you are indicating with a lot of the libfabric guys proposing stuff like this.
241 days ago [-]
bobmcnamara 241 days ago [-]
Æthernet!
241 days ago [-]
dist-epoch 241 days ago [-]
Ethernet + AI
JHONWALTER 241 days ago [-]
[dead]
Rendered at 11:13:30 GMT+0000 (Coordinated Universal Time) with Vercel.
https://ultraethernet.org/ultra-ethernet-consortium-uec-laun...
Their noteworthy features list is:
> Modern RDMA for Ethernet and IP – Supporting intelligent, low-latency transport for high-throughput environments.
> Open Standards and Interoperability – Avoids vendor lock-in while accelerating ecosystem-wide innovation.
> End-to-End Scalability – From routing and provisioning to operations and testing, UEC scales to millions of endpoints.
I'm interested to see what the optional hardware features are. And what their relationship is to the different UE profiles (AI Base, AI Full, and HPC).
> The Ultra Ethernet Transport (UET) layer is designed to handle the most challenging application scale, deliver packets reliably and securely, manage and avoid congestion within the network, and react to contention at the endpoints. Its goals are minimal tail latency and highest network utilization. At the same time, UET is designed to enable simple hardware and software implementations – such as what might be required for accelerator-integrated endpoints. UET can be programmed through the OFI libfabric standard interface. It sets out to address the shortcomings of RoCEv2, specifically its semantics, transport layer, wire operations, implementation complexities, and scale limits
I'm sure there's a lot more nuanced to it all. But I think predictability/utilization/latency in RoCE are worse, that it relies on Explicit Congestion Notification more for flow control. Where-as UE is using Infiniband style credit based flow control, which should insure that any data sent has sufficient throughout allocated to it to be received.
UE seems to be a more direct creation of an Infiniband like network, atop Ethernet but where all players are agreeing to behave in an Infiniband like way with Infiniband predictability, where-as RoCE encapsulates Infiniband data but still behaves more like an Ethernet network lacking the coordination of Infiniband. I'm far from certain; it'd be so fun to have some extensive material to go over to really find out.
I'm not familiar with infiniband, so can't comment on the pricing.
* https://www.intel.com/content/dam/support/us/en/documents/ne... (PDF)
Officially there are a bunch of folks in the IB alliance:
* https://www.infinibandta.org/member-listing/
Omni-Path was sold off to Cornelis Networks in 2020. Cornelis just released new 400Gbps products last week, their CN5000 line, with 48 and 576 port switches. https://www.crn.com/news/components-peripherals/intel-spins-...
While Intel may have had the 'IB-adjacent' Omni-Path, they also had plain-old IB as well. I ran these/similar switches for an Isilon back-end network (starting when Isilon was still an independent company, pre-EMC buyout, pre-Dell buyout, in the OneFS 6.x days):
* https://www.dell.com/support/kbdoc/en-us/000019215/install-a...
Before that, Mellanox ate Voltaire, who was the other large vendor in the IB space.
So at this point, I believe NVIDIA's Mellanox devices are the only people selling IB chips these days, and glancing at the TOP500 from June, seems like a good half (45.8%) of the supercomputers listed there are using either Ethernet-based or otherwise non-IB stacks.
I've done HPC for the last little while, but mostly Eth-based. Lately I have done a smattering more of IB, and (AFIACT) the main advantage IB gets you is lower latency (both 'innately' at the network layer, and with more 'out-of-box' RDMA).
Infiniband is, if not purpose-built for HPC, at least more specifically suited for it. Ethernet mostly gets to be usable for HPC setups, IIRC, by implementing IB verbs over Ethernet and some Ethernet extensions that let it pretend _really_ hard that it's a guaranteed delivery transport.
But yes, I think IB's main advantage these days is reliably lower latency (which some kinds of HPC workloads certainly care more about). I believe you also used to get better performance on large switch fabrics when you started saturating significant fractions of the links, but I don't know that that's true any more, since some markets of Ethernet switch started caring about that and listing it as a bullet point.
25GBASE-T and 40GBASE-T are dead-on-arrival: the spec exists, but nobody ever bothered to actually ship it. It requires yet another cable upgrade and the maximum distance is too short to be usable. And that's before we even looked at power consumption.
The obvious answer is fiber, but that's orders of magnitude more complicated to roll out. The fiber itself is a massive pain to install when you want to make it to-length on site, the currently-popular connectors require a lot of babysitting, it can't be backwards compatible with the old RJ45 stuff, and it can't do PoE. Pulling a standard single-mode fiber pair with LC connectors to every cubicle and access point? Not exactly an attractive option.
I personally think we're probably going to see some kind of Frankensteined (think GG45-like) RJ45-with-fiber pop up, but that won't be any time soon.
https://www.tomshardware.com/networking/realteks-usd10-tiny-...
Change is coming! I feel like 5Gbit is mostly useful for switches, that a lot of other products are gonna jump direct to 10GBase-T.
25GBase-T or 40GBase-T is probably going to be a bit more energy intensive for a while, and 30m is ok but not great range if you're trying to span an office.
Ideally I'd really like to see optics costs fall, for consumers to start picking it up. Maybe the future is SFP everywhere and perhaps that makes sense (swap out transciever if one fails! Direct Attach Copper maybe or Base-T for convenience). But I really hope we see some real inroads, some of this separation between consumer & commercial go away.
I got a 150' 40Gbps fiber optic display port cable for ~$50, over half a decade ago. Unidirectional (I believe) but still just totally reset my expectations of what stuff really has to cost! Seemed too good to be true but still using it! We're spending more and more energy and effort trying to squeeze bits down a difficult copper channel, but optics right there, getting easier and easier.
I wish Intel hadn't given up on on-package interconnects (available on package on Xeon & Phi, 2016-2019). High end at the time. But now PCIe, USB, and DisplayPort all have optical capable specs, and the energy cost and distance of copper keeps making less and less sense. We gotta go forward, and the break from scant tiny consumer bandwidth to something much different feels like a potential dam burst.
I hope Intel's still doing some optical stuff with the recent shake ups. Loved their embarrassingly parallel RISC-V beast with 16 optical links. And it was a year ago they demoed a new all optical switch (rip Intel/Barefoot Tofino, you wonderful p4 programmable beast) with copious 4Tb/s PCIe switching. https://www.theregister.com/2023/09/01/intel_graph_analytics... https://newsroom.intel.com/artificial-intelligence/intel-unv...
10GBASE-T dates back to 2006, and we've been seeing it on the odd prosumer workstation / elite gamer motherboard for quite a while now. But those markets aren't going to lead to a full office rewiring.
The entry-level market is more interesting: when will we see it on motherboards that retail for $100? When will we see it on the average Dell Optiplex? When will your ISP router come with 10G ports?
Will some people be using 10G before that? Definitely, and I'll be one of them. Will it be something to consider for the average office environment? Nah.
I would assume that max distance wouldn't be 100meters, but even if 30, that's more than plenty for lots of use cases
And as you mentioned, poe devices will be a huge pain that will have to be resolved
Ultra Ethernet will be primarily for cluster/AI/Super computing, trading, server, and backplane scenarios where high speed low-latency throughput turns into money or bang for your buck.
Debbie in accounting will practically never need more than 10g networking to do her job.
And 640K of RAM ought to be enough for anyone. Who knows, perhaps in 2050 Edward will be mailing Debbie iterations of their 500GB AI fraud model for local inference?
The question isn't if 10G will need to be replaced with something better, but when.
Further, almost every person on an internal network still uses 1g, which has been going strong for 25 years so far and will likely only cease being the standard when it's no longer economical for manufacturers to produce 1g hardware.
It's a bad idea though. This is the reason it's not adopted in any product.
Ultimately it's just a different (IMO slightly more refined) look at how to support RDMA on Ethernet vs RoCE which is a more ham fisted implementation.
RoCE took an encapsulation approach that has some drawbacks (namely it's reliance on PFC/ECN for congestion management).
This takes a different approach that attempts to actually do first-class re-implementations of Infiniband-ish congestion control with end-to-end credit based flow control similar to Infiniband virtual lanes.
UEC was formed/announced in July 2023:
* https://ultraethernet.org/leading-cloud-service-semiconducto...
ChatGPT was launched in November 2022:
* https://en.wikipedia.org/wiki/ChatGPT
* https://en.wikipedia.org/wiki/GPT-4