Why would you call colocation "building your own data center"? You could call it "colocation" or "renting space in a data center". What are you building? You're racking. Can you say what you mean?
macintux 10 minutes ago [-]
Dealing with power at that scale, arranging your own ISPs, seems a bit beyond your normal colocation project, but I haven’t bee in the data center space in a very long time.
0xbadcafebee 4 minutes ago [-]
[delayed]
sitkack 6 minutes ago [-]
It would be nice to have a lot more detail. The WTF sections are the best part. Sounds like your gear needs "this side towards enemy" sign and/or the right affordances so it only goes in one way.
Did you standardize on layout at the rack level? What poke-yoke processes did you put into place to prevent mistakes?
What does your metal->boot stack look like?
Having worked for two different cloud providers and built my own internal clouds with PXE booted hosts, I find too find this stuff fascinating.
Also take utmost advantage of a new DC when you are booting it to try out all the failure scenarios you can think of and the ones you can't through randomized fault injection.
coolkil 9 minutes ago [-]
Awesome!! Hope to see more companies go this route. I had the pleasure to do something similar for a company(lot smaller scale though)
It was my first job out of university. I will never forget the awesome experience of walking into the datacenter and start plugging cables and stuff
j-b 18 minutes ago [-]
Love these kinds of posts. Tried railway for the first time a few days ago. It was a delightful experience. Great work!
__fst__ 24 minutes ago [-]
Can anyone recommend some engineering reading for building and running DC infrastructure?
ca508 13 minutes ago [-]
We didn't find many good up-to-date resources online on the hardware side of things - kinda why we wanted to write about it. The networking aspect was the most mystical - I highly recommend "BGP in the datacenter" by Dinesh Dutt on that (I think it's available for free via NVidia). Our design is heavily influenced by the ideas discussed there.
dban 2 hours ago [-]
This is our first post about building out data centers. If you have any questions, we're happy to answer them here :)
gschier 1 hours ago [-]
How do you deal with drive failures? How often does a Railway team member need to visit a DC? What's it like inside?
justjake 51 minutes ago [-]
Everything is dual redundancy. We run RAID so if a drive fails it's fine; alerting will page oncall which will trigger remote hands onsite, where we have spares for everything in each datacenter
gschier 47 minutes ago [-]
How much additional overhead is there for managing the bare-metal vs cloud? Is it mostly fine after the big effort for initial setup?
ca508 25 minutes ago [-]
We built some internal tooling to help manage the hosts. Once a host is onboarded onto it, it's a few button clicks on an internal dashboard to provision a QEMU VM. We made a custom ansible inventory plugin so we can manage these VMs the same as we do machines on GCP.
The host runs a custom daemon that programs FRR (an OSS routing stack), so that it advertises addresses assigned to a VM to the rest of the cluster via BGP. So zero config of network switches, etc... required after initial setup.
We'll blog about this system at some point in the coming months.
nextworddev 36 minutes ago [-]
First time checking out railway product- it seems like a “low code” and visual way to define and operate infrastructure?
Like, if Terraform had a nice UI?
justjake 27 minutes ago [-]
Kinda. It's like if you had everything from an infra stack but didn't need to manage it (Kubernetes for resilience, Argo for rollouts, Terraform for safely evolving infrastructure, DataDog for observability)
If you've heard of serverless, this is one step farther; infraless
Give us your code, we will spin it up, keep it up, automate rollouts service discovery, cluster scaling, monitoring, etc
exabrial 52 minutes ago [-]
I'm surprised you guys are building new!
Tons of Colocation available nearly everywhere in the US, and in the KCMO area, there are even a few dark datacenters available for sale!
cool project none-the-less. Bit jealous actually :P
justjake 48 minutes ago [-]
The requirements end up being pretty specific, based on workloads/power draw/supply chain
So, while we could have bought something off the shelf, that would have been suboptimal from a specs perspective. Plus then we'd have to source supply chain etc.
By owning not just the servers but the whole supply chain, we have redundancy at every layer, from the machine, to the parts on site (for failures), to the supply chain (refilling those spare parts/expanding capacity/etc)
CMCDragonkai 38 minutes ago [-]
Can you share a list of dark datacenters that are for sale. They sound interesting as a business.
gschier 51 minutes ago [-]
More info on the cost comparison between all the options would be interesting
dban 38 minutes ago [-]
We pulled some cost stuff out of the post in final review because we weren't sure it was interesting ... we'll bring it back for a future post
ramon156 37 minutes ago [-]
weird to think my final internship was running on one of these things. thanks for all the free minutes! it was a nice experience
Rendered at 22:46:45 GMT+0000 (Coordinated Universal Time) with Vercel.
Did you standardize on layout at the rack level? What poke-yoke processes did you put into place to prevent mistakes?
What does your metal->boot stack look like?
Having worked for two different cloud providers and built my own internal clouds with PXE booted hosts, I find too find this stuff fascinating.
Also take utmost advantage of a new DC when you are booting it to try out all the failure scenarios you can think of and the ones you can't through randomized fault injection.
It was my first job out of university. I will never forget the awesome experience of walking into the datacenter and start plugging cables and stuff
The host runs a custom daemon that programs FRR (an OSS routing stack), so that it advertises addresses assigned to a VM to the rest of the cluster via BGP. So zero config of network switches, etc... required after initial setup.
We'll blog about this system at some point in the coming months.
Like, if Terraform had a nice UI?
If you've heard of serverless, this is one step farther; infraless
Give us your code, we will spin it up, keep it up, automate rollouts service discovery, cluster scaling, monitoring, etc
Tons of Colocation available nearly everywhere in the US, and in the KCMO area, there are even a few dark datacenters available for sale!
cool project none-the-less. Bit jealous actually :P
So, while we could have bought something off the shelf, that would have been suboptimal from a specs perspective. Plus then we'd have to source supply chain etc.
By owning not just the servers but the whole supply chain, we have redundancy at every layer, from the machine, to the parts on site (for failures), to the supply chain (refilling those spare parts/expanding capacity/etc)