In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun.
It can be a pain to setup a break-glass, especially if you have a lot of legacy CI cruft to deal with. But it pays off in spades during outages.
I'm biased because we (Dagger) provide tooling that makes this break-glass setup easier, by decoupling the CI logic from CI infrastructure. But it doesn't matter what tools you use: just make sure you can run a bootstrap CI pipeline from your local machine. You'll thank me later.
cpfohl 33 minutes ago [-]
I swear this is my fault. I can go weeks without doing infra work. Github does fine, I don't see any hiccups, status page is all green.
But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....
I would so very much love to see GitHub switch gears from building stuff like Copilot etc and focus on availability
overshard 28 minutes ago [-]
I've taken to hosting everything critical like this myself on a single system with Docker Compose with regular off premises backups and a restore process that I know works because I test it every 6 months. I can swap from local hosting to a VPS in 30 mins if I need to. It seems like the majority of large services like GitHub have had increasingly annoying downtime while I try to get work done. If you know what you're doing it's a false premise that you'll just have more issues with self hosting. If you don't know what you are doing it's becoming an increasingly good time to learn. I've had 4 years of continuous uptime on my services at this point. I still push to third parties like GitHub as yet another backup and see the occasional 500 and my workflow keeps chugging along. I've gotten old and grumpy and rather just do it myself.
granzymes 4 minutes ago [-]
I have a bug bash in an hour and fixes that need to go in beforehand. So of course GitHub is down.
pothamk 25 minutes ago [-]
What’s interesting about outages like this is how many things depend on GitHub now beyond just git hosting.
CI pipelines, package registries, release automation, deployment triggers, webhooks — a lot of infrastructure quietly assumes GitHub is always available.
When GitHub degrades, the blast radius is surprisingly large because it breaks entire build and release chains, not just repo browsing.
littlestymaar 21 minutes ago [-]
> a lot of infrastructure quietly assumes GitHub is always available
Which is really baffling when talking about a service that has at least weekly hicups even when it's not a complete outage.
Part of it is probably historical momentum.
GitHub started as “just git hosting,” so a lot of tooling gradually grew around it over the years — Actions, package registries, webhooks, release automation, etc. Once teams start wiring all those pieces together, replacing or decoupling them becomes surprisingly hard, even if everyone knows it’s a single point of failure.
Imustaskforhelp 23 seconds ago [-]
Are we serious?
akoumjian 33 minutes ago [-]
Is this related to Cloudflare?
I'm getting cf-mitigated: challenge on openai API requests.
codeberg might be a little slower on git cli, but at least it's not becoming a weekly 'URL returned error: 500' situation...
popcornricecake 28 minutes ago [-]
These days it feels like people have simply forgotten that you could also just have a bare repository on a VPS and use it over ssh.
yoyohello13 15 minutes ago [-]
Most developers don’t even know git and GitHub are different things…
mynameisvlad 33 minutes ago [-]
I mean, this isn't a 'URL returned error: 500' situation for anything that Codeberg provides considering this is an issue with Copilot and Actions.
joecool1029 30 minutes ago [-]
Except actually it was, that was what my git client was reporting trying to run a pull.
mynameisvlad 24 minutes ago [-]
I'm going to trust the constant stream of updates from the company itself which shows exactly what went down and came back up rather than a random anecdote.
ocdtrekkie 35 minutes ago [-]
I rarely successfully get Codeberg URLs to load. Which is sad because I actually would very much like to recommend it but I find it unreliable as a source.
That being said, GitHub is Microsoft now, known for that Microsoft 360 uptime.
cyberax 26 minutes ago [-]
> Microsoft 360 uptime
I mean... It's right in the name! It's up for 360 days a year.
IshKebab 32 minutes ago [-]
I mean... you understand the scale difference right?
garciasn 32 minutes ago [-]
How reliable is githubstatus.com? I know that status pages are generally not updated until Leadership and/or PR has a chance to approve the changes; is that the case here?
Our health check checks against githubstatus.com to verify 'why' there may be a GHA failure and reports it, e.g.
Cannot run: repo clone failed — GitHub is reporting issues (Partial System Outage: 'Incident with Copilot and Actions'). No cached manifests available.
But, if it's not updated, we get more generic responses. Are there better ways that you all employ (other than to not use GHA, you silly haters :-))
duckkg5 28 minutes ago [-]
Right now the page says Copilot and Actions are affected but I can't even push anything to a repo from the CLI.
alemanek 17 minutes ago [-]
Yep getting 500 errors intermittently on fetch and checkout operations in my CI pretty consistently at the moment. Like 1 in 2 attempts
jjice 24 minutes ago [-]
Agreed. I believe that's marked under "Git Operations" and it's all green. Just began being able to push again a minute ago.
joshrw 37 minutes ago [-]
Happening very often lately
risyachka 10 minutes ago [-]
and we all know why
m_w_ 20 minutes ago [-]
Seems like the xkcd [1] for internet infrastructure that was posted earlier [2] should have github somewhere on it, even if just for how often it breaks. Maybe it falls under "whatever microsoft is doing"
Most individual services have two nines... but not all of them.
modeless 11 minutes ago [-]
90 day non-degraded uptime of Github Actions is 98.8% if the official numbers can be believed
kibwen 11 minutes ago [-]
Github proudly boasts an industry-leading seven 9s of uptime. 49.999999%
banga 21 minutes ago [-]
Only on days with a "y"...
41 minutes ago [-]
littlestymaar 23 minutes ago [-]
In many companies I worked for, there were a bunch of infrastructure astronauts who made everything very complicated in the name of zero downtime and sold them to management as “downtime would kill pur credibility and our businesses ”, and then you have billion dollar companies everyone relies on (GitHub, Cloudflare) who have repeated downtime yet it doesn't seem to affect their business in any way.
Krutonium 20 minutes ago [-]
To be fair - it SUPER does. Being down frequently makes your competition look better.
Of course, once you have the momentum it doesn't matter nearly as much, at least for a while. If it happens too much though, people will start looking for alternatives.
The key to remember is Momentum is hard to redirect, but with enough force (reasons), it will.
baggy_trough 22 minutes ago [-]
The reality is that consumers don't really care about downtime unless it's truly frequent.
This is very worrying if their mandate doesn't include quality control.
khaledh 23 minutes ago [-]
I figured that it would be something like that. But it's been so frequent that I expect the leadership to act decisively towards a long-term reliability plan. Unfortunately they have near monopoly in this space, so I guess there's not enough incentive to fix the situation.
gobalini 6 minutes ago [-]
How frequent? I think the obsession with uptime is annoying. If GitHub is down, if there’s something so critical, then you need some more control of the system. Otherwise take a couple hours and get a coffee or an early lunch.
drcongo 21 minutes ago [-]
Does anything running on Azure have an acceptable uptime?
boxingdog 41 minutes ago [-]
[dead]
Rendered at 19:51:05 GMT+0000 (Coordinated Universal Time) with Vercel.
It can be a pain to setup a break-glass, especially if you have a lot of legacy CI cruft to deal with. But it pays off in spades during outages.
I'm biased because we (Dagger) provide tooling that makes this break-glass setup easier, by decoupling the CI logic from CI infrastructure. But it doesn't matter what tools you use: just make sure you can run a bootstrap CI pipeline from your local machine. You'll thank me later.
But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....
Which is really baffling when talking about a service that has at least weekly hicups even when it's not a complete outage.
There's almost 20 outages listed on HN over the past two months: https://news.ycombinator.com/from?site=githubstatus.com so much for “always available”.
I'm getting cf-mitigated: challenge on openai API requests.
https://www.cloudflarestatus.com/ https://status.openai.com/
That being said, GitHub is Microsoft now, known for that Microsoft 360 uptime.
I mean... It's right in the name! It's up for 360 days a year.
Our health check checks against githubstatus.com to verify 'why' there may be a GHA failure and reports it, e.g.
Cannot run: repo clone failed — GitHub is reporting issues (Partial System Outage: 'Incident with Copilot and Actions'). No cached manifests available.
But, if it's not updated, we get more generic responses. Are there better ways that you all employ (other than to not use GHA, you silly haters :-))
[1]: https://www.reddit.com/r/ProgrammerHumor/comments/1p204nx/ac... [2]: https://news.ycombinator.com/item?id=47230704
https://mrshu.github.io/github-statuses/
Most individual services have two nines... but not all of them.
Of course, once you have the momentum it doesn't matter nearly as much, at least for a while. If it happens too much though, people will start looking for alternatives.
The key to remember is Momentum is hard to redirect, but with enough force (reasons), it will.
And the frequency they can tolerate is surprisingly high given that we're talking about the 20th or so outage of 2026 for github. (See: https://news.ycombinator.com/from?site=githubstatus.com)
https://www.windowscentral.com/microsoft/using-ai-is-no-long...
https://thenewstack.io/github-will-prioritize-migrating-to-a...