NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
▲'Duffy Lied': Controllers Refute Transportation Secretary's Ground Stop Story (commondreams.org)
alexpotato 3 hours ago [-]
Long time SRE/DevOps with 20 year of FinTech experience in banks/hedge funds/crypto and having seen many, many outages (some listed here [0]):

- There is almost always a standard operating procedure (SOP)

- Just b/c there is a SOP for releases etc, that doesn't mean that people always follow it

- This can be due to many reasons e.g. laziness, being distracted, pressure from higher ups ("just push it out there and I'll take the risk" ha!), tired from having a newborn etc etc

- Doing things outside the "green zone" deployment windows therefore always carries risk

- All of this being said, EVEN IF you ARE following SOP, things still break in all kinds of weird ways

- And the end user is not often the best observer nor do they have all of the context to determine what broke, how it broke and why. (There are exceptions to this though)

0 - https://x.com/alexpotato/status/1215876962809339904

fooqux 1 hours ago [-]
> banks/hedge funds/crypto

Most of us here at HN work (or have worked) at a place where if you screw up a SOP the company could lose lots of money. So we're careful.

That's different than working at a place where if you screw up, people die. It's a totally different environment (or at least it used to be, can't say what happened over the last ten years). There's a lot more 9's, and a lot more consideration of making changes during off periods just in case. I don't believe this was planned for that reason alone.

mcswell 1 hours ago [-]
"just push it out there and I'll take the risk": https://www.youtube.com/watch?v=U2qgTrqBS_I

(and SkyNet no less)

UltraSane 2 hours ago [-]
Duffy has been very loud about doing upgrades faster so this kind of result is predictable.
WaitWaitWha 2 hours ago [-]
yep. most IR peeps have plenty of stories where a single event is just a symptom of multiple failures, unfortunate accidents, and alike. most red teams will also regale chained attacks is where it is today. it is rarely a single piece of lock that breaks, it needs several other things to be in the "wrong" state.

Your third bullet is a bane of gov contractors. Not the sales people, but the people actually doing the work. Imagine if you had an FFP contract and you finished the job within 90% of the paid time. At $1mm that is sweet $100K that the big cheese keeps. (not you, you go back to the mines and since you demoed you can do it at 90%, next time you have to do it 80% time. and, no i am not sore ;) )

tl;dr My first reaction to Duffy's statement was "you have no clear idea yet, but this is your best guess". And, we may never know.

jml7c5 3 hours ago [-]
Asking air traffic controllers for their opinion on this is a bit odd — would they even know about networking equipment swaps? I would expect that to be handled by the engineers and technicians who work on the system, not the people who direct the planes.
jm4 2 hours ago [-]
They wouldn't necessarily know the specifics of upgrades, but they would likely know when upgrades typically happen because it's normal to send out emails about maintenance windows.
sippingabonedry 1 hours ago [-]
The quote is the headline isn't even from controllers, it's a troll post on BlueSky.
ratelimitsteve 3 hours ago [-]
they might know about how the last 10 equipment swaps went and why this one is different.
llm_nerd 2 hours ago [-]
I imagine equipment is regularly updated and upgraded without them knowing.

I'm extremely skeptical of this administration's competence, but this story is 100% finding anyone saying what you want to hear, however unreliable the narration might be.

Jtsummers 1 hours ago [-]
> I imagine equipment is regularly updated and upgraded without them knowing.

Part of the reason for this outage was that these systems are not regularly upgraded. That's why they're dealing with decades old network hardware and lines and marrying them up to modern fiber. It's expensive to do the upgrades, but instead of doing regular rolling upgrades to their network components they generally do big-bang upgrades like the one happening now (that apparently caused the outage). In 20-30 years there will be another big-bang upgrade replacing the Juniper routers and fiber lines they've been installing now, and it'll probably cause similar problems.

This also happens a lot in DOD (not surprising, same contractors doing the work and probably the same companies in the program offices for FAA as DOD giving the same bad advice), it's incredibly frustrating because it's very obvious that there are better ways to do things, but the funding isn't there until it's urgent. That's why they're still using 1980s computers with floppy disks in their critical path for ATC as well. Now that those systems are nearly impossible to maintain, there is enough noise to push funding a proper replacement effort, about 20-30 years after it should have been done.

esseph 2 hours ago [-]
> I imagine equipment is regularly updated and upgraded without them knowing

I highly doubt this.

FedGov loves some ITIL, even for the contractors. A change of this magnitude would have gone through CAB with stakeholder sign-off.

dylan604 1 hours ago [-]
> I imagine equipment is regularly updated and upgraded without them knowing.

It's a good thing you imagined and did not assume. Equipment regularly updated? You mean like the floppies they are still using?

xvxvx 2 hours ago [-]
Unless I’m just tired, the article, reporter and air traffic controllers are confused about hardware switches and routers having issues, and the act of switching out hardware. Duffy stated that the switches and routers were having issues and needed to be shut down. This seems to have been read as physical hardware switches, read ‘replacements’, were happening during the day.
aperrien 2 hours ago [-]
It's not just you. It reads like a confused mess to mee too.
sippingabonedry 2 hours ago [-]
TFA is a self-proclaimed progressive opinion site.

Its purpose is ragebait not accurate news reporting.

They may as well have linked to graffiti scrawled above a gloryhole.

The quote in the title is not from controllers as implied, but from a BlueSky post of a well-known social media provocateur.

sega_sai 2 hours ago [-]
I don't know what happened, but for me a red flag was the specific explanation given while the event was still ongoing: "overloaded router", followed by request for more money. If the cause was so simple , then it was unclear why it wasn't fixed immediately, so something does not quite add up (at least till an independent investigation)
cmurf 46 minutes ago [-]
Duffy is asserting everyone in this administration is permanently unaccountable and irresponsible.

https://bsky.app/profile/atrupar.com/post/3n2ebpzqe3s27

noefingway 2 hours ago [-]
I don't believe anything that Sean Duffy or his ilk say. I would hope there is a detailed and accurate post-mortem published to explain exactly what happened. However, IMHO this is just another example of the destruction wrought by Trump/Musk/DOGE. I recommend reading this article by James Fallows: https://web.archive.org/web/20261114090809/https://www.nytim...
dylan604 1 hours ago [-]
Why would you hope that? Who would be releasing said post-mortem? Would it not be from the "or his ilk" portion of the comment? Sure, some government employee or team might produce an honest and factual report, but there's no way this administration would release it. The only way for that to happen would be for it to be leaked
sheikhnbake 29 minutes ago [-]
> The only way for that to happen would be for it to be leaked

Which is very plausible considering the number of high level leaks and whistleblowing that's happened just in this term alone

schiffern 3 hours ago [-]
"Refute" makes it sound like there's juicy insider info, but it's just skepticism that FAA scheduled upgrades on a Friday daytime.

This was a cascading multicast failure, which can take hours to build up. Earliest reports of disruptions came no later than 10:30 am, so it's quite possible the upgrades were performed in the overnight hours on Thursday.

LorenPechtel 3 hours ago [-]
Very good point on the timing. You do stuff like that at minimum load, not at maximum.
nobodyandproud 2 hours ago [-]
Duffy is incompetent. This administration is incompetent.

I’ve more scared to fly since 2025 when they started cutting deep, than the months and the year after 9/11.

jmclnx 3 hours ago [-]
Tin foil hat on: Microsoft Cloud issues ? aka Office 365 over the same period :)
api 3 hours ago [-]
That's just a regular hat. No foil needed.

Even if MS stuff was not in any critical path it still could have played havoc with coordination of an update or a change, contributing to the failure.

Also FYI foil hats are actually resonant cavities and amplify the mind control waves. All true paranoids use Faraday mesh, preferably as a full head covering.

procarch2019 2 hours ago [-]
I saw a post…I think on HN…the other day. It was a technical post about making a faraday cage out of a metal trash can. Afterwards I thought to myself let me check Amazon to see how saturated the faraday phone bag market is. Lo and behold I find out there’s faraday bags for all sorts of electronics now and head meshes. I’d be curious how well any of them work, but I wasn’t expecting to see it.

I guess it makes a little sense given a lot of electronics don’t have a removable battery or physical power switch nowadays.

derektank 2 hours ago [-]
I’m not sure which version of Microsoft the FAA uses, but GCC High and the DoD version of Microsoft 365 have been working just fine through the commercial outage of Microsoft 365
unpleasurable 2 hours ago [-]
Don't forget the foramen magnum!
ferrouswheel 3 hours ago [-]
[flagged]
rc5150 2 hours ago [-]
[flagged]
collingreen 2 hours ago [-]
Your bar for criminal behavior is really low!
tocs3 1 hours ago [-]
The ones holding the bar up are getting tired.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 19:25:39 GMT+0000 (Coordinated Universal Time) with Vercel.