"The hackers stated that they attempted to contact Red Hat with an extortion demand but received no response other than a templated reply instructing them to submit a vulnerability report to their security team."
Just hilarious
1970-01-01 1 days ago [-]
You didn't give the kicker:
"According to them, the created ticket was repeatedly assigned to additional people, including Red Hat's legal and security staff members."
Summarized: Given enough eyeballs, all extortion demands are fallow.
elicash 1 days ago [-]
And then there's more, via 404:
“Since RedHat doesn't want to answer to us,” the hackers wrote in a channel on Telegram viewed by 404 Media, suggesting they have attempted to contact Red Hat. [...]
“We have given them too much time already to answer lol instead of just starting a discussion they kept ignoring the emails,” the message added. In another message, the group said it had “gained access to some of their clients' infrastructure as well, already warned them but yeah they preferred ignoring us.”
First rule of having someone reply: spell their name correctly.
andmarios 1 days ago [-]
To be fair, once your data has been stolen, it doesn't make sense to engage with the hackers. There is no way to guarantee that the stolen data won't be used.
What you must do immediately is notify the affected customers, bring down or lock the affected services, and contact the authorities.
poemxo 1 days ago [-]
I'm a customer and the first I'm hearing about this is from HN.
eviks 16 hours ago [-]
There is no guarantee anywhere (strictly speaking, including in the legal market), but that doesn't mean the paying has no effect of the probability of data being dumped.
Notification is an independent requirement.
jasonjayr 1 days ago [-]
There is an interesting dynamic/risk in play:
If an attacker make an extortion threat, but then still follows through on the release/damage after being paid, then people are not incentivized to engage with you, and will go into attack mode right away, making it riskier for you.
HOWEVER, if the attacker make the extortion threat, takes payment, and then honors the agreement, and ends the transaction, then parties are more inclined to just pay to make the problem go away. They know that the upfront price is the full cost of the problem.
I've seen that there are 'ethical attackers' out there that move on after an attack, but you never know what kind you're dealing with :-/ "Never negotiate...."
Loudergood 1 days ago [-]
Then the hacker org spins up a new name(like a shitty construction llc) and robs the next guy.
Reputation isn't all that useful for extortion.
Running all your crimes as the "Wet Bandits" makes it much easier for law enforcement if they do catch up with you.
themafia 1 days ago [-]
There's no way to guarantee that I won't get in a car accident. So I pay for insurance. I may never need it, it may never come in handy, but it still makes sense to carry the policy.
nomilk 1 days ago [-]
fallow == marked by inactivity
Thanks, hadn't encountered this word before.
bombcar 1 days ago [-]
Normally used with farming; you run the land two years, and then leave it fallow for a year to recover.
jayd16 1 days ago [-]
Corpo cyberpunk
devonkim 24 hours ago [-]
This whole process happening is exactly what happens in a quest in Cyberpunk 2077. There’s an e-mail chain where a gang tried to extort a corporation and gave up after being unable to reach a person.
I sincerely hope that the game doesn’t become prophetic in the manner Idiocracy has.
behnamoh 1 days ago [-]
Can't be extorted if you can't be reached. such a two-brain move!
Before even reading the article, the title made me lol as I thought the open sourced code was stolen XD
INTPenis 1 days ago [-]
Red Hat, I am very disappointed. We're all ISO27001 everywhere, separation of data, and separation of network resources. But you keep our data in your github repo?
mmh0000 1 days ago [-]
You've got to look at ISO27001 from the perspective of the Sales Rep, not from an Engineer.
In theory, being ISO27001 means that you're environment follows best practices and has a somewhat sane security posture.
To the business people, a new customer demands that you have ISO27001 certification before they'll sign the $$$$ contract. The salesperson does not care HOW you get the certificate, just that you have it, they need this contract signed!
The department wasn't designed with security in mind, so implementing everything required by ISO will take many months. But sales needs $$$$ now! The CEO, CFO, and CTO are aligned: money now!
So, there's high pressure to pass the audit quickly. You implement what you can, you weasle your way around the things that will take too long. Those things are "out of scope" or "testing databases". You implement MFA while the auditor is auditing, but you know it breaks developers' workflows and there isn't a quick fix, so you turn MFA back off once the audit is complete....
TA-DA! We're ISO27001 certified! But we're no more secure than we were before.
Grikbdl 1 days ago [-]
> In theory, being ISO27001 means that you're environment follows best practices and has a somewhat sane security posture.
Nah, it just means you have defined, documented processes and document that you stick to them. They actual processes can be shit and maybe you also have something on the side the auditors don't get shown, but ultimately the certification is a total joke. Source: Worked at a place that got certified despite being a security joke.
johannes1234321 1 days ago [-]
> ultimately the certification is a total joke.
Yes and no. Even if it is a joke there is one thing it qualifies: You at least spent time looking at the process. This already is a gain over complete wild west.
1oooqooq 1 days ago [-]
that makes absolutely no sense at all.
do you mean you rather be lied to than not be lied to?
quicksilver03 18 hours ago [-]
That looks more like SOC2 than ISO-27001 though.
lima 15 hours ago [-]
It's the same with ISO27001. A bad actor can always weasel their way through.
throwaway127482 1 days ago [-]
Engineers who are smart enough / talented enough, and who feel secure, can push back on security issues even if it will hold up a deal. This tells me that the most valuable engineers at Red Hat either do not push back enough on security concerns, or don't care enough (or aren't experienced enough) to know that the concerns exist in the first place, or they feel insecure in their position.
array_key_first 1 days ago [-]
Ultimately, devs can't get sales reps fired, but sales reps can absolutely get devs fired.
Depending on how dysfunctional the org is, there's no super dev anywhere who can fix it. You just shut up, do bad things knowing theyre bad, or get fired.
anthk 1 days ago [-]
It should be the opposite. For every big engineering issue happened because of the sales' dept pressures, the sales reps would have their asses out of any company.
typpilol 1 days ago [-]
When I was working in MSP land this was the worst.
I had a sales guy sell a a company a replacement for their terminal server, with OneDrive lol
I almost died laughing when he explained to me the project.
I said.. you want to run cad files off OneDrive in place of a terminal/storage server?
"Yes"
Let's just say we ended up just moving their server to the cloud and VPN access onsite and for external developers.
latchkey 1 days ago [-]
100%! Insert SOC2, HIPAA, etc...
baobun 1 days ago [-]
> Correction: After publishing, Red Hat confirmed that it was a breach of one of its GitLab instances, and not GitHub. Title and story updated.
> After publishing our story, Red Hat confirmed that the security incident was a breach of its GitLab instance used solely for Red Hat Consulting on consulting engagements, and not GitHub.
> While Red Hat did not respond to any further questions about the breach, the hackers told BleepingComputer that the intrusion occurred approximately two weeks ago.
ExoticPearTree 1 days ago [-]
Data is separated in different repositories, per the story.
And, never forget: what a company preaches and advertises is not the same with what the company is actually doing.
zingababba 1 days ago [-]
Yes, I've been in fortune 100 IT security for awhile now. When it comes to passing audits its a shit ton of misdirection.
Every time I've been involved in audits at a company, my boss will tell me "let me tell you how to talk to auditors," which ends up meaning lie by omission, imply that things are in good standing without making strictly false statements, and otherwise just make the auditors go away. It all seems silly, but maybe it should be thought of like the court system? An adversarial process whereby each side is vying for its own interests?
ExoticPearTree 1 days ago [-]
With auditors you talk like you would either talk on a deposition or on the witness stand: do not say more than what you were asked, do not make assumptions, do not try to be helpful in any way, do not offer more data than asked for.
Is it really OK? Not necessarily, but on the other hand you don't want to spend the rest of your life answering even more questions from other people the auditors might bring in to help them understand your helpful explanations.
I learned this the hard way, assuming auditors are logical and understand technology.
Spooky23 1 days ago [-]
That’s a good point of view.
90% of the time, they are checking boxes. But if they are fishing, you have to be careful because they generally are bad at understanding anything, but good at manipulating the audit rules to frame things in such a way so they can “catch a big fish”.
dijit 1 days ago [-]
The issue is, it’s very easy to understand what’s not being said for a reasonably intelligent person.
A person who is used to interviewing people will be able to tell right away.
baobun 1 days ago [-]
s/should/could/
Your boss is bad apple and so are you if you adopt their ways.
behnamoh 1 days ago [-]
if you find it unethical, can't you leave anonymous tips for the auditers?
1 days ago [-]
delusional 1 days ago [-]
Sure, if you don't want a job.
unethical_ban 1 days ago [-]
That may sound bad or immoral by the company, but know that auditors have the own ambition and mo ey to think about, and will try to mark any possible thing as a serious problem regardless of whether it is.
Yes, it is highly adversarial and the best compromise I've seen is to have an internal audit team that is separate organizationally from IT, but has to withstand peer review if they claim anything is a real problem.
jandrusk 1 days ago [-]
VPN profiles? Best start revoking/renewing certs and keys. Geez, why would you store VPN profiles there?
6c696e7578 1 days ago [-]
GitLab, not GitHub. I think the distinction is that you can have a on-prem GitLab (as well as hosted online). The implication here being that RedHat probably had very relaxed account security.
zb3 1 days ago [-]
Telegram name: "thecrimsoncollective"
1 days ago [-]
temphaaa 1 days ago [-]
[dead]
dang 1 days ago [-]
[stub for offtopicness]
(title fixed now)
erikerikson 1 days ago [-]
"Correction: After publishing, Red Hat confirmed that it was a breach of one of its GitLab instances, and not GitHub. Title and story updated."
Title needs updating
stingrae 1 days ago [-]
Correction Posted: "After publishing, Red Hat confirmed that it was a GitLab account breach, not GitHub."
sateesh 1 days ago [-]
It's GitLab not GitHub
jmclnx 1 days ago [-]
[flagged]
reactordev 1 days ago [-]
>”They allegedly found authentication tokens, full database URIs, and other private information in Red Hat code and CERs, which they claimed to use to gain access to downstream customer infrastructure.”
They published CERs somewhere that had access keys and urls. Probably to somewhere that it wasn’t authorized to be published or shared and they got a hold of it. Using that, they got a hold of everything else. More CERs, GitHub repos using access tokens, vpn credentials to all the hottest players. At this point, you’d have to tear down and rebuild to undo the damage. Rotating certs, keys, IP’s, the whole nine yards.
jasonjmcghee 1 days ago [-]
Sounds like they got their hands on authentication tokens. Through RedHat, not GitHub.
tzs 1 days ago [-]
It looks to me like the article is talking about authentication tokens that they found in the data they took from the breach.
gryfft 1 days ago [-]
[flagged]
Kaytaro 1 days ago [-]
No this was all Red Hat. IBM would never let consultants use something convenient like GitHub, they would be forced to use some crappy internal webapp powered by watson.
bonzini 1 days ago [-]
IBM uses GitHub Enterprise.
RickJWagner 1 days ago [-]
Retired Red Hatter here.
I wouldn’t be quick to blame IBM. Red Hat and IBM both take security very seriously, and regard it as central to the mission. IBM also has deep enough pockets to devote serious resources to whatever they put their mind to.
Security is just hard. Procedures can be written, but people make mistakes, forget rules, etc. The procedures also have to be constantly updated to keep up with new and innovative attacks. It’s a never ending battle.
Sorry to see this happen to Red Hat. I am confident that right now all hands are on deck working on remediation.
loudmax 1 days ago [-]
Security is hard, but running a large organization is also hard. The danger is that the entire organization becomes overrun by managers whose guiding principle is CYA, and then things slowly grind to a halt. Eventually you wind up with security mandates that have no relation to what's actually deployed because the people nominally responsible for security are evaluating metrics that are years out of date.
1oooqooq 1 days ago [-]
i doubt old red hat wouldn't have warned clients right away.
Rendered at 01:44:06 GMT+0000 (Coordinated Universal Time) with Vercel.
Just hilarious
"According to them, the created ticket was repeatedly assigned to additional people, including Red Hat's legal and security staff members."
Summarized: Given enough eyeballs, all extortion demands are fallow.
“Since RedHat doesn't want to answer to us,” the hackers wrote in a channel on Telegram viewed by 404 Media, suggesting they have attempted to contact Red Hat. [...]
“We have given them too much time already to answer lol instead of just starting a discussion they kept ignoring the emails,” the message added. In another message, the group said it had “gained access to some of their clients' infrastructure as well, already warned them but yeah they preferred ignoring us.”
https://www.404media.co/red-hat-investigating-breach-impacti...
First rule of having someone reply: spell their name correctly.
What you must do immediately is notify the affected customers, bring down or lock the affected services, and contact the authorities.
If an attacker make an extortion threat, but then still follows through on the release/damage after being paid, then people are not incentivized to engage with you, and will go into attack mode right away, making it riskier for you.
HOWEVER, if the attacker make the extortion threat, takes payment, and then honors the agreement, and ends the transaction, then parties are more inclined to just pay to make the problem go away. They know that the upfront price is the full cost of the problem.
I've seen that there are 'ethical attackers' out there that move on after an attack, but you never know what kind you're dealing with :-/ "Never negotiate...."
Reputation isn't all that useful for extortion.
Running all your crimes as the "Wet Bandits" makes it much easier for law enforcement if they do catch up with you.
Thanks, hadn't encountered this word before.
I sincerely hope that the game doesn’t become prophetic in the manner Idiocracy has.
In theory, being ISO27001 means that you're environment follows best practices and has a somewhat sane security posture.
To the business people, a new customer demands that you have ISO27001 certification before they'll sign the $$$$ contract. The salesperson does not care HOW you get the certificate, just that you have it, they need this contract signed!
The department wasn't designed with security in mind, so implementing everything required by ISO will take many months. But sales needs $$$$ now! The CEO, CFO, and CTO are aligned: money now!
So, there's high pressure to pass the audit quickly. You implement what you can, you weasle your way around the things that will take too long. Those things are "out of scope" or "testing databases". You implement MFA while the auditor is auditing, but you know it breaks developers' workflows and there isn't a quick fix, so you turn MFA back off once the audit is complete....
TA-DA! We're ISO27001 certified! But we're no more secure than we were before.
Nah, it just means you have defined, documented processes and document that you stick to them. They actual processes can be shit and maybe you also have something on the side the auditors don't get shown, but ultimately the certification is a total joke. Source: Worked at a place that got certified despite being a security joke.
Yes and no. Even if it is a joke there is one thing it qualifies: You at least spent time looking at the process. This already is a gain over complete wild west.
do you mean you rather be lied to than not be lied to?
Depending on how dysfunctional the org is, there's no super dev anywhere who can fix it. You just shut up, do bad things knowing theyre bad, or get fired.
I had a sales guy sell a a company a replacement for their terminal server, with OneDrive lol
I almost died laughing when he explained to me the project.
I said.. you want to run cad files off OneDrive in place of a terminal/storage server?
"Yes"
Let's just say we ended up just moving their server to the cloud and VPN access onsite and for external developers.
> After publishing our story, Red Hat confirmed that the security incident was a breach of its GitLab instance used solely for Red Hat Consulting on consulting engagements, and not GitHub.
> While Red Hat did not respond to any further questions about the breach, the hackers told BleepingComputer that the intrusion occurred approximately two weeks ago.
And, never forget: what a company preaches and advertises is not the same with what the company is actually doing.
Also, here is some more information about this breach: https://x.com/intcyberdigest/status/1973422846396473765
Is it really OK? Not necessarily, but on the other hand you don't want to spend the rest of your life answering even more questions from other people the auditors might bring in to help them understand your helpful explanations.
I learned this the hard way, assuming auditors are logical and understand technology.
90% of the time, they are checking boxes. But if they are fishing, you have to be careful because they generally are bad at understanding anything, but good at manipulating the audit rules to frame things in such a way so they can “catch a big fish”.
A person who is used to interviewing people will be able to tell right away.
Your boss is bad apple and so are you if you adopt their ways.
Yes, it is highly adversarial and the best compromise I've seen is to have an internal audit team that is separate organizationally from IT, but has to withstand peer review if they claim anything is a real problem.
(title fixed now)
Title needs updating
They published CERs somewhere that had access keys and urls. Probably to somewhere that it wasn’t authorized to be published or shared and they got a hold of it. Using that, they got a hold of everything else. More CERs, GitHub repos using access tokens, vpn credentials to all the hottest players. At this point, you’d have to tear down and rebuild to undo the damage. Rotating certs, keys, IP’s, the whole nine yards.
I wouldn’t be quick to blame IBM. Red Hat and IBM both take security very seriously, and regard it as central to the mission. IBM also has deep enough pockets to devote serious resources to whatever they put their mind to.
Security is just hard. Procedures can be written, but people make mistakes, forget rules, etc. The procedures also have to be constantly updated to keep up with new and innovative attacks. It’s a never ending battle.
Sorry to see this happen to Red Hat. I am confident that right now all hands are on deck working on remediation.