Why is a 300k mrr company (his tiwtter bio) running on raleway?
jeremyjh 26 minutes ago [-]
You may think that sounds like a very nice salary but $4MM annual gross is a very small business. I can’t imagine how Railway would be in business if that were considered a large customer.
rco8786 48 minutes ago [-]
Lots of companies run on railway
SkyPuncher 31 minutes ago [-]
Why not?
posabsolute 1 hours ago [-]
Why are you even commenting? And it's Railway not raleway.
halfmatthalfcat 47 minutes ago [-]
User since 2011 and you’re asking this question?
fbrncci 2 hours ago [-]
Right just as I take my first day off (for the entire year!) from work to fly out for a vacation. The mother of all security bombs starts dropping all around stacks and services that I am responsible for. How else could have taken time off go for me…
jazzyjackson 1 hours ago [-]
Brother do not cancel anything for these people, they do not care about you, it's their fault for not considering bus factor in their org chart. Why can't the person above you handle that?
jjulius 2 hours ago [-]
Prioritize your mental health, go enjoy your vacation.
fbrncci 40 minutes ago [-]
Thanks, yes I will. Left without the laptop(s). Let the chips fall where they lay. If they need me back so urgently, they will have to go through the full cancellation of booked flights and hotels first. Lets enjoy the time off.
boringg 30 minutes ago [-]
I hope you don't check in on hackernews after this post here!
bryancoxwell 1 hours ago [-]
Strongly concur
Liuser 1 hours ago [-]
Not your problem. Enjoy your vacation. Bus factor of 1 is poor planning of leadership.
balderdash 44 minutes ago [-]
Either a) your company will reimburse you for your costs incurred and give you a different week off, or b) they don’t need you there that badly.
oofdere 1 hours ago [-]
I say this as a Railway customer, go take your vacation. The only time you should even reconsider is a massive bonus plus more time off for staying.
eecks 1 hours ago [-]
Correct me if I am wrong - if the API keys are compromised, then a large bill could be racked up?
dansquizsoft 57 minutes ago [-]
Oh you're right, that does sound like a good reason for the original commenter to not take his vacation. Lucky the organisation he works for made him the single point of failure in circumstances such as these.
eecks 54 minutes ago [-]
I had assumed the original commenter OWNED the company. Yes, I agree with you that if he is an employee then take the time off.
dansquizsoft 28 minutes ago [-]
Ah yep, that would be a different story if that's the case (from reading his original comment I still think he is an employee though)
reassess_blind 31 minutes ago [-]
None of this matters if you zoom out.
MattIPv4 2 hours ago [-]
Seen multiple folks in the Railway Discord confirming that their secrets stored in Railway have been used by third parties, so it definitely sounds like the attackers were able to read decrypted environment variables.
numbsafari 1 hours ago [-]
Environment variables are a terrible place to store secrets.
I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.
The 12 Factor manifesto was 100% wrong on this point.
Use environment variables to point to where secrets are located, not to store their values.
skinfaxi 48 minutes ago [-]
> Use environment variables to point to where secrets are located, not to store their values.
Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.
jrochkind1 45 minutes ago [-]
It doesn't sound like they got them via a shell environment, they just hacked into the place that stored them. If the place you store secrets can be hacked into, it doesn't really matter if they are injected into environment variables or not.
tptacek 1 hours ago [-]
Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself.
IanCal 48 minutes ago [-]
(pre-caveat thing to not be massively annoyingly pedantic throughout, I'm not a security buff and I know who it is I'm talking to, I'm not trying to correct them)
I'm not sure I get the difference - the issue I would have thought was having the system code have access to the secrets themselves? Env file vs env vars injected at runtime seems the same there - either can be exfiltrated. The difference would be having it injected when used.
Is the problem with files about what happens outside that, being committed, etc?
tptacek 38 minutes ago [-]
Everything can be exfiltrated. If you never store the environment in a durable file, it's not meaningfully easier to exfiltrate from the environment than from process memory. Most of the wisdom about the insecurity of environment variables falls out of people setting them in .profile or in .envrc files --- that's very bad, yes, as are configuration files (the thing 12 Factor Apps was responding to).
The headline here is that there's no reason to believe "environment variables" were the reason this data was breached; presumably, the database those variables came from was what actually got hit.
I have a bad gut feeling that every day is going to look a bit like today with endless urgent patches and breaches.
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
0x1F8B 2 hours ago [-]
The security team at my company is working with a frontier lab to scan all our code/systems. They are in a borderline panic to close every possible issue, even innocuous ones that are buried under defense in depth. Whatever they are seeing from our partner has them spooked -- as in "acceptable risk" is no longer a thing anymore.
Fordec 2 hours ago [-]
I fully believe the only sustainable future is that hot patching will need to itself become fully automated. Hot patch now with a minimal allowed amount of line changes to head off the AIs, review it in office hours for auditing and the team is actually awake.
My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.
munificent 2 hours ago [-]
> hot patching will need to itself become fully automated.
Great, even less control over the things we use and more power ceded to corporations.
Fordec 2 hours ago [-]
Due to the small nature of hot patching (eg. bump this dependency, rotate these keys, turn off that config, follow the runbook) it will unlikely need to be a frontier model and is likely able to be done LocalLLM/Jev style decision making rather than having to hand all the keys to frontier model operators.
stabbles 2 hours ago [-]
Hot patching is itself a liability
Fordec 1 hours ago [-]
Not reacting is a liability. Blue-Green deployment is a liability. Code is itself a liability. Choosing this career at all is a liability.
soulofmischief 20 minutes ago [-]
Great, but the fact remains that easy hot patching which skips normal review is going to open you up to many of the same liabilities you're trying to minimize by introducing it. The target just shifts, and people will attack that infrastructure.
The actual solution largely involves the same technologies but in reverse: leveraging AI for live security monitoring, better correctness validation and security testing before code goes live, etc. And of course more pressure to use battle-tested widely-adopted libraries and systems, even though of course that comes with its own tradeoffs.
Importantly, as an engineer, when someone tells you something has liabilities or tradeoffs, the response is not "Everything is a liability, existing is a liability"; that is unprofessional. The appropriate response is to engage with them on these tradeoffs and working through the best ones, because that's your job. Otherwise defer to security engineers who are ready to back up their arguments. It could be as simple as following up with, "But do you think those liabilities outweigh the benefits of hot patching without proper review? What are some of the main liabilities you see, and what methods/technologies do you think properly address them?"
xtracto 1 hours ago [-]
The internet is over. We are going back to static htmls for public pages, and Point to Point VPN bases connections for B2B service provision.
B2C? too risky to publish services online haha.
mosura 44 minutes ago [-]
Still some industries where you cable colocated servers in a shared DC together where neither server has internet access.
blantonl 15 minutes ago [-]
[flagged]
yakz 2 hours ago [-]
It's only just starting. It'll get worse before it gets better.
reconnecting 2 hours ago [-]
I think the only way out is to go back to on-prem for everything, like in the 00s.
LAN cables will go up in price soon.
seanw444 1 hours ago [-]
Ha, Cyberpunk 2077 is becoming reality. We're proposing the Blackwall.
soulofmischief 19 minutes ago [-]
That was always going to happen. The future of the internet is AIs scamming you and your grandmother, ransomwaring you, draining your back accounts, etc. There will be no reason or rhyme, it's simply a function of available compute and ease of automation, both of which have reached a critical threshold.
cyberax 1 hours ago [-]
I think we'll see a reduction in complexity and an increase in transparency, rather than just on-prem.
One medium-sized server can now easily serve the entire banking system of a large country. So you don't need clusters of servers with careful tradeoffs between consistency and availability because of the CAP theorem.
Just get a big honking host, slap Postres on it, and run your application server in a VM on the same host. For backup, export the WAL logs from Postgres onto your favorite off-site blob storage. UI for it? There's Pigsty.
It still makes sense to host this in a cloud provider, just because you probably don't want to deal with actual hardware. And maybe get a second standby server for redundancy.
That's it.
You'll need a bunch of external dependencies for email, 2FA, observability, LLMs, and so on. But they should all be outgoing dependencies that should not allow access back into your system.
mosura 42 minutes ago [-]
Real loads are too spiky for that.
solyin 2 hours ago [-]
How does on-prem help if your product is a SaaS.
lawrenceduk 2 hours ago [-]
You open a SaaS shop where people come to use it
codeulike 2 hours ago [-]
Instead of being a SaaS it will need to just become a S
80fj0348i 2 hours ago [-]
You put the SaaS into a Box your customers can install into their datacenter!
doubled112 1 hours ago [-]
What a brave new concept!
Sometimes I do miss owning the hardware to run a thing.
What I don't miss was having all of those appliances run a different base distro of Linux, each with varying levels of update support, while the security team keeps reading security scans to me.
reaperducer 1 hours ago [-]
Like back when you could rent a Google appliance.
2 hours ago [-]
Sivart13 2 hours ago [-]
surely once there's been enough patches the code will be perfect though?
DrawTR 2 hours ago [-]
Jeez, what a rough past 24 hours for all security folk. Hang in there
jacquesm 2 hours ago [-]
It's going to be a rough decade.
rectang 2 hours ago [-]
I really don't know how we are going to deal with accelerated exploit generation via LLMs — it blows up the disclosure processes we've evolved.
The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.
faizshah 12 minutes ago [-]
In new software I write these days I limit the number of external dependencies and mostly vibe code anything I need besides the most critical foundational libraries (increasingly I am replacing these as well).
I think this, the supply chain attacks and binary reverse engineering will cause us to go back to thin clients, html over the wire and vibe coded libs instead of external open source deps.
jacquesm 2 hours ago [-]
I think it will kill open source to some degree. I found it already very hard to imagine how the endless queues of issues were being dealt with so diligently, but at least there was a human on the other side. The torrent that is unleashed now is not manageable on a human scale. It's a highly automated arms race with a bunch of good-willing humans caught in the crossfire between large companies and criminals duking it out.
rectang 1 hours ago [-]
Could open source activity transition towards provably correct software, with far more modest feature sets?
I don't have experience with proof based tools, so I'm not sure what's feasible.
jacquesm 1 hours ago [-]
A lot of people in the open source world actually enjoy the act of programming. Not all of them will enjoy prompting to the same degree, but to keep pace with the bad guys you will have to either join or find something else to do. That's going to be a big loss.
rectang 49 minutes ago [-]
It's funny, that's true although I personally can't relate — I enjoy having built something robust, but the act of building isn't something I love.
I can imagine prompting an LLM to build provably correct software. Aren't we already doing something similar already when we remind it to write unit tests and account for all edge cases?
Actually writing provably correct software from scratch sounds terribly tedious, which is why I've never explored it. But maybe an LLM could make that task more approachable.
jacquesm 46 minutes ago [-]
I switch between the two modes. I love algorithmic stuff and I really don't like working on boilerplate and plumbing. So I use AI for the second and read every line and all the other stuff I roll by hand. The speedup isn't super like that but the amount of control is pretty good.
aaron_m04 1 hours ago [-]
Business requirements will have to change. All your competitors are under the same pressure with the same shitty set of options, after all.
jacquesm 1 hours ago [-]
That would have to start with businesses shelling out more funds for the packages that their businesses depend on.
kilroy123 1 hours ago [-]
It already _is_ a rough decade. I have hated the 20s so damn much.
cortesoft 2 hours ago [-]
It feels like the only solution is going to be proactively running your own AI agents looking for patches and applying them.
rectang 2 hours ago [-]
The issue is where to direct the LLM's attention.
Prompting the LLM to "look for vulnerabilities in PHP's multipart upload handling code" is going to find stuff that "look for vulnerabilities in PHP" won't.
cortesoft 2 minutes ago [-]
I was more talking about staying up on security issues with your stack after they are patched. We will need to check for security updates constantly, or risk being exploited in the gap between a security release and you finding out that you need to update your software.
cyberpunk 1 hours ago [-]
Or, you know, use client certificates so the bots can’t hit you.
Those can leak / be stolen of course, but a bunch of my clients php apis survived as without a cert they’re told to fuck right off.
It does make their client onboarding a bit harder but it’s a trade off which seems to have been the right one.
cortesoft 2 minutes ago [-]
This doesn’t help you if you need to run a public website.
jacquesm 2 hours ago [-]
Yes, so who will pay for that? Open source so far meant you would use your brains and your creativity, now you need to use your money too if you want to keep pace. From a 'guts, credits & glory' proposition to 'slop processor if you can afford it' is not really an upgrade.
throwaway613746 2 hours ago [-]
[dead]
llama052 2 hours ago [-]
I'm already tired, between pushing back on vibe coded internal projects being put onto the internet and then the constant barrage of CVEs that mean nothing.
Then you have real security incidents that you need to tune into constantly.
merlindru 2 hours ago [-]
job security!
AJRF 2 hours ago [-]
Do cloud providers have a lock-down mode where you can press 1 button and stop all spend?
Aboutplants 2 hours ago [-]
Not a bad idea
tom1337 1 hours ago [-]
It is… for their business.
1 hours ago [-]
fyzix 2 hours ago [-]
How is it possible rotate keys if you disable deployments? You're basically telling us to take our services offline...
tossstone 2 hours ago [-]
I've already taken my applications offline and will be migrating to a new host...
frio 1 hours ago [-]
Yep. What an incredibly frustrating notice. I like Railway and I guess their hands are tied here, but wow.
eecks 57 minutes ago [-]
I rotated mine. If you just change the keys and hit deploy it works
xmcp123 40 minutes ago [-]
Saw someone on Twitter indicating that they did this, and new keys got compromised.
alpineman 2 hours ago [-]
Train wreck. People were writing on discord 3 hours ago, 10 hours after the incident started, that keys created just for railway were running up usage. Meanwhile builds have been down for more than 12 hours. Only NOW did they just send out an email to all customers. I wouldn't have found out until then if I wasn't checking the discord periodically.
dylanz 32 minutes ago [-]
I'm rotating production secrets all night... great. I confirmed that they've actually used many of my secret keys already.
1-6 1 hours ago [-]
It's time for competent programmers to get rehired to help patch massive exploits that are about to unfold before us.
minraws 2 hours ago [-]
how are the secrets decrypted, I would like some explanation so that I can have some understanding here, I am starting to feel like all these companies are just meme-ing with their security posture.
Like did someone vibe code too hard, how do env secrets leak for everyone...
9cb14c1ec0 2 hours ago [-]
Easy, all the bad guys need to do is access whatever system stores the decryption keys.
barelysapient 13 minutes ago [-]
Ugh. Love Railway.
eecks 2 hours ago [-]
Huh, I've been using Railway tonight and this was the first I've seen of this. Just got an email about it now too at 23:36
i revoked my api keys, changed with new ones but guess what? railway does not allow deployments at the moment, so my app is down.
shinycode 1 hours ago [-]
I’m lucky I moved away from it to host my own db with coolify
dgellow 1 hours ago [-]
That’s coming for all platforms unfortunately
wateralien 2 hours ago [-]
Is this our new AI daily reality?
iririririr 2 hours ago [-]
companies probably have php and Python applications exposed to the web and nobody knows.
the speed of irresponsible deployments is infinite now, by design
paxy 2 hours ago [-]
It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!” as if it absolves you from having to maintain a secure system. If anything it makes the situation more damning. If it was possible to find exploits using AI then your own red teaming should have found them before the attacker did.
tokioyoyo 2 hours ago [-]
Defending is so much harder than attacking. You don't even have to focus on one target, as you can just mass-AI-script a lot of attacks wherever it's possible, and see what sticks. Even if you did everything right, there's always the social-engineering angle, or a single compromised device, or just accidental bad upgrade through supply chain. If you can mass-spread the seed, and plant something that "calls home" constantly, you can just have an agent that would start the attack the minute it gets infected, and go deeper.
It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.
Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.
nrmitchi 2 hours ago [-]
> It’s funny that every security breach disclosure now has a mandatory “the attacker used AI!”
This isn't in the article? They do not at all claim "the attacked used AI!".
The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.
arcanemachiner 2 hours ago [-]
> your own red teaming should have found them before the attacker did
Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.
gkoberger 2 hours ago [-]
Unless they edited the post, they never said this. You should delete this; it's false information.
mbreese 2 hours ago [-]
I didn’t actually see that in this post at all. They mention that the end goal of the attackers might be to get to AI inference providers. But I don’t think they are crying “AI was used” here.
Or did I miss something?
barrrrald 2 hours ago [-]
This one doesn’t?
gbnwl 2 hours ago [-]
The past two days have been awful as someone who uses Railway.
sergiotapia 2 hours ago [-]
If you're on Railway do not wait, ROTATE every API key at the source, and do not set new values because Railway is still compromised and not secured.
sergiotapia 1 hours ago [-]
Actually I will move my projects away from Railway. With AI what it is today, no reason to shy away from hosting on actual VPS.
What is the best VPS to use today? Is it still hetzner or ovh?
lijok 2 hours ago [-]
Damn I guess all those l33t hiring practices were for nothing
rvz 2 hours ago [-]
Of course, it was a giant scam that tested for nothing and optimized for how to pass a test.
Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.
Keloran 2 hours ago [-]
Well this is certainly interesting, I wonder how their secrets were being stored, I assumed they were stored in vault and then injected with a sidecar, but I guess not
goforbg 1 hours ago [-]
the worst service I've signed up for has to be railway. What a terrible product.
esseph 2 hours ago [-]
Fuck, I thought this was about trains
jon128 11 minutes ago [-]
[dead]
dayyan 56 minutes ago [-]
[flagged]
torben-friis 1 hours ago [-]
>We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers.
Is that an accusation? As in, major AI players taking out competitors ?
Rendered at 00:38:10 GMT+0000 (Coordinated Universal Time) with Vercel.
And it’s all just getting started imo.
https://blog.cloudflare.com/emudhra-certificate-compromise
https://www.wiz.io/blog/php2shell-php-rce
[0] https://news.ycombinator.com/item?id=50173716
https://docs.gitlab.com/releases/patches/patch-release-gitla...
https://x.com/wickedguro/status/2113015432974405658
I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain.
The 12 Factor manifesto was 100% wrong on this point.
Use environment variables to point to where secrets are located, not to store their values.
Now your apps need to be able to read the secret that is pointed to, as opposed to the task runner populating the secret and the app would not know about the details.Not to mention the changes it entails to development.
I'm not sure I get the difference - the issue I would have thought was having the system code have access to the secrets themselves? Env file vs env vars injected at runtime seems the same there - either can be exfiltrated. The difference would be having it injected when used.
Is the problem with files about what happens outside that, being committed, etc?
The headline here is that there's no reason to believe "environment variables" were the reason this data was breached; presumably, the database those variables came from was what actually got hit.
Fwiw it sounds similar to the hack over the weekend that opencode suffered.
My second belief is that environment variables as a development concept will need to be retired. If it's something that causes a security breach if it by its mere existence on a public git repo, it is information that by design cannot ever pass to a filesystem out of a chain of trust, not by mere development conventions people choose to adhere to.
Great, even less control over the things we use and more power ceded to corporations.
The actual solution largely involves the same technologies but in reverse: leveraging AI for live security monitoring, better correctness validation and security testing before code goes live, etc. And of course more pressure to use battle-tested widely-adopted libraries and systems, even though of course that comes with its own tradeoffs.
Importantly, as an engineer, when someone tells you something has liabilities or tradeoffs, the response is not "Everything is a liability, existing is a liability"; that is unprofessional. The appropriate response is to engage with them on these tradeoffs and working through the best ones, because that's your job. Otherwise defer to security engineers who are ready to back up their arguments. It could be as simple as following up with, "But do you think those liabilities outweigh the benefits of hot patching without proper review? What are some of the main liabilities you see, and what methods/technologies do you think properly address them?"
B2C? too risky to publish services online haha.
LAN cables will go up in price soon.
One medium-sized server can now easily serve the entire banking system of a large country. So you don't need clusters of servers with careful tradeoffs between consistency and availability because of the CAP theorem.
Just get a big honking host, slap Postres on it, and run your application server in a VM on the same host. For backup, export the WAL logs from Postgres onto your favorite off-site blob storage. UI for it? There's Pigsty.
It still makes sense to host this in a cloud provider, just because you probably don't want to deal with actual hardware. And maybe get a second standby server for redundancy.
That's it.
You'll need a bunch of external dependencies for email, 2FA, observability, LLMs, and so on. But they should all be outgoing dependencies that should not allow access back into your system.
Sometimes I do miss owning the hardware to run a thing.
What I don't miss was having all of those appliances run a different base distro of Linux, each with varying levels of update support, while the security team keeps reading security scans to me.
The only thing I can think of is hunkering down with static HTML to reduce attack surface, but that's laughably incompatible with many many business requirements.
I think this, the supply chain attacks and binary reverse engineering will cause us to go back to thin clients, html over the wire and vibe coded libs instead of external open source deps.
I don't have experience with proof based tools, so I'm not sure what's feasible.
I can imagine prompting an LLM to build provably correct software. Aren't we already doing something similar already when we remind it to write unit tests and account for all edge cases?
Actually writing provably correct software from scratch sounds terribly tedious, which is why I've never explored it. But maybe an LLM could make that task more approachable.
Prompting the LLM to "look for vulnerabilities in PHP's multipart upload handling code" is going to find stuff that "look for vulnerabilities in PHP" won't.
Those can leak / be stolen of course, but a bunch of my clients php apis survived as without a cert they’re told to fuck right off.
It does make their client onboarding a bit harder but it’s a trade off which seems to have been the right one.
Then you have real security incidents that you need to tune into constantly.
Like did someone vibe code too hard, how do env secrets leak for everyone...
the speed of irresponsible deployments is infinite now, by design
It's really a wild west right now, have no idea how we fix it. I'm assuming everything will get more concentrated in big co's hands, as having a very robust security is extremely hard.
Also, we have to agree that younger people will start companies. There will be small companies that cannot afford red teams. There will be companies that are just not equipped, and their profit margins cannot sustain constant attacks.
This isn't in the article? They do not at all claim "the attacked used AI!".
The only mention of "AI" is "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." This is the same (apparently) end-goal as the Opencode breach last week, and is not blaming or finger-point at AI. It is stating (rightfully) that AI inference is very easy to resell and turn stolen credentials into cash.
Easier said than done. The defender has to do everything right, all the time, to prevent a headline like this. The attacker only needs to be right once, and it's game over. AI or no AI.
Or did I miss something?
What is the best VPS to use today? Is it still hetzner or ovh?
Let's see if companies are ready to hire more people who know what they are doing, against a super-intelligent swarm of millions of agents with cyber capabilities.
Is that an accusation? As in, major AI players taking out competitors ?