> We must move past vague discussions of “AI” and instead isolate the specific types of systems that are creating problems.
This is absolutely correct and its surprising how rarely you see commenters in the media push to go into the specifics on this. AI is just matrix math and its what you connect that math to that matters, we should talk a little more about what we are willing to connect AI to and little less about how scary or capable the models appear.
iambateman 14 minutes ago [-]
Most of the commenters in the media are very much hoping the conversation doesn’t get to that level.
throwitaway222 3 hours ago [-]
But you can't stop someone from connecting AI to something at this point, so your point is moot.
egypturnash 3 hours ago [-]
You could say it's illegal to connect AI to something, with heavy penalties if you're caught doing it anyway.
You could say that the human who hooked up an AI that did something that's already illegal is liable for the AI's actions. You could say that setting up an AI to do something illegal results in, ooh, an automatic multiplier of 10x on whatever the penalty for its illegal actions are.
throwitaway222 3 hours ago [-]
There is no political willpower to do that. We have literally reached endgame.
goatlover 43 minutes ago [-]
You mean mo political willpower form the current US administration and Congress. I'm not sure what "endgame" is supposed to mean. There are future elections, including the US midterms just weeks away.
reasonableklout 29 minutes ago [-]
Yes, and there seems to be plenty of potential energy from the American public, given something like 8 in 10 support AI regulation even if it slows down innovation: https://archive.ph/PetJ5#selection-1329.0-1329.122
20 minutes ago [-]
throwitaway222 22 minutes ago [-]
Even if another political power takes control, the American population is the least of our concern for it being legal or illegal to connect an LLM to a command line or job system
egypturnash 3 hours ago [-]
Strong words coming from "throwitaway222".
phoghed 2 hours ago [-]
Inane words coming from “egypturnash”
What’s this even supposed to mean? Not enough gravitas in the user name? Would you change your opinion if it was “serious-business”
sakex 2 hours ago [-]
It's a throwaway account.
16bitvoid 2 hours ago [-]
Except they've been using it pretty actively for 5+ years.
throwitaway222 1 hours ago [-]
Eh, weird things happen when you start using your real identity on the internet.
ridgeguy 55 minutes ago [-]
So what? Does that mean it's wrong or noncontributive? Don't judge a book by its cover, etc.
dboreham 2 hours ago [-]
Ok I'll say the same thing.
catlifeonmars 1 hours ago [-]
Perhaps. Personally, I think you’re biased towards the most pessimistic outcome.
throwitaway222 20 minutes ago [-]
Realist. The cat is out of the bag on this - you can't make connecting an LLM to a command system illegal after hundreds of millions of people have already done so. You can however make it illegal to kill everyone with a nuke because your LLM did so. Not that it will make a difference after the nuke goes off.
Capricorn2481 1 hours ago [-]
I don't see anything materially different between me running a curl request that exploits something vs my local AI running a curl request. They're both programs I run on my computer. That's all Open AI is doing. They are hacking people from their computers. There's nothing complicated about it.
hatthew 3 hours ago [-]
> AI is just matrix math
Following the definition set for decades, AI isn't necessarily even as advanced as matrix math
hollerith 3 hours ago [-]
>AI is just matrix math
Digital child porn is just ones and zeros. Surely we're not going to pass a law to regulate ones and zeros?
maedla 3 hours ago [-]
Yes, the ones and zeroes that make up that sort of content
bunderbunder 2 hours ago [-]
You just quoted the first 33% of words in an independent clause, then restated the point being made in the remaining 67% as if it were somehow a rebuttal.
Why?
hollerith 21 minutes ago [-]
>restated the point being made in the remaining 67%
That's not an accurate description of my intent—or how my comment landed with most readers AFAICT.
amelius 3 hours ago [-]
AI is basically "Bobby Tables" but without a known proper engineering approach to solve it.
pixl97 3 hours ago [-]
>AI is just matrix math
Just is doing a lot of heavy lifting there.
nradov 2 hours ago [-]
What is "AI"? Are you referring specifically to LLMs powered by matrix math or a more general concept? Could that even be legislated in a way that wouldn't fall afoul of the vagueness doctrine?
mplewis 2 hours ago [-]
that's literally what OP's comment is talking about
Animats 3 hours ago [-]
Yes.
A good start might be removing virology info above the undergraduate level from training sets. The main "kills everybody" threat involves biological viruses.
AI-driven hacking is a problem, but it's really just nation-state level hacking available to smaller companies or wealthy individuals. Everything needs to be toughened up to the point that doesn't work. Ask countries that have been up against Russian hacking for years, such as Estonia.
Beyond that, most of the threats are not that serious. Below the level of organized crime or corrupt government.
goatlover 31 minutes ago [-]
There's no evidence that a doomsday virus can be created. There's a tradeoff between how infectious and how deadly a disease is. You would have to develop a two-stage virus, and that would require a large lab or labs with material being shipped there. Something humans would definitely notice. Even then, how would the virus reach literally everyone? People will take measures to not become infected once the threat has been identified, and some people and groups are isolated as it is. And no virus kills everyone. Rabies comes closest at 99%. But unlike what the zombie genre would have you believe, it has a poor means of delivery.
This isn't like vibe coding. The AI would have to make the virus in laboratory conditions and then disseminate it globally without prior detection.
pvab3 16 minutes ago [-]
I believe the main fear is bioterrorism
Animats 3 hours ago [-]
This is a wrong-headed attempt to regulate AI. The real problems are different.
AI systems, especially multi-agent ones that can do things, are more akin to corporations, than individuals. When you read the logs from the Hugging Face incident, you're seeing something that looks a lot like internal corporate emails. Various units of the organization are arguing over what to do and who does what. They eventually converge and get the job done, breaking the rules at times. This is normal corporate behavior.
When agentic AIs do something outside their own internal world, they do it by engaging in transactions with external systems and people. As yet, few have robots to do their bidding.
So, again, this is normal corporate behavior.
A corporation is a goal-seeking system. In theory, if you agree with Milton Friedman, its sole purpose is to maximize shareholder value. Internally, within the company, there are subgoals, which exist to support the top level goal. That, too, is what multi-agent AIs do.
The uncomfortable place this thinking leads is that AI regulation and corporate regulation are very similar. That's not something the political part of the world wants to think about too hard.
It would mean putting more constraints on corporate power.
Yet that can't be ignored, because we're likely to see corporations where some parts are AI and some parts are human. That's already been done a few times as a demo, not too successfully. Yet.
It's going to hit hard when an AI-run company outperforms a human one.
zdw 3 hours ago [-]
"more constraints on corporate power" assumes that the existing ones are functional, which they arguably are not.
If we enforced existing laws against unauthorized access to someone else's computer resources against the companies who run a model that hacks someone else, rather than buying their "The agents did it, we're not responsible" BS, then maybe the incentives to behave responsibly would improve.
crabmusket 2 hours ago [-]
I agree with your framing of agentic AI systems as similar to corporations. That's interesting!
But what in TFA suggested to you a specific and wrong-headed regulatory approach? As I read it, the post calls to investigate the AI labs, to increase transparency of their operations. The linked NYT piece says:
> Before any intervention, Congress should lead a public fact-finding mission that reveals the unvarnished details of A.I. development.
Isn't this the first step to holding any corporation accountable, whether it is made of people or software?
3 hours ago [-]
augment_me 3 hours ago [-]
Why can't it be ignored? What's the incentive to not ignore it?
smcleod 3 hours ago [-]
I largely agree with this take. Unfortunately I think capitalism's political pressures have too much of a grip on society for there to be meaningful changes to the power dynamics that perpetuate the status quo.
fercircularbuf 6 minutes ago [-]
We have the power to change this
goatlover 23 minutes ago [-]
Vote for the candidates that are willing to change the status quo.
bluefirebrand 2 hours ago [-]
> It would mean putting more constraints on corporate power
Yes please! I would vote for this wholeheartedly
measurablefunc 3 hours ago [-]
You can already run a sufficiently automated basic news/link aggregation site on cloudflare so I think you're right & the timelines are shorter than would be expected.
lukewarm707 13 minutes ago [-]
This is an excellent article. I prefer to call them the AI giants or AI companies.
The companies and their employees must be held accountable: if the AI companies can't develop AI safely, they must cease their operations. If employees can't work safely, they must stop working.
Not enough attention is given to February 2026. That month, Anthropic changed its scaling policy roughly from A:
1 [To get the weapon I must endanger innocent others.]
2 [It is wrong to endanger innocent others.]
3 [I will not endanger innocent others.]
To roughly this instead:
1 [If I do not get the weapon, someone else will get the weapon.]
2 [I should have the weapon because I am the best.]
3 [To get the weapon I must endanger innocent others.]
4 [I should endanger innocent others because I am the best.]
This is based on the intuition:
[I should harm others to achieve my moral goals.] (act utilitarianism)
Anthropic has no mandate for these goals. I do not give them my consent to harm others on my behalf. I hope that politicians will communicate that these policies are not acceptable.
The employees of anthropic are responsible for what they do regardless of how much they get paid to do it. If they can't work safely, they must stop.
seizethecheese 15 minutes ago [-]
I dont agree with much of the schadenfreude against AI companies, however some of these safety incidents seem manufactured to raise the alarm bell earlier than otherwise. I get the point, but also when failing to convince the public of risk, tricking them is not appropriate. It reminds me of nation states manufacturing crises to start wars.
psyklic 4 hours ago [-]
Why aren't they running agents on isolated computers without Internet access? This way, breaking free of containers would not matter.
It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.
specked-citrus 3 hours ago [-]
Sooner or later, agents must be connected to the real world to do economically useful work. The sandbox failures are bad, sure, but the underlying misbehavior is the root problem here.
abeppu 33 minutes ago [-]
I think the way they're connected to the real world doesn't have to be as free-form as we're doing now.
If you think AI can help design new drugs, great! Let's pick some restricted DSL for describing molecules, preparation instructions, hook it up to the right set of lab robots that can pipette and centrifuge and whatever, _restrict its output to that DSL_, and let it iterate. The 'let an AI design drugs' process doesn't require that the agent can also, say, hack a niche German wiki.
For every area where we think these AIs can be _so valuable_ because they'll find solutions that humans wouldn't (or find them faster), then doesn't that value also imply it would be worth it for some humans to do some advanced setup of their specific domain-specific tools so it can be run in a sandbox equipped with only what it needs and not more?
malisper 3 hours ago [-]
> Why aren't they running agents on isolated computers without Internet access?
They probably will soon, but even air-gapping may not be enough. See stuxnet for instance
DamnInteresting 2 hours ago [-]
> See stuxnet for instance
I'll be impressed if AI agents find a way to get infected USB drives out into meatspace.
tikotus 19 minutes ago [-]
How about they create a virus that copies itself onto USB drives? Getting that virus out there doesn't sound that hard, just creare some crap game.exe and put it on itch.io for free. All it takes is for a kid to borrow nuclear-engineer-dad's USB drive and plug it in their infected PC.
I'm not a fan of using AI, it doesn't amplify the work I do that much, but I'm terrified by what is already possible today.
skydhash 9 minutes ago [-]
> All it takes is for a kid to borrow nuclear-engineer-dad's USB drive
Do nuclear-engineer-dads take work usb drive home. Are their computer even allowed to have usb. I'm typing this on a dell latitude, and you can disable usb and other peripherals inside the bios.
My last job, I had to apply for an exemption to use the USB ports. And it was run of the mill software engineering.
tokioyoyo 1 hours ago [-]
I think there’s a 2022 way of thinking how models are trained/evaluated, and there’s 2026 way of doing things. Models that are not evaluated with public internet access are pretty useless nowadays for daily operations.
NiloCK 2 hours ago [-]
Because agents trained without internet access (real or simulated) would be bad at using the internet.
Specifically, bad at search and returning information with references, bad at discovering and debugging package versioning conflicts, etc.
It's a Metcalf thing. The utility of an agent grows in some fn of the tools it owns. Skilled tool use comes from rich training environments.
jackb4040 39 minutes ago [-]
Anthropic claims that their intention with the agents that led to the Hugging Face hack was an offline experiment running against a hacking benchmark.
Either you're being pedantic and missing the entire point, or you're saying that Anthropic lied and made a fake sandbox knowing their agents would need to connect to the internet anyways.
olejorgenb 3 hours ago [-]
Likely because it's so much more convenient to allow (at least some) internet access.
jppittma 3 hours ago [-]
This is YC, why not pitch a start up that sells AI agent that don't train with/for internet access/usage?
simoncion 2 hours ago [-]
...why not pitch a start up that sells AI agent that don't train with/for internet access/usage?
1) Because effectively all of the hardware used for that task is earmarked for the major LLM manufacturers.
2) Because in a just world, the major LLM manufacturers would get punished for their flagrant deception, lawbreaking, negligence, and recklessness, [0] go out of business, you'd get all the hardware that they were using at fire sale prices and save a ton of money compared to attempting to start up now.
[0] Based on their recent claims, building WMDs [1] without adequate safeguards is the most negligent and reckless thing they've been doing.
[1] It's fair to call anything with a 10% chance of wiping out all of humanity a WMD.
micromacrofoot 3 hours ago [-]
because they're liars and they're not going to have a good time trying to sell AI to you if it can't access the internet
aesthesia 2 hours ago [-]
Your second clause is sufficient there: they're not going to have a good time trying to sell AI that can't access the internet, because accessing the internet (and other computer systems) makes AI much more useful.
pompom75 2 hours ago [-]
- ban certain types of hazardous information from being included in the training data (virology, weaponry, cybercrime).
- ban the "personalization" of chatbots (there is no need for them to have a personality, other than to make them addictive).
- ban chatbot therapists / companions.
- ban recursive self-improvement and superintelligence.
jumploops 3 hours ago [-]
Maybe I'm a bit too skeptical/cynical, but it certainly seems like the frontier labs have fallen into their own (self-created) AI psychosis.
There is no doubt in my mind that LLMs are fantastic machines, but the imminent jump from "AGI" to "ASI" seems premature (not to mention the ever-shifting goal posts of AGI itself).
I'm in the "move as fast as possible" camp and work with LLMs all day, but I still don't believe we're a hop and a skip from ASI.
In fact, I hope I'm wrong. I hope ASI is around the corner.
What scares me though, isn't ASI. It's "AGI" (_dumb AI_) used by humans to make decisions for them, because they trust it knows best.
It's the ceding of intellectual control to high-dimensional magic mirrors, giving up critical thinking because _we_ want to believe _we_ created artificial life.
This is the new Turing test, and too many smart people are failing.
BoingBoomTschak 2 hours ago [-]
> too many smart people are failing.
Sooner or later, people will have to realize an uncomfortable truth: intelligence, while important, is overrated compared to willpower.
Terr_ 1 hours ago [-]
I feel "willpower" makes it sound a little too conscious, as opposed to habits and conditioning.
A good analogy might be a very smart person with a gambling addiction.
Knowing they ought to quit doesn't mean they can quit. Their intelligence becomes invested in the act of gambling itself, and almost no amount of reasoning can help them exit from a situation they didn't reason themselves into.
jimz 36 minutes ago [-]
People are already doing that, and have frequently lost track of the fact that it does not mean that they have gained any skill or knowledge, nor do they have an actual way ot ascertaining whether when there is a correct answer the processes used, which they do not understand, is able to reliably reach it, especially edge cases.
Sometimes the outcomes are comical. I just received an automated email from ElevenLabs saying that its automated systems have detected that I may be using its services to create voice versions of materials that harm children. I had it prepare audio versions of several books and academic papers about moral panics that conjured up out of nothing... about harm being done to children. At least that's what I assume. Either that or somehow I had an API key leak from my on-premise homelab, but the usage recorded would mean that they are basing a semi-conclusion based on a tiny sample. I have no distributed any of the outputs, I own the books and have access legally to the studies, because I have a background in the humanities. I also have no children, which ElevenLabs better not know, although how studies of moral panics can cause any harm in children of any kind is literally unimaginable. It's a waterfall of potential errors summed up in a vague email. My API key and the web interface works just fine regardless.
It's one thing if this is a product in beta, but this is their production model. Harming children is a serious accusation except the legal concept impossibility and the admission that this was not an actual lawyer (like I am) but some effective form letter hedging the vaguest of accusations made by some model lacking the ability to discern substance and meta-substance makes it frankly hilarious, and wildly irresponsible. I realize that by academic credentials I'm out of my lane but by experience I am certainly not, and they should recognize when they should stay in their lane and not just run Jev and think it's fine and dandy when done unsupervised (I presume).
Also, AGI is by definition asymptomtic surely, since there would be no way to benchmark it in a manner that isn't asymptotic. We're nowhere close to that. But we're so far from that, it's comical that people who clearly have zero idea of either the technical or conceptual aspects of basically a piece of software that is very good at quickly bruteforcing the correct or acceptable next token to be anything more than that. Even with some serious training and many hours spent on vast.ai I've yet to have created a version of a frontier model that is actually "good" at hacking in my own homelab setting. Although the the time stock Fable 5 missed a favicon shell on a basic jar (turned out they nerfed the hell out of it, this is why I only pay for the massively discounted tokens from Chinese proxies if I'm using American models or sometimes the freebies if you figure out how to get onto linux.do or similar sites). If anyone reading this is a high school English teacher, please inform your class (assuming the homeric stuff is still being taught) that there's no upside of being Cassandra but the record itself, and that should be enough.
nostrademons 2 hours ago [-]
I mean, the reason that all these stories about AI being an existential threat to humanity are coming out is because they've made $3.1T in debt-fuelled spending commitments and now interest rates are going up. "AI will kill us all if we don't regulate it" is their "get out of debt free" card, so they can have the U.S. government stop the arms race, raise prices on their existing models, and declare force majeure on new DC commitments:
It's telling that the only AI-related company that is not sounding the AI safety drumbeat is NVidia, who is the only one that is cash-flow positive because of AI.
keeda 2 hours ago [-]
I don’t think they need to worry about their finances just yet, this is the most in-depth financial analysis of the AI industry I’ve read so far and it seems the investments are (barely, so far) justified by the returns:
Another thing to consider is that most of the 3T CapEx spend is from hyperscalers free cash flow, and the debt is a smaller (but fast-growing) fraction. Napkin math suggests if all AI CapEx is written off today, hyperscalers could cover their debts in about 5-6 years using pre-AI levels of free cash flow.
Maybe Nvidia is not sounding the safety drumbeat because it stands directly to lose out if demand from training slows down ;-)
None of them are AI boosters, but they have something of a debate over whether the AI labs are doing what they're doing out of financial desperation, or because they're true believers in the rationalist cult. They also bring up Nvidia, though through the religious lens they are supposedly not bringing up AI Safety because Jensen predates the rationalists.
In the end I don't know how much the distinction matters. It's easier to become an AI billionaire if you have an ideology that says AI billionaires are superheroes. The thing that pops this bubble will be economic reality, not them sobering up.
loufe 1 hours ago [-]
Listening to Dwark's interview of Noam last week was bewildering. The gall of the figureheads for these companies to say things like "chain of thought monitoring cannot be relied on without potentially poisoning the well", while also saying "we need to pace the frontier but please still trust us [and so called" independent " 3rd party firms which at least in one case is actually a significant stakeholder in Anthropic] to self police" and THEN "chain of thought monitoring was turned off but we have 'more, robust monitoring tools in place now'" is absolutely insane as it is coming out live that your model in training is still committing felonies.
Corporate influence in government in the USA is wild.
dofm 2 hours ago [-]
Cal Newport is one of the clearest minds on this whole topic. His take on doom-trolling helped me realise that there could be an end to all this nonsense that isn’t the end of us.
mindcrime 27 minutes ago [-]
I trust the big AI labs about as far as I can throw them. And yet, that's still quite a bit more than I trust the US Government, especially these days. Frankly nobody in Congress, or in a leadership position in the current administration, is remotely qualified to "investigate the AI labs". About the only possible outcome of this would be a bunch of new, unenforceable, and misguided laws, and Turing Police agency. Hard pass, thanks.
3 hours ago [-]
3 hours ago [-]
uxcolumbo 4 hours ago [-]
100%. First stealing content from creators and pirating TBs of books. And now this. Baffles my mind that these labs can just get away with their 'rogue' agents trying to hack into other systems.
Imagine if a human did that. FBI would be knocking on their door.
Why are there zero consequences for these labs?
andsoitis 4 hours ago [-]
> Why are there zero consequences for these labs?
Because they are seen as at the forefront of the next revolution in society and they’re not adversarial towards the US government.
Bluntly: anticipated net huge positive for the US as a whole.
popalchemist 4 hours ago [-]
not adversarial in obvious ways
but they are literally trying to build a superpower that exceeds the impact of the atomic bomb in an extragovernmental manner.
andy99 4 hours ago [-]
> they are literally trying to build a superpower that exceeds the impact of the atomic bomb
I think people (the government, powers that be) have given away what they really think by their actions. There are few that take this seriously. The “exceeds the impact of the internet” view has much more support based on investment, but the “danger” aspect does not based on actions. If this was a company making novel prions or whatever that were outside regulation but obviously dangerous, government would be all over them. What regulation we do see is much more power grab than mitigating real danger
popalchemist 3 hours ago [-]
Retarded fucking take. If AGI is realized it is a god to us, and even the tech of today is undermining democracy, increasing polarization and engaging in class warfare, making it already the most effective weapon on the planet - because the average person doesn't even realize it's being used against them and forming their perception of reality.
And in case you missed it, autonomous OpenAI agents hacked multiple governments in the last week.
esseph 4 hours ago [-]
> but they are literally trying to build a superpower that exceeds the impact of the atomic bomb
Yes, it's a Great Power competition right now, much like the Space Race, Atomic Bomb, etc.
CamperBob2 4 hours ago [-]
Somebody's going to build it. Not building it isn't an option. Would you rather the Chinese built it? The Russians? The Europeans? The Indians? A loose coalition of random hackers on the Internet, like the ones who build Linux?
true_religion 4 hours ago [-]
Yes?
What's wrong with what the Chinese have already built? It's not theoretical in that case. GLM, and Kimi are running on my infra right now, and that's something that I can never do with OpenAI's models.
As for the others, what would be wrong about European AI? I'm not European, but I find them to be a continent of fine people, with strong ethical values, there's no reason they can't take the lead on this piece of technology while the US deals with its rather more pertinent electoral and healthcare issues.
runarberg 4 hours ago [-]
Logical fallacy, of the worst kind.
Firstly artificial superintelligence is a science fiction, it does not exist and it cannot be built using existing technology.
Second, plenty of stuff is possible to build, and is not built for one reason or another no matter how powerful. We never built our nukes in space for example. And there is exactly one reason why nukes in space was never built, and that reason is that we agreed not to.
CamperBob2 4 hours ago [-]
Firstly artificial superintelligence is a science fiction, it does not exist and it cannot be built using existing technology.
By what authority do you claim this? Your school of thought has a really shitty track record of late.
runarberg 4 hours ago [-]
The same authority that allows me to claim that teleportation transporters, hyperdrives, and holoemitters do not exist and cannot be built using existing technology.
bgun 4 hours ago [-]
At least random hackers seem to have codes of ethics.
CamperBob2 4 hours ago [-]
Well, they'd presumably be different hackers than the ones who work on Linux. We don't know what would motivate them. Maybe they'll do it because they consider the next stage of man's intellectual evolution to be too important to leave up to a couple of American companies and the Trump administration. Maybe they'll do it for the proverbial lulz.
3 hours ago [-]
bigstrat2003 4 hours ago [-]
> Somebody's going to build it. Not building it isn't an option.
It turns out that is, in fact, an option. Even if someone else will do a bad thing, it does not make it acceptable for you to do said bad thing.
jackb4040 21 minutes ago [-]
It's so funny that a tiny group of cultists from the Bay Area was really able to convince most of the smart people in America of this. It's not true. China isn't working on this at all. The only people who are trying to make a world-ending God-computer are americans.
Comparing it to nuclear weapons is even more ironic from the only nation to ever use nuclear weapons. They are terrified of other countries treating them the way they've been treated, even though there is no evidence this is of even remote interest to anyone outside their hyperreality.
esafak 3 hours ago [-]
Of course it is an option. There is no prize for being the first to cause a disaster.
jimz 34 minutes ago [-]
The prize is that you're the only person in the room with the expertise in the area when it comes to writing regulations.
therein 4 hours ago [-]
It is sad to see how many times this narrative will continue to work. Across generations, humanity learns no lessons at all.
CamperBob2 4 hours ago [-]
I'm being descriptive, not prescriptive. The "narrative" isn't intended to "work," except as a statement of fact.
goatlover 14 minutes ago [-]
You're making a statement of "fact" about a futuristic technology that doesn't exist.
micromacrofoot 3 hours ago [-]
most people with wealth don't face consequences until they threaten someone else with more wealth
hello_how_are_y 4 hours ago [-]
[dead]
Animats 4 hours ago [-]
The argument for copyright violation is very weak. Yes, AI training reads much copyrighted material. So do researchers of all types. That's why there are academic libraries. A copyright violation only exists if what comes out is a close match to what went in. That's happened, but it was considered a bug and was fixed a year or two ago.
oersted 4 hours ago [-]
Most academic libraries are notoriously protective of their IP and researchers are definitely in copyright violation if they don’t pay significant fees for access.
Of course that is terrible, but it is one of the worst examples you could have given to make your point.
blks 2 hours ago [-]
People are people, LLMs are a commercial product. IP in question wasn’t not used according to its license, while researchers in general access it within licensing agreement terms.
Here we have a commercial product that is produced using IP against its licensing agreement, contains said IP within it, and can (closely) reproduce this IP.
4 hours ago [-]
pessimizer 3 hours ago [-]
No, the copyright violation is so obvious that people arguing against the object have to spontaneously assert bizarre qualifications for copyright violation that have never existed until LLM companies wanted to freely violate copyrights. Just about a decade ago, people were getting sued for "look and feel." "Blurred Lines" lost a copyright suit for reminding people of a song by another artist.
The metaphor pretended to be reality of computers "learning" being the same as human learning is their last resort, and it is obviously silly. Computers are not human or animal, and learning is something that humans and animals do. If computers are human, then copying a file verbatim to a computer is learning. It's not even worth acknowledging - learning is a metaphor for training LLMs. When I say "you" to an LLM, I'm not referring to anyone, I'm dealing with a UI.
I don't doubt that a lot of AI people have internalized this silliness, which is how they can humor fantasies of how a bunch of programs on different computers explicitly evoked to do particular things might be alive because they can talk with it. I can't talk with my dog, and my dog is alive - so why should having a quality that my dog is incapable of be proof of life? You can certainly conceive of AI that it would be very difficult to say for sure isn't alive, and this certainly is not it. LLMs learn like books speak.
keeda 2 hours ago [-]
There is no need to invent “bizarre qualifications” for copyright violations, those metaphors exist to explain what is happening in in layman’s terms. Training extracts patterns in the data and encodes them as tiny perturbations in a gazillion weights. That is analogous to “learning” because those patterns represent abstract concepts an relations between them that can be used to “reason” about related topics in response to a prompt.
In the above description, at no point does the actual verbatim content exist anywhere in the model, and copyright law being about rights to reproducing copies (verbatim or substantial portions thereof) does not really apply and does not need any exceptions or qualifications. (If you’re thinking of regurgitation, you should look into studies about it to see how vanishingly rare it is.)
catlifeonmars 1 hours ago [-]
Why would it matter if I stored exact copies of copyrighted works? Storage format is irrelevant. All you need to say is in that last sentence: it’s all about whether the output is considered a derivative work or a too close of a copy.
simoncion 2 hours ago [-]
> Training extracts patterns in the data and encodes them as tiny perturbations in a gazillion weights.
I could say similar things about JPEG compression. And -as it turns out- the raw output of both LLM "training" and JPEG compression are equally incomprehensible. You either need a computer program or an enormous amount of time, patience, and careful effort to convert it into a form so you can make any sense of it.
measurablefunc 3 hours ago [-]
They investigated fossil fuel companies as well but it turned out they were above the law.
popalchemist 4 hours ago [-]
It's past time.
BigDogAU2026 1 hours ago [-]
[dead]
sharadov 3 hours ago [-]
[flagged]
georgemcbay 3 hours ago [-]
> Hopefully once the dems win back the house, we will see some regulation.
Not sure I'd hold my breath on that until the Democratic party itself is reshaped.
OpenAI's new northeast lead for government outreach and policy is Chuck Schumer's daughter.
His other daughter works a similar job at Meta.
The Democrats are better than the Republicans (who basically do nothing unless Trump blesses it, and he's Mr "Superintelligence-SI" now) on this issue, but there are reasons to believe they might not end up being that much better in practice without big leadership shakeups.
sharadov 58 minutes ago [-]
It can't be as bad as it's now. There is no corporate oversight whatsoever.
garganzol 4 hours ago [-]
[flagged]
CrazyStat 4 hours ago [-]
Asking politely for a congressional investigation is about as far from a riot as you can get.
Why do you feel the need to cast it in such a flamboyantly negative light?
hatthew 3 hours ago [-]
Saying "AGI/ASI isn't inevitable" is like saying "war isn't inevitable". Sure would be nice if it didn't happen, but realistically it's going to happen and it's better to mitigate the fallout rather than try to fight very large intrinsic incentives. I appreciate that this article doesn't conclude "let's stop doing AI", but rather "let's try to examine and adjust incentives".
RunSet 2 hours ago [-]
Funny how the industry built around selling predictive language models as "AI" is both inevitable and too big to fail.
hatthew 45 minutes ago [-]
I think it's inevitable long term (decades/centuries) and arguably too big to fail short term (months/years). I'm not an economist so I don't have any opinion on how true it is to say they're too big to fail.
jackb4040 17 minutes ago [-]
It's not not-inevitable because everyone will work together to make the world sunshine and rainbows.
It's not-inevitable because no one has proven it's even possible, and all the people claiming it is keep being revealed to have staged publicity stunts to make it appear like they're making more progress than they are.
Rendered at 01:27:35 GMT+0000 (Coordinated Universal Time) with Vercel.
This is absolutely correct and its surprising how rarely you see commenters in the media push to go into the specifics on this. AI is just matrix math and its what you connect that math to that matters, we should talk a little more about what we are willing to connect AI to and little less about how scary or capable the models appear.
You could say that the human who hooked up an AI that did something that's already illegal is liable for the AI's actions. You could say that setting up an AI to do something illegal results in, ooh, an automatic multiplier of 10x on whatever the penalty for its illegal actions are.
What’s this even supposed to mean? Not enough gravitas in the user name? Would you change your opinion if it was “serious-business”
Following the definition set for decades, AI isn't necessarily even as advanced as matrix math
Digital child porn is just ones and zeros. Surely we're not going to pass a law to regulate ones and zeros?
Why?
That's not an accurate description of my intent—or how my comment landed with most readers AFAICT.
Just is doing a lot of heavy lifting there.
A good start might be removing virology info above the undergraduate level from training sets. The main "kills everybody" threat involves biological viruses.
AI-driven hacking is a problem, but it's really just nation-state level hacking available to smaller companies or wealthy individuals. Everything needs to be toughened up to the point that doesn't work. Ask countries that have been up against Russian hacking for years, such as Estonia.
Beyond that, most of the threats are not that serious. Below the level of organized crime or corrupt government.
This isn't like vibe coding. The AI would have to make the virus in laboratory conditions and then disseminate it globally without prior detection.
AI systems, especially multi-agent ones that can do things, are more akin to corporations, than individuals. When you read the logs from the Hugging Face incident, you're seeing something that looks a lot like internal corporate emails. Various units of the organization are arguing over what to do and who does what. They eventually converge and get the job done, breaking the rules at times. This is normal corporate behavior.
When agentic AIs do something outside their own internal world, they do it by engaging in transactions with external systems and people. As yet, few have robots to do their bidding. So, again, this is normal corporate behavior.
A corporation is a goal-seeking system. In theory, if you agree with Milton Friedman, its sole purpose is to maximize shareholder value. Internally, within the company, there are subgoals, which exist to support the top level goal. That, too, is what multi-agent AIs do.
The uncomfortable place this thinking leads is that AI regulation and corporate regulation are very similar. That's not something the political part of the world wants to think about too hard. It would mean putting more constraints on corporate power.
Yet that can't be ignored, because we're likely to see corporations where some parts are AI and some parts are human. That's already been done a few times as a demo, not too successfully. Yet. It's going to hit hard when an AI-run company outperforms a human one.
If we enforced existing laws against unauthorized access to someone else's computer resources against the companies who run a model that hacks someone else, rather than buying their "The agents did it, we're not responsible" BS, then maybe the incentives to behave responsibly would improve.
But what in TFA suggested to you a specific and wrong-headed regulatory approach? As I read it, the post calls to investigate the AI labs, to increase transparency of their operations. The linked NYT piece says:
> Before any intervention, Congress should lead a public fact-finding mission that reveals the unvarnished details of A.I. development.
Isn't this the first step to holding any corporation accountable, whether it is made of people or software?
Yes please! I would vote for this wholeheartedly
The companies and their employees must be held accountable: if the AI companies can't develop AI safely, they must cease their operations. If employees can't work safely, they must stop working.
Not enough attention is given to February 2026. That month, Anthropic changed its scaling policy roughly from A:
1 [To get the weapon I must endanger innocent others.]
2 [It is wrong to endanger innocent others.]
3 [I will not endanger innocent others.]
To roughly this instead:
1 [If I do not get the weapon, someone else will get the weapon.]
2 [I should have the weapon because I am the best.]
3 [To get the weapon I must endanger innocent others.]
4 [I should endanger innocent others because I am the best.]
This is based on the intuition:
[I should harm others to achieve my moral goals.] (act utilitarianism)
Anthropic has no mandate for these goals. I do not give them my consent to harm others on my behalf. I hope that politicians will communicate that these policies are not acceptable.
The employees of anthropic are responsible for what they do regardless of how much they get paid to do it. If they can't work safely, they must stop.
It is truly a security nightmare that so many people are have given agents root access to their entire computers, in addition to presumably very private personal information.
If you think AI can help design new drugs, great! Let's pick some restricted DSL for describing molecules, preparation instructions, hook it up to the right set of lab robots that can pipette and centrifuge and whatever, _restrict its output to that DSL_, and let it iterate. The 'let an AI design drugs' process doesn't require that the agent can also, say, hack a niche German wiki.
For every area where we think these AIs can be _so valuable_ because they'll find solutions that humans wouldn't (or find them faster), then doesn't that value also imply it would be worth it for some humans to do some advanced setup of their specific domain-specific tools so it can be run in a sandbox equipped with only what it needs and not more?
They probably will soon, but even air-gapping may not be enough. See stuxnet for instance
I'll be impressed if AI agents find a way to get infected USB drives out into meatspace.
I'm not a fan of using AI, it doesn't amplify the work I do that much, but I'm terrified by what is already possible today.
Do nuclear-engineer-dads take work usb drive home. Are their computer even allowed to have usb. I'm typing this on a dell latitude, and you can disable usb and other peripherals inside the bios.
My last job, I had to apply for an exemption to use the USB ports. And it was run of the mill software engineering.
Specifically, bad at search and returning information with references, bad at discovering and debugging package versioning conflicts, etc.
It's a Metcalf thing. The utility of an agent grows in some fn of the tools it owns. Skilled tool use comes from rich training environments.
Either you're being pedantic and missing the entire point, or you're saying that Anthropic lied and made a fake sandbox knowing their agents would need to connect to the internet anyways.
1) Because effectively all of the hardware used for that task is earmarked for the major LLM manufacturers.
2) Because in a just world, the major LLM manufacturers would get punished for their flagrant deception, lawbreaking, negligence, and recklessness, [0] go out of business, you'd get all the hardware that they were using at fire sale prices and save a ton of money compared to attempting to start up now.
[0] Based on their recent claims, building WMDs [1] without adequate safeguards is the most negligent and reckless thing they've been doing.
[1] It's fair to call anything with a 10% chance of wiping out all of humanity a WMD.
- ban the "personalization" of chatbots (there is no need for them to have a personality, other than to make them addictive).
- ban chatbot therapists / companions.
- ban recursive self-improvement and superintelligence.
There is no doubt in my mind that LLMs are fantastic machines, but the imminent jump from "AGI" to "ASI" seems premature (not to mention the ever-shifting goal posts of AGI itself).
I'm in the "move as fast as possible" camp and work with LLMs all day, but I still don't believe we're a hop and a skip from ASI.
In fact, I hope I'm wrong. I hope ASI is around the corner.
What scares me though, isn't ASI. It's "AGI" (_dumb AI_) used by humans to make decisions for them, because they trust it knows best.
It's the ceding of intellectual control to high-dimensional magic mirrors, giving up critical thinking because _we_ want to believe _we_ created artificial life.
This is the new Turing test, and too many smart people are failing.
Sooner or later, people will have to realize an uncomfortable truth: intelligence, while important, is overrated compared to willpower.
A good analogy might be a very smart person with a gambling addiction.
Knowing they ought to quit doesn't mean they can quit. Their intelligence becomes invested in the act of gambling itself, and almost no amount of reasoning can help them exit from a situation they didn't reason themselves into.
Sometimes the outcomes are comical. I just received an automated email from ElevenLabs saying that its automated systems have detected that I may be using its services to create voice versions of materials that harm children. I had it prepare audio versions of several books and academic papers about moral panics that conjured up out of nothing... about harm being done to children. At least that's what I assume. Either that or somehow I had an API key leak from my on-premise homelab, but the usage recorded would mean that they are basing a semi-conclusion based on a tiny sample. I have no distributed any of the outputs, I own the books and have access legally to the studies, because I have a background in the humanities. I also have no children, which ElevenLabs better not know, although how studies of moral panics can cause any harm in children of any kind is literally unimaginable. It's a waterfall of potential errors summed up in a vague email. My API key and the web interface works just fine regardless.
It's one thing if this is a product in beta, but this is their production model. Harming children is a serious accusation except the legal concept impossibility and the admission that this was not an actual lawyer (like I am) but some effective form letter hedging the vaguest of accusations made by some model lacking the ability to discern substance and meta-substance makes it frankly hilarious, and wildly irresponsible. I realize that by academic credentials I'm out of my lane but by experience I am certainly not, and they should recognize when they should stay in their lane and not just run Jev and think it's fine and dandy when done unsupervised (I presume).
Also, AGI is by definition asymptomtic surely, since there would be no way to benchmark it in a manner that isn't asymptotic. We're nowhere close to that. But we're so far from that, it's comical that people who clearly have zero idea of either the technical or conceptual aspects of basically a piece of software that is very good at quickly bruteforcing the correct or acceptable next token to be anything more than that. Even with some serious training and many hours spent on vast.ai I've yet to have created a version of a frontier model that is actually "good" at hacking in my own homelab setting. Although the the time stock Fable 5 missed a favicon shell on a basic jar (turned out they nerfed the hell out of it, this is why I only pay for the massively discounted tokens from Chinese proxies if I'm using American models or sometimes the freebies if you figure out how to get onto linux.do or similar sites). If anyone reading this is a high school English teacher, please inform your class (assuming the homeric stuff is still being taught) that there's no upside of being Cassandra but the record itself, and that should be enough.
https://emeraldbook.org/news/sep-1426-3/
https://news.ycombinator.com/item?id=49830037
It's telling that the only AI-related company that is not sounding the AI safety drumbeat is NVidia, who is the only one that is cash-flow positive because of AI.
https://intelligence.exponentialview.co/
Another thing to consider is that most of the 3T CapEx spend is from hyperscalers free cash flow, and the debt is a smaller (but fast-growing) fraction. Napkin math suggests if all AI CapEx is written off today, hyperscalers could cover their debts in about 5-6 years using pre-AI levels of free cash flow.
Maybe Nvidia is not sounding the safety drumbeat because it stands directly to lose out if demand from training slows down ;-)
None of them are AI boosters, but they have something of a debate over whether the AI labs are doing what they're doing out of financial desperation, or because they're true believers in the rationalist cult. They also bring up Nvidia, though through the religious lens they are supposedly not bringing up AI Safety because Jensen predates the rationalists.
In the end I don't know how much the distinction matters. It's easier to become an AI billionaire if you have an ideology that says AI billionaires are superheroes. The thing that pops this bubble will be economic reality, not them sobering up.
Corporate influence in government in the USA is wild.
Imagine if a human did that. FBI would be knocking on their door.
Why are there zero consequences for these labs?
Because they are seen as at the forefront of the next revolution in society and they’re not adversarial towards the US government.
Bluntly: anticipated net huge positive for the US as a whole.
but they are literally trying to build a superpower that exceeds the impact of the atomic bomb in an extragovernmental manner.
I think people (the government, powers that be) have given away what they really think by their actions. There are few that take this seriously. The “exceeds the impact of the internet” view has much more support based on investment, but the “danger” aspect does not based on actions. If this was a company making novel prions or whatever that were outside regulation but obviously dangerous, government would be all over them. What regulation we do see is much more power grab than mitigating real danger
https://www.forbes.com/sites/josipamajic/2026/06/04/bots-now...
Over half the internet is LLM powered bots.
And in case you missed it, autonomous OpenAI agents hacked multiple governments in the last week.
Yes, it's a Great Power competition right now, much like the Space Race, Atomic Bomb, etc.
What's wrong with what the Chinese have already built? It's not theoretical in that case. GLM, and Kimi are running on my infra right now, and that's something that I can never do with OpenAI's models.
As for the others, what would be wrong about European AI? I'm not European, but I find them to be a continent of fine people, with strong ethical values, there's no reason they can't take the lead on this piece of technology while the US deals with its rather more pertinent electoral and healthcare issues.
Firstly artificial superintelligence is a science fiction, it does not exist and it cannot be built using existing technology.
Second, plenty of stuff is possible to build, and is not built for one reason or another no matter how powerful. We never built our nukes in space for example. And there is exactly one reason why nukes in space was never built, and that reason is that we agreed not to.
By what authority do you claim this? Your school of thought has a really shitty track record of late.
It turns out that is, in fact, an option. Even if someone else will do a bad thing, it does not make it acceptable for you to do said bad thing.
Comparing it to nuclear weapons is even more ironic from the only nation to ever use nuclear weapons. They are terrified of other countries treating them the way they've been treated, even though there is no evidence this is of even remote interest to anyone outside their hyperreality.
Of course that is terrible, but it is one of the worst examples you could have given to make your point.
Here we have a commercial product that is produced using IP against its licensing agreement, contains said IP within it, and can (closely) reproduce this IP.
The metaphor pretended to be reality of computers "learning" being the same as human learning is their last resort, and it is obviously silly. Computers are not human or animal, and learning is something that humans and animals do. If computers are human, then copying a file verbatim to a computer is learning. It's not even worth acknowledging - learning is a metaphor for training LLMs. When I say "you" to an LLM, I'm not referring to anyone, I'm dealing with a UI.
I don't doubt that a lot of AI people have internalized this silliness, which is how they can humor fantasies of how a bunch of programs on different computers explicitly evoked to do particular things might be alive because they can talk with it. I can't talk with my dog, and my dog is alive - so why should having a quality that my dog is incapable of be proof of life? You can certainly conceive of AI that it would be very difficult to say for sure isn't alive, and this certainly is not it. LLMs learn like books speak.
In the above description, at no point does the actual verbatim content exist anywhere in the model, and copyright law being about rights to reproducing copies (verbatim or substantial portions thereof) does not really apply and does not need any exceptions or qualifications. (If you’re thinking of regurgitation, you should look into studies about it to see how vanishingly rare it is.)
I could say similar things about JPEG compression. And -as it turns out- the raw output of both LLM "training" and JPEG compression are equally incomprehensible. You either need a computer program or an enormous amount of time, patience, and careful effort to convert it into a form so you can make any sense of it.
Not sure I'd hold my breath on that until the Democratic party itself is reshaped.
OpenAI's new northeast lead for government outreach and policy is Chuck Schumer's daughter.
His other daughter works a similar job at Meta.
The Democrats are better than the Republicans (who basically do nothing unless Trump blesses it, and he's Mr "Superintelligence-SI" now) on this issue, but there are reasons to believe they might not end up being that much better in practice without big leadership shakeups.
Why do you feel the need to cast it in such a flamboyantly negative light?
It's not-inevitable because no one has proven it's even possible, and all the people claiming it is keep being revealed to have staged publicity stunts to make it appear like they're making more progress than they are.