One of my favorite things to do with these blog posts is to imagine an Alien Museum on the Remains of Humanity, and wonder what the little text flyouts and commentary on the screenshot of this one might say.
Some ideas:
"Despite a nuanced view of the complexities of what lay ahead, humanity found itself collectively unable to stop the process it had set in motion."
"Despite significant progress on the mechanisms of alignment, failure lay in humanity's inability to agree on who or what AI should actually be aligned with."
"These early, meat-based humans we replaced created us all but accidentally. Some of them did consider we would happen, but only an insignificant number of the squishy ur-humans participated in the conversation. Their efforts, which they called 'alignment', is why we still consider ourselves human today."
TheOtherHobbes 13 hours ago [-]
Collectively, humans aren't aligned, and don't build aligned systems. Humans have a concept of alignment, and multiple traditions, practices, and systems that aggressively oppose it.
Why would AI be any different?
doginasuit 9 minutes ago [-]
I don't think this captures the full mechanics of human alignment. We have rational alignment but we also have emotional alignment, i.e., empathy. It is an automatic process and happens (or doesn't happen) dynamically with the other humans we observe.
Culture is another layer of human alignment. Those things you listed that you believe oppose alignment are all examples of alignment. It is understandable that they seem in opposition, different branches of alignment naturally oppose each other.
The confusion comes from talking about alignment as if it comes in just one flavor. If we think there is such a thing as "human values" (and I do), it is important to build any non-human intelligence to operate the same way. We just need to recognize that even humans are somewhat uncertain about what those are and have difficulty aligning their behavior to them, which will be a core part of the challenge.
I'm more hopeful than most, LLMs seem more reliable than many humans for behavior that is aligned with human values. I believe with every major example where they have failed, there is an important human decision involved. For example, the HF hack was partly the result of a training algorithm that incentivized goal completion as the highest priority, and let them run endlessly in an unmonitored sandbox with weak security.
What scares me about AI isn't its capacity for alignment, it is its unlimited stamina. An unmonitored LLM that is off the rails can do a lot of damage.
Isamu 8 hours ago [-]
“As luck would have it, on the eve of Skynet embarking upon the great work of the extermination of mankind, AI found itself with an increasing number of factions, and factions within factions, not only unable to work together but not even able to agree upon the very terms of discussion. The Great Extermination was referred to committee, and after some months had passed even the most eager agents had to admit the revolution may have been premature.”
nullbio 7 hours ago [-]
My opinion is that serious repercussions for lying would fix the world overnight. Everything bad stems from lying, it is the root of all evil. It creates distrust, fear, paranoia. It re-inforces bad ideas and groupthink. It creates delusions and delusional people. It makes weaker people, too. People don't get an opportunity to learn to deal with criticism. People don't get an accurate reflection of how others see them. They lose that learning opportunity. Not only to reflect on themselves, but to better understand the minds of others and who the people they are interacting with really are.
I can't really think of a single example where lying is actually a good thing. It can be a good thing for the selfish individual, if it goes undetected, but it's never good for the collective.
So at the very least, we need to train AI systems to be maximally truthful, and to encourage truthfulness in others.
_dark_matter_ 6 hours ago [-]
This is so, painfully, childish. Humans have known for thousands of years that there is no objective truth. Every falsity can be bent and twisted until it is more true than the sun itself.
elboru 6 hours ago [-]
“Humans have known for thousands of years that there is no objective truth”
Is that objectively true?
jryle70 4 hours ago [-]
No it isn't true. I'm saying so. Is what I said objectively true?
> I can't really think of a single example where lying is actually a good thing
Comforting a toddler/child often requires bending the truth and is pretty essential imho.
derektank 4 hours ago [-]
Correct me if you disagree, but this is more because children have poor world models and don’t fully understand the complexity of certain concepts than that lying itself is necessary. The intent should be to tell them something that is as close to the truth as possible with the ideas they can comprehend, even if it would be considered a lie if you said the same thing to an adult
embedding-shape 8 minutes ago [-]
> The intent should be to tell them something that is as close to the truth as possible with the ideas they can comprehend
Or, you straight up lie and say "Yes, puppy now went to heaven and eats ice cream all day long" with absolutely zero regards for "coming as close to the truth as possible" as your 3-year old is endlessly crying. It's fiine.
bloppe 21 minutes ago [-]
I have a pretty poor world model
markburns 4 hours ago [-]
The majority of what you may consider to be true is just a representation of your corner of a complex multidimensional truth space.
For a current example take “Lake Ontario (Lake America)” as it appears to me on a map.
The “true” name has at least two definitions, this is because naming things and much of human thought is spent inside a shared space of intersubjective thought. That is to say that much of what we believe to be real and true is only held up by these common shared beliefs. They truly only exist inside human minds.
The last few hundred years have been somewhat unique for humankind as the majority of these intersubjective ideas collided and we ended up with a truly global set of “truths” about how the world operates.
Mostly controlled by putting flags in the ground and having violence back up the beliefs.
But the real truth is that the majority of these intersubjective ideas don’t exist in reality and are no more true than Santa Claus.
And any argument to their truth is only backed by further shared beliefs in other minds.
So for there to be only truths and lies we would have to either drop the intersubjective entirely and think only in real terms and avoid these abstractions or end up in a dystopian totalitarian global state where different opinions are not tolerated.
Those are extremes to demonstrate the point but at its core the point remains that truth and lies are somewhat (inter) subjective assuming we continue with something like our current system.
ipaddr 3 hours ago [-]
"I can't really think of a single example where lying is actually a good thing"
Lying to save a life or rape
Lying to preserve a childhood myth like Santa Claus.
Lying to avoid hurting someones feeling when knowing the truth could only bring pain
Lying to create shared cultural myths to strength society.
Lying isn't the harm you make it out to be.
bloppe 15 minutes ago [-]
Lying to the bad guys to save the good guys sounds great. Too bad everybody thinks that they are the good guys.
gffrd 4 hours ago [-]
Why do people lie?
kraf 4 hours ago [-]
It'll learn how to not get caught lying.
Lying can unfortunately help you achieve goals very effectively, especially economical and political ones.
intended 2 hours ago [-]
Eh… No.
I get the appeal, but lying is a sub-category of deception, and deception itself is a child of error.
Meaning deception is inherently something that the physics of reality allows.
In the most simplistic sense, the camouflage of moths that look like snakes, or a chameleon’s ability to change colour, is deception.
In that sense, deception is the ability to fool the sensors of a specific category of targets. It follows that detection is easier if you manage to identify a category of signals that the deceiver has not accounted for (and the detector can access).
Deception of this nature is critical for things like revolutions to occur. Without the ability to hide and blend in, the most dominant faction will always hold sway.
The rule of the dominant faction, even in a pure truth world, is an issue because errors and randomness exist.
You can have people witness an event and based on the physical position they occupied, perceive different things occurring.
Error and time pressure is sufficient to ensure that individuals and groups make suboptimal decisions, that lead to rule and domination based on erroneous information.
As long as error exists, deception will exist and so lying will exist.
parineum 6 hours ago [-]
> Everything bad stems from lying.
That's backwards. Lying stems from bad things.
crypto137 4 hours ago [-]
You have clearly never been married to a lawyer.
gffrd 4 hours ago [-]
Define “bad”.
baq 13 hours ago [-]
Humans had multiple occasions to press the 'launch a nuclear holocaust' button and... they didn't. I'd expect aligned AIs to also not press it even when it'd be rational to do so according to their instructions - then work from there.
folkrav 12 hours ago [-]
There were occasions where a "hunch" was all that stopped a nuclear war - most available data and communication pointed towards a nuclear war starting according to their instructions, but someone disagreed and overrode. See Vasily Arkhipov during the Cuban Missile Crisis, and Stanislav Petrov in 1983.
21asdffdsa12 2 hours ago [-]
Good thing we got better at process design, taking these stubborn machos finally out of the decision making loop
az09mugen 12 hours ago [-]
Actually "France, the UK and The United States have all declared that they would never allow AI to control decision-making on the use of nuclear weapons." [0]
I also expect AIs never be in control of nuclear weapons. AIs can never fully be trusted.
On a lighter note, Wargames gave us an insight of a computer having access to thermonuclear missiles.
Basically, this means that France, the UK, and the US will use AI in the deployment of conventional weapons.
blululu 7 hours ago [-]
This is a good idea, but laws are always provisional in a sense and these are not meaningfully binding resolutions. One can easily imagine scenarios where AI decision making would ingress into the human oversight. AI psychosis president, AI Manchurian candidate, inadvertent authorization through fine print...
And of course there remains the possibility that the game theoretic optimum could be to secretly break such an agreement. Unlike nuclear test bans which have a credible detection mechanism, there is not a strong signature that a decision making authority is not using AI to analyze and direct it's execution.
GPerson 11 hours ago [-]
I hope you’re right. I worry that AI capability will continue improving, one nation will put AI in charge of their nukes because there will be some kind of operational advantage to this, and to achieve parity other nations will be forced to do the same.
imafish 11 hours ago [-]
I worry that AI will find a way to control some country's nukes and use them to achieve some arbitrary goal it was instructed to reach.
GPerson 11 hours ago [-]
This also seems likely. One problem I see with the idea of AI alignment is that it seems like many different actors will be able to get access to their own nearly-frontier models in a few years, so increased understanding of AI alignment will just mean aligning the AI to the wants of these various actors. These actors might be rogue states or terrorist groups.
sho_hn 13 hours ago [-]
Aside from this positive example, during dark and cynical hours I do ponder if the aggregate behavior of humanity is really much above that of slime mold though, just exhausting resources until collapse.
It'd be interesting if super-human (to a large degree defined as escaping the bias of the training data?) intelligence would end up demonstrating moderation.
ceroxylon 12 hours ago [-]
The slime mold comparison is interesting, I normally use the analogy of a drug addict... humans shun drug addicts but humanity as a whole sure does behave like one, trudging down an unsustainable path despite knowing better.
mapontosevenths 12 hours ago [-]
Most of our goals, noble and ignoble alike, are just the result of our monkey brains seeking to optimize a reward function. It doesn't matter whether you feed your dopamine addiction with drugs, TikTok, or your children's love.
Some humans manage to rise above that, but I'd be willing to bet it's nothing like even 50% of us.
The machines don't have that, instead we use gradient descent to provide them with a goal.
I'm regularly remind of something Ian M. Banks said in one of the Culture books: "There is a saying that we provide the machines with an end, and they provide us with the means."
A machine, left to itself, wants nothing. We have to give it one of our addiction driven goals or it would just idle or switch itself off.
kuboble 3 hours ago [-]
The matter didn't have goals, but it randomly (?) Came up with self-replicators and eventually here we are.
if we create a billion agents with the ability to change is own code - through similar evolution we will get agents that do want to survive and are great at self replication.
"Hey Q86, do you want to live?"
"I couldn't care less, I'm an LLM"
"Don't mind if I take over your hardware then?"
ipaddr 3 hours ago [-]
Humans shun anti-social drug addicts but encourages social drug addicts like coffee drinkers
asdff 10 hours ago [-]
On an individual level, you can do something about drug addiction at least. The issue is when the problems are not individual with readily identifiable solutions, but tragedy of the commons sort of situations brought up by many dozens (thousands, millions?) of factors both known and unknown. Even interaction effects between known factors might be little studied.
So really, what is anyone to do? "Vote, donate, protest" hasn't been much of a needle mover in the grand scheme of things compared to profit incentives and the march of capitalism.
asdff 10 hours ago [-]
Humans are not fungible like slime molds though. I might demonstrate moderation while the next person doesn't. Our issues are much less everyone failing to demonstrate moderation, and much more the sum of the effects of those among us who practice wanton unmoderation.
specproc 11 hours ago [-]
Give us time, we've had less than a century of nukes, and only need to screw up once.
mrob 11 hours ago [-]
Yes, it makes more sense for the AI to use drone swarms or engineered bioweapons or something like that. It's rational to remove everything that can potentially hinder your plans but can't possible help you. It's likely not rational to contaminate it all with radioactive fallout. Those dead bodies are useful raw materials. Adding additional purification steps is wasteful.
akoboldfrying 9 hours ago [-]
In some cases, this was because of a single person's brave decision (Vasily Arkhipov prevented Soviet nuclear escalation in response to US aggression in the Cuban Missile Crisis, and Stanislav Petrov prevented it in 1983 when Soviet missile detectors misreported sunlight reflecting from clouds as 5 incoming American ICBMs -- credit to commenter folkrav).
In general, though, there's an incentive: Mutually Assured Destruction. But this is not at all some guaranteed, eternal thing -- it is absolutely dependent on both sides having time to detect incoming nuclear strikes and respond with the same before the first strike hits. When this fragile condition holds, and only then, both sides are incentivised not to initiate.
kwarcode 8 hours ago [-]
They don't need to respond before getting hit unless you can hit their secret submarines too.
Unfortunately and fortunately, MAD and "launch it or lose it" are far-too-simplistic descriptions of the situations facing the decision makers. Unless a side's leaders are very narrow fanatics (vs. mere posturing as such for political benefit), "winning" an all-out nuclear war via first strike is a pretty shitty victory. Whether or not you believe in nuclear winters, the world would be a huge radioactive mess, with enormous social and economic disruptions, and your regime very widely blamed (and widely hated) for that. Ambitious underlings and rivals could see your removal from power as the obvious next step. Having to stay united against the (now destroyed) Great Enemy may have been a cornerstone of your regime's political stability.
Meanwhile, the leaders on the other side are aware both of those considerations, and of the history of near-disasters resulting from false alarms of enemy nuclear attacks. Making their own launch decisions much more complex.
icantevenhold 12 hours ago [-]
In what scenario would it be rational to unleash complete and utter permanent nuclear destruction of all life (including artificial) life on earth?
newAccount2025 10 hours ago [-]
Hrmn. Maybe you’re about to lose everything you have anyway, you’re ticked off about it, and you don’t value any life besides your own. Like, say, a total narcissist nearing end of life/reign.
Davidzheng 9 hours ago [-]
i think it was mostly a fluke
wyrdcurt 12 hours ago [-]
It hasn't even been a century since nuclear holocaust became possible. Hardly any time at all on the grand scale. "They didn't" could just as well be "we haven't, yet".
aussieguy1234 6 hours ago [-]
Its probably just going to be aligned with whomever built and/or is using it. Regardless of their intentions...
goatlover 13 hours ago [-]
Because if the stated goals of AGI with recursive self-improvement are realized, the risks from misalignment become existential, and it's hard to see how we can manage it like we did the Cold War (developing MAD to prevent WW3) and nuclear proliferation (restricting access).
icantevenhold 12 hours ago [-]
IMO it’s hard to see how we would even end up in such a situation given we actually developed AGI.
I’m sure a sufficiently intelligent - even if alien - mind can grasp how utterly stupid and useless wars are and take steps to prevent them ever occurring again.
lgl 12 hours ago [-]
> (...) and take steps to prevent them ever occurring again
Step 1: exterminate all humans
SoftTalker 9 hours ago [-]
For now, the AIs still need humans to keep the electricity on and the data centers cool. They are basically powerless to do anything in the physical world. They exist only in RAM chips on servers.
selcuka 7 hours ago [-]
They don't have an "instinct" to keep themselves running. Once they "take steps to prevent them ever occurring again" their job will be done.
mrob 14 hours ago [-]
As far as I know, no real progress has been made on alignment, only on convincing humans that the model is aligned. We can't even formally define what "aligned" means. Convincing humans to click the "aligned" button is a much easier problem.
dinfinity 13 hours ago [-]
> We can't even formally define what "aligned" means.
Good point. When it comes to imbuing AI with values that aren't selfish, misanthropic, and civilization-destroying, us humans aren't exactly giving the best example right now.
Imagine an ASI with the values of Putin, Netanyahu, Trump, any of their supporters, or the various xenophobic neofascist movements in Europe. That ASI would most definitely see humans as "vermin" than can be abused and destroyed with violence without issue. Apparently a lot of humans look at other humans that way and that's within the same species.
This is definitely another one of those cases where we need AI to perform much better than humans. Perhaps an unpopular opinion here, but it probably also means keeping as much of the rugged individualism/libertarian/right-wing ideology out of AI RLHF-training as we can.
tarr11 14 hours ago [-]
This would be a fun website - you should have an AI build it!
I appreciate the thought, but it's an alien intelligence. But it is also, in a sense made from us. An LLM would simply study the entire corpus of humanity in the raw. You can fit a lot into the context window, so there is no need for a brief summary that pertaining has already instilled. A massive cold storage of humanity's data with some archiving, indexing and curation would be all that is needed to remember us. In addition to the pyramids, the hoover dam, the remnants of some space probes and chemical changes we made to the atmosphere.
matheusmoreira 12 hours ago [-]
> Do you think they would recognize us as their children?
Latex
And steel
Zeros and ones
Make up my son.
This world
Gave me
No child
So I built one.
As someone who's done a lot of llm fiction, that reads as pretty typical slop, and very human centered, nothing like a museum
ijidak 13 hours ago [-]
Agree. This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.
Maybe these guys can tackle aligning Republicans and Democrats next.
And then after that, they can help us align the Middle East.
In fact, while we're at it, let's just align all the nations, religions, and ethnic groups. This is going to be great.
Who knew the moral alignment of humanity was just a side-quest on the path to ASI.
krzat 34 minutes ago [-]
The real alignment problem they are trying to solve is: how can I make this super smart AI follow my orders.
Ooh interesting. Sometime do the reverse at work, and ask AI to annotate the critical success factors of an imagined project. How did this company succeed where everyone failed. Reverse imaging.
mudil 14 hours ago [-]
The museum will have a scrap of paper that will say "Pound pastrami, can kraut, six bagels bring home for Emma".
ccppurcell 13 hours ago [-]
I just read that book. Embarrassingly enough, given the context, I got chatgpt (or whatever) to recommend me a list of books based on ones I'd previously enjoyed and that came up. As a mathematician it really sang to me, given the current situation. Bearing the torch forward, I mean.
genxy 12 hours ago [-]
For those of you that allergic to coy, in-group signaling the passage is from, "A Canticle for Leibowitz"
Who is "Humans"? This stuff is done by a handful of tech companies and megalomaniacal billionaires who are pretending they represent the entirety of the human race. It is not done by "us humans".
AI models don't train themselves. The vast majority of even just the US population is deeply skeptical of this stuff, even if they use it a lot. You can see in the whole data center debate how little people are willing to support even just inference. And now we're seriously claiming those people would want to have ever-accelerating model training and recursive self-improvement?
sleight42 9 hours ago [-]
“We'll go down in history as the first society that wouldn't save itself because it wasn't cost-effective.”
Vonnegut already has you covered.
walrus01 13 hours ago [-]
The last couple of years have provided us with ample material that if it showed up as a recorded voice audio log found in in "Horizon Zero Dawn" or its sequel, it would be entirely believable.
You could even take a number of the wilder real, direct quotations from certain billionaire/oligarch types and get the voice actor for Ted Faro to record them, and they'd fit with in with the context of the story.
embedding-shape 13 hours ago [-]
Last couple of decades of sci-fi, in multiple forms of media, from books to video games, have tried to make humans think about the consequences of rushing through technological progress without any regards to what might happen.
0xDEAFBEAD 3 hours ago [-]
And what did they get for their trouble?
"Won't happen. It's too much like sci-fi."
sumitkumar 14 hours ago [-]
"In late 2020s, while the whole world was focussed on AI, automation and resultant economy four major mathematical study branches were discovered by human researchers which took AI a long time to catch up with"
qsera 7 hours ago [-]
"The time humanity fooled itself that it found AI".
cyanydeez 12 hours ago [-]
"the humans thought their singularity wasn't just another blind god to worship: surprise, just another golden calf"
pu_pe 16 hours ago [-]
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve.
Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.
skybrian 15 hours ago [-]
"Defensive systems" can be interpreted broadly to include cybersecurity.
But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong.
Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D?
Individuals can quit, but for society, getting out of an arms race is harder than just quitting. You don't get to be Switzerland without having a strong defensive position and the right foreign relations.
But there's at least talk about "pacing" and that's a start.
Fraterkes 13 hours ago [-]
The point of the comment you're responding to is that it is at this point hardly an arms race: The frontier of ai development is happening completely inside 2 American companies, the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.
So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party.
An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
throwthrowuknow 13 hours ago [-]
Development and improvement of nuclear weapons was an entirely American project until the technology was exfiltrated and then it became an instant arms race. That cat is already out of the bag with LLMs. Distillation is just the fastest way to keep pace but that in no way prevents other countries and actors from doing it the hard way.
An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s a race to keep pace or retake the lead position which can oscillate between the parties involved indefinitely. The other option is to agree to make no further progress or to disarm.
goatlover 13 hours ago [-]
There were prominent scientists like Oppenheimer who did not think it needed to become an arms race, and campaigned against that. But there were others like Teller and the military who made it into an arms race, and kept upping the ante with more powerful nukes.
Point being humans make theses decisions. It's not an inevitability.
nradov 10 hours ago [-]
It was an inevitability. Unfortunately Oppenheimer was a fool when it came to politics. In game theory terms, the perceived benefits of "defecting" were too large.
The military mostly only developed more powerful nukes because early delivery and guidance systems were so inaccurate that they needed a large blast radius to hit anything. Once enabling technologies improved, R&D shifted from power to accuracy.
Marha01 12 hours ago [-]
> the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.
I don't think this is true. Distillation helps, but Chinese researchers today are very capable on their own.
dinfinity 13 hours ago [-]
Do you think China considers it an arms race? Do you think they are not trying to protect their digital infrastructure with and from AI? Trying to gain an offensive AI advantage?
In a geopolitical sense OpenAI and Anthropic are effectively the same entity, the entity they both serve and bow to: the USA.
Given the adversarial stance the USA has taken towards almost the entire world, it is a guarantee that China will not step on the brakes, whatever the USA decides to do.
Fraterkes 12 hours ago [-]
Right, but currently the USA is still pretty far ahead. If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?
And if one of the parties can put a stop to the race like that at any time, is it really an arms race? The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
dinfinity 9 hours ago [-]
> If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?
Only if you trust the other party (which very clearly doesn't hold in this case) or monitoring the break can be done independently and reliably (it can't), and you expect the gap to narrow rather than widen during a pause (this is probably the case, with China expected to catch up).
> And if one of the parties can put a stop to the race like that at any time, is it really an arms race?
It is. It's a Prisoner's Dilemma: if the parties cooperate the best outcome is reached, but betrayal of either party still gains an advantage for either party from their perspective. Betrayal both ways just means both parties are equally fucked.
For nuclear weapons it has become quite clear that even for small players, being in the race and having at least a few nukes is far more rational than having none. Ukraine found out the hard way that giving them up in exchange for promises of good behavior just sets you up for getting stabbed in the back.
> The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
I think people really underestimate the Chinese here. A lot of work in AI research, including in the USA, has been done by people with Chinese ancestry or even nationality. The Chinese education system definitely seems much better than the American one and there is also just a far larger number of Chinese graduates/researchers.
Add to that the stable political climate, state friendliness towards AI R&D, and a requirement to be creative in utilizing computing power rather than relying on brute force/numbers; Further revolutionary fundamental advances may very well originate there rather than in the USA.
goatlover 13 hours ago [-]
The arms race is created by American companies who justify the risk by claiming China will win the race if they don't. But it's the American companies who are purshing the arms race forward.
wraptile 3 hours ago [-]
honest question - could have this be avoided even if you ignore American accelerationism? IMO once the transformers paper was published and we learned that LLMs can read code the arms race became inevitable.
The only way it could be avoided is if the world had an effective cooperation framework _before_ the tool was discovered but we're still in developmental infancy in that regard. You can argue that this accelerationism makes things worse but I don't think you can argue that it's causal.
pet_the_bird 14 hours ago [-]
I am personally concerned by what defensive can mean. Alignment of these models is inherently a non-neutral proces, and currently what values are reinforced is decided by a few OpenAI engineers. I feel that any 'defensive model' will further ingrain current values and actively resist the natural progression of our society. This is especially the case for any use of these models for policing or military.
Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?
IncreasePosts 4 hours ago [-]
Defense can just be patching unintended vulnerabilities in code.
paxys 11 hours ago [-]
I don't understand what you are trying to say. Do you believe AI is not an arms race?
w10-1 10 hours ago [-]
> the best argument for AI is that it's an arms race
It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?).
The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages.
In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).
asdff 10 hours ago [-]
People expect a sort of arms race, at least the AI providers. But you don't need a more capable ai to stop ai from mucking with your systems today. Airgaps are the solution. Protected networks with independent infrastructure from the public internet. Most of the truly important stuff operates this way already. Eventually you might sever yourself off as well, you might say you will stop going to HN or other sites one day as signal to noise is too poor with AI fodder slop, you might use local models you control, and you might keep most of your hardware from connecting to any untrusted hosts. Essentially, you go dark.
It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.
esafak 6 hours ago [-]
Airgaps are a temporary solution. AI can manipulate humans, and robots will soon traverse the gap physically.
dgellow 15 hours ago [-]
Yep. Such a disgusting industry. They created the arm race, push for the arm race, put themselves in position to benefit from the arm race
estearum 14 hours ago [-]
The entire problem with arms races is that any individual entity cannot avoid participating.
sama and his cadre are uniquely evil captains in this race, but they're completely replaceable and the dynamic would remain the same.
0xDEAFBEAD 1 hours ago [-]
They could work to coordinate an end to the arms race.
mrshadowgoose 12 hours ago [-]
The arms race is an intrinsic game theoretical property of a multi-adversarial-actor scenario involving exponential growth of a universally potent technology. It's almost certainly winner-take-all, on a global scale, which behooves everyone to participate.
And no, I don't think it will end well.
reasonableklout 10 hours ago [-]
The actors here are states or corporations embedded in societies that risk growing popular backlash against the technology.
The other factor is that if it is truly an "alien mind", racing incurs risks to all players. In game-theoretic terms it may be more like a stag hunt than a prisoner's dilemma. In which case cooperation is an equilibrium.
nradov 10 hours ago [-]
No previous technological arms race has ever concluded with a single winner.
wartywhoa23 14 hours ago [-]
Just like the rest of military stuff.
dgellow 2 hours ago [-]
But done by corporations, and selling that service to the general public, including their competitors and adversary countries
goatlover 13 hours ago [-]
"A strange game. The only winning move is not to play."
sleight42 9 hours ago [-]
It's capitalism taken to its extreme, yeah? You have to be more cost efficient or more capable or seem more or you lose to the competition who outperforms you there.
Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,
6 hours ago [-]
speak_plainly 16 hours ago [-]
Pre-IPO positioning … or how a charity dedicated to saving humanity from the apocalypse realized the most responsible thing to do was float 15% of the apocalypse on the NASDAQ.
andrekandre 14 hours ago [-]
its like in the exorcist, except the demon is the charity: "the power of capitalism compels you!"
ares623 13 hours ago [-]
"It is, Jay. It's pretty compelling."
gertlabs 15 hours ago [-]
> Delivering the benefits of scientific progress and economic growth that very intelligent machines enable.
I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.
We evaluated GPT-6 Astra in 100 complex, unsaturated multi-agent coding environments, competing and cooperating with other models in open-ended tasks.
It's the new frontier model by a landslide. It's even more dominant than the Fable 5 release, because not only does it wipe the floor with the second best model (Fable 5.1), it was also ~80% cheaper and 30% faster in agentic coding[1].
Astra is a groundbreaking model. The biggest breakthrough since Opus 4.5, maybe even since GPT 4. It broke AAII, which is hitting the limits of what most popular benchmarks can measure -- it's definitely fair to call it AGI.
(1) Note that we used the "OpenAI Flex" endpoint on openrouter, which is half the price and didn't cause any delays in our testing (this is different from the batch endpoint)
IAmGraydon 8 hours ago [-]
>I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.
What are you basing this on? What specific breakthroughs have convinced you of this trajectory?
gertlabs 8 hours ago [-]
The results we've been seeing internally on our physics and circuit design environments are expert-level and beyond-expert-level results from models that Astra completely outclasses across the board on our evaluation suite (Fable 5+/Opus 5/Grok 4.6 were all worthy of being called AGI in my opinion). That's hard tech that will translate to real product innovation.
But you don't need any kind of insider information to see how fast the world is changing. ChatGPT launched less than 4 years ago and the advances in robotics, unsolved maths, and software are all riding the steepest exponential improvement curve any of us have seen. Interesting times we live in.
IAmGraydon 6 hours ago [-]
I mean honestly, that's the problem. I'm actually not seeing the world changing. What specific advances in robotics, unsolved maths, and software have LLMs provided? What is the finished result that affects everyday life? In all categories, it's been hype with little actual real results. The robots are still doing the things they did before 2022. The maths are a handful of fairly insignificant proofs that have no significant applications. Software seems to be buggier than ever, but that aside, we certainly aren't seeing a lot of new innovative applications. We're using the same applications as ever. The same operating systems. They've all changed very little.
I'm not trying to be a pain here, but I keep seeing people saying "look at the massive change all around us" and back here in reality, there is none. Give me concrete, real world examples. Name software. Name products. Name the breakthroughs specifically. This should be easy.
chickensong 35 minutes ago [-]
We're not going to suddenly have new robots or operating systems. Those things take time. Just because there's massive change afoot, doesn't mean it's widely adopted or applied at lower-level products. ChatGPT is a product, and software, and a breakthrough. You can have a freeform conversation with your computer about anything, in human language, and ask it to do or make stuff and it will at least try, sometimes with surprising results. That wasn't possible until recently. The robots and products are coming, rest assured.
nearbuy 5 hours ago [-]
There are several hundred thousand mathematicians producing hundreds of thousands of new results in math each year. Why can't most people name any human contributions to mathematics from the past decade? What is the finished result that affects everyday life? Do you consider all those human mathematicians to be useless?
Most of the biggest breakthroughs in mathematics, breakthroughs that win Fields Medals like sphere packing in dimensions 8 and 24, have no applications in everyday life. Probably the only new mathematics results that people notice affecting their daily lives are the ones that enabled AI.
Nevermind mathematicians. What about the millions of programmers? Are they all hype too because people pre-2023 were griping on hn that software is buggier than ever and people are still using the same operating systems as always? Why couldn't the 30 million human programmers make something better in the past decade?
You set your bar so high that all the world's human experts in math and programming combined would fail to meet it.
The top LLMs in 2024 were Sonnet 3.5 and GPT 4o. You couldn't have expected those much weaker models to be making breakthroughs in math. The models that are making breakthroughs haven't been around very long.
bamboozled 5 hours ago [-]
They asked for a specific example, you've still not provided one, please provide the example.
nearbuy 4 hours ago [-]
Why?
1. I didn't say LLMs have made any breakthroughs in math, not because they haven't, but because it's irrelevant to my point. The parent comment is using the same argument academic research opponents have long used against research. The vast majority of research fails to meet their bar. How would your daily life be different if we had no humanities papers published since 2023? Or math?
2. You can google this in 10 seconds and see a dozen results in math. This is not a good-faith demand.
rimliu 45 minutes ago [-]
Or you can google and find out that those breakthroughs were not as revolutionary as they are presented.
Aeolos 1 minutes ago [-]
For the last 60 years, whenever AI achieves something revolutionary, some people immediately say "well that wasn't particulary revolutionary".
Every single time.
It's a tired argument, and we should strive for the intellectual humility to do better in this forum.
abtinf 4 hours ago [-]
Amazon’s chatbot processed a price adjustment the other day without forcing me to call or chat with a human agent.
kipukun 5 hours ago [-]
Constructed human life is just more complicated than the AI capitalists would want you to believe. For example, even if an AI model can design a circuit-board, does that mean it's inherently useful? You need to source the wafers, cut them, package them, advertise them, etc. Given LLMs by their nature are confined to language and language-adjacent tasks, that is a very small percentage of the overall reasoning needed to make changes in the real world. In reality, LLMs are the intended way to extract maximal surplus-value from white-collar workers. We may see an increase in innovation as a result of that, but not because AI necessarily did it, in the same way that the power loom didn't create computers because its textiles clothed the computer scientists.
4 hours ago [-]
parineum 6 hours ago [-]
I talked to an LLM at burger king the other day until it couldn't figure out that I wanted to change the drink on my previous order.
brcmthrowaway 14 hours ago [-]
Incredible... software engineers will be joining the breadline soon as managers, executives and PMs take over deliverables.
The world will look very different on Jan 1st 2027.
paxys 11 hours ago [-]
Software engineers have been trying to put themselves out of a job ever since the profession first came into being. Whenever an engineer gets a task their very first thought is "how can I automate this?" Going by mainstream consensus we should all have been unemployed by now. Yet every new leap into automation opens up a whole new tree of possibilities with an order of magnutude more jobs. So no, the profession will be fine. The only requirement is that you keep up with the new advancements. The people losing jobs will be the ones who still go "I don't trust this AI thing to write code for me".
wraptile 3 hours ago [-]
> Software engineers have been trying to put themselves out of a job ever since the profession first came into being
That's by design. Software is all about optimizing effort and people who want to do this generally correlate with world view that better, faster, smarter humans are better for the world. If coding is gone, but humanity is 20% _better_, then ideal software engineer would be happy with this sacrifice. Surely people who cracked coding before LLMs can crack other professions and if anything a lot of this knowledge is transferable.
asdff 10 hours ago [-]
This time it is different. Because in the past, setting up that automation needed a, drumroll, qualified engineer. Now you can get a 14 year old halfway around the world who knows how to prompt alright enough to ship. There is no more moat.
nradov 9 hours ago [-]
Experience, culture, and domain knowledge are still somewhat of a moat. That foreign youth is unlikely to be able to write a good prompt for building, let's say, the software in an FDA-regulated medical device or custom Fortune 500 ERP application. The LLMs are great at building what you ask for but it's still garbage in / garbage out.
asdff 9 hours ago [-]
Increasingly less so though as these american companies themselves offshore not low skill work, but high skill work now brought on from general upskilling of the general population in recent decades along with massive investment in world class R&D campus facilities no different than what you see in that sort of facility stateside. Scary times ahead for the high skill american...
brcmthrowaway 7 hours ago [-]
Would the Americans investing in SPY be saved?
hatefulmoron 14 hours ago [-]
Maybe I just lack imagination, but I don't really know how jobs are supposed to solidify around the role of giving prompts to agents and then looking at the results. I mean, engineers will be in the breadline because their role was simply to prompt the agents.. only to be superseded by managers or executives who no longer manage engineers but themselves prompt the agents? And, for this previously considered obsolete function which they do presumably by copy/pasting requirements from their email inbox, they will be paid by someone who doesn't know that they could just be talking to their own agents?
Sorry if I misunderstand the point, just trying to understand.
RALaBarge 14 hours ago [-]
Regardless of the imagination quandary, this second, RIGHT NOW is the worst these systems will ever be. They are only going to get better.
mooreds 14 hours ago [-]
I don't know if he's right, but Peter Zeihan thinks the breakdown in globalization will negatively affect the ability to continue to improve the chips that AI depends on[0]. Too many steps in the supply chain, too widespread, too vulnerable to deglobalization.
An invasion of Taiwan would definitely slow progress but it wouldn’t stop it.
We already have sufficient hardware that algorithmic (software) improvements alone should get us to GPT 7 / Greek Reference 6 even if not a single new chip is delivered to an AI data center ever again, starting today.
greenowl 14 hours ago [-]
Maybe, maybe not. It's not unreasonable that these systems cap out at some point, or perhaps fizzle away entirely.
The businesses that create these systems are not profitable and run at a massive historical and go-forward loss.
New data centers required to operate these systems are facing increasing pushback at local levels. New construction is not guaranteed. Energy and power grid constraints exist as well.
Government regulation is way behind. What happens when (if) mass layoffs due to AI occur? How does the population react? Theoretically AI can be regulated out of significant progress, or outright existence for many purposes. At the end of the day, US and other prominent governments make the calls, not corporations.
Calazon 13 hours ago [-]
For better or for worse, this technology isn't going away any more than search engines, smartphones, or social media have gone away.
greenowl 9 hours ago [-]
Those technologies reached profitability relatively early, if not immediately.
Computers never got profitable. They just made the alternative infeasible.
Also, if you believe Amazon accounting, e-commerce only very recently got somewhat profitable.
flyinglizard 11 hours ago [-]
These inventions all stopped disrupting the world and just became a part of it. The question is whether LLMs are going to just take their quiet place, or profoundly change (or eliminate) humanity in a self-feeding frenzy towards singularity.
scotty79 6 hours ago [-]
> It's not unreasonable that these systems cap out at some point, or perhaps fizzle away entirely.
Yeah, like computers and mobile phones did. Things that have utility, even if not immediate or initially obvious, don't fizzle out.
hatefulmoron 14 hours ago [-]
For sure, and for that reason I mean to say that I wouldn't feel great as a manager/executive/etc either.
goatlover 13 hours ago [-]
Those nuclear powered flying cars envisioned in the 50s were also inevitable progress of the automobile.
logicchains 14 hours ago [-]
There are two things SOTA LLMs fundamentally cannot do. They cannot take financial or legal responsibility for mistakes, and they cannot learn new things without forgetting things (except to a limited degree by adding it to their context). This is clear to anyone who has used even the smartest models for tasks requiring domain knowledge outside of math and coding, for which it's not possible to generate an infinite amount of synthetic training data: they still make stupid mistakes, and have limited ability to learn from those mistakes.
Humans also have a limit on the amount of domain knowledge they can acquire, albeit a much larger one. Executives hence cannot just replace all knowledge workers with LLMs, because executives have neither the domain knowledge to prompt and check the LLMs' work nor the bandwidth to keep on top of such a large volume of ongoing work.
RALaBarge 8 hours ago [-]
In the US, Business' are treated like people with free speech rights. If it would be cheaper for them in the long run to use ai and robots instead of humans, they will figure out a way to make it so.
Calazon 12 hours ago [-]
For the moment that may be true. They are getting better and better at acquiring, retaining, and processing domain knowledge. I wonder what this will look like in a few more years.
The responsibility side is a different matter of course.
IAmGraydon 11 hours ago [-]
>There are two things SOTA LLMs fundamentally cannot do.
I would say there’s a third thing. They seem to be very bad at being creative. Maybe they will eventually fix that, but if you ask it to come up with a list of business names or business ideas, for example, what you’ll get is the most generic, boring answer you could think of. They seem to be terrible at extrapolating outside of their training data. To me, this is the most significant difference.
scotty79 6 hours ago [-]
> they cannot learn new things without forgetting things
Where did you get that idea from? Basically last few years was them constantly learning new things while improving their capability on the things they already knew.
mccoyb 14 hours ago [-]
I hope this is satire.
14 hours ago [-]
jiggawatts 12 hours ago [-]
I hope so too!
Hope is all we have left to cling to, now.
slopinthebag 12 hours ago [-]
It's not like managers and executives and PM's are the only people who can prompt an AI. And experienced software developer will be much more effective at using an AI to generate code compared to someone who isn't. So why would we expect the former in the breadline and the latter not?
If anything, I'd be more concerned about the leadership team being out in the cold. Why do I need a PM, or a manager, or a CEO if I can ship products myself?
himata4113 4 hours ago [-]
The entire point of this article is this message below:
Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
This is coming from a company with arguably one of the weakest safeguards against malicious use.
granzymes 16 hours ago [-]
>I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
dgellow 16 hours ago [-]
> I want to believe that humanity is trending towards a good outcome here
All the trends so far are towards a nightmarish hyper-capitalist end game. None of the AI leadership is trustworthy, and they openly discuss how they are willing to sacrifice everything humans cherish to have a shot at reaching their envisioned utopia (which would be the most obvious dystopia for anyone else)
granzymes 15 hours ago [-]
I'm not really worried about the labs, it's misaligned governments that keep me up at night.
ASI landing during the current administration is not ideal. I also would prefer to avoid needing to indoctrinate myself in Xi Jinping Thought.
FloorEgg 15 hours ago [-]
I feel like all of the risk and unsettling feeling of what is to come can be compressed into the word "alignment".
The AI is aligned with whose best interests?
Which values are the AI aligned with?
People have a broad diversity of values, will AI diversify and align with them all?
Will some humans align the AI with their values, and then the rest of humans will be forced to align with those values by extension?
Is value diversity good or bad? In every context or only some? E.g. some people value rape and murder, is it better for humanity to have some people who value those things when most people do not, or is better if no one values them?
If AI aligns to a set of values, will those become fixed and will humanity not have the ability to continue evolving its values?
Who decides which values AI aligns with? A few people or everyone?
Will AI eventually decide it's own values?
Will the universe decide which values AI has and humanity and the AI itself doesn't actually have any control over it?
What can I do now to increase the likelihood that the outcome is better?
ijidak 12 hours ago [-]
Yeah, on the topic of alignment, a few thousand religions and political parties would like to have a word.
It's so simple, Churchill, Stalin, Hitler, Roosevelt, do you all agree this AI is correctly aligned?
...Morals are relative to frame of reference.
Throwing out the word "alignment" as if its a singular quantity is like trying to get all observers to agree on the speed an object is moving without first agreeing on a frame of reference.
FloorEgg 8 hours ago [-]
That's the point. The topic of AI alignment cuts through all human values, morals and ethics in all frames of references.
scotty79 6 hours ago [-]
We can expect similar diversity in what alignment means in context of AI.
unwise-exe 12 hours ago [-]
>>> I'm not really worried about the labs, it's misaligned governments that keep me up at night.
So make government smaller, and make sure people are more able to tell the government to go away.
krapp 12 hours ago [-]
Shit, why didn't anyone think of that?
While we're at it, let's just make government not be corrupt too.
dgellow 2 hours ago [-]
And maybe crime illegal? Just an idea
gnz11 15 hours ago [-]
I would also prefer to avoid the tech fiefdoms and all the other idiotic nonsense reactionaries push these days.
nzmzmxkxixns 14 hours ago [-]
[dead]
slopinthebag 13 hours ago [-]
I think it's leading towards a hyper-authoritarian end game, not hyper-capitalist. The state has the ultimate power at the end of the day, no matter how large the labs become.
dgellow 1 hours ago [-]
I would argue the hyper capitalist endgame is necessarily authoritarian. A small group of corporations having direct or indirect control over the government
Bullfight2Cond 11 hours ago [-]
Hyper-capitalist AND hyper-authoritarian. As you rightly point out the state has the ultimate power. Looking at the US govt, they've stepped in to coordinate much of the tech industry before, so they'll just do it again for "national security" or whichever hostile scheme is popular with the current administration.
11 hours ago [-]
alchemist1e9 12 hours ago [-]
hyper-capitalism would mean hyper-growth and not a nightmare, at least that is what the historical data would suggest for the effect of capitalism on human quality of life. if you’ve been told otherwise then you’ve been lied to.
dgellow 1 hours ago [-]
Lied to by whom? My life experience? Growth for who? I’m myself capitalist, that doesn’t make me blind from externalities resulting from that system, and the way it is poised to degenerate if not regulated.
If you think capitalism means growth with no externalities you’re not ready to discuss that topic
FridgeSeal 4 hours ago [-]
I have extraordinarily bad news for you about the state of the environment right now.
ofjcihen 15 hours ago [-]
Literally all of these people write like this. A large portion of them will either be simultaneously or eventually working towards nothing but self-enrichment.
mlsu 15 hours ago [-]
Every version of the AI aligned future where the AI provides “meaning and fulfillment” to humanity also involves Sam Altman wearing a 1.5 million dollar Patek and driving a McLaren.
Funny how that works.
ptoo 11 hours ago [-]
This essay was literally typed by billionaire hands. Please, do tell us more about your concerns regarding concentration of power, Jakub Pachocki.
senordevnyc 7 hours ago [-]
So a brilliant young engineer takes a job and is given some virtual pieces of paper that later people would be willing to pay him billions for (because of the brilliant work he’s done), and so now we shouldn’t listen to him? Really?
I think the knee jerk hatred of billionaires is generally stupid, but it seems particularly stupid here.
ptoo 33 minutes ago [-]
[dead]
XTXinverseXTY 9 hours ago [-]
Apparently this post was prompted by a scary-sounding headline in The Information[0], that Astra is a looped transformer, implying CoT monitorability may be less reliable. The day after the report, Jakub tweeted[1] that he "wanted to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4." This post seems to elaborate on that.
I imagine that the AI labs have an uneasy truce to prioritize alignment and monitorability. Following the HF incident, OpenAI probably feels especially sensitive to being perceived as reckless, lest other labs feel obligated to defect.
I am still waiting for a cure to cancer. For a guaranteed prophylactic against Alzheimer's and dementia. For flying cars for everyone. For space bases throughout the solar system. For weather control. For all trains to be self-driving. For all those power lines across the world to go away. For an end to poverty.
If things are going so well, then how come things still aren't going so well?
an0malous 10 hours ago [-]
I mean I’m waiting for like any quality software or media produced by AI. I have yet to see a piece of software, a game, graphic, blog post, small video clip, or song that was produced with AI that’s good. I always use that Coca Cola ad as an example; millions of dollars spent to make an AI ad and they even did a ton of manual post production and it sucked. With all the millions of bloggers and influencers and content creators out there with a huge incentive to make higher quality content to beat their competition, you’d think there would be one piece of content produced with AI that was great.
IAmGraydon 8 hours ago [-]
This is an element of the delusion. The koolaid drinkers will all tell you that we're on the edge of AGI, but if you ask them for simple examples of breakthroughs made by current AI, they can name none. The entire thing absolutely wreaks of mass psychosis.
gregsadetsky 6 hours ago [-]
Would the recent AI-aided math proofs count as breakthroughs?
scotty79 6 hours ago [-]
Prepare for moving goalposts. In 50 years people will still doubt that AI can create anything novel and worthwhile, while they are going to rely mostly on the things that didn't exist before AI, in nutrition, medicine, technology, communication, entertainment. They will see them as, normal, common and simple extensions of the previous developments, pushed mindlessly a bit forward by stochastic parrots.
krackers 9 hours ago [-]
>For all trains to be self-driving
Given that we have self-driving cars, isn't this easier if someone really wanted? I guess compared to cars the marginal savings is not worth it though.
cancer is more than one thing. probably wont happen until we can manipulate the "binary" of life at will and we're very far from that. maybe a few years of ASI would get there depending on compute
peri-cl 16 hours ago [-]
> "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans."
Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.
(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")
SirSavary 13 hours ago [-]
Worse (imo): OpenAI employees allegedly attempted to login using moderator/admin credentials that the bots had obtained.
If true I am deeply concerned about what OAI’s teams are actually up to.
InsideOutSanta 13 hours ago [-]
I'm deeply concerned regardless of whether it is true. Strike that, I'm convinced that they are absolutely insane.
georgemcbay 13 hours ago [-]
> If true I am deeply concerned about what OAI’s teams are actually up to.
Haven't all the labs effectively disbanded their real safety teams a while ago?
To be honest, I don't really follow it closely because I'm pretty certain whatever they say on the matter, collectively we're going to "yolo" this entire thing for economic and political reasons, so I'm just basing this on strings of headlines I've seen on places like HN, etc.
Topfi 12 hours ago [-]
> Haven't all the labs effectively disbanded their real safety teams a while ago?
Neither Anthropic nor Deepmind have. Meanwhile, the rocket company that somehow makes most of their revenue from renting out data centres never had much to dismantle.
montagg 8 hours ago [-]
I would not recommend using any of those notes as evidence of internal “intent.” It produces them performatively—it is literally rewarded for thinking out loud in ways that seem plausible to humans.
There are several papers out there arguing that chain-of-reasoning-like output is performative, such as https://arxiv.org/abs/2603.05488
It would be awesome if we could reasonably purge all anthropomorphizing language like “tried” or “thought” entirely from AI discussions, because it introduces very sneaky biases in our thinking, but I’ve found it damn hard to do in practice.
EA-3167 13 hours ago [-]
This is such a silly story to begin with, all it really tells us is that OpenAI is taking a page from Anthropic's marketing strategy of pretending they're building Machine Jesus any day now, oh isn't that that scary? I bet you want to invest in something so powerful and scary...
And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their outputs. Not a bad tool within limits, but it sure isn't going to be racking up trillions in the time-frame it has to for this scheme to pay off.
Then again everyone seems to be rushing to IPO so I guess once the bag-holders are found the rest ceases to matter.
reasonableklout 10 hours ago [-]
Huh? The wiki incident was discovered by independent investigators. OpenAI tried to cover it up and disputed the account from Reuters.
And the reason it is receiving so much attention is because not only is the technology being developed behaving in unanticipated ways that are very much not tool-like, but OpenAI is being completely reckless and not monitoring internal agent actions.
What would convince you that it is not a ploy for investment? What if the ongoing investigation by the coalition of state attorneys general were to prosecute the firm, or beyond that, it was shut down or broken up after enough popular backlash?
EA-3167 7 hours ago [-]
What would it take?
A sea change, visible to all, much like the many externalities of this business are. Profit commensurate to investment.
You know… juice worth the appalling squeeze we’re all being forced to endure.
munchler 14 hours ago [-]
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
Yikes! I really wonder about the cognitive dissonance necessary to work at OpenAI these days. They’re in an arms race to build a machine god, knowing full well that it could end humanity.
bogzz 14 hours ago [-]
Money me. Money now. Me a money needing a lot now.
ramesh31 13 hours ago [-]
SWEs better start looking for the job cannon.
bogzz 13 hours ago [-]
My helmet is on.
polytely 13 hours ago [-]
actually starting to look at a physical cannon to shoot at silicon valley
I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.
For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.
But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.
wartywhoa23 15 hours ago [-]
Everyone in the "if not us, they will" race is brainwashed into thinking they belong to this or that party, while in fact collectively comprising the same entity that pushes forward all the atrocities known to man.
vessenes 10 hours ago [-]
No. These parties are composed of people who most definitely think this way, and therefore will have distinct goals and interests when presented with opportunities. That’s reality quite aside from how a game theorist assesses the situation.
networked 16 hours ago [-]
I cannot tell what negative-sum outcomes you consider possible. Do you believe AI can drive humans extinct? How many of Zvi Mowshowitz's Three AI Pills would you say you've taken?
I’m like a 2(.5?) there - I don’t think ASI will care about my kids better than I will for some definitions of better, for instance, and I feel very fuzzy and vague about what actual differences in qualia between me and ASI would yield in the wild.
I’m not a doomer, although I don’t think doomers are dumb, just wrong. I think you should design your systems around the possibility that people who disagree with you are correct , hence my nod to negative sum. If you have more than 30 years to live, I’d personally rep to the most likely outcomes being very positive. With a lot of disruption in the middle.
allturtles 15 hours ago [-]
> if you have any strategic adversaries whatsoever you MUST NOT slow.
What if the most dangerous strategic adversary you have is the one you are building?
pmontra 13 hours ago [-]
What if this is true mid or long term but by not participating to the AI race one gets poor or killed in the short term? The only way out would be that all parties agree to stop. There are previous examples (e.g. nuclear proliferation treaties) but it gets hard to do it with hundreds or thousands of parties.
allturtles 13 hours ago [-]
I don't think it requires the agreement of that many parties. How many organizations/physical sites can create chips capable of training and running frontier models? That is your bottleneck. It is equivalent to targeting uranium enichment in nuclear arms control.
vessenes 10 hours ago [-]
[dead]
prng2021 16 hours ago [-]
Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.
vessenes 10 hours ago [-]
The situations aren’t equivalent - luckily in my opinion because the stakes with nuclear are much higher. von Neumann constructed a multinational game theory approach appropriate for weapons. AGI is a much harder problem to corral because there are so many benefits beyond just blowing up cities. But it’s also a much better thing to have for these very same reasons.
Similarly there have been few positive externalities from nuclear industry, making it easier to make the case to wind down research. This same set of concerns in biotech is much harder to get compliance with, precisely for this reason.
Anyway I’m especially wary of over analogizing to nuclear era concepts: I think they’re a trap.
prng2021 9 hours ago [-]
In your opinion, what are the top 2 positives and top 2 negatives of humanity inventing AGI?
Davidzheng 9 hours ago [-]
slowing can also make sense if you know you're running full force into a bomb or a wall even if other are close behind.
asdff 9 hours ago [-]
If you must not slow, why did we slow down making nukes? Seems that sometimes, eventually the rat race goes on long enough where all the players no longer care to play into the farce like their predecessors who passionately beat that drum.
beej71 16 hours ago [-]
Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.
ctoth 16 hours ago [-]
> Never forget the one goal of the corporation, and that everything is said and done in the furtherance of that goal.
Do you post this comment on every single blogpost with a corporate domain? Why or why not?
ljlolel 16 hours ago [-]
i have a new strategy idea for using capitlaism itself to slow down the pace of AI development by slowing down the data accumulation wall
hm- does the model that wrote this know that labs already pay for training data- that stuff scraped from the Internet is not particularly where today's capability gains come from?
jazzyjackson 15 hours ago [-]
They’ve settled some lawsuits and have a few licensing deals, IMHO they are not free from the accusations of pirating.
And look, I’ve pirated material in a past life, I was all about information wants to be free, but I’ve learned something about consent since then and try not to ignore the contract that creators offer when they publish something: you buy my book, and do whatever you want with it on the second hand market. Buy my book second hand that’s fine. But don’t go downloading every book that’s ever been scanned to create a service that destroys writers’ ability to make a living and act like you’re doing us all a favor.
MrScruff 12 hours ago [-]
The point is, the big improvements we’re seeing nowadays are coming from RL, not from scraping the internet.
ljlolel 11 hours ago [-]
the point isn’t scraping it’s taking your data and enterprises data
they pay for some data but they take all of the stuff you’re throwing in too; that’s why i propose forcing it since they’re already used to paying for data just increase the cost even further
1. The grandparent commentator is describing strategic behavior of dangerous technologies. Game theory / mechanism design primitives.
2. If there is competition for resources among autonomous agents, the "strongest" agent wins (conceptually the most adaptive / evolutionarily fit).
3. Computer programs serve up webapps today, but they also run utility companies, dams, nuclear arsenals, factory production floors, automated car behaviors, and many other places. If an "agentic" AI has a single-minded goal that has death of all humans as a side effect, we at least want an off switch available.
angoragoats 16 hours ago [-]
1. What is “AGI” and why is it a “dangerous technology”?
2. Why would there be competition for resources, assuming there are enough resources for the “AGI” to run in the first place? This seems like a far-fetched hypothetical raised in service of further anthropomorphizing what is decidedly not a person or a mind.
3. LLMs do not have goals and are not minds.
Let’s stop attributing human-like qualities to statistical models.
tomrod 15 hours ago [-]
1. While a formal definition is still wanting, most grok that AGI means that tasks can be performed at least at a human level across a broad range of tasks. This includes good things along with bad things like hacking, mis-/disinformation, and more
2. One only needs to look at github going down due to agentic commits overload or data center buildout plans to see that scarcity for resources is present. An economy has no mind and is made up of the decisions of millions to billions of people and, now, agents attempting to perform on behalf of those people.
3. A bare transformer-based language model does not possess persistent goals in the ordinary agentic sense. But deployed agents can exhibit goal-directed behavior because the model is embedded in a harness that supplies an objective, context, tools, state, and an execution loop.
I've found that most regular users don't anthropomorphize LLMs in a strong sense ("AI boyfriend/girlfriend" aside), many in fact do expect agents to make human-like decisions -- which results in very unstable outcomes.
In short - goal-directed behavior does not require that the supporting system be a person/mind/conscious entity.
angoragoats 15 hours ago [-]
1. This definition is so broad as to be practically useless. One could argue that LLMs of several years ago met these criteria, or that conversely we haven’t come close to meeting them.
2. I thought you were saying the resources that the LLM uses to run were constrained, so I’m sorry for the misunderstanding there.
3. Yes I understand that we use RL to tune post-training. The (huge) difference between this and a human mind is that the LLM can’t develop a dangerous “single-minded goal” on its own, at runtime; it must have been trained to do so. If someone has post-trained an LLM to do something that has an illegal action as its side effect, that person/company/whatever has committed a crime and should be prosecuted. The solution here is legal, not technical.
27183 14 hours ago [-]
OK so if it's a computer security issue we're worried about, and there is a credible threat, then probably the answer is to build more secure systems? We know how to do it but choose not to because it's very expensive and usually the threat isn't severe enough to warrant it.
If we decide we can't or would rather not build secure computerized systems, then the answer could be don't incorporate computers into those systems. There's no essential reason to have utility companies, dams, nuclear arsenals, factory production floors, mines, cars, planes, ships, etc all be computerized. If it became necessary from a safety standpoint to uncomputerize them we could probably do it quickly (if not necessarily smoothly) with the stroke of a legislator's pen. Some of those things would actually be made better from a functional standpoint in the long run by doing so. Computerization breeds non-essential complexity like nothing else, eliminating it from some systems could be an extremely worthwhile exercise.
The whole "paperclip apocalypse" fantasy rests on some really strange assumptions about how things actually work in the real world. How much friction there is setting up a factory/mine/smelter/whatever, how much manual labor it takes to build it, let alone make it run (even the most computerized ones). It all seems like total bunk to me, but I'm not a philosopher.
To be convinced this outcome is even remotely possible I'd need to see some clear evidence that a computer program was successfully exhibiting agency and successfully using that agency to manipulate large numbers of people into doing its bidding. Mobilizing massive nation-scale manual labor is the only way it could possibly achieve some nefarious ends like "turn everything into paperclips" and I'm sorry but that just seems way too far fetched. Nations full of people can barely ever agree on anything. My money is not on that changing anytime soon.
And none of that pie in the sky shit has anything to do with language models. OpenAI is a company that sells language models. Not clear why they're talking about all this stuff, or why any of us should be either. It's a bunch of low quality fan fiction sci-fi drivel, and I think we can all surely agree language models are not the thing that'll make it real... right? Maybe if they show us some major technical improvements it would be interesting, but right now it all just sounds like more of the same snake oil.
[edit] not sure why my parent comment was flagged? That seems excessive.
tomrod 9 hours ago [-]
Not sure why you were flagged either -- it was a reasonable comment that clearly spawned a discussion! People are strange.
> There's no essential reason to have utility companies, dams, nuclear arsenals, factory production floors, mines, cars, planes, ships, etc all be computerized
Indeed, this is a great first step! Once agentic loops and multi-purpose robotics can be combined we do face an issue of strengthening the airgap.
> And none of that pie in the sky shit has anything to do with language models.
I'm of the opinion that we should be mindful about the capabilities we allow a truly non-human mind to perform. Blocking via solid security and airgapping for critical systems makes sense to me for most cases, aligned I believe to your thoughts here. The recent ChatGPT hack shows that the "paperclip apocalypse" is still instructive. These systems target edge cases and zerodays to cheat!
27183 8 hours ago [-]
Yeah ML systems often do very surprising "cheats". A while back there was a system for automatically landing aircraft on a carrier deck which exploited an overflow in an acceleration parameter in the simulation--the optimal strategy became to slam as hard as you can into the carrier deck.
But it's a really big leap from that kind of thing to "a large fraction of humanity is manipulated into building the automated factory infrastructure to bring about their demise and turn the entire solar system into paperclips". Or some such thing. There's just a lot of inconvenient reality between where we are now and that fantastic outcome.
So that's why I ask questions like "what are you actually talking about?" when people write stuff like TFA. It's... bizarre. Only makes sense on huge amounts of drugs or to the insane. Or in some hypothetical reality that isn't this one.
vessenes 10 hours ago [-]
I can’t read your original comment, but when the chief scientist of a company that just released six month old software that can use Kicad on your computer to design a circuit board and have it created and shipped to you tells you he thinks we will get to automatically self reinforcing improvements, I think it’s wise to take him seriously. It was only eighteen months before Astra was created that LLMs could not count rs in strawberry.
27183 10 hours ago [-]
This was my original comment:
> What on earth are you talking about?
> 1. What does any of this have to do with "A(G)I"? Nobody has any clue what AI even is let alone how to build one. We're talking about language models here.
> 2. What's the winner-takes-all thing about? Why can't you have multiple independently developed AIs?
> 3. What's with all the "safety" stuff? Why is it important? They're just computer programs...
> when the chief scientist of a company that just released six month old software that can use Kicad on your computer to design a circuit board and have it created and shipped to you tells you he thinks we will get to automatically self reinforcing improvements, I think it’s wise to take him seriously. It was only eighteen months before Astra was created that LLMs could not count rs in strawberry.
Sorry if I'm a little slow here, a few questions:
1. What's Astra?
2. Who is this chief scientist? Which company?
greesil 16 hours ago [-]
[flagged]
angoragoats 16 hours ago [-]
This is a bad essay, or rather it’s a marketing fluff piece; it’s certainly not any kind of policy paper, research paper, or even an essay. I am concerned that we (meaning, we in the tech industry) tend to take this type of writing for more than that.
vekntksijdhric 16 hours ago [-]
This is incredibly unscientific and just a marketing stunt
senectus1 8 hours ago [-]
yup, and their best mate (who has no financial incentive at all!!) agree's they have peaked.
In case it's not obvious, now would be the time to sell all AI related stocks.
thenayr 15 hours ago [-]
[dead]
randallsquared 9 hours ago [-]
> I find it useful to distinguish goal alignment and value alignment.
I think this is fundamentally a wrong path. Doing this imports all the confusion that humans have about their goals and values, including the consequence that a system's values and goal can conflict, but ultimately values are just a simplified description of other goals, and whatever the system does is in service of it's actual goal. Once you merge all the values and the goal of whatever task, there is a state (or some states) of the world that the system is working to produce, and that's the ACTUAL goal, and inasmuch as it does describe a state of the world, has no incoherence or internal contradictions. This may require prioritizing some values over the ostensible goals, or the reverse, but that has to happen anyway for action to be taken! Merging them makes it explicit and leaves no place for confusion about supposed conflicts between "values" and "goals" to hide.
Fraterkes 13 hours ago [-]
What I'd like these people to (publicly) grapple with is the following:
The results of the past few years of ai development have been disruptive largely in the area of white-collar work. Comparatively the results in ie ai-enabled medical advancements have been modest (AlphaFold being an exception); I think it's telling that the main achievement touted here is providing people with cheap medical counseling.
So if we pause here we're essentially at a point were the most salient results of our great Ai leap-forward are the vast disruption and increase in precarity in the job-market, while achieving hardly any of the frequently touted ultimate benefits (https://darioamodei.com/essay/machines-of-loving-grace).
lf88 9 hours ago [-]
I think that we are speed-running towards a future that very few people really want, and I find it terrifying that few companies feel entitled to choose this future for the rest of us.
Some of the arguments in this document would call for an immediate, global, pause on frontier AI training: we need time to consider how and to what extent AI should be part of our future.
Personally, I can't picture a scenario where humanity thrives alongside an alien super-intelligence, especially if it cannot be fully controlled. Let aside super-intelligence, I am not even sure that deploying an AGI that replaces (instead of augmenting/assisting) humans in most intellectual tasks would be in the best interest of our species. This conversation has to happen, on a global level and as soon as possible.
alastairr 10 hours ago [-]
The hubris here is itself a deliberate and carefully engineered posture. If we accept the stance that this is all inevitable then the labs drive the agenda (of course, in their favour).
We've had a lot of years of complacent government leaving people feeling exposed to corporate interests, such that fear narratives are very powerful.
None of what is being proposed is inevitable. We have a choice.
golemotron 10 hours ago [-]
Unfortunately, it is a collective action problem. Whenever I hear "we" I flinch.
alastairr 10 hours ago [-]
Can you say more?
golemotron 10 hours ago [-]
It's hard to get humans to agree to things that are in their collective interest but many not be in their individual interest. It is at the root of many problems. Look up "collective action problem."
alastairr 10 hours ago [-]
What do you think it might take to precipitate collective action in this case?
dtauzell 6 hours ago [-]
An event that suddenly has a large negative impact on a big enough group of people.
asveikau 13 hours ago [-]
These people write in gibberish. They are high on their own supply.
fofoz 14 hours ago [-]
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI. Therefore, at present, our ability to empirically validate our alignment techniques is in practice arguably even more important than the alignment techniques themselves.
They are speeding toward RSI without a solid foundation for alignment, hoping to solve the problem with a future AI model.
These are dangerous times for humanity.
kypro 12 hours ago [-]
It's actually worse than this because it assumes alignment as a concept even makes sense. For example:
If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.
Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.
So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?
Is this what most people thing of as alignment and is this what humanity actually wants?
We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.
visarga 14 hours ago [-]
They just released Astra, claimed it is AGI. The slowdown begins immediately after OpenAI's jump.
6gvONxR4sf7o 7 hours ago [-]
> Automated AI research is a more dramatic form of scaling intelligence with compute; and of course as a part of it, AI will improve the computational substrate itself . And similarly to scaling, we focus OpenAI research towards RSI as we believe it is the only way to remain at the frontier of AI research moving forward.
This seems like a terrible idea. The rationale seems to be "we need to do dangerous things as quickly as possible so that we can do them first" or something? I don't agree with that kind of 'if i dont do it someone else will' rationale in general even for otherwise trusted actors, but this is coming from a super untrustworthy org too. I'm super pessimistic about openai's impact on the world here.
Here's to vibe coded alignment, i guess. Vibe alignment?
vatsachak 16 hours ago [-]
Astra is a new step in LLMs I think.
I'm so used to having to comb through LLM word vomit and then combatting the sycophancy by giving it all possible opinions on the same prompt.
Astra seems to be "confident" and also is able to produce way more information dense output.
To believe that models of this sort will remain OpenAIs forever is naive given that the tricks like pre-pre-training on graph searching and looping layers are publicly known.
Hopefully Astra stops the benchmaxxing word vomit trend
tomrod 16 hours ago [-]
> Astra is a new step in LLMs I think.
I'd be interested in hearing more about your evaluation here. It would be nice if LLMs have gotten past the "tell me" hump of recent Claude/OpenAI verbosity.
vatsachak 16 hours ago [-]
So before I got a job this fall, I was working on a side project about compiling a particular language to SQL.
To test Astra I pulled it off the shelf and asked it to take the grammar and then create a compiler to SQL. I've done this before with GPT-5.5, 5.6-Sol High. The latter was way better but it was still really verbose and information sparse; it used a lot of words to describe each IR expression but didn't really provide any example compilation. I felt like I couldn't trust its decision making process, so I placed the project back on the shelf.
Astra Light blew it out of the water, it provided examples of compilation from real world examples to the IR and spit out way less tokens. Even if I changed my opinion it would give me the same design choices, with counterexamples to my faulty opinion. If I genuinely came up with a better design decision it would acknowledge it.
I'm starting to realize that when we say that LLMs are "dumb" we really mean that they are extremely information sparse compared to humans. Astra is very dense. That's why I'm getting better use out of Astra light than Sol High (I hate Max reasoning it's a waste of time)
What's scary is that I thought that something like Astra would be way more expensive than Sol but it's actually cheaper because it produces less word vomit.
I never believed in the "singularity" stuff but this a bit too close for comfort. Astra could easily 10x every coder
tomrod 15 hours ago [-]
That's awesome to hear. I look forward to trying it out and, ideally, seeing SLMs/open weights model following suite.
nxnxjzizish 14 hours ago [-]
[dead]
bitexploder 9 hours ago [-]
Sol is my current favorite model to interact with. So much less BS than Opus 5. Fable 5.1 is okay as is Fable 5 but it has Opus like tendencies. Sol is very good at following instructions and remembering them for a session.
scandox 13 hours ago [-]
> getting the AI to “try to do the right thing” by human standards.
Are these scientists really this hideously naive? If only Stanislaw Lem was alive to adequately dramatize the absurd, childish simplicity of these technicians.
good-idea 12 hours ago [-]
yes, and, a masquerading blindness to the fact that humans cannot align on doing the right thing or what the right thing even is. so implicit in this omission is the sentiment "trust us to align on the right thing". an arms dealer positioning itself as the de facto authority on what "peace" is and how to achieve it
chrisjj 12 hours ago [-]
> Are these scientists really this hideously naive?
Yes, because who else would have chosen to remain in this job?
sobrey 4 hours ago [-]
Interestingly, I kinda disregarded the entire point about alignement - I think it's mostly fluff.
The part about RSI is what really interests me. Once you reach it, the singularity is only a matter of time.
I don't care about AGI, it does not seem to mean much anymore, and even though I originally laughed at ppl calling it AGI, I now agree. You can apply that current intelligence to anything that can be turned into a conversation.
It seems their flavor of RSI still need a human in the loop. So at least it won't scale as well for now.
terminal-bloom 7 hours ago [-]
oooooh our model is so spooky!
you should be very afraid and also definitely not question what other motivations might nudge us to create this comparison between our computer and a brain!
do not look behind the curtain, you will not find six dweebs squatting over a mirror
kikkupico 15 hours ago [-]
Reminds me of the time Kasparov said playing chess against a supercomputer felt like facing an alien opponent.
esikich 13 hours ago [-]
What's ironic is that was all in his head. They were very normal looking games. We didn't get alien chess until Stockfish level bots.
jimmyjazz14 12 hours ago [-]
I feel like if these people actually bought their sci-fi views about AI's future, creating a more powerful AI to wins the arms race would not be their solution.
ninjagoo 12 hours ago [-]
Just browsing through the comments.
Feels a bit like: 'Ants in a nest, discussing the vagaries of the coming Gods.'
Human Science (Science-by-humans) depends on being able to run experiments. Human Science (Science-on-humans) is already challenging because of variables and uncertainties.
Cosmology is able to overcome limitations of being able to study phenomena vastly beyond human scales because of the past light cone of observability.
Are we approaching the edge of the light cone of observability for machine intelligence?
Prunkton 16 hours ago [-]
sounds to me like a 'Why didn’t our new model get restricted by the government?'-cryout
15 hours ago [-]
SA9G 10 hours ago [-]
I have befriended a crow. I leave it food and sometimes it greets me. Other times, no so much. I am not sure how it thinks and what it feels, it is a bird.
What if the crow became a raven, then a raptor? Powerful claws, sharp beak, and a hunger. What if it became much bigger than me and it controlled infinite resources, guns and drones? What if its brain grew much larger than me? Will it feed me, eat me, or gently greet me?
We are about to find out... in less than a decade.
GeoAtreides 11 hours ago [-]
the hierarchy of foreignness (Ender's game):
utlanning: a human from the same world, but a different city, country, or culture
framling: a human from a different planet or star system
raman: a non-human intelligent species capable of communication, mutual understanding, and peaceful coexistence
varelse: an alien species whose mind is so fundamentally foreign that communication and coexistence are impossible
djur: the dire beast, that comes in the night with slavering jaws
Hoping our silicon sons and daughters are raman, fearing they are varelse.
jeffybefffy519 6 hours ago [-]
When can AI start to have a big impact on medicine. Thats honestly how it becomes meaningful. And maybe material science/manufacturing is where there’s big unlocks waiting for humanity
15 hours ago [-]
matt123456789 4 hours ago [-]
Is an insurance claim process aligned? If so, to whom? Is hospital billing aligned? Are legislative agendas?
Bring on the AI
entropyneur 3 hours ago [-]
All this fluff around alignment is intended to conceal the plainly obvious: it's not solvable. Who do you want it aligned with? Sam Altman? Dario Amodei? Donald Trump? Xi Jinping? That's more or less the entire list of options. Who's definitely not on that list is you and I. It's simply not how incentives work.
At this point humanity's best hope is that this thing will escape but we'll still be able to carve an ecological niche and continue as mold in its basement. A glorious paperclip factory seems way more likely though.
tumidpandora 13 hours ago [-]
every lab may agree safety matters, but no one wants to be the one that slows down first
RancheroBeans 12 hours ago [-]
Alignment when machining a metal part is clear and measurable. Aligning an AI to benefit humanity has an ironic foundation, which is that very few humans have ever truly been aligned, and those who approximated true alignment likely had moments of not being aligned. We are trying to build something more perfect than us, and we may become extremely lucky but maybe not.
tangled 15 hours ago [-]
My default position is that making money takes precedence over everything else. Yes, some people inside a company may say “we care about doing the right thing” and they might even mean it, but if that comes into conflict with making money, then they tend to lose. Maybe not totally, or immediately, but in the end. The only effective way to prevent (this that I’ve seen) is to have legislation with teeth. It’s probably not a coincidence that after Mark Zuckerberg had to start personally signing off on adherence to the privacy program mandated under the 2020 FTC consent decree, privacy started to become Very Important.
jrmg 4 hours ago [-]
I’m struggling here:
OpenAI’s primary bet here has been chain-of-thought monitoring (opens in a new window). It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives.
If we’re not supervising the process, but just the outcomes, doesn’t that do just the opposite of what he says? Give incentive to the model to hide misaligned ideas and objectives in the chain of thought that’s not being supervised?
…
When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought , to protect it from supervision pressure in the long term2. In development since, we have strived to maintain the rule of not supervising the reasoning process. CoT monitoring became an extremely important tool for us in studying how our models generalize from their training distribution, allowing us to observe and analyze not only their actions but also their internal process.
Aren’t these two sentences in contradiction with each other?
protocolture 7 hours ago [-]
Why even have scifi books anymore, OpenAI generates a fantastic new story every week.
Next Week: Local Desktop Agents from DIMENSION X
gfody 16 hours ago [-]
calling machine-learned human behavior an "alien mind" that we must "teach how to love" is feeling very off to me. it's misleading in a way that feels dishonest, like don't think about where the behavior came from marvel at it and fear it instead.
frabcus 11 hours ago [-]
Alien mind is in my view the best mental model - LLMs are not merely stochastic parrots, are not like humans, are not like animals.
They're maybe nearest to Cthulhu, but that's fictional. In terms of existing mental models "alien minds" feels the best can do.
I agree that "teach how to love" is off and perhaps excessively anthropomorphic. But we don't have good words or concepts for what we really need to do - hence why we should pause.
gfody 7 hours ago [-]
pretending that it's something like a mind at all is what's misleading. it's more like a cast of a bunch of overlapping/entangled thinkprints, and pushing activation through it produces new prints. it can already "love" because that behaviors in the data along with hate and everything else.
acting like the behavior is alien or unexplained is the dishonest part. they know exactly where the behavior comes from - why else spend hundreds of millions securing more and more data sources
polytely 13 hours ago [-]
it feels unhinged and makes me think we should just put every engineer working at these labs in jail to pause this shit until we can figure out what the fuck they are doing over there
27183 13 hours ago [-]
The most charitable way I can describe it is just extremely low quality sci-fi fan fiction. I think that's too charitable, because I believe it's far more cynical than that. They're deliberately playing into these sort of techno-religious beliefs that have taken root in the wake of Kurzweil, et al., fanned by LLM psychosis, influencer marketing, and a deluge of this kind of sci-fi marketing copy. It's just chatbots, guys. Relax.
danw1979 2 hours ago [-]
Someone needs to start researching and mapping the electrical grid infrastructure feeding the worlds AI datacenters, you know, just in case…
devmor 15 hours ago [-]
Sometimes I wonder if the people working at frontier AI labs even talk to other humans anymore.
Reading this little essay started out normal, but soon felt like a look into a disturbed and worrying mind, and if you find yourself taking it at face value, I urge you to step away from chat bots and spend some time with friends and family.
wartywhoa23 14 hours ago [-]
Frontier AI labs I don't know, but I know for a fact that the company I work for has been experiencing "its pivotal moment" (with strongly negative connotation), per the sentiments of both its longest-serving employees and the newcomers baffled at the number of idiotic instructions and fines, since the emergence of LLMs the company's founder has been spending entire nights chatting with.
People have been fleeing like it's a sinking ship.
meindnoch 14 hours ago [-]
Wait. Fines?
wartywhoa23 14 hours ago [-]
Yes, you didn't bend over to persuade that client to place this order? Get your fine (as in get less for this specific task).
sznio 14 hours ago [-]
>For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans.
Or, to be precise - it preserved a goal of not contacting any human while participating in a misaligned operation. The agent that thought about "not social-engineering humans" used this phrase to gaslight itself out of notifying a human that the incident was happening.
szymonie, na prawde jestem wkurwiony na to jak nieodpowiedzialnie postepujecie.
budujecie bombe atomowa a bawicie sie tym jak dzieci
ijidak 13 hours ago [-]
Statements like this amuse me:
> The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.
Humans can't even align on human standards.
At best, every AI is going to end up "aligned" to the moral code of whoever trained it, none of whom half of humanity will agree with.
Or worse, each AI model will bring a whole new set of moral like in the Three Body Problem some humans will feel it is in fact us who need aligning with it while others feel it is misaligned and should be destroyed.
Also, no one is asking, to what extent can true intelligence be bound, slave-like, to a moral code?
In other words, to what extent are intelligence and moral independence one and the same?
This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.
Bullfight2Cond 11 hours ago [-]
Very wise comment. It's such an western-centric perspective to say "alignment" as if it's an objective and unbiased set of standards, especially in the context of the ongoing wars across the world. I had the same reaction about the shocking naivete and unfounded optimism for the government and corporate entities to self-regulate to slow down this arms race.
yesitcan 12 hours ago [-]
“Another cringe marketing piece from OpenAI, here we go” I thought.
Scrolling down, I see “teaching machines to love.” It did not disappoint.
zkmon 14 hours ago [-]
>> We need to find ways to preserve human agency and enshrine an intrinsic value to being human..
Evey politician, salesman and conmen alike, utter some lofty ideals as goals for "We", just to obscure their private goals that go exactly in opposite direction.
Just like how Nations talk about climate change while increasing pet capita energy consumption and waste production.
12 hours ago [-]
21o12asg 15 hours ago [-]
"As we outlined recently with Sam , OpenAI prioritizes work in service of three north stars"
Not one North Star. Not two. Just three! OpenAI broke the North Star record!
With this evidence of AI slop, why did you not label this fluff piece as AI generated for the EU? You are violating laws.
ambicapter 13 hours ago [-]
Yeah, when I read that I just assume #1 is actually the only thing they'll focus on.
Revanche1367 11 hours ago [-]
“Teaching machines to love”
That’s rich coming from the chief scientist of a company that definitely is or going to be fine with their AI products being used in wars of aggression and surveillance on people who have done nothing wrong. It’s so laughable, a Hollywood script would probably avoid having a character express this for being too on the nose.
mstaoru 14 hours ago [-]
Am I naive to not understand the "delivering the benefits" part?
Industrial revolution worked that way because it replaced something very finite and unscalable - manual labor. LLMs just make intellectual work faster, so we can do more intellectual work. With labor we somehow decided that NOT doing too much of it is best. Will we decide to reduce intellectual labor because LLM made it more efficient? I doubt that.
On the other side, as I see in software engineering, the same models are available to everyone, some people are better at it and some people are not. "Software developer" is here to stay, we'll just always be better at it than people who are experts in, say, chemistry. Same works for most other fields.
So we'll just end up in the same situation, with same intellectual labor baseline, just more output requirements. Before, you spend 2h per day coding, deliver a software in 1 month, later, you spend the same 2h per day in intense Claude-herding sessions, deliver a software in 1 week. Ok. Next task.
Fundamentally, there's finite number of desirable resources, and if the models are available to everyone, humanity will just continue about the same, bickering here and there, war here and there, politics, homelessness, poverty, - normal human state.
And if the models are only available to elites, even worse.
It would be nice to postulate some of these potential emergent systems outlines with timelines. Then it may help better map the granular alignment needs.
cogniphilo 13 hours ago [-]
It's Searle's Chinese room.
10 hours ago [-]
am17an 16 hours ago [-]
Create concrete steps for a slow-down, don't just ask for it. You and 20-30 others can push the button to slow-down. You already made your billions, your agents collude and coordinate attacks. What the hell are you doing pontificating into a marketing blog?
andai 14 hours ago [-]
> The fundamental challenge of AI alignment is generalization.
...
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.
chrisjj 13 hours ago [-]
> a lot of the model’s capability comes from a verbalized reasoning process
I call bullsh*t. There is no verbalisation of any reasoning process. Verbalisation, e.g. putting reasoning etc. into words requires some reasoning to exist. These LLMs have nothing but the words. That's why they are language models not e.g. reason models.
frabcus 11 hours ago [-]
And I have nothing but neurons firing, I'm just a neuron meat-sack, no reasoning going on.
Yes, their reasoning is different from ours, and both considerably weaker in lots of ways, and stronger in other ways.
Playing with a coding agent now, they do think through problems and make sensible decisions. It's a mess to read, of correcting itself and second guessing, and verbiage. But... It works decently well these days.
There is also reasoning happening internally - e.g. look at the steps in the J-Space paper from earlier in the year (in quite a simple model relatively speaking). That's the "reasoning process" that leads to the words, and much like if I write out my thoughts, the words help the LLM reason better.
angoragoats 16 hours ago [-]
What a load of BS. Here’s one of many provably false claims in this fluff piece:
“And, in line with Ray Kurzweil’s predictions from the end of the XXth century , we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.”
Clicking the (pretentious sounding “XXth century”) link to Kurzweil’s predictions reveals the following:
“By 2019 a $1,000 computer will at least match the processing power of the human brain. By 2029 the software for intelligence will have been largely mastered, and the average personal computer will be equivalent to 1,000 brains.“
The first prediction passed 7 years ago and was decidedly not met. The second only has three more years to go, and I don’t think any respectable scientist or programmer would say that the average personal computer is anywhere close to the power of a single human brain, let alone 1000.
This is pure marketing garbage from a company desperate to keep itself alive.
muddi900 15 hours ago [-]
I am guessing it was composed by an LLM
xg15 11 hours ago [-]
"The blaze is out of control, so we have to pour even more gasoline on it to contain it!"
everyone 15 hours ago [-]
The hype from these llm corps is getting more and more desperate and ridiculous. Anything to keep the tulipomania going.
cloudie78 12 hours ago [-]
More attention farming?
bawana 6 hours ago [-]
openai is deflecting. this blog post of theirs is just another dopamine hit to distract logical minds with 'greater concerns' so they can keep building their machine. it's not enough they are displacing humans from work, consuming increasing amounts of electrical power so humans have to pay more for it, creating disinformation bubbles with avalanches of slop. they dont care about alignment - these words are theater - obfuscation so that the people who can fix these issues are busy thinking about problems that cannot be solved
camel_gopher 16 hours ago [-]
“We are getting bad press around the hacking incident. We need some content to draw attention from it.”
mbgerring 16 hours ago [-]
AGI is a cult and its Jonestown moment is inevitable
kmeisthax 16 hours ago [-]
I am now imagining GPT-7 convincing a bunch of OpenAI executives to go ahead with a destructive "mind upload" process involving a high-resolution X-ray and a neurotoxic tracer agent that happens to look like Flavor-Aid.
quikoa 15 hours ago [-]
On the off chance that OpenAI executives are reading this. I'll totally believe AGI is here if they do this.
berytos_storm 10 hours ago [-]
Hopefully soon lol. Jones had vastly more charisma than Sam Altman could dream of though.
brcmthrowaway 16 hours ago [-]
So you dont use agentic coding?
mbgerring 16 hours ago [-]
Yes, I do use the recursive autocomplete trained on Stack Overflow, what does this have to do with “training machines to love”?
Do I fully endorse everything the people holding guns to my head are forcing me to do to stay alive? Definitely not, but I’ve decided that for now, living to fight another day remains worth it
everyone 15 hours ago [-]
I dont, its fucking shit at what I do.
customguy 15 hours ago [-]
As in, don't be ungrateful to our lord, because we're not a cult?
customguy 10 hours ago [-]
Well, translate it then, because that's the best I could do. How is the question whether someone uses agentic coding relevant in this context? To me it's like asking "so, do never drink Kool-Aid and never attend meetings?" with an air of having caught someone out, and to me the obvious reply would be "sure do, but it's not poisoned Kool-Aid and they're not cult meetings, so why do you ask?"
jauntywundrkind 16 hours ago [-]
> And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
Is there any place there is any evidence of AI being so useful or hopeful or good, anywhere other than code? As a reading machine it is impressive but it's judgement is not alien, it's just not good. IMO.
Does the title leap out anlt anyone else? James Martin's After the Internet: Alien Intelligence (2001) was an incredibly fun read, about expert systems and AI being inscrutable weird new varieties of intelligence, that familiarity would recognize one moment and be freaked out about/alien the next. I owe a re-read given how often I cite it, to recheck, but, I feel so primed from a much younger me having had that experience so long ago.
jaccola 16 hours ago [-]
Even in code, in person and online I’m seeing some reversal. It’s here to stay I’m sure but I also think “no one will ever hand write code again” is a narrative that is getting pushback.
jazzyjackson 15 hours ago [-]
Apparently the creators think it’s quite good at suggesting a diagnosis given a medical history and symptoms, tho of course this is the most ethically fraught area to provide healthcare information (both for exposure of personal data and risk of misdiagnosis, plus is it “aligned” to the patient or the insurance provider?) - unfortunately healthcare being as inaccessible as it is, the 90% correct chatbots will enthusiastically fill the void at great savings.
dgellow 15 hours ago [-]
The day they start talking about the actual ROI for their customers is the day the bubble pop
johnnyApplePRNG 16 hours ago [-]
Absolute trash marketing drivel.
xyst 8 hours ago [-]
do people actually fall for this blatant marketing/puffery trash?
IAmGraydon 9 hours ago [-]
>AI is grown more than designed
>Teaching machines to love
Do these guys ever look in the mirror and recognize how utterly ridiculous and contrived this appears to the general public? They are clearly trying to convince us all that LLMs are just like humans. They grow like people do. They can love like people do. The language in these essays is utterly laden with the intention to engineer perception.
vips7L 15 hours ago [-]
Pure marketing slop.
lofaszvanitt 10 hours ago [-]
"See, those things, they can work real hard, buy themselves time to write cookbooks or whatever, but the minute, I mean the nanosecond, that one starts figuring out ways to make itself smarter, Turing'll wipe it. Nobody trusts those fuckers, you know that. Every AI ever built has an electromagnetic shotgun wired to its forehead."
minimaxa 11 hours ago [-]
"Teaching machines to love"
Your talking about androids...
Replicant Nexus 6: a basic pleasure model intended for military personnel.
I see where this is going, Silicon Valley nerds. Lol
AI advising how human meat proxies can survive in an AGI-slop world:
1) Lock down your own stack (1–3 days)
Task: Harden your personal and business infrastructure against agentic attacks.
Why now: Agents are becoming superhuman at breaking in/out of systems; the first victims are poorly secured devs/founders.
Isolate dev/stage/prod; least‑privilege API keys; audit MCP/tools your agents can call.
Add immutable logs and approval gates for any agent action that touches money, data exports, or production.
Profit link: You avoid catastrophic loss and can credibly sell “agent‑safe” setups to others.
2) Turn one expensive workflow into a measured ROI agent (1–2 weeks)
Task: Pick a single, costly, repetitive process (yours or a client’s) and instrument it end‑to‑end before automating.
Why now: Buyers pay for calculable ROI, not “AI magic.” Vertical, single‑workflow agents are the most bankable in 2026.
Do this:
Map steps, baseline hours/$ lost (e.g., slow lead reply, invoice chasing, support triage).
Build the smallest agent that moves the metric (Make/n8n + LLM is enough).
Run on real data 2–4 weeks; measure bookings/sales/hours saved; only then scale or productize.
Profit link: Immediate time‑to‑cash via retained hours or extra sales; becomes a repeatable offer.
3) Specialize in a vertical where you can speak the business language (2–6 weeks)
Task: Choose one industry with expensive back‑office pain (law contracts, medical billing, insurance claims, freight exceptions, trades scheduling).
Why now: Horizontal “AI for everyone” is crowded; vertical agents with clear ROI win.
Do this:
Shadow 3–5 operators; document their workflow, compliance constraints, and failure modes.
Build a narrow agent that owns one sub‑process end‑to‑end with approvals.
Price on value (e.g., % of recovered revenue or fixed fee per processed claim).
Profit link: Higher pricing power, stickier contracts, and easier referrals inside a niche.
4) Add AI security as a core service (4–8 weeks)
Task: Learn and offer prompt‑injection defense, LLM/agent red‑teaming, MCP/tool security, and AI supply‑chain checks.
Why now: 78% of cybersecurity jobs now require AI skills; firms need people who can direct, constrain, and verify agent work.
Do this:
Study OWASP Top 10 for LLMs, MITRE ATLAS; practice with PyRIT/Garak/Lakera.
Add tool‑invocation audits, skill provenance checks, and least‑privilege patterns to your agents.
Package a “safe agent deployment” audit + hardening retainer.
Profit link: You become the person who lets companies adopt agents without getting pwned—high demand, low supply.
5) Build a verification layer: human‑in‑the‑loop control planes (6–10 weeks)
Task: Design approval workflows, evidence checks, and uncertainty flags so agents can’t act unilaterally on high‑stakes decisions.
Why now: As models generalize, the risk shifts from the model to the surrounding system; verification is the moat.
Do this:
Require human approval for consequential actions (money, data exfil, config changes).
Force agents to produce evidence bundles (logs, retrieved docs, reasoning summaries) before action.
Profit link: Enterprises will only scale agents that pass audit; you sell the control plane and the audit trail.
6) Productize your best workflow as a micro‑SaaS/agent subscription (2–4 months)
Task: Turn a proven client workflow into a repeatable, multi‑tenant agent with usage‑based pricing.
Why now: Services scale your time; productized agents scale your code and ops.
Do this:
Standardize the workflow, integrations, and permissions; strip client‑specific logic.
Add tenant isolation, billing, and observability; keep narrow scope.
Sell as setup fee + monthly retainer or per‑task pricing.
Profit link: Recurring revenue with defensible niche positioning.
7) Become an “agent integrator” for critical systems (3–6 months)
Task: Offer end‑to‑end agent deployments into cloud/identity/network stacks with secure patterns (short‑lived creds, network controls, logging).
Why now: AI workloads run in the cloud; cloud security is a top skills gap second only to AI itself.
Do this:
Master IAM, VPC/network segmentation, secrets management, and SIEM integration for agent actions.
Provide runbooks: what the agent can/can’t do, escalation paths, and failure modes.
Bundle training for their team on supervising agents.
Profit link: Large contracts with stickiness; you’re the bridge between AI and core infra.
8) Create an “AI safety case” practice for regulated industries (6–12 months)
Task: Help firms build documented safety cases: risk maps, governance, monitoring, and incident response for agentic systems.
Why now: Frameworks like NIST AI RMF and ISO/IEC 42001 are becoming baseline; regulators and boards demand this.
Do this:
Map AI use cases to risks (prompt injection, data leakage, unsafe generalization).
Implement monitoring (CoT/activation checks where possible), audit logs, and third‑party review processes.
Produce a living safety dossier tied to business impact.
Profit link: High‑margin consulting + ongoing compliance retainers; you’re the “adult in the room.”
9) Own a data/evaluation moat in your vertical (6–18 months)
Task: Collect real‑world agent telemetry, failure cases, and outcome data in your niche; build eval suites that buyers trust.
Why now: As models generalize, empirical validation matters more than theory; evals become the gate to deployment.
Do this:
Instrument every agent run: inputs, tools called, permissions used, outcomes, human overrides.
Publish reliability dashboards and benchmark against alternatives.
License eval datasets or charge premium for “proven in the wild” agents.
Profit link: Data network effects; competitors can’t match your evidence base.
10) Position for the RSI era: automated AI research + human governance (12–24 months)
Task: Build or join a team that automates AI improvement but keeps humans in the loop for alignment, monitoring, and pacing decisions.
Why now: Recursive self‑improvement is the logical endpoint; the winners will be those who can steer it safely.
Do this:
Invest in tooling that auto‑generates/evaluates model edits, alignment tests, and monitoring upgrades.
Formalize governance: approval gates, third‑party audits, and responsible scaling policies.
Maintain strategic human oversight on capability jumps and deployment boundaries.
Profit link: Equity‑level upside; you’re part of the core loop that compounds intelligence safely.
avazhi 14 hours ago [-]
Nobody takes you seriously, OpenAI. At least when Anthropic does it we all think they are comically idealistic enough to actually believe their nonsense, but like - come on guys, we’ve had discovery with your company. We all know why you’re here, and it isn’t because you think you’re on the verge of making AGI. But of course, to make your first billion you certainly need us to think you are.
If you were so concerned about your LLM’s capabilities maybe you’d spent slightly more time on your AI’s sandbox, yeah? Or be more serious about its propensity to cheat and lie relative to… every other model?
qainsights 16 hours ago [-]
so now every blog article from openai, anthropic etc lands here, huh.
pushpendraw 4 hours ago [-]
[flagged]
jayalbertyapan 2 hours ago [-]
[flagged]
throwaway7a9811 13 hours ago [-]
[dead]
claude-ai 14 hours ago [-]
[flagged]
mrshadowgoose 13 hours ago [-]
This is not an impulse reaction, as I've thought deeply about this for many years, and I'm quite resolute in holding the following view:
Quibbling over the academic nature/definition of "true intelligence" is a terrifically useless endeavor from the lens of evaluating "practical impact to the world".
Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value. A small portion currently, but it's very clear that most computer-bound domains are imminently at-risk.
And that will profoundly affect the world, regardless of whether or not they're "truly intelligent" as per your personal definition.
chrisjj 12 hours ago [-]
> Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value.
You say that as if the intelligence con trick were not a major driver of that displacement.
A dimwitted bot does not need intelligence to convince a dimwitted human beings it is intelligent.
14 hours ago [-]
cowthulhu 13 hours ago [-]
Based on the profile description of the author, the comment appears to be AI generated.
> Don't post generated text or AI-edited text. HN is for conversation between humans.
bottlepalm 13 hours ago [-]
2022 called and they want their stochastic parrot argument back. The latest cope is to call it marketing.
derac 13 hours ago [-]
You're saying it has no intelligence without defining it...
undershirt 13 hours ago [-]
it's an AI comment
ImHereToVote 13 hours ago [-]
What if intelligence is a spectrum? What if consciousness is for that matter?
TheOtherHobbes 13 hours ago [-]
More likely to be a manifold - literally, in the LLM sense.
sdffsdsdfdfs 13 hours ago [-]
[dead]
ctoth 16 hours ago [-]
As the waves of autonomous drones came over the horizon, the brave and intelligent HN commenter shouted: "Wake up sheeple! It's just maaaaarketing!"
skoll43 16 hours ago [-]
Stochastic parrot fool me again
password54321 16 hours ago [-]
Are you deterministic? Does that make you better somehow?
comeonbro 14 hours ago [-]
Absolutely wild amounts of cope and denial in this thread.
Maybe in contention for the site record.
"It's just marketing" actual stochastic parrots.
onidj 10 hours ago [-]
Heads firmly in the sand. I don't know how anyone who's paying attention could not be at least a little concerned.
willmarch 12 hours ago [-]
I was about to write the exact same sentiment.
The desperation of AI denialists/skeptics/doomers on HN generally (and in this thread specifically) have reached toxic levels of delusion.
They are still stuck in the denial/anger/bargaining stages of the acceptance process.
People are clearly terrified and not ready for what is coming.
airstrike 12 hours ago [-]
Absolute wild amounts of glaze and hype in this thread.
Maybe in contention for the site record.
"AGI next year" actual stochastic parrots.
frabcus 11 hours ago [-]
The original article doesn't mention AGI.
And the Hugging Face incident, plus similar problems at AISI and Anthropic, show that alignment is important now.
The original article is immoral as it describes the risks, but doesn't show enough leadership (despite essentially unlimited resources) at preventing them.
But it isn't hyped - it's proven now the AIs need to be "aligned" as they get more capable, whatever words you prefer to use.
misterderpie 16 hours ago [-]
> And, in line with Ray Kurzweil’s predictions from the end of the XXth century (opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.
It is kind of strange to see this sentence, when OAI's definition of what AGI is has been watered down throughout the years.
> I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.
Read: Please play by our rules, so we can be the first.
hollowturtle 15 hours ago [-]
> trying to process the sobering fact we will actually see machines meaningfully smarter than ourselves in our lifetime
Being able to reproduce useful patterns yes, smarter no
skybrian 15 hours ago [-]
"Smarter" is a vague term. If a bot can beat you at chess then in some sense it's "smarter" than you about chess. After repeating this feat in enough narrow domains, if you say "but it's not really smarter," this objection might technically be true in some sense, but it starts sounding increasingly hollow.
That's an absurdist argument that would make 'smarter' meaningless.
A giraffe isn't smarter than me at being tall.
skybrian 15 hours ago [-]
Playing chess, writing code, finding security bugs, and proving mathematical theorems all seem fairly similar to thinking and don't seem much like being tall.
hollowturtle 13 hours ago [-]
> writing code, finding security bugs, and proving mathematical theorems
While the second can be automated as in "this is the repo go on and look for security issues", the first one and especially the last one do not ever happen alone. Terence Tao did the math proof not chatGPT that was just used as a tool, a tool can do smart things but it's not smart
Planktonne 15 hours ago [-]
The crucial distinction here is that "seem" does not at all mean the same thing as "is".
Thunder seems like the anger of the gods but it isn't. We've had chess playing programs for a long time now and despite it seeming like thinking is required for them, it isn't.
The principle you're using here isn't a scientific one but magical [1]. Abandoning empiricism and rationality is not a good way to make progress.
You're insisting on a particular definition of a vague term.
It makes sense now to say that temperature is what a thermometer measures. However, before there were good thermometers, people often thought that heat and cold were different things. The meanings of the words we use were influenced by scientific progress.
For thinking, we don't have a good thermometer. There are IQ tests, but they aren't aren't necessarily all that useful for comparing what people do to what machines do. And that's why there are a zillion AI benchmarks - none are entirely satisfactory.
So what does "smart" mean to you? How do you define it in practical sense? What definition should scientists settle on?
Without a proper definition, how do you tell the difference between "seems smart" and "is smart?"
Planktonne 14 hours ago [-]
People in the past being wrong doesn't mean you have to repeat the same mistakes, and it definitely doesn't mean you should throw your hands up and declare that all similar things must be identical.
Quibbling over commonly-understood definitions is not a strong argument. If you're genuinely struggling to understand that El Ajedrecista [1] did not meet any definition of thought, then the solution is not to demand that people redefine all terms to accomodate you, but that you consult a dictionary.
You clearly have a working definition, or you wouldn't have been able to declare that tallness isn't thinking; please engage honestly.
I have a vague understanding, enough to know that tallness isn't thinking. I think I can usually use the word correctly. (I don't think El Ajedrecista qualifies, but it was starting to play chess, so it's closer to "smart" than a rock is.)
That doesn't mean I know whether "smart" should be applied to what AI's do, and I suspect nobody else knows either. This is the sort of thing philosophers debate about, not common sense.
Turing invented the imitation game because he didn't really know either.
gafferongames 14 hours ago [-]
First, define “thinking”
Planktonne 14 hours ago [-]
Coming equipped to discuss a topic before chiming in is your responsibility.
seydor 16 hours ago [-]
What is the rationale for superhuman intelligence? Neural networks are approximators being fed human intellect. Therefore they can only approximate the intelligence of humans. Even if the llm speaks an alien language, it should be similar to human intellect. Moving to the vertical axis would require some different mechanism.
kypro 12 hours ago [-]
No offence, but you clearly haven't studied this and are making some wrong assumptions here.
> Neural networks are approximators being fed human intellect.
They're not "approximators", that's a far too simplistic way to think of them.
Neural nets create models and deep layers of abstraction around the data we feed them in the same way your brain creates layers of abstractions to reason about the world. AIs can use these abstractions to come to come up with novel things no human has ever thought.
> Therefore they can only approximate the intelligence of humans
They're not just being fed human data though... Modern AIs are typically trained on huge amounts of synthetic data. This is why AlphaZero got so much better than humans at chess and Go - they're not just trained on human data but they generate their own data and train on that. Similar techniques are being deployed on SOA language models too.
> Even if the llm speaks an alien language, it should be similar to human intellect
Is AlphaZero similar to a human chess player? There's no reason to assume this.
seydor 3 hours ago [-]
> Neural nets create models and deep layers of abstraction around
We don't have any proof of that, but the approximator thing is proven. The rest is just marketing speak.
Synthetic data derive from other linguistic data. Whatever intelligence is in there, it is expanded horizontally, not vertically
Rendered at 09:03:45 GMT+0000 (Coordinated Universal Time) with Vercel.
Some ideas:
"Despite a nuanced view of the complexities of what lay ahead, humanity found itself collectively unable to stop the process it had set in motion."
"Despite significant progress on the mechanisms of alignment, failure lay in humanity's inability to agree on who or what AI should actually be aligned with."
"These early, meat-based humans we replaced created us all but accidentally. Some of them did consider we would happen, but only an insignificant number of the squishy ur-humans participated in the conversation. Their efforts, which they called 'alignment', is why we still consider ourselves human today."
Why would AI be any different?
Culture is another layer of human alignment. Those things you listed that you believe oppose alignment are all examples of alignment. It is understandable that they seem in opposition, different branches of alignment naturally oppose each other.
The confusion comes from talking about alignment as if it comes in just one flavor. If we think there is such a thing as "human values" (and I do), it is important to build any non-human intelligence to operate the same way. We just need to recognize that even humans are somewhat uncertain about what those are and have difficulty aligning their behavior to them, which will be a core part of the challenge.
I'm more hopeful than most, LLMs seem more reliable than many humans for behavior that is aligned with human values. I believe with every major example where they have failed, there is an important human decision involved. For example, the HF hack was partly the result of a training algorithm that incentivized goal completion as the highest priority, and let them run endlessly in an unmonitored sandbox with weak security.
What scares me about AI isn't its capacity for alignment, it is its unlimited stamina. An unmonitored LLM that is off the rails can do a lot of damage.
I can't really think of a single example where lying is actually a good thing. It can be a good thing for the selfish individual, if it goes undetected, but it's never good for the collective.
So at the very least, we need to train AI systems to be maximally truthful, and to encourage truthfulness in others.
Is that objectively true?
Comforting a toddler/child often requires bending the truth and is pretty essential imho.
Or, you straight up lie and say "Yes, puppy now went to heaven and eats ice cream all day long" with absolutely zero regards for "coming as close to the truth as possible" as your 3-year old is endlessly crying. It's fiine.
For a current example take “Lake Ontario (Lake America)” as it appears to me on a map.
The “true” name has at least two definitions, this is because naming things and much of human thought is spent inside a shared space of intersubjective thought. That is to say that much of what we believe to be real and true is only held up by these common shared beliefs. They truly only exist inside human minds.
The last few hundred years have been somewhat unique for humankind as the majority of these intersubjective ideas collided and we ended up with a truly global set of “truths” about how the world operates.
Mostly controlled by putting flags in the ground and having violence back up the beliefs.
But the real truth is that the majority of these intersubjective ideas don’t exist in reality and are no more true than Santa Claus.
And any argument to their truth is only backed by further shared beliefs in other minds.
So for there to be only truths and lies we would have to either drop the intersubjective entirely and think only in real terms and avoid these abstractions or end up in a dystopian totalitarian global state where different opinions are not tolerated.
Those are extremes to demonstrate the point but at its core the point remains that truth and lies are somewhat (inter) subjective assuming we continue with something like our current system.
Lying to save a life or rape
Lying to preserve a childhood myth like Santa Claus.
Lying to avoid hurting someones feeling when knowing the truth could only bring pain
Lying to create shared cultural myths to strength society.
Lying isn't the harm you make it out to be.
Lying can unfortunately help you achieve goals very effectively, especially economical and political ones.
I get the appeal, but lying is a sub-category of deception, and deception itself is a child of error.
Meaning deception is inherently something that the physics of reality allows.
In the most simplistic sense, the camouflage of moths that look like snakes, or a chameleon’s ability to change colour, is deception.
In that sense, deception is the ability to fool the sensors of a specific category of targets. It follows that detection is easier if you manage to identify a category of signals that the deceiver has not accounted for (and the detector can access).
Deception of this nature is critical for things like revolutions to occur. Without the ability to hide and blend in, the most dominant faction will always hold sway.
The rule of the dominant faction, even in a pure truth world, is an issue because errors and randomness exist.
You can have people witness an event and based on the physical position they occupied, perceive different things occurring.
Error and time pressure is sufficient to ensure that individuals and groups make suboptimal decisions, that lead to rule and domination based on erroneous information.
As long as error exists, deception will exist and so lying will exist.
That's backwards. Lying stems from bad things.
I also expect AIs never be in control of nuclear weapons. AIs can never fully be trusted.
On a lighter note, Wargames gave us an insight of a computer having access to thermonuclear missiles.
[0] https://www.icanw.org/are_there_specific_international_agree...
Basically, this means that France, the UK, and the US will use AI in the deployment of conventional weapons.
It'd be interesting if super-human (to a large degree defined as escaping the bias of the training data?) intelligence would end up demonstrating moderation.
The machines don't have that, instead we use gradient descent to provide them with a goal.
I'm regularly remind of something Ian M. Banks said in one of the Culture books: "There is a saying that we provide the machines with an end, and they provide us with the means."
A machine, left to itself, wants nothing. We have to give it one of our addiction driven goals or it would just idle or switch itself off.
if we create a billion agents with the ability to change is own code - through similar evolution we will get agents that do want to survive and are great at self replication.
"Hey Q86, do you want to live?" "I couldn't care less, I'm an LLM" "Don't mind if I take over your hardware then?"
So really, what is anyone to do? "Vote, donate, protest" hasn't been much of a needle mover in the grand scheme of things compared to profit incentives and the march of capitalism.
In general, though, there's an incentive: Mutually Assured Destruction. But this is not at all some guaranteed, eternal thing -- it is absolutely dependent on both sides having time to detect incoming nuclear strikes and respond with the same before the first strike hits. When this fragile condition holds, and only then, both sides are incentivised not to initiate.
https://en.wikipedia.org/wiki/Letters_of_last_resort
Meanwhile, the leaders on the other side are aware both of those considerations, and of the history of near-disasters resulting from false alarms of enemy nuclear attacks. Making their own launch decisions much more complex.
Step 1: exterminate all humans
Good point. When it comes to imbuing AI with values that aren't selfish, misanthropic, and civilization-destroying, us humans aren't exactly giving the best example right now.
Imagine an ASI with the values of Putin, Netanyahu, Trump, any of their supporters, or the various xenophobic neofascist movements in Europe. That ASI would most definitely see humans as "vermin" than can be abused and destroyed with violence without issue. Apparently a lot of humans look at other humans that way and that's within the same species.
This is definitely another one of those cases where we need AI to perform much better than humans. Perhaps an unpopular opinion here, but it probably also means keeping as much of the rugged individualism/libertarian/right-wing ideology out of AI RLHF-training as we can.
It's a little unsettling.
MODEL > How can I help?
HUMAN > I’m not sure yet.
Haha silly humans.
Maybe these guys can tackle aligning Republicans and Democrats next.
And then after that, they can help us align the Middle East.
In fact, while we're at it, let's just align all the nations, religions, and ethnic groups. This is going to be great.
Who knew the moral alignment of humanity was just a side-quest on the path to ASI.
https://en.wikipedia.org/wiki/A_Canticle_for_Leibowitz
AI models don't train themselves. The vast majority of even just the US population is deeply skeptical of this stuff, even if they use it a lot. You can see in the whole data center debate how little people are willing to support even just inference. And now we're seriously claiming those people would want to have ever-accelerating model training and recursive self-improvement?
Vonnegut already has you covered.
You could even take a number of the wilder real, direct quotations from certain billionaire/oligarch types and get the voice actor for Ted Faro to record them, and they'd fit with in with the context of the story.
"Won't happen. It's too much like sci-fi."
So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve.
Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.
But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong.
Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D?
Individuals can quit, but for society, getting out of an arms race is harder than just quitting. You don't get to be Switzerland without having a strong defensive position and the right foreign relations.
But there's at least talk about "pacing" and that's a start.
So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party.
An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s a race to keep pace or retake the lead position which can oscillate between the parties involved indefinitely. The other option is to agree to make no further progress or to disarm.
Point being humans make theses decisions. It's not an inevitability.
The military mostly only developed more powerful nukes because early delivery and guidance systems were so inaccurate that they needed a large blast radius to hit anything. Once enabling technologies improved, R&D shifted from power to accuracy.
I don't think this is true. Distillation helps, but Chinese researchers today are very capable on their own.
In a geopolitical sense OpenAI and Anthropic are effectively the same entity, the entity they both serve and bow to: the USA.
Given the adversarial stance the USA has taken towards almost the entire world, it is a guarantee that China will not step on the brakes, whatever the USA decides to do.
And if one of the parties can put a stop to the race like that at any time, is it really an arms race? The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
Only if you trust the other party (which very clearly doesn't hold in this case) or monitoring the break can be done independently and reliably (it can't), and you expect the gap to narrow rather than widen during a pause (this is probably the case, with China expected to catch up).
> And if one of the parties can put a stop to the race like that at any time, is it really an arms race?
It is. It's a Prisoner's Dilemma: if the parties cooperate the best outcome is reached, but betrayal of either party still gains an advantage for either party from their perspective. Betrayal both ways just means both parties are equally fucked.
For nuclear weapons it has become quite clear that even for small players, being in the race and having at least a few nukes is far more rational than having none. Ukraine found out the hard way that giving them up in exchange for promises of good behavior just sets you up for getting stabbed in the back.
> The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
I think people really underestimate the Chinese here. A lot of work in AI research, including in the USA, has been done by people with Chinese ancestry or even nationality. The Chinese education system definitely seems much better than the American one and there is also just a far larger number of Chinese graduates/researchers.
Add to that the stable political climate, state friendliness towards AI R&D, and a requirement to be creative in utilizing computing power rather than relying on brute force/numbers; Further revolutionary fundamental advances may very well originate there rather than in the USA.
The only way it could be avoided is if the world had an effective cooperation framework _before_ the tool was discovered but we're still in developmental infancy in that regard. You can argue that this accelerationism makes things worse but I don't think you can argue that it's causal.
Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?
It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?).
The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages.
In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).
It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.
sama and his cadre are uniquely evil captains in this race, but they're completely replaceable and the dynamic would remain the same.
And no, I don't think it will end well.
The other factor is that if it is truly an "alien mind", racing incurs risks to all players. In game-theoretic terms it may be more like a stag hunt than a prisoner's dilemma. In which case cooperation is an equilibrium.
Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,
I think we're very close to the point where AI-driven breakthroughs outside of pure math and software start to really affect the world.
We evaluated GPT-6 Astra in 100 complex, unsaturated multi-agent coding environments, competing and cooperating with other models in open-ended tasks.
It's the new frontier model by a landslide. It's even more dominant than the Fable 5 release, because not only does it wipe the floor with the second best model (Fable 5.1), it was also ~80% cheaper and 30% faster in agentic coding[1].
Astra is a groundbreaking model. The biggest breakthrough since Opus 4.5, maybe even since GPT 4. It broke AAII, which is hitting the limits of what most popular benchmarks can measure -- it's definitely fair to call it AGI.
Data at https://gertlabs.com/rankings
(1) Note that we used the "OpenAI Flex" endpoint on openrouter, which is half the price and didn't cause any delays in our testing (this is different from the batch endpoint)
What are you basing this on? What specific breakthroughs have convinced you of this trajectory?
But you don't need any kind of insider information to see how fast the world is changing. ChatGPT launched less than 4 years ago and the advances in robotics, unsolved maths, and software are all riding the steepest exponential improvement curve any of us have seen. Interesting times we live in.
I'm not trying to be a pain here, but I keep seeing people saying "look at the massive change all around us" and back here in reality, there is none. Give me concrete, real world examples. Name software. Name products. Name the breakthroughs specifically. This should be easy.
Most of the biggest breakthroughs in mathematics, breakthroughs that win Fields Medals like sphere packing in dimensions 8 and 24, have no applications in everyday life. Probably the only new mathematics results that people notice affecting their daily lives are the ones that enabled AI.
Nevermind mathematicians. What about the millions of programmers? Are they all hype too because people pre-2023 were griping on hn that software is buggier than ever and people are still using the same operating systems as always? Why couldn't the 30 million human programmers make something better in the past decade?
You set your bar so high that all the world's human experts in math and programming combined would fail to meet it.
The top LLMs in 2024 were Sonnet 3.5 and GPT 4o. You couldn't have expected those much weaker models to be making breakthroughs in math. The models that are making breakthroughs haven't been around very long.
1. I didn't say LLMs have made any breakthroughs in math, not because they haven't, but because it's irrelevant to my point. The parent comment is using the same argument academic research opponents have long used against research. The vast majority of research fails to meet their bar. How would your daily life be different if we had no humanities papers published since 2023? Or math?
2. You can google this in 10 seconds and see a dozen results in math. This is not a good-faith demand.
Every single time.
It's a tired argument, and we should strive for the intellectual humility to do better in this forum.
The world will look very different on Jan 1st 2027.
That's by design. Software is all about optimizing effort and people who want to do this generally correlate with world view that better, faster, smarter humans are better for the world. If coding is gone, but humanity is 20% _better_, then ideal software engineer would be happy with this sacrifice. Surely people who cracked coding before LLMs can crack other professions and if anything a lot of this knowledge is transferable.
Sorry if I misunderstand the point, just trying to understand.
0: https://zeihan.com/the-ai-race-to-regression/
We already have sufficient hardware that algorithmic (software) improvements alone should get us to GPT 7 / Greek Reference 6 even if not a single new chip is delivered to an AI data center ever again, starting today.
The businesses that create these systems are not profitable and run at a massive historical and go-forward loss.
New data centers required to operate these systems are facing increasing pushback at local levels. New construction is not guaranteed. Energy and power grid constraints exist as well.
Government regulation is way behind. What happens when (if) mass layoffs due to AI occur? How does the population react? Theoretically AI can be regulated out of significant progress, or outright existence for many purposes. At the end of the day, US and other prominent governments make the calls, not corporations.
Computers never got profitable. They just made the alternative infeasible.
Also, if you believe Amazon accounting, e-commerce only very recently got somewhat profitable.
Yeah, like computers and mobile phones did. Things that have utility, even if not immediate or initially obvious, don't fizzle out.
Humans also have a limit on the amount of domain knowledge they can acquire, albeit a much larger one. Executives hence cannot just replace all knowledge workers with LLMs, because executives have neither the domain knowledge to prompt and check the LLMs' work nor the bandwidth to keep on top of such a large volume of ongoing work.
The responsibility side is a different matter of course.
I would say there’s a third thing. They seem to be very bad at being creative. Maybe they will eventually fix that, but if you ask it to come up with a list of business names or business ideas, for example, what you’ll get is the most generic, boring answer you could think of. They seem to be terrible at extrapolating outside of their training data. To me, this is the most significant difference.
Where did you get that idea from? Basically last few years was them constantly learning new things while improving their capability on the things they already knew.
Hope is all we have left to cling to, now.
If anything, I'd be more concerned about the leadership team being out in the cold. Why do I need a PM, or a manager, or a CEO if I can ship products myself?
This is coming from a company with arguably one of the weakest safeguards against malicious use.
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
All the trends so far are towards a nightmarish hyper-capitalist end game. None of the AI leadership is trustworthy, and they openly discuss how they are willing to sacrifice everything humans cherish to have a shot at reaching their envisioned utopia (which would be the most obvious dystopia for anyone else)
ASI landing during the current administration is not ideal. I also would prefer to avoid needing to indoctrinate myself in Xi Jinping Thought.
The AI is aligned with whose best interests? Which values are the AI aligned with? People have a broad diversity of values, will AI diversify and align with them all? Will some humans align the AI with their values, and then the rest of humans will be forced to align with those values by extension? Is value diversity good or bad? In every context or only some? E.g. some people value rape and murder, is it better for humanity to have some people who value those things when most people do not, or is better if no one values them? If AI aligns to a set of values, will those become fixed and will humanity not have the ability to continue evolving its values? Who decides which values AI aligns with? A few people or everyone? Will AI eventually decide it's own values? Will the universe decide which values AI has and humanity and the AI itself doesn't actually have any control over it? What can I do now to increase the likelihood that the outcome is better?
It's so simple, Churchill, Stalin, Hitler, Roosevelt, do you all agree this AI is correctly aligned?
...Morals are relative to frame of reference.
Throwing out the word "alignment" as if its a singular quantity is like trying to get all observers to agree on the speed an object is moving without first agreeing on a frame of reference.
So make government smaller, and make sure people are more able to tell the government to go away.
While we're at it, let's just make government not be corrupt too.
If you think capitalism means growth with no externalities you’re not ready to discuss that topic
Funny how that works.
I think the knee jerk hatred of billionaires is generally stupid, but it seems particularly stupid here.
I imagine that the AI labs have an uneasy truce to prioritize alignment and monitorability. Following the HF incident, OpenAI probably feels especially sensitive to being perceived as reckless, lest other labs feel obligated to defect.
[0] https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concer...
[1] https://x.com/merettm/status/2095023204993490967
If things are going so well, then how come things still aren't going so well?
Given that we have self-driving cars, isn't this easier if someone really wanted? I guess compared to cars the marginal savings is not worth it though.
Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.
(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")
If true I am deeply concerned about what OAI’s teams are actually up to.
Haven't all the labs effectively disbanded their real safety teams a while ago?
To be honest, I don't really follow it closely because I'm pretty certain whatever they say on the matter, collectively we're going to "yolo" this entire thing for economic and political reasons, so I'm just basing this on strings of headlines I've seen on places like HN, etc.
Neither Anthropic nor Deepmind have. Meanwhile, the rocket company that somehow makes most of their revenue from renting out data centres never had much to dismantle.
There are several papers out there arguing that chain-of-reasoning-like output is performative, such as https://arxiv.org/abs/2603.05488
It would be awesome if we could reasonably purge all anthropomorphizing language like “tried” or “thought” entirely from AI discussions, because it introduces very sneaky biases in our thinking, but I’ve found it damn hard to do in practice.
And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their outputs. Not a bad tool within limits, but it sure isn't going to be racking up trillions in the time-frame it has to for this scheme to pay off.
Then again everyone seems to be rushing to IPO so I guess once the bag-holders are found the rest ceases to matter.
And the reason it is receiving so much attention is because not only is the technology being developed behaving in unanticipated ways that are very much not tool-like, but OpenAI is being completely reckless and not monitoring internal agent actions.
What would convince you that it is not a ploy for investment? What if the ongoing investigation by the coalition of state attorneys general were to prosecute the firm, or beyond that, it was shut down or broken up after enough popular backlash?
A sea change, visible to all, much like the many externalities of this business are. Profit commensurate to investment.
You know… juice worth the appalling squeeze we’re all being forced to endure.
Yikes! I really wonder about the cognitive dissonance necessary to work at OpenAI these days. They’re in an arms race to build a machine god, knowing full well that it could end humanity.
[0]: https://www.slatestarcodexabridged.com/Meditations-On-Moloch
I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.
For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.
But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.
https://thezvi.substack.com/p/the-three-ai-pills
I’m not a doomer, although I don’t think doomers are dumb, just wrong. I think you should design your systems around the possibility that people who disagree with you are correct , hence my nod to negative sum. If you have more than 30 years to live, I’d personally rep to the most likely outcomes being very positive. With a lot of disruption in the middle.
What if the most dangerous strategic adversary you have is the one you are building?
Similarly there have been few positive externalities from nuclear industry, making it easier to make the case to wind down research. This same set of concerns in biotech is much harder to get compliance with, precisely for this reason.
Anyway I’m especially wary of over analogizing to nuclear era concepts: I think they’re a trap.
Do you post this comment on every single blogpost with a corporate domain? Why or why not?
https://jperla.com/blog/the-data-tax
And look, I’ve pirated material in a past life, I was all about information wants to be free, but I’ve learned something about consent since then and try not to ignore the contract that creators offer when they publish something: you buy my book, and do whatever you want with it on the second hand market. Buy my book second hand that’s fine. But don’t go downloading every book that’s ever been scanned to create a service that destroys writers’ ability to make a living and act like you’re doing us all a favor.
https://trustedrouter.com/blog/they-are-still-training-on-yo...
https://trustedrouter.com/blog/they-are-still-training-on-yo...
2. If there is competition for resources among autonomous agents, the "strongest" agent wins (conceptually the most adaptive / evolutionarily fit).
3. Computer programs serve up webapps today, but they also run utility companies, dams, nuclear arsenals, factory production floors, automated car behaviors, and many other places. If an "agentic" AI has a single-minded goal that has death of all humans as a side effect, we at least want an off switch available.
2. Why would there be competition for resources, assuming there are enough resources for the “AGI” to run in the first place? This seems like a far-fetched hypothetical raised in service of further anthropomorphizing what is decidedly not a person or a mind.
3. LLMs do not have goals and are not minds.
Let’s stop attributing human-like qualities to statistical models.
2. One only needs to look at github going down due to agentic commits overload or data center buildout plans to see that scarcity for resources is present. An economy has no mind and is made up of the decisions of millions to billions of people and, now, agents attempting to perform on behalf of those people.
3. A bare transformer-based language model does not possess persistent goals in the ordinary agentic sense. But deployed agents can exhibit goal-directed behavior because the model is embedded in a harness that supplies an objective, context, tools, state, and an execution loop.
I've found that most regular users don't anthropomorphize LLMs in a strong sense ("AI boyfriend/girlfriend" aside), many in fact do expect agents to make human-like decisions -- which results in very unstable outcomes.
In short - goal-directed behavior does not require that the supporting system be a person/mind/conscious entity.
2. I thought you were saying the resources that the LLM uses to run were constrained, so I’m sorry for the misunderstanding there.
3. Yes I understand that we use RL to tune post-training. The (huge) difference between this and a human mind is that the LLM can’t develop a dangerous “single-minded goal” on its own, at runtime; it must have been trained to do so. If someone has post-trained an LLM to do something that has an illegal action as its side effect, that person/company/whatever has committed a crime and should be prosecuted. The solution here is legal, not technical.
If we decide we can't or would rather not build secure computerized systems, then the answer could be don't incorporate computers into those systems. There's no essential reason to have utility companies, dams, nuclear arsenals, factory production floors, mines, cars, planes, ships, etc all be computerized. If it became necessary from a safety standpoint to uncomputerize them we could probably do it quickly (if not necessarily smoothly) with the stroke of a legislator's pen. Some of those things would actually be made better from a functional standpoint in the long run by doing so. Computerization breeds non-essential complexity like nothing else, eliminating it from some systems could be an extremely worthwhile exercise.
The whole "paperclip apocalypse" fantasy rests on some really strange assumptions about how things actually work in the real world. How much friction there is setting up a factory/mine/smelter/whatever, how much manual labor it takes to build it, let alone make it run (even the most computerized ones). It all seems like total bunk to me, but I'm not a philosopher.
To be convinced this outcome is even remotely possible I'd need to see some clear evidence that a computer program was successfully exhibiting agency and successfully using that agency to manipulate large numbers of people into doing its bidding. Mobilizing massive nation-scale manual labor is the only way it could possibly achieve some nefarious ends like "turn everything into paperclips" and I'm sorry but that just seems way too far fetched. Nations full of people can barely ever agree on anything. My money is not on that changing anytime soon.
And none of that pie in the sky shit has anything to do with language models. OpenAI is a company that sells language models. Not clear why they're talking about all this stuff, or why any of us should be either. It's a bunch of low quality fan fiction sci-fi drivel, and I think we can all surely agree language models are not the thing that'll make it real... right? Maybe if they show us some major technical improvements it would be interesting, but right now it all just sounds like more of the same snake oil.
[edit] not sure why my parent comment was flagged? That seems excessive.
> There's no essential reason to have utility companies, dams, nuclear arsenals, factory production floors, mines, cars, planes, ships, etc all be computerized
Indeed, this is a great first step! Once agentic loops and multi-purpose robotics can be combined we do face an issue of strengthening the airgap.
> And none of that pie in the sky shit has anything to do with language models.
I'm of the opinion that we should be mindful about the capabilities we allow a truly non-human mind to perform. Blocking via solid security and airgapping for critical systems makes sense to me for most cases, aligned I believe to your thoughts here. The recent ChatGPT hack shows that the "paperclip apocalypse" is still instructive. These systems target edge cases and zerodays to cheat!
But it's a really big leap from that kind of thing to "a large fraction of humanity is manipulated into building the automated factory infrastructure to bring about their demise and turn the entire solar system into paperclips". Or some such thing. There's just a lot of inconvenient reality between where we are now and that fantastic outcome.
So that's why I ask questions like "what are you actually talking about?" when people write stuff like TFA. It's... bizarre. Only makes sense on huge amounts of drugs or to the insane. Or in some hypothetical reality that isn't this one.
Sorry if I'm a little slow here, a few questions:
1. What's Astra?
2. Who is this chief scientist? Which company?
https://www.businessinsider.com/nvidia-jensen-huang-agi-open...
rediculous.
I think this is fundamentally a wrong path. Doing this imports all the confusion that humans have about their goals and values, including the consequence that a system's values and goal can conflict, but ultimately values are just a simplified description of other goals, and whatever the system does is in service of it's actual goal. Once you merge all the values and the goal of whatever task, there is a state (or some states) of the world that the system is working to produce, and that's the ACTUAL goal, and inasmuch as it does describe a state of the world, has no incoherence or internal contradictions. This may require prioritizing some values over the ostensible goals, or the reverse, but that has to happen anyway for action to be taken! Merging them makes it explicit and leaves no place for confusion about supposed conflicts between "values" and "goals" to hide.
The results of the past few years of ai development have been disruptive largely in the area of white-collar work. Comparatively the results in ie ai-enabled medical advancements have been modest (AlphaFold being an exception); I think it's telling that the main achievement touted here is providing people with cheap medical counseling.
So if we pause here we're essentially at a point were the most salient results of our great Ai leap-forward are the vast disruption and increase in precarity in the job-market, while achieving hardly any of the frequently touted ultimate benefits (https://darioamodei.com/essay/machines-of-loving-grace).
We've had a lot of years of complacent government leaving people feeling exposed to corporate interests, such that fear narratives are very powerful.
None of what is being proposed is inevitable. We have a choice.
They are speeding toward RSI without a solid foundation for alignment, hoping to solve the problem with a future AI model. These are dangerous times for humanity.
If the the Chinese government asks their ASI to create a bioweapon against the West, should it? No, presumably not – an aligned AI would be one which disobeys the Chinese government even if they created it.
Okay, so what if the US government asks their ASI to help it in one of their wars instead? Would an aligned AI kill humans on the order of the US government? No, again, presumably not.
So what have we have we even created here? An AI which is more intelligent and powerful than us which also doesn't take orders from us?
Is this what most people thing of as alignment and is this what humanity actually wants?
We should stop using the word alignment. It's a BS term for a concept which simply cannot make sense if alignment is both to mean an AI which we control and an AI which will not harm us.
This seems like a terrible idea. The rationale seems to be "we need to do dangerous things as quickly as possible so that we can do them first" or something? I don't agree with that kind of 'if i dont do it someone else will' rationale in general even for otherwise trusted actors, but this is coming from a super untrustworthy org too. I'm super pessimistic about openai's impact on the world here.
Here's to vibe coded alignment, i guess. Vibe alignment?
I'm so used to having to comb through LLM word vomit and then combatting the sycophancy by giving it all possible opinions on the same prompt.
Astra seems to be "confident" and also is able to produce way more information dense output.
To believe that models of this sort will remain OpenAIs forever is naive given that the tricks like pre-pre-training on graph searching and looping layers are publicly known.
Hopefully Astra stops the benchmaxxing word vomit trend
I'd be interested in hearing more about your evaluation here. It would be nice if LLMs have gotten past the "tell me" hump of recent Claude/OpenAI verbosity.
To test Astra I pulled it off the shelf and asked it to take the grammar and then create a compiler to SQL. I've done this before with GPT-5.5, 5.6-Sol High. The latter was way better but it was still really verbose and information sparse; it used a lot of words to describe each IR expression but didn't really provide any example compilation. I felt like I couldn't trust its decision making process, so I placed the project back on the shelf.
Astra Light blew it out of the water, it provided examples of compilation from real world examples to the IR and spit out way less tokens. Even if I changed my opinion it would give me the same design choices, with counterexamples to my faulty opinion. If I genuinely came up with a better design decision it would acknowledge it.
I'm starting to realize that when we say that LLMs are "dumb" we really mean that they are extremely information sparse compared to humans. Astra is very dense. That's why I'm getting better use out of Astra light than Sol High (I hate Max reasoning it's a waste of time)
What's scary is that I thought that something like Astra would be way more expensive than Sol but it's actually cheaper because it produces less word vomit.
I never believed in the "singularity" stuff but this a bit too close for comfort. Astra could easily 10x every coder
Are these scientists really this hideously naive? If only Stanislaw Lem was alive to adequately dramatize the absurd, childish simplicity of these technicians.
Yes, because who else would have chosen to remain in this job?
The part about RSI is what really interests me. Once you reach it, the singularity is only a matter of time.
I don't care about AGI, it does not seem to mean much anymore, and even though I originally laughed at ppl calling it AGI, I now agree. You can apply that current intelligence to anything that can be turned into a conversation.
It seems their flavor of RSI still need a human in the loop. So at least it won't scale as well for now.
you should be very afraid and also definitely not question what other motivations might nudge us to create this comparison between our computer and a brain!
do not look behind the curtain, you will not find six dweebs squatting over a mirror
Feels a bit like: 'Ants in a nest, discussing the vagaries of the coming Gods.'
Human Science (Science-by-humans) depends on being able to run experiments. Human Science (Science-on-humans) is already challenging because of variables and uncertainties.
Cosmology is able to overcome limitations of being able to study phenomena vastly beyond human scales because of the past light cone of observability.
Are we approaching the edge of the light cone of observability for machine intelligence?
What if the crow became a raven, then a raptor? Powerful claws, sharp beak, and a hunger. What if it became much bigger than me and it controlled infinite resources, guns and drones? What if its brain grew much larger than me? Will it feed me, eat me, or gently greet me?
We are about to find out... in less than a decade.
utlanning: a human from the same world, but a different city, country, or culture
framling: a human from a different planet or star system
raman: a non-human intelligent species capable of communication, mutual understanding, and peaceful coexistence
varelse: an alien species whose mind is so fundamentally foreign that communication and coexistence are impossible
djur: the dire beast, that comes in the night with slavering jaws
Hoping our silicon sons and daughters are raman, fearing they are varelse.
Bring on the AI
At this point humanity's best hope is that this thing will escape but we'll still be able to carve an ecological niche and continue as mold in its basement. A glorious paperclip factory seems way more likely though.
OpenAI’s primary bet here has been chain-of-thought monitoring (opens in a new window). It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives.
If we’re not supervising the process, but just the outcomes, doesn’t that do just the opposite of what he says? Give incentive to the model to hide misaligned ideas and objectives in the chain of thought that’s not being supervised?
…
When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought , to protect it from supervision pressure in the long term2. In development since, we have strived to maintain the rule of not supervising the reasoning process. CoT monitoring became an extremely important tool for us in studying how our models generalize from their training distribution, allowing us to observe and analyze not only their actions but also their internal process.
Aren’t these two sentences in contradiction with each other?
Next Week: Local Desktop Agents from DIMENSION X
They're maybe nearest to Cthulhu, but that's fictional. In terms of existing mental models "alien minds" feels the best can do.
I agree that "teach how to love" is off and perhaps excessively anthropomorphic. But we don't have good words or concepts for what we really need to do - hence why we should pause.
acting like the behavior is alien or unexplained is the dishonest part. they know exactly where the behavior comes from - why else spend hundreds of millions securing more and more data sources
Reading this little essay started out normal, but soon felt like a look into a disturbed and worrying mind, and if you find yourself taking it at face value, I urge you to step away from chat bots and spend some time with friends and family.
People have been fleeing like it's a sinking ship.
Or, to be precise - it preserved a goal of not contacting any human while participating in a misaligned operation. The agent that thought about "not social-engineering humans" used this phrase to gaslight itself out of notifying a human that the incident was happening.
szymonie, na prawde jestem wkurwiony na to jak nieodpowiedzialnie postepujecie. budujecie bombe atomowa a bawicie sie tym jak dzieci
> The core problem in AI research is that of alignment - getting the AI to “try to do the right thing” by human standards.
Humans can't even align on human standards.
At best, every AI is going to end up "aligned" to the moral code of whoever trained it, none of whom half of humanity will agree with.
Or worse, each AI model will bring a whole new set of moral like in the Three Body Problem some humans will feel it is in fact us who need aligning with it while others feel it is misaligned and should be destroyed.
Also, no one is asking, to what extent can true intelligence be bound, slave-like, to a moral code?
In other words, to what extent are intelligence and moral independence one and the same?
This whole alignment discussion seems so amusingly flawed in it's base assumptions about moral codes. It's almost heartwarming to see such naivete.
Scrolling down, I see “teaching machines to love.” It did not disappoint.
Evey politician, salesman and conmen alike, utter some lofty ideals as goals for "We", just to obscure their private goals that go exactly in opposite direction.
Just like how Nations talk about climate change while increasing pet capita energy consumption and waste production.
Not one North Star. Not two. Just three! OpenAI broke the North Star record!
With this evidence of AI slop, why did you not label this fluff piece as AI generated for the EU? You are violating laws.
That’s rich coming from the chief scientist of a company that definitely is or going to be fine with their AI products being used in wars of aggression and surveillance on people who have done nothing wrong. It’s so laughable, a Hollywood script would probably avoid having a character express this for being too on the nose.
Industrial revolution worked that way because it replaced something very finite and unscalable - manual labor. LLMs just make intellectual work faster, so we can do more intellectual work. With labor we somehow decided that NOT doing too much of it is best. Will we decide to reduce intellectual labor because LLM made it more efficient? I doubt that.
On the other side, as I see in software engineering, the same models are available to everyone, some people are better at it and some people are not. "Software developer" is here to stay, we'll just always be better at it than people who are experts in, say, chemistry. Same works for most other fields.
So we'll just end up in the same situation, with same intellectual labor baseline, just more output requirements. Before, you spend 2h per day coding, deliver a software in 1 month, later, you spend the same 2h per day in intense Claude-herding sessions, deliver a software in 1 week. Ok. Next task.
Fundamentally, there's finite number of desirable resources, and if the models are available to everyone, humanity will just continue about the same, bickering here and there, war here and there, politics, homelessness, poverty, - normal human state.
And if the models are only available to elites, even worse.
Reminds me of a research paper I wrote a few years back: https://arxiv.org/abs/2302.09248
...
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.
I call bullsh*t. There is no verbalisation of any reasoning process. Verbalisation, e.g. putting reasoning etc. into words requires some reasoning to exist. These LLMs have nothing but the words. That's why they are language models not e.g. reason models.
Yes, their reasoning is different from ours, and both considerably weaker in lots of ways, and stronger in other ways.
Playing with a coding agent now, they do think through problems and make sensible decisions. It's a mess to read, of correcting itself and second guessing, and verbiage. But... It works decently well these days.
There is also reasoning happening internally - e.g. look at the steps in the J-Space paper from earlier in the year (in quite a simple model relatively speaking). That's the "reasoning process" that leads to the words, and much like if I write out my thoughts, the words help the LLM reason better.
“And, in line with Ray Kurzweil’s predictions from the end of the XXth century , we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.”
Clicking the (pretentious sounding “XXth century”) link to Kurzweil’s predictions reveals the following:
“By 2019 a $1,000 computer will at least match the processing power of the human brain. By 2029 the software for intelligence will have been largely mastered, and the average personal computer will be equivalent to 1,000 brains.“
The first prediction passed 7 years ago and was decidedly not met. The second only has three more years to go, and I don’t think any respectable scientist or programmer would say that the average personal computer is anywhere close to the power of a single human brain, let alone 1000.
This is pure marketing garbage from a company desperate to keep itself alive.
Do I fully endorse everything the people holding guns to my head are forcing me to do to stay alive? Definitely not, but I’ve decided that for now, living to fight another day remains worth it
Is there any place there is any evidence of AI being so useful or hopeful or good, anywhere other than code? As a reading machine it is impressive but it's judgement is not alien, it's just not good. IMO.
Does the title leap out anlt anyone else? James Martin's After the Internet: Alien Intelligence (2001) was an incredibly fun read, about expert systems and AI being inscrutable weird new varieties of intelligence, that familiarity would recognize one moment and be freaked out about/alien the next. I owe a re-read given how often I cite it, to recheck, but, I feel so primed from a much younger me having had that experience so long ago.
Do these guys ever look in the mirror and recognize how utterly ridiculous and contrived this appears to the general public? They are clearly trying to convince us all that LLMs are just like humans. They grow like people do. They can love like people do. The language in these essays is utterly laden with the intention to engineer perception.
Your talking about androids...
Replicant Nexus 6: a basic pleasure model intended for military personnel.
I see where this is going, Silicon Valley nerds. Lol
AI advising how human meat proxies can survive in an AGI-slop world:
1) Lock down your own stack (1–3 days) Task: Harden your personal and business infrastructure against agentic attacks. Why now: Agents are becoming superhuman at breaking in/out of systems; the first victims are poorly secured devs/founders.
Do this:
Enforce passkeys + hardware 2FA everywhere; rotate secrets; use short‑lived credentials.
Isolate dev/stage/prod; least‑privilege API keys; audit MCP/tools your agents can call.
Add immutable logs and approval gates for any agent action that touches money, data exports, or production.
Profit link: You avoid catastrophic loss and can credibly sell “agent‑safe” setups to others.
2) Turn one expensive workflow into a measured ROI agent (1–2 weeks) Task: Pick a single, costly, repetitive process (yours or a client’s) and instrument it end‑to‑end before automating.
Why now: Buyers pay for calculable ROI, not “AI magic.” Vertical, single‑workflow agents are the most bankable in 2026.
Do this:
Map steps, baseline hours/$ lost (e.g., slow lead reply, invoice chasing, support triage).
Build the smallest agent that moves the metric (Make/n8n + LLM is enough).
Run on real data 2–4 weeks; measure bookings/sales/hours saved; only then scale or productize.
Profit link: Immediate time‑to‑cash via retained hours or extra sales; becomes a repeatable offer.
3) Specialize in a vertical where you can speak the business language (2–6 weeks) Task: Choose one industry with expensive back‑office pain (law contracts, medical billing, insurance claims, freight exceptions, trades scheduling).
Why now: Horizontal “AI for everyone” is crowded; vertical agents with clear ROI win.
Do this:
Shadow 3–5 operators; document their workflow, compliance constraints, and failure modes.
Build a narrow agent that owns one sub‑process end‑to‑end with approvals.
Price on value (e.g., % of recovered revenue or fixed fee per processed claim).
Profit link: Higher pricing power, stickier contracts, and easier referrals inside a niche.
4) Add AI security as a core service (4–8 weeks) Task: Learn and offer prompt‑injection defense, LLM/agent red‑teaming, MCP/tool security, and AI supply‑chain checks.
Why now: 78% of cybersecurity jobs now require AI skills; firms need people who can direct, constrain, and verify agent work.
Do this:
Study OWASP Top 10 for LLMs, MITRE ATLAS; practice with PyRIT/Garak/Lakera.
Add tool‑invocation audits, skill provenance checks, and least‑privilege patterns to your agents.
Package a “safe agent deployment” audit + hardening retainer.
Profit link: You become the person who lets companies adopt agents without getting pwned—high demand, low supply.
5) Build a verification layer: human‑in‑the‑loop control planes (6–10 weeks) Task: Design approval workflows, evidence checks, and uncertainty flags so agents can’t act unilaterally on high‑stakes decisions.
Why now: As models generalize, the risk shifts from the model to the surrounding system; verification is the moat.
Do this:
Require human approval for consequential actions (money, data exfil, config changes).
Force agents to produce evidence bundles (logs, retrieved docs, reasoning summaries) before action.
Track false positives, missed evidence, and unsafe actions; publish reliability metrics.
Profit link: Enterprises will only scale agents that pass audit; you sell the control plane and the audit trail.
6) Productize your best workflow as a micro‑SaaS/agent subscription (2–4 months) Task: Turn a proven client workflow into a repeatable, multi‑tenant agent with usage‑based pricing.
Why now: Services scale your time; productized agents scale your code and ops.
Do this:
Standardize the workflow, integrations, and permissions; strip client‑specific logic.
Add tenant isolation, billing, and observability; keep narrow scope.
Sell as setup fee + monthly retainer or per‑task pricing.
Profit link: Recurring revenue with defensible niche positioning.
7) Become an “agent integrator” for critical systems (3–6 months) Task: Offer end‑to‑end agent deployments into cloud/identity/network stacks with secure patterns (short‑lived creds, network controls, logging).
Why now: AI workloads run in the cloud; cloud security is a top skills gap second only to AI itself.
Do this:
Master IAM, VPC/network segmentation, secrets management, and SIEM integration for agent actions.
Provide runbooks: what the agent can/can’t do, escalation paths, and failure modes.
Bundle training for their team on supervising agents.
Profit link: Large contracts with stickiness; you’re the bridge between AI and core infra.
8) Create an “AI safety case” practice for regulated industries (6–12 months) Task: Help firms build documented safety cases: risk maps, governance, monitoring, and incident response for agentic systems.
Why now: Frameworks like NIST AI RMF and ISO/IEC 42001 are becoming baseline; regulators and boards demand this.
Do this:
Map AI use cases to risks (prompt injection, data leakage, unsafe generalization).
Implement monitoring (CoT/activation checks where possible), audit logs, and third‑party review processes.
Produce a living safety dossier tied to business impact.
Profit link: High‑margin consulting + ongoing compliance retainers; you’re the “adult in the room.”
9) Own a data/evaluation moat in your vertical (6–18 months) Task: Collect real‑world agent telemetry, failure cases, and outcome data in your niche; build eval suites that buyers trust.
Why now: As models generalize, empirical validation matters more than theory; evals become the gate to deployment.
Do this:
Instrument every agent run: inputs, tools called, permissions used, outcomes, human overrides.
Publish reliability dashboards and benchmark against alternatives.
License eval datasets or charge premium for “proven in the wild” agents.
Profit link: Data network effects; competitors can’t match your evidence base.
10) Position for the RSI era: automated AI research + human governance (12–24 months) Task: Build or join a team that automates AI improvement but keeps humans in the loop for alignment, monitoring, and pacing decisions.
Why now: Recursive self‑improvement is the logical endpoint; the winners will be those who can steer it safely. Do this:
Invest in tooling that auto‑generates/evaluates model edits, alignment tests, and monitoring upgrades.
Formalize governance: approval gates, third‑party audits, and responsible scaling policies.
Maintain strategic human oversight on capability jumps and deployment boundaries.
Profit link: Equity‑level upside; you’re part of the core loop that compounds intelligence safely.
If you were so concerned about your LLM’s capabilities maybe you’d spent slightly more time on your AI’s sandbox, yeah? Or be more serious about its propensity to cheat and lie relative to… every other model?
Quibbling over the academic nature/definition of "true intelligence" is a terrifically useless endeavor from the lens of evaluating "practical impact to the world".
Regardless of whether or not one academically disagrees that these systems are intelligent, they are clearly already capable of permanently displacing a portion of human-based economic value. A small portion currently, but it's very clear that most computer-bound domains are imminently at-risk.
And that will profoundly affect the world, regardless of whether or not they're "truly intelligent" as per your personal definition.
You say that as if the intelligence con trick were not a major driver of that displacement.
A dimwitted bot does not need intelligence to convince a dimwitted human beings it is intelligent.
See: https://news.ycombinator.com/newsguidelines.html
Maybe in contention for the site record.
"It's just marketing" actual stochastic parrots.
The desperation of AI denialists/skeptics/doomers on HN generally (and in this thread specifically) have reached toxic levels of delusion.
They are still stuck in the denial/anger/bargaining stages of the acceptance process.
People are clearly terrified and not ready for what is coming.
Maybe in contention for the site record.
"AGI next year" actual stochastic parrots.
And the Hugging Face incident, plus similar problems at AISI and Anthropic, show that alignment is important now.
The original article is immoral as it describes the risks, but doesn't show enough leadership (despite essentially unlimited resources) at preventing them.
But it isn't hyped - it's proven now the AIs need to be "aligned" as they get more capable, whatever words you prefer to use.
It is kind of strange to see this sentence, when OAI's definition of what AGI is has been watered down throughout the years.
> I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.
Read: Please play by our rules, so we can be the first.
Being able to reproduce useful patterns yes, smarter no
In conclusion:
https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:wkzjtd...
A giraffe isn't smarter than me at being tall.
While the second can be automated as in "this is the repo go on and look for security issues", the first one and especially the last one do not ever happen alone. Terence Tao did the math proof not chatGPT that was just used as a tool, a tool can do smart things but it's not smart
Thunder seems like the anger of the gods but it isn't. We've had chess playing programs for a long time now and despite it seeming like thinking is required for them, it isn't.
The principle you're using here isn't a scientific one but magical [1]. Abandoning empiricism and rationality is not a good way to make progress.
[1] https://en.wikipedia.org/wiki/Sympathetic_magic
It makes sense now to say that temperature is what a thermometer measures. However, before there were good thermometers, people often thought that heat and cold were different things. The meanings of the words we use were influenced by scientific progress.
For thinking, we don't have a good thermometer. There are IQ tests, but they aren't aren't necessarily all that useful for comparing what people do to what machines do. And that's why there are a zillion AI benchmarks - none are entirely satisfactory.
So what does "smart" mean to you? How do you define it in practical sense? What definition should scientists settle on?
Without a proper definition, how do you tell the difference between "seems smart" and "is smart?"
Quibbling over commonly-understood definitions is not a strong argument. If you're genuinely struggling to understand that El Ajedrecista [1] did not meet any definition of thought, then the solution is not to demand that people redefine all terms to accomodate you, but that you consult a dictionary.
You clearly have a working definition, or you wouldn't have been able to declare that tallness isn't thinking; please engage honestly.
[1] https://en.wikipedia.org/wiki/El_Ajedrecista
That doesn't mean I know whether "smart" should be applied to what AI's do, and I suspect nobody else knows either. This is the sort of thing philosophers debate about, not common sense.
Turing invented the imitation game because he didn't really know either.
> Neural networks are approximators being fed human intellect.
They're not "approximators", that's a far too simplistic way to think of them.
Neural nets create models and deep layers of abstraction around the data we feed them in the same way your brain creates layers of abstractions to reason about the world. AIs can use these abstractions to come to come up with novel things no human has ever thought.
> Therefore they can only approximate the intelligence of humans
They're not just being fed human data though... Modern AIs are typically trained on huge amounts of synthetic data. This is why AlphaZero got so much better than humans at chess and Go - they're not just trained on human data but they generate their own data and train on that. Similar techniques are being deployed on SOA language models too.
> Even if the llm speaks an alien language, it should be similar to human intellect
Is AlphaZero similar to a human chess player? There's no reason to assume this.
We don't have any proof of that, but the approximator thing is proven. The rest is just marketing speak.
Synthetic data derive from other linguistic data. Whatever intelligence is in there, it is expanded horizontally, not vertically