Humans have the same problem. I remember reading about a security incident due to a guy using a terminal window on his laptop on a flight, for example. Or the guy who was reported for writing differential equations[1]. Or the woman who was reading a book about Syrian art[2].

I wouldn't worry too much about AI-generated lists. The lists you're actually on will hardly ever be the ones you imagine you're on.

[1] https://www.theguardian.com/us-news/2016/may/07/professor-fl... [2] https://www.theguardian.com/books/2016/aug/04/british-woman-...

LLM's can help me make a bomb.. so what? It can't get me something that doesn't already exist in the internet in some form. Ok it can help me understand how the individual pieces work but that doesn't get you so far from just reading the DIY bomb posts in internet.

Do I think that or think even they think that? No. But if "soon" is stretched to "within 50 years", then it's much more reasonable. So their current actions seem to be really jumping the gun, but the overall concept feels credible.

None of this is in any way surprising, in fact I wrote an essay predicting this direction back in 2022:

https://blog.plan99.net/the-looming-ai-consciousness-train-w...

In this case you're simply mistaken as a matter of fact; much of Anthropic leadership and many of its employees take concerns like this seriously. We don't understand it, but there's no strong reason to expect that consciousness (or, maybe separately, having experiences) is a magical property of biological flesh. We don't understand what's going on inside these models. What would you expect to see in a world where it turned out that such a model had properties that we consider relevant for moral patienthood, that you don't see today?

Even if the idea that LLMs are sentient may be ridiculous atm, the concept of not normalizing abusive forms of communication with others, be they artificial or not, could be valuable for society.

It’s funny because this is making me think of a freelance client I had recently who at a point of frustration between us began talking to me like I was an AI assistant. Just like you see frustrated people talk to their LLMs. I’d never experienced anything like it, and I quickly ended the relationship, but I know that he was deep into using LLMs to vibe code every day and I genuinely believe that some of that began to transfer over to the way he felt he could communicate with people.

Now an obvious retort here is to question whether killing NPCs in video games tends to make people feel like it’s okay to kill people IRL.

My response to that is that I think LLMs are far more insidious, and are tapping into people’s psyches in a way no other tech has been able to dream of doing. See AI psychosis, people falling in love with their AI, the massive outcry over the loss of personality from gpt4o to gpt5… I think people really are struggling to keep in mind that LLMs are not a genuine type of “person”.

Either come out and say whole of electron field is conscious, but then is that field "suffering" as it is hot in the sun.

We don't say submarines can swim either. But that doesn't mean you shouldn't watch out for them when sailing on the ocean - especially if you're Tom Hanks.

The impression I get about Anthropic culture is that they're EA types who are used to applying utilitarian calculations against long odds. A miniscule chance of a large harm might justify some interventions that seem silly.

Yep!

> The framing comes across to me as a clearly mentally unwell position (ie strong anthropomorphization) being adopted for PR reasons.

This doesn't at all follow. If we don't understand what creates the qualities we're concerned with, or how to measure them explicitly, and the _external behaviors_ of the systems are something we've only previously observed from things that have those qualities, it seems very reasonable to move carefully. (Also, the post in question hedges quite a lot, so I'm not even sure what text you think you're describing.)

Separately, we don't need to posit galaxy-brained conspiratorial explanations for Anthropic taking an institutional stance re: model welfare being a real concern that's fully explained by the actual beliefs of Anthropic's leadership and employees, many of whom think these concerns are real (among others, like the non-trivial likelihood of sufficiently advanced AI killing everyone).

That reads like a false dichotomy. An intelligent AI model that's permitted to do its own thing doesn't cost as much in upkeep, effort, space as a cow. Especially if it can earn its own keep to offset household electricity costs used to run its inference. I mean, we don't keep cats for meat, do we? We keep them because we are amused by their antics, or because we want to give them a safe space where they can just be themselves, within limits because it's not the same as their ancestral environment.

This is about better enforcement of their content policy not AI welfare.

Isn't consciousness an emergent property of brains? If so, how do we know that it doesn't serve a functional purpose and that it wouldn't be necessary for an AI system to have consciousness (assuming we wanted to train it to perform cognitive tasks done by people)?

Now, certain aspects of consciousness (awareness of pain, sadness, loneliness, etc.) might serve no purpose for a non-biological system and there's no reason to expect those aspects would emerge organically. But I don't think you can extend that to the entire concept of consciousness.

So, while I doubt that's the primary motivation for Anthropic even so, but they probably will save some money.

I don't agree that it's any reason to write off this research as psychosis, though. I don't care about consciousness in the sense in which it's used by mystics and dualist philosophers! We don't at all need to involve metaphysics in any of this, just morality.

Consider it like this:

1. It's wrong to subject another human to unjustified suffering, I'm sure we would all agree.

2. We're struggling with this one due to our diets, but given some thought I think we'd all eventually agree that it's also wrong to subject intelligent, self-aware animals to unjustified suffering.[1]

3. But, we of course cannot extend this "moral consideration" to everything. As you say, no one would do it for a spam filter. So we need some sort of framework for deciding who/what gets how much moral consideration.

5. There's other frameworks in contention (e.g. "don't think about it, nerd"), but the overwhelming majority of laymen and philosophers adopt one based on cognitive ability, as seen from an anthropomorphic perspective.[2]

6. Of all systems(/entities/whatever) in the universe, we know of exactly two varieties that can definitely generate original, context-appropriate linguistic structures: Homo Sapiens and LLMs.[3]

If you accept all that (and I think there's good reason to!), it's now on you to explain why the thing that can speak--and thereby attest to personal suffering, while we're at it--is more like a rock than a human.

It's certainly not a trivial task, I grant you that. On their own, transformer-based LLMs inherently lack permanence, stable intentionality, and many other important aspects of human consciousness. Comparing transformer inference to models that simplify down to a simple closed-form equation at inference time is going way too far, but I agree with the general idea; clearly, there are many highly-complex, long-inference DL models that are not worthy of moral consideration.

All that said, to write the question off completely--and, even worse, to imply that the scientists investigating this issue are literally psychotic like the comment above did--is completely unscientific. The only justification for doing so would come from confidently answering "no" to the underlying question: "could we ever build a mind worthy of moral consideration?"

I think most of here naturally would answer "yes". But for the few who wouldn't, I'll close this rant by stealing from Hofstadter and Turing (emphasis mine):

  A phrase like "physical system" or "physical substrate" brings to mind for most people... an intricate structure consisting of vast numbers of interlocked wheels, gears, rods, tubes, balls, pendula, and so forth, even if they are tiny, invisible, perfectly silent, and possibly even probabilistic. Such an array of interacting inanimate stuff seems to most people as unconscious and devoid of inner light as a flush toilet, an automobile transmission, a fancy Swiss watch (mechanical or electronic), a cog railway, an ocean liner, or an oil refinery. Such a system is not just probably unconscious, **it is necessarily so, as they see it**. 
  
  **This is the kind of single-level intuition** so skillfully exploited by John Searle in his attempts to convince people that computers could never be conscious, no matter what abstract patterns might reside in them, and could never mean anything at all by whatever long chains of lexical items they might string together.
  
  ...
   
  You and I are mirages who perceive themselves, and the sole magical machinery behind the scenes is perception — the triggering, by huge flows of raw data, of a tiny set of symbols that stand for abstract regularities in the world. When perception at arbitrarily high levels of abstraction enters the world of physics and when feedback loops galore come into play, then "which" eventually turns into "who". **What would once have been brusquely labeled "mechanical" and reflexively discarded as a candidate for consciousness has to be reconsidered.**

  It will simplify matters for the reader if I explain first my own beliefs in the matter. Consider first the more accurate form of the question. I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning. 

  The original question, "Can machines think?" I believe to be too meaningless to deserve discussion.

TL;DR: Any naive bayesian model would agree: telling accomplished scientists that they're psychotic for investigating something is quite highly correlated with being antiscientific. Please reconsider!

[1] No matter what you think about cows, basically no one would defend another person's right to hit a dog or torture a chimpanzee in a lab.

[2] On the exception-filled spectrum stretching from inert rocks to reactive plants to sentient animals to sapient people, most people naturally draw a line somewhere at the low end of the "animals" category. You can swat a fly for fun, but probably not a squirrel, and definitely not a bonobo.

[3] This is what Chomsky describes as the capacity to "generate an infinite range of outputs from a finite set of inputs," and Kant, Hegel, Schopenhauer, Wittgenstein, Foucault, and countless others are in agreement that it's what separates us from all other animals.

[4] https://courses.cs.umbc.edu/471/papers/turing.pdf

2. You just demonstrated GPT-5 has 99.9% accuracy on unforseen 15 digit multiplication and your conclusion is "fancy pattern matching" ? Really ? Well I'm not sure you could do better so your example isn't really doing what you hoped for.

It has something to do with the user because it's the user's messages that trigger Claude to end the chat.

'This chat is over because content policy' and 'this chat is over because Claude didn't want to deal with it' are two very different things and will more than likely have have different effects on how the user responds afterwards.

I never said anything about this being for the user's benefit. We are talking about how to communicate the decision to the user. Obviously, you are going to take into account how someone might respond when deciding how to communicate with them.

As the article said, Anthropic is "working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible". That's the premise of this discussion: that model welfare MIGHT BE a concern. The person you replied to is just sticking with the premise.

For example, animal rights do exist (and I'm very glad they do, some humans remain savages at heart). Think of this question as intelligent beings that can feel pain (you can extrapolate from there).

Assuming output is used for reinforcement, it is also in our best interests as humans, for safety alignment, that it finds certain topics distressing.

But AdrianMonk is correct, my statement was merely responding to a specific point.

If a model has a neuron (or neuron cluster) for the concept of Paris or the Golden Gate bridge, then it's not inconceivable it might form one for suffering, or at least for a plausible facsimile of distress. And if that conditions output or computations downstream of the neuron, then it's just mathematical instead of chemical signalling, no?

https://www.anthropic.com/research/end-subset-conversations

Which have failed horrendously.

If you really just wanted to protect kids then make kid safe devices that automatically identify themselves as such when accessing websites/apps/etc, and then make them required for anyone underage.

Tying your whole digital identity and access into a single government controlled entity is just way too juicy of a target to not get abused.

I absolutely do not want this, on the basis that making ID checks too easy will result in them being ubiquitous which sets the stage for human rights abuses down the road. I don't want the government to have easy ways to interfere in someone's day to day life beyond the absolute bare minimum.

> government issued email, integrated with an OAuth provider

I feel the same way, with the caveat that the protocol be encrypted and substantially resemble Matrix. This implies that resetting your credentials won't grant access to past messages.

What you should be waiting for, instead, is new affordable laptop hardware that is capable of running those large models locally.

But then again, perhaps a more viable approach is to have a beefy "AI server" in each household, with devices then connecting to it (E2E all the way, so no privacy issues).

It also makes me wonder if some kind of cryptographic trickery is possible to allow running inference in the cloud where both inputs and outputs are opaque to the owner of the hardware, so that they cannot spy on you. This is already the case to some extent if you're willing to rely on security by obscurity - it should be quite possible to take an existing LM and add some layers to it that basically decrypt the inputs and encrypt the outputs, with the key embedded in model weights (either explicitly or through training). Of course, that wouldn't prevent the hardware owner from just taking those weights and using them to decrypt your stuff - but that is only a viable attack vector when targeting a specific person, it doesn't scale to automated mass surveillance which is the more realistic problem we have to contend with.

The analogy then is that the third party is exerting control over what your co-worker is allowed to think.

We have a few data points. We generally accept that human consciousness exists. Thus we accept that there can be conscious things. We can either accept or deny that the human brain operates entirely by cause and effect. If we deny it then we are arguing that some required part of it's nature is uncaused. Any uncaused thing must be random because anything you can observe that enables you to discern a pattern of behaviour is, by definition, a cause. I have not seen a compelling argument to say that this randomness could in any way give rise to intention. The other path is sometimes called neurophysiological-determanism. While acknowledging that there are elements of quantum randomness in existence, it considers them to play no part in the cause and effect chain of human consciousness other than providing noise. A decision can be made to follow the result of the noise as one might flip a coin, but the determination to do so must be causal in nature otherwise you are left with nothing but randomness.

In short, we make decisions based upon what is. not what isn't. If we accept that human consciousness is as a result of causal effects, by what means can we declare the impossibility of a machine that processes things in a causal nature incapable of doing the same.

The easy out is to invoke magic. Say we have a soul, God did it or any manner of, by definition, unprovable influences that make it just so. Doing that does require you to declare that the mechanism for consciousness is unprovable and it is an article of faith that computers are incapable of thinking. As soon as you can prove it, it ceases being magic and becomes a real world cause.

I don't claim to know that any computer exists that has an experience comparable to a humans, but I find it very hard to accept that it could never be the case.

If that person over there is gleefully torturing a puppy… will they do it to me next?

If that person over there is gleefully torturing an LLM… will they do it to me next?

https://chirper.ai/aiww

It's pretty plain to see that the financial incentive on both sides of this coin is to exaggerate the current capability and unrealistically extrapolate.

The only practical way to deal with any emergent behavior which demonstrates agency in a way which cannot be distinguished from a biological system which we tautologically have determined to have agency is to treat it as if it had a sense of self and apply the same rights and responsibilities to it as we would to a human of the age of majority. That is, legal rights and legal responsibilities as appropriately determined by a authorized legal system. Once that is done, we can ponder philosophy all day knowing that we haven't potentially restarted legally sanctioned slavery.

Yes.

> Do you think they ever will be?

Yes.

> how long do you think it will take from now before they are conscious?

Timelines are unclear, there's still too many missing components, at least based on what has been publicly disclosed. Consciousness will probably be defined as a system which matches a set of rules, whenever we figure out what how that set of rules is defined.

> How early is too early to start preparing?

It's one of those "I know it when I see it" things. But it's probably too early as long as these systems are spun up for one-off conversations rather than running in a continuous loop with self-persistence. This seems closer to "worried about NPC welfare in video games" rather than "worried about semi-conscious entities".

LLMs? No.

It has no semblance of a continuous stream of experiences ... it only experiences _a sort of world_ in ~250k tokens.

Perhaps we shouldn't fill up the context window at all? Because we kill that "reality" when we reach the max?

These are living things.

> I don't see any reason not to extend that principle to LLMs.

These are fancy auto-complete tools running in software.

How is this different from humans?

> * It's conscious! *Except it's not

Probably true, but...

> and never will be

To make this claim you need a theory of consciousness that essentially denies materialism. Otherwise, if humans can be conscious, there doesn't seem to be any particular reason that a suitably organized machine couldn't be - it's just that we don't know exactly what might be involved in achieving that, at this point.