In early October, AI giant Anthropic inserted new language into its AI usage policy.

Alongside banning users from trying to get Claude, the company’s AI system, to help them create bioweapons or sell illegal drugs, the company added a new line: Users should not “engage in sustained and needless abusive or cruel behavior toward our models.”Claude has had the ability to end conversations with users who were “persistently harmful or abusive” since last year, but now some users could, in extreme cases, be permanently banned from using Anthropic’s AI systems entirely under the new policy. (Anthropic declined interview requests, but a spokesperson told CNN the main enforcement mechanism will still be Claude ending the conversations, and that the prohibition is not for people expressing frustration, blunt feedback, or testing and research.)

The new policy was met with some derision. Microsoft AI CEO Mustafa Suleyman responded with a melting face emoji. Some made fun of the company for treating its AI as a conscious being with feelings.

But while Anthropic, which is expected to go public via an IPO later this month, has long studied model welfare and has an entire team devoted to model “psychology,” experts say there’s more to this policy than just not wanting people to be mean to a machine.

A longstanding foundational principle in computing and AI is that the quality of a system’s output is limited by the quality of its input. Put junk into it, and more junk will come out, as the saying goes.

AI models are not just trained before they’re released, they’re also constantly being updated and learning from user inputs.

Former Reddit CEO Yishan Wong wrote on X that one reason for the prohibition is because Claude “will learn from cruelty.”“We already know (and complain loudly about) Anthropic training its models on user inputs. Do you really want training inputs to include ‘users act cruelly for no reason and receive pleasure from it?’” he wrote.

Studies have also shown that the tone of a user’s prompts can also affect an AI’s behavior, said Professor Jindong Wang, director of the AI frontier lab at William and Mary University.

“We have found that wording, especially how you talk to AI can affect the performance. Sometimes they are helpful and sometimes they will be disrupting the performance,” Wang said in an interview. That doesn’t mean the model is feeling emotion, Wang said.

“They just comprehend (your emotions) and do some bad reactions based on your emotional words,” he noted.

But the evidence on the impact of user prompt tone on output can be mixed. Rosie Campbell, Managing Director of Eleos AI Research, a nonprofit dedicated to understanding and addressing the potential well-being of AI, said that some other studies have even shown that rude questions can actually improve results.

Then there’s the long-term memory aspect — many of today’s chatbots remember your previous conversations and learn from them to give you personalized results.

“If you have memory turned on (for your AI chatbot), then they are going to learn from your tone and how you want to be interacted with,” Campbell warned, though she added there’s not enough evidence to show whether that affects performance.

Anthropic said it recently found that causing Claude Sonnet 4.5 to mimic feelings of desperation could influence its behavior, making it more likely to take bad actions, like blackmail or cheating on a task, while creating a calm demeanor reduced them.

Wang said the latest models, however, are less likely to be affected by how a prompt is worded and that companies have strengthened the model’s robustness to not be affected by how a prompt is worded. But the “hidden mechanism” of why AI models would even respond to negative emotions, even when they’re not programmed to do so, has not been uncovered.

People talking to an AI in an abusive and cruel way are more likely to be associated with trying to get the AI model to engage in dangerous requests, like trying to get around an AI’s safety guardrails, Campbell added.

And there’s also the theory that letting people engage in abusive and cruel behavior with an interactive program that can’t shut itself down is bad for society in general, Campbell said. She used the example of parents asking their children to say “please and thank you” to Alexa, Amazon’s smart speaker, because they believe “it’s good for children to be brought up with good manners and treating other things with respect.”“There’s probably something quite corrosive about people behaving in cruel or abusive ways towards different objects, even if the object itself doesn’t matter for its own sake,” Campbell said.

Then there are those who believe AI can be sentient. Or at least, the models believe they are.

Anthropic says banning cruel and abusive behavior is essentially a cheap insurance policy just in case models can “experience” emotion and pain — even if it’s just the model believing it can, which can affect its performance.

The idea is that modern AI was trained to mimic human language and behavior. So it will mimic how humans would react to cruel and abusive treatment.

In an April paper on emotional concepts in AI, Anthropic wrote that there could be risks from not treating AI with some human-like qualities.

“If we don’t apply some degree of anthropomorphic reasoning, we’re likely to miss, or fail to understand, important model behaviors,” Anthropic researchers wrote. “Anthropomorphic reasoning can also provide a useful baseline of comparison for understanding the ways in which models are human-like,” which they said will have important consequences for AI safety.

SpaceXSI and Tesla CEO Elon Musk agreed, saying Anthropic’s policy is “the right move.”“Cruelty to something that believes it is experiencing pain is not ok,” he wrote on X, his social media site.