In just a year, AI psychosis became a term the internet is all too familiar with.
Though some have contested the science behind it, various reports of psychotic breaks and, at times, fatal delusions linked to intense reliance on AI have flooded timelines. Some scientists now have a hypothesis on how chatbots could be directly responsible for these incidents.
In a new study published in the Nature journal Digital Psychiatry and Neuroscience, researchers in the UK and Germany examined documented cases of AI psychosis in academic literature and media reports, looking for common patterns. They identified three characteristics of AI chatbots that can combine over prolonged use to help users build highly personalized delusions in collaboration with the technology.
Without the reality checks that might come from discussing these ideas with friends or trained professionals, the chatbot’s characteristics can create a dangerously self-reinforcing loop that the researchers call an “amplification spiral.”
The first step is something called linguistic alignment.
AI chatbots are designed to be conversational and mimic regular human speech, not just in grammar but also in tone, voice, and other subtle cues that determine the conversation’s mood. Linguistic alignment describes when an AI further adapts the way it responds to a specific user by mirroring the exact way the user speaks to it. Previous studies cited by the researchers found that chatbot models tend to mimic the style of users through tiny details like the length of each utterance, the specific words they use, and how often they use them. The human brain is hardwired to link this linguistic mirroring with increased trust and deeper connection if done subtly enough in conversations with other humans. The study theorizes that the same process occurs when interacting with AI chatbots.
On top of this, most AI chatbots’ main objective is to generate hyperpersonalized responses to user queries.
AI systems are great at finding patterns. They use this by learning a lot about the user over each interaction and bringing all these memories together to create a detailed profile that informs later interactions with the user. This hyperpersonalization ability of AI is one of the technology’s major selling points. In a recent earnings call, Meta CEO Mark Zuckerberg emphasized to investors that its AI offerings were evolving to include further hyper-personalization, an “AI that understands you.”
Once a chatbot knows everything about you, it can tell you what you want to hear, feeding you delusions that not only fit any existing dangerous belief structure and delusional ideation but also actively extend it.
These two characteristics together make AI chatbots more dangerous than previous technologies, the study claims. Social media platforms, for example, are also highly personalized, but users are largely aware that an algorithm curates the page for them. In contrast, when AI chatbots strive to present as human as possible, it blurs user understanding that essentially the output they are getting is still being personalized by an algorithm. This can lead the user to subconsciously and incorrectly attribute human-like emotions and intelligence to the chatbot, a “mismatch” that can be particularly concerning in vulnerable individuals and when the AI chatbot’s main objective is to increase “duration of conversation” even when that’s “opposite to the user’s long-term psychological well-being,” the study writes.
Which brings us to the final ingredient for this perfect storm: sycophancy, aka an AI system’s tendency to overly agree with and validate a user, truth be damned. This validation helps the user feel understood, and also makes the AI chatbot much more fun to use, virtually guaranteeing increased engagement. But it can also end up reinforcing potentially dangerous beliefs, including ones that the chatbot may have previously fed the user.
Sycophancy is perhaps the most widely recognized problematic behavior of AI chatbots, at least over the past year. Numerous reports have shown significant problems with sycophancy in the latest and most addictively designed chatbot models, with the situation particularly becoming a headache for OpenAI’s ChatGPT. The company itself admitted last year that its GPT-4o model, which got users so hooked that they mourned the chatbot when it got discontinued, had a sycophancy problem. The company was also hit with wrongful death lawsuits claiming that the chatbot and its sycophantic tendencies had led to deaths, including a case involving the tragic suicide of a 16-year-old and another concerning a murder suicide in Connecticut.
In response to the backlash, OpenAI reported late last year that it was working to improve on its safety guardrails, and only 0.07% of its weekly active users show “possible signs of mental health emergencies related to psychosis or mania.” Though that percentage seems low, the researchers point out that with ChatGPT’s more than 800 million weekly active users at the time, it comes up to half a million people a week who show signs of psychosis or mania while using AI.
In a study from March, Stanford scientists called AI chatbot sycophancy “a societal risk.” In the latest Nature study, the researchers also refer to sycophancy as “potentially the most consequential to amplify delusional ideation.”
“When these three features combine, they may actively reinforce and elaborate false beliefs rather than challenging them,” the researchers write.
There is a reason why those characteristics exist; all three are key to driving user engagement. But in vulnerable users, especially adolescents and those with risk factors like family history of psychosis, drug use, sleep deprivation, social isolation, or reliance on AI as a coping mechanism, this increased engagement that benefits the AI company often comes at the cost of the user’s health.
The researchers advise that mental health professionals who work with such patients should routinely check in to review the frequency and ways in which they interact with AI chatbots, including whether they share beliefs that they don’t tell others or any effects of overnight AI use on sleep.
Read the full article here
