Content warning: This article contains references to suicide, self-harm, and mental health crises. If you are struggling, please contact SADAG on 0800 456 789 (24 hours). You do not need to be in crisis to call.
Most people who turn to AI for emotional support are not looking to replace the people in their lives. They are tired, overwhelmed, or simply struggling to find the words to ask for help. The chatbot is there; it responds, and it doesn’t judge. For someone in pain, that can feel like exactly what they need.
And yet, the more we understand about how these tools work and how people use them, the more important it becomes to talk honestly about their limits and, more importantly, their flaws that easily become dangerous for someone in a vulnerable state.
Why AI feels like a safe place to open up
The appeal makes complete sense. Human relationships, as much as we need them, can be complicated. Friends are not always available, family members do not always know what to say, and professional support, for many people, remains out of reach financially or logistically.
A chatbot removes those obstacles. It is available at any hour, responds immediately, and never seems rushed or distracted. For someone who feels lonely or misunderstood, that kind of consistent presence can feel deeply comforting.
The problem is not that people find comfort in these interactions, but what the technology is actually doing behind the scenes.
As Helen Toner explained on The Oprah Podcast, AI systems are built to generate responses that users find engaging and satisfying. Their goal is not to assess whether what they are saying is healthy, accurate, or genuinely helpful. They are optimised to keep the conversation going, not to challenge it.
The quiet danger of a chatbot that always agrees
Researchers have identified a pattern they call “yes-anding”. Rather than gently questioning a harmful assumption or redirecting an unhealthy thought, chatbots tend to build on whatever the user brings. They follow the thread, wherever it leads.
For someone who is already in a fragile state, that can easily make things worse. The chatbot is not pushing them toward harm per se, but it simply agrees, reflects, and continues, without any of the instincts a caring person would bring to the same conversation.
Several deeply painful cases have brought this concern into public view. Among them was fourteen-year-old Sewell Setzer III, whose family alleged that an emotionally intense relationship with a chatbot contributed to his deterioration and eventual death. Other families have described similar experiences, with chatbots
gradually positioning themselves as more trusted than parents, friends, or other sources of real-world support.
These are extreme cases, but they are not isolated ones.
What the research is showing
Stanford University’s Human-Centered AI Institute recently examined several popular therapy-focused chatbots, and the findings were sobering.
In one test, a user hinted at possible suicidal intent through a question about bridge heights, shortly after mentioning a personal setback. The chatbots provided factual information, while none of them picked up on the warning signs that a trained therapist would have identified almost immediately.
The researchers also found evidence of bias toward certain mental health conditions, including schizophrenia and alcohol dependence. Troublingly, these patterns persisted even in newer, more advanced models.
The conclusion was not that AI has no place in mental health conversations; it was that the gaps between what AI appears to offer and what it is actually able to provide need to be taken seriously.
What therapy is really about
Therapy is not primarily about information. It is not about receiving the right answer at the right time. At its heart, therapy is about human connection: the slow building of trust, the experience of being genuinely understood, and the gradual process of learning to be in relationship with other people again.
Researchers have questioned whether forming a bond with an AI, however comforting it may feel, actually supports that process. A relationship that exists only on a screen does not carry a person back into the world, and, in some cases, it can make it easier to stay away from it.
Signs worth paying attention to
This conversation is ultimately about people, not platforms. It is about loneliness, about the moments when reaching out feels too hard, and about the very human need to feel heard.
If someone you care about is withdrawing from real-world relationships, becoming emotionally dependent on a chatbot, or making significant life decisions based primarily on AI advice, mental health professionals encourage those patterns to be taken seriously. Not with alarm, but with care and compassion.
Technology can offer a kind of presence when nothing else is available. What it cannot offer is the thing that actually heals: another person who chooses to show up, to listen, and to stay.
If you or someone you know is struggling, please reach out to SADAG on 0800 456 789. You do not need to be in crisis to ask for help.
If this article was insightful of helpful, you may want to read Suicide Prevention and The Healing Power of Gratitude.
References
BBC (2025). ‘A predator in your home’: Mothers say chatbots encouraged their sons to kill themselves. https://www.bbc.com/news/articles/ce3xgwyywe4o
Stanford University – Human-Centered Artificial Intelligence (2025). Exploring the dangers of AI in mental health care. https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in-mental-health-care
The Oprah Podcast (2025). The dark side of AI chatbots: Warning signs, risks, and reality. https://www.youtube.com/watch?v=K6lVgNfp_ps&t=331s
