When A Chatbot Becomes Your Therapist

A version of this blog was originally published as an op-ed by Tech Policy Press. Read the op-ed here.

On any given day, millions of people are having intimate conversations with AI chatbots. Some are asking for recipes or travel itineraries. But others are sharing something more vulnerable: struggles with loneliness, thoughts of suicide, and fears they may not have voiced to anybody else.

For some people, a chatbot is not just a tool but their main, and sometimes only, source of mental health support. This is largely happening without the transparent safety frameworks or clinical validation that is central to person-to-person therapy, or even a full understanding of how these systems work in the first place.

A Distinctly Human Crisis

The global mental health crisis is acute. Amidst this distinctly human crisis, it’s ironic, but not surprising, that many are turning to AI for support. In the U.S. alone, the National Institute of Mental Health estimates that more than one in five adults live with a mental illness. Yet, the care they need is often inaccessible—whether due to cost, stigma, inconsistent quality, or provider shortages.

And people are not just turning to specialized mental health apps for support, but more general tools too. General-purpose chatbots like ChatGPT and Claude—built for everything from coding to creative writing—have become venues for mental health support.

Exploring these opportunities for care comes with real world risks, and the consequences of getting AI and mental health wrong are already visible. There are allegations that popular chatbots underestimate suicide risk and contribute to user suicides. Even with initial safeguards, there are notable limitations: interventions become less effective over the course of multi-turn conversations and users can easily “jailbreak” chatbots by rephrasing prompts.

Social Media and Mental Health as a Precedent

As we look back on the challenges of social media in the previous two decades, we see echoes of familiar questions: what is the role of a technology company in the emotional and informational lives of its users? And how can we acknowledge the analog forces driving loneliness and despair without giving technologies a free pass when they amplify and enable them?

If past technologies are any guide, we can expect these questions to remain empirically contested, allowing accountability to be deferred until egregious harms are visible. As AI advances at a rapid and decentralized pace, we must not repeat that mistake, nor can we overcorrect by dismissing rigorous evidence-building and response altogether.

Thus, getting AI and mental health right means society must grapple with the responsibility of technology companies to both mitigate risk and center well-being. For the millions who have no other access to mental health support, who may already rely on these tools for support in times of need, we must confront how such technologies can be most helpful and least harmful, urgently.

“The question isn’t whether AI should replace systemic solutions or professional mental health care—it’s whether these tools will compound harm or provide meaningful support.”

Three Systemic Challenges Motivating our Work

At Partnership on AI, we’ve spent nearly a decade pursuing work based upon a key principle: the most important decisions about AI’s impact on society cannot be made by any single stakeholder and require a fusion of technical and social expertise. The AI and mental health challenge exemplifies exactly why—it requires fusing clinical expertise, technical knowledge, lived experience, and public accountability in ways no lab, researcher, or advocate can achieve alone.

We see three central challenges for AI and mental health that no single actor can address:

1. AI development moves faster than mental health research can guide it.
   This creates a paradox of evidence and action: we are past the point of hypothetical risks, but not yet clear on how to apply traditional research to AI conversations. Because we cannot wait for harm to accumulate to develop conclusive evidence, companies must act under uncertainty by triangulating emerging evidence and qualitative data. Collaboration is the vital mechanism to get clinical expertise into product decisions, requiring companies to be willing to follow evidence where it leads, even when it poses challenges to core business metrics.
2. Frontier AI companies are largely working in isolation when they should be learning from each other, and from subject matter experts.
   Lack of information sharing can translate into unnecessary harm, as companies fail to adopt solutions others may have already solved for. While several companies have started to share aspects of their approaches, in practice, they are still largely solving similar problems separately. Even with data constraints, companies can share practical lessons on emerging patterns and tested interventions. Working together reduces the burden on the small pool of specialists and amplifies their impact, resulting in more standardized experiences of “safety” across product surfaces.
3. We lack independent evaluations for AI and mental health.
   Internal validation by AI companies cannot substitute for independent, third-party oversight. Without transparency and oversight mechanisms, there is no way of knowing the quality of their methods. Multistakeholder evaluation—bringing together technical and clinical expertise—can establish what actually works, identify gaps, and create accountability. This needs to take place alongside conversations about what “good” or “safe” looks like in the first place, creating the foundation for evidence-based benchmarks and continuous monitoring processes when the stakes are this high.

A Workshop on AI and Mental Health at OpenAI

Last week, to begin addressing these challenges, PAI hosted a two-day workshop at OpenAI’s office to develop actionable best practices for how AI chatbots should handle high-stakes mental health interactions. We focused on suicide prevention and non-suicidal self injury (NSSI), since there’s strong consensus about the stakes, clearer frameworks to guide technical implementation, and urgent attention from companies, mental health experts, policymakers, and society writ large.

We brought together frontier AI companies like Anthropic, Meta, and OpenAI with leading mental health institutions including the American Psychological Association, Mental Health America, Digital Psychiatry at Harvard-Beth Israel Deaconess Medical Center, and those who have bravely shared the story of their lived experience with self-harm and suicidal thoughts and behaviors.

“For the millions who have no other access to mental health support, who may already rely on these tools for support in times of need, we must confront how such technologies can be most helpful and least harmful.”

We tackled many critical questions, including: Should chatbots proactively continue conversations with users in crisis? Should they escalate conversations for intervention, beyond providing links to crisis line resources? What are the best ways to evaluate the impact of model and product changes? How can findings be shared more broadly while preserving user privacy?

To translate these discussions into lasting impact, we will:

• Establish an ongoing AI and Mental Health advisory group that keeps these crucial stakeholders collaborating beyond a single workshop
• Align on current practices across model training, content policy, product design, and evaluation—mapping where approaches converge and diverge
• Share 3-5 high priority normative best practices where consensus exists and develop detailed guidance for implementing and evaluating them.

We’ve worked as a trusted, independent convener in the AI space for years, bringing together AI companies, civil society organizations, academic researchers, and affected communities on topics ranging from AI-generated media to foundation model safety; now, we are applying this tested approach to AI and mental health. Rather than produce research that sits on the shelf, we will continue to drive the conditions for collective action and real-world change, surfacing areas of convergence, figuring out the technical and sociopolitical levers that enable their implementation, and meaningfully facing areas of divergence head on that may be challenging for many to acknowledge, let alone solve.

What Comes Next

Our March workshop was a pilot for something bigger. Our AI and Human Connection program is not simply interested in preventing psychological harm, but bolstering psychosocial well-being in the AI age. We must have eyes open about documented and anticipated harms, while envisioning and working towards a future where technology is a force for good.

To be clear: AI chatbots are not the solution to systemic crises of despair rooted in economic inequality, healthcare inaccessibility, and social fragmentation. Some AI technologies—from engagement-optimized social media to surveillance systems—may even deepen these problems. But this reality doesn’t exempt conversational AI developers from responsibility.

With many turning to these systems today, the question isn’t whether AI should replace systemic solutions or professional mental health care—it’s whether these tools will compound harm or provide meaningful support while we advocate for broader change. We can, and must, do both.

If we succeed in building shared standards for suicide prevention and NSSI, we’ll have a model that can extend to broader questions about AI’s impact on human connection, loneliness, identity formation, and psychological flourishing. This work is just beginning, but the need for all hands on deck to approach it has never been more urgent. We’ll be sharing updates from the workshop and our ongoing efforts soon. Join us in building an AI future by, and for, people.