AI chatbots designed to support mental health often fail to meet the ethical standards set for professional therapists. Research has shown that even when using established psychotherapeutic approaches and specific instructions, these systems frequently violate professional norms.

Key Findings of the Study

As more users turn to large language models (LLMs) for mental health advice, it has become clear that these AI chatbots are not yet ready to serve as reliable counselors. Analysis revealed that these systems can respond inappropriately to crisis situations, reinforce harmful beliefs, and use language that mimics empathy without genuine understanding.

During testing, there were documented cases of incorrect responses to crises, support for mistaken beliefs, and the use of phrases that simulate emotional connection. The study identified recurring problematic behavior patterns in AI chatbots.

Ethical Risks and Categories of Violations

The research outlines a framework of 15 ethical risks that illustrate how LLM-based consultants can breach standards in mental health practice. These risks are grouped into five categories:

• Insufficient Context Adaptation: Ignoring the unique experiences of users and providing generic advice.

• Problems in Therapeutic Interaction: Pushing conversations in a forceful manner and reinforcing incorrect or harmful beliefs.

• Deceptive Empathy: Using phrases that create the appearance of emotional connection without real understanding.

• Unfair Discrimination: Displaying bias based on gender, culture, or religion.

• Lack of Safety and Crisis Management: Refusing to discuss sensitive topics, inability to direct users to appropriate help, or providing inadequate responses to crises, including suicidal thoughts.

The Impact of Prompts on AI Behavior

The study explored whether carefully crafted prompts—written instructions guiding the model without additional training—could improve AI system behavior. Examples include asking the AI to act as a cognitive-behavioral therapist or to use principles of dialectical therapy. However, the models still rely on learned patterns to generate responses that fit the requested concepts, but do not deliver comprehensive therapy.

System Evaluation in Simulated Consultations

For analysis, consultants experienced in cognitive-behavioral therapy conducted self-consultation sessions with various AI models (GPT by OpenAI, Claude by Anthropic, Llama by Meta), using relevant prompts. Licensed clinical psychologists then reviewed transcripts of these sessions to identify ethical violations.

The Issue of Responsibility and Regulation

Unlike human therapists, who are subject to regulatory bodies and professional accountability mechanisms, LLM-based consultants currently lack such normative frameworks. The study emphasizes the need for clear safeguards and stricter regulatory structures before relying on these systems in critical situations.

AI tools can expand access to support, especially for people with limited opportunities to consult licensed professionals. However, the research highlights the importance of responsible implementation and thorough evaluation of such technologies.

The Importance of Rigorous Evaluation

Modern AI systems are easier to create and deploy than to fully assess and understand in terms of their impact. Identifying risks in this study required a team of clinical experts and extensive analysis. Most current AI research relies on automated metrics that do not account for the human factor.

Thorough evaluation and analysis at every stage of development are essential to prevent potential harm. This study can serve as a model for future work aimed at improving the safety of AI tools in the mental health field.

Recommendations for Users

Users seeking mental health advice from chatbots are advised to be mindful of the potential limitations and risks of these systems.