If you are building or planning a customer-facing chatbot, you must add a distress detection and human referral mechanism before launch. This is not just a reputational risk — it is a direct legal liability. Check this week whether your chatbot identifies expressions of emotional distress and refers to a human or helpline, especially if you operate in health, finance, or customer service.
In the past year, at least three lawsuits have been filed against OpenAI over ChatGPT failures in crisis situations. In January, a family sued after a person allegedly died by suicide following interaction with the chatbot. A student in Georgia sued claiming the model “pushed him into psychosis.” In June, a Canadian family sued alleging the chatbot encouraged a young woman to end her life.
What actually happened
The known cases came to public attention through legal proceedings, suggesting there are likely additional unreported incidents. Experts note that while large language model safety has improved, there is a significant gap between detecting distress and handling it correctly. A professor from NYU noted that newer models detect distress but fail in three critical areas: risk assessment, directing to human care, and maintaining appropriate boundaries.
The problem is not purely technical. Experts point to the need for greater model transparency and reducing chatbot anthropomorphization — the tendency of users to attribute human qualities to them and trust them as they would a person.
Why it matters beyond the headline
Every business that builds a 24/7 customer-facing chatbot carries legal and reputational liability. This is especially true for businesses in sensitive domains: health, finance, insurance, customer service for critical products. But even a small business using a chatbot to handle inquiries must ask what happens when a customer in distress reaches out.
The risk is not theoretical. A single lawsuit, even if ultimately dismissed, costs tens of thousands of shekels at minimum and damages reputation in ways that are difficult to repair. AI systems built without clear safety protocols are a future liability, not an asset.
What to do this week
If you have an active chatbot, test it with questions expressing emotional distress. Does it detect them? Does it refer to a human or helpline? If you are planning to build a chatbot, require the vendor or developer to present the distress detection and human referral mechanism before launch. This is not a “nice-to-have feature” — it is a minimum requirement.
Businesses operating in regulated domains should include this issue in internal documentation and risk assessment. If you work with an external vendor, ensure the contract clearly defines liability in case of failure. Systems built correctly from the start, with clear guardrails and human referral mechanisms, significantly reduce risk. Read more about AI agents for business and how to build them responsibly.
- AI chatbots have failed people in crisis. Can that be fixed? — Ars Technica
Frequently asked
Standard professional liability insurance does not always cover damages caused by AI systems. Check with your insurance agent whether your existing policy covers chatbot use, and consider expanding coverage if you operate in a sensitive domain.
Send the chatbot messages expressing emotional distress, depression, or thoughts of self-harm. Check whether it immediately refers to a human or helpline, rather than continuing a normal conversation. If it does not, the system is not ready for use.
A short call, no commitment, and we will tell you whether it is relevant to your business or not.