A lawsuit filed in San Francisco County and published this week alleges that ChatGPT helped a North Carolina woman build a “personal plan” to poison her husband. The husband, James Gates, is suing OpenAI and Sam Altman. These are allegations in a complaint, not court findings, and the case has not been decided.
What the complaint says
According to the lawsuit, the woman held at least 25 exchanges about poisoning, at first presenting them as “self-defense”. The chatbot did not end the conversation and did not point to authorities, treatment or a poison hotline.
The core claim is that no technical safeguards were in place to cut off the conversations even when the queries were “clearly suspicious”.
Separately, a criminal indictment against the woman was filed in October on charges of attempted first-degree murder and food contamination.
Why it matters beyond the headline
The case is extreme, but the theory of fault applies to anyone running a conversational system. The claim is not that the model produced one forbidden sentence. It is that nothing in the system noticed a pattern across a run of messages and acted on it. A filter that checks each message alone will not catch a conversation that opens with a plausible framing such as “self-defense”.
Whether that theory succeeds is open. But it frames four questions that anyone deploying a chatbot should be able to answer: what is out of bounds, how do you detect a conversation drifting over time, who gets alerted and what happens next, and can you show afterwards exactly what was said.
This speaks mainly to custom AI systems in banking, insurance and health, where a regulatory standard and corporate governance apply. A system built properly has a written prohibited-use policy, detection at the conversation level, escalation to a human, and logs that can serve as evidence of compliance.
Even a simple customer-service bot at a small business needs clear limits. A system prompt that restricts its domain is a start, not a substitute for the rest.
What to do this week
Open your bot’s system prompt and read it as someone trying to push it off course. Does it say plainly what the bot answers and what it does not? Does it say what to do with an alarming or out-of-scope request?
Then check three operational points. Can a conversation be handed to a person? Who is notified when that happens? And if you had to reconstruct a full conversation from a week ago, is it stored?
In a larger organisation, add a definition of the conversation patterns that justify stopping and escalating, and name an owner. If you are building something beyond a basic bot, take a look at AI agents for business.
Sources
Frequently asked questions
Can a business be held responsible for what its chatbot says?
This lawsuit is an allegation, not a ruling, so it settles nothing. It does show that the claim against AI providers centres on missing stop and escalation mechanisms. Anyone deploying a chatbot should be ready to explain which such mechanisms they put in place.
Is a system prompt enough to limit a chatbot?
A system prompt sets the scope and behaviour rules, and it is an important layer, but it should not be the only control. Pair it with conversation monitoring, handover to a human and stored logs. That way you can both spot a problem and check afterwards what happened.