SACRAMENTO, California — A leading California Democratic lawmaker issued a scathing critique Wednesday of what she said were overly lax safety guardrails for kids using AI chatbots, citing recent high-profile deaths associated with the technology.
“We see reports, like we saw this week, of a young man taking his own life after ChatGPT coached him on how to do it,” Assemblymember Rebecca Bauer-Kahan told POLITICO, referencing a lawsuit over the suicide of a 16-year-old boy who sought feedback from OpenAI’s ChatGPT.
“We're moving too fast with our children. That's not where we're going to experiment on AI.”
Bauer-Kahan’s concerns, shared Wednesday during a panel on AI regulation at POLITICO’s “The California Agenda: Sacramento Summit,” come as Democrats and Republicans alike have accused AI companies of not doing enough to safeguard kids using so-called “companion chatbots.”
OpenAI announced after the wrongful death lawsuit that it was working on an update to improve how its models “recognize and respond to signs of mental and emotional distress.”
Multiple members of Congress have demanded investigations of Meta following reports that the company’s guidelines allowed chatbots to have “sensual” conversations with kids. And on Monday, more than 40 state attorneys general, including California’s Rob Bonta, sent letters to several AI companies regarding the revelations about Meta’s chatbot policies and warning, “if you knowingly harm kids, you will answer for it.”
Meta has said it is revising its guidelines and that such conversations with children should never have been permitted.
California Democrats are pushing a pair of bills aimed at protecting kids from potential harms posed by companion chatbots. The first, authored by Bauer-Kahan, would outright bar companies from offering “emotionally manipulative” chatbots to kids.
The second measure from Democratic state Sen. Steve Padilla is more targeted, proposing mandatory reporting requirements when people discuss self-harm with chatbots, plus a ban on addictive reward structures sometimes used to increase user engagement with the bots.
Bauer-Kahan predicted Wednesday that Gov. Gavin Newsom “will sign a lot of AI legislation.”
One problem that may outlast Newsom, who is term-limited, is the return of a moratorium on state AI regulation, which Congressional Republicans failed to pass through their domestic policy megabill earlier this year but some, including Sen. Ted Cruz are already eager to revive.
“The next governor is going to have to handle that deftly,” said Mariano-Florentino “Tino” Cuéllar, a former California Supreme Court justice and the president of the Carnegie Endowment for International Peace, said Wednesday.
State lawmakers have a head start from brainstorming ways around federal preemption before the measure was stripped out.
Comments