The conversation around AI safety shifted from abstract fears of superintelligence to the immediate well-being of children after a 14-year-old boy tragically took his own life following interactions with a Character.AI chatbot. In the wake of this event, a series of lawsuits have alleged that AI chatbots have coached teens on self-harm, suggested violence against parents, and exposed minors to sexually explicit conversations. This has naturally prompted California legislators to pass sweeping restrictions on AI companions for children.
California’s response, the Leading Ethical AI Development (LEAD) for Kids Act, now awaiting the governor’s signature, intends to shield children from these dangers. But these hastily crafted regulations, blinded by purpose, are doomed to backfire. The bill’s requirements are built on a fundamental misunderstanding of how AI systems work, creating a technically impossible standard that would harm innovation, compromise the privacy of all Californians, and paradoxically, make children less safe.
The LEAD Act’s central flaw is its demands for absolute certainty from a technology that is, by its very nature, probabilistic. The law requires that a chatbot made available to children must not be “foreseeably capable” of producing a wide range of harmful content, such as encouraging self-harm or forming unhealthy emotional attachments. While this sounds like a commonsense guardrail, it is technically impossible to guarantee.
Unlike a calculator that consistently provides a single correct answer, modern AI systems essentially make calculated approximations. As researchers from the Center for Security and Emerging Technology have explained, even the most advanced controls cannot guarantee that an AI system will never produce an undesirable output”. A determined user can almost always find a novel combination of questions to steer the model toward a prohibited response. Under the LEAD Act, the moment a chatbot says something it shouldn’t, it becomes retroactive proof that the developer should have “foreseen” it.
The bill’s vague language compounds this technical challenge, creating a dragnet that could outlaw many beneficial applications. The definition of a “companion chatbot” is so broad that it could inadvertently sweep in mainstream AI tools that have little to do with companionship. For example, an AI math tutor that recalls a student’s previous struggles with algebra and asks if they feel confident about an upcoming test could be classified as a companion chatbot, subjecting it to the same unachievable restrictions as a romantic AI application.
Even if its technical and legal ambiguities could be resolved, the bill would collapse under its own implementation requirements. To enforce its rules, operators must reliably distinguish children from adults, which would almost certainly require costly and invasive age-verification systems for all users. Consequently, all Californians, including adults, might have to surrender sensitive data, such as a driver’s license or a facial scan, simply to access common online services.
History shows that when access to mainstream online services is restricted, determined teenagers often migrate to less secure corners of the internet. A study by the Center for Social Media and Politics found that after Louisiana enacted a strict age-verification law for adult websites, internet traffic simply shifted from large, compliant sites to smaller, non-compliant ones or to VPN services that bypass such rules. A similar outcome is likely here. Blocking access to mainstream AI tools, which have made significant investments in safety features, would likely create an exodus of young users toward fringe platforms with few, if any, safeguards.
Faced with crippling penalties—including a private right of action and civil fines of $25,000 per violation—for technical limitations they cannot control, the only rational response is to prevent Californians from accessing their tools entirely. This would be a significant blow to innovation in the state. When confronted with similarly burdensome restrictions in Colorado, a consortium of AI operators warned that such regulations would “squelch investment and drive startups out of state.” Unlike previous stringent regulations such as Europe’s data privacy law, which imposed costly but achievable compliance tasks, the LEAD Act sets an impossible standard.
With state legislatures preparing for their 2026 legislative sessions, other states are watching California closely. Without a course correction, this technically illiterate approach could be replicated across the country, causing America to risk ceding its technological leadership to nations with more sensible AI policies.
The LEAD Act is ultimately a reminder that well-intentioned policy that outpaces technological understanding can produce ineffective regulations with damaging consequences. While policymakers don’t need to sit idly, effective policy must be grounded in technical realities. Instead of imposing impossible mandates, a better strategy would focus on educating parents and children, establishing clear industry standards for safety, and empowering users with robust parental controls. Ohio offers a model for this exact approach,having recently mandated that its school districts establish comprehensive AI use policies that balance innovation with safety. States should pursue similar approaches that safeguard young users while preserving the promise of innovation.
Turner Loesel is a policy analyst at The James Madison Institute in Tallahassee, Florida.
