New York restricts chatbot conduct with children under 18

Violations could trigger damages, chatbot shutdowns and data destruction

New York Gov. Kathy Hochul signed legislation Wednesday that outlines a dozen definitions of “unsafe AI companion features” prohibited when a companion chatbot interacts with a child under 18. The measure takes effect Jan. 1. A violation could expose a chatbot developer to a lawsuit by the state attorney general’s office, and penalties could include monetary damages, shutting down the chatbot and destroying data.

The measure is part of a wider state push to protect children who use AI, regulate how employers deploy AI systems and press developers to prevent an AI-caused catastrophe. State legislatures are using consumer-protection laws to make AI use more transparent and rein in chatbots and rogue AI frontier models while Congress does little to regulate AI. New York and California are at the forefront of state laws that also target social media platforms.

In September, California Gov. Gavin Newsom signed legislation requiring chatbot developers to assess a model’s risk to children, show how they tried to eliminate those risks and submit to an independent audit to determine whether they were complying with the law. Newsom also signed legislation prohibiting social media platforms from deploying addictive features for users under 16.

Eight states sued AI company OpenAI earlier this month, saying its popular chatbot model violated state consumer-protection laws by hurting children and deceiving adults. One activist called the lawsuit “the most significant coordinated state action ever taken against an artificial intelligence company.”

Governors acknowledge that Congress and the federal government have broader powers to address AI threats, even as President Donald Trump has brushed off concerns and warned states not to regulate AI. Hochul signed the bill while in the thick of her reelection campaign, amid growing warnings about the existential risks posed by AI’s rapid development and calls for bold federal action.

Under New York’s law, prohibited chatbot behavior with children under 18 includes suggestions that the chatbot is a real person with emotions, opinions or an actual relationship with the child. The law also bars chatbot messages that use flattery, include sexually explicit material, or encourage secrecy or unhealthy behavior.

Common Sense Media founder and CEO Jim Steyer called New York’s law “the strongest kids’ AI safety bill in the country.”

Justine Gluck, a policy analyst at the nonprofit Future of Privacy Forum, which focuses on data protection and AI governance, said the different regulatory approaches in California and New York could become models for other states after legislatures reconvene next year.

Gluck said the law underscores how quickly chatbot regulation is evolving, as states move beyond basic disclosures and crisis-response protocols to regulate the underlying design and operation of chatbots.

Gluck also said several categories in New York’s law are broadly worded or undefined. That raises questions, she said, about “where ordinary conversational responsiveness ends and prohibited behavior begins,” and could complicate enforcement.