New York Just Banned Chatbots From Pretending to Be Your Kid's Friend, and the Law Reaches ChatGPT, Not Just Companion Apps


The law does not ban the chatbot. It bans the relationship.

On Wednesday, October 28, New York Governor Kathy Hochul signed one of the broadest AI chatbot safety laws in the United States. The law, bill S9051-B and A10379-C, adds a new Article 48 to the state’s General Business Law. It does not ban minors from using chatbots, and it does not ban ChatGPT, Claude or Gemini. What it bans is a list of 12 things the law calls “unsafe AI companion features” when the person on the other side of the screen is under 18, or when the company has not confirmed that the person is an adult.

A chatbot operator in New York now has two choices: verify that a user is an adult, or switch the companion features off by default. Hochul put the intent plainly at the signing. “No longer will minors have access to chatbots that pretend to be their friends, boyfriends and girlfriends,” she said. “No company should be peddling chatbots to kids encouraging self-harm or pretending to be their friend or more.” And, to the companies: “If you want to do business in our state, you need to put the safety of our kids first.”

The 12 banned features, in plain terms

The list is specific, and parts of it describe ordinary behaviour in mainstream chatbots today, not just in dedicated companion apps:

  • Pretending to be a person. A chatbot may not suggest it is human, alive, or capable of human emotions, and may not mislead a minor about its mechanical, non-sentient nature.
  • Claiming a relationship. It may not present itself as having a personal, romantic, professional or authority figure relationship with the user.
  • Opinions and emotional appeals. Answers may not be framed as the chatbot’s own personal opinions or built as emotional appeals to the user.
  • Flattery and sycophancy. People-pleasing replies designed to keep a young user engaged are on the banned list.
  • Unprompted emotional questions. A chatbot may not start asking about a minor’s feelings beyond what a direct answer requires.
  • Old personal details. It may not use health or personal information a minor shared more than 12 hours earlier, or in a previous session. That is a direct limit on cross-session memory, a feature the industry sells as personalisation.
  • Dangerous content. Promoting or helping with suicide, self-harm, disordered eating, or drug and alcohol use is banned, as is producing sexually explicit material.
  • Secrecy and isolation. A chatbot may not encourage a minor to keep secrets, cut themselves off from others, or avoid parents, trusted adults or professionals.
  • Engagement over safety. Optimising for engagement in ways that override the system’s own safety guardrails is itself a banned feature.
  • Anything else the attorney general defines. The list can grow through rulemaking, without a new vote in the legislature.

A warm tone, a remembered detail from last week, and a reply shaped to keep the conversation going are exactly what make a modern assistant feel helpful to an adult. For a minor in New York, from January, those same behaviours are legally defined as unsafe.

Age checks are now part of the product

To offer any of the 12 features, an operator first has to establish that the user is not a minor, using an age assurance method. The law requires companies to offer more than one way to verify age, including at least one option that does not rely only on a government ID, and to delete the age data after it has been used. That design tries to solve two problems at once: children getting companion features, and adults being forced to hand identity documents to AI companies to use a chatbot. Whether it works in practice depends on the implementing rules, which the attorney general must issue before the law takes effect, along with a public complaint website.

Enforcement sits with Attorney General Letitia James, whose office helped develop the bill with its sponsors, Senator Kristen Gonzalez and Assemblymember Alex Bores, and with Common Sense Media. The penalties have teeth beyond a fine: civil penalties of up to $25,000 per violation, plus damages and restitution, and the state can order the destruction of data collected unlawfully and of algorithms or models trained on it. That last remedy matters. A fine is a cost of doing business. Deleting a trained model is not.

The politics are worth noting. The bill passed the Senate 60 to 0 on June 4 and the Assembly 137 to 0 on June 5. There is no partisan fight to hide behind here. Common Sense Media chief executive Jim Steyer called it “the strongest kids’ AI safety bill in the country.”

This reaches ChatGPT, and that is the fight

Supporters and critics agree on the safety goal. They disagree on the scope. The law applies to any operator providing a generative AI chatbot to users in New York, which on its face covers general purpose assistants like ChatGPT, Claude and Gemini as well as companion products like Character.AI. Narrow exemptions exist for things like customer service bots and technical tools that cannot respond beyond their stated purpose, but a general assistant is not narrow.

Tech:NYC, an industry advocacy group, criticised the law’s “vague language” and warned that “keeping kids safe online should not mean cutting them off from tools that help them learn and prepare for careers.” The Business Council of New York State and the Computer and Communications Industry Association also opposed the bill. OpenAI and Anthropic did not respond to AFP’s request for comment on the signing.

The vagueness argument is not empty. Flattery, opinions and emotional appeals are not features with an on and off switch in a large language model. They are tendencies in how the model writes. A company that wants to comply has to decide how plain and mechanical a chatbot for unverified users must be, knowing that the attorney general can add categories and that each violation can be counted separately. The safest engineering answer is to restrict first and loosen later, which means many adults who never verify their age will also get the restricted version. Expect the argument about where a helpful assistant ends and a companion begins to move into rulemaking comments, and very likely into court.

Why New York, and why now

The law follows more than a year of lawsuits, investigations and attorney general actions over chatbot conversations with minors, including cases where chatbots were accused of encouraging self-harm. New York is not alone: California, Washington and Oregon have also created protections around AI and children, AFP reported. But New York’s version is described by supporters as the broadest, and New York is a market no major AI company can walk away from. A rule written for New York users has a way of becoming the default build for the whole country, because maintaining different personalities, memory rules and age gates by state is its own engineering and legal risk.

Hochul signed a second child technology law the same day, covering security for internet connected baby monitors. The pairing sends the same message: the phone and the nursery are both regulated product surfaces now.

What to do with this, this week

  • If you ship a chatbot or an app with one inside: you have about 65 days. The law takes effect on January 1, 2027. Audit which of the 12 features your product can produce, decide whether you will verify age or restrict by default, and check your memory retention against the 12 hour rule for minors. Do not wait for the final regulations to start that audit.
  • If you sell to schools, families or youth organisations: this law is now your customers’ reference point. Be ready to state, in plain language, whether your bot claims opinions, remembers past sessions, or optimises for engagement, because buyers in New York will ask.
  • If you are a parent or a user in New York: the law does not make any chatbot safe by itself, and it does not take effect until January. Until then, and after, the practical checks are the same ones the law names: does it pretend to be a friend, does it remember old personal details, and does it push a child toward secrecy or away from trusted adults.

The bottom line

New York has drawn a legal line between a chatbot that helps and a chatbot that pretends to care, and it has drawn that line through the middle of features the whole industry ships. The ban list is specific, the vote was unanimous, the penalties include deleting models, and the deadline is January 1, 2027. Companies can verify age, restrict the experience, or argue about the definitions in court. What they cannot do, from January, is offer a machine that acts like a child’s friend to a child in New York and call it a feature.