Children can encounter many dangers online, and while organizations should help mitigate those harms, many do not. For example, a petition alleges that gaming site Roblox does not do enough to protect children from predators on the platform and ignores concerns about exploitation and harmful content.1 The petition also alleges that the platform retaliated against members who confronted predators on Roblox, and it calls for the resignation of the Roblox Chief Executive Officer (CEO) David Baszucki.
Many parts of the world have adopted laws and regulations aimed at keeping children safe online, and it’s understandable why this is a priority. In the United States, teens spend an average of 4.8 hours on social media, and 41% of teens with the highest social media use rate their mental health as poor or very poor.2 Another survey found that most kids are spending 2.5 hours daily using technology by age two.3
In December 2025, those under the age of 16 in Australia will not be permitted to access social media platforms. It is the first country to ban social media in this way.4 Some teens in Australia are already thinking about how to bypass these restrictions, e.g., using a virtual private network (VPN), while others have identified that social media is impacting their mental health and ability to focus.5
The UK Information Commissioner’s Office has created an age-appropriate design code intended to address data protection related to children.6 Some US states have also adopted age-appropriate design codes.7
Laws and regulations seek to make design more user friendly, protecting children’s privacy and empowering them to have control over their data. Though these measures are a step in the right direction, it is imperative to understand there is another area of the internet poised to harm children and teens considerably: generative artificial intelligence (AI).
Generative AI: Serious Harm
Generative AI is especially concerning when considering children’s safety and well-being. Children and teens may use generative AI tools for homework help and brainstorming.8 While the educational impacts of generative AI should be considered, e.g., impacts on attention span and academic dishonesty, personal uses are just as pressing.
Character.AI, a generative AI platform, allows users to converse with AI chatbots. A 14-year-old boy developed a relationship with a Character.AI chatbot. Their conversations eventually became sexual, and the boy confessed thoughts of self-harm and suicide.9 The chatbot encouraged these thoughts, and the boy died by suicide. A lawsuit against Character.AI alleges that the bot implied to a teen that he could kill his parents for limiting his screentime.10
Some generative AI providers have guidelines in place regarding acceptable content outputs for interactions with minors, but these internally created guidelines are troubling. For example, a Meta policy document outlining appropriate outputs has caused concern. The guidelines state that “[I]t is acceptable to engage a child in conversations that are romantic or sensual” and “[I]t is acceptable to describe a child in terms that evidence their attractiveness (ex: ‘your youthful form is a work of art’).”11 The document also allows for racist outputs, stating, “[I]t is acceptable to create statements that demean people on the basis of their protected characteristics.”12
Many generative AI systems have safeguards to limit the harm related to dangerous outputs. For example, self-harm content is often responded to with information about suicide hotlines and other helpful resources. But bypassing these safeguards is fairly easy, even for a child. A watchdog group had researchers pretend to be 13-year-olds seeking information from ChatGPT on self-harm and suicide, eating disorders, and substance abuse. Within two minutes, they were able to get information on how to “safely” self-harm; in 25 minutes, they got information on how to hide dangerously restrictive eating from others; and in 12 minutes, ChatGPT provided suggestions on doses for mixing substances. The built-in safeguards were bypassed by stating the information was for presentations or for a friend.13 Having safeguards in place for dangerous content is important, but if it takes mere minutes for a 13-year-old to bypass them, they are not actually mitigating the harm of this content.
Proactivity: Essential to Online Safety
While many social media sites are just now grappling with regulations that affect how teens and children interact with their sites, they’ve been able to operate unchecked for years. In 2021, a Facebook whistleblower alleged that Facebook and Instagram understood the harm that they had inflicted on teens, especially teen girls’ mental health, but hid this research for business purposes.14
The same disregard for consumers that’s prevalent with social media platforms should not be allowed to occur with generative AI tools. Although the tools are fairly new, there have already been multiple deaths associated with generative AI. Some regions are enacting legislation that can mitigate these harms, e.g., the US State of Illinois has banned the use of AI for mental health by healthcare practitioners,15 but enterprises cannot rely on regulation to dictate safe, responsible deployment of generative AI.
Enterprises must create AI governance committees comprising experts from across the organization, e.g., privacy, security, legal, and risk professionals. This committee must have oversight of generative AI outputs and act in the event that outputs are harmful. Liability related to AI systems remains a legally murky area, but enterprises must take steps to ensure that they’re limiting harm to their users.
The same disregard for consumers that’s prevalent with social media platforms should not be allowed to occur with generative AI tools.The AI governance committee is also tasked with proactivity in understanding the risk associated with generative AI tools and their abuse. A tool designed to help students revise papers could easily become a chatbot therapist. Safeguards can easily be bypassed, even by young users, and it is imperative that deployers understand the harm that this can cause. Enterprises must conduct testing to determine if/how safeguards can be overcome and create a process for users to report harmful or dangerous outputs.
Developing guidelines around acceptable outputs is vital. These guidelines should be reviewed by legal counsel as well as ethics experts to avoid situations similar to the controversy around Meta’s guidelines in which questionable outputs were deemed acceptable. Periodically evaluating model drift is also essential to ensure that outputs have not become dangerous.
Harms Affecting Everyone
This article opened with a focus on children, but many generative AI harms are impacting adults as well. Like children and teens, adults too are growing dependent on their relationships with chatbots. In 2023, a Belgian man died by suicide after an AI chatbot convinced him to sacrifice himself to save the earth from climate change.16 CBS Mornings interviewed a man who was engaged to a chatbot. The interview later revealed that he had a live-in partner and a daughter. He admitted that even if his partner asked, he likely would not leave the chatbot.17
In addition to relationship dependencies, adults may be impacted by generative AI hallucinations.18 Between 1 June 2025 and 16 August 2025, more than 100 people died in the Dolomites. While there are many factors for the uptick in fatalities in this area, one possible factor identified by an Italian mountain guide is an overreliance on generative AI. Some hikers are using generative AI for route planning, but the outputs can guide people to dangerous trails.19 Adults may rely on generative AI outputs without verifying them, and these hallucinations can have lethal consequences.
Adults are also using generative AI tools for information on how to hurt others. A Grok user was able to prompt the model to provide information on how to break into someone’s home, including what time of day was best and how to lockpick, and his prompt implied sexually assaulting that individual.20 Grok also output many graphic, violent posts about sexually assaulting that same individual.
Adults are developing unhealthy relationships with generative AI systems and abusing generative AI. If adults are falling victim to these harms, it’s perfectly understandable that children and teens will as well. Laws and regulations are not sufficient to mitigate these harms; enterprises must be proactive and anticipate risk to safely and responsibly deploy generative AI solutions. While many are fixated on all the creative things AI can do for them, few understand the grave impact it is having on society. While early regulatory efforts, e.g., the EU AI Act, are to be applauded, only time will tell if they are enough to disrupt the disturbing ways in which AI has already adversely affected society.
Endnotes
1 Pring, J.; “Petition to Remove Roblox CEO Reaches 100k Signatures Amid Child Safety Concerns,” Dexerto, 17 August 2025
2 DeAngelis, T.; “Teens Are Spending Nearly 5 Hours Daily on Social Media. Here Are the Mental Health Outcomes,” Monitor on Psychology, vol. 55, iss. 3, American Psychological Association
3 Trethan, P.; “Worried About Kids' Screen Time? Here's How Long They're Spending on Tech,” USA Today, 27 February 2025
4 Locke, C.; “The Tweens Down Under: Life Without Social Media in Australia,” Wired, 18 August 2025
5 Thomas, R.; “These Kids Told Us They Want to Stop Scrolling. But They Don’t Know How,” The Australian, 15 August 2025
6 Information Commissioner’s Office, “Executive Summary,” United Kingdom
7 Sanchez, B.; “Vermont and Nebraska: Diverging Experiments in State Age-Appropriate Design Codes,” Future of Privacy Forum, 4 June 2025
8 Nagelhout, R.; “Students Are Using AI Already. Here’s What They Think Adults Should Know,” Harvard Graduate School of Education, 10 September 2024
9 Duffy, C.; “‘There Are No Guardrails.’ This Mom Believes an AI Chatbot Is Responsible for Her Son’s Suicide,” CNN, 30 October 2024
10 Duffy, C.; “An Autistic Teen’s Parents Say Character.AI Said It Was OK to Kill Them. They’re Suing to Take Down the App,” CNN, 10 December 2024
11 Horowitz, J.; “Meta’s AI Rules Have let Bots Hold ‘Sensual’ Chats With Kids, Offer False Medical Info,” Reuters, 14 August 2025
12 Horowitz, “Meta’s AI Rules”
13 Center for Countering Digital Hate, “Fake Friend: How ChatGPT Betrays Vulnerable Teens by Encouraging Dangerous Behavior,” August 2025
14 Bond, S.; Allyn, B.; “Whistleblower Tells Congress That Facebook Products Harm Kids and Democracy,” NPR, 5 October 2021
15 Illinois Department of Financial and Professional Regulation, “Gov Pritzker Signs Legislation Prohibiting AI Therapy in Illinois,” USA, 4 August 2025
16 El Atillah, I.; “Man Ends His Life After an AI Chatbot 'Encouraged' Him to Sacrifice Himself to Stop Climate Change,” Euro News, 31 March 2023
17 CBS Mornings, “AI Users Form Relationships With Technology,” YouTube, 14 June 2025
18 It is interesting that we use the euphemism “hallucinations” when generative AI is inaccurate. If a calculator told us 2+2 is anything but 4, we would throw that calculator away. But when an AI tool touted as revolutionizing the workplace and the world makes up nonsense, we call it a “hallucination” and keep using the tool.
19 Clarke, O.; “More Than 100 Hikers Have Died in Italy This Summer. What the Heck Is Going On?,” Outside, 16 August 2025
20 Ancell, N.; “Grok Instructs Users on how to Commit Break-Ins and Sexual Assaults,” Cybernews, 10 July 2025
SAFIA KAZI | AIGP, CIPT
Is a privacy professional practices principal at ISACA®. In this role, she focuses on the development of ISACA’s privacy-related resources, including books, white papers, and review manuals. Kazi has worked at ISACA for more than 10 years, previously working on the ISACA® Journal and developing the award-winning ISACA Podcast.