Roblox, a popular online gaming platform among children and teenagers, is releasing an open-source version of its artificial intelligence system designed to detect predatory language in game chats. The move comes as the company faces lawsuits and criticism over claims that it has not done enough to safeguard young users.
One recent lawsuit in Iowa alleges that a 13-year-old girl met an adult predator on Roblox, was kidnapped, trafficked across multiple states, and assaulted. The case argues that Roblox’s features make children “easy prey for pedophiles.”
The AI tool, named Sentinel, is built to detect early signs of potential child endangerment, such as sexually exploitative language. Roblox reported that in the first half of 2025 alone, the system enabled the company to submit 1,200 potential child exploitation cases to the National Center for Missing and Exploited Children. The company now plans to make Sentinel available for other platforms.
Detecting risks is challenging because harmful conversations often begin innocently. Questions such as “how old are you?” or “where are you from?” may seem harmless but can take on a different meaning over time. Roblox does not allow users to share images or videos in chats and tries to block personal details like phone numbers. Children under 13 cannot chat outside games without parental approval, and private chats are not encrypted so they can be monitored.
“We have had filters in place all along, but those filters tend to focus on what is said in a single line or within just a few lines of text,” said Matt Kaufman, chief safety officer at Roblox. “When it comes to child endangerment or grooming, these behaviours appear over a very long period of time.”
Sentinel reviews one-minute chat snapshots, analysing around 6 billion messages daily. It uses two data sets — one with harmless conversations and another with messages involving child endangerment — to identify patterns that go beyond single flagged words. “That index gets better as we detect more bad actors,” said Naren Koneru, vice president of engineering for trust and safety.
The system scores users based on their conversations over time. If scores suggest a risk, Roblox investigates the user’s activity, friends, and games, and human moderators review and escalate serious cases to law enforcement.
Also read: Viksit Workforce for a Viksit Bharat
Do Follow: The Mainstream formerly known as CIO News LinkedIn Account | The Mainstream formerly known as CIO News Facebook | The Mainstream formerly known as CIO News Youtube | The Mainstream formerly known as CIO News Twitter |The Mainstream formerly known as CIO News Whatsapp Channel | The Mainstream formerly known as CIO News Instagram
About us:
The Mainstream formerly known as CIO News is a premier platform dedicated to delivering latest news, updates, and insights from the tech industry. With its strong foundation of intellectual property and thought leadership, the platform is well-positioned to stay ahead of the curve and lead conversations about how technology shapes our world. From its early days as CIO News to its rebranding as The Mainstream on November 28, 2024, it has been expanding its global reach, targeting key markets in the Middle East & Africa, ASEAN, the USA, and the UK. The Mainstream is a vision to put technology at the center of every conversation, inspiring professionals and organizations to embrace the future of tech.
