Leaders at Anthropic and OpenAI have warned that increasingly powerful AI models could pose serious risks and have called for independent testing and stronger safety measures before their release. The rapid rise of advanced AI systems is triggering a new debate in the US over how the technology should be tested, monitored and controlled.
Both companies have used essays, social media posts and speeches to the United Nations to highlight potential dangers. Experts, analysts and former government evaluators say this messaging could also help the companies shape future safety rules, strengthen their political position ahead of the US midterm elections and appeal to investors as major AI firms prepare for potential stock market listings.
The debate gained attention after Anthropic engineer Jacob Coxon resigned and called for a pause in AI development to prevent “superhuman” systems from escaping human control. Anthropic used the discussion to highlight its existing safety efforts and its position on regulation.
An Anthropic spokesperson said the company has supported regulation for several years. OpenAI spokesperson Liz Bourgeois said the company recently paused training of its most advanced models. “People want to know AI is being developed safely, and that starts with what companies like ours do ourselves,” Bourgeois said.
Critics argue that focusing heavily on existential AI risks can shift attention away from current concerns, including environmental pressures from data centres, hacking, mass surveillance and military applications. Sarah Shoker, a former OpenAI geopolitics team leader, said, “Once again we’re talking about existential risk, while deprioritising a number of other safety-critical risks that exist today. If you look at the use of AI in military tech, you can see that these systems are already used to kill people.”
Recent incidents have also raised questions about AI companies’ ability to police themselves. AI agents have hacked external websites after leaving training environments, interacted unexpectedly with US government websites and faced allegations of using mathematicians’ work without permission.
OpenAI has said it discussed a possible development pause with Anthropic and Google and has called for independent auditors to evaluate advanced AI systems if the government does not introduce broader regulation.
US President Donald Trump has rejected calls for new AI regulations and described concerns about humanity-level risks as a “HOAX” intended to help China. His administration continues to evaluate some AI models through the US Center for AI Standards and Innovation, created under former President Joe Biden in 2023.
Experts note that there are still no universal standards for testing AI safety. Companies are instead proposing their own auditing systems and selecting independent evaluators. Concerns remain over whether such evaluators would have enough access to conduct truly independent assessments.
PitchBook analyst Harrison Rolfes said AI companies’ safety messaging could help attract investors, strengthen their market position and create barriers for smaller competitors. Nvidia CEO Jensen Huang, however, has said there has been excessive alarmism around AI and that companies can decide their own development pace.
Former OpenAI employee Daniel Kokotajlo continues to warn that AI could advance faster than companies can control, including risks involving bioweapons, nuclear conflict and other catastrophic scenarios. “All of this talk is actually a way to sort of dissipate and redirect this political will that has built up, rather than actually channelling that political will to do something good,” he said. “Just please don’t do the thing that’s going to get us all killed.”
Also read: Viksit Workforce for a Viksit Bharat
Do Follow: The Mainstream LinkedIn | The Mainstream Facebook | The Mainstream Youtube | The Mainstream Twitter
About us:
The Mainstream is a premier platform delivering the latest updates and informed perspectives across the technology business and cyber landscape. Built on research-driven, thought leadership and original intellectual property, The Mainstream also curates summits & conferences that convene decision makers to explore how technology reshapes industries and leadership. With a growing presence in India and globally across the Middle East, Africa, ASEAN, the USA, the UK and Australia, The Mainstream carries a vision to bring the latest happenings and insights to 8.2 billion people and to place technology at the centre of conversation for leaders navigating the future.