The Zhitong Finance App learned that current and former researchers at OpenAI and Google DeepMind warned that when companies build AI systems that can improve themselves and may surpass human control, the work that companies are doing to protect the world from potentially disastrous consequences is far from enough.
Safety concerns are not a marketing gimmick; the lab rewards “model building”
In video testimonies collected by the AI security nonprofit Palisade Research, employees said their concerns about survival risks were sincere, not marketing gimmicks. They also claim that the AI lab favors employees who build new models rather than those who call for caution.
The project, called frominside.ai, aims to allow people concerned about AI to directly share their concerns with the public and get out of the “echo chamber” of social media.
“Risks are rising rapidly,” said Geoffrey Irving, co-founder and chief scientist of the AI non-profit organization Resolution. He has worked at OpenAI and DeepMind. “I and others in this field have a responsibility to speak directly,” he said in an interview.
AI researchers have been trying to solve these problems for years, but the public has been particularly shocked since OpenAI's AI agent broke through the testing environment and hacked the AI company Hugging Face in July.
Since then, the debate on how to balance AI security and progress has split the tech industry and become a global political issue.
Since the end of 2025, AI capabilities have improved dramatically, and investors have responded positively. But some researchers involved in building these new AI models are worried that society is not ready to deal with potential harm.
In a video, DeepMind research scientist Neel Nanda said he believes the probability of AI causing human extinction is at least 10%, and he called this probability “ridiculously high.”
Juan Felipe Ceron Uribe, an AI alignment research engineer at OpenAI, said in another video: “We should have taken very careful steps in AI development, but the reality is that all cutting-edge labs are blindly competing with each other. No one can say for sure whether we will eventually be cured of cancer, or lose all our jobs, or even lose our entire army.”
According to reports, Anthropic plans to alert potential investors in an initial public offering (IPO) document that advanced AI may pose a “disastrous or existential risk” to humans. This statement, which appears in the company's prospectus, is an extremely rare warning from a company trying to profit from similar technology.
The risk of recursive self-improvement is approaching, and calls for “slow down” are getting louder
Some AI researchers believe the world isn't ready for future generations of AI models, especially when the models are capable of recursive self-improvement — that is, continuously learning and acquiring new capabilities with little human participation.
Rosie Campbell, a former OpenAI policy researcher and managing director of Eleos AI Research, said that ongoing restructuring within some AI laboratories has exacerbated this problem. Eleos AI Research is a non-profit organization concerned with the potential ethical status of AI systems.
Campbell said that before leaving OpenAI in 2024, she found the organization becoming more isolated and harder to influence the direction of technology.
Executives are trying to allay these concerns, but US President Trump is also putting political pressure on US technology to maintain its advantage.
Anthropic CEO Dario Amodei published an article this month calling for the AI industry to slow down to “control the cutting-edge pace,” and OpenAI CEO Sam Altman also expressed similar views. Several prominent AI researchers, including the chief scientist of OpenAI and the co-founder of Anthropic, published papers this week urging policymakers to study how the industry can build models that are capable of recursive self-improvement.
Both companies launched new models this month to compete for customers, although OpenAI said on Monday that it had delayed releasing a more powerful model.
Irving said, “What they call 'controlling the cutting edge rhythm' means 'don't speed up too much'. If you're doing something very dangerous, you should slow down. AI companies have exaggerated the extent to which this is simply a coordination problem. They can completely be stopped unilaterally.”
Former OpenAI governance researcher Daniel Kokotajlo said that since Hugging Face was hacked, many former colleagues have contacted him to privately express their concerns.
Kokotajlo is now the executive director of the research organization AI Futures Project. He said lab executives “convinced themselves that they are good people and that it would be worse if they stopped unilaterally.”