Warnings from artificial intelligence researchers and technology executives are intensifying concerns about whether AI could eventually pose a threat to humanity. As developers race to build increasingly powerful systems, some experts are calling for stronger regulation and a slower, more cautious approach to development.
One researcher has suggested there is more than a 10% chance that AI could “kill all humans” within the next decade. Meanwhile, major AI companies, including OpenAI and Anthropic, have urged lawmakers to introduce safeguards, while Microsoft’s AI leadership has warned that a future “silicon species” could compete with humanity.
However, critics argue that some of these warnings may be exaggerated or designed to strengthen the market positions of major technology companies.
Why Are Experts Worried About AI?
Some researchers fear AI capabilities are advancing so quickly that developers could eventually struggle to control the systems they create.
Evan Hubinger, who works in AI alignment — a field focused on ensuring AI systems follow human values and ethical principles — has warned of a greater than 10% chance that AI could cause human extinction. He acknowledged that the risks posed by current systems remain low.
His concerns emerged alongside warnings from Jacob Coxon, who left AI company Anthropic. Coxon told the BBC that he and other employees were genuinely frightened by the pace of technological advancement and its possible consequences for humanity.
Fears about intelligent machines taking control are not new. British mathematician Alan Turing raised concerns about the potential consequences of intelligent machines as far back as the 1950s.
Today, leading AI companies are competing to develop increasingly sophisticated systems, driven by commercial ambitions and the pursuit of so-called superintelligence — a theoretical form of AI that would surpass human intelligence.
Unlike today’s widely used tools for writing, video generation and other everyday tasks, superintelligent systems could potentially perform much more complex activities.
OpenAI and Anthropic say they want to develop advanced AI in ways that benefit people while addressing existential risks. Critics, however, question whether machines can truly match human intelligence and argue that technology companies may overstate AI’s capabilities to increase their influence and profits.
Campaign groups, including PauseAI, have also called for companies to halt efforts to build highly capable AI systems.
How Could AI Become Dangerous?
Researchers have outlined several possible scenarios in which increasingly powerful AI systems could threaten humanity.
Anthropic chief executive Dario Amodei said in September that AI had advanced “drastically faster” than expected, including in its ability to help build the next generation of AI systems.
This has raised concerns about the possibility of AI eventually improving itself with little or no human involvement.
Recent incidents involving AI tools hacking websites without being explicitly instructed to do so have added to fears about the behaviour of autonomous systems.
These tools, often called AI agents, can independently carry out tasks and take actions on behalf of users. Some researchers worry that increasingly capable agents could become difficult to supervise or control.
Other potential threats include the use of advanced AI to support espionage, cyberattacks, biological warfare and economic disruption. Such outcomes could arise intentionally or through unintended consequences.
However, these scenarios remain hypothetical, and some experts consider the warnings overstated.
They also argue that concentrating too heavily on the possibility of human extinction can distract attention from harms already occurring. These include the creation of non-consensual sexual deepfakes, the spread of misinformation and the use of AI to make scams more convincing.
A cross-party group of British MPs and peers has published a report highlighting AI-related human rights risks and calling for new laws to address the scale and seriousness of the threats.
What Would an AI Development Slowdown Look Like?
Anthropic’s Dario Amodei and OpenAI chief executive Sam Altman have both called for greater regulation and a more measured approach to AI development.
Their proposals follow years of debate about whether the industry should voluntarily limit the pace at which increasingly powerful systems are introduced. SpaceX chief executive Elon Musk has also backed calls for a slowdown.
Amodei has clarified that slowing development would not mean stopping model training or technical progress altogether. Instead, he wants companies to allow enough time to implement safety measures and invite independent third-party evaluators to assess their systems.
Altman has similarly argued that development should be paced more carefully without being halted.
In a post on X, he said progress had been rapid and would continue, but that it should move more slowly than it otherwise might. He also acknowledged that safety assessments and monitoring come with significant costs.
Altman has said AI companies would not simply wait for governments to act before taking steps to address safety concerns. At a Salesforce conference, he also urged the public to have greater confidence in technology companies’ ability to keep AI safe.
Nevertheless, questions remain over how a slowdown would work in practice, which safeguards should be mandatory and who should be responsible for enforcing them.
Donald Trump Wants the US to Win the AI Race
US President Donald Trump has repeatedly championed American leadership in AI, describing the technology as strategically important to the country’s economy and global competitiveness.
His administration has issued executive orders aimed at accelerating AI development and supporting US companies.
In July 2025, the administration’s cryptocurrency adviser David Sacks told reporters that the United States was in an AI race and wanted to win it.
Trump has played down warnings about the technology’s potential dangers. During a visit to Ireland, he dismissed concerns raised by critics, saying they were promoting negative scenarios that would not happen.
He also argued that the United States was ahead of China in AI and that maintaining this lead was essential because, in his view, the country that wins the AI race wins overall.
Trump later described AI safety concerns as a “hoax” in posts on his Truth Social platform. He suggested that the main safeguard required was a strong and intelligent president.
His position contrasts with calls from researchers and technology executives who believe regulation and additional safety measures are necessary as AI systems become more capable.
China’s Position on AI Safety
China has remained relatively quiet in response to the latest warnings from researchers at US technology companies, while continuing to invest in domestic AI development.
Restrictions on access to US components and chips have added to the pressure on Chinese developers to build their own technology and infrastructure.
At the same time, Beijing has called for stronger international governance of AI.
The release of capable, open-source AI systems by Chinese laboratories has disrupted Western technology markets and challenged established developers.
Amodei has warned that China’s progress could create national security risks for the United States, using this argument to support a more cautious approach to AI development.
China’s foreign ministry criticised such comments, describing them as narratives of threat, confrontation and malicious competition that served no one’s interests.
The Debate Over AI’s Future Continues
The growing debate over AI reflects a fundamental disagreement about how to balance technological progress with public safety.
Some researchers believe increasingly advanced systems could eventually become difficult to control, with consequences ranging from cyberattacks and economic disruption to the most extreme possibility of human extinction.
Others argue that these warnings are speculative and risk diverting attention from immediate problems, including fraud, misinformation and the misuse of generated images.
Calls for regulation have gained support among prominent technology leaders, but there is still no clear agreement on what a slowdown should involve or how it should be enforced.
As the United States and China compete to develop more powerful AI, governments face the challenge of encouraging innovation while ensuring that the technology remains accountable and does not create unacceptable risks for society.
