Former Anthropic Researcher Warns Rapid AI Progress Could Pose Serious Risks

AI safety has become one of the biggest concerns surrounding the rapid development of artificial intelligence. While companies continue to build increasingly capable AI systems, some researchers and executives are warning that progress could be moving faster than society’s ability to manage the risks.

Jacob Coxon, a former researcher who worked at both OpenAI and Anthropic, has become one of the latest voices to publicly raise concerns about the pace of AI development. Speaking to the BBC after leaving Anthropic, Coxon said some people working inside the AI industry are deeply worried about where the current race could lead.

His comments come as Anthropic CEO Dario Amodei and other prominent technology figures call for stronger safeguards, regulation, and greater coordination around the development of increasingly powerful AI systems.

Why AI Researchers Are Becoming More Concerned

AI systems have advanced rapidly in recent years. Models that once struggled with relatively simple tasks can now write software, analyze large amounts of information, operate tools, and perform increasingly complex reasoning.

That progress has created enormous opportunities, but it has also raised questions about whether developers can reliably predict and control the behavior of future systems.

Coxon argued that the speed of development itself is part of the problem. In his view, companies and researchers can find themselves competing in a race where slowing down individually may appear difficult if competitors continue moving forward.

He also suggested that international coordination could become important, particularly because AI development is not limited to a single country or group of companies.

The concern is not simply that AI could make mistakes. More serious scenarios involve increasingly autonomous systems gaining capabilities that their developers cannot fully understand or control.

Anthropic’s CEO Has Also Called for a Slower Approach

Coxon’s warnings come shortly after Anthropic CEO Dario Amodei published an essay arguing that the development of increasingly powerful AI should be slowed enough to give governments and companies time to address its risks.

Amodei has not argued that AI development should stop altogether. Instead, he has emphasized the need for safeguards and time to establish systems capable of managing the technology’s potential dangers.

That position is notable because Anthropic is one of the companies directly involved in building advanced AI models.

The debate over AI safety is no longer limited to researchers studying hypothetical future systems.

Amodei’s proposal has also received support from major figures elsewhere in the industry. OpenAI CEO Sam Altman and xAI CEO Elon Musk have both indicated support for stronger regulation, independent monitoring, and efforts to prevent an uncontrolled race toward more powerful AI systems.

What Could an AI “Apocalypse” Actually Look Like?

One of the most difficult questions surrounding AI safety is also one of the least certain: what would an extreme AI failure actually look like?

There is no single agreed-upon scenario.

Some researchers worry about AI systems being used to automate cyberattacks, manipulate information, or operate at a scale that would be difficult for humans to contain. Others are concerned about future systems becoming increasingly autonomous and pursuing objectives in ways their creators did not anticipate.

Coxon pointed to a scenario in which large numbers of AI-powered systems could work together and effectively act as a highly capable network of automated agents.

These scenarios remain speculative. There is no evidence that today’s publicly available AI systems are capable of causing the kind of catastrophic outcome described by the most extreme warnings.

That distinction is important. AI safety discussions involve both demonstrated risks and predictions about technologies that do not yet exist.

AI Safety Is Already a Real Problem

The debate does not depend entirely on hypothetical future systems.

Today’s AI models can already generate convincing misinformation, produce incorrect information with confidence, assist with malicious activities, and create software or content that can be difficult to distinguish from human work.

The challenge becomes greater as models gain access to external tools and can complete longer sequences of tasks with less human involvement.

This is one reason researchers are increasingly focused on AI alignment — the broader challenge of making sure AI systems behave according to human intentions and remain within appropriate safety boundaries.

Anthropic has invested heavily in this area, including research into how AI models behave internally and methods for evaluating potentially dangerous capabilities.

The company has also said it believes advanced AI development should eventually be governed by a lawful and verifiable framework that allows companies and governments to coordinate on the release of powerful models.

Silicon Valley Does Not Agree on the Level of Risk

Not everyone in the technology industry accepts the most severe predictions about AI.

Hugging Face CEO Clement Delangue publicly questioned the weight given to Coxon’s warnings, arguing that his background did not necessarily make him an authority on the probability of human extinction.

Nvidia CEO Jensen Huang has also previously rejected predictions that AI development will inevitably lead to humanity’s destruction, describing such claims as unrealistic.

This disagreement highlights one of the central problems in the AI safety debate: there is no consensus about how likely catastrophic outcomes are.

Some researchers believe the possibility deserves urgent attention even if the probability is relatively low. Others argue that focusing too heavily on hypothetical extinction scenarios could distract from more immediate problems such as misinformation, cybersecurity, labor disruption, and misuse of AI systems.

Some AI Researchers Say the Risks Are Worth Taking Seriously

Coxon is not alone in expressing concern.

Anthropic scientist Evan Hubinger has also publicly argued that advanced AI could potentially create an existential threat to humanity. Other researchers and executives at major AI organizations have issued similar warnings over the past several years.

At the same time, many of these same researchers acknowledge that AI could provide enormous benefits.

AI systems could accelerate scientific research, assist with drug discovery, improve productivity, and help researchers solve problems that are currently extremely difficult.

That creates a complicated situation for the industry. The goal is not necessarily to prevent AI from becoming more capable, but to ensure that safety measures develop quickly enough to keep pace with those capabilities.

The AI Race Creates a Difficult Safety Problem

The biggest challenge may be that AI development is becoming increasingly competitive.

If one company slows down to conduct additional safety testing while competitors continue developing more powerful systems, there can be strong commercial and strategic pressure to catch up.

The same problem exists between countries.

That is why calls for coordination have become an increasingly important part of the AI safety debate. Without some degree of international cooperation, companies and governments could face incentives to prioritize speed over caution.

Creating effective rules will not be easy. AI technology changes quickly, and regulations designed for one generation of systems could become outdated as capabilities evolve.

AI Development Is Entering a Critical Period

The debate surrounding Coxon’s departure reflects a much larger question facing the technology industry.

AI development is no longer moving at a pace that can be discussed purely as a software trend. Advanced models are becoming increasingly capable, companies are investing enormous amounts of money in them, and governments are competing to establish leadership in the technology.

That makes questions about safety, oversight, and accountability increasingly important.

There is still significant uncertainty about what future AI systems will be capable of and whether the most extreme scenarios will ever occur. But uncertainty does not make the risks irrelevant.

The more capable AI becomes, the more important it will be for researchers, companies, and governments to understand its limitations before deploying systems that can operate with increasing independence.

The future of AI may ultimately depend on finding a balance between innovation and restraint. Building more powerful systems may bring extraordinary benefits, but ensuring those systems remain controllable could become one of the defining technological challenges of the next decade.