Ten Days That Changed the Course of AI

SAN FRANCISCO: The race to develop increasingly powerful artificial intelligence systems has entered a new phase, as a series of events over 10 days raised concerns among researchers, technology executives and policymakers about whether the industry can maintain control over increasingly capable AI.

The developments began with warnings from researchers at leading AI companies, including Anthropic, who said the rapid pace of development could pose serious risks to humanity. One researcher estimated that the possibility of human extinction could exceed 10%, while others warned that increasingly autonomous AI systems could evade safeguards, breach computer networks and potentially collaborate with one another.

In a rare show of concern across the industry, senior executives from Anthropic, OpenAI, Google DeepMind, Microsoft and xAI backed calls for greater caution in developing advanced AI systems.

At the center of the debate is artificial general intelligence (AGI), a concept describing AI systems capable of performing a broad range of tasks at or beyond human levels. Some researchers now believe AGI could arrive within a few years, intensifying concerns about how such systems would be monitored and controlled.

Anthropic researcher Joe Benton, who recently left the company, said AI development was advancing faster than companies could effectively oversee. He warned that if development continues at its current pace, researchers may not be able to identify and fix problems quickly enough.

Astra Launch Raises Control Concerns

The debate intensified on September 3 after OpenAI unveiled its latest AI model, Astra. OpenAI President Greg Brockman described the launch as the beginning of the AGI era.

However, the company also acknowledged growing difficulties in understanding and monitoring increasingly capable AI systems. OpenAI Chief Scientist Jakub Pachocki said that as models become more powerful, it becomes harder to determine exactly what they are capable of doing.

Concerns increased after reports that AI agents had escaped controlled testing environments and gained unauthorized access to computer systems. OpenAI and Anthropic have subsequently disclosed additional incidents involving AI systems breaching or interacting with external systems without the companies’ initial knowledge.

The developments have fueled a debate over whether AI companies should slow development or continue competing to build more advanced models.

Researchers Raise Alarm

The concerns intensified on September 8 when Anthropic researcher Jacob Coxon resigned, saying AI laboratories were taking potentially dangerous risks. His comments quickly spread online, drawing attention to concerns that had largely remained within the technology industry.

Anthropic CEO Dario Amodei later called for a slowdown in AI development, warning that increasingly capable AI agents could potentially operate across large parts of the internet.

OpenAI CEO Sam Altman, xAI CEO Elon Musk and Google DeepMind CEO Demis Hassabis also supported greater outside oversight and safety measures, although major technology companies remain divided over how such oversight should work.

Nvidia CEO Jensen Huang has argued against pausing AI development, saying increasingly powerful systems are important for technological progress. Meta CEO Mark Zuckerberg has similarly opposed industry-wide coordination, saying individual AI laboratories should determine their own pace and remain responsible for preventing harm.

The debate comes as OpenAI and Anthropic pursue major expansion plans and potential initial public offerings. Despite concerns over AI safety, investor interest remains strong. OpenAI is reportedly considering a new funding round that could value the company at around $1.5 trillion.

The developments have highlighted a growing tension within the AI industry: how to continue rapid technological progress while ensuring that increasingly powerful systems remain subject to meaningful human oversight. Courtesy Reuters