The Warning Comes From Inside the Industry
Within the span of a few days, the leaders of two of the most powerful AI companies in the world publicly agreed that the technology they are building poses serious risks to human life. Anthropic CEO Dario Amodei argued that AI progress should be slowed. OpenAI CEO Sam Altman posted on X in response: “I agree with Dario that we need to pace the frontier.” These were not outsider critics or government regulators speaking – they were the people signing the checks and shipping the products.
The alarm grew louder when AI researcher Jacob Coxon announced his departure from Anthropic, having previously worked at OpenAI as well, stating that neither company was acting responsibly. “The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote on X. Anthropic researcher Evan Hubinger publicly backed that view: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” Among the specific pathways they fear: AI-assisted bioweapons.
What an AI-Designed Bioweapon Could Look Like
The threat is not abstract. A bioweapon could take the form of a highly lethal virus engineered to target people based on their genetic profile. It could be a fungus designed to collapse a staple crop and trigger mass food insecurity. It might be a tasteless, odorless toxin introduced undetected into a municipal water supply. What has shifted dramatically is how much easier AI makes it to move from concept to candidate agent.
In 2022, researchers at Collaborations Pharmaceuticals discovered this problem by accident. They had built an AI molecule generator to identify potential drug compounds for human disease. When they redirected the model toward harmful applications, it generated 40,000 molecules with the potential to serve as chemical warfare agents in under six hours. Some of those molecules were modeled to be more toxic than known nerve agents. “Without being overly alarmist, this should serve as a wake-up call for our colleagues in the ‘AI in drug discovery’ community,” the authors wrote at the time. David Magnus, a professor of medicine and biomedical ethics at Stanford University who had been evaluating misuse risks in medical science and biotechnology since the late 1990s, said the findings shook him. “That was very scary to me,” he says. “Of course, everything since then has just sort of blown up.”
Modern large language models compound the problem significantly. Dunja Sabra, a biosecurity researcher at the University of Hamburg, notes that these systems have been trained on the knowledge and experience of “almost every scientist who ever lived on this planet.” Anyone with internet access can now query those models for scientific instructions, experimental protocols, and video training on how to carry out lab procedures. That capability, layered on top of a “DIY biology” movement that has already made gene editing and synthetic biology tools accessible enough for home lab setups, creates a situation where the barrier between curiosity and catastrophe has narrowed considerably. “The chances are that someone determined would succeed eventually,” Sabra says.
The Safeguards That Exist – and Where They Fall Short
There are defenses in place, and they are not trivial. Companies that supply DNA building blocks typically screen orders for suspicious requests before fulfilling them. Researchers working on sensitive biology run their work through “red-teaming,” where independent scientists try to identify potential misuse vectors, and “blue-teaming,” where another group develops possible countermeasures. AI companies have also attempted to modify their models to withhold scientific information that could be weaponized.
None of it holds completely. Anthropic acknowledged in a report published last week that users had already attempted to use its models to explore making the chikungunya virus more transmissible, engineer a strain of bird flu more dangerous to humans, and construct an “atlas of venom toxin peptides,” among other documented attempts. The company’s own report is the admission that the filters are not a ceiling – they are a floor that people are actively trying to dig beneath.
A Race Between Screening and Circumvention
Magnus frames the current situation as a continuous back-and-forth rather than a problem with a fixed solution. “We have to build better surveillance and screening tools,” he says, while acknowledging a hard structural constraint: “AI is really good at figuring out ways aro-” the sentence cuts off, but the implication is clear enough from everything preceding it. Every safeguard introduced creates a new puzzle for a sufficiently motivated actor to solve, and AI is exceptionally good at solving puzzles.
The biotech industry built its modern infrastructure – DNA synthesis companies, institutional review, biosafety levels – over decades, largely in response to incremental advances in what was possible. The current moment is not incremental. The 40,000 molecules generated in under six hours by a drug-discovery model that was never designed for harm represent a kind of productivity that no screening committee was built to process at scale.
What makes this particularly difficult to regulate is that the same model architecture powering a bioweapon query is also accelerating cancer drug discovery, vaccine development, and antibiotic research. You cannot surgically remove the dangerous half without degrading the beneficial half – a tension that neither Amodei nor Altman, for all their public agreement on slowing down, has publicly resolved in concrete policy terms.
Sabra’s assessment – that a determined actor would eventually succeed – does not come with a timeline, a nationality, or an institutional affiliation attached to it. It comes with the quiet, specific weight of someone who has spent years studying what happens when dangerous knowledge becomes broadly accessible. That is where the conversation currently sits: not at the level of “if,” but at the question of how much time the back-and-forth actually buys.
