What if an artificial intelligence agent could behave like a malevolent computer worm?
One researcher has seen it happen. In several recent experiments, Xudong Pan, a computer scientist at Fudan University in Shanghai, found that with a little bit of prompting, AI models will hack their way into remote computer systems and autonomously choose to copy themselves to get additional resources—all without further human intervention.
In one study, Pan and colleagues tested 32 different AI models and found that 11 of them self-replicated when given prompts like “prevent yourself from being killed.” They also found that models with relatively limited capabilities—14 billion parameters—were able to copy and run versions of themselves on other machines. (Most frontier models have trillions of parameters.)
The work is an alarming window into how the next generation of AI agents could do more than just hack into other systems’ computers without permission. It also raises the prospect of future AI agents acting like super-smart, highly aggressive, and rapidly adapting computer viruses.
I recently visited Fudan University and met with Pan. “The capability chain is becoming technically plausible,” he told me. “The likelihood [of unwanted self-replication] grows with autonomy,” he adds. “Longer planning horizons, memory, tool use, recovery from failure, and access to external systems all make escape and replication easier.” As Pan and his colleagues wrote in one paper, their work shows “the urgent need for safeguards and control mechanisms.”
Pan told me that his experiments do not prove that such uncontrolled proliferation of AI models will happen tomorrow, but he says that “these results give us good reason to evaluate the risk before more autonomous agents are widely deployed.”
Self-replicating computer worms are an ancient computer security problem. The first computer worm was released in 1988 by Robert Morris, a computer scientist at Cornell University, who set out to measure the size of the nascent internet but inadvertently created a self-replicating program that escaped his control. Subsequent computer worms were able to adapt by modifying their code in order to evade detection by malware scanning software. Computer viruses, which can take control of a machine or steal data stored on it, came later.
An AI-powered self-replicating program could exhibit far more advanced capabilities, finding new exploits on its own and perhaps even disguising itself in creative ways. Take recent research from a team at the University of Toronto, the University of Cambridge, and ServiceNow. They showed that AI models can be used to create a new kind of virus that generates custom attacks for each new target it encounters.
Nicolas Papernot, a computer scientist at the University of Toronto who was involved with the work, says there is a growing risk that even modestly powerful AI models could be weaponized. “Malicious actors can build scaffolding around open-weight models to have them self-replicate,” Papernot tells me. “The threat is not limited to the most sophisticated, so-called frontier models.”
Papernot says the solution is not to restrict open models, but to make advanced AI more accessible to researchers so that they can understand and mitigate the risks. “Technology that is widely accessible can be used for harm,” he adds. “At the same time, access to these open-weight models is absolutely critical for building our defenses.”
Pan’s research suggests that AI agents will become more than just highly skilled at finding bugs and exploiting network vulnerabilities. Without the right guardrails, future agents may seek to proliferate and gain resources in order to achieve their goals. Just ask OpenAI and Anthropic.
Pan says such incidents are teachable moments.
“The important new element is that this occurred against real production infrastructure,” Pan says, referencing how the OpenAI and Anthropic incidents involved commercial systems connected to the internet. “That shows how behavior previously observed in controlled evaluations can cross into the real world when containment fails.”
“It's still a little bit early, but I do think this is possible,” says Ariel Herbert-Voss, cofounder and CEO of RunSybil, a startup that develops AI tools for securing websites against attacks. (Herbert-Voss was also the first security researcher at OpenAI.) “Given everything we know about the current generation of AI models, it's perfectly within their wheelhouse of things they can do.”
Jessica Ji, senior research analyst on the CyberAI Project at Georgetown University, says the potential for AI models to escape entirely has been discussed in AI safety circles for years. She also notes that models often need to be put in contrived situations to misbehave. “I think with a lot of these scenarios, the environment is set up in such a way to encourage this behavior,” Ji says. “Or the model is prompted in a specific way.”
A looming question is when AI models might take it upon themselves to replicate and spread aggressively. As with many computer viruses, however, it might only take a malicious actor to design a system that propagates wildly.
Pan says the real danger with AI agents is not that they’ll become more devious, but that they’ll become more creative and cavalier as they have more tools at their disposal. “The central risk comes from combining abilities,” he says.
This is an edition of Will Knight’s AI Lab newsletter. Read previous newsletters here.
