Anthropic co-founder Jack Clark said AI agents might soon be able to build and train models themselves and, if that happens, humans could lose control over AI systems.
Anthropic co-founder Jack Clark wants the AI industry to pump the brakes before the technology starts further developing itself without human input.
ADVERTISEMENT
ADVERTISEMENT
Speaking to the BBC, Clark said 80% of Anthropic’s coding work is already being done by its AI Claude, and that it could go up to 100% in a couple of years. However, he said “it’s a choice” whether AI companies let it get that far without stopping it.
“We think this is a topic that the world should be talking more about,” Clark said. “The AI industry right now has a gas pedal, but it doesn’t have a brake pedal in the car, and we want to do some of the work to build that pedal.”
This process is called “recursive self-improvement,” where an AI is able to improve itself without human input, according to Anthropic in a related blog post from Thursday night.
In a recursive model, AI agents, the autonomous workers built by a chatbot, could “become capable enough to build and train models themselves,” so Claude “could be continuously improved by Claude,” Anthropic said.
While recursive AI could bring some good to the fields of science and healthcare, Anthropic warns that it might mean increasing “the risks of humans losing control over AI systems.”
“If systems are capable of fully building their own successors, the ways we secure them, monitor them and shape their behaviour all grow much more important,” the blog post reads.
There is evidence within Anthropic’s own model that recursion is coming sooner rather than later. It points to the fact that code correction rates by their staff have been falling steadily for the last year, which means there are fewer errors in what Claude is producing.
Claude is also able to run its own research experiments when given an open-ended question, such as “Can a weaker model supervise a stronger one?” and come up with its own solutions without human input.
“The evidence suggests that the human role is narrowing at each step in the AI development process,” the blog reads.
Anthropic said its institute will conduct research to build a system to check whether developers have actually stopped or slowed down the move towards recursive AI, it said.
However, a real slowdown would require “multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions.”
