Marina Favaro and Jack Clark, in a very long article on Anthropic’s website:
The technical trends discussed in this piece suggest that AI systems are going to become much more capable in coming years. These trends have huge implications. AI that can build itself would be a major development in the history of technology—one that could bring enormous good for the world in science, healthcare, and beyond. But full recursive self-improvement also might increase the risks of humans losing control over AI systems. If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important.
Then, scrolling waaaaaaay down, they lay out three possible futures that could play out as AI systems become better and faster than humans, especially at coding tasks:
- The trend stalls, but today’s AI capabilities are widely diffused.
- AI labs continue to see compounding efficiency gains.
- AI systems themselves become capable of full recursive self-improvement, and begin building their successors.
They continue:
We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret. If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.
A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions. It would also require that each can verify that the others have actually stopped. Due to the unique characteristics of AI systems, the detectability (a lower standard than verifiability) element of this arms control problem is much more challenging than with other technologies. Training runs are far easier to conceal than missile silos, their inputs are general-purpose, and the incentive to defect quietly is enormous, because whoever continues while others pause could inherit the lead. A credible pause also has to specify what triggers it, what lifts it, and who adjudicates.
None of this is necessarily impossible in principle—the world has built verification regimes for other complex technologies (e.g., the Intermediate-Range Nuclear Forces Treaty)—but those regimes took decades to build both the infrastructure and the trust. We don’t have that long. A unilateral pause by one lab, by contrast, is achievable immediately, but accomplishes much less: it would change who the front-runner is, but it would not create the wider deliberative process that is currently missing.
The article closes with a promise to “organize conversations where policymakers, researchers, civil society, and other AI companies can help answer some of the questions this piece raises, especially around full recursive self-improvement and how to create better options for coordination and deliberation” in “the coming months.”
Meanwhile, Anthropic is barreling toward an IPO , even if there’s no clear return. Marina Temkin, Techcrunch:
Private investors have been falling over themselves to get a piece of Anthropic, given the AI model maker is growing at a dizzying pace. Multiple investors told TechCrunch that the company’s $65 billion fundraise at a $965 billion valuation, announced last week, was greatly oversubscribed. Now, with that private demand still strong, Anthropic has revealed that it’s taking steps toward a public listing by filing confidentially for an IPO.
Co-founder Daniela Amodei, speaking at the Bloomberg Tech conference on Thursday, said the decision comes down to capital. “It’s a really big upfront cost to train the models and to serve inference on them,” she said. “My guess is that over time, the sort of core set of companies that are working to advance the frontier are just going to need access to capital, and I think the public market is very well suited to that.”
Temkin continues:
That isn’t fazing Amodei, who believes businesses are still early in figuring out how to deploy AI effectively.
“The use cases today, I expect will continue to be the primary driver of efficiency or creativity, whether that’s coding, financial services, legal, [or] health care,” she said. “But as the business community gets more familiar with the tools, we’re all going to learn together. My hope is that over time it’ll be more incorporated into the day-to-day of how humans do our work, and there will actually be a lot more value realized.”
On one hand, Anthropic is worried about AI systems growing out of our control and wants everyone to take a breath. On the other, it wants a bunch of cash from investors to keep pushing the envelope.
Got it.
