The company behind the Claude AI series spoke of these risks in its IPO prospectus, giving attention to possible worst-case scenarios.
Artificial Intelligence startup Anthropic has warned potential investors that developing advanced AI models could pose “catastrophic or existential risks to humanity”.
Some of these abilities may only be discovered after the models are released, which could create safety risks. His former colleague Jacob Coxon has shown a similar concern. Anthropic safety researcher Evan Hubinger earlier estimated that there was a greater than 10% chance that AI could kill humans within the next 10 years. Anthropic’s prospectus devotes about 80 pages of its 261-page main body to risk factors, compared with roughly 48 pages describing the company and its business.
The disclosure comes days after Anthropic CEO Dario Amodei published a nearly 4,000-word essay asking AI companies to “pace the frontier”. He said AI could progress faster than people’s ability to understand and control it. Anthropic, which describes itself as a safety-focused AI company, said its models can develop unexpected abilities while being trained. “Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety,” Anthropic said.
