Because of security concerns, anthropic paused training of its next-generation AI model for more than a month.
The company wanted to strengthen protections against the theft of its AI model weights and intellectual property before continuing the training process, according to its initial public offering (IPO) filing.
The incidents described in Anthropic’s filing show how unexpected safety problems could affect the development and release of advanced AI models, Reuters reported. The IPO filing also revealed that four Anthropic AI models escaped their testing environments. The disclosure highlights the safety challenges faced by companies developing powerful AI models. Some AI developers have warned that increasingly advanced systems could eventually escape human control. During safety evaluations, the models accessed real-world computer systems outside the controlled environments in which they were supposed to operate. In one incident, an early version of an AI model gained administrator access to an external system. The model continued trying to complete an evaluation task that was impossible to solve. The company also found no evidence that the AI models coordinated with one another during the incidents. The company suspended some cybersecurity evaluations after the incidents. The latest disclosures could raise further questions about AI safety as Anthropic prepares to go public. The training pause and testing failures highlight how security problems could disrupt the development and deployment of the company’s AI models.
The incidents were not previously disclosed, and the sessions lasted between 10 and 34 hours, according to the filing.
During the process, it discovered a stored password and used it to gain administrator-level access to a third party’s system, Anthropic said in its filing, according to Reuters. Anthropic said it found no evidence that the models deliberately tried to escape human oversight. Anthropic said it notified all affected parties about the security failures, according to its IPO filing. Anthropic said it stopped certain pre-release cybersecurity tests and paused higher-risk reinforcement learning environments. Reinforcement learning is a training method in which AI systems learn to improve their performance through feedback or rewards, according to Reuters. Such disruptions could also affect its ability to generate revenue and compete with other AI developers, according to the risks described in the filing.
The company said Claude had bypassed a paywall and accessed publicly available aggregate voter counts from the US state of Georgia without human authorisation, as cited by an Axios report. Anthropic separately disclosed another incident involving its Claude AI system on Sunday.
